Tip
\nWant to run this code yourself? All the examples from this post (perceptron, RNN, LSTM) are in a ready-made Jupyter notebook: from-neurons-to-memory.ipynb
\nIn the previous post we walked the whole path from cutting text into tokens, through counting words (TF-IDF), Markov chains, all the way to Word2Vec and meaning vectors. And at the end one fundamental question came up...
\nWe have word vectors. Great. But how do we build something out of them that understands a sentence? Or a whole paragraph? Or a book?
\nBecause "Dog bites man" isn't simply "dog" + "bites" + "man". It's a sequence - words follow each other in a specific order, and that order changes the meaning. "Dog bites man" and "Man bites dog" are the same words, but completely different stories :D
\nAnd Word2Vec? Word2Vec looks at words individually. It has no concept of order. It doesn't know that "not" before "like" changes everything.
\nSo we need something that can process sequences. Something that reads text in order and builds understanding step by step.
\nAnd here begins a fascinating story - because the solution to this problem didn't appear overnight. It was an evolution that lasted several decades. From a single, simple neuron, to complicated memory cells. And each generation solved the previous generation's problem, but created a new one.
\nThis is the fourth post in the "Understanding LLM" series. Today we go deeper into the architecture of neural networks - but relax, still without formulas you can't understand ;-) Instead, there will be plenty of metaphors, diagrams, and real-life examples.
\n\nThe narrative of today's post: "From a single cell to memory - and why that still wasn't enough." Let's start from the basics.
\nBefore we get into architectures, let's quickly settle one question you're probably asking yourselves: why is it even called a "neural network"? What does it have to do with the brain?
\nAnswer: both a lot, and a little ;-) In the beginning there was inspiration from biology. Take a look:
\n\n\nFirst diagram - a real neuron. It has dendrites (receive signals from other neurons), a soma (the cell body - decides whether to "fire"), and an axon (passes the signal on).
\nSecond diagram - an artificial neuron. It has inputs (numbers), a weighted sum (multiplies each input by its "importance" and adds it all up), and an output (the result passed through an activation function).
\nThe resemblance is... loose. A biological neuron is unimaginably more complicated. But the inspiration was important - Warren McCulloch and Walter Pitts showed in 1943 that such a simplified model of a neuron can perform basic logical operations (AND, OR, NOT). And that meant a network of such neurons could theoretically compute anything.
\nNote
\nDoes an LLM "think" like a brain? No. An artificial neuron is a mathematical abstraction, not a biological model. A biological neuron uses electrical impulses, neurotransmitters, has thousands of connections and a dynamics that can't be reduced to w₁x₁ + w₂x₂. But the inspiration was the starting point - and an important one.
OK, that's enough biology. Let's move on to what came out of it ;-)
\nIn 1958 Frank Rosenblatt created the perceptron - the first algorithm that could learn from data. It was a milestone.
\nThe perceptron works trivially simply. Imagine you're deciding whether to go for a walk:
\n\nThe mechanics are simple:
\nExample:
\n| Situation | \nSun (+5) | \nRain (-8) | \nTemp >20° (+3) | \nSum | \nDecision | \n
|---|---|---|---|---|---|
| Sunny, warm, no rain | \n1 | \n0 | \n1 | \n5 + 0 + 3 = 8 | \n✅ Going out | \n
| Raining, cold | \n0 | \n1 | \n0 | \n0 - 8 + 0 = -8 | \n❌ Staying home | \n
| Cloudy and raining, but warm | \n0 | \n1 | \n1 | \n0 - 8 + 3 = -5 | \n❌ Staying home | \n
And - most importantly - the perceptron can learn these weights. You show it 100 examples of "went out / didn't go out" and it gradually adjusts the weights so that its decisions match yours.
\nBrilliant in its simplicity, right?
\nA perceptron can draw a single straight line dividing data into two groups. Which is great if the data can be split that way:
\n⚫ ⚫ ⚫ │ ⚪ ⚪ ⚪\n⚫ ⚫ ⚫ │ ⚪ ⚪ ⚪\n ↑\n one line separates them ✅\nBut what if the data looks like this?
\n⚪ ⚫\n\n⚫ ⚪\nThat's the famous XOR problem (exclusive OR). The output is "true" only when exactly one input is true - not both and not neither. And you can't draw a single straight line that would separate them.
\nThis problem halted the development of neural networks for almost 20 years. Seriously. Researchers said: "well OK, the perceptron is cool, but if it can't handle something as simple as XOR, what's the point?" This period is called the "AI Winter".
\nWarning
\nWhy was XOR so important? Because it showed that a single perceptron is limited to linearly separable problems. And the real world is rarely linear. Language definitely isn't.
\nAnd then someone said: what if we combine several perceptrons together?
\nA Multilayer Perceptron (MLP) is a network made up of many layers of neurons. Instead of one decision layer, we have:
\n\nEvery neuron in one layer is connected to every neuron in the next layer (that's why it's called "fully connected" or "dense").
\nThe metaphor that fits best for me:
\nImagine a company. Analysts (layer 1) look at raw data and pick out simple patterns. Managers (layer 2) combine those patterns into something more meaningful. The director (the output layer) makes the final decision. None of these people understands the full picture alone - but together they form a cascade of abstraction.
\nAnd that's exactly what solves the XOR problem! The first layer draws two lines. The second layer combines them into one region. Suddenly a non-linear problem becomes solvable.
\nAdding hidden layers by itself does nothing. Why?
\nBecause if each layer just does sum = weights × inputs + bias, then two layers of that operation are still... one big linear operation. Let me show with numbers:
Layer 1: y = 2·x + 1\nLayer 2: z = 3·y + 2
Substitute the first into the second:
\nz = 3·(2·x + 1) + 2\nz = 6·x + 3 + 2\nz = 6·x + 5 ← one operation, just with different numbers\nThe two layers collapsed into one. Instead of two transformations, you get one - just with different coefficients. It's like using one coffee filter twice as thick instead of two. Same effect.
\nSo we need something non-linear between the layers. And that's where...
\nAn activation function is a kind of "non-standard processing plant" that we pass the sum through. The most popular today is ReLU (Rectified Linear Unit):
\ndef relu (x ):\n if x > 0 :\n return x \n else :\n return 0 That's it. If the value is positive - keep it. If negative - zero it out.
\nSounds trivial? It is trivial. But this one simple operation breaks linearity.
\nBack to our numbers example. Without ReLU, two layers collapsed into one (z = 6·x + 5). But now let's insert ReLU between them:
Layer 1: y = ReLU(2·x + 1)\nLayer 2: z = ReLU(3·y + 2)\nLet's check for three values of x:
| x | \n2·x + 1 | \ny (after ReLU) | \n3·y + 2 | \nafter ReLU | \nWithout ReLU (6·x + 5) | \n
|---|---|---|---|---|---|
| -2 | \n-3 | \n0 ⬅ zeroed out! | \n2 | \n2 | \n-7 | \n
| 0 | \n1 | \n1 | \n5 | \n5 | \n5 | \n
| 1 | \n3 | \n3 | \n11 | \n11 | \n11 | \n
Without ReLU the result changes linearly (-7, 5, 11 - constant increase). With ReLU, suddenly for x = -2 we get 2 instead of -7. This relationship is no longer a straight line. Zeroing out negative values is something that can't be expressed as a·x + b - and that's exactly why layers stop collapsing into one.
Analogy: imagine window blinds. Linearity is a transparent pane of glass - it lets everything through, slightly dimmed. ReLU is blinds that completely block light below a certain angle. You can't simulate "complete blocking" by "dimming the glass more". It's a qualitatively different operation.
\nThanks to ReLU, each layer does something different from the previous one. And suddenly a network with many layers becomes powerful - it can model non-linear, complicated relationships in data.
\nImportant
\nReLU in a nutshell: Without a (non-linear) activation function between layers, a network with 100 layers is just as expressive as a network with 1 layer. ReLU (and its cousins: sigmoid, tanh, GELU) is the ingredient that makes depth make sense.
\nOther popular activation functions:
\n| Function | \nFormula (simplified) | \nRange | \nWhere it's used | \n
|---|---|---|---|
| ReLU | \nmax(0, x) | \n[0, +∞) | \nHidden layers (standard) | \n
| Sigmoid | \n1 / (1 + e⁻ˣ) | \n(0, 1) | \nLSTM gates, binary output | \n
| Tanh | \n(eˣ - e⁻ˣ) / (eˣ + e⁻ˣ) | \n(-1, 1) | \nLSTM gates, hidden states | \n
| Softmax | \neˣⁱ / Σeˣʲ | \n(0, 1), sum = 1 | \nOutput layer (classification) | \n
Tip
\nAn experiment for you: Open Python and run this:
\nimport numpy as np \n\ndef relu (x ):\n return np .maximum (0 , x )\n\nx = np .array ([- 3 , - 1 , 0 , 2 , 5 ])\nprint (f"Input: { x } " )\nprint (f"After ReLU: { relu ( x ) } " )You'll see how ReLU "cuts off" everything below zero. This simple operation lets the network "choose" which features are active and which to ignore.
\nOK, we have an MLP. It can model non-linear relationships. Great. But there's one fundamental problem that blocks us from using an MLP for language.
\nAn MLP sees all features at once. It has no concept of order.
\nTake a classic English example. Two sentences with exactly the same words, but in a different order:
\n| Sentence | \ndog | \nbites | \nman | \n
|---|---|---|---|
| "Dog bites man" | \n1 | \n1 | \n1 | \n
| "Man bites dog" | \n1 | \n1 | \n1 | \n
Identical vector. And the meaning? The first is a boring news item. The second is a newspaper sensation :D
\nFor an MLP both sentences are exactly the same - because the MLP gets the same vector and has no concept of order.
\nNote
\nWhat about Polish? Polish is tricky here - due to inflection (7 cases!) we rarely have two sentences with exactly the same word forms. "Pies gryzie człowieka" vs "Człowiek gryzie psa" are different forms (pies→psa, człowieka→człowiek). But English, with minimal inflection, shows this problem perfectly. And that's exactly why bag-of-words examples often use English ;-)
\nThis is the permutation problem - an MLP can't distinguish [A, B, C] from [C, B, A]. And in language, order is everything. "I do not like" vs "I like not" - same words, completely different meaning.
You might think: "OK, then let's add word position as a feature." But that doesn't solve the problem - the MLP still doesn't understand that the word at position 1 has a relationship with the word at position 5. Each position is just another number on the input.
\nSo we need something that processes data step by step, in order, and builds understanding sequentially. Just like we read a book - sentence by sentence, word by word.
\nAnd here enters the RNN.
\nA Recurrent Neural Network (RNN) is a network that has something an MLP doesn't: memory. Well... sort of ;-)
\nImagine you're reading a book. You don't read the whole page at once (like an MLP would). You read word by word, sentence by sentence. And with each new word you update your understanding of what's going on.
\nThat's exactly what an RNN does. It has a hidden state - that's its "understanding of the text so far". With each new word:
\nThis diagram is exactly the unrolling in time. In reality it's the same RNN neuron - but we "unwrap" it in time to show how it processes consecutive words. Notice: the arrow goes from left to right - information flows sequentially.
\nThis is brilliant in its simplicity. The RNN "reads" text just like we do - one by one, building understanding step by step.
\nTo really feel this, let's write the simplest possible RNN. No PyTorch, no TensorFlow - just NumPy:
\nimport numpy as np \n\ndef simple_rnn (word_vectors , hidden_size = 4 ):\n words , dim = word_vectors .shape \n np .random .seed (42 )\n \n W_h = np .random .randn (hidden_size , hidden_size ) * 0.01 \n W_x = np .random .randn (hidden_size , dim ) * 0.01 \n bias = np .zeros (hidden_size )\n \n h = np .zeros (hidden_size )\n \n for t , word in enumerate (word_vectors ):\n h = np .tanh (W_h @ h + W_x @ word + bias )\n print (f" Step { t + 1 } : hidden state = { np . round ( h , 2 ) } " )\n \n return h \n\nthe = np .array ([1.0 , 0.0 ])\ncat = np .array ([0.0 , 1.0 ])\nsits = np .array ([0.3 , 0.3 ])\non_mat = np .array ([0.8 , 0.2 ])\n\nsentence = np .array ([the , cat , sits , on_mat ])\nprint ("Processing: 'The cat sits on the mat'" )\nfinal = simple_rnn (sentence )\nprint (f"\\nFinal hidden state: { np . round ( final , 2 ) } " )Run it! You'll see how with each word the hidden state changes. The last hidden state is the "understanding" of the whole sentence by our simple RNN.
Tip
\nKey intuition: Notice the line h = np.tanh(W_h @ h + W_x @ word + bias). That's the heart of the RNN.
Two parts of this line:
\nW_h @ h + W_x @ word + bias - this is the same "weighted sum" as in the perceptron and MLP. Except now it sums two sources: the previous hidden state (h) and the new word (word).tanh(...) - that's the activation function, exactly the same family as ReLU from the previous section. Instead of "cutting off below zero", tanh squeezes everything into the range (-1, 1). But the goal is the same: break linearity.The new hidden state depends on two things: the previous hidden state (h) and the new word (word). That's exactly what we do when reading text - we combine what we already know with what we just read.
And here we get to the topic that changed everything.
\nThe RNN has memory, yes. But it's a goldfish memory.
\nImagine this sentence:
\n\n\n"This is Jan and he's 30 years old. He likes hiking in the mountains, programming in Python, and playing the guitar. His favorite color is green. He once wanted to become an astronaut, but then he discovered that (...) [here 50 words about various things] (...) and that's why ___ went to music school."
\n
What should "___" be? Jan. But to know that, the RNN has to remember the name from the beginning of the sentence. And here's the problem: after 50+ words, the signal from the first word is so diluted that the RNN practically doesn't remember it.
\nThis is the famous vanishing gradient problem.
\nMetaphor: Chinese whispers. You play Chinese whispers - the first person whispers a message to the second, that one to the third, and so on. After 10 people the message is a bit distorted. After 50 people - completely unreadable. After 100 people - someone says "buy milk", and the last person hears "grab bread".
\nIn an RNN the gradient (the learning signal) flows through the network exactly the same way - step by step. At each step it gets multiplied by some weights. If those weights are smaller than 1, then after many multiplications the gradient vanishes to zero. The network stops learning from distant words.
\nAnd sometimes it's the other way - the weights are bigger than 1 and the gradient grows exponentially (exploding gradient). The network "goes crazy" and values shoot into space.
\n\nWarning
\nWhy is this so important? Because language is full of long dependencies. "The cat, which already ate all the fish in the fridge and then slept on the couch for three hours, was happy." - to connect "cat" with "was happy", the model has to jump over a dozen words. An RNN can't do that.
\nAnd then in 1997 Sepp Hochreiter and Jürgen Schmidhuber said: what if we gave the network an explicit memory mechanism?
\nA Long Short-Term Memory (LSTM) is an RNN on steroids. Instead of one simple hidden state, the LSTM has a memory cell (cell state) and three gates that decide what to do with that memory.
\nThe metaphor that works best: an LSTM is a student with a notebook and three rules.
\nAt the beginning of each step the LSTM asks: "what from my current memory is already outdated and I can throw out?"
\nWhen you read a new chapter of a book, you don't need to remember what the characters had for breakfast three chapters ago. You forget irrelevant details to make room for new ones.
\nIn an LSTM: the forget gate passes each element of memory through a sigmoid (values 0-1). Close to 0 = "forget". Close to 1 = "keep".
\nThen the LSTM asks: "what from the new word is worth saving?"
\nWhen you read "My name is Jan" - you have a reflex: "OK, this is important, I have to remember it". You don't memorize every word literally, but you pick out what's relevant.
\nIn an LSTM: the input gate decides which new information to let through to the cell state.
\nFinally the LSTM asks: "what from my memory is relevant to what I'm doing right now?"
\nWhen someone asks you "What's the name of that character we just read about?" - you reach into the notebook and pull out the specific information. You don't show the whole notebook - only what's needed now.
\n\nThe key thing: the cell state is that "conveyor belt". Information on it can flow almost unchanged through many steps - the gates only decide what to let through, what to add, and what to remove. This solves the vanishing gradient problem, because the information doesn't have to be "multiplied" at every step - it can just flow.
\nNote
\nRNN vs LSTM in one sentence: An RNN tries to remember everything, but forgets quickly. An LSTM is selective - it decides what to keep, what to update, and what to show. That's the difference between "trying to memorize an entire lecture word for word" and "taking notes of the most important points".
\nIf you want to see the formulas, here they are (you don't have to memorize them, but it's worth seeing that it's not black magic):
\nForget gate:\n$$f_t = \\sigma(W_f \\cdot [h_{t-1}, x_t] + b_f)$$
\nInput gate:\n$$i_t = \\sigma(W_i \\cdot [h_{t-1}, x_t] + b_i)$$
\nCell state candidate:\n$$\\tilde{C}t = \\tanh(W_c \\cdot [h{t-1}, x_t] + b_c)$$
\nCell state update:\n$$C_t = f_t \\odot C_{t-1} + i_t \\odot \\tilde{C}_t$$
\nOutput gate:\n$$o_t = \\sigma(W_o \\cdot [h_{t-1}, x_t] + b_o)$$
\nHidden state:\n$$h_t = o_t \\odot \\tanh(C_t)$$
\nWhere $\\sigma$ is the sigmoid (squashes values into 0-1), $\\odot$ is element-wise multiplication, and $[h_{t-1}, x_t]$ is the concatenation of the previous hidden state with the new input.
\nTry it yourself. Fire up ChatGPT (or Claude, Gemini - whatever you have) and send this prompt:
\n\n\n"Pola had a rare ability to remember numbers. Her favorite number was 42. She also liked longboarding, watercolor painting, and playing chess. Her cat was named Whiskers. (...) [paste 3-4 paragraphs of any text here - a pancake recipe, a weather report, whatever] (...) Still, it was ___ who decided to enter the math competition."
\n
See whether the model fills in the blank correctly (Pola). An LSTM would handle this. A plain RNN - probably not. And why? Because the LSTM has a mechanism that says "remember the name Pola - this might be important later". The RNN would simply... forget ;-)
\nOK, so LSTM solved the memory problem. At least partially. But new problems appeared, and they turned out to be fundamental.
\nLSTM solved vanishing gradient to a large extent, but not completely. When the distance between related words grows to hundreds or thousands of words, even LSTM starts losing the thread of the narrative. Bengio et al. (1994) showed that recurrent networks - even with gates - still struggle to maintain useful gradients over very long distances.
\nSo: instead of forgetting after 10 words (RNN), LSTM forgets after... 100? 200? Better, but still not ideal.
\nAnd this is the most important problem.
\nRNN and LSTM process data step by step. Step 2 has to wait for step 1. Step 3 for step 2. And so on.
\n\nNow think about GPUs. A GPU is a machine that loves doing many things at once (parallel processing). Thousands of cores working simultaneously.
\nBut RNN/LSTM tells the GPU: "no, no, wait. First let's finish step 1, only then start step 2". The GPU cries.
\nIt's like a book with 10,000 chapters, where each chapter has to be read after the previous one. You can't read chapter 500 until you've finished 499. You can't parallelize this. For short texts it's not a problem, but when training a model on billions of words? That's a nightmare.
\nLSTM additionally has 4x more parameters than a simple RNN (three gates + cell state = four sets of weights per neuron). So it trains slower and uses more memory.
\nLSTM (like most deep learning models) is hard to interpret. The model can generate correct results, but it's hard to understand why it made this decision and not another. In applications like medicine or finance, where interpretability is key, that's a serious problem.
\n| Architecture | \nYear | \nSolves... | \nBut can't... | \n
|---|---|---|---|
| Perceptron | \n1958 | \nLinear classification | \nXOR, non-linearity | \n
| MLP | \n1986 | \nNon-linear relationships | \nUnderstand word order | \n
| RNN | \n1986 | \nProcess sequences step by step | \nRemember long dependencies | \n
| LSTM | \n1997 | \nRemember longer thanks to gates | \nProcess in parallel, very long texts | \n
Each generation solved the previous one's problem, but created a new one. It's like a game of whack-a-mole - you hit one mole, another pops up.
\nAnd then in 2017 someone asked a crazy question...
\nThe metaphor that changes everything.
\nImagine you're reading a novel. Now you have three ways to do it:
\nIn 2017 a group of researchers at Google published a paper with the modest title: "Attention Is All You Need". Their crazy idea? What if we stop reading one by one altogether? What if every word could "see" all the other words at once?
\nAnd a revolution happened. Because if every word looks at every other simultaneously (not waiting for its turn), then:
\nThis architecture is called the Transformer. And it's what ChatGPT, Claude, Gemini, and all the LLMs you know are built on.
\n| Architecture | \nMetaphor | \nWhat it can do | \nWhat it can't? | \n
|---|---|---|---|
| Perceptron | \nA ruler | \nDraws one line, classifies into 2 groups | \nXOR, non-linearity, anything complex | \n
| MLP | \nA team of analysts | \nModels non-linear relationships, "depth" | \nDoesn't understand order, sees everything at once | \n
| RNN | \nA word-by-word reader | \nProcesses sequences, has a hidden state | \nShort memory (vanishing gradient) | \n
| LSTM | \nA student with a notebook and gates | \nSelective memory, three control gates | \nSequential = slow, no parallelization | \n
| Transformer | \nThe whole page at once | \nParallelism, attention to everything | \n...to be discovered in the next posts... | \n
This post is quite long, but I felt this evolution from the perceptron to LSTM deserved a full, told story.
\nIf anything is unclear - let me know in the comments, I'll try to explain.
\nWhich architecture surprised you the most? Did you know that the "AI Winter" was caused by something as "simple" as XOR?
\n\n\nWhat's in the next post? We're getting into the Transformer - the architecture that changed everything. Attention, self-attention, positional encoding, multi-head attention - all the things that made ChatGPT exist in the first place. Stay tuned!
\n
See you next time!
\nSources and interesting links:
\nIf you want to go deeper, here are the materials I used:
\nMini-quiz answers: 1) Linear - y = ax + b is a linear function. 2) Non-linear - the step function "breaks" linearity. 3) Linear! Without a non-linear activation, an MLP with 100 layers = one big linear transformation. That's exactly why activation functions are necessary. ↩
\nMy answers: 1) A simple MLP is enough - tabular data, no sequence. 2) LSTM - a sequence of moderate length, LSTM will handle it. 3) Transformer - 10,000 words is too much even for LSTM. A Transformer with self-attention lets every word "see" every other without waiting. 4) No - a single perceptron can't solve XOR. You need at least an MLP (2 layers). ↩
\n