From simple neurons to memory - the evolution of language models

In the previous post we walked the whole path from cutting text into tokens, through counting words (TF-IDF), Markov chains, all the way to Word2Vec and meaning vectors. And at the end one fundamental question came up...

We have word vectors. Great. But how do we build something out of them that understands a sentence? Or a whole paragraph? Or a book?

Because "Dog bites man" isn't simply "dog" + "bites" + "man". It's a sequence - words follow each other in a specific order, and that order changes the meaning. "Dog bites man" and "Man bites dog" are the same words, but completely different stories :D

And Word2Vec? Word2Vec looks at words individually. It has no concept of order. It doesn't know that "not" before "like" changes everything.

So we need something that can process sequences. Something that reads text in order and builds understanding step by step.

And here begins a fascinating story - because the solution to this problem didn't appear overnight. It was an evolution that lasted several decades. From a single, simple neuron, to complicated memory cells. And each generation solved the previous generation's problem, but created a new one.

This is the fourth post in the "Understanding LLM" series. Today we go deeper into the architecture of neural networks - but relax, still without formulas you can't understand ;-) Instead, there will be plenty of metaphors, diagrams, and real-life examples.

The narrative of today's post: "From a single cell to memory - and why that still wasn't enough." Let's start from the basics.

Biological neuron vs artificial - where does the idea even come from?

Before we get into architectures, let's quickly settle one question you're probably asking yourselves: why is it even called a "neural network"? What does it have to do with the brain?

Answer: both a lot, and a little ;-) In the beginning there was inspiration from biology. Take a look:

First diagram - a real neuron. It has dendrites (receive signals from other neurons), a soma (the cell body - decides whether to "fire"), and an axon (passes the signal on).

Second diagram - an artificial neuron. It has inputs (numbers), a weighted sum (multiplies each input by its "importance" and adds it all up), and an output (the result passed through an activation function).

The resemblance is... loose. A biological neuron is unimaginably more complicated. But the inspiration was important - Warren McCulloch and Walter Pitts showed in 1943 that such a simplified model of a neuron can perform basic logical operations (AND, OR, NOT). And that meant a network of such neurons could theoretically compute anything.

OK, that's enough biology. Let's move on to what came out of it ;-)

Perceptron - the simplest decision in the world

In 1958 Frank Rosenblatt created the perceptron - the first algorithm that could learn from data. It was a milestone.

The perceptron works trivially simply. Imagine you're deciding whether to go for a walk:

The mechanics are simple:

1. Each input has its own weight (importance) - sun has weight +5, rain -8, temperature +3
2. The perceptron multiplies each input by its weight and adds it all up (that's the "weighted sum")
3. If the sum is greater than or equal to the threshold (in our case threshold = 0) - it fires "yes". Otherwise - "no".

Example:

And - most importantly - the perceptron can learn these weights. You show it 100 examples of "went out / didn't go out" and it gradually adjusts the weights so that its decisions match yours.

Brilliant in its simplicity, right?

But the perceptron has one big problem...

A perceptron can draw a single straight line dividing data into two groups. Which is great if the data can be split that way:

⚫ ⚫ ⚫    │    ⚪ ⚪ ⚪
⚫ ⚫ ⚫    │    ⚪ ⚪ ⚪
           ↑
      one line separates them ✅

But what if the data looks like this?

⚪  ⚫

⚫  ⚪

That's the famous XOR problem (exclusive OR). The output is "true" only when exactly one input is true - not both and not neither. And you can't draw a single straight line that would separate them.

This problem halted the development of neural networks for almost 20 years. Seriously. Researchers said: "well OK, the perceptron is cool, but if it can't handle something as simple as XOR, what's the point?" This period is called the "AI Winter".

And then someone said: what if we combine several perceptrons together?

MLP - a team that can do more than an individual

A Multilayer Perceptron (MLP) is a network made up of many layers of neurons. Instead of one decision layer, we have:

Every neuron in one layer is connected to every neuron in the next layer (that's why it's called "fully connected" or "dense").

The metaphor that fits best for me:

Imagine a company. Analysts (layer 1) look at raw data and pick out simple patterns. Managers (layer 2) combine those patterns into something more meaningful. The director (the output layer) makes the final decision. None of these people understands the full picture alone - but together they form a cascade of abstraction.

And that's exactly what solves the XOR problem! The first layer draws two lines. The second layer combines them into one region. Suddenly a non-linear problem becomes solvable.

But careful - there's a catch!

Adding hidden layers by itself does nothing. Why?

Because if each layer just does sum = weights × inputs + bias, then two layers of that operation are still... one big linear operation. Let me show with numbers:

Layer 1: y = 2·x + 1 Layer 2: z = 3·y + 2

Substitute the first into the second:

z = 3·(2·x + 1) + 2
z = 6·x + 3 + 2
z = 6·x + 5    ← one operation, just with different numbers

The two layers collapsed into one. Instead of two transformations, you get one - just with different coefficients. It's like using one coffee filter twice as thick instead of two. Same effect.

So we need something non-linear between the layers. And that's where...

The activation function - the unsung hero of deep learning

An activation function is a kind of "non-standard processing plant" that we pass the sum through. The most popular today is ReLU (Rectified Linear Unit):

def relu(x):
    if x > 0:
        return x
    else:
        return 0

That's it. If the value is positive - keep it. If negative - zero it out.

Sounds trivial? It is trivial. But this one simple operation breaks linearity.

Back to our numbers example. Without ReLU, two layers collapsed into one (z = 6·x + 5). But now let's insert ReLU between them:

Layer 1:  y = ReLU(2·x + 1)
Layer 2:  z = ReLU(3·y + 2)

Let's check for three values of x:

Without ReLU the result changes linearly (-7, 5, 11 - constant increase). With ReLU, suddenly for x = -2 we get 2 instead of -7. This relationship is no longer a straight line. Zeroing out negative values is something that can't be expressed as a·x + b - and that's exactly why layers stop collapsing into one.

Analogy: imagine window blinds. Linearity is a transparent pane of glass - it lets everything through, slightly dimmed. ReLU is blinds that completely block light below a certain angle. You can't simulate "complete blocking" by "dimming the glass more". It's a qualitatively different operation.

Thanks to ReLU, each layer does something different from the previous one. And suddenly a network with many layers becomes powerful - it can model non-linear, complicated relationships in data.

Other popular activation functions:

import numpy as np

def relu(x):
    return np.maximum(0, x)

x = np.array([-3, -1, 0, 2, 5])
print(f"Input: {x}")
print(f"After ReLU: {relu(x)}")

Mini-quiz: linear or not?¹

1. Plain linear regression (y = ax + b) - linear or non-linear?
2. A perceptron with a step function (0 or 1) - linear or non-linear?
3. An MLP without an activation function between layers - linear or non-linear?

Problem: an MLP doesn't understand order

OK, we have an MLP. It can model non-linear relationships. Great. But there's one fundamental problem that blocks us from using an MLP for language.

An MLP sees all features at once. It has no concept of order.

Take a classic English example. Two sentences with exactly the same words, but in a different order:

Identical vector. And the meaning? The first is a boring news item. The second is a newspaper sensation :D

For an MLP both sentences are exactly the same - because the MLP gets the same vector and has no concept of order.

This is the permutation problem - an MLP can't distinguish [A, B, C] from [C, B, A]. And in language, order is everything. "I do not like" vs "I like not" - same words, completely different meaning.

You might think: "OK, then let's add word position as a feature." But that doesn't solve the problem - the MLP still doesn't understand that the word at position 1 has a relationship with the word at position 5. Each position is just another number on the input.

So we need something that processes data step by step, in order, and builds understanding sequentially. Just like we read a book - sentence by sentence, word by word.

And here enters the RNN.

RNN - "reads text one by one"

A Recurrent Neural Network (RNN) is a network that has something an MLP doesn't: memory. Well... sort of ;-)

Imagine you're reading a book. You don't read the whole page at once (like an MLP would). You read word by word, sentence by sentence. And with each new word you update your understanding of what's going on.

That's exactly what an RNN does. It has a hidden state - that's its "understanding of the text so far". With each new word:

1. It takes the new word (x_t)
2. It takes its current understanding (h_{t-1})
3. It combines them and creates a new understanding (h_t)

This diagram is exactly the unrolling in time. In reality it's the same RNN neuron - but we "unwrap" it in time to show how it processes consecutive words. Notice: the arrow goes from left to right - information flows sequentially.

This is brilliant in its simplicity. The RNN "reads" text just like we do - one by one, building understanding step by step.

RNN in code - minimal

To really feel this, let's write the simplest possible RNN. No PyTorch, no TensorFlow - just NumPy:

import numpy as np

def simple_rnn(word_vectors, hidden_size=4):
    words, dim = word_vectors.shape
    np.random.seed(42)
    
    W_h = np.random.randn(hidden_size, hidden_size) * 0.01
    W_x = np.random.randn(hidden_size, dim) * 0.01
    bias = np.zeros(hidden_size)
    
    h = np.zeros(hidden_size)
    
    for t, word in enumerate(word_vectors):
        h = np.tanh(W_h @ h + W_x @ word + bias)
        print(f"  Step {t+1}: hidden state = {np.round(h, 2)}")
    
    return h

the = np.array([1.0, 0.0])
cat = np.array([0.0, 1.0])
sits = np.array([0.3, 0.3])
on_mat = np.array([0.8, 0.2])

sentence = np.array([the, cat, sits, on_mat])
print("Processing: 'The cat sits on the mat'")
final = simple_rnn(sentence)
print(f"\nFinal hidden state: {np.round(final, 2)}")

Run it! You'll see how with each word the hidden state changes. The last hidden state is the "understanding" of the whole sentence by our simple RNN.

But the RNN has one big problem...

And here we get to the topic that changed everything.

The goldfish problem - why the RNN forgets

The RNN has memory, yes. But it's a goldfish memory.

Imagine this sentence:

"This is Jan and he's 30 years old. He likes hiking in the mountains, programming in Python, and playing the guitar. His favorite color is green. He once wanted to become an astronaut, but then he discovered that (...) [here 50 words about various things] (...) and that's why ___ went to music school."

What should "___" be? Jan. But to know that, the RNN has to remember the name from the beginning of the sentence. And here's the problem: after 50+ words, the signal from the first word is so diluted that the RNN practically doesn't remember it.

This is the famous vanishing gradient problem.

Metaphor: Chinese whispers. You play Chinese whispers - the first person whispers a message to the second, that one to the third, and so on. After 10 people the message is a bit distorted. After 50 people - completely unreadable. After 100 people - someone says "buy milk", and the last person hears "grab bread".

In an RNN the gradient (the learning signal) flows through the network exactly the same way - step by step. At each step it gets multiplied by some weights. If those weights are smaller than 1, then after many multiplications the gradient vanishes to zero. The network stops learning from distant words.

And sometimes it's the other way - the weights are bigger than 1 and the gradient grows exponentially (exploding gradient). The network "goes crazy" and values shoot into space.

And then in 1997 Sepp Hochreiter and Jürgen Schmidhuber said: what if we gave the network an explicit memory mechanism?

LSTM - the network with a notebook

A Long Short-Term Memory (LSTM) is an RNN on steroids. Instead of one simple hidden state, the LSTM has a memory cell (cell state) and three gates that decide what to do with that memory.

The metaphor that works best: an LSTM is a student with a notebook and three rules.

The forget gate - "throw out old notes"

At the beginning of each step the LSTM asks: "what from my current memory is already outdated and I can throw out?"

When you read a new chapter of a book, you don't need to remember what the characters had for breakfast three chapters ago. You forget irrelevant details to make room for new ones.

In an LSTM: the forget gate passes each element of memory through a sigmoid (values 0-1). Close to 0 = "forget". Close to 1 = "keep".

The input gate - "save this, it's important"

Then the LSTM asks: "what from the new word is worth saving?"

When you read "My name is Jan" - you have a reflex: "OK, this is important, I have to remember it". You don't memorize every word literally, but you pick out what's relevant.

In an LSTM: the input gate decides which new information to let through to the cell state.

The output gate - "show me what I need right now"

Finally the LSTM asks: "what from my memory is relevant to what I'm doing right now?"

When someone asks you "What's the name of that character we just read about?" - you reach into the notebook and pull out the specific information. You don't show the whole notebook - only what's needed now.

The key thing: the cell state is that "conveyor belt". Information on it can flow almost unchanged through many steps - the gates only decide what to let through, what to add, and what to remove. This solves the vanishing gradient problem, because the information doesn't have to be "multiplied" at every step - it can just flow.

An experiment for you

Try it yourself. Fire up ChatGPT (or Claude, Gemini - whatever you have) and send this prompt:

"Pola had a rare ability to remember numbers. Her favorite number was 42. She also liked longboarding, watercolor painting, and playing chess. Her cat was named Whiskers. (...) [paste 3-4 paragraphs of any text here - a pancake recipe, a weather report, whatever] (...) Still, it was ___ who decided to enter the math competition."

See whether the model fills in the blank correctly (Pola). An LSTM would handle this. A plain RNN - probably not. And why? Because the LSTM has a mechanism that says "remember the name Pola - this might be important later". The RNN would simply... forget ;-)

Why LSTM still wasn't enough

OK, so LSTM solved the memory problem. At least partially. But new problems appeared, and they turned out to be fundamental.

Problem 1: Even LSTM has its limits

LSTM solved vanishing gradient to a large extent, but not completely. When the distance between related words grows to hundreds or thousands of words, even LSTM starts losing the thread of the narrative. Bengio et al. (1994) showed that recurrent networks - even with gates - still struggle to maintain useful gradients over very long distances.

So: instead of forgetting after 10 words (RNN), LSTM forgets after... 100? 200? Better, but still not ideal.

Problem 2: Sequentiality = bottleneck

And this is the most important problem.

RNN and LSTM process data step by step. Step 2 has to wait for step 1. Step 3 for step 2. And so on.

Now think about GPUs. A GPU is a machine that loves doing many things at once (parallel processing). Thousands of cores working simultaneously.

But RNN/LSTM tells the GPU: "no, no, wait. First let's finish step 1, only then start step 2". The GPU cries.

It's like a book with 10,000 chapters, where each chapter has to be read after the previous one. You can't read chapter 500 until you've finished 499. You can't parallelize this. For short texts it's not a problem, but when training a model on billions of words? That's a nightmare.

LSTM additionally has 4x more parameters than a simple RNN (three gates + cell state = four sets of weights per neuron). So it trains slower and uses more memory.

Problem 3: "Black box"

LSTM (like most deep learning models) is hard to interpret. The model can generate correct results, but it's hard to understand why it made this decision and not another. In applications like medicine or finance, where interpretability is key, that's a serious problem.

Evolution summary - what solved what

Each generation solved the previous one's problem, but created a new one. It's like a game of whack-a-mole - you hit one mole, another pops up.

And then in 2017 someone asked a crazy question...

"What if we stop reading one by one?"

The metaphor that changes everything.

Imagine you're reading a novel. Now you have three ways to do it:

• RNN - you remember the last few sentences. The rest blurs.
• LSTM - you hold onto the main plot, forget the small details. Better, but you still read linearly.
• ??? - instead of reading word by word, you look at the whole page at once. Your attention jumps to the key characters, important plot moments, unexpected twists. You don't read in order - you grasp the whole thing simultaneously!

In 2017 a group of researchers at Google published a paper with the modest title: "Attention Is All You Need". Their crazy idea? What if we stop reading one by one altogether? What if every word could "see" all the other words at once?

And a revolution happened. Because if every word looks at every other simultaneously (not waiting for its turn), then:

• There's no sequential bottleneck - everything happens in parallel
• The GPU is in heaven - thousands of cores working at once
• Long dependencies? No problem - word no. 1 and word no. 1000 "see" each other directly, without 999 intermediate steps

This architecture is called the Transformer. And it's what ChatGPT, Claude, Gemini, and all the LLMs you know are built on.

Summary - the whole evolution in one place

Final quiz: match the architecture²

1. You want to predict the price of a house based on square footage, number of rooms, and neighborhood - which architecture is enough?
2. You need to process a 50-word sentence and determine its sentiment (positive/negative) - what do you choose?
3. You're training a model on 10,000-word texts and every word has to "see" every other - RNN, LSTM, or something else?
4. You want to classify points on a plane into two groups, but they're arranged in an X shape (XOR) - is a single perceptron enough?

This post is quite long, but I felt this evolution from the perceptron to LSTM deserved a full, told story.

If anything is unclear - let me know in the comments, I'll try to explain.

Which architecture surprised you the most? Did you know that the "AI Winter" was caused by something as "simple" as XOR?

What's in the next post? We're getting into the Transformer - the architecture that changed everything. Attention, self-attention, positional encoding, multi-head attention - all the things that made ChatGPT exist in the first place. Stay tuned!

See you next time!

Sources and interesting links:

If you want to go deeper, here are the materials I used:

• Multilayer Perceptrons - d2l.ai - an excellent, visual introduction to MLP: layers, activation functions, the transition from linear regression to multi-layer networks
• Multilayer perceptron - Wikipedia - definitions, network diagrams, a timeline of the history from 1943 to the present
• Multilayer Perceptron (MLP): A Practical Way to Understand Neural Networks - dev.to - a fresh, very intuitive article with metaphors and a comparison of MLP vs CNN vs Transformer
• Perceptron Explained - PlainEnglish - the perceptron as the "building block" of networks, components, geometric intuition, limitations
• Multi-Layer Perceptron Learning in TensorFlow - GeeksforGeeks - clear diagrams, forward propagation, backpropagation with implementation
• Introduction to Long Short Term Memory - GeeksforGeeks - an accessible introduction to LSTM: memory cell, gates, the long-term dependencies problem
• Drawbacks of LSTM Algorithm: A Case Study - SSRN - the drawbacks of LSTM: computational complexity, overfitting, lack of parallelization, sensitivity to hyperparameters
• Understanding Transformer Model Types: The Evolution from RNN to Modern AI - dev.to - a narrative account of the transition from RNNs to Transformers, encoder/decoder types
• LLM: Large Language Models Evolution - YouTube - a video telling the development line from sequential models to attention-based architectures