How a computer reads text - from counting words to vectors

In the two previous posts we built a solid foundation. In the first we took language apart - phonetics, morphology, syntax, semantics, pragmatics. In the second we asked: "OK, but does an LLM even understand what it generates?" and concluded that an LLM is a machine of signs, not a mind.

But one fundamental question remained unanswered: how does a computer even "take" text in?

Because a computer doesn't see letters. It doesn't see words. A computer sees numbers. So how is it that you type "The cat sits on the mat" and ChatGPT somehow processes it? What's the path from text to number, from number to "understanding"?

That path is a fascinating story of evolution - from the simplest word counting, through ever smarter mathematical tricks, to vectors that can capture semantic similarities. Every step on that path was an answer to the limitations of the previous one.

This is the third post in the "Understanding LLM" series. Today we switch the perspective from linguistic and philosophical to technical. But don't worry - there will still be plenty of examples, plenty of "aha!" moments, and zero formulas you can't understand ;-)

The narrative of this post: "First you cut the text into pieces, then you count, then you understand." Simple? We'll see ;-) Let's start with the cutting.

---

Tokenization - how text gets split into pieces

The problem: a computer doesn't see words

Imagine you're a computer. Someone shows you text:

"Alice has a cat."

What do you see? Letters? Words? No. You see a sequence of bytes: 41 6c 69 63 65 20 68 61 73 20 61 20 63 61 74 2e. Zero idea where one word starts and another ends. Zero idea that "cat" is an animal and not three random characters.

For a computer to do anything with text, it first has to cut it into pieces. And that process is called tokenization.

Tokenization is step zero. Without it there's no TF-IDF, no n-grams, no embeddings, no LLM. Everything starts with cutting.

The naive approach: split on spaces

The simplest idea: split the text on spaces.

"Alice has a cat." -> ["Alice", "has", "a", "cat."]

Did you notice? "cat." has a period glued to it. That's not the word "cat" - it's the word "cat" with punctuation. And what about texts like these:

• "un-believ-able" - one word or three?
• "Hi!" - word + punctuation?
• "New York" - one word or two?
• ":) " - a word? a symbol? an emotion?

In short: splitting on spaces doesn't work. The world is too complicated for such simple rules.

Token != word != morpheme

In the first post we met morphemes - the smallest meaningful units. "Unhappiness" is three morphemes: "un-happy-ness". And now note: a token is not the same as a morpheme.

A token is simply a piece of text determined by the tokenizer. It can be a whole word, a part of a word, or a single character. It depends on the algorithm.

BPE - Byte Pair Encoding

And here comes BPE (Byte Pair Encoding) - the most popular tokenization algorithm, used by GPT-2, GPT-3, GPT-4, Llama, and many other models.

The idea of BPE can be reduced to one sentence: find the most frequent pair of characters and merge it into one token. And repeat.

Sounds simple? Because it is simple ;-) Let's see it on an example.

BPE step by step

We have a small corpus with five English words and their frequencies:

"low"  ×10     "lower"  ×5      "newest"  ×12
"widest"  ×4    "new"  ×5

Step 0: We split everything into single characters:

l o w       ×10     l o w e r       ×5      n e w e s t       ×12
w i d e s t  ×4     n e w           ×5

Our starting dictionary is: ["d", "e", "i", "l", "n", "o", "r", "s", "t", "w"] - simply all unique characters.

Step 1: We find the most frequent pair of adjacent characters:

• (e, s) appears in "newest" (12) and "widest" (4) = 16 times <- winner!
• (n, e) appears in "newest" (12) and "new" (5) = 17 times
• (l, o) appears in "low" (10) and "lower" (5) = 15 times

Wait, let's recount. (n, e): "newest" (12) + "new" (5) = 17. Actually (n,e) = 17 times <- winner!

Let's merge (n, e) -> "ne". Our dictionary grows by one token:

Dictionary: ["d", "e", "i", "l", "n", "o", "r", "s", "t", "w", "ne"]

l o w       ×10     l o w e r       ×5      ne w e s t        ×12
w i d e s t  ×4     ne w            ×5

Step 2: We look for the most frequent pair again:

• (ne, w) appears in "newest" (12) and "new" (5) = 17 times <- winner!
• (e, s) appears in "newest" (12) and "widest" (4) = 16 times
• (l, o) appears in "low" (10) and "lower" (5) = 15 times

Merge (ne, w) -> "new":

Dictionary: [..., "ne", "new"]

l o w       ×10     l o w e r       ×5      new e s t         ×12
w i d e s t  ×4     new             ×5

Step 3: The most frequent pair is now (e, s) = 16 times. Merge -> "es":

Dictionary: [..., "ne", "new", "es"]

l o w       ×10     l o w e r       ×5      new es t          ×12
w i d es t   ×4     new             ×5

Step 4: The most frequent pair is (l, o) = 15 times. Merge -> "lo":

Dictionary: [..., "ne", "new", "es", "lo"]

lo w        ×10     lo w e r        ×5      new es t          ×12
w i d es t   ×4     new             ×5

Step 5: The most frequent pair is (lo, w) = 15 times. Merge -> "low":

Dictionary: [..., "ne", "new", "es", "lo", "low"]

low         ×10     lo w e r        ×5      new es t          ×12
w i d es t   ×4     new             ×5

And so on, until we reach the target vocabulary size. In real models:

• GPT-2 has a vocabulary of 50 257 tokens (256 bytes + 50 000 merges + 1 special)
• GPT-4/o has a vocabulary of ~100 000 tokens
• Llama 3 also uses around ~100 000 tokens

When is the tokenizer used?

That's an important question, and the answer is: always. At both stages.

1. Training: first you train the tokenizer on a huge corpus (it learns which pairs to merge). Then you tokenize the ENTIRE training corpus into IDs. The model learns on those IDs.

2. Prompting: when you write to ChatGPT, your text goes through the SAME tokenizer -> IDs -> the model generates new IDs -> a decoder turns them back into text.

The key point: the tokenizer is trained separately, before the model. Then it's "frozen" - it never changes again. GPT-2 has its tokenizer (a 50K vocabulary), GPT-4 has its own (a 100K vocabulary). And that's the reason Polish gets cut up worse in GPT-2 - because GPT-2's tokenizer was trained mainly on English, so few Polish character sequences got merged. The model itself might "understand" Polish better, but the tokenizer already cut the text into tiny pieces.

Different models - different cuts

The key thing: every model has its own tokenizer. The same text can be cut completely differently:

import tiktoken

text = "Incredibly happy"

gpt2 = tiktoken.get_encoding("gpt2")
gpt4 = tiktoken.get_encoding("cl100k_base")

print("GPT-2:", gpt2.encode(text))
print("GPT-4:", gpt4.encode(text))

print("GPT-2:", [gpt2.decode([t]) for t in gpt2.encode(text)])
print("GPT-4:", [gpt4.decode([t]) for t in gpt4.encode(text)])

GPT-2 will probably cut the text into many small fragments (because it doesn't "know" it well), and GPT-4 will do it much more efficiently (because it has seen more of this text).

The example above uses Polish ("Nieprawdopodobnie szczęśliwy" = "Incredibly happy") because it perfectly illustrates how badly a tokenizer trained on one language handles another.

What about Polish tokenizers?

There are models trained specifically on Polish text - and their tokenizers cut Polish much better:

from transformers import AutoTokenizer

text = "Nieprawdopodobnie szczęśliwy"

herbert = AutoTokenizer.from_pretrained("allegro/herbert-base-cased")
print("HerBERT (Allegro, Polish WordPiece):", herbert.tokenize(text))

roberta = AutoTokenizer.from_pretrained("sdadas/polish-roberta-base-v2")
print("Polish RoBERTa (Polish SentencePiece):", roberta.tokenize(text))

HerBERT (Allegro, Polish WordPiece):  ['Nie', 'prawdopodobnie</w>', 'szczęśliwy</w>']  -> 3 tokens
Polish RoBERTa (Polish SentencePiece): ['▁Nie', 'prawdopodobnie', '▁szczęśliwy']        -> 3 tokens

Three tokens! "prawdopodobnie" and "szczęśliwy" are single tokens for a Polish tokenizer. Because that tokenizer "saw" these words so many times on a Polish corpus that it merged them into whole units.

Two Polish tokenizers worth knowing:

Tokenizer	Creator	Algorithm	Implemented in
HerBERT	Allegro	WordPiece	Polish BERT on the KGR10 corpus
Polish RoBERTa	sdadas	SentencePiece (Unigram)	Polish RoBERTa on a large corpus

Build your own tokenizer!

Now that we understand how BPE works, let's try to train our own tokenizer. We'll use the tokenizers library from HuggingFace (not to be confused with tiktoken):

from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
from tokenizers.pre_tokenizers import Whitespace

tokenizer = Tokenizer(BPE(unk_token="[UNK]"))
tokenizer.pre_tokenizer = Whitespace()

trainer = BpeTrainer(vocab_size=300, special_tokens=["[UNK]"])

corpus = [
    "Alice has a cat and a dog",
    "The cat sits on the mat",
    "The dog runs in the park",
    "Alice has a cat and a cat and once more a cat",
    "The cat likes milk and the cat likes to sleep",
    "The dog likes to run in the park and chase the cat",
    "A happy cat is a cat that has lots of milk",
    "An incredibly happy dog runs in the park",
    "Happy Alice has a cat and a dog and a happy home",
    "A home is a place where a happy family lives",
]

tokenizer.train_from_iterator(corpus, trainer)
print(f"Vocabulary size: {tokenizer.get_vocab_size()}")

test = "Incredibly happy"
encoding = tokenizer.encode(test)
print(f'"{test}" -> {encoding.tokens}')

Vocabulary size: 130
"Incredibly happy" -> ['Incredibly', 'happy']

Two tokens! Because our small corpus is in English - and "Incredibly" and "happy" appeared often enough for BPE to merge them into single tokens.

But wait - what do those parameters in the code actually mean?

Parameter	What it does	Example
vocab_size	The maximum number of tokens in the vocabulary. BPE will merge pairs until the vocabulary reaches this size. The bigger, the longer the tokens (whole words). The smaller, the smaller the pieces (single letters).	GPT-2: 50 257, ours: 300
special_tokens	Tokens with a special meaning for the model. They're always in the vocabulary, regardless of training.	[UNK], <PAD>, <S> (start), </S> (end)
unk_token	The "unknown" token - replaces every character the tokenizer can't recognize. E.g. if an emoji appears in the text and the tokenizer doesn't have it in the vocabulary, it inserts [UNK].	[UNK] = "I don't know what this is"

A simple analogy: vocab_size is the thickness of your dictionary - how many entries fit in it. unk_token is the entry meaning "no such word". And special_tokens are "reserved pages" - they're always in the dictionary, regardless of what you teach the tokenizer.

Other tokenization algorithms

BPE isn't the only player in town. Here's a quick comparison:

Algorithm	Who uses it	How it works	Key difference
BPE	GPT, Llama, many others	Merges the most frequent pair	Simple, bottom-up
WordPiece	BERT, DistilBERT, Electra	Merges the pair with the highest "score"	Score = freq(pair) / (freq(a) x freq(b))
Unigram	T5, Pegasus, ALBERT	Starts from a large vocabulary, removes the least useful	Probabilistic, can yield different tokenizations
SentencePiece	Multilingual models	BPE or Unigram on raw text (even without spaces!)	Works with languages without spaces (Chinese, Japanese)

The difference between BPE and WordPiece is subtle: BPE simply merges the most frequent pair. WordPiece merges the pair that is most surprising - i.e. one that appears together more often than would follow from the frequency of the individual elements. It's a bit like a relationship that is "more than the sum of its parts" ;-)

Quiz: how will BPE cut this?¹

We have our merges from the example above: n+e->ne, ne+w->new, e+s->es, l+o->lo, lo+w->low. How will BPE split these new words?

1. "lowest" (assuming it wasn't in the corpus)
2. "newer" (assuming it wasn't in the corpus)
3. "widower" (a real English word!)

---

Corpora, Bag-of-Words, and TF-IDF

What is a corpus?

OK, we have tokens. But how does the tokenizer know which character pairs to merge? From a corpus! BPE learns the most frequent combinations from text. And to do anything with tokens - count them, find patterns, train a model - we also need a corpus.

A corpus is simply a collection of texts. It can be:

• the set of all Wikipedia articles
• a set of product reviews from Amazon
• the emails in your inbox
• a set of all posts on Reddit (knock on wood)

Each individual text in a corpus we call a document. Simple?

And these are exactly the datasets on which real models are trained - both tokenizers and the models themselves:

Corpus	What it contains	Size	Fun fact
Wikitext	Articles from (English) Wikipedia	~500 MB	A standard benchmark for evaluating language models
Wiki-40B	Wikipedia in 59 languages (including Polish!)	~40 GB	The first step of many multilingual models
EuroParl	Transcriptions from the EU parliament (21 languages)	~2 GB	High-quality official texts, great for translation
Common Crawl	Dumps of web content	petabytes (PB!)	The largest publicly available web corpus. Most LLMs use it
OpenWebText	A copy of Reddit links with >3 upvotes	~38 GB	GPT-2 was trained on it. Filtered "quality" from Reddit
The Pile	A mix of 22 sources (arXiv, GitHub, Wikipedia, books...)	~825 GB	Created by EleutherAI as a "do-everything" dataset
RedPajama	An open replica of LLaMA's training data	~1.2 TB	That's why LLaMA is so good - trained on a huge, diverse set
OSCAR	Web dumps, filtered by language	~6.5 TB	Has separate subsets for each language, including Polish OSCAR

And Polish corpora? Here are the most important ones:

• NKJP (Narodowy Korpus Jezyka Polskiego / National Corpus of Polish) - millions of Polish texts, a balanced set from various fields
• Polish-ROBERTa corpus - ~20 GB of Polish text from the web, Polish RoBERTa was trained on it
• OSCAR (the Polish part) - Polish pages from Common Crawl, several hundred GB
• Polish Wikipedia - ~2 GB of articles, often the starting point for Polish models

Key observation: the same corpus is used to train both the tokenizer and the model. First you train the tokenizer on the corpus (it learns which character pairs to merge), then you tokenize the whole corpus with that tokenizer, and on those tokens you train the model.

Bag-of-Words - a bag full of words

We have a corpus. Now we want to turn each document into numbers so the computer can do something with it.

The simplest way: Bag-of-Words (BoW) - a bag of words. We count how many times each word appears in the document.

Example. We have three short movie reviews:

Document 1: "This movie is great"
Document 2: "This movie is terrible"
Document 3: "Great movie I recommend"

We build a vocabulary of all unique words:

{this, movie, is, great, terrible, i, recommend}

And now we turn each document into a vector of counts:

	this	movie	is	great	terrible	i	recommend
Doc 1	1	1	1	1	0	0	0
Doc 2	1	1	1	0	1	0	0
Doc 3	0	1	0	1	0	1	1

Each document is now just a sequence of numbers. The computer is happy.

But we're not entirely ;-). Because notice the problem:

Document 1: "This movie is great" -> [1, 1, 1, 1, 0, 0, 0] Document 2: "This movie is terrible" -> [1, 1, 1, 0, 1, 0, 0]

These vectors are almost identical! They differ in one position. Yet one says "great" and the other "terrible" - completely different meanings.

And there's a second problem: "Dog bites man" and "Man bites dog" will give exactly the same BoW vector. BoW completely ignores word order.

Bag-of-Words in code

from sklearn.feature_extraction.text import CountVectorizer
import pandas as pd

corpus = [
    "This movie is great",
    "This movie is terrible",
    "Great movie I recommend",
]

vectorizer = CountVectorizer()
X = vectorizer.fit_transform(corpus)

df = pd.DataFrame(
    X.toarray(),
    columns=vectorizer.get_feature_names_out(),
    index=["Doc 1", "Doc 2", "Doc 3"]
)
print(df)

Result:

        great  i  is  movie  recommend  terrible  this
Doc 1      1  0   1      1          0         0     1
Doc 2      0  0   1      1          0         1     1
Doc 3      1  1   0      1          1         0     0

Exactly the same table as above - only now generated by code. Notice: "movie" is everywhere (value 1 in every row), and "terrible" only in Document 2. That's the whole magic of BoW - simple counting.

TF-IDF - what if not all words are equal?

BoW treats every word the same. But not every word is equally important!

The word "is" appears in almost every English sentence. The word "semiotics" - rather rarely. Which is more informative? Obviously "semiotics" - because if you see it in a document, it tells you much more about its content than "is".

TF-IDF (Term Frequency - Inverse Document Frequency) is a way to capture that.

Recall our corpus - three short movie reviews:

Document 1: "This movie is great"          (4 words)
Document 2: "This movie is terrible"       (4 words)
Document 3: "Great movie I recommend"      (3 words)

Let's compute TF-IDF for the word "great" in Document 1:

TF (Term Frequency) - how often the word appears in one document:

TF("great", Document 1) = 1 / 4 = 0.25

(1 occurrence in a 4-word document)

IDF (Inverse Document Frequency) - how "rare" the word is across the whole corpus:

IDF("great") = log(3 / 2) = 0.176

(3 documents in the corpus, 2 of them contain "great")

TF-IDF = TF x IDF:

TF-IDF("great", Document 1) = 0.25 x 0.176 = 0.044

And the word "is", which is EVERYWHERE?

IDF("is") = log(3 / 2) = 0.176  (also in 2 of 3 documents)
IDF("movie") = log(3 / 3) = 0      (in ALL documents!)

"Movie" gets zero in IDF! Because if a word is in every document, it carries no information that would help tell one document apart from another.

TF-IDF in code

Try it yourself. Here's a complete example in Python:

from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd

corpus = [
    "This movie is great",
    "This movie is terrible",
    "Great movie I recommend",
]

vectorizer = TfidfVectorizer()
tfidf_matrix = vectorizer.fit_transform(corpus)

df = pd.DataFrame(
    tfidf_matrix.toarray(),
    columns=vectorizer.get_feature_names_out(),
    index=["Doc 1", "Doc 2", "Doc 3"]
)
print(df.round(2))

The result will look roughly like this:

        great    i    is  movie  recommend  terrible  this
Doc 1     0.41  0.0  0.49   0.37        0.0       0.0  0.49
Doc 2     0.00  0.0  0.46   0.35        0.0       0.58  0.46
Doc 3     0.41  0.54  0.0   0.41        0.54       0.0  0.00

Notice: "movie" has low values everywhere (because it's everywhere). "terrible" has a high value only in Document 2 (because it only appears there). TF-IDF works!

TF-IDF as a search engine

TF-IDF has one super practical application: text search. This is basically how the first search engines worked.

The idea is simple: you turn the user's query into a TF-IDF vector, and look for the document whose vector is most similar to it (so-called cosine similarity).

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

corpus = [
    "This movie is great",
    "This movie is terrible",
    "Great movie I recommend",
]

vectorizer = TfidfVectorizer()
tfidf_matrix = vectorizer.fit_transform(corpus)

query = "great movie"
query_vec = vectorizer.transform([query])

similarities = cosine_similarity(query_vec, tfidf_matrix)[0]
ranked = sorted(zip(corpus, similarities), key=lambda x: -x[1])
for i, (doc, sim) in enumerate(ranked, 1):
    print(f'{i}. "{doc}" -> {sim:.3f}')

Query: "great movie"

1. "Great movie I recommend"        -> 0.694  <- best result!
2. "This movie is great"            -> 0.667
3. "This movie is terrible"         -> 0.229

Document 3 wins - because "great" and "movie" are high-TF-IDF keywords for it. Document 2 has "movie" but not "great" - so its similarity is low.

---

N-grams - or, adding context

The problem: words don't live in a vacuum

TF-IDF is better than pure BoW, because at least it weights words by importance. But it still treats every word separately. It doesn't know that "do" + "not" + "like" = something completely different than "like" alone.

Or an English example that shows the problem perfectly:

• "big red carpet and machine" - we're talking about a carpet
• "big red machine and carpet" - we're talking about a machine

The same words! But the order changes everything. BoW and TF-IDF don't see it.

What are n-grams?

An n-gram is simply a sequence of N consecutive elements (usually words or characters).

Example with the sentence "Alice has a cat":

Type	N	Result
Unigrams	1	Alice, has, a, cat
Bigrams	2	Alice has, has a, a cat
Trigrams	3	Alice has a, has a cat

And now the magic: instead of counting single words, we count pairs of words (bigrams). And suddenly:

• "do not" becomes a single entity (negation + verb = negative sense)
• "big red" and "red carpet" are different things
• "New York" is one unit, not two words

You can also combine n-grams with TF-IDF! In sklearn it's enough to change one parameter:

from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd

corpus = [
    "This movie is great",
    "This movie is terrible",
    "Great movie I recommend",
]

vectorizer = TfidfVectorizer(ngram_range=(1, 2))
tfidf_matrix = vectorizer.fit_transform(corpus)

df = pd.DataFrame(
    tfidf_matrix.toarray(),
    columns=vectorizer.get_feature_names_out(),
    index=["Doc 1", "Doc 2", "Doc 3"]
)
print(df.round(2))

Look what happened - the feature vocabulary grew! Besides single words ("movie", "is", "great") we now have pairs: "movie is", "is great", "is terrible", "great movie", "this movie", "movie recommend". Each pair is a separate column with its own TF-IDF.

Thanks to this, the model sees that "is great" (Doc 1) and "is terrible" (Doc 2) are different things - because they're different bigrams with different TF-IDF values.

Markov chains - when n-grams start "predicting"

N-grams by themselves are just counts - like BoW, they don't generate text. But if we add transition probabilities to them, we suddenly get something that generates new text.

The idea is simple: for each word we look at what words most often follow it. And we pick the next word probabilistically.

Let's take a small corpus:

"alice has a cat alice has a dog the cat likes milk alice likes the cat"

We count bigrams and transition probabilities:

Word	Next word	Probability
alice	has	2/3 = 67%
alice	likes	1/3 = 33%
has	a	2/2 = 100%
a	cat	2/3 = 67%
a	dog	1/3 = 33%
the	cat	1/1 = 100%
cat	likes	1/1 = 100%
likes	milk	1/2 = 50%
likes	the	1/2 = 50%

That is exactly a Markov chain - a model in which the probability of the next state depends only on the current state (or the last few).

"alice" -> most often "has" (67%) -> "a" -> "cat" (67%) or "dog" (33%). So we generate e.g.: "alice has a cat". Or "alice has a dog". Or "alice likes the cat". All these sentences are "new" - they didn't appear as wholes in the corpus - but the model stitched them together from transition probabilities.

How it works under the hood - step by step

Before we get to the code, let's trace the whole process on our fingers. We have the same corpus:

"alice has a cat alice has a dog the cat likes milk alice likes the cat"

Step 1: Build the trigram dictionary. We slide a "window" of 3 words and look at what follows:

Window (trigram)	Next word
alice has a	cat
has a cat	alice
a cat alice	has
cat alice has	a
alice has a	dog
has a dog	the
a dog the	cat
dog the cat	likes
the cat likes	milk
cat likes milk	alice
likes milk alice	likes
milk alice likes	the

Each trigram has exactly one possible next word (because our corpus is tiny).

Step 2: Generate. We start from the starter trigram ("alice", "has", "a"):

Start:    alice has a
Step 1:   alice has a [cat]     <- after (alice,has,a) comes "cat"
Step 2:   alice has a cat [alice]  <- after (has,a,cat) comes "alice"
Step 3:   alice has a cat alice [has] <- after (a,cat,alice) comes "has"
Step 4:   ... cat alice has [a]   <- after (cat,alice,has) comes "a"
Step 5:   ... alice has a [dog]   <- after (alice,has,a) comes "dog"
Step 6:   ... has a dog [the]     <- after (has,a,dog) comes "the"
Step 7:   ... a dog the [cat]     <- after (a,dog,the) comes "cat"
Step 8:   ... dog the cat [likes] <- after (dog,the,cat) comes "likes"
Step 9:   ... the cat likes [milk] <- after (the,cat,likes) comes "milk"
Step 10:  STOP <- trigram (cat,likes,milk) -> "alice"... it loops back

Result: "alice has a cat alice has a dog the cat likes milk" - basically our corpus, "re-glued" from the inside. On a larger corpus the result would be different every time.

A simple text generator from n-grams

Here's a complete (and genuinely short!) text generator in Python:

import random

corpus = "alice has a cat alice has a dog the cat likes milk alice likes the cat"
tokens = corpus.split()

trigrams = {}
for i in range(len(tokens) - 3):
    key = (tokens[i], tokens[i + 1], tokens[i + 2])
    next_word = tokens[i + 3]
    if key not in trigrams:
        trigrams[key] = []
    trigrams[key].append(next_word)

current = ("alice", "has", "a")  # <- this is our "prompt"!
output = list(current)

for _ in range(10):
    if tuple(output[-3:]) not in trigrams:
        break
    possibilities = trigrams[tuple(output[-3:])]
    output.append(random.choice(possibilities))

print(" ".join(output))

Possible output: "alice has a cat alice has a dog the cat likes milk alice likes the cat"

Nothing spectacular, right? But that's because our corpus is microscopic. On a real corpus (e.g. all of Wikipedia) a trigram generator can produce sentences that sound sensible, even though they never appeared before.

---

Bayesian techniques - or, classifying text

What if we don't want to generate, but CLASSIFY?

Markov chains are great for generating text. But in practice we often want something else: assign a text to a category.

• Is this email spam or not-spam?
• Is this review positive or negative?
• Is this article about sports, politics, or technology?

And here comes Naive Bayes - one of the simplest yet most useful text classification algorithms.

Bayes' theorem - the intuition

Without formulas, just an example:

Imagine a colleague comes up to you and says: "I'm coughing." What's the probability he has a cold?

It depends! If it's November and everyone in the office is sick - high. If it's July and he coughs once - low.

Bayes' theorem is a formal way of thinking about such situations: we update our knowledge based on new evidence.

And now let's translate that to text. The question is:

"If a document contains the word 'viagra', what's the probability it's spam?"

Naive Bayes in action

We have a small set of emails:

Email	Content	Label
1	"Buy cheap viagra now"	Spam
2	"Free viagra offer"	Spam
3	"Meeting tomorrow at 10"	Not-spam
4	"Send me the report tomorrow"	Not-spam

A new email arrives: "Viagra meeting tomorrow". Spam or not?

Naive Bayes computes:

1. P(Spam) = 2/4 = 0.5, P(Not-spam) = 2/4 = 0.5
2. P("viagra" | Spam) = 2/2 = 1.0 (both spams have "viagra")
3. P("viagra" | Not-spam) = 0/2 = 0.0 (no not-spam has "viagra")
4. P("tomorrow" | Spam) = 0/2 = 0.0
5. P("tomorrow" | Not-spam) = 2/2 = 1.0

P(Spam | "viagra meeting tomorrow") ∝ 0.5 x 1.0 x 0.0 x ... = 0!

P(Not-spam | "viagra meeting tomorrow") ∝ 0.5 x 0.0 x 1.0 x ... = 0!

Oops. The presence of "tomorrow" (a not-spam word) zeroed out spam, and the presence of "viagra" (a spam word) zeroed out not-spam. This particular email is tricky ;-)

In practice, so-called Laplace smoothing is used (we add 1 to each count) to avoid zeroing:

• P("viagra" | Spam) = (2 + 1) / (7 + 14) = 0.214 (we add 1, divide by all words in spam + unique words)
• P("viagra" | Not-spam) = (0 + 1) / (7 + 14) = 0.048

Now: P(Spam | email) ∝ 0.5 x 0.214 x 0.048 x 0.048 ≈ 0.000246 P(Not-spam | email) ∝ 0.5 x 0.048 x 0.143 x 0.143 ≈ 0.000490

Not-spam wins! Because "tomorrow" and "meeting" strongly indicate a normal email.

Why "Naive"?

Because we assume words are independent. That is: the probability of "viagra" appearing doesn't depend on whether "cheap" also appears. Of course that's not true! "Viagra" and "cheap" appear together more often than by chance. But the model still works surprisingly well...

It's a bit like assuming the weather in London doesn't depend on the weather in Paris. Of course it does a bit! But if you want to quickly estimate whether you need an umbrella, the independence assumption gives you a reasonably good answer ;-)

A spam classifier in Python

Now that we understand the math, let's do it in a few lines of code. The scikit-learn library has a ready Naive Bayes classifier:

• CountVectorizer - turns text into a word-count vector (our BoW from the previous section)
• MultinomialNB - that's Naive Bayes, the "multinomial" variant (because it counts word probabilities), with alpha=1.0 i.e. Laplace smoothing

from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import CountVectorizer

emails = [
    "Buy cheap viagra now",
    "Free viagra offer",
    "Meeting tomorrow at 10",
    "Send me the report tomorrow",
    "Cheap pills online buy now",
    "Report from yesterday send",
]
labels = [1, 1, 0, 0, 1, 0]  # 1=spam, 0=not-spam

vectorizer = CountVectorizer()
X = vectorizer.fit_transform(emails)

classifier = MultinomialNB(alpha=1.0)
classifier.fit(X, labels)

new_email = vectorizer.transform(["Viagra meeting tomorrow"])
prediction = classifier.predict(new_email)
print("Spam!" if prediction[0] == 1 else "Not spam.")

---

Word2Vec - words become geometry

The "eureka" moment

All the methods we've met so far have one problem in common: they treat words as discrete, independent entities. In BoW "cat" is at position 47 in the vector, "dog" at position 138. There's no relationship between them.

But WE know that "cat" and "dog" are similar - both are pets. "Cat" and "car" - completely different. How do we make a computer "know" that too?

The answer from 2013, from Tomáš Mikolov's team at Google: let's turn words into points in a high-dimensional space. Semantically similar words will be close to each other. Different words - far apart.

That's Word2Vec. And it's a breakthrough.

One-hot encoding: the starting point

Before Word2Vec, the standard was so-called one-hot encoding - each word is a vector with a 1 at its position and 0 everywhere else:

"cat"  -> [0, 0, 0, 1, 0, 0, ...]  (1 at position 3)
"dog"  -> [0, 0, 1, 0, 0, 0, ...]  (1 at position 2)
"car"  -> [1, 0, 0, 0, 0, 0, ...]  (1 at position 0)

The problem? Every word is the same distance from every other word. The distance between "cat" and "dog" is the same as between "cat" and "car". Zero information about meaning.

Word2Vec: "Show me your neighbors and I'll tell you who you are"

Word2Vec is based on a brilliant linguistic intuition: words that appear in similar contexts have similar meanings.

If you see: "___ runs in the park and chases pigeons", what fits there? Dog? Cat? Rather not "car" or "democracy". Context defines the word.

"I feed ___ kibble" - both dog and cat fit. That means the context of these words is partly shared. And that's exactly what Word2Vec exploits.

Remember Saussure from the second post? Meaning is relational. "Cat" means what it means because it is NOT "dog", it is NOT "house". Word2Vec is the mathematical implementation of that idea!

CBOW and Skip-gram - two sides of the same coin

Word2Vec has two variants. Let's see both:

CBOW (Continuous Bag of Words): from context we guess the middle word.

"The cat ___ on the mat" -> sits? lies? sleeps?

Skip-gram: from the middle word we guess the context.

"sits" -> The? cat? on? the? mat?

How does it work inside? A neural network with one hidden layer:

1. Each word starts as a one-hot vector (e.g. [0, 0, 1, 0, ...])
2. We multiply by a weight matrix W (size: vocabulary x embedding dimension, e.g. 50 000 x 300)
3. The result is a vector of size E (e.g. 300 numbers) - that's our embedding!
4. In CBOW: we average the context vectors and predict the middle. In Skip-gram: we take the middle and predict the neighbors.
5. The network trains on millions of (word, context) pairs, updating the matrix W
6. After training, the row of the matrix W corresponding to a given word is its embedding

Sounds complicated? Let's do it in code - from scratch, without any ML libraries:

import numpy as np

np.random.seed(42)

# --- 1. Vocabulary and one-hot encoding ---
corpus = "cat sits on the mat and sleeps dog sits on the rug and sleeps cat runs around the room dog runs in the park".split()
vocab = list(set(corpus))
word2idx = {w: i for i, w in enumerate(vocab)}
vocab_size = len(vocab)

def one_hot(word):
    vec = np.zeros(vocab_size)
    vec[word2idx[word]] = 1
    return vec

print(f'One-hot "cat": {one_hot("cat")}')
# [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1.]  - only a single one!

# --- 2. Weight matrix (these will be our embeddings) ---
embed_dim = 5  # in real Word2Vec this is 100-300
W1 = np.random.randn(vocab_size, embed_dim) * 0.01  # vocabulary -> embedding
W2 = np.random.randn(embed_dim, vocab_size) * 0.01  # embedding -> vocabulary

# --- 3. Training pairs (CBOW: context -> middle) ---
window = 2
pairs = []
for i in range(window, len(corpus) - window):
    context = corpus[i - window:i] + corpus[i + 1:i + window + 1]
    target = corpus[i]
    pairs.append((context, target))

print(f'Context: {pairs[0][0]} -> Target: {pairs[0][1]}')
# Context: ['cat', 'sits', 'the', 'mat'] -> Target: on

# --- 4. Training (gradient descent) ---
def softmax(x):
    e = np.exp(x - np.max(x))
    return e / e.sum()

for epoch in range(500):
    loss = 0
    for context_words, target_word in pairs:
        context_idx = [word2idx[w] for w in context_words]
        target_idx = word2idx[target_word]

        # forward: averaged context embeddings -> predict the middle
        hidden = np.mean(W1[context_idx], axis=0)
        output = softmax(hidden @ W2)
        loss -= np.log(output[target_idx] + 1e-8)

        # backward: update the weights
        grad = output.copy()
        grad[target_idx] -= 1
        W2 -= 0.05 * np.outer(hidden, grad)
        grad_hidden = grad @ W2.T
        for idx in context_idx:
            W1[idx] -= 0.05 * grad_hidden / len(context_idx)

# --- 5. Result: embeddings! ---
for word in ["cat", "dog", "sits", "runs"]:
    print(f"{word}: {W1[word2idx[word]].round(3)}")

# --- 6. Cosine similarity ---
def cosine(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b) + 1e-8)

print(f'cat <-> dog: {cosine(W1[word2idx["cat"]], W1[word2idx["dog"]]):.3f}')
print(f'cat <-> sits: {cosine(W1[word2idx["cat"]], W1[word2idx["sits"]]):.3f}')
print(f'cat <-> runs: {cosine(W1[word2idx["cat"]], W1[word2idx["runs"]]):.3f}')

One-hot "cat": [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1.]
Context: ['cat', 'sits', 'the', 'mat'] -> Target: on
cat: [-2.866 -0.264 -0.225 -1.822  2.783]
dog: [-1.133  0.678 -1.537  1.989  2.241]
sits: [ 1.524 -0.023  1.804 -2.355 -0.788]
runs: [ 2.     4.098 -0.315  2.971  0.815]
cat <-> dog: 0.378
cat <-> sits: -0.177
cat <-> runs: -0.407

Notice: cat and dog (0.378) are more similar than cat and runs (-0.407). The network discovered on its own that "cat" and "dog" are animals, because they appear in similar contexts. Zero labels, zero supervision - just text and math.

CBOW vs Skip-gram

	CBOW	Skip-gram
Direction	Context -> Middle word	Middle word -> Context
Speed	Faster	Slower
Rare words	Handles worse	Handles better
Frequent words	Handles better	Handles worse
When to use	Large corpus, frequent words	Small corpus, rare words

According to Mikolov et al.'s original paper: Skip-gram is better for rare words and small datasets. CBOW is faster and better for frequent words.

The magical vector space

After training Word2Vec, each word is a vector (e.g. 300 numbers). And this space has remarkable properties:

Similar words are close:

distance("cat", "dog") < distance("cat", "car")
distance("France", "Germany") < distance("France", "banana")

Analogies work like adding and subtracting vectors:

vector("king") - vector("man") + vector("woman") ≈ vector("queen")

Why? Because the "king" vector encodes many semantic dimensions - in one of them is information about "gender" (man/woman), in another about "power" (monarchy). By subtracting "man" and adding "woman", we change the gender dimension while keeping the rest.

Other analogy examples:

France - Paris + Berlin ≈ Germany
small - smaller + big ≈ bigger
boat - water + air ≈ airplane

This is what we talked about in the first post on the occasion of semantics. Now you see how it works under the hood.

Word2Vec in code

First let's train our own model on a small corpus to see how it works from scratch:

from gensim.models import Word2Vec

corpus = [
    ["cat", "sits", "on", "the", "mat", "and", "sleeps"],
    ["dog", "sits", "on", "the", "rug", "and", "sleeps"],
    ["cat", "runs", "around", "the", "room"],
    ["dog", "runs", "in", "the", "park"],
    ["cat", "and", "dog", "play", "in", "the", "garden"],
    ["cat", "eats", "kibble", "from", "the", "bowl"],
    ["dog", "eats", "kibble", "from", "the", "bowl"],
    ["cat", "catches", "a", "mouse", "in", "the", "house"],
    ["dog", "chases", "the", "cat", "in", "the", "garden"],
    ["bird", "sits", "on", "a", "branch", "of", "the", "tree"],
    ["fish", "swims", "in", "the", "aquarium"],
    ["car", "drives", "on", "the", "road"],
    ["bike", "rides", "on", "the", "path"],
    ["cat", "purrs", "when", "you", "pet", "it"],
    ["dog", "wags", "its", "tail", "when", "it", "sees", "its", "owner"],
]

model = Word2Vec(
    sentences=corpus,
    vector_size=10,   # vector dimension (small = educational)
    window=3,         # context window size
    min_count=1,      # ignore words rarer than N
    sg=0,             # 0 = CBOW, 1 = Skip-gram
    epochs=200,       # how many passes over the data
)

print(model.wv.most_similar("cat", topn=3))
# [('dog', 0.96), ('sleeps', 0.95), ('on', 0.95)]

print(model.wv.similarity("cat", "dog"))    # ~0.96
print(model.wv.similarity("cat", "car"))    # ~0.83

And here's how we use a pretrained model trained on billions of words:

from gensim.downloader import load

model = load("glove-wiki-gigaword-50")

result = model.most_similar(
    positive=["king", "woman"],
    negative=["man"],
    topn=5
)

for word, score in result:
    print(f"{word}: {score:.3f}")

# queen: 0.852
# throne: 0.737
# ...

You can also check the similarity between words:

print(model.similarity("cat", "dog"))    # ~0.92
print(model.similarity("cat", "car"))    # ~0.15

"Cat" and "dog" - similarity 0.92. "Cat" and "car" - 0.15. The model "knows" that a cat is closer to a dog than to a car.

From sparse vectors to dense

Let's make the comparison again, to make it stick:

	BoW / TF-IDF	Word2Vec
Type	Sparse	Dense
Vector length	= vocabulary size (can be 100 000+)	= embedding dimension (usually 100-300)
Values	Mostly zeros	All non-zero
Similarity	Hard to capture	Cosine similarity works great
Context	None	Captured through the context window
Analogies	None	Work (king - man + woman ≈ queen)

Can Word2Vec generate text?

No. Word2Vec creates a map of meanings - it tells you that "cat" is close to "dog", but it can't compose a sentence out of that. It's like a synonym dictionary: you know what's similar, but you won't write a poem.

But... what if we combine Word2Vec with Markov chains? A Markov chain picks the next word based on frequency. And if instead of drawing equally, we make words more similar to the context have a higher chance?

from gensim.models import Word2Vec
import numpy as np
from collections import defaultdict
import random

corpus = [
    "cat sits on the mat and sleeps",
    "dog sits on the rug and sleeps",
    "cat runs around the room",
    "dog runs in the park",
    "cat and dog play in the garden",
    "cat eats kibble from the bowl",
    "dog eats kibble from the bowl",
    "cat catches a mouse in the house",
    "dog chases the cat in the garden",
    "bird sits on a branch of the tree",
    "cat looks through the window and purrs",
    "dog barks at the mailman",
    "cat sleeps all day on the couch",
]

tokenized = [sentence.split() for sentence in corpus]

w2v_model = Word2Vec(sentences=tokenized, vector_size=10, window=3, min_count=1, epochs=200)

# build the transition matrix (like in Markov chains)
transitions = defaultdict(list)
for sentence in tokenized:
    for i in range(len(sentence) - 1):
        transitions[sentence[i]].append(sentence[i + 1])

def generate(start, length=6):
    words = [start]
    for _ in range(length):
        current = words[-1]
        if current not in transitions:
            break

        candidates = transitions[current]

        # averaged vector of the last words = context
        context_vector = np.mean(
            [w2v_model.wv[w] for w in words[-3:] if w in w2v_model.wv],
            axis=0,
        )

        # score candidates by similarity to the context
        scored = []
        for candidate in candidates:
            if candidate in w2v_model.wv:
                similarity = np.dot(context_vector, w2v_model.wv[candidate]) / (
                    np.linalg.norm(context_vector) * np.linalg.norm(w2v_model.wv[candidate]) + 1e-8
                )
                scored.append((candidate, similarity))

        if scored:
            # draw, but with weights - similar words have a higher chance
            weights = [score + 1 for _, score in scored]
            chosen = random.choices([w for w, _ in scored], weights=weights, k=1)[0]
            words.append(chosen)
        else:
            words.append(random.choice(candidates))
    return " ".join(words)

for _ in range(5):
    print(generate("cat"))

cat sits on the mat and sleeps
cat looks through the window and purrs
cat eats kibble from the bowl
cat sleeps all day on the couch
cat catches a mouse in the house

It's not Shakespeare, but the texts are more coherent than pure random Markov. The idea is simple:

1. The Markov chain gives candidates for the next word
2. Word2Vec scores the candidates based on similarity to the context
3. We pick randomly, but with a bias - words that fit better have a higher chance

This is a tiny step toward what today's LLMs do. They also predict the next word, but instead of a simple transition matrix they have multi-layer Transformer networks with an attention mechanism. There, "scoring candidates" is much, much more advanced.

---

Summary - the whole path in one place

Here's our roadmap, from cutting text to the geometry of meanings:

Method	What it does	Context?	What it can't do
Tokenization (BPE)	Cuts text into pieces	No	Doesn't understand what it cuts
BoW / TF-IDF	Counts words, weights by rarity	No	Ignores order
N-grams	Looks at word sequences	Local (2-3 words)	Short context, fast-growing vocabulary
Naive Bayes	Classifies based on probability	No (words "independent")	Doesn't capture dependencies between words
Word2Vec	Turns words into meaning vectors	Yes (context window)	One word = one vector (ignores polysemy)

And the key perspective: from each of these methods, the LLM took something for itself.

• Tokenization (BPE) is the first step of every LLM's pipeline. ChatGPT cuts your text into tokens BEFORE it does anything with it.
• TF-IDF and BoW are the foundations of thinking about text as numbers. Without that idea, that words can be "counted", there would be no representation learning.
• N-grams and Markov chains are the prototype of next-token prediction. That's exactly what an LLM does - except an LLM has a context of thousands of tokens, not 2-3.
• Naive Bayes showed that a probabilistic approach to text works surprisingly well. An LLM is also a probabilistic model - it predicts the probability of the next token.
• Word2Vec is the ancestor of what we today call the "embedding layer" in Transformers. GPT no longer uses Word2Vec as a separate step, but its embedding layer realizes the same idea: token -> vector.

The LLM didn't fall from the sky. It stands on the shoulders of giants.

---

I know this post is a lot, but I wanted the path from "cutting text" to "meaning vectors" to be complete and understandable.

I hope you'll say "aha!" at least a few times. If anything is unclear - let me know in the comments, I'll try to explain. And if you have better Word2Vec analogy examples - even more reason to let me know.

Which method surprised you the most? Did you know that ChatGPT "thinks" in tokens, not words? And that its "thinking" is really a descendant of Markov chains?

See you next time!

---

Sources and interesting links:

If you want to go deeper, here are the materials I used:

• Byte-Pair Encoding tokenization - Hugging Face - a great, step-by-step introduction to BPE with code
• Tokenization algorithms - Hugging Face - an overview of BPE, WordPiece, Unigram, SentencePiece
• BPE Tokenizer From Scratch - Sebastian Raschka - a BPE implementation from scratch in Python, very educational
• Introduction to TF-IDF Vectorization in NLP - CodeSignal - a readable introduction to TF-IDF with examples
• Python for NLP: Developing an Automatic Text Filler using N-grams - StackAbuse - a great article on n-grams with a text generator
• Brief Introduction to N-gram and TF-IDF - Medium - a comparison of TF-IDF vs n-grams with code
• Word Embeddings: CBOW vs Skip-Gram - Baeldung - a clear comparison of both Word2Vec architectures
• NLP 101: Word2Vec - Skip-gram and CBOW - Medium - good pictures and intuition for Word2Vec
• TF-IDF Character N-grams versus Word Embedding-based Models - ACL Anthology - a contrast between classic features and embeddings
• TensorFlow Embedding Projector - an interactive visualization of the vector space (mandatory!)

1. Tokenization quiz answers: 1) "lowest" -> ["low", "est"] (l+o->lo, lo+w->low; "e" and "s" might be merged to "es", then "est"? depends on the merges - with our merges "es" exists, so "lowest" -> ["low", "es", "t"]). 2) "newer" -> ["new", "er"] (n+e->ne, ne+w->new; "er" is not merged in our dictionary). 3) "widower" -> ["w", "i", "d", "o", "w", "er"] or similar - "widow" isn't a merge we have, so it gets split (w+i+d+ow... but "ow" isn't merged either, only "lo"). The exact split depends on which merges made it into the dictionary. ↩