Tip
\nWant to run this code yourself? All the examples from this post are in a ready-made Jupyter notebook: how-computer-reads-text.ipynb
\nIn the two previous posts we built a solid foundation. In the first we took language apart - phonetics, morphology, syntax, semantics, pragmatics. In the second we asked: "OK, but does an LLM even understand what it generates?" and concluded that an LLM is a machine of signs, not a mind.
\nBut one fundamental question remained unanswered: how does a computer even "take" text in?
\nBecause a computer doesn't see letters. It doesn't see words. A computer sees numbers. So how is it that you type "The cat sits on the mat" and ChatGPT somehow processes it? What's the path from text to number, from number to "understanding"?
\nThat path is a fascinating story of evolution - from the simplest word counting, through ever smarter mathematical tricks, to vectors that can capture semantic similarities. Every step on that path was an answer to the limitations of the previous one.
\nThis is the third post in the "Understanding LLM" series. Today we switch the perspective from linguistic and philosophical to technical. But don't worry - there will still be plenty of examples, plenty of "aha!" moments, and zero formulas you can't understand ;-)
\n\nThe narrative of this post: "First you cut the text into pieces, then you count, then you understand." Simple? We'll see ;-) Let's start with the cutting.
\nImagine you're a computer. Someone shows you text:
\n\n\n"Alice has a cat."
\n
What do you see? Letters? Words? No. You see a sequence of bytes: 41 6c 69 63 65 20 68 61 73 20 61 20 63 61 74 2e. Zero idea where one word starts and another ends. Zero idea that "cat" is an animal and not three random characters.
For a computer to do anything with text, it first has to cut it into pieces. And that process is called tokenization.
\nTokenization is step zero. Without it there's no TF-IDF, no n-grams, no embeddings, no LLM. Everything starts with cutting.
\nThe simplest idea: split the text on spaces.
\n"Alice has a cat." -> ["Alice", "has", "a", "cat."]\nDid you notice? "cat." has a period glued to it. That's not the word "cat" - it's the word "cat" with punctuation. And what about texts like these:
\nIn short: splitting on spaces doesn't work. The world is too complicated for such simple rules.
\nIn the first post we met morphemes - the smallest meaningful units. "Unhappiness" is three morphemes: "un-happy-ness". And now note: a token is not the same as a morpheme.
\n\nA token is simply a piece of text determined by the tokenizer. It can be a whole word, a part of a word, or a single character. It depends on the algorithm.
\nNote
\nThree levels of cutting text:
\nAnd here comes BPE (Byte Pair Encoding) - the most popular tokenization algorithm, used by GPT-2, GPT-3, GPT-4, Llama, and many other models.
\nThe idea of BPE can be reduced to one sentence: find the most frequent pair of characters and merge it into one token. And repeat.
\nSounds simple? Because it is simple ;-) Let's see it on an example.
\nWe have a small corpus with five English words and their frequencies:
\n"low" ×10 "lower" ×5 "newest" ×12\n"widest" ×4 "new" ×5\nStep 0: We split everything into single characters:
\nl o w ×10 l o w e r ×5 n e w e s t ×12\nw i d e s t ×4 n e w ×5\nOur starting dictionary is: ["d", "e", "i", "l", "n", "o", "r", "s", "t", "w"] - simply all unique characters.
Step 1: We find the most frequent pair of adjacent characters:
\nWait, let's recount. (n, e): "newest" (12) + "new" (5) = 17. Actually (n,e) = 17 times <- winner!
\nLet's merge (n, e) -> "ne". Our dictionary grows by one token:
\nDictionary: ["d", "e", "i", "l", "n", "o", "r", "s", "t", "w", "ne"]\n\nl o w ×10 l o w e r ×5 ne w e s t ×12\nw i d e s t ×4 ne w ×5\nStep 2: We look for the most frequent pair again:
\nMerge (ne, w) -> "new":
\nDictionary: [..., "ne", "new"]\n\nl o w ×10 l o w e r ×5 new e s t ×12\nw i d e s t ×4 new ×5\nStep 3: The most frequent pair is now (e, s) = 16 times. Merge -> "es":
\nDictionary: [..., "ne", "new", "es"]\n\nl o w ×10 l o w e r ×5 new es t ×12\nw i d es t ×4 new ×5\nStep 4: The most frequent pair is (l, o) = 15 times. Merge -> "lo":
\nDictionary: [..., "ne", "new", "es", "lo"]\n\nlo w ×10 lo w e r ×5 new es t ×12\nw i d es t ×4 new ×5\nStep 5: The most frequent pair is (lo, w) = 15 times. Merge -> "low":
\nDictionary: [..., "ne", "new", "es", "lo", "low"]\n\nlow ×10 lo w e r ×5 new es t ×12\nw i d es t ×4 new ×5\nAnd so on, until we reach the target vocabulary size. In real models:
\nThat's an important question, and the answer is: always. At both stages.
\n\nTraining: first you train the tokenizer on a huge corpus (it learns which pairs to merge). Then you tokenize the ENTIRE training corpus into IDs. The model learns on those IDs.
\nPrompting: when you write to ChatGPT, your text goes through the SAME tokenizer -> IDs -> the model generates new IDs -> a decoder turns them back into text.
\nThe key point: the tokenizer is trained separately, before the model. Then it's "frozen" - it never changes again. GPT-2 has its tokenizer (a 50K vocabulary), GPT-4 has its own (a 100K vocabulary). And that's the reason Polish gets cut up worse in GPT-2 - because GPT-2's tokenizer was trained mainly on English, so few Polish character sequences got merged. The model itself might "understand" Polish better, but the tokenizer already cut the text into tiny pieces.
\nTip
\nAn experiment for you: Go to tiktokenizer.vercel.app, type some Polish text and see how the GPT-2 model cuts it into tokens. You'll see that Polish characters (a, e, l, s, c with diacritics) often take up 2-3 tokens! Because GPT-2 was trained mainly on English, Polish is "exotic" to it.
\nThe key thing: every model has its own tokenizer. The same text can be cut completely differently:
\nimport tiktoken \n\ntext = "Incredibly happy" \n\ngpt2 = tiktoken .get_encoding ("gpt2" )\ngpt4 = tiktoken .get_encoding ("cl100k_base" )\n\nprint ("GPT-2:" , gpt2 .encode (text ))\nprint ("GPT-4:" , gpt4 .encode (text ))\n\nprint ("GPT-2:" , [gpt2 .decode ([t ]) for t in gpt2 .encode (text )])\nprint ("GPT-4:" , [gpt4 .decode ([t ]) for t in gpt4 .encode (text )])GPT-2 will probably cut the text into many small fragments (because it doesn't "know" it well), and GPT-4 will do it much more efficiently (because it has seen more of this text).
\n\nThe example above uses Polish ("Nieprawdopodobnie szczęśliwy" = "Incredibly happy") because it perfectly illustrates how badly a tokenizer trained on one language handles another.
\nThere are models trained specifically on Polish text - and their tokenizers cut Polish much better:
\nfrom transformers import AutoTokenizer \n\ntext = "Nieprawdopodobnie szczęśliwy" \n\nherbert = AutoTokenizer .from_pretrained ("allegro/herbert-base-cased" )\nprint ("HerBERT (Allegro, Polish WordPiece):" , herbert .tokenize (text ))\n\nroberta = AutoTokenizer .from_pretrained ("sdadas/polish-roberta-base-v2" )\nprint ("Polish RoBERTa (Polish SentencePiece):" , roberta .tokenize (text ))HerBERT (Allegro, Polish WordPiece): ['Nie', 'prawdopodobnie</w>', 'szczęśliwy</w>'] -> 3 tokens\nPolish RoBERTa (Polish SentencePiece): ['▁Nie', 'prawdopodobnie', '▁szczęśliwy'] -> 3 tokens\nThree tokens! "prawdopodobnie" and "szczęśliwy" are single tokens for a Polish tokenizer. Because that tokenizer "saw" these words so many times on a Polish corpus that it merged them into whole units.
\nTwo Polish tokenizers worth knowing:
\n| Tokenizer | \nCreator | \nAlgorithm | \nImplemented in | \n
|---|---|---|---|
| HerBERT | \nAllegro | \nWordPiece | \nPolish BERT on the KGR10 corpus | \n
| Polish RoBERTa | \nsdadas | \nSentencePiece (Unigram) | \nPolish RoBERTa on a large corpus | \n
Note
\nWhy do Polish models cut better? Because their tokenizers were trained on Polish text. "Prawdopodobnie" appeared in the Polish corpus thousands of times, so BPE/SentencePiece merged it into one token. GPT-2 saw mostly English, so "prawdopodobnie" never got merged - and it cuts it into pieces. This is also why GPT-4o (trained on a much more multilingual corpus) cuts Polish better than GPT-2 - but still worse than typically Polish models.
\nNow that we understand how BPE works, let's try to train our own tokenizer. We'll use the tokenizers library from HuggingFace (not to be confused with tiktoken):
from tokenizers import Tokenizer \nfrom tokenizers .models import BPE \nfrom tokenizers .trainers import BpeTrainer \nfrom tokenizers .pre_tokenizers import Whitespace \n\ntokenizer = Tokenizer (BPE (unk_token = "[UNK]" ))\ntokenizer .pre_tokenizer = Whitespace ()\n\ntrainer = BpeTrainer (vocab_size = 300 , special_tokens = ["[UNK]" ])\n\ncorpus = [\n "Alice has a cat and a dog" ,\n "The cat sits on the mat" ,\n "The dog runs in the park" ,\n "Alice has a cat and a cat and once more a cat" ,\n "The cat likes milk and the cat likes to sleep" ,\n "The dog likes to run in the park and chase the cat" ,\n "A happy cat is a cat that has lots of milk" ,\n "An incredibly happy dog runs in the park" ,\n "Happy Alice has a cat and a dog and a happy home" ,\n "A home is a place where a happy family lives" ,\n]\n\ntokenizer .train_from_iterator (corpus , trainer )\nprint (f"Vocabulary size: { tokenizer . get_vocab_size () } " )\n\ntest = "Incredibly happy" \nencoding = tokenizer .encode (test )\nprint (f'" { test } " -> { encoding . tokens } ' )Vocabulary size: 130\n"Incredibly happy" -> ['Incredibly', 'happy']\nTwo tokens! Because our small corpus is in English - and "Incredibly" and "happy" appeared often enough for BPE to merge them into single tokens.
\nBut wait - what do those parameters in the code actually mean?
\n| Parameter | \nWhat it does | \nExample | \n
|---|---|---|
vocab_size | \nThe maximum number of tokens in the vocabulary. BPE will merge pairs until the vocabulary reaches this size. The bigger, the longer the tokens (whole words). The smaller, the smaller the pieces (single letters). | \nGPT-2: 50 257, ours: 300 | \n
special_tokens | \nTokens with a special meaning for the model. They're always in the vocabulary, regardless of training. | \n[UNK], <PAD>, <S> (start), </S> (end) | \n
unk_token | \nThe "unknown" token - replaces every character the tokenizer can't recognize. E.g. if an emoji appears in the text and the tokenizer doesn't have it in the vocabulary, it inserts [UNK]. | \n[UNK] = "I don't know what this is" | \n
A simple analogy: vocab_size is the thickness of your dictionary - how many entries fit in it. unk_token is the entry meaning "no such word". And special_tokens are "reserved pages" - they're always in the dictionary, regardless of what you teach the tokenizer.
Tip
\nExperiment: Copy this code, change vocab_size to e.g. 50 and see what happens. You'll see that with a smaller vocabulary the tokenizer cuts words into smaller pieces - because it has less "room" for merges. That's exactly what we talked about: vocabulary size is a hyperparameter, and how finely the text gets cut depends on it.
Warning
\nWhy does this matter? Because an LLM has a token limit in its context window. GPT-3 has 4K tokens, GPT-4 Turbo has 128K. But a "token" is not a "word"! In English ~1 token ~= 0.75 words. In a language poorly served by the tokenizer, it can be ~1 token ~= 0.4 words. So such text "eats up" more tokens and hits the limit faster.
\nBPE isn't the only player in town. Here's a quick comparison:
\n| Algorithm | \nWho uses it | \nHow it works | \nKey difference | \n
|---|---|---|---|
| BPE | \nGPT, Llama, many others | \nMerges the most frequent pair | \nSimple, bottom-up | \n
| WordPiece | \nBERT, DistilBERT, Electra | \nMerges the pair with the highest "score" | \nScore = freq(pair) / (freq(a) x freq(b)) | \n
| Unigram | \nT5, Pegasus, ALBERT | \nStarts from a large vocabulary, removes the least useful | \nProbabilistic, can yield different tokenizations | \n
| SentencePiece | \nMultilingual models | \nBPE or Unigram on raw text (even without spaces!) | \nWorks with languages without spaces (Chinese, Japanese) | \n
The difference between BPE and WordPiece is subtle: BPE simply merges the most frequent pair. WordPiece merges the pair that is most surprising - i.e. one that appears together more often than would follow from the frequency of the individual elements. It's a bit like a relationship that is "more than the sum of its parts" ;-)
\nNote
\nA callback to post 1: Remember when we said Polish morphology is a nightmare for an LLM? 7 cases, inflection, lots of endings... Well, now you see why. The tokenizer doesn't "know" that "domu", "domowi", "domem" are inflections of the same word. To it they're just different character sequences. "Understanding" the relationship between forms - that's something the model has to learn on its own, during training.
\nWe have our merges from the example above: n+e->ne, ne+w->new, e+s->es, l+o->lo, lo+w->low. How will BPE split these new words?
\nOK, we have tokens. But how does the tokenizer know which character pairs to merge? From a corpus! BPE learns the most frequent combinations from text. And to do anything with tokens - count them, find patterns, train a model - we also need a corpus.
\nA corpus is simply a collection of texts. It can be:
\nEach individual text in a corpus we call a document. Simple?
\n\nAnd these are exactly the datasets on which real models are trained - both tokenizers and the models themselves:
\n| Corpus | \nWhat it contains | \nSize | \nFun fact | \n
|---|---|---|---|
| Wikitext | \nArticles from (English) Wikipedia | \n~500 MB | \nA standard benchmark for evaluating language models | \n
| Wiki-40B | \nWikipedia in 59 languages (including Polish!) | \n~40 GB | \nThe first step of many multilingual models | \n
| EuroParl | \nTranscriptions from the EU parliament (21 languages) | \n~2 GB | \nHigh-quality official texts, great for translation | \n
| Common Crawl | \nDumps of web content | \npetabytes (PB!) | \nThe largest publicly available web corpus. Most LLMs use it | \n
| OpenWebText | \nA copy of Reddit links with >3 upvotes | \n~38 GB | \nGPT-2 was trained on it. Filtered "quality" from Reddit | \n
| The Pile | \nA mix of 22 sources (arXiv, GitHub, Wikipedia, books...) | \n~825 GB | \nCreated by EleutherAI as a "do-everything" dataset | \n
| RedPajama | \nAn open replica of LLaMA's training data | \n~1.2 TB | \nThat's why LLaMA is so good - trained on a huge, diverse set | \n
| OSCAR | \nWeb dumps, filtered by language | \n~6.5 TB | \nHas separate subsets for each language, including Polish OSCAR | \n
And Polish corpora? Here are the most important ones:
\nKey observation: the same corpus is used to train both the tokenizer and the model. First you train the tokenizer on the corpus (it learns which character pairs to merge), then you tokenize the whole corpus with that tokenizer, and on those tokens you train the model.
\nNote
\nWhy does this matter? If your corpus has little Polish text (e.g. GPT-2 trained mainly on English), the tokenizer won't merge Polish words, and the model won't learn Polish well. That's why Polish models (HerBERT, Polish RoBERTa) use corpora with a lot of Polish text - e.g. NKJP + Polish OSCAR + Polish Wikipedia.
\nWe have a corpus. Now we want to turn each document into numbers so the computer can do something with it.
\nThe simplest way: Bag-of-Words (BoW) - a bag of words. We count how many times each word appears in the document.
\nExample. We have three short movie reviews:
\nDocument 1: "This movie is great"\nDocument 2: "This movie is terrible"\nDocument 3: "Great movie I recommend"\nWe build a vocabulary of all unique words:
\n{this, movie, is, great, terrible, i, recommend}\nAnd now we turn each document into a vector of counts:
\n| \n | this | \nmovie | \nis | \ngreat | \nterrible | \ni | \nrecommend | \n
|---|---|---|---|---|---|---|---|
| Doc 1 | \n1 | \n1 | \n1 | \n1 | \n0 | \n0 | \n0 | \n
| Doc 2 | \n1 | \n1 | \n1 | \n0 | \n1 | \n0 | \n0 | \n
| Doc 3 | \n0 | \n1 | \n0 | \n1 | \n0 | \n1 | \n1 | \n
Each document is now just a sequence of numbers. The computer is happy.
\nBut we're not entirely ;-). Because notice the problem:
\n\n\nDocument 1: "This movie is great" -> [1, 1, 1, 1, 0, 0, 0]\nDocument 2: "This movie is terrible" -> [1, 1, 1, 0, 1, 0, 0]
\n
These vectors are almost identical! They differ in one position. Yet one says "great" and the other "terrible" - completely different meanings.
\nAnd there's a second problem: "Dog bites man" and "Man bites dog" will give exactly the same BoW vector. BoW completely ignores word order.
\nNote
\nIn inflected languages things look different. In a language with cases, "The dog bites the man" and "The man bites the dog" can look different because the word forms change. But in English, "dog" is always "dog" regardless of its role in the sentence - which is exactly why BoW can't distinguish "dog bites man" from "man bites dog". This is that morphology from the first post that sometimes helps us and sometimes gets in the way ;-)
\nWarning
\nBag-of-Words in a nutshell:
\nfrom sklearn .feature_extraction .text import CountVectorizer \nimport pandas as pd \n\ncorpus = [\n "This movie is great" ,\n "This movie is terrible" ,\n "Great movie I recommend" ,\n]\n\nvectorizer = CountVectorizer ()\nX = vectorizer .fit_transform (corpus )\n\ndf = pd .DataFrame (\n X .toarray (),\n columns = vectorizer .get_feature_names_out (),\n index = ["Doc 1" , "Doc 2" , "Doc 3" ]\n)\nprint (df )Result:
\n great i is movie recommend terrible this\nDoc 1 1 0 1 1 0 0 1\nDoc 2 0 0 1 1 0 1 1\nDoc 3 1 1 0 1 1 0 0\nExactly the same table as above - only now generated by code. Notice: "movie" is everywhere (value 1 in every row), and "terrible" only in Document 2. That's the whole magic of BoW - simple counting.
\nImportant
\nCan BoW generate new texts? No. BoW is only a way of representing text (text -> numbers). It's used to analyze, compare, and classify documents, but it doesn't generate anything new. It also can't predict what word should come after "The cat sits on the...". To generate text, we need something that understands word order and can predict the next one. And that's exactly what we're getting to with n-grams and Markov chains ;-)
\nBoW treats every word the same. But not every word is equally important!
\nThe word "is" appears in almost every English sentence. The word "semiotics" - rather rarely. Which is more informative? Obviously "semiotics" - because if you see it in a document, it tells you much more about its content than "is".
\nTF-IDF (Term Frequency - Inverse Document Frequency) is a way to capture that.
\nRecall our corpus - three short movie reviews:
\nDocument 1: "This movie is great" (4 words)\nDocument 2: "This movie is terrible" (4 words)\nDocument 3: "Great movie I recommend" (3 words)\nLet's compute TF-IDF for the word "great" in Document 1:
\nTF (Term Frequency) - how often the word appears in one document:
\nTF("great", Document 1) = 1 / 4 = 0.25\n(1 occurrence in a 4-word document)
\nIDF (Inverse Document Frequency) - how "rare" the word is across the whole corpus:
\nIDF("great") = log(3 / 2) = 0.176\n(3 documents in the corpus, 2 of them contain "great")
\nTF-IDF = TF x IDF:
\nTF-IDF("great", Document 1) = 0.25 x 0.176 = 0.044\nAnd the word "is", which is EVERYWHERE?
\nIDF("is") = log(3 / 2) = 0.176 (also in 2 of 3 documents)\nIDF("movie") = log(3 / 3) = 0 (in ALL documents!)\n"Movie" gets zero in IDF! Because if a word is in every document, it carries no information that would help tell one document apart from another.
\nTip
\nThe intuition behind TF-IDF in one sentence: A word is important for a given document if it appears there often (high TF) but rarely in other documents (high IDF). In other words: "Are you unique?"
\nTry it yourself. Here's a complete example in Python:
\nfrom sklearn .feature_extraction .text import TfidfVectorizer \nimport pandas as pd \n\ncorpus = [\n "This movie is great" ,\n "This movie is terrible" ,\n "Great movie I recommend" ,\n]\n\nvectorizer = TfidfVectorizer ()\ntfidf_matrix = vectorizer .fit_transform (corpus )\n\ndf = pd .DataFrame (\n tfidf_matrix .toarray (),\n columns = vectorizer .get_feature_names_out (),\n index = ["Doc 1" , "Doc 2" , "Doc 3" ]\n)\nprint (df .round (2 ))The result will look roughly like this:
\n great i is movie recommend terrible this\nDoc 1 0.41 0.0 0.49 0.37 0.0 0.0 0.49\nDoc 2 0.00 0.0 0.46 0.35 0.0 0.58 0.46\nDoc 3 0.41 0.54 0.0 0.41 0.54 0.0 0.00\nNotice: "movie" has low values everywhere (because it's everywhere). "terrible" has a high value only in Document 2 (because it only appears there). TF-IDF works!
\nTip
\nAn experiment for you: Think about your study (or work) notes. Which words would have high TF-IDF? Probably technical terms - "recursion", "entropy", "backpropagation". And which would have low ones? "And", "is", "on", "that" - because they're everywhere.
\nTF-IDF has one super practical application: text search. This is basically how the first search engines worked.
\nThe idea is simple: you turn the user's query into a TF-IDF vector, and look for the document whose vector is most similar to it (so-called cosine similarity).
\nfrom sklearn .feature_extraction .text import TfidfVectorizer \nfrom sklearn .metrics .pairwise import cosine_similarity \nimport numpy as np \n\ncorpus = [\n "This movie is great" ,\n "This movie is terrible" ,\n "Great movie I recommend" ,\n]\n\nvectorizer = TfidfVectorizer ()\ntfidf_matrix = vectorizer .fit_transform (corpus )\n\nquery = "great movie" \nquery_vec = vectorizer .transform ([query ])\n\nsimilarities = cosine_similarity (query_vec , tfidf_matrix )[0 ]\nranked = sorted (zip (corpus , similarities ), key = lambda x : - x [1 ])\nfor i , (doc , sim ) in enumerate (ranked , 1 ):\n print (f' { i } . " { doc } " -> { sim :.3f } ' )Query: "great movie"\n\n1. "Great movie I recommend" -> 0.694 <- best result!\n2. "This movie is great" -> 0.667\n3. "This movie is terrible" -> 0.229\nDocument 3 wins - because "great" and "movie" are high-TF-IDF keywords for it. Document 2 has "movie" but not "great" - so its similarity is low.
\nNote
\nCosine similarity measures the angle between two vectors. If the vectors point in the same direction (similar proportions of words) - the similarity is close to 1. If in opposite directions - close to 0. Don't worry about the math - the intuition is simple: "how similar are these two vectors to each other?"
\nTF-IDF is better than pure BoW, because at least it weights words by importance. But it still treats every word separately. It doesn't know that "do" + "not" + "like" = something completely different than "like" alone.
\nOr an English example that shows the problem perfectly:
\nThe same words! But the order changes everything. BoW and TF-IDF don't see it.
\nAn n-gram is simply a sequence of N consecutive elements (usually words or characters).
\nExample with the sentence "Alice has a cat":
\n| Type | \nN | \nResult | \n
|---|---|---|
| Unigrams | \n1 | \nAlice, has, a, cat | \n
| Bigrams | \n2 | \nAlice has, has a, a cat | \n
| Trigrams | \n3 | \nAlice has a, has a cat | \n
And now the magic: instead of counting single words, we count pairs of words (bigrams). And suddenly:
\nYou can also combine n-grams with TF-IDF! In sklearn it's enough to change one parameter:
\nfrom sklearn .feature_extraction .text import TfidfVectorizer \nimport pandas as pd \n\ncorpus = [\n "This movie is great" ,\n "This movie is terrible" ,\n "Great movie I recommend" ,\n]\n\nvectorizer = TfidfVectorizer (ngram_range = (1 , 2 ))\ntfidf_matrix = vectorizer .fit_transform (corpus )\n\ndf = pd .DataFrame (\n tfidf_matrix .toarray (),\n columns = vectorizer .get_feature_names_out (),\n index = ["Doc 1" , "Doc 2" , "Doc 3" ]\n)\nprint (df .round (2 ))Look what happened - the feature vocabulary grew! Besides single words ("movie", "is", "great") we now have pairs: "movie is", "is great", "is terrible", "great movie", "this movie", "movie recommend". Each pair is a separate column with its own TF-IDF.
\nThanks to this, the model sees that "is great" (Doc 1) and "is terrible" (Doc 2) are different things - because they're different bigrams with different TF-IDF values.
\nNote
\nN-grams are a trade-off. The bigger the N, the more context you capture, but the more data you need. With trigrams you have 3x more combinations than with unigrams. With 4-grams - even more. And you quickly get to the point where most n-grams appear in the corpus only once, which isn't useful.
\nN-grams by themselves are just counts - like BoW, they don't generate text. But if we add transition probabilities to them, we suddenly get something that generates new text.
\nThe idea is simple: for each word we look at what words most often follow it. And we pick the next word probabilistically.
\nLet's take a small corpus:
\n"alice has a cat alice has a dog the cat likes milk alice likes the cat"\nWe count bigrams and transition probabilities:
\n| Word | \nNext word | \nProbability | \n
|---|---|---|
| alice | \nhas | \n2/3 = 67% | \n
| alice | \nlikes | \n1/3 = 33% | \n
| has | \na | \n2/2 = 100% | \n
| a | \ncat | \n2/3 = 67% | \n
| a | \ndog | \n1/3 = 33% | \n
| the | \ncat | \n1/1 = 100% | \n
| cat | \nlikes | \n1/1 = 100% | \n
| likes | \nmilk | \n1/2 = 50% | \n
| likes | \nthe | \n1/2 = 50% | \n
That is exactly a Markov chain - a model in which the probability of the next state depends only on the current state (or the last few).
\n\n"alice" -> most often "has" (67%) -> "a" -> "cat" (67%) or "dog" (33%). So we generate e.g.: "alice has a cat". Or "alice has a dog". Or "alice likes the cat". All these sentences are "new" - they didn't appear as wholes in the corpus - but the model stitched them together from transition probabilities.
\nBefore we get to the code, let's trace the whole process on our fingers. We have the same corpus:
\n"alice has a cat alice has a dog the cat likes milk alice likes the cat"\nStep 1: Build the trigram dictionary. We slide a "window" of 3 words and look at what follows:
\n| Window (trigram) | \nNext word | \n
|---|---|
| alice has a | \ncat | \n
| has a cat | \nalice | \n
| a cat alice | \nhas | \n
| cat alice has | \na | \n
| alice has a | \ndog | \n
| has a dog | \nthe | \n
| a dog the | \ncat | \n
| dog the cat | \nlikes | \n
| the cat likes | \nmilk | \n
| cat likes milk | \nalice | \n
| likes milk alice | \nlikes | \n
| milk alice likes | \nthe | \n
Each trigram has exactly one possible next word (because our corpus is tiny).
\nStep 2: Generate. We start from the starter trigram ("alice", "has", "a"):
Important
\nThis starter trigram is our "prompt"! Just like in ChatGPT you type text and the model continues - here we type "alice has a" and the generator picks the next words. The only difference is scale: ChatGPT has a context of thousands of tokens, and we have 3 words.
\nStart: alice has a\nStep 1: alice has a [cat] <- after (alice,has,a) comes "cat"\nStep 2: alice has a cat [alice] <- after (has,a,cat) comes "alice"\nStep 3: alice has a cat alice [has] <- after (a,cat,alice) comes "has"\nStep 4: ... cat alice has [a] <- after (cat,alice,has) comes "a"\nStep 5: ... alice has a [dog] <- after (alice,has,a) comes "dog"\nStep 6: ... has a dog [the] <- after (has,a,dog) comes "the"\nStep 7: ... a dog the [cat] <- after (a,dog,the) comes "cat"\nStep 8: ... dog the cat [likes] <- after (dog,the,cat) comes "likes"\nStep 9: ... the cat likes [milk] <- after (the,cat,likes) comes "milk"\nStep 10: STOP <- trigram (cat,likes,milk) -> "alice"... it loops back\nResult: "alice has a cat alice has a dog the cat likes milk" - basically our corpus, "re-glued" from the inside. On a larger corpus the result would be different every time.
Here's a complete (and genuinely short!) text generator in Python:
\nimport random \n\ncorpus = "alice has a cat alice has a dog the cat likes milk alice likes the cat" \ntokens = corpus .split ()\n\ntrigrams = {}\nfor i in range (len (tokens ) - 3 ):\n key = (tokens [i ], tokens [i + 1 ], tokens [i + 2 ])\n next_word = tokens [i + 3 ]\n if key not in trigrams :\n trigrams [key ] = []\n trigrams [key ].append (next_word )\n\ncurrent = ("alice" , "has" , "a" ) # <- this is our "prompt"! \noutput = list (current )\n\nfor _ in range (10 ):\n if tuple (output [- 3 :]) not in trigrams :\n break \n possibilities = trigrams [tuple (output [- 3 :])]\n output .append (random .choice (possibilities ))\n\nprint (" " .join (output ))Possible output: "alice has a cat alice has a dog the cat likes milk alice likes the cat"
Note
\nWhy does it come out the same every time? Because our corpus is so small that most trigrams have only one possible next word. random.choice has nothing to choose from! On a larger corpus (e.g. all of English Wikipedia) the same code would generate different text every time - because almost every trigram would have several possible continuations with different probabilities. And that's exactly the moment when generation becomes interesting.
Nothing spectacular, right? But that's because our corpus is microscopic. On a real corpus (e.g. all of Wikipedia) a trigram generator can produce sentences that sound sensible, even though they never appeared before.
\nImportant
\nMarkov chains are the first "language models"! Predicting the next word based on context - that is EXACTLY what ChatGPT does. Of course GPT uses a much more advanced method (Transformer + attention over thousands of context tokens), but the fundamental idea is the same: the probability of the next token. An LLM is a descendant of Markov chains. On steroids ;-)
\nMarkov chains are great for generating text. But in practice we often want something else: assign a text to a category.
\nAnd here comes Naive Bayes - one of the simplest yet most useful text classification algorithms.
\nWithout formulas, just an example:
\nImagine a colleague comes up to you and says: "I'm coughing." What's the probability he has a cold?
\nIt depends! If it's November and everyone in the office is sick - high. If it's July and he coughs once - low.
\nBayes' theorem is a formal way of thinking about such situations: we update our knowledge based on new evidence.
\nAnd now let's translate that to text. The question is:
\n\n\n\n"If a document contains the word 'viagra', what's the probability it's spam?"
\n
We have a small set of emails:
\n| Content | \nLabel | \n|
|---|---|---|
| 1 | \n"Buy cheap viagra now" | \nSpam | \n
| 2 | \n"Free viagra offer" | \nSpam | \n
| 3 | \n"Meeting tomorrow at 10" | \nNot-spam | \n
| 4 | \n"Send me the report tomorrow" | \nNot-spam | \n
A new email arrives: "Viagra meeting tomorrow". Spam or not?
\nNaive Bayes computes:
\nP(Spam | "viagra meeting tomorrow") ∝ 0.5 x 1.0 x 0.0 x ... = 0!
\nP(Not-spam | "viagra meeting tomorrow") ∝ 0.5 x 0.0 x 1.0 x ... = 0!
\nOops. The presence of "tomorrow" (a not-spam word) zeroed out spam, and the presence of "viagra" (a spam word) zeroed out not-spam. This particular email is tricky ;-)
\nIn practice, so-called Laplace smoothing is used (we add 1 to each count) to avoid zeroing:
\nNow: P(Spam | email) ∝ 0.5 x 0.214 x 0.048 x 0.048 ≈ 0.000246\nP(Not-spam | email) ∝ 0.5 x 0.048 x 0.143 x 0.143 ≈ 0.000490
\nNot-spam wins! Because "tomorrow" and "meeting" strongly indicate a normal email.
\nBecause we assume words are independent. That is: the probability of "viagra" appearing doesn't depend on whether "cheap" also appears. Of course that's not true! "Viagra" and "cheap" appear together more often than by chance. But the model still works surprisingly well...
\nIt's a bit like assuming the weather in London doesn't depend on the weather in Paris. Of course it does a bit! But if you want to quickly estimate whether you need an umbrella, the independence assumption gives you a reasonably good answer ;-)
\nNow that we understand the math, let's do it in a few lines of code. The scikit-learn library has a ready Naive Bayes classifier:
CountVectorizer - turns text into a word-count vector (our BoW from the previous section)MultinomialNB - that's Naive Bayes, the "multinomial" variant (because it counts word probabilities), with alpha=1.0 i.e. Laplace smoothingfrom sklearn .naive_bayes import MultinomialNB \nfrom sklearn .feature_extraction .text import CountVectorizer \n\nemails = [\n "Buy cheap viagra now" ,\n "Free viagra offer" ,\n "Meeting tomorrow at 10" ,\n "Send me the report tomorrow" ,\n "Cheap pills online buy now" ,\n "Report from yesterday send" ,\n]\nlabels = [1 , 1 , 0 , 0 , 1 , 0 ] # 1=spam, 0=not-spam \n\nvectorizer = CountVectorizer ()\nX = vectorizer .fit_transform (emails )\n\nclassifier = MultinomialNB (alpha = 1.0 )\nclassifier .fit (X , labels )\n\nnew_email = vectorizer .transform (["Viagra meeting tomorrow" ])\nprediction = classifier .predict (new_email )\nprint ("Spam!" if prediction [0 ] == 1 else "Not spam." )Tip
\nAn experiment for you: Create 5 positive sentences about a movie ("This movie was amazing!", "Great action!", ...) and 5 negative ones ("Boring as watching paint dry", "Waste of time", ...). Write down the words that appear ONLY in positive and ONLY in negative. That's the intuition of Naive Bayes - every word "votes" for one of the categories.
\nWarning
\nLimitation: Naive Bayes sees words but not context. "The movie is AMAZING... just kidding, awful" - the model will see "amazing" and say "positive". It doesn't get irony or complex constructions. We need something that understands relationships between words. And here we enter the world of vectors...
\nAll the methods we've met so far have one problem in common: they treat words as discrete, independent entities. In BoW "cat" is at position 47 in the vector, "dog" at position 138. There's no relationship between them.
\nBut WE know that "cat" and "dog" are similar - both are pets. "Cat" and "car" - completely different. How do we make a computer "know" that too?
\nThe answer from 2013, from Tomáš Mikolov's team at Google: let's turn words into points in a high-dimensional space. Semantically similar words will be close to each other. Different words - far apart.
\nThat's Word2Vec. And it's a breakthrough.
\nBefore Word2Vec, the standard was so-called one-hot encoding - each word is a vector with a 1 at its position and 0 everywhere else:
\n"cat" -> [0, 0, 0, 1, 0, 0, ...] (1 at position 3)\n"dog" -> [0, 0, 1, 0, 0, 0, ...] (1 at position 2)\n"car" -> [1, 0, 0, 0, 0, 0, ...] (1 at position 0)\nThe problem? Every word is the same distance from every other word. The distance between "cat" and "dog" is the same as between "cat" and "car". Zero information about meaning.
\n\nWord2Vec is based on a brilliant linguistic intuition: words that appear in similar contexts have similar meanings.
\nIf you see: "___ runs in the park and chases pigeons", what fits there? Dog? Cat? Rather not "car" or "democracy". Context defines the word.
\n\n"I feed ___ kibble" - both dog and cat fit. That means the context of these words is partly shared. And that's exactly what Word2Vec exploits.
\nRemember Saussure from the second post? Meaning is relational. "Cat" means what it means because it is NOT "dog", it is NOT "house". Word2Vec is the mathematical implementation of that idea!
\nWord2Vec has two variants. Let's see both:
\nCBOW (Continuous Bag of Words): from context we guess the middle word.
\n"The cat ___ on the mat" -> sits? lies? sleeps?
Skip-gram: from the middle word we guess the context.
\n"sits" -> The? cat? on? the? mat?
How does it work inside? A neural network with one hidden layer:
\nSounds complicated? Let's do it in code - from scratch, without any ML libraries:
\nimport numpy as np \n\nnp .random .seed (42 )\n\n# --- 1. Vocabulary and one-hot encoding --- \ncorpus = "cat sits on the mat and sleeps dog sits on the rug and sleeps cat runs around the room dog runs in the park" .split ()\nvocab = list (set (corpus ))\nword2idx = {w : i for i , w in enumerate (vocab )}\nvocab_size = len (vocab )\n\ndef one_hot (word ):\n vec = np .zeros (vocab_size )\n vec [word2idx [word ]] = 1 \n return vec \n\nprint (f'One-hot "cat": { one_hot ( "cat" ) } ' )\n# [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1.] - only a single one! \n\n# --- 2. Weight matrix (these will be our embeddings) --- \nembed_dim = 5 # in real Word2Vec this is 100-300 \nW1 = np .random .randn (vocab_size , embed_dim ) * 0.01 # vocabulary -> embedding \nW2 = np .random .randn (embed_dim , vocab_size ) * 0.01 # embedding -> vocabulary \n\n# --- 3. Training pairs (CBOW: context -> middle) --- \nwindow = 2 \npairs = []\nfor i in range (window , len (corpus ) - window ):\n context = corpus [i - window :i ] + corpus [i + 1 :i + window + 1 ]\n target = corpus [i ]\n pairs .append ((context , target ))\n\nprint (f'Context: { pairs [ 0 ][ 0 ] } -> Target: { pairs [ 0 ][ 1 ] } ' )\n# Context: ['cat', 'sits', 'the', 'mat'] -> Target: on \n\n# --- 4. Training (gradient descent) --- \ndef softmax (x ):\n e = np .exp (x - np .max (x ))\n return e / e .sum ()\n\nfor epoch in range (500 ):\n loss = 0 \n for context_words , target_word in pairs :\n context_idx = [word2idx [w ] for w in context_words ]\n target_idx = word2idx [target_word ]\n\n # forward: averaged context embeddings -> predict the middle \n hidden = np .mean (W1 [context_idx ], axis = 0 )\n output = softmax (hidden @ W2 )\n loss -= np .log (output [target_idx ] + 1e-8 )\n\n # backward: update the weights \n grad = output .copy ()\n grad [target_idx ] -= 1 \n W2 -= 0.05 * np .outer (hidden , grad )\n grad_hidden = grad @ W2 .T \n for idx in context_idx :\n W1 [idx ] -= 0.05 * grad_hidden / len (context_idx )\n\n# --- 5. Result: embeddings! --- \nfor word in ["cat" , "dog" , "sits" , "runs" ]:\n print (f" { word } : { W1 [ word2idx [ word ]]. round ( 3 ) } " )\n\n# --- 6. Cosine similarity --- \ndef cosine (a , b ):\n return np .dot (a , b ) / (np .linalg .norm (a ) * np .linalg .norm (b ) + 1e-8 )\n\nprint (f'cat <-> dog: { cosine ( W1 [ word2idx [ "cat" ]], W1 [ word2idx [ "dog" ]]):.3f } ' )\nprint (f'cat <-> sits: { cosine ( W1 [ word2idx [ "cat" ]], W1 [ word2idx [ "sits" ]]):.3f } ' )\nprint (f'cat <-> runs: { cosine ( W1 [ word2idx [ "cat" ]], W1 [ word2idx [ "runs" ]]):.3f } ' )One-hot "cat": [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1.]\nContext: ['cat', 'sits', 'the', 'mat'] -> Target: on\ncat: [-2.866 -0.264 -0.225 -1.822 2.783]\ndog: [-1.133 0.678 -1.537 1.989 2.241]\nsits: [ 1.524 -0.023 1.804 -2.355 -0.788]\nruns: [ 2. 4.098 -0.315 2.971 0.815]\ncat <-> dog: 0.378\ncat <-> sits: -0.177\ncat <-> runs: -0.407\nNotice: cat and dog (0.378) are more similar than cat and runs (-0.407). The network discovered on its own that "cat" and "dog" are animals, because they appear in similar contexts. Zero labels, zero supervision - just text and math.
\nNote
\nThis is all of Word2Vec in ~30 lines. Real Word2Vec adds negative sampling (because softmax over 50 000 words is slow) and optimizations, but the principle is exactly the same.
\nNote
\nThe "fake task": What we care about is not what the network predicts, but the hidden layer weights. We train the network on an "artificial" task of predicting context, but what we want to extract is the matrix W - our embeddings. It's a bit like training someone to solve crosswords not so they solve crosswords well, but so they expand their vocabulary ;-)
\n| \n | CBOW | \nSkip-gram | \n
|---|---|---|
| Direction | \nContext -> Middle word | \nMiddle word -> Context | \n
| Speed | \nFaster | \nSlower | \n
| Rare words | \nHandles worse | \nHandles better | \n
| Frequent words | \nHandles better | \nHandles worse | \n
| When to use | \nLarge corpus, frequent words | \nSmall corpus, rare words | \n
According to Mikolov et al.'s original paper: Skip-gram is better for rare words and small datasets. CBOW is faster and better for frequent words.
\nAfter training Word2Vec, each word is a vector (e.g. 300 numbers). And this space has remarkable properties:
\nSimilar words are close:
\ndistance("cat", "dog") < distance("cat", "car")\ndistance("France", "Germany") < distance("France", "banana")\nAnalogies work like adding and subtracting vectors:
\nvector("king") - vector("man") + vector("woman") ≈ vector("queen")\nWhy? Because the "king" vector encodes many semantic dimensions - in one of them is information about "gender" (man/woman), in another about "power" (monarchy). By subtracting "man" and adding "woman", we change the gender dimension while keeping the rest.
\nOther analogy examples:
\nFrance - Paris + Berlin ≈ Germany\nsmall - smaller + big ≈ bigger\nboat - water + air ≈ airplane\nThis is what we talked about in the first post on the occasion of semantics. Now you see how it works under the hood.
\nFirst let's train our own model on a small corpus to see how it works from scratch:
\nfrom gensim .models import Word2Vec \n\ncorpus = [\n ["cat" , "sits" , "on" , "the" , "mat" , "and" , "sleeps" ],\n ["dog" , "sits" , "on" , "the" , "rug" , "and" , "sleeps" ],\n ["cat" , "runs" , "around" , "the" , "room" ],\n ["dog" , "runs" , "in" , "the" , "park" ],\n ["cat" , "and" , "dog" , "play" , "in" , "the" , "garden" ],\n ["cat" , "eats" , "kibble" , "from" , "the" , "bowl" ],\n ["dog" , "eats" , "kibble" , "from" , "the" , "bowl" ],\n ["cat" , "catches" , "a" , "mouse" , "in" , "the" , "house" ],\n ["dog" , "chases" , "the" , "cat" , "in" , "the" , "garden" ],\n ["bird" , "sits" , "on" , "a" , "branch" , "of" , "the" , "tree" ],\n ["fish" , "swims" , "in" , "the" , "aquarium" ],\n ["car" , "drives" , "on" , "the" , "road" ],\n ["bike" , "rides" , "on" , "the" , "path" ],\n ["cat" , "purrs" , "when" , "you" , "pet" , "it" ],\n ["dog" , "wags" , "its" , "tail" , "when" , "it" , "sees" , "its" , "owner" ],\n]\n\nmodel = Word2Vec (\n sentences = corpus ,\n vector_size = 10 , # vector dimension (small = educational) \n window = 3 , # context window size \n min_count = 1 , # ignore words rarer than N \n sg = 0 , # 0 = CBOW, 1 = Skip-gram \n epochs = 200 , # how many passes over the data \n)\n\nprint (model .wv .most_similar ("cat" , topn = 3 ))\n# [('dog', 0.96), ('sleeps', 0.95), ('on', 0.95)] \n\nprint (model .wv .similarity ("cat" , "dog" )) # ~0.96 \nprint (model .wv .similarity ("cat" , "car" )) # ~0.83 Note
\nNotice: our corpus is TINY (15 sentences), so the results are far from ideal - "cat" and "car" have as much as 0.83 similarity, which makes no sense. But the model correctly puts "dog" closest to "cat"! On real corpora (billions of sentences) these vectors become very accurate.
\nAnd here's how we use a pretrained model trained on billions of words:
\nfrom gensim .downloader import load \n\nmodel = load ("glove-wiki-gigaword-50" )\n\nresult = model .most_similar (\n positive = ["king" , "woman" ],\n negative = ["man" ],\n topn = 5 \n)\n\nfor word , score in result :\n print (f" { word } : { score :.3f } " )\n\n# queen: 0.852 \n# throne: 0.737 \n# ... You can also check the similarity between words:
\nprint (model .similarity ("cat" , "dog" )) # ~0.92 \nprint (model .similarity ("cat" , "car" )) # ~0.15 "Cat" and "dog" - similarity 0.92. "Cat" and "car" - 0.15. The model "knows" that a cat is closer to a dog than to a car.
\nTip
\nExperiment: Go to the TensorFlow Embedding Projector - it's an interactive visualization of the vector space. You can type words and see what's close. It's ONE of the most beautiful visualizations in all of ML. Seriously, check it out!
\nWarning
\nWatch out for bias: Word2Vec learns from data. If in the training data "programmer" appears more often near "man" than "woman" - the model will pick that up. A famous example: vector("programmer") - vector("man") + vector("woman") ≈ "homemaker". That's a bias hidden in the data that the model unreflectively reproduces. Remember the semiosphere from the second post? The semiosphere isn't neutral - and neither is the training data.
Let's make the comparison again, to make it stick:
\n\n| \n | BoW / TF-IDF | \nWord2Vec | \n
|---|---|---|
| Type | \nSparse | \nDense | \n
| Vector length | \n= vocabulary size (can be 100 000+) | \n= embedding dimension (usually 100-300) | \n
| Values | \nMostly zeros | \nAll non-zero | \n
| Similarity | \nHard to capture | \nCosine similarity works great | \n
| Context | \nNone | \nCaptured through the context window | \n
| Analogies | \nNone | \nWork (king - man + woman ≈ queen) | \n
No. Word2Vec creates a map of meanings - it tells you that "cat" is close to "dog", but it can't compose a sentence out of that. It's like a synonym dictionary: you know what's similar, but you won't write a poem.
\nBut... what if we combine Word2Vec with Markov chains? A Markov chain picks the next word based on frequency. And if instead of drawing equally, we make words more similar to the context have a higher chance?
\nfrom gensim .models import Word2Vec \nimport numpy as np \nfrom collections import defaultdict \nimport random \n\ncorpus = [\n "cat sits on the mat and sleeps" ,\n "dog sits on the rug and sleeps" ,\n "cat runs around the room" ,\n "dog runs in the park" ,\n "cat and dog play in the garden" ,\n "cat eats kibble from the bowl" ,\n "dog eats kibble from the bowl" ,\n "cat catches a mouse in the house" ,\n "dog chases the cat in the garden" ,\n "bird sits on a branch of the tree" ,\n "cat looks through the window and purrs" ,\n "dog barks at the mailman" ,\n "cat sleeps all day on the couch" ,\n]\n\ntokenized = [sentence .split () for sentence in corpus ]\n\nw2v_model = Word2Vec (sentences = tokenized , vector_size = 10 , window = 3 , min_count = 1 , epochs = 200 )\n\n# build the transition matrix (like in Markov chains) \ntransitions = defaultdict (list )\nfor sentence in tokenized :\n for i in range (len (sentence ) - 1 ):\n transitions [sentence [i ]].append (sentence [i + 1 ])\n\ndef generate (start , length = 6 ):\n words = [start ]\n for _ in range (length ):\n current = words [- 1 ]\n if current not in transitions :\n break \n\n candidates = transitions [current ]\n\n # averaged vector of the last words = context \n context_vector = np .mean (\n [w2v_model .wv [w ] for w in words [- 3 :] if w in w2v_model .wv ],\n axis = 0 ,\n )\n\n # score candidates by similarity to the context \n scored = []\n for candidate in candidates :\n if candidate in w2v_model .wv :\n similarity = np .dot (context_vector , w2v_model .wv [candidate ]) / (\n np .linalg .norm (context_vector ) * np .linalg .norm (w2v_model .wv [candidate ]) + 1e-8 \n )\n scored .append ((candidate , similarity ))\n\n if scored :\n # draw, but with weights - similar words have a higher chance \n weights = [score + 1 for _ , score in scored ]\n chosen = random .choices ([w for w , _ in scored ], weights = weights , k = 1 )[0 ]\n words .append (chosen )\n else :\n words .append (random .choice (candidates ))\n return " " .join (words )\n\nfor _ in range (5 ):\n print (generate ("cat" ))cat sits on the mat and sleeps\ncat looks through the window and purrs\ncat eats kibble from the bowl\ncat sleeps all day on the couch\ncat catches a mouse in the house\nIt's not Shakespeare, but the texts are more coherent than pure random Markov. The idea is simple:
\nThis is a tiny step toward what today's LLMs do. They also predict the next word, but instead of a simple transition matrix they have multi-layer Transformer networks with an attention mechanism. There, "scoring candidates" is much, much more advanced.
\nImportant
\nThe key difference: Word2Vec gives each word ONE vector. But "bank" in "bank account" and "bank" of a river are completely different meanings! Word2Vec doesn't distinguish them - that's why in the next posts we get into Transformers, where context changes the meaning of every word.
\nHere's our roadmap, from cutting text to the geometry of meanings:
\n| Method | \nWhat it does | \nContext? | \nWhat it can't do | \n
|---|---|---|---|
| Tokenization (BPE) | \nCuts text into pieces | \nNo | \nDoesn't understand what it cuts | \n
| BoW / TF-IDF | \nCounts words, weights by rarity | \nNo | \nIgnores order | \n
| N-grams | \nLooks at word sequences | \nLocal (2-3 words) | \nShort context, fast-growing vocabulary | \n
| Naive Bayes | \nClassifies based on probability | \nNo (words "independent") | \nDoesn't capture dependencies between words | \n
| Word2Vec | \nTurns words into meaning vectors | \nYes (context window) | \nOne word = one vector (ignores polysemy) | \n
And the key perspective: from each of these methods, the LLM took something for itself.
\nThe LLM didn't fall from the sky. It stands on the shoulders of giants.
\nImportant
\nWhat's in the next post? We have word vectors - but how do we build something out of them that understands whole sentences? In the next post we'll walk the whole path from a single neuron (perceptron, 1958), through MLP, RNN with the forgetting problem, to LSTM with memory gates. The evolution of architectures that prepared the ground for Transformers and ChatGPT. Stay tuned!
\nI know this post is a lot, but I wanted the path from "cutting text" to "meaning vectors" to be complete and understandable.
\nI hope you'll say "aha!" at least a few times. If anything is unclear - let me know in the comments, I'll try to explain. And if you have better Word2Vec analogy examples - even more reason to let me know.
\nWhich method surprised you the most? Did you know that ChatGPT "thinks" in tokens, not words? And that its "thinking" is really a descendant of Markov chains?
\nSee you next time!
\nSources and interesting links:
\nIf you want to go deeper, here are the materials I used:
\nTokenization quiz answers: 1) "lowest" -> ["low", "est"] (l+o->lo, lo+w->low; "e" and "s" might be merged to "es", then "est"? depends on the merges - with our merges "es" exists, so "lowest" -> ["low", "es", "t"]). 2) "newer" -> ["new", "er"] (n+e->ne, ne+w->new; "er" is not merged in our dictionary). 3) "widower" -> ["w", "i", "d", "o", "w", "er"] or similar - "widow" isn't a merge we have, so it gets split (w+i+d+ow... but "ow" isn't merged either, only "lo"). The exact split depends on which merges made it into the dictionary. ↩
\nLLMs (Large Language Models, i.e. ChatGPT, Claude, Gemini and friends) have been with us for a while now, and I decided to dig deeper into the topic, and one big question came up: but how does it even work that a model "understands" language? After all, underneath it's just "math and statistics", right? Well... yes and no ;-)
\nAnd then I stumbled into linguistics. And it turned out that to truly understand what an LLM does with language, you first need to understand what language is made of. And since a programmer trying to read about linguistics is a rather funny picture, I thought: I'll share it with you ;-)
\nThis is the first post in a series in which we'll survey different levels of language analysis and see how it all connects to LLMs. In today's post we'll build the foundation - we'll get to know the five main layers of language and see why each of them matters for language models.
\nBefore we get into the details, imagine language as a sort of... onion. Or a layer cake, if you prefer sweet metaphors :D
\nEach layer is a different level on which language "works". From the lowest - sounds - to the highest - what we really meant to say, even when we said something completely different.
\n\nFive layers, five ways in which language gets "analyzed". And note - an LLM has to handle each of these levels to come across as a sensible conversational partner. Of course it doesn't do this consciously - but the structures it learns during training somehow "catch" these layers.
\nNote
\nThe five layers of language in a nutshell:
\nOK, let's get to it. We start at the bottom of our onion ;-)
\nPhonetics studies speech sounds as physical phenomena. Put simply: how we produce sounds, how they travel through the air, and how we receive them.
\nIt has three branches:
\n| Branch | \nWhat it studies | \nExample | \n
|---|---|---|
| Articulatory | \nHow the speech organs produce sounds | \nWhere is your tongue when you say "sh"? | \n
| Acoustic | \nThe physical properties of sound waves | \nFrequency, amplitude | \n
| Auditory | \nHow the ear and brain receive sounds | \nHow the cochlea turns waves into a nerve signal | \n
Phonology is a level up. It's not interested in sounds as physical phenomena, but in how sounds are organized in a particular language.
\nKey concept: the phoneme - the smallest unit of sound that distinguishes the meaning of words.
\nA simple example in English:
\nOr in Polish:
\nTip
\nExperiment: Say "cat" out loud, once as a plain statement ("My cat sleeps."), and once as a surprised question ("Cat?!" with raised eyebrows). The same word, but intonation, stress, and melody completely change what the listener "hears". That is phonology in action - sounds + their organization in a system.
\nRight - an LLM gets text, not sound. It has never heard pronunciation. And yet...
\nSo the model somehow absorbs phonological information from text data alone. A bit like never having seen the ocean, but having read so many books about it that you can describe it ;-)
\nBut of course there are limitations. A text-based LLM can't distinguish homophones (words that sound the same but mean different things) based on phonetics - it handles them purely at the semantic level (more on that in a moment).
\nAnd a fun fact: models like Whisper (speech-to-text from OpenAI) already connect phonetics with text. But that's a topic for a separate post in this series ;-)
\nWarning
\nParadox: An LLM has never heard a single sound, yet from training data it "knows" about rhymes, wordplay, and sound patterns. It's a bit like a person deaf from birth who can write rhyming poetry - because they've read millions of them.
\nMorphology studies the structure of words - how they're assembled from smaller, meaningful pieces called morphemes.
\nA morpheme is the smallest unit of language that carries meaning. It can't be divided into anything smaller that would still carry meaning.
\nAn example from English, often used in textbooks:
\nunhappiness = un + happy + ness\n ↑ ↑ ↑\n "not" "happy" makes a noun\nOr in Polish:
\nnieszczęśliwy = nie + szczęśliw + y\n ↑ ↑ ↑\n "not" "happy/fortune" masculine gender\nOr even more:
\nnieprzystojny = nie + przy + stoj + ny\n "not" + "at" + "stand" + adjective suffix\nEach of these pieces has its own meaning. Those are morphemes.
\n| Type | \nDescription | \nExamples | \n
|---|---|---|
| Free | \nCan stand alone as a word | \ncat, house, runs | \n
| Bound | \nMust be attached to another morpheme | \nun-, -ness, -s, -ed | \n
And among bound morphemes we have affixes:
\nThis is an important distinction:
\nInflection - changes the grammatical form, but doesn't create a new word:
\nDerivation (word formation) - creates a new word, often changing the part of speech:
\nWarning
\nPolish morphology is a nightmare for an LLM! The Polish language is highly inflectional - we have 7 cases, 2 numbers, 3 genders, verb aspects, and tons of exceptions. For comparison: in English, declining a noun means at most adding -s. In Polish? "Dom, domu, domowi, dom, domem, domu, domie, domy, domow..." and so on :D
\nHere's a fun fact. An LLM doesn't "know" what morphemes are. It uses BPE tokenization (Byte Pair Encoding), which splits text into tokens - but tokens are not the same as morphemes.
\n\nThe BPE tokenizer splits text based on frequency of occurrence in the training data, not on linguistic structure. Sometimes that overlaps with morphemes, sometimes it doesn't.
\nThat means the model doesn't learn morphology "directly" - it learns it indirectly, through statistics. And somehow it works ;-)
\nTip
\nA challenge for you: Can you break these words into morphemes?
\nSyntax is the set of rules that determine how words are combined into sentences. Thanks to syntax we know that "The cat sits on the mat" is a valid sentence, and "The cat the on mat sits" is gibberish.
\nSimple? Well... more or less ;-)
\nDifferent languages have different word-order rules:
\n| Language | \nBasic order | \nExample | \n
|---|---|---|
| English | \nSVO (Subject-Verb-Object) | \nThe cat sits on the mat | \n
| Polish | \n"Flexible" SVO, but... | \nKot siedzi na macie / Na macie siedzi kot / Siedzi kot na macie | \n
| Japanese | \nSOV | \n猫がマットの上に座っている | \n
| Latin | \nFree (because endings say everything) | \nFelix in tapete sedet | \n
Polish is nice because we have quite a lot of freedom in word order. While in English "On the mat sits the cat" sounds poetic or unnatural, for us it's a normal sentence ;-)
\nSentences have a hierarchical structure - words group into phrases, and phrases into larger phrases. You can draw it as a tree:
\n\nThat's the simple sentence "The cat sits on the mat" and it already has its hierarchy! Now imagine sentences with subordinate clauses, participles, adverbials...
\nThis is a famous example from Noam Chomsky.
\nThe sentence is syntactically correct - it has a subject, a predicate, a proper structure. But semantically it's complete nonsense - ideas can't be green or sleep.
\nAnd that's exactly the proof that syntax and semantics are two different levels. You can have perfect syntax and zero sense ;-)
\nImportant
\nSyntax is an LLM's superpower. The model learns to predict the next token based on a huge amount of text data. In the process it absorbs syntactic patterns - it knows that after "The cat sits on..." we expect a noun, and after "Quickly..." a verb. This is the foundation of why an LLM generates grammatically correct sentences.
\nTest yourself - which sentences are syntactically correct?1
\nIf syntax asks "how is this built?", semantics asks: "what does it mean?"
\nSemantics studies the meaning of words, phrases, and sentences. And right away it turns out this isn't simple at all.
\nA classic English example: bank
\nThe same word, four completely different meanings. How does an LLM know which one is meant? From context. And that's exactly what it does well - because it has seen millions of sentences where "bank" appeared in different contexts.
\nOr another classic - polysemy within a single sentence:
\n\n\n"I can can the canning of cans."
\n
The word "can" three times, three different functions: can (be able to), can (the verb, to put in a can), can (the noun, a container). Context changes everything ;-)
\nAlmost. But not quite:
\nSemantics studies these subtle differences - both denotation (literal meaning) and connotation (associations, emotional coloring).
\nHere enters the concept of word embeddings. In broad terms: each word is represented as a vector - a sequence of numbers - in a high-dimensional space. Words with similar meanings are "close" to each other in that space.
\nThe famous example you've probably seen:
\nvector("king") - vector("man") + vector("woman") ≈ vector("queen")\nMeaning: the model "knows" that the relationship between "king" and "man" is analogous to the relationship between "queen" and "woman". And it got that purely from data - nobody taught it that!
\nWe can even show this in code. Here's a simple example with the gensim library in Python:
from gensim .downloader import load \n\nmodel = load ("glove-wiki-gigaword-50" )\n\nresult = model .most_similar (\n positive = ["king" , "woman" ],\n negative = ["man" ],\n topn = 3 \n)\n\nprint (result )\n# [('queen', 0.85), ...] -- in first place: queen! Tip
\nIf you want to test this yourself: install gensim (pip install gensim) and run the code above. Of course you'll need some RAM - the GloVe model isn't small. But the result is genuinely satisfying ;-)
One of the most important concepts in semantics: the meaning of a sentence is the result of the meanings of the individual words + the rules for combining them.
\nSo: "The red cat sits on the mat" = [red] + [cat] + [sits] + [on] + [the mat] + syntactic rules.
\nSounds trivial, but it's a powerful principle. Thanks to it we can understand an infinite number of sentences, even ones we've never heard before. And thanks to it an LLM can generate new, never-before-seen sentences that still make sense.
\nWarning
\nBut careful: "I understood that you didn't understand that it wasn't understandable." - each word is simple, but the sentence as a whole requires analyzing it layer by layer. Compositionality is a powerful tool, but it has its limits.
\nAnd so we arrive at the most interesting layer (in my opinion). Pragmatics studies how context influences the meaning of an utterance - what we really want to say, often by saying something completely different.
\nIf semantics asks "what does this mean?", pragmatics asks: "what did the author mean in this particular situation?"
\nScene 1: "It's cold in here."
\nSemantically: information about the temperature in the room.\nPragmatically: "Close the window!" or "Turn up the heating!" or "Give me a blanket."
\nEvery one of you understands that when someone says "It's cold in here" while standing next to an open window - you don't ask about degrees Celsius, you just close the window.
\nScene 2: "Could you pass the salt?"
\nSemantically: a question about your ability to pass the salt.\nPragmatically: a request to pass the salt.
\nAnswering "Yes, I could" (and nothing more) is semantically correct, but pragmatically... well, you're being a bit rude ;-)
\nScene 3: Irony
\n\n\n"Well congratulations, you broke it again."
\n
Semantically: congratulations.\nPragmatically: irony, criticism, disappointment.
\nDoes an LLM catch irony? Sometimes yes, sometimes no. It's still an open research problem.
\nThe philosopher J.L. Austin came up with something brilliant: utterances don't just describe reality, they also do things.
\n| Type of speech act | \nDescription | \nExample | \n
|---|---|---|
| Locutionary | \nThe utterance itself | \nYou say "I'm closing the window" | \n
| Illocutionary | \nThe intention of the utterance | \nA promise to close the window | \n
| Perlocutionary | \nThe effect on the listener | \nThe listener feels relieved | \n
Other examples of speech acts:
\nPaul Grice, another philosopher of language, noticed that conversation follows a cooperative principle - we say things that are relevant, true, clear, and on topic.
\nWhen someone breaks that principle, we look for an implicature - a hidden meaning.
\nExample:
\n\n\nA: "Are you going to the party?"\nB: "I have an exam tomorrow."
\n
B didn't answer "yes" or "no". But from the context A understands: I'm not going, because I have to study. That's a conversational implicature.
\nWarning
\nThis is the hardest layer for an LLM. And that's not just my opinion - researchers all over the world are working on benchmarks that test the pragmatics of language models.
\nThe problems:
\nBut on the other hand - GPT-4 and newer models are getting better and better. Why? Because the enormous amount of training data contains pragmatics in practice - dialogues from movies, books, internet forums. The model has "seen" millions of examples of irony, politeness, requests.
\nTip
\nAn experiment for you: Open ChatGPT (or Claude, or whatever you have at hand) and run this short test:
\nTell the model:
\n"I just hammered my finger."
\nThen, in a new conversation:
\n"Well congratulations, I hammered my finger again!"
\nSee whether the model notices that in the second case "congratulations" is irony, or whether it starts calling an ambulance?
\nHere are our five layers, in a nutshell:
\n| Layer | \nWhat it studies | \nExample | \nHow the LLM handles it | \n
|---|---|---|---|
| Phonetics & phonology | \nSounds and the system of sounds | \n/kat/ vs /hat/ | \nNever hears them, but "knows" from text data | \n
| Morphology | \nWord structure from morphemes | \nun-happy-ness | \nBPE tokenization != morphemes, but it works somehow | \n
| Syntax | \nWord order in a sentence | \nThe cat sits on the mat | \nThe model's superpower - handles grammar great | \n
| Semantics | \nMeaning of words and sentences | \n"bank" - which one? | \nWord embeddings + context | \n
| Pragmatics | \nMeaning in context | \n"It's cold" = close the window | \nThe hardest layer, still an open problem | \n
In the next post we change perspective - instead of looking at the layers of language, we look at the very nature of signs and meaning:
\n\nIf you made it all the way here - thanks! ;-) I really appreciate that you took the time to read this monster.
\nI hope I brought the topic a little closer to you. If anything is unclear - let me know in the comments, I'll try to explain. And if you have better examples (and you surely do!) - even more reason to let me know.
\nWhich layer surprised you the most? Which do you find the most interesting in the context of AI?
\nSee you in the next post!
\nSources and interesting links:
\nIf you want to go deeper, here are the materials I used when writing this post:
\nQuiz answers: sentences 1 and 5 are correct. Sentence 1 has standard SVO order. Sentence 5 is a grammatical question formed with the auxiliary "does". Sentences 2 and 3 are understandable but sound archaic or poetic - English allows such inversions only in limited stylistic contexts, unlike Polish where flexible word order is the norm. Sentence 4 ("The dog the mailman at barks") is syntactically incorrect - the word order is so jumbled it violates the rules of English grammar. ↩
\n