Linguistic features - what you need to know before you understand how an LLM thinks

LLMs (Large Language Models, i.e. ChatGPT, Claude, Gemini and friends) have been with us for a while now, and I decided to dig deeper into the topic, and one big question came up: but how does it even work that a model "understands" language? After all, underneath it's just "math and statistics", right? Well... yes and no ;-)

And then I stumbled into linguistics. And it turned out that to truly understand what an LLM does with language, you first need to understand what language is made of. And since a programmer trying to read about linguistics is a rather funny picture, I thought: I'll share it with you ;-)

This is the first post in a series in which we'll survey different levels of language analysis and see how it all connects to LLMs. In today's post we'll build the foundation - we'll get to know the five main layers of language and see why each of them matters for language models.

---

Language is like an onion - layer upon layer

Before we get into the details, imagine language as a sort of... onion. Or a layer cake, if you prefer sweet metaphors :D

Each layer is a different level on which language "works". From the lowest - sounds - to the highest - what we really meant to say, even when we said something completely different.

Five layers, five ways in which language gets "analyzed". And note - an LLM has to handle each of these levels to come across as a sensible conversational partner. Of course it doesn't do this consciously - but the structures it learns during training somehow "catch" these layers.

OK, let's get to it. We start at the bottom of our onion ;-)

---

Phonetics and phonology - the sounds an LLM never hears

What is phonetics?

Phonetics studies speech sounds as physical phenomena. Put simply: how we produce sounds, how they travel through the air, and how we receive them.

It has three branches:

Branch	What it studies	Example
Articulatory	How the speech organs produce sounds	Where is your tongue when you say "sh"?
Acoustic	The physical properties of sound waves	Frequency, amplitude
Auditory	How the ear and brain receive sounds	How the cochlea turns waves into a nerve signal

And phonology?

Phonology is a level up. It's not interested in sounds as physical phenomena, but in how sounds are organized in a particular language.

Key concept: the phoneme - the smallest unit of sound that distinguishes the meaning of words.

A simple example in English:

• pat vs bat - the words differ by one sound (/p/ vs /b/), and the meaning is completely different
• rat vs mat - /r/ vs /m/ again changes everything

Or in Polish:

• kot (cat) vs lot (flight) - /k/ vs /l/ and we have a totally different word

Why does this matter for an LLM?

Right - an LLM gets text, not sound. It has never heard pronunciation. And yet...

• it knows that "cat" and "hat" rhyme (because it saw it in texts)
• it handles puns and wordplay based on sounds
• it can write rhyming poetry

So the model somehow absorbs phonological information from text data alone. A bit like never having seen the ocean, but having read so many books about it that you can describe it ;-)

But of course there are limitations. A text-based LLM can't distinguish homophones (words that sound the same but mean different things) based on phonetics - it handles them purely at the semantic level (more on that in a moment).

And a fun fact: models like Whisper (speech-to-text from OpenAI) already connect phonetics with text. But that's a topic for a separate post in this series ;-)

---

Morphology - how words are built from blocks

What is morphology?

Morphology studies the structure of words - how they're assembled from smaller, meaningful pieces called morphemes.

A morpheme is the smallest unit of language that carries meaning. It can't be divided into anything smaller that would still carry meaning.

An example from English, often used in textbooks:

unhappiness = un + happy + ness
               ↑       ↑       ↑
            "not"   "happy"   makes a noun

Or in Polish:

nieszczęśliwy = nie + szczęśliw + y
                   ↑        ↑       ↑
               "not"  "happy/fortune"  masculine gender

Or even more:

nieprzystojny = nie + przy + stoj + ny
              "not" + "at" + "stand" + adjective suffix

Each of these pieces has its own meaning. Those are morphemes.

Two kinds of morphemes

Type	Description	Examples
Free	Can stand alone as a word	cat, house, runs
Bound	Must be attached to another morpheme	un-, -ness, -s, -ed

And among bound morphemes we have affixes:

• Prefixes (before the root): un-, re-, pre-, dis-
• Suffixes (after the root): -ness, -ful, -ize, -ed
• Infixes (inside the root): practically absent in English, but present in other languages! E.g. in Tagalog (Philippines): sulat (to write) -> sumulat (to write something) - the morpheme -um- is inserted into the middle of the root. Or think of the English abso-bloody-lutely ;-)

Inflection vs derivation

This is an important distinction:

Inflection - changes the grammatical form, but doesn't create a new word:

• house -> houses (plural)
• run -> ran (past tense)

Derivation (word formation) - creates a new word, often changing the part of speech:

• teach -> teacher (verb -> noun)
• house -> household (noun -> adjective/noun)

How does an LLM handle morphology?

Here's a fun fact. An LLM doesn't "know" what morphemes are. It uses BPE tokenization (Byte Pair Encoding), which splits text into tokens - but tokens are not the same as morphemes.

The BPE tokenizer splits text based on frequency of occurrence in the training data, not on linguistic structure. Sometimes that overlaps with morphemes, sometimes it doesn't.

That means the model doesn't learn morphology "directly" - it learns it indirectly, through statistics. And somehow it works ;-)

---

Syntax - the rules of the sentence-arranging game

What is syntax?

Syntax is the set of rules that determine how words are combined into sentences. Thanks to syntax we know that "The cat sits on the mat" is a valid sentence, and "The cat the on mat sits" is gibberish.

Simple? Well... more or less ;-)

Word order

Different languages have different word-order rules:

Language	Basic order	Example
English	SVO (Subject-Verb-Object)	The cat sits on the mat
Polish	"Flexible" SVO, but...	Kot siedzi na macie / Na macie siedzi kot / Siedzi kot na macie
Japanese	SOV	猫がマットの上に座っている
Latin	Free (because endings say everything)	Felix in tapete sedet

Polish is nice because we have quite a lot of freedom in word order. While in English "On the mat sits the cat" sounds poetic or unnatural, for us it's a normal sentence ;-)

The syntax tree

Sentences have a hierarchical structure - words group into phrases, and phrases into larger phrases. You can draw it as a tree:

That's the simple sentence "The cat sits on the mat" and it already has its hierarchy! Now imagine sentences with subordinate clauses, participles, adverbials...

"Colorless green ideas sleep furiously"

This is a famous example from Noam Chomsky.

The sentence is syntactically correct - it has a subject, a predicate, a proper structure. But semantically it's complete nonsense - ideas can't be green or sleep.

And that's exactly the proof that syntax and semantics are two different levels. You can have perfect syntax and zero sense ;-)

Quiz: correct or not?

Test yourself - which sentences are syntactically correct?¹

1. The dog barks at the mailman.
2. At the mailman barks the dog.
3. Barks the dog at the mailman.
4. The dog the mailman at barks.
5. Does the dog bark at the mailman?

---

Semantics - what does "meaning" even mean?

What is semantics?

If syntax asks "how is this built?", semantics asks: "what does it mean?"

Semantics studies the meaning of words, phrases, and sentences. And right away it turns out this isn't simple at all.

Polysemy - one word, many meanings

A classic English example: bank

• Bank - a financial institution ("I deposited money at the bank")
• Bank - the side of a river ("The river bank was muddy")
• Bank - a row of similar things ("A bank of computers")
• Bank - to tilt an aircraft ("The plane banked to the left")

The same word, four completely different meanings. How does an LLM know which one is meant? From context. And that's exactly what it does well - because it has seen millions of sentences where "bank" appeared in different contexts.

Or another classic - polysemy within a single sentence:

"I can can the canning of cans."

The word "can" three times, three different functions: can (be able to), can (the verb, to put in a can), can (the noun, a container). Context changes everything ;-)

Synonyms - are "big", "large", and "huge" the same?

Almost. But not quite:

• A big house - normal, just sizable
• A large house - sounds a bit more formal
• A huge house - now that's a mansion :D

Semantics studies these subtle differences - both denotation (literal meaning) and connotation (associations, emotional coloring).

How does an LLM "understand" meaning?

Here enters the concept of word embeddings. In broad terms: each word is represented as a vector - a sequence of numbers - in a high-dimensional space. Words with similar meanings are "close" to each other in that space.

The famous example you've probably seen:

vector("king") - vector("man") + vector("woman") ≈ vector("queen")

Meaning: the model "knows" that the relationship between "king" and "man" is analogous to the relationship between "queen" and "woman". And it got that purely from data - nobody taught it that!

We can even show this in code. Here's a simple example with the gensim library in Python:

from gensim.downloader import load

model = load("glove-wiki-gigaword-50")

result = model.most_similar(
    positive=["king", "woman"],
    negative=["man"],
    topn=3
)

print(result)
# [('queen', 0.85), ...]  -- in first place: queen!

Compositionality

One of the most important concepts in semantics: the meaning of a sentence is the result of the meanings of the individual words + the rules for combining them.

So: "The red cat sits on the mat" = [red] + [cat] + [sits] + [on] + [the mat] + syntactic rules.

Sounds trivial, but it's a powerful principle. Thanks to it we can understand an infinite number of sentences, even ones we've never heard before. And thanks to it an LLM can generate new, never-before-seen sentences that still make sense.

---

Pragmatics - what I really meant

What is pragmatics?

And so we arrive at the most interesting layer (in my opinion). Pragmatics studies how context influences the meaning of an utterance - what we really want to say, often by saying something completely different.

If semantics asks "what does this mean?", pragmatics asks: "what did the author mean in this particular situation?"

Scenes from life

Scene 1: "It's cold in here."

Semantically: information about the temperature in the room. Pragmatically: "Close the window!" or "Turn up the heating!" or "Give me a blanket."

Every one of you understands that when someone says "It's cold in here" while standing next to an open window - you don't ask about degrees Celsius, you just close the window.

Scene 2: "Could you pass the salt?"

Semantically: a question about your ability to pass the salt. Pragmatically: a request to pass the salt.

Answering "Yes, I could" (and nothing more) is semantically correct, but pragmatically... well, you're being a bit rude ;-)

Scene 3: Irony

"Well congratulations, you broke it again."

Semantically: congratulations. Pragmatically: irony, criticism, disappointment.

Does an LLM catch irony? Sometimes yes, sometimes no. It's still an open research problem.

Speech act theory

The philosopher J.L. Austin came up with something brilliant: utterances don't just describe reality, they also do things.

Type of speech act	Description	Example
Locutionary	The utterance itself	You say "I'm closing the window"
Illocutionary	The intention of the utterance	A promise to close the window
Perlocutionary	The effect on the listener	The listener feels relieved

Other examples of speech acts:

• A promise: "I promise I'll do it." - by saying it, you perform the act of promising
• A declaration: "I now pronounce you married." - someone must have the authority for it, but it works!
• An apology: "I'm sorry." - that's not a description of a state of affairs, it is the act of apologizing

Grice's implicatures

Paul Grice, another philosopher of language, noticed that conversation follows a cooperative principle - we say things that are relevant, true, clear, and on topic.

When someone breaks that principle, we look for an implicature - a hidden meaning.

Example:

A: "Are you going to the party?" B: "I have an exam tomorrow."

B didn't answer "yes" or "no". But from the context A understands: I'm not going, because I have to study. That's a conversational implicature.

How does an LLM handle pragmatics?

The problems:

• Irony and sarcasm - the model often takes it literally
• Implicatures - it doesn't always "catch" what's between the lines
• Speech acts - it may not recognize whether someone is promising, asking, or threatening
• Cultural context - what's polite in one culture is rude in another

But on the other hand - GPT-4 and newer models are getting better and better. Why? Because the enormous amount of training data contains pragmatics in practice - dialogues from movies, books, internet forums. The model has "seen" millions of examples of irony, politeness, requests.

---

Summary - the whole onion in one place

Here are our five layers, in a nutshell:

Layer	What it studies	Example	How the LLM handles it
Phonetics & phonology	Sounds and the system of sounds	/kat/ vs /hat/	Never hears them, but "knows" from text data
Morphology	Word structure from morphemes	un-happy-ness	BPE tokenization != morphemes, but it works somehow
Syntax	Word order in a sentence	The cat sits on the mat	The model's superpower - handles grammar great
Semantics	Meaning of words and sentences	"bank" - which one?	Word embeddings + context
Pragmatics	Meaning in context	"It's cold" = close the window	The hardest layer, still an open problem

What's next?

In the next post we change perspective - instead of looking at the layers of language, we look at the very nature of signs and meaning:

• Semiotics - why an LLM doesn't "think", yet still means something

---

If you made it all the way here - thanks! ;-) I really appreciate that you took the time to read this monster.

I hope I brought the topic a little closer to you. If anything is unclear - let me know in the comments, I'll try to explain. And if you have better examples (and you surely do!) - even more reason to let me know.

Which layer surprised you the most? Which do you find the most interesting in the context of AI?

See you in the next post!

---

Sources and interesting links:

If you want to go deeper, here are the materials I used when writing this post:

• Branches of linguistics - Fiveable - a great overview of linguistics subfields with examples
• Levels of Linguistic Analysis - Fiveable - a synthesis of the levels of language analysis
• The Five Language Domains - Relay Graduate School - a simple, didactic take on the five language domains
• Linguistic-Informed Approach to Production LLM Systems - ZenML - a bridge between linguistics and LLM practice
• Evaluating Large Language Models on Linguistic Competence - LMU Munich - how the linguistic competence of models is studied
• Pragmatics in the Era of LLMs: A Survey - arXiv - a survey of research on pragmatics in LLMs

1. Quiz answers: sentences 1 and 5 are correct. Sentence 1 has standard SVO order. Sentence 5 is a grammatical question formed with the auxiliary "does". Sentences 2 and 3 are understandable but sound archaic or poetic - English allows such inversions only in limited stylistic contexts, unlike Polish where flexible word order is the norm. Sentence 4 ("The dog the mailman at barks") is syntactically incorrect - the word order is so jumbled it violates the rules of English grammar. ↩