Hello, my name is Shana.

Welcome to this course on how LLMs actually work. No AI background is required.

When I used to start thinking about LLMs in the beginning, I imagined this magic brain where we give a prompt or some kind of question and magic happens and the answer magically comes out. We hear buzzwords like neural networks. So then that makes me think of this like nodes
and lines connecting the nodes and I don’t know what the nodes are for, and I don’t know what the connecting lines are because it’s all rather magical. That’s how I used to think about it.

The aim of this course is to, bit by bit, step by step, break down that magic and replace it with actual technical knowledge about the mechanics that’s happening inside the LLM. There is still a bit of magic at the end, but not half as much as there is currently.

So what this lesson is and what this lesson is not. This is not a lesson that teaches you how to write a prompt. There is a lot of very basic lessons out there that are already do that. This is not a lesson that teaches you how to use someone else’s AI tool. And this is not a lesson that sits all the way on the other end of the spectrum that expects you to be a mathematical genius in order to understand what the LLM’s doing. It’s not.

This lesson sits squarely in the middle. Where we’re going to cover the technical details of an LLM and the architecture of the LLM, in-depth, but without using any code. So you’ll be able to understand this, even if you’ve got no engineering experience at all.

Throughout the lessons, we’re going to be posing a bunch of questions and answering them, right?
Questions that should be relevant to you. We will also show newspaper articles, excerpts from them, and you’ll see that you’re able to read them and understand them now.

You’ll be able to be in the room with other people, talking about LLMs, and be part of the conversation, as you make informed decisions that are relevant for your company, because you understand the architecture of the LLM, not just what you’ve heard people parrot back at you.

These are some examples of some of the questions that we’ll be going through throughout this course. Here’s one.

Your risk committee asked the vendor “is this model biased?” and the vendor replies that it was trained on diverse, high quality, publicly available data. Well, does that answer the question?
Does the data being diverse, and high quality and publicly available, stop it from being biased? Hmm… let’s see.

Another question. Your API bill has two line items for the same model. You’ve got your input tokens with one price, and then your output tokens, with a different price that’s usually much higher. Why does it cost more for the API to produce output, than to read the input?

The team has been pasting customer records into an AI tool. And eventually, someone asks, “so do you think this model is actually learning from what we type into the LLM?” That’s a very relevant question. A lot of people have this question, and we’re going to clear this up once and for all.

Your team pastes a 200 page policy into the tool, and asks it a question about a specific page, say page 190, and they get a very confident answer from the LLM. How do you know, that the model even looked at your page? How do you know?

Next question, your vendor offers the same model in three different sizes at three price points, and you’re told that the big one is really, really much more capable than the others. Well, what are you actually buying more of? Do you need it? The only way to answer that, is to understand the architecture of the LLM, understand why a bigger one makes a difference.

There’s many AI announcements, and you see them in newspapers, 10 trillion parameters, 70 billion parameters, or three billion parameters, small enough to run on your phone, and your vendor will quote one, and your board’s going to ask you, what does it mean? Well, what’s a parameter, and what does a bigger number, or more of them mean? Does that make it a better model? And more specifically does it make it a better model for your use case?

So throughout this course, we’re going to be answering these questions, and more. We’ve got 16 lessons that we’ll be going through, starting all the way from the very very beginning of what’s a token, and finishing off at how you’re going to train or fine tune an LLM to solve your business needs.

I’m really excited to walk through this journey with you. It’s going to get complex, but we’re going to take all those complexities, split them up, bit by bit by bit, until we’ve got a really deep mental model of what’s happening in the LLM. You’ll be able to make decisions for your team, based on knowledge, real, deep understanding of what an LLM is doing. Not just parroting back information that you hear from other people.

With that, see you in lesson number one.

Chapter 1: Tokenization

Segmenting raw text to build an LLM’s vocabulary. This is a lightly edited transcript of Chapter 1 of “How LLMs Actually Work.” No AI background required.

The question we’re trying to answer

Hi, my name is Shana, and welcome to Chapter 1 of my course on how LLMs actually work. This chapter is about tokenization.

Here’s the question we’re going to answer: does the tokenizer matter when choosing an LLM for your business? Two models can receive exactly the same text and turn it into different numbers of tokens. When does that difference actually matter?

To get there, we first have to know what a token is, why we don’t use full words, and why we use subwords instead. Throughout this course I’ll be presenting definition boxes, and at the end there will be a glossary you can download from my website.

What is a language model?

The whole course depends on us knowing what a language model is. It’s an AI system that processes and predicts human speech or text. What it’s doing is learning statistical patterns and relationships in text fragments.

We often want to think it understands, that it’s thinking. I don’t know if it’s doing any of that. But what we do know is that what it’s doing is math. It’s learning statistical patterns and relationships between text fragments, and that allows it to complete sentences, translate languages and recognize context.

A large language model is the same thing, but trained at massive, massive scale — billions of parameters and vast amounts of text. Claude is an example of an LLM. We haven’t covered the term “parameters” yet, but that’s coming later in the course.

Three types of LLM architecture

There are three types of LLM architecture: decoder-only, encoder-only, and encoder-decoder. This course only focuses on decoder-only LLMs — the ones that predict the next token, like ChatGPT, Claude and Llama.

Vocabulary

Let’s start at the beginning. Whenever anyone learns a language, they have to build up their vocabulary. LLMs are no different. In this case, the vocabulary we give the LLM to learn is made up of tokens, which are pieces of text — not necessarily full words.

The tokenizer’s job is to take text and turn it into tokens. It does that twice. It tokenizes the text when the LLM is being trained on its initial training material, and it tokenizes any prompt we give it afterwards so the LLM can process it.

For the LLM to understand any prompt we write, that prompt has to be given in the language the LLM understands, using the vocabulary it was trained on. That’s why the tokenizer and the LLM go hand in hand. Whatever was used to create the initial vocabulary the LLM was trained on is what has to be used whenever we give that LLM something to process.

Phase 1: Building the tokenizer

The job of the tokenizer is to decide what chunks of text the model should treat as individual pieces.

It starts with a large text corpus. In my example I’ve shown only four lines — obviously that’s not what it would really be given. It would be given huge amounts of text. And it has to decide how to break those lines into small pieces that the LLM can learn. Those pieces are called tokens. At this stage, all we have is characters and words.

The simple approach, and why it fails

The simplest possible tokenizer would say: every word is a token. Just take all that text and split it on spaces. Now you have a list of tokens — the, cat, sat, on, the, mat — and that’s your vocabulary.

But this is very problematic. If the model is given a word it has never seen before, such as “computes,” it would have no way to represent it.

So then we’d say: fine, we’ll give it more words. We’ll have cat, and cats, and catlike, and catwalk. We’d need so many words to make sure the model is never blindsided, never given a token it doesn’t understand. But very quickly you find the number of words you’d need is just colossal.

That’s why this approach is impractical. We don’t have unlimited space for every possible token. A better approach is to store reusable pieces: cat, and then s, and then like, and walk. From there the tokenizer can put them together and represent words it has never seen.

Subword merge rules: Byte Pair Encoding

From this idea we get the concept of subword merge rules — how a vocabulary is built up iteratively, via Byte Pair Encoding.

Imagine the corpus of text we’re giving our tokenizer contains these words: computer three times, computing twice, faster four times, and similarly master, lower and newer.

Round 1. The tokenizer looks for adjacent pairs. It sees “e” and “r” next to each other three times in computer, because computer appears three times. Then it appears in faster four times. It adds up every place that pair occurs across the whole corpus, and the total is 19.

Then it looks at other pairs. How many times do “t” and “e” appear next to each other? That’s only 10 times across the whole corpus. It’s not in computing, but it is in faster. It’s not in lower. And it keeps adding them up until it has a tally — and “e” and “r” win.

So the first merge rule becomes: merge the adjacent pair “e” and “r” to make “er.” That becomes a token, added to our list, and it can be used in future rounds.

Round 2. Now it asks how many times “t” appears adjacent to our new token “er.” Three times in computer, four in faster, three in master — 10 times. When it does that for all the other adjacent pairs, “t” next to “er” wins. So our second merge rule creates “ter.”

Round 3. It continues. It looks at “w” next to “er.” It looks at “s” next to the token “ter” that we made in round 2. In this case the winner is “wer,” and that becomes the third merge rule.

Notice it doesn’t matter where in the word the pair occurs. In this case “ter” is at the end. Beginning, middle or end — it makes no difference. The tokenizer is simply looking for an efficient way to merge adjacent pairs, choosing whichever pair occurs most frequently. And it keeps track of the merge rules it performed to create each token, in each round.

How the vocabulary gets built

Let’s lay out what’s happening.

• It starts with single characters and gradually creates larger, reusable pieces.
• It counts every adjacent pair in the text corpus.
• The most frequent pair is merged everywhere at once, and the new token is added to the vocabulary.
• The winning pair can sit anywhere in a word — beginning, middle or end.
• Each new token is then eligible to take part in future merges.
• It does this thousands and thousands of times.

And this is the part I want to say twice, because it matters: the number of merges is limited by the target vocabulary size.

There is not an unlimited number of tokens we allow it to find. We don’t let it keep going until it has found every variation and combination. When the engineers design their LLM — and the tokenizer that will go hand in hand with it — they set a limit on how many tokens that LLM will be trained on, and therefore how many tokens the tokenizer will find. It cannot go on forever.

So the process is finite. The tokenizer does not learn every possible intermediate form of every word. At the end of it, the result is a vocabulary of reusable fragments, which we call tokens — and the merge rules, in the order they were created.

A word the tokenizer has never seen

Before the nuances, remember that the tokenizer is used in two places: to find the initial vocabulary the LLM is trained on, and again every time we use that finished LLM, to turn our prompt into the tokens it understands.

Now imagine the LLM is prompted with a word it has never seen before. The tokenizer will split that word into tokens it does have. For “computes,” it might have “comp” and “ut” and “e” and “s.” It has never seen the whole word, but it has these four tokens.

So there is no requirement that every word becomes a single token — that would make the vocabulary far too big. Some words end up as one token, others as two, three or more. The tokenizer’s goal isn’t to capture words. It’s to build an efficient vocabulary of reusable text fragments.

Token IDs

Once the tokenizer has finished creating its vocabulary, it assigns a unique ID number to each token. In our case, “comp” might be 0, “ut” might be 1, “e” might be 2, and so on.

The ID itself has no meaning. It’s just an index, like a row number in a database. There’s no special significance to it — it exists purely to uniquely identify each token.

Storing the merge rules

In our example, the tokenizer started with small pieces — the word computer, each letter separate. Then it learned that “e” and “r” are adjacent and can merge into “er.” Then that “t” is adjacent to “er,” giving “ter.” And so on.

The tokenizer stores these merge rules. Why? Here’s the thing I want you to understand: every future prompt must be tokenized using exactly the same rules that were learned from the initial text corpus.

It stores the rules in the order it learned them, because that order sets the priority of each rule. Think back a few steps — the first merge rule came from the adjacent pair that appeared most frequently in the corpus. That priority matters. Every prompt you give your LLM goes through the same tokenizing process, using the same established merge rules, created when the tokenizer was first built.

The tokenizer is complete

At this point the tokenizer is finished. The vocabulary has been determined, the merge rules are established, and the token IDs have been assigned. That’s phase one.

And here’s the thing to sit with: at this stage the LLM has not learned anything yet. All we have is a tokenizer, with a vocabulary, with merge rules, and with token IDs. The LLM has not been trained.

So, again — the LLM is not trained on words. It’s trained on the token sequence produced by this particular tokenizer, the one that goes hand in hand with it. How the LLM actually gets trained is something we’ll cover in future chapters.

Tokens in, detokenization out

We’re back to the magic brain picture. It still seems magical, but it’s slightly less magical than it was, because we know something we didn’t know before. It isn’t just a question going into the LLM and an answer coming out.

We know the prompt we type is tokenized, and that the tokens are what get sent in. And we can extrapolate the other side: if the LLM is processing tokens, then what comes out must be tokens too — so there must be a detokenization process that turns them back into text we can read.

Tokenization going in. Processing in the middle. Detokenization coming out. The brain is slightly less magical than it was.

How a prompt gets tokenized

First the text is split on whitespace and punctuation into small chunks. By the way, the spaces don’t get thrown away — each space gets attached to the front of the word that follows it. We’ll see that in the hands-on demo shortly.

Inside each chunk, the tokenizer applies its stored merge rules. Those rules are an ordered list: rule one was learned first, so it has the highest priority and is applied first.

In each round, the tokenizer looks at every adjacent pair in the chunk and merges the highest-priority one — wherever it sits, beginning, middle or end. It repeats until no adjacent pairs appear in the merge rules. When nothing left in the chunk matches a rule, it stops merging and goes with the tokens it has found.

Worked example: a word the tokenizer knows

Let’s take a word the tokenizer knows — computer. It’s in the original vocabulary, and the LLM was trained on it.

The tokenizer breaks the word into individual characters and goes through the merge rules in order:

• Rule 1 — “e” and “r” are next to each other, so they merge into “er.”
• Rule 2 — “t” can merge with “er,” giving “ter.”
• Rules 3, 4 and 5 — “wer,” “as” and “aster” are not applicable here, so they’re skipped.
• Rule 6 — “c” and “o” merge into “co.”
• Rule 7 — “m” and “p” merge into “mp.”
• Rule 8 — “co” and “mp” merge into “comp.”
• Rules 9 and 10 — and finally you have “computer.”

The result is a single token.

Notice the merges did not run left to right. We started at the end, went to the beginning with “co,” then did “m” and “p” in the middle. It wasn’t going in the order the letters were typed. The word was assembled from the end, then the front, then the middle — always taking whichever pair had the highest priority.

And when you look at the rule numbers going down the list, they always climb. You’ll never see them jump around, because the highest-priority rule is always applied first.

Worked example: a word it has never seen

Now a word the tokenizer has never seen: computes. Same merge rules.

• Rule 6 — “c” and “o” merge into “co.”
• Rule 7 — “m” and “p” merge into “mp.”
• Rule 8 — “co” and “mp” merge into “comp.”
• Rule 11 — “u” and “t” merge into “ut.”

And then it stops. There are no more applicable rules. Rule 12 makes “in” and rule 13 makes “ing,” but neither applies to “computes.”

The result is four tokens: comp / ut / e / s.

The tokenizer has never seen the word “computes.” But it has seen these individual tokens, and that’s what gets given to the LLM. A word the tokenizer never saw is built from pieces it already has. Some rules are never used, simply because the pair isn’t present in that word.

Two different jobs

So the tokenizer does two quite different things.

When it’s created, it builds an efficient vocabulary through statistical rule-building. Its goal at that point is to discover an efficient vocabulary and the merge rules, keeping track of both.

At inference — when the LLM has been trained and is being used — we use that same tokenizer to tokenize our prompt. It reapplies those same merge rules in the order they were learned. Its goal now is to segment new text exactly as the training corpus was segmented.

The tokenizer is entirely bound by its original vocabulary list.

Tokenizer algorithms in real models

Everything above describes the tokenizer used by GPT-2, released in 2019. It’s the most widely taught example because it’s the clearest to follow.

Modern models use a variety of tokenizer algorithms. GPT-2, GPT-4 and Llama all use some form of Byte Pair Encoding — the one we’ve just been through. Others use different approaches: T5 uses SentencePiece Unigram, BERT uses WordPiece. Claude and Gemini have not publicly documented what their tokenizer algorithms are.

There are open-source libraries for all of this. If you want to try Byte Pair Encoding, you can use tiktoken. There’s also sentencepiece, and transformers, which will run any of them. So: different models, different algorithms — and most of them are public. If you’re handy with code, this is something you can actually run yourself.

The tokenizer isn’t the hard part

The algorithms behind tokenizers are not proprietary. Anyone can train a tokenizer using open-source libraries.

The challenge in building an LLM isn’t creating the tokenizer. The real challenge is training the neural network on huge, huge amounts of tokenized text — and we haven’t even begun to talk about that yet.

Why should a business care how text becomes tokens?

Because tokens are the unit you are billed in, limited by, and throttled by. Not words.

• Price is measured in dollars per million tokens.
• The context window — the number of tokens that can be processed per request — decides how much of your document fits in a single call.
• Latency is time per token, not per word. That’s how fast your user gets an answer.
• Rate limits are tokens per minute. That’s how many users you can serve at once.

We care about tokens because that’s what we’re working with. The LLM is working in tokens, not words.

Not all text costs the same

Merge rules are learned from the training corpus, and that corpus is mostly English. So the tokenizer builds long, efficient tokens for English — it keeps merging adjacent pairs until the tokens have quite a lot of characters in them.

But non-ASCII characters start out taking more bytes, and more bytes cost more. Remember what BPE stands for: Byte Pair Encoding. It works on bytes, not letters.

So here’s a quick primer on how characters are represented in computing.

• 0–127 — one byte. This is ASCII. The letter “a” is 97, which sits in the middle of that range, so it’s one byte.
• 128–2,047 — two bytes. Hebrew, Arabic, Greek, Cyrillic, and accented European characters. The Hebrew letter mem is 1,502, so it takes two bytes in UTF-8.
• 2,048–65,535 — three bytes. Japanese, Chinese, Thai, Hindi.
• 65,536 and above — four bytes. Emoji.

Because our tokenizer is trained mostly on English text, it’s going to be most efficient with ASCII characters, which only use one byte. Whether it compresses the two-, three- and four-byte characters back down depends on how much of that language it was trained on. It might. You’ll have to test it.

So the tokenization process has more financial significance to you if you’re dealing with characters that aren’t ASCII. It’s going to start costing you more, and that might be a factor in your business decisions.

Why does the same prompt cost more in one model than another?

Simply because different models use different tokenizers — and different tokenizers have different vocabularies, different rules, and different ways of dividing the same text. So the same prompt does not produce the same token sequence. The LLM receives whatever its own tokenizer produced.

What you can’t do is say, “I love this LLM, but I prefer that tokenizer, so let me use them together.” You can’t mix and match. The tokenizer and the LLM go hand in hand. Part of the choice you make as a business is whether a given tokenizer-and-LLM combination works well for the kind of text you’ll be feeding it.

Does the tokenizer matter when choosing an LLM?

Potentially, yes. If two tokenizers split the same business workload differently, one may need more tokens to represent the same information.

That matters if you’re processing large documents, large volumes of documents, very domain-specific terminology, or high-volume AI workloads. The tokenizer is therefore one factor to consider when evaluating an LLM.

Does fewer tokens mean a better model?

No. Token efficiency is only one factor. You also have to consider:

• Model capability — does it perform the required task well?
• Token efficiency — how many tokens does it need to represent your workload?
• Workload volume — how much text will the business actually process?
• Business economics — what does the difference mean for cost and capacity?

Is the difference in tokenization significant enough to matter for your workload? That’s going to be very specific to your domain.

How to evaluate it

You can evaluate this. It isn’t a hidden secret, and it isn’t where the magic brain comes in. Here’s how:

1. Take a representative sample of the text your business expects to process.
2. Tokenize the same sample with each model’s tokenizer.
3. Compare the token counts.
4. Combine those counts with your expected workload, volume and pricing.

So you don’t have to guess. You measure before you select your model. And what that means is significant: tokenization is now a measurable business characteristic. It’s something you can make a reasoned decision about. You can go to your board having done the homework.

Hands-on exercise

There’s a great tool for this, and I’d encourage you to go and try it yourself:

https://gptforwork.com/tools/tokenizer

Find a piece of text whose content you are comfortable pasting into a third-party tool — and I’ll say that again, text you are comfortable pasting into a third-party tool. Paste it in, and it will show you the token count for each model. Don’t just look at the count; look at how the text was divided.

What the demo showed

I started with a basic sentence: “The fox sat on the rock.” As you’d expect, each word is its own token — and you can see that the space before each word is counted as part of that token. Six words, six tokens. Grok gave six as well. Claude gave thirteen for the same sentence.

Then I tried something more interesting — a passage without too many strange words, but containing things like “COVID-19,” which became three tokens. The name “Bryn Mawr” was split: it found “Bryn,” but had to break “Mawr” into two tokens, with the “r” separated off. The year 1918 was split too: “191” came out as one token and the “8” as another.

Across that passage the models were fairly close. 111 words came to 130 tokens in one, slightly more in another, and about twenty more again for Claude. Not a dramatic difference.

Then the interesting one. I had a complex medical text — words I couldn’t begin to pronounce. 188 words became 438 tokens in one model, and only 341 in another. More than double the number of tokens to words in the first case, and a very significant gap between the two models.

If you’re going to be processing medical text, that’s a cost you’ll be paying, every single request.

So play around with the tool. Paste in different kinds of text — text you’re comfortable pasting in — and see how it performs. Come away with the knowledge that different LLMs use different tokenizers, that each one will divide your text differently, and that your choice should take into account the kind of text you’ll actually be giving it.

Summary

• An LLM never sees words. It sees tokens.
• A tokenizer builds a vocabulary of reusable text fragments from a training corpus.
• Byte Pair Encoding builds that vocabulary by repeatedly merging the most frequently occurring adjacent pair.
• Every token gets an ID. The ID is just a unique identifier and carries no special meaning.
• The merge rules are stored in the order they were established, so every future prompt is split exactly the same way.
• The tokenizer and the model are permanently paired. They go hand in hand.
• Not all text costs the same. Non-English text, code and ID numbers need more tokens than plain English.
• Different tokenizers divide the same text differently — so you can measure it before you choose a model.

A glossary of every term from this chapter is available to download. I’ll see you in the next chapter.