Chapter 1 · Chapter 1: Models and Tokens
Unlocks: every part of the masterclass. This is the ground floor: what the numbers are, what goes in, what comes out, and why generation behaves the way it does.
Parameters: the whole model is a file of numbers
A model’s parameters are ordinary floating point numbers, stored in a file. When someone says a model is 7B, they mean roughly seven billion parameters. When they say it is a 14 GB download, they are saying the same thing in bytes: seven billion numbers at two bytes each (a 16-bit format called BF16, covered in Chapter 4) is about 14 GB. Parameter count is the model’s size the way disk size is a volume’s size. It tells you what it costs to store and load, and roughly how much machinery runs per prediction, but by itself it does not tell you how good the model is, any more than binary size tells you how good a program is.
You will meet two refinements of this number constantly in the masterclass, so learn them now.
Total versus active parameters. Some architectures (Mixture of Experts, Chapter 3) keep most parameters idle for any given token and use only a slice. Then the honest description is two numbers: total parameters (what you store) and active parameters (what computes per token). A “30B-A3B” model stores 30 billion and computes with about 3 billion per token. Confusing these is the single most common error in model comparisons, and the masterclass’s Piece 0 builds its whole comparison spine on keeping them separate.
Nameplate versus actual. Marketing names round, exclude, or flatter. A model sold as “7B” can hold 9 billion parameters once you count the embedding table (below). Chapter 6 teaches you to read the config file and count for yourself. Until then, treat headline sizes the way you treat a vendor’s throughput claim: a starting point for verification, not a fact.
Tokens: the atoms of text
The function does not read characters and does not read words. It reads tokens: chunks of text from a fixed vocabulary, typically somewhere between 32,000 and 256,000 distinct chunks depending on the model. A tokenizer converts text to a sequence of token IDs (integers) before the model sees it, and converts IDs back to text afterward.
Tokens are usually subwords. Common words are single tokens. Rare words split into pieces. Code, numbers, and high-entropy strings like hashes and base64 split into many small pieces, because the vocabulary was built from common patterns in training text and a random hex string has none. A rough rule for English prose is one token per three to four characters, but that rule collapses exactly where security work lives: logs, shell commands, encoded payloads. The masterclass’s Part 3 measures this per family, because a tokenizer that shreds your domain’s text taxes everything downstream: more tokens per document means more compute, more memory, and less of your context window spent on content.
Why subwords at all? A character-level model would need very long sequences to say anything. A word-level model would need a vocabulary of millions and would still fail on any word it had never seen, a hostname, a hash, a new CVE identifier. Subwords are the engineering compromise: a bounded vocabulary that can spell anything, at the price of uneven chunk sizes.
The vocabulary itself is learned, not hand-designed, and the learning algorithm explains the unevenness. The dominant method is byte pair encoding (BPE), adapted for language models from a compression algorithm (Sennrich et al., 2016). Start with a vocabulary of individual characters or bytes. Count every adjacent pair in a large training corpus, merge the most frequent pair into a single new token, and repeat, tens of thousands of times, until the vocabulary reaches its target size. Frequent strings earn merges early: common words, “ing”, ” http”, whole code keywords become single tokens. Rare strings never earn a merge and stay shredded into characters, which is precisely the hash and base64 behavior above. A tokenizer is a compression pass over a particular corpus, so whose text was in that corpus decides whose text tokenizes cheaply. Part 3 prices the consequences per family.
Embeddings: turning IDs into geometry
An integer ID is useless to a function; you cannot do arithmetic on “token 41,207” and learn anything. So the first thing the model does is look each ID up in a table, the embedding table, and replace it with a vector: a list of numbers, typically a few thousand long. Every token in the vocabulary has its own vector, and those vectors are parameters too, learned during training like everything else. The embedding table for a 256K vocabulary at 4,096 numbers per vector is over a billion parameters by itself, which is why vocabulary size is a real budget decision (Part 3 does this accounting).
The learned vectors end up encoding meaning geometrically. Tokens used in similar contexts get similar vectors, meaning their lists of numbers point in similar directions. “Password” and “credential” end up near each other; “password” and “photosynthesis” do not. A useful analogy is fuzzy hashing for files: similar inputs produce similar digests, and similarity becomes a distance you can measure. The analogy breaks in an instructive place: a fuzzy hash is fixed and hand-designed, while embeddings are learned and high-dimensional, so the “similarity” they encode is whatever the training data rewarded, including biases and spurious patterns, not a designed notion of sameness.
At the far end of the model, the process reverses. The final layer produces a score for every token in the vocabulary (often using the same embedding table, a weight-sharing trick called tying), and those scores become the probability distribution over the next token.
The forward pass and the generation loop
One run of the function over the current sequence is a forward pass. It processes the whole sequence at once and produces, at the final position, the distribution over what comes next. Generation is then a loop: sample one token from the distribution, append it, run the forward pass again. A 500-token answer is 500 forward passes. This is why the first token of a response arrives after a delay (the prompt itself must be processed, a phase called prefill) and the rest stream at a steadier rate (the decode phase). Part 10 of the masterclass is built on the economics of these two phases.
Notice what this implies about “thinking.” The model does not plan the answer and then type it. Each token is chosen with only the previous tokens as context, which is why getting a model to reason step by step in its output works at all: the intermediate tokens become part of the input for the later ones. The scratchpad is the context itself.
Temperature and sampling: controlled randomness
The forward pass gives probabilities, not a decision. Something has to pick. The simplest rule, always pick the most probable token (greedy decoding), produces deterministic but often dull and repetitive text. Real systems sample: pick randomly, weighted by probability. Two knobs control how adventurous the sampling is.
Temperature rescales the distribution before sampling. Below 1.0 it sharpens the distribution, favoring the top choices, more deterministic, more conservative. Above 1.0 it flattens it, giving unlikely tokens a real chance, more varied and more error-prone. Temperature 0 is the greedy limit. Think of it as a randomness budget, like jitter injected into a retry policy: sometimes you want none (a config file), sometimes you want some (brainstorming exploit variants).
Top-k and top-p truncate the distribution before sampling: consider only the k most likely tokens, or only the smallest set whose probabilities sum to p, and sample within that set. Truncation is a guardrail against the long tail, where individually improbable tokens include outright garbage.
The engineering consequence the masterclass relies on: an evaluation score is partly a property of these settings. The same model at temperature 0 and at temperature 1.0 is, behaviorally, two different systems, and a benchmark number that does not state its sampling settings is incomplete. Part 9 returns to this with receipts.
Context window: the model’s working memory
The context window is the maximum number of tokens the model can take in at once, input plus output together. Everything the model “knows” about your conversation lives in that window. When the window fills, older content falls out or must be summarized by something outside the model; the model itself has no memory across sessions at all.
Sizes range from a few thousand tokens in older models to 512K and beyond in the current open families. Bigger is not free: processing cost and the memory used to track the sequence grow with window occupancy, and models are usually trained mostly on shorter sequences and extended to long windows late in training (Part 6). A claimed window and an effective window are also different things; a model can accept 512K tokens and still be bad at finding the one relevant log line buried at position 300,000. Treat context claims the way you treat a SIEM’s advertised events-per-second: verify with your own workload shape.
A term you will meet everywhere: the KV cache. During generation, the model saves intermediate results for every token it has processed (the keys and values of Chapter 3) so it does not recompute them each step. The cache is the real memory bill of serving long contexts, and half of Part 10 is ways to shrink it.
Check yourself
- A model is described as 30B-A3B. What do the two numbers mean, and which one decides the download size?
- Why does the model not simply read characters?
- What is an embedding, and why does vocabulary size cost parameters?
- How many forward passes produce a 200-token answer, roughly?
- What do temperature and top-p change, and what do they not change?
- Your context window is 128K tokens and your log file tokenizes badly. What goes wrong before the model says anything wrong?
Answers
- 30B total parameters stored; about 3B active per token in its MoE layers. Total parameters decide the download and memory size; active parameters decide per-token compute.
- Character sequences would be far longer for the same text, making every step slower and long-range patterns harder to learn. Subword tokens are the compromise between sequence length and a bounded vocabulary.
- An embedding is the learned vector that replaces a token ID at the model’s input. The table holds one vector per vocabulary entry, so a bigger vocabulary means a bigger table, over a billion parameters at 256K entries.
- About 200, one per generated token, plus the initial pass that processes the prompt (prefill).
- They change how the next token is picked from the probability distribution the model outputs: how much randomness, and how much of the tail is eligible. They do not change the model’s parameters or knowledge.
- The log consumes far more tokens than its character count suggests, so less of it fits, and what does fit costs more compute and cache. The failure is a budget failure, not a comprehension failure.
Go deeper
- Andrej Karpathy, “Let’s Build GPT” (https://www.youtube.com/watch?v=kCc8FmEb1nY). Builds the forward pass and generation loop in code, token by token. The single best two hours for this chapter’s material.
- Andrej Karpathy, “Let’s Build the GPT Tokenizer” (https://www.youtube.com/watch?v=zduSFxRajkE). Tokenization from first principles, including why code and weird strings tokenize the way they do.
- Brendan Bycroft, LLM Visualization (https://bbycroft.net/llm). A 3D, step-through visualization of a real model processing tokens. Use it to make the forward pass physical.
- Transformer Explainer (https://poloclub.github.io/transformer-explainer/). Interactive, in-browser, with temperature and sampling controls you can push around to feel what they do.
- Sebastian Raschka, Build a Large Language Model (From Scratch), Chapter 2 and its repository (https://github.com/rasbt/LLMs-from-scratch). The code-level version of tokens and embeddings, for when the videos leave you wanting to type it yourself.
This chapter unlocks every part of the Model-Making Masterclass. Its vocabulary appears on nearly every page of the tome.