Foundations: Prerequisites for the Model-Making Masterclass

Chapter 3 · Ch 3: Transformers

Chapter 3 · Chapter 3: Transformers

Listen: audio twin

Unlocks: Masterclass Parts 3 (tokenizers), 4 (architecture), and 13 (the Nemotron generations). The transformer is the architecture nearly every model in the masterclass uses, including the hybrids, which are transformers with some layers swapped. This chapter builds it from the one mechanism that matters: attention.

The problem attention solves

A language model reads a sequence of token vectors. To predict what comes next, each token’s vector needs to absorb information from the tokens around it. In the sentence “the exploit failed because the service was patched,” the word “failed” only makes sense in light of “patched,” several tokens away. Older architectures passed information along step by step, like a bucket brigade, and long-range connections faded. The transformer replaced the brigade with a lookup: every token can pull information directly from every earlier token, in proportion to how relevant each one is. That lookup is attention.

The honest analogy is a content-addressable store, closer to a database query than to an address. A normal array is accessed by position: give me element 5. Attention is accessed by content: give me a weighted blend of everything whose description matches what I am looking for. Think of each token running a search query over the log of all previous tokens and reading back a weighted summary of the hits. The analogy breaks in one place worth knowing: the “documents” being searched are not stored text but vectors computed on the fly, and the blend is a smooth average, not a set of discrete hits. Attention never retrieves a token. It mixes representations.

Queries, keys, values: the three roles

Each token’s vector plays three roles, produced by multiplying the vector by three different learned matrices:

  • The query is what this token is looking for. (“I am a pronoun; find the noun I refer to.”)
  • The key is what this token advertises about itself. (“I am a noun, a service name, a past-tense verb.”)
  • The value is what this token actually hands over if selected: the information content it contributes.

Scoring works by dot product (Chapter 2): the current token’s query is dotted with every previous token’s key. High score means “relevant to me.” The scores are normalized into weights that sum to one (a softmax), and the token’s new representation becomes the weighted sum of all the value vectors, weighted by those scores. A token that matches nothing much ends up with a diffuse blend; a token with one perfect match effectively copies that value in.

Three details complete the mechanism. First, causality: when predicting text, a token may only attend to tokens before it, never ahead. The future is masked out, which is what makes the model a predictor rather than a summarizer of a complete document. Second, scaling: the raw dot products are divided down before the softmax to keep the weights from collapsing onto a single winner early in training. Third, the KV cache from Chapter 1: keys and values, once computed, never change, so generation stores them and each new token only computes its own query, keys, and values, then attends over the stored set. The cache is why generation is feasible, and its size is why long contexts are expensive.

Heads and layers: many lookups, stacked deep

One attention operation per layer would force a single relevance criterion. Real models run many in parallel, called heads. Each head has its own query, key, and value matrices, so each learns a different notion of relevance: one head tracks subject to verb, another tracks the most recent similar token, another tracks delimiter structure in code. A typical layer might have 32 heads. Their outputs are concatenated and mixed by another matrix. Multi-head attention is less like one analyst reading the logs and more like 32 analysts each filing a one-line summary, which a fusion step merges.

A transformer layer is attention plus a second block: a feed-forward network (FFN), a plain two-matrix transformation applied to each token separately, where much of the model’s stored “knowledge” is believed to live. Attention moves information between tokens; the FFN processes what each token now holds. Layers stack: a modern model has anywhere from a dozen to over a hundred layers, and the token vectors are refined at every one.

The stacking works because of the residual stream. Each block does not replace the token’s vector; it adds its contribution to it. The vector that flows from embedding to output is a running accumulator, and each layer reads from it and writes back an increment. Think of it as a shared message bus or an append-only log that every stage annotates rather than overwrites. This is why very deep stacks train at all: the original signal has a clean path through, and each layer only has to learn a useful delta.

Position: attention is order-blind, so order is injected

Nothing in the lookup mechanism itself knows which token came first. A bag of tokens with the same content in different order would produce the same blends, which is fatal for language, where order is meaning. So position information is added deliberately.

The current default is RoPE (rotary position embedding): before the dot products, each query and key vector is rotated by an angle that depends on its position, so the score between two tokens ends up depending on their relative distance. RoPE’s practical virtue is that relative distance falls out of the geometry, and its practical headache is extrapolation: stretch the model to contexts far beyond its training length and the rotations enter territory the model never trained in, degrading quality until fixes like rescaling the rotation frequencies (the YaRN family of methods, Part 6) are applied. Alternatives you will meet in Part 4: some models drop explicit position encoding entirely (NoPE) and let the causal mask itself imply order, and some restrict most layers to a local window of recent tokens (sliding window attention, next section). All are answers to the same question: how does a position-blind lookup stay order-aware and affordable?

Second gear: sliding window attention

Full attention lets every token consult every earlier token, and the KV cache grows with every token served. Sliding window attention puts a horizon on the lookup: each token attends only to the most recent W tokens, a few thousand, say, and everything older falls out of direct reach. The cache stops growing once the window fills, and per-token cost flattens, which is the entire point. The design is not as myopic as it sounds, because layers stack. A token outside your window influenced tokens inside it at earlier layers, so information from far back still arrives, relayed hop by hop through the stack, with the effective reach growing with depth. What is lost is the direct, exact grab: “quote the line from 40,000 tokens ago” is a long relay chain, not a lookup. Most designs that use windows therefore keep a minority of layers at full attention for genuine long-range reach and run the cheap windowed version everywhere else. Part 4 reads each family’s ratio of windowed to full layers as a cost-versus-recall decision.

Second gear: Multi-head Latent Attention (MLA)

Sliding windows shrink the cache by restricting reach. MLA, introduced by DeepSeek (DeepSeek-V2, 2024) and carried into the models Part 4 and Part 10 price, shrinks the cache while keeping full reach, by changing what gets stored. Standard attention stores every token’s full key and value vectors, per head. MLA instead stores one compressed latent vector per token, a small summary produced by a learned down-projection, and reconstructs each head’s keys and values from that summary, on the fly, when attention needs them. The cache holds the summary, not the expanded set: DeepSeek reported a KV cache reduction of over 90 percent against its own earlier dense model, with matching quality. The trade is compute and complexity for memory: the decompression math runs on every step, and some standard tricks do not compose cleanly with the compressed form. Read MLA as the third answer to this chapter’s recurring bill: where sliding windows pay in reach and Mamba (below) pays in exact recall, MLA pays in arithmetic to keep both reach and recall while gutting the cache.

Mixture of Experts: conditional compute

A dense model runs every parameter on every token. A Mixture of Experts (MoE) layer replaces the single FFN with a bank of FFNs, the experts, plus a small router network. For each token, the router picks a few experts, often 6 to 8 of 128 or more, and only those process the token. The rest sit idle.

This is conditional compute, and the systems analogy is a load balancer in front of specialist services: requests route to the two or three services competent for them instead of hitting a monolith. It buys capacity cheaply. Total parameters (the shelf size, and the knowledge capacity) can grow large while active parameters (the per-token compute bill) stay small. The costs are real and Part 4 prices them: all the experts must be stored and loaded even though few fire, routing can collapse if a few experts hog the traffic (load balancing becomes a training problem of its own), and the routing decisions complicate distributed training.

One design note that recurs in the masterclass: most MoE models keep one shared expert that every token passes through, a commons for general patterns, alongside the routed specialists. When you read “30B-A3B” in a headline, you are looking at this design: 30 billion stored, about 3 billion active per token.

The hybrid paragraph: Mamba in one pass

Attention’s weakness is cost: the KV cache and the per-token work grow with context length. Mamba, a state-space architecture, takes the opposite trade. It processes the sequence recurrently, maintaining a fixed-size internal state that it updates per token, the way a streaming log processor maintains a running aggregate instead of storing the raw stream. Per-token cost and memory stay constant no matter how long the context gets. What it gives up is exact recall: the state is a lossy compression of history, and “find the exact token from 40,000 tokens ago” is precisely attention’s home turf. Hybrid models, the Nemotron line above all, interleave the two: most layers Mamba for cheap length, a minority of attention layers for exact recall. Part 4 compares the ratios the field has converged on, and Part 13 shows NVIDIA arriving at the hybrid design generation by generation.

Check yourself

  1. In attention, what are the query, key, and value of a token, in one phrase each?
  2. Why can a token not attend to tokens after it during generation?
  3. What does the residual stream do, and why does it make deep stacks trainable?
  4. Why does the model need position information injected at all, and what is RoPE’s answer?
  5. An MoE model has 128 experts per layer and activates 8. What does that do to total versus active parameters?
  6. What does Mamba give up to get constant per-token cost?
Answers
  1. Query: what this token is looking for. Key: what this token advertises about itself, used for matching. Value: the information this token contributes if matched.
  2. Because the model is trained to predict the next token; attending to the future would leak the answer during training and is impossible during real generation, where the future does not exist yet. A causal mask enforces it.
  3. It carries each token’s vector through all layers as an accumulator that blocks read from and add to, rather than replacing. The clean path preserves the signal and lets each layer learn only an incremental refinement.
  4. Attention scores depend on content alone, so without position the mechanism cannot distinguish word order. RoPE rotates query and key vectors by position-dependent angles so that attention scores reflect relative distance.
  5. Total parameters count all 128 experts (plus everything else), so the model is large to store. Active parameters count only the 8 experts a token uses, so per-token compute stays closer to a much smaller dense model.
  6. Exact recall over long ranges. Its fixed-size state compresses history lossily, so precise retrieval of a specific distant token is weaker than attention’s direct lookup.

Go deeper

Where this leads

This chapter unlocks Part 3, Part 4, Part 13 of the Model-Making Masterclass.