Chapter 2 · Chapter 2: Just Enough Math
Unlocks: Masterclass Parts 5 (pretraining), 6 (midtraining), and 8 (RL). The masterclass talks about loss curves, learning rates, optimizers, and gradients as if they were weather. This chapter makes them concrete with arithmetic you can check on a napkin. No proofs. If you can follow a weighted average and a feedback loop, you already have the prerequisites.
Vectors: lists of numbers with a job
A vector is an ordered list of numbers. You met one in Chapter 1: an embedding is a vector, a few thousand numbers that stand for a token. Inside the model, everything is vectors. A token enters as a vector, each layer transforms it into a new vector, and the final vector becomes scores over the vocabulary.
Why lists? Because a list can encode many soft facts at once. One position in the vector might respond to “this token is part of a URL,” another to “this is a verb form,” thousands more to patterns nobody has named. No single number means anything clean by itself. The meaning is in the pattern across the whole list, the way a single log line means little and a correlated cluster of them means an incident.
Two operations on vectors do almost all the work.
Weighted sum. Take several numbers, multiply each by a weight, add the results. You already trust this operation: a risk score is a weighted sum. Severity times 0.5, plus asset criticality times 0.3, plus exposure times 0.2. The weights decide what matters. In a model, the weights are the parameters, and the weighted sums are vastly bigger, but the operation is the same one.
Dot product. Take two vectors of the same length, multiply them position by position, and add everything up. The result is a single number that measures alignment: large when the vectors point the same way, near zero when they are unrelated, negative when they oppose. Attention (Chapter 3) is built from dot products: one token’s query vector dotted with another token’s key vector produces the score for how much the first should listen to the second. When the masterclass says two representations are “similar,” a dot product (or its normalized cousin, cosine similarity) is almost always the measuring instrument.
Matrix multiplication: weighted sums, industrialized
A matrix is a table of numbers, which is to say a stack of vectors. Multiplying a matrix by a vector runs one weighted sum per row of the matrix: each output number is the dot product of the input vector with one row. A layer of a neural network is, at its core, exactly this: take the token’s vector, multiply by a big matrix of learned weights, get a new vector.
Tiny worked example. Input vector x = [2, 1]. Weight matrix W with rows [1, 0] and [0.5, 3]. Output row one: 2 times 1 plus 1 times 0 equals 2. Output row two: 2 times 0.5 plus 1 times 3 equals 4. Output: [2, 4]. That is a neural network layer, minus scale and a nonlinearity (a simple bend applied to the outputs, without which stacked linear layers collapse into one). Scale it up: real layers multiply vectors of a few thousand numbers by matrices with millions of entries, billions of times per training run. This is why training is a GPU problem. Matrix multiplication is the one operation GPUs exist to do fast, and a model’s appetite for it is measured in FLOPs, floating point operations, the currency Chapter 4 prices.
Loss: one number that says how wrong you are
Training needs a score for wrongness. The loss is that score: a single number, computed from the model’s output and the correct answer, where lower is better.
For next-token prediction the loss works like this. The model outputs a probability for every token. The training text says which token actually came next. The loss is essentially the negative log of the probability the model assigned to the true token. If the model gave the right token probability 0.9, the loss contribution is about 0.105, small. If it gave it 0.1, the loss is about 2.3. If it gave it 0.01, the loss is about 4.6. Being confidently wrong is punished much harder than being uncertain, which is exactly the incentive you want, and the logarithm is what creates that shape. Average this over a batch of millions of tokens and you get the training loss you see on every loss curve in the masterclass.
When Part 5 describes a loss spike, this is the number that jumped. When a tech report quotes a final loss of 1.8 versus 1.9, that gap, averaged over trillions of tokens, represents a real quality difference, the way a small difference in detection rate over a year of traffic is real. Loss is a proxy, though. It measures prediction of the training distribution, not usefulness, which is why Part 9 (evaluation) exists and why nobody ships on loss alone.
Gradients: a compass for every knob at once
Here is the central problem of training. You have billions of knobs and one loss number. Which knobs do you turn, which way, and by how much?
The gradient answers all three. The gradient of the loss with respect to a parameter is the answer to a tiny experiment: if I nudged this one parameter up by a hair, would the loss go up or down, and how steeply? Compute that for every parameter and you have a vector as long as the parameter list, pointing in the direction of steepest increase of the loss. Training walks the other way.
Computing gradients for billions of parameters sounds impossible until you know the trick: backpropagation. The model is a chain of operations. The loss depends on the last layer’s output, which depends on the previous layer, and so on back to the input. The chain rule from calculus lets you propagate the “how much did you contribute to the error” signal backward through the chain, one layer at a time, reusing intermediate results. It is bookkeeping, not magic, and its cost is predictable: roughly twice the cost of the forward pass. Karpathy’s micrograd video in the Go deeper list builds this in about 100 lines of Python if you want to see the bookkeeping bare.
The security-engineer analogy: the gradient is like a blast-radius report for a configuration change, except it covers every setting in the system simultaneously and reports not just direction but sensitivity. The analogy breaks on scale and on meaning: gradients are local, valid only for tiny nudges near the current settings, which is why training takes millions of small steps instead of one computed leap.
Gradient descent: walking downhill in fog
The training loop, complete:
- Run a batch of training text through the model (forward pass).
- Compute the loss against the true next tokens.
- Compute the gradient of the loss for every parameter (backward pass, backpropagation).
- Update each parameter: new value equals old value minus the learning rate times the gradient.
- Repeat, for millions of steps.
Step 4 is the whole algorithm: gradient descent. The learning rate is the step size, and it is the most consequential single setting in a training run. Too large and the walk overshoots, bouncing out of good valleys or diverging entirely, the loss explodes and the run is dead or rolled back. Too small and the walk takes forever and may stall in a mediocre spot. Real runs do not use a constant rate: they warm up from near zero (a cold model’s early gradients are garbage, and full-size steps on garbage can wreck the run before it starts), hold or decay through the run, and taper toward zero at the end so the model settles. Part 5 compares the schedule shapes, cosine versus warmup-stable-decay, and Part 6 is about the final decay phase, annealing, where the best data is spent. You now know enough to read those arguments as what they are: step-size policy.
One refinement matters for reading Part 5. Plain gradient descent uses the raw gradient. Production optimizers, AdamW above all, keep running averages of gradients and their squares and scale each parameter’s step by how noisy its signal has been: steady signal, confident step; noisy signal, cautious step. That is the entire intuition behind the optimizer wars in Part 5. The contenders differ in how they summarize gradient history; the walking is the same.
A second optimizer family changes the walking itself, and Part 5 spends real time on it. Muon ignores the per-parameter view: for each weight matrix, it takes the gradient of the whole matrix and orthogonalizes it (a few Newton-Schulz iterations), which reshapes the update so its directions contribute more evenly instead of a few dominant directions swamping the rest. In plain terms, AdamW adjusts each knob’s step size; Muon adjusts the shape of the whole step. It applies to two-dimensional weight matrices, while embeddings and the output head usually stay on AdamW. Labs experiment with it because Moonshot’s scaling study (Liu et al., 2025) reported roughly twice AdamW’s compute efficiency and trained a production model with it; if even part of that holds at scale, the same budget buys a better model, which is why Part 5 audits optimizer claims like budget claims.
Training is inference plus correction, at scale
Assemble the pieces and the mystery drains out of pretraining. Inference is the forward pass. Training is the forward pass, plus a loss, plus a backward pass, plus a small knob adjustment, repeated over trillions of tokens on a cluster. The model’s “intelligence” is the accumulated residue of those adjustments. Nothing in the loop understands anything. The understanding, such as it is, lives in where the knobs ended up, which is Chapter 1’s point about fitted functions, now with the fitting machinery visible.
Two consequences worth carrying into the masterclass. First, training can only install what the data and the loss reward. A model trained to predict web text learns web text’s patterns, including its errors and biases, and post-training (Chapter 5) exists to reshape that raw material. Second, every training claim in the masterclass, every recipe, schedule, and optimizer choice, is a claim about this loop: that this variant of the walk ends at better knobs for the same compute. When a family publishes its loss curves and its spike incidents, it is showing you its walk, wrong turns included. That is what makes the open families in this study worth more than their weights.
Check yourself
- What does a dot product measure, and where does the model use it?
- A model assigns the true next token probability 0.5 in one case and 0.05 in another. Which contributes more loss, and roughly by what shape of rule?
- What is a gradient, in one sentence, and why does training step in the opposite direction?
- Name the five steps of the training loop.
- What breaks if the learning rate is far too large? Far too small?
- Why is backpropagation affordable instead of costing one experiment per parameter?
Answers
- Alignment between two vectors: how much they point the same way. Attention uses dot products between query and key vectors to score how much one token should attend to another.
- The 0.05 case, by a lot. Loss grows like the negative logarithm of the assigned probability, so confident wrongness is punished disproportionately: about 0.69 for 0.5 versus about 3.0 for 0.05.
- The gradient is, for every parameter, the direction and steepness with which a small nudge to that parameter would change the loss. Training steps against it because the gradient points uphill, toward higher loss.
- Forward pass on a batch; compute loss against the true tokens; backward pass to get gradients; update parameters by learning rate times gradient; repeat.
- Too large: overshooting, oscillation, loss spikes or outright divergence, and a dead or rolled-back run. Too small: glacial progress and settling into mediocre parameters, wasting the compute budget.
- Because the model is a chain of operations, error contributions can be propagated backward layer by layer with the chain rule, reusing intermediate results. The backward pass costs roughly twice the forward pass, total, not once per parameter.
Go deeper
- 3Blue1Brown, Essence of Linear Algebra (https://www.youtube.com/playlist?list=PLZHQObOWTQDMsr9K-rj53DwVRMYO3t5Yr). Chapters 1 to 3 are enough for this prerequisite: vectors, linear transformations, and matrix multiplication as geometry. Watch when the weighted-sum picture feels thin.
- 3Blue1Brown, “Gradient descent, how neural networks learn” (https://www.youtube.com/watch?v=IHZwWFHWa-w). The loss landscape and the walk downhill, drawn properly. Twenty minutes, watch before Part 5.
- Andrej Karpathy, “The spelled-out intro to neural networks and backpropagation: building micrograd” (https://www.youtube.com/watch?v=VMj-3S1tku0). Backpropagation implemented in plain sight, about 100 lines. For readers who only believe what they can run.
- 3Blue1Brown, Neural Networks series (https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi). The gentlest full telling of networks, loss, and learning, if this chapter went by too fast.
- Andrej Karpathy, Neural Networks: Zero to Hero (https://www.youtube.com/playlist?list=PLAqhIrjkxbuWI23v9cThsA9GvCAUhRvKZ). The whole build path from micrograd to a working GPT. The standing reference when any single chapter here needs a deeper second pass.
This chapter unlocks Part 5, Part 6, Part 8 of the Model-Making Masterclass.