Chapter 4 · Chapter 4: Training at Scale
Unlocks: Masterclass Parts 5 (pretraining) and 10 (serving). Chapter 2 gave you the training loop for one batch on one machine. This chapter is about what changes when the model has 100 billion parameters, the corpus has 15 trillion tokens, and the machine is a warehouse. Almost every number in Part 5 is downstream of the arithmetic in this chapter.
Tokens are the unit of work, batches are the unit of scheduling
Training does not proceed document by document. Text is packed into sequences of a fixed length (say 8,192 tokens), and a batch is a pile of those sequences processed together: for example 1,024 sequences times 8,192 tokens is about 8.4 million tokens per optimization step. One step is one forward pass, one backward pass, one parameter update over that whole batch. A pretraining run is millions of such steps. When a tech report says “trained on 25T tokens,” it is stating total work the way a build farm would state total compile jobs: batch tokens times steps.
Batch size is a real trade-off, not a free dial. Bigger batches parallelize beautifully and keep GPUs fed, but each step sees more data before correcting course, and past a size that grows with the run, extra batch buys little learning per token. Reports quote batch size in tokens per step precisely because that is the comparable unit across models of different shapes.
GPU memory anatomy: where the bytes actually go
The first rude surprise of training at scale is that the weights are not the memory problem. Four tenants share each GPU’s memory during training, and the weights are only one of them.
Take a 7B model trained the standard way, mixed precision with the AdamW optimizer, and count bytes per parameter:
- Weights, compute copy (BF16): 2 bytes. This is the copy the matrix math runs on.
- Weights, master copy (FP32): 4 bytes. A high-precision original is kept because tiny updates would vanish if rounded into the 16-bit copy directly, the way small deposits disappear in an account kept only in whole dollars.
- Gradients: 2 to 4 bytes. One gradient per parameter, produced by every backward pass.
- Optimizer state: 8 bytes. AdamW keeps two running averages per parameter (Chapter 2), each a full FP32 number.
Total: roughly 16 to 18 bytes per parameter. Your 7B model needs about 120 GB just to exist in a trainable state, before a single token is processed. That exceeds one high-end GPU’s memory, which is the entire reason the parallelism section below exists.
The fifth tenant is activations: the intermediate results of the forward pass, which must be kept because the backward pass needs them. Activation memory scales with batch size times sequence length times layers, and it is the tenant that grows when you raise the batch. One standard trick trades compute for memory here: gradient checkpointing discards most activations and recomputes them during the backward pass, paying roughly a third more compute to cut activation memory dramatically. When Part 5 reports a run’s configuration, these five numbers are the invisible spreadsheet behind it.
Precision: bits are a budget line
A floating point number costs bytes and time in proportion to its width, so precision is a budget decision wearing a numerics costume. The formats that matter:
- FP32 (32-bit): the reference. Used for master weights and optimizer state, where small errors accumulate dangerously.
- BF16 (16-bit): the training default for the compute itself. Half the bytes of FP32, roughly double the matrix throughput on modern GPUs, and enough range for forward and backward passes when the master copy stays FP32.
- FP8 (8-bit): the proven next step. DeepSeek-V3 trained in FP8 with careful scaling and reported a relative loss error under 0.25 percent against BF16. Memory and bandwidth halve again.
- NVFP4 (4-bit): the current frontier, NVIDIA Blackwell only. Training in 4-bit floats needs compensating tricks (block scaling, stochastic rounding, a minority of sensitive layers kept at higher precision) and buys another large throughput multiple.
The pattern to carry into Part 5: nobody quantizes everything. Embeddings, normalization, accumulations, and optimizer state stay in high precision in every serious recipe, because those are the places where rounding error compounds. Precision arguments in the masterclass are always arguments about which tensors may be cheap, never about whether cheap is free. The same formats return in Part 10 for serving, where the stakes flip: there, quantization shrinks a finished model for deployment and the question is how much quality the shipped artifact loses, which is why Part 10 insists benchmarks be measured on the quantized artifact itself.
FLOPs: the currency, and MFU: the exchange rate you actually get
Compute is counted in FLOPs, floating point operations. A useful approximation covers most of what the masterclass needs: training a dense model costs about 6 FLOPs per parameter per token (2 for the forward pass, 4 for the backward). A 7B model over 1T tokens is therefore about 6 times 7e9 times 1e12, roughly 4.2e22 FLOPs. A top GPU delivers on the order of 1e15 BF16 FLOPs per second at peak. Divide, and one GPU needs on the order of 10 million seconds, months, for that run; a thousand GPUs need hours to days. This back-of-envelope method, parameters times tokens times 6, divided by hardware peak times GPU count, is how every training budget in the field is first estimated, and the masterclass’s Part 5 uses it to convert recipes into dollar figures.
The catch is the exchange rate. MFU, model FLOPs utilization, is the share of the hardware’s theoretical peak a run actually achieves. Real large runs land around 30 to 55 percent. The rest is lost to communication between GPUs, memory stalls, pipeline bubbles, and recompute. MFU is the honest efficiency number because it cannot be flattered by buying more hardware: a run at 35 percent MFU is burning 65 percent of a very expensive cluster on overhead. When a report omits MFU, you cannot cost the run, and Part 5 treats that omission as a finding in itself.
The companion concept is tokens per parameter, the training diet’s richness. The Chinchilla result (2022) found compute-optimal training at roughly 20 tokens per parameter, and the modern practice deliberately overtrains small models far past that point, thousands of tokens per parameter for the smallest, because inference cost over a model’s lifetime dwarfs training cost. Part 5 reads each family’s ratio as a strategy statement: giants train near optimal, serving workhorses train long.
Parallelism in one page
No single GPU holds a frontier run, so the run is carved across GPUs four ways, and real clusters compose all of them.
Data parallelism copies the whole training setup onto many GPUs and gives each copy a different slice of the batch. After the backward pass, the copies average their gradients so everyone updates identically. Simple, and it multiplies the memory problem instead of solving it, since every copy holds everything. The fix is sharding: ZeRO and its descendant FSDP split the optimizer state, gradients, and eventually the weights themselves across the copies, so each GPU stores a fraction and fetches what it needs. With full sharding, data parallelism becomes the default substrate of most open training frameworks.
Tensor parallelism splits individual layers: one matrix multiplication is cut across several GPUs, each computing a slice, with results combined over the fastest interconnect. Communication is constant and heavy, so tensor parallelism lives inside a single node or rack where the wiring is fastest.
Pipeline parallelism splits the model by depth: layers 1 to 20 on one GPU group, 21 to 40 on the next, and batches flow through the stages like an assembly line. It crosses between nodes gracefully, but the line has bubbles: stages idle while the pipeline fills and drains, and scheduling tricks (splitting batches into micro-batches) exist to shrink them.
Expert and context parallelism are the specialists: expert parallelism places different MoE experts on different GPUs (Chapter 3’s routing becomes a network scheduling problem), and context parallelism splits very long sequences across GPUs. Part 5 names which composition each family chose; you now know what each choice is buying and what it costs in communication.
The systems analogy writes itself: this is horizontal scaling of a stateful workload, with the familiar lesson that the interconnect, not the node, is usually the binding constraint. It breaks where all ML analogies break: the “state” being synchronized is a gradient for every one of billions of parameters, every step, so the synchronization traffic is enormous and constant, more like a database in synchronous replication at microsecond stakes than like a stateless web tier.
Second gear: predicting ahead, then drafting and verifying
Chapter 1’s generation loop is serial: one forward pass, one token. Two related ideas attack that serial cost, and both are decided partly at training time, which is why they belong in this chapter rather than only in Part 10’s serving discussion.
Multi-token prediction (MTP) changes the training target. Instead of asking the model to predict only the next token, extra output heads ask it to predict the token after next, and the one after that, from the same internal state (Gloeckle et al., 2024). As a training signal this forces richer planning into the representation, and the paper reports better models for the same training time, especially on code. At serving time those extra heads become a built-in draft mechanism: the model can propose its own next few tokens cheaply.
Speculative decoding is the serving pattern that consumes such proposals. A fast drafter, a small separate model, or the target model’s own MTP heads, guesses several tokens ahead. The big model then verifies all the guesses in a single parallel forward pass, accepts the longest prefix that matches what it would have produced itself, and resamples at the first disagreement. Done with the correct acceptance rule, the output distribution is provably identical to the big model generating alone; only the speed changes, typically two to three times faster (Leviathan et al., 2023). The systems name is exact: this is speculative execution, the branch-prediction trick from CPU design, applied to tokens.
The lesson to carry into Part 10 is that serving economics reach back into architecture. Whether a model ships MTP heads is decided before pretraining starts, the acceptance rate that decides the speedup depends on how well the drafter agrees with the target, and a family that designs for cheap inference makes that decision in the architecture stage and pays for it in the training run. When Part 10 quotes draft acceptance rates for real models, this is the machinery being measured.
Check yourself
- A run uses batches of 4 million tokens and takes 3 million steps. How many tokens did it train on?
- Why does a 7B model need far more than 14 GB of memory to train?
- Which tensors stay in high precision even in an FP8 or NVFP4 run, and why?
- Estimate the training FLOPs for an 8B dense model on 2T tokens. What else do you need to turn that into wall-clock time?
- What does 40 percent MFU actually mean, in money terms?
- Which parallelism strategy splits a single layer across GPUs, and where is it safe to use?
Answers
- 4 million times 3 million equals 12 trillion tokens (12T).
- Because training holds four tenants per parameter: the BF16 compute copy, an FP32 master copy, gradients, and two FP32 optimizer averages, roughly 16 to 18 bytes per parameter, about 120 GB at 7B, plus activations on top.
- Master weights, optimizer state, accumulations, and sensitive spots like embeddings and normalization. Rounding error in those compounds across millions of steps; the cheap formats are spent where error stays local.
- About 6 times 8e9 times 2e12, roughly 9.6e22 FLOPs. To get time you need the GPU count, each GPU’s peak FLOPs at the training precision, and the MFU you expect to actually achieve.
- Sixty percent of the cluster you are paying for is producing no model improvement: it is spent on communication, stalls, bubbles, and recompute. MFU converts directly into cost per trained token.
- Tensor parallelism. It communicates constantly, so it is confined to where interconnect is fastest, typically inside a node or rack.
Go deeper
- Hugging Face, The Ultra-Scale Playbook (https://huggingface.co/spaces/nanotron/ultrascale-playbook). The parallelism and memory chapters are the full version of this chapter, with measured configurations. The reference for Part 5’s systems arguments.
- Hugging Face, The Smol Training Playbook (https://huggingface.co/spaces/HuggingFaceTB/smol-training-playbook). A complete small-model run narrated end to end: data, ablations, stability incidents, and what the numbers cost.
- Andrej Karpathy, “Let’s reproduce GPT-2 (124M)” (https://www.youtube.com/watch?v=l8pRSuU81PU). The best single video on throughput as an engineering discipline: precision, compilation, and memory, measured live.
- Hoffmann et al., “Training Compute-Optimal Large Language Models” (the Chinchilla paper) (https://arxiv.org/abs/2203.15556). The source of the 20 tokens per parameter rule and the cleanest example of a scaling-law study. Skim the method, read the conclusions.
- Stanford CS336, course site (https://cs336.stanford.edu/). Lectures 2 (resource accounting), 7, and 8 (parallelism) are this chapter at university depth, with assignments that make you compute the budgets yourself.
This chapter unlocks Part 5, Part 10 of the Model-Making Masterclass.