Chapter 7 · Chapter 7: Second Gear
Unlocks: Masterclass Parts 6 (midtraining), 8 (RL and distillation), 9 (evaluation), and 11 (the cyber specialization). Chapters 1 to 6 walk the pipeline once, in order. This chapter covers the four ideas the later parts assume without stopping to teach: averaging weights, forgetting, sampling metrics, and keeping a long training run alive. None is a pipeline stage. All of them decide whether the stages work.
Weight averaging and model souping: the average is a model
A checkpoint is a file of numbers (Chapter 1). If two checkpoints have the same architecture, they have the same number of parameters in the same order, so you can average them: walk the two files in parallel, take the mean of each pair of numbers, write a third file. That third file is a working model. This is not an ensemble, where several models vote at inference time and you pay for all of them. The average is a single model with a single model’s serving cost.
Why can this possibly work? Because of where fine-tuned models land. Chapter 2 pictured training as walking downhill into a valley of low loss. Models fine-tuned or annealed from the same starting checkpoint tend to land in the same wide valley, the same loss basin. Points inside a wide, shallow bowl average to another point inside the bowl, often a flatter, more central one, and flat spots generalize better than sharp ones. Average models that started from different initializations and the picture collapses: their parameters never agreed on a coordinate system, the same slot in the file means different things in each, and the average is noise. Souping has a hard precondition, shared ancestry, and every use in the masterclass respects it.
The practical recipe comes from Wortsman et al. (2022), the model soups paper. A uniform soup averages every ingredient equally. A greedy soup ranks the ingredients on a validation set and adds each one only if the running average improves, which guarantees the soup is never worse than its best ingredient on that set. The gain mechanism is variance reduction: each run’s result wobbles around the recipe’s true quality because of data order and other luck, and averaging cancels part of the wobble.
Where the masterclass spends it:
- Anneal soups (Part 6). OLMo 2 ran its 50B-token anneal three times from the same checkpoint, changing only the data order, and averaged the three results. The soup beat the best single run on the team’s math aggregate, 77.0 against 75.6, and an independent group (LLM-jp) reproduced the pattern on a different base, soup at 59.1 on GSM8K against 53.5 for the best single seed. Souping also hands you a free variance estimate: the spread of the ingredients tells you how much of a score is luck.
- Checkpoint averaging as a schedule (Parts 5 and 6). K2-Horizon starts its midtraining from an average of the last few pretraining checkpoints. A later result (the WSM paper) explains why that is sound: averaging recent checkpoints emulates the effect of decaying the learning rate, performed after the fact, in weight space.
- Merged RL experts (Part 8). K2-Horizon trained five separate RL expert models from one mid-training checkpoint, one per domain, then merged them by weight averaging into a single flagship. The merge takes minutes where distillation would take a training run. What the average cannot do is reconcile genuinely conflicting behavior, one expert’s brevity against another’s thoroughness partially cancels, which is why a targeted SFT phase still had to patch the merged model afterward.
- Repair merges (Part 6). SmolLM3 fixed a capability its preference stage had damaged by merging 0.9 of the aligned checkpoint with 0.1 of the earlier midtraining checkpoint: a weighted average restoring a behavior without retraining.
Catastrophic forgetting and replay: new training overwrites old skill
A model’s knowledge is not filed in compartments. It is distributed across shared weights (Chapter 1), and every gradient step moves those weights toward whatever the current batch rewards. Train a model on a narrow new diet and the steps that install the new skill simultaneously uninstall pieces of everything else. The effect has a dramatic name, catastrophic forgetting, and it is the default outcome of sequential training, not an edge case. The later data wins because it is the only data still pushing.
The security analogy is a shared library problem. Fine-tuning is patching a system by editing libraries every other service links against. The patched service improves; three others quietly change behavior, and nothing pages you, because there is no test run scheduled for them.
The standard mitigation is replay: mix a slice of the old distribution into the new training data, so the gradient keeps receiving “stay good at this too” signal while it learns the new material. Part 6’s mixes are built this way. OLMo’s Dolmino anneal keeps roughly 47 percent elite web text as a general anchor alongside its math and instruction patch slices. Its long-context extension mix is two-thirds short data, insurance against the extension damaging ordinary short-context skill. Nobody in the open roster runs a pure-domain stage at base-model scale, and the mix proportions are the visible price of that discipline: a large share of every domain stage is spent not forgetting.
Supporting measures exist: lower learning rates (smaller steps, less overwrite), adapters such as LoRA (the frozen base cannot forget what it never relearns, which caps both damage and gain), and penalties that slow movement on weights important to earlier skills. But replay plus measurement is the load-bearing pair. Every serious conversion stage in the masterclass is bracketed by the same eval suite run before and after, because forgetting you do not measure is forgetting you ship.
Why LAB2 cares (Part 11): a cyber fine-tune is exactly the dangerous shape, a narrow, high-signal domain diet applied to a general base. Run it pure and you should expect eroded general capability and, subtler, shifted refusal behavior before post-training ever starts, because the anneal sets the prior post-training inherits. The mix design, roughly half general replay with patch slices upsampled hard, is the mechanism by which the model keeps both skill sets.
pass@k, avg@k, pass^k: three questions about the same attempts
Sampled generation (Chapter 1) means one attempt proves little. Evaluations therefore draw k attempts per task and summarize them, and Part 9’s whole argument is that the three standard summaries answer different questions. Take one model on one task, and suppose each attempt independently succeeds with probability 0.8. Draw 5 attempts.
pass@k: did any attempt succeed? The chance at least one of k succeeds is 1 minus the chance all fail: 1 − 0.2^5, which is about 0.9997. This is the capability-with-a-selector number. It is the right headline when something downstream can pick the winning attempt: a test suite, a compiler, a verifier, a human reviewing candidates. A coding assistant inside a CI loop lives here.
avg@k: what does one attempt score, on average? Average the outcomes and you estimate the single-attempt success rate, here 0.80, with the sampling noise squeezed down by the k draws. This is the comparison instrument: which checkpoint is better, did the RL stage help. It supports no claim about any individual deployment experience, only about expected quality.
pass^k: did every attempt succeed? The chance all k succeed is 0.8^5, about 0.33. This is tau-bench’s reliability metric, and its published results are the standing warning: a frontier agent solving about 61 percent of retail tasks on one attempt succeeded on all eight attempts under a quarter of the time.
Same model, same task, three headlines: 99.97 percent, 80 percent, 33 percent. None is a lie. Each is a different question, and quoting the wrong one is how a capable but inconsistent system gets sold as a dependable one.
The defensive agent is judged by pass^k for a structural reason: it has no selector. A containment agent that acts unattended serves every attempt it makes, including the failures, to a real network. At 80 percent per-attempt reliability it mishandles roughly one incident in five, and no verifier stands downstream to discard the bad runs the way a test suite discards bad code. Part 9’s rule follows: report pass@k where a verifier is genuinely in the loop, and pass^k for anything expected to act alone. The gap between the two at the same k is the inconsistency budget, and for agent work it often decides whether the system ships.
Training stability operations: spikes, rollbacks, and checkpoint cadence
Chapter 2 introduced the loss spike: the wrongness number jumps, sometimes to several times its value, because of a bad data shard, a too-hot learning rate, or a numerical edge case. A run of millions of steps will spike. Stability work is the operations discipline around that fact, and Part 5 grades labs on whether they publish their incident accounting.
The response ladder, cheapest first:
- Standing guards. Gradient clipping caps any single step’s size. Skip-step guards (OLMo 3’s SkipStepAdamW) refuse the update entirely when the loss or gradient norm sits far outside its recent range: you lose one batch, not the run.
- Rollback. Restore the last known-good checkpoint and resume from it, discarding every step since. This is the expensive, reliable option, and K2-Horizon’s published accounting shows its real scale: their 7B run needed 43 rollbacks, the deepest rewinding 22,124 steps, and their automated loss monitor triggers rollbacks without a human in the loop. The smooth published loss curve is the accepted trajectory only; the erased branches are where the incidents live.
Which makes checkpoint cadence an operational decision with a price on both sides. Save every 250 steps, as K2-Horizon does, and a rollback costs at most 250 steps of compute, but you pay storage and I/O millions of times over the run. Save every 10,000 steps and the same incident costs 10,000 steps. This is backup policy, unchanged from systems work: the cadence is your recovery point objective, set by how much work you can afford to lose. Checkpoints earn their keep a second way: they are the ingredients for the checkpoint averaging above, and the stage-by-stage artifacts Part 8 diffs to attribute behavior changes, refusal included, to the stage that caused them.
Check yourself
- Two models were fine-tuned from the same base; two others were pretrained independently. Which pair can be averaged into a working model, and why?
- What does a greedy soup guarantee that a uniform soup does not?
- Why does training on new data erase old capability instead of just adding to it?
- A cyber anneal mix is 100 percent vulnerability data. Name two things you expect to go wrong.
- A model succeeds on a task with probability 0.8 per attempt. Give pass@5, avg@5, and pass^5, approximately.
- Your run spikes and the automated guard rolls back 20,000 steps. Which earlier decision set that price?
Answers
- The fine-tuned pair. Shared initialization means they land in the same loss basin with parameters in the same coordinate system, so the element-wise average is still a good model. Independently pretrained models never agreed on coordinates; their average is noise.
- A greedy soup adds each ingredient only if validation improves, so the result is never worse than its best single ingredient on that validation set. A uniform soup averages everything, including an ingredient that wandered out of the basin.
- Because capabilities share the same weights. Gradient steps on the new data move those shared weights toward the new distribution, overwriting the settings the old capabilities depended on. There is no separate storage for old skills.
- Catastrophic forgetting of general capability (the base’s broad skill erodes), and a shift in the model’s behavioral priors, including refusal tendencies, before post-training starts. The standard fix is replay: a large share of general high-quality data mixed in, with evals run before and after.
- pass@5 is about 0.9997 (at least one success). avg@5 is about 0.80 (the expected single-attempt rate). pass^5 is about 0.33 (all five succeed).
- Checkpoint cadence. Rollback can only return to a saved checkpoint, so the spacing between saves is the maximum work any incident can cost. More frequent saves buy a cheaper recovery point at storage and I/O cost.
Go deeper
- Wortsman et al., “Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time” (https://arxiv.org/abs/2203.05482). The souping paper: the basin argument, uniform versus greedy recipes, and the failure cases. Read before Part 6’s anneal section.
- Kirkpatrick et al., “Overcoming catastrophic forgetting in neural networks” (https://arxiv.org/abs/1612.00796). The EWC paper. Its method is one mitigation among several, but it is the cleanest formal statement of why sequential training erases earlier skills.
- Chen et al., “Evaluating Large Language Models Trained on Code” (https://arxiv.org/abs/2107.03374). The Codex paper, source of pass@k and its unbiased estimator.
- Yao et al., “τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains” (https://arxiv.org/abs/2406.12045). Introduces pass^k and shows a strong agent’s reliability collapsing as k grows.
- Hugging Face, The Smol Training Playbook (https://huggingface.co/spaces/HuggingFaceTB/smol-training-playbook). A complete small run narrated with its stability incidents, guard settings, and checkpointing choices. The practitioner version of this chapter’s last section.
This chapter unlocks Part 6, Part 8, Part 9, Part 11 of the Model-Making Masterclass.