Foundations: Prerequisites for the Model-Making Masterclass

Chapter 5 · Ch 5: Post-Training

Chapter 5 · Chapter 5: Post-Training

Listen: audio twin

Unlocks: Masterclass Parts 7 (SFT), 8 (preference, RL, distillation), and 11 (the cyber specialization). Pretraining produces a base model: a superb text predictor that will happily continue your question with more questions, imitate the style of whatever you paste, and comply with or refuse requests essentially at random. Everything users actually experience as “the model” is installed afterward, in post-training, for a tiny fraction of the pretraining budget. That asymmetry, cheap stage, decisive effect, is why the masterclass spends two full parts here.

SFT: behavior cloning on demonstrations

Supervised fine-tuning is the first stage, and the mechanism is deliberately boring: it is the Chapter 2 training loop, run on a special dataset. The dataset is pairs: a prompt, and the response a good assistant should have given. Chat transcripts with role markers, questions with worked answers, instructions with ideal completions, tool calls with correct arguments. The model predicts the response token by token, the loss scores it against the demonstration, and the knobs adjust. The model is quite literally cloned from the behavior in the data.

The security-training analogy is close: this is a junior analyst shadowing a senior’s closed tickets, absorbing format, thoroughness, and house style by imitation. It works the same way and fails the same way. The student learns the ceiling of the demonstrations, including their errors. The student learns style faster than substance, so fluent, well-formatted wrongness is SFT’s signature failure. And the student never learns to do better than the examples, because nothing in the loop rewards exceeding them. Those three limits are why SFT is the beginning of post-training and not the end, and Part 7’s real subject is the dataset: whose demonstrations, how many, how filtered, in what order. In SFT, data curation is the whole craft.

Where do demonstrations come from at scale? Humans write some. Increasingly, stronger models generate most of them, a fact that connects SFT to distillation below and to the masterclass’s recurring legal question of whose outputs may train whom.

A practical variant dominates at small scale, including in LAB2’s plans: instead of updating all parameters, you freeze the base model and train a small set of added matrices, an adapter (LoRA; quantized base plus adapter is QLoRA). The adapter is a small file that steers the frozen model. It cannot teach as much as full training, and it cannot install knowledge the base lacks, but it is cheap enough to run on one good GPU, which is why it is the workhorse of specialization projects. Part 7 treats adapter choices as first-class recipe decisions.

Preference data: teaching taste

SFT teaches the model to imitate. It does not teach the model to choose. Preference tuning starts where imitation ends: show the model two responses to the same prompt, one marked better, and train it to favor the better one’s style of answer. The “better” judgments come from human raters, from stronger models acting as judges, or from rubrics. Aggregated over hundreds of thousands of pairs, this installs taste: prefer the answer that is correct over the one that is merely confident, prefer concise over padded, prefer declining gracefully over moralizing.

The widely used direct method is DPO (Direct Preference Optimization), which trains the model on the pairs with a classification-style loss: raise the probability of the chosen response, lower the rejected one, stay close to the SFT model. Its popularity is an engineering story: it skips the separate reward model and the RL loop that the older RLHF pipeline required, and it is stable and cheap. Its structural limit matters for Part 8: it learns only from the fixed pairs it is given, so it cannot discover responses better than anything in its dataset. It polishes; it does not explore.

RL in one honest page

Reinforcement learning removes the ceiling. The setup has three pieces. The policy is the model itself, the thing that generates responses. The environment is wherever the response lands: a chat, a test suite, a sandbox. The reward is a score for how the response did. RL training is a loop: the policy generates responses to many prompts, each response gets scored, and the knobs adjust so that high-scoring behavior becomes more likely and low-scoring behavior less. It is gradient descent (Chapter 2) where the training data is the model’s own fresh output and the loss is built from rewards.

Two flavors of reward decide everything.

Learned rewards come from a reward model, itself trained on human preference data. This is classic RLHF. It can score anything, including style, which is its power, and it can be gamed, which is its curse. A policy trained long enough against a learned reward learns to exploit the reward model’s blind spots: verbosity that reads as thoroughness, confidence that reads as competence. The field calls this reward hacking, and a security reader needs no translation: any scored system gets optimized into its own loopholes, exactly like a detection rule tuned until the red team routes around it.

Verifiable rewards come from a checker that cannot be flattered. The code compiles and the tests pass, or they do not. The math answer matches. The exploit runs on the replica, or it crashes. RL with verifiable rewards (RLVR) is the engine of the reasoning-model era, because the reward is ground truth: free, tireless, and ungameable by style. Its boundary is equally clear: it only works where a reliable checker exists. Part 8 compares how each family balances the two flavors, and Part 11 builds the cyber program on verifiable rewards precisely because cyber work is unusually checkable.

The algorithm name you will see most is GRPO: sample a group of responses to the same prompt, score them all, and push the policy toward the ones that beat the group average. No separate value model, just relative judgment inside the group. One honest caution to carry: RL amplifies what the reward measures and quietly taxes what it does not. Reports in Part 8 show safety and general scores dipping as reasoning scores climb under RL, a trade the field now budgets for rather than discovers afterward.

Distillation: compression of one model into another

Distillation trains a student model on a teacher model’s behavior. In the common form, the teacher generates a large dataset of responses, and the student is SFT-trained on them, which makes distillation a way to buy SFT data with a capability ceiling far above human-written demonstrations. In the stronger form, the student matches the teacher’s full probability distributions, learning not just what the teacher said but how sure it was.

The point is economics. A frontier teacher is expensive to run and may be closed entirely. A distilled student carries a useful fraction of the skill at a fraction of the serving cost, on hardware you own. The inheritance warning applies with force: the student absorbs the teacher’s errors, style, and blind spots along with its skill, and the teacher’s license terms decide whether its outputs may legally train your student at all, a constraint Part 8 and LAB2 treat as binding. A modern refinement, on-policy distillation, has the student generate its own responses and the teacher grade them token by token, fixing the mismatch where a student trained only on teacher text fails in situations the teacher never visited. Nemotron 3 Ultra’s multi-teacher version of this (MOPD) is Part 8’s centerpiece.

Refusal: a trained behavior, not a filter

It is worth stating plainly, because Part 11 depends on it: what a model refuses is decided in post-training, by data. The SFT set includes demonstrations of declining certain requests; preference and RL stages reward declining and punish complying, category by category. The resulting behavior is a property of the weights, distributed and statistical, not a rule list in front of the model. Three consequences follow. Refusal can be measured, per category, the way Part 11 measures it per vulnerability class. It can be shifted by further training, in either direction, including the weight-surgery techniques (abliteration) that Part 8 documents for stripping it back out. And it is never absolute: it is a tendency with a rate, which is why the masterclass reports refusal as a measured percentage under stated conditions, never as a binary.

Check yourself

  1. What exactly is the training data in SFT, and what is the model trained to do with it?
  2. Name SFT’s three structural limits.
  3. What are the three pieces of any RL setup?
  4. Why is a verifiable reward harder to game than a learned reward, and what is its limitation?
  5. How does distillation differ from ordinary SFT?
  6. A vendor says their model “has a safety filter.” Given this chapter, what do you ask next?
Answers
  1. Prompt and desired-response pairs (demonstrations). The model is trained by the standard loss loop to predict the demonstrated response, cloning the behavior in the data.
  2. It cannot exceed the quality of its demonstrations; it teaches style faster than substance, producing fluent wrongness; and nothing in the loop rewards discovering better responses than the examples.
  3. The policy (the model generating), the environment (where responses land and get used), and the reward (the score that drives the update).
  4. A checker returns ground truth (tests pass, the exploit runs), so style cannot inflate the score; there is nothing to flatter. Its limitation is scope: it only exists for tasks with a reliable checker.
  5. The demonstrations are generated by a stronger teacher model rather than written by humans, and in the strong form the student matches the teacher’s probability distributions, not just its final text. The student inherits the teacher’s errors and the teacher’s license terms along with its skill.
  6. Whether the refusal behavior lives in the weights (trained, statistical, measurable as a rate) or in a wrapper around them, and either way: measured how, on what categories, at what rate, on the exact artifact being sold.

Go deeper

  • Nathan Lambert, The RLHF Book (https://rlhfbook.com/). The full arc from SFT through reward modeling, DPO, and RLVR, free online, written for engineers. This is the textbook for Parts 7 and 8; read the SFT and direct-alignment chapters first.
  • Sebastian Raschka, Build a Reasoning Model (From Scratch), repository (https://github.com/rasbt/reasoning-from-scratch). GRPO and distillation implemented small enough to read, on a real base model. Brandon owns the book in audio; the repo is its code twin.
  • Hugging Face smol-course (https://github.com/huggingface/smol-course). Hands-on SFT and preference tuning with the actual tooling (TRL) on a small open model. The fastest route from this chapter to running the loop yourself.
  • Andrej Karpathy, “Deep Dive into LLMs like ChatGPT” (https://www.youtube.com/watch?v=7xTGNNLPyMI). Its final third is the clearest spoken explanation of RLHF and reward hacking available; watch it with Part 8 open.
  • Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models” (https://arxiv.org/abs/2106.09685). The adapter paper, short and foundational. Read it before Part 7’s adapter discussions; QLoRA is its quantized descendant.
Where this leads

This chapter unlocks Part 7, Part 8, Part 11 of the Model-Making Masterclass.