Foundations: Prerequisites for the Model-Making Masterclass

Chapter 6 · Ch 6: Reading the Artifacts

Chapter 6 · Chapter 6: Reading the Artifacts

Listen: audio twin

Unlocks: Masterclass Parts 1 (openness), 9 (evaluation), 12 (synthesis), and 13 (generations). The masterclass’s core skill is not training models. It is reading the evidence labs publish about models and knowing which claims the evidence supports. Labs publish three artifacts that matter: the model card, the config file, and the tech report. Each has a different evidentiary weight, and confusing them is how marketing becomes fact. This chapter teaches the reading order the masterclass uses: card for claims, config for truth about shape, report for method, license for permission.

The model card: a claim sheet, not a measurement

A model card is the README that ships with the weights, usually on Hugging Face. It typically contains the model’s name and lineage, a benchmark table, a license tag, training data described in prose, and usage notes. Read it the way you read a vendor datasheet or an SBOM: as a structured set of claims by an interested party, valuable precisely because it commits the vendor to specifics you can check.

Four habits make card reading reliable.

Find the lineage. The card should say what the model was trained or fine-tuned from. A checkpoint post-trained from another lab’s base inherits that base’s license and some of its behavior, and cards do not always volunteer this in the headline. Part 1 calls the stacked result a license chain and grades a derivative by the weakest link in it.

Date the numbers. Benchmark tables are snapshots. Note which benchmark versions and which dates; suites get revised, and a score against last year’s version of a test is not comparable to this year’s.

Ask what is missing. A card with no training data section, no evaluation settings, and no stated limitations is telling you something by omission. The masterclass flags absences rather than filling them with assumptions, and Part 1’s openness ladder is essentially a formalized list of what might be missing: data, code, configs, intermediate checkpoints, logs, the eval harness itself.

Never trust the license tag alone. The card’s license field is a label. The governing text is the license file and the card’s own terms block, and Part 1 documents cases where the tag, the card text, and redistributors’ claims disagree. When they disagree, the release is not commercially cleared until the actual text is read. That is a legal read, not a benchmark read, and it comes first in LAB2’s discipline for a reason.

The config.json: the shape, in the open

Every Hugging Face release ships a small JSON file, config.json, that describes the architecture exactly, because the serving code needs it to build the model. It is the least marketable and most honest document in the release. Nobody writes ad copy in a config file.

Here is a real one, trimmed: SmolLM3-3B, a dense 3B model from Hugging Face. The fields that matter:

  • hidden_size: 2048. The width of the vectors flowing through the model (Chapter 3’s residual stream is 2,048 numbers wide).
  • num_hidden_layers: 36. Depth: 36 transformer layers.
  • num_attention_heads: 16 and num_key_value_heads: 4. Sixteen query heads sharing 4 key-value heads: grouped-query attention, the KV cache compression from Chapter 3, visible in two integers.
  • vocab_size: 128256. The tokenizer’s vocabulary (Chapter 1). Multiply by hidden size and you get the embedding table: 128,256 times 2,048 is about 263 million parameters, roughly 9 percent of this model, spent before the first layer.
  • tie_word_embeddings: true. The output head reuses the input embedding table, so that 263M is spent once, not twice. In models that do not tie, the table costs double, which is how a “7B” nameplate quietly becomes 9B of actual parameters (Part 3’s central lesson).
  • max_position_embeddings: 65536. The trained context ceiling, 64K tokens, whatever the launch post rounds it to.
  • no_rope_layer_interval: 4. Every fourth layer skips rotary position encoding: the NoPE pattern from Chapter 3, disclosed in the config even if the blog post never mentions it.

For a Mixture of Experts model, the config adds the fields that produce the honest size: num_experts (or routed experts per layer), num_experts_per_tok (how many fire), and a shared-expert flag. Total parameters come from the whole config; active parameters come from the per-token path through it. When a headline says “30B” and the config says 128 experts with 6 active, you can compute both numbers yourself and check the headline against them. That computation, nameplate versus actual, total versus active, is the single most repeated move in the masterclass, and it takes ninety seconds per model once you know the fields.

The weights themselves ship as safetensors files: a simple, auditable format that stores tensors and nothing else, and cannot execute code on load, unlike the older pickle-based formats. Treat that the way you treat any supply chain property: the format is a security decision the ecosystem already made for you, and a release that still ships pickle checkpoints is telling you something about its hygiene.

The tech report: method, with an interest

A tech report is the long document: architecture rationale, data recipe, training configuration, stability incidents, ablations, evaluation tables with settings. It is the most valuable artifact and the one to read most skeptically, because it is a scientific paper written by the marketing department’s colleagues. The masterclass’s reading protocol:

Separate measurement from narrative. The tables are measurements (under stated conditions). The surrounding prose is the lab’s theory of why the measurements came out that way. Theories are cheap; Part 13’s discipline is to record what each generation’s report credits for its gains, labeled as the lab’s own attribution, never upgraded to established fact.

Hunt the conditions. Every score has a harness (the software that ran the eval), a prompt format, sampling settings (Chapter 1), and a precision (Chapter 4). A score measured on the BF16 original does not describe the 4-bit quantized build users actually download. The masterclass’s rule, stated in Part 10 and enforced in Part 11: a benchmark is a property of an artifact at a precision, not of a model name. When a report quotes a number for a different artifact than the one shipping, that is a finding, and Part 9 collects cases.

Value the negative space. The most credible reports publish what failed: loss spikes and rollbacks, ablations that lost, data mixes that were tried and dropped. A report with no failures is a brochure. The open families in this study earn their anchor status partly by publishing their incidents, which is why their recipes can be learned from rather than merely admired.

Check the harness against the claim. Scores are comparable only when the same harness, prompts, and scoring produced them. The standard open harness is EleutherAI’s lm-evaluation-harness; serious families publish their harness or their exact settings with the numbers. Two scores for the same checkpoint from different harnesses can differ by points, and neither is lying. They are different measurements wearing the same benchmark name.

The license: the artifact that outranks the benchmarks

You already know how to read licenses from open source work; the model world adds three wrinkles. First, weights licenses are often custom texts, not Apache or MIT, with clauses software licenses never carry, including termination triggers tied to guardrail violations and rights for the vendor to update terms unilaterally. Second, licenses are per checkpoint, not per family: one lab’s roster can span six different license names, and the same model can carry different terms on different distribution surfaces. Third, derivatives stack: your obligations are the union of every license in the chain from base to checkpoint. Part 1 works the cases, including one where a license conflict leaves a whole model line commercially unusable until someone reads one card’s text directly. The takeaway for this prerequisite is procedural: in any build decision, the license read happens before the benchmark read, because a model you cannot ship has no performance characteristics that matter.

Check yourself

  1. What is a model card, evidentiary speaking, and what are two things you check on it first?
  2. From a config with vocab 128,256 and hidden size 2,048, how many embedding parameters are there, and what changes if embeddings are not tied?
  3. A headline says 30B. The config shows 128 experts per layer, 6 active per token. Why is the headline incomplete rather than wrong?
  4. A report quotes a benchmark for the BF16 model; the download page ships a 4-bit build. What is the status of that number?
  5. Two harnesses report scores 4 points apart for the same checkpoint on the same benchmark. Who is lying?
  6. Why does the masterclass read the license before the benchmarks?
Answers
  1. A structured claim sheet by an interested party. Check the lineage (what it was derived from, since licenses and behavior inherit) and the completeness of its disclosures (data, eval settings, limitations), treating absences as findings.
  2. About 263 million. If embeddings are not tied, the output head is a second table of the same size, so the vocabulary costs roughly 526 million parameters in total.
  3. Because an MoE model has two sizes: total parameters stored (the headline) and active parameters per token (what computes). The config lets you compute the active figure; without it, cost and speed comparisons against dense models are meaningless.
  4. It describes a different artifact than the one being shipped. Until the suite is rerun on the 4-bit build, the number is a claim about a model nobody can download.
  5. Probably neither. Different prompt formats, sampling settings, and scoring rules produce different measurements under the same benchmark name. Comparability requires the same harness and settings, which is why families that publish their harness earn more trust.
  6. Because commercial usability is a legal property decided by the license chain, including termination and update clauses and stacked upstream terms. A model that cannot be shipped has no benchmark that matters, and license conflicts have voided otherwise strong candidates in this study.

Go deeper

Where this leads

This chapter unlocks Part 1, Part 9, Part 12, Part 13 of the Model-Making Masterclass.