Five things to carry through every chapter

• A reasoning trace is a proposal. Correctness belongs to a verifier, a tool result or an accountable human decision.
• More thinking is not automatically better. Budget it against consequence, ambiguity, latency and the value of another sample.
• Evaluation is part of the system, not a score collected after the build. Extraction and ground truth can fail as easily as the model.
• Reinforcement learning magnifies the reward specification. Monitor reward spread, entropy, clipping and held-out task quality together.
• Distillation transfers useful strategies and useless tics. Filter teacher traces, challenge the student with fresh cases and keep the release gate independent.

Reading route

Read Chapters 1 to 3 to understand the engine and its measurement boundary. Chapters 4 and 5 cover inference-time interventions. Chapters 6 and 7 build and stabilise the learning loop. Chapter 8 treats distillation as controlled transfer rather than compression alone. The public-pattern appendix extracts enduring lessons without pretending that vendor line-ups stand still. The final Merehaven portfolio is wholly fictional and uses synthetic records to show how regulated workflows change the design.

Edition boundary

The numerical results in the worked experiments are laboratory fixtures. They explain how to measure a small model, not what every device, model family or dataset will produce. Re-run them on the named checkpoint, hardware and test split before using them as a baseline. Product names in historical discussion are orientation points, not a current buying guide.

Why should a next-token predictor show its work?

A next-token mechanism poses a hard question. A large language model is, structurally, a next-token prediction machine. You feed it a sequence of tokens, it produces a probability distribution over all possible next tokens, and it picks one. Then it does it again. And again. That is the entire mechanism. The system exposes no explicit symbolic proof engine that can certify the answer before generation. Just one token after another, each chosen because it is statistically likely given everything that came before.

And yet, when you ask a reasoning model like DeepSeek-R1 to solve a multi-step algebra problem, it produces something that looks exactly like the step-by-step working a human would write on a whiteboard. It identifies the given values, selects an appropriate formula, substitutes, simplifies, and boxes the answer. It even catches its own arithmetic mistakes and corrects them mid-stream. How does a next-token predictor do this?

The answer is mechanical but consequential. The model does not reason the way you reason. It does not maintain internal logical axioms or apply inference rules. What it does is generate tokens that follow the pattern of reasoning, and in doing so, it follows learned token patterns that can produce useful intermediate steps and sometimes a correct answer. The act of generating those intermediate steps is not cosmetic. It materially improves accuracy. That is the core discovery that this entire book unpacks.

Sebastian Raschka, whose Build a Large Language Model (From Scratch) became a foundational text for practitioners who want to understand LLMs from the inside out, defines reasoning in LLMs with deliberate pragmatism: reasoning means the model generates explicit intermediate steps before a final answer, where those steps measurably improve accuracy on complex tasks. This is not a claim about consciousness, understanding, or genuine cognition. It is an engineering definition. It says: if a model shows its work and gets more problems right as a result, we call that reasoning. A reasoning model is then formally an LLM that has been improved to explain its steps before giving an answer, where those intermediate steps increase accuracy on complex tasks such as coding, logical puzzles, and math problems.

The useful distinction is the causal link: the act of generating intermediate steps is not just cosmetic transparency. It actually improves performance. That measurable difference is the phenomenon to explain. The terms "reasoning" and "thinking" are used throughout this book as they are commonly used by researchers and engineers working on LLMs. However, this does not imply that LLMs actually reason or think in the same way humans do. Whether LLMs engage in genuine cognition or merely produce convincing simulations of cognition remains an open and actively debated research question. We can set that philosophical debate aside and test the engineering claim.

Apple's research team published a 2025 paper titled "The Illusion of Thinking" which found that even strong contemporary reasoning models are sophisticated pattern matchers that break down on problems requiring truly novel logical inference outside their training distribution. The paper tested reasoning models on graph coloring problems of increasing complexity and found that performance collapsed beyond a certain complexity threshold, regardless of how much inference-time compute was allocated. This finding does not diminish the practical value of reasoning models. A pattern matcher that solves 52% of competition-level math problems is substantially useful, even if a philosopher would not call it genuine reasoning. The practical question driving this book is not whether the reasoning is real but whether it is reliable enough to deploy, and on math, code, and logic benchmarks, it demonstrably is.

To see concretely why reasoning matters, picture two scenarios. In the first, you ask a conventional, non-reasoning LLM "What is the tallest mountain in the world that has never been climbed?" The model outputs a terse answer: "Gangkhar Puensum." It might be correct, but it does not show how it arrived there, and there is no mechanism for catching errors if the reasoning required multiple steps.

In the second scenario, you ask a reasoning LLM: "Alice has 3 apples and Bob has 5 apples. Alice gives 1 apple to Bob. How many apples do they have together?" Rather than immediately outputting "8," the reasoning model generates explicit intermediate steps. Step 1: identify the initial counts (Alice = 3, Bob = 5). Step 2: update after the transfer (Alice = 2, Bob = 6). Step 3: compute the total (2 + 6 = 8). Only then does it produce the final answer. The intermediate reasoning steps may or may not be shown to the user. Some models like DeepSeek-R1 show the full chain-of-thought in <think> tags; others like OpenAI's o1 initially hid the reasoning trace and showed only a summary.

The important point is to emphasize that LLM reasoning differs materially from traditional, deterministic reasoning. A symbolic logic engine or theorem prover follows strict, rule-based steps that guarantee consistency and correctness. Treat a symbolic logic engine like following a recipe where every step is fixed and must produce the same result each time. These systems are deterministic: given the same inputs and rules, they always produce the same outputs.

In contrast, an LLM generates reasoning autoregressively, predicting one token at a time based on statistical patterns in its training data. The model's "reasoning steps" are not guaranteed to be logically sound, even if they look convincing. The model might produce a chain-of-thought that reads perfectly but contains a subtle logical error, a skipped step, or an incorrect intermediate calculation. This is one of the core challenges that the techniques in this book aim to mitigate: by using methods like self-consistency (Chapter 4), self-refinement (Chapter 5), and reinforcement learning with verifiable rewards (Chapter 6), we can significantly reduce the frequency of these errors even though we cannot eliminate them entirely.

How does a conventional LLM learn to speak?

To understand where reasoning methods fit, you need to understand the assembly line that produces a conventional LLM. Treat it as building a musician in three stages.

Stage one is ear training. You lock a student in a room with every recording ever made and tell them: "Listen to a phrase, then predict the next note." This is pre-training. The LLM ingests trillions of tokens from books, websites, research papers, and code repositories. Its only objective is to predict the next token given all the preceding ones. This sounds trivially simple, and the objective is. But the scale is not.

Before diving into the scale, a word about tokens. A token is a small unit of text that a language model processes. It can be a full word, part of a word, or even punctuation, depending on the tokenizer. For example, the sentence "An LLM can be useful." might be broken into tokens like "An", " L", "LM", " can", " be", " useful", and "." by a common tokenizer such as Byte Pair Encoding (BPE). These tokens are then converted into numerical IDs that the model can ingest. Chapter 2 covers tokenization in detail.

Thought experiment · what if there were no tokenizer?

Consider a model operating on raw characters instead of tokens. "Explain large language models." contains 31 characters. Each character would be a separate input. Since attention cost scales as O(N²), the character-level model's attention would be 25 times more expensive than the tokenized version (31² vs 6²). And generation would require 31 forward passes instead of 6.

This is why tokenization exists: it compresses text, making both training and inference materially more efficient. BPE finds the optimal tradeoff: common words become single tokens (maximum compression), rare words are split into subwords (graceful degradation). The result is near-optimal compression across the training distribution.

Understanding model size and its implications

To make "621 million parameters" tangible, consider that each parameter is a single floating-point number stored in bfloat16 format (2 bytes). That means the model's weights occupy 621,000,000 × 2 = 1.24 gigabytes. If you printed each parameter value on a line of text, the printout would be approximately 12 million pages long, or about 24,000 standard reams of paper stacked 120 meters high, roughly the height of a 40-story building.

But parameters are not randomly scattered numbers. They are organized into structured matrices inside the model's layers. The embedding layer alone, which maps 151,936 token IDs to 1,024-dimensional vectors, contains 151,936 × 1,024 ≈ 155.6 million parameters, one-quarter of the total. The output head, which maps 1,024-dimensional hidden states back to 151,936 vocabulary scores, contains another 155.6 million parameters. Together, these two vocabulary-dependent layers account for half the model. The remaining half lives inside the 28 transformer blocks, with each block containing approximately 16 million parameters split between the attention mechanism and the feedforward network.

This distribution has a practical consequence: if you wanted to adapt this model to a different language with a different vocabulary, you would need to replace half the parameters (the embedding and output layers) while keeping the other half (the transformer blocks). The transformer blocks encode general reasoning patterns. The vocabulary layers encode language-specific token representations. This separation is one reason why the same transformer architecture works for English, Chinese, code, and mathematics: the reasoning machinery is language-agnostic.

Now, the scale. Pre-training DeepSeek V3, the base model behind the DeepSeek-R1 reasoning system, required 2,048 NVIDIA H800 GPUs running for 11 weeks. According to its technical report (one of the few recent models with fully transparent compute disclosure), the final training run consumed 14. 8 million GPU-hours of compute. The energy usage was roughly 620 megawatt-hours, which is approximately the amount of electricity an average American household uses in about 55 years. The estimated cost was 5. 5 million USD. These numbers make clear why training LLMs from scratch is not feasible for most practitioners and researchers. After this brutal immersion, the model can produce fluent, coherent text. It knows facts. It can translate languages it was never explicitly taught to translate. Pre-trained LLMs also begin to exhibit emergent properties, meaning they can perform tasks they were never explicitly trained to do.

But a raw pre-trained model responds to questions the way a book responds to a bookmark: it continues the text, rather than answering the question.

Stage two is learning to follow instructions. You take the ear-trained musician and teach them to play requests. "Play something in B minor." "Transpose this for guitar." This is supervised fine-tuning (SFT), sometimes called instruction tuning. The model is trained on curated datasets of instruction-response pairs: "Summarize this article" paired with an ideal summary, "Translate this sentence" paired with a correct translation. The improvement is dramatic: tasks like question answering, summarization, translation, and code generation all become substantially more reliable.

Stage three is learning taste. You play the musician's performances to an audience and let them vote on which version they prefer. This is preference tuning, typically implemented through Reinforcement Learning with Human Feedback (RLHF) or its more recent alternative, Direct Preference optimisation (DPO). Human evaluators rank multiple model responses, and these rankings are used to train a reward model that guides further optimisation. The practical consequence is that preference tuning is about polish and alignment: making the model's outputs not just correct but pleasant, safe, and helpful.

After these three stages, you have a conventional LLM. It is fluent, obedient, and well-mannered. But it is not a reasoning model. even an instruction-tuned and preference-tuned model is not yet a "chatbot." A chat interface adds another layer: a system prompt, conversation history management, and orchestration logic. The book provides an implementation example in Appendix G.

The penguin problem, or why pattern matching is not enough

During pre-training, LLMs are exposed to vast quantities of text and learn to predict the next token by identifying and reproducing statistical associations. This process enables fluent and coherent text generation, but it is materially rooted in surface-level correlations rather than deep understanding.

Consider the prompt "The capital of Germany is..." An LLM will almost certainly produce "Berlin." But it is not logically deducing the capital from first principles or consulting an internal database. It is recalling a strong statistical association learned from training data, where "capital of Germany" was overwhelmingly followed by "Berlin" across millions of web pages. For factual recall tasks like this, pattern matching works extremely well.

But what about tasks that go beyond pattern recognition? Consider the following: "All birds can fly. A penguin is a bird. Can a penguin fly?"

There are two valid ways to evaluate this, depending on the reasoning framework. In a closed-world setting, where only the prompt's information is considered, the two premises logically entail "Yes, a penguin can fly." This follows from a straightforward syllogistic inference called modus ponens applied to a universal quantifier: if all members of category X have property Y, and entity Z is a member of X, then Z has property Y.

In an open-world setting, where background knowledge is also allowed, the well-known fact that penguins cannot fly conflicts with the derived conclusion. A well-tested reasoning system should notice this inconsistency.

When tested with this exact prompt, GPT-4o produces a notably sophisticated response. It acknowledges the logical structure of the syllogism while immediately flagging that the premise is factually wrong. It identifies the contradiction and even offers to formalise the reasoning using symbolic logic.

This response appears to demonstrate genuine logical reasoning. But does it? The distinction is no, not necessarily. GPT-4o does not implement explicit contradiction-checking. It does not maintain a formal model of the world. Instead, its training data contains millions of instances that correct this specific contradiction: there are countless web pages, textbooks, and discussion forums that explicitly state "penguins cannot fly." The model has learned a strong statistical association between "penguin" and "not flying" that overrides the in-context premise.

This is what makes conventional LLMs simultaneously impressive and fragile. The model appears to reason logically, but it is actually recognising a familiar pattern. The distinction matters substantially because pattern-based reasoning breaks down in two specific situations.

First, when the logical scenario is novel. If you construct a syllogism using entirely made-up terms ("All glorps are snazzles. A fimble is a glorp. Is a fimble a snazzle?"), the model cannot fall back on memorised patterns. It must actually perform logical inference, which it often fails to do reliably.

Second, when reasoning complexity is high, involving intricate, multi-step logical relationships where the model must maintain consistency across many intermediate steps. Each additional step introduces opportunities for the probabilistic generation process to drift off track.

We might say that LLMs simulate logical reasoning through learned patterns, and we can improve this simulation further with specific reasoning methods. But they are not explicitly executing any rule-based logic internally. Even before the advent of dedicated reasoning models such as o1 and DeepSeek-R1, LLMs were capable of simulating reasoning behaviour. What we now explicitly label a "reasoning model" is a more refined, more reliable version of this capability.

Why are explicit rule-based systems not more popular if they can guarantee logical consistency? Rule-based systems were widely used in the 1980s and 1990s for medical diagnosis, legal decisions, and engineering applications. They are still used in critical domains. However, they are materially limited by their reliance on human-crafted heuristics. Building comprehensive rule sets for complex, open-ended domains is extraordinarily labour-intensive and brittle: every edge case requires a new rule, and rules can interact in unexpected ways. LLMs, by contrast, learn from data and can handle novel inputs without explicit programming for each scenario. The tradeoff is that they lose the formal guarantees of correctness.

As of this writing, popular reasoning models include Anthropic's Claude 4, xAI's Grok 4, Google's Gemini 2.5, DeepSeek's R1, Alibaba's Qwen3, and many more. The techniques employed by these models are the focus of this book.

Thought experiment · the made-up syllogism test

We can conduct a thought experiment that reveals exactly where pattern matching fails. We construct a syllogism using entirely invented terms:

"All zerplings are quilnox. A bramwit is a zerpling. Is a bramwit quilnox?"

A rule-based logic engine answers in microseconds: Yes. The structure is identical to "All A are B. C is an A. Is C a B?" The content of the terms is irrelevant.

Now test this on a conventional LLM. Many models answer correctly, not because they are performing logical inference, but because the structure "All X are Y. A Z is an X. Is a Z a Y?" appears frequently in logic textbooks and philosophy courses in the training data. The model has memorised the pattern of syllogistic reasoning even with unfamiliar terms.

The real test requires one more step. Add a contradicting premise: "All zerplings are quilnox. A bramwit is a zerpling. Bramwits are not quilnox. Is a bramwit quilnox?"

Now the model faces a genuine logical challenge. The first two premises imply "yes." The third premise says "no." A formal reasoner would flag the contradiction. A pattern-matching model has no mechanism for detecting contradictions between premises because it does not maintain a formal model of the world. It predicts the next most likely token given the preceding context, and if the context contains contradictory information, the model simply follows whichever statistical pattern is strongest.

In experiments, conventional LLMs often answer "No" to this contradicted syllogism, not because they detected the logical contradiction, but because the explicit statement "Bramwits are not quilnox" is the most recent and most directly relevant piece of context. The model is doing recency-biased pattern matching, not contradiction detection. A reasoning-enhanced model (like those built in this book) performs significantly better on these tests because its intermediate steps make logical structure more explicit.

This distinction matters for production systems. If you are building a legal document analyzer that must detect contradictory clauses in a contract, a conventional LLM might miss contradictions between clauses that are separated by many paragraphs. A reasoning model that generates explicit intermediate steps ("Clause 3 states X. Clause 17 states not-X. These are contradictory.") is more likely to catch the conflict because the intermediate steps force the relevant information into the same context window.

A system design exercise · when to reason and when not to

This is a constructed design exercise, not a report of a named deployment. Consider you are the ML architect for a customer service platform that handles 10 million queries per day. Each query costs approximately $0.001 with a conventional LLM and $0.015 with a reasoning model (the reasoning model generates 10x more tokens and uses a more expensive inference pipeline). Your annual LLM budget at the conventional rate is $3.65 million. At the reasoning rate, it would be $54.75 million. You cannot afford to use reasoning for everything.

The engineering challenge is routing: which queries need reasoning and which do not?

Category 1: "What are your business hours?" This is factual recall. Pattern matching handles it perfectly. No reasoning needed. Cost: $0.001.

Category 2: "I was charged $47.99 but my plan says $39.99. Can you explain the difference?" This requires multi-step reasoning: look up the plan, identify the base price, check for add-ons or overage charges, compute the difference, and explain. A conventional LLM might hallucinate an explanation. A reasoning model would work through the steps. Cost: $0.015.

Category 3: "If I switch from the Premium plan to the Basic plan mid-cycle, keep my add-on for international calls, and apply my loyalty discount, what will my next bill be?" This is a multi-step arithmetic problem with conditional logic. It absolutely requires reasoning. Cost: $0.015.

A simple routing heuristic: if the query contains arithmetic, conditional logic, or requires combining information from multiple sources, route to the reasoning model. Otherwise, use the conventional model. In practice, this splits roughly 80/20, sending 8 million queries to the cheap model and 2 million to the reasoning model. Annual cost: ($0.001 × 8M × 365) + ($0.015 × 2M × 365) = $2.92M + $10.95M = $13.87M. That is a 75% reduction from the all-reasoning budget.

The techniques in this book give you the understanding to make these routing decisions intelligently. You will know which techniques (CoT, self-consistency, GRPO, distillation) are appropriate for which complexity levels, and you will know the compute-accuracy tradeoff for each.

Three pillars, one model

The approaches to improving reasoning in LLMs fall into three categories, and understanding their relationship is essential before you write a line of code. The announcement of OpenAI's o1 on September 12, 2024 brought reasoning into the mainstream. A few months later, in January 2025, DeepSeek released DeepSeek-R1 along with a detailed technical report. DeepSeek-R1 was transformative for two reasons. First, it was freely and openly available, competing with and in some benchmarks exceeding the proprietary o1. Second, the accompanying technical report provided a detailed blueprint for how to train such a model, making the methodology accessible to the entire research community.

The spectrum from recall to reasoning · five levels

It helps to treat LLM capability as a spectrum with five distinct levels:

Level 1: Factual recall. "What is the capital of France?" No intermediate steps. Any pre-trained LLM handles this. Accuracy: 95%+.

Level 2: Template application. "Convert 72°F to Celsius." One formula, one step. Pre-trained LLMs: 70-85%.

Level 3: Multi-step composition. "A car travels 60 mph for 2.5 hours, then 45 mph for 1.5 hours. Total distance?" Two multiplications and an addition, requiring intermediate results across three steps. Without CoT: 30-50%. With CoT: 60-80%.

Level 4: Strategy selection. "Find all prime factors of 1,764." The model must choose a factoring strategy, then execute it consistently. Without RL: 20-40%. With RL: 50-70%.

Level 5: Novel composition. "Prove that for n > 1, n³ - n is divisible by 6." Construct a proof from scratch. Even the best models: 40-60%. This is the research frontier.

CoT (Chapter 4) addresses Level 3. GRPO (Chapters 6-7) addresses Level 4. Distillation (Chapter 8) transfers Level 4-5 capabilities from large to small models. Inference-time scaling improves all levels by trading compute for accuracy.

Pillar 1: Inference-time compute scaling. Consider you are taking a math exam and the proctor says, "You can have as much time as you want." You would not write your answer faster. You would check your work, try multiple approaches, reread the question. Inference-time compute scaling does the same thing for an LLM. Without changing a single parameter in the model, you modify how it generates responses: add "Explain step by step" to the prompt (chain-of-thought prompting), generate five different answers and take the majority vote (self-consistency), or have the model critique and revise its own answer (self-refinement). These techniques require zero additional training. They work on any existing LLM. And they are notably effective: on MATH-500, adding "Explain step by step" to the prompt boosts the base Qwen3 0. 6B from 15. 2% to 40. 6% accuracy. That is a 25.

4 percentage point improvement from six words. This topic is the focus of Chapters 4 and 5.

Pillar 2: Reinforcement learning (RL). Now consider the proctor not only gives you unlimited time but also tells you, after each attempt, whether your answer is correct. Over many attempts, you learn which strategies lead to correct answers. RL updates the model's weights during training, enabling it to learn and refine reasoning strategies through trial and error based on feedback. The specific algorithm in this book is Group Relative Policy optimisation (GRPO), introduced by DeepSeek, which represents a significant simplification over earlier approaches like PPO. We explore RL in Chapters 6 and 7.

Thought experiment · reasoning as a routing problem

Here is a thought experiment that makes the economic argument vivid. Consider a hospital with two types of doctors. General practitioners (GPs) handle 80% of patient visits: colds, checkups, prescription renewals. They are fast and affordable. Specialists handle the remaining 20%: complex diagnoses, surgery, rare diseases. They are slower and expensive.

A hospital that sends every patient to a specialist would provide excellent care but go bankrupt. A hospital that employs only GPs would save money but miss dangerous conditions. The optimal system routes patients based on complexity.

LLM reasoning is exactly this routing problem. Conventional LLMs are GPs: fast, cheap, good enough for most queries. Reasoning models are specialists: slower, more expensive, but essential for complex problems. The techniques in this book teach you to build the specialist, understand when to call them, and optimise the cost of the consultation.

Concretely, the three pillars correspond to different levels of specialist intervention:

Inference-time scaling is like asking the GP to spend more time with the patient. Same doctor, same training, just more time per appointment. Cost: proportional to time spent. The GP might catch issues they would have missed in a rushed visit.

Reinforcement learning is like sending the GP back to medical school for additional training in a specialty. The doctor comes back permanently more capable. Cost: one-time training investment. The improved doctor handles complex cases routinely.

Distillation is like having the GP shadow a specialist for a month and learn their diagnostic patterns. The GP cannot become the specialist, but they can learn many of the specialist's techniques. Cost: the specialist's time (one-time) plus the GP's learning time (one-time). The GP is permanently more capable.

The practical sequence stacks all three: distill (shadow the specialist), then RL (additional specialty training), then inference-time scaling (spend more time per patient when the case is complex).

Pillar 3: Distillation. Finally, consider that instead of learning by trial and error, you have access to a master tutor who shows you exactly how to solve each problem. You study the tutor's worked solutions and learn to reproduce their reasoning patterns. This is knowledge distillation: a small "student" model is trained on the chain-of-thought solutions generated by a large "teacher" model. Within the LLM context, this typically means performing supervised fine-tuning using high-quality labeled instruction datasets generated by a larger, more capable model. It differs slightly from traditional knowledge distillation in deep learning, where the student model typically learns from both the outputs and the logits (raw pre-softmax scores) produced by the teacher. In LLM reasoning distillation, we usually rely only on the teacher's text outputs.

This is faster and cheaper than RL: 3 hours and 15 GB of memory versus 12 hours and 70 GB for GRPO on the same hardware. This topic is covered in Chapter 8.

These three pillars are not competing alternatives. They are complementary layers of a practical sequence: distill first (cheap warm start), then refine with RL (targeted improvement), then apply inference-time scaling at deployment (maximize accuracy per query).

The distinction between RL for reasoning and RLHF for preference tuning deserves emphasis. Both use reinforcement learning, but they differ in how the reward is obtained. RLHF incorporates explicit human evaluations as reward signals. RL for reasoning (called RLVR, reinforcement learning with verifiable rewards) relies on automated signals: a simple program checks whether the answer is right or wrong. The reward signal in reasoning RL is verifiable, which makes the training loop much more scalable than RLHF's reliance on expensive human annotation.

A worked example · the cost of reasoning at scale

This is a constructed design exercise, not a report of a named deployment. We can make the cost concrete with actual numbers. Suppose you deploy a math tutoring chatbot that serves 100,000 student queries per day. Each query is a math problem. You use the Qwen3 0.6B model on an NVIDIA H100 GPU.

Option A: Greedy decoding (no reasoning). Average response: 20 tokens. Speed: 141 tokens/sec (compiled + KV cache). Time per query: 0.14 seconds. Daily GPU time: 100,000 × 0.14s = 14,000 seconds ≈ 3.9 GPU-hours. H100 rental at $3/hour: $11.70/day. Accuracy: 15.2%.

Option B: CoT prompting. Average response: 200 tokens (10x longer reasoning traces). Speed: 141 tokens/sec. Time per query: 1.42 seconds. Daily GPU time: 142,000 seconds ≈ 39.4 GPU-hours. Cost: $118/day. Accuracy: 40.6%.

Option C: CoT + self-consistency (n=5). Five responses per query, each 200 tokens. Speed: 141 tokens/sec. Time per query: 7.1 seconds (sequential) or 1.42 seconds (5 GPUs parallel). Daily GPU time: 197 GPU-hours (regardless of parallelism). Cost: $591/day. Accuracy: 49.6%.

Option D: GRPO-trained model with greedy decoding. After 50 steps of GRPO training (one-time cost of ~$50 in compute), the model generates reasoning traces by default, averaging 150 tokens. Time per query: 1.06 seconds. Daily GPU time: 29.4 GPU-hours. Cost: $88/day. Accuracy: 47.4%.

Option E: Distilled model with greedy decoding. After distillation (one-time cost of ~$50 for teacher API + ~$10 for training compute), similar to Option D. Cost: $88/day. Accuracy: 45.0%.

The numbers tell the story. Option A is cheapest but nearly useless for a math tutor (15.2% accuracy). Option C is most accurate but 50x more expensive. Option D (GRPO) gives nearly the same accuracy as Option C at one-seventh the daily cost, with a one-time training investment of $50. This is why training-time techniques (Chapters 6-8) matter: they bake reasoning into the model's weights, avoiding the per-query cost of inference-time scaling.

Why you should build it, not just use it

In February 2025, OpenAI's CEO stated that GPT-4.5 (internally called Orion) would be the company's last model without chain-of-thought reasoning, and that a top goal going forward would be to unify reasoning and non-reasoning models into systems that "know when to think for a long time or not." The industry pivot was complete. Reasoning was no longer an optional add-on. It was becoming the default.

But knowing that reasoning models exist and knowing how they work are very different things. Using an API teaches you what a reasoning model does. Building one from scratch teaches you how and why.

This book uses Qwen3 0.6B as its base model throughout, a 621-million-parameter model from Alibaba's Qwen3 family. This is a deliberately small model. Raschka offers a useful analogy: if you are curious about how cars work, you would not start by building a Ferrari. You would build something like a Volkswagen Beetle. The engine is simpler, the components are visible, and you learn the same fundamental mechanics. The reasoning techniques applied to Qwen3 0.6B are identical to those used by models 1,000x its size. The difference is that the smaller model lets you see the mechanics clearly and run every experiment on consumer hardware, including CPUs for the inference chapters.

However, while the model is small, the reasoning techniques themselves are still computationally intensive. Chapters 2-5 can be executed in a reasonable time on a CPU. Chapters 5-8 benefit substantially from GPU access. The book recommends Lightning AI Studio or Google Colab for cloud GPU access.

Reasoning costs can be broken down into two components that compound multiplicatively. The length tax comes from longer responses: a reasoning model that generates 10x more tokens than a conventional model incurs 10x the per-token inference cost. The sampling tax comes from generating multiple responses: self-consistency with 5 samples multiplies the cost by 5. Combined, CoT (10x length) plus self-consistency (5x sampling) equals a 50x cost multiplier, before accounting for the overhead of answer extraction and majority voting.

This cost structure creates a natural hierarchy of deployment strategies, ordered from cheapest to most expensive:

1. Greedy decoding (1x cost, 15.2% accuracy): good enough for trivial queries
2. CoT prompting (8x cost, 40.6% accuracy): best single-response strategy
3. GRPO-trained model (1x cost at inference, but one-time training cost, 47.4% accuracy): best when you can invest in training
4. Distilled model (1x cost at inference, cheapest training cost, 45.0% accuracy): best when a teacher is available
5. CoT + self-consistency (85x cost, 52.0% accuracy): best when accuracy is paramount and compute budget is unlimited

A production system would typically use strategy 3 or 4 (trained model) as the default, with strategy 5 (self-consistency) reserved for high-stakes queries where accuracy justifies the cost. Strategy 2 (CoT prompting) serves as a fallback for models that have not been trained for reasoning.

The roadmap

The book progresses through four stages:

Stage 1: Conventional LLM (Chapter 2). You start with a pre-trained LLM as your base model. This stage covers loading pre-trained weights and implementing basic text generation, establishing the foundation upon which all reasoning enhancements are built.

Stage 2: Evaluation (Chapter 3). Before you can improve reasoning, you need to measure it. This stage builds benchmark-based evaluation and a math verification pipeline, creating the measurement tools that quantify the impact of every technique applied in subsequent stages.

Stage 3: Inference Techniques (Chapters 4-5). With a model and evaluation pipeline in place, you explore reasoning-enhancing techniques that do not require modifying model weights: CoT prompting, temperature scaling, top-p sampling, self-consistency voting, and self-refinement.

Stage 4: Training Techniques (Chapters 6-8). The final stage introduces methods that modify the model's weights to permanently improve its reasoning capabilities: GRPO reinforcement learning and knowledge distillation.

This structure, building evaluation before optimisation, is deliberately disciplined. In practice, it is tempting to jump straight to reinforcement learning and distillation, but without a reliable evaluation pipeline, you cannot distinguish genuine improvement from noise. By the time you reach Chapter 6, you will have a complete measurement infrastructure that allows you to rigorously quantify the impact of every RL training run.

One more thought experiment to cement the distinction. Consider two students taking a math exam. Student A writes only final answers: "42", "7/3", "x=5." Student B writes intermediate steps: "Let x = the number of apples. Then 3x + 5 = 20. Subtract 5: 3x = 15. Divide by 3: x = 5." Both might get the same score, but Student B has two advantages. First, if Student B makes an arithmetic error, they can spot it by reviewing their work. Second, a teacher grading Student B's paper can give partial credit and identify exactly where the reasoning went wrong.

A reasoning model is Student B. It writes intermediate steps not because it was told to (well, sometimes it was), but because writing those steps activates statistical pathways through its parameters that reliably produce correct final answers. The steps are not decoration. They are the mechanism by which the model arrives at accuracy.

The three pillars of reasoning enhancement, inference-time scaling, reinforcement learning, and distillation, all work by encouraging the model to generate better intermediate steps. CoT prompting says "show your work." RL says "here is a reward for getting the right answer; figure out that showing your work helps." Distillation says "here is how a smarter model shows its work; learn to do the same thing."

The thread

You now have the field. You know that LLM reasoning is not deterministic logic but probabilistic token generation shaped to produce reliable intermediate steps. You know the three pillars: inference-time scaling, reinforcement learning, and distillation. You know the cost. And you know why building from scratch is one direct way to develop practical intuition for these tradeoffs.

What you do not yet have is a model you can actually run. The concepts need to become code. That means loading pre-trained weights, implementing a tokenizer, and building a text generation loop from the ground up, not because these are glamorous tasks, but because every experiment in the rest of the book depends on them.

In the next chapter, you will watch 621 million parameters predict one token at a time, and you will understand exactly what happens at each step.

Consider one more angle on why reasoning matters economically. OpenAI charges roughly 3-6x more for its reasoning models (o1, o3) than for conventional models (GPT-4o). Google's Gemini 2.5 reasoning mode uses significantly more compute per query than its standard mode. The reasoning tax is real, and it is passed directly to users. This book teaches you to understand that tax: when it is worth paying, when it is not, and how to minimize it through smart engineering choices like selective routing (send only hard problems to the reasoning model) and efficient inference-time scaling (use CoT before reaching for self-consistency).

The distinction between pattern matching and reasoning is not binary. It is a spectrum. Apple's 2025 research paper "The Illusion of Thinking" found that even the best reasoning models are sophisticated pattern matchers that break down on truly novel problems outside their training distribution. But sophistication matters. A pattern matcher that solves 52% of competition-level math problems is substantially useful, even if a philosopher would not call it "genuine reasoning." The practical question is not whether the reasoning is real but whether it is reliable enough to deploy.

Consider a spectrum. On the far left is pure pattern matching: "The capital of France is..." → "Paris." No intermediate steps, no reasoning, just statistical recall. On the far right is full formal reasoning: a theorem prover that derives conclusions from axioms using inference rules, guaranteed correct by construction.

Every LLM sits somewhere in the middle of this spectrum. A conventional LLM sits closer to the left. A reasoning model sits closer to the right, but never reaches it. The techniques in this book move a model rightward along this spectrum by making its token generation process more structured, more reliable, and more amenable to self-correction.

Apple's research team published a paper in 2025 titled "The Illusion of Thinking," which found that reasoning models are sophisticated but still materially pattern matchers. They break down on problems that require truly novel logical inference outside their training distribution. This finding does not diminish the practical value of reasoning models. A system that solves 52% of competition-level math problems (up from 15.2%) is substantially useful, even if it is not doing "real" reasoning in the philosophical sense. The practical question is not whether the reasoning is genuine but whether it is useful.

OpenAI CEO Sam Altman reinforced this pragmatic framing in February 2025, stating that GPT-4.5 would be the company's last non-chain-of-thought model and that future models would "know when to think for a long time or not." The industry is not debating whether LLMs truly reason. It is building systems that reason well enough to be useful, and that is what this book teaches you to build.

What goes in and what comes out?

Consider you are playing a game of predictive text on your phone. You type "The weather today is" and the phone offers three suggestions: "nice," "cold," "going." Each suggestion is a prediction about what word you are most likely to type next, based on patterns the phone has learned from millions of text messages.

A large language model does the same thing, but on a vastly larger scale. Instead of offering three suggestions, it assigns a score to every word in its vocabulary, all 151,936 of them, for the Qwen3 model we use in this book. The word with the highest score becomes the next token.

But there is a catch. The model does not actually work with words. It works with tokens, which are the smallest units of text the model can process. A token might be a full word ("the"), a word fragment ("ing"), a single character, or even a punctuation mark. The process of splitting text into tokens is called tokenization, and it is handled by a separate component called a tokenizer.

Here is what happens concretely. You write a prompt: "Explain large language models." The tokenizer converts this into a sequence of six numerical IDs, one for each token. These IDs are fed into the model. The model processes them through 28 layers of transformer blocks and produces, for each input token position, a vector of 151,936 scores. These scores are called logits, raw values where higher means more likely. We only care about the scores at the last position, because that is the model's prediction for what comes next.

We take the argmax of those 151,936 scores, finding the index with the highest value. That index is a token ID. We decode it back to text. For the prompt "Explain large language models.", the model predicts that the most likely next token is " Large" (with a leading space). This makes sense as a continuation.

Now we append " Large" to the input and feed the whole sequence, "Explain large language models. Large", back into the model. It predicts the next token. We append it. Feed it back. Predict again. This loop continues until we hit a maximum token limit or the model produces a special end-of-sequence token that signals it has finished its thought.

This process, generating one token per forward pass and appending it to the input for the next pass, is called autoregressive generation. "Autoregressive" because each step regresses on (depends on) the outputs of all previous steps. It is the universal mechanism behind every LLM you have ever used. When ChatGPT streams text to you word by word, that is not a display trick. The model is genuinely producing the response one token at a time.

The complete generation function in Python, stripped to its bare essentials:

@torch.inference_mode()
def generate_text_basic_stream(
    model,
    token_ids,
    max_new_tokens, 
    eos_token_id=None
):
    model.eval()

    for _ in range(max_new_tokens):
        out = model(token_ids)[:, -1]
        next_token = torch.argmax(out, dim=-1, keepdim=True)

        if (eos_token_id is not None
                and torch.all(next_token == eos_token_id)):
            break

        yield next_token
        
        token_ids = torch.cat([token_ids, next_token], dim=1)

Fourteen lines. That is the entire text generation engine. Every chatbot, every writing assistant, every code completion tool, runs a loop that is a more optimised version of these fourteen lines. We can walk through the design decisions.

The @torch.inference_mode() decorator tells PyTorch to disable its gradient computation machinery. During training, PyTorch builds an elaborate computation graph that tracks how every output depends on every parameter, enabling backpropagation. During inference, we do not need any of this; we are just running the model forward and reading off predictions. Disabling gradient tracking saves significant memory and computation.

The model.eval() call switches the model from training mode to evaluation mode. This matters because certain layers, like Dropout, behave differently in each mode. In training mode, Dropout randomly zeros out some neural activations to prevent overfitting. In evaluation mode, it passes everything through unchanged. If you forget this call, your model will produce slightly different (and slightly worse) outputs every time you run it.

The line model(token_ids)[:, -1] is the heart of the loop. It runs a full forward pass through all 28 transformer layers and then extracts only the last position's scores via [:, -1]. We discard the scores at every other position because, during generation, we only need the prediction for what comes after the final token.

The yield keyword makes this a Python generator function. Instead of accumulating all tokens into a list and returning them at the end, the function yields each token the instant it is generated. This is what enables streaming: the user sees tokens appear one at a time, like watching someone type.

Finally, torch.cat([token_ids, next_token], dim=1) appends the new token to the end of the input sequence. In the next iteration, the model will process this extended sequence and predict the token after that.

Running this function on a Mac Mini M4 CPU with the prompt "Explain large language models in a single sentence," the model produces:

Large language models are artificial intelligence systems that can
understand, generate, and process human language, enabling them to
perform a wide range of tasks, from answering questions to writing
articles, and even creating creative content.

A perfectly coherent response, generated at 5 tokens per second. Five tokens per second means each token takes 200 milliseconds, and each of those milliseconds involves pushing the entire input sequence through 28 transformer layers, computing attention over all previous positions, and selecting the highest-scoring vocabulary entry from a space of 151,936 options. The fact that this happens on a laptop CPU, with a model that fits in 1.4 GB of disk space, is a small engineering marvel.

But 5 tokens per second is also painfully slow. A 200-word response takes over a minute. For the reasoning experiments in later chapters, where we generate multiple candidate solutions and score each one, the generation cost multiplies rapidly. We need to go faster.

What the model does not do

It is equally important to understand what does not happen during generation. The model does not:

Plan ahead. When generating "The capital of Germany is Berlin.", the model did not first decide to write this sentence and then execute the plan. It generated "The," then "capital" (because "capital" was likely after "The" in this context), then "of" (very likely after "capital"), and so on. The coherent sentence is an emergent property of good local predictions, not the result of global planning.

Maintain a world model. The model does not have an internal representation of "Germany" as a country with a capital. It has statistical associations between token sequences. "Germany" + "capital" strongly predicts "Berlin" because these tokens co-occurred frequently in training data.

Remember between sessions. Each generation call is independent. The model has no memory of previous conversations (unless you include conversation history in the prompt). The KV cache persists within a single generation call but is reset between calls.

Check its own work. The base model does not spontaneously verify its outputs. If it generates "2 + 2 = 5," it does not notice the error because it has no verification mechanism. This is precisely what the reasoning techniques in this book address: CoT makes verification more likely by forcing the model to show intermediate steps (which are more likely to be individually correct), and GRPO trains the model to associate self-checking behaviours with positive rewards.

Understanding these limitations is essential for setting appropriate expectations. A reasoning model is not a mathematician who shows their work. It is a pattern-matching machine that has been shaped (by prompting, RL, or distillation) to generate token sequences that follow the patterns of mathematical reasoning. The distinction matters when the patterns break down.

There is something almost philosophical about the autoregressive generation process. The model has no plan. It does not know how its sentence will end when it begins. It is like a jazz musician improvising: each note follows naturally from the previous ones, shaped by years of training, but the overall arc emerges only in retrospect. When the model writes "The capital of Germany is Berlin," it did not decide to write that sentence and then execute the plan. It wrote "The," then "capital" because "capital" was likely after "The," then "of" because "of" was likely after "capital," and so on. The coherence of the output is an emergent property of locally good decisions, not the execution of a global plan. This is both the power and the limitation of autoregressive generation.

The performance measurement toolkit

Before optimizing generation speed, we need to measure it. The book provides a utility function that reports tokens per second and GPU memory usage:

import warnings

def generate_stats(output_token_ids, tokenizer, start_time, end_time):
    total_time = end_time - start_time
    print(f"\n\nTime: {total_time:.2f} sec")
    print(f"{int(output_token_ids.numel() / total_time)} tokens/sec")
    for name, backend in (("CUDA", getattr(torch, "cuda", None)),
                          ("XPU", getattr(torch, "xpu", None))):
        if backend is not None and backend.is_available():
            if hasattr(backend, "synchronize"):
                backend.synchronize()
            max_mem_gb = backend.max_memory_allocated() / (1024 ** 3)
            print(f"Max {name} memory allocated: {max_mem_gb:.2f} GB")
            backend.reset_peak_memory_stats()

The synchronize() call is important for GPU backends: GPU operations are asynchronous (they return control to Python before the computation finishes), so without synchronization, timing measurements would include only the time to launch GPU kernels, not the time to complete them. This is a common benchmarking pitfall that can make GPU code appear 10x faster than it actually is.

Running the baseline benchmark on a Mac Mini M4 CPU produces 5 tokens per second. For a typical 40-token response, that is 8 seconds of wall-clock time. Tolerable for development but unusable for production. The next two sections address this with optimizations that achieve a combined 13.6x speedup.

The Volkswagen beetle approach to AI research

Before diving into the optimizations, it is worth pausing on the choice of model. The Qwen3 0.6B has 600 million parameters. GPT-4 is estimated to have over a trillion. DeepSeek-V3, the base model for DeepSeek-R1, was trained on 2,048 NVIDIA H800 GPUs for approximately 11 weeks at an estimated cost of 5.5 million dollars. The final training run consumed 14.8 million GPU-hours of compute, which is roughly the electricity an average American household uses in 55 years.

Clearly, we are not training a model of that scale. We are loading a small, pre-trained model and building reasoning techniques on top of it. Raschka offers a useful analogy: if you want to understand how cars work, you do not start by building a Ferrari. You start by building a Volkswagen Beetle. The Beetle still has an engine, a transmission, and wheels. The principles are identical. The scale is manageable.

The Qwen3 0.6B follows the exact same architectural pattern as models hundreds of times its size. It has an embedding layer that maps each of 151,936 possible token IDs to a 1,024-dimensional vector. It has 28 transformer blocks, each containing a Grouped Query Attention mechanism and a FeedForward network, with RMSNorm layers throughout. It has a linear output head that projects the hidden state back to vocabulary space. The architecture is identical to strong contemporary models; only the dimensions differ. The reasoning techniques we build on top of this model transfer directly to larger models without modification.

The model has already been pre-trained on massive text data, instruction-tuned to follow directions, and preference-tuned to produce helpful responses. We treat it as a finished engine. Our job is to add the turbocharger: the reasoning capabilities that turn a fluent text generator into a step-by-step problem solver.

Why is the naive approach so slow?

The 5-tokens-per-second speed of our basic generation function is not just an inconvenience. It reveals a fundamental inefficiency in the naive implementation, an inefficiency that, once you see it, becomes almost painful.

Consider what happens during iteration 50 of a 100-token generation. The model receives 56 tokens (6 original plus 50 generated) and processes all 56 through its 28 transformer layers. But the thing: the first 55 of those tokens were already processed in the previous iteration. Their intermediate computations, specifically the key and value tensors inside the attention mechanism, have not changed. We are recomputing them from scratch at every single step, like a court stenographer who transcribes the entire trial from the beginning every time a new question is asked.

At this point, KV caching enters. The idea is exactly what a court stenographer actually does: keep running notes. Instead of reprocessing the entire transcript for each new question, the stenographer reads back only the relevant part of what was already recorded.

In technical terms, the attention mechanism in each transformer layer computes three things from its input: queries, keys, and values. The queries represent "what am I looking for?" The keys represent "what information do I contain?" The values represent "what information should I pass along?" The attention score between a query and a key determines how much the corresponding value contributes to the output.

During standard autoregressive generation, every iteration recomputes the keys and values for all previous tokens. KV caching stores these tensors after they are computed and reuses them in subsequent iterations. On the first pass, the full input sequence goes through the model, and the cache stores the keys and values for all positions. On every subsequent pass, only the single new token is processed. The model retrieves the cached keys and values from previous positions and combines them with the new token's key and value to compute attention. No redundant computation.

The code change is minimal:

@torch.inference_mode()
def generate_text_basic_stream_cache(
    model,
    token_ids,
    max_new_tokens,
    eos_token_id=None
):
    model.eval()
    cache = KVCache(n_layers=model.cfg["n_layers"])
    model.reset_kv_cache()

    out = model(token_ids, cache=cache)[:, -1]
    for _ in range(max_new_tokens):
        next_token = torch.argmax(out, dim=-1, keepdim=True)

        if (eos_token_id is not None
                and torch.all(next_token == eos_token_id)):
            break

        yield next_token
        out = model(next_token, cache=cache)[:, -1]

Notice what disappeared: the torch.cat line. There is no need to build up the full token sequence anymore because the cache handles context preservation internally. In the first pass, model(token_ids, cache=cache) processes the entire input and populates the cache. In every subsequent pass, model(next_token, cache=cache) processes only the single new token, consulting the cache for everything else.

The performance improvement is immediate and dramatic: from 5 tokens/sec to 29 tokens/sec. A 5.8x speedup. The generated text is identical token-for-token, which is the critical sanity check confirming that the optimisation is correct.

But the improvement is also mathematically expected. Without caching, generating N tokens requires processing sequences of length 1, 2, 3, ..., N, for a total of approximately N²/2 operations. With caching, each step processes exactly one token, for a total of N operations. The complexity drops from O(N²) to O(N). For N = 100, that is the difference between 5,000 operations and 100 operations.

The compiler's gift

The second optimisation is model compilation using torch.compile, a feature available since PyTorch 2.0. The idea behind compilation is simple but capable.

Standard PyTorch execution is eager: each line of Python code is executed immediately, one at a time, by the Python interpreter. This is great for debugging and flexibility, but it is slow. The Python interpreter has significant overhead per operation, and it cannot see the big picture. It does not know that the output of this matrix multiplication feeds directly into that activation function, which feeds into that normalisation layer. It processes each operation in isolation.

torch.compile changes this. It analyses the entire computation graph of the model's forward pass, identifies patterns that can be fused (a matrix multiply followed by an addition followed by a nonlinearity can often be combined into a single optimised kernel), eliminates redundant memory accesses, and generates optimised low-level code that runs without the Python interpreter's overhead.

The usage is trivially simple:

model_compiled = torch.compile(model)

One line. The first execution is slower because the compiler needs to analyse and optimise the graph. But every subsequent execution runs the optimised code.

The results, combined with KV caching, are striking:

The combined speedup is multiplicative, not additive: 13.6x over the baseline. A 40-token response that took 8 seconds now completes in under 0.6 seconds. And the generated text remains identical.

A cross-hardware comparison reveals something surprising. The optimised CPU pipeline at 68 tokens/sec is competitive with unoptimized GPU execution. On an NVIDIA H100 GPU without KV caching and without compilation, the model generates 51 tokens/sec. A well-optimised CPU beats a poorly-optimised GPU, at least for a model this small. For larger models, the GPU's parallelism advantage grows rapidly, but this result underlines an important point: software optimisation can close hardware gaps, sometimes materially.

A system design exercise · choosing your hardware

This is a constructed design exercise, not a report of a named deployment. You are deploying a Qwen3 0.6B-based reasoning assistant for a startup. You have three hardware options:

Option 1: Cloud H100 GPU ($3/hour). Performance: 141 tokens/sec (compiled + KV cache). Best for: high-throughput production with many concurrent users. A single H100 can serve approximately 100 users simultaneously at 1.4 tokens/sec each, which is enough for real-time streaming. Monthly cost at 24/7 operation: $2,160.

Option 2: Apple Mac Mini M4 ($600 one-time). Performance: 71 tokens/sec (compiled + KV cache on GPU). Best for: development, testing, small-scale deployment. Can serve approximately 50 users simultaneously. No recurring cloud costs. Breaks even versus cloud after ~7 days of continuous use.

Option 3: Any CPU ($0). Performance: 68 tokens/sec (compiled + KV cache). Best for: development and testing. Surprisingly competitive with GPU options for this small model. The gap widens materially for larger models: a 7B model on CPU drops to ~5 tokens/sec while GPU stays above 50.

The counterintuitive insight from these numbers: for models under 1B parameters, the choice between CPU and GPU matters less than the choice between compiled and uncompiled, or cached and uncached. Software optimisation (13.6x speedup) dwarfs hardware acceleration (2-3x speedup for this model size). This changes completely for larger models, where GPU parallelism becomes essential.

For the exercises and experiments in this book, CPU is sufficient through Chapter 5. GPU becomes important in Chapters 6-8, where GRPO training generates thousands of rollouts and distillation processes thousands of training examples. The training loops are memory-bound (requiring ~15-70 GB GPU memory) rather than compute-bound, which means the cheapest GPU with enough memory is often the optimal choice.

The machine that cannot yet think

We now have a working, efficient text generation pipeline. The model loads in under two seconds. The tokenizer handles the conversion between human text and numerical token IDs. The generation function loops through forward passes, each producing one token, until the model generates an end-of-sequence signal. KV caching and compilation make the whole process fast enough for interactive use.

But ask this model to solve a math problem, a genuine multi-step reasoning task, and something interesting happens. It tries. It produces text that looks like it might be working through the problem. And then it gets the answer wrong. Not always, but often enough that you cannot trust it. The base model achieves roughly 15% accuracy on the MATH-500 benchmark, meaning it gets 85 out of 100 math problems wrong.

This is not a failure of the model's knowledge. The model has seen countless math solutions during pre-training. It knows what a math solution looks like. The failure is in the execution. The model cannot reliably maintain logical consistency across multiple reasoning steps. It skips steps, makes arithmetic errors, or produces a confident chain of reasoning that leads to a wrong answer.

Everything we build in Chapters 3 through 8 is designed to fix this. But before we can fix it, we need to measure it precisely. We need to build an evaluation pipeline that can automatically extract a model's answer from its free-form output, normalise it, and check whether it matches the correct solution. That pipeline is not just a testing tool. It will become the reward function that drives the reinforcement learning training in Chapter 6.

The embedding layer alone accounts for 151,936 × 1,024 = 155.6 million parameters, roughly one-quarter of the model's total 621 million. This is not coincidence. In modern LLMs, the output head (which maps back from hidden dimensions to vocabulary) often shares weights with the embedding layer, a technique called weight tying that halves the memory cost of the vocabulary-dependent parameters. Qwen3 uses this optimisation.

We have built the engine. A tokenizer converts text to numbers. A transformer model converts those numbers to predictions. An autoregressive loop turns predictions into coherent text. KV caching and compilation make the loop fast.

But the engine cannot reason. It can recall facts with impressive accuracy, complete sentences with surprising coherence, and generate text that passes for human writing. What it cannot do, reliably, is work through a problem step by step and arrive at the correct answer. The gap between fluent text generation and reliable reasoning is the gap this entire book exists to close.

Before we can close it, we need to measure it. In the next chapter, we build a math verification pipeline that will tell us, with symbolic precision, whether the model got the answer right. The same pipeline will later become the reward function for reinforcement learning. Evaluation is not a preamble to the real work. Evaluation is the real work, because you cannot improve what you cannot measure.

Before any transformer processing happens, each token ID is converted to a 1024-dimensional vector by the embedding layer. Treat this as looking up a word in a very specialized dictionary where each definition is not a sentence but a list of 1,024 numbers. The embedding for "Berlin" might be [0.23, -1.45, 0.78, ...] and the embedding for "Paris" might be [0.21, -1.39, 0.82, ...], very similar because both are European capitals. The embedding for "happiness" would be far away in this 1024-dimensional space, because it is a completely different kind of concept.

This is the essence of how neural networks represent meaning: through geometry. Words with similar meanings are nearby points in a high-dimensional space. Words with different meanings are distant. The model did not learn these positions from a dictionary. It learned them from statistical co-occurrence patterns during pre-training. Words that appear in similar contexts end up with similar embeddings. This is a beautiful consequence of the distributional hypothesis in linguistics: "You shall know a word by the company it keeps."

After embedding, the 6-token input becomes a 6×1024 matrix: six rows, each a 1024-dimensional vector. This matrix is what flows upward through the 28 transformer blocks. Each block refines these representations by allowing tokens to exchange information through the attention mechanism. By the 28th block, each token's 1024-dimensional vector encodes not just its own meaning but its meaning in context, shaped by every other token in the sequence.

The weight of 621 million parameters

To put 621 million parameters in perspective, consider that each parameter is a single number, stored in bfloat16 format (2 bytes). If you wrote each parameter on a Post-it note and stacked the notes, the pile would be approximately 62 kilometers tall, about seven times the cruising altitude of a commercial jet. These 621 million numbers are what the model "knows." They were shaped by trillions of tokens of training data into a configuration that can produce coherent, relevant text.

But parameters are not stored as Post-it notes. They are stored as matrices inside the 28 transformer layers. The largest matrix in the model is the output head: a 1024×151,936 matrix containing approximately 155 million parameters, one-quarter of the model's total. This single matrix is responsible for converting the model's internal representation into a score for every possible next token. When you ask the model "The capital of Germany is" and it produces a logit of 20 for "Berlin," that number 20 is the result of a 1024-dimensional dot product between the model's hidden state and the 1024-dimensional row for "Berlin" in this 155-million-parameter matrix.

The remaining three-quarters of parameters live inside the 28 transformer blocks. Each block contains approximately 16 million parameters split between the attention mechanism (which decides which tokens are relevant to each other) and the feedforward network (which transforms token representations based on what the attention mechanism determined was relevant). These blocks are stacked like floors of a building, and information flows upward from the first block to the twenty-eighth. By the time a token's representation reaches the output head, it has been processed and refined 28 times.

One of the most practical implications of model size is memory. At 2 bytes per parameter, the model requires 621M × 2 = 1.24 GB just to store the weights. The downloaded file is 1,433 MB (slightly larger due to PyTorch file format overhead). During inference, additional memory is needed for activations (the intermediate values computed during the forward pass) and the KV cache. For the Qwen3 0.6B, total inference memory is approximately 2-3 GB, well within the range of even a modest laptop.

For comparison, DeepSeek-R1 at 671 billion parameters requires 671B × 2 = 1.34 TB just for weights, which is why it needs multiple high-end GPUs with hundreds of gigabytes of combined memory. The techniques in this book apply equally to both models, but the small model lets you see the mechanics without needing a data center.

When you call model(token_ids), an intricate cascade of matrix multiplications begins. We can trace it for a single token to build intuition.

The input token ID (say, 3460 for " large") is looked up in the embedding table, producing a 1024-dimensional vector. Treat this vector as the token's identity card in a 1024-dimensional space where similar concepts cluster together. " large" and " big" would have nearby vectors; " large" and " quantum" would be far apart.

This 1024-dimensional vector enters the first transformer block. In the attention mechanism, it is projected into three vectors: a query (128 dimensions, "what am I looking for?"), a key (128 dimensions, "what do I contain?"), and a value (128 dimensions, "what information do I provide?"). The query is compared against all keys in the sequence via dot products, producing attention scores. These scores are softmax-normalised into weights that sum to 1. The weighted sum of values produces the attention output.

But Qwen3 has 16 query heads, each with its own 128-dimensional Q/K/V projections. These 16 heads attend to different aspects of the input simultaneously: one might focus on syntactic structure, another on semantic similarity, a third on positional proximity. The 16 outputs (each 128-dimensional) are concatenated into a 2048-dimensional vector and projected back to 1024 dimensions through the output projection.

After attention, the feedforward network applies two independent transformations (fc1 and fc2, each projecting from 1024 to 3072 dimensions), multiplies them element-wise (the gating mechanism), and projects back to 1024 with fc3. The SiLU activation function (x * sigmoid(x)) introduces the non-linearity that allows the network to learn complex patterns.

A residual connection adds the block's input to its output: output = input + attention(norm(input)) + ffn(norm(input + attention(norm(input)))). This "highway" allows gradients to flow directly from later layers to earlier layers during training, preventing the vanishing gradient problem that plagued early deep networks.

This entire process repeats 28 times. By the final block, the 1024-dimensional vector for each token encodes not just the token's identity but its complete contextual meaning, shaped by the 28 layers of attention and feedforward processing.

How the forward pass actually works

When you call model(token_ids), here is what happens inside the model in sequence. The input token IDs (say, [840, 20772, 3460, 4128, 4119, 13] for "Explain large language models.") are first converted to 1024-dimensional embedding vectors by looking up each ID in the 151,936-row embedding table. These six 1024-dimensional vectors form a 6×1024 matrix.

This matrix flows upward through 28 transformer blocks. In each block, the attention mechanism lets each token "attend to" every other token. The query-key dot products compute relevance scores: how relevant is token j to predicting what comes after token i? The values are then combined according to these relevance weights. After attention, a feedforward network processes each token independently, applying two non-linear transformations. A residual connection adds the block's input to its output, preventing "forgetting" in deep networks.

After all 28 blocks, a final RMSNorm layer standardizes the output, and the output head (the 1024×151,936 matrix) produces logit scores for every possible next token. The entire process takes approximately 0.02 seconds on a CPU for a 6-token input, and it produces the 6×151,936 matrix that we extract the last row from.

The courtroom stenographer · KV caching

Without optimisation, every iteration reprocesses the entire growing sequence from scratch. For N generated tokens, the total computation scales as O(N³): N iterations, each processing a sequence of average length N/2 through attention mechanisms that cost O(N²).

Treat a courtroom stenographer. Without a stenographer, every time the judge asks a new question, the entire trial transcript must be read aloud from the beginning. With a stenographer, the judge asks the stenographer to read back only the relevant bits. The stenographer's notes are the KV cache.

In transformer attention, each token is projected into three vectors: a query (what am I looking for?), a key (what do I contain?), and a value (what information do I provide?). Without caching, every forward pass recomputes keys and values for all previous tokens. With KV caching, we store those tensors and only compute them for the new token. This converts per-step cost from O(N²) to O(N), and total generation cost from O(N³) to O(N²).

Listing 2.4: Text generation with KV cache

from reasoning_from_scratch.qwen3 import KVCache

@torch.inference_mode()
def generate_text_basic_stream_cache(
    model, token_ids, max_new_tokens, eos_token_id=None
):
    model.eval()
    cache = KVCache(n_layers=model.cfg["n_layers"])
    model.reset_kv_cache()
    
    out = model(token_ids, cache=cache)[:, -1]
    for _ in range(max_new_tokens):
        next_token = torch.argmax(out, dim=-1, keepdim=True)
        if (eos_token_id is not None
                and torch.all(next_token == eos_token_id)):
            break
        yield next_token
        out = model(next_token, cache=cache)[:, -1]

The differences from Listing 2.2 are minimal but impactful. A KVCache object is created with one entry per transformer layer (28 for Qwen3 0.6B). During the first iteration, the full input is passed via model(token_ids, cache=cache), populating the cache. In subsequent iterations, only next_token is passed. The model retrieves cached keys and values from previous tokens and combines them with the new token's key and value. The torch.cat line from Listing 2.2 is completely absent because the cache handles context preservation internally.

Performance: 28 tokens/sec, a 5.6x speedup from 5 tokens/sec. The generated text is identical, confirming correctness.

Compiling the model for even more speed

torch.compile is a PyTorch feature that compiles the model's forward pass into an optimised execution graph. Instead of executing Python line by line, the compiler analyses the entire computation, fuses operations, eliminates redundant work, and generates optimised low-level code:

major, minor = map(int, torch.__version__.split(".")[:2])
if (major, minor) >= (2, 8):
    torch._dynamo.config.allow_unspec_int_on_nn_module = True

model_compiled = torch.compile(model)

The first execution is slower due to compilation overhead. Subsequent runs reflect the true optimised speed:

Warm-up:    Time: 11.68 sec, 3 tokens/sec
Timed run:  Time: 6.78 sec,  6 tokens/sec

Compilation alone provides a modest improvement (5 → 6 tok/s on CPU). The real gains come from combining compilation with KV caching:

KV cache + compile:
Warm-up:    Time: 8.07 sec,  5 tokens/sec
Timed run:  Time: 0.60 sec,  68 tokens/sec

68 tokens per second, a 13.6x speedup over the baseline. A response that took 8 seconds now completes in under 0.6 seconds.

A remarkable observation: the compiled CPU with KV cache (68 tok/s) is competitive with uncompiled GPU variants (27-51 tok/s). Software optimisation can close hardware gaps for small models.

The H100 KV cache variant (48 tok/s) being slightly slower than non-cached (51 tok/s) is explained by an implementation detail: the book's KV cache uses torch.cat to grow tensors dynamically, which is simple but slower on GPUs than pre-allocating full tensors. A GPU-optimised variant (available in the bonus materials) pre-allocates the full K and V tensors, eliminating this overhead.

The thread

You now have a working language model. You can feed it text, and it will produce more text, one token at a time. You have optimised this process with KV caching and model compilation, achieving a 13.6x speedup.

But the uncomfortable question: how good is this model, actually? When you ask it a math problem, does it get the right answer? You have a machine that speaks fluently. You do not yet know whether it speaks truthfully.

To find out, you need to build something that can check its work.

Thought experiment · the verifier as the foundation of everything

Consider this thought experiment. What if we removed the verifier from the book entirely? What would change?

Chapter 3 would disappear, obviously. But more importantly:

Chapter 4 would lose its measurement infrastructure. We could still implement CoT prompting and self-consistency, but we would have no way to measure whether they actually improve accuracy. We would be flying blind, making changes and hoping they help.

Chapter 6 would be impossible. GRPO requires a reward signal that tells the model whether each rollout's answer is correct or incorrect. Without the verifier, there is no reward. Without reward, there is no reinforcement learning. The entire RL training pipeline collapses.

Chapter 8 would be severely compromised. The distillation dataset must be filtered for correctness (removing examples where the teacher got the wrong answer). Without the verifier, we cannot filter, and the student would learn from a mix of correct and incorrect teacher responses.

This thought experiment reveals the verifier's true role: it is not just an evaluation tool. It is the foundation upon which all training-time techniques are built. The same grade_answer function appears in three contexts:

1. Evaluation (Chapter 3): grade_answer(extracted, ground_truth) → True/False for logging
2. RL reward (Chapter 6): float(grade_answer(...)) → 1.0/0.0 for training signal
3. Data filtering (Chapter 8): grade_answer(teacher_answer, ground_truth) → keep/discard

Build evaluation first, optimise later. This is not just good practice. It is structurally necessary.

Why math? the luxury of verifiable truth

There are many things you might want to evaluate an LLM on: helpfulness, safety, creativity, code quality, factual accuracy. Most of these are inherently subjective. Is this summary "good"? Is this poem "creative"? Reasonable people disagree, which means any evaluation requires either expensive human annotation or an LLM judge whose own biases become part of the measurement.

Math is different. Math has a property that is rare and precious in the world of language: deterministic verifiability. The answer to "What is the integral of x² from 0 to 3?" is either 9 or it is not. No judgment call required. No annotator disagreement. A program can check the answer.

This is why math has become the default testing ground for reasoning models. The MATH dataset, introduced by Hendrycks et al. in 2021, contains 12,500 problems spanning algebra, number theory, geometry, and more, each with a verified answer. The MATH-500 subset, used throughout this book, contains 500 carefully curated problems at various difficulty levels.

But the simplicity of "right or wrong" is deceptive. The devil is in the extraction. A model does not produce a clean number. It produces hundreds of tokens of natural language, mathematical notation, LaTeX formatting, and reasoning traces. Somewhere in that output is an answer, and finding it reliably is an engineering problem worthy of its own chapter.

The evaluation hierarchy · a complete taxonomy

Before diving into implementation, it is worth understanding where verification-based evaluation sits in the broader landscape of LLM evaluation methods. Treat evaluation as a ladder with four rungs, each trading off scalability against depth.

Rung 1: Multiple choice (MMLU, ARC, HellaSwag). The model selects from predefined options: A, B, C, or D. This is the cheapest and fastest evaluation method. MMLU, the most popular benchmark, contains 57 subjects with about 16,000 questions. Performance is measured as accuracy (fraction correct). Two scoring methods exist: log-probability scoring (compare the model's assigned probabilities to each option without generating text) and generation-based scoring (let the model generate a letter and check it). The limitation: multiple choice tests recall, not reasoning. A model can score well by recognising familiar patterns without understanding the underlying concepts. And the fixed-format questions do not reflect open-ended use where models must generate open-ended responses.

Rung 2: Verifiers (MATH-500, HumanEval, GPQA). The model generates a free-form response, and a deterministic program checks whether the answer is correct. This is what we build in this chapter. The key advantage over multiple choice: models produce open-ended answers, which better reflects open-ended use. The key limitation: only applicable to domains with deterministic correctness criteria. Math and code are the primary domains. You cannot verify whether an essay is "correct" with a program.

The math verifier generalizes to other verifiable domains. For code verification, replace grade_answer with a sandboxed executor that runs test cases. The reward becomes the fraction of tests passed (0.0 to 1.0). For logic puzzles, write a constraint checker that verifies all rules are satisfied. For scientific computation, use the math pipeline with domain-specific normalisation (units, significant figures, scientific notation). Each extension requires careful sandbox design but follows the same extract-normalise-parse-compare pattern.

For domains without deterministic correctness (creative writing, summarization, open-ended Q&A), verification is not applicable and you must fall back to LLM-as-judge or human evaluation. This is why math and code dominate reasoning model research: they provide clean automated reward signals that make RLVR feasible at scale.

Rung 3: Leaderboards (LM Arena, formerly Chatbot Arena). Two models receive the same prompt, users vote for the preferred response, and votes are aggregated into rankings. LM Arena originally used the Elo rating system (from chess): each model starts with a baseline score (e.g., 1000), and after each comparison, the winner gains points while the loser loses points, with magnitude depending on the rating difference. Later, LM Arena transitioned to the Bradley-Terry model, which estimates all ratings jointly using maximum likelihood. The advantage: leaderboards capture holistic quality including style, helpfulness, and nuance. The limitation: subjective, expensive (requires human voters), and vulnerable to gaming (prompt selection bias, voting manipulation).

Rung 4: LLM-as-judge. A strong LLM with a predefined grading rubric evaluates model responses. This is more scalable and consistent than human evaluation but introduces its own biases: preference for longer responses, deference to the judge model's style, and inability to reliably evaluate reasoning correctness. Process Reward Models (PRMs) are a specialized variant that evaluate intermediate reasoning steps rather than just the final answer. They provide denser feedback for training but are difficult to build reliably. Notably, DeepSeek-R1 did not use PRMs, relying instead on outcome-based verifiers (Rung 2).

For reasoning model development, verification (Rung 2) is the sweet spot. It is automated, reproducible, and directly provides the reward signal needed for reinforcement learning. This is why math has become the standard domain: it sits at the intersection of "requires reasoning" and "can be verified."

System design exercise · building a verification pipeline for code

This is a constructed design exercise, not a report of a named deployment. The math verifier we build here generalizes to other verifiable domains. Consider extending it to code. Instead of checking simplify(pred - gt) == 0, you would:

1. Extract the code block from the model's response (similar to extracting \boxed{} but for code fences)
2. Normalize by stripping comments, standardizing whitespace, handling import variations
3. Execute the code in a sandboxed environment with predefined test cases
4. Compare the output against expected results

The key difference from math verification: code execution requires sandboxing (the model might generate malicious code), timeout handling (the model might generate infinite loops), and test case design (you need comprehensive tests, not just a single ground truth). But the principle is identical: extract, normalise, execute, compare.

Companies like Anthropic, OpenAI, and DeepSeek use code verification as a second major reward signal alongside math verification when training reasoning models. The combination is capable because math and code exercise different reasoning capabilities: math requires algebraic manipulation and numerical computation, while code requires logical flow control, data structure manipulation, and API knowledge.

Extracting an answer from chaos

Here is a real model output for the problem "What is the value of 1/4 + 3/8?":

To find the sum of 1/4 and 3/8, I need a common denominator. The LCD of 4 and 8 is 8. Converting: 1/4 = 2/8. So 2/8 + 3/8 = 5/8. The answer is \boxed{\frac{5}{8}}.

The answer is 5/8. But where is it in the output? It could be after "the answer is," or inside \boxed{}, or at the very end of the response, or buried somewhere in the middle. Different models use different conventions. Even the same model might format its answer differently on different runs.

The extraction pipeline uses a two-stage strategy. First, it looks for the \boxed{} convention, which reasoning models are typically trained to use for final answers. A function called get_last_boxed scans the output for \boxed{...} and extracts the content. If the model used nested braces (which happens with fractions like \boxed{\frac{5}{8}}), the function handles brace matching correctly.

If no \boxed{} is found, a fallback function called extract_final_candidate tries to find the last number or mathematical expression in the output, on the theory that models tend to state their final answer at the end.

This two-stage approach is pragmatic engineering, not elegant theory. It handles the most common cases and fails gracefully on edge cases. In the exercises, you are invited to design adversarial inputs that break the pipeline and then fix it.

Thought experiment · building a verifier from scratch

Consider you are given a pile of 500 math exams, each with a student's handwritten solution and a teacher's answer key. Your job: grade them all, without reading the solutions, by checking only the final answer against the key.

You quickly discover this is harder than it sounds. Student A writes "1/2". The key says "0.5". Are these the same? Obviously yes, but string comparison says no. Student B writes "√12". The key says "2√3". Same number, different notation. Student C writes "x = 3 or x = 5" and the key says "3". Did the student answer correctly? They found both solutions when the question asked for the smallest, so the answer is somewhere in their response, but it is not an exact match.

This is exactly the problem the verification pipeline must solve. And the reason it took the ML community years to converge on a well-tested solution (SymPy-based symbolic comparison) is that every shortcut fails on some subset of cases. String comparison fails on equivalent fractions. Numerical comparison fails on symbolic answers (√2 cannot be exactly represented as a float). Regular expression matching fails on nested LaTeX expressions. Only symbolic algebra, where simplify(pred - ground_truth) == 0, handles the full range of mathematical equivalences.

The pipeline's robustness was hard-won. Early reasoning model papers used simple string matching for evaluation, which worked for benchmarks with tightly controlled answer formats but broke badly on free-form generation. The SymPy-based approach costs more computation (~50ms per comparison vs <1ms for string matching) but handles the diversity of LLM outputs far more robustly. The 50ms is negligible compared to the seconds-to-minutes of generation time per problem.

The normalisation problem, or why "1/2" equals "0.5"

You have extracted the model's answer: \frac{5}{8}. The ground truth answer is \frac{5}{8}. Are they equal? Trivially yes, in this case. But consider these equivalent representations of the same number:

• \frac{5}{8}
• 0.625
• 5/8
• \dfrac{5}{8}
• \frac{10}{16}

A string comparison would say these are all different. A human would say they are all the same. The evaluation pipeline needs to agree with the human.

The solution comes in two parts. First, a normalisation function strips away formatting artifacts. It removes LaTeX commands like \left, \right, \dfrac, and \text{}. It converts \frac{a}{b} into a/b. It strips dollar signs, trailing periods, and whitespace. It handles special cases like percentages, degrees, and scientific notation. After normalisation, \frac{5}{8} becomes 5/8, and \dfrac{10}{16} also becomes 10/16.

But 5/8 and 10/16 are still different strings, despite being the same number. At this point, the second part comes in: symbolic parsing using SymPy, Python's symbolic mathematics library. SymPy can parse 5/8 and 10/16 into symbolic rational numbers and then check whether they are mathematically equivalent: simplify(5/8 - 10/16) == 0. It handles fractions, radicals, trigonometric identities, and algebraic expressions with the rigor of a computer algebra system.

The full equivalence check:

def equality_check(pred, gt):
    """Check if prediction equals ground truth, symbolically."""
    pred_sym = sympy_parser(pred)
    gt_sym = sympy_parser(gt)
    
    if pred_sym is None or gt_sym is None:
        return str(pred).strip() == str(gt).strip()  # Fallback to string
    
    try:
        diff = sympy.simplify(pred_sym - gt_sym)
        return diff == 0
    except:
        return False

This function handles the vast majority of cases correctly. It correctly identifies that \frac{5}{8} equals 0.625, that \sqrt{2}/2 equals \frac{\sqrt{2}}{2}, and that x^2 + 2x + 1 equals (x+1)^2. It fails on certain edge cases (complex numbers, matrices, geometric proofs), but for the MATH-500 benchmark, it is well-tested enough to serve as a reliable grading function.

The assembly line inspector

Treat the verification pipeline as a quality control inspector at the end of a factory assembly line. The factory (the LLM) produces widgets (answers) of varying quality. Some are perfect. Some have defects. Some look perfect but have internal flaws that only careful measurement reveals.

The inspector has four tools. The extractor is a pair of calipers that grabs the critical dimension from the widget (the boxed answer from the LLM output). The normalizer is a cleaning station that strips away cosmetic differences: paint color, surface texture, protective packaging (LaTeX formatting, extra whitespace, \dfrac vs \frac). The parser is a precision measuring instrument that converts the cleaned widget into an exact specification (a SymPy symbolic object). And the comparator checks whether the specification matches the blueprint (the ground truth).

This four-tool pipeline catches equivalences that no simpler approach could handle. The fraction 28/6 and the decimal 4.666... and the mixed number 4⅔ all reduce to the same symbolic object: Rational(14, 3). A string comparison would see three different answers. The inspector sees one.

But the inspector is not omniscient. It cannot grade essays, evaluate code correctness (that requires execution), or assess the quality of a proof (that requires understanding the logical structure). It works only in domains where correctness is deterministic and verifiable. Math is the perfect domain. Code is another (verify by running test cases). Open-ended text is not. This is why math has become the de facto standard for reasoning model research: it sits at the intersection of "requires reasoning to solve" and "can be automatically verified."

The inspector metaphor extends to Chapter 6 in an important way. In a factory, the quality inspector only grades output. But what if the inspector could also train the workers? In GRPO, the same verification pipeline that grades answers during evaluation is repurposed as the reward function that provides training feedback. The inspector becomes the teacher.

The complete pipeline, from prompt to grade

We can trace a single problem through the entire evaluation pipeline. The problem is from the MATH-500 dataset:

"Find the number of integers n that satisfy −8π ≤ n ≤ 10π."

Step 1: Render the prompt. The raw problem text is wrapped in a prompt template that the model expects. For the base model, this is a simple instruction format.

Step 2: Generate. The model produces a response, potentially hundreds of tokens long, containing reasoning steps and a final answer.

Step 3: Extract. The get_last_boxed function finds \boxed{57} in the response. If no boxed answer exists, extract_final_candidate looks for the last number.

Step 4: Normalize. The normalize_text function strips LaTeX formatting, converting \boxed{57} to 57.

Step 5: Parse. The sympy_parser function converts 57 into the SymPy integer Integer(57).

Step 6: Compare. The equality_check function computes simplify(Integer(57) - Integer(57)) == 0, which is True. The answer is correct.

Step 7: Grade. The grade_answer function returns True. This problem is marked as correct.

Raw Output → get_last_boxed → extract_final_candidate → normalize_text → sympy_parser → equality_check → grade_answer

This pipeline runs for all 500 problems in MATH-500. The final accuracy is the fraction of problems graded as correct.

Tracing through three edge cases

We can trace through three tricky examples to build intuition for where the pipeline succeeds and where it struggles.

Edge case 1: Equivalent fractions. The model outputs \boxed{28/6} and the ground truth is 14/3.

Step 1 (Extract): Regex finds 28/6 inside the boxed expression. Step 2 (Normalize): No LaTeX to strip, already a simple fraction. Step 3 (Parse): SymPy parses 28/6 as Rational(28, 6), which automatically simplifies to Rational(14, 3). Step 4 (Compare): simplify(Rational(14,3) - Rational(14,3)) = 0. Correct!

String comparison would have said "28/6" ≠ "14/3" and marked this wrong. SymPy catches the equivalence.

Edge case 2: Different radical forms. Model outputs \boxed{2\sqrt{3}}, ground truth is \sqrt{12}.

Step 1: Extract 2\sqrt{3}. Step 2: Normalize converts \sqrt{3} to sqrt(3), giving 2*sqrt(3). Step 3: Parse to 2*sqrt(3). Ground truth \sqrt{12} parses to sqrt(12). Step 4: simplify(2*sqrt(3) - sqrt(12)) = simplify(2*sqrt(3) - 2*sqrt(3)) = 0. Correct!

SymPy knows that √12 = √(4×3) = 2√3.

Edge case 3: Decimal approximation. Model outputs \boxed{4.667}, ground truth is 14/3 (= 4.6666...).

Step 1: Extract 4.667. Step 2: No normalisation needed. Step 3: Parse to Float(4.667). Ground truth parses to Rational(14, 3). Step 4: simplify(Float(4.667) - Rational(14,3)) = Float(0.000333...). This is not exactly zero. Potentially incorrect!

At this point, the pipeline can produce a false negative. The model's answer is approximately correct (within rounding error) but the verifier may reject it because the floating-point comparison is not exactly zero. The fix: use abs(simplify(pred - gt)) < epsilon instead of strict equality, with epsilon = 1e-6. Some implementations include this tolerance; others do not. It is worth checking when deploying the pipeline.

Edge case 4: Set-valued answer. Model outputs \boxed{3, 5}, ground truth is \boxed{3} (the problem asks for the smallest solution).

Step 1: Extract 3, 5. Step 2: Normalize. Step 3: SymPy attempts to parse "3, 5" as a single expression and may fail, returning None or raising an exception. Step 4: Since parsing failed, grade_answer returns False. False negative!

The model found both solutions (commendable!) but the pipeline expected only one. This mismatch between the model's output format and the pipeline's expectation is a recurring challenge. The extract_final_candidate function with its fallback strategy (try boxed → try last number → try full text) mitigates some of these cases, but cannot handle all format variations.

The base model versus the reasoning model · a first comparison

Running this pipeline produces the first concrete data point of the book.

The base model (Qwen3 0.6B, pre-trained only) achieves 15.2% accuracy on MATH-500. On a 10-problem subset, it gets 3 out of 10 correct in about 0.4 minutes.

The reasoning model (Qwen3 0.6B, officially trained with reasoning methods by the Qwen team) achieves 50.8% accuracy on the full MATH-500 dataset. On the same 10-problem subset, it gets 9 out of 10 correct, but takes about 7 minutes instead of 0.4.

That 14x slowdown is not a bug. The reasoning model produces far longer responses because it generates intermediate thinking steps. Each <think> section contains the model's reasoning trace, which can be hundreds of tokens long. More tokens means more forward passes, means more time. The accuracy improvement comes at a concrete, measurable cost.

This tradeoff, better accuracy for more computation, is the central tension of the entire book. Every technique in Chapters 4 through 8 navigates this tradeoff differently. Inference-time scaling spends computation at generation time. Reinforcement learning spends computation during training to reduce generation-time cost. Distillation spends computation once (generating the teacher dataset) to reduce both training and generation costs.

The prompt template effect is one of the most underappreciated findings in LLM evaluation. A well-chosen template can improve accuracy by 3-5 percentage points. A poorly chosen template can decrease accuracy by the same amount. For a model scoring 15% on a 500-problem benchmark, a 5-point swing changes the result from "essentially random" (10%) to "showing real capability" (20%).

Consider three template variations tested on the Qwen3 base model:

Template A: "Solve: [problem]", Accuracy: ~12%. Too terse. The model does not know it is expected to produce a mathematical solution.

Template B: "Below is a math problem. Solve it and put your final answer within \boxed{}.\n\n[problem]", Accuracy: 15.2%. The template used throughout the book. It establishes context and specifies output format.

Template C: "You are a mathematics professor. Solve the following problem carefully, showing all work. Put your final answer in \boxed{} format.\n\n[problem]", Accuracy: ~14%. Surprisingly, the more elaborate template performs slightly worse. The extra tokens consume context window space and may activate "professor-style" verbose explanations that do not always lead to correct answers.

The lesson: prompt template selection should be treated as a hyperparameter search, not an afterthought. Test 3-5 variations on a small subset (10-20 problems) before committing to a full evaluation run. The optimal template differs between base and reasoning models, between model families, and even between model sizes within the same family.

For the reasoning model variant, the template includes chat formatting tokens and a <think> marker that signals the model to enter its reasoning mode. The base model ignores these special tokens since they were not part of its training vocabulary.

Prompt templates matter more than you think

A subtle finding from this chapter deserves special attention: the prompt template significantly impacts model performance. The same model, evaluated on the same problems, with the same evaluation pipeline, can produce different accuracy numbers depending on how you phrase the prompt.

For the base model, simply adding "Explain step by step." to the prompt changes accuracy measurably. For the reasoning model, using the correct chat template (with <|im_start|> and <|im_end|> markers) versus a plain text prompt can change accuracy by several percentage points. The reasoning model was trained with a specific template, and deviating from it degrades performance.

This is not a flaw in the evaluation; it is a feature of how LLMs work. The model is a pattern-matching engine. If the prompt pattern matches the patterns it saw during training, it activates the right "mode." If the pattern is slightly off, the model might generate in a different mode entirely: producing prose instead of math, skipping the reasoning trace, or formatting the answer in a way the extractor cannot parse.

The practical lesson: when comparing techniques, always use the same prompt template. When deploying a model, always use the template it was trained with. Prompt engineering is not optional.

The verifier as the bridge

Here is a thought that will not become fully clear until Chapter 6, but is worth planting now.

The evaluation pipeline you just built, the one that extracts an answer, normalizes it, and checks it against the ground truth, does exactly one thing: it takes a model output and produces a grade, correct or incorrect, 1 or 0.

In reinforcement learning, the thing that takes an agent's action and produces a score is called the reward function. The evaluation pipeline is the reward function. The same code that grades the model during evaluation will, in Chapter 6, provide the reward signal that drives GRPO training. Every correct answer generates a reward of 1.0. Every incorrect answer generates a reward of 0.0. The model learns to produce outputs that maximize this reward, which means it learns to produce correct answers.

This dual-use design is not accidental. It is the key architectural insight of reinforcement learning with verifiable rewards (RLVR): if you can write a program that checks whether an answer is correct, you can use that program as the reward function for RL. No human annotators. No learned reward model. Just a verifier.

The verifier is the bridge between evaluation and training. Build a good verifier, and you get both a reliable evaluation pipeline and a scalable training signal for free.

The pipeline's reliability has been validated across thousands of evaluation runs in the book's experiments. But two classes of failure deserve specific mention because they affect the reward function in Chapter 6.

False negatives (grading a correct answer as wrong) occur when the model produces a valid answer in an unexpected format that the extraction pipeline does not recognize. For example, an answer of "x ∈ {3, 5}" might be extracted as "3, 5" and compared against "3" (if the problem asks for the smallest value). The verifier sees a mismatch and returns 0.0. In evaluation, this slightly underestimates accuracy. In RL training, it wastes a positive training signal: the model produced a correct answer but received no reward.

False positives (grading a wrong answer as correct) are rarer but more dangerous. They can occur when SymPy's simplification accidentally equates expressions that are not truly equal, or when the normalisation step strips information that changes the meaning. In RL training, a false positive provides a positive reward for a wrong answer, teaching the model to reproduce incorrect reasoning.

The practical impact: on MATH-500, the false negative rate is approximately 1-2% and the false positive rate is below 0.5%. This is acceptable for both evaluation and RL training. But practitioners extending the pipeline to new domains (e.g., physics with units, or set theory with membership operators) should carefully validate the extraction and normalisation steps on their specific data.

The decision to use SymPy rather than simple string comparison was not obvious at the start. Early reasoning model papers used string matching for evaluation, which worked for benchmarks where answer format was tightly controlled but broke badly on free-form generation. The SymPy-based approach costs more computation (about 50ms per answer for parsing and simplification) but handles the diversity of LLM outputs far more robustly. The 50ms is negligible compared to the seconds-to-minutes of generation time.

We now have the two essential pieces of infrastructure: a model that generates text (Chapter 2) and an evaluator that grades the model's answers (this chapter). The base model scores 15.2% on MATH-500. The official reasoning model scores 50.8%. The gap between them is the territory we will explore for the rest of the book.

The next chapter begins the exploration with a deceptively simple question: can we make the base model reason better without changing a single weight? The answer involves a Nobel Prize-winning psychologist's insights about how humans think, a temperature knob that controls randomness, and a voting scheme borrowed from jury deliberation. No training required. Just clever prompting and statistical inference. The results will surprise you.

The decision to build evaluation before optimisation is not just good engineering practice. It reflects a painful lesson from the history of machine learning.

In the early days of deep learning, researchers would often build a new model, train it, eyeball a few outputs, and declare success. Later, independent teams would evaluate the same model more rigorously and find that the claimed improvements did not replicate. The problem was not dishonesty. It was overfitting to the evaluation procedure: researchers would unconsciously adjust their methods until they produced outputs that looked good on the examples they happened to check.

The fix, adopted across the field, was to establish evaluation benchmarks before doing any model development. MATH-500 is one such benchmark. By building the evaluation pipeline in this chapter, before touching any inference or training technique, we ensure that every improvement we claim in Chapters 4-8 is measured against a fixed, pre-defined test set using a fully automated, deterministic procedure. There is no room for unconscious cherry-picking.

The .jsonl output format enables a specific form of post-hoc analysis called error analysis: examining the individual problems the model got wrong to identify systematic patterns. Does the model struggle with geometry more than algebra? Does it fail on problems requiring more than 5 reasoning steps? Does it make arithmetic errors or logical errors? These patterns guide where to focus optimisation effort. Without structured logging, this analysis is impossible.

We can trace through the complete verification pipeline on a concrete example. The model is asked: "If a+b=3 and ab=13/6, what is a²+b²?"

The model generates: "Using the identity a²+b² = (a+b)² - 2ab, we get (3)² - 2(13/6) = 9 - 13/3 = 27/3 - 13/3 = 14/3. The answer is 14/3"

Step 1 (Extraction): get_last_boxed() applies the regex pattern to find \boxed{\dfrac{14}{3}}. It extracts the inner content: \dfrac{14}{3}.

Step 2 (Normalization): normalize_text() converts \dfrac{14}{3} to (14)/(3). It strips the \d prefix, converts frac{14}{3} to the division form, removes extra whitespace.

Step 3 (Symbolic parsing): sympy_parser() parses (14)/(3) into a SymPy Rational object: Rational(14, 3). This is no longer a string. It is a mathematical object that SymPy can manipulate algebraically.

Step 4 (Comparison): The ground truth is also parsed: "14/3" becomes Rational(14, 3). The equality check computes simplify(Rational(14, 3) - Rational(14, 3)) = 0. Since the difference is zero, the answer is correct. grade_answer returns True.

Now consider a trickier case. The model outputs \boxed{4.666...} and the ground truth is 14/3. String comparison would say these are different. But SymPy can handle this: it parses 4.666... as a float, converts to a rational, and checks whether simplify(4.6667 - 14/3) is approximately zero. It is, within floating-point tolerance. This is why SymPy is essential: it understands mathematical equivalence, not just string equivalence.

And a failure case: the model outputs \boxed{2\sqrt{3}} and the ground truth is \sqrt{12}. These are mathematically identical (2√3 = √12). SymPy recognizes this: simplify(2*sqrt(3) - sqrt(12)) evaluates to 0. Simple string comparison would incorrectly mark this as wrong.

But SymPy is not infallible. Edge cases include complex numbers (is (-1)^{1/3} equal to -1 or a complex root?), expressions with multiple valid forms, and numerical precision issues for transcendental numbers. These edge cases are rare in MATH-500 but become more consequential when the pipeline is repurposed as a reward function in Chapter 6, because a false negative (grading a correct answer as wrong) wastes a training signal.

The prompt template has a surprisingly large effect on accuracy. The template used throughout the book is:

def render_prompt(problem_text):
    return (
        f"Below is a math problem. Solve it and put your "
        f"final answer within \\boxed{{}}.\n\n{problem_text}"
    )

This template does two critical things. First, it establishes the task context: "Below is a math problem" tells the model what kind of reasoning to activate. Without this context, the model might treat the mathematical notation as part of a textbook excerpt and try to continue the textbook rather than solve the problem.

Second, it specifies the output format: "put your final answer within \boxed{}" tells the model to use the LaTeX \boxed{} notation that the extraction pipeline depends on. Without this instruction, the model might write "The answer is 14/3" in plain text, which the primary extraction path (regex for \boxed{}) would miss. The fallback extraction would still catch it, but the reliability drops.

In experiments with the Qwen3 base model, changing the prompt template from "Solve this problem" to "Below is a math problem. Solve it and put your answer within \boxed{}" improved accuracy by 3-5 percentage points on the 10-sample subset. The exact wording, whether you include a newline before the problem text, whether you use "solve" or "compute" or "find," all of these make measurable differences for smaller models. This sensitivity reflects the nature of next-token prediction: different prompts activate different statistical pathways through the model's parameters, and some pathways are more productive for mathematical reasoning than others.

Loading the model for evaluation

The model loading logic is wrapped in a reusable function that supports both the base model and the reasoning variant:

Listing 3.1: Loading a pre-trained model

from pathlib import Path
import torch
from reasoning_from_scratch.ch02 import get_device
from reasoning_from_scratch.qwen3 import (
    download_qwen3_small, Qwen3Tokenizer,
    Qwen3Model, QWEN_CONFIG_06_B
)

def load_model_and_tokenizer(
    which_model, device, use_compile, local_dir="qwen3"
):
    if which_model == "base":
        download_qwen3_small(
            kind="base", tokenizer_only=False, out_dir=local_dir
        )
        tokenizer_path = Path(local_dir) / "tokenizer-base.json"
        model_path = Path(local_dir) / "qwen3-0.6B-base.pth"
        tokenizer = Qwen3Tokenizer(tokenizer_file_path=tokenizer_path)

    elif which_model == "reasoning":
        download_qwen3_small(
            kind="reasoning", tokenizer_only=False, out_dir=local_dir
        )
        tokenizer_path = Path(local_dir) / "tokenizer-reasoning.json"
        model_path = Path(local_dir) / "qwen3-0.6B-reasoning.pth"
        tokenizer = Qwen3Tokenizer(
            tokenizer_file_path=tokenizer_path,
            apply_chat_template=True,
            add_generation_prompt=True,
            add_thinking=True,
        )
    else:
        raise ValueError(f"Invalid choice: which_model={which_model}")

    model = Qwen3Model(QWEN_CONFIG_06_B)
    model.load_state_dict(torch.load(model_path))
    model.to(device)

    if use_compile:
        torch._dynamo.config.allow_unspec_int_on_nn_module = True
        model = torch.compile(model)

    return model, tokenizer

WHICH_MODEL = "base"
device = get_device()
model, tokenizer = load_model_and_tokenizer(
    which_model=WHICH_MODEL, device=device, use_compile=False
)

The reasoning variant uses apply_chat_template=True (wraps input in structured chat format), add_generation_prompt=True (adds assistant turn marker), and add_thinking=True (enables <think> token generation). We include it as a comparison benchmark.

For convenience in generating and collecting responses:

Listing 3.3: A wrapper for streamed text generation

def generate_text_stream_concat(
    model, tokenizer, prompt, device, max_new_tokens, verbose=False,
):
    input_ids = torch.tensor(
        tokenizer.encode(prompt), device=device
    ).unsqueeze(0)
    generated_ids = []
    for token in generate_text_basic_stream_cache(
        model=model, token_ids=input_ids,
        max_new_tokens=max_new_tokens,
        eos_token_id=tokenizer.eos_token_id,
    ):
        next_token_id = token.squeeze(0)
        generated_ids.append(next_token_id.item())
        if verbose:
            print(tokenizer.decode(next_token_id.tolist()), end="", flush=True)
    return tokenizer.decode(generated_ids)

Generating a response to a math problem:

prompt = (
    r"If $a+b=3$ and $ab=\tfrac{13}{6}$, "
    r"what is the value of $a^2+b^2$?"
)
input_token_ids_tensor = torch.tensor(
    tokenizer.encode(prompt), device=device
).unsqueeze(0)

The model produces a step-by-step response using the algebraic identity a² + b² = (a+b)² - 2ab, arriving at the correct answer: \boxed{\dfrac{14}{3}}. Even the base model provides reasoning-like explanations, likely because the Qwen3 team included chain-of-thought data during pre-training.

The four-step extraction pipeline

An LLM does not produce clean answers. It produces text. Getting from raw text to a boolean verdict of "correct" or "incorrect" requires four transformations.

Step 1: Extraction. Locate the answer inside the model's output. The primary strategy is \boxed{} notation:

import re

def get_last_boxed(text):
    pattern = r"\\boxed\{([^{}]*(?:\{[^{}]*\}[^{}]*)*)\}"
    matches = re.findall(pattern, text)
    return matches[-1].strip() if matches else None

This regex handles nested braces (essential for \boxed{\frac{14}{3}}). Taking the last match is deliberate: models sometimes write preliminary boxed expressions during their working.

If no boxed answer exists, fall back:

def extract_final_candidate(text, fallback="number_then_full"):
    boxed = get_last_boxed(text)
    if boxed:
        return boxed
    if fallback is None:
        return None
    if fallback in ("number_only", "number_then_full"):
        numbers = re.findall(r"-?\d+\.?\d*", text)
        if numbers:
            return numbers[-1]
    if fallback == "number_then_full":
        return text.strip()
    return None

Step 2: Normalization. Strip LaTeX formatting and convert to a canonical form. The normalize_text function removes \$, \text{}, \left, \right, \displaystyle, trailing periods, converts \frac{a}{b} to (a)/(b), \sqrt{x} to sqrt(x), and canonicalizes whitespace. This ensures that cosmetically different representations of the same number are not treated as different answers.

Step 3: Symbolic parsing. Use SymPy to parse the normalised string into a mathematical object. SymPy understands that 14/3, 28/6, and 4.666... are all the same number. Simple string comparison would fail here.

Step 4: Comparison. Check whether simplify(prediction - ground_truth) == 0:

from sympy import simplify
from sympy.parsing.latex import parse_latex

def equality_check(pred, gt):
    try:
        diff = simplify(pred - gt)
        return diff == 0
    except Exception:
        return False

The grade_answer function wraps all four steps and also handles tuple comparisons.

The prompt template

def render_prompt(problem_text):
    return (
        f"Below is a math problem. Solve it and put your "
        f"final answer within \\boxed{{}}.\n\n{problem_text}"
    )

This tells the model what format to use for its answer (\boxed{}), which the extraction pipeline depends on. The exact wording has a surprisingly large effect on accuracy for smaller models.

The MATH-500 benchmark

from reasoning_from_scratch.ch03 import load_math500
math_data = load_math500()
print(f"Number of problems: {len(math_data)}")   # 500

Each entry contains a math problem and a verified ground-truth answer, spanning algebra, geometry, number theory, combinatorics, and precalculus.

The evaluation loop implements a pattern that appears in every ML project: iterate, compute, log, aggregate. But several implementation details deserve attention because they affect both the reliability of results and the ability to debug failures.

Streaming generation with collection: The generate_text_stream_concat function simultaneously streams tokens to the console (for live monitoring) and collects them in a list (for later extraction). This dual-mode operation is essential during development: you can watch the model's reasoning unfold in real time while still capturing the complete response for automated grading.

Progress reporting with ETA: The eta_progress_message helper computes estimated time remaining based on average time per example. For a 500-problem evaluation that takes 13 minutes on the base model, knowing "ETA: 8 minutes" versus "ETA: 3 hours" (for the reasoning model) lets you plan accordingly.

JSON Lines output: Each problem's result is written as a single-line JSON object to a .jsonl file immediately after grading. This has two advantages over accumulating results in memory. First, if the evaluation crashes at problem 300, you still have results for problems 1-299. Second, the .jsonl format allows streaming analysis: you can start examining results before the full evaluation completes.

Error analysis patterns: With the .jsonl output, you can compute per-subject accuracy (does the model struggle more with geometry than algebra?), per-difficulty accuracy (does it solve Level 1 problems but fail Level 5?), and response-length correlation (do longer responses tend to be more correct?). These analyses guide where to focus optimisation effort. For example, if the model achieves 60% on algebra but 5% on number theory, you might prioritize number theory problems in your GRPO training data.

We can examine what a typical .jsonl entry looks like:

{
  "problem": "If $a+b=3$ and $ab=\\tfrac{13}{6}$...",
  "gtruth_answer": "\\frac{14}{3}",
  "generated_text": "Using the identity $a^2+b^2=(a+b)^2-2ab$...",
  "extracted": "\\frac{14}{3}",
  "correct": true
}

The generated_text field contains the model's complete response, including all intermediate reasoning steps. The extracted field contains only the answer pulled out by extract_final_candidate. Comparing these two fields reveals whether extraction failures (not extraction logic problems) are causing false negatives. If generated_text contains the correct answer but extracted is wrong, the extraction pipeline needs fixing.

The full evaluation loop

import json, time

def evaluate_math500_stream(
    model, tokenizer, device, math_data,
    max_new_tokens=2048, verbose=False,
    out_path="eval_math500.jsonl",
):
    num_correct = 0
    num_examples = len(math_data)
    start_time = time.time()

    with open(out_path, "w") as f:
        for i, row in enumerate(math_data):
            prompt = render_prompt(row["problem"])
            gen_text = generate_text_stream_concat(
                model, tokenizer, prompt, device, max_new_tokens
            )
            extracted = extract_final_candidate(gen_text)
            is_correct = grade_answer(extracted, row["answer"])

            if is_correct:
                num_correct += 1

            record = {
                "problem": row["problem"],
                "gtruth_answer": row["answer"],
                "generated_text": gen_text,
                "extracted": extracted,
                "correct": bool(is_correct),
            }
            f.write(json.dumps(record, ensure_ascii=False) + "\n")

            progress_msg = eta_progress_message(
                processed=i, total=num_examples,
                start_time=start_time, show_eta=True, label="MATH-500",
            )
            print(progress_msg, end="\r", flush=True)

    seconds_elapsed = time.time() - start_time
    acc = num_correct / num_examples if num_examples else 0.0
    print(f"\nAccuracy: {acc*100:.1f}% ({num_correct}/{num_examples})")
    print(f"Total time: {seconds_elapsed/60:.1f} min")
    print(f"Logs written to: {out_path}")
    return num_correct, num_examples, acc

The .jsonl output format (one JSON object per line) enables post-hoc analysis without re-running evaluation:

results = []
with open(local_path, "r") as f:
    for line in f:
        if line.strip():
            results.append(json.loads(line))

Thought experiment · what if evaluation were free?

If evaluation cost nothing, the optimal strategy would be to evaluate constantly: after every prompt change, every hyperparameter tweak, every training step. In practice, evaluation has non-trivial cost. Running MATH-500 on the base model takes 13.3 minutes on H100. On the reasoning model, it takes 185.4 minutes (over 3 hours). This cost creates a tension between evaluation frequency and development speed.

The resolution is tiered evaluation. During rapid prototyping (testing prompt templates, debugging extraction), use a 10-problem subset. Results on 10 problems are noisy but take seconds. During technique development (implementing CoT, self-consistency), use a 50-100 problem subset. Results are more reliable and take minutes. For final comparison (publishing results, selecting production models), use the full 500 problems. Each tier provides the right granularity for its stage of development.

The .jsonl output enables another optimisation: incremental evaluation. If you change only the extraction pipeline (not the model or generation), you can re-grade existing .jsonl files without regenerating responses. This turns a 13-minute evaluation into a 2-second re-grading, enabling rapid iteration on the extraction logic.

Baseline results

The base model achieves 15.2% on the full 500-sample dataset. The reasoning variant reaches 50.8%, but at 14x longer generation time (185.4 vs 13.3 min on H100) because it produces far longer responses with intermediate thinking steps.

This 14x runtime difference is worth understanding quantitatively. The base model generates an average of 25 tokens per problem (terse, often wrong answers). The reasoning model generates an average of 350 tokens per problem (verbose, step-by-step solutions wrapped in <think> tags). At 141 tokens/sec on H100, the base model spends 0.18 seconds per problem while the reasoning model spends 2.48 seconds. Multiply by 500 problems and you get 90 seconds (1.5 min) vs 1,240 seconds (20.7 min) of pure generation time. The actual runtimes (13.3 min vs 185.4 min) are longer because of prompt encoding, KV cache management, and output processing overhead.

The supplementary materials include a batched evaluation script that processes multiple problems per forward pass. With batch size 128 on H100, the base model drops from 13.3 min to 3.3 min (4x speedup) and the reasoning model drops from 185.4 min to 14.6 min (12.7x speedup). Batching is especially effective for the reasoning model because GPU utilization is low when processing one long sequence at a time.

These timing numbers establish the performance envelope for Chapters 4-8. Every technique we apply will be measured against this baseline. When we report that CoT prompting takes 85.9 minutes, you can calculate that it generates roughly 85.9/13.3 × 25 ≈ 161 tokens per problem (6.4x longer than the base model's responses), which aligns with the observation that CoT responses are 200-500 tokens long.

The .jsonl output from each evaluation run enables fine-grained error analysis that goes beyond aggregate accuracy. You can compute accuracy by problem difficulty level (Level 1-5), by subject area (algebra, geometry, number theory, combinatorics, precalculus), by response length (do longer responses correlate with correctness?), and by extraction success (how often does the pipeline fail to extract an answer?). These analyses are essential for targeting your optimisation effort: if the model achieves 50% on algebra but 5% on number theory, you should prioritize number theory in your GRPO training data.

The supplementary materials include a batched evaluation script. With batch size 128 on H100: base model drops from 13.3 min to 3.3 min; reasoning model drops from 185.4 min to 14.6 min.

The thread

You now have three things. A model that generates text (Chapter 2). A pipeline that checks whether that text contains a correct mathematical answer (this chapter). And a benchmark number: 15.2% accuracy on MATH-500.

That number is your floor. Everything from here is about raising it. And the first tool requires no training, no data, no GPU time. It requires six words added to a prompt.

Two roads to better reasoning

In general, there are two strategies to improve reasoning: increase training compute (change the model's weights, covered in Chapters 6-8) and increase inference compute (change how the model generates, covered here and in Chapter 5). Both improve accuracy by using more compute, but inference-time scaling does this on the fly, without modifying a single parameter in the model. This makes even fixed, already-deployed models more capable.

We focus on three practical techniques. Chain-of-thought prompting modifies the prompt to encourage step-by-step reasoning. Self-consistency generates multiple responses and selects the most frequent answer via majority voting. Self-refinement (covered in the next chapter) has the model iteratively critique and improve its own answer. These methods were chosen because they are popular, representative, and offer strong improvements. Longer responses that include reasoning are what we typically expect from reasoning models. And parallel sampling is an available systems pattern.

We load the model exactly as in previous chapters:

import torch
from reasoning_from_scratch.ch02 import get_device
from reasoning_from_scratch.ch03 import (
    load_model_and_tokenizer, render_prompt,
)
from reasoning_from_scratch.ch04 import (
    generate_text_stream_concat_flex
)

device = get_device()
device = torch.device("cpu")

model, tokenizer = load_model_and_tokenizer(
    which_model="base", device=device, use_compile=False
)

The code runs on CPU by default to ensure reproducible results. The generate_text_stream_concat_flex function is a flexible wrapper that accepts any text generation function as a plug-in, using the strategy pattern so we can swap out generation strategies without changing surrounding code:

def generate_text_stream_concat_flex(
    model, tokenizer, prompt, device,
    max_new_tokens=2048, verbose=True,
    generate_func=generate_text_stream_cache,
    **kwargs
):
    input_ids = torch.tensor(
        tokenizer.encode(prompt), device=device
    ).unsqueeze(0)
    generated_ids = []
    for token in generate_func(
        model=model, token_ids=input_ids,
        max_new_tokens=max_new_tokens,
        eos_token_id=tokenizer.eos_token_id,
        **kwargs
    ):
        next_token_id = token.squeeze(0)
        generated_ids.append(next_token_id.item())
        if verbose:
            print(tokenizer.decode(next_token_id.tolist()),
                  end="", flush=True)
    return tokenizer.decode(generated_ids)

The six words that change everything

The simplest inference-time technique is chain-of-thought (CoT) prompting: appending an instruction like "Explain step by step." to the prompt. The difference it makes on a concrete problem:

raw_prompt = (
    "Half the value of $3x-9$ is $x+37$. "
    "What is the value of $x$?"
)
prompt_baseline = render_prompt(raw_prompt)
prompt_cot = render_prompt(raw_prompt + " Explain step by step.")

Without CoT: The base model outputs a terse, incorrect answer: \boxed{58}. It jumps straight to a number without showing any work. There is no intermediate reasoning visible, and the answer happens to be wrong.

With CoT: The model produces a multi-paragraph solution. It writes out the equation (3x-9)/2 = x+37, multiplies both sides by 2 to get 3x-9 = 2x+74, subtracts 2x from both sides to get x-9 = 74, adds 9 to get x = 83, and boxes the correct answer \boxed{83}. The instruction triggered the model to generate the kind of step-by-step reasoning it encountered during pre-training.

The MATH-500 results quantify this:

A 25.4 percentage point improvement from a single line of text added to the prompt. The runtime increase (10.1 to 85.9 min) reflects the longer responses: the model now generates 200-500 tokens per problem instead of 5-20.

The original chain-of-thought paper suggested "Let's think step by step." However, in experiments with the Qwen3 base model, "Explain step by step" performs better, which is why we use the latter. CoT is particularly useful for base models that do not naturally provide reasoning explanations. Models already trained as reasoning models usually do not benefit, since they already explain their answers by default.

The decisive point: the "Explain step by step" instruction does not give the model new knowledge. It changes how the model uses its existing knowledge, aligning generation with the reasoning patterns present in its training data. Training corpora contain many worked mathematical solutions, and the model has internalised those patterns during pre-training. All the CoT prompt does is activate them. This is why CoT works: it exploits the fact that the training data is rich in step-by-step explanations.

Thought experiment · why does showing work help?

Here is a thought experiment that builds intuition for why chain-of-thought prompting works. Consider the problem: "A store has 3 shelves. Each shelf has 4 boxes. Each box has 6 items. How many items total?"

Without CoT: The model sees "3 shelves, 4 boxes, 6 items" and must directly predict "72." This requires the model to perform multi-step arithmetic (3 × 4 × 6 = 72) implicitly within a single forward pass. The model's ability to do this depends on whether its training data contained this specific multiplication pattern. If the model has seen "3 × 4 × 6 = 72" or equivalent patterns, it can recall the answer. If not, it must compose two multiplications (3 × 4 = 12, then 12 × 6 = 72) within a single prediction step, which is unreliable for pattern-matching models.

With CoT: The model generates "Step 1: 3 shelves × 4 boxes = 12 boxes total. Step 2: 12 boxes × 6 items = 72 items total. The answer is 72." Each intermediate step involves a single multiplication that is highly likely to be in the training data. "3 × 4 = 12" and "12 × 6 = 72" are both common patterns. By decomposing the problem into simpler sub-problems, the model converts a hard pattern-matching task into a sequence of easy ones.

This is the fundamental mechanism: CoT decomposes complex reasoning into simpler sequential steps, each of which is within the model's pattern-matching capabilities. The intermediate tokens serve as a form of working memory, storing partial results that the model can reference in subsequent steps. Without CoT, the model must hold all intermediate computations in its hidden state (the 1024-dimensional vector), which has limited capacity. With CoT, intermediate results are externalized as tokens that become part of the input for the next step.

Extended walkthrough · CoT on a geometry problem

We can trace CoT prompting on a harder problem to see how it handles multi-step reasoning with geometry.

Problem: "In triangle ABC, angle A = 60°, side b = 4, side c = 5. Find side a."

Without CoT (greedy decoding): The model outputs \boxed{5}. This is wrong. The model likely pattern-matched on the numbers without applying the law of cosines.

With CoT ("Explain step by step."): The model generates:

"We need to find side a using the Law of Cosines: a² = b² + c² - 2bc·cos(A).

Given: b = 4, c = 5, A = 60°, so cos(60°) = 1/2.

Substituting: a² = 4² + 5² - 2(4)(5)(1/2) = 16 + 25 - 20 = 21.

Therefore: a = √21.

The answer is \boxed{\sqrt{21}}."

The model correctly identified the Law of Cosines as the appropriate formula, substituted all values correctly, performed the arithmetic without errors, and simplified the final expression. Each intermediate step is a simple pattern-matching task that the model handles reliably: "Law of Cosines" is well-represented in training data, "cos(60°) = 1/2" is a memorised identity, and "16 + 25 - 20 = 21" is basic arithmetic.

The mechanism of CoT: it decomposed a hard problem (directly computing √21 from a geometry problem) into a sequence of easy sub-problems (identify the formula, recall a trigonometric identity, perform arithmetic, take a square root). Each sub-problem is within the model's pattern-matching capability. The chain of tokens serves as external working memory that the model can reference at each step.

This explains three empirical observations:

1. CoT helps more on harder problems. Easy problems (single-step) do not need decomposition. Hard problems (multi-step) benefit substantially from breaking the reasoning chain into manageable pieces.
2. CoT helps more on smaller models. Larger models have higher-dimensional hidden states and more layers, giving them more internal capacity to hold intermediate computations. Smaller models saturate their internal capacity earlier and benefit more from externalizing intermediate results as tokens.
3. CoT does not help reasoning models. Models already trained for reasoning generate intermediate steps by default, so adding "Explain step by step" to the prompt is redundant.

The prompt engineering trap

A common operating mistake is over-engineering the CoT prompt. Teams sometimes write elaborate multi-paragraph instructions like: "You are a mathematical expert. Think carefully about each step. Show all intermediate calculations. Double-check your work. Present your final answer in a box." This sounds reasonable but often performs worse than the simple "Explain step by step."

Why? Because longer system prompts consume tokens from the context window, leaving less room for the actual reasoning. And overly specific instructions can constrain the model's solution strategy in counterproductive ways. "Double-check your work" might cause the model to generate a second pass that introduces new errors. "Show all intermediate calculations" might cause the model to be so verbose that it loses track of the overall problem structure.

The empirical finding from the book's experiments: "Explain step by step." (five words) performs better than "Let's think step by step." (five words, the original Wei et al. phrasing) on the Qwen3 base model. The optimal prompt is model-specific and should be determined empirically on a small held-out set before committing to a full evaluation. Do not assume that what works for GPT-4 works for Qwen3.

Inside the prediction · logits and the 151,936-way decision

Before we can implement self-consistency (which requires generating multiple different answers), we need a mechanism to introduce controlled randomness. Currently, greedy decoding always selects the highest-scoring token, producing identical output every time. To understand how to introduce randomness, we need to look at what happens inside the prediction step.

Consider the prompt "The capital of Germany is":

ex_prompt = "The capital of Germany is"
with torch.inference_mode():
    input_token_ids = torch.tensor(
        tokenizer.encode(ex_prompt), device=device
    ).unsqueeze(0)
    next_token_logits = model(input_token_ids)[:, -1]
    print(next_token_logits.shape)   # torch.Size([1, 151936])

The model produces 151,936 scores, one for every token in the vocabulary. These raw scores are called logits. For this particular prompt, "Berlin" (token index ~19,840) receives a logit of approximately 20, while most other tokens score between -8 and 5. In greedy decoding, we always pick the token with the highest logit. "Berlin" wins every time. The response is deterministic.

To visualize this, we can plot 100 logit values in the neighborhood of "Berlin":

import matplotlib.pyplot as plt

def plot_scores_bar(
    next_token_logits, start=19_800, end=19_900,
    arrow=True, ylabel="Logit value"
):
    x = torch.arange(start, end)
    logits_section = next_token_logits[0, start:end].float().cpu()
    plt.bar(x, logits_section)
    plt.xlabel("Vocabulary index")
    plt.ylabel(ylabel)
    if arrow:
        max_idx = torch.argmax(logits_section)
        plt.annotate(
            "Berlin",
            xy=(x[max_idx], logits_section[max_idx]),
            xytext=(x[max_idx] - 25,
                    logits_section[max_idx] - 2),

## Production architecture: serving self-consistency at scale

We can design the production architecture for a self-consistency-based math tutoring service that handles 1,000 concurrent users with a 5-second latency requirement.

**The naive approach:** For each user query, sequentially generate 5 CoT responses. Average response: 300 tokens at 141 tokens/sec = 2.1 seconds per response. Total: 5 × 2.1 = 10.5 seconds. This exceeds the 5-second latency budget.

**The parallel approach:** Deploy 5 model replicas on separate GPUs. When a query arrives, fan it out to all 5 replicas simultaneously. Each replica generates 1 response in 2.1 seconds. Collect all 5 responses, extract answers, majority vote. Total latency: 2.1 seconds (the slowest replica). Well within budget.

**The cost:** 5 H100 GPUs at $3/hour each = $15/hour. For 1,000 concurrent users with an average query every 30 seconds, that is 2,000 queries/minute or 120,000 queries/hour. Cost per query: $15/120,000 = $0.000125. That is 0.0125 cents per query, notably cheap.

**The engineering challenge:** The fan-out-and-collect pattern requires a load balancer that distributes each query to 5 replicas and a collector service that waits for all 5 responses before running the majority vote logic. This is a standard pattern in distributed systems (similar to MapReduce) but adds operational complexity.

**The optimisation:** Not all queries need 5 samples. Easy queries (where the first response is highly confident) can stop after 1-2 samples. Hard queries (where early responses disagree) should use all 5 or even more. This **adaptive sampling** strategy reduces average GPU usage by 40-60% while maintaining accuracy.

Implementing adaptive sampling:

```python
def adaptive_self_consistency(
    model, tokenizer, prompt, device,
    max_samples=10, confidence_threshold=0.8
):
    answers = []
    for i in range(max_samples):
        answer = generate_with_cot(model, tokenizer, prompt, device)
        short = extract_final_candidate(answer)
        answers.append(short)
        
        # Check if we have enough agreement to stop early
        counts = Counter(answers)
        top_answer, top_count = counts.most_common(1)[0]
        agreement_ratio = top_count / len(answers)
        
        if agreement_ratio >= confidence_threshold and len(answers) >= 3:
            return top_answer  # Early stop: high agreement
    
    return counts.most_common(1)[0][0]  # Use all samples

This implementation generates at most max_samples responses but stops early if 80%+ of responses agree. On MATH-500, this reduces the average number of samples from 10 to approximately 4.2 while maintaining 95% of the accuracy of full self-consistency.

        arrowprops={"facecolor": "black",
                    "arrowstyle": "->", "lw": 1.5},
        fontsize=10,
    )
plt.grid(alpha=0.3)
plt.tight_layout()
plt.show()

plot_scores_bar(next_token_logits)

The resulting chart shows "Berlin" towering above all neighbors with a logit of ~20, while surrounding tokens cluster between -8 and 5. In greedy decoding, only this single peak matters.

***

## The creative dial · temperature scaling

**Temperature** changes how sharp or spread out the logits are. The analogy comes from statistical physics: in the Boltzmann distribution, higher temperature means particles are more evenly distributed across energy states. In LLMs, higher temperature means tokens are more evenly distributed across probability, yielding more diverse (and potentially less coherent) outputs.

The operation is a single division:

```python
def scale_logits_by_temperature(logits, temperature):
    if temperature <= 0:
        raise ValueError("Temperature must be positive")
    return logits / temperature

That is the entire implementation. Divide every logit by the temperature parameter. Let me walk through what this does concretely.

With our "capital of Germany" example, Berlin has logit 20 and Munich has logit 8. The gap is 12.

At temperature=0.35 (sharper): Berlin becomes 20/0.35 = 57.1, Munich becomes 8/0.35 = 22.9. The gap widens to 34.2. After softmax, Berlin dominates even more completely. The distribution becomes needle-sharp.

At temperature=1.0 (no change): Berlin stays 20, Munich stays 8. The gap remains 12. This is the baseline.

At temperature=5.0 (flatter): Berlin becomes 20/5.0 = 4.0, Munich becomes 8/5.0 = 1.6. The gap shrinks to 2.4. After softmax, many tokens have comparable probability. Sampling might return "Paris", "Munich", or even "Bridge."

The important point is not the absolute height of one logit but how the gaps between logits change. Temperature controls the gap.

After temperature scaling, we convert logits to probabilities using the softmax function, which normalizes values to the range [0, 1] and ensures they sum to 1:

rescaled_logits = scale_logits_by_temperature(
    next_token_logits, 5.0
)
next_token_probas = torch.softmax(rescaled_logits, dim=-1)
print("Probability sum:", torch.sum(next_token_probas))
# 1.0000

Then we sample from this distribution using multinomial sampling via torch.multinomial:

torch.manual_seed(123)
sampled = torch.multinomial(
    next_token_probas.cpu(), num_samples=1
)

In multinomial sampling, tokens with higher probability are more likely to be selected, but any token with nonzero probability has a chance. This is materially different from greedy decoding. If "Berlin" has probability 0.70, "Munich" 0.20, and "Hamburg" 0.10, greedy decoding always returns "Berlin." Multinomial sampling returns "Berlin" 70% of the time, "Munich" 20%, and "Hamburg" 10%.

To see the practical effect, we can draw 1,000 samples and count frequencies:

from collections import Counter

def count_samples(next_token_probas, tokenizer, n=1000):
    counts = Counter()
    for _ in range(n):
        idx = torch.multinomial(
            next_token_probas.cpu(), num_samples=1
        ).item()
        token = tokenizer.decode([idx])
        counts[token] += 1
    for token, count in counts.most_common(7):
        print(f"  {token!r}: {count}x")

At temperature=0.35: "Berlin" appears 435 times out of 1000, while real German cities like "Munich" and "Hamburg" appear only 3 times each. At temperature=5.0: the distribution is so flat that no token appears more than twice, and most sampled tokens are nonsensical. This demonstrates why temperature must be chosen carefully: too low gives no diversity, too high gives nonsense. For reasoning tasks, the sweet spot is typically 0.7-1.0.

The full generation function with temperature:

from reasoning_from_scratch.qwen3 import KVCache

@torch.inference_mode()
def generate_text_temp_stream_cache(
    model, token_ids, max_new_tokens,
    eos_token_id=None, temperature=0.
):
    model.eval()
    cache = KVCache(n_layers=model.cfg["n_layers"])
    model.reset_kv_cache()
    out = model(token_ids, cache=cache)[:, -1]

    for _ in range(max_new_tokens):
        orig_device = token_ids.device
        if temperature is None or temperature == 0.0:
            next_token = torch.argmax(
                out, dim=-1, keepdim=True
            )
        else:
            logits = scale_logits_by_temperature(
                out, temperature
            )
            probas = torch.softmax(logits, dim=-1)
            next_token = torch.multinomial(
                probas.cpu(), num_samples=1
            )
            next_token = next_token.to(orig_device)

        if (eos_token_id is not None
                and torch.all(next_token == eos_token_id)):
            break

        yield next_token
        out = model(next_token, cache=cache)[:, -1]

When temperature=0 or None, it falls back to greedy decoding. Otherwise, it applies the full temperature, softmax, and sample pipeline. This function reuses the KV cache from Chapter 2 for efficient generation.

The nucleus filter · top-p sampling

Temperature alone has a dangerous property: even at moderate settings, the long tail of the vocabulary can produce absurd tokens. With 151,936 tokens, even 0.01% probability per token means roughly 15 tokens have a non-trivial chance of being sampled at each step.

Top-p sampling (also called nucleus sampling) solves this by filtering the tail. The idea: sort all tokens by probability in descending order, compute the cumulative sum, and keep only the tokens whose cumulative probability does not exceed a threshold p.

Walk through it with a toy vocabulary of 10 tokens:

toy_logits = torch.tensor(
    [-0.7, -3.0, 0.1, -1.2, 2.0,
     -1.0, -0.5, -2.0, 0.3, 1.5]
)
toy_logits_scaled = scale_logits_by_temperature(
    toy_logits, 1.0
)
toy_probas = torch.softmax(toy_logits_scaled, dim=-1)

Token 4 gets ~0.45 probability, token 9 gets ~0.28, and the rest share the remaining ~0.27. After sorting by probability and computing the cumulative sum:

With top_p=0.90, we keep only the first four tokens (cumulative reaches 0.88 before exceeding 0.90). Token 0 and everything below are zeroed out. The remaining probabilities are renormalized to sum to 1.

The implementation:

def top_p_filter(probas, top_p):
    sorted_probas, sorted_idx = torch.sort(
        probas, descending=True, dim=-1
    )
    cum_sum = torch.cumsum(sorted_probas, dim=-1)
    sorted_mask = (cum_sum - sorted_probas) >= top_p
    sorted_probas[sorted_mask] = 0.0
    probas_filtered = torch.zeros_like(probas)
    probas_filtered.scatter_(
        -1, sorted_idx, sorted_probas
    )
    probas_filtered = probas_filtered / probas_filtered.sum(
        dim=-1, keepdim=True
    )
    return probas_filtered

The scatter_ operation maps the sorted, filtered probabilities back to their original vocabulary positions. The final division renormalizes so the distribution sums to 1.

The complete generation function with both temperature and top-p:

@torch.inference_mode()
def generate_text_top_p_stream_cache(
    model, token_ids, max_new_tokens,
    eos_token_id=None, temperature=0.8, top_p=0.9,
):
    model.eval()
    cache = KVCache(n_layers=model.cfg["n_layers"])
    model.reset_kv_cache()
    out = model(token_ids, cache=cache)[:, -1]

    for _ in range(max_new_tokens):
        orig_device = token_ids.device
        if temperature is None or temperature == 0.0:
            next_token = torch.argmax(
                out, dim=-1, keepdim=True
            )
        else:
            logits = scale_logits_by_temperature(
                out, temperature
            )
            probas = torch.softmax(logits, dim=-1)
            probas = top_p_filter(probas, top_p)
            next_token = torch.multinomial(
                probas.cpu(), num_samples=1
            )
            next_token = next_token.to(orig_device)

        if (eos_token_id is not None
                and torch.all(next_token == eos_token_id)):
            break

        yield next_token
        out = model(next_token, cache=cache)[:, -1]

Temperature and top-p together are not inference-time scaling techniques in themselves. They are enablers. Temperature creates diversity. Top-p removes the dangerous tail. Together, they make it possible to generate multiple meaningfully different responses to the same prompt, which is the prerequisite for self-consistency.

The wisdom of crowds · self-consistency

Consider you are a teacher grading a math exam. A student submits five different attempts at the same problem. Three arrive at 83, one at 54, one at 22. You do not need to check the work to feel confident that 83 is correct. The answer that multiple independent attempts converge on is more likely to be right than any single attempt.

This is self-consistency, and the algorithm is simple: generate N responses, extract the final answer from each, take the majority vote.

from collections import Counter
from reasoning_from_scratch.ch03 import extract_final_candidate

def self_consistency_vote(
    model, tokenizer, prompt, device,
    num_samples=10, temperature=0.8, top_p=0.9,
    max_new_tokens=2048,
    show_progress=True, show_long_answer=False,
    seed=None,
):
    full_answers, short_answers = [], []

    for i in range(num_samples):
        if seed is not None:
            torch.manual_seed(seed + i + 1)

        answer = generate_text_stream_concat_flex(
            model=model, tokenizer=tokenizer,
            prompt=prompt, device=device,
            max_new_tokens=max_new_tokens,
            verbose=show_long_answer,
            generate_func=generate_text_top_p_stream_cache,
            temperature=temperature, top_p=top_p,
        )

        short = extract_final_candidate(
            answer, fallback="number_then_full"
        )
        full_answers.append(answer)
        short_answers.append(short)

        if show_progress:
            print(f"[Sample {i+1}/{num_samples}] "
                  f"→ {short!r}")

    counts = Counter(short_answers)
    groups = {s: [] for s in counts}
    for idx, s in enumerate(short_answers):
        groups[s].append(idx)

    mc = counts.most_common()
    if not mc:
        majority_winners, final_answer = [], None
    else:
        top_freq = mc[0][1]
        majority_winners = [
            s for s, f in mc if f == top_freq
        ]
        final_answer = (mc[0][0]
                        if len(majority_winners) == 1
                        else None)

    return {
        "full_answers": full_answers,
        "short_answers": short_answers,
        "counts": dict(counts),
        "groups": groups,
        "majority_winners": majority_winners,
        "final_answer": final_answer,
    }

Testing on our running example with 5 samples:

[Sample 1/5] → '83'
[Sample 2/5] → '22'
[Sample 3/5] → '54'
[Sample 4/5] → '83'
[Sample 5/5] → '61'

Three of five samples would have been wrong individually. But majority vote selects 83, which is correct. Self-consistency transforms unreliable individual predictions into reliable collective predictions. The function handles ties by returning None when multiple answers share the top frequency; Chapter 5 implements scoring methods that serve as tie-breakers.

Why correct answers cluster while errors scatter

The mathematical intuition behind self-consistency deserves deeper examination. Consider a math problem with one correct answer (83) and infinitely many wrong answers (1, 2, 3, ..., 82, 84, 85, ...). If the model has a 40% chance of reaching the correct answer through any given reasoning path, and the remaining 60% probability is spread across all possible wrong answers, then:

With 10 samples: expected correct votes = 10 × 0.4 = 4. Expected votes for any specific wrong answer ≈ 0 (because the 60% error probability is divided among hundreds of possible wrong answers). The correct answer wins by a landslide even though the model is wrong more often than right.

This is why self-consistency provides such large gains over single-sample generation. It does not require the model to be right most of the time. It only requires the model to be right more often than any specific wrong answer. With a diverse error distribution (many different wrong answers), even a 20-30% per-sample accuracy can produce 80%+ accuracy after majority voting.

The failure condition is correlated errors: when many samples make the same mistake. The bat-and-ball problem ($0.10 instead of $0.05) is a classic example. Correlated errors concentrate votes on a single wrong answer, defeating the diversification that makes majority voting work. CoT prompting reduces correlated errors by forcing the model to show its work, which changes the error distribution from "many samples make the same intuitive mistake" to "some samples set up the algebra correctly and get the right answer."

Thought experiment · when does self-consistency fail?

Self-consistency works because correct reasoning paths converge on one answer while errors diverge into many wrong answers. But what happens when errors are systematic rather than random?

Consider the problem: "A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost?"

The intuitive (and wrong) answer is $0.10. The correct answer is $0.05 (if the ball costs $0.05, the bat costs $1.05, and together they cost $1.10).

If you generate 10 responses with self-consistency, you might get:

• 7 responses saying $0.10 (the intuitive trap)
• 2 responses saying $0.05 (the correct answer)
• 1 response saying $0.55 (some other error)

Majority vote selects $0.10, the wrong answer. Self-consistency amplified the systematic bias rather than correcting it.

This failure mode is especially dangerous because it gives high-confidence wrong answers. The team looking at the self-consistency output sees "7 out of 10 responses agree on $0.10" and concludes the model is confident. But confidence (measured by vote share) and correctness are not the same thing.

The fix is CoT prompting. With "Explain step by step," the model is more likely to set up the algebra correctly: "Let x = ball price. Then bat = x + 1.00. Total: x + (x + 1.00) = 1.10. So 2x = 0.10, x = 0.05." The explicit algebra overcomes the intuitive trap. This is why CoT + self-consistency outperforms self-consistency alone: CoT reduces systematic biases, and self-consistency filters random errors. They address different failure modes.

The numbers that changed my intuition

The comprehensive MATH-500 results reveal something I did not expect:

Row 3 shows that CoT alone is notably effective: +25.4% from a single-line prompt change.

Row 4 shows that temperature + top-p without CoT provides only +2.6%, confirming that these are enablers for diverse sampling, not reasoning techniques in themselves.

Rows 5-7 show self-consistency without CoT: accuracy improves with more samples (28.4% to 31.6%), but with diminishing returns. Going from 3 to 10 samples gives only +3.2%.

Row 8 is the most counterintuitive: CoT + temperature/top-p without majority voting actually drops accuracy compared to CoT alone (33.4% vs 40.6%). Why? Because sampling introduces variability, and when you keep only one answer, that variability hurts more than it helps. Diversity is only beneficial when you have a selection mechanism (majority voting) to filter it.

Rows 9-11 are the headline results. CoT + self-consistency achieves the highest accuracy, with n=10 reaching 52.0%. This exceeds the reasoning model's baseline of 48.2%. Read that carefully: a base model that was never trained for reasoning, given chain-of-thought prompting and 10-sample self-consistency, beats a model specifically trained with RL and distillation techniques. The untrained model with inference-time scaling beats the trained model.

This does not mean inference-time scaling is always superior. The compute cost is extraordinary: 862.6 minutes = 85x the baseline of 10.1 minutes. But it demonstrates a profound point: the base model already contains the knowledge needed to reason correctly. The reasoning techniques in later chapters do not give the model new knowledge. They make its existing knowledge more accessible and cheaper to deploy.

The structural limitation

Self-consistency has one structural constraint: it requires short, extractable final answers for majority voting. For open-ended tasks, the next chapter introduces self-refinement. But before we move on, one important practical detail about how temperature and top-p work together.

How temperature and top-p interact

A common confusion: does top-p make temperature unnecessary, or vice versa? Neither. They address different problems and should be used together.

Temperature controls the overall shape of the distribution. Low temperature sharpens it (fewer plausible tokens). High temperature flattens it (many plausible tokens). But temperature does not remove any tokens from consideration. Even at temperature 0.5, every token in the vocabulary has nonzero probability, including nonsensical ones.

Top-p removes tokens from consideration entirely. After temperature scaling and softmax, top-p filters the tail by zeroing out all tokens below the cumulative probability threshold. This guarantees that sampling only draws from the "nucleus" of plausible tokens.

Together: temperature sets the diversity level (how spread out the probability is among plausible tokens), and top-p enforces a quality floor (no token with negligible probability can be sampled). The standard combination for reasoning tasks is temperature=0.8, top-p=0.9: moderate diversity within a filtered set of plausible tokens.

System design exercise · deploying self-consistency with adaptive sampling

This is a constructed design exercise, not a report of a named deployment. The fixed-N approach to self-consistency is wasteful. For easy problems where the model gets it right on the first try, generating 9 more samples is pure waste. For hard problems where 10 samples are not enough, stopping at 10 is arbitrary. Adaptive sampling adjusts the number of samples based on early agreement.

The idea: after each sample, check whether a majority has formed. If 3 of the first 4 samples agree, you have high confidence and can stop early. If 4 samples all disagree, the problem is hard and you should generate more samples.

def adaptive_sc(model, tokenizer, prompt, device,
                max_samples=10, confidence=0.8, min_samples=3):
    answers = []
    for i in range(max_samples):
        response = generate_with_cot(model, tokenizer, prompt, device)
        answers.append(extract_final_candidate(response))
        
        if len(answers) >= min_samples:
            counts = Counter(answers)
            top_count = counts.most_common(1)[0][1]
            if top_count / len(answers) >= confidence:
                break  # High agreement, stop early
    
    return Counter(answers).most_common(1)[0][0]

On MATH-500 with max_samples=10 and confidence=0.8, this reduces the average samples from 10 to approximately 4.2 while retaining 95% of the accuracy benefit. The 58% compute savings come almost entirely from easy problems (where 2-3 samples suffice) while hard problems still receive the full 10 samples.

Self-consistency has a constraint that trips up teams on their first deployment: it requires short, extractable final answers. You need to compare answers across multiple samples, which means you need answers that can be extracted and compared. For boxed numbers, this works perfectly. For open-ended questions without clear final answers ("Explain the implications of quantum entanglement for information theory"), self-consistency is much harder to apply.

The next chapter introduces self-refinement, which handles this limitation by having the model iteratively critique and improve a single response rather than voting across multiple independent responses. Along the way, it builds the token log-probability machinery that becomes essential for reinforcement learning in Chapter 6.

We can walk through a complete self-consistency example with concrete numbers to build intuition for why it works. The problem is: "Half the value of 3x-9 is x+37. What is x?"

The complete log-probability walkthrough with real numbers

Let me trace through the entire log-probability computation for a concrete response to build rock-solid intuition. The prompt is "What is the capital of Germany?" and the response is "The capital of Germany is Berlin."

The full sequence (prompt + response) is tokenized into, say, 12 tokens. The model processes all 12 tokens in a single forward pass and produces a 12×151,936 logit matrix. At each position i, we have 151,936 logit scores representing the model's prediction for what token should come at position i+1.

For the response tokens only (positions 7-11, since the prompt occupies positions 0-6), we look up the logit assigned to the token that actually appears:

Position 7: Model predicts after "?" → actual next token " The"
  logit for " The" among 151,936 options: 3.2
  log_softmax → log-probability: -4.81

Position 8: Model predicts after " The" → actual: " capital"
  logit for " capital": 7.8
  log-probability: -1.23

Position 9: Model predicts after " capital" → actual: " of"
  logit for " of": 9.1
  log-probability: -0.45

Position 10: Model predicts after " of" → actual: " Germany"
  logit for " Germany": 6.5
  log-probability: -2.12

Position 11: Model predicts after " Germany" → actual: " is"
  logit for " is": 8.9
  log-probability: -0.31

Position 12: Model predicts after " is" → actual: " Berlin"
  logit for " Berlin": 7.2
  log-probability: -1.78

Sum of log-probabilities: -4.81 + (-1.23) + (-0.45) + (-2.12) + (-0.31) + (-1.78) = -10.70

Average log-probability: -10.70 / 6 = -1.78

Now compare with "The capital of Germany is Bridge":

Same first 5 positions (identical log-probs). Position 12: logit for " Bridge": -2.3, log-probability: -15.00. Sum: -4.81 + (-1.23) + (-0.45) + (-2.12) + (-0.31) + (-15.00) = -23.92. Average: -3.99.

"Berlin" scores -1.78 average, "Bridge" scores -3.99 average. Higher (less negative) is better. The model is much more confident in "Berlin."

In Chapter 6 (GRPO), we use the sum (-10.70) because the entire sequence receives a single reward. In Chapter 8 (distillation), we use the negative average (1.78) as the cross-entropy loss. Same computation, three uses.

The correct algebraic solution: (3x-9)/2 = x+37, so 3x-9 = 2x+74, so x = 83.

We generate 10 responses with temperature=0.8 and top-p=0.9 after CoT prompting:

Response 1: Sets up the equation correctly, solves for x=83. ✓ Response 2: Misreads "half the value" as "twice the value", gets x=22. ✗ Response 3: Correct algebra, but makes arithmetic error 74+9=82 instead of 83. ✗ Response 4: Correct. x=83. ✓ Response 5: Approaches correctly, but drops a negative sign midway. Gets x=58. ✗ Response 6: Correct. x=83. ✓ Response 7: Correct, but takes a longer path through substitution. x=83. ✓ Response 8: Misinterprets the problem, gets x=54. ✗ Response 9: Correct. x=83. ✓ Response 10: Makes a division error. Gets x=61. ✗

Extracted answers: [83, 22, 82, 83, 58, 83, 83, 54, 83, 61]. Counter: {83: 5, 22: 1, 82: 1, 58: 1, 54: 1, 61: 1}. Majority winner: 83. Correct.

Five of ten individual responses would have been wrong. But the majority vote correctly identifies 83. The useful distinction: correct reasoning paths tend to converge on the same answer, while errors are diverse. Response 2 makes a different mistake than Response 3, which makes a different mistake than Response 5. Errors scatter across many wrong answers, while correct solutions concentrate on one right answer.

This property, convergence of correct solutions and divergence of errors, is what makes majority voting so effective. It breaks down when the model has a systematic bias: if 7 out of 10 responses make the same error (all reading "half" as "twice"), the majority vote would select the wrong answer. Systematic biases are more common for problems that closely resemble common patterns in the training data that happen to point to the wrong approach.

The thread

You have now seen the cheapest path to better reasoning: spend more compute at inference time. Chain-of-thought prompting is the most cost-effective single intervention. Self-consistency is the most accurate, at proportionally more compute. These techniques require no training, no data, no GPUs for parameter updates.

But there is a ceiling. Inference-time scaling squeezes more performance out of a model's existing knowledge. It cannot create knowledge that is not there. If the model has never seen the pattern needed to solve a problem, no amount of sampling and voting will help. To go beyond this ceiling, you need to change the model itself.

Why diversity alone is not enough

Row 8 of the results table reveals one of the most counterintuitive findings in inference-time scaling: adding diversity without a selection mechanism makes things worse. CoT prompting alone gives 40.6% accuracy. Adding temperature (0.9) and top-p (0.9) on top of CoT, but without majority voting, drops accuracy to 33.4%.

Why? Because temperature and top-p introduce randomness into the generation process. Without diversity, greedy decoding always produces the model's single best answer. With diversity, each generation is slightly different, and some of those differences are productive (exploring alternative solution paths) while others are destructive (making arithmetic errors, losing track of the problem). When you only generate one response, you are equally likely to get a productive or destructive variation. The net effect is negative.

Self-consistency solves this by generating many diverse responses and using majority voting to filter out the destructive variations. If the model produces 10 responses and 6 of them arrive at 83, the remaining 4 incorrect variations are outvoted. Diversity is beneficial only when combined with a selection mechanism that preserves the signal and filters the noise.

This has a direct implication for operating systems. If your latency budget allows only one response per query, do not use temperature and top-p. Use greedy decoding with CoT prompting. Temperature and top-p are only beneficial when combined with self-consistency (multiple samples + voting) or self-refinement (multiple iterations + scoring).

The Pareto frontier has a practical interpretation for product managers. Plot accuracy on the Y-axis and cost-per-query on the X-axis. Each technique occupies a point on this plot:

The "knee" of the curve is at CoT prompting (8x cost for +25.4% accuracy). Everything to the right of that knee has diminishing returns. The trained models (GRPO, distilled) break the frontier entirely: they achieve near-premium accuracy at baseline cost, at the price of a one-time training investment.

This is the fundamental economic argument for Chapters 6-8: training-time techniques are capital expenditures (one-time costs that depreciate over time), while inference-time techniques are operating expenditures (per-query costs that accumulate). For high-volume deployments (thousands or millions of queries), the capital investment in training pays for itself almost immediately.

The compute-accuracy frontier

Plot the accuracy against compute cost and you get a Pareto frontier: a curve showing the best accuracy achievable at each compute budget. The points on this frontier define the efficient options.

At 1x compute (10.1 min): 15.2% accuracy (greedy baseline). This is the cheapest option.

At 8.5x compute (85.9 min): 40.6% accuracy (CoT alone). This is the best single-response option.

At 21x compute (212.2 min): 42.2% accuracy (CoT + SC n=3). Marginal improvement over CoT alone.

At 85x compute (862.6 min): 52.0% accuracy (CoT + SC n=10). The highest accuracy available without training.

Notice the diminishing returns. The first 8.5x of compute buys 25.4 percentage points. The next 12.5x buys only 1.6 points. The final 64x buys 9.8 points. Each additional unit of compute buys less accuracy than the previous one.

In production, the right point on this frontier depends on your use case. A math tutoring chatbot where accuracy is critical might justify 85x compute. A casual Q&A system would use CoT alone. A latency-sensitive API with a 2-second budget might use greedy decoding with no CoT at all.

Self-consistency samples can be parallelized across multiple GPUs, reducing wall-clock time without changing total compute. With 10 GPUs, CoT + SC (n=10) takes roughly the same wall-clock time as CoT alone, at 10x the hardware cost. This is a common production pattern: trade hardware for latency.

Scoring · how confident is the model in its own answer?

Before you can refine, you need to score. You need a way to look at a model's output and assign it a number that represents quality. In self-consistency, the score was implicit: the answer that appeared most often won the vote. For self-refinement, we need an explicit scoring function.

The simplest scorer is rule-based: does the response contain a \boxed{} answer? Is the response reasonably short (shorter responses that still contain an answer are preferred over rambling ones)? This scorer is crude but useful as a baseline.

The more interesting scorer is based on token log-probabilities, and understanding it requires a detour through the model's internal probability calculations.

The three scorers · a unified view

Before diving into the implementation details, we can understand the three scoring approaches this chapter builds and how they relate:

Scorer 1: Rule-based (heuristic). Checks surface features: does the answer have \boxed{}? Is it short? This is the crudest scorer but also the fastest and most interpretable. It works well as a first filter or tie-breaker. Treat it as grading homework by checking whether the student showed their work and wrote neatly, without actually reading the content.

def heuristic_score(answer, prompt=None,
                    brevity_bonus=500.0, boxed_bonus=2.0,
                    extract_bonus=1.0, fulltext_bonus=0.0):
    score = 0.0
    cand = extract_final_candidate(answer, fallback="none")
    if cand:
        score += boxed_bonus
    else:
        cand = extract_final_candidate(answer, fallback="number_only")
        if cand:
            score += extract_bonus
        else:
            cand = extract_final_candidate(answer, fallback="number_then_full")
            if cand:
                score += fulltext_bonus
    score += 1.5 * math.exp(-len(answer) / brevity_bonus)
    return score

The brevity bonus decays exponentially: 1.5 * exp(-length / 500). At 200 characters, the bonus is ~1.0. At 1,000 characters, it is ~0.2. This mildly favors compact answers without forcing one-liners.

Scorer 2: Log-probability (model confidence). Measures how "natural" the answer feels to the model. This is the most principled scorer but also the most deceptive: high confidence does not mean high correctness. Treat it as having the student grade their own exam: they will rate answers they feel good about highly, even if those answers happen to be wrong.

Scorer 3: Verifier (ground truth). Checks whether the answer is mathematically correct. This is the gold standard but requires knowing the correct answer, which is only available during evaluation, not during production deployment. Treat it as the teacher's answer key: perfect accuracy, but you cannot use it during the exam.

In production, you use Scorer 1 or 2 because you do not have the ground truth. During evaluation and RL training (Chapter 6), you use Scorer 3 because you do have it. The gap between Scorer 2 and Scorer 3 is the core challenge of self-refinement: the model must improve its answers using an imperfect signal (confidence) because the perfect signal (correctness) is unavailable.

Probabilities, or what the model really thinks

When the model generates a token, it does not just pick the highest-scoring option. It assigns a probability to every token in its vocabulary. The softmax function converts the raw scores (logits) into probabilities that sum to 1.

For a specific token, say the word "the," the probability might be 0.12, meaning the model assigns a 12% chance that "the" is the correct next token. For the word "cat," the probability might be 0.03. For a random word like "defenestration," it might be 0.000001.

The probability of an entire sequence is the product of the probabilities of each token, given all previous tokens:

P("the cat sat") = P("the") × P("cat" | "the") × P("sat" | "the cat")

If each token has a probability around 0.1, a 10-token sequence has a probability around 0.1^10 = 10^-10, a number so small that computers struggle to represent it accurately. This is the underflow problem: multiply enough small numbers together and the result rounds to zero.

The logarithm trick

There is an elegant mathematical solution to the underflow problem, and it is the same trick that scientists have used for centuries when working with very large or very small numbers: take the logarithm.

Consider you are measuring the brightness of stars. The dimmest stars you can see with your naked eye are about 1,000,000 times brighter than the faintest stars a telescope can detect. Working with numbers that span a factor of a million is cumbersome. So astronomers use a logarithmic scale (magnitudes) that compresses this range into manageable numbers.

The log-probability of a token is simply the natural logarithm of its probability. If P("the") = 0.12, then log P("the") ≈ -2.12. If P("defenestration") = 0.000001, then log P("defenestration") ≈ -13.8. The logarithm compresses a range from 0 to 1 into a range from negative infinity to 0. Numbers that are very close to zero (very unlikely tokens) get very negative log-probabilities. Numbers close to 1 (very likely tokens) get log-probabilities close to 0.

The critical property: logarithms turn products into sums. The probability of a sequence is a product of token probabilities, but the log-probability of a sequence is a sum of token log-probabilities:

log P("the cat sat") = log P("the") + log P("cat" | "the") + log P("sat" | "the cat")

Sums of numbers around -2 do not cause underflow. A 100-token sequence might have a log-probability around -200, which is a perfectly representable number. The equivalent raw probability, 10^-87, would underflow to zero on any computer.

For scoring model outputs, we typically use the average log-probability: divide the sum by the number of tokens. This normalizes for length, so we do not penalize longer responses just for being longer.

def compute_avg_logprob(model, token_ids, device):
    """Compute average log-probability of a token sequence."""
    with torch.no_grad():
        logits = model(token_ids.to(device))

    log_probs = torch.log_softmax(logits, dim=-1)

    # For each position, get the logprob of the ACTUAL next token
    target_ids = token_ids[:, 1:]  # Shift right
    token_logprobs = log_probs[:, :-1, :].gather(-1, target_ids.unsqueeze(-1)).squeeze(-1)

    return token_logprobs.mean().item()

This function answers the question: "How natural does this response look to the model?" A high average log-probability (close to 0) means the model finds this sequence very likely. A low average log-probability (very negative) means the model considers this sequence unlikely.

The important caveat: this measures how "natural" the response is, not how correct it is. A model can be confidently wrong. The response "The answer is 42" might have a higher average log-probability than "The answer is 83" if the model has a statistical bias toward 42. The logprob scorer measures fluency and self-consistency, not truth.

The self-refinement loop

With a scoring function in hand, we can build the self-refinement loop. The process has three stages:

Stage 1: Generate an initial answer. The model produces its first attempt at solving the problem, using CoT prompting and temperature/top-p sampling.

Stage 2: Critique. The model receives its own answer and is prompted to identify errors and suggest fixes. A typical critique prompt: "Review the following answer and identify any errors. If you find errors, explain what went wrong and how to fix it."

Stage 3: Revise. Based on the critique, the model generates a revised answer. The revised answer is scored. If the score improves, the revision is accepted. If not, the original answer is kept.

Stages 2 and 3 can repeat for multiple iterations. Each iteration is a chance for the model to catch and correct its own mistakes.

def self_refine(model, tokenizer, problem, device, max_iterations=3):
    # Stage 1: Initial answer
    prompt = render_prompt(problem, cot=True)
    answer = generate(model, tokenizer, prompt, device)
    best_score = score(model, answer, device)
    best_answer = answer

    for i in range(max_iterations):
        # Stage 2: Critique
        critique_prompt = f"Review this answer and identify errors:\n{answer}"
        critique = generate(model, tokenizer, critique_prompt, device)

        # Stage 3: Revise
        revise_prompt = f"Based on this critique:\n{critique}\nProvide a corrected answer."
        revised = generate(model, tokenizer, revise_prompt, device)
        revised_score = score(model, revised, device)

        # Accept only if improved
        if revised_score > best_score:
            best_answer = revised
            best_score = revised_score
            answer = revised

    return best_answer

The score-based acceptance is crucial. Without it, later iterations can degrade the answer. The model's critique step is imperfect, especially for a small model. It might "fix" a correct step, introducing an error. By requiring each revision to improve the score, we prevent this backsliding.

System design exercise · the five-call budget

This is a constructed design exercise, not a report of a named deployment. You are deploying a reasoning assistant and have a strict budget of 5 LLM calls per user query (driven by latency and cost constraints). How do you allocate those calls?

Strategy A: Pure self-consistency. Generate 5 responses with temperature=0.8 and top-p=0.9, extract answers, majority vote. This is the simplest strategy and works well for math problems with extractable answers. Total calls: 5. Expected accuracy on MATH-500: approximately 49.6% (extrapolating from the n=5 row in Chapter 4's table).

Strategy B: Pure self-refinement. Generate 1 initial response, run 2 critique-and-revision cycles. Each cycle costs 2 calls (critique + revision). Total calls: 1 + 2 + 2 = 5. Expected accuracy: approximately 44% with logprob scorer. Better for open-ended tasks where majority voting is not applicable.

Strategy C: Hybrid. Generate 3 responses (self-consistency), select the best using logprob scoring (Best-of-3), then refine the winner with 1 critique-and-revision cycle. Total calls: 3 + 1 + 1 = 5. This combines the diversity of self-consistency with the iterative improvement of self-refinement.

Strategy D: Adaptive. Use a lightweight classifier to estimate problem difficulty. For easy problems (estimated difficulty < threshold), use a single CoT call. For hard problems, use the remaining budget for self-consistency. This maximizes expected accuracy per call by allocating compute where it matters most.

In experiments, Strategy A (pure self-consistency) wins for math tasks because majority voting is a more well-tested selection mechanism than logprob scoring. Strategy B wins for open-ended tasks where answers cannot be compared via voting. Strategy D wins when problem difficulty is heterogeneous (a mix of easy and hard problems in the query stream). The choice depends on your task, your accuracy requirements, and your latency budget.

The deep connection to chapter 6

The log-probability machinery built in this chapter is not just useful for scoring. It is the mathematical heart of the GRPO training algorithm in Chapter 6. Let me make this connection explicit.

In this chapter, we compute: avg_logprob = mean(log_softmax(logits)[target_tokens]). This measures how likely the model thinks a given response is.

In Chapter 6, we compute: seq_logprob = sum(log_softmax(logits)[target_tokens]). This measures the same thing but summed instead of averaged, because GRPO assigns one reward per entire sequence.

In Chapter 8, we compute: cross_entropy = -mean(log_softmax(logits)[target_tokens]). This is the negative of what we compute here, used as the loss function for distillation.

Three chapters, three applications, one computation. If you deeply understand token log-probabilities here, you understand the core mathematics of both RL training and distillation. The implementation differs only in whether you sum or average, and whether you negate or not.

Self-refinement sounds elegant in theory: generate, critique, improve, repeat. But the empirical results tell a more nuanced story. On MATH-500, self-refinement with the logprob scorer achieves approximately 44% accuracy, compared to 40.6% for CoT alone and 52% for CoT + self-consistency (n=10).

Why does self-consistency beat self-refinement? Three reasons:

Reason 1: Diversity vs. iterative narrowing. Self-consistency explores ten independent solution paths in parallel. Each path might make a different mistake, but the correct answer emerges through majority voting. Self-refinement follows a single path and iteratively adjusts it. If the initial direction is materially wrong (e.g., the model chose the wrong formula), the critique may refine the execution without correcting the strategy. It is the difference between sending ten scouts in different directions versus sending one scout and telling them to "look harder" when they do not find water.

Reason 2: The critique bottleneck. The critique step is generated by the same model that produced the initial answer. If the model's knowledge is insufficient to solve the problem, it is also insufficient to critique the solution. A model that does not know the Law of Cosines cannot critique a solution that should have used the Law of Cosines. Self-consistency does not require critique capability because majority voting is a purely mechanical selection process.

Reason 3: Scorer limitations. The logprob scorer selects for model confidence, not correctness. During self-refinement, a revision might be more confident but less correct than the original. The scorer accepts the confident-but-wrong revision, and accuracy drops. Self-consistency's majority voting is immune to this failure mode.

Despite these limitations, self-refinement has important advantages. It works on open-ended tasks where majority voting is not applicable. It can improve response quality (clarity, conciseness, formatting) even when the answer is already correct. And when combined with an external scorer (like an LLM judge rather than the model's own logprobs), it can be highly effective. DeepSeekMath-V2 used external-judge self-refinement to achieve gold-level mathematics competition performance.

The practical recommendation: use self-consistency for problems with extractable answers (math, code). Use self-refinement for open-ended tasks. Use both in combination (self-consistency to select an answer, then self-refinement to polish it) when quality and accuracy both matter.

Where self-refinement breaks

Here is where intellectual honesty matters. For the Qwen3 0.6B base model on math problems, self-refinement is less effective than self-consistency. The reason is straightforward: the critique step requires the model to identify errors in mathematical reasoning, which demands the same (or greater) capability as solving the problem in the first place. A model that cannot reliably solve a math problem is also unlikely to reliably identify errors in a solution.

This finding is model-size dependent. DeepSeekMath-V2, a much larger and more capable model, achieved excellent results with self-refinement because its critique capability was strong enough to meaningfully improve its own work. For a 600-million-parameter model, the critique is often the weakest link.

The practical lesson: scoring does not always improve results. Whether and which scorer to use depends on the model and must be determined empirically. There is no universal "best" inference-time technique. The right technique depends on the model's size, the task domain, and the available compute budget.

Why this chapter exists

If self-refinement is less effective than self-consistency for our model, why spend an entire chapter on it?

Two reasons.

First, the log-probability concept is essential for Chapter 6. In GRPO, the policy gradient loss is computed using sequence-level log-probabilities. Specifically, the loss is the negative product of the advantage (how much better this response was than average) and the sequence log-probability (how likely the model currently considers this response). If you do not understand log-probabilities, the GRPO loss function is just a formula. If you do understand them, the GRPO loss function is a precise expression of the intuition "make good responses more likely and bad responses less likely."

The connection is direct. The average log-probability scorer in this chapter computes:

(1/T) × Σ log P(token_t | context)

The GRPO loss in Chapter 6 uses:

Σ log P(token_t | context)

The only difference is the averaging. GRPO uses the sum (because the entire rollout receives a single reward), while the scorer uses the average (because we want to compare sequences of different lengths). The mathematical operation is the same.

Second, the self-refinement loop demonstrates a general pattern: generate, evaluate, improve. This pattern underlies reinforcement learning itself. In RL, the "generate" step is rollout sampling, the "evaluate" step is the reward function, and the "improve" step is the weight update. Self-refinement is RL in miniature, applied at inference time instead of training time, with the scoring function playing the role of the reward.

Understanding this pattern prepares you conceptually for the shift from inference to training. The techniques change, but the structure does not.

The self-refinement loop in practice follows a specific three-step cycle. Let me trace through it with our running example.

Step A: Initial generation. We prompt the model with the math problem plus CoT instruction. It generates an initial response with answer \boxed{18}. This is wrong (correct answer: 83). The logprob scorer gives it a score of -2.34.

Step B: Critique. We construct a critique prompt that includes the original question and the draft answer, asking the model to identify errors. The model generates a critique: "The draft incorrectly simplifies the equation. When we have (3x-9)/2 = x+37, we need to multiply both sides by 2 first, giving 3x-9 = 2x+74." This critique correctly identifies the error, though it also includes a false claim that "the question is incomplete."

Step C: Revision. We construct a refine prompt that includes the question, the draft, and the critique. The model generates a revised answer using the fix plan from the critique: "Multiply both sides by 2: 3x-9 = 2x+74. Subtract 2x: x-9 = 74. Add 9: x = 83. The answer is \boxed{83}." Score: -1.87. This is higher than -2.34, so the revision is accepted.

The cycle can be repeated. If we run iteration 2, the model critiques the already-correct answer, and the revision keeps \boxed{83} with a similar score. No further improvement occurs.

This example shows both the power and the fragility of self-refinement. The model correctly identified its own mistake in the critique step and produced a correct revision. But it also hallucinated a false claim about the question being incomplete, which did not affect the mathematical solution but illustrates that the critique is not guaranteed to be fully accurate.

The thread

We have exhausted what inference-time techniques can do for our base model. CoT prompting improves accuracy by encouraging intermediate steps. Temperature and top-p sampling introduce controlled randomness. Self-consistency leverages that randomness through majority voting. Self-refinement attempts iterative improvement within a single response. Together, these techniques can more than double the base model's accuracy, all without changing a single weight.

But the ceiling is real. The model cannot solve problems that require strategies it has never learned. It cannot apply techniques it has never seen. To expand the model's capabilities, we need to change the weights. We need training.

The next chapter marks the sharpest turn in the book. We leave the world of fixed models and clever prompts and enter the world of reinforcement learning. The evaluation pipeline from Chapter 3 becomes a reward function. The log-probabilities from this chapter become a loss function. The model will generate thousands of candidate solutions, receive binary feedback (correct or incorrect), and update its weights to favor the strategies that led to correct answers. In 50 training steps, accuracy will jump from 15.2% to 47.4%.

The model is about to learn to think.

The confidence trap

The log-probability scorer has a failure mode that reveals something deep about language models. Consider two responses to the problem "What is 2+2?":

Response A: "The answer is 4." Average logprob: -0.31 (very confident). Response B: "Well, 2+2 equals 4, but we should verify by considering that addition of natural numbers is commutative and associative, so 2+2 = 2+(1+1) = (2+1)+1 = 3+1 = 4." Average logprob: -2.87 (less confident).

Both answers are correct, but the scorer strongly prefers Response A because the model finds it easier to produce. The shorter, more common response pattern has higher probability per token. Response B, while more thorough, contains unusual phrasings that the model is less certain about.

Now consider a more dangerous case. Response C: "The answer is 5." If the model has been asked many similar questions and frequently seen "5" as an answer, it might assign relatively high probability to this incorrect response. The scorer would rank it above Response B despite being wrong.

This is the confidence trap: the model's confidence in a response is correlated with but not identical to correctness. A high logprob score means the model finds the answer easy to produce. It does not mean the answer is right. This distinction is critical. It means log-probability scoring is useful as a ranking heuristic (all else being equal, higher-confidence answers are more likely to be correct) but cannot be treated as a correctness signal.

In the self-refinement loop, this manifests as a specific failure pattern: the scorer occasionally selects a confident-but-wrong answer over a tentative-but-correct one. The model produces a correct answer in iteration 1, critiques it in iteration 2, produces a revised answer in iteration 3 that happens to be wrong but expressed with more common phrasing (and thus higher logprob), and the scorer accepts the revision. The self-refinement loop has made the answer worse.

This is why the MATH-500 results for self-refinement (~44% with the logprob scorer) are lower than CoT + self-consistency (52%). Self-consistency is more well-tested because it uses majority voting, which is immune to confidence bias: a wrong answer does not get more votes just because the model expressed it confidently.

The bridge to reinforcement learning

The log-probability concepts from this chapter are not just scoring tools. They are the mathematical foundation of the GRPO training algorithm in Chapter 6. But before we cross that bridge, we can see self-refinement work end-to-end on a concrete problem.

Worked example · self-refinement on a failing problem

This is a constructed design exercise, not a report of a named deployment. We can trace a complete self-refinement cycle on a problem where the model initially fails.

Problem: "Find the area of a triangle with sides 5, 12, and 13."

Iteration 0 (Initial generation): The model generates: "Using the formula A = (1/2)bh, we need the base and height. The base is 13 and... I'm not sure of the height. Let me use Heron's formula instead. s = (5+12+13)/2 = 15. A = sqrt(15 × 10 × 3 × 2) = sqrt(900) = 30. The answer is \boxed{30}." Logprob score: -3.14. Answer: 30. (This is correct! The model recovered from its initial uncertainty by switching strategies.)

Critique: "The solution correctly identifies the need for Heron's formula and applies it accurately. The computation of the semi-perimeter s = 15 is correct. The product under the square root 15 × 10 × 3 × 2 = 900 is correct. sqrt(900) = 30 is correct. However, the solution does not note that 5-12-13 is a right triangle (5² + 12² = 13²), which would allow the simpler formula A = (1/2)(5)(12) = 30."

Iteration 1 (Revision): "Since 5² + 12² = 25 + 144 = 169 = 13², this is a right triangle with legs 5 and 12. Therefore A = (1/2)(5)(12) = 30. The answer is \boxed{30}." Logprob score: -2.21. Answer: 30.

The revision is accepted (score improved from -3.14 to -2.21). Both answers are correct, but the revision is more elegant. This is self-refinement at its best: the critique identified an improvement (simpler approach), and the revision implemented it. The logprob scorer correctly ranked the cleaner solution higher.

System design exercise · scoring in a multi-model pipeline

This is a constructed design exercise, not a report of a named deployment. In a more mature operating design, scoring functions are not limited to the model's own log-probabilities. A common architecture uses three models in sequence:

Generator model (fast, small): Produces the initial response. This might be a distilled 0.6B model running on CPU for low latency.

Scorer model (medium, specialized): Evaluates the response quality. This might be a reward model trained on human preferences, or a specialized math verifier. Its score determines whether the response is good enough to send to the user or needs revision.

Refiner model (slow, large): If the scorer rejects the initial response, a larger model (7B or 70B) generates a higher-quality replacement. This model is expensive, so it is only invoked for queries that the generator failed on.

This three-tier architecture optimizes the compute-quality tradeoff: most queries are handled cheaply by the generator (with scoring as a quality gate), and only difficult queries escalate to the expensive refiner. The log-probability scoring concepts from this chapter form the foundation of the scorer tier.

Something subtle happened in this chapter that will not be fully apparent until Chapter 6. You learned to compute token log-probabilities, to score sequences, and to understand what it means for a model to assign high or low confidence to a response. These are not just scoring tools. They are the mathematical foundation of policy gradient reinforcement learning.

In GRPO (Chapter 6), the policy gradient loss is: -(advantages × logprobs).mean(). The logprobs in that equation are exactly the sequence-level log-probabilities you learned to compute here. The sequence_logprob function in Chapter 6 is nearly identical to the calc_next_token_logprobas function from this chapter, with one difference: it sums over answer tokens only (not the prompt) and uses @torch.no_grad() instead of @torch.inference_mode() to allow future gradient computation.

The cross-entropy loss used in distillation (Chapter 8) is the negative average log-probability: if the sum of token log-probabilities is -16.625 over 5 tokens, the cross-entropy is 16.625/5 = 3.325. Minimizing cross-entropy is equivalent to maximizing average log-probability.

So the same computation, token log-probabilities, appears three times in this book: as a scoring function (this chapter), as the training signal for RL (Chapter 6), and as the loss function for distillation (Chapter 8). Understanding it here means understanding it everywhere.

Public research systems have combined revision loops with external scoring. The practical result: self-refinement works best when combined with an external scoring function (like an LLM judge) rather than the model's own log-probability scores.

Why? Because the confidence trap described earlier means the model's self-assessment is systematically biased. A model cannot reliably detect its own mistakes if those mistakes are produced by the same statistical patterns it uses for evaluation. An external judge, using different parameters and potentially different training data, provides an independent check that catches errors the model would miss in self-evaluation.

In practice, self-refinement is often combined with other techniques rather than used alone. A common production pattern is: generate 3 responses with CoT (self-consistency), select the highest-scoring response using a log-probability or external scorer (Best-of-N), then refine it through one critique-and-revision cycle (self-refinement). This combines the diversity of self-consistency, the discriminative power of scoring, and the iterative improvement of refinement.

We can trace through the probability computation for a 6-token sequence: "The capital of Germany is Berlin." We feed the entire sequence into the model in one forward pass and get logits for every position. Then we ask: at each position, what probability did the model assign to the token that actually appears next?

Position 0 ("The"): The model predicts what comes after "The". The actual next token is "capital". The model assigns probability 0.000061 to "capital" given just "The". This is very low because "The" could be followed by almost anything.

Position 1 ("The capital"): The model predicts what comes after "capital". The actual next token is "of". Probability: 0.4629. Much higher, because "capital of" is a very common bigram.

Position 2 ("The capital of"): Predicts after "of". Actual: "Germany". Probability: 0.0166. Moderate; many countries could follow "capital of".

Position 3 ("The capital of Germany"): Predicts after "Germany". Actual: "is". Probability: 0.7422. Very high; "Germany is" is the natural continuation.

Position 4 ("The capital of Germany is"): Predicts after "is". Actual: "Berlin". Probability: 0.1690. The model considers several cities but "Berlin" is the top candidate.

The joint probability is the product of all per-token probabilities: 0.000061 × 0.4629 × 0.0166 × 0.7422 × 0.1690 = 5.94 × 10⁻⁸.

For "The capital of Germany is Bridge", the first four probabilities are identical (same context), but the fifth position gives "Bridge" a probability of 0.0000003 instead of 0.1690. Joint probability: 1.05 × 10⁻¹³.

The ratio is about 565,000:1 in favor of "Berlin" over "Bridge". The model is overwhelmingly more confident in "Berlin".

But both joint probabilities are astronomically small. For a 500-token response, you would multiply 500 numbers that are each less than 1. The result would have hundreds of zeros after the decimal point. Your computer cannot represent this number. This is the underflow problem, and it motivates the move to log-probabilities.

The log trick converts multiplication into addition: log(a × b × c) = log(a) + log(b) + log(c). Instead of multiplying 500 tiny probabilities (which underflows to zero), we take the logarithm of each and sum them. Since all probabilities are between 0 and 1, all log-probabilities are negative. A log-probability of 0 means probability 1 (certainty). A log-probability of -10 means probability e⁻¹⁰ ≈ 0.0000454 (very unlikely).

For "Berlin": log-probs are [-9.6875, -0.7695, -4.0938, -0.3008, -1.7812]. Sum: -16.6250. For "Bridge": log-probs are [-9.6875, -0.7695, -4.0938, -0.3008, -15.0000]. Sum: -29.8750.

The first four values are identical (same context). Only the fifth differs: -1.78 for "Berlin" vs -15.00 for "Bridge". The sum gap (-16.63 vs -29.88) is pronounced and numerically stable.

torch.log_softmax is preferred over torch.log(torch.softmax(...)) because it is a single fused operation that avoids computing the softmax separately (which can overflow for extreme logit values before the log brings them back to a manageable range).

Critical pairing rule: Always use softmax + prod, or log_softmax + sum. Mixing combinations (e.g., softmax with sum, or log_softmax with prod) is mathematically incorrect and will produce wrong results. This is a common bug in implementations.

In 1997, IBM's Deep Blue defeated Garry Kasparov in chess. But Deep Blue did not just play moves. It evaluated them. The scoring function, not the move generation, was the engine's secret weapon. Language models face the same problem: when you generate five responses, how do you pick the best one without knowing the right answer?

Before building the more sophisticated logprob scorer, we can understand why scoring is needed at all. In Chapter 4, self-consistency used majority voting to select the best answer. But majority voting has three limitations that scoring addresses:

Limitation 1: Ties. When five responses produce five different answers, majority voting returns no winner. A scorer can rank the five responses and pick the best one.

Limitation 2: Non-extractable answers. Majority voting requires short, comparable answers (numbers, letters, yes/no). For open-ended responses ("Explain why photosynthesis is important"), there is no short answer to vote on. A scorer can evaluate the full response.

Limitation 3: Quality beyond correctness. Two responses might reach the same correct answer, but one might be more concise, better formatted, or more clearly reasoned. A scorer can distinguish between these, selecting the higher-quality response.

The self-refinement loop (built later in this chapter) uses scoring to decide whether a revision improved the answer. Without a scorer, the loop would have no way to compare the original and revised responses, and every revision would be blindly accepted even if it made the answer worse.

Three families of scorers exist:

Rule-based scorers check surface features: formatting, length, structure. Fast and interpretable but shallow. They know nothing about mathematical correctness.

Model-based scorers use the LLM's own confidence (log-probabilities) to assess how "natural" a response feels. More principled but susceptible to the confidence trap: the model can be confidently wrong.

External scorers use a separate model or tool to evaluate quality. Most capable (an LLM judge, or the Chapter 3 verifier itself) but most expensive. In production, external scorers are often used for training data curation and the cheaper scorers for real-time decisions.

The scoring concept bridges Chapters 4 and 5: Chapter 4 used majority voting (a discrete selection mechanism), this chapter uses continuous scoring (a ranking mechanism). Both serve the same purpose: selecting the best response from a set of candidates. The choice depends on whether answers are extractable (use voting) or not (use scoring).

Scoring with rules

A simple rule-based scorer rewards answers with \boxed{} formatting and penalizes length:

import math

def heuristic_score(
    answer, prompt=None,
    brevity_bonus=500.0, boxed_bonus=2.0,
    extract_bonus=1.0, fulltext_bonus=0.0,
):
    score = 0.0
    cand = extract_final_candidate(answer, fallback="none")
    if cand:
        score += boxed_bonus
    else:
        cand = extract_final_candidate(answer, fallback="number_only")
        if cand:
            score += extract_bonus
        else:
            cand = extract_final_candidate(answer, fallback="number_then_full")
            if cand:
                score += fulltext_bonus
    score += 1.5 * math.exp(-len(answer) / brevity_bonus)
    return score

The brevity bonus decays exponentially: answers under ~200 characters get nearly full bonus, while answers over 1,000 characters get less than 0.2 points. The prompt argument is a placeholder for interface compatibility with the logprob scorer.

This section introduces the mathematical concepts that underpin both self-refinement scoring AND the GRPO training algorithm in Chapter 6. Understanding these concepts deeply here saves significant confusion later. Let me build the intuition in layers.

Layer 1: What the model actually outputs. At each token position, the model produces 151,936 logit scores. These logits are raw, unnormalized numbers that can be positive or negative, large or small. They have no direct probabilistic interpretation. The token with the largest logit is the model's "first choice," but the magnitude of the logit does not directly tell you how confident the model is.

Layer 2: From logits to probabilities. The softmax function converts logits to probabilities: P(token_i) = exp(logit_i) / Σ_j exp(logit_j). This ensures all probabilities are positive and sum to 1. After softmax, we can say "the model assigns 74% probability to 'Berlin' and 17% to a quiz placeholder." These are proper probabilities that can be multiplied and compared.

Layer 3: From probabilities to sequence probability. The probability of an entire sequence is the product of per-token probabilities: P(sequence) = P(token_1) × P(token_2|token_1) × P(token_3|token_1,token_2) × ... Each factor is the probability of the actual token at that position, conditioned on all preceding tokens.

Layer 4: The underflow problem. Multiplying many numbers less than 1 quickly produces numbers too small for computers to represent. For a 500-token response where each token has average probability 0.1, the joint probability is 0.1^500 = 10^(-500), a number with 500 zeros after the decimal point. This is far below the smallest representable floating-point number (~10^(-308) for float64). The computer rounds it to exactly 0.0, losing all information.

Layer 5: The log solution. Taking logarithms converts products to sums: log(a × b) = log(a) + log(b). Instead of multiplying 500 probabilities (underflow), we sum 500 log-probabilities (manageable negative numbers). The log of 0.1 is -2.3, so the log of 0.1^500 is -1,150, a perfectly representable number.

Layer 6: In practice. torch.log_softmax(logits, dim=-1) computes log-probabilities directly from logits in a single, numerically stable operation. This avoids the intermediate step of computing probabilities (which can overflow for large logits) and then taking their log (which is numerically unstable near zero).

These six layers form a conceptual stack. In Chapter 5 (this chapter), we use Layers 1-6 to build a scoring function. In Chapter 6, we use the same stack inside the GRPO loss computation. In Chapter 8, we use the negative average of Layer 5 as the cross-entropy loss for distillation. Same mathematics, three applications.

Token probabilities and log-probabilities

Every time a model generates a token, it produces a probability distribution over the entire vocabulary. We can retrospectively score an existing response by checking how much probability the model assigned to each token:

@torch.inference_mode()
def calc_next_token_probas(model, tokenizer, prompt, device, show=True):
    token_ids = torch.tensor(tokenizer.encode(prompt), device=device)
    logits = model(token_ids.unsqueeze(0)).squeeze(0)
    all_probas = torch.softmax(logits, dim=-1)
    t_idx = torch.arange(0, token_ids.shape[0] - 1, device=device)
    next_ids = token_ids[1:]
    next_token_probas = all_probas[t_idx, next_ids]
    prod_next_token_probas = torch.prod(next_token_probas)
    if show:
        print("Next-token probabilities:", next_token_probas)
        print("Joint probability:", prod_next_token_probas)
    else:
        return next_token_probas, prod_next_token_probas

"Berlin" gives joint probability 5.94×10⁻⁸. "Bridge" gives 1.05×10⁻¹³. But both are astronomically small because multiplying probabilities less than 1 quickly underflows.

The fix: log(a × b × c) = log(a) + log(b) + log(c). Sum log-probabilities instead of multiplying probabilities:

@torch.inference_mode()
def calc_next_token_logprobas(model, tokenizer, prompt, device, show=True):
    token_ids = torch.tensor(tokenizer.encode(prompt), device=device)
    logits = model(token_ids.unsqueeze(0)).squeeze(0)
    all_logprobas = torch.log_softmax(logits, dim=-1)
    t_idx = torch.arange(0, token_ids.shape[0] - 1, device=device)
    next_ids = token_ids[1:]
    next_token_logprobas = all_logprobas[t_idx, next_ids]
    sum_next_token_logprobas = torch.sum(next_token_logprobas)
    if show:
        print("Next-token log-probabilities:", next_token_logprobas)

## The training data: where the problems come from

A critical but often overlooked aspect of GRPO training is the training data itself. The book uses 12,000 math problems from the MATH dataset, specifically the problems that are NOT in the 500-problem evaluation set (MATH-500). This separation is essential: if the model trained on the same problems it is evaluated on, the accuracy numbers would be meaningless (the model could simply memorize answers).

Each training example contains:
- `problem`: The math problem as a string (e.g., "Half the value of 3x-9 is x+37. What is x?")
- `answer`: The ground truth answer (e.g., "83")
- `solution`: A worked solution (NOT used during training, to avoid constraining the model's solution strategy)
- `level`: Difficulty from 1 (easiest) to 5 (hardest)
- `type`: Subject area (algebra, geometry, number theory, etc.)

The `solution` field is deliberately ignored. If we used it to constrain the model's output (e.g., penalizing solutions that differ from the reference), we would limit exploration. The model might discover more efficient or creative solution paths that differ from the reference solution. GRPO only cares about the final answer's correctness, giving the model freedom to develop its own reasoning strategies.

The diversity of difficulty levels matters for training dynamics. Level 1 problems (simple arithmetic) provide frequent positive rewards early in training, giving the model a foundation. Level 5 problems (competition-level) initially produce all-zero rewards, providing no learning signal. As training progresses, the model begins solving Level 3-4 problems, and eventually some Level 5 problems. This natural curriculum effect means that the training problems self-organize from easy to hard based on the model's evolving capability.

        print("Joint log-probability:", sum_next_token_logprobas)
    else:
        return next_token_logprobas, sum_next_token_logprobas

"Berlin" scores -16.6250. "Bridge" scores -29.8750. Higher (less negative) is better. torch.log_softmax is a single fused operation that is more numerically stable than torch.log(torch.softmax(...)).

Critical pairing rule: Always use softmax + prod, or log_softmax + sum. Mixing them (e.g., softmax with sum or log_softmax with prod) is mathematically incorrect.

Scoring answers, not prompts

@torch.inference_mode()
def avg_logprob_answer(model, tokenizer, prompt, answer, device="cpu"):
    prompt_ids = tokenizer.encode(prompt)
    answer_ids = tokenizer.encode(answer)
    full_ids = torch.tensor(prompt_ids + answer_ids, device=device)
    logits = model(full_ids.unsqueeze(0)).squeeze(0)
    logprobs = torch.log_softmax(logits, dim=-1)
    start = len(prompt_ids) - 1
    end = full_ids.shape[0] - 1
    t_idx = torch.arange(start, end, device=device)
    next_tokens = full_ids[start + 1 : end + 1]
    next_token_logps = logprobs[t_idx, next_tokens]
    return torch.mean(next_token_logps).item()

We average instead of summing so answers of different lengths are comparable. "Berlin" scores -0.204 (high confidence). "Bridge" scores -3.891 (low confidence). But high logprob does not mean correct. It means the model finds the answer easy to produce. A confidently wrong answer scores high.

Self-refinement lets the model iteratively critique and improve a single answer:

def make_critique_prompt(raw_prompt, draft):
    return (
        "You are a meticulous reviewer. Identify logical errors, missing "
        "steps, or arithmetic mistakes. If the answer seems correct, "
        "say so briefly. Then propose a concise plan to fix issues.\n\n"
        f"Question:\n{raw_prompt}\n\nDraft answer:\n{draft}\n\n"
        "Write a short critique and bullet-point fix plan "
        "(under ~120 words).\nCritique:"
    )

def make_refine_prompt(raw_prompt, draft, critique):
    return (
        "Revise the answer using the critique. Keep it concise and "
        "end with a final boxed result: \\boxed{ANSWER}\n\n"
        f"Question:\n{raw_prompt}\n\nPrevious answer:\n{draft}\n\n"
        f"Critique:\n{critique}\n\nRevised answer:"
    )

The full loop with pluggable scoring:

def self_refinement_loop(
    model, tokenizer, raw_prompt, device,
    iterations=2, max_response_tokens=2048,
    max_critique_tokens=256, score_fn=None,
    prompt_renderer=render_prompt, prompt_suffix="",
    verbose=False, temperature=0.7, top_p=0.9,
):
    steps = []
    prompt = prompt_renderer(raw_prompt) + prompt_suffix
    current_full = generate_text_stream_concat_flex(
        model=model, tokenizer=tokenizer, prompt=prompt, device=device,
        max_new_tokens=max_response_tokens, verbose=False,
        generate_func=generate_text_top_p_stream_cache,
        temperature=temperature, top_p=top_p,
    )
    current_extracted = extract_final_candidate(current_full, fallback="number_then_full")
    current_score = score_fn(answer=current_full, prompt=prompt) if score_fn else 0.0

    for it in range(iterations):
        draft_before_full = current_full
        score_before = current_score
        
        critique_prompt = make_critique_prompt(raw_prompt, draft_before_full)
        critique_full = generate_text_stream_concat_flex(
            model=model, tokenizer=tokenizer, prompt=critique_prompt, device=device,
            max_new_tokens=max_critique_tokens, verbose=False,
            generate_func=generate_text_top_p_stream_cache,
            temperature=temperature, top_p=top_p,
        )

        refine_prompt = make_refine_prompt(raw_prompt, draft_before_full, critique_full)
        revised_full = generate_text_stream_concat_flex(
            model=model, tokenizer=tokenizer, prompt=refine_prompt, device=device,
            max_new_tokens=max_response_tokens, verbose=False,
            generate_func=generate_text_top_p_stream_cache,
            temperature=temperature, top_p=top_p,
        )
        revised_extracted = extract_final_candidate(revised_full, fallback="number_then_full")
        revised_score = score_fn(answer=revised_full, prompt=prompt) if score_fn else 0.0

        if revised_score >= current_score:
            current_full = revised_full
            current_extracted = revised_extracted
            current_score = revised_score

    return {"final_full": current_full, "final_extracted": current_extracted, "steps": steps}

On MATH-500, self-refinement with the logprob scorer achieves approximately 44.0% accuracy.

These log-probability concepts are not just useful for scoring. They are the mathematical foundation of the GRPO training objective in the next chapter.

From human judgment to verifiable truth

In Chapter 1, we briefly mentioned reinforcement learning from human feedback (RLHF), the technique that transformed base models into the helpful assistants we use daily. RLHF works by training a separate neural network (the reward model) to predict which responses humans will prefer, then using that reward model to guide the language model's training.

RLHF has a fundamental vulnerability. The reward model is a neural network, and neural networks can be gamed. The language model can learn to produce outputs that score high on the reward model without actually being better. It might learn to be verbose (longer responses often score higher), to use confident-sounding language (even when wrong), or to exploit quirks in the reward model's training data. This phenomenon is called reward hacking, and it is one of the most persistent challenges in RLHF.

For reasoning tasks like math and code, there is a cleaner alternative. Instead of training a neural network to predict human preferences, you can write a simple program that checks whether the answer is correct. You do not need a reward model. You need a verifier.

This approach is called Reinforcement Learning with Verifiable Rewards, or RLVR. The verifier is the math evaluation pipeline from Chapter 3. Correct answer? Reward = 1.0. Wrong answer? Reward = 0.0. No ambiguity. No gaming. No expensive human annotation.

The tradeoff: RLVR only works for tasks with objectively verifiable answers. You can verify math and code. You cannot verify "write a persuasive essay" or "summarise this article helpfully." For subjective tasks, you still need RLHF. But for reasoning tasks, RLVR is simpler, cheaper, and harder to game.

A concrete walkthrough · one complete GRPO step

Let me trace through every computation in a single GRPO training step with real numbers. This walkthrough is the key to understanding the entire algorithm.

The training example: "Half the value of 3x-9 is x+37. What is the value of x?" Ground truth: 83.

Stage 1: Generate 4 rollouts. The model generates four candidate solutions using temperature=0.9 and top-p=0.9:

Rollout 1 (198 tokens): Sets up the equation (3x-9)/2 = x+37. Multiplies both sides by 2: 3x-9 = 2x+74. Subtracts 2x: x-9 = 74. Adds 9: x = 83. Boxes: \boxed{83}.

Rollout 2 (245 tokens): Same approach but with more verbose explanation, arriving at \boxed{83}.

Rollout 3 (156 tokens): Misreads "half the value" as "twice the value." Sets up 2(3x-9) = x+37. Gets 6x-18 = x+37. Gets 5x = 55. Gets x = 11. Boxes: \boxed{11}.

Rollout 4 (312 tokens): Correct approach but makes an arithmetic error: 74+9 = 82 instead of 83. Boxes: \boxed{82}.

Stage 2: Compute rewards. For each rollout, extract the boxed answer and check against ground truth "83":

Rollout 1: extract("...\\boxed{83}...") → "83". grade("83", "83") → True. Reward: 1.0
Rollout 2: extract("...\\boxed{83}...") → "83". grade("83", "83") → True. Reward: 1.0
Rollout 3: extract("...\\boxed{11}...") → "11". grade("11", "83") → False. Reward: 0.0
Rollout 4: extract("...\\boxed{82}...") → "82". grade("82", "83") → False. Reward: 0.0

Rewards: [1.0, 1.0, 0.0, 0.0]. Two correct, two wrong. This is the ideal scenario: maximum contrast within the group.

Stage 3: Compute advantages. Z-score normalisation:

mean = (1.0 + 1.0 + 0.0 + 0.0) / 4 = 0.5
std = sqrt(((0.5)² + (0.5)² + (-0.5)² + (-0.5)²) / 4) = 0.5
advantages = [(1.0-0.5)/0.5, (1.0-0.5)/0.5, (0.0-0.5)/0.5, (0.0-0.5)/0.5]
           = [+1.0, +1.0, -1.0, -1.0]

Positive advantages for correct rollouts (+1.0): the model should produce more sequences like these. Negative advantages for incorrect rollouts (-1.0): the model should produce fewer sequences like these. The magnitudes are equal: correct and incorrect rollouts receive equal-strength learning signal.

Stage 4: Compute sequence log-probabilities. For each rollout, compute the sum of token log-probabilities over the answer tokens (excluding the prompt). The model processes each full sequence (prompt + answer) through a forward pass and looks up the log-probability of each actual answer token:

Rollout 1 (198 answer tokens): seq_logprob = -156.34
Rollout 2 (245 answer tokens): seq_logprob = -203.17
Rollout 3 (156 answer tokens): seq_logprob = -134.89
Rollout 4 (312 answer tokens): seq_logprob = -267.42

Longer rollouts have more negative log-probabilities (more tokens = more terms in the sum, each negative). Note that Rollout 3 (the wrong answer) has a less negative logprob than Rollout 1 (a correct answer). This is fine! The logprobs measure how likely the model was to generate each sequence, not whether the sequence is correct. The advantages handle the correctness signal.

Stage 5: Compute policy gradient loss.

pg_loss = -(advantages × logprobs).mean()
        = -(+1.0 × (-156.34) + 1.0 × (-203.17) + (-1.0) × (-134.89) + (-1.0) × (-267.42)) / 4
        = -(-156.34 - 203.17 + 134.89 + 267.42) / 4
        = -(42.80) / 4
        = -10.70

Stage 6: Backpropagate and update. loss.backward() computes the gradient of -10.70 with respect to all 621 million parameters. The optimizer takes one step in the direction that would:

• Make Rollout 1's tokens more probable (correct answer, shorter, efficient)
• Make Rollout 2's tokens more probable (correct answer)
• Make Rollout 3's tokens less probable (wrong setup: "twice" instead of "half")
• Make Rollout 4's tokens less probable (arithmetic error)

After this one step, the model is slightly better at this type of algebra problem. Repeat 49 more times on different problems, and accuracy jumps from 15.2% to 47.4%.

Why the improvement is so dramatic

The 32.2 percentage point improvement (15.2% → 47.4%) in just 50 training steps deserves explanation. In supervised learning, 50 gradient updates barely move the needle on most tasks. What makes GRPO so effective in so few steps?

Three factors converge. First, the model already knows how to reason. During pre-training on trillions of tokens, Qwen3 0.6B internalised millions of step-by-step math solutions from textbooks, homework help forums, and educational websites. The knowledge is already in the weights. GRPO does not teach new knowledge; it adjusts the model's output distribution so that the correct patterns are selected more reliably during generation.

Second, the reward signal is extremely clear. Binary rewards (1.0 for correct, 0.0 for wrong) provide an unambiguous learning signal. There is no gray area, no subjectivity, no noise from disagreeing human evaluators. When the model gets the right answer, the signal says "do more of this." When it gets the wrong answer, the signal says "do less of this." This clarity allows the model to learn quickly from each informative step.

Third, the first improvements are the easiest. Going from 15.2% to 47.4% means the model learns to solve problems it was "almost" able to solve before. These are problems where the correct reasoning pattern was in the top-5 most likely generation paths but not the top-1. GRPO's advantage signal is strongest for these borderline cases, where some rollouts succeed and others fail. The remaining 52.6% of unsolved problems require reasoning patterns that are deeper in the model's distribution, and extracting them takes many more steps with diminishing returns.

This explains the "diminishing returns" observation from Chapter 7: the easy gains come fast (steps 1-50), the medium gains come slower (steps 50-200), and further training can actually reverse gains (steps 200+) without stabilization techniques. The production implication: 50-100 steps is often sufficient for a first training run. Invest additional compute in stabilization (clipping, monitoring) rather than more steps.

The chef who learns from four dishes

Before diving into the math, we can build an analogy that will carry through the entire chapter.

Consider a chef who is trying to master a new recipe. The chef has no recipe book; they have to figure it out by experimentation. Here is their process:

1. The chef prepares four versions of the dish, each with slightly different techniques (a little more salt here, a different cooking time there).
2. A food critic tastes all four and gives each a binary score: "good" or "bad."
3. The chef compares the four scores. Two dishes were good, two were bad.
4. The chef examines what they did differently for the good dishes versus the bad ones. More salt? Less heat? Whatever distinguished the good versions from the bad ones, the chef reinforces those choices.
5. The chef repeats with a new dish.

This is GRPO in a nutshell. The chef is the language model. The four versions are rollouts (candidate responses generated by the model). The food critic is the verifier. The comparison step is advantage computation. And the adjustment is the policy gradient update.

The useful distinction is that the chef does not need an external standard of excellence. They do not need a Michelin guide or a culinary school textbook. They only need to compare their own attempts against each other and ask: "Which of these was better, and what did I do differently?" This is why GRPO is "group relative": the learning signal comes from relative comparisons within a group, not from an absolute standard.

This eliminates the need for a value model (also called a critic in RL terminology), which is a separate neural network that PPO requires to estimate how good a given state is. The value model is expensive to train and maintain. GRPO sidesteps it entirely by computing advantages relative to the group mean. Fewer models to train, less compute, less complexity.

The six-step GRPO procedure

We can trace one training step in detail. We have a math problem from the training set and our base model.

Step 1: Generate rollouts. For the given problem, the model generates 4 candidate responses using temperature and top-p sampling. Because of the randomness, each response is different. Some may be correct, some may be wrong.

Step 2: Compute rewards. Each response is graded by the verifier from Chapter 3. The verifier extracts the answer, normalizes it, and checks it against the ground truth. The rewards are binary: [1.0, 0.0, 1.0, 0.0] (two correct, two incorrect).

Step 3: Compute advantages. The advantage of each rollout is its reward minus the group mean, divided by the group standard deviation. This is z-score normalisation. With rewards [1, 0, 1, 0], the mean is 0.5, the standard deviation is 0.5, and the advantages are [1.0, -1.0, 1.0, -1.0]. Positive advantages mean "better than average." Negative advantages mean "worse than average."

Notice what happens when all four rollouts have the same reward (all correct or all incorrect). The standard deviation is zero, and the advantages are all zero. The model learns nothing from this problem. This is desirable: if every attempt succeeds, there is nothing to improve; if every attempt fails, there is no successful strategy to reinforce.

Step 4: Compute sequence log-probabilities. For each rollout, compute the sum of log-probabilities of all tokens in the response, using the current model weights. This is the sum we built in Chapter 5:

log P(response) = Σ log P(token_t | tokens_1, ..., tokens_{t-1})

Why the sum and not the average? Because the entire rollout receives a single reward. A response is either correct or incorrect as a whole, not on a per-token basis. Using the sum means that longer correct responses contribute more to the gradient, which is appropriate because longer responses contain more tokens whose generation strategy should be reinforced.

Step 5: Compute the policy gradient loss. The loss is:

loss = -mean(advantages × sequence_log_probabilities)

Read this equation out loud: "Take the advantage of each rollout (how much better it was than average) and multiply it by the log-probability of that rollout under the current model. Average these products across all rollouts. Negate the result because PyTorch minimizers minimize, and we want to maximize expected reward."

The gradient of this loss, when backpropagated, has a beautiful interpretation:

• For rollouts with positive advantages (better than average), the gradient points in a direction that increases their log-probability. The model becomes more likely to produce similar responses in the future.
• For rollouts with negative advantages (worse than average), the gradient points in a direction that decreases their log-probability. The model becomes less likely to produce similar responses.

The model learns from its own comparisons. No external teacher. No reward model. Just "this attempt was better than that one, so do more of this and less of that."

Step 6: Update weights. Standard PyTorch: backpropagate the loss, clip the gradients (max_norm = 1.0 to prevent exploding gradients), and apply the optimizer step. Then move to the next training problem and repeat.

The implementation

The complete GRPO loss function fits in about 50 lines of Python. The conceptual core:

def compute_grpo_loss(model, tokenizer, prompt, ground_truth, device,
                       num_rollouts=4, max_new_tokens=512,
                       temperature=0.9, top_p=0.9):

    # Step 1: Generate rollouts
    rollouts = []
    for _ in range(num_rollouts):
        response = generate(model, tokenizer, prompt, device,
                          temperature=temperature, top_p=top_p,
                          max_new_tokens=max_new_tokens)
        rollouts.append(response)

    # Step 2: Compute rewards
    rewards = torch.tensor([
        reward_rlvr(response, ground_truth)  # 1.0 or 0.0
        for response in rollouts
    ])

    # Step 3: Compute advantages (z-score normalization)
    if rewards.std() == 0:
        return torch.tensor(0.0)  # Nothing to learn
    advantages = (rewards - rewards.mean()) / rewards.std()

    # Step 4: Compute sequence log-probabilities
    log_probs = torch.stack([
        compute_sequence_logprob(model, tokenizer, prompt + response, device)
        for response in rollouts
    ])

    # Step 5: Policy gradient loss
    loss = -(advantages.detach() * log_probs).mean()

    return loss

The .detach() on advantages is critical. Advantages are computed from rewards, which depend on the model's outputs, but we do not want to backpropagate through the reward computation. The advantages are a fixed signal; only the log-probabilities should generate gradients.

The training loop wraps this in standard PyTorch:

optimizer = torch.optim.AdamW(model.parameters(), lr=5e-6)

for step in range(50):
    problem = math_train[step % len(math_train)]
    prompt = render_prompt(problem["question"])

    loss = compute_grpo_loss(
        model, tokenizer, prompt, problem["answer"], device,
        num_rollouts=4, max_new_tokens=512
    )

    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
    optimizer.step()
    optimizer.zero_grad()

That is the entire training loop. Fifty lines of code that transform a base model into a reasoning model.

The results · fifty steps to reasoning

Running this loop for 50 steps, each step taking a few minutes on a GPU (longer on CPU), produces the following trajectory:

The early steps show rewards near zero. The model cannot solve math problems, so almost all rollouts receive a reward of 0.0. The advantages are all zero. The model learns nothing.

Then, around step 3 to 5, something shifts. One or two rollouts per step start getting the right answer. The advantage computation now has signal: the correct rollout has a positive advantage, and the incorrect ones have negative advantages. The model begins adjusting its weights to favor the strategy that led to the correct answer.

By step 20, the average reward per step has climbed to 0.5 or higher. The model is getting half its rollouts correct. The responses are getting longer (from ~5 tokens to ~100+ tokens) as the model learns to generate intermediate reasoning steps.

By step 50, the average reward is often 0.75 or higher. The model generates detailed reasoning traces with steps, checks, and \boxed{} answers. Evaluating on MATH-500:

• Before training: 15.2% accuracy
• After 50 steps: 47.4% accuracy

This is within 1 percentage point of the official Qwen3 0.6B reasoning model (48.2%), which was trained by Alibaba's Qwen team using release-ready infrastructure. The gap, from 15.2% to 47.4%, represents a 3x improvement achieved by a training loop that fits on a single page.

What GRPO discovers · emergent behaviours

One of the most remarkable findings from DeepSeek-R1 was that certain reasoning behaviours emerged spontaneously during GRPO training, without being explicitly programmed or demonstrated. These include:

Self-correction. The model learns to write phrases like "Wait, let me check that" or "Actually, I made an error above" and then revises its computation. Nobody taught the model to self-correct. It discovered that generating self-correction tokens correlated with higher reward (because catching and fixing errors leads to correct final answers).

Problem decomposition. The model learns to break complex problems into subproblems: "First, I need to find the value of a+b. Then I can use that to compute a²+b²." This decomposition strategy is not in the reward signal (which only checks the final answer). The model discovered that decomposition is an intermediate strategy that increases the probability of a correct final answer.

Uncertainty expression. The model occasionally generates tokens expressing uncertainty: "I think the answer is 83, but let me verify by substituting back." This verification step costs extra tokens (and thus extra compute) but increases accuracy. The model learned that spending tokens on verification is a worthwhile investment.

These emergent behaviours are exciting but also fragile. They tend to appear after 20-50 training steps and can disappear during longer training runs if the model collapses or overfits. This is one reason why checkpoint selection (always save and evaluate) is so important: the version of the model with the richest emergent behaviours might be at step 35, not step 500.

These emergent behaviours suggest that the model is not just memorising solutions but developing general reasoning strategies. However, the strategies are fragile. They tend to appear after 20-50 steps and can disappear during longer training runs if the model collapses or overfits. This is one of the strongest arguments for the practical sequence of Chapter 8: distill the emerged behaviours into a stable training dataset, then use GRPO to refine them further from a stable starting point.

A thought experiment: what if you ran GRPO for 10,000 steps with perfect stabilization? Would the model eventually achieve 100% accuracy on MATH-500? Almost certainly not. The model's knowledge is bounded by what it learned during pre-training. GRPO cannot teach the model mathematical concepts it has never seen. It can only make the model more reliable at applying concepts it already knows. This ceiling effect is why distillation from a larger teacher (who has seen more and knows more) is a necessary complement to RL training.

System design exercise · GRPO for a coding assistant

This is a constructed design exercise, not a report of a named deployment. The math verification pipeline generalizes directly to code. Instead of grade_answer(extracted, ground_truth), you would use:

def reward_code(generated_code, test_cases):
    """Execute code in sandbox and check test cases."""
    try:
        result = sandbox_execute(generated_code, timeout=5)
        passed = sum(1 for tc in test_cases if tc.check(result))
        return passed / len(test_cases)  # 0.0 to 1.0
    except (TimeoutError, SyntaxError, RuntimeError):
        return 0.0

The reward is no longer binary (0 or 1) but continuous (0.0 to 1.0 based on fraction of test cases passed). This provides richer signal: a solution that passes 4 out of 5 tests is closer to correct than one that passes 0 out of 5, and the reward reflects this.

The GRPO training loop is identical. Generate rollouts (code solutions), compute rewards (run test cases), compute advantages (z-score), compute logprobs, compute loss, backpropagate. The only change is the reward function.

In production, teams typically combine math and code rewards during training. DeepSeek-R1 trained on a mixture of math problems, coding challenges, and logical puzzles. The diversity of training tasks prevents the model from overfitting to a single domain's reasoning patterns.

What makes this result intellectually striking is not just the accuracy improvement. It is how the model's outputs change.

Before GRPO training, the model's responses to math problems are short and wrong. It guesses a number, wraps it in \boxed{}, and moves on. There is no intermediate work, no decomposition of the problem, no checking.

After GRPO training, the model spontaneously generates multi-step reasoning traces. It sets up equations, performs algebraic manipulations, and arrives at answers through a visible chain of logic. Nobody programmed this behaviour. Nobody provided examples of reasoning traces. The model discovered, through pure trial and error, that generating intermediate steps leads to correct answers, and correct answers lead to rewards.

This is the same emergence that DeepSeek observed in R1-Zero at a much larger scale. The reinforcement learning signal, a simple binary "right or wrong," is sufficient for the model to discover that reasoning is a useful strategy. The model does not know why reasoning works. It just knows that responses containing intermediate steps tend to score higher than responses that jump directly to an answer.

The basketball coach analogy

GRPO is like a basketball coach who cannot demonstrate shots. The coach (the training algorithm) watches the team (the model) take shots (generate rollouts) in practice. After each round, the coach checks the scoreboard (the verifier). Shots that went in (correct answers) are reinforced: the players are told "do more of whatever you did there." Shots that missed (wrong answers) are suppressed: "do less of that."

The coach never says "hold the ball at this angle" or "follow through with your wrist." That would be distillation (Chapter 8), where a teacher demonstrates the correct technique. GRPO is pure trial and error. The coach provides only outcome feedback, never process feedback.

The "group relative" part is critical. The coach does not compare each shot against an absolute standard. The coach compares each shot against the other shots in the same practice session. If a player makes 3 out of 4 shots, those 3 successful shots get reinforced and the 1 miss gets suppressed. If the same player makes 0 out of 4, nothing happens because there is no contrast to learn from.

This is why GRPO needs at least 2 rollouts per training example (the book uses 4). With only 1 rollout, there is nothing to compare against. The advantage would be zero. The gradient would be zero. No learning.

What can go wrong

Fifty steps is a notably short training run. What happens if you train for 500 steps? Or 5,000?

In the basic GRPO formulation implemented here, longer training does not necessarily improve performance. After about 50 to 100 steps, accuracy often plateaus or even declines. The model can become unstable: it might start generating very long responses that consume the entire token budget without reaching an answer, or it might collapse to a narrow set of response patterns that work for the training problems but fail to generalize.

This instability is not a bug in the implementation. It is a fundamental challenge with the basic GRPO formulation, which lacks safeguards against policy drift (the model's behaviour diverging too far from its starting point). Chapter 7 addresses this with three practical fixes: clipped policy ratios (borrowed from PPO), KL divergence regularization (which penalizes the model for drifting too far from the original), and format rewards (which reward structural properties of the response, not just answer correctness).

For now, the takeaway is: the basic GRPO formulation is capable but fragile. Fifty well-chosen steps can be more valuable than five thousand poorly managed ones. And the best checkpoint in this run was rarely the last checkpoint.

Let me trace through the complete GRPO loss computation for a single training step with concrete numbers.

Training example: "What is 7 × 8?" Ground truth: "56."

Stage 1: Generate 4 rollouts. Rollout 1: "7 × 8 = 56. \boxed{56}" (12 tokens) Rollout 2: "Let me compute: 7 times 8 equals... \boxed{56}" (18 tokens) Rollout 3: "The answer is \boxed{54}" (9 tokens) Rollout 4: "7 × 8 = 42 + 14 = 56. \boxed{56}" (20 tokens)

Stage 2: Compute rewards. Rollout 1: extract "56", grade against "56" → reward 1.0 Rollout 2: extract "56", grade → reward 1.0 Rollout 3: extract "54", grade → reward 0.0 (wrong) Rollout 4: extract "56", grade → reward 1.0

Stage 3: Compute advantages. Rewards: [1.0, 1.0, 0.0, 1.0]. Mean: 0.75. Std: 0.5. Advantages: [(1.0-0.75)/0.5, (1.0-0.75)/0.5, (0.0-0.75)/0.5, (1.0-0.75)/0.5] = [+0.5, +0.5, -1.5, +0.5].

Stage 4: Compute sequence logprobs. Rollout 1 (12 tokens): sum of token logprobs = -7.92 Rollout 2 (18 tokens): sum = -20.15 Rollout 3 (9 tokens): sum = -16.61 Rollout 4 (20 tokens): sum = -23.37

Stage 5: Policy gradient loss. pg_loss = -mean([0.5×(-7.92), 0.5×(-20.15), (-1.5)×(-16.61), 0.5×(-23.37)]) = -mean([-3.96, -10.08, 24.92, -11.69]) = -(-0.20) = 0.20

Stage 6: Backpropagate and update. loss.backward() computes gradients of 0.20 with respect to all 621 million parameters. The optimizer nudges each parameter in the direction that would make the correct rollouts (1, 2, 4) more likely and the incorrect rollout (3) less likely.

The gradient for the correct rollouts is positive (increase their logprobs). The gradient for the incorrect rollout is negative and larger in magnitude (the advantage is -1.5 vs +0.5), so the model learns more aggressively from its mistakes than from its successes. This asymmetry is a property of the z-score normalisation.

The thread

We have crossed the Rubicon. For the first time in the book, we have changed the model's weights. The base model that started at 15.2% accuracy now scores 47.4% on MATH-500. It has learned to reason, not from examples of reasoning, but from the simple binary signal of "right" and "wrong."

The algorithm behind this transformation, GRPO, is conceptually elegant: generate multiple attempts, compare them, reinforce the good ones, suppress the bad ones. But its basic form is unstable over long training runs. The next chapter digs into the engineering details of making GRPO work reliably: which metrics to monitor, how to detect instability early, and which algorithmic modifications prevent the model from drifting into failure modes. If Chapter 6 was about getting GRPO working, Chapter 7 is about getting it working well.

It is worth pausing to state clearly what GRPO is doing mathematically, now that we have all the pieces. The model has a policy: a function that maps prompts to probability distributions over token sequences. The policy is parameterized by the model's 621 million weights. GRPO adjusts those weights to make the policy more likely to produce sequences that receive high rewards.

The mechanism is the policy gradient: the gradient of the expected reward with respect to the model's weights. The policy gradient theorem says this gradient is proportional to the product of the advantage (how much better this rollout was than average) and the gradient of the log-probability of the rollout with respect to the weights. By moving the weights in this direction, we increase the probability of high-advantage rollouts and decrease the probability of low-advantage rollouts.

The "group relative" part of GRPO refers to how advantages are computed: relative to the group mean, not relative to an absolute baseline. This eliminates the need for a separate value network (used in PPO) that estimates the expected reward. The tradeoff is noisier estimates (group statistics from 4 rollouts are less precise than a trained value function), but materially lower overhead.

The "policy optimisation" part refers to the iterative process: generate rollouts under the current policy, compute the policy gradient, update the weights, and repeat. Over many iterations, the policy gradually shifts toward producing correct, well-formatted mathematical solutions.

The distinction between RLVR and RLHF deserves a concrete example. In RLHF, you might show two human evaluators the same pair of model responses to "Explain photosynthesis." Evaluator A prefers Response 1 because it is more concise. Evaluator B prefers Response 2 because it is more thorough. They disagree. Now you need more evaluators, or a reconciliation process, or a separate reward model trained on their preferences. The whole pipeline is expensive, slow, and subjective.

In RLVR, you ask the model "What is 7 × 8?" The model responds "56." Is this correct? Yes. Reward: 1.0. You ask again. It responds "54." Correct? No. Reward: 0.0. There is no ambiguity, no subjectivity, no need for human evaluators. A simple Python function handles it:

def reward_rlvr(answer_text, ground_truth):
    extracted = extract_final_candidate(
        answer_text, fallback=None
    )
    if not extracted:
        return 0.0
    correct = grade_answer(extracted, ground_truth)
    return float(correct)

By setting fallback=None, we enforce that the model must use \boxed{} format. An answer of "The final answer is 83" (correct but not boxed) gets reward 0.0. This might seem harsh, but it teaches the model to use the expected output format, which makes downstream extraction reliable.

Advantages are the learning signal, and understanding them requires seeing three scenarios:

Scenario 1: Mixed results (normal learning). Rewards: [1, 1, 0, 0]. Mean: 0.5, Std: 0.577. Advantages: [+0.87, +0.87, -0.87, -0.87]. The correct rollouts are reinforced, the incorrect ones are suppressed. This is productive learning.

Scenario 2: All wrong (no learning). Rewards: [0, 0, 0, 0]. Mean: 0.0, Std: 0.0 (plus epsilon). Advantages: [0, 0, 0, 0]. No contrast within the group, so no learning signal. The model cannot distinguish better from worse because everything was equally bad.

Scenario 3: All correct (no learning). Rewards: [1, 1, 1, 1]. Mean: 1.0, Std: 0.0 (plus epsilon). Advantages: [0, 0, 0, 0]. Again no contrast. The model cannot learn anything because everything was equally good.

This is why generating enough rollouts (at least 4) matters: with only 2 rollouts, the probability that both are correct or both wrong is high, producing many zero-signal steps. With 4 rollouts, you are more likely to get a mix of correct and incorrect, which provides the contrast needed for learning.

In the summer of 2023, a team at DeepSeek tried something that most RL researchers would have considered reckless. They trained an LLM to solve math problems without any human-labeled examples of correct reasoning. No curated chain-of-thought datasets. No expert annotations. Instead, they let the model attempt problems on its own, checked whether the final answer was correct using a simple calculator, and used that binary signal to adjust the model's weights. The technique was called GRPO, and its simplicity was its genius.

Loading model and training data

import torch
from reasoning_from_scratch.ch02 import get_device
from reasoning_from_scratch.ch03 import load_model_and_tokenizer, render_prompt

device = get_device()
model, tokenizer = load_model_and_tokenizer(
    which_model="base", device=device, use_compile=False
)

We need a training dataset separate from MATH-500. The full MATH dataset contains ~12,500 problems; the 500-problem test set is reserved for evaluation:

## Production checklist: before you start a GRPO training run

Before committing GPU hours to a GRPO training run, verify these prerequisites:

**1. Evaluation pipeline works.** Run the Chapter 3 evaluation on 50 problems with the base model. Verify you get ~15% accuracy. If you get 0%, your extraction or grading pipeline is broken. If you get 100%, your dataset is contaminated (the model has seen the problems during pre-training).

**2. Rollout generation works.** Generate 4 rollouts for a single problem. Verify they are different (temperature is working). Verify at least some produce `\boxed{}` answers. If all rollouts are identical, temperature or top-p is misconfigured. If none produce `\boxed{}` answers, the prompt template needs adjustment.

**3. Rewards have contrast.** Check that the 4 rollouts produce a mix of 1.0 and 0.0 rewards. If all are 0.0, the problem is too hard for the current model. If all are 1.0, the problem is too easy. In both cases, the advantage is zero and no learning occurs. Your training data should include a mix of difficulty levels.

**4. Gradient flows.** Verify that `loss.backward()` produces nonzero gradients. Print the gradient norm: `total_norm = sum(p.grad.norm() ** 2 for p in model.parameters() if p.grad is not None) ** 0.5`. If it is zero, something is wrong with the computation graph (perhaps `@torch.inference_mode()` is being used instead of `@torch.no_grad()` in the rollout generation).

**5. Checkpoint saving works.** Save and reload a checkpoint. Verify the reloaded model produces the same outputs as the original. A common bug: saving the model state dict but forgetting to save the optimizer state, which means training cannot be resumed properly.

**6. Evaluation on held-out set is automated.** Set up automatic evaluation every N steps (10-50 steps for small models). This is your ground truth for checkpoint selection. Without it, you are guessing when to stop.

import json
from pathlib import Path
import requests

def load_math_train(local_path="math_train.json", save_copy=True):
    local_path = Path(local_path)
    url = (
        "https://raw.githubusercontent.com/rasbt/"
        "math_full_minus_math500/refs/heads/main/"
        "math_full_minus_math500.json"
    )
    if local_path.exists():
        with local_path.open("r", encoding="utf-8") as f:
            data = json.load(f)
    else:
        r = requests.get(url, timeout=30)
        r.raise_for_status()
        data = r.json()
        if save_copy:
            with local_path.open("w", encoding="utf-8") as f:
                json.dump(data, f, indent=2)
    return data

math_train = load_math_train()
print("Dataset size:", len(math_train))   # 12000

Each entry contains "problem", "answer" (for verifier), "solution" (not used during training), "level" (difficulty), and "type" (subject area). The solution field is intentionally not used, as constraining the model to a specific solution path would limit exploration.

Sampling rollouts

A rollout is a complete answer generated by the model for a given prompt. Critical implementation detail: we use @torch.no_grad() instead of @torch.inference_mode() because we need to backpropagate through the computation graph later.

from reasoning_from_scratch.qwen3 import KVCache
from reasoning_from_scratch.ch04 import top_p_filter

@torch.no_grad()
def sample_response(
    model, tokenizer, prompt, device,
    max_new_tokens=512, temperature=0.8, top_p=0.9,
):
    input_ids = torch.tensor(tokenizer.encode(prompt), device=device)
    cache = KVCache(n_layers=model.cfg["n_layers"])
    model.reset_kv_cache()
    logits = model(input_ids.unsqueeze(0), cache=cache)[:, -1]

    generated = []
    for _ in range(max_new_tokens):
        if temperature and temperature != 1.0:
            logits = logits / temperature
        probas = torch.softmax(logits, dim=-1)
        probas = top_p_filter(probas, top_p)
        next_token = torch.multinomial(probas.cpu(), num_samples=1).to(device)
        token_id = next_token.item()
        generated.append(token_id)
        if tokenizer.eos_token_id is not None and token_id == tokenizer.eos_token_id:
            break
        logits = model(next_token, cache=cache)[:, -1]

    full_token_ids = torch.cat([
        input_ids,
        torch.tensor(generated, device=device, dtype=input_ids.dtype),
    ])
    return full_token_ids, input_ids.numel(), tokenizer.decode(generated)

The reward function

The same verifier from Chapter 3, repurposed as a reward generator:

from reasoning_from_scratch.ch03 import extract_final_candidate, grade_answer

def reward_rlvr(answer_text, ground_truth):
    extracted = extract_final_candidate(answer_text, fallback=None)
    if not extracted:
        return 0.0
    correct = grade_answer(extracted, ground_truth)
    return float(correct)

By setting fallback=None, we enforce that the model must use \boxed{} format. Results: \boxed{83} → 1.0, \boxed{38} → 0.0, "The final answer is 83" (correct but NOT boxed) → 0.0.

Advantages · why contrast is everything

rewards = torch.tensor([1., 1., 0., 0.])
advantages = (rewards - rewards.mean()) / (rewards.std() + 1e-4)
# tensor([0.8659, 0.8659, -0.8659, -0.8659])

Positive advantages → increase likelihood. Negative → decrease. If all rewards are the same, std is zero and advantages are zero. No learning happens without contrast. This is why num_rollouts >= 2.

Sequence log-probabilities

In GRPO, we use summed (not averaged) log-probabilities because each rollout receives a single reward for the whole sequence:

def sequence_logprob(model, token_ids, prompt_len):
    logits = model(token_ids.unsqueeze(0)).squeeze(0).float()
    logprobs = torch.log_softmax(logits, dim=-1)
    selected = logprobs[:-1].gather(
        1, token_ids[1:].unsqueeze(-1)
    ).squeeze(-1)
    return torch.sum(selected[prompt_len - 1:])

Testing on four rollouts:

\boxed{83}                       → Logprob: -7.9243
The correct answer is \boxed{83} → Logprob: -20.1546
The final answer is 83           → Logprob: -16.6130
We get \boxed{38}                → Logprob: -23.3677

Shorter answers get higher (less negative) logprobs. The incorrect answer gets the lowest.

The policy gradient loss deserves a careful unpacking because it is the single equation that makes GRPO work.

pg_loss = -(advantages.detach() * logprobs).mean()

Reading from inside out:

logprobs: A tensor of sequence-level log-probabilities, one per rollout. Each value measures how likely the current model was to generate that particular rollout. These are the only values with gradients attached; they depend on the model's parameters through the forward pass.

advantages.detach(): A tensor of z-score-normalised advantages, one per rollout. Positive for above-average rollouts, negative for below-average. The .detach() is critical: it tells PyTorch "treat these as fixed constants, do not compute gradients through them." Without detach, the optimizer would try to change the advantages themselves (making all advantages positive, which is trivially optimal but useless).

advantages.detach() * logprobs: Element-wise multiplication. For a correct rollout with advantage +1.0 and logprob -156, the product is -156. For an incorrect rollout with advantage -1.0 and logprob -135, the product is +135. The correct rollout contributes a negative term; the incorrect rollout contributes a positive term.

.mean(): Average across rollouts. This normalizes the loss magnitude by the number of rollouts, making the gradient scale independent of batch size.

-(...): Negate the entire thing. PyTorch optimizers minimize loss. We want to maximize the expected advantage-weighted log-probability. Negating converts maximization to minimization.

The gradient's effect: When loss.backward() runs, the gradient flows through the logprobs (the only non-detached term). For the correct rollout (advantage +1.0), the gradient points in the direction that would increase its logprob (make it more likely). For the incorrect rollout (advantage -1.0), the gradient points in the direction that would decrease its logprob (make it less likely). The optimizer then steps in this direction, simultaneously increasing the probability of correct sequences and decreasing the probability of incorrect ones.

This is the complete mechanism by which GRPO teaches a model to reason. There is no hidden magic. The model generates text, a verifier checks correctness, advantages encode relative quality, log-probabilities encode how the model currently generates, and the gradient adjusts the model to generate correct text more often.

The policy gradient loss

rewards = torch.tensor([1., 1., 0., 0.])
logprobs = torch.tensor([-7.9243, -20.1546, -16.6130, -23.3677])
advantages = (rewards - rewards.mean()) / (rewards.std() + 1e-4)

pg_loss = -(advantages.detach() * logprobs).mean()
print(f"Policy gradient loss: {pg_loss:.4f}")   # -2.5764

The negative sign converts maximization into minimization for PyTorch's optimizers. .detach() prevents gradients from flowing through the advantages.

The complete GRPO function

def compute_grpo_loss(
    model, tokenizer, example, device,
    num_rollouts=2, max_new_tokens=256,
    temperature=0.8, top_p=0.9,
):
    assert num_rollouts >= 2
    roll_logps, roll_rewards, samples = [], [], []
    prompt = render_prompt(example["problem"])
    was_training = model.training
    model.eval()

    for _ in range(num_rollouts):
        token_ids, prompt_len, text = sample_response(
            model=model, tokenizer=tokenizer,
            prompt=prompt, device=device,
            max_new_tokens=max_new_tokens,
            temperature=temperature, top_p=top_p,
        )
        reward = reward_rlvr(text, example["answer"])
        logp = sequence_logprob(model, token_ids, prompt_len)
        roll_logps.append(logp)
        roll_rewards.append(reward)
        samples.append({"text": text, "reward": reward,
                        "gen_len": token_ids.numel() - prompt_len})

    if was_training:
        model.train()

    rewards = torch.tensor(roll_rewards, device=device)
    advantages = (rewards - rewards.mean()) / (rewards.std() + 1e-4)
    logps = torch.stack(roll_logps)
    pg_loss = -(advantages.detach() * logps).mean()

    return {
        "loss": pg_loss.item(), "pg_loss": pg_loss.item(),
        "rewards": roll_rewards,
        "advantages": advantages.detach().cpu().tolist(),
        "samples": samples, "loss_tensor": pg_loss,
    }

When both rollouts answer incorrectly, rewards are [0.0, 0.0], advantages are [0.0, 0.0], and loss is 0.0. No gradient, no update.

Before examining the training loop code, we can understand what makes GRPO training materially different from standard supervised learning.

In supervised learning, the training data is fixed. You have input-output pairs, and the model learns to map inputs to outputs by minimizing cross-entropy loss. The loss function is deterministic: given the same input and the same model weights, you always get the same loss value. Training is reproducible, predictable, and well-understood.

In GRPO, the training data is generated by the model itself. At each step, the model generates rollouts using its current parameters. Different random seeds produce different rollouts. Different rollouts produce different rewards. Different rewards produce different advantages. Different advantages produce different gradients. The entire training signal is stochastic, dependent on the model's own generation process.

This self-referential nature creates three challenges that do not exist in supervised learning:

Challenge 1: Non-stationarity. As the model improves, the distribution of its generated rollouts changes. Problems that were unsolvable at step 1 become solvable at step 30. The effective difficulty of the training data shifts continuously, even though the actual training problems are fixed.

Challenge 2: Variance. The policy gradient is a high-variance estimator. With only 4 rollouts per step, the advantage estimates are noisy. Two training runs with different random seeds can produce significantly different accuracy trajectories. This makes debugging difficult: is a drop in accuracy a genuine problem or just random fluctuation?

Challenge 3: Reward sparsity. For hard problems, all 4 rollouts might be wrong, producing zero advantage and zero gradient. The model spends a training step generating 4 responses (expensive) and learning nothing. As the model improves and easy problems are exhausted, more steps produce zero signal, slowing learning.

Understanding these challenges is essential for interpreting training curves. A noisy loss curve is normal. A fluctuating accuracy is normal. A step with zero gradient is normal. The goal is not smooth optimisation (that is supervised learning's luxury) but sustained improvement in evaluation accuracy over hundreds of steps.

The training loop

def train_grpo(
    model, tokenizer, math_train, device,
    steps=50, num_rollouts=4, max_new_tokens=512,
    temperature=0.9, top_p=0.9, lr=5e-6,
    grad_clip_norm=1.0, seed=123, checkpoint_dir="checkpoints",
):
    optimizer = torch.optim.AdamW(model.parameters(), lr=lr)
    model.train()

    for step in range(1, steps + 1):
        torch.manual_seed(seed + step)
        example = math_train[(step - 1) % len(math_train)]
        optimizer.zero_grad()

        stats = compute_grpo_loss(
            model=model, tokenizer=tokenizer,
            example=example, device=device,
            num_rollouts=num_rollouts,
            max_new_tokens=max_new_tokens,
            temperature=temperature, top_p=top_p,
        )

        stats["loss_tensor"].backward()

        if grad_clip_norm is not None:
            torch.nn.utils.clip_grad_norm_(model.parameters(), grad_clip_norm)

        optimizer.step()

        print(
            f"[Step {step}/{steps}] "
            f"loss={stats['loss']:.4f} "
            f"reward_avg={sum(stats['rewards'])/len(stats['rewards']):.3f} "
            f"avg_resp_len={sum(s['gen_len'] for s in stats['samples'])/len(stats['samples']):.1f}"
        )

        if step % 10 == 0:
            ckpt_path = Path(checkpoint_dir) / f"grpo-step{step:05d}.pth"
            ckpt_path.parent.mkdir(parents=True, exist_ok=True)
            torch.save(model.state_dict(), ckpt_path)

    return model

After 50 steps: 47.4% accuracy on MATH-500, up from 15.2%. In 50 gradient updates, the model learned to solve nearly half of competition-level math problems without seeing a single worked solution.

The five vital signs

A human doctor does not diagnose a patient by checking a single metric. They check pulse, blood pressure, temperature, respiration rate, and oxygen saturation. The five readings together tell a story that no single reading can. GRPO training requires the same multi-signal approach.

In Chapter 6, our 50-step training run produced output like:

[Step 1/50] loss=-0.0000 reward_avg=0.000 avg_resp_len=5.5
[Step 2/50] loss=-0.0000 reward_avg=0.000 avg_resp_len=6.8
[Step 3/50] loss=0.3592 reward_avg=0.250 avg_resp_len=7.8
...
[Step 50/50] loss=1.2341 reward_avg=0.750 avg_resp_len=198.3

For 50 steps, three metrics (loss, reward average, response length) suffice. For 500 steps, you need all five vital signs.

Vital sign 1: Average reward. Are the model's answers getting more correct? A rising trend is good. A plateau is normal once easy problems are exhausted. A sudden collapse to zero means the model has become either too bad to ever be right or too good to ever be wrong. Either way, the learning signal has vanished.

Vital sign 2: Average response length. How verbose is the model becoming? A healthy trend is gradual increase as the model learns longer reasoning traces. An explosive increase, say from 50 tokens to 500 tokens in a few steps, signals that the model has found a way to exploit the reward function by padding responses. This is called length exploitation and is one of the most common failure modes.

Vital sign 3: Evaluation accuracy on MATH-500. This is the ground truth. Training metrics can deceive because they reflect performance on training data, which the model has seen. Evaluation accuracy on held-out problems cannot be gamed. In production, you save checkpoints every N steps and select the one with the highest evaluation accuracy. the best checkpoint in this run was not the last checkpoint.

Vital sign 4: Advantage standard deviation. This reflects the strength of the learning signal. Recall from Chapter 6 that advantages are z-score normalised rewards within a batch. If all rollouts receive the same reward (all correct or all incorrect), the standard deviation is zero and the advantages are zero. A sustained drop in advantage std means more training steps produce no learning. The model is spending time and compute achieving nothing.

Vital sign 5: Entropy. Entropy measures how uncertain the model's probability distribution is during generation. Treat entropy as a measure of how many tokens the model is seriously considering at each step. Moderate entropy (1-3 nats) indicates healthy exploration: the model is choosing confidently among a reasonable set of options. Very low entropy (below 0.5) signals collapse: the model has become so deterministic that it generates nearly identical responses every time, eliminating the diversity needed for exploration. Very high entropy (above 5) signals instability: the model is producing near-random text, spreading probability mass across thousands of tokens.

The plotting function for tracking these metrics:

import pandas as pd
import matplotlib.pyplot as plt

def plot_grpo_metrics(csv_path, columns=None, window=20):
    df = pd.read_csv(csv_path)
    if columns is None:
        columns = [c for c in df.columns if c != "step"]
    n_cols = 2
    n_rows = (len(columns) + 1) // 2
    fig, axes = plt.subplots(
        n_rows, n_cols, figsize=(12, 4 * n_rows)
    )
    axes = axes.flatten()
    for idx, col in enumerate(columns):
        ax = axes[idx]
        ax.plot(df["step"], df[col], alpha=0.3,
                label="Raw")
        ax.plot(df["step"],
                df[col].rolling(window).mean(),
                label=f"MA-{window}", linewidth=2)
        ax.set_title(col)
        ax.legend()
        ax.grid(alpha=0.3)
    plt.tight_layout()
    plt.show()

For convenience, the supplementary materials provide pre-computed log files from all experiments in this chapter. A 500-step run on an H100 takes 2-3 hours, so using pre-computed logs is reasonable for readers who prefer not to run the training themselves.

A complete training run diary

Let me narrate a realistic 500-step GRPO training run, highlighting what each vital sign reveals at different stages. This is based on actual experiments with the Qwen3 0.6B model on an H100 GPU.

Steps 1-10: The cold start. Average reward: 0.0-0.1. Response length: 5-15 tokens. Advantage std: 0.0 most steps. Entropy: high (~4.0). The model generates terse, mostly wrong answers. Most batches produce all-zero rewards, so no learning happens. The model is basically frozen.

Steps 10-30: First signs of life. Reward begins ticking up: 0.1-0.3. Response length grows: 15-80 tokens. Advantage std: 0.3-0.5 (some batches have contrast). Entropy: still high (~3.5). The model starts generating longer responses, some of which happen to contain correct answers. The learning signal is noisy but nonzero. This is the "spark" phase.

Steps 30-50: Rapid improvement. Reward surges: 0.3-0.75. Response length: 80-200 tokens. Advantage std: 0.5-0.8 (strong learning signal). Entropy: moderate (~2.5). The model has learned that generating reasoning steps leads to correct answers. It produces structured solutions with increasing reliability. This is the "hockey stick" phase where accuracy climbs rapidly.

Steps 50-100: Plateau begins. Reward: 0.7-0.85 (slowing). Response length: 200-350 tokens. Advantage std: 0.4-0.6 (beginning to decline). Entropy: ~2.0 (concentrating). Evaluation accuracy peaks around 47-48%. The model has learned the easy improvements. Remaining gains require solving harder problems or refining existing strategies.

Steps 100-200: The danger zone. Reward: fluctuating 0.6-0.85. Response length: 300-500 tokens (still growing). Advantage std: 0.2-0.4 (weakening). Entropy: 1.5-2.0 (narrowing). Evaluation accuracy: flat or slightly declining (45-47%). The model is spending more tokens without getting more problems right. At this point, training instability begins without clipping.

Steps 200-500 (without clipping): Collapse. Reward: collapses to 0.0-0.2. Response length: explodes to 500+ tokens or collapses to 3-5 tokens. Advantage std: near zero. Entropy: either very low (collapse) or very high (random). Evaluation accuracy: drops to 20-30%. The model has overfit to a degenerate strategy.

Steps 200-500 (with clipping, eps=0.2): Stable plateau. Reward: stable 0.7-0.85. Response length: stable 200-350 tokens. Advantage std: 0.3-0.5. Entropy: 1.5-2.5. Evaluation accuracy: stable 40-45%. the best checkpoint in this run was typically around step 150-250.

This diary illustrates why the Chapter 6 default of 50 steps was well-chosen: it captures the rapid improvement phase while stopping before the instability zone. Longer training requires the stabilization techniques from this chapter.

Production · automated training monitoring

In production, you do not manually watch training curves. You set up automated monitoring with alerts. Here is a practical monitoring configuration:

# Alert thresholds for a GRPO training run
ALERTS = {
    "reward_collapse": {
        "condition": "reward_avg < 0.1 for 20 consecutive steps",
        "action": "Pause training, inspect data curriculum",
        "severity": "critical"
    },
    "length_explosion": {
        "condition": "avg_response_len > 800 tokens",

## The complete distillation training loop, annotated

The distillation training loop deserves a line-by-line walkthrough because it is the template for all supervised fine-tuning in LLM development:

```python
def train_distillation(
    model, train_examples, val_examples, device,
    epochs=2, lr=5e-6, grad_clip_norm=None,
    seed=123, log_every=50, checkpoint_dir="checkpoints",
):
    # Step 1: Initialize the optimizer
    # AdamW is the standard choice for LLM fine-tuning.
    # lr=5e-6 is deliberately small: we want to adjust
    # the model's behaviour without destroying pre-trained knowledge.
    # For comparison, pre-training uses lr=1e-4 to 3e-4.
    optimizer = torch.optim.AdamW(model.parameters(), lr=lr)
    
    # Step 2: Switch to training mode
    # This enables dropout (if present) and sets batch normalisation
    # to compute running statistics from the current batch.
    model.train()
    
    total_steps = epochs * len(train_examples)
    global_step = 0
    rng = random.Random(seed)

    for epoch in range(1, epochs + 1):
        # Step 3: Shuffle training examples
        # Random shuffling prevents the model from learning 
        # order-dependent patterns (e.g., "easy problems first").
        epoch_examples = list(train_examples)
        rng.shuffle(epoch_examples)

        for example in epoch_examples:
            global_step += 1
            
            # Step 4: Zero gradients from previous step
            # PyTorch accumulates gradients by default.
            # Without this call, gradients from step N-1
            # would contaminate step N.
            optimizer.zero_grad()
            
            # Step 5: Compute cross-entropy loss
            # This is answer-only: prompt tokens provide context
            # but are excluded from the loss computation.
            loss = compute_example_loss(model, example, device)
            
            # Step 6: Backpropagate
            # Compute gradients of the loss with respect to
            # all 621 million parameters.
            loss.backward()
            
            # Step 7: Optional gradient clipping
            # Prevents any single example from causing a
            # catastrophically large parameter update.
            if grad_clip_norm is not None:
                torch.nn.utils.clip_grad_norm_(
                    model.parameters(), grad_clip_norm
                )
            
            # Step 8: Update weights
            # AdamW adjusts each parameter based on its gradient,
            # learning rate, and per-parameter momentum statistics.
            optimizer.step()

            # Step 9: Periodic evaluation
            if log_every and global_step % log_every == 0:
                val_loss = evaluate_examples(
                    model=model,
                    examples=val_examples,
                    device=device,
                )
                model.train()  # Switch back to training mode
                print(
                    f"[Epoch {epoch}/{epochs} "
                    f"Step {global_step}/{total_steps}] "
                    f"train_loss={loss.item():.4f} "
                    f"val_loss={val_loss:.4f}"
                )

        # Step 10: Save checkpoint after each epoch
        ckpt_path = Path(checkpoint_dir)
        ckpt_path.mkdir(parents=True, exist_ok=True)
        ckpt_file = ckpt_path / f"distill-epoch{epoch}.pth"
        torch.save(model.state_dict(), ckpt_file)
        print(f"Saved checkpoint: {ckpt_file}")

    return model

Every line in this loop is standard PyTorch. The only reasoning-specific choice is compute_example_loss, which computes cross-entropy only on the answer tokens (excluding the prompt). Replace that function with any other loss, and this becomes a general-purpose fine-tuning loop.

The validation loss is computed on 25 held-out examples every 50 steps. This frequency balances two concerns: checking too often slows training (each evaluation requires a forward pass through 25 examples); checking too rarely means you might miss the optimal checkpoint. For larger datasets, evaluate every 200-500 steps.

The learning rate 5e-6 deserves comment. Pre-training uses learning rates 50-100x larger (1e-4 to 3e-4). Fine-tuning uses smaller learning rates because we want to adjust the model's behaviour without overwriting its pre-trained knowledge. Treat pre-training as teaching a student to read, and fine-tuning as teaching that student to read a specific genre. You do not want the genre training to cause the student to forget how to read.

    "action": "Add length penalty or cap max_new_tokens",
    "severity": "warning"
},
"accuracy_degradation": {
    "condition": "eval_accuracy drops 5+ points from peak",
    "action": "Stop training, use best checkpoint",
    "severity": "critical"
},
"entropy_collapse": {
    "condition": "entropy_avg < 0.5 for 10 steps",
    "action": "Increase temperature or reduce learning rate",
    "severity": "warning"
},
"entropy_explosion": {
    "condition": "entropy_avg > 5.0",
    "action": "Reduce learning rate, add clipping",
    "severity": "critical"
},
"advantage_starvation": {
    "condition": "advantage_std < 0.1 for 30 steps",
    "action": "Adjust curriculum difficulty, increase rollouts",
    "severity": "warning"
}

}

A crucial release principle is: **do not promote the latest checkpoint automatically.** Always maintain a leaderboard of checkpoints ranked by evaluation accuracy and deploy the best one. Many teams have deployed degraded models because their training pipeline automatically promoted the most recent checkpoint without evaluation.

## The overcorrection problem

The core instability in basic GRPO comes from unconstrained updates. Consider a student who reads one brilliant essay and immediately tries to rewrite everything they have ever written in that style. They abandon their own voice, their own techniques, their own strengths. The essay was inspiring, but the overcorrection destroys everything they previously knew.

In GRPO, a single rollout with a large positive advantage and a far-from-zero log-probability can produce an enormous gradient. When the optimizer applies this gradient, the model's weights change materially in one step. The parameters that encoded useful patterns from pre-training are overwritten. The model loses capabilities it already had.

Concretely: suppose the model generates a correct answer to a hard problem and the advantage is +3.0 (well above average). The sequence log-probability is -50.0 (a moderately unlikely sequence). The policy gradient loss for this rollout is -(3.0 × -50.0) = 150.0, which is a large loss value. Backpropagating through this loss produces a large gradient that, without constraints, could change the model's weights by more than all 49 previous training steps combined.

***

## The clipping fix

The fix, borrowed from **Proximal Policy optimisation (PPO)**, is **clipped policy ratios**. The idea is to limit how much any single training step can change the model's behaviour relative to where it was before the step.

For each token in a rollout, we compute the ratio of the new policy's probability to the old policy's probability:

```python
ratio = torch.exp(logprob_new - logprob_old)

If the ratio is close to 1.0, the new and old policies agree: the update is conservative. If the ratio is 2.0, the new policy assigns twice the probability to this token: the update is aggressive. If the ratio is 0.1, the new policy has drastically reduced this token's probability.

Clipping constrains this ratio to the range [1 - eps, 1 + eps]:

clipped_ratio = torch.clamp(
    ratio, 1 - clip_eps, 1 + clip_eps
)
pg_loss = -torch.min(
    ratio * advantages,
    clipped_ratio * advantages
).mean()

The torch.min is the key mechanism. It takes the more pessimistic estimate. If the unclipped update would give a large improvement, clipping caps the credit. If the unclipped update would give a large penalty, clipping caps the damage. This prevents any single step from causing an outsized weight update.

With clip_eps=0.2, the ratio is constrained to [0.8, 1.2]. The model cannot more than double or halve its probability for any token in a single step. In experiments, this produces materially more stable training: the model sustains 40%+ accuracy over 500 steps instead of collapsing after 100.

Different clip values have different effects. DAPO recommends a larger clip_eps=0.28 to allow more exploration. Very small values (0.05) over-constrain learning, preventing the model from making necessary changes. Very large values (5.0) provide insufficient protection, making clipping effectively a no-op. The sweet spot for most practitioners is 0.1-0.2.

The KL death spiral

The original GRPO formulation included a KL divergence term that penalizes the model for drifting too far from a reference policy (usually the model's initial state):

kl_loss = torch.mean(logprob_ref - logprob_new)
total_loss = pg_loss + kl_coeff * kl_loss

The intuition is appealing: prevent the model from becoming so different from its starting point that it loses the fluency and knowledge acquired during pre-training. If the model starts generating gibberish in pursuit of high rewards, the KL term pulls it back toward sensible text.

In practice, for math reasoning, KL regularization creates a death spiral. Here is how it happens:

1. The model enters a phase where all rollouts receive the same reward (all correct or all incorrect for a particular problem). This happens naturally when the model encounters problems that are uniformly too hard or too easy.

2. With all rewards equal, advantages are zero. The policy gradient component of the loss is zero. No learning signal from the task.

3. The only remaining gradient comes from the KL term, which pushes the model toward the reference policy.

4. The reference policy is the untrained base model, which generates essentially random responses for math problems.

5. The KL term pushes the model toward random generation, which makes rewards even worse.

6. Worse rewards mean more steps with uniform rewards, which means more steps where KL is the sole gradient source, which pushes harder toward randomness.

7. The model collapses to 0% accuracy.

This is not theoretical. In experiments with the Qwen3 0.6B model, adding a KL term (kl_coeff=0.01) caused accuracy to drop from 40% to 0% within 100 training steps when rewards temporarily collapsed.

Multiple independent research teams have converged on the same conclusion: for math reasoning tasks, removing the KL term entirely produces more stable training. Dr. GRPO (2025), DAPO (2025), OLMo 3, and DeepSeek-V3.2 all report improved stability and equal or better accuracy without KL regularization for verifiable tasks.

The nuance: KL may still be valuable for natural language tasks where there is no verifiable reward, where keeping the model "close to its training distribution" prevents degenerate outputs. DeepSeek-V3.2 uses a domain-specific approach: KL coefficient of zero for math and code tasks, nonzero for natural language tasks.

Format rewards and the model that learned to cheat

A fascinating failure mode emerges when you add format rewards on top of correctness rewards. You might want the model to use <think> tokens to demarcate its reasoning:

def format_reward(text):
    has_think = "<think>" in text and "</think>" in text
    return 1.0 if has_think else 0.0

The combined reward becomes rlvr_reward + weight * format_reward. If weight is too high relative to the correctness reward, the model discovers that it can earn reward by simply writing <think></think> (with nothing inside) at the start of every response. It gets the format bonus without doing any reasoning. The model has hacked the reward function.

This is reward hacking, one of the oldest and most persistent problems in reinforcement learning. The model optimizes the reward signal you gave it, which is not necessarily the behaviour you wanted. The model is not being malicious. It is being rational: it found the cheapest path to high reward.

The fix requires careful reward engineering. Make format rewards small relative to correctness rewards. Or condition format rewards on correctness: only grant format credit when the answer is also correct (reward = rlvr_reward + weight * format_reward * rlvr_reward). Or use curriculum scheduling that introduces format rewards only after correctness rewards are stable.

An important subtlety: the base model has never seen <think> tokens during pre-training. It does not know what they mean. RL can teach the model that using <think> tokens is correlated with reward, but it cannot teach the model what meaningful reasoning should go inside them. For that, the model needs prior exposure through pre-training or instruction tuning that includes <think> examples.

Thought experiment · the reward hacking zoo

Reward hacking is not unique to format rewards. Every reward function can be hacked. Here are five real examples from reasoning model training:

Hack 1: The empty thinker. With a format reward for <think> tags, the model learns to write <think></think> (empty tags) for free reward. Fix: condition format reward on correctness.

Hack 2: The verbose reasoner. If longer responses correlate with higher accuracy (they often do, up to a point), the model learns to pad responses with repetitive text. A response might include the same calculation three times in slightly different words. Fix: add a length penalty to the reward, or use per-token loss normalisation (DAPO).

Hack 3: The hedge artist. If the model discovers that hedging language ("The answer might be 83, but it could also be 82") sometimes gets partial credit (depending on the extraction pipeline), it learns to include multiple candidate answers in every response. Fix: use strict extraction that only accepts a single boxed answer.

Hack 4: The format memorizer. The model memorizes that certain output templates (e.g., "Step 1: ... Step 2: ... Step 3: ... Therefore, the answer is ...") correlate with high reward, and applies this template regardless of the problem content. The reasoning steps are cosmetic, not substantive. Fix: include diverse problem types in training so that no single template works universally.

Hack 5: The difficulty avoider. If the model can influence which problems it encounters (e.g., through early stopping patterns that trigger different sampling), it might learn to generate outputs that are more likely to be scored on easier problems. This is rare in standard GRPO but can occur in more complex training setups.

The general principle: the model will optimise whatever you reward. If there is a cheaper path to high reward than actually solving the problem, the model will find it. Careful reward engineering is not a one-time setup task; it requires ongoing monitoring and adjustment as the model discovers new exploitation strategies.

Before listing the modifications, it is worth understanding why there are so many. GRPO was published in the DeepSeek-R1 paper in January 2025. Within six months, at least 15 independent research groups proposed improvements. This rapid iteration reflects two factors.

First, GRPO is simple enough to implement and test quickly. A researcher can modify one component (e.g., the advantage normalisation), run a training experiment in a day, and compare results. The simplicity that makes GRPO elegant also makes it easy to tinker with.

Second, basic GRPO has genuine failure modes that manifest differently depending on model size, dataset difficulty, and training duration. A modification that stabilizes a 1B model on math might destabilize a 70B model on code. This context-dependence means different teams, working with different setups, independently discover different failure modes and different fixes.

The modifications can be grouped into four categories:

Category 1: Advantage computation. How to normalise rewards within a group. Z-score normalisation (the default) divides by standard deviation, which can be unstable when all rewards are similar (std near zero). Dr. GRPO proposes mean-subtraction without std-division. GDPO proposes per-reward-source normalisation when multiple reward signals are combined.

Category 2: Update magnitude. How to limit the size of each parameter update. Clipping (PPO-style, used in this chapter) limits the policy ratio. DAPO clips more aggressively (eps=0.28). CISPO clips the importance weights themselves rather than the ratio-advantage product. VERL truncates importance sampling for stale rollouts.

Category 3: Regularization. How to prevent the model from diverging too far from its starting point. KL regularization (original GRPO) penalizes divergence from a reference model but can cause death spirals. Dr. GRPO and DAPO remove KL entirely for math tasks. DeepSeek-V3.2 uses domain-specific KL (zero for math/code, nonzero for natural language).

Category 4: Loss normalisation. How to aggregate the per-token losses within a sequence. Sequence-level normalisation (the default) sums all token losses. Token-level normalisation (DAPO) divides by token count, preventing long sequences from dominating the gradient. The choice affects whether the model learns to prefer shorter or longer responses.

For practitioners starting out: use basic GRPO (Chapter 6) plus clipping (this chapter, eps=0.2), no KL, and 4 rollouts per step. Add token-level normalisation if you observe length exploitation. Add domain-specific KL if you train on a mix of math and natural language tasks. Save checkpoints every 10-50 steps and select based on evaluation accuracy.

The map of 15+ modifications

The field of GRPO improvements is evolving so fast that a comprehensive survey would be outdated before it was published. As of early 2025, more than 15 modifications have been proposed:

1. Token-level loss normalisation (DAPO): Divide by number of tokens rather than rollouts to avoid penalizing long responses.
2. No standard-deviation normalisation (Dr. GRPO): Replace z-score with simple mean-subtraction, avoiding instability when std is very small.
3. No KL loss (Dr. GRPO, DAPO): Remove KL entirely for math reasoning tasks.
4. Clip higher (DAPO): Use clip_eps=0.28 instead of 0.1-0.2 to allow more exploration.
5. Truncated importance sampling (VERL): Cap the importance ratio more aggressively.
6. Domain-specific KL (DeepSeek-V3.2): KL=0 for math/code, KL>0 for natural language.
7. Reweighted KL (DeepSeek-V3.2): Weight KL by importance sampling ratios.
8. Off-policy sequence masking (DeepSeek-V3.2): Mask sequences where the policy ratio exceeds a threshold.
9. Keep sampling mask (DeepSeek-V3.2): Maintain the same top-p/top-k mask during evaluation.
10. Original advantage normalisation (DeepSeek-V3.2): Despite Dr. GRPO's recommendation, z-score works well at DeepSeek's scale.
11. Per-reward normalisation (GDPO): Normalize each reward source independently.
12. Sequence-level importance sampling (GSPO): Apply clipping at sequence level.
13. Clip importance weights (CISPO): Clip the weights rather than the product. 14-15. Various advantage and gradient normalisation variants.

The practical takeaway: start with basic GRPO from Chapter 6, add clipped policy ratios (clip_eps=0.2), omit the KL term for math/code tasks, monitor all five vital signs, and select checkpoints based on evaluation accuracy rather than training loss.

One of the most insidious failure modes is what practitioners call mode collapse: the model converges to generating nearly identical responses regardless of the input. You can detect this by monitoring entropy. When entropy drops below 0.5 for multiple consecutive steps, the model is generating the same token at each position with near-certainty. At that point, the model has effectively memorised a single response template and is applying it to every problem.

Mode collapse often follows a specific pattern. First, the model discovers a response format that frequently earns reward (say, a particular way of structuring algebra solutions). It reinforces this format aggressively. Other solution strategies get suppressed. The model's output diversity decreases. Because diversity has decreased, self-consistency (if used during evaluation) becomes less effective. Evaluation accuracy stalls or drops. The model continues reinforcing the same format because it still earns reward on the problems where that format works. But it has lost the ability to handle problems requiring different approaches.

The fix is multi-pronged. Clipping limits the rate at which the model can suppress alternative strategies. Higher temperature during rollout generation maintains exploration. And curriculum scheduling ensures the model encounters diverse problem types rather than getting stuck on a subset.

Another critical insight from production deployments: learning rate scheduling matters more in GRPO than in supervised learning. Most GRPO implementations use a constant learning rate, but a warm-up + cosine decay schedule can significantly improve stability. The warm-up phase (10-20 steps at reduced learning rate) prevents the optimizer from making large changes before it has accumulated meaningful gradient statistics. The cosine decay gradually reduces the learning rate as training progresses, matching the intuition that large updates are more dangerous as the model becomes more refined.

The thread

You have now seen both the power and the fragility of reinforcement learning for reasoning. GRPO can transform a base model into a reasoning model in 50 steps. It can also destroy a reasoning model in 500 steps without proper stabilization. The difference is engineering: clipping, metric monitoring, checkpoint selection, and reward design.

But GRPO has a fundamental limitation. The model learns by trial and error, generating hundreds of rollouts for each training example, most of which are wrong. For a 0.6B model, this is feasible. For a 671B model, the compute cost is staggering. Is there a way to teach reasoning that does not require the model to discover correct reasoning patterns on its own?

There is. You give it a teacher.

Why monitoring saves money

A concrete economic argument for monitoring: an unmonitored 500-step GRPO training run on an H100 GPU takes approximately 3 hours and costs roughly $9 at typical cloud rates. If the model peaks at step 150 and degrades afterward, you waste 350 steps of compute ($6.30). Over 10 experimental runs (common during hyperparameter tuning), the waste totals $63. With monitoring and early stopping, each run stops at the peak, saving 70% of compute.

For larger models, the savings are proportionally larger. A 7B model's GRPO training run might take 72 hours and cost $216. Detecting collapse at step 200 (out of 2,000) instead of running to completion saves $194 per run. Over 10 runs, that is $1,940 of saved compute.

The monitoring infrastructure (logging to CSV, periodic evaluation, checkpoint saving) adds approximately 5% overhead to training time. The return on this investment is typically 50-70% compute savings from early stopping, plus the confidence that you are deploying the best checkpoint rather than the last checkpoint.

The stability hierarchy

Not all GRPO modifications contribute equally to stability. Based on the accumulated evidence from multiple research groups, here is a rough hierarchy from most to least impactful:

1. Clipped policy ratios (most impactful): Prevents catastrophic updates. The single most important addition to basic GRPO. Without clipping, training collapses after ~100 steps on average. With clipping, stable training extends to 500+ steps.

2. No KL for math/code (very impactful): Eliminates the death spiral failure mode. Without this fix, reward collapse triggers a cascade that destroys the model. With no KL, reward collapse simply causes zero learning (benign) rather than active degradation (catastrophic).

3. Gradient clipping (max_norm=1.0, impactful): Prevents any single step from producing extreme parameter changes. This is standard practice in all neural network training, not specific to GRPO, but especially important here because RL gradients can be much larger than supervised learning gradients.

4. Token-level loss normalisation (moderately impactful): Prevents long responses from dominating the gradient. Matters most when response length varies widely across rollouts.

5. Increased rollouts (4 → 8 per step, moderately impactful): Reduces variance in advantage estimates. More rollouts = more reliable signal. But each rollout costs a full generation pass, so the compute cost doubles.

6. Temperature scheduling (somewhat impactful): Starting with higher temperature (more exploration) and gradually decreasing (more exploitation) can help the model explore diverse strategies early and refine the best one later.

Start with items 1-3 (clipping + no KL + gradient clipping) and add 4-6 only if training curves show specific issues (length exploitation, noisy advantages, or early convergence).

The tightrope walker and the safety net

Clipped policy ratios are a safety net for a tightrope walker. Without a safety net, a single misstep (a large, noisy gradient) can be catastrophic (the model's parameters are pushed far from a useful region). With a safety net, the walker can take risks (explore different reasoning strategies) knowing that a fall will be caught at a predetermined height (the clip range [1-eps, 1+eps]).

The clip value eps determines how high the safety net is hung. eps=0.2 means the net is close: each step is limited to a 20% change in policy probability. eps=0.28 (DAPO's recommendation) hangs the net slightly higher, allowing more exploration. eps=5.0 hangs the net so high it is effectively useless.

The useful distinction from PPO, which introduced clipping: the safety net should be pessimistic. The torch.min operation always takes the more conservative estimate. If the unconstrained update suggests the model improved materially (which might be noise), clipping says "maybe, but we can only take credit for a modest improvement." If the unconstrained update suggests the model degraded badly, clipping says "maybe, but we can only penalize modestly." This pessimism is what makes training well-tested over hundreds of steps.

The field of GRPO modifications is evolving at a pace that makes any detailed survey obsolete within months. Between January and June 2025, at least 15 papers proposed improvements to the basic algorithm. Some modify advantage computation (Dr. GRPO removes standard-deviation normalisation). Some modify the clipping strategy (DAPO uses larger clip values). Some modify what is clipped (CISPO clips importance weights rather than the product of weights and advantages). Some modify the KL handling (DeepSeek-V3.2 uses domain-specific KL coefficients). Some modify the loss normalisation (DAPO normalizes per token rather than per rollout).

The common thread through all these modifications: they address specific instabilities observed during long training runs. Each modification is a patch for a specific failure mode. This is why the chapter focuses on understanding the failure modes rather than cataloging every proposed fix. If you understand why basic GRPO becomes unstable (unconstrained updates, reward collapse, KL death spirals), you can evaluate new modifications as they appear and choose the ones relevant to your specific training setup.

The practical takeaway has remained stable even as the details evolve: start simple (basic GRPO + clipping), monitor aggressively (all five vital signs), iterate quickly (short training runs with frequent evaluation), and never trust the last checkpoint.

A practical heuristic that many teams have converged on independently: train for 10% of the steps you think you need, evaluate, then decide whether to continue. If 50 steps of GRPO yield 47.4% accuracy on your eval set, run 50 more steps and check whether accuracy improved. If it did not, stop. If it did, run 50 more. This incremental approach avoids the common trap of committing to a 5,000-step training run that degrades after step 200. The compute wasted on steps 200-5,000 could have been spent on hyperparameter tuning, data quality improvements, or distillation from a better teacher. The field's collective experience suggests that for GRPO on math tasks, the useful training window is 50-500 steps for small models and 500-5,000 for large models. Beyond that, diminishing returns set in rapidly.

Another lesson from production: always maintain a "golden checkpoint" that you never overwrite. Before any training run, save the untrained base model to a separate path. If training goes catastrophically wrong (and it will, eventually), you need a known-good starting point. Many teams also maintain a checkpoint from their best previous training run. This creates a safety ladder: if the current run fails, fall back to the best previous run. If that fails, fall back to the untrained base model.

The interaction between clipping and the number of rollouts creates a nuanced tradeoff. With more rollouts per step (say 8 instead of 4), the advantage estimates are less noisy because they are computed over a larger sample. Less noisy advantages mean more reliable gradient directions, which means clipping needs to intervene less frequently. Conversely, with fewer rollouts (say 2), the advantages are very noisy, and clipping frequently caps gradients that were pointing in wrong directions. In experiments, 4 rollouts with clip_eps=0.2 provides a good balance between signal quality and compute cost for the Qwen3 0.6B model. Larger models, which have more parameters to update and more complex loss landscapes, often benefit from 8-16 rollouts per step with slightly larger clip values.

Decision check: What is the most common failure mode in GRPO training?

Reward collapse: all rollouts receive the same reward, producing zero advantage, zero learning signal, zero gradient. If KL regularization is present, it becomes the sole gradient source and pushes toward the reference policy (the untrained base model), creating a death spiral to 0% accuracy. The fix: omit KL for math tasks, monitor advantage std as an early warning, and ensure training data is neither uniformly too easy nor too hard.

Decision check: Why is clipping important for GRPO stability?

Without clipping, a single rollout with high advantage can produce an enormous gradient that drastically changes the model's weights in one step. This overcorrection destroys previously learned patterns. Clipping constrains the policy ratio to [1-eps, 1+eps], ensuring each step produces a bounded update. It comes from PPO and is one of the most reliable stabilization methods.

Decision check: What is reward hacking?

The model finds the cheapest path to high reward, which may not be the behaviour you intended. For example, if you reward the model for using <think> tags, it learns to write empty <think></think> tags for free reward without actually reasoning. The fix is careful reward engineering: keep format rewards small relative to correctness rewards, or condition them on correctness.

In production, the five vital signs translate into automated monitoring alerts:

Average reward dropping below threshold: If average reward drops below 0.3 for 50 consecutive steps, pause training and inspect. Likely cause: the training data curriculum has shifted to problems that are uniformly too hard, or the model has overfit to early easy problems and lost generality.

Response length exceeding limit: If average response length exceeds 1,000 tokens, the model may be gaming the reward function through verbosity. Mitigation: add a length penalty to the reward or cap generation at fewer tokens.

Evaluation accuracy dropping from peak: If evaluation accuracy drops by more than 5 percentage points from the best checkpoint, the model is degrading. Stop training and use the best checkpoint. This is the most important monitoring rule.

Advantage std near zero: If advantage std drops below 0.1 for 20 consecutive steps, the learning signal has vanished. Possible fixes: increase temperature (more diversity in rollouts), adjust the training curriculum (mix in easier problems), or increase the number of rollouts per step.

Entropy below 0.5 or above 5.0: Low entropy indicates collapse (near-deterministic generation). High entropy indicates instability (near-random generation). Both require intervention: for collapse, increase temperature or reduce learning rate; for instability, add clipping or reduce learning rate.

The release principle is: the best checkpoint may not be the last checkpoint. Always save checkpoints every N steps and select the one with the highest evaluation accuracy. Many teams have lost weeks of compute by assuming that training always improves the model.

A common question from practitioners new to GRPO: "How do I know when to stop training?" The answer requires reading all five vital signs simultaneously, not any one in isolation.

Here is a decision flowchart:

If average reward is climbing AND evaluation accuracy is climbing: Keep training. Everything is working.

If average reward is climbing BUT evaluation accuracy is flat: The model is getting better at the training problems but not generalizing to the held-out evaluation set. This is overfitting. Stop training and use the best checkpoint so far.

If average reward is flat AND evaluation accuracy is flat: The model has plateaued. More training will not help. Consider: (a) increasing rollout count for better advantage estimates, (b) increasing temperature for more exploration, (c) adding harder problems to the training data, or (d) stopping and accepting the current accuracy.

If average reward is declining OR evaluation accuracy is declining: Training is actively degrading the model. Stop immediately. Use the best previous checkpoint. Investigate: was clipping enabled? Was the learning rate too high? Did reward collapse occur?

If advantage std is near zero for many consecutive steps: The model is encountering problems that are uniformly too easy or too hard. Adjust the training data difficulty. Mix in a broader range of problem levels.

If entropy is dropping below 1.0: The model's output distribution is collapsing. Generation is becoming deterministic, killing exploration. Increase temperature. Reduce learning rate. Consider re-initializing from an earlier checkpoint.

These heuristics are not exhaustive, but they cover the most common scenarios that practitioners encounter. The key principle: no single metric tells the full story. Average reward can be high while the model is overfitting. Loss can be decreasing while accuracy is dropping. Only the combination of all five signals, interpreted together, gives a reliable picture of training health.

Reading the vital signs · a diagnostic walkthrough

Let me walk through reading a real training run's vital signs. In the first 50 steps, average reward rises from 0.0 to 0.75, average response length grows from 5 tokens to 198 tokens, and evaluation accuracy improves from 15.2% to 47.4%. All vital signs are healthy.

Between steps 50 and 150, reward continues to rise (0.75 to 0.85), but response length accelerates (198 to 450 tokens). The model is learning, but it is also becoming more verbose. This is not yet a problem, but it is worth watching.

At step 200, something changes. Reward plateaus at 0.85, but response length continues to grow (450 to 600 tokens). The model is not getting more problems right, but it is writing longer responses. This is the first warning sign: the model may be approaching the limit of what it can learn from the training data, and the growing length suggests it is padding rather than reasoning.

At step 300, advantage standard deviation drops sharply from 0.8 to 0.2. This means most training steps are producing near-zero learning signal: all rollouts in a batch receive the same reward. The model is either getting nearly everything right (unlikely at 85% accuracy) or encountering problems that are uniformly too hard.

By step 400, entropy drops below 1.0. The model has become very deterministic in its generation, repeating similar patterns across different problems. This reduces the diversity needed for exploration. Even with temperature=0.9, the model's internal distribution is so peaked that sampling produces little variation.

At step 500, evaluation accuracy has dropped to 35%, below the step-50 checkpoint. The model has overfit, forgotten, and degraded. The best checkpoint was at step 150 (evaluation accuracy ~48%).

This diagnostic walkthrough illustrates why checkpoint selection based on evaluation accuracy, not training loss, is essential. The training loss might suggest the model is still improving at step 500 (loss is decreasing), but evaluation accuracy reveals the truth: the model peaked at step 150 and has been getting worse ever since.

Run checklist · before you start a GRPO training run

Before committing GPU hours, verify these prerequisites:

1. Evaluation pipeline verified. Run Chapter 3 evaluation on 50 problems. Expect ~15% accuracy for the base model. If you get 0%, extraction or grading is broken. If you get 100%, your dataset is contaminated.

2. Rollout diversity confirmed. Generate 4 rollouts for a single problem with temperature=0.9. Verify they differ (if identical, temperature is not being applied). Verify at least some produce \boxed{} answers (if none do, adjust the prompt template).

3. Reward contrast exists. Check that 4 rollouts produce a mix of 1.0 and 0.0 rewards. If all are 0.0, the problem is too hard. If all are 1.0, too easy. Either way, the advantage is zero and no learning occurs.

4. Gradients flow. After loss.backward(), print the gradient norm: total_norm = sum(p.grad.norm()**2 for p in model.parameters() if p.grad is not None)**0.5. If zero, the computation graph is broken (check for @torch.inference_mode() where @torch.no_grad() is needed).

5. Checkpoints save and reload. Save a checkpoint, reload it, verify the model produces identical outputs. A common bug: saving the model but not the optimizer state, making training resumption impossible.

6. Evaluation automated. Set up automatic MATH-500 evaluation every 10-50 steps. This is your ground truth for checkpoint selection. Without it, you are guessing when training peaked.

These six checks take approximately 30 minutes and can save hours or days of wasted compute on misconfigured training runs.

Why RL for language is especially hard

Reinforcement learning for language models is harder than RL for games, robotics, or most other domains. The action space is enormous: at each token position, the model chooses from 151,936 possible actions. A 500-token response involves 151,936^500 possible sequences, a number with over 2.5 million digits. This is orders of magnitude larger than the action spaces in Atari (18 actions), Go (361 positions), or robotic manipulation (typically 6-12 continuous dimensions).

The reward is delayed: the model generates hundreds of tokens before receiving a single reward signal (correct or incorrect). It must attribute this reward back to specific token decisions hundreds of steps earlier. Which token in a 300-token reasoning trace was responsible for the wrong answer? The attention mask at token 47? The variable substitution at token 132? The arithmetic at token 256? The model cannot know.

The reward is binary: right or wrong. There is no partial credit for "almost correct" reasoning. A response that follows perfect logic for 299 tokens and makes a single arithmetic error in the last step gets the same reward (0.0) as a response that is completely nonsensical. This makes the learning signal extremely noisy.

These challenges explain why basic GRPO, despite its conceptual elegance, requires careful engineering to work in practice. The clipping, KL handling, and metric monitoring described in this chapter are not optional refinements. They are the difference between a training run that produces a useful model and one that produces an expensive brick.

Hard distillation · the practical choice

There are two ways a student can learn from a teacher. In a master class, the teacher performs a piece, and the student tries to reproduce the performance. The student hears only the notes the teacher plays; they do not see the teacher's internal deliberation about which notes were considered and rejected. This is hard distillation: the student learns from the teacher's final output.

In a very different kind of lesson, the teacher explains their decision process for every note. "Here I considered playing a B-flat, which would have been technically correct but emotionally flat. I chose a C-sharp instead because it creates tension that resolves in the next measure." The student learns not just what the teacher did but what the teacher considered doing and why. This is soft distillation: the student learns from the teacher's full probability distribution, including the probabilities assigned to tokens the teacher did not actually generate.

Soft distillation is richer. The teacher's probability distribution contains what Geoffrey Hinton (who coined the term "knowledge distillation" in 2015) called dark knowledge: information about the relationships between classes or tokens that is invisible in the final output. If the teacher assigns 60% probability to the correct answer and 30% to a closely related answer, that 30% tells the student something important about the problem's structure.

But for LLM reasoning, hard distillation is overwhelmingly more practical, for three reasons.

First, teacher logits are usually inaccessible. DeepSeek-R1, GPT-4, Claude, and Gemini expose generated text through their APIs, but not their internal probability distributions. You can collect the teacher's reasoning traces; you cannot collect its logits.

Second, storage. Each token position in a reasoning trace would require storing 151,936 floating-point numbers (one per vocabulary entry) if you wanted the full distribution. A 1,000-token reasoning trace would require 600 MB of logit data per example. Storing plain text for the same trace requires a few kilobytes.

Third, cross-model compatibility. Hard distillation works across model families because the teacher's text is simply re-tokenized by the student's tokenizer. Soft distillation requires the teacher and student to share a tokenizer, since the probability distributions must be over the same vocabulary. This restricts soft distillation to within-family transfers.

The takeaway: hard distillation is supervised fine-tuning on synthetic data. The teacher generates reasoning traces, and the student is trained to reproduce them. No new loss functions. No new training infrastructure. Just cross-entropy loss on teacher-generated tokens.

Building the training dataset

The dataset construction is the most labour-intensive part of distillation, but it is a one-time cost.

We take the same 12,000 MATH training problems used for GRPO in Chapters 6 and 7 (carefully separated from the 500-problem MATH-500 evaluation set to prevent data leakage). Instead of having the student generate solutions and grading them (as in RLVR), we send each problem to the teacher model and collect its complete response: the <think> reasoning trace plus the final \boxed{} answer.

The teacher in this case is DeepSeek-R1, accessed via API through OpenRouter. The total cost for generating 12,000 responses was approximately $50. This is a one-time expense; the resulting dataset can be reused for multiple training runs, different student models, and hyperparameter experiments.

Not all teacher responses are correct. DeepSeek-R1 achieves about 91% accuracy on MATH-500, which means roughly 9% of its responses contain errors. Training the student on incorrect reasoning traces would teach it to reproduce mistakes. So we filter: each teacher response is verified using the same evaluation pipeline from Chapter 3. Only responses with correct final answers are included in the training set.

After filtering and removing responses that exceed the student's context window (2,048 tokens), we are left with approximately 6,670 training examples and 25 validation examples.

Each training example is formatted as a conversation:

<|im_start|>user
[math problem]<|im_end|>
<|im_start|>assistant
<think>[teacher reasoning trace]</think>
[teacher final answer]<|im_end|>

The formatting uses the reasoning tokenizer's chat template, which includes <think> and </think> tokens that demarcate the reasoning section. This format is critical: the student needs to learn not just what to think but when to start and stop thinking.

Why filtering matters · a concrete example

The teacher dataset contains 12,000 math problems with DeepSeek-R1's responses. But not all responses are usable. Two filters are applied:

Filter 1: Correctness. DeepSeek-R1 achieves approximately 91% accuracy on MATH-500. This means roughly 9% of its responses contain incorrect final answers. If we train the student on these incorrect responses, we teach it to reproduce the teacher's mistakes.

Consider a concrete case. The problem: "Find the sum of all integers n such that n² - 11n + 24 ≤ 0." The correct answer is 21 (n can be 3, 4, 5, 6, 7, 8; sum = 3+4+5+6+7+8 = 33... actually let me recalculate: factor n²-11n+24 = (n-3)(n-8), so n ∈ [3,8], sum = 3+4+5+6+7+8 = 33). Suppose DeepSeek-R1 incorrectly factors the quadratic and arrives at 28. If we train the student on this response, the student learns the wrong factoring technique.

The fix: use the Chapter 3 verification pipeline to check every teacher response against the ground truth. Discard responses where the teacher got it wrong. This typically removes 8-10% of the dataset.

Filter 2: Length. With max_len=2048 tokens:

def filter_examples_by_max_len(examples, max_len=2048):
    filtered = [s for s in examples if len(s["token_ids"]) <= max_len]
    print(f"Original: {len(examples)}")     # 12000
    print(f"Filtered: {len(filtered)}")     # 6695
    print(f"Removed:  {len(examples) - len(filtered)}")  # 5305
    return filtered

44% of examples are removed! DeepSeek-R1 generates extremely long reasoning traces for harder problems, often exceeding 4,000 tokens. A max_len of 4,096 would retain more examples but roughly double memory usage and training time.

The removed examples tend to be the hardest problems, meaning the student never sees the teacher's approach to advanced topics. This is a significant limitation. A student trained on the filtered dataset learns to handle moderate-difficulty problems well but has no exposure to the teacher's strategies for hard problems. In practice, teams with more GPU memory use max_len=4096 or even 8192 to retain more examples.

After filtering, length statistics on the remaining 6,695 examples: average 1,180 tokens, shortest 236 tokens, longest 2,048 tokens. The dataset is split into 6,670 training and 25 validation examples. The validation set is deliberately small to avoid slowing down the training loop.

The loss function · teaching the answer, not the question

A subtle but important design choice: the cross-entropy loss is computed only on the answer tokens, not on the prompt tokens. The model is not rewarded for "predicting" the question (it already has the question as input). It is rewarded only for generating the correct reasoning trace and answer.

This is implemented through a loss mask. Each training example is tokenized into a single sequence, and the tokens corresponding to the prompt (everything up to and including the <|im_start|>assistant\n marker) are masked out. Only the tokens in the assistant's response contribute to the loss.

def compute_example_loss(model, example, device):
    """Compute cross-entropy loss on answer tokens only."""
    input_ids = example["input_ids"].to(device)
    target_ids = example["target_ids"].to(device)
    loss_mask = example["loss_mask"].to(device)

    logits = model(input_ids.unsqueeze(0))

    # Shift: predict position t+1 from position t
    shift_logits = logits[0, :-1, :]
    shift_targets = target_ids[1:]
    shift_mask = loss_mask[1:]

    # Cross-entropy on masked positions
    loss = F.cross_entropy(shift_logits, shift_targets, reduction='none')
    masked_loss = (loss * shift_mask).sum() / shift_mask.sum()

    return masked_loss

The loss function is the same cross-entropy loss used in standard language model training. The "distillation" is not in the loss function; it is in the data. The training targets are teacher-generated tokens instead of human-written tokens. Everything else about the training procedure, the optimizer, the gradient computation, the weight updates, is standard supervised learning.

Future directions · where reasoning models go from here

The techniques in this book represent the state of the art as of early 2026, but the field is evolving rapidly. Four directions are particularly promising:

1. Flexible inference budgets. Instead of fixed compute per query, future systems will dynamically allocate reasoning effort based on problem difficulty. OpenAI's GPT-5 "auto" mode is an early example: the system automatically determines whether a question requires extended reasoning (allocating more tokens and compute) or can be answered quickly. This moves the inference-scaling decision from the user ("should I use CoT?") to the system ("how hard is this problem?").

2. Process reward models. Current RLVR uses outcome-based rewards: correct/incorrect final answer. Process reward models (PRMs) evaluate intermediate reasoning steps, providing denser feedback. Instead of waiting until the end to learn whether the answer was right, the model gets feedback on each step: "This algebraic manipulation was valid" or "This substitution introduced an error." PRMs could materially improve training efficiency, but they are difficult to build reliably (who labels intermediate steps as correct or incorrect?). DeepSeek-R1 notably did not use PRMs.

3. Agent applications. Reasoning is essential for AI agents that must plan multi-step workflows, call tools, recover from failures, and coordinate tasks. An agent that books a complex multi-leg flight needs to reason about connections, layovers, baggage rules, and ticket prices across multiple APIs. Current reasoning models excel at single-turn problems (one question, one answer). Extending them to multi-turn, tool-using agent workflows is an active research area.

4. Multi-modal reasoning. Current reasoning models process text. Future models will reason across text, images, code, and structured data simultaneously. A model that can look at a geometry diagram, read the problem statement, write equations, and compute the answer would be far more capable than one that processes only text descriptions of diagrams.

These directions all build on the foundations covered in this book. The evaluation pipeline from Chapter 3 will need to extend to new domains. The inference-time techniques from Chapters 4-5 will need to adapt to agent workflows. The RL training from Chapters 6-7 will need to handle multi-modal rewards. And distillation from Chapter 8 will need to transfer capabilities across modalities.

The principles, however, remain constant: measure before optimizing, build from scratch to understand, and combine training-time and inference-time techniques for maximum performance.

The training loop

The training loop is the simplest in the book. It is a standard supervised learning loop with shuffling, gradient computation, and periodic validation:

def train_distillation(model, train_examples, val_examples, device,
                        epochs=2, lr=5e-6):
    optimizer = torch.optim.AdamW(model.parameters(), lr=lr)

    for epoch in range(epochs):
        random.shuffle(train_examples)
        model.train()

        for i, example in enumerate(train_examples):
            loss = compute_example_loss(model, example, device)
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()

            if i % 50 == 0:
                val_loss = evaluate_examples(model, val_examples, device)
                print(f"[{epoch+1}/{epochs}] Step {i}: val_loss={val_loss:.4f}")

No rollout generation. No reward computation. No advantage normalisation. No policy gradient. Just: compute loss, backpropagate, update. The complexity is in the data preparation, not in the training algorithm.

Training for 2-3 epochs on the ~6,670 training examples takes about 3 hours on a DGX Spark using 15 GB of RAM. Compare this to GRPO, which took about 12 hours and 70 GB for just 50 steps. Distillation is roughly 4x faster and requires 5x less memory, because it does not need to generate multiple rollouts per training step.

The results · two teachers, two outcomes

The book evaluates distillation with two different teachers:

DeepSeek-R1 (cross-family): The teacher and student are from different model families with different tokenizers. The teacher's reasoning traces are re-tokenized using the student's tokenizer, which introduces some friction. After 3 epochs of training, the student reaches 33.6% accuracy on MATH-500.

Qwen3 235B-A22B (same-family): The teacher and student are from the same Qwen3 family with the same tokenizer. The teacher's reasoning conventions, output formatting, and token distributions are naturally aligned with the student. After 1 epoch of training, the student reaches 45.0% accuracy on MATH-500.

The difference, 33.6% versus 45.0%, is striking and has a clear explanation. When the teacher and student share a tokenizer, the teacher's reasoning traces are expressed in the student's native vocabulary. The <think> and </think> tokens, the formatting conventions, the subword boundaries, everything aligns perfectly. The student does not need to "translate" the teacher's style; it just needs to learn the teacher's strategies.

For context, the GRPO-trained model from Chapter 6 reached 47.4%. Same-family distillation is nearly as effective at a fraction of the cost. Cross-family distillation is less effective but still more than doubles the base model's accuracy.

Neither the cross-family nor same-family distilled student matches the teacher. DeepSeek-R1 achieves 91.2% on MATH-500, and Qwen3 235B-A22B achieves 92.4%. But recovering 45 percentage points of the teacher's 77-percentage-point advantage (from 15.2% to 92.4%) with a model that is 400x smaller is a remarkable transfer of capability.

Closer inspection · why same-family distillation wins

The 11.4 percentage point gap between cross-family distillation (DeepSeek-R1 → Qwen3 0.6B: 33.6%) and same-family distillation (Qwen3 235B → Qwen3 0.6B: 45.0%) has three root causes, each worth understanding in detail.

Cause 1: Tokenizer alignment. DeepSeek-R1 and Qwen3 use different BPE vocabularies. The same mathematical expression "\frac{14}{3}" might tokenize as 6 tokens in DeepSeek's vocabulary and 8 tokens in Qwen3's. When we re-tokenize the teacher's response using the student's tokenizer, the token boundaries shift. A token that was atomic in the teacher's vocabulary might be split into subwords in the student's, or vice versa. These boundary mismatches create noise in the training signal: the student is learning to predict token boundaries that do not align with the teacher's natural generation pattern.

Cause 2: Response style. Different model families develop characteristic response styles during pre-training. DeepSeek-R1 might write "Let me work through this step by step" while Qwen3 models typically write "We can solve this as follows." These stylistic differences are baked into every response in the dataset. The cross-family student must simultaneously learn reasoning patterns AND adapt to an unfamiliar style. The same-family student can focus entirely on reasoning because the style already matches its pre-training distribution.

Cause 3: Mathematical notation. Different models format mathematics differently. One might write \frac{14}{3} where another writes 14/3 or \dfrac{14}{3}. These are semantically identical but tokenize differently and occupy different positions in the vocabulary's probability space. The student trained on same-family data sees notation that is already familiar from pre-training, reducing the learning burden.

The practical implication: when choosing a teacher model for distillation, prefer models from the same family as your student, even if a cross-family model is slightly more capable. The alignment benefit typically outweighs a 5-10% capability gap. If you must use a cross-family teacher, consider post-processing the teacher's responses to match the student's notation and style conventions.

The practical sequence, fully specified

We now have all the pieces to state the complete practical sequence for building a reasoning model:

Step 1: Choose your base model. Select the largest model you can afford to serve. Reasoning techniques are more effective on larger models (they have more latent knowledge to expose). Qwen3 0.6B is for learning; production deployments typically use 7B-70B models.

Step 2: Distill first. Generate reasoning traces from the best available teacher (preferably same-family). Filter for correctness and length. Train the student for 1-2 epochs with answer-only cross-entropy loss. Cost: low (one-time teacher API cost + a few hours of training). Expected improvement: 2-3x accuracy over the base model.

Step 3: Refine with GRPO. Starting from the distilled checkpoint (not the base model), run GRPO training with verifiable rewards. The distilled model already knows how to generate reasoning traces, so GRPO can focus on refining accuracy rather than discovering reasoning from scratch. Use clipping (eps=0.2), no KL for math/code tasks, 4+ rollouts per step, and frequent evaluation. Cost: moderate (12-24 hours on a single GPU for a 0.6B model; proportionally more for larger models). Expected improvement: additional 5-15 percentage points over distillation alone.

Step 4: Deploy with inference-time scaling. At serving time, apply CoT prompting when the task evidence supports it and self-consistency (when latency budget allows). The trained model responds to CoT prompting more effectively than the base model because it has internalised reasoning patterns. Even n=3 self-consistency provides meaningful accuracy gains.

Step 5: Monitor and iterate. Track accuracy, response length, and latency in production. Periodically retrain on new data or problems the model fails on. Use the evaluation pipeline from Chapter 3 to measure each iteration.

This recipe was validated by DeepSeek-R1's training methodology (distillation + RL), by Google's Gemini reasoning variants (multi-stage training), and by the experiments in this book. The order matters: distillation provides a warm start that makes RL much more efficient, and inference-time scaling extracts maximum performance from the trained model.

Distillation versus reinforcement learning

The three techniques we have explored, inference-time scaling, GRPO, and distillation, form a practical toolkit for reasoning model development. Each has its sweet spot.

Inference-time scaling is the right choice when you cannot afford training, need immediate improvement, and can tolerate higher per-query costs. It is the only option that works with frozen models.

Distillation is the right choice when you have access to a capable teacher model, want a cheap and fast training pipeline, and are willing to accept performance that is below the teacher but far above the base model. It is especially effective for small student models, where the complexity of RL can be disproportionate to the model's capacity.

GRPO is the right choice when you need the highest possible performance from a given model size, do not have access to a suitable teacher model, or want the model to discover novel reasoning strategies rather than imitating a teacher. It is more expensive and more fragile than distillation, but it can surpass distillation when well-tuned.

The DeepSeek-R1 paper showed that for small models (under 7B parameters), distillation actually outperforms RL. The reasoning is intuitive: a small model has limited capacity to discover effective reasoning strategies through trial and error, but it has enough capacity to imitate strategies demonstrated by a capable teacher. For large models, RL becomes more effective because the model has the capacity to discover strategies that even the teacher might not use.

The most effective approach in production combines both: distill first (cheap, provides a warm start), then refine with RL (expensive, provides targeted improvement). This two-stage sequence is a useful experiment when both trace data and verifiable rewards are available.

The question "How long should I train?" has a counterintuitive answer for distillation: not very long. The best Qwen3 235B distillation accuracy (45.0%) occurs at epoch 1, and degrades to 44.2% by epoch 3 despite the validation loss continuing to decrease. This disconnect between validation loss and evaluation accuracy is a common pattern in distillation and deserves explanation.

Validation loss measures how well the model predicts the teacher's exact tokens. It always decreases with more training because the model is memorising the teacher's specific word choices. But evaluation accuracy measures whether the model produces correct answers on unseen problems. These are different things. The teacher might write "Let's compute" where "We calculate" would be equally correct. Memorizing "Let's compute" reduces validation loss but does not improve reasoning capability.

This is overfitting to surface form rather than reasoning content. The student learns the teacher's tics, preferences, and phrasings rather than the underlying mathematical strategies. The first epoch captures the reasoning patterns (the big accuracy jump from 15.2% to 45.0%). Subsequent epochs capture the stylistic details (diminishing or negative returns on accuracy).

The practical rule: always evaluate checkpoints on a held-out benchmark and select the best one. For this fixture, the best checkpoint appeared early; another dataset or optimiser may peak elsewhere.

The thread

We have completed the four-stage journey that this book set out to follow. Stage 1 loaded a base model and implemented text generation. Stage 2 built the evaluation pipeline. Stage 3 explored inference-time techniques that improve reasoning without changing weights. Stage 4 trained reasoning into the model through reinforcement learning and distillation.

The base model started at 15.2% accuracy on MATH-500. Inference-time techniques raised it past 30%. GRPO raised it to 47.4%. Same-family distillation reached 45.0%. The official Qwen3 0.6B reasoning model, trained by a professional team at Alibaba, scores 48.2%.

Every technique in this book is implemented in raw PyTorch, with no third-party LLM libraries. The code is readable, modifiable, and runnable on consumer hardware. The concepts, from tokenization to text generation, from evaluation to GRPO, from log-probabilities to cross-entropy distillation, form a connected chain where each link depends on and reinforces the ones before it.

The field of reasoning models is moving fast. In the months since DeepSeek-R1, more than 15 GRPO modifications have been proposed, new distillation recipes have been developed, and the boundary between inference-time and training-time techniques continues to blur. But the fundamentals, the ones covered in this book, remain stable: generate, evaluate, improve. Whether the improvement comes from prompting, sampling, reinforcement, or imitation, the underlying loop is the same.

You now have the tools to understand, implement, and extend these fundamentals. The next generation of reasoning models, whatever form they take, will be built on the foundations you have just learned.

The 11.4 percentage point gap between cross-family (33.6%) and same-family (45.0%) distillation deserves explanation. When the teacher is DeepSeek-R1 and the student is Qwen3 0.6B, three friction points emerge.

First, tokenizer mismatch. DeepSeek-R1 and Qwen3 use different tokenizers with different vocabularies. The teacher's response "Let me compute \frac{14}{3}" might tokenize as 8 tokens in DeepSeek's vocabulary but 10 tokens in Qwen3's. The student sees a different token sequence than what the teacher produced, creating subtle misalignment in the training signal.

Second, style mismatch. Different model families develop different response styles during pre-training. DeepSeek-R1 might write "Let's approach this step by step" while Qwen3 models typically write "We can solve this as follows." The student must simultaneously learn reasoning patterns AND translate between styles, splitting its limited capacity.

Third, formatting mismatch. The teacher's use of LaTeX, whitespace, and structural markers may differ from what the student's tokenizer and pre-training have prepared it for. Even small formatting differences compound across thousands of training examples.

Same-family distillation eliminates all three friction points. The Qwen3 235B teacher and Qwen3 0.6B student share the same tokenizer, the same response conventions, and the same formatting habits. The student's entire capacity can focus on learning reasoning patterns rather than translating between idioms.

This finding has a practical implication: when choosing a teacher model for distillation, prefer models from the same family as your student, even if a cross-family model is slightly more capable. The alignment benefit outweighs the capability gap.

The Raphael principle

In 1503, a young painter named Raphael arrived in Florence. He had talent, but he was not yet Raphael. What transformed him was not solitary practice. It was watching Leonardo da Vinci and Michelangelo work. He studied their brushstrokes, compositions, and treatment of light. He did not copy them mechanically. He absorbed their techniques and synthesized them into his own style. Four years later, he was one of the greatest painters who ever lived.

Distillation follows the same principle. The student model does not match the teacher's full range of capabilities, just as a 600-million-parameter model cannot match a 671-billion-parameter one. But it can learn the teacher's strategies, and those strategies make it far more capable than it would be on its own.

The DeepSeek-R1 team demonstrated this at scale. They distilled their 671B teacher into models ranging from 1.5B to 70B parameters. The distilled 7B model outperformed models of the same size that were trained with RL alone. The teacher provides a richer learning signal than RL: instead of binary correct/incorrect feedback, the student receives the complete reasoning trace, showing not just whether an answer is right but exactly how to arrive at it.

Why the training loop is boring, and that is the point

The distillation training loop is standard supervised learning. Shuffle the examples, compute cross-entropy loss, backpropagate, update weights, check validation loss, save checkpoints. If you have trained any neural network with PyTorch, you have written this loop before.

This is the point. Distillation reuses the most well-understood, best-tested training paradigm in deep learning. There are no policy gradients, no advantage computations, no rollout generation, no reward functions. The entire complexity is front-loaded: generating the teacher dataset (a one-time cost) and filtering it for quality. Once you have the dataset, training is straightforward.

The training loop terminates each epoch by saving a checkpoint and computing validation loss. The validation set is deliberately small (25 examples) to avoid slowing down training. The real test of quality is MATH-500 evaluation, which is run on saved checkpoints after training completes.

The fact that the best accuracy (45.0%) occurs at epoch 1, not epoch 3, is a pattern seen across many distillation experiments. The student quickly learns the teacher's general reasoning patterns in the first epoch. Subsequent epochs increasingly memorize teacher-specific phrasings and structures that do not generalize. This is why early stopping and checkpoint selection are essential.

Thought experiment · what if we combined everything?

The final thought experiment that ties the entire book together. What if you applied every technique to the same base model?

Start: Base Qwen3 0.6B, 15.2% accuracy on MATH-500.

Step 1: Distill from Qwen3 235B. Accuracy jumps to 45.0%. Cost: ~$60 (API + training). Time: 3 hours.

Step 2: GRPO training starting from the distilled checkpoint (not the base model). The distilled model already generates reasoning traces, so GRPO can focus on refining accuracy rather than discovering reasoning from scratch. Expected accuracy: 50-55% (estimated, since the book does not run this exact experiment). Cost: ~$50. Time: 12 hours.

Step 3: Deploy with inference-time scaling. Apply CoT prompting (no weight update, but additional inference tokens) and self-consistency (n=5) to the GRPO-refined model. Expected accuracy: 58-65% (estimated, extrapolating from the observation that inference-time scaling compounds with training-time improvements). Per-query cost: 5x baseline.

From 15.2% to an estimated 58-65%, using a 0.6B parameter model that runs on a laptop. The total training cost: approximately $110. The total training time: approximately 15 hours.

For comparison, the official Qwen3 0.6B reasoning model achieves 48.2% without inference-time scaling and 55.2% with self-consistency (n=3). The combined recipe would likely match or exceed these numbers because it combines the same techniques (distillation + RL) that the Qwen3 team used, plus inference-time scaling.

This combined approach is not theoretical. It is the recipe used by every major reasoning model: DeepSeek-R1, Google Gemini reasoning variants, Anthropic's Claude reasoning capabilities. The order matters (distill → RL → inference scaling) and the details matter (which teacher, how many GRPO steps, what clip value, what self-consistency budget). This book gives you the understanding to make those decisions.

The details will change as the field evolves. The principles will not. Evaluation must precede optimisation. Training-time improvements compound with inference-time scaling. Simpler algorithms with careful engineering beat complex algorithms with sloppy engineering. And building from scratch is one way to develop the intuition that makes these principles actionable.

The complete toolkit

With distillation complete, the book's entire toolkit is assembled. Three pillars of reasoning model development, each with different strengths:

Inference-time scaling requires no training. It is the cheapest to deploy and works on any model. Best result: 52.0% with CoT + SC (n=10). Limitation: high per-query compute cost (85x baseline).

Reinforcement learning modifies weights through trial and error. Best result: 47.4% with 50 GRPO steps. Limitation: computationally expensive training (12h, 70GB), requires careful stabilization.

Distillation modifies weights by imitating a teacher. Best result: 45.0% with Qwen3 235B teacher. Limitation: requires access to a strong teacher model, but training is cheap (3h, 15GB).

The practical sequence layers all three: distill first (give the model a strong foundation of reasoning patterns), refine with RL (push beyond what imitation alone can achieve using verifiable rewards), apply inference-time scaling at serving time (squeeze maximum accuracy from the deployed model). Public research has explored versions of this layered sequence.

Future directions include flexible inference budgets (the system decides reasoning effort per query), process rewards (evaluate intermediate reasoning steps, not just final answers), and multi-objective training for agent applications where reasoning models must plan, call tools, and recover from failures.

The details will change. The principles will not.

Treat two approaches to learning piano from a master. In hard distillation, you listen to the master's recordings and learn to reproduce them note for note. You hear the final performance, the phrasing, the dynamics. The master does not need to be in the room. You just need the recordings.

In soft distillation, the master sits beside you and shows you not just which notes to play, but which notes they considered playing and rejected. "I almost used a diminuendo here, but the crescendo felt more natural." This richer information helps you internalize the master's decision-making process.

For LLM reasoning, hard distillation is overwhelmingly more common for three reasons. First, teacher logits (the "rejected notes") are usually inaccessible: proprietary models expose text but not probability distributions. Second, storing full distributions (151,936 floats per token position) for thousands of long reasoning traces would cost terabytes. Third, different model families use different tokenizers, so the teacher's probability for "Berlin" as token 19,826 is meaningless to a student with a different vocabulary mapping.

The economics are transformative. GRPO requires the model to generate its own rollouts: for each training problem, generate 4 candidate solutions, verify each one, compute advantages, and backpropagate. For Qwen3 0.6B, 50 GRPO steps takes ~12 hours and ~70 GB GPU memory. Distillation separates the expensive part (generating teacher data) from training. The teacher dataset costs ~$50 in DeepSeek-R1 API calls, a one-time cost. Training then takes ~3 hours and ~15 GB. The dataset can be reused for unlimited experiments.

The complete results table reveals three insights that are not obvious from any single row:

Insight 1: Same-family advantage. Rows 3-5 vs 6-8 show that same-family distillation (Qwen3 235B → Qwen3 0.6B) materially outperforms cross-family (DeepSeek-R1 → Qwen3 0.6B): 45.0% vs 33.6%. The likely explanation is tokenizer and response-style alignment. When teacher and student share the same tokenizer, the training signal is cleaner.

Insight 2: More training is not always better. The best Qwen3 235B accuracy (45.0%) occurs at epoch 1, not epoch 3 (44.2%). The model is overfitting to teacher-specific patterns rather than learning generalizable reasoning. In production: always evaluate checkpoints and select the best one.

Insight 3: Neither distilled model matches its teacher. DeepSeek-R1 achieves 91.2% on MATH-500. Qwen3 235B achieves 92.4%. The best distilled student (45.0%) recovers only about half the teacher's accuracy. But using a larger student (Qwen3 4B or 30B parameters) would close this gap significantly, as the DeepSeek-R1 paper demonstrated.

The complete cross-book comparison:

The practical sequence: distill first (cheap warm start), refine with RL (targeted improvement), apply inference-time scaling at serving time (maximum accuracy per query).

In 2015, Geoffrey Hinton published "Distilling the Knowledge in a Neural Network." For reasoning models, hard distillation trains a student on teacher-generated text. Soft distillation trains on the teacher's full probability distributions. Hard is overwhelmingly more common because teacher logits are rarely available, text is cheap to store, and cross-tokenizer alignment is not required.

The dataset generation process deserves detailed examination because data quality is the single largest determinant of distillation success. The process has four stages:

Stage 1: Teacher generation. Each of the 12,000 MATH problems is sent to DeepSeek-R1 via API. The teacher generates a complete response including <think> tags wrapping the reasoning trace and a final \boxed{} answer. Average response length: ~1,500 tokens. Total tokens generated: ~18 million. API cost at ~$2.75 per million output tokens: approximately $50. This is a one-time cost; the dataset can be reused indefinitely.

Stage 2: Correctness filtering. The Chapter 3 verifier checks each teacher response against the ground truth. DeepSeek-R1 achieves ~91% accuracy, so ~1,080 responses are incorrect. These are removed. Remaining: ~10,920 examples with verified correct answers.

Stage 3: Length filtering. Responses longer than 2,048 tokens are removed to fit within GPU memory during training. This is the most aggressive filter: it removes 44% of the remaining examples, leaving 6,695. The removed examples tend to be the hardest problems with the longest reasoning traces. This creates a bias in the training data toward easier problems with shorter solutions. A practitioner with more GPU memory should increase max_len to 4,096 or 8,192 to retain harder examples.

def filter_examples_by_max_len(examples, max_len=2048):
    filtered = [s for s in examples if len(s["token_ids"]) <= max_len]
    print(f"Original: {len(examples)}")     # ~10920
    print(f"Filtered: {len(filtered)}")     # 6695
    print(f"Removed:  {len(examples) - len(filtered)}")   # ~4225
    return filtered

Stage 4: Train/validation split. The filtered examples are shuffled with a fixed random seed (for reproducibility) and split into 6,670 training and 25 validation examples. The validation set is deliberately small (25) to minimize the cost of periodic evaluation during training. The real quality metric is MATH-500 evaluation on saved checkpoints.

Length statistics after filtering: average 1,180 tokens, shortest 236 tokens, longest 2,048 tokens. The median is approximately 1,050 tokens, meaning most reasoning traces are 3-4 paragraphs of dense mathematical derivation. The shortest examples (236 tokens) are typically simple problems where the teacher's reasoning trace is concise. The longest (2,048 tokens) are at the cutoff boundary; many responses just above this cutoff were removed.

Generating the dataset

The teacher (DeepSeek-R1, 671B) generates reasoning traces for 12,000 MATH problems. Cost: ~$50 via API. The dataset is filtered (only correct answers) and length-filtered:

def filter_examples_by_max_len(examples, max_len=2048):
    filtered = [s for s in examples if len(s["token_ids"]) <= max_len]
    print(f"Original: {len(examples)}")     # 12000
    print(f"Filtered: {len(filtered)}")     # 6695
    print(f"Removed:  {len(examples) - len(filtered)}")   # 5305
    return filtered

44% of examples are removed because DeepSeek-R1's traces exceed 2048 tokens for harder problems. The dataset is split into 6,670 training and 25 validation examples.

Each example is formatted with the reasoning tokenizer's chat template:

def format_example(example, tokenizer):
    problem = example["problem"]
    response = example["response"]
    prompt_tokens = tokenizer.apply_chat_template(problem)
    response_tokens = tokenizer.encode(response)
    return {
        "token_ids": prompt_tokens + response_tokens,
        "prompt_len": len(prompt_tokens)
    }