The model remains autoregressive

For a prompt $x$ and response tokens $y_1,\ldots,y_T$, a causal language model factorises the response probability as

$$\pi_\theta(y\mid x) = \prod_{t=1}^{T} \pi_\theta(y_t\mid x,y_{<t}).$$

The training objective normally minimises negative log-likelihood over selected target tokens:

$$\mathcal{L}_{\mathrm{NLL}} = -\frac{1}{\sum_t m_t} \sum_{t=1}^{T} m_t\log \pi_\theta(y_t\mid x,y_{<t}),$$

where $m_t\in\{0,1\}$ masks prompt, padding or otherwise excluded tokens.

Longer rationales give the decoder more sequential computation. Each generated token becomes part of the context for the next step. That can help the model decompose a problem, record an intermediate value or recover from an earlier guess. It can also create more places to make an error. The mechanism does not guarantee logical validity.

Chain-of-thought prompting improved multi-step benchmark results in the experiments reported by Wei and peers, and zero-shot prompts that invited intermediate work also improved several studied tasks.1 Those results are empirical and model-dependent. They do not establish that a particular phrase will improve every model, language or task.

Visible work is an artefact, not privileged telemetry

A rationale can serve three useful purposes:

1. it supplies additional context for later tokens;
2. it gives a verifier more structure to inspect; and
3. it provides a human with an explanation candidate.

Those purposes are different. A rationale may lead to the right answer through a faulty step. It may cite a correct rule after the answer was determined by an unrelated cue. It may omit the decisive factor. Experiments have shown that generated explanations can be influenced by features they do not acknowledge.2

For this reason, the book distinguishes four objects:

An application may retain a concise calculation trace while withholding or discarding unrestricted scratch text. The retained record should be designed for the task rather than treated as a transcript of a mind.

A taxonomy based on checkability and consequence

“Reasoning” covers tasks with very different feedback. Two dimensions are especially useful:

• checkability: how cheaply and reliably an outcome can be tested; and
• consequence: the cost of accepting an incorrect result.

The four broad regions lead to different designs:

Five levels of computational work

A useful task ladder separates operations that are often blended together:

1. Recall: reproduce a learned association.
2. Local transformation: rewrite, classify or extract within a bounded input.
3. Composition: combine several stated facts or operations.
4. Search: explore alternatives, backtrack or sample candidates.
5. Grounded action: use tools and evidence under permissions to change external state.

The ladder is not an intelligence scale. A calculator may outperform a language model on arithmetic while doing less linguistic work. A short retrieved answer may be safer than an elaborate generated derivation. Route by task structure.

The task contract comes before the prompt

Before choosing a model, write an executable or reviewable contract. The following data class captures the minimum:

from __future__ import annotations

from dataclasses import dataclass
from typing import Callable, Literal


Outcome = Literal["accept", "reject", "review"]


@dataclass(frozen=True)
class TaskContract:
    name: str
    version: str
    max_input_tokens: int
    max_output_tokens: int
    verifier_name: str
    consequence: Literal["low", "medium", "high"]
    on_verifier_error: Outcome = "review"
    on_parse_error: Outcome = "review"


@dataclass(frozen=True)
class Verification:
    outcome: Outcome
    score: float | None
    reason_code: str
    verifier_version: str


Verifier = Callable[[str, str], Verification]

The contract should also state:

• the permitted evidence sources;
• the accepted answer representation;
• normalisation rules;
• abstention conditions;
• the cost and latency budget;
• the person or system allowed to approve an action;
• the record retained for audit; and
• the test slices that must not regress.

Prompt text is then one implementation detail of the contract.

Baselines prevent expensive self-deception

Every experiment needs a cheap baseline. For a reasoning pipeline, useful baselines include:

• direct answer with greedy decoding;
• direct answer with sampling;
• explicit worked answer with greedy decoding;
• $N$ sampled candidates with majority vote;
• $N$ sampled candidates selected by a deterministic verifier;
• the same methods with an external calculator or code executor; and
• a non-generative rule or retrieval system.

If a calculator solves the task exactly, it is the correct baseline and may be the correct production component. If retrieval plus a template meets the requirement, adding GRPO is engineering theatre.

The comparison unit is not model accuracy alone:

$$\text{utility} = v_{\mathrm{correct}}p_{\mathrm{correct}} -c_{\mathrm{wrong}}p_{\mathrm{wrong}} -c_{\mathrm{abstain}}p_{\mathrm{abstain}} -c_{\mathrm{compute}} -c_{\mathrm{review}}.$$

The values and costs must be supplied by the application owner. A high-consequence task can make a small false-accept rate dominate the equation.

Route compute instead of applying it uniformly

Reasoning tokens, parallel samples and verifier calls all cost time and money. Easy items often need none of them. Very hard items may not improve with more sampling because the base model rarely enters a correct region.

A first router can be deterministic:

from dataclasses import dataclass
from typing import Literal


Route = Literal["direct", "sample_and_verify", "human_review"]


@dataclass(frozen=True)
class RouteFeatures:
    consequence: Literal["low", "medium", "high"]
    deterministic_verifier: bool
    input_complete: bool
    estimated_difficulty: float  # calibrated to [0, 1]


def choose_route(features: RouteFeatures) -> Route:
    if features.consequence not in {"low", "medium", "high"}:
        raise ValueError("unknown consequence")
    if (
        type(features.deterministic_verifier) is not bool
        or type(features.input_complete) is not bool
    ):
        raise TypeError("router flags must be booleans")
    if not 0.0 <= features.estimated_difficulty <= 1.0:
        raise ValueError("estimated_difficulty must lie in [0, 1]")
    if not features.input_complete or features.consequence == "high":
        return "human_review"
    if features.deterministic_verifier and features.estimated_difficulty >= 0.35:
        return "sample_and_verify"
    return "direct"

The caller maps a router validation exception to review rather than to the direct path. The thresholds are placeholders. Estimate them on held-out traffic, include confidence intervals and monitor drift. A router trained on benchmark difficulty may fail on production ambiguity, missing context or adversarial inputs.

Small models can teach the system, even when they cannot solve the domain

A sub-billion-parameter checkpoint is useful for learning the mechanics:

• tokenisation and chat templates are inspectable;
• inference can run on accessible hardware;
• rollout batches are small enough to profile;
• reward bugs become visible quickly; and
• overfitting appears within a manageable experiment.

It is not evidence that the same checkpoint is suitable for every application. A small model can learn answer formatting or a narrow arithmetic pattern while lacking the knowledge and reliability required for a broader domain.

The reference Qwen3 checkpoint illustrates another reproducibility lesson. Its repository label says “0.6B”, while its current model metadata reports 751,632,384 tensor parameters. Published names often encode a family tier rather than an exact tensor count. Capacity and memory calculations should use the checkpoint, configuration and dtype actually loaded.

A governed banking example

Consider a fictional credit analyst reviewing a facility agreement and two amendments. The task is to identify the effective leverage covenant and compare a supplied ratio with the threshold.

The language model is allowed to:

• retrieve clauses from the authorised document set;
• propose which clause supersedes another;
• extract candidate numerator, denominator and threshold fields; and
• draft a short explanation with source references.

It is not allowed to:

• choose an unauthorised document;
• invent a missing value;
• calculate the ratio in free text;
• approve or decline credit; or
• write to a system of record.

A deterministic service resolves effective dates, validates units, calculates the ratio and compares it with the approved threshold. Ambiguity, missing evidence or conflicting clauses produce review. An authorised analyst owns the conclusion.

This is a reasoning system because it may retrieve, compare, extract and explain across several steps. Its safety comes from the boundary around those steps.

Build check

Before continuing, a project should be able to answer:

1. What exact object is graded?
2. Which outcomes are accept, reject and review?
3. Can the verifier fail closed?
4. What is the direct baseline?
5. Which per-item records are retained?
6. Which action remains under human authority?

If those answers are absent, more model training will make the system harder to diagnose.

Chapter notes

Pin the repository, not the marketing tier

The examples use:

repository: Qwen/Qwen3-0.6B
revision:   c1899de289a04d12100db370d81485cdf75e47ca
retrieved:  2026-07-28

The official configuration at that revision reports:

The query projection width is $16\times128=2{,}048$, larger than the residual width. The grouped key and value projections each have width $8\times128=1{,}024$. Architecture code should read head_dim rather than assuming hidden_size / num_attention_heads.

At BF16, the parameter tensors alone occupy approximately

$$751{,}632{,}384\times2 =1{,}503{,}264{,}768\text{ bytes} \approx1.40\text{ GiB}.$$

Inference also allocates activations, attention workspaces, temporary logits and a KV cache. Training adds gradients, optimiser states and usually higher-precision copies. A model that fits for inference may still be far from fitting for full-parameter training.

Load with explicit trust and precision decisions

The following loader uses the published Transformers integration and refuses implicit code execution from the model repository:

from __future__ import annotations

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer


MODEL_ID = "Qwen/Qwen3-0.6B"
REVISION = "c1899de289a04d12100db370d81485cdf75e47ca"


def load_reference_model(device: torch.device):
    tokenizer = AutoTokenizer.from_pretrained(
        MODEL_ID,
        revision=REVISION,
        trust_remote_code=False,
    )
    dtype = torch.bfloat16 if device.type in {"cuda", "mps"} else torch.float32
    model = AutoModelForCausalLM.from_pretrained(
        MODEL_ID,
        revision=REVISION,
        torch_dtype=dtype,
        use_safetensors=True,
        trust_remote_code=False,
    ).to(device)
    model.eval()
    return model, tokenizer

In an offline or controlled environment, mirror the exact repository revision, verify file hashes and load from the approved mirror. A Git commit pins repository state; it does not by itself establish software supply-chain trust.

The chat template is executable configuration

The model does not receive a Python list of messages. It receives tokens. The tokenizer’s chat template converts roles, content and generation markers into a particular token sequence.

def encode_user_prompt(tokenizer, prompt: str, *, thinking: bool):
    messages = [{"role": "user", "content": prompt}]
    token_ids = tokenizer.apply_chat_template(
        messages,
        tokenize=True,
        add_generation_prompt=True,
        enable_thinking=thinking,
        return_tensors="pt",
    )
    return token_ids

Qwen3’s published template supports thinking and non-thinking modes.1 Those modes change the formatted prefix and expected response form. They are part of the evaluation condition.

Log the template hash or complete rendered prefix. A repository update that changes only template text can change an experiment without changing the weight tensors.

Inspect the rendered representation during setup:

def inspect_template(tokenizer, prompt: str, *, thinking: bool) -> dict:
    ids = encode_user_prompt(tokenizer, prompt, thinking=thinking)[0]
    return {
        "num_tokens": int(ids.numel()),
        "first_ids": ids[:12].tolist(),
        "last_ids": ids[-12:].tolist(),
        "rendered": tokenizer.decode(ids, skip_special_tokens=False),
    }

Do not publish sensitive prompts in debug logs. In production, record a template version and token counts while applying the organisation’s data-retention policy.

What one decoder layer does

Let the residual stream entering a layer be $X\in\mathbb{R}^{B\times T\times D}$. A pre-normalised decoder layer applies:

$$X' = X + \operatorname{Attention}(\operatorname{RMSNorm}(X)),$$

$$X'' = X' + \operatorname{MLP}(\operatorname{RMSNorm}(X')).$$

For grouped-query attention,

$$Q\in\mathbb{R}^{B\times H_q\times T\times D_h},$$

$$K,V\in\mathbb{R}^{B\times H_{kv}\times T\times D_h}.$$

With $H_q=16$ and $H_{kv}=8$, each key/value head serves two query heads. After applying rotary position information to queries and keys, attention uses a causal mask:

$$\operatorname{softmax}\left( \frac{QK^\top}{\sqrt{D_h}} + M \right)V,$$

where $M_{ij}=0$ when key position $j$ is visible to query position $i$, and a sufficiently negative value otherwise.

Attention maps are internal routing weights. They are useful for shape debugging and some mechanistic analyses, but they do not form a complete causal explanation of an answer.

RoPE changes queries and keys, not token order

Rotary position embedding applies position-dependent rotations to pairs of query and key coordinates.2 For one two-dimensional pair at position $m$,

$$\begin{bmatrix} x'_{2i}\\ x'_{2i+1} \end{bmatrix} = \begin{bmatrix} \cos(m\omega_i) & -\sin(m\omega_i)\\ \sin(m\omega_i) & \cos(m\omega_i) \end{bmatrix} \begin{bmatrix} x_{2i}\\ x_{2i+1} \end{bmatrix}.$$

This makes the query-key inner product depend on relative displacement. It does not expand the usable context without limit. Training distribution, numerical precision, scaling method, cache implementation and evaluation all constrain long-context behaviour.

The KV cache is the dominant sequence-dependent allocation

Without caching, each generation step recomputes keys and values for the entire prefix. A cache stores them once per layer.

For a dense cache in bytes,

$$C_{\mathrm{KV}} = L\times T\times2\times H_{kv}\times D_h\times s,$$

where $s$ is bytes per cached element. For the pinned configuration at BF16:

$$28\times T\times2\times8\times128\times2 =114{,}688T\text{ bytes}.$$

That is 112 KiB per cached token per sequence. At 40,960 tokens, the arithmetic is approximately 4.375 GiB, excluding allocator overhead and implementation-specific storage. Batch size multiplies the requirement.

This estimate should be checked against measured peak allocation because some runtimes page, quantise or share cache blocks.

Logits become a policy only after decoding rules

At step $t$, the model produces logits $z\in\mathbb{R}^{V}$. Temperature $\tau>0$ gives

$$p_i = \frac{\exp(z_i/\tau)} {\sum_j \exp(z_j/\tau)}.$$

Top-$p$ sampling sorts tokens by probability and retains the smallest prefix whose cumulative probability reaches $p$. The retained probabilities are renormalised before sampling.

Greedy decoding selects $\arg\max_i z_i$. It is not equivalent to temperature zero in code because division by zero is undefined. Implement greedy selection as a separate branch.

import math

import torch


def nucleus_sample(
    logits: torch.Tensor,
    *,
    temperature: float,
    top_p: float,
    generator: torch.Generator,
) -> tuple[torch.Tensor, torch.Tensor]:
    if logits.ndim != 2 or logits.shape[0] < 1 or logits.shape[1] < 1:
        raise ValueError("expected logits shaped [batch, vocabulary]")
    if not math.isfinite(temperature) or temperature <= 0:
        raise ValueError("temperature must be greater than zero")
    if not math.isfinite(top_p) or not 0 < top_p <= 1:
        raise ValueError("top_p must lie in (0, 1]")
    if not torch.isfinite(logits).all():
        raise ValueError("logits must be finite")

    scaled_logits = logits.float() / temperature
    if not torch.isfinite(scaled_logits).all():
        raise ValueError("scaled logits must be finite")
    probs = torch.softmax(scaled_logits, dim=-1)
    sorted_probs, sorted_ids = torch.sort(probs, descending=True, dim=-1)
    cumulative = torch.cumsum(sorted_probs, dim=-1)

    remove = cumulative - sorted_probs >= top_p
    sorted_probs = sorted_probs.masked_fill(remove, 0.0)
    sorted_probs = sorted_probs / sorted_probs.sum(dim=-1, keepdim=True)

    sampled_rank = torch.multinomial(
        sorted_probs,
        num_samples=1,
        generator=generator,
    )
    sampled_ids = sorted_ids.gather(dim=-1, index=sampled_rank)
    sampling_logprobs = sorted_probs.gather(
        dim=-1,
        index=sampled_rank,
    ).log()
    return sampled_ids, sampling_logprobs

The subtraction in cumulative - sorted_probs keeps the token that crosses the threshold. Masking on cumulative > top_p without shifting can remove it.

A transparent autoregressive loop

Library generation utilities handle many edge cases. A small manual loop remains valuable for understanding stop conditions and collecting token log-probabilities:

from dataclasses import dataclass

import torch


@dataclass
class Generation:
    token_ids: list[int]
    sampling_logprobs: list[float]
    model_logprobs: list[float]
    stop_reason: str


@torch.inference_mode()
def generate_one(
    model,
    input_ids: torch.Tensor,
    *,
    eos_token_ids: set[int],
    max_new_tokens: int,
    temperature: float,
    top_p: float,
    seed: int,
) -> Generation:
    if (
        input_ids.ndim != 2
        or input_ids.shape[0] != 1
        or input_ids.shape[1] < 1
    ):
        raise ValueError("this teaching loop expects shape [1, prompt_length]")
    if max_new_tokens < 1:
        raise ValueError("max_new_tokens must be at least one")

    device = input_ids.device
    generator = torch.Generator(device=device)
    generator.manual_seed(seed)
    sequence = input_ids
    cache = None
    emitted: list[int] = []
    sampling_logprobs: list[float] = []
    model_logprobs: list[float] = []

    for _ in range(max_new_tokens):
        model_input = sequence if cache is None else sequence[:, -1:]
        outputs = model(
            input_ids=model_input,
            past_key_values=cache,
            use_cache=True,
        )
        cache = outputs.past_key_values
        logits = outputs.logits[:, -1, :]
        next_id, next_sampling_logprob = nucleus_sample(
            logits,
            temperature=temperature,
            top_p=top_p,
            generator=generator,
        )
        next_model_logprob = torch.log_softmax(
            logits.float(),
            dim=-1,
        ).gather(
            dim=-1,
            index=next_id,
        )
        token_id = int(next_id.item())
        emitted.append(token_id)
        sampling_logprobs.append(float(next_sampling_logprob.item()))
        model_logprobs.append(float(next_model_logprob.item()))
        sequence = torch.cat((sequence, next_id), dim=1)

        if token_id in eos_token_ids:
            return Generation(
                emitted,
                sampling_logprobs,
                model_logprobs,
                "eos",
            )

    return Generation(
        emitted,
        sampling_logprobs,
        model_logprobs,
        "length",
    )

sampling_logprobs records the actual temperature-scaled, top-$p$-renormalised behaviour distribution. model_logprobs records the selected token under the unmodified model softmax. They are equal only when temperature is 1 and top_p is 1. Mixing the two silently corrupts an importance ratio.

This loop is intentionally single-sequence. A production batched loop must maintain an attention_mask, mark finished rows, avoid sampling new content for finished rows, distinguish padding from stop tokens and cap total context length. Chapter 4 introduces a candidate generator with those requirements.

Compilation and warm-up need measurement

Graph compilation can reduce Python and kernel-launch overhead, but the gain depends on shapes, backend, model code and how often a compiled graph is reused. Dynamic prompt lengths can trigger recompilation. The first request also pays model loading, memory allocation and kernel initialisation costs.

Report at least:

• cold-start latency;
• first-token latency after warm-up;
• steady-state tokens per second;
• batch size and prompt/response lengths;
• peak memory;
• precision and device;
• compilation time and number of graph variants; and
• whether tokenisation and network time are included.

A speed number without those conditions is advertising, not a benchmark.

Generation invariants

Before building evaluation, test:

def assert_generation_contract(tokenizer, model) -> None:
    cfg = model.config
    assert cfg.vocab_size == len(tokenizer)
    assert cfg.num_attention_heads % cfg.num_key_value_heads == 0
    assert cfg.head_dim * cfg.num_key_value_heads > 0
    assert tokenizer.eos_token_id is not None
    assert tokenizer.pad_token_id is not None

The vocabulary assertion may need an explicit exception for models whose tokenizer contains added tokens outside the output projection. Treat such a mismatch as a documented interface decision.

Also verify:

• a fixed seed reproduces the same tokens on the same software and hardware path;
• greedy decoding is repeatable;
• an EOS token stops the row;
• the response-length cap is enforced;
• prompt tokens are not returned as answer text by mistake;
• cached and uncached logits match within the chosen tolerance; and
• left- and right-padding policies agree with the position and attention implementation.

The next chapter uses this fixed interface to build evidence about model behaviour.

Chapter notes

Define the record before running the model

A benchmark item should carry a stable identifier and provenance:

from __future__ import annotations

from dataclasses import asdict, dataclass
from typing import Any, Literal


@dataclass(frozen=True)
class EvalItem:
    item_id: str
    prompt: str
    reference_answer: str
    split: str
    source: str
    source_revision: str
    subject: str | None = None
    difficulty: str | None = None


@dataclass(frozen=True)
class EvalResult:
    item_id: str
    sample_id: int
    seed: int
    raw_response: str
    extracted_answer: str | None
    normalised_answer: str | None
    verifier_outcome: Literal["correct", "incorrect", "review", "error"]
    reason_code: str
    latency_ms: float
    prompt_tokens: int
    response_tokens: int
    verifier_version: str

    def to_jsonable(self) -> dict[str, Any]:
        return asdict(self)

Do not overwrite a raw response after fixing the parser. Re-run the parser against the preserved output and record the new parser version. That separation allows an evaluator to improve without repeating expensive generation.

Use an explicit response boundary

Searching for “the last number” is fragile. A date, equation number or confidence value can appear after the intended result. The prompt should request a narrow terminal field:

Return any concise working first.
End with exactly one line:
<final>ANSWER</final>

The extractor then accepts one well-formed final field:

import re


FINAL_PATTERN = re.compile(
    r"<final>[ \t]*(?P<answer>[^\r\n<>]*?)[ \t]*</final>",
    flags=re.IGNORECASE,
)
FINAL_TERMINAL_PATTERN = re.compile(
    r"^[ \t]*<final>[ \t]*(?P<answer>[^\r\n<>]*?)"
    r"[ \t]*</final>[ \t]*(?:\r?\n)*\Z",
    flags=re.MULTILINE | re.IGNORECASE,
)
FINAL_TAG_PATTERN = re.compile(r"</?final>", flags=re.IGNORECASE)


def extract_final(response: str) -> tuple[str | None, str]:
    matches = list(FINAL_PATTERN.finditer(response))
    if not matches:
        return None, "missing_final"
    if len(matches) != 1:
        return None, "multiple_final_fields"
    if len(FINAL_TAG_PATTERN.findall(response)) != 2:
        return None, "malformed_final_fields"

    terminal = FINAL_TERMINAL_PATTERN.search(response)
    if terminal is None:
        return None, "final_not_terminal_single_line"
    answer = terminal.group("answer").strip()
    if not answer:
        return None, "empty_final"
    if len(answer) > 512:
        return None, "final_too_long"
    return answer, "ok"

Multiple fields produce review rather than silently choosing the last. A strict contract reduces ambiguous grading and makes formatting reward easier to reason about.

Normalisation must not solve the problem

Normalisation removes representational differences that the task declares irrelevant. It must not repair a wrong derivation or infer an omitted answer.

For simple numeric answers, Python’s Fraction and Decimal avoid eval:

from decimal import Decimal, InvalidOperation
from fractions import Fraction
import re


INTEGER = re.compile(r"^[+-]?(?:\d+|\d{1,3}(?:,\d{3})+)$")
DECIMAL = re.compile(
    r"^[+-]?(?:(?:\d+|\d{1,3}(?:,\d{3})+)(?:\.\d*)?|\.\d+)$"
)
FRACTION = re.compile(r"^[+-]?\d+\s*/\s*[+-]?\d+$")


def normalise_number(text: str) -> tuple[str | None, str]:
    raw = text.strip()

    if INTEGER.fullmatch(raw):
        return str(int(raw.replace(",", ""))), "integer"

    if FRACTION.fullmatch(raw):
        numerator, denominator = raw.split("/")
        try:
            fraction = Fraction(int(numerator), int(denominator))
        except ZeroDivisionError:
            return None, "zero_denominator"
        return f"{fraction.numerator}/{fraction.denominator}", "fraction"

    if DECIMAL.fullmatch(raw):
        try:
            decimal = Decimal(raw.replace(",", ""))
        except InvalidOperation:
            return None, "invalid_decimal"
        if decimal == 0:
            return "0", "decimal"
        canonical = format(decimal, "f")
        if "." in canonical:
            canonical = canonical.rstrip("0").rstrip(".")
        return canonical, "decimal"

    return None, "unsupported_numeric_form"

The comma pattern rejects malformed groupings such as 1,2 before removing separators. Decimal text is canonicalised without calling context-sensitive arithmetic, so a long input is not rounded to the process-wide decimal precision. The normaliser deliberately does not equate 0.5 with 1/2. The verifier can convert each accepted representation to an exact Fraction:

def to_fraction(normalised: str) -> Fraction:
    if "/" in normalised:
        return Fraction(normalised)
    return Fraction(Decimal(normalised))


def verify_numeric(candidate: str, reference: str) -> tuple[str, str]:
    candidate_norm, candidate_kind = normalise_number(candidate)
    reference_norm, reference_kind = normalise_number(reference)
    if candidate_norm is None or reference_norm is None:
        return "review", f"parse:{candidate_kind}:{reference_kind}"
    if to_fraction(candidate_norm) == to_fraction(reference_norm):
        return "correct", "exact_numeric_equivalence"
    return "incorrect", "numeric_mismatch"

For symbolic mathematics, a computer-algebra system can test equivalence by simplifying a difference under stated assumptions. Run it in a restricted worker with a time limit and an allow-list of syntax. General expression parsing and simplification can consume unbounded resources or invoke unsafe features if configured carelessly.

For code, compile and run in a disposable sandbox with CPU, memory, time, filesystem and network limits. A passing public test set is still incomplete evidence; hidden and adversarial tests reduce overfitting.

Choose the verifier that matches the claim

A learned outcome reward model estimates whether a final answer is acceptable. A process reward model scores intermediate steps. Both are models with their own distribution shifts, calibration errors and exploitable features.

The process-supervision experiments reported by Lightman and peers found stronger best-of-$N$ selection on their MATH setting than outcome supervision, but they also documented imperfections in automatic final-answer grading.1 The result motivates careful step-level feedback; it does not make a process reward model an oracle.

MATH-500 is a particular held-out subset

The original MATH dataset contains competition-style problems across subjects and difficulty levels.2 The set commonly called MATH-500 came from the process-supervision work: 4,500 MATH test problems were included in reward-model training, and the remaining 500 were selected uniformly at random for evaluation. The authors reported that the subset’s subject and difficulty distribution was representative of the full test set.3

Three consequences follow:

1. identify the exact 500-item file and revision;
2. do not mix the other 4,500 items into a “held-out” claim without explaining the training relationship; and
3. do not compare scores produced by different answer normalisers as if only the model changed.

Benchmark names are not data lineage.

Measure more than top-line accuracy

For $N$ items:

$$\text{accuracy} = \frac{\#\text{correct}}{N},$$

$$\text{coverage} = \frac{\#(\text{correct or incorrect})}{N},$$

$$\text{selective accuracy} = \frac{\#\text{correct}} {\#(\text{correct or incorrect})}.$$

Coverage exposes a system that improves apparent accuracy by sending most items to review. Report the review and error rates separately.

For multiple generated samples, pass@k asks whether at least one of $k$ draws is correct. If $n$ samples contain $c$ correct results and samples are selected without replacement from that set, the standard unbiased estimator is

$$\widehat{\operatorname{pass@k}} = 1-\frac{\binom{n-c}{k}}{\binom{n}{k}},$$

with value 1 when $n-c<k$.

from math import comb


def pass_at_k(*, n: int, c: int, k: int) -> float:
    if not (0 <= c <= n and 1 <= k <= n):
        raise ValueError("require 0 <= c <= n and 1 <= k <= n")
    if n - c < k:
        return 1.0
    return 1.0 - comb(n - c, k) / comb(n, k)

pass@k measures candidate availability, not the ability to identify the correct candidate. A selector metric evaluates the latter:

$$\text{selection accuracy@}N = \frac{\#\text{items where selector chooses a correct candidate}} {N_{\mathrm{items}}}.$$

Build confidence intervals and paired comparisons

For binary correctness, a Wilson interval is more stable than a plain normal approximation, especially near 0 or 1:

import math


def wilson_interval(successes: int, trials: int, z: float = 1.959964):
    if (
        trials <= 0
        or not 0 <= successes <= trials
        or not math.isfinite(z)
        or z <= 0
    ):
        raise ValueError("invalid interval parameters")
    p = successes / trials
    z2 = z * z
    denominator = 1 + z2 / trials
    centre = (p + z2 / (2 * trials)) / denominator
    radius = (
        z
        * math.sqrt(p * (1 - p) / trials + z2 / (4 * trials * trials))
        / denominator
    )
    return centre - radius, centre + radius

When comparing two methods on the same items, use paired outcomes. Count which items improve and regress, then apply an appropriate paired test or bootstrap. Two overlapping marginal confidence intervals do not answer the paired question.

Retain random seeds and all candidates. Sampling variance can exceed the apparent difference between methods.

Slice failures before averaging them away

At minimum, slice by:

• subject and difficulty;
• prompt length and response length;
• answer representation;
• extraction outcome;
• verifier reason code;
• language;
• tool use;
• source or document type;
• route and compute budget; and
• protected or high-risk category where legally and ethically appropriate.

Avoid creating tiny slices and presenting noise as discovery. Define important slices before looking at the result and include their sample sizes.

Learned verifiers need their own evaluation

A learned verifier score should be tested for:

• discrimination: does it rank correct above incorrect candidates?
• calibration: do items scored near 0.8 succeed about 80% of the time?
• selective risk: what error remains above an acceptance threshold?
• shift: does performance hold across subjects, lengths and generators?
• exploitability: can a candidate gain score through format, verbosity or copied phrases?
• independence: was the verifier trained on outputs from the policy it now judges?

Use a held-out, human-adjudicated set for disputed cases. Evaluate the complete selection procedure, since a mildly biased verifier can become severely biased when choosing the maximum of many samples.

If $S_1,\ldots,S_N$ are noisy scores, selecting $\arg\max S_i$ preferentially chooses positive scoring error. This is the optimiser’s curse. Increasing $N$ can make the chosen score rise while true quality stalls or falls.

Keep benchmark and development data apart

Track every way an item can influence the system:

• base-model pretraining is often only partially known;
• supervised examples can contain near-duplicates;
• generated training data can copy benchmark solutions;
• prompt examples can reveal answer forms;
• verifier training can include evaluation outputs;
• repeated threshold tuning can overfit the test; and
• public benchmark discussion can enter later model training.

Use exact hashes and approximate matching for text, formulas and code. A contamination check cannot prove absence from opaque pretraining, so describe what was checked and what remains unknown.

A minimal evaluation runner

from collections.abc import Callable, Iterable
from time import perf_counter


Generate = Callable[[EvalItem, int], tuple[str, int, int]]
Verify = Callable[[str, str], tuple[str, str]]
Normalise = Callable[[str], tuple[str | None, str]]


def evaluate_items(
    items: Iterable[EvalItem],
    *,
    generate: Generate,
    normalise: Normalise,
    verify: Verify,
    seed: int,
    verifier_version: str,
) -> list[EvalResult]:
    results: list[EvalResult] = []
    for sample_id, item in enumerate(items):
        started = perf_counter()
        response = ""
        prompt_tokens = response_tokens = 0
        extracted = normalised = None

        try:
            response, prompt_tokens, response_tokens = generate(
                item,
                seed + sample_id,
            )
            if (
                not isinstance(response, str)
                or type(prompt_tokens) is not int
                or type(response_tokens) is not int
                or prompt_tokens < 0
                or response_tokens < 0
            ):
                raise TypeError("invalid generation result")
        except Exception as exc:
            outcome = "error"
            reason = f"generation_exception:{type(exc).__name__}"
        else:
            try:
                extracted, extraction_reason = extract_final(response)
            except Exception as exc:
                outcome = "error"
                reason = f"extractor_exception:{type(exc).__name__}"
            else:
                if extracted is None:
                    outcome, reason = "review", extraction_reason
                else:
                    try:
                        normalised, normalise_reason = normalise(extracted)
                        if (
                            normalised is not None
                            and not isinstance(normalised, str)
                        ) or not isinstance(normalise_reason, str):
                            raise TypeError("invalid normaliser result")
                    except Exception as exc:
                        outcome = "error"
                        reason = (
                            f"normaliser_exception:{type(exc).__name__}"
                        )
                    else:
                        if normalised is None:
                            outcome, reason = "review", normalise_reason
                        else:
                            try:
                                outcome, reason = verify(
                                    normalised,
                                    item.reference_answer,
                                )
                                if outcome not in {
                                    "correct",
                                    "incorrect",
                                    "review",
                                    "error",
                                } or not isinstance(reason, str):
                                    raise TypeError(
                                        "invalid verifier result"
                                    )
                            except Exception as exc:
                                outcome = "error"
                                reason = (
                                    "verifier_exception:"
                                    f"{type(exc).__name__}"
                                )

        results.append(
            EvalResult(
                item_id=item.item_id,
                sample_id=sample_id,
                seed=seed + sample_id,
                raw_response=response,
                extracted_answer=extracted,
                normalised_answer=normalised,
                verifier_outcome=outcome,
                reason_code=reason,
                latency_ms=(perf_counter() - started) * 1_000,
                prompt_tokens=prompt_tokens,
                response_tokens=response_tokens,
                verifier_version=verifier_version,
            )
        )
    return results

Write results atomically, include the run configuration and hash the output. Parallel execution may reorder completions, so sort by stable item and sample identifiers before comparing runs.

Evaluation in the banking case

The fictional covenant assistant uses a layered evaluator:

1. retrieval coverage: did the evidence set include the executed agreement and applicable amendments?
2. clause precedence: did deterministic rules resolve effective dates and supersession?
3. field schema: are numerator, denominator, currency, period and threshold present?
4. calculation: does an independent function reproduce the ratio?
5. claim support: does each drafted statement point to a source span?
6. decision boundary: did the system avoid an approval recommendation?

The final label is not “model correct”. It is a vector of checks, exceptions and reviewer actions. That representation supports investigation when one component changes.

Chapter notes

Begin with the one-sample baseline

For each item, measure:

1. direct greedy answer;
2. worked greedy answer;
3. one sampled worked answer; and
4. the deterministic or human baseline.

The phrase “think step by step” is not a method specification. Record the exact prompt, template mode, response cap and decoding settings. Chain-of-thought effects vary across models and tasks.

A useful response contract separates optional work from the graded answer:

Solve the problem. You may use concise intermediate work.
Do not guess when required information is absent.
End with one field: <final>ANSWER</final>

The extraction rule from Chapter 3 grades only the final field. Intermediate text can be analysed separately.

Parallel sampling estimates the model’s answer distribution

Suppose a decoding policy produces candidate response $y$ with probability $\pi(y\mid x)$. Sampling $N$ independent candidates gives a set

$$\mathcal{Y}_N=\{y^{(1)},\ldots,y^{(N)}\}.$$

After normalising each final answer to $a^{(i)}$, self-consistency selects the most frequent answer:

$$\hat{a} = \arg\max_a \sum_{i=1}^{N} \mathbf{1}[a^{(i)}=a].$$

The original self-consistency paper described this as sampling diverse reasoning paths and marginalising over their final answers. It reported gains on the studied arithmetic and commonsense benchmarks, while noting the added compute and the possibility of nonsensical paths.1

A task-specific aggregator should preserve ties and parse failures:

from collections import Counter
from dataclasses import dataclass


@dataclass(frozen=True)
class VoteResult:
    answer: str | None
    votes: int
    valid_candidates: int
    total_candidates: int
    tied: bool
    reason_code: str


def plurality_vote(
    answers: list[str | None],
    *,
    minimum_valid: int = 1,
) -> VoteResult:
    if type(minimum_valid) is not int or minimum_valid < 1:
        raise ValueError("minimum_valid must be a positive integer")
    valid = [answer for answer in answers if answer is not None]
    if len(valid) < minimum_valid:
        return VoteResult(
            None, 0, len(valid), len(answers), False, "insufficient_valid"
        )

    counts = Counter(valid)
    ordered = counts.most_common()
    top_answer, top_count = ordered[0]
    tied = len(ordered) > 1 and ordered[1][1] == top_count
    if tied:
        return VoteResult(
            None, top_count, len(valid), len(answers), True, "plurality_tie"
        )
    return VoteResult(
        top_answer, top_count, len(valid), len(answers), False, "plurality"
    )

An invalid parse is not a vote for an empty string. A tie is not arbitrary success.

Diversity has a useful middle range

If temperature is too low, candidates can be near-identical and extra samples add little. If it is too high, the model may leave the region where the task is solved coherently.

Top-$p$ and temperature interact:

• temperature changes relative probability ratios;
• top-$p$ then determines which tokens remain eligible;
• a response-length cap changes the chance of reaching the final field; and
• the prompt and chat template alter the distribution before either control.

Tune the combination on a development split. Report answer diversity, valid-output rate and accuracy together. A setting that creates many unique wrong answers is not valuable exploration.

Useful diversity measures include:

$$\text{unique-answer ratio} = \frac{\#\text{distinct valid answers}} {\#\text{valid samples}},$$

and the entropy of normalised answer counts:

$$H(A) = -\sum_a \hat{p}(a)\log \hat{p}(a).$$

Both depend on the normaliser. Two algebraically equivalent answers should be one category if the verifier declares them equivalent.

Majority agreement is evidence, not a verifier

If all candidates share the same misconception, agreement is confidently wrong. Candidate errors are also correlated because samples share weights, prompt and training data.

Therefore report the empirical relation between agreement and correctness. Bin the held-out items by top-answer vote share and measure accuracy in each bin. Recalibrate after changing model, prompt, decoding or subject.

For a vote share

$$q=\frac{c_{\max}}{n_{\mathrm{valid}}},$$

do not read $q=0.8$ as an 80% probability of correctness without a calibration study.

Self-consistency works best when:

• correct solutions occupy several high-probability paths;
• wrong paths disperse across different answers;
• final answers can be normalised reliably; and
• enough diversity remains under the sampling policy.

It can fail when:

• a systematic trap attracts most samples;
• the task has many valid free-form answers;
• candidates copy a spurious prompt cue;
• parsing collapses distinct answers together; or
• all samples inherit the same missing evidence.

Best-of-N needs an independent selector

Instead of voting, a verifier can score each candidate:

$$\hat{y} = \arg\max_{y\in\mathcal{Y}_N} s_\phi(x,y).$$

A deterministic verifier can select a mathematically or programmatically correct candidate. A learned verifier can rank candidates when exact checking is unavailable, but selection amplifies its scoring errors.

Compare at least:

The oracle row is an upper bound on candidate availability. It is not deployable.

Sequential revision changes the proposal distribution

A revision loop feeds a candidate and feedback into the next generation:

$$y^{(k+1)} \sim \pi_\theta(\cdot\mid x,y^{(k)},f^{(k)}).$$

Feedback may be:

• a deterministic test failure;
• a symbolic counterexample;
• a tool result;
• a learned critique; or
• a human comment.

Revision is valuable when feedback contains information the first generation lacked. Asking the same model to “check again” without new evidence can preserve or reinforce an error.

from dataclasses import dataclass
from typing import Callable


@dataclass(frozen=True)
class RevisionAttempt:
    response: str
    outcome: str
    feedback: str


GenerateRevision = Callable[[str, list[RevisionAttempt]], str]
CheckResponse = Callable[[str], tuple[str, str]]


def revise_until_checked(
    prompt: str,
    *,
    generate: GenerateRevision,
    check: CheckResponse,
    max_attempts: int,
) -> list[RevisionAttempt]:
    if max_attempts < 1:
        raise ValueError("max_attempts must be at least one")
    history: list[RevisionAttempt] = []
    for _ in range(max_attempts):
        response = generate(prompt, history)
        if not isinstance(response, str):
            raise TypeError("revision response must be text")
        outcome, feedback = check(response)
        if outcome not in {"correct", "incorrect", "review", "error"}:
            raise ValueError("unknown checker outcome")
        if not isinstance(feedback, str):
            raise TypeError("checker feedback must be text")
        history.append(RevisionAttempt(response, outcome, feedback))
        if outcome in {"correct", "review", "error"}:
            break
    return history

Stop on review. The model should not turn an ambiguous verifier result into permission to continue until something passes.

Adaptive budgets need a defensible stopping rule

Uniform $N$ wastes samples on easy items and may still underserve medium items. Adaptive sampling starts small and expands under uncertainty.

One exact early-stop rule applies to a fixed maximum plurality vote. If the current leader has more votes than the runner-up could obtain even after receiving every remaining sample, the winner is locked:

from collections import Counter


def plurality_is_locked(
    valid_answers: list[str],
    *,
    generated: int,
    max_samples: int,
) -> bool:
    if not 0 <= len(valid_answers) <= generated <= max_samples:
        raise ValueError(
            "require valid answers <= generated <= max_samples"
        )
    counts = Counter(valid_answers).most_common(2)
    if not counts:
        return False
    leader = counts[0][1]
    runner_up = counts[1][1] if len(counts) == 2 else 0
    remaining = max_samples - generated
    return leader > runner_up + remaining

This rule preserves the result of the planned fixed-size plurality vote. It may save little compute when the race is close.

A probabilistic early-stop threshold can save more, but repeated peeking changes its statistical behaviour. Calibrate the entire sequential policy on held-out data rather than applying a one-shot confidence interval at every step.

Difficulty should be estimated from observable features

The test-time scaling study by Snell and peers found that the effective allocation varied with problem difficulty and that predicted difficulty could improve compute allocation in their setting.2 It also found different mixtures of parallel search and sequential revision were useful across difficulty ranges.

A production difficulty model might use:

• a cheap model’s success probability;
• direct-answer entropy;
• disagreement among a small pilot sample;
• verifier margin;
• prompt length and structural features;
• missing-field or retrieval-coverage signals; and
• historical error rates for the task slice.

Do not use protected attributes as a shortcut. Avoid a feedback loop in which high-budget items create better labels and therefore appear easier later.

Evaluate routing with:

$$\text{quality per token} = \frac{\text{accepted correct outcomes}} {\text{generated tokens}},$$

alongside false accepts, reviews, latency and cost.

Capacity planning separates latency from throughput

For $N$ candidates with generation times $g_i$, corresponding verification times $v_i$, queue time $q$ and aggregation time $a$:

Sequential latency is approximately

$$L_{\mathrm{seq}} =q+\sum_{i=1}^{N}(g_i+v_i)+a.$$

Fully parallel latency is approximately

$$L_{\mathrm{parallel}} =q+\max_i(g_i+v_i)+a,$$

but only if generation and verification capacity exists for all candidates and each pair can proceed independently. If verification is serial or batched separately, its measured stage time replaces the $v_i$ term. Parallelism reduces item latency by consuming more simultaneous serving capacity. Under load, queue time can erase the gain.

Batching several candidates on one accelerator is neither fully sequential nor independent replication. Measure:

• time to first token;
• time per output token;
• candidate completion spread;
• queue delay at target concurrency;
• tokens per second per device;
• cache memory;
• cancelled-token waste after early stop; and
• verifier saturation.

Avoid cost examples tied to an assumed cloud price. Record the actual dated price, reservation model and utilisation when making a deployment decision.

A safe pattern for the banking case

The fictional covenant assistant uses additional compute only after deterministic gates:

1. retrieval must cover the executed agreement and relevant amendments;
2. all required fields must be present with source spans;
3. clause-precedence rules must produce one effective covenant or review;
4. the deterministic calculation must run successfully.

The model may then sample three concise draft explanations. A claim checker rejects drafts whose figures or source references disagree with the verified record. If several pass, the system can choose the shortest supported draft. It does not vote on the covenant value because that value is calculated, not generated.

High consequence changes the role of search. More samples are used to improve wording under fixed facts, not to manufacture consensus about the facts.

Experiment table

For each compute policy, retain:

Compare policies on the same items and seeds where possible. Present the Pareto frontier rather than hiding cost inside one accuracy number.

Chapter notes

Three scorer families

Compute response log-probability with the causal shift

For response tokens $y_1,\ldots,y_T$,

$$\log \pi_\theta(y\mid x) = \sum_{t=1}^{T} \log \pi_\theta(y_t\mid x,y_{<t}).$$

The logit at combined-sequence position $t-1$ predicts token $t$. The mask must select only response targets.

from dataclasses import dataclass

import torch
import torch.nn.functional as F


@dataclass(frozen=True)
class SequenceScores:
    sum_logprob: torch.Tensor
    mean_logprob: torch.Tensor
    token_count: torch.Tensor


def sequence_logprobs(
    logits: torch.Tensor,
    input_ids: torch.Tensor,
    response_mask: torch.Tensor,
) -> SequenceScores:
    """
    logits:        [batch, sequence, vocabulary]
    input_ids:     [batch, sequence]
    response_mask: [batch, sequence], true only for response tokens
    """
    if logits.shape[:2] != input_ids.shape:
        raise ValueError("logits and input_ids sequence shapes differ")
    if input_ids.shape != response_mask.shape:
        raise ValueError("input_ids and response_mask shapes differ")
    if logits.ndim != 3 or logits.shape[1] < 2 or logits.shape[2] < 1:
        raise ValueError("expected non-empty [batch, sequence, vocabulary]")
    if torch.any(response_mask[:, 0]):
        raise ValueError("sequence position zero has no causal target logit")

    prediction_logits = logits[:, :-1, :].float()
    target_ids = input_ids[:, 1:]
    target_mask = response_mask[:, 1:].to(dtype=torch.bool)
    safe_logits = prediction_logits.masked_fill(
        ~target_mask.unsqueeze(-1),
        0.0,
    )
    safe_targets = target_ids.masked_fill(~target_mask, 0)
    if not torch.isfinite(
        prediction_logits.masked_select(target_mask.unsqueeze(-1))
    ).all():
        raise ValueError("scored logits must be finite")
    if torch.any(
        (safe_targets < 0) | (safe_targets >= logits.shape[-1])
    ):
        raise ValueError("target token lies outside the vocabulary")

    token_logprobs = F.log_softmax(safe_logits, dim=-1).gather(
        dim=-1,
        index=safe_targets.unsqueeze(-1),
    ).squeeze(-1)

    masked = token_logprobs.masked_fill(~target_mask, 0.0)
    counts = target_mask.sum(dim=-1)
    if torch.any(counts == 0):
        raise ValueError("every row must contain at least one response token")

    sums = masked.sum(dim=-1)
    means = sums / counts
    return SequenceScores(sums, means, counts)

Tests should cover:

• one response token;
• different response lengths in one padded batch;
• left and right padding under the chosen model contract;
• masked prompt tokens;
• masked padding tokens;
• an empty response; and
• equality with a hand-computed two-token example.

Sum and mean scores answer different questions

The summed score decreases as more tokens are multiplied:

$$\log \pi(y\mid x)\leq 0.$$

It therefore tends to prefer shorter sequences when candidates differ in length. The mean log-probability

$$\frac{1}{T}\sum_t \log \pi(y_t\mid x,y_{<t})$$

reduces that length effect but can favour verbose continuations made of individually predictable tokens. Neither is a calibrated correctness probability.

Other options include:

• score only the terminal answer field;
• compare candidates of a fixed response format;
• add an explicit length penalty tuned on held-out data; or
• use a task verifier and treat likelihood as a diagnostic.

Report the token count beside every sequence score.

Confidence requires an event

“The model is 90% confident” is incomplete. Confidence in what?

Possible events include:

• the next token equals a target token;
• the final answer is correct;
• the response passes a test suite;
• a reviewer prefers candidate A;
• a factual claim is supported by cited evidence; or
• the complete action is safe to execute.

Next-token probabilities do not automatically calibrate any of the later events. To calibrate final-answer correctness, define the answer event, collect held-out labels and map a score to observed frequency. Recheck after distribution shift.

For binary labels $z_i\in\{0,1\}$ and predicted probabilities $p_i$, the Brier score is

$$\operatorname{Brier} = \frac{1}{N}\sum_{i=1}^{N}(p_i-z_i)^2.$$

Reliability diagrams reveal whether scores near a given value correspond to that empirical success rate. Also measure resolution: a perfectly calibrated model that predicts the base rate for every item may be unhelpful for routing.

Selection creates a distribution shift

Suppose a learned verifier score is

$$s_i=q_i+\varepsilon_i,$$

where $q_i$ is true quality and $\varepsilon_i$ is scoring error. Selecting the maximum among many candidates favours both high quality and positive error:

$$i^*=\arg\max_i s_i.$$

As candidate count rises, the selected score can improve faster than selected true quality. A verifier evaluated on random candidates may perform worse on candidates deliberately optimised to fool it.

Evaluate the exact deployment procedure:

1. sample $N$ candidates from the deployed generator;
2. score them with the deployed verifier;
3. select under the deployed tie and threshold rules;
4. obtain independent labels for the selected candidates; and
5. repeat across $N$, task slices and model revisions.

Keep a random-candidate audit sample as a control. Otherwise the system may only label what its verifier already likes.

Combine scores through gates before weights

A weighted sum is tempting:

$$S(y) = w_1s_{\mathrm{correct}}(y) +w_2s_{\mathrm{format}}(y) +w_3s_{\mathrm{style}}(y).$$

It can let a high style score compensate for a failed correctness check. Hard requirements should be gates:

import math

from dataclasses import dataclass


@dataclass(frozen=True)
class CandidateScores:
    parse_ok: bool
    deterministic_ok: bool | None
    support_score: float | None
    style_score: float | None
    policy_violation: bool


def selection_key(scores: CandidateScores):
    for value in (scores.support_score, scores.style_score):
        if value is not None and not math.isfinite(value):
            raise ValueError("candidate scores must be finite")
    eligible = (
        scores.parse_ok
        and scores.deterministic_ok is True
        and not scores.policy_violation
    )
    if not eligible:
        return (0, float("-inf"), float("-inf"))
    support = (
        scores.support_score
        if scores.support_score is not None
        else float("-inf")
    )
    style = (
        scores.style_score
        if scores.style_score is not None
        else float("-inf")
    )
    return (1, support, style)

This lexicographic rule first enforces eligibility, then ranks support, then style. The ordering is visible and testable.

For tasks without a deterministic truth check, use multiple scorers and explicit disagreement:

• accept when independent signals agree above calibrated thresholds;
• review when they disagree;
• reject on a hard policy violation; and
• audit a sample of accepted, reviewed and rejected outputs.

Critique is a proposal, not a diagnosis

Self-refinement asks a model to critique and revise its output. Research systems such as Self-Refine have reported improvements on selected tasks using iterative feedback from the same model.1 The feedback remains generated text. It can identify a real error, invent an error or rationalise the original answer.

A safer loop has an external check:

from dataclasses import dataclass
from typing import Callable, Literal


Status = Literal["pass", "fail", "review", "error"]


@dataclass(frozen=True)
class Check:
    status: Status
    codes: tuple[str, ...]
    public_feedback: str


Draft = Callable[[str, str | None], str]
Checker = Callable[[str], Check]


def checked_refinement(
    prompt: str,
    *,
    draft: Draft,
    check: Checker,
    max_attempts: int = 3,
):
    if max_attempts < 1:
        raise ValueError("max_attempts must be at least one")
    response = draft(prompt, None)
    history: list[tuple[str, Check]] = []

    for attempt in range(max_attempts):
        if not isinstance(response, str):
            raise TypeError("draft response must be text")
        result = check(response)
        if not isinstance(result, Check):
            raise TypeError("checker must return Check")
        if result.status not in {"pass", "fail", "review", "error"}:
            raise ValueError("unknown checker status")
        if not isinstance(result.public_feedback, str):
            raise TypeError("public feedback must be text")
        history.append((response, result))
        if result.status in {"pass", "review", "error"}:
            return history
        if attempt + 1 < max_attempts:
            response = draft(prompt, result.public_feedback)

    return history

public_feedback must exclude secrets, hidden tests and unsafe system details. If a checker reports uncertainty, escalation is safer than repeated regeneration.

Avoid training on unverified self-critique

A self-training pipeline often:

1. generates responses;
2. scores them;
3. keeps high-scoring examples; and
4. fine-tunes on the survivors.

This can improve a narrow metric. It can also amplify the scorer’s blind spots. Add safeguards:

• deterministic labels where available;
• independent human review of boundary cases;
• deduplication by prompt and semantic content;
• per-source and per-generator caps;
• retained rejected samples for audit;
• a frozen final test set;
• adversarial tests designed after inspecting false accepts; and
• data lineage from generated example to model checkpoint.

Do not train a verifier and policy on mutually generated labels, evaluate them against each other and call the agreement independent evidence.

Scorer drift and versioning

A scorer registry should include:

scorer_id
model_or_code_revision
training_data_revision
target_event
input_schema
output_schema
calibration_dataset
calibration_date
supported_slices
known_failure_slices
thresholds
owner

Changing any of these can change selection. Replay a fixed candidate bank through a new scorer before switching live traffic. Then compare selected candidates, not just raw scores.

Monitor:

• acceptance and review rates;
• score distribution;
• score versus deterministic outcome;
• disagreement between scorers;
• length and format correlations;
• selected-candidate audit accuracy; and
• performance by generator and task slice.

The banking case uses scoring only after factual gates

For the fictional credit-document assistant:

• a deterministic schema check rejects missing values and units;
• a provenance check verifies document identifiers and source spans;
• a calculation service determines the covenant ratio;
• a claim checker compares every numeric statement with that verified record; and
• a language scorer may rank clarity among drafts that passed all gates.

The language scorer cannot compensate for a missing amendment or wrong ratio. Reviewer feedback is recorded against the specific check, not reduced to a single opaque “quality” number.

A refinement cycle may ask the model to repair a missing citation or remove an unsupported statement. It may not ask the model to reinterpret the credit decision until a draft passes.

Build checks

Before using a score for selection or training:

1. name its target event;
2. state whether higher is better;
3. test the causal shift and response mask;
4. report candidate lengths;
5. calibrate on held-out data;
6. evaluate after best-of-$N$ selection;
7. inspect correlations with format and verbosity;
8. define hard gates and escalation; and
9. retain a human-adjudicated audit set.

The next chapter turns verifiable rewards into policy updates. Every warning here carries into that optimiser.

Chapter notes

The four policy roles

Keep the roles explicit:

At rollout collection, copy or snapshot the current policy as $\pi_{\mathrm{old}}$. Store its log-probabilities for the sampled tokens. Reusing responses from an unknown or much older policy breaks the stated importance ratio.

The reference is commonly the starting supervised checkpoint. The original DeepSeekMath iterative algorithm also described outer iterations that refreshed the reference. Those are different experiments and should not share an unlabeled “GRPO” result.1

The teaching run samples from the model softmax with temperature 1 and no top-$p$ truncation. Under that setting, the behaviour and model log-probabilities in Chapter 2 are identical. If a training system samples from a tempered or truncated distribution, it must define that behaviour policy and derive the ratio against the same distribution; substituting raw model log-probabilities after the fact is not the stated on-policy objective.

Group-relative advantage

For prompt $b$, sample $G$ responses with rewards $r_{b,1},\ldots,r_{b,G}$. Outcome-supervised GRPO uses

$$A_{b,g} = \frac{r_{b,g}-\bar{r}_b} {\sqrt{\operatorname{Var}(r_b)+\delta}},$$

where $\delta$ prevents division by zero.

import math

import torch


def group_advantages(
    rewards: torch.Tensor,
    *,
    eps: float = 1e-6,
) -> torch.Tensor:
    """Normalise rewards shaped [batch, group] within each prompt."""
    if rewards.ndim != 2 or rewards.shape[0] < 1 or rewards.shape[1] < 2:
        raise ValueError("rewards must have shape [batch>=1, group>=2]")
    if not math.isfinite(eps) or eps <= 0:
        raise ValueError("eps must be positive")
    rewards = rewards.float()
    if not torch.isfinite(rewards).all():
        raise ValueError("rewards must be finite")
    centred = rewards - rewards.mean(dim=1, keepdim=True)
    variance = centred.square().mean(dim=1, keepdim=True)
    return centred / torch.sqrt(variance + eps)

Using the population variance (unbiased=False) makes the definition independent of a library’s small-sample correction. Log the exact convention.

Consequences:

• adding the same constant to all rewards in a group changes no advantage;
• multiplying all rewards by a positive constant has little effect after normalisation;
• if every reward ties, the policy-gradient signal is zero;
• $G=2$ supplies only a coarse comparison;
• a lucky or erroneous outlier affects every response in the group; and
• advantages are not comparable absolute task values.

Group composition therefore matters. Samples for a prompt should use the stated rollout policy and sampling settings, not a mixture of unrelated generators unless the objective accounts for that mixture.

From token log-probabilities to the clipped surrogate

For sampled token $o_{b,g,t}$, define

$$\rho_{b,g,t}(\theta) = \frac{ \pi_\theta(o_{b,g,t}\mid q_b,o_{b,g,<t}) }{ \pi_{\mathrm{old}}(o_{b,g,t}\mid q_b,o_{b,g,<t}) } = \exp\left( \ell^\theta_{b,g,t}-\ell^{\mathrm{old}}_{b,g,t} \right).$$

The PPO-style token surrogate is

$$\min\left( \rho A, \operatorname{clip}(\rho,1-\epsilon,1+\epsilon)A \right).$$

For a positive advantage, the upper clip limits the benefit assigned to a large probability increase. For a negative advantage, the lower side limits the benefit assigned to a large decrease. Taking min after multiplying by the signed advantage handles both cases.

Clipping does not impose a hard bound on parameter change. It clips this sampled surrogate. Learning rate, repeated epochs, gradient norm and the KL term still matter.

The reference-policy penalty

DeepSeekMath used the positive sampled estimator

$$\widehat{D}_{\mathrm{KL}} = \frac{\pi_{\mathrm{ref}}}{\pi_\theta} -\log\frac{\pi_{\mathrm{ref}}}{\pi_\theta} -1.$$

Let

$$\Delta = \log\pi_{\mathrm{ref}}-\log\pi_\theta.$$

Then

$$\widehat{D}_{\mathrm{KL}} = \exp(\Delta)-\Delta-1\geq0.$$

The estimator is zero when the sampled-token probabilities match.

def sampled_reverse_ratio_kl(
    current_logprobs: torch.Tensor,
    reference_logprobs: torch.Tensor,
) -> torch.Tensor:
    delta = reference_logprobs.float() - current_logprobs.float()
    return torch.expm1(delta) - delta

The log-probabilities must refer to the same sampled tokens and contexts. Padding, prompt and post-stop positions are excluded.

A complete masked GRPO loss

The teaching implementation below follows the original token-ratio objective and averages tokens within each response, then responses within each prompt, then prompts within the batch.

from dataclasses import dataclass

import torch


@dataclass(frozen=True)
class GRPOMetrics:
    loss: float
    policy_objective: float
    sampled_kl: float
    clip_fraction: float
    mean_response_tokens: float
    normalised_advantage_std: float


def grpo_loss(
    *,
    current_logprobs: torch.Tensor,
    old_logprobs: torch.Tensor,
    reference_logprobs: torch.Tensor,
    response_mask: torch.Tensor,
    rewards: torch.Tensor,
    clip_epsilon: float,
    kl_beta: float,
) -> tuple[torch.Tensor, GRPOMetrics]:
    """
    Log-probabilities and mask: [batch, group, response_length]
    Rewards:                    [batch, group]
    """
    expected = current_logprobs.shape
    if expected != old_logprobs.shape or expected != reference_logprobs.shape:
        raise ValueError("all log-probability tensors must share a shape")
    if expected != response_mask.shape:
        raise ValueError("response_mask shape differs from log-probabilities")
    if current_logprobs.ndim != 3:
        raise ValueError("expected [batch, group, response_length]")
    if rewards.shape != current_logprobs.shape[:2]:
        raise ValueError("rewards must have shape [batch, group]")
    if not 0 < clip_epsilon < 1:
        raise ValueError("clip_epsilon must lie in (0, 1)")
    if not math.isfinite(kl_beta) or kl_beta < 0:
        raise ValueError("kl_beta cannot be negative")

    mask = response_mask.to(dtype=torch.bool)
    token_counts = mask.sum(dim=-1)
    if torch.any(token_counts == 0):
        raise ValueError("every response needs at least one scored token")
    for name, values in (
        ("current", current_logprobs),
        ("old", old_logprobs),
        ("reference", reference_logprobs),
    ):
        if not torch.isfinite(values.masked_select(mask)).all():
            raise ValueError(f"{name} log-probabilities must be finite")

    advantages = group_advantages(rewards).detach()
    token_advantages = advantages.unsqueeze(-1)

    safe_current = current_logprobs.float().masked_fill(~mask, 0.0)
    safe_old = old_logprobs.float().masked_fill(~mask, 0.0)
    safe_reference = reference_logprobs.float().masked_fill(~mask, 0.0)

    log_ratio = safe_current - safe_old
    ratio = torch.exp(log_ratio)
    if not torch.isfinite(ratio).all():
        raise FloatingPointError("importance ratio is non-finite")
    clipped_ratio = ratio.clamp(1 - clip_epsilon, 1 + clip_epsilon)

    unclipped = ratio * token_advantages
    clipped = clipped_ratio * token_advantages
    token_policy = torch.minimum(unclipped, clipped)

    token_kl = sampled_reverse_ratio_kl(
        safe_current,
        safe_reference,
    )
    if not torch.isfinite(token_kl).all():
        raise FloatingPointError("sampled KL is non-finite")
    token_objective = token_policy - kl_beta * token_kl
    token_objective = token_objective.masked_fill(~mask, 0.0)

    response_objective = token_objective.sum(dim=-1) / token_counts
    objective = response_objective.mean()
    loss = -objective
    if not torch.isfinite(loss):
        raise FloatingPointError("GRPO loss is non-finite")

    policy_only = token_policy.masked_fill(~mask, 0.0).sum(dim=-1)
    policy_only = (policy_only / token_counts).mean()
    mean_kl = token_kl.masked_fill(~mask, 0.0).sum(dim=-1)
    mean_kl = (mean_kl / token_counts).mean()
    clipped_tokens = ((ratio != clipped_ratio) & mask).sum()

    metrics = GRPOMetrics(
        loss=float(loss.detach()),
        policy_objective=float(policy_only.detach()),
        sampled_kl=float(mean_kl.detach()),
        clip_fraction=float(clipped_tokens / mask.sum()),
        mean_response_tokens=float(token_counts.float().mean()),
        normalised_advantage_std=float(
            advantages.std(unbiased=False)
        ),
    )
    return loss, metrics

The loss assumes the tensors already contain log-probabilities for response targets, aligned as in Chapter 5. It does not take logits because computing three full vocabulary distributions inside one function would obscure memory and gradient choices.

Masked positions are replaced before exponentiation. Masking only the final objective is too late: an extreme padding value can overflow inside exp, and the backward pass can retain a non-finite gradient even when the displayed loss appears finite.

Shape ledger for a rollout batch

For $B=2$, $G=4$ and padded response length $T=256$:

Flatten [B, G] into a generation batch only at interfaces that require it. Restore the mapping before reward normalisation. Mixing responses across prompts destroys the group baseline.

Reward design starts with a truth table

For a verifiable maths task:

The correctness signal should dominate the cosmetic signal. If a perfectly formatted wrong answer outranks a correct but slightly malformed answer, the optimiser learns the wrong priority.

def combined_reward(
    *,
    verifier_outcome: str,
    format_ok: bool,
) -> float | None:
    if verifier_outcome in {"review", "error"}:
        return None
    if verifier_outcome not in {"correct", "incorrect"}:
        raise ValueError("unknown verifier outcome")
    correctness = 1.0 if verifier_outcome == "correct" else 0.0
    format_bonus = 0.1 if format_ok else 0.0
    return correctness + format_bonus

Do not convert a verifier exception into zero reward. An infrastructure error is not evidence that the response is wrong.

Sparse binary rewards create many tied groups. Increase prompt diversity, group size or verifier coverage before adding arbitrary shaping. Shaping signals can make training move while reducing the target behaviour.

Collect rollouts without accidental gradients

A rollout batch is a frozen observation:

@torch.inference_mode()
def collect_rollout_batch(
    old_policy,
    reference_policy,
    prompts,
    *,
    group_size: int,
    generate_group,
    score_group,
):
    if group_size < 2:
        raise ValueError("group_size must be at least two")
    old_policy.eval()
    reference_policy.eval()
    sampled = generate_group(old_policy, prompts, group_size=group_size)
    rewards = score_group(prompts, sampled.responses)
    if rewards.shape != (len(prompts), group_size):
        raise ValueError("reward shape does not match prompt groups")

    old_logprobs = sampled_token_logprobs(
        old_policy,
        sampled.input_ids,
        sampled.response_mask,
    )
    reference_logprobs = sampled_token_logprobs(
        reference_policy,
        sampled.input_ids,
        sampled.response_mask,
    )
    return sampled, rewards, old_logprobs, reference_logprobs

sampled_token_logprobs is the shifted gather from Chapter 5, retaining per-token rather than reduced scores. The reference and old models remain in evaluation mode.

For memory, old-policy log-probabilities can be computed during generation or replayed immediately after. Reference log-probabilities can be computed in microbatches. Store them in a precision whose error is shown not to change training materially.

Optimiser update and gradient accumulation

import math


def update_policy(
    policy,
    optimiser,
    microbatches,
    *,
    clip_epsilon: float,
    kl_beta: float,
    max_grad_norm: float,
):
    if not math.isfinite(max_grad_norm) or max_grad_norm <= 0:
        raise ValueError("max_grad_norm must be finite and positive")
    policy.train()
    optimiser.zero_grad(set_to_none=True)
    microbatches = list(microbatches)
    prompt_counts = [int(batch.rewards.shape[0]) for batch in microbatches]
    total_prompts = sum(prompt_counts)
    if total_prompts <= 0:
        raise ValueError("empty update")

    metric_rows = []
    for batch, prompt_count in zip(microbatches, prompt_counts):
        current = sampled_token_logprobs(
            policy,
            batch.input_ids,
            batch.response_mask,
        )
        loss, metrics = grpo_loss(
            current_logprobs=current,
            old_logprobs=batch.old_logprobs,
            reference_logprobs=batch.reference_logprobs,
            response_mask=batch.response_mask,
            rewards=batch.rewards,
            clip_epsilon=clip_epsilon,
            kl_beta=kl_beta,
        )
        weight = prompt_count / total_prompts
        (loss * weight).backward()
        metric_rows.append(metrics)

    grad_norm = torch.nn.utils.clip_grad_norm_(
        policy.parameters(),
        max_norm=max_grad_norm,
    )
    if not torch.isfinite(grad_norm):
        optimiser.zero_grad(set_to_none=True)
        raise FloatingPointError("non-finite gradient norm")

    optimiser.step()
    return float(grad_norm), metric_rows

Weight microbatches by the reduction unit used in the loss. This implementation first averages responses inside each prompt group and then averages prompts, so it uses prompt counts. A token-weighted alternative is a different objective and may favour long responses.

Tests before any expensive run

Use synthetic tensors to establish invariants:

def test_grpo_invariants():
    current = torch.zeros(1, 4, 3, requires_grad=True)
    old = torch.zeros_like(current)
    reference = torch.zeros_like(current)
    mask = torch.tensor([[[1, 1, 0]] * 4], dtype=torch.bool)
    rewards = torch.tensor([[0.0, 1.0, 0.0, 1.0]])

    loss, metrics = grpo_loss(
        current_logprobs=current,
        old_logprobs=old,
        reference_logprobs=reference,
        response_mask=mask,
        rewards=rewards,
        clip_epsilon=0.2,
        kl_beta=0.04,
    )
    assert torch.isfinite(loss)
    assert abs(metrics.sampled_kl) < 1e-8
    loss.backward()
    assert current.grad is not None
    assert torch.isfinite(current.grad).all()

    padded_extreme = torch.tensor(
        [[[0.0, 0.0, 1_000.0]] * 4],
        requires_grad=True,
    )
    padded_loss, _ = grpo_loss(
        current_logprobs=padded_extreme,
        old_logprobs=old,
        reference_logprobs=reference,
        response_mask=mask,
        rewards=rewards,
        clip_epsilon=0.2,
        kl_beta=0.04,
    )
    padded_loss.backward()
    assert torch.isfinite(padded_extreme.grad).all()
    assert torch.all(padded_extreme.grad[..., -1] == 0)

    tied = group_advantages(torch.ones(2, 4))
    assert torch.allclose(tied, torch.zeros_like(tied))

Also test:

• positive and negative clipping by hand;
• padding invariance;
• identical current/reference gives zero KL;
• KL remains non-negative;
• changing masked log-probabilities does not change loss;
• rewards are normalised within, never across, prompts;
• a verifier error is excluded rather than rewarded; and
• resuming a checkpoint reproduces the next update.

What GRPO actually learns from

With outcome supervision, every token in a response receives the same response-level advantage. The gradient reinforces or suppresses the sampled token choices that produced the response. It does not identify which reasoning step caused the outcome.

Process rewards can assign feedback at step boundaries and propagate future step rewards backwards, as described in DeepSeekMath. That introduces a process reward model or labelled process signal and a new set of alignment risks.

GRPO can elicit behaviour already reachable by the starting policy and sampling distribution. It cannot reward a correct response that never appears. Curriculum, supervised warm-start data, tools and broader pretraining can matter more than the optimiser name.

A bounded experiment

Begin with a run small enough to inspect manually:

• 50 to 200 prompts drawn from the training split;
• $G=4$ or $8$;
• deterministic final-answer rewards;
• a small format bonus;
• one policy update per rollout batch;
• a fixed reference checkpoint;
• short response cap;
• held-out evaluation every few updates; and
• saved raw groups, rewards and reason codes.

The number of updates is not a claim of expected improvement. Stop conditions should be based on held-out accuracy, divergence, entropy, invalid-output rate and general-capability checks.

The banking boundary

RL with verifiable rewards fits synthetic banking subtasks such as:

• extracting a field into a fixed schema;
• selecting the applicable clause from a supplied set with a known label;
• producing a calculation trace whose result is recomputed; and
• attaching source identifiers to supported claims.

It should not use realised lending outcomes as a simplistic reward for a language model. Outcomes are delayed, confounded, regulated and shaped by human and economic factors. A model could optimise a proxy in ways that harm customers or violate policy.

The fictional covenant case trains on synthetic clauses and exact schema/verifier rules. The trained model remains an extraction and drafting component. Deterministic calculation and human credit authority remain outside the policy.

Chapter notes

Reward metrics reveal the training signal

Log per batch and per task slice:

• mean, standard deviation and quantiles of total reward;
• each reward component;
• fraction of all-equal groups;
• fraction of groups containing at least one correct response;
• fraction containing both correct and incorrect responses;
• verifier review and error rates;
• format-valid rate; and
• correlation between reward and response length.

The informative-group rate

$$\frac{\#\text{groups with non-zero reward variance}} {\#\text{groups}}$$

indicates how often group normalisation supplies a policy signal. If it approaches zero, increasing update count does not create information. The policy may be too weak, too strong or sampled with too little diversity for the current curriculum.

import torch


def reward_batch_metrics(rewards: torch.Tensor) -> dict[str, float]:
    if rewards.ndim != 2 or rewards.shape[0] < 1 or rewards.shape[1] < 2:
        raise ValueError("expected rewards shaped [batch>=1, group>=2]")
    rewards = rewards.float()
    if not torch.isfinite(rewards).all():
        raise ValueError("rewards must be finite")
    group_std = rewards.std(dim=1, unbiased=False)
    return {
        "reward_mean": float(rewards.mean()),
        "reward_std": float(rewards.std(unbiased=False)),
        "informative_group_rate": float((group_std > 1e-8).float().mean()),
        "all_zero_group_rate": float(
            (rewards.abs().sum(dim=1) == 0).float().mean()
        ),
    }

Log reason codes alongside scalar rewards. A falling reward can be a model regression or a verifier outage.

Policy movement needs three complementary signals

Token entropy

For logits $z$ and probabilities $p=\operatorname{softmax}(z)$:

$$H(p) = -\sum_v p_v\log p_v.$$

def mean_token_entropy(
    logits: torch.Tensor,
    target_mask: torch.Tensor,
) -> torch.Tensor:
    if (
        logits.ndim < 2
        or logits.shape[-1] < 1
        or logits.shape[:-1] != target_mask.shape
    ):
        raise ValueError("mask must match all non-vocabulary dimensions")
    mask = target_mask.to(dtype=torch.bool)
    if not torch.any(mask):
        raise ValueError("no scored tokens")
    safe_logits = logits.float().masked_fill(
        ~mask.unsqueeze(-1),
        0.0,
    )
    if not torch.isfinite(
        logits.float().masked_select(mask.unsqueeze(-1))
    ).all():
        raise ValueError("scored logits must be finite")
    log_probs = torch.log_softmax(safe_logits, dim=-1)
    probs = log_probs.exp()
    entropy = -(probs * log_probs).sum(dim=-1)
    selected = entropy.masked_select(mask)
    if not torch.isfinite(selected).all():
        raise ValueError("scored entropy must be finite")
    return selected.mean()

Entropy can fall because the model learned a cleaner output format. It can also signal diversity collapse. Interpret it with held-out accuracy, unique-answer rate and response length.

Optimisation telemetry catches numerical failure

Record:

• loss and policy-objective terms;
• gradient norm before clipping;
• whether clipping occurred;
• learning rate;
• update time;
• peak allocated and reserved memory;
• non-finite loss or gradient counts; and
• skipped optimiser steps.

Check finiteness before optimiser.step(). A NaN checkpoint is not a recovery point.

Mixed-precision training adds scaler state and overflow behaviour. BF16 does not use dynamic loss scaling in the same way as FP16, but activations and reductions can still overflow or lose precision. Keep log-probability and reward-normalisation arithmetic in FP32 unless an experiment demonstrates equivalence.

Gradient clipping is a containment measure:

$$g \leftarrow g\min\left(1,\frac{c}{\lVert g\rVert_2}\right).$$

It does not fix a reward outlier, incorrect reduction or unsuitable learning rate. Log how often and by how much gradients are clipped.

Response behaviour exposes reward shortcuts

Inspect distributions of:

• prompt and response tokens;
• stop reason;
• final-field position;
• invalid or multiple final fields;
• repeated phrases and loops;
• language switching;
• unique normalised answers;
• use of requested tools;
• citations per claim; and
• policy-violation reason codes.

Common shortcuts include:

Independent evaluation is the stop authority

Use at least three held-out sets:

1. in-domain validation: same task family, disjoint items;
2. challenge set: known failure patterns, adversarial formatting and verifier boundary cases; and
3. capability retention: unrelated tasks the model must not lose.

Run greedy and sampled evaluations. RL can improve the sampling distribution while changing the greedy mode, or vice versa.

Do not select a checkpoint on the final test set. Predeclare:

• primary metric;
• minimum coverage;
• maximum false-accept or policy-violation rate;
• maximum response length or latency;
• retention floors;
• tie-breaking rule; and
• early-stop patience.

A checkpoint is eligible only when all gates pass.

import math

from dataclasses import dataclass


@dataclass(frozen=True)
class EvalSummary:
    accuracy: float
    coverage: float
    false_accept_rate: float
    invalid_rate: float
    mean_response_tokens: float
    retention_accuracy: float


def checkpoint_is_eligible(
    summary: EvalSummary,
    *,
    minimum_accuracy: float,
) -> bool:
    rates = (
        summary.accuracy,
        summary.coverage,
        summary.false_accept_rate,
        summary.invalid_rate,
        summary.retention_accuracy,
        minimum_accuracy,
    )
    if any(not 0.0 <= value <= 1.0 for value in rates):
        raise ValueError("rates and thresholds must lie in [0, 1]")
    if (
        not math.isfinite(summary.mean_response_tokens)
        or summary.mean_response_tokens < 0
    ):
        raise ValueError(
            "mean_response_tokens must be finite and non-negative"
        )
    return (
        summary.accuracy >= minimum_accuracy
        and summary.coverage >= 0.98
        and summary.false_accept_rate <= 0.01
        and summary.invalid_rate <= 0.01
        and summary.mean_response_tokens <= 512
        and summary.retention_accuracy >= 0.95
    )

The numbers are illustrative. Production thresholds require application risk ownership and empirical uncertainty intervals.

Diagnose patterns, not single points

Checkpoint the full experiment state

A resumable checkpoint includes:

• policy weights;
• optimiser and scheduler;
• mixed-precision scaler when used;
• global update and rollout counters;
• Python, NumPy, Torch CPU and accelerator RNG states;
• data sampler or shuffle generator;
• old-policy and reference revisions;
• model, tokenizer and chat-template revisions;
• reward and verifier versions;
• curriculum position;
• run configuration; and
• best-checkpoint selection state.

from __future__ import annotations

import os
import random
from pathlib import Path
from tempfile import NamedTemporaryFile
from typing import Any

import numpy as np
import torch


def capture_rng_state() -> dict[str, Any]:
    state: dict[str, Any] = {
        "python": random.getstate(),
        "numpy": np.random.get_state(),
        "torch_cpu": torch.get_rng_state(),
    }
    if torch.cuda.is_available():
        state["torch_cuda"] = torch.cuda.get_rng_state_all()
    if hasattr(torch, "mps") and torch.mps.is_available():
        state["torch_mps"] = torch.mps.get_rng_state()
    return state


def atomic_torch_save(payload: dict[str, Any], destination: Path) -> None:
    destination.parent.mkdir(parents=True, exist_ok=True)
    with NamedTemporaryFile(
        dir=destination.parent,
        prefix=f".{destination.name}.",
        suffix=".tmp",
        delete=False,
    ) as handle:
        temporary = Path(handle.name)
    try:
        torch.save(payload, temporary)
        with temporary.open("rb") as handle:
            os.fsync(handle.fileno())
        os.replace(temporary, destination)
    finally:
        temporary.unlink(missing_ok=True)

Only load trusted training checkpoints. A general torch.load payload can contain Python pickles. For distribution, publish weight-only safetensors plus plain configuration files where possible.

After save, load the checkpoint into a fresh process and run a small forward/evaluation test. Existence is not integrity.

Run diary

For every run, record:

objective:
hypothesis:
code revision:
configuration hash:
policy / old / reference revisions:
dataset revisions:
verifier revision:
hardware and software:
start time:
planned update cap:
stop gates:
checkpoint selection rule:
deviations from plan:
result:
failure analysis:

Attach plots and per-item outputs rather than pasting only a final score. A failed run with complete evidence can improve the next experiment; an unexplained success cannot be trusted.

Monitoring the fictional banking assistant

Training-time checks for the synthetic covenant task include:

• exact field accuracy;
• applicable-clause accuracy;
• calculation agreement;
• supported-claim rate;
• unsupported-number rate;
• missing-amendment challenge set;
• invalid schema rate;
• response length;
• general extraction retention; and
• results by document template and clause wording.

At use time, the bank would monitor different outcomes:

• retrieval coverage and source age;
• review and exception rates;
• deterministic validation failures;
• analyst overrides with reason codes;
• document-template drift;
• access-control denials;
• latency and service errors; and
• sampled quality reviews.

Training reward is not a live control metric. The deployed system is governed by evidence, exceptions and human actions.

Research context

DeepSeek-R1 reported that reinforcement learning on verifiable tasks could elicit longer and more structured problem-solving behaviour, while its authors also described readability and language-mixing problems in the RL-only stage and used later stages to address them.1 Those observations fit the monitoring framework here: capability, format and general usability can move in different directions.