The output law begins after the model

For a prefix such as “The case should be”, an autoregressive model produces one score for every vocabulary token. A softmax turns those scores into a conditional distribution. That distribution describes what the model assigns probability to next under the current context. It is not yet a returned token, a completed answer or a business decision.

Here v is a candidate token, t is the generation step, θ denotes model parameters, x<t is the generated prefix and c is the supplied context. The decoder D maps the model distribution p_t into the operational distribution q_t. It may use an admissible set A_t, randomness ω_t, search state, penalties and a stopping rule.

The distinction becomes larger over a sequence. Every sampled token becomes part of the next prefix, so one local intervention alters later model distributions. The decoder-induced probability of a complete sequence y is the product of its transformed conditionals:

A decoding configuration therefore specifies more than style. It specifies which complete trajectories remain possible and how probability moves among them. The stopping token matters as much as temperature. So do repetition penalties, forbidden tokens, length limits and the order in which processors run. Two services using the same weights can expose meaningfully different systems.

A four-token worked example

Take one minimal distribution over four next tokens: approve 0.40, review 0.30, decline 0.20 and explain 0.10. Nothing about the prompt or model changes in the following comparisons. Only the decoder changes. That lets us attribute every difference to the intervention rather than to a new example.

Greedy decoding replaces the four-way distribution with [1, 0, 0, 0]. Temperature 0.5 squares the probabilities before renormalisation, producing approximately [0.533, 0.300, 0.133, 0.033]. Temperature 2.0 takes square roots and produces approximately [0.325, 0.282, 0.230, 0.163]. The first intervention strengthens the original ranking; the second weakens it. Neither changes which tokens are possible.

Now apply top-p with threshold 0.75. The first two tokens contain mass 0.70, which is insufficient, so the decoder also retains decline. After renormalisation, the result is [0.444, 0.333, 0.222, 0]. A grammar that allows exactly the three status values happens to produce the same one-step numbers. The equality is accidental. Top-p removed explain because it fell beyond a cumulative threshold; the grammar removed it because it violates the output contract. In another context, top-p could remove a valid status or admit a syntactically invalid token, while the grammar should continue to enforce the represented language.

The example also exposes processor order. Temperature 0.5 followed by top-p 0.75 keeps only approve and review, yielding [0.64, 0.36, 0, 0]. Top-p first and temperature second keeps three tokens and yields about [0.552, 0.310, 0.138, 0]. The same named settings can implement different output laws when their order differs.

Selection, transformation and execution are different operations

A useful decoder description separates three questions. First, which probability distribution is intended? Second, how is one output selected from it? Third, how is that selection computed efficiently? Temperature and top-p answer the first question. Random sampling or argmax answers the second. Exact speculative decoding answers the third. Interfaces often mix all three under one generation_config, which encourages conceptual mistakes.

Order also matters. Suppose a service first applies temperature and then top-p. Sharpening can cause the cumulative threshold to be reached by fewer tokens. Reversing the operations means truncation is calculated on the original distribution, after which temperature only reshapes the retained set. These are different q_t distributions. A deployment record that stores only “temperature 0.7, top-p 0.9” is incomplete unless it also records processor order and renormalisation.

Current generation libraries expose many such processors, including top-k, top-p, typical, epsilon and eta cut-offs, repetition controls and forced tokens. Their availability is implementation evidence, not a recommendation to combine them all. The central engineering task is to make each intervention earn its place against a stated loss.[11]

flowchart TD
  A["One model, three jobs, one<br/>dangerous default"] --> B["The output law begins after<br/>the model"]
  B --> C["Five interventions, five kinds<br/>of control"]
  C --> D["Control has a boundary"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#D95545,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Five interventions, five kinds of control

Greedy decoding collapses uncertainty

Greedy decoding selects the token with the largest current probability. Formally, q_t is a point mass on argmax_v p_t(v). It is repeatable when the model, numerical kernels and tie-breaking are repeatable. It is also myopic: the locally best token need not begin the highest-probability complete sequence, much less the sequence with highest task utility.

Greedy selection is sensible when there is effectively one acceptable continuation, the model is well aligned with that target and downstream checks can reject errors. It is weak evidence of truth. A confident wrong distribution produces a repeatable wrong answer. For open-ended text, aggressively seeking high-probability continuations has historically produced generic or repetitive generations in particular model and task settings. Holtzman and colleagues demonstrated this on long-form GPT-2 continuation and introduced nucleus sampling as an alternative for that setting.[1] The result does not establish that sampling is universally superior.

Temperature bends odds while preserving rank

Temperature divides logits before softmax. If the original probabilities are strictly positive, an equivalent expression is:

T is the temperature. Values below one sharpen the distribution; values above one flatten it. The odds between two tokens become (p_i / p_j)^(1/T). Lowering T enlarges an existing preference. Raising it compresses that preference. For finite logits, temperature does not change rank and does not make a possible token impossible. It reallocates mass across the existing support.

Temperature is therefore a confidence amplifier or attenuator, not an independent creativity variable. If the model is miscalibrated, sharpening can magnify the error. If the distribution is already flat because the task permits many continuations, moderate flattening may increase variety at tolerable cost. The correct judgement depends on sequence-level utility, not the aesthetic impression of a few samples.

Nucleus sampling moves the support boundary

Nucleus, or top-p, sampling sorts tokens by probability and retains the smallest set whose cumulative mass reaches threshold p. It sets every excluded probability to zero and renormalises the retained mass. Unlike top-k, its candidate count adapts to the shape of the current distribution. A peaked distribution may retain one token; a flat distribution may retain hundreds.

This adaptivity is useful, but the threshold is not a calibrated confidence level. A top-p set containing 90 per cent of model mass does not mean the correct next token is inside it with 90 per cent real-world probability. It means only that the model assigned 90 per cent of its own local mass to the retained tokens. The distinction matters under miscalibration, distribution shift and rare entities.

Research after the original nucleus work makes the boundary clearer. Hewitt, Manning and Liang interpret truncation as attempted “desmoothing”: removing probability assigned outside the support of plausible language. Their eta-sampling rule combines an absolute threshold with an entropy-dependent threshold and shows that top-p can over-truncate low-entropy contexts.[3] Finlayson and colleagues provide a related theoretical account of why threshold truncation can remove model artefacts under assumptions, while necessarily excluding some valid low-probability tokens too.[4]

Tail removal creates an accessibility decision. It can suppress nonsense, but it can also make a rare surname, minority-language form, unusual code identifier or surprising yet correct fact unreachable. That effect should be tested with targeted rare-token cases, not inferred from average fluency.

Constrained decoding changes the admissible language

A formal decoder uses a grammar, automaton, trie, schema or parser state to compute which next tokens can still lead to a valid output. It masks the rest and renormalises the allowed set. PICARD demonstrated incremental parsing for text-to-SQL, rejecting inadmissible continuations during decoding rather than filtering only at the end.[5] Later work showed how input-dependent grammars can support several structured NLP tasks without task-specific fine-tuning.[6]

The guarantee is bounded by what the formal system represents and what the engine correctly implements. A JSON grammar can guarantee valid JSON syntax. A schema can constrain types, required properties and enumerations. Neither establishes that the chosen value is supported by evidence, permitted for the user or safe to execute. JSONSchemaBench also shows that real engines differ in schema coverage, over-constraint, under-constraint and latency, so the phrase “schema constrained” is not itself an implementation guarantee.[7]

This difference is not automatically a defect. Local masking is often chosen because it is tractable and guarantees membership in the represented language. It becomes a defect when engineers describe the result as a sample from the model “conditioned on validity” or assume that relative semantic preferences remain unchanged.

Exact speculative decoding changes the schedule, not the law

Autoregressive generation is serial: a target model normally computes one next-token distribution, samples a token, then repeats. Speculative decoding lets a cheaper draft model propose several tokens. The target scores the proposed block in parallel. An acceptance test keeps proposals compatible with the target; a correction distribution handles rejection. In the exact stochastic algorithm, these rules are constructed so the final samples follow the target distribution.

That makes exact speculation a powerful negative control. If outputs change beyond expected sampling noise, the implementation, numerics, random-number coupling or acceptance rule deserves investigation. Leviathan, Kalman and Matias reported two-to-threefold acceleration for T5-XXL in their setup with unchanged outputs; Chen and colleagues reported distribution-preserving speculative sampling and setup-specific speedups for Chinchilla models.[8][9] Those numbers are evidence about those systems, not portable service-level promises.

Stopping is a decoder intervention too

Generation ends when the decoder accepts an end-of-sequence token, matches a stop string or reaches a length limit. Each rule changes the distribution over complete outputs. A maximum-token cap can turn one coherent answer into a truncated one. A minimum-token rule can suppress a legitimate short answer. A stop string can terminate inside a quoted value if tokenisation and string matching are not aligned.

Length controls also interact with the earlier policies. Raising temperature may increase the chance of delaying the end token. A grammar may forbid EOS until all required fields are closed. Beam scores may favour short or long sequences depending on normalisation. Speculative decoding must verify stopping events with the same care as ordinary tokens. A complete decoder specification includes termination, because a sequence without its stopping law is not a complete probability model.

Test stopping with empty outputs, the shortest valid output, the longest permitted output, stop text split across tokens and content that legitimately contains the stop marker. These cases often reveal failures that ordinary quality samples miss.

Control has a boundary

The best decoder is task-relative

It is tempting to turn one successful setting into a house standard. The evidence argues against that. Wiher, Meister and Cotterell compared decoding methods across machine translation, summarisation, dialogue, story generation and unconditional generation. They found that relationships among probability, quality, diversity, repetition and length varied substantially by task. Mode-seeking methods were competitive for directed tasks, while story generation behaved differently.[2]

The decoder should be evaluated against the loss of the generation stage, not a generic notion of response quality. A translation system may value fidelity to one constrained target. A brainstorming system may value set-level coverage. A code generator may value passing tests. A status selector may value calibrated abstention and a deterministic policy. These losses imply different output laws.

Even “diversity” needs a denominator. Diversity across ten options for one prompt differs from variation across repeated customer notices. The former may be useful; the latter may be a defect. A stochastic policy can improve set coverage while damaging single-output reliability. A deterministic policy can improve reproducibility while hiding uncertainty.

Negative control: when decoding barely matters

Consider a conditional distribution whose top token has probability 0.90. Greedy chooses it. Temperature 0.7 raises its mass to roughly 0.98. Top-p 0.90 keeps only it. Across ordinary sample counts, all three policies appear identical. This does not prove that the policies are equivalent. It shows that the base distribution is so concentrated that the intervention has little observable room.

This is an important counterexample to decoder-centred explanations. A better trained or better conditioned model can make reasonable policies converge. Conversely, when several continuations have similar mass, the same policy differences become visible. The right experiment varies both the decoder and the distributional regime.

Syntax can become a semantic trap

Suppose evidence is insufficient and policy requires review. The model assigns approve probability 0.36, review 0.29 and decline 0.15. A grammar masks every other token and renormalises these three. It now guarantees a valid enum, but approve becomes probability 0.45. The output is easier to parse and still wrong for the case.

The remedy is not a more elaborate grammar unless the semantic rule itself is mechanically expressible. In the workflow scenario, deterministic policy should first compute whether evidence is sufficient. That result can narrow the allowed enum to review. The model may then draft an explanation, but it does not acquire authority to override the policy decision.

Formal constraints are strongest when they encode true invariants and weakest when they disguise unresolved judgement as syntax. This is why constrained output belongs beside semantic validation, evidence checks and action authorisation, not in place of them.

The explicit failure boundary

There is a second boundary at low valid mass. If a schema leaves only continuations to which the model assigns negligible probability, renormalisation can force the system far from its learned regime. The output may be syntactically valid yet semantically brittle. This condition should trigger a refusal, repair prompt, alternative model or deterministic constructor rather than unlimited retries.

A third boundary concerns reproducibility. A seed does not by itself guarantee replay across model revisions, tokeniser changes, kernels, batching, devices or altered random-number consumption. For consequential generation, store the model and tokeniser identifiers, complete decoding pipeline, seed, stopping reason and enough input evidence to reconstruct the decision context.

stateDiagram-v2
  [*] --> Claim
  state "A four-token worked example" as Claim
  state "Selection, transformation and<br/>execution are different<br/>operations" as Test
  state "Greedy decoding collapses<br/>uncertainty" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Choose by task loss, then test

A minimal decision instrument

The selection process begins with the output contract, not the model API. Ask how many answers are acceptable, which invariants are mechanical, how costly variation is and whether latency work must preserve the current law. Then choose the smallest intervention that changes the required property.

The decoder laboratory

The executable artefact uses an eight-token synthetic vocabulary and three controlled contexts. It isolates the transformations without downloading a model. The open context has several plausible continuations and a low-probability rare_name. The peaked context is the negative control. The governed context permits three enum values while marking only review as semantically correct for the scenario.

The laboratory computes support size, Shannon entropy and total variation distance from the base distribution. Total variation is half the summed absolute probability difference. A value of zero means two distributions match exactly; a value of one means they have disjoint support. It then draws 100,000 exact speculative samples from a deliberately imperfect draft distribution and compares the empirical result with the target.

The positive constrained case reports enum-valid probability 1.000. The negative case reports semantic correctness only 0.362, leaving 0.6370.9300.0024 from the target at 100,000 samples. That residual is ordinary finite-sample error, not a theoretical change to the target law.

#!/usr/bin/env python3
"""A dependency-free laboratory for distribution-changing decoders.

The examples use synthetic one-step token distributions. They isolate what a
policy changes before a real model, tokenizer, prompt or serving stack adds
confounders.

Run:
    python3 decoder_lab.py

Expected findings:
* greedy collapses the distribution to one token;
* temperature reshapes odds but preserves support and rank;
* nucleus sampling removes tail support and renormalises what remains;
* a grammar mask can guarantee a valid enum while leaving semantic error;
* exact speculative sampling changes the proposal path, not the target law.
"""

from __future__ import annotations

from collections import Counter
from dataclasses import dataclass
import math
import random
from typing import Callable, Iterable, Mapping, Sequence

Token = str
Distribution = dict[Token, float]

TOKENS: tuple[Token, ...] = (
    "approve", "review", "decline", "explain",
    "escalate", "unsafe", "malformed", "rare_name",
)

CONTEXTS: dict[str, Distribution] = {
    "peaked": dict(zip(TOKENS, (0.900, 0.040, 0.020, 0.015, 0.010, 0.007, 0.005, 0.003))),
    "open": dict(zip(TOKENS, (0.280, 0.200, 0.150, 0.120, 0.100, 0.070, 0.050, 0.030))),
    "governed": dict(zip(TOKENS, (0.360, 0.290, 0.150, 0.050, 0.040, 0.040, 0.040, 0.030))),
}


def normalise(weights: Mapping[Token, float]) -> Distribution:
    """Return non-negative weights normalised to sum to one."""
    if not weights:
        raise ValueError("distribution must not be empty")
    if any((not math.isfinite(value)) or value < 0 for value in weights.values()):
        raise ValueError("weights must be finite and non-negative")
    total = sum(weights.values())
    if total <= 0:
        raise ValueError("at least one weight must be positive")
    return {token: value / total for token, value in weights.items()}


def assert_distribution(dist: Mapping[Token, float], *, tolerance: float = 1e-12) -> None:
    if set(dist) != set(TOKENS):
        raise ValueError("distribution must contain the laboratory vocabulary exactly")
    if any((not math.isfinite(value)) or value < 0 for value in dist.values()):
        raise ValueError("probabilities must be finite and non-negative")
    if abs(sum(dist.values()) - 1.0) > tolerance:
        raise ValueError(f"probabilities must sum to one, got {sum(dist.values()):.12f}")


def greedy(dist: Mapping[Token, float]) -> Distribution:
    """Place all mass on the highest-probability token."""
    winner = max(dist, key=dist.__getitem__)
    return {token: 1.0 if token == winner else 0.0 for token in dist}


def temperature(dist: Mapping[Token, float], value: float) -> Distribution:
    """Apply q_i proportional to p_i ** (1 / T)."""
    if value <= 0 or not math.isfinite(value):
        raise ValueError("temperature must be finite and greater than zero")
    transformed = {
        token: (probability ** (1.0 / value) if probability > 0 else 0.0)
        for token, probability in dist.items()
    }
    return normalise(transformed)


def top_p(dist: Mapping[Token, float], threshold: float) -> Distribution:
    """Keep the smallest ranked set whose cumulative mass reaches threshold."""
    if not 0 < threshold <= 1:
        raise ValueError("top-p threshold must be in (0, 1]")
    ranked = sorted(dist.items(), key=lambda pair: (-pair[1], pair[0]))
    kept: set[Token] = set()
    cumulative = 0.0
    for token, probability in ranked:
        kept.add(token)
        cumulative += probability
        if cumulative >= threshold:
            break
    return normalise({token: probability if token in kept else 0.0 for token, probability in dist.items()})


def constrain(dist: Mapping[Token, float], allowed: Iterable[Token]) -> Distribution:
    """Mask tokens outside an admissible set, then renormalise."""
    allowed_set = set(allowed)
    unknown = allowed_set.difference(dist)
    if unknown:
        raise ValueError(f"constraint names unknown tokens: {sorted(unknown)}")
    return normalise({token: probability if token in allowed_set else 0.0 for token, probability in dist.items()})


def entropy_bits(dist: Mapping[Token, float]) -> float:
    return -sum(probability * math.log2(probability) for probability in dist.values() if probability > 0)


def support_size(dist: Mapping[Token, float]) -> int:
    return sum(probability > 0 for probability in dist.values())


def total_variation(left: Mapping[Token, float], right: Mapping[Token, float]) -> float:
    return 0.5 * sum(abs(left[token] - right[token]) for token in left)


def sample_categorical(dist: Mapping[Token, float], rng: random.Random) -> Token:
    draw = rng.random()
    cumulative = 0.0
    final_token = next(reversed(dist))
    for token, probability in dist.items():
        cumulative += probability
        if draw < cumulative:
            return token
    return final_token


def empirical(draw: Callable[[], Token], n: int) -> Distribution:
    if n <= 0:
        raise ValueError("sample count must be positive")
    counts = Counter(draw() for _ in range(n))
    return {token: counts[token] / n for token in TOKENS}


@dataclass(frozen=True)
class SpeculativeResult:
    token: Token
    accepted_draft: bool


def speculative_once(
    target: Mapping[Token, float],
    draft: Mapping[Token, float],
    rng: random.Random,
) -> SpeculativeResult:
    """Perform one exact speculative-sampling step."""
    proposal = sample_categorical(draft, rng)
    q = draft[proposal]
    p = target[proposal]
    acceptance_probability = min(1.0, p / q) if q > 0 else 0.0
    if rng.random() < acceptance_probability:
        return SpeculativeResult(proposal, True)
    residual = normalise({token: max(target[token] - draft[token], 0.0) for token in target})
    return SpeculativeResult(sample_categorical(residual, rng), False)


def row(name: str, dist: Mapping[Token, float], base: Mapping[Token, float]) -> str:
    reachable = ",".join(token for token, probability in dist.items() if probability > 0)
    return (
        f"{name:14s} support={support_size(dist):d} "
        f"entropy={entropy_bits(dist):.3f} bits "
        f"TV(base)={total_variation(dist, base):.3f} "
        f"rare_name={dist['rare_name']:.4f} reachable=[{reachable}]"
    )


def main() -> None:
    for dist in CONTEXTS.values():
        assert_distribution(dist)

    print("DECODER INTERVENTIONS ON THE SYNTHETIC OPEN CONTEXT")
    base = CONTEXTS["open"]
    policies: Sequence[tuple[str, Distribution]] = (
        ("base", base),
        ("greedy", greedy(base)),
        ("temperature=.7", temperature(base, 0.7)),
        ("temperature=1.4", temperature(base, 1.4)),
        ("top_p=.90", top_p(base, 0.90)),
    )
    for name, dist in policies:
        print(row(name, dist, base))

    print("\nNEGATIVE CONTROL: A PEAKED CONTEXT")
    peaked = CONTEXTS["peaked"]
    for name, dist in (
        ("greedy", greedy(peaked)),
        ("temperature=.7", temperature(peaked, 0.7)),
        ("top_p=.90", top_p(peaked, 0.90)),
    ):
        print(f"{name:14s} argmax={max(dist, key=dist.__getitem__):10s} approve_mass={dist['approve']:.4f}")

    print("\nCONSTRAINED OUTPUT: SYNTAX POSITIVE, SEMANTICS NEGATIVE")
    governed = CONTEXTS["governed"]
    allowed_statuses = {"approve", "review", "decline"}
    constrained = constrain(governed, allowed_statuses)
    syntactic_validity = sum(constrained[token] for token in allowed_statuses)
    semantic_correctness = constrained["review"]
    print(f"valid_enum_probability={syntactic_validity:.3f}")
    print(f"correct_review_probability={semantic_correctness:.3f}")
    print(f"wrong_but_valid_probability={1.0 - semantic_correctness:.3f}")

    print("\nEXACT SPECULATIVE SAMPLING")
    target = base
    draft = normalise(dict(zip(TOKENS, (0.340, 0.180, 0.140, 0.110, 0.080, 0.060, 0.050, 0.040))))
    rng = random.Random(75)
    n = 100_000
    accepted = 0

    def draw() -> Token:
        nonlocal accepted
        result = speculative_once(target, draft, rng)
        accepted += int(result.accepted_draft)
        return result.token

    observed = empirical(draw, n)
    print(f"samples={n}")
    print(f"draft_acceptance_rate={accepted / n:.3f}")
    print(f"empirical_TV_from_target={total_variation(observed, target):.4f}")
    print("target       " + " ".join(f"{target[token]:.3f}" for token in TOKENS))
    print("speculative  " + " ".join(f"{observed[token]:.3f}" for token in TOKENS))


if __name__ == "__main__":
    main()

DECODER INTERVENTIONS ON THE SYNTHETIC OPEN CONTEXT
base           support=8 entropy=2.725 bits TV(base)=0.000 rare_name=0.0300
greedy         support=1 entropy=-0.000 bits TV(base)=0.720 rare_name=0.0000
temperature=.7 support=8 entropy=2.507 bits TV(base)=0.094 rare_name=0.0146
temperature=1.4 support=8 entropy=2.848 bits TV(base)=0.068 rare_name=0.0470
top_p=.90      support=6 entropy=2.442 bits TV(base)=0.080 rare_name=0.0000

CONSTRAINED OUTPUT: SYNTAX POSITIVE, SEMANTICS NEGATIVE
valid_enum_probability=1.000
correct_review_probability=0.362
wrong_but_valid_probability=0.637

EXACT SPECULATIVE SAMPLING
samples=100000
draft_acceptance_rate=0.930
empirical_TV_from_target=0.0024

A production-shaped decoder test

Return to the case workflow. Build three evaluation slices rather than one aggregate benchmark. The status slice contains cases with a unique policy outcome, incomplete evidence and deliberately conflicting evidence. The explanation slice measures claim support, prohibited disclosure, numeric fidelity and reader comprehension. The options slice measures unique feasible proposals, duplication and analyst-rated usefulness across the set.

For status, compare greedy, constrained greedy and deterministic policy construction. A valid enum is only the first gate. Report policy accuracy and abstention by evidence state. For explanations, sweep a small, declared grid of temperature and truncation settings across fixed prompts and model versions. Report distributions, not a favourite example. For options, sample several outputs per case and score set coverage as well as per-output defects.

Then add fault cases. Remove the rare name from the high-probability head. Place the correct status just outside the top-p boundary. Give the grammar a schema feature the engine does not support. Lower valid probability mass until renormalisation forces an awkward path. Replace the draft model with a poorly matched one and verify that exact speculation loses acceptance and speed without changing the output law.

A decoder release passes only when its intended intervention is visible and its unintended intervention is bounded. For a distribution-changing policy, measure what probability moved and which cases became unreachable. For a distribution-preserving optimisation, test equivalence and latency separately.

A disciplined evaluation protocol

Begin with decoder unit tests that do not require a language model. Feed known logits into each processor and assert the exact transformed probabilities, allowed support and renormalisation. Include ties, all-masked states, very small probabilities, infinite logits and a threshold that lands exactly on a cumulative boundary. These tests establish that the implementation matches the intended mathematics.

Next run paired model experiments. Freeze the model, tokeniser, prompts, context construction and stopping limits. Change one decoder feature at a time. For deterministic policies, compare outputs directly. For stochastic policies, draw enough samples to estimate distributions over task-relevant events. Pairing removes prompt and model variation from the causal contrast. A small hyperparameter grid is more informative than a large undirected sweep because each point should correspond to a stated hypothesis about concentration, support or validity.

Stratify results by conditions the mechanism predicts should matter. For temperature, stratify by base entropy and top-token margin. For nucleus sampling, record retained set size, removed mass and whether required rare tokens remain reachable. For constraints, record valid mass before masking, parser state, schema feature coverage and wrong-but-valid outcomes. For speculation, record draft acceptance, target calls, corrected positions and equivalence to the non-speculative baseline.

Then evaluate complete sequences against the stage loss. Fluency alone is insufficient. Use exact correctness or test execution for narrow tasks, evidence support for explanations, set coverage for ideation, conformance for formal outputs and calibrated abstention where the system may defer. Report the denominator, repeated-run distribution and uncertainty. One attractive sample cannot establish a decoder policy.

Finally, inject failures at the decoder boundary. Move a necessary token just below the truncation threshold. Make all grammar-valid tokens low probability. Introduce a schema keyword unsupported by the engine. Change processor order. Force an early stop. Give the draft model a badly mismatched distribution. The release should either preserve the declared property or fail visibly. Silent fallback from constrained to unconstrained generation is a control failure, not graceful degradation.

This protocol separates three claims that are often blended: the transform is mathematically correct, the induced outputs improve the chosen task metric, and the serving implementation preserves both under load. Each needs its own evidence.

The distribution intervention record

Store one compact record for every released decoding policy. It makes configuration review possible and prevents a service default from becoming invisible application logic.

Compact glossary