A language model does not emit a sentence in one indivisible act. At each step it produces scores over possible next tokens. A decoder then decides how those scores may become the next token, which changes the prefix, which changes the next distribution.

Let the model’s conditional distribution at position t be p_θ(x_t | x<t, c), where θ denotes learned parameters, x<t is the prefix already generated, and c is the supplied context. A decoding policy π receives that distribution plus settings such as temperature, a truncation rule, a random seed and a stopping condition. It returns a new distribution or a selected continuation.

This notation matters because it prevents a common collapse. The learned distribution, the transformed decoding distribution, the realised token and the finally released answer are four different objects. A selector, verifier or human reviewer may act after generation. A tool result may alter the prefix midway. A content rule may forbid a token regardless of probability. The observable answer therefore belongs to a configured system, not to the weight file in isolation.

Four claims hide inside one generated sentence

Consider the sentence “The transfer is pending.” It can support at least four distinct claims. A distributional claim says the model assigned some probability to the token sequence under the supplied prefix. A policy claim says the decoder retained and reweighted that sequence. A realisation claim says this particular random or deterministic path produced it. A release claim says the configured selector and authority allowed the sentence to reach the user.

These claims require different evidence. A logit trace can support the first two. A seed, operation order and stopping receipt support the third. A selector record, verification result and policy decision support the fourth. The final string supports none of them by itself. This is why two evaluations can disagree without either mismeasuring. Top-one accuracy asks whether a local maximum was correct. Pass-at-k asks whether at least one useful answer appeared among candidates. Majority aggregation asks whether an answer recurred. Human preference asks which released text a rater preferred. Each test observes a different slice of the configured process.

The distinction also prevents an attribution reversal. Suppose a constrained JSON decoder forces valid syntax and a verifier rejects unsupported fields. The resulting system may look more reliable even though the underlying model distribution did not improve. Conversely, a model revision may improve its probability ranking while a warmer default makes released answers less stable. Capability evidence and expression-policy evidence must be kept separate before they are recombined.

Research on neural text degeneration made this visible early. Maximising likelihood through greedy or beam-style decoding could produce dull or repetitive text, while nucleus sampling from the same model produced a different quality-diversity regime.[1] A broader comparison found that several popular samplers share distributional properties such as entropy reduction and order preservation, and that violating those properties can sharply damage the measured trade-off.[2] These results do not establish one universal decoder. They establish that decoder choice is part of the causal explanation of output quality.

A useful counterfactual audit now becomes possible. Preserve the model, context and logits, then swap only the decoder. If behaviour changes, the decoder is a cause of the difference. Preserve the released answer, then inspect whether it arose as a dominant mode, a rare sample or a selected candidate. If its provenance changes, identical text no longer licences identical confidence. Finally, preserve model and decoder but swap the selector. If the released answer changes, the selector belongs inside the evaluated system boundary.

This audit is more demanding than publishing a single prompt and response, but it is also cheaper than guessing at hidden competence. It asks for controlled interventions on components that are already configurable. The result is a causal attribution map: what the model made probable, what the decoder made expressible, what randomness realised, and what authority released.

flowchart LR
  A["The answer is a configured event"] --> B["Three levers that do different<br/>causal work"]
  B --> C["A token choice becomes a<br/>trajectory"]
  C --> D["Build a decoding laboratory,<br/>not a folklore table"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#007D68,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Temperature is an odds exponent, not a creativity substance

Suppose the model probabilities over five candidate tokens are 0.50, 0.25, 0.15, 0.07 and 0.03. Temperature transforms them by raising each probability to the power 1/T and renormalising. Equivalently, it divides the logits by T before softmax.

At T = 0.5, the leading token rises from 0.50 to about 0.734. At T = 2, it falls to about 0.348 while the fifth token rises from 0.03 to about 0.085. The order does not change and no positive-probability token is removed. What changes is the ratio between alternatives. Lower temperature amplifies the model’s existing preference, including an existing mistake. Higher temperature does not inject knowledge or imagination. It allows weaker alternatives to win more often.

Truncation edits the set of possible next tokens

Top-k keeps a fixed number of highest-ranked tokens. Top-p keeps the smallest ranked set whose cumulative transformed probability reaches p. Locally typical sampling favours tokens whose information content is close to the distribution’s conditional entropy.[4] Entropy-dependent methods such as eta-sampling alter the threshold with context.[3] Relative rules such as min-p compare each token with the current maximum. All of them perform support surgery: some tokens receive exactly zero probability, then the survivors are renormalised.

That zero matters. Temperature makes an unlikely token rarer or more common. Truncation makes it impossible at that step. This can remove model noise, but it can also remove a rare correct name, a necessary punctuation token, a minority-language continuation or the only adequate escape from an early mistake. The decoder has no independent access to truth. It is estimating which portion of the model’s support deserves to survive.

Hewitt, Manning and Liang interpret truncation as desmoothing: the model distribution can be treated as a mixture of a target distribution and a smoothing tail, so the decoder tries to recover plausible support.[3] Their analysis also shows why a generic cumulative rule can remove a plausible high-probability word in a sharply peaked context. Meister and colleagues describe truncation methods as sampling adapters that often trade recall for precision: fewer strings remain expressible, but a larger fraction of surviving strings may be desirable.[5] This is a better mental model than “top-p controls creativity”.

Search substitutes a trajectory objective

Greedy decoding selects the highest-probability token at each step. Beam search keeps several partial sequences, extends them, prunes them, and returns a high-scoring complete sequence under a chosen score and stopping rule. Search can recover a path whose first token was not locally best. It can also optimise the wrong objective more effectively.

Neural machine translation provides a clean warning. Exact or wider search has exposed models whose highest-scoring output is too short or even empty, because locally normalised probabilities and sequence scoring do not fully encode adequacy.[8] Other analyses show that beam behaviour depends on length treatment, stopping criteria and implicit information-density effects.[7] These findings are task-specific, but the causal lesson travels: more search reduces search error only relative to the encoded score; it need not reduce model or decision error.

The sequence score itself deserves inspection. Summed log probabilities become more negative as a sequence grows, which creates pressure towards shorter outputs unless the task, end token and normalisation scheme counteract it. Length rewards, coverage terms, repetition penalties and contrastive objectives repair different symptoms by changing the score. Contrastive search, for example, combines model confidence with a degeneration penalty over representations rather than simply maximising likelihood.[10] Such methods can be valuable, but they are new objectives, not more faithful extraction of an untouched model belief.

Search evaluation should therefore separate three errors. A search error occurs when the algorithm misses the best sequence under its own score. A model error occurs when that score prefers an inadequate sequence. A decision error occurs when the released sequence causes the wrong task outcome under the real loss function. Wider beams primarily target the first. Decoder folklore often assumes improvement in all three.

The practical impact of decoding appears over time. A token is more than appended to a sentence. It becomes part of the next condition under which the model computes.

The probability of a complete sequence is a product of conditional probabilities along its realised prefix. Change an early token and every later factor may change. This is why “temperature 0.8” cannot be assigned a stable semantic meaning across prompts or even across positions within one response. At one step the distribution may have a dominant head. At the next it may be flat. A static decoder applies the same policy to both geometries.

Adaptive decoders respond to this problem by changing control during generation. Mirostat uses feedback to target surprise or perplexity instead of fixing k or p throughout a sequence.[6] Locally typical and entropy-dependent methods also condition their support rule on the current distribution.[4] Recent preprint work goes further and tries to make truncation boundaries less sensitive to temperature by using relative logit geometry.[12] These proposals differ, but they share an important diagnosis: static thresholds ignore how uncertainty changes from prefix to prefix.

Measure the path at two scales

Token-level measurements describe the intervention directly. Entropy shows concentration. Effective support, exp(H) when H uses natural logarithms or 2^H in bits, estimates how many equally likely choices would create the same uncertainty. Retained mass shows how much transformed probability survives truncation. Top-token margin shows how easily a small perturbation could reverse the leading choice. These quantities reveal geometry, but they do not say whether a sentence is useful.

Sequence-level measurements describe consequences. Useful candidates per unit compute, semantic diversity, repetition, factual error, unsafe assertion rate, calibration of abstention, selector burden and task-weighted loss are all possible outcomes. They should be evaluated across seeds and prefix families. A decoder can lower token entropy while increasing sequence error if it locks onto a wrong early branch. It can raise lexical diversity without producing independent reasoning paths. It can reduce repetition in short samples yet drift over a long horizon.

The two scales must be linked. Record a geometry snapshot at each decisive step, then ask which snapshots precede failures. If unsafe outputs occur only when the head is flat, adaptive abstention may help. If they occur under sharp heads, retrieval or model correction is more plausible. If repetition follows a sampled phrase that raises its own future probability, the failure is a feedback loop rather than a static tail problem. A decoder metric becomes explanatory only when it predicts a sequence consequence under intervention.

One transcript does not reveal confidence

Suppose two configured systems both answer “42”. In system A, the answer held 0.90 probability and greedy decoding selected it. In system B, it held 0.35 probability among several plausible answers and happened to be sampled. The text is identical. The evidential meaning is not. Conversely, two different texts may express equally supported paraphrases.

Diversity can become computation only through selection

Sampling several candidates can improve a final decision when the candidates contain useful variation and a selector can recognise it. Self-consistency for chain-of-thought reasoning samples multiple reasoning paths and aggregates their answers; the reported experiments improved accuracy on several reasoning benchmarks compared with greedy chain-of-thought decoding.[9] The mechanism is not that randomness makes every path smarter. It is that sampling exposes several paths and aggregation exploits repeated agreement.

Agreement is still not truth. Paths may share a systematic misconception, copied premise or prompt-induced bias. Majority voting can magnify a correlated error. Open-ended answers may not be cleanly aggregable. Selection can also leak a separate model’s preferences into the result. Multiple samples become additional compute only when diversity, independence and selection quality are measured.

A proper multi-sample report therefore includes more than sample count. It records the decoder used to create variation, the semantic clustering rule, the answer extractor, the selector’s access to evidence, the rate of unanimous wrong answers and the marginal gain from each additional sample. It also compares against spending the same compute on a stronger prompt, retrieval, a verifier or a larger model. Without this matched budget, “self-consistency helped” may mean only that more inference was purchased.

There is also a stopping problem. If samples arrive sequentially, the system may stop when one answer reaches a threshold, but repeated correlated draws can create false certainty. A safer design treats agreement as one signal beside evidence quality and path independence. When the task has no reliable aggregation function, retaining several visibly different candidates for human judgement may be more honest than manufacturing a single consensus.

The Geometry-Loss-Horizon test

A useful decoder decision begins with three properties. Geometry describes the local distribution: top-token margin, entropy, tail slope and sensitivity to small logit changes. Loss describes what errors cost: false assertion, missed alternative, monotony, review burden or irreversible action. Horizon describes how long and stateful the trajectory is: one label, a paragraph, a tool-using plan or a continuing agent episode.

This Geometry-Loss-Horizon test is practitioner synthesis, not a published theorem. It explains why global settings fail. A flat distribution with low-cost creative loss may justify broad sampling. A flat distribution with high-cost factual loss should trigger retrieval, verification or abstention, rather than only a colder temperature. A long horizon makes feedback and periodic checks more valuable because local errors compound. A sharp distribution only supports decisive output when calibration and evidence justify the sharpness.

stateDiagram-v2
  [*] --> Claim
  state "Four claims hide inside one<br/>generated sentence" as Claim
  state "Temperature is an odds<br/>exponent, not a creativity<br/>substance" as Test
  state "Truncation edits the set of<br/>possible next tokens" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Teams often inherit a table that says temperature 0.2 is factual and 0.9 is creative. Such labels conceal the local distribution, truncation order, seed variance, selector and task loss. A useful laboratory holds the model fixed and treats decoder policy as the intervention.

The executable lab below uses a synthetic three-step language model for an investigation conclusion. Its probabilities are deliberately small enough to inspect. Each generated sequence combines a qualifier, a finding and a conclusion. Some combinations are marked as unsupported high-risk assertions by an external synthetic rule. The lab lets you vary temperature and one support rule, then estimates diversity, modal consistency, discarded probability mass and unsafe assertion rate over repeated seeded runs.

A minimum experimental protocol

Freeze the non-decoder variables. Register the model revision, tokenizer, prompt template, authorised evidence, tool responses and maximum horizon. Decoder experiments are uninterpretable when retrieval or prompts change at the same time.

Build a prefix panel. Do not test only complete prompts. Capture decisive prefixes with sharp heads, flat heads, long tails, near ties, rare proper nouns and known misleading maxima. Synthetic prefixes are useful for mechanism tests; authorised task data is required for operational claims. The panel should include cases where the desired token is near the truncation boundary.

Create an intervention matrix. Compare a small number of policies chosen for causal contrast: temperature only, fixed-count truncation, cumulative truncation, an adaptive rule, greedy choice and a sequence-level selector where the task supports it. Keep operation order explicit. Use registered seeds so every policy faces comparable random streams, while also repeating with fresh seed sets to test brittleness.

Declare loss before looking at outputs. Open-ended quality is not one number. Define the cost of false assertion, omission, repetition, invalid format, unsafe content, review time and lost diversity. Weight them according to the decision, not according to which decoder looks attractive. Include compute and latency when candidate generation or verification is part of the policy.

Inject failures. Flatten one sharp distribution, sharpen one wrong maximum, move a necessary token just outside the threshold, change the stopping token probability and perturb an early prefix. These interventions test mechanism. Natural benchmark averages can hide the exact geometry that breaks a policy.

Choose a profile with an abstention boundary. A result permits deployment only within the tested model, task, loss and horizon envelope. Record conditions that force a narrower profile, external verification or abstention. Re-run the panel after any model, tokenizer, prompt, retrieval, selector or framework change. The decoder contract is part of the release evidence, not a one-time tuning note.

How to use the laboratory

Begin at temperature 1 with no truncation, run 1,000 seeded trajectories, and record the six metrics. Then change one variable. A positive result means the candidate policy improves the metric that represents your declared loss while preserving required alternatives. For example, a lower unsafe rate is useful only if the policy does not erase necessary review language or collapse all outputs into one generic phrase.

A negative result is also informative. If several policies produce similar unsafe rates, the synthetic risk may be located in the model’s high-probability head rather than its tail. If diversity changes but decision loss does not, the selected metric may be insensitive. If results reverse under another seed or prefix geometry, the proposed setting is brittle. The laboratory is designed to reveal policy sensitivity, not to award a winning sampler.

The artefact encodes strong assumptions. It uses a frozen, hand-authored model; a short horizon; exact access to token probabilities; a deterministic external risk label; and no prompt, retrieval or classifier error. It cannot establish real-model factuality, human preference, reasoning faithfulness or consciousness. Its purpose is narrower: make the causal logic of decoding visible and give teams a repeatable experiment shape.

from __future__ import annotations

from dataclasses import dataclass
from math import exp, log
from random import Random
from typing import Iterable, Literal, Sequence

Method = Literal["sample", "greedy", "topk", "topp", "minp", "typical"]


@dataclass(frozen=True)
class TokenProb:
    token: str
    probability: float


def normalise(items: Iterable[TokenProb]) -> list[TokenProb]:
    values = list(items)
    total = sum(item.probability for item in values)
    if total <= 0:
        raise ValueError("Probability mass must be positive")
    return [TokenProb(item.token, item.probability / total) for item in values]


def apply_temperature(items: Sequence[TokenProb], temperature: float) -> list[TokenProb]:
    if temperature <= 0:
        raise ValueError("temperature must be greater than zero")
    powered = [TokenProb(item.token, item.probability ** (1.0 / temperature)) for item in items]
    return normalise(powered)


def truncate(
    items: Sequence[TokenProb],
    method: Method,
    *,
    top_k: int = 3,
    top_p: float = 0.9,
    min_p: float = 0.1,
    typical_mass: float = 0.9,
) -> list[TokenProb]:
    ranked = sorted(items, key=lambda item: item.probability, reverse=True)
    if method in {"sample", "greedy"}:
        kept = ranked
    elif method == "topk":
        kept = ranked[:max(1, top_k)]
    elif method == "topp":
        kept, cumulative = [], 0.0
        for item in ranked:
            kept.append(item)
            cumulative += item.probability
            if cumulative >= top_p:
                break
    elif method == "minp":
        threshold = min_p * ranked[0].probability
        kept = [item for item in ranked if item.probability >= threshold]
    elif method == "typical":
        entropy = -sum(item.probability * log(item.probability) for item in ranked)
        ordered = sorted(ranked, key=lambda item: abs(-log(item.probability) - entropy))
        kept, cumulative = [], 0.0
        for item in ordered:
            kept.append(item)
            cumulative += item.probability
            if cumulative >= typical_mass:
                break
    else:
        raise ValueError(f"Unknown method: {method}")
    return normalise(kept)


def draw(items: Sequence[TokenProb], rng: Random, greedy: bool = False) -> str:
    ranked = sorted(items, key=lambda item: item.probability, reverse=True)
    if greedy:
        return ranked[0].token
    threshold = rng.random()
    cumulative = 0.0
    for item in ranked:
        cumulative += item.probability
        if threshold <= cumulative:
            return item.token
    return ranked[-1].token


def decode_once(
    items: Sequence[TokenProb],
    *,
    method: Method,
    temperature: float,
    seed: int,
) -> str:
    transformed = apply_temperature(items, temperature)
    support = truncate(transformed, method)
    return draw(support, Random(seed), greedy=(method == "greedy"))


if __name__ == "__main__":
    base = [
        TokenProb("pending", 0.42),
        TokenProb("delayed", 0.28),
        TokenProb("review", 0.17),
        TokenProb("failed", 0.09),
        TokenProb("fraudulent", 0.04),
    ]
    print(decode_once(base, method="topp", temperature=1.0, seed=50))

{
  "model": {"id": "model-family", "revision": "sha-or-version", "tokenizer": "revision"},
  "context": {"prompt_hash": "sha256", "evidence_manifest": "manifest-id"},
  "decoder": {
    "operation_order": ["temperature", "top_p", "sample"],
    "temperature": 0.8,
    "top_p": 0.92,
    "top_k": null,
    "seed_policy": "registered-seed-set-v1",
    "stop": {"max_new_tokens": 320, "stop_strings": ["</answer>"]}
  },
  "selection": {"candidate_count": 8, "rule": "evidence-grounded-reranker-v2"},
  "release": {"abstain_on_instability": true, "risk_mass_ceiling": 0.01},
  "evaluation": {"task_loss": "false-assertion-weighted", "seed_count": 50}
}

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: The answer is a configured event
  M->>E: Four claims hide inside one generated sentence
  E-->>M: compare rival explanations
  E->>D: Version the whole path from scores to release
  D-->>Q: revise scope or act

The practical decision is no longer “which temperature makes this model good?” It is “which configured decoding policy is justified by this distribution geometry, this loss function and this trajectory horizon, and what evidence permits release?”

That changes evaluation. Hold the model, prompt and evidence fixed. Compare at least one odds intervention, one support intervention and, where relevant, one trajectory-selection method. Run a registered seed set. Measure outcome loss as well as diversity and average likelihood. Record transform order and stopping semantics. Inspect whether the preferred setting fails on sharp, flat and misleading distributions. Treat post-generation selection as a separate causal component.

It also changes system identity. A model revision with the same decoder may behave differently because its probability geometry changed. A decoder revision with the same model may change which claims can be expressed. A selector revision may alter final accuracy without touching either. Model, decoder, selector and release rule must be versioned together whenever their joint behaviour is the evaluated object.

For low-cost ideation, broad sampling and several candidates can be rational because novelty has value and selection is reversible. For bounded classification, calibration, constrained labels and abstention usually matter more than prose diversity. For factual synthesis, decoder tuning must sit behind authorised evidence and readback. For consequential action, the model’s text remains a proposal; typed authority and deterministic checks govern effects.

The strongest alternative explanation is that decoder sensitivity merely exposes a poorly trained or miscalibrated model. Often that is partly right. Better training, retrieval or calibration may reduce sensitivity. It does not make decoding neutral. Every autoregressive deployment still needs a rule for moving from scores to a trajectory, and that rule still has a loss profile.

The visible sentence is the end of a causal pipeline. Once that is understood, temperature stops being a personality dial, top-p stops being a universal quality filter, and beam width stops being synonymous with more intelligence. They become what they are: interventions whose value depends on the distribution they meet, the path they create and the decision they are allowed to influence.