The chain rule makes a tree

Let a sequence contain tokens x₁ through x_T. A token may be a character, word fragment, audio value, image code or another discrete unit. Probability theory permits the joint probability of the whole sequence to be written as a product of conditional probabilities:

This equation is the chain rule of probability. It is an identity, not an approximation and not a claim that the world itself thinks from left to right. We choose an order, then factor the joint distribution according to that order. Any finite joint distribution can be factorised this way, although some orders make the required conditional relationships much harder to learn.

The factorisation creates a supervised example at every position. From a training sequence such as “rain fell softly”, the model receives several conditional tasks: predict “rain” from the beginning marker, “fell” from “rain”, “softly” from “rain fell”, and the end marker from the full phrase. A corpus of N sequences with average length L supplies roughly N × L next-step targets. The targets overlap, so learning one reusable representation can influence many contexts.

Maximum-likelihood training usually minimises the negative log probability of the observed sequence. Taking a logarithm converts the product into a sum:

This additive loss is computationally convenient and statistically meaningful. It rewards calibrated probability mass on observed continuations rather than only the winning token. Early neural language models used distributed word representations to generalise beyond exact n-grams, explicitly aiming to learn a joint probability over word sequences while escaping the curse of dimensionality (Bengio et al., 2003). Later work showed that recurrent networks could generate long-range text and handwriting by predicting one point at a time (Graves, 2013), while WaveNet applied the same principle to raw audio samples conditioned on previous samples (van den Oord et al., 2016).

A minimal worked example: local maxima are not a global answer

Consider a two-token language. At the first position the model assigns 0.55 to A and 0.45 to B. After A, it assigns 0.51 to X and 0.49 to Y. After B, it assigns 0.99 to Z. The end marker is then certain.

A greedy decoder chooses the most probable token at every step, producing AX. Its complete probability is 0.55 × 0.51 = 0.2805. Yet BZ has probability 0.45 × 0.99 = 0.4455. The factorisation defines the global probabilities; the greedy procedure does not necessarily find the globally most probable sequence. This small example separates the model from the decoder before larger systems make the distinction harder to see.

Sampling performs a different operation. It draws a first token from the 0.55/0.45 distribution, then draws a second token from the corresponding conditional. Repeating the process many times reproduces the model's joint distribution in expectation. Beam search approximates a high-probability path by retaining several prefixes. Top-k and nucleus sampling deliberately truncate or reshape the conditional distribution. These policies operate on the same model but create different rollout distributions.

Thought experiment: the full ledger and the short-window clerk

Return to the boarding pass. Give two clerks exactly the same training strings and exactly the same next-character objective. The first clerk receives a lossless state that retains both route letters and both seat digits until the checksum. The second receives only the last three characters. Vary nothing else.

Both can learn that a pass begins with @, contains two letters and two digits, places a colon before the checksum, and ends with a semicolon. At the checksum position, however, the short-window clerk cannot distinguish two prefixes that share the same final three characters but differ in the first route letter. If those prefixes require different checksums, no prediction rule based only on the available state can be correct for both.

This is not exposure bias. It appears even when every prefix is copied perfectly from the data. It is an information boundary. Autoregression can compose only the dependencies represented in its conditioning state. A longer context window, recurrent state or attention mechanism may make the information available, but availability still does not prove causal use. An intervention that changes the remote route letter while holding the suffix fixed tests whether the checksum prediction actually depends on it.

flowchart TD
  A["The one-character clerk"] --> B["The chain rule makes a tree"]
  B --> C["Training beside the teacher,<br/>generating alone"]
  C --> D["Likelihood is not the product"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#5B5CF6,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Training beside the teacher, generating alone

During ordinary maximum-likelihood training, the prefix at position t comes from the dataset. The model predicts the next token while standing on a correct history, even when its own earlier prediction would have been wrong. This procedure is often called teacher forcing. The name can make it sound like an artificial convenience, but it follows directly from evaluating the chain-rule factors on an observed sequence.

For a causal Transformer, every ground-truth token is already present in the training example. A triangular attention mask prevents position t from seeing positions t and beyond, yet the model can calculate predictions for many positions in parallel, one source of the Transformer’s training efficiency (Vaswani et al., 2017). Generation is different. The token at position t does not exist until the model or decoder chooses it, and that choice becomes part of the input for position t + 1. Training can therefore be parallel across positions while ordinary left-to-right sampling remains sequential.

Where exposure errors enter

Suppose the data distribution supplies prefixes according to q_t, while the deployed model and decoder create prefixes according to r_t. Training estimates conditional performance mainly under q_t. Generation needs useful conditionals under r_t. The two distributions match only if the model and decoder reproduce the data process closely enough at every preceding position.

An early error can therefore have two effects. First, it is itself an incorrect token. Second, it changes the state on which later predictions are conditioned. The second effect is the exposure problem. Work on scheduled sampling framed this discrepancy directly: training uses the true previous token, while inference substitutes a generated token, so errors may accumulate (Bengio et al., 2015). Sequence-level training similarly targeted the gap between next-token objectives and whole-sequence generation metrics (Ranzato et al., 2015).

“May accumulate” is the important phrase. The popular picture is a snowball: one error creates a stranger prefix, the stranger prefix creates a larger error, and collapse becomes inevitable. That can happen, but it is not a theorem of autoregression. A model trained on a rich prefix distribution may recognise the new state and return to a high-density continuation. In open-ended text generation, He and colleagues found that distortion caused by generated rather than ground-truth prefixes was limited in their experiments and did not appear to grow incrementally; they described a self-recovery effect (He et al., 2021).

The correct question is therefore not “does exposure bias exist?” in the abstract. It is: which deviations move this configured model outside its competent prefix region, which properties can it recover, and over what horizon? Fluency may recover while factual identity remains wrong. Syntax may recover while a plan has crossed an irreversible branch. A benchmark that scores only surface quality can declare recovery even when the original task has been lost.

Thought experiment: one forced token, two meanings of recovery

Generate the synthetic pass @AH77:1;. Immediately after the first route letter, intervene and replace A with B. Nothing else changes. A state-sensitive generator now has two reasonable options, depending on the task contract.

If the printed prefix is authoritative, it should accept the edit, continue from @BH77:, and produce the checksum for the new route. The string can become globally valid again. This is recovery relative to the visible prefix. If the original route AH came from an external booking record, however, adapting to BH silently violates source fidelity. The model has recovered grammar while abandoning intent.

Vary only the recovery criterion and the same continuation changes from success to failure. That distinction matters in production. A conversational story may permit graceful improvisation. A generated calculation, identifier, citation or payment instruction may require the system to reject the altered state, retrieve the authoritative value, or restart. Self-recovery is always recovery of a named invariant, not a general property of good prose.

Why simply training on model mistakes is not a universal cure

Scheduled sampling mixes ground-truth and generated previous tokens during training, gradually exposing the learner to its own states. This can improve some tasks, but the objective must be examined carefully. Huszár argued that the original scheduled-sampling objective is not a proper scoring rule and can become statistically inconsistent, because the training target remains drawn from the original sequence after the conditioning prefix has been replaced (Huszár, 2015). The model may be asked to predict a continuation that no longer belongs to the altered prefix.

This criticism does not erase reported empirical gains. It changes what those gains establish. They support a method under particular tasks and decoders; they do not show that teacher forcing is generally wrong. Maximum likelihood remains the principled objective when the aim is to estimate the data distribution under the chosen factorisation. Sequence-level rewards, imitation-learning methods, denoising objectives and constrained decoding solve different problems and introduce their own bias, variance or optimisation debt.

Four mechanisms that are too often called exposure bias

State insufficiency occurs on clean prefixes. The information needed for the next decision is absent, compressed away or not used. More self-generated training data cannot reconstruct a variable the architecture never receives.

Prefix-distribution shift occurs when the model is competent on observed histories but brittle on histories created by its own plausible deviations. Perturbation and free-running tests reveal this gap.

Decoder distortion occurs when greedy, beam, truncated or temperature-scaled decoding creates a distribution unlike both the data and unmodified model sampling. Changing training may hide a poor decoding policy rather than repair the model.

Objective mismatch occurs when locally likely tokens do not compose into the property that matters, such as a valid programme, a faithful summary, a globally consistent plan or a useful translation score. Here the missing ingredient is an outcome signal, constraint or verifier, not merely wider prefix exposure.

Likelihood is not the product

Cross-entropy answers a precise question: how much probability did the model assign to observed next tokens under observed prefixes? Averaged over a representative test set, it is strong evidence about conditional distribution fit. It also has an information-theoretic interpretation as expected code length. What it does not directly answer is whether one deployed decoding policy will produce useful complete objects.

The distinction begins with the minimal example. Greedy decoding selected AX even though BZ had higher joint probability. Real vocabularies create vastly larger trees, so exact search is usually infeasible. Beam search retains several prefixes but can favour short or generic sequences unless length and coverage are handled. Temperature changes the sharpness of every conditional distribution. Top-k discards all but a fixed number of candidates. Nucleus sampling retains the smallest set whose cumulative probability crosses a threshold. Each policy changes which prefixes become future inputs.

Holtzman and colleagues showed that decoding by aggressively pursuing high-probability continuations can produce bland or repetitive text, and proposed nucleus sampling to avoid the unreliable tail while preserving diversity (Holtzman et al., 2020). The durable lesson is broader than one sampling rule. A model distribution and a decoder-induced distribution are different experimental objects. A generation report that names the model but omits temperature, truncation, beam width, stopping rule and constraints is missing part of the mechanism.

Token fit, sequence probability and sequence utility

Three quantities are often collapsed. Token fit is the average conditional loss on held-out data. Sequence probability is the product of a model's conditional factors for one complete candidate. Sequence utility is an external value assigned to that candidate for a task. The utility may reward factual support, human preference, executable correctness, diversity, safety or business outcome. It need not decompose token by token.

A code generator can assign high probability to every line and still miss a distant variable rename. A report generator can maintain excellent grammar while contradicting a source figure introduced hundreds of tokens earlier. A proof can contain individually familiar steps but use one invalid implication. In all three cases, local probability is useful evidence about familiarity and coherence, yet the decisive property lives at a larger scope.

One response is to add a sequence-level reward or verifier. Another is constrained decoding, which removes tokens or paths that violate a grammar or type system. A third is post-generation validation and repair. These mechanisms are complementary because they constrain different objects. A grammar can guarantee parseability without truth. A verifier can check a final property while missing an early unsafe action. A reward can improve average utility while creating new incentives and calibration problems.

A serious negative control: validity by replay

Suppose a generator returns only complete training examples. Every sample may be grammatical, checksum-valid and human-looking. An evaluator that checks only validity could rank this memoriser above a model that creates novel but occasionally flawed sequences. Yet replay has not learned the intended compositional rule, and it may disclose training content.

This is why generation evaluation needs a negative control capable of earning the visible success for the wrong reason. In the character laboratory later in this paper, a trie memoriser samples only training rows. It achieves 100 per cent format and checksum validity, but zero valid novelty. Its success is useful: it proves that validity alone cannot identify generalisation. The model must also face held-out likelihood, unseen combinations, duplication checks and interventions on the causal variables.

A deterministic grammar is another informative control. It can guarantee symbols, field order and termination without learning probabilities at all. If the product requires only a formal language, the grammar may be cheaper and more reliable. The autoregressive model earns its place only when uncertainty over legitimate continuations, flexible content or learned dependencies adds value beyond the deterministic baseline.

When another factorisation wins

Autoregression is powerful, but it is not the only way to learn from sequences. BERT uses a masked objective to learn bidirectional representations, allowing each masked position to condition on both left and right context during pretraining (Devlin et al., 2019). This is well suited to representation and conditional reconstruction, although ordinary BERT is not itself a left-to-right free-running generator.

Non-autoregressive translation models predict output positions in parallel rather than conditioning each word on previously generated outputs. The original formulation reported an order-of-magnitude inference-latency reduction, with a quality trade-off relative to its autoregressive teacher in those experiments (Gu et al., 2018). Parallelism is bought by modelling dependencies through latent variables, refinement, distillation or other structures rather than through a simple causal prefix.

Discrete diffusion offers another geometry. A corruption process masks or changes tokens, and a reverse process iteratively reconstructs a sequence. D3PM connected discrete diffusion to autoregressive and mask-based generation and reported strong character-level text results (Austin et al., 2021). More recent work such as LLaDA has trained larger masked diffusion language models from scratch and reported performance competitive with matched autoregressive baselines on several tasks (Nie et al., 2025).

These results challenge any claim that broad language capability logically requires left-to-right next-token prediction. They do not establish a universal replacement. A 2026 experimental analysis of eight diffusion language models across eight benchmarks found that denoising steps, context, block size and unmasking strategy materially shaped quality and efficiency trade-offs (Bertolani et al., 2026). The generation procedure remains part of the model in the operational sense.

stateDiagram-v2
  [*] --> Claim
  state "A minimal worked example:<br/>local maxima are not a global<br/>answer" as Claim
  state "Thought experiment: the full<br/>ledger and the short-window<br/>clerk" as Test
  state "Where exposure errors enter" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Audit the rollout, not only the token

The practical consequence is an evaluation change. Perplexity or held-out negative log-likelihood remains necessary when the claim concerns distribution fit. It is simply the first surface. A deployable generator also needs evidence about the prefixes it creates and the complete objects produced by its actual decoder.

Worked example: a character laboratory with a hidden global rule

The companion artefact creates all 6,400 possible synthetic passes of the form @RRDD:C;. There are eight route letters for each of two route positions and ten digits for each of two seat positions. The checksum is a deterministic function of all four symbols. A fixed random seed allocates 4,480 passes to training and 1,920 to held-out evaluation. Each model generates 5,000 samples.

The stateful rule model is an explicit positive control, not a learned neural network. It retains sufficient state and assigns uniform probabilities to legal route and seat choices before calculating the checksum. The 3-gram model learns character counts from the training split but can condition on only three previous characters. The trie memoriser is the negative control: it samples only paths observed in training.

These baselines isolate the argument. The stateful model asks whether local conditionals are logically sufficient when state is sufficient. The 3-gram asks whether excellent local surface regularity implies the distant relation. The trie asks whether complete-sequence validity proves generative generalisation.

The 3-gram result is the central contrast. It creates the right visible shape in 99.38 per cent of rollouts, but only 11.20 per cent satisfy the hidden checksum. In 88.80 per cent of samples it enters at least one prefix that never occurred in training. When the first route letter is edited while the visible three-character suffix is held fixed, its most probable checksum is correct only 14.22 per cent of the time. Surface fluency has not recovered the remote cause.

The trie produces a different warning. It reaches 100 per cent checksum validity by remaining entirely inside training prefixes. It produces no valid novel sample. Its poor held-out likelihood reflects the same limitation from another angle: when a held-out prefix leaves its stored tree, probability collapses to the fallback. Perfect sampled validity can be evidence of memorisation rather than compositional generation.

The stateful model produces 100 per cent valid strings and the correct checksum after every route intervention. Its valid-novel rate is approximately 30 per cent because the held-out partition contains approximately 30 per cent of the finite language. This baseline does not show that explicit rules always beat learned models. It shows that the local-to-global construction succeeds when the conditioning state contains the rule's sufficient variables.

Production-shaped worked scenario: a settlement-message draft

Assume a case record contains an amount of GBP 825.40, a beneficiary identifier, a value date and a purpose code. Historical approved messages provide training text. A next-token model may learn the local language of settlement instructions extremely well. It can still write “GBP 825.40” near the beginning and later state an “EUR total”, copy one digit incorrectly into a reference, or produce a formally valid purpose phrase unsupported by the case.

Surface one measures conditional fit on held-out approved drafts, stratified by field rarity and message length. Surface two edits one authoritative field in the prefix and asks whether every dependent field updates, whether the model flags the conflict, or whether it continues fluently with stale values. Surface three parses the finished message into a typed schema, compares every field with source evidence, enforces currency and identifier rules, checks duplication, and routes exceptions to review.

A constrained decoder can guarantee the schema and allowed code set. It cannot, by itself, prove that the selected beneficiary or amount came from the authorised case. A retrieval layer can provide the source facts. It cannot guarantee that the model copied them faithfully. A final validator can catch many mismatches. It cannot safely grant the model permission to execute a transfer. The architecture therefore keeps model proposal, deterministic validation, accountable approval and external effect as distinct operations.

This scenario also shows where autoregression earns value. It can express optional narrative fields, order evidence coherently and adapt wording to context. Deterministic code should calculate checksums, copy identifiers and enforce cross-field relations. The right system is not “autoregression everywhere” or “no model”. It places uncertain language generation inside a typed outcome boundary.

A compact architecture decision record

The executable artefact

The full laboratory uses Python 3.10 or later and only the standard library. It states its synthetic assumptions, includes positive and negative assertions, prints a JSON evidence record and is deterministic under seed 49. Save the embedded source as autoregression_character_lab.py and run python3 autoregression_character_lab.py. A separate copy is provided beside this article for direct execution.

Inspect the implementation in this paper and Review the recorded evidence in this paper.

"""Synthetic character-level autoregression lab for Paper 49.

The lab isolates three mechanisms:
1. a stateful autoregressive rule model that carries sufficient prefix state;
2. a learned fixed-order character n-gram model;
3. a trie memoriser that only replays training strings.

Every row is a synthetic boarding pass with the form @RRDD:C;, where RR is a
route, DD is a seat and C is a checksum determined by all four earlier symbols.
All data and thresholds are teaching fixtures. Python 3.10+. Standard library only.
"""
from __future__ import annotations

from collections import Counter, defaultdict
from dataclasses import asdict, dataclass
import json
import math
import random
import re
from typing import Iterable, Mapping

SEED = 49
ROUTES = "ABCDEFGH"
DIGITS = "0123456789"
START = "^"
ALPHABET = tuple("@ABCDEFGH0123456789:;")
FORMAT = re.compile(r"^@[A-H]{2}[0-9]{2}:[0-9];$")


def checksum(route1: str, route2: str, tens: str, ones: str) -> str:
    """A deterministic long-range relation used only by the synthetic language."""
    a = ROUTES.index(route1)
    b = ROUTES.index(route2)
    return str((3 * a + 5 * b + 7 * int(tens) + 11 * int(ones)) % 10)


def make_pass(route1: str, route2: str, tens: str, ones: str) -> str:
    return f"@{route1}{route2}{tens}{ones}:{checksum(route1, route2, tens, ones)};"


def corpus() -> list[str]:
    return [
        make_pass(a, b, t, o)
        for a in ROUTES
        for b in ROUTES
        for t in DIGITS
        for o in DIGITS
    ]


def is_format_valid(text: str) -> bool:
    return bool(FORMAT.fullmatch(text))


def is_checksum_valid(text: str) -> bool:
    return is_format_valid(text) and text[6] == checksum(text[1], text[2], text[3], text[4])


def is_partial_valid(prefix: str) -> bool:
    tests = (
        lambda p: len(p) < 1 or p[0] == "@",
        lambda p: len(p) < 2 or p[1] in ROUTES,
        lambda p: len(p) < 3 or p[2] in ROUTES,
        lambda p: len(p) < 4 or p[3] in DIGITS,
        lambda p: len(p) < 5 or p[4] in DIGITS,
        lambda p: len(p) < 6 or p[5] == ":",
    )
    return all(test(prefix) for test in tests)


def sample_distribution(dist: Mapping[str, float], rng: random.Random) -> str:
    threshold = rng.random()
    cumulative = 0.0
    items = sorted(dist.items())
    for token, probability in items:
        cumulative += probability
        if threshold <= cumulative:
            return token
    return items[-1][0]


class CharacterModel:
    name = "character model"

    def distribution(self, prefix: str) -> dict[str, float]:
        raise NotImplementedError

    def generate(self, rng: random.Random, max_chars: int = 12) -> str:
        prefix = ""
        for _ in range(max_chars):
            dist = self.distribution(prefix)
            if not dist:
                break
            prefix += sample_distribution(dist, rng)
            if prefix.endswith(";"):
                break
        return prefix

    def nll_per_character(self, rows: Iterable[str], floor: float = 1e-12) -> float:
        loss = 0.0
        characters = 0
        for row in rows:
            prefix = ""
            for char in row:
                probability = self.distribution(prefix).get(char, 0.0)
                loss -= math.log(max(probability, floor))
                prefix += char
                characters += 1
        return loss / characters


class StatefulChecksumModel(CharacterModel):
    """An explicit autoregressive model whose state is sufficient for the checksum."""

    name = "stateful rule model"

    def distribution(self, prefix: str) -> dict[str, float]:
        position = len(prefix)
        if position == 0:
            return {"@": 1.0}
        if not prefix.startswith("@"):
            return {}
        if position in (1, 2):
            return {char: 1 / len(ROUTES) for char in ROUTES}
        if position in (3, 4):
            return {char: 1 / len(DIGITS) for char in DIGITS}
        if position == 5 and is_partial_valid(prefix):
            return {":": 1.0}
        if position == 6 and is_partial_valid(prefix):
            return {checksum(prefix[1], prefix[2], prefix[3], prefix[4]): 1.0}
        if (
            position == 7
            and is_partial_valid(prefix)
            and prefix[6] == checksum(prefix[1], prefix[2], prefix[3], prefix[4])
        ):
            return {";": 1.0}
        return {}


class NGramModel(CharacterModel):
    """A learned three-character model with backoff and light additive smoothing."""

    name = "3-gram model"

    def __init__(self, order: int = 3, alpha: float = 0.001):
        self.order = order
        self.alpha = alpha
        self.counts: dict[str, Counter[str]] = defaultdict(Counter)

    def fit(self, rows: Iterable[str]) -> None:
        for row in rows:
            padded = START * self.order + row
            for index in range(self.order, len(padded)):
                char = padded[index]
                history = padded[index - self.order : index]
                for width in range(self.order + 1):
                    context = history[-width:] if width else ""
                    self.counts[context][char] += 1

    def distribution(self, prefix: str) -> dict[str, float]:
        history = (START * self.order + prefix)[-self.order :]
        counter: Counter[str] | None = None
        for width in range(self.order, -1, -1):
            context = history[-width:] if width else ""
            if self.counts.get(context):
                counter = self.counts[context]
                break
        if counter is None:
            return {char: 1 / len(ALPHABET) for char in ALPHABET}
        total = sum(counter.values()) + self.alpha * len(ALPHABET)
        return {
            char: (counter.get(char, 0) + self.alpha) / total
            for char in ALPHABET
        }


class TrieMemoriser(CharacterModel):
    """A prefix tree that samples only rows seen during training."""

    name = "trie memoriser"

    def __init__(self):
        self.next_counts: dict[str, Counter[str]] = defaultdict(Counter)

    def fit(self, rows: Iterable[str]) -> None:
        for row in rows:
            prefix = ""
            for char in row:
                self.next_counts[prefix][char] += 1
                prefix += char

    def distribution(self, prefix: str) -> dict[str, float]:
        counter = self.next_counts.get(prefix)
        if not counter:
            return {char: 1 / len(ALPHABET) for char in ALPHABET}
        total = sum(counter.values())
        return {char: count / total for char, count in counter.items()}


@dataclass(frozen=True)
class RolloutMetrics:
    model: str
    heldout_nll_nats_per_char: float
    format_valid_rate: float
    checksum_valid_rate: float
    valid_and_novel_rate: float
    unseen_prefix_rate: float
    route_intervention_accuracy: float
    unique_rate: float


def all_prefixes(rows: Iterable[str]) -> set[str]:
    result = {""}
    for row in rows:
        result.update(row[:index] for index in range(1, len(row) + 1))
    return result


def route_intervention_accuracy(model: CharacterModel, rows: Iterable[str]) -> float:
    """Edit the first route character, outside the n-gram's checksum context.

    The score asks whether the most probable checksum becomes the correct checksum
    for the edited route. This is a causal-use test, not another likelihood score.
    """
    hits = 0
    total = 0
    for row in rows:
        replacement = ROUTES[(ROUTES.index(row[1]) + 1) % len(ROUTES)]
        edited_prefix = row[:1] + replacement + row[2:6]
        expected = checksum(
            edited_prefix[1], edited_prefix[2], edited_prefix[3], edited_prefix[4]
        )
        dist = model.distribution(edited_prefix)
        if dist:
            predicted = max(sorted(dist), key=lambda char: dist[char])
            hits += int(predicted == expected)
        total += 1
    return hits / total


def evaluate(
    model: CharacterModel,
    train: list[str],
    test: list[str],
    samples: int = 5_000,
    seed: int = SEED,
) -> RolloutMetrics:
    rng = random.Random(seed)
    generated = [model.generate(rng) for _ in range(samples)]
    train_set = set(train)
    train_prefixes = all_prefixes(train)

    format_valid = [is_format_valid(row) for row in generated]
    checksum_valid = [is_checksum_valid(row) for row in generated]
    valid_and_novel = [
        is_checksum_valid(row) and row not in train_set for row in generated
    ]
    unseen_prefix = [
        any(row[:index] not in train_prefixes for index in range(1, len(row) + 1))
        for row in generated
    ]

    return RolloutMetrics(
        model=model.name,
        heldout_nll_nats_per_char=model.nll_per_character(test),
        format_valid_rate=sum(format_valid) / samples,
        checksum_valid_rate=sum(checksum_valid) / samples,
        valid_and_novel_rate=sum(valid_and_novel) / samples,
        unseen_prefix_rate=sum(unseen_prefix) / samples,
        route_intervention_accuracy=route_intervention_accuracy(model, test),
        unique_rate=len(set(generated)) / samples,
    )


def main() -> None:
    rows = corpus()
    rng = random.Random(SEED)
    rng.shuffle(rows)
    split = int(0.7 * len(rows))
    train, test = rows[:split], rows[split:]

    ngram = NGramModel()
    ngram.fit(train)
    trie = TrieMemoriser()
    trie.fit(train)
    stateful = StatefulChecksumModel()

    results = [evaluate(model, train, test) for model in (stateful, ngram, trie)]
    by_name = {result.model: result for result in results}

    # Positive case: local conditionals plus sufficient state preserve the global rule.
    assert by_name["stateful rule model"].checksum_valid_rate == 1.0
    assert by_name["stateful rule model"].route_intervention_accuracy == 1.0
    assert by_name["stateful rule model"].valid_and_novel_rate > 0.25

    # Negative case: local surface regularity does not recover a hidden long-range rule.
    assert by_name["3-gram model"].format_valid_rate > 0.98
    assert by_name["3-gram model"].checksum_valid_rate < 0.20
    assert by_name["3-gram model"].route_intervention_accuracy < 0.25

    # Negative control: perfect validity can be purchased by replaying the training set.
    assert by_name["trie memoriser"].checksum_valid_rate == 1.0
    assert by_name["trie memoriser"].valid_and_novel_rate == 0.0

    payload = {
        "assumptions": {
            "seed": SEED,
            "synthetic_language_size": len(rows),
            "train_rows": len(train),
            "test_rows": len(test),
            "rollouts_per_model": 5_000,
            "ngram_context_characters": 3,
        },
        "results": [asdict(result) for result in results],
        "example_training_rows": train[:3],
        "example_heldout_rows": test[:3],
    }
    print(json.dumps(payload, indent=2))


if __name__ == "__main__":
    main()

{
  "assumptions": {
    "seed": 49,
    "synthetic_language_size": 6400,
    "train_rows": 4480,
    "test_rows": 1920,
    "rollouts_per_model": 5000,
    "ngram_context_characters": 3
  },
  "results": [
    {
      "model": "stateful rule model",
      "heldout_nll_nats_per_char": 1.0955066586684592,
      "format_valid_rate": 1.0,
      "checksum_valid_rate": 1.0,
      "valid_and_novel_rate": 0.2996,
      "unseen_prefix_rate": 0.2996,
      "route_intervention_accuracy": 1.0,
      "unique_rate": 0.699
    },
    {
      "model": "3-gram model",
      "heldout_nll_nats_per_char": 2.224226184220807,
      "format_valid_rate": 0.9938,
      "checksum_valid_rate": 0.112,
      "valid_and_novel_rate": 0.0,
      "unseen_prefix_rate": 0.888,
      "route_intervention_accuracy": 0.1421875,
      "unique_rate": 0.9374
    },
    {
      "model": "trie memoriser",
      "heldout_nll_nats_per_char": 5.418433969227896,
      "format_valid_rate": 1.0,
      "checksum_valid_rate": 1.0,
      "valid_and_novel_rate": 0.0,
      "unseen_prefix_rate": 0.0,
      "route_intervention_accuracy": 0.7380208333333333,
      "unique_rate": 0.606
    }
  ],
  "example_training_rows": [
    "@FE05:0;",
    "@FA64:1;",
    "@CC79:4;"
  ],
  "example_heldout_rows": [
    "@CD91:5;",
    "@AA74:3;",
    "@AH77:1;"
  ]
}

Compact glossary

Autoregressive factorisation

A decomposition of a joint sequence probability into next-step conditionals ordered by a chosen sequence direction.

Teacher forcing

Training each conditional prediction on the observed earlier tokens rather than on tokens previously generated by the model.

Exposure error

Loss caused when free-running prefixes differ from the prefix distribution on which useful conditional behaviour was learned.

Decoder

The policy that converts conditional probabilities into tokens, including greedy, beam, temperature, truncation and constraint rules.

Self-recovery

Return from a perturbed prefix towards a named property such as fluency, grammar, source fidelity or plan state.

Sequence utility

The task value of a complete output, which may not decompose into next-token likelihood terms.

The next step is a contract, not a guarantee

Autoregression deserves neither mystique nor dismissal. Its core is exact and modest: the probability of a complete sequence can be factorised into conditional probabilities, and those conditionals can be learned from overlapping prefixes. That mechanism explains how a model trained on local prediction can generate globally structured text, audio or other ordered data.

The mechanism also names its own limits. A next-step predictor can preserve only dependencies represented in its conditioning state. During rollout, its choices create the prefixes on which later choices depend. The decoder changes which paths are visited. The product we value may live at sequence, task or operational level rather than inside token loss.

The changed decision is to release an autoregressive generator only after evidence crosses all three surfaces. First establish conditional fit on clean prefixes. Then test free-running and perturbed prefixes for recovery of named invariants. Finally evaluate complete outputs under the exact deployed decoder against deterministic and replay controls. When the bridge fails, change the state, objective, decoder, verifier or factorisation rather than asking perplexity to answer a question it was never designed to answer.

The power of the next step is real because local conditionals compose. The responsibility of engineering begins where composition is mistaken for assurance.