Two freedoms hidden inside one debate

The common baseline bundles several decisions. A decoder-only Transformer uses attention as its principal token mixer, applies a causal mask, factorises a sequence from left to right and usually caches prior keys and values during decoding. Each choice can be changed without changing all the others. A Transformer can be bidirectional. A recurrent backbone can remain autoregressive. A diffusion process can use a Transformer. A block-diffusion model can recover caching across blocks.

The cleanest map therefore uses two axes. The first asks how a forward pass carries information through a sequence: dense token-to-token access, local attention, sparse attention, recurrent state, state-space dynamics or a mixture. The second asks which output positions may condition one another before they are final: strict prefix order, blockwise order, arbitrary masked refinement or another partial order.

The factorisation creates the commitment

For a token sequence x_1:T conditioned on context c, an autoregressive model writes the joint probability as a product of next-token conditionals:

This factorisation does not say that the model lacks a global representation. During training it can learn broad regularities, and during inference the hidden state can encode a plan. It says something narrower: the sampler must choose x_t before it can condition on the realised value of x_t+1. Later tokens cannot directly repair earlier ones inside the same sampling trajectory.

The Transformer made training highly parallel because all positions of a known training sequence can be processed together under a causal mask. Decoding remains sequential because generated tokens do not yet exist. This distinction between parallel training and serial sampling is the first source of confusion in claims that one architecture “breaks” another.

Statistical dependence and runtime dependence are different ledgers

A factorisation describes which realised variables a probability can condition on. A runtime graph describes which computations must wait for which others on a particular implementation. The two are related, but they are not identical. Teacher-forced autoregressive training can evaluate every next-token loss in parallel because the complete target sequence is available. Cached decoding evaluates only the new position, yet must wait for the sampled prefix. Masked diffusion can shorten that chain by updating several positions per round, while each round may perform substantially more work.

This yields three quantities that should never be collapsed. Dependency depth counts the longest sequence of model decisions that cannot overlap. Total model work counts token positions, layers or FLOPs evaluated. Realised service time includes hardware utilisation, memory movement, batching and scheduler overhead. A method can improve the first, worsen the second and still win or lose the third depending on the serving regime.

The distinction also prevents an architectural category error. Replacing attention with recurrent state changes the cost of each forward pass and its reusable history. It does not, by itself, reduce the number of output commitments. Replacing left-to-right sampling with diffusion changes the commitment graph. It does not, by itself, make each pass cheap. The two interventions can be combined, but their evidence must be recorded separately.

Thought experiment: the ledger and the poem

Give two systems the same backbone and the same parameter count. The first task is to continue a live market transcript where every emitted word must appear immediately. The second is to complete a six-line poem whose final words must form an acrostic and whose third line determines the rhyme of the first. Only the output dependency changes.

Strict left-to-right commitment matches the transcript. Revision adds no useful freedom because a listener has already consumed the prefix. The poem has mutual constraints across positions. A model that can hold uncertain words open, fill the strongest positions first and revise weak positions can use evidence unavailable to a committed prefix. Generation order is useful only when the task contains dependencies that the order can expose.

Now hold the output order fixed and replace a short prompt with a million-token event stream. The poem advantage says nothing about that workload. Efficient history transport becomes the binding issue. That is where hybrid backbones enter.

flowchart TD
  A["The sentence that discovers<br/>its ending too late"] --> B["Two freedoms hidden inside one<br/>debate"]
  B --> C["Hybrid backbones change memory<br/>transport"]
  C --> D["Diffusion changes commitment order"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#D95545,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Hybrid backbones change memory transport

Attention offers content-addressed access. A query at the current position can compare itself with many stored token representations and place high weight on a distant item. The price is explicit history. During autoregressive decoding, the key-value cache grows with context length for every attention layer. Prefill also creates a large token-to-token interaction problem, even when efficient kernels soften the hardware cost.

A recurrent or state-space layer follows a different contract. It updates a state as each token arrives and uses that state to produce the next representation. A simplified selective update can be written as:

Mamba made the state update input-selective and paired it with a hardware-aware scan, showing that a recurrent sequence model could become a serious language backbone. Paper 86 in this canon treats that mechanism directly. The narrower question here is why a designer would keep some attention after adopting it.

Griffin mixes gated linear recurrences with local attention. Jamba interleaves Transformer and Mamba layers, also using mixture-of-experts capacity in selected layers. Samba combines selective state-space layers with sliding-window attention and reports long-context extrapolation and throughput gains under its evaluated settings. These are not three proofs of one universal ratio. They are evidence for the compositional idea that different mixers can carry different information burdens.

Thought experiment: one lost account number

Imagine a recurrent state with enough capacity to preserve topic, sentiment, active entities and the broad trajectory of a conversation. One account number appears once, 80,000 tokens earlier, among many similar numbers. A compact state may retain that an account was discussed while losing the exact digits. Dense attention can, in principle, address the original token representation directly. Local attention cannot reach it unless another mechanism carries or retrieves it.

Now change the distant item from an arbitrary number to the document’s dominant topic. Compression becomes appropriate. The state can integrate repeated evidence, and storing every token verbatim may be wasteful. A hybrid earns its complexity when the workload contains both compressible continuity and sparse demands for exact recall. If every relevant feature can be compressed, pure recurrence may suffice. If exact random access dominates, reducing attention layers can create a false economy.

Layer ratios are systems decisions

The attractive algebra of fewer attention layers does not guarantee an attractive serving system. Recurrent updates contain scans and elementwise operations that can map differently to accelerators than large matrix multiplications. Attention and state-space layers may prefer different tensor-parallel, pipeline or memory layouts. Kernel maturity, batching, prefill length and decode length can reverse a theoretical advantage.

A fair hybrid experiment therefore measures at least four things separately: quality under matched training compute, prefill latency, decode latency across concurrency levels, and persistent state memory. It also includes exact-recall probes and intervention tasks rather than relying only on perplexity. A lower cache footprint that destroys a rare but consequential identifier is not an efficiency gain for that workload.

A matched hybrid ablation needs more than a leaderboard

Start with one trained baseline and vary the information path as narrowly as the implementation permits. Compare all-attention, all-recurrent and several interleavings under the same tokenizer, data order, optimisation budget and active parameter count. Report the extra training needed for a hybrid to recover baseline quality. A model that needs materially more tokens to compensate for lossy state may still be attractive at inference, but the trade must be visible.

Then use probes designed around the claimed division of labour. A long copy task tests exact distant retrieval. A slowly changing latent-state task tests compression. A distractor stream tests selective forgetting. An intervention that zeroes recurrent state after a named event should damage continuity while leaving recent local lookup partly intact. Removing the attention checkpoint expected to retrieve an arbitrary identifier should selectively damage that identifier. When both ablations cause undifferentiated collapse, the proposed role separation has not been demonstrated.

Finally, run the serving comparison at several prompt and output lengths. Hybrid gains often depend on whether the request is prefill-heavy, decode-heavy or memory-constrained. At high concurrency, a smaller per-request state may admit more active sequences. At low concurrency, an irregular recurrent kernel may underuse the accelerator. The useful unit is the workload-level Pareto frontier, not the layer-level complexity class.

Diffusion changes commitment order

Image diffusion begins with noise and repeatedly moves a continuous array towards a sample. Text consists of discrete symbols. Early language approaches therefore had to decide where the noising process should live. Diffusion-LM diffused continuous word-vector representations, which made gradient-based control natural but left a mapping back to discrete tokens. D3PM defined diffusion-like corruption directly over discrete states, including an absorbing mask state. Later work such as SEDD and masked diffusion language modelling strengthened discrete objectives and sampling.

The most intuitive masked process starts with a clean sequence x₀. A forward corruption schedule replaces more positions with a mask as time t increases. The learned reverse model sees a partially masked sequence x_t and predicts the original tokens at masked positions. A simplified training loss is:

At generation time, the model begins with masks in the output region. Each pass predicts many positions. A sampler decides which predictions to keep, which to leave masked and sometimes which committed tokens to mask again. The next pass uses the increasingly complete sequence. The critical path can depend on a chosen number of refinement rounds rather than directly on output length.

The sampler is part of the model’s behaviour

Two checkpoints with identical weights can behave differently under different unmasking policies. One sampler may reveal the most confident positions first. Another may follow a fixed schedule. A third may reserve punctuation, delimiters or low-entropy spans while repeatedly remasking uncertain content. Temperature, block size, confidence threshold and stopping criteria all change the speed-quality frontier.

This matters because simultaneous token predictions are conditionally coupled through an incomplete sequence. Several positions may each look plausible while being mutually inconsistent. Committing all of them maximises apparent parallelism and can freeze a bad joint choice. Committing only one or two preserves quality and approaches autoregressive seriality. Remasking allows repair, but adds work and can produce oscillation when two alternatives repeatedly displace one another.

A useful diffusion evaluation stores the denoising trajectory: first proposal, commitment round, remasking events, final token and confidence for each position. It reveals whether speed came from genuine multi-token progress or weak early guesses. A recent LLaDA and Dream study under text corruption found that reported certainty could remain high as answer-ranking usefulness degraded, while hidden states retained error signals. The setting is narrow, but unmasking confidence is not calibrated correctness. Failure injection should force early commitment, disable remasking and perturb order.

Scale establishes possibility, not dominance

LLaDA trained an 8-billion-parameter masked diffusion model from scratch under pretraining and supervised fine-tuning, showing that in-context learning and instruction following do not logically require autoregressive factorisation. Dream 7B used autoregressive model initialisation followed by diffusion training and emphasised arbitrary-order generation and infilling. iLLaDA later scaled fully bidirectional diffusion pretraining further and reported stronger base-model results, while its instruction-tuned model still trailed a strong autoregressive comparator on several mathematics and code evaluations.

That evidence changes a conceptual claim. Diffusion is no longer confined to tiny demonstrations or continuous latent text. It does not establish that every serving workload should migrate. Model scale, data, post-training, evaluation protocol and inference implementation remain entangled. A diffusion model trained for trillions of tokens and an autoregressive model with a different data mixture are not a controlled test of factorisation alone.

Block diffusion makes the continuum explicit

Full diffusion has practical friction. It often assumes an output region or maximum length in advance. Every denoising pass may revisit the whole active sequence. Standard causal key-value caching does not transfer directly because a token’s representation can change when tokens on either side change. Streaming the first token can also be awkward when the sequence is meant to remain revisable.

Block Diffusion factorises a sequence autoregressively across blocks while modelling each block with discrete denoising. If a sequence is divided into blocks b = 1…B, the distribution becomes:

The strongest objection: parallel updates can preserve a serial problem

A model may predict many token distributions in one pass, yet only a few predictions may be safe to commit. Repeated full-sequence evaluations can consume more total work than cached autoregressive decoding. The number of denoising steps is therefore an incomplete efficiency metric. Batch size, prompt length, output length, precision, kernel implementation, remasking policy and quality target all affect wall-clock behaviour.

A critical efficiency study found that open diffusion models in its tested configurations often lagged autoregressive counterparts as batching changed. A later single-H200 serving characterisation of LLaDA-8B-Instruct with a D2F adapter attributed only 24% of single-request wall time to GPU computation and found large gains from synchronised batching relative to per-request dispatch. In that workload, host dispatch, request-step variance and batching policy dominated the algorithmic story. Every method still needs end-to-end measurement.

The deeper limitation concerns correctness. A theoretical analysis of masked diffusion distinguishes token-level likelihood from sequence-level error. Under its assumptions, near-optimal perplexity can require a number of sampling steps independent of sequence length, while obtaining a fully correct sequence can require steps that grow linearly with length. This matters for code, arithmetic and regulated instructions, where one wrong token can invalidate the whole output.

Compare dependency ledgers, not labels

A fair architecture decision starts with the dependency that consumes time, memory or correctness. “Use diffusion because it is parallel” is too coarse. “Use a hybrid because attention is quadratic” is equally incomplete. The configured system includes the prompt path, backbone, sampler, cache, scheduler, quantisation, batch policy, stopping rule and any deterministic postprocessor.

Composition is possible, but the interfaces matter

Hybrid does not name one model family. A diffusion model can use recurrent or state-space layers for compressible continuity, attention for exact lookup, and a masked sampler for revisable output. FLARE converts hybrid-attention autoregressive checkpoints into one checkpoint that supports autoregressive verified decoding and diffusion-style denoising. A block-diffusion Mamba hybrid retains a cacheable prefix while refining an active block. The backbone and commitment schedule can be combined because they answer different questions.

The combination is not automatic. A causal recurrent state summarises a fixed prefix. During full masked refinement, tokens on both sides may change, so state derived from them becomes stale. The block-Mamba design confines reverse-direction recurrence to the active block and keeps the completed prefix forward-only. FLARE, meanwhile, reports that transfer data quality and block-objective training efficiency constrain capability preservation. Recomputing state, running directional sweeps, retaining selected attention or freezing completed blocks each restores a dependency the label appeared to remove.

Three interfaces deserve explicit tests. The receptive-field interface asks which positions may influence each update. The state-invalidation interface states which cached activations become unusable after a token changes. The commitment interface defines when a tentative token joins an immutable prefix, visible output or executable artefact. Without these contracts, “hybrid diffusion” can hide repeated recomputation behind a low denoising-step count.

A clean experiment changes one axis at a time. Compare dense and hybrid autoregressive backbones to isolate memory transport. Compare dense autoregressive and dense masked-diffusion models under matched data and compute to isolate commitment order. Then compare dense and hybrid backbones under the same diffusion objective and sampler. The factorial design reveals whether an improvement belongs to the backbone, factorisation, scheduler or their interaction. The combination earns its complexity only when it moves the quality, latency or memory frontier beyond both simpler alternatives.

Production-shaped worked example: generating a governed credit condition

The output contains a borrower name, facility amount, currency, covenant threshold, testing date, evidence requirement and escalation clause. Several values are copied exactly from authoritative fields. Others are phrased from policy. A final legal consistency check may reveal that an earlier phrase should change from “quarterly” to “monthly”.

A full diffusion model could keep the clause globally revisable. That sounds attractive until the simpler controls are listed. Typed slots can force exact values. A grammar can constrain the clause shape. A deterministic validator can reject inconsistent periods. A second-pass editor can revise only the affected span. The diffusion model earns a place only if its joint refinement improves quality or latency beyond this structured baseline under the same evidence and effect controls.

The backbone decision is separate. The source pack may contain years of account events. A hybrid model could reduce history-state cost, but exact covenant values should still come from typed context rather than compressed neural state. Alternative model geometry does not replace authority, provenance or deterministic validation. It changes how a proposal is formed.

Serious negative control: the live interpreter

Consider real-time speech interpretation. Each translated phrase must reach the listener with low delay. Later source words may clarify an earlier ambiguity, but the acoustic stream keeps moving and the listener has already acted on the translation. A full-sequence diffusion decoder that waits for a large output canvas and repeatedly revises it may produce an elegant final transcript while failing the actual service objective.

The negative control is a strong streaming autoregressive model with prefix caching and bounded correction messages. If it matches final quality while delivering much lower time to first audio, diffusion’s revision freedom is unnecessary for that workload. This control matters because offline benchmark scoring often ignores when a token became usable.

Measure the first usable token, not merely the first emitted token

Time to first token can flatter a system that emits speculative text which is later revised or rejected. For a chat interface, the first visible token may be useful immediately. For a code patch, a single token is unusable until the syntax closes and tests pass. For a payment instruction, no part should be actionable before typed fields and policy checks agree. The evaluation unit must reflect the release contract.

Record at least four clocks. The first begins when the request is admitted. The second stops when the first user-usable unit is available. The third stops when the complete candidate exists. The fourth stops after validation, repair and effect-safe release. Alongside these clocks, record total model evaluations, positions scored, bytes of reusable state and rejected trajectories. A model that wins candidate latency but loses validated latency has not accelerated the service.

This measurement also clarifies where diffusion may shine. If the usable unit is an entire constrained span, parallel refinement can shorten the path to that span. If usable units stream token by token, revision creates a release conflict. If the output is later consumed by a deterministic compiler or verifier, the model should be judged on accepted artefacts per unit cost, not raw token throughput.

stateDiagram-v2
  [*] --> Claim
  state "Figure 1. Four alternative<br/>sequence geometries" as Claim
  state "Figure 2. Backbone transport<br/>and commitment order form<br/>separate axes" as Test
  state "The factorisation creates the<br/>commitment" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Executable sequence-geometry lab

The companion lab isolates the central causal claim with synthetic binary sequences. In the positive case, the second half mirrors the first. Each mirrored pair provides two noisy observations of one latent bit. A left-to-right decoder commits the first observation before it sees the second. A four-round masked-refinement scheduler combines both observations and commits high-confidence pairs first.

The negative control removes the mirror relation. Positions become independent. Confidence-ordered refinement changes the schedule but has no information to combine, so token and exact-sequence accuracy must match the left-to-right decoder. This prevents the preferred result from being attributed to extra passes alone.

The scheduler ledger in the script also compares illustrative reusable-history units for dense AR, hybrid AR, full masked refinement and block diffusion. Those units are structural accounting, not bytes, FLOPs or wall-clock measurements.

"""Synthetic sequence-geometry lab for 'Hybrid and Diffusion Language Models'.

Python 3.10+. Standard library only.
This is a teaching instrument, not a benchmark of trained language models.
"""
from __future__ import annotations

from dataclasses import asdict, dataclass
import json
import math
import random
from statistics import fmean


@dataclass(frozen=True)
class Result:
    workload: str
    decoder: str
    token_accuracy: float
    exact_sequence_accuracy: float
    serial_model_calls: int
    token_positions_scored: int


def bit_from_logit(value: float) -> int:
    return int(value >= 0.0)


def make_observations(target, sigma, rng):
    return [(1.0 if bit else -1.0) + rng.gauss(0.0, sigma) for bit in target]


def autoregressive_decode(logits, mirrored):
    if not mirrored:
        return [bit_from_logit(value) for value in logits]
    half = len(logits) // 2
    prefix = [bit_from_logit(value) for value in logits[:half]]
    return prefix + list(reversed(prefix))


def masked_refinement_decode(logits, mirrored, rounds):
    if rounds < 1:
        raise ValueError("rounds must be positive")
    output = [None] * len(logits)

    if not mirrored:
        items = sorted(range(len(logits)), key=lambda i: abs(logits[i]), reverse=True)
        batch = math.ceil(len(items) / rounds)
        for start in range(0, len(items), batch):
            for i in items[start:start + batch]:
                output[i] = bit_from_logit(logits[i])
    else:
        half = len(logits) // 2
        pairs = []
        for i in range(half):
            j = len(logits) - 1 - i
            pair_logit = logits[i] + logits[j]
            pairs.append((abs(pair_logit), i, j, bit_from_logit(pair_logit)))
        pairs.sort(reverse=True)
        batch = math.ceil(len(pairs) / rounds)
        for start in range(0, len(pairs), batch):
            for _, i, j, bit in pairs[start:start + batch]:
                output[i] = bit
                output[j] = bit

    assert all(value is not None for value in output)
    return [int(value) for value in output]


def evaluate(*, name, mirrored, length, sigma, rounds, trials, seed):
    rng = random.Random(seed)
    ar_token_scores, ref_token_scores, ar_exact, ref_exact = [], [], [], []

    for _ in range(trials):
        if mirrored:
            payload = [rng.randrange(2) for _ in range(length // 2)]
            target = payload + list(reversed(payload))
        else:
            target = [rng.randrange(2) for _ in range(length)]

        logits = make_observations(target, sigma, rng)
        ar = autoregressive_decode(logits, mirrored)
        ref = masked_refinement_decode(logits, mirrored, rounds)
        ar_token_scores.append(sum(a == b for a, b in zip(ar, target)) / length)
        ref_token_scores.append(sum(a == b for a, b in zip(ref, target)) / length)
        ar_exact.append(int(ar == target))
        ref_exact.append(int(ref == target))

    return [
        Result(name, "left-to-right commit", fmean(ar_token_scores),
               fmean(ar_exact), length, length),
        Result(name, f"{rounds}-round masked refinement", fmean(ref_token_scores),
               fmean(ref_exact), rounds, length * rounds),
    ]


def scheduler_ledger(length=128, layers=32, attention_fraction=0.25,
                     ssm_state_equivalent_tokens=4, full_diffusion_rounds=8,
                     block_size=16, block_rounds=4):
    attention_layers = round(layers * attention_fraction)
    recurrent_layers = layers - attention_layers
    blocks = math.ceil(length / block_size)
    return [
        {"geometry": "dense autoregressive Transformer",
         "serial_model_calls": length,
         "token_positions_scored": length,
         "reusable_history_state_units": layers * length},
        {"geometry": "hybrid autoregressive backbone",
         "serial_model_calls": length,
         "token_positions_scored": length,
         "reusable_history_state_units": attention_layers * length +
             recurrent_layers * ssm_state_equivalent_tokens},
        {"geometry": "full masked refinement",
         "serial_model_calls": full_diffusion_rounds,
         "token_positions_scored": length * full_diffusion_rounds,
         "reusable_history_state_units": 0},
        {"geometry": "block diffusion",
         "serial_model_calls": blocks * block_rounds,
         "token_positions_scored": blocks * block_size * block_rounds,
         "reusable_history_state_units": layers * length},
    ]


def main():
    rounds = 4
    results = []
    results.extend(evaluate(name="mirrored sequence (positive case)", mirrored=True,
                            length=16, sigma=1.15, rounds=rounds,
                            trials=5_000, seed=87))
    results.extend(evaluate(name="independent positions (negative control)",
                            mirrored=False, length=16, sigma=1.15,
                            rounds=rounds, trials=5_000, seed=88))

    mirror_ar, mirror_ref, control_ar, control_ref = results
    assert mirror_ref.exact_sequence_accuracy > mirror_ar.exact_sequence_accuracy + 0.10
    assert abs(control_ref.token_accuracy - control_ar.token_accuracy) < 1e-12
    assert mirror_ref.serial_model_calls < mirror_ar.serial_model_calls
    assert mirror_ref.token_positions_scored > mirror_ar.token_positions_scored

    print(json.dumps({
        "warning": "Synthetic causal illustration; not a trained-model benchmark.",
        "results": [asdict(item) for item in results],
        "scheduler_ledger": scheduler_ledger(),
    }, indent=2))


if __name__ == "__main__":
    main()

The decision that changes: break the expensive dependency, then stop

Hybrid and diffusion language models are best understood as interventions on different dependency graphs. A hybrid backbone asks how much of the past must remain explicitly addressable at every layer. Diffusion asks how many output positions must wait for a committed prefix. Block diffusion, local attention and recurrent-attention mixtures show that the design space is continuous rather than tribal.

The practical decision is no longer “Which model family wins?” It is “Which dependency is binding, and what is the smallest intervention that removes it without weakening the required evidence?” Long streaming histories may justify recurrent transport with attention checkpoints. Globally constrained infilling may justify masked refinement. Flexible-length chat with immediate streaming may continue to favour cached autoregression. Structured outputs may be solved more reliably by typed placeholders and deterministic validation than by changing the generative paradigm.

A defensible experiment therefore keeps the strongest simple baseline, matches training and serving conditions, reports critical path and total work, and tests the failure mode the new geometry claims to repair. If the advantage disappears when late cross-position evidence is removed, the mechanism has been isolated. If it survives only in unmatched demos, the label has outrun the evidence.