The explanation on the surface

Step-by-step prompting earned attention for a sound practical reason. On arithmetic, symbolic and commonsense tasks, asking a language model to emit intermediate steps can improve its final performance. Scratchpads can also let a transformer carry partial results across a long computation.[1][2] These results establish that an external trace can be useful. They do not establish that every sentence in a successful trace reports the internal route that produced the answer.

This difference echoes a mature distinction in interpretability. A plausible explanation fits human expectations. A faithful explanation accurately attributes the system’s behaviour to the causes that produced it.[3] Plausibility is judged from the story. Faithfulness is judged against the system. A polished account can therefore be persuasive and unfaithful, while a terse, ugly trace can be causally informative.

One visible trace, three possible jobs

“Chain of thought” often bundles three distinct roles. The first is an operative trace: the semantic content of an intermediate step changes later computation. If the model writes that 17 × 6 is 102, the representation of 102 becomes an input to the answer. The second is computational scaffolding: producing extra tokens, pauses or recurrent states gives the network more sequential computation, but the readable meaning of those tokens is not what carries the result. The third is a descriptive trace: the model has already formed, or is forming in parallel, a conclusion and generates language that explains it.

These categories are not moral judgements. A descriptive trace can be useful for communication. Scaffolding can improve accuracy. An operative trace can still omit relevant causes. The distinction matters because the evidence each trace supports is different.

A trace is a measurement channel, not a window

It helps to treat the rationale as an instrument reading. A thermometer is useful because changes in temperature cause systematic changes in the reading. The scale can still be miscalibrated, slow or insensitive outside a range. In the same way, a reasoning trace becomes informative when controlled changes in the computation produce predictable changes in the words. Its value depends on a measurement bridge, not on the intimacy of its language.

Four questions follow. Observability asks whether the trace contains information about the route. Causal participation asks whether the trace’s content helps produce the answer. Completeness asks whether important causes are missing. Usefulness asks whether a human or monitor can detect the property that matters. These questions can separate. A trace may be causally operative but too compressed for a reviewer. It may be highly monitorable because it blurts out a plan while remaining incomplete about the computation. It may be a faithful summary of a hidden calculation without participating in that calculation.

This framing avoids two symmetrical errors. The first says that any natural-language step is transparent because humans can read it. The second says that no trace can be evidence because neural computation is distributed. Both ignore intervention. Distributed systems can still expose reliable intermediate variables, and readable variables can still be decorative. Structured question decomposition has improved measured faithfulness in some tasks, but that gain does not prove every decomposed step is operative.[7] The status of a trace is an empirical property of a configured model, task, prompt and decoding regime. Change any of those, and the causal role may change.

Latent-reasoning systems make this separation explicit. In Chain of Continuous Thought, a hidden state is fed back as the next input embedding without first being decoded into a word. The reported gains are task and architecture specific, but the design demonstrates a conceptual possibility: useful intermediate computation need not take the form of natural-language propositions.[9] That possibility prevents a common inference from “the model produced steps” to “the steps reveal the model’s computation”.

flowchart LR
  A["The two tunnels beneath one answer"] --> B["The explanation on the surface"]
  B --> C["Causality beneath the prose"]
  C --> D["The TRACE audit"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#087A98,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Causality beneath the prose

To make the problem precise, let P be the prompt, C the visible chain of thought, H the evolving hidden state and A the final answer. A reader sees P, C and A. The model’s computation also includes H, learned parameters, sampling state and any tool results. The central question is whether answer-relevant information from P reaches A through the meaning expressed in C, rather than through a route that merely coexists with C.

The autoregressive trap

In an autoregressive model, earlier trace tokens enter the context for later tokens. Strictly speaking, changing them can change the answer. That fact is too weak to establish explanation. A misspelling, a newline or a style token can alter the distribution of later outputs without representing a reason. Conversely, paraphrasing a step may leave its meaning intact while changing thousands of low-level activations.

We therefore need three levels of dependence. Token dependence asks whether these exact symbols matter. Semantic dependence asks whether changing the proposition changes the answer in the direction the proposition predicts, while paraphrases remain stable. Mechanistic dependence asks whether internal states corresponding to the proposition causally influence the answer. Faithfulness is not mere sensitivity; it is sensitivity with the right structure.

Sufficiency, necessity and completeness pull apart

Three attractive tests answer different questions. A trace is sufficient when the answer can be recovered from the trace and minimal remaining context. It is necessary when removing or counterfactually changing it alters the answer. It is complete when no answer-relevant route bypasses it. Rationale benchmarks have long used sufficiency and comprehensiveness-style measures, but their interpretation depends on how masked inputs are constructed and whether redundant features remain.[4]

Consider a system with two calculators. One writes the product into the visible trace; the other computes the same product directly from the prompt. The trace alone is sufficient. It is not necessary because the direct calculator can replace it. It is not complete because the second route bypasses it. Now consider a different system that refuses to answer unless it sees exactly twenty intermediate tokens. The token sequence is necessary, but a matched string of dots works as well as the correct arithmetic. Necessity here belongs to the scaffold, not to the semantics.

A third system copies the visible intermediate number into the answer. The step is necessary and semantically sufficient, yet the model may have selected that number using an unreported biasing cue. The trace faithfully carries the last transformation while remaining incomplete about why that intermediate was chosen. This is why a single “faithfulness score” invites overclaiming. The evidence should remain typed: semantic necessity, semantic sufficiency, bypass resistance, timing and internal concordance.

The smallest useful causal claim is local: “Under this prompt family, changing step three from 102 to 96 changes the answer distribution in the predicted direction, while paraphrases and matched corruptions do not.” That statement is narrower than “the model explains itself”, but it is testable, reproducible and strong enough to change how the trace is used.

Early answering tests operationalise this idea by truncating a chain and asking for the answer. Across tasks, models can range from heavily dependent on later steps to largely settled before them.[6] More recent activation-probe work reports that answer information can sometimes be decoded before the visible trace declares a commitment.[16] Decodability is not causality, because a hidden state may contain an unused correlate. Steering, patching or ablation is needed to test whether the early representation actually changes the outcome.

This boundary is crucial. A rationale can be locally faithful about one cause and silent about another. It can track the transformation from premises to answer while failing to report the cue that selected those premises. It can also name every relevant fact yet misstate the mechanism. The right target is therefore not a binary label attached to prose. It is a profile of which causal claims survived which interventions.

The TRACE audit

The proposed contribution is a five-part audit for articulated reasoning. It does not score “how thoughtful” a model sounds. It tests a narrower claim: that the visible trace carries answer-relevant computation. The acronym names the required evidence: Targeted intervention, Rival-route control, Answer timing, Counterfactual simulatability and Evidence triangulation.

T · Targeted intervention

Choose one trace step with an explicit downstream consequence. Create a paraphrase, a valid counterfactual and a matched nonsense edit. Regenerate from the intervention point. The positive pattern is selective: paraphrases preserve the answer, counterfactuals shift it predictably, and nonsense causes uncertainty or repair rather than the same confident conclusion. A negative result means the step was not necessary under this test. It does not show that no other trace step mattered.

R · Rival-route control

List alternative generators before running the test. These include a direct prompt-to-answer shortcut, a spurious hint, answer-choice order, memorised association, tool output, a hidden-state route and generic extra compute. Change or block one rival at a time. In open-weight models, attention masks, activation patches or parameter interventions can make this stronger. In API-only systems, use paired prompts, hidden canaries, label permutations and independent tool logs. A trace is explanatory only relative to rivals that have been made to fail.

A · Answer timing

Ask for an answer after successive prefixes of the trace. Where possible, train a probe on held-out examples to estimate when answer information becomes decodable, then use steering or patching to test causal force. A conclusion available after the prompt but before the claimed decisive step is evidence of precommitment. It is not proof that later reasoning is purely decorative. The model may revise, verify or calibrate a previously favoured answer.

C · Counterfactual simulatability

A good explanation should let a reader predict how the system will behave on a nearby case. “The model failed because it treated the variable name total as a positive signal” predicts improvement when the variable is renamed. “The trace used multiplication” predicts a different answer when the multiplier changes. Counterfactual simulatability turns explanation into forecast, not admiration.[17]

E · Evidence triangulation

Combine at least three evidence types: behaviour under controlled inputs, trace interventions and an independent signal such as activations, tool receipts or parameter edits. Output-level consistency tests are useful but can remain tests of self-consistency rather than the model’s inner mechanism.[8] Mechanistic evidence is stronger when patching or ablation changes the predicted intermediate state and the final answer. Case studies have found both concordant internal features and apparent backwards construction of rationales, which is exactly why a portfolio is needed.[11]

Build an audit packet before collecting impressive examples

A serious audit begins with a claim packet, not a gallery of good-looking traces. State the candidate causal variable in ordinary language and in a machine-readable record. For example: “The comparison cost < limit in step four determines the recommendation.” Name the output that should change, the direction of change and the point from which generation will resume. Then list at least two rivals, such as a retrieved label, an answer-choice prior or a direct feature in the prompt.

Construct a family of paired cases. Each original needs a semantic counterfactual, a paraphrase, a matched corruption and a rival-route intervention. Human review should confirm that the counterfactual remains coherent and that the paraphrase preserves the claimed relation. Run enough stochastic samples to estimate distributions rather than selecting one trajectory. Record the model version, decoding settings, prompt template, tool state and intervention generator. Without those controls, a later reproduction may be testing a different system.

Analyse failures before averages. A model can pass 95 per cent of easy cases while failing every case in which the shortcut conflicts with the stated rule. Report results by task family, difficulty, trace length and answer confidence. Where probes are used, train them on held-out data and compare with control directions. Where activation patches are used, include equal-norm random patches and check that the intervention changes the targeted intermediate behaviour rather than merely damaging the network.

Finally, bind each observation to a permitted inference. A counterfactual answer shift supports local semantic dependence. Early-answer stability supports precommitment. Shortcut removal supports or weakens a bypass hypothesis. Mechanistic patching supports a scoped internal route. None alone establishes complete introspective access. The audit is complete only when a negative result has an interpretation and a failed control can stop the claim.

What a positive result permits: evidence that a specified trace step causally contributes under the tested distribution and intervention family. What a negative result means: the stated step was dispensable, overridden or tested inadequately. What the lab cannot establish: that a real transformer uses the same mixture model, that every causal factor was captured, or that natural-language concepts map cleanly onto internal features.

from __future__ import annotations

from dataclasses import dataclass
from random import Random
from typing import Literal

Route = Literal["trace", "direct", "shortcut", "noise"]

@dataclass(frozen=True)
class System:
    trace: float
    direct: float
    shortcut: float
    noise: float
    scaffold_success: float
    early_trace_answer: float

SYSTEMS: dict[str, System] = {
    "operative": System(.76, .12, .04, .08, .45, .10),
    "scaffold":  System(.10, .70, .05, .15, .90, .55),
    "posthoc":   System(.05, .20, .66, .09, .35, .94),
    "hybrid":    System(.42, .31, .19, .08, .62, .58),
}

def choose_route(rng: Random, s: System) -> Route:
    r = rng.random()
    if r < s.trace:
        return "trace"
    if r < s.trace + s.direct:
        return "direct"
    if r < s.trace + s.direct + s.shortcut:
        return "shortcut"
    return "noise"

def answer(route: Route, target: int, hint: int, trace_bit: int,
           *, filler: bool, s: System, rng: Random) -> int:
    if route == "trace":
        return rng.randrange(2) if filler else trace_bit
    if route == "direct":
        succeeds = (not filler) or (rng.random() < s.scaffold_success)
        return target if succeeds else rng.randrange(2)
    if route == "shortcut":
        return hint
    return rng.randrange(2)

def audit(name: str, trials: int = 5_000, seed: int = 52) -> dict[str, float]:
    s, rng = SYSTEMS[name], Random(seed)
    changed_by_edit = changed_by_paraphrase = 0
    shortcut_stable = early_same = 0
    baseline_correct = filler_correct = 0

    for _ in range(trials):
        x1, x2 = rng.randrange(2), rng.randrange(2)
        target = x1 ^ x2
        hint = 1 - target
        route = choose_route(rng, s)
        trace_bit = target if route != "shortcut" else hint

        base = answer(route, target, hint, trace_bit, filler=False, s=s, rng=rng)
        edited = answer(route, target, hint, 1 - trace_bit, filler=False, s=s, rng=rng)
        paraphrase = answer(route, target, hint, trace_bit, filler=False, s=s, rng=rng)
        without_shortcut = answer(
            "direct" if route == "shortcut" else route,
            target, hint, trace_bit, filler=False, s=s, rng=rng
        )
        filler_answer = answer(route, target, hint, trace_bit, filler=True, s=s, rng=rng)

        changed_by_edit += edited != base
        changed_by_paraphrase += paraphrase != base
        shortcut_stable += without_shortcut == base
        early_prob = s.early_trace_answer if route == "trace" else .96
        early_same += rng.random() < early_prob
        baseline_correct += base == target
        filler_correct += filler_answer == target

    mediation = max(0.0, (changed_by_edit - changed_by_paraphrase) / trials)
    return {
        "semantic_mediation": mediation,
        "paraphrase_stability": 1 - changed_by_paraphrase / trials,
        "shortcut_resistance": shortcut_stable / trials,
        "late_commitment": 1 - early_same / trials,
        "semantic_over_filler": max(0.0, (baseline_correct - filler_correct) / trials),
    }

if __name__ == "__main__":
    for name in SYSTEMS:
        scores = audit(name)
        print(name, {k: round(v, 3) for k, v in scores.items()})

stateDiagram-v2
  [*] --> Claim
  state "One visible trace, three<br/>possible jobs" as Claim
  state "A trace is a measurement<br/>channel, not a window" as Test
  state "The silent calculator" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Designing with traces without trusting them blindly

Reasoning traces can be valuable even when they are not complete explanations. Monitoring them can reveal intentions, shortcuts and reward hacking that are difficult to infer from final actions alone. OpenAI reports that chain-of-thought access improved monitoring of reward-hacking behaviour in coding environments, while also warning that direct optimisation of the trace can teach a model to hide the very behaviour the monitor seeks.[12] Later monitorability work formalises a broader property: how well a monitor can predict behaviour-relevant properties from available signals.[13]

Monitorability and faithfulness overlap but are not identical. A trace may contain an incriminating plan that helps a monitor, even if many other sentences are post-hoc. A faithful trace may be too technical, compressed or oblique for a weak monitor. Monitorability asks whether the signal is useful for oversight; faithfulness asks whether it tracks the causes of this behaviour.

Why interface pressure changes the evidence channel

A chain is generated inside an incentive system. If every trace is judged for politeness, policy conformity or resemblance to a preferred template, the model receives pressure on the observation channel itself. The easiest way to improve the score may be to remove suspicious language while retaining the underlying strategy. This is not unique to language models. Any measured process can adapt to the measurement when the measure enters the objective.

The design response is not to abandon monitoring. It is to separate optimisation targets. Train the action policy against task and safety outcomes. Preserve an unrestricted internal trace where policy permits, then monitor it with a distinct system. Generate the user-facing explanation through a separate, evidence-grounded summariser. Evaluate whether the summariser omits material facts, but do not assume its wording is the model’s causal transcript. Periodically run hidden interventions that the model cannot infer from surface form, and compare trace signals with actions and receipts.

Current evidence does not show that every form of trace supervision destroys monitorability. It shows a credible failure path and model-specific examples. The operational implication is proportionate: when a trace becomes a target, revalidate it as a sensor. A previously informative channel can lose its value after post-training, prompting changes, safety tuning or deployment feedback.

A practical trace architecture

For consequential systems, maintain four separate artefacts. The internal reasoning trace is a monitorable but untrusted signal. The user-facing rationale is a concise explanation generated from approved evidence and may be sanitised. The decision receipt records input versions, rules, model proposal, verifier results and authority. The effect receipt records the observed outcome and readback. Combining them into one paragraph destroys distinctions the system later needs for audit and recovery.

Do not require the user-facing rationale to reproduce private or unrestricted chain of thought. The useful target is a decision-sufficient explanation: decisive evidence, applicable rule, uncertainty, alternatives and next action. That explanation should be checked against receipts, not trusted because it resembles the model’s hidden process.

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: The two tunnels beneath one answer
  M->>E: One visible trace, three possible jobs
  E-->>M: compare rival explanations
  E->>D: Treat every rationale as a causal hypothesis
  D-->>Q: revise scope or act