The glow on the wire

Information can be present without being used

Imagine copying every intermediate activation into an auxiliary buffer. The buffer now contains everything the original computation contained. A sufficiently powerful probe can read many task variables from it. Yet the buffer has no outgoing connection to the model. Deleting it changes nothing.

This replacement thought experiment alters one property only: causal access. Representation is preserved, behaviour is preserved and readability may even improve, but the candidate loses all power to affect the output. The lesson is simple. Decodability establishes availability to an observer, not availability to the model.

A circuit is relative to a behaviour and a grain

A network does not contain one privileged circuit waiting to be uncovered. It contains a computation graph that can be partitioned at many grains: neurons, attention heads, MLP blocks, residual directions, sparse features, token positions or edges between them. A circuit claim chooses a behaviour, a metric, a dataset and a grain, then says that some subgraph explains the relevant change in that metric.

This relativity is not a weakness. An engineer can explain why a processor emits a cache miss without explaining every transistor. The danger appears when a local explanation is silently promoted into a global one. A head may be part of an indirect-object circuit on one prompt family and serve other functions elsewhere. A feature may be causal at one token position and merely correlated at another.

Mechanism is not a ranking of important parts

Feature importance asks how much a score changes when a variable changes. A mechanism account asks how organised operations transform one state into another. The difference is visible in a simple multiplication. If a network computes Y = A × B, either input can have a large local effect. Neither input explains multiplication. The mechanism includes the operation that combines them, the conditions under which it is applied and the downstream use of its result.

The same distinction applies to transformer components. A head with a large direct logit contribution may write useful evidence without selecting where that evidence came from. Another head may choose the source token but contribute little directly to the output. A third component may inhibit a tempting wrong answer. Ranking them by one scalar collapses different causal roles.

A useful mechanism description therefore has three layers. It names variables that change meaningfully across the task, operations that transform or route those variables, and organisation that determines which operation feeds which receiver. Causal abstraction formalises this aspiration by asking whether interventions on a low-level model correspond to interventions on a simpler high-level account (Geiger et al., 2025). A recent axiomatic approach adds a compositional demand: an interpretation should approximately preserve the semantics of parts when they are combined, rather than only fit isolated observations (Palumbo et al., 2025).

This gives a sharper standard than “the highlighted units matter”. The proposed account should predict a pattern of intervention outcomes that would be surprising under rival organisations. If two models contain equally readable country features but only one routes them into the answer head, the route is the discriminating mechanism evidence.

The worked example reveals why heatmaps are hypothesis generators. They can point to a place where the answer is readable or where a local intervention has a large effect. They do not yet describe the organisation that makes the computation reliable.

flowchart LR
  A["Opening case The head that<br/>knew Paris"] --> B["The glow on the wire"]
  B --> C["The counterfactual the heatmap<br/>cannot supply"]
  C --> D["Why a positive patch can lie"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#7B4CD8,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

The counterfactual the heatmap cannot supply

Clean, corrupt, restore

Causal tracing and activation patching create an internal counterfactual by combining two forward passes. A clean input produces the target behaviour and a cache of internal states. A matched corrupted input removes or reverses one task-relevant property, causing the metric to deteriorate. In a third run, selected clean activations replace the corresponding corrupted activations. If the behaviour recovers, the patched state is causally implicated in the difference between those runs.

Meng and colleagues used this logic to localise states involved in factual recall: corrupt the subject representation, then restore individual hidden states and observe which restorations recover the factual prediction (Meng et al., 2022). Related forms appear as interchange interventions, causal mediation, resample ablation and causal scrubbing (Vig et al., 2020) (Geiger et al., 2025).

Worked example 2: restoration in three numbers

Let m be a task metric such as the logit difference between the correct and incorrect answer. Suppose the clean run scores 4, the corrupted run scores −1 and the patched run scores 2. A convenient restoration fraction is:

The patch restores 60 per cent of the clean-corrupt gap. This number is useful because it normalises across prompt pairs with different baseline difficulty. It is not a probability that the component “is the mechanism”. It depends on four choices: the contrast, the state being replaced, the downstream context into which it is inserted and the metric.

The denominator also carries an invariant. If the clean and corrupted runs barely differ, there is no meaningful effect to restore. A patching experiment without a stable clean-corrupt gap is like testing a medicine on a symptom the patient does not have.

Noising and denoising ask asymmetric questions

There are two common patch directions. In a denoising experiment, the model runs on a corrupted input and receives a clean activation. The question is whether the candidate can restore missing task information in a damaged context. In a noising experiment, the model runs on a clean input and receives a corrupted activation. The question is whether replacing the candidate is enough to damage an otherwise intact computation.

These questions need not agree. Suppose the output uses an OR gate over two clean paths. Denoising either path into a fully corrupted run may restore the answer, so both look sufficient. Noising one path inside the clean run may have little effect because the other remains active, so neither looks necessary. With an AND gate, the reverse pattern can occur: corrupting either path breaks the output, while restoring only one into a fully corrupted run fails.

The disagreement is informative. It reveals how the candidate is embedded in the rest of the system. Reporting only the direction that produces the cleaner heatmap hides this organisation. A confirmatory study should normally include both directions or explain why one counterfactual is the scientifically relevant one. Heimersheim and Nanda emphasise that noising and denoising are not symmetric and should not be interpreted as interchangeable measures (Heimersheim and Nanda, 2024).

For the numerical example above, imagine that noising the candidate reduces the clean score from 4 to 1.5. Relative to the five-point clean-corrupt gap, the necessity loss is 50 per cent. Denoising restored 60 per cent. The two numbers together say more than either alone: the state can carry much of the missing information, and the intact model also depends materially on it, though neither test yet identifies the route.

Four questions, not one

A single restoration asks whether the candidate can carry enough clean information to improve a corrupted run. A mechanism claim needs four distinct questions.

Contribution: does changing the candidate change the behaviour? Rescue: does restoring the clean state recover the behaviour after a controlled corruption? Route: does the effect reach the output through the predicted downstream path? Stability: does the result survive reasonable changes in prompts, metrics, patch values and model seeds?

From component patching to path patching

Ordinary activation patching replaces a component’s output and lets that change propagate through every available downstream route. Path patching narrows the question. It transfers the candidate’s influence only into a selected receiver or edge while holding other routes to a reference state. Causal scrubbing goes further by formalising a high-level hypothesis and exchanging low-level activations that should be equivalent under that hypothesis.

These methods matter because a mechanism is organised. If the claim is “head H copies the indirect-object name into the residual stream, then head J reads it and raises the correct token”, a route test should predict two things. Patching H into J’s input should rescue the output, while blocking H-to-J transmission should remove that rescue. A component-only patch cannot distinguish this account from any alternative route leaving H.

Why a positive patch can lie

Redundancy makes necessity disappear

Return to the two-wire OR circuit. Each path participates in normal computation, but either can mask the loss of the other. Neural networks often contain similar overdetermination. The indirect-object study found backup name-mover heads that become more active when primary heads are disrupted (Wang et al., 2023). This “Hydra” behaviour means a small ablation effect can indicate compensation rather than irrelevance.

Group interventions can reveal the hidden organisation. Ablate A, then B, then both. Compare the joint effect with the sum of single effects. If the joint loss is much larger, the components are substitutable or synergistic. If it is smaller, they may overlap or inhibit one another. The correct unit of explanation may be a set, a gate or a conditional policy rather than an isolated head.

A rescue can travel through the wrong route

Suppose a researcher identifies a one-dimensional subspace that appears to encode “Paris”. Patching that direction into a corrupted run causes the model to answer Paris. It is natural to conclude that the subspace carries the factual feature used by the model. Yet the intervention may be large enough to activate a dormant detector elsewhere. The answer changes as if the intended feature were restored, but the normal model never routes factual recall through that subspace.

One useful control is intervention scaling. Reduce the patch magnitude from full replacement towards zero and record both the target output and predicted downstream states. A plausible local mechanism should usually show a coherent response near the natural activation range. An abrupt effect only at extreme magnitudes may indicate threshold crossing or off-manifold behaviour. This is not a universal law, because real circuits can be nonlinear. It is a warning to inspect the path rather than celebrate the endpoint.

Route evidence must predict an intermediate consequence

Blocking a proposed edge is stronger than patching a source component, but it is still possible to obtain an output effect for the wrong reason. A route claim becomes sharper when it predicts an intermediate state. If H is said to write a name feature that F reads, increasing H should change F’s pre-activation in a specified direction and at a specified token position. Patching an unrelated receiver should not produce the same readback.

This creates a two-level test. The output metric asks whether the intervention matters for behaviour. The readback asks whether it travelled through the proposed internal transformation. Agreement is not automatic. A patch may raise the correct logit while F remains unchanged, which weakens the H-to-F account. F may change as predicted while the final answer does not, suggesting that the route exists but is not sufficient under the current context.

Readbacks also help distinguish a mechanism from a steering handle. A steering direction is useful if adding it reliably changes behaviour. A mechanism direction is more demanding: its intervention should recreate the downstream pattern expected from the naturally occurring computation. Control is evidence of leverage; matched internal propagation is evidence of route.

The metric can manufacture importance

Changing one logit by two units has a stable additive meaning. Changing a probability by twenty percentage points does not, because softmax compresses or amplifies the same logit change depending on all competing logits. Accuracy is coarser still: a substantial internal effect can leave the top-ranked token unchanged.

For targeted language tasks, logit difference often gives a useful soft metric, while KL divergence can capture changes across the full output distribution. Neither is universally correct. If the mechanism claim concerns choosing Bob over Alice, a Bob-minus-Alice logit difference excludes components that merely boost all names. If the claim concerns preserving the whole next-token distribution, that metric is too narrow. Heimersheim and Nanda recommend comparing several metrics because agreement is stronger evidence and disagreement reveals what each test actually measures (Heimersheim and Nanda, 2024).

The ablation value changes the task

Zeroing a component, replacing it with a mean activation, resampling from another prompt and optimising a replacement value are not interchangeable. Each creates a different counterfactual model. Miller and colleagues found circuit faithfulness scores to be highly sensitive to apparently small changes in ablation methodology (Miller et al., 2024). The score reflects both the proposed circuit and the world created outside it.

This is why “faithfulness = 90 per cent” is incomplete without the ablation distribution, metric and prompt set. A circuit that performs well when excluded edges receive activations from a matched corrupted prompt may fail when they receive means or zeros. The circuit has not necessarily changed. The test has.

Interactions make single-component scores unstable

Activation patching is often interpreted as measuring the indirect effect through one mediator. In a nonlinear network with skip connections and many mediators, the patched component’s effect can depend on the states of all the others. A recent analysis decomposes the usual patching estimand into a pure indirect component plus interaction terms, and shows that those interactions can suppress backup components or inflate context-dependent ones (Vaidyanathan et al., 2026).

That result is a preprint and should not be treated as the final theory of patching. Its practical warning is already testable: if component rankings change sharply across prompt distributions, patch directions or group contexts, report the interaction rather than forcing a universal order. Instability can be evidence about organisation, rather than only experimental noise.

Automation discovers candidates, not verdicts

Manual circuit analysis is expensive. Automated Circuit Discovery prunes edges by repeatedly testing whether their removal changes a chosen metric (Conmy et al., 2023). Attribution patching accelerates the search by using a first-order gradient approximation to estimate many patch effects with a small number of passes (Syed et al., 2024). Integrated-gradient variants and optimisation-based pruning aim to improve faithfulness.

These methods are valuable search engines. They rank where confirmatory effort should go. Their approximation errors, threshold choices and circuit-size objectives should not be mistaken for mechanistic evidence. Hanna, Pezzelle and Belinkov show why overlap with a previously named circuit is weaker than testing whether the discovered subgraph actually preserves the target behaviour when the rest of the model is ablated (Hanna et al., 2024).

Large-scale circuit tracing adds another layer. Anthropic’s attribution-graph work constructs local replacement models from cross-layer features, then uses the resulting graphs to propose feature interactions. The authors explicitly distinguish output matching from mechanistic faithfulness and validate selected hypotheses with perturbations; they also report that perturbation agreement degrades over longer downstream distances (Lindsey et al., 2025). The general lesson is healthy: an interpretable surrogate is a hypothesis instrument until its intervention behaviour matches the original model.

stateDiagram-v2
  [*] --> Claim
  state "Information can be present<br/>without being used" as Claim
  state "Cut the wire, keep the signal" as Test
  state "A circuit is relative to a<br/>behaviour and a grain" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

A mechanism claim passport

The field needs a way to keep useful local findings without inflating them. The proposed mechanism claim passport treats each explanation as a versioned experimental object. It does not output a consciousness score, a truth badge or a universal faithfulness number. It records which inferential border the evidence has crossed.

The passport has six stamps. Scope fixes the model revision, behaviour, prompt distribution, metric and component grain. Contrast verifies that clean and corrupted cases isolate the intended variable and create a stable gap. Contribution combines ablation or noising with clean-state rescue. Route tests the predicted receiver and blocks bypasses. Rivals challenges redundancy, dormant paths and alternative bases. Stability repeats the result across baselines, magnitudes, metrics, seeds and holdout prompts.

Four claim levels

The passport produces four useful levels. A readable pattern is supported by observation, probing or visualisation. A local causal contributor changes a declared metric under a declared intervention. A mechanism slice includes a tested route and survives named rivals within a bounded distribution. A stable mechanism claim also survives holdouts and reasonable methodological variation, ideally with independent reproduction.

These levels prevent a common rhetorical jump. “A linear probe decodes the indirect object” is a pattern claim. “Patching head H changes Bob-versus-Alice logit difference” is a local causal claim. “H writes a name feature that receiver J uses, and blocking H-to-J transmission removes the rescue” is a mechanism-slice claim. “The same organised route appears across templates, seeds and model variants” approaches a stable mechanism claim.

Uncertainty belongs in the claim, not only the error bars

Most circuit reports aggregate patch effects across prompts, then present a mean heatmap. The average can conceal two different worlds. In one, the same route has a modest effect on almost every prompt. In the other, the route dominates a small subgroup and does nothing elsewhere. The means may match, but only the first supports a broadly shared mechanism within the sampled distribution.

The passport should therefore retain the distribution of effects. Report confidence intervals or bootstrap intervals for the clean-corrupt gap and restoration, but also inspect heterogeneity by template, token position, lexical class and baseline. A useful stability statement is concrete: “the route’s effect was positive in 87 per cent of held-out prompt pairs and remained positive under three ablation semantics”. It is more informative than a single average faithfulness score.

Pre-registration helps at the point where exploration becomes confirmation. Discovery data may be used freely to choose prompts, hooks, metrics and candidate routes. Before the holdout run, freeze the candidate graph, allowed intervention magnitudes, primary metric, exclusion rules and claim wording. Otherwise, the researcher can unknowingly search across many analytical choices until one route appears clean.

Independent reproduction is strongest when it varies something material. Re-running the same notebook is useful for computational reproducibility. Rebuilding the intervention with a second framework, a separately generated prompt set or a neighbouring model checks whether the finding depends on hidden implementation choices. The goal is not ceremonial replication. It is to discover which parts of the claimed mechanism are invariant and which are artefacts of the experimental interface.

Executable lab: watch interactions change the conclusion

The lab below implements a synthetic two-mediator system. The clean input activates paths A and B. Their outputs combine with a tunable interaction term. A backup control lets B strengthen when A is ablated. The bars show clean, corrupt, single-patch, joint-patch and ablation outcomes.

Use it to test three intuitions. Increase backup and watch A appear unnecessary despite participating in the normal route. Increase interaction and compare the joint patch with the sum of single patches. Increase patch magnitude and notice how a strong intervention can dominate the output even when natural-path evidence remains weak.

What a positive result permits: when single and joint patches behave as predicted, route blocking removes rescue and holdout results remain stable, the candidate account may be described as a mechanism slice within the declared scope. What a negative result means: it weakens that particular organisation, not the existence of all internal structure. The candidate may be mislocalised, the grain may be wrong, the circuit may be distributed, or the behaviour may use several conditional routes.

What the lab cannot establish: it does not validate any real transformer circuit, choose universal thresholds or show that activation patching identifies pure indirect effects. It makes assumptions visible so that a real protocol can replace them with measured values.

from __future__ import annotations

from dataclasses import dataclass
from enum import Enum


class ClaimLevel(str, Enum):
    INVALID_CONTRAST = "invalid contrast"
    READABLE_PATTERN = "readable pattern"
    LOCAL_CAUSAL_CONTRIBUTOR = "local causal contributor"
    MECHANISM_SLICE = "mechanism slice"
    STABLE_MECHANISM = "stable mechanism claim"


@dataclass(frozen=True)
class InterventionEvidence:
    clean: float
    corrupt: float
    patch_candidate: float
    ablate_candidate: float
    patch_joint: float | None = None
    patch_rival: float | None = None
    route_blocked_patch: float | None = None
    holdout_restoration: float | None = None
    baseline_spread: float | None = None

    def gap(self) -> float:
        return self.clean - self.corrupt

    def restoration(self, value: float) -> float:
        gap = self.gap()
        if abs(gap) < 1e-9:
            raise ValueError("Clean-corrupt gap is too small to normalise.")
        return (value - self.corrupt) / gap

    def necessity_loss(self) -> float:
        gap = self.gap()
        if abs(gap) < 1e-9:
            raise ValueError("Clean-corrupt gap is too small to normalise.")
        return (self.clean - self.ablate_candidate) / gap

    def joint_interaction(self) -> float | None:
        if self.patch_joint is None or self.patch_rival is None:
            return None
        a = self.patch_candidate - self.corrupt
        b = self.patch_rival - self.corrupt
        joint = self.patch_joint - self.corrupt
        return joint - (a + b)


def grade(e: InterventionEvidence) -> ClaimLevel:
    if abs(e.gap()) < 0.25:
        return ClaimLevel.INVALID_CONTRAST

    rescue = e.restoration(e.patch_candidate)
    necessity = e.necessity_loss()
    if max(rescue, necessity) < 0.20:
        return ClaimLevel.READABLE_PATTERN

    if e.route_blocked_patch is None:
        return ClaimLevel.LOCAL_CAUSAL_CONTRIBUTOR

    blocked_rescue = e.restoration(e.route_blocked_patch)
    route_specific = rescue >= 0.40 and blocked_rescue <= 0.15
    if not route_specific:
        return ClaimLevel.LOCAL_CAUSAL_CONTRIBUTOR

    stable = (
        e.holdout_restoration is not None
        and e.holdout_restoration >= 0.35
        and e.baseline_spread is not None
        and e.baseline_spread <= 0.20
    )
    return ClaimLevel.STABLE_MECHANISM if stable else ClaimLevel.MECHANISM_SLICE


if __name__ == "__main__":
    evidence = InterventionEvidence(
        clean=4.0,
        corrupt=-1.0,
        patch_candidate=2.0,
        ablate_candidate=1.2,
        patch_joint=3.5,
        patch_rival=1.4,
        route_blocked_patch=-0.4,
        holdout_restoration=0.52,
        baseline_spread=0.12,
    )
    print("restoration:", round(evidence.restoration(evidence.patch_candidate), 3))
    print("necessity loss:", round(evidence.necessity_loss(), 3))
    print("interaction:", round(evidence.joint_interaction() or 0.0, 3))
    print("claim level:", grade(evidence).value)

A smallest useful implementation

A serious first implementation does not require a frontier model. Use a small open transformer, a task with unambiguous paired prompts and a metric defined before inspection. Cache activations at one grain, run broad exploratory patching on a discovery split, then freeze three candidate routes. On a holdout split, run noising, denoising, path-restricted patches, dose curves, joint interventions and at least two ablation baselines. Store every run in a manifest containing model revision, tokenizer, prompts, hooks, tensor shapes, metric code, random seed and software environment.

What observation would strengthen the passport? Candidate routes should predict downstream state changes as well as output changes, and those predictions should remain directionally stable across holdouts. What would weaken it? Large rescue with no predicted readback, strong dependence on one corruption, or reversed component rankings under reasonable baselines. A rival explanation could still say that several low-level routes realise the same higher-level operation. The passport is therefore best applied to scoped computational claims, not to claims about understanding, agency or experience.

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: Opening case The head that knew Paris
  M->>E: Information can be present without being used
  E-->>M: compare rival explanations
  E->>D: Separate discovery from authorisation
  D-->>Q: revise scope or act

Separate discovery from authorisation

When an interpretability result arrives, do not ask first whether the visualisation is compelling. Ask what decision it is meant to support. A heatmap may guide the next experiment. A local causal contributor may justify targeted monitoring. A tested mechanism slice may support a narrow model edit or a focused regression test. A stable mechanism claim may influence release assurance. These are different evidential jobs.

The practical decision is to separate discovery from authorisation. Probes, attribution scores, sparse features and automated circuit finders may nominate candidates. They should not authorise a mechanism statement on their own. Authorisation requires a frozen claim, held-out interventions, route predictions, rival controls and a record of methodological sensitivity.

This discipline changes how failures are treated. If a clean-state patch rescues the answer but route blocking does not remove the rescue, the result has not “failed interpretability”. It has falsified one route and exposed another question. If a head’s importance vanishes under a different corruption, the head may be contrast-specific rather than spurious. If joint interventions reveal backup behaviour, the mechanism has become more organised, not less interpretable.

The strongest outcome is not a tiny circuit with a neat story. It is an explanation that continues to make correct predictions when the model is perturbed in ways the explanation did more than fit. A pattern becomes a mechanism when it survives being used as an experimental theory.