A circuit is a causal claim, not a drawing

A railway map can show rails, stations or passenger journeys. Those are related objects, but they answer different questions. The metal track says what movement is physically possible. A route map says which sequence serves a journey. A timetable says when the route is actually used. Calling all three “the railway” creates the same confusion that surrounds neural circuits.

Early circuits work proposed studying neural networks by decomposing them into features and connections, then asking how those parts implement a behaviour.[1] Transformer-specific work sharpened the physical substrate: residual streams, attention heads, query-key selection and output-value transformations can be written as interacting computational paths.[2] That decomposition is indispensable because it provides places to measure and intervene. It does not decide which level of description deserves explanatory status.

Three objects called a circuit

The first object is the physical subgraph: named heads, MLP blocks, feature directions or edges chosen from a particular model and hook vocabulary. It is executable and precise. It is also coordinate-bound. Reparameterisation, head permutation, feature rotation or continued training can change those coordinates while leaving behaviour and algorithmic organisation similar.

The second object is the role graph: semantic encoder, duplicate detector, candidate suppressor, value mover, answer writer. Roles are claims about what components do and how information flows among them. They can survive component substitution, but only if their names predict interventions. Calling a head a “name mover” after viewing examples is description. Predicting which donor activation should change which answer, and which paraphrases should preserve that effect, begins explanation.

The third object is a causal abstraction: a simpler high-level computation plus a mapping from low-level model states to its variables. Causal abstraction work formalises mechanistic interpretation as finding intelligible algorithms that are faithful simplifications of lower-level mechanisms.[5] The abstraction need not reproduce every activation. It must preserve the intervention relations relevant to the claim.

A compact way to state the burden is to let F denote the full model, C the proposed high-level circuit and α a mapping from model states into high-level variables. For each input x and intervention i inside the declared test family, two routes should agree within a stated tolerance. One route intervenes on the model and then maps its state upward. The other maps the ordinary model state upward and applies the corresponding high-level intervention inside C.

This condition blocks two easy escapes. Behavioural equivalence alone is too weak because a lookup table can reproduce answers without sharing a mechanism. Exact activation equality is too strong because redundant coordinates and implementation changes can preserve a causal role. The informative question is whether the abstraction predicts what changes, what remains fixed and where the prediction fails.

It also forces semantic discipline. If the high-level variable is called “candidate identity”, then interventions that change identity while preserving position, token frequency and syntax should follow the abstraction. Interventions that change only those surface properties should not. A role name earns its meaning by the counterfactual partitions it predicts.

Thought experiment one: swap the technicians

Two technicians run a theatre cue sequence. One opens the curtain, one changes the light, one starts the sound. The next night, the staff exchange consoles but perform the same roles in the same dependency order. A camera-based account tied to faces declares a new mechanism. A script-level account declares the same mechanism. Which is better depends on the intervention: if disabling the “light” role still produces the predicted dark stage regardless of operator, the script captures the stable cause.

Now weaken the account further and say only that “someone prepares the stage”. That description survives every staff change, but it no longer predicts which interruption causes which failure. Mechanistic stability sits between coordinate worship and vacuous functionalism. The accepted level is the coarsest abstraction that continues to predict discriminating interventions.

This view aligns with evidence that some transformer mechanisms can remain algorithmically recognisable while their implementing heads change during training. Tigges and colleagues tracked circuits across 300 billion training tokens and models from 70 million to 2.8 billion parameters. Their study found that functional component types and overarching algorithms could persist even when particular attention heads changed.[14] That is evidence for role-level stability in the studied tasks, not a licence to assume it for every behaviour or architecture.

Circuit granularity is part of the hypothesis

A whole attention head is convenient because it has a clean tensor boundary. It can still perform several input-dependent operations. An edge between two modules is finer, but may bundle many feature directions. A learned feature or subspace is finer again, yet its basis may be non-unique and an intervention may leave the model's normal activation manifold. No granularity is automatically the “real” circuit.

The choice changes both discovery and stability. A head-level circuit may replicate because the same broad module remains useful, while its feature-level computation changes. A feature-level circuit may preserve a semantic direction across prompts while moving between heads. An edge-level circuit may look sparse because a replacement baseline suppresses distributed background support. Every circuit claim should therefore name its unit of analysis and test at least one adjacent resolution.

Induction-head research offers a useful example of role-first reasoning. The proposed motif links a previous-token detector to a head that copies the token following an earlier match, and the work connects that motif to in-context learning behaviour across studied models.[3] The explanatory object is more than a head number. It is a pattern of attention, value transport and behavioural effect. The same standard should apply when a circuit is less tidy.

A practical multi-resolution check begins coarse, because exhaustive feature-level intervention is expensive. Localise with residual or component patches. Split the strongest components into heads, positions, edges or subspaces. Then ask whether the refined pieces jointly reconstruct the coarse effect and whether the semantic interpretation improves on held-out counterfactuals. If refinement changes the story, publish the discrepancy rather than selecting the prettier level.

flowchart LR
  A["The circuit that moved while<br/>the behaviour stayed put"] --> B["A circuit is a causal claim,<br/>not a drawing"]
  B --> C["Activation patching asks one<br/>conditional counterfactual"]
  C --> D["Stability has several<br/>dimensions, and they can<br/>disagree"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#007D68,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Activation patching asks one conditional counterfactual

Suppose a clean prompt makes the model prefer answer A, while a carefully corrupted prompt makes it prefer B. Cache an internal activation from the clean run. Insert that activation at the same hook in the corrupted run, then recompute everything downstream. If the answer moves towards A, the donor activation carried something causally useful in that recipient context.

That sentence contains every important qualification. The result is conditional on the clean donor, the corrupted recipient, the exact slice replaced, the downstream network left available and the metric used to score recovery. Activation patching is stronger than observing correlation, but it does not name the transported content by itself. Practical guidance on patching therefore emphasises corruption design, metric choice, hook placement and interpretation boundaries.[8]

Five choices determine what a patch means

The corruption defines the question. Replacing one name with another, shuffling all names, changing syntax and inserting random tokens do not create equivalent recipients. A corruption should remove or alter the variable under study while preserving enough of the ordinary task that downstream computation remains interpretable. If the corrupt run already fails for unrelated reasons, recovery may measure general repair rather than the proposed mechanism.

The donor defines the alternative world. A clean activation from the matched counterfactual preserves many hidden properties besides the named variable. A random donor changes too much. A mean donor may be off-distribution. Resampling within an equivalence class can be more discriminating, provided that class is specified before results are inspected.

The patch direction defines the dependence tested. Clean-to-corrupt patching asks whether donor information can rescue the corrupt context. Corrupt-to-clean patching asks whether introducing the corrupt state harms the clean computation. Non-linearity, saturation and repair can make these effects asymmetric. Reporting only the larger direction chooses a story after observing the data.

The hook defines the causal boundary. Patching before layer normalisation lets the donor alter subsequent centring and scaling. Patching a head output differs from patching its attention pattern, values or residual contribution. Token alignment matters when a corruption changes tokenisation. The experiment receipt must identify layer, position, head or feature axes and whether the hook is pre- or post-operation.

The metric defines visible success. Top-token accuracy can miss a large movement that stops short of a rank flip. A logit difference can conceal changes elsewhere in the distribution. Normalised restoration becomes unstable when clean and corrupt margins are close. Use one sensitive metric, one behavioural outcome and the unnormalised values. Add no-op hooks, unchanged donor slices, irrelevant positions and matched random donors to detect implementation drift.

Worked example one: the pair that crowns two mechanisms

The laboratory accompanying this article has two input variables. The semantic variable determines the correct answer. A surface cue is correlated with it in the discovery examples and contributes half as much to the output. The model contains a semantic encoder, a surface encoder, a primary router, a conditional backup and an answer writer. Every value is inspectable.

In the first pair, the clean input is (semantic=+1, surface=+1) and the corrupt input is (−1, −1). The clean margin is +1.50; the corrupt margin is −1.50. Patching the semantic encoder restores the margin to +0.50, or 0.667 restoration. Patching the surface encoder restores it to −0.50, or 0.333. A discovery scan therefore assigns causal credit to both paths.

Now change one feature at a time. Keep the surface cue at −1 in both clean and corrupt inputs while changing only the semantic variable. The semantic patch restores 1.000; the surface patch restores 0.000 because donor and recipient surface states are identical. In a third family, reverse the surface cue relative to the semantic answer. The semantic patch overshoots to 2.000 restoration, while the shortcut patch produces −1.000 and moves the output in the wrong direction.

This is the article's serious negative control. The shortcut has the same granularity as the semantic route and earns a visible patch score on the discovery contrast. It is rejected only when the experiment changes a feature that the mechanism claim says should be irrelevant. A random component baseline would be easier to beat, but less discriminating. The best negative control is a plausible rival mechanism that succeeds wherever the original experiment is confounded.

The example also explains why a patch score is not an intrinsic property of a component. Restoration changes from 0.667 to 1.000 to 2.000 for the same semantic node because the clean-corrupt denominator and the recipient's remaining surface path change. Metric-robustness studies have likewise warned that circuit rankings and faithfulness conclusions can depend on the evaluation design.[13] Keep raw clean, corrupt and patched metrics beside every normalised score.

Thought experiment two: the bridge that repairs itself

A town has a main bridge and an emergency ferry. Under ordinary traffic, almost nobody uses the ferry. Close the bridge and the ferry starts running. Measuring the traffic loss after closing only the bridge suggests the bridge is nearly unnecessary. Closing only the dormant ferry also changes little. Close both and the route collapses.

A one-at-a-time circuit scan can produce exactly this mistake. A component's effect depends on what the rest of the model does after the intervention. The Hydra effect study reported compensation by later language-model layers after an attention layer was ablated.[9] More recent causal-mediation analysis argues that activation-patching estimates can contain interaction terms with other mediators, making conditionally important components look small or inflating effects attributed to a single component.[17] The latter is a recent preprint and should be treated as an active methodological claim, not settled consensus.

Necessity and sufficiency must therefore be stated relative to an intervention regime. A circuit may be sufficient when all excluded edges are replaced by a baseline, yet not be the route ordinarily used. A component may be unnecessary under single ablation yet conditionally necessary when a backup is blocked. Path patching, co-ablation, interaction tests and causal scrubbing can refine the hypothesis, but each requires an explicit account of which distinctions should be preserved.[7]

The correct response is not to demand every higher-order interaction. That becomes combinatorial. Use a staged design: first identify plausible routes, then test interactions where compensation is predicted or where the decision consequence warrants the cost. Report the untested interaction boundary. A bounded circuit claim may be scientifically useful; an unbounded one is merely convenient.

Stability has several dimensions, and they can disagree

“The circuit is stable” is incomplete. Stable across what? A mechanism claim normally crosses at least three families: inputs, interventions and implementations. An input family includes paraphrases, tokenisations, distractor structures and task variants. An intervention family includes patch direction, donor choice, baseline, granularity and joint disturbances. An implementation family includes seeds, checkpoints, model sizes or architectures.

Design the stability family before discovering the graph

A useful test family resembles a factorial experiment. List the variables the proposed mechanism says are causally relevant, such as entity identity and grammatical role. List plausible nuisances, such as name frequency, token length, position, voice and punctuation. Then construct pairs that vary one relevant variable while holding nuisances fixed, vary nuisances while holding the relevant variable fixed, and deliberately reverse common correlations.

Not every combination will be linguistically natural. Naturalness is part of the boundary, not an excuse to abandon control. Use generated templates for precise separation, human-written paraphrases for ecological breadth and corpus examples for distributional realism. Keep the three strata distinct because success on one does not imply success on the others.

Before circuit discovery, verify that the full model performs the intended behaviour on each family and that the clean-corrupt contrast is large enough for the metric. A circuit cannot explain competence the model does not reliably display. Separate model failures from explanation failures: when the full model changes strategy or loses accuracy, the correct result may be a mechanism boundary rather than a failed replication.

Pre-register the invariance predictions where the stakes justify it. Which role should survive passive voice? Which path should disappear when the surface cue is held fixed? Which component may be substituted across checkpoints? Which observation would falsify the semantic label? This turns a collection of patches into a test of a causal theory and reduces the temptation to rename components after every surprising result.

The claim should name which transformations preserve the high-level variables. If passive voice changes the syntax but not the binding relation, a semantic binding circuit ought to preserve its role-level predictions. If a new task uses the same selection primitive for different content, component reuse may strengthen a general algorithmic interpretation rather than weaken task specificity. If a checkpoint moves the role to another head, physical overlap may fall while causal abstraction remains intact.

1. Causal fidelity

Does the proposed subgraph preserve the full model's relevant behaviour when excluded computation is replaced according to a declared baseline? IOI work evaluated an end-to-end circuit using faithfulness, completeness and minimality, while acknowledging remaining gaps.[4] Automated circuit discovery later showed that compact subgraphs can be recovered at scale, but discovery success is still metric- and task-dependent.[6] Faithfulness is a necessary compression test. It does not alone validate the semantic names attached to the nodes.

2. Discriminating counterfactuals

Can the circuit distinguish the preferred mechanism from a plausible rival? The surface-held-fixed pair in the laboratory does more work than a random ablation because it preserves the shortcut while changing the target variable. Causal scrubbing expresses this idea by testing whether activations can be resampled while preserving exactly the input distinctions the interpretation says each node may depend on.[7] A hypothesis that cannot generate its own adversarial family is not ready to be called a mechanism.

3. Recurrence within the claimed input family

Does the role recur across prompt instances, or is the reported graph an average that few examples actually use? A recent preprint proposes separating within-task consistency from task specificity. Across six tasks and seven models, it reports substantial within-task component reuse and causal impact relative to capacity-matched random ablations.[16] Its own scope is bounded by one extraction method, head-and-MLP granularity and the chosen tasks. The broader lesson is durable: report the distribution of per-example circuit effects, not only the union graph.

4. Specificity or justified reuse

A component shared across tasks is not automatically non-explanatory. It may implement a general operation such as copying, comparison or suppression. Merullo and colleagues found substantial reuse between the IOI and Coloured Objects tasks in their studied GPT-2 models, including about 78 per cent overlap among in-circuit attention heads and a targeted intervention that improved the latter task.[11] Conversely, the recent consistency study found that many circuit components were not unique to one task.[16] The right question is whether the shared role predicts both tasks, not whether each task owns private hardware.

5. Repair and interaction awareness

Have plausible backup paths and joint effects been tested? Single-component necessity scores are weakest where the model can compensate. Co-ablation, path restriction and interaction-aware sampling should be targeted at candidate redundancies. Where these tests are infeasible, the explanation must say that it identifies a useful route under first-order interventions, not the unique or complete mechanism.

6. Role-level replication

Do the same causal roles and dependency predictions recur across seeds, checkpoints or scales even when component coordinates move? Physical overlap is strong evidence when it appears, but it should not be the only acceptance rule. A stable algorithm can be realised by different heads; identical heads can also change function. Replication therefore needs both coordinate-level and role-level comparisons.

Benchmarks can make these dimensions more comparable. MIB separates circuit localisation from causal-variable localisation across multiple tasks and models, evaluating whether methods recover concise causal pathways or variables.[15] Such benchmarks measure methods under known evaluation structures. They do not eliminate the need to define the behaviour, corruption and invariance family for a new mechanism claim.

Overlap is especially seductive because it is easy to visualise. Two methods can recover many of the same nodes while producing circuits with different faithfulness. Hanna and colleagues found that an integrated-gradients variant of edge attribution patching yielded more faithful circuits than ordinary edge attribution patching even when both had high overlap with previously identified components.[12] Use overlap to compare physical selections; use intervention fidelity to compare explanatory adequacy.

The mechanism stability dossier

A circuit explanation should be released like a scientific instrument, not a decorative annotation. The dossier below is an acceptance protocol for research, model diagnosis or internal monitoring. It is deliberately stricter when the proposed use changes a model, blocks a release or supports a consequential control.

Worked example two: from a name-binding graph to a release decision

The discovery dataset contains active-voice sentences with two single-token names. Patching identifies an early entity state, two middle attention paths and a late answer writer. The subgraph retains 94 per cent of the full model's answer-margin contrast under the chosen resample baseline. That is useful causal fidelity, but the release question is not yet answered.

The team builds a family of contrasts. Passive voice preserves the liable party while changing word order. Token-split names preserve identity while changing lexical boundaries. A third distractor preserves syntax while adding another organisation. A negation family changes liability while keeping names and positions fixed. An authorised exception family changes the decision rule rather than the evidence. Each family exists because it separates one candidate mechanism from another.

The role-binding path recurs across active and passive voice, but one physical head is replaced by another after continued training. The role-level patch prediction remains correct. Token-split names recruit an extra early MLP, so the original physical graph is incomplete while the abstraction survives with an expanded encoder role. The negation family fails: the proposed circuit writes the extracted party, but a separate policy path determines whether that party is actually liable. The correct conclusion is bounded.

The circuit is accepted as a regression sentinel for entity-to-answer transport within affirmative liability clauses, not as an explanation of legal liability and never as an authority to approve a document. The dossier records the failed negation scope, the component substitution across checkpoints and the prompt families tested. A later release can extend the claim only by adding evidence.

Treat the dossier as a living assurance argument

The dossier is not a certificate that the model has been understood. It is a structured argument with a claim, supporting evidence and named defeaters. The claim might be “this role graph explains entity transport for affirmative two-party clauses in model version V”. Evidence includes patch results, held-out families, co-ablation tests and replication receipts. Defeaters include tokenisations not covered, a failed negation family, unexplored higher-order interactions and a model update.

Each item needs an owner and expiry trigger. A tokenizer change invalidates position-aligned patches. Quantisation can alter activation geometry. Fine-tuning may preserve output accuracy while moving roles. A new prompt distribution may introduce structures absent from the stability family. When a trigger fires, the claim returns to review rather than remaining true by documentation inertia.

Do not compress the dossier into one green score. Some gaps are conditional holds; others defeat the interpretation. Missing cross-seed replication may permit a snapshot-bound research claim. Failure of the discriminating counterfactual rejects the semantic role. Missing use boundaries prevents operational adoption even when the science is strong. The gate should follow the consequence of being wrong.

Executable artefact: the circuit stability laboratory

The accompanying standard-library Python script implements the minimal model used in Figures 3 to 6. It runs three prompt families, patches five named sites, performs single and joint ablations, compares two synthetic checkpoints and emits a machine-readable dossier. Assertions enforce the positive semantic case, the inert negative control, the counter-correlated shortcut failure, the self-repair interaction and the role-level replication result.

The artefact is intentionally smaller than a transformer. Its purpose is to make experimental logic executable. Replacing it with a real model changes tensor sizes, hook names and compute cost, but not the evidence contract: paired inputs, named donor and recipient states, downstream recomputation, raw metrics, rival mechanisms, interaction tests and a reproduction receipt.

def forward(prompt, *, patches=None, ablate=()):
    patches = dict(patches or {})
    ablated = set(ablate)

    semantic = 0.0 if "semantic_encoder" in ablated else prompt.semantic
    semantic = patches.get("semantic_encoder", semantic)

    surface = 0.0 if "shortcut_encoder" in ablated else prompt.surface
    surface = patches.get("shortcut_encoder", surface)

    primary = 0.0 if "primary_router" in ablated else semantic
    primary = patches.get("primary_router", primary)

    if "backup_router" in ablated:
        backup = 0.0
    elif "primary_router" in ablated:
        backup = 0.85 * semantic       # conditional self-repair
    else:
        backup = 0.0

    margin = primary + backup + 0.5 * surface
    return margin

family              site                 restoration
confounded          semantic_encoder          0.667
confounded          shortcut_encoder          0.333
surface-held-fixed  semantic_encoder          1.000
surface-held-fixed  shortcut_encoder          0.000
counter-correlated  semantic_encoder          2.000
counter-correlated  shortcut_encoder         -1.000

primary-only ablation drop:                  0.15
primary-plus-backup ablation drop:           1.00
physical component set Jaccard:              0.00
role graph edge overlap:                     1.00

Operational use requires a stricter boundary

A circuit finding can guide a regression test, a targeted dataset, a hypothesis about model editing or a more efficient monitor. It should not become a safety or authorisation control merely because it is internal. The model may route around the measured path, the extraction method may miss features, an update may move the role, and the monitor itself may alter performance. External outcome tests and model-independent controls remain necessary.

Activation traces may also encode prompt content. Store only what the experiment requires, protect the trace as sensitive derived data, and record model, tokenizer, precision, prompt-pair hash, hook semantics, patch rule, metric code and raw outcome. A screenshot of a graph cannot be replayed after a model update. A decision receipt can.