The forward pass is a stack of counterfactual handles

A textbook diagram tells us what tensors are computed. An instrumented model adds a stronger property: every tensor can be named, cached, replaced and followed through the remaining computation.

The original Transformer introduced attention-only sequence modelling with residual connections, layer normalisation, position-wise feed-forward networks and masking for autoregressive decoding. Later decoder designs such as GPT-2 moved layer normalisation to the input of each sub-block and added a final normalisation; subsequent analysis has compared the optimisation consequences of pre-LN and post-LN. The pre-normalised arrangement is the specimen used here because it exposes two clean update paths into one evolving residual state. It is one transformer design, not the definition of all transformers.

Start with the state that persists

Let token x_t occupy position t. Its first residual vector is the sum of a learned token embedding E[x_t] and a position vector P_t:

The symbol r names the residual state, and its superscript names depth. Changing the token embedding changes lexical identity. Changing the position vector changes where the same identity appears. Patching the sum tests their combined downstream effect, while patching either input separately can distinguish identity from location. This distinction is already causal: two inputs can produce similar final behaviour through different combinations of those terms.

In a pre-normalised block, the model first computes u_t = LN(r_t). Layer normalisation centres and rescales features within that token vector before the attention projections. It does not mix token positions. A patch before normalisation asks what the whole residual state contributes; a patch after normalisation asks what the attention sub-block actually receives. The two interventions need not agree because normalisation can change the direction and scale of a replacement.

Routing and payload are separate transformations

For head h, the normalised vector is projected into a query, key and value:

The query at destination t helps express what that destination seeks. The key at source s helps express how that source may be addressed. Their dot product affects a routing score. The value is the source-specific payload available after routing. These interpretations are structural rather than semantic: a head need not correspond to a human-readable question, address or fact.

The attention weight from destination t to source s is:

Here d_h is head width, and m_t,s is zero for allowed reads and negative infinity for masked future positions. Raising one compatible score usually raises its weight while reducing others because softmax normalises the row. Patching a score row or pattern row therefore changes routing. It still says nothing by itself about what the routed values contain.

The head output is a weighted sum z_t,h = Σ_s α_t,s,hv_s,h. Heads are concatenated, projected by W^O, and added to the residual state:

The addition matters. Even if the attention update is removed, the earlier state remains available through the residual bypass. An output can therefore depend on information that attention copied, information already present at the destination, or an interaction between them.

The MLP is another write, not a detachable explanation

The attention-updated state is normalised again and passed through a position-wise MLP:

The MLP does not mix positions directly; it transforms the information already assembled at each position. Increasing one hidden pre-activation can change several output directions after the second linear map. Patching the MLP activation tests the nonlinear hidden state, while patching its output tests the actual residual write. Because the earlier residual remains, neither intervention isolates a stand-alone “MLP answer”.

After all blocks, final normalisation and the unembedding matrix produce logits ℓ_t = W_ULN(r_t^(L)). A logit is an unnormalised score for a vocabulary item. The difference between two relevant logits is often a more stable behavioural metric than one probability because it directly measures preference between the contrasted answers.

Intervention geometry is part of the hypothesis

A tensor name is still too coarse. The value tensor has axes for batch, head, source position and head feature. The attention pattern has axes for batch, head, destination and source. A residual state has batch, position and model feature. Replacing a whole tensor answers a different question from replacing one head at one source, one destination row, one feature direction or one token vector.

Suppose the clean source token is at position one and the final query is at position five. Patching V[:, :, 1, :] asks whether clean payload from that source is sufficient when every destination may read it. Patching pattern[:, :, 5, :] asks whether the clean routing distribution at the query is sufficient while values remain corrupt. Patching resid_post[:, 5, :] replaces the complete accumulated query state after the block, including all routes and bypasses. A high score at the last site is less local than a high score at the first.

Hook placement also matters. A pre-hook cache records what the unmodified computation produced. A replacement then enters the next operation. If the cache is taken after the hook, repeated experiments may accidentally use previously patched states as donors. Patching before layer normalisation differs from patching after it, because the destination's mean and variance transform the replacement. Patching scores after the causal mask with an unconstrained replacement can revive a future position; patching before the mask lets the ordinary masking operation be recomputed.

The intervention slice is therefore part of the causal claim, not an implementation footnote. A defensible record names model version, layer, tensor, axes, source and destination positions, donor run, hook side, precision and downstream metric. Without that geometry, a result cannot be reproduced and may not mean what its label suggests.

Each formula now names a legitimate intervention site, but naming sites is not causal discovery. The next step is to specify two worlds whose difference has a clear meaning, then ask which replacements transfer that difference through the fixed model.

flowchart LR
  A["The identical heatmap"] --> B["The forward pass is a stack of<br/>counterfactual handles"]
  B --> C["Clean and corrupted runs turn<br/>handles into causal tests"]
  C --> D["A successful patch is local<br/>evidence, not a map of truth"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#087A98,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Clean and corrupted runs turn handles into causal tests

Activation patching is a controlled transplant. It copies an internal state from one run into another and lets the rest of the network respond normally.

Choose a clean prompt x^C that elicits the behaviour of interest and a corrupted prompt x^R that removes or changes one causal feature. Run both through the same frozen model and cache activation h_j at site j. Then rerun the corrupted prompt while replacing only the chosen tensor slice with its clean counterpart. In intervention notation, the rerun applies do(h_j = h_j^C).

This operation differs from reading a heatmap or training a probe. A heatmap records a quantity produced in the original computation. A probe establishes that a chosen decoder can recover information. A patch asks whether the downstream model changes when a particular internal state is supplied. Causal mediation, causal tracing, circuit discovery and related methods differ in their formal claims and search procedures, but they share this move from association to controlled internal change. A practical activation-patching guide by Heimersheim and Nanda catalogues the design choices and scope limits of these experiments.

Measure recovery without hiding the denominator

Let M be a scalar metric, such as the final TRUE logit minus the FALSE logit. Define clean score M_C, corrupt score M_R, and patched-corrupt score M_j←C. Denoising recovery is:

A value near zero means the patch did not transfer the clean contrast on this metric. A value near one means it restored the full clean-corrupt gap. Values can be negative or exceed one, so clamping them would discard evidence about reversal or overshoot. If the denominator is small, the ratio is unstable and the pair is unsuitable.

Reverse the donor and destination to perform noising. Patch the corrupt activation into the clean run and compute N_j = (M_C − M_j←R)/(M_C − M_R). Denoising asks whether the clean state at that site is sufficient to restore the contrast within the intact downstream system. Noising asks whether the corrupt state is sufficient to destroy it within the clean system. These questions coincide only under restrictive assumptions about linearity, redundancy and interaction.

Node patches and path patches ask different questions

A component patch replaces everything a node contributes at one site. It can show that the donor state is sufficient for downstream recovery, but it does not identify which upstream source created the relevant part or which downstream edge consumed it. Path patching narrows the question by allowing a changed activation to travel through selected connections while other inputs are held to a reference run. Causal mediation similarly distinguishes total effects from effects transmitted through a named mediator.

Consider a head output at the query position. Full patching may restore the answer. Three explanations remain compatible: the head copied the changed source bit, the head changed a nuisance feature that the MLP interprets, or the replacement perturbed layer-normalisation geometry and activated a downstream route. To distinguish them, patch source values, destination queries, pattern rows and head mixtures separately; then test the edge from the head output into the next MLP or logit contribution. Each refinement removes rival causal stories, but only within the chosen clean-corrupt worlds.

There is a practical search trade-off. Exhaustively patching every element is expensive and produces a multiple-comparison problem. Coarse residual sweeps can locate candidate layers and positions; finer head, route, value and feature tests can follow. The coarse sweep is exploratory, so its strongest cells should not be reported as confirmatory evidence on the same prompts. Reserve held-out pairs or preregister the refinement before treating a discovered site as replicated.

Worked example: the one-block XOR specimen

The executable model uses a fourteen-token vocabulary, model width 24, three attention heads, MLP width 48 and one causal decoder block. Its input is <BOS> A{bit} D{i} B{bit} D{j} ?. The target is TRUE when the two bits differ. Training contains 128 combinations; testing withholds all sixteen examples in which both distractors are drawn from D4 and D5. The distractors are therefore familiar individually but unseen together.

After 800 deterministic full-batch updates, train and held-out accuracy are both 1.000. For the displayed pair, the clean TRUE − FALSE logit difference is 11.758 and the corrupt difference is −11.887. The gap is large enough for normalised patching. Results are then averaged over eight held-out clean-corrupt pairs, four distractor combinations under each setting of the second bit.

The first result is deliberately awkward for heatmap storytelling. At the final query, heads zero and one attend almost entirely to the first bit in both clean and corrupt runs. Across the worked pair, the maximum change in any attention weight is below one millionth. Patching the clean query, key, score row or attention pattern into the corrupt run produces approximately zero recovery. Patching the clean source value produces full recovery.

Counterexample: the perfect-looking head with a disconnected write

Now build a negative control that could make an attention explanation unnecessary. Give a model two parallel branches. One branch contains an attention head whose pattern always points perfectly to the causal source token. Set that head's output projection to zero. The other branch reads both bits through the residual state and solves XOR in its MLP. The attention heatmap is flawless, stable and entirely irrelevant to the answer.

This counterexample preserves the visual evidence while removing causal use. A probe on the head values may still decode the source bit because the input contains it. Only interventions downstream of the head reveal that its write is disconnected. Conversely, a head with diffuse-looking weights could be causally useful if a small weighted mixture supplies the exact feature another component needs. Visual concentration is neither necessary nor sufficient.

The synthetic trained specimen is not this hand-built control, because its source-value and head-output patches do transfer the answer. The control matters because it identifies the rival explanation the experiment must defeat: apparent selection may be causally inert. A patch programme earns its conclusion by ruling out the disconnected-head baseline, not by producing a more persuasive picture.

The downstream path adds a second lesson. Patching the head mixture, projected attention output or residual state after attention gives full recovery. Patching only the MLP output restores 0.765 on average. The remaining gap travels through the residual bypass. Replacing the full post-MLP residual restores one. Therefore “the MLP contains 76.5 per cent of the answer” would be careless. The ratio is recovery for one intervention geometry, one contrast family and one logit metric; nonlinear downstream normalisation prevents a simple additive decomposition.

Negative controls keep recovery honest

A successful transplant is persuasive only relative to alternatives. Patching the position vector from the clean source produces zero recovery because position is unchanged. Patching a value from the irrelevant distractor position produces approximately zero. Patching a downstream residual from another FALSE example also produces zero. These controls show that neither “any activation from another run” nor “any value vector” is sufficient.

A same-norm random displacement at the causal value site recovers 0.375 of the logit gap. That is not a failed control to hide. It shows that a large arbitrary movement can steer the metric partway, even though the matched clean value restores it completely. The appropriate conclusion is specificity by comparison, not purity by assertion. Stronger studies should repeat random directions, match covariance or local geometry, and test whether the proposed effect generalises across prompts and model seeds.

A successful patch is local evidence, not a map of truth

Interventions are stronger than observation, but their strength has a boundary: they establish what happened under a specified replacement, contrast, metric and downstream context.

The first temptation is to call a high-recovery site “the mechanism”. That wording quietly adds uniqueness, minimality and semantic stability. A whole-vector patch can restore behaviour even when only a subspace matters. A subspace patch can activate a dormant downstream pathway that the intact computation does not normally use. A component can look unimportant because another component repairs its loss. A donor activation can also be implausible in the destination context.

These are not philosophical loopholes. Research on transformer circuits has found distributed and backup behaviour; the Hydra effect documents compensatory changes after ablation. Work on subspace patching demonstrates that an intervention may create an interpretability illusion by driving a normally dormant path. Recent conditional co-ablation results argue that some backup components become visible only after the primary set is removed. The common lesson is that component importance is conditional on the rest of the system.

Donor plausibility and the manifold problem

A clean activation is usually more plausible than a zero vector because it came from the same model on a related prompt. “Came from the model” is not enough. The donor may encode a different token identity, syntactic role, sequence length or surrounding context. Inserted into the destination, it can create a combination the model never encounters naturally. Downstream change may then reflect generic sensitivity to an abnormal state rather than transfer of the proposed feature.

Matched counterfactual pairs reduce this risk by keeping context and position stable. So do interchange interventions that swap only examples sharing an abstract causal variable, rather than one convenient pair. Further controls can interpolate between clean and corrupt states, sample donors from the same label, match activation norm or covariance, and check whether the patched state remains near naturally observed activations. None certifies on-manifold status in a high-dimensional network, but agreement across them makes a damage-only explanation less economical.

The metric must also be causally aligned. Restoring one answer logit may worsen calibration, increase another error or exploit a shortcut specific to the contrast. Report the complete relevant logit set, behavioural accuracy on held-out pairs and side effects on control tasks. A mechanistic intervention should narrow uncertainty about a behaviour, not merely maximise one scalar.

Thought experiment: one bridge, then a spare bridge

Imagine a river crossed by one bridge. Closing it stops all traffic. Reopening it restores traffic. Noising and denoising agree, and the bridge appears necessary and sufficient for the observed journey. Now add a normally quiet spare bridge controlled by an automatic diversion system. Closing the main bridge reroutes traffic, so its single ablation has little effect. Closing the spare bridge in the intact world also has little effect. Yet closing both stops traffic.

Only one causal feature changed: redundancy. The single-bridge observation supports a simple component claim; the two-bridge observation requires interaction tests. Transformers can implement analogous redundancy through parallel heads, residual bypasses, later-layer compensation or distributed features. Therefore a low single-site effect can mean irrelevance, or it can mean successful repair.

Denoising and noising can disagree without contradiction

In the synthetic model, MLP-output denoising averages 0.765 with little variation, while noising averages 0.552 with a population standard deviation of 0.248. The clean MLP state is consistently useful for restoring the corrupt run, but the corrupt MLP state has prompt-dependent power to damage the clean run. Nonlinear normalisation, residual content and direction-specific interactions can all contribute.

The right response is not to select the prettier direction. Report both, retain raw scores, and explain the intervention graph. Denoising is often useful for tracing where a changed input becomes sufficient for recovery. Noising can test necessity, but may trigger backup behaviour or create abnormal states. Zero and mean ablations are easier to construct, yet they may be more off-distribution than a matched counterfactual donor. None of these operations supplies a universal causal truth.

The explicit failure boundary

Three dimensions determine how much weight a patch deserves. First is donor plausibility: does the replacement resemble a state the destination could naturally reach? Second is contrast breadth: does the effect replicate across paraphrases, entities, templates and seeds, or only one prompt pair? Third is interaction coverage: have parallel components, bypasses and conditional backups been tested?

A useful claim therefore has the form: “For this prompt distribution and metric, replacing activation slice j with a matched clean donor transfers the contrast, while stated controls do not.” That sentence is longer than “head 3 understands negation”, but it is testable. It also tells the next investigator exactly what to vary.

stateDiagram-v2
  [*] --> Claim
  state "Start with the state that persists" as Claim
  state "Routing and payload are<br/>separate transformations" as Test
  state "The MLP is another write, not<br/>a detachable explanation" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

The intervention ledger changes the engineering decision

Mechanistic work becomes operationally useful when its result is a bounded evidence record rather than an attractive internal visualisation.

Production-shaped worked scenario: an amendment-sensitive router

The team first observes that a linear probe can decode the amendment flag from middle-layer states. Several attention heads also concentrate on the amended clause. Neither result tells them whether the final recommendation uses that information. Retraining immediately could erase unrelated capability, while a confident story about one head could direct effort to the wrong component.

They construct matched document pairs. Each clean document contains an amended clause and should favour review. Its corrupt counterpart replaces only the amendment marker with an otherwise plausible unamended form. Names, dates, amounts, document length and surrounding clauses stay fixed. The metric is the manual_review − straight_through logit difference, and the denominator must exceed a predeclared threshold.

The intervention sweep begins coarse. It patches residual states by layer and token position to locate where the amendment contrast first becomes sufficient. It then decomposes that region into attention pattern, value, head output and MLP writes. Denoising and noising are both run. Irrelevant-clause donors, same-label donors, random directions matched by layer covariance, paraphrased clauses and multiple model seeds form the controls. If a suspected head disappears under noising because another head repairs the effect, the team tests their conditional interaction rather than declaring both unimportant.

The result changes one of three decisions. A stable, specific path may justify a targeted training or architectural experiment, followed by ordinary behavioural and operational evaluation. A broad residual effect with weak component specificity suggests that local surgery is unjustified; the safer response is data correction, model rollback or a deterministic review rule. If random and irrelevant controls rival the clean patch, the mechanistic hypothesis is rejected even if the original heatmap looked compelling.

The intervention does not authorise release. It narrows a causal hypothesis about the model component. Release still depends on end-to-end error rates, distribution shift, calibration, policy controls, auditability and recovery. The model's internal evidence belongs inside a wider assurance case, not above it.

The forward-pass intervention ledger

The decision instrument below forces every mechanistic claim to carry its experimental conditions. It can be stored as a JSON record, review worksheet or evaluation manifest. The important property is typed traceability from behaviour to contrast, intervention, control and decision.

When the ledger says stop

Some investigations should end before a circuit claim. Stop when the behavioural contrast is unstable, when tokenisation changes the supposedly fixed context, when clean and corrupt prompts differ along several uncontrolled features, or when the model's output gap is too small to support a reliable recovery ratio. Also stop when an API exposes no stable internal state: simulated “activations” from another model are not evidence about the inaccessible system.

A null result is useful. It may show that the proposed representation is readable but not used, that the effect is too distributed for the chosen resolution, or that the experiment lacks power. Record those alternatives rather than promoting the largest noisy cell. The ledger improves decisions partly by preventing an internal story from being manufactured where the intervention evidence is indecisive.

The executable artefact

The accompanying Python program implements the entire specimen with PyTorch 2.x. It contains no pretrained weights, network calls or private data. It creates the synthetic dataset, trains the 5,640-parameter model on CPU, caches every named activation, applies denoising and noising hooks, runs specificity controls, asserts the expected positive and negative cases, and writes a JSON evidence record.

Inspect the implementation in this paper and Review the recorded evidence in this paper. Run python instrumented_transformer.py. Under the stated seed and dependency, the headline output reports held-out accuracy 1.000, full source-value recovery, and approximately zero query and pattern recovery.

def tap(name, value, cache, hooks):
    cache[name] = value.detach().clone()
    patched = hooks[name](value) if name in hooks else value
    cache[f"{name}:post"] = patched.detach().clone()
    return patched

# Denoising intervention: clean source V into the corrupt run.
def replace_source_value(destination):
    result = destination.clone()
    result[:, :, SOURCE_POSITION] = clean_cache["blocks.0.v"][:, :, SOURCE_POSITION]
    return result

patched_logits = model(
    corrupt_ids,
    hooks={"blocks.0.v": replace_source_value},
)

The hook is intentionally simple. It records the tensor before intervention, replaces only the registered slice, and records what entered the downstream graph. In a larger model, the difficult work is not writing the assignment. It is maintaining shape, device, precision and batch alignment; choosing a plausible donor; preventing accidental patches to future or unrelated tokens; and tracking exactly which model, prompt set and metric produced the result.

Compact glossary

Activation

A tensor produced during the forward pass, such as a residual state, query, value, attention pattern or MLP output.

Activation patching

Replacing an activation in one run with a donor activation from another run, then measuring downstream change.

Denoising

Clean activation into a corrupted run, usually asking whether the clean state can restore a behavioural contrast.

Noising

Corrupt activation into a clean run, usually asking whether the corrupt state can transfer loss of the contrast.

Residual stream

The evolving per-token vector to which attention and MLP sub-blocks add updates.

Self-repair

A downstream or parallel response that compensates after a component is altered, masking its conditional importance.

Make the forward pass falsifiable

A transformer forward pass is no longer opaque once its transformations are named. Token and position vectors create the first residual state. Layer normalisation prepares reads. Queries and keys shape routing. Values supply payload. Attention and the MLP write updates. Residual connections preserve alternative paths. Final normalisation and unembedding turn the accumulated state into logits.

That anatomy is necessary and insufficient. A cache shows what occurred, while an intervention tests what downstream behaviour depends on under a specified replacement. In the measured specimen, the attention pattern remains visually fixed across opposite answers. Value patching restores the clean answer; pattern patching does not. The MLP carries a strong partial effect, and the residual bypass prevents a simple ownership story.

The changed decision is to treat internal visualisations as hypothesis generators and intervention ledgers as the minimum unit of causal evidence. Every claim should name the clean-corrupt contrast, metric, tensor geometry, donor, direction, controls, replication range and failure conditions. Heatmaps, probes and patch sweeps can then work together without being mistaken for one another.

The strongest result is not a colourful circuit. It is a claim narrow enough to fail: under these prompts, in this model, this replacement transfers this contrast, while these rivals do not. That discipline turns a forward pass from a diagram to an experiment, and an experiment into a better research or engineering decision.