Part I: What a probe actually proves

The first discipline is linguistic. “The model represents property z” can mean several different things. A conventional probe answers only one of them unless the experiment earns more.

Recoverability is relative to a reader

Take a fixed model and stop its forward pass at layer ℓ. The activation is a vector, tensor or collection of vectors. A probing experiment labels examples with a property such as grammatical number, object count, sentiment, policy status or the top item in a stack. It then trains a separate predictor on the frozen activations.

If the probe generalises, the most defensible statement is conditional: under this distribution, sample size, probe family and protocol, the property is recoverable from this activation. Change the reader and the result may change. A linear probe asks whether a simple hyperplane separates the labels. A nonlinear probe asks whether a more capable auxiliary model can extract them. Neither reader is automatically the computation used by the original model.

Early linear-probe work provided a lightweight way to compare layers without retraining the underlying network. Later work made the hidden cost visible. Hewitt and Liang showed why control tasks are needed to distinguish information in a representation from information memorised by an expressive probe. Minimum-description-length and information-theoretic approaches ask how many examples or bits a reader needs, rather than reporting only eventual accuracy.^{1, 2, 3, 4}

The spectator and the controller

Hidden states often preserve broad traces of the input because shared embeddings, residual pathways and skip connections carry information later computation may ignore. Ravichander and colleagues demonstrated the broader point in natural-language inference: a model could encode a property unnecessary for its trained task, with pretrained embeddings accounting for much of the recoverability.⁵

The correct reaction is not to discard probes. Use them to locate candidate layers, compare representational accessibility, find failures of invariance, and define hypotheses for intervention. A probe is a map of where an external reader can find a signal. It is not yet a wiring diagram.

What a defensible probe report contains

A probing result should expose its conditions. Name the target variable and labelling rule. State the layer, token positions and aggregation. Declare the probe family, parameter count, optimisation, sample size and split. Report a chance or majority baseline, but also a control task that tests whether the probe can memorise arbitrary labels. Show variation across seeds and split by document, speaker or source when examples share those dependencies.

Capacity is not only parameter count. A linear model trained on millions of near-duplicate points may find a stable separator unavailable in a small-data regime. Conversely, a property needing few examples may be more accessible than one reaching the same final accuracy after extensive supervision. Learning curves, description length and cross-domain transfer therefore add information hidden by one percentage.

The target needs a negative control too. If “policy status” can be predicted from template, year token or source system, the probe may read a proxy rather than the relation between effective time and supersession. Test matched examples that hold those cues fixed, then test a source where the cue-label correlation reverses. A good report says exactly which accessibility claim survived and which alternatives remain.

The readability and causal-use quadrants

The top-right quadrant is where many claims want to arrive, but probe accuracy gives only vertical position. The top-left is the central warning: the property is easy to recover but not on the answer path. The bottom-right matters equally. A representation can be causally indispensable yet hard for the chosen reader because the code is nonlinear, distributed, rotated or expressed only through interactions.

Decodability and causal use are neither equivalent nor ordered. A stronger reader may increase measured readability without changing the model. A better intervention may reveal use without making the property linearly accessible. Controlled studies in counting vision transformers and bracket-sequence transformers report examples of both dissociations, although these are task-specific results rather than universal laws.^{16, 17}

flowchart TD
  A["The warning lamp that was not<br/>wired to the brake"] --> B["Part I, What a probe actually<br/>proves"]
  B --> C["Part II, From readout to<br/>causal use"]
  C --> D["Part III, Interventions can<br/>also lie"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#5B5CF6,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Part II: From readout to causal use

To ask whether the model uses a property, the experiment must alter the relevant internal variable and observe the rest of the model. This is a different measurement system.

Define the counterfactual before touching the activation

Let the remainder of the model after layer ℓ be F_ℓ. Its output is y = F_ℓ(h_ℓ). A causal-use test proposes an intervention I that changes the hidden state's target property from z to z′. The key comparison is not simply whether the output changes. It is whether the output moves towards the answer expected under the named counterfactual.

This formulation exposes four questions. What exactly counts as changing z? Which other properties must remain fixed? Is the edited state plausible for the model? Does the output move in the predicted direction rather than merely becoming worse? Without these questions, a destructive ablation can masquerade as explanation.

A nonlinear wheel code

The example explains why “the probe failed, therefore the model lacks the concept” is as unsafe as the opposite inference. A simple reader may be chosen because simplicity is scientifically useful. The result should then be reported as linear accessibility, not absence.

Necessity, sufficiency, mediation and mechanism

Interventions answer different questions. Removing a component and observing failure supports a form of necessity under that removal. Injecting a component and obtaining the target behaviour supports sufficiency under that injection. Showing that a source variable changes an internal mediator which changes the output supports mediation. A mechanistic explanation asks for more: stable components, their transformations, and evidence that the simplified account predicts behaviour across relevant inputs and interventions.

Causal mediation analysis has been used to move beyond encoding claims by estimating the contribution of internal components to behaviour. Interchange interventions align a high-level causal variable with a low-level neural representation and test whether swapping that variable produces the high-level model's predicted counterfactual. Causal abstraction gives this aspiration a formal language.^{8, 9, 13}

A causal effect is stronger than a readout, but it is not automatically a mechanism. An ablation may reveal fragility rather than meaning. A successful patch may exploit a route the unmodified model never uses. A mediator may matter only for one prompt template or answer token. The claim must carry its intervention, distribution, layer, position, metric and controls.

Why the two objectives diverge

The probe is rewarded for covariance with a label across observed activations. The mediator is defined by sensitivity of the downstream function. These objectives can coincide, but no general principle forces them to. A residual stream may contain several correlated versions of a feature, while the next block reads only one. A probe may choose the statistically cleanest version even when the model reads a noisier, entangled one. Conversely, the model may use a direction whose natural variation is small, making it hard to identify from observational data.

The strongest competing possibility deserves a fair test: training may often organise task-relevant variables into simple, accessible directions, so linear probe directions could frequently overlap with causal mediators. That would make probes excellent heuristics. It would not turn the overlap into an identity. Measure alignment against matched random subspaces, predict intervention effects before running them, and test whether the relationship survives prompts, tasks and model versions.

Worked example: the best readout layer may not be the control site

Consider a twelve-layer sequence model trained to classify whether a bracket string is balanced. A probe for current stack depth rises from 61 per cent at layer two to 97 per cent at layer ten. It is tempting to declare layer ten the place where the model “does stack reasoning”. Yet the late layer may simply expose a summary produced earlier so that the output head can use a different variable, such as whether any unmatched opener remains.

Test three layers with matched bracket sequences. At layer six, swapping a compact top-of-stack representation changes long-distance closing-bracket predictions while leaving local token identity stable. At layer ten, moving along the depth probe direction changes the probe's own answer but not the model's next-token distribution. Both observations can coexist: depth is most readable late, while a related stack operation is causally active earlier.

The example changes layer selection. Do not intervene only where probe accuracy peaks. Sample layers across the rise, plateau and fall of readability. For each, compare probe direction, activation-patching effect and downstream path. A mediator can appear before a clean readout because later transformations reorganise a distributed computation into a convenient summary. It can also appear after the first readable trace because the model preserves information before consulting it.

This is why a layerwise probe curve should be treated as a candidate-location map. Peaks guide scarce intervention effort, but they do not name the mechanism. Reports that show only the maximum layer suppress the very temporal structure needed to distinguish early availability, active transformation and late readout.

Part III: Interventions can also lie

Moving from observation to intervention is necessary, but the verb “intervene” does not certify the edit. A bad intervention can manufacture the causal story it appears to test.

Four intervention families and their debts

Zero ablation replaces a component with zero. It is simple, but zero may be far outside the normal activation distribution. Mean replacement preserves a typical scale and asks whether example-specific variation matters, yet it can erase several entangled properties. Concept erasure projects away directions from which a probe reads a property. It tests sensitivity to a declared subspace, but the subspace may include correlated information or leave nonlinear traces.⁶ Counterfactual patching transplants an activation from a matched source example, often keeping the state closer to the model's own manifold.

Activation patching has helped localise causal dependencies, including factual recall. Its details matter. Different corruption methods, positions and output metrics can produce different conclusions, so the corruption and recovery metric belong inside the claim.^{10, 12}

Intervention design starts with a causal sketch

Before choosing an edit, draw the simplest high-level model that could generate the behaviour. For the policy example, candidate variables include clause identity, policy status, jurisdiction, retrieved evidence and disposition. Add arrows only where the hypothesis predicts influence. The intervention should correspond to one high-level change: switch policy status while preserving the other variables. Without that sketch, “patch layer 18” describes an operation but not a counterfactual.

Use increasingly demanding tests. Sweep edit magnitude and look for a smooth, signed effect rather than a one-off collapse. Patch from several matched sources. Compare the target subspace with random subspaces matched for dimension, norm and layer. Intervene upstream and downstream of the proposed mediator. A path hypothesis predicts where the effect should appear and where it should disappear. Restore the original activation and verify recovery.

Repeat with more than one intervention family when feasible. Agreement between a matched patch, validated erasure and interchange test is harder to explain as one method's artefact. Disagreement is useful: it exposes sensitivity to the edit definition. A causal sketch earns value by predicting a pattern of successes and failures, not one changed output.

Completeness and selectivity

A reliable causal probe has two minimum jobs. Completeness asks whether the intervention thoroughly changed the target property. If an independent reader still recovers the original value, the edit may be incomplete. Selectivity asks whether relevant non-target properties survived. If the edit also destroys syntax, identity, position or task state, a behavioural change cannot be cleanly attributed to the named concept.

Canby and colleagues formalise these desiderata and report a practical trade-off across causal probing methods: stronger target transformation can come with greater collateral change. Their study also shows why a linear edit should not be assumed clean merely because the original probe was linear.¹⁵

A serious negative control: the dormant route

Suppose an activation contains a subspace correlated with a feature, but that subspace is normally ignored by the output head. A patch injects a strong vector into it. Downstream nonlinearities activate a dormant parallel route, and the answer changes as though the feature had been edited. The intervention looks targeted and successful, yet it has not identified the route used during ordinary inference.

Makelov, Lange and Nanda construct this kind of interpretability illusion for subspace activation patching. The result does not make patching useless. It changes the burden. Compare with random subspaces matched for dimension and norm. Test whether natural variation along the proposed subspace predicts output sensitivity. Verify path-specific mediation under clean inputs, not only under large injections. Reduce edit magnitude and ask whether the effect is smooth, monotonic and reversible.¹¹

Where the mechanism stops helping

Probe-and-intervene methods become weak when the target variable is poorly specified; when equivalent information is distributed across redundant paths; when the edit leaves the activation manifold; when autoregressive feedback amplifies a tiny perturbation into an unrelated trajectory; or when collateral properties cannot be measured. They also struggle when a concept is contextual rather than a stable binary label.

If completeness and selectivity cannot be assessed independently, retreat to a perturbation claim. Report that a component is sensitive, an edit changes a logit, or a probe reads a correlation. Do not name a causal concept. This boundary matters especially for large language models, where plausible semantic labels can outrun intervention precision.

What a null intervention result means

A clean null result is not proof that the model never uses the property. The chosen edit may miss the true code, target the wrong position, act after the relevant computation, or be compensated by redundant pathways. Concept-erasure studies make the same point from another direction: removing what one classifier reads may leave alternative or nonlinear information available. Unit ablations can likewise show weak effects even when a distributed population matters.^{7, 14}

The null still constrains the claim. Suppose the probe reads policy status at 98 per cent, an independently validated edit flips that status, non-target probes remain stable, and the model's answer does not move across hundreds of matched cases. Under those conditions, the specific edited subspace is unlikely to mediate the tested answer. The correct conclusion is local: this readable code is not required along this tested route and distribution. It does not erase every possible representation of policy status.

Nulls become stronger when several interventions converge. If matched patching, low-magnitude steering and selective erasure all fail at the same layer while positive controls change the output, the passenger interpretation gains weight. If an earlier path-specific intervention succeeds, the study can replace a vague absence claim with a temporal one: the property is consulted earlier, then copied or reconstructed into a late readable state.

Publishing these negative results matters. A field that reports only successful patches will overestimate alignment between convenient readouts and true mediators. Failed interventions are part of the mechanism map when their target validity, positive controls and statistical power are visible.

Part IV: The probe and intervention lab

The fastest way to feel the distinction is to build four tiny mechanisms whose probe readability and causal use vary independently.

Worked example 1: four matched micro-models

The lab below uses balanced synthetic binary data. A brute-force linear reader searches 720 directions and every midpoint threshold for the best separation. A matched intervention changes the hidden state as though the target had flipped, then checks whether the mechanism's output follows the counterfactual. All computation runs in the browser.

The mechanisms differ in one causal feature at a time. In readable and used, the first coordinate stores the target and drives the output. In readable and unused, the same coordinate stores the target, but a parallel channel supplies the answer. In nonlinear and used, alternating target labels occupy sixty-four points around a circle; a nonlinear parity readout is exact while the best linear separator is near chance. In opaque and unused, every hidden point appears with both labels and the answer again comes from a separate channel.

The positive case is the upper-right mechanism: the target is linearly readable and the matched edit flips every answer. The serious negative case is the upper-left mechanism: the target is equally readable, yet editing the probed coordinate changes no answer. The same 100 per cent probe score accompanies two different mechanisms. Only intervention separates them.

The lower-right case blocks the opposite mistake. A near-chance linear readout does not show absence. The wheel's target is carried in an alternating nonlinear code, and a one-step rotation flips the model's answer on every example. A reader matched to the computation would recover it. The declared linear reader does not.

readableUsed.hidden = (z,k) => [z, 0.2*cos(2πk/32)]
readableUsed.output = h => sign(h[0])

readableUnused.hidden = readableUsed.hidden
readableUnused.output = (h,z) => z  // separate channel

nonlinearUsed.hidden = (z,k) => pointOnCircle(2k + parity(z))
nonlinearUsed.output = h => sign(cos(32*angle(h)))

opaqueUnused.hidden = (z,k) => pointOnCircle(k) // same states for both labels
opaqueUnused.output = (h,z) => z

Worked example 2: a production-shaped monitor

Return to the synthetic document-review system. Its output affects a human review queue, so the team wants a monitor for “the model used the currently effective policy”. A layerwise probe is attractive because it appears to inspect internal recognition rather than merely compare the final answer with a deterministic rule.

The first design decision is to separate purposes. As a diagnostic monitor, the probe can flag distribution shifts, compare model versions and select cases for deeper tests. As an authoritative control, it would block, release or escalate work. The second purpose has a much higher evidence burden.

Build matched cases in which policy wording stays constant while effective date, supersession relation and jurisdiction vary independently. Train on one partition and test on unseen policy families. Add control labels that preserve token frequencies but randomise status. Measure sample efficiency or description length as well as final accuracy. Then create counterfactual pairs and intervene at the candidate layer.

The edit must receive four receipts. An independent reader should verify that current versus superseded status changed. Probes for clause identity, jurisdiction, dates and document position should remain stable. The edited activation should resemble activations produced by natural counterfactual documents. Finally, the answer and cited evidence should move towards the counterfactual result, reverse under the opposite edit, and replicate after a model update.

Until those receipts agree, the probe must not become the authority kernel. A deterministic policy service should still resolve which rule is effective; the model can propose an interpretation and the probe can supply operating evidence. Access to an internal activation does not grant authority to approve a decision.

Even as a diagnostic, the probe needs release discipline. Store the model version, activation hook, probe weights, label definition, calibration set and threshold together. A model upgrade can rotate or redistribute a representation while preserving task performance, making an old probe silently unreliable. Re-run readability controls and the intervention battery before carrying the monitor across versions.

In operation, compare probe drift with final-answer drift and known policy-service outcomes. A changing score may indicate representational reorganisation, input shift, label drift or loss of causal use. Those hypotheses require different responses. The monitor can detect that an internal signal changed and trigger investigation. It cannot, without renewed causal evidence, certify why the model answered as it did.

stateDiagram-v2
  [*] --> Claim
  state "Recoverability is relative to<br/>a reader" as Claim
  state "The spectator and the controller" as Test
  state "What a defensible probe report<br/>contains" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

The four-receipt probe-to-mechanism assay

The assay deliberately ends with a bounded claim. Mechanistic interpretability seeks faithful, intelligible simplifications of low-level computation, and causal abstraction supplies one rigorous framework for testing them. A probe-plus-intervention result may become one component in such an account. It does not identify every interacting feature, prove stability under shift, or justify an anthropomorphic statement about what the model “believes”.

An observation that would weaken this paper's thesis is clear. Across diverse tasks and architectures, probe directions would need to predict selective intervention effects, beat matched random baselines and remain aligned under prompts, training seeds and model updates. Such a result would show that readability is a reliable surrogate for use in a defined regime. Until then, alignment between readout and use must be measured.

Compact glossary

Probe

An auxiliary reader trained on frozen activations to predict a labelled property.

Readability

Recoverability under a specified reader, data regime and protocol.

Causal use

A bounded claim that changing an internal variable changes output along a predicted path.

Completeness

The degree to which an intervention transforms its target representation.

Selectivity

The degree to which relevant non-target information is preserved.

Causal abstraction

A tested alignment between a high-level causal model and lower-level neural transformations.