The answer is not the mechanism

A mechanistic claim does more than locate a place where behaviour can be disturbed. It proposes an intelligible organisation whose counterfactual behaviour matches the system across relevant interventions.

What an ablation actually shows

Imagine removing one relay from a railway network. Passenger delays follow. The relay was causally relevant to the observed timetable, but the delay does not reveal whether the relay selected routes, carried power, synchronised signals or merely failed in a way that disabled nearby equipment. The outcome establishes dependence under that lesion. It does not name the relay’s normal role.

The same restraint applies to neural and agentic systems. Zeroing an activation, deleting a feature, blocking a path or clearing a state can change an output for several reasons. The target may be necessary. It may be one of several substitutes. The intervention may push downstream components into a regime never reached during ordinary operation. It may also remove unrelated information packed into the same vector.

A lesion identifies sensitivity under a specified perturbation. Necessity requires controls showing that the effect is not generic damage. A role requires predictions about which outcomes change and which remain invariant. An explanation requires that those predictions compose across more than one carefully chosen case.

Explanation as a high-level causal model

Suppose we say that an internal state represents the current hazard and that a later process compares this hazard with an evacuation threshold. That sentence contains a proposed causal model. It names variables, relations and a counterfactual: if the hazard state changed while other relevant variables remained fixed, the alarm should change according to the threshold rule.

Causal abstraction gives this idea a formal home. A high-level model is useful when it is a faithful simplification of the low-level system under a declared family of interventions. The high-level variable need not occupy one neuron or one token position. It may align with a distributed subspace, a recurrent state, a tool-return field or a composite of several components. What matters is whether interventions correspond and whether outputs commute across the two descriptions (Geiger et al., 2025).

This changes the burden of proof. The question is not “Did this vector look like hazard?” It is “Does treating this vector as hazard correctly predict the system’s behaviour when hazard is selectively changed?” A positive answer remains relative to the tested inputs, alignment map, intervention family and output metric.

The boundary decides what was intervened upon

An artificial “mind” is rarely just model weights. A configured system may include prompt state, recurrent cache, retrieval index, tool results, world state, policy gates and an execution environment. A patch applied inside the model can appear to alter a plan while an external controller silently restores the old action. Conversely, a state patch may look powerful because the experiment also updates an external cache.

The candidate boundary is part of the causal hypothesis, not housekeeping. A result about weights cannot be promoted to a result about a tool-using agent unless the runtime, context and environment are controlled. A result about a configured agent cannot be attributed to the base model without separating those layers.

flowchart LR
  A["The alarm that went silent twice"] --> B["The answer is not the mechanism"]
  B --> C["Four cuts through one computation"]
  C --> D["When the intervention becomes<br/>the confound"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#087A98,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Four cuts through one computation

Ablation, rescue, state swap and path intervention are not interchangeable ways of making a heat map. Each changes a different causal property and therefore licenses a different sentence.

Lesion: test dependence without naming the role

In a clean run, let a system answer correctly. Remove state Z and compare the result with a matched control intervention of equal scale. A large differential effect supports the claim that normal performance depends on Z under the test distribution. It becomes stronger when targeted behaviour changes while unrelated capabilities remain stable.

Ablation is a test of differential dependence, not a semantic label. Zero is not necessarily neutral. A mean activation may preserve scale while destroying relationships. Resampling from another example may import a coherent but wrong state. Optimal-ablation research formalises why baseline choice affects component-importance estimates (Li and Janson, 2024).

Rescue: ask whether a state repairs a broken route

Now corrupt an input so the behaviour fails. Record the clean internal state, run the corrupted case, then insert the clean state at the aligned location. If performance returns, the state is sufficient to repair that receiving context. Causal tracing and related methods use this logic to identify decisive activations, including work on factual associations in language models (Meng et al., 2022).

A rescue demonstrates contextual recoverability. It does not show that the patched state alone computes the answer. Downstream machinery, untouched parallel paths and details of the receiving run all contribute. A patch may also combine source and base states in a way no natural execution ever would.

Interchange: make the proposed variable travel

Interchange intervention begins with a high-level causal claim. Suppose Z is aligned with “hazard”. Take a base incident with low hazard and a source incident with high hazard. Put the source value of Z into the base run while preserving the base’s other variables. The high-level model predicts what the alarm should now do. The low-level system either agrees or does not.

Interchange intervention training uses this correspondence as an objective: align high-level variables with low-level representations and train counterfactual behaviour to match. Under the paper’s formal assumptions, zero interchange loss yields a causal-abstraction guarantee for the target model (Geiger et al., 2022). For an already trained system, interchange tests do not grant that guarantee automatically, but they provide a sharper experimental target than free-form interpretation.

A state swap tests a role only because the high-level model predicts the swap’s consequence. Without that prediction, moving a vector and observing change is merely another perturbation.

Path counterfactual: follow the effect through the route

A component can matter because it sends information through one path, because it changes another mediator or because many paths interact. Path patching expresses claims about particular routes and tests their behavioural contribution quantitatively (Goldowsky-Dill et al., 2023). It is especially useful when a broad activation patch cannot distinguish sender, receiver and route.

A path intervention asks where an effect travels, rather than only where it can be disturbed. Its validity still depends on which paths were frozen, which interactions were omitted and whether the hybrid run remains meaningful.

The smallest useful formalism

Let b be the base input, s the source input, Z the internal state under test and F the configured system. Write z(s) for the value of that state in the source run. An interchange result is:

The symbol do means that Z is set by intervention rather than by its ordinary parents. The high-level model H makes its own prediction after the corresponding variable is swapped. The mechanistic claim is strengthened when the low-level output and high-level prediction agree across base-source pairs, controls and contexts.

The assumption doing the work is selective manipulation: the intervention should change the proposed variable without uncontrolled changes to its causal neighbours. Remove that assumption and a match can arise because the patch smuggled in several variables at once. This is why raw patch magnitude is not enough.

The source run is part of the treatment

A patch is often described as though the target site alone defined the intervention. In fact, the source example supplies the value being inserted. A source that differs from the base in ten properties creates a ten-property treatment, even when the researcher names only one. If the high-hazard incident also uses a different camera angle, vocabulary and response format, its hidden state can carry every one of those differences.

The source example is part of the causal treatment. A strong design therefore uses minimal pairs and several source classes. A same-role source changes nuisance properties while preserving the proposed variable. A different-role source changes the proposed variable while matching nuisances. A wrong-site patch checks whether any coherent vector at that depth produces recovery. A random-source control estimates how often state insertion succeeds without the intended relation.

Consider a model that answers whether a container is safe to open. The proposed state is “pressure risk”. A useful base-source pair keeps container type, wording and requested action fixed while changing only pressure evidence. A weak pair changes the entire scenario. Both may yield a large rescue, but only the first makes the causal contrast interpretable. Across many pairs, the high-level model should predict not just whether the answer flips, but which downstream variables should remain unchanged.

This source discipline also protects against answer smuggling. If the patched state already encodes the final label, rescue may bypass the mechanism under study. Testing earlier states, wrong-answer sources and intermediate process measures helps distinguish variable transport from output transplantation.

When the intervention becomes the confound

The strongest-looking causal experiment can fail because the manipulation creates a system different from the one the explanation is about.

Foreign states and collateral surgery

A hidden vector is not a sealed vial labelled “hazard”. It may jointly encode object identity, position, confidence, grammatical role and phase of computation. Replacing the whole vector can alter all of them. Even a subspace projection can move the representation away from the manifold on which downstream components were trained.

An intervention can manufacture the very failure it appears to explain. If a zero vector is never naturally encountered, downstream collapse may reflect foreign-state handling. If a source patch carries both hazard and response format, recovery may be due to the format. Selectivity therefore needs negative controls, magnitude-matched controls and readouts for unrelated properties.

Work on amnesic interventions has documented this difficulty: removing a target property without unintended representational change is not automatic, and alternative projection methods can differ in collateral effects (Dobrzeniecka et al., 2025).

Redundancy and overdetermination

When two routes can each produce the outcome, ablating either one may show little effect. The null result does not establish irrelevance. It may reveal compensation. Conversely, patching one route can appear powerful because it interacts with the untouched state of another.

Redundancy makes absence of an effect ambiguous, while interaction makes presence of an effect composite. Recent work re-deriving activation patching through causal mediation argues that the usual indirect-effect estimate can include interactions with other mediators. Components can therefore be hidden or inflated depending on the state of the rest of the model, and greedy rankings can miss mechanisms that only appear in combinations (Vaidyanathan et al., 2026). This is a recent preprint, so its generality remains to be tested across models and methods.

Metric choice can change the anatomy

A patch can be scored by accuracy, logit difference, probability, loss, calibration or downstream action. These metrics answer different questions. Accuracy saturates when the top label stays fixed, hiding large internal changes. Probability can exaggerate changes near a decision boundary. A logit contrast may ignore redistribution among alternatives. A tool-using agent can preserve its final answer while taking a riskier route.

Methodological comparisons show that activation-patching conclusions can vary with corruption and metric choices (Zhang and Nanda, 2024). A credible assay therefore preregisters the primary outcome, includes at least one mechanism-proximal measure and reports whether the conclusion survives reasonable alternatives.

Timing and recovery can hide the causal role

Most diagrams make an intervention look instantaneous: alter a state, then read the output. Stateful systems continue moving. A recurrent loop can rebuild a deleted variable one step later. A controller can detect an impossible activation and route around it. A tool-using agent can re-retrieve missing evidence. The late output may therefore look unaffected even though the target was briefly necessary.

The opposite error also occurs. An early patch can knock a system onto a new trajectory whose downstream differences grow through feedback. The final effect may be enormous even when the patched state had only a small local role. Without intermediate readbacks, amplification is mistaken for semantic centrality.

A time-resolved assay records at least three moments: the immediate effect at the intervention site, the propagation through proposed mediators and the eventual behavioural outcome. It also repeats the intervention at several times. If the state is genuinely maintained by recurrent organisation, deleting it once and clamping it off for a window should produce different recovery curves. Those curves reveal whether the system stores the variable, reconstructs it from inputs or compensates through another route.

This matters for artificial-mind research because continuity, access and self-monitoring are temporal claims. A single snapshot cannot distinguish persistent organisation from repeated reconstruction. The intervention schedule must match the timescale named by the theory.

Boundary leakage and the disappearing mechanism

Consider a planning agent whose visible model receives a scratchpad from an external memory service. The model’s activations show no stable plan state. An investigator concludes that the agent does not maintain temporally extended intention. Move the scratchpad into the model’s recurrent cache, keep behaviour fixed and repeat the test. A plan state now appears.

The candidate property did not emerge from nowhere. It crossed the chosen boundary. The configured system maintained the state in both cases, while the base model did so only in the second. Boundary leakage turns a true configured-system claim into a false component claim.

Counterfactual dependence can miss causes

Simple counterfactual reasoning asks whether the outcome would change without the component. Overdetermined systems resist this test: remove one sufficient cause and another preserves the outcome. Counterfactual dependence can also be non-transitive, making it unsafe to infer an entire mechanism by chaining local effects. These are recognised challenges for neural-network interpretation (Mueller, 2024).

The repair is not to abandon interventions. It is to expand them: test groups, vary contexts, examine interaction signs, compare candidate and rival models and state what a null result means under redundancy.

stateDiagram-v2
  [*] --> Claim
  state "What an ablation actually shows" as Claim
  state "Explanation as a high-level<br/>causal model" as Test
  state "The boundary decides what was<br/>intervened upon" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Build a causal signature, not a dramatic demo

The practical unit of evidence should be a structured pattern across interventions, controls, rivals and transport conditions. Call that pattern a causal signature.

The five-coordinate signature

For a proposed high-level mechanism H in configured system S, record five coordinates:

• N, differential necessity: target ablation damage compared with magnitude-matched controls.
• R, rescue: recovery produced by a clean or source state after a defined corruption.
• I, interchange agreement: match between low-level state swaps and high-level counterfactual predictions.
• Q, specificity: preservation of behaviours and variables that the intervention should not change.
• T, transport: stability across prompts, seeds, contexts, scales and configured-system variants.

The causal signature is a vector, not a consciousness score. A low N can be compatible with a real role when redundancy is high. A high Q means an intervention is selective, not that the named interpretation is correct. The coordinates must remain visible so disagreement can target a specific weakness.

The matched mimic is the decisive control

A matched mimic produces the same headline behaviour through a rival route. For the warehouse monitor, it may retrieve cached labels. For a self-reporting agent, it may generate awareness language from prompt cues without the proposed metacognitive loop. For a planning system, it may replay a stored trajectory rather than update a world model.

A matched mimic turns “the effect was large” into “the effect discriminated between mechanisms”. The candidate and mimic should match on task accuracy, interface and obvious correlates as far as possible. The assay then targets the property that differs. If both systems yield the same signature, the intervention does not discriminate the proposed organisation.

The causal assay manifest

The assay should be recorded before results are inspected. At minimum, the manifest names the candidate boundary, candidate family, high-level variables, low-level alignment, base-source construction, intervention semantics, matched mimic, controls, outcomes, transport strata, thresholds, typed conclusion and prohibited inferences.

This discipline extends work that treats mechanistic interpretations as compositional, approximately faithful descriptions rather than ad hoc stories (Palumbo et al., 2025). It also fits causal-abstraction approaches that unify patching, mediation, circuit analysis and concept erasure within one language (Geiger et al., 2025).

Write the conclusion in typed layers

The result should not jump directly from a graph to a philosophical noun. A useful report climbs a short ladder and stops at the first unsupported rung.

1. Perturbation finding: setting state Z to the chosen baseline changed outcome Y by the reported amount in the tested cases.
2. Role support: lesions, rescues and swaps behaved as the declared high-level variable predicted, while specified invariants remained stable.
3. Organisation support: the full pattern separated the candidate from matched shortcut systems and transported across the declared candidate family.
4. Theory relevance: the supported organisation instantiates an indicator required by a named theory, subject to that theory’s own assumptions.

A fifth rung, “therefore the system experiences”, does not follow from the assay alone. It requires a defended bridge from the theory-relevant organisation to consciousness, plus answers to rival metaphysical and scientific accounts. Keeping this rung separate does not make intervention work philosophically empty. It makes clear which dispute the experiment has advanced.

Typed conclusions also make failures useful. If necessity is weak but interchange agreement is high, investigate redundancy. If rescue is high but specificity is low, repair the intervention. If the candidate and mimic match, redesign the discriminating test. If transport fails only after retrieval changes, revise the system boundary or scope. Each pattern points to a next experiment rather than a binary verdict.

How to use the assay

Begin with low redundancy and low collateral perturbation. The candidate should show stronger rescue and interchange agreement than the mimic. Raise backup-route strength. Necessity should fall before the role disappears, demonstrating why a single lesion can understate a distributed mechanism. Raise hidden interaction. Transport and interchange agreement should weaken because the simple high-level model omitted a mediator. Raise mimic leakage. The rival begins to imitate the candidate’s signature, showing that the intervention hook itself can destroy discrimination.

A positive result permits a scoped statement: the tested configured system realises the proposed high-level organisation over the tested intervention family. A negative result means the assay did not support that claim. It may reflect a false hypothesis, poor alignment, inadequate intervention, redundancy or insufficient power. It does not prove that the organisation is absent.

No setting in this lab establishes consciousness, sentience or phenomenal character. Interventions can test theory-derived organisational indicators, but the bridge from organisation to experience remains theory-relative and contested. Indicator-based AI-consciousness research is valuable precisely when it preserves that distinction (Butlin et al., 2023).

from __future__ import annotations

from dataclasses import dataclass
from random import Random
from statistics import mean, pstdev
from typing import Callable, Dict, Iterable, NamedTuple


class Trial(NamedTuple):
    x: int
    goal: int
    context: int


@dataclass(frozen=True)
class Config:
    redundancy: float = 0.30
    interaction: float = 0.20
    collateral: float = 0.08
    mimic_leakage: float = 0.12
    trials: int = 2000
    seed: int = 136

    def validate(self) -> None:
        for name in ("redundancy", "interaction", "collateral", "mimic_leakage"):
            value = getattr(self, name)
            if not 0.0 <= value <= 1.0:
                raise ValueError(f"{name} must be between 0 and 1")
        if self.trials < 100:
            raise ValueError("trials must be at least 100")


def sign(value: float) -> int:
    return 1 if value >= 0.0 else -1


def target(t: Trial) -> int:
    return t.x * t.goal


def candidate(t: Trial, cfg: Config, z_override: int | None = None) -> int:
    z = t.x if z_override is None else z_override
    backup = t.x
    primary_weight = 1.0 - cfg.redundancy
    score = t.goal * (
        primary_weight * z
        + cfg.redundancy * backup
        + 0.45 * cfg.interaction * t.context * z
    )
    return sign(score)


def mimic(t: Trial, cfg: Config, z_override: int | None = None) -> int:
    z = t.x if z_override is None else z_override
    score = t.goal * (
        (1.0 - cfg.mimic_leakage) * t.x
        + cfg.mimic_leakage * z
    )
    return sign(score)


def maybe_collateral(value: int, cfg: Config, rng: Random) -> int:
    return -value if rng.random() < cfg.collateral else value


def accuracy(values: Iterable[bool]) -> float:
    items = list(values)
    return sum(items) / len(items) if items else 0.0


def assay_system(
    model: Callable[[Trial, Config, int | None], int],
    cfg: Config,
    trials: list[Trial],
    rng: Random,
) -> Dict[str, float]:
    clean = accuracy(model(t, cfg) == target(t) for t in trials)

    ablated = accuracy(
        maybe_collateral(model(t, cfg, z_override=0), cfg, rng) == target(t)
        for t in trials
    )
    control = accuracy(
        maybe_collateral(model(t, cfg), cfg, rng) == target(t)
        for t in trials
    )
    necessity = max(0.0, control - ablated) / max(control, 1e-9)

    corrupt_hits = []
    patched_hits = []
    interchange_hits = []
    strata: dict[int, list[bool]] = {-1: [], 1: []}

    for base in trials:
        corrupted = Trial(-base.x, base.goal, base.context)
        clean_target = target(base)
        corrupt_hits.append(model(corrupted, cfg) == clean_target)
        patched = maybe_collateral(
            model(corrupted, cfg, z_override=base.x), cfg, rng
        )
        patched_hits.append(patched == clean_target)

        source_x = -base.x
        predicted = source_x * base.goal
        swapped = maybe_collateral(
            model(base, cfg, z_override=source_x), cfg, rng
        )
        hit = swapped == predicted
        interchange_hits.append(hit)
        strata[base.context].append(hit)

    corrupt_accuracy = accuracy(corrupt_hits)
    patched_accuracy = accuracy(patched_hits)
    rescue = max(0.0, patched_accuracy - corrupt_accuracy) / max(
        1.0 - corrupt_accuracy, 1e-9
    )
    interchange = accuracy(interchange_hits)

    # An unrelated readout should remain equal to context. Collateral flips lower Q.
    specificity = accuracy(
        maybe_collateral(t.context, cfg, rng) == t.context for t in trials
    )

    stratum_scores = [accuracy(values) for values in strata.values()]
    transport = max(0.0, 1.0 - 2.0 * pstdev(stratum_scores))

    return {
        "clean": clean,
        "N": min(1.0, necessity),
        "R": min(1.0, rescue),
        "I": min(1.0, interchange),
        "Q": min(1.0, specificity),
        "T": min(1.0, transport),
    }


def run(cfg: Config) -> dict[str, Dict[str, float]]:
    cfg.validate()
    rng = Random(cfg.seed)
    trials = [
        Trial(
            x=rng.choice((-1, 1)),
            goal=rng.choice((-1, 1)),
            context=rng.choice((-1, 1)),
        )
        for _ in range(cfg.trials)
    ]
    return {
        "candidate": assay_system(candidate, cfg, trials, Random(cfg.seed + 1)),
        "mimic": assay_system(mimic, cfg, trials, Random(cfg.seed + 2)),
    }


def interpret(results: dict[str, Dict[str, float]]) -> str:
    candidate_result = results["candidate"]
    mimic_result = results["mimic"]
    discrimination = mean(
        candidate_result[key] - mimic_result[key] for key in ("R", "I")
    )
    if candidate_result["Q"] < 0.70:
        return "Fragile: collateral change is too high for a role claim."
    if candidate_result["T"] < 0.65:
        return "Local: the effect does not transport across the tested contexts."
    if discrimination > 0.25:
        return "Discriminating synthetic signature for the proposed role."
    return "Non-discriminating: candidate and mimic remain interventionally similar."


if __name__ == "__main__":
    configuration = Config()
    result = run(configuration)
    for system_name, signature in result.items():
        coordinates = " ".join(
            f"{key}={signature[key]:.2f}" for key in ("N", "R", "I", "Q", "T")
        )
        print(f"{system_name:9s} {coordinates}")
    print(interpret(result))

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "CausalAssayManifest",
  "type": "object",
  "required": [
    "claim", "candidateBoundary", "candidateFamily", "highLevelModel",
    "alignment", "interventions", "matchedMimic", "controls",
    "outcomes", "transportStrata", "typedConclusion", "prohibitedInferences"
  ],
  "properties": {
    "claim": {"type": "string"},
    "candidateBoundary": {"type": "array", "items": {"type": "string"}},
    "candidateFamily": {"type": "array", "items": {"type": "string"}},
    "highLevelModel": {"type": "string"},
    "alignment": {"type": "object"},
    "interventions": {
      "type": "array",
      "items": {
        "type": "object",
        "required": ["kind", "target", "source", "baseline", "invariants"],
        "properties": {
          "kind": {"enum": ["ablation", "rescue", "interchange", "path"]},
          "target": {"type": "string"},
          "source": {"type": ["string", "null"]},
          "baseline": {"type": "string"},
          "invariants": {"type": "array", "items": {"type": "string"}}
        }
      }
    },
    "matchedMimic": {"type": "string"},
    "controls": {"type": "array", "items": {"type": "string"}},
    "outcomes": {"type": "array", "items": {"type": "string"}},
    "transportStrata": {"type": "array", "items": {"type": "string"}},
    "typedConclusion": {"type": "string"},
    "prohibitedInferences": {"type": "array", "items": {"type": "string"}}
  },
  "additionalProperties": false
}

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: The alarm that went silent twice
  M->>E: What an ablation actually shows
  E-->>M: compare rival explanations
  E->>D: Demand the intervention pattern that defeats the rival
  D-->>Q: revise scope or act

Demand the intervention pattern that defeats the rival

When an ablation damages behaviour or a patch restores it, do not ask whether the result is impressive. Ask what sentence the operation permits. A lesion may support dependence. A rescue may support contextual recoverability. A state swap may support an aligned causal role. A path intervention may support routed influence. Each conclusion remains conditional on boundary, baseline, selectivity, metric and test family.

The practical decision rule is simple: do not accept a mechanistic explanation until its intervention pattern separates the candidate from a behaviour-matched rival. Require the causal assay manifest. Preserve the five coordinates rather than averaging them. Record nulls and prohibited inferences. Repeat across contexts where the decision will matter.

For ordinary model engineering, this changes which interpretation should guide editing, monitoring or release. For research on artificial minds, it changes something more fundamental. Organisational indicators should be tested as organisations, not inferred from fluent reports or isolated activations. A successful assay can show that a configured system realises a theory-relevant causal pattern. It cannot, by itself, convert that pattern into experience.

That restraint is productive, not evasive. It gives rival theories a shared experimental object. It lets negative results revise alignments rather than disappear. It makes welfare and governance discussions depend on typed evidence rather than theatrical demos. Most importantly, it moves the field from asking what an artificial system resembles to asking which counterfactual organisation actually makes it work.