Geometry is a view of the state, not yet a story of the computation

A deep network converts one representation into another. At layer l, each input becomes a point in a high-dimensional space. Geometry is the collection of relations we choose to inspect among those points.

Let hl(x) ∈ ℝd be the activation vector produced for input x at layer l. For n inputs, stack those vectors into a matrix Hl ∈ ℝn×d. A cluster asks which rows are near one another. A direction asks whether moving along a vector changes a labelled property. A subspace asks whether several directions together carry a family of variation. A manifold description asks whether examples occupy a lower-dimensional, curved region inside the ambient space.

These are not interchangeable observations. Euclidean distance, cosine similarity, neighbourhood rank, linear separability, intrinsic dimension and class-manifold radius each inspect a different relation. A two-dimensional projection adds another choice: it compresses the hidden state according to the objective of PCA, t-SNE, UMAP or another method. A clean picture can therefore be a faithful answer to a narrow geometric question and a poor answer to the causal question the reader silently substitutes for it.

Visibility is the weakest rung. A pattern visible after projection says that the projection exposed it. It does not say the pattern was easy to read in the original space, stable across samples, used by downstream layers, or responsible for the output.

A probe measures accessibility under a chosen reader

A probe freezes the network, collects hidden states and trains a separate model to predict a property. A linear probe uses a hyperplane. A nonlinear probe may use a small multilayer network. The result is conditional on the probe family, training set, regularisation and evaluation split.

The classic linear-probe experiment by Alain and Bengio found that class information became increasingly linearly separable through the layers of the image classifiers they studied. That is useful diagnostic evidence: the network progressively arranged its states so a simple reader could classify them.

Yet a probe creates a second learning system. Hewitt and Liang introduced control tasks to distinguish information supplied by the representation from mappings learned by a capable probe. Ravichander, Belinkov and Hovy then showed, in their natural-language-inference setting and controlled synthetic tasks, that properties could be encoded even when the trained task did not require them. High probing accuracy did not entail task relevance.

Minimal worked example: the archive beside the calculator

Suppose a hidden state is exactly h = (c, n). The first coordinate c is a calibrated defect signal. The second coordinate n is camera identity. The deployed output is σ(6c), where σ is the logistic function. A probe can recover n perfectly. Camera identity forms a clean axis. But the output weight is (6, 0), so changing n while holding c fixed cannot alter the answer.

This is not a paradox. The hidden state is a shared workspace. Upstream computation may preserve information for reconstruction, auxiliary objectives, residual flow, later tokens, or no useful reason at all. The downstream head reads only some directions. Decodable means available to a new reader; used means influential in the existing reader.

Coordinates can move while the network’s function stays fixed

Imagine rotating a transparent sheet carrying all hidden-state points. The clusters rotate, every named axis changes, and the pairwise distances stay the same. If the next layer rotates its weights back, the network computes exactly the same function. A more general invertible transformation can stretch and shear the space while a compensating downstream map preserves behaviour.

This freedom creates an invariance requirement for interpretation. An explanation should not depend on a coordinate choice that the model’s function does not distinguish. Individual neurons can still matter, especially where nonlinearities, sparsity or architectural routing privilege them. But the burden is empirical. One cannot infer it from the convenience of the basis supplied by the implementation.

Thought experiment: rotate the explanation while holding the machine fixed

Give two audit teams copies of one trained network. Team One receives the ordinary checkpoint. It finds a unit whose activation tracks surface roughness and reports a “roughness neuron”. Before Team Two receives its copy, apply an orthogonal rotation to the whole hidden interface and the compensating inverse rotation to the next linear map. Inputs, logits, losses and every ordinary counterfactual output remain identical to numerical precision. No single unit in Team Two’s checkpoint now tracks roughness strongly; a distributed direction does.

Only one causal feature changed: the coordinate basis used to write the same intermediate computation. The network’s external function did not change. If the first report says the model depends on unit 317, while the second says it depends on a twelve-unit mixture, both reports cannot be fundamental descriptions of the same function. The basis-invariant statement is narrower: roughness information occupies a subspace that the downstream computation reads under a declared intervention. The named neuron is an implementation coordinate, unless architecture or intervention evidence gives that coordinate a special role.

Now change a second fact. Insert a rectified nonlinearity between the rotated state and the compensating map. An arbitrary rotation can alter which values are clipped at zero, so the two networks may no longer be functionally equivalent. The symmetry has been broken by the architecture. This does not rescue every neuron story. It tells the investigator what must be tested: whether the proposed unit remains selectively necessary across naturally occurring inputs, whether its effect survives basis-compatible controls, and whether another direction can substitute for it.

A geometric explanation should live at the coarsest level that survives the model’s genuine symmetries. Sometimes that level is a direction, sometimes a subspace, sometimes a local manifold chart, and occasionally a privileged unit or routed component. The right unit is discovered by invariance and intervention together, not selected because the plotting library labels columns.

Similarity measures answer comparison questions

Representation-similarity methods make coordinate choices explicit. SVCCA, canonical-correlation variants and centred kernel alignment compare activation matrices while ignoring selected transformations. Kornblith and colleagues introduced CKA as a similarity index that avoids a high-dimensional failure of fully invertible-linear-invariant measures and can identify correspondences between networks trained from different initialisations.

That does not turn CKA into a measure of shared mechanism. It asks whether two sets of sample relations align under its invariances. Nanda and colleagues showed why ordinary representation similarity does not by itself measure shared invariance to input perturbations, then repurposed similarity inside a directional intervention-based comparison. The lesson generalises: a metric is meaningful only after its invariances match the claim.

flowchart TD
  A["The perfect cluster that did<br/>nothing"] --> B["Geometry is a view of the<br/>state, not yet a story of the<br/>computation"]
  B --> C["A probe measures accessibility<br/>under a chosen reader"]
  C --> D["Coordinates can move while the<br/>network’s function stays fixed"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#B96D00,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Geometry can matter without every visible feature becoming a cause

The caution against causal overreach should not be confused with a claim that geometry is cosmetic. Geometry can lower the complexity of a downstream decision. If class manifolds shrink, separate and reduce their correlations, a linear readout can classify more objects. Cohen and colleagues measured such changes through hierarchies of several deep vision networks and connected them to classification capacity.

Late in training, class features and classifier weights can also approach a highly regular arrangement known as neural collapse. Papyan, Han and Donoho documented this terminal-phase pattern across image-classification settings. These results show that objectives and architectures can create recurring geometry with functional relationships to readout.

The boundary remains important. A geometry-to-capacity result concerns a declared readout family and task. It does not license a claim that any attractive two-dimensional cluster is a causal circuit, that a single direction is the unique representation of a concept, or that a model reasons through the labels humans attach to its plots.

Thought experiment: swap visibility while holding causal influence fixed

Consider two networks with the same output accuracy. In Network A, colour forms a crisp cluster at the last hidden layer, but the output head projects it away. Shape is represented across several overlapping dimensions and drives the decision. In Network B, colour is only weakly linearly decodable, yet a narrow colour-sensitive subspace gates which downstream expert runs. The visible feature is inert in A and causal in B.

Now vary one fact at a time. Improve the colour probe in A without changing the head. Decodability rises while causal influence stays zero. Strengthen the gate in B without making colour easier to decode globally. Causal influence rises while probe accuracy may barely move. The two quantities can covary in ordinary training, but neither logically determines the other.

A 2025 preprint by Huang and Chang reports this kind of divergence in counting vision transformers: some middle-layer object tokens were causally influential under activation patching despite weak count decodability, while some later tokens were highly decodable yet functionally inert. The result is recent and should not be treated as a universal law, but it is a direct warning against substituting one measurement for the other.

Superposition makes the candidate unit uncertain

A second complication is that a feature need not align with one neuron or one orthogonal axis. In toy models of superposition, networks represent more sparse features than they have dimensions by placing features in overlapping directions. Sparse autoencoders attempt to recover a more interpretable dictionary, but evaluation remains difficult. Karvonen and colleagues explicitly frame the absence of a ground-truth feature set as an evaluation problem.

Consequently, a failed single-direction intervention may miss a distributed subspace, while a successful broad ablation may remove several unrelated computations. The causal unit must be discovered and tested, not assumed to be a neuron, probe vector or sparse-autoencoder feature.

Intervention is the missing test, provided the intervention is valid

An intervention asks what the frozen network would do if a candidate internal variable were different. The simplest version removes a subspace. Let PS project onto candidate subspace S, and let μ be a reference centre. An erasure intervention is:

Removal is not the only option. Activation patching replaces a state from one input with the corresponding state from another. Interchange interventions swap aligned internal variables while a high-level causal model predicts the counterfactual output. Geiger and colleagues use such swaps in interchange intervention training, and later work on distributed alignment search learns rotations that align high-level variables with distributed neural representations.

Steering provides a complementary positive intervention. Instead of erasing a direction, add it and predict a signed behavioural change. Inference-Time Intervention, for example, shifts selected attention-head activations along learned directions and measures benchmark behaviour. Such results demonstrate controllability under a specified intervention. They do not by themselves prove that the edited direction is a complete natural variable used in ordinary unedited computation.

Necessity, sufficiency and mediation are different tests

One intervention score cannot answer every causal question. Erasure asks a version of necessity: does performance fall when this candidate is removed while the rest of the recorded state is retained? Addition asks a version of sufficiency: can placing the candidate into an otherwise suitable state induce the predicted behaviour? Patching asks about mediation: when two inputs differ in a high-level property, does transplanting this internal state transmit the corresponding part of the output difference?

The distinctions matter whenever computation is redundant or conjunctive. Suppose two parallel pathways each carry enough defect evidence to trigger inspection. Removing either alone has little effect, so neither appears necessary, yet removing both changes the decision. Conversely, an added “defect direction” may be sufficient to raise a logit because the downstream head responds to it, even though ordinary images reach the same decision through another pathway. A successful steering vector proves an available control surface more readily than it proves the natural route of computation.

Conjunction creates the opposite pattern. Imagine an XOR-like rule in which the decision depends on the relation between two latent bits. Neither bit has a monotonic marginal effect across a balanced dataset, and a one-dimensional probe may look weak. Swapping one bit conditionally on the other nevertheless flips the answer exactly as the rule predicts. Here a global average effect hides a sharp conditional mechanism. Layer, input family, baseline state and intervention strength are therefore part of the causal claim, not details to be discarded after the headline result.

A useful report presents an intervention profile rather than one dramatic number: removal effect, addition effect, paired patching effect, dose response, specificity, rescue and matched-control percentile. Agreement across these tests supports a bounded mechanism. Disagreement is not an inconvenience to average away. It identifies whether the candidate is redundant, merely controllable, conditionally used or entangled with a wider computation.

Six controls that make an intervention informative

First, use a held-out probe and a probe-capacity control. If a flexible probe also learns shuffled or arbitrary labels, raw accuracy exaggerates what the representation supplies. Report the probe class, regularisation, sample size and uncertainty.

Second, compare matched random directions or subspaces. Removing any activation component can hurt a network. Match dimensionality, activation variance and intervention norm, then ask whether the candidate effect is exceptional relative to that distribution.

Third, sweep intervention strength. A semantic mechanism should usually show a directional, interpretable dose response over some range. A discontinuous collapse at a large dose is more consistent with generic off-manifold damage.

Fourth, test specificity. The intervention should alter the predicted target more than unrelated behaviours. If removing a “camera” direction changes every class probability, calibration and reconstruction metric, the edit may be destructive rather than explanatory.

Fifth, rescue. Restore the removed component, patch the original state back, or add an opposite signed direction. Recovery is stronger evidence than damage alone because it links the effect to the controlled variable rather than irreversible corruption.

Sixth, repeat across the system boundary that matters. A direction found on one dataset, prompt template or checkpoint may not be stable. Mechanism claims should survive new inputs, seeds and relevant shifts, or state their narrower boundary.

Where the causal interpretation breaks

The strongest objection to intervention-based interpretation is that hidden states are not ordinary treatment variables. They are jointly produced by earlier layers and usually occupy a thin region of the ambient space. Editing one coordinate can create a state the network would never naturally produce. Downstream change then measures response to an alien activation, not the role of a familiar feature.

Erasure also conflates information with implementation. If the target feature is entangled with another required feature, removing the probe subspace damages both. Conversely, no observed effect does not prove irrelevance. The model may represent the same information redundantly, reconstruct it downstream, route around the edited component, or use a nonlinear feature that a linear erasure misses.

What a null intervention does, and does not, rule out

An inert edit has at least three explanations. The candidate information may be present but genuinely unused. The intervention may target the wrong realisation, such as one linear direction when the model uses a nonlinear or input-dependent code. Or the edit may remove a working signal that another pathway reconstructs before the measured output. These explanations make the same immediate prediction, little output change, but imply different architectures and different release decisions.

Separate them by changing the test rather than repeating the same ablation. Search for alternative probes and nonlinear readers, intervene at several layers, remove candidate subspaces jointly, and measure whether downstream layers reconstruct the property. Patch naturally paired states when possible. A successful joint intervention after two inert single edits supports redundancy. Rapid downstream recovery supports reconstruction. Persistent inertness across justified candidate units strengthens the archived-information account, although it never proves that every possible decoder is irrelevant.

The negative control should also preserve ordinary damage while removing semantic alignment. Draw random subspaces matched for dimension, activation variance and intervention norm; build probe directions from shuffled labels; or patch states between inputs that share the claimed property but differ in incidental details. If these controls move the output as much as the candidate edit, the observed effect belongs to intervention geometry rather than the named feature. A null result becomes informative only after the test demonstrates that it could have detected the proposed use.

Even a clean effect may be non-unique. Several rotated subspaces can support equivalent readouts. A direction can combine multiple human concepts. A high-level variable can be realised by different low-level states on different inputs. Causal evidence should therefore be expressed as a bounded mapping: in this model, layer, input family and intervention class, changing this subspace predicts these output changes.

A probe-and-intervention lab that separates presence from use

The executable artefact creates a six-dimensional synthetic world. One latent bit is causal for the target. A second bit is a nuisance. Four Gaussian dimensions add irrelevant variation. A fixed two-layer nonlinear encoder preserves both bits; only the logistic output head is trained. This isolates the question of what the existing head reads from what a new probe can recover.

The lab runs two regimes. In the independent regime, nuisance and target are independent during head training. In the shortcut regime, they agree in 98 per cent of head-training examples, while evaluation restores independence. Probes are trained on a separate balanced calibration set in both regimes. The same architecture, layer dimensions and intervention code are used throughout.

For each layer, the script reports output accuracy, causal and nuisance probe accuracy, a shuffled-label control, the mean absolute probability change after removing each probe direction, prediction-flip rate, and the distribution of effects from 256 random directions. It also applies an orthogonal rotation to the last hidden layer, rotates the output weights back and asserts that logits remain unchanged.

The independent regime is the positive demonstration of the thesis. The final layer exposes nuisance identity at 99.6 per cent probe accuracy. Removing that direction changes mean probability by less than 0.001 after rounding and flips no predictions. Removing the causal direction flips 47.3 per cent. Information is present, but the head does not use it.

The shortcut regime is the negative case that prevents complacency. The final-layer nuisance probe is similarly excellent at 99.9 per cent, but removal now flips 20.9 per cent of predictions. Balanced output accuracy falls to 79.0 per cent because training made the clean nuisance signal attractive. Here the intervention reveals that the representation has become a working shortcut, not merely an archive.

Random-direction effects overlap the target effects in some layers, which is itself instructive. A single ablation is not self-authenticating. The candidate direction must be compared with a distribution of matched edits, and the semantic claim should be supported by specificity and rescue. The code includes random controls as a warning, not as a complete causal estimator.

The executable core

The artefact uses NumPy, synthetic data and fixed seeds. It contains assertions for the decodable-but-unused positive case, the shortcut negative case and the coordinated-rotation control. Save the embedded code as representation_probe_intervention_lab.py and run python3 representation_probe_intervention_lab.py.

#!/usr/bin/env python3
"""Probe-and-intervention laboratory for latent representation claims.

Synthetic data only. Requires NumPy. The experiment creates two deep feature
stacks with identical architecture:

1. independent: a nuisance variable is linearly decodable but the trained head
   does not use it;
2. shortcut: the nuisance variable is spuriously correlated with the target
   during head training, so it becomes both decodable and causally used.

Assumptions:
- calibration and test data balance target and nuisance labels;
- the fixed encoder preserves both factors while only the output head trains;
- direction removal uses the origin, appropriate here because the balanced
  synthetic activations are approximately centred.

Expected output:
- independent final layer: nuisance probe above 98%, nuisance flips below 1%;
- shortcut final layer: nuisance probe above 98%, nuisance flips above 15%;
- coordinated orthogonal rotation: maximum logit difference below 1e-10.

Run:
    python3 representation_probe_intervention_lab.py
"""
from __future__ import annotations

from dataclasses import dataclass
import numpy as np


Array = np.ndarray


def sigmoid(z: Array) -> Array:
    return 1.0 / (1.0 + np.exp(-np.clip(z, -30.0, 30.0)))


def orthogonal(dimension: int, seed: int) -> Array:
    rng = np.random.default_rng(seed)
    q, _ = np.linalg.qr(rng.normal(size=(dimension, dimension)))
    return q


W1 = orthogonal(6, 1)
W2 = orthogonal(6, 2)


def make_world(
    size: int,
    *,
    correlation: float,
    causal_noise: float,
    nuisance_noise: float,
    seed: int,
) -> tuple[Array, Array, Array]:
    """Return inputs, target labels and nuisance labels.

    The target is the sign of a latent causal bit. The nuisance bit agrees with
    the causal bit with probability ``correlation``. Four additional dimensions
    are irrelevant Gaussian variation.
    """
    rng = np.random.default_rng(seed)
    causal = rng.choice((-1.0, 1.0), size=size)
    agrees = rng.random(size) < correlation
    nuisance = np.where(agrees, causal, -causal)
    inputs = np.column_stack(
        (
            causal + rng.normal(0.0, causal_noise, size),
            nuisance + rng.normal(0.0, nuisance_noise, size),
            rng.normal(0.0, 0.8, size),
            rng.normal(0.0, 0.8, size),
            rng.normal(0.0, 0.8, size),
            rng.normal(0.0, 0.8, size),
        )
    )
    return inputs, (causal > 0.0).astype(float), (nuisance > 0.0).astype(float)


def representation_stack(inputs: Array) -> list[Array]:
    """A fixed, two-layer nonlinear encoder used to isolate head behaviour."""
    h0 = inputs
    h1 = np.tanh(1.2 * (h0 @ W1))
    h2 = np.tanh(1.2 * (h1 @ W2))
    return [h0, h1, h2]


def fit_logistic(
    features: Array,
    labels: Array,
    *,
    learning_rate: float = 0.1,
    steps: int = 3_000,
    l2: float = 1e-3,
) -> tuple[Array, float]:
    """Fit deterministic binary logistic regression with gradient descent."""
    mean = features.mean(axis=0)
    scale = features.std(axis=0) + 1e-8
    standardised = (features - mean) / scale
    weights = np.zeros(features.shape[1])
    bias = 0.0
    for _ in range(steps):
        probability = sigmoid(standardised @ weights + bias)
        weights -= learning_rate * (
            standardised.T @ (probability - labels) / len(labels) + l2 * weights
        )
        bias -= learning_rate * np.mean(probability - labels)
    original_weights = weights / scale
    original_bias = bias - mean @ original_weights
    return original_weights, float(original_bias)


def accuracy(features: Array, labels: Array, weights: Array, bias: float) -> float:
    predicted = sigmoid(features @ weights + bias) >= 0.5
    return float(np.mean(predicted == labels))


def remove_direction(features: Array, direction: Array) -> Array:
    unit = direction / np.linalg.norm(direction)
    return features - np.outer(features @ unit, unit)


@dataclass(frozen=True)
class Regime:
    name: str
    training_correlation: float
    causal_noise: float
    nuisance_noise: float
    output_l2: float


@dataclass(frozen=True)
class LayerResult:
    regime: str
    layer: int
    output_accuracy: float
    causal_probe_accuracy: float
    nuisance_probe_accuracy: float
    shuffled_control_accuracy: float
    causal_delta_probability: float
    nuisance_delta_probability: float
    causal_flip_rate: float
    nuisance_flip_rate: float
    random_delta_median: float
    random_delta_p95: float


def analyse(regime: Regime) -> tuple[list[LayerResult], dict[str, float]]:
    train_x, train_y, _ = make_world(
        5_000,
        correlation=regime.training_correlation,
        causal_noise=regime.causal_noise,
        nuisance_noise=regime.nuisance_noise,
        seed=11,
    )
    calibration_x, calibration_y, calibration_n = make_world(
        4_000,
        correlation=0.5,
        causal_noise=regime.causal_noise,
        nuisance_noise=regime.nuisance_noise,
        seed=12,
    )
    test_x, test_y, test_n = make_world(
        3_000,
        correlation=0.5,
        causal_noise=regime.causal_noise,
        nuisance_noise=regime.nuisance_noise,
        seed=21,
    )

    train_h = representation_stack(train_x)
    calibration_h = representation_stack(calibration_x)
    test_h = representation_stack(test_x)

    output_w, output_b = fit_logistic(
        train_h[2], train_y, steps=5_000, l2=regime.output_l2
    )

    def forward_from(layer: int, hidden: Array) -> Array:
        if layer == 0:
            hidden = np.tanh(1.2 * (hidden @ W1))
            hidden = np.tanh(1.2 * (hidden @ W2))
        elif layer == 1:
            hidden = np.tanh(1.2 * (hidden @ W2))
        elif layer != 2:
            raise ValueError(f"unknown layer: {layer}")
        return sigmoid(hidden @ output_w + output_b)

    baseline_probability = forward_from(2, test_h[2])
    output_accuracy = float(np.mean((baseline_probability >= 0.5) == test_y))
    shuffled_rng = np.random.default_rng(47)
    shuffled = shuffled_rng.permutation(calibration_y)
    shuffled_test = shuffled_rng.integers(0, 2, size=len(test_y)).astype(float)
    rows: list[LayerResult] = []

    for layer in range(3):
        causal_w, causal_b = fit_logistic(calibration_h[layer], calibration_y)
        nuisance_w, nuisance_b = fit_logistic(calibration_h[layer], calibration_n)
        shuffled_w, shuffled_b = fit_logistic(calibration_h[layer], shuffled)

        causal_probability = forward_from(
            layer, remove_direction(test_h[layer], causal_w)
        )
        nuisance_probability = forward_from(
            layer, remove_direction(test_h[layer], nuisance_w)
        )

        random_effects = []
        rng = np.random.default_rng(700 + layer)
        for _ in range(256):
            direction = rng.normal(size=test_h[layer].shape[1])
            direction /= np.linalg.norm(direction)
            probability = forward_from(
                layer, remove_direction(test_h[layer], direction)
            )
            random_effects.append(
                float(np.mean(np.abs(probability - baseline_probability)))
            )

        rows.append(
            LayerResult(
                regime=regime.name,
                layer=layer,
                output_accuracy=output_accuracy,
                causal_probe_accuracy=accuracy(
                    test_h[layer], test_y, causal_w, causal_b
                ),
                nuisance_probe_accuracy=accuracy(
                    test_h[layer], test_n, nuisance_w, nuisance_b
                ),
                shuffled_control_accuracy=accuracy(
                    test_h[layer], shuffled_test, shuffled_w, shuffled_b
                ),
                causal_delta_probability=float(
                    np.mean(np.abs(causal_probability - baseline_probability))
                ),
                nuisance_delta_probability=float(
                    np.mean(np.abs(nuisance_probability - baseline_probability))
                ),
                causal_flip_rate=float(
                    np.mean(
                        (causal_probability >= 0.5)
                        != (baseline_probability >= 0.5)
                    )
                ),
                nuisance_flip_rate=float(
                    np.mean(
                        (nuisance_probability >= 0.5)
                        != (baseline_probability >= 0.5)
                    )
                ),
                random_delta_median=float(np.median(random_effects)),
                random_delta_p95=float(np.quantile(random_effects, 0.95)),
            )
        )

    # Coordinate-change control at the final layer. Orthogonal rotation changes
    # axis labels but not pairwise Euclidean geometry or model behaviour when the
    # downstream weights are transformed consistently.
    rotation = orthogonal(test_h[2].shape[1], 99)
    rotated_h = test_h[2] @ rotation
    rotated_w = rotation.T @ output_w
    original_logits = test_h[2] @ output_w + output_b
    rotated_logits = rotated_h @ rotated_w + output_b
    max_logit_difference = float(np.max(np.abs(original_logits - rotated_logits)))

    return rows, {
        "max_logit_difference_after_rotation": max_logit_difference,
        "test_examples": float(len(test_y)),
    }


def print_results(rows: list[LayerResult], controls: dict[str, float]) -> None:
    print(
        "regime      layer  output  probe(c) probe(n) shuffle  "
        "drop-c Δp/flip  drop-n Δp/flip  random Δp median/p95"
    )
    print("-" * 111)
    for row in rows:
        print(
            f"{row.regime:<11} {row.layer:>5}  "
            f"{row.output_accuracy:>6.1%}  "
            f"{row.causal_probe_accuracy:>7.1%}  "
            f"{row.nuisance_probe_accuracy:>7.1%}  "
            f"{row.shuffled_control_accuracy:>6.1%}  "
            f"{row.causal_delta_probability:>5.3f}/{row.causal_flip_rate:>5.1%}  "
            f"{row.nuisance_delta_probability:>5.3f}/{row.nuisance_flip_rate:>5.1%}  "
            f"{row.random_delta_median:>5.3f}/{row.random_delta_p95:>5.3f}"
        )
    print()
    print(
        "maximum logit difference after coordinated orthogonal rotation: "
        f"{controls['max_logit_difference_after_rotation']:.3e}"
    )


def main() -> None:
    regimes = (
        Regime("independent", 0.50, 0.35, 0.35, 1e-3),
        Regime("shortcut", 0.98, 0.50, 0.15, 1e-2),
    )
    all_rows: list[LayerResult] = []
    all_controls: dict[str, float] = {}
    for regime in regimes:
        rows, controls = analyse(regime)
        all_rows.extend(rows)
        all_controls[regime.name] = controls[
            "max_logit_difference_after_rotation"
        ]

    # Positive case: nuisance is accessible but almost unused in the independent
    # regime. Negative case: nuisance remains accessible and now affects output in
    # the shortcut regime. The thresholds are deliberately loose enough to remain
    # stable across ordinary NumPy versions while still testing the mechanism.
    independent_final = next(
        row for row in all_rows if row.regime == "independent" and row.layer == 2
    )
    shortcut_final = next(
        row for row in all_rows if row.regime == "shortcut" and row.layer == 2
    )
    assert independent_final.nuisance_probe_accuracy > 0.98
    assert independent_final.nuisance_flip_rate < 0.01
    assert shortcut_final.nuisance_probe_accuracy > 0.98
    assert shortcut_final.nuisance_flip_rate > 0.15
    assert max(all_controls.values()) < 1e-10

    print_results(
        all_rows,
        {
            "max_logit_difference_after_rotation": max(all_controls.values()),
        },
    )


if __name__ == "__main__":
    main()

stateDiagram-v2
  [*] --> Claim
  state "Minimal worked example: the<br/>archive beside the calculator" as Claim
  state "Thought experiment: rotate the<br/>explanation while holding the<br/>machine fixed" as Test
  state "Similarity measures answer<br/>comparison questions" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Production-shaped worked scenario: a site shortcut investigation

Worked scenario

This scenario is synthetic. It describes an assurance workflow for a visual inspection model, not a deployed system.

A manufacturer has images from four plants. Plant A uses a darker camera enclosure and also has a higher historical defect rate. A frozen model performs well on the ordinary test set. A site probe reaches 98 per cent accuracy at several layers, and a two-dimensional projection forms site clusters. The assurance question is not “does the model know the site?” The image makes site recoverable. The question is whether site evidence changes defect decisions when defect evidence is controlled.

The team first creates matched image pairs: the same component and defect state under controlled enclosure and colour transformations, plus a smaller set captured on two rigs. It records the naturally induced movement in each candidate layer. This defines a reachable site-change direction more plausibly than a global probe alone.

Second, it compares three interventions: patch the hidden state from the alternate rig image, remove the site subspace, and add a signed site direction at several strengths. For every edit it measures the target logit, calibration, non-target class logits and reconstruction of downstream features. Random subspaces are matched by dimension and activation variance.

Third, it performs rescue. A rig-A state patched into a rig-B example should move the decision in the predicted direction if site is causally used; restoring the original state should restore the output. The team repeats the test across plants, defect types, lighting conditions and checkpoints.

Four decisions can follow. If site is decodable but interventions are inert, the cluster is documented as retained nuisance information and does not by itself block release. If targeted effects are strong only under extreme off-manifold edits, the team reports unresolved sensitivity and improves the counterfactual data. If valid site changes alter decisions, it retrains with balanced or invariant data and makes the intervention suite a release regression. If effects vary by defect type, it narrows the claim and investigates a conditional shortcut rather than publishing one global explanation.

The geometry claim card

The following instrument keeps a team from publishing a mechanism claim stronger than its evidence. Start with the decision the analysis must change. Then stop at the first row whose evidence is not met.

The card changes the default burden. A plot no longer needs to be dismissed because it is “only correlational”; it is accepted as descriptive evidence. It simply cannot carry the release, scientific or governance decision reserved for a causal test. Equally, an intervention need not be inflated into a complete explanation. It can remain a useful, bounded sensitivity result.

Compact glossary

Representation

The activation state produced for an input at a specified layer or component.

Geometry

Relations among representation states under a declared metric, projection or transformation class.

Probe

A separately trained reader used to test whether a property is recoverable from frozen states.

Subspace intervention

An edit that removes, replaces or adds activation components associated with a candidate set of directions.

Activation patching

Replacing an internal activation from one run with an activation from another to measure output change.

Rescue

Restoring the candidate state or applying an opposite edit to recover the predicted behaviour.

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: The perfect cluster that did nothing
  M->>E: Minimal worked example: the archive beside the calculator
  E-->>M: compare rival explanations
  E->>D: Promote geometry only after intervention
  D-->>Q: revise scope or act

Promote geometry only after intervention

Deep networks make internal structure visible because their activations are measurable. That visibility is scientifically valuable. It lets us compare layers, find accessible factors, study manifold separation, select counterfactuals and generate candidate mechanisms. The error begins when the language of seeing silently becomes the language of causing.

The durable distinction is between information that a new reader can extract and information the existing computation uses. A probe answers the first question. A targeted, controlled and rescued intervention is needed for the second. Similarity metrics answer yet another question: which sample relations survive selected transformations. None should be treated as a universal proxy for the others.

The practical change is direct. Do not retrain a model, certify an explanation, block a release or deploy an activation edit because a cluster is clean or a probe is accurate. Use that result to design the intervention. Compare it with random and shuffled controls. Test reachability, specificity, redundancy and rescue. Preserve the exact model, layer, data and counterfactual boundary in the claim.

Representation geometry then becomes more useful, not less. It stops serving as a persuasive picture of what the analyst hopes the network is doing. It becomes an experimental map: a disciplined way to choose where to intervene, what outcome to predict and what evidence would prove the proposed mechanism wrong.