The basis that looks like a discovery

Why superposition appears

Start with a model that has two internal dimensions but needs to track three properties. Call them square, red and moving. If these properties were always present together, two dimensions would be hopelessly insufficient. If they are sparse, most inputs contain only one, the model can place the three properties along different directions in the same plane. Each property remains recoverable most of the time, but the directions interfere when several co-occur.

That is the core intuition behind superposition. In toy models, sparse features can be packed into fewer dimensions because the network accepts bounded interference on the uncommon examples where features collide.[1] The model is not storing three clean labelled drawers inside two neurons. It is using geometry. A neuron then becomes a coordinate axis through several properties rather than a property itself.

What an SAE actually optimises

A sparse autoencoder receives an activation vector x, maps it to latent activations z, and reconstructs x̂ from a learned dictionary of decoder directions. A common training objective balances reconstruction error against sparse use of the dictionary:

Nothing in this objective asks whether latent j is stable across random seeds. Nothing requires it to align with a variable the original model uses downstream. Nothing forces the same grain of description across dictionary widths. Nothing guarantees that a one-dimensional direction is the right shape. The objective says: reconstruct these sampled activations with a sparse code.

This is not a criticism of the method. It is its contract. Sparse autoencoders have recovered highly interpretable latents, scaled to millions of candidate patterns, and supported detailed intervention studies.[2][4][5] The mistake is to smuggle extra guarantees into the contract. Classical dictionary learning can identify a generating dictionary only under assumptions about sparsity, incoherence, sampling and the data-generating process.[14] Real model activations do not arrive with a certificate that those assumptions hold.

Reconstruction answers whether the dictionary can describe an activation distribution. It does not, by itself, answer whether the dictionary matches the variables that carry a computation. This decomposition-mechanism gap is the load-bearing distinction for everything that follows.

flowchart TD
  A["The feature that survived<br/>every example"] --> B["The basis that looks like a<br/>discovery"]
  B --> C["Where a sparse coordinate<br/>earns causal meaning"]
  C --> D["The stability lab"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#5B5CF6,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Where a sparse coordinate earns causal meaning

Suppose a latent has a compelling label. What additional evidence would turn it from a readable coordinate into a mechanism claim? The answer becomes clearer by changing one causal property at a time. Four failures matter because each defeats a different inference.

Failure one: the latent changes when the seed changes

Training an SAE is a non-convex optimisation problem. Different initialisations can settle on dictionaries with similar reconstruction and sparsity but different atoms. Paulo and Belrose trained SAEs on the same model activations and data while changing the random seed. In one 131,000-latent Llama 3 8B setting, only 30 per cent of features were shared between runs under their matching procedure; the broader study found seed dependence across models, layers and SAE variants.[6] This number should not be universalised, but the phenomenon is enough to change practice.

A feature that appears in one run may still be valuable. Its absence elsewhere has several possible causes: optimisation failure, insufficient samples, an overly flexible objective, correlated variables, or genuinely non-unique structure. The correct conclusion is not “the feature is false”. It is “single-run identity is unlicensed”.

Failure two: the feature changes when the dictionary becomes wider

A small dictionary may allocate one latent to “mathematics”. A wider dictionary can split it into algebra, geometry and proof language. Wider still, algebra may split into group theory, linear equations and ring notation. The change is not necessarily a march towards indivisible atoms. Leask and colleagues found that larger SAE latents could themselves be compositions of interpretable meta-latents, and argued that tested SAEs did not converge on one unique, complete and irreducible dictionary.[8]

Failure three: a clear feature has arbitrary exceptions

Imagine a latent that activates on words beginning with the letter S. Its dashboard looks monosemantic. Yet some S-words do not activate it. Other token-aligned latents carry pieces of the same direction on those cases. Chanin and colleagues called this feature absorption. In their first-letter experiments on Gemma 2 2B with JumpReLU SAEs, apparently interpretable latents missed cases that causal ablation traced to absorbing latents. Changing width or sparsity did not simply remove the problem, and some configurations showed more absorption.[7]

Failure four: the model's variable is not one-dimensional

Standard SAEs assign one decoder direction to each latent. That works naturally when a property behaves like a scalar direction. Some model variables have internal geometry. Engels and colleagues found circular representations of days of the week and months of the year in language models, then used interventions to show that the circular structure supported modular arithmetic tasks.[9] A circle cannot be represented faithfully by one ray. A vector dictionary must approximate it with several local directions.

The causal twin: marker or mediator?

Even a stable, well-labelled latent may be downstream of the computation we care about. A “refusal language” latent can activate because the model has already decided to refuse. A different latent may carry the information that causes the refusal. Both correlate with refusal tokens. Only one mediates the decision.

From coordinate feature to mechanism feature

The four failures suggest a disciplined claim ladder. A latent may be readable before it is stable, stable before it is complete, and causally effective before it is a normal mediator. Each rung permits a stronger use.

A mechanism feature is not defined by one high score. It is a bounded claim backed by a pattern of tests. “Bounded” matters. A feature can mediate refusal at layer 18 for English text and fail at layer 12, in another language or after fine-tuning. The warrant should name the model, checkpoint, activation site, data distribution, intervention scale, downstream variable and known exceptions.

The five levels should be recorded as a profile rather than collapsed into an average. A perfect label cannot compensate for zero coverage. Strong seed agreement cannot compensate for an intervention that changes many unrelated outputs. Restoration cannot rescue a feature whose clean-run activation was never necessary. The dimensions answer different questions, so arithmetic aggregation would hide the exact reason a use is unsafe.

A claim packet should therefore contain the candidate label, the sampling rule that produced its examples, the alternative dictionaries searched, the matching method, contrast-set results, intervention magnitudes, control directions, downstream readbacks and a list of failed contexts. The natural-language description is only one field. This packet lets another researcher reproduce the inference rather than merely admire the examples.

Discovery and confirmation should also be separated. Use one slice of data to find and name the candidate. Freeze the label and matching rule. Test recurrence, false negatives and causal effects on held-out slices. Otherwise the same examples that inspired the interpretation also certify it, making a flexible label look more stable than it is.

This turns interpretability from feature collecting into claim engineering. The next part makes that discipline executable.

The stability lab

A good interpretability instrument should make its own uncertainty visible. The lab below does not attempt to reproduce a frontier model. It creates a two-dimensional activation field from known sparse causes, trains several small dictionaries from different random initialisations, and then asks whether equal reconstruction implies equal features. The answer changes with correlation, noise, dictionary width and the number of active atoms permitted.

The simplicity is deliberate. Because the generating directions are known, the lab can distinguish three quantities that real-model studies often collapse: fit to activations, agreement among decompositions and recovery of the planted causes. Low reconstruction error is a statement about the first quantity only.

What the lab tests

The synthetic data contain two coordinates, several planted directions and sparse non-negative activations. A small Top-K dictionary learns unit decoder vectors. Each iteration encodes a sample by its strongest positive dot products, reconstructs the sample and nudges the selected vectors towards the remaining error. Four seeds see identical data. Their atoms are compared by activation correlation and angular matching.

The experiment encodes four assumptions. The activation site is fixed. Features combine approximately linearly. Sparsity is a sensible prior. The downstream causal target is a known direction. Removing any assumption changes the meaning of success. In a real transformer, the data distribution is selected, the encoder is approximate, downstream computations are nonlinear, and the intervention may leave the native activation manifold.

The lab cannot establish that SAE latents are canonical in large language models. It cannot validate an automated natural-language label. It cannot show that an intervention follows the model's normal causal route. Its value is narrower and more important: it exposes the logical gap between reconstruction and identification in a setting where the ground truth is available.

import numpy as np


def unit(v: np.ndarray) -> np.ndarray:
    return v / max(np.linalg.norm(v), 1e-12)


def make_data(seed: int = 7, n: int = 1200) -> tuple[np.ndarray, np.ndarray]:
    rng = np.random.default_rng(seed)
    angles = np.deg2rad([0, 32, 118, 205, 286])
    truth = np.c_[np.cos(angles), np.sin(angles)]
    xs = []
    for _ in range(n):
        active = [0, 1] if rng.random() < 0.55 else [rng.integers(0, len(truth))]
        coeff = rng.uniform(0.35, 1.25, len(active))
        x = sum(c * truth[i] for c, i in zip(coeff, active))
        xs.append(x + rng.normal(0, 0.03, 2))
    return np.asarray(xs), truth


def fit_dictionary(x: np.ndarray, width: int, top_k: int, seed: int) -> np.ndarray:
    rng = np.random.default_rng(seed)
    d = np.asarray([unit(rng.normal(size=2)) for _ in range(width)])
    for epoch in range(45):
        for i in rng.permutation(len(x)):
            scores = np.maximum(0.0, d @ x[i])
            chosen = np.argpartition(scores, -top_k)[-top_k:]
            z = np.zeros(width)
            z[chosen] = scores[chosen]
            residual = x[i] - z @ d
            d[chosen] += (0.055 / (1 + epoch / 18)) * z[chosen, None] * residual
            d[chosen] = np.asarray([unit(v) for v in d[chosen]])
    return d


def encode(x: np.ndarray, d: np.ndarray, top_k: int) -> np.ndarray:
    scores = np.maximum(0.0, x @ d.T)
    keep = np.argpartition(scores, -top_k, axis=1)[:, -top_k:]
    z = np.zeros_like(scores)
    rows = np.arange(len(x))[:, None]
    z[rows, keep] = scores[rows, keep]
    return z


def mse(x: np.ndarray, d: np.ndarray, top_k: int) -> float:
    z = encode(x, d, top_k)
    return float(np.mean((x - z @ d) ** 2))


def signature_agreement(x: np.ndarray, a: np.ndarray, b: np.ndarray, top_k: int) -> float:
    """Match latents by held-out activation signatures, not decoder angle alone."""
    za, zb = encode(x, a, top_k), encode(x, b, top_k)
    scores = []
    for j in range(za.shape[1]):
        va = za[:, j]
        if np.std(va) < 1e-10:
            scores.append(0.0)
            continue
        best = 0.0
        for k in range(zb.shape[1]):
            vb = zb[:, k]
            if np.std(vb) >= 1e-10:
                best = max(best, abs(float(np.corrcoef(va, vb)[0, 1])))
        scores.append(best)
    return float(np.mean(scores))


x, truth = make_data()
runs = [fit_dictionary(x, width=12, top_k=1, seed=s) for s in range(4)]
print("mean reconstruction MSE:", np.mean([mse(x, d, 1) for d in runs]))
print("mean cross-seed signature agreement:", np.mean([
    signature_agreement(x, runs[0], d, 1) for d in runs[1:]
]))
print("planted directions:\n", truth)
print("first learned dictionary:\n", runs[0])

From toy geometry to a configured model

This architecture also prevents a common category error. An SAE intervention and a model action are not the same event. The former changes an internal state. The latter is the verified downstream consequence. Recording both makes it possible to discover that a latent moved while the claimed capability did not, or that the output changed through collateral damage.

Match feature families without choosing the answer

Cross-seed matching is itself an inference problem. Decoder cosine similarity can join directions that look alike but activate on different examples. Activation correlation can join latents that share a dataset shortcut. Label similarity can make an automated interpreter repeat the researcher's preferred concept. Downstream-effect similarity is stronger, but two broad perturbations can damage the same output for unrelated reasons.

A defensible matcher uses several views and declares them before examining the headline result. One practical sequence is to generate candidate pairs from activation correlation, reject pairs with incompatible minimal contrasts, compare decoder subspaces, and then reserve downstream intervention effects for held-out confirmation. Matching can be many-to-one because a broad latent at one width may correspond to a family at another. Unmatched mass is reported rather than forced into the nearest label.

The matching rule should be tested on synthetic mixtures where the planted relation is known. It should also face negative controls: unrelated features with similar frequency, random rotations that preserve reconstruction, and features whose labels are semantically close but whose interventions differ. A feature family earns stability only when the matcher can fail visibly. Otherwise “the feature returned” may mean only that the comparison procedure was flexible enough to find something familiar.

Checkpoint changes require a new boundary. Fine-tuning can rotate, split or reroute a representation while preserving behaviour. A monitoring feature that matched across seeds in the base model must not be assumed to survive an adapted model. Re-run the family matcher, the contrast set and at least the decisive intervention. Version the result with the checkpoint rather than treating the label as a permanent asset.

stateDiagram-v2
  [*] --> Claim
  state "The sparse lens separates a<br/>crowded sky, but the lens<br/>chooses the constellations" as Claim
  state "Why superposition appears" as Test
  state "Rotate every instrument,<br/>preserve every prediction" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

The strongest boundaries

The strongest alternative explanation is not that sparse autoencoders are useless. It is that they are useful scientific instruments whose axes are partly chosen by objective, data and regularisation. A microscope can expose real cells without making its pixels biological units. A coordinate chart can support navigation without being carved into the terrain. On this view, SAE latents remain valuable for search, summarisation, anomaly detection and intervention proposal even when no single dictionary is canonical.

Classical dictionary-learning results show that sparse dictionaries can become identifiable under explicit assumptions about data generation, sparsity and sampling.[14] Recent work seeks analogous guarantees or stronger stability for SAEs.[15] This is a genuine route forward. It is also a boundary on present claims. A large model's activations may violate independence, linearity, stationarity or adequate sampling. Identifiability is not transferred by borrowing the word “sparse”. The assumptions must be tested at the activation site being interpreted.

Residual error creates a second boundary. An SAE can reconstruct most activation variance while losing a small component that matters disproportionately to the downstream loss. Conversely, some predictable reconstruction error may be irrelevant to the behaviour under study.[10] Reconstruction metrics therefore require a task-conditioned readback. For a safety claim, ask what the residual contributes to the refusal or compliance decision. For factual recall, ask what it contributes to the target token distribution. End-to-end sparse dictionary learning makes this principle explicit by optimising preservation of the model output rather than activation reconstruction alone.[11] Coverage is about the causal variable, not the percentage of activation energy retained.

A third boundary is geometric. Weekdays and months have been recovered as circular structures in language-model representations, with causal interventions that move the model around the circle.[9] A set of one-dimensional SAE atoms can approximate such a structure, but the atoms may be local tangents rather than separate concepts. Subspace-aware methods explicitly explore this possibility.[16] When latents split and recombine with width, the correct candidate may be a subspace, manifold or distributed circuit.

A fourth boundary sits below activations. Feature decompositions do not automatically explain how weights implement the computation. Recent circuit work shows that interference can remain in weights even after a useful feature basis is chosen.[17] A sparse feature can be the right variable for one causal experiment while leaving the weight-level algorithm unresolved. Representation and implementation are connected questions, not interchangeable answers.

The matrix resists two opposite mistakes. One is inflation: calling every crisp latent a mechanism. The other is nihilism: discarding a coordinate because it is not unique. Coordinate features are often excellent instruments for finding examples and generating interventions. Their value becomes unsafe only when the use outruns the evidence.

Non-uniqueness is harmless when the decision is invariant to the choice of basis. If several dictionaries retrieve the same case set, support the same bounded intervention and preserve the same downstream readback, the analyst may choose the one that is cheaper or easier to communicate. Non-uniqueness becomes consequential when a monitor misses different cases, an edit produces different side effects, or a mechanistic story changes with the dictionary. The scientific target is not uniqueness for its own sake. It is decision-relevant invariance.