Why sparse coordinates are attractive

A network layer supplies a vector of numbers. Those coordinates are convenient for computation, but an individual coordinate need not correspond to an individual human concept. A neuron may activate for legal language, place names and a punctuation pattern. Conversely, a useful property may be spread across many neurons. Looking at neurons one by one can therefore make a structured representation appear arbitrary.

Superposition offers one explanation. Suppose a layer has d dimensions but the model sometimes needs more than d features. If only a small subset is active for any one input, the model can place feature directions at angles to one another in the same space. Most pairs rarely co-occur, so the capacity gained may outweigh the interference introduced. Toy models demonstrate this trade between sparsity, importance, capacity and interference; they do not by themselves prove that every language-model representation follows the same geometry.[1]

The sparse-autoencoder bargain

A sparse autoencoder, or SAE, learns two maps around a frozen model activation h. The encoder converts h into a wider latent vector z, usually non-negative and mostly zero. The decoder maps that sparse vector back to an approximation of the original activation. In a common abstraction:

h is the original activation; W_e and W_d are encoder and decoder weights; b terms are biases; g enforces sparsity; z contains latent activations; and ĥ is the reconstruction.

Training balances reconstruction against sparsity. An L1 SAE penalises the total magnitude of latent activations. A TopK SAE directly keeps a fixed number of the largest activations. More reconstruction capacity usually reduces error, while stronger sparsity can make examples easier to inspect. The two aims can conflict. Early transformer dictionary-learning work showed that sparse features could expose structure effectively invisible in the neuron basis.[10] High reconstruction fidelity says that the dictionary preserves much of the activation variance. It does not say that each dictionary atom is semantically pure, uniquely recoverable or causally used.

One latent, three claims

The phrase “we found a feature” often compresses three different propositions. Separating them prevents evidence from one proposition silently carrying the other two.

Semantic readability is a claim about the relationship between latent activation and a human description. We inspect highly activating text, images or tokens and decide that a label compresses them well. A rigorous version measures both precision and recall over held-out examples, includes hard negatives and tests whether independent annotators or automated explainers can predict activation. Top examples alone mainly estimate precision in a favourable tail. They say little about cases the feature misses.

Geometric recovery is a claim about representation. A decoder atom may align with a known direction, reconstruct a controlled ground-truth feature or span a useful subspace. Cosine similarity, reconstruction error and probe recovery belong here. Geometry matters because it tells us where information can be written. Yet the decoder is only half the autoencoder. A decoder atom can be well aligned while its corresponding encoder gate rarely or never activates on the inputs used to justify the label.

Causal participation is a claim about a specified computation. On a defined distribution and at a defined layer, changing the candidate mediator should alter a defined output through the model’s downstream computation. The result must survive controls for reconstruction error, collateral movement, alternative pathways and interaction effects. This is narrower than saying that the model “has the concept”. It may support only: “at layer 12, token position 37, this latent mediates part of the logit difference between these two prompt families.”

This distinction also clarifies disagreement. A researcher may call a latent successful because it organises examples and supports editing. Another may reject it because it is unstable across seeds or fails natural activation recall. They can both be correct if they are evaluating different claims. The right question is not whether an SAE feature is “real” in the abstract, but which operation and decision the evidence licenses.

Three falsifiers, not one celebration

Each claim should come with an observation that would weaken it. A semantic label weakens when independent examples activate for an unrelated cause or when genuine positives remain silent. A geometric-recovery claim weakens when the match disappears under signed comparison, held-out synthetic features or a different but equally faithful dictionary. A causal claim weakens when the effect vanishes after retaining the residual, using a more local corruption, matching intervention magnitude or conditioning on interacting components.

This matters because interpretability workflows are selected for discovery. Researchers inspect the examples that look meaningful, the directions that steer, and the prompts that yield crisp graphs. Selection is not misconduct; it is how hypotheses are found. It becomes a validity problem when the same selected cases are reported as evaluation. The confirmatory set must be frozen after the hypothesis is stated, and failures must remain in the denominator.

A useful feature report should therefore contain a claim, a scope and a defeater. The claim states what the latent allegedly detects or mediates. The scope fixes model, layer, position, distribution and metric. The defeater states what result would demote it. Without that third field, every new anomaly can be absorbed into a more elaborate story, and the account ceases to be experimentally discriminating.

flowchart TD
  A["The perfect feature that did<br/>nothing"] --> B["Why sparse coordinates are<br/>attractive"]
  B --> C["One latent, three claims"]
  C --> D["Why a clean latent may still<br/>not be a mechanism"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#D95545,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Why a clean latent may still not be a mechanism

Thought experiment: the perfect compass

Place two compass needles on a table. Both point exactly north. The first is connected to an aircraft’s navigation system; the second is painted on the dashboard. Their geometric alignment is identical. Rotating either painted or connected needle by hand may give an observer the same visual reading. Only one participates in navigation.

An SAE can create the analogous ambiguity. The decoder column tells us the direction added back into activation space when a latent is changed. The encoder row tells us which natural activations cause that latent to change. If a decoder atom aligns with a candidate feature but its encoder gate is inactive on those examples, the atom is like the painted needle. It may remain a powerful intervention direction because external steering bypasses the natural detector.

The antipodal trap

Non-negative SAE latents make orientation especially important. Consider two ground-truth features on opposite rays of one axis: alpha is +v and beta is −v. If evaluation uses absolute cosine similarity, a decoder atom +v is a perfect geometric match for both. A ReLU encoder aligned with +v activates for alpha and returns zero for beta. Adding −v by external steering can still create beta-like behaviour. The axis is steerable, but the named latent does not detect beta.

A recent preprint calls this family of mismatches causal inertness and reports controlled cases in which geometrically recovered atoms fail to activate for the matched feature. Its most important contribution is methodological: report decoder geometry, encoder behaviour and intervention results separately.[11] Because the work is a recent preprint and includes toy and selected production-SAE audits, its numerical rates should not be treated as a settled prevalence estimate across models.

Splitting, absorption and silent holes

Even when orientation is correct, one human concept need not map to one latent. A broad feature may split into context-specific children. That can be useful when the children correspond to distinct computations. It becomes a monitoring problem when a general latent appears semantically clean but fails on cases captured by its children. Work on a controlled first-letter task calls this feature absorption: token-specific latents absorb part of a general letter feature, leaving holes where the apparent general feature should fire.[7]

Absorption exposes the weakness of inspecting only maximal activations. A latent can have excellent-looking examples and still have low recall. Increasing dictionary width or changing sparsity may move the holes rather than eliminate them. For oversight, a sparse feature that silently misses a concept is more dangerous than a visibly noisy feature because the label invites unwarranted confidence.

Thought experiment: two atlases, neither canonical

Give two cartographers the same city. One builds a railway atlas; the other builds a flood-risk atlas. Both are sparse: each omits details irrelevant to its purpose. A station may be atomic in the first map and a small part of a catchment in the second. Asking which map contains the city’s uniquely true units mistakes a representation for the territory.

SAE dictionaries can behave similarly. Training two SAEs on the same activations with different random seeds can yield materially different feature sets. One study reported that, for a 131,000-latent SAE on a Llama 3 8B feed-forward layer, only about 30 per cent of features were shared across seeds; the pattern appeared across multiple layers, models, datasets and SAE variants.[5] Meta-SAE and stitching experiments also challenge the idea that any one dictionary is simultaneously complete, unique and atomic: larger-SAE latents can add information missing from smaller dictionaries, while apparently specific latents can decompose into interpretable combinations.[6]

Non-uniqueness does not eliminate causal explanation. Biology uses several useful decompositions of the same organism. Software can be described by functions, data flows or state machines. What non-uniqueness removes is the shortcut from “this dictionary has a clean atom” to “the model’s ontology contains this indivisible concept”. A mechanistic unit can be task-relative and still earn its status through interventions.

Recent work further suggests that some concepts may occupy low-dimensional curved manifolds rather than isolated rays. An SAE may represent such a manifold globally through a group of atoms, locally through tiled features, or through a fragmented mixture. In that setting the appropriate object of analysis may be a feature group and its geometry, not a single latent.[13] This remains an active research direction, but it offers a concrete explanation for why individual-feature labels can look partial even when the dictionary reconstructs well.

The strongest objection: mechanisms need not be unique

A critic might accept every instability result and still reject the paper’s caution. Scientific mechanisms are often described at several grains. Electrical engineers can explain one circuit through currents, transfer functions or state variables. If an SAE coordinate supports reliable intervention, why demand that it be a unique atom?

The objection is correct about uniqueness. Causal usefulness does not require a privileged ontology. It does require the claim to follow the invariant. Suppose two SAE seeds recover different atoms, but each atom belongs to a matched two-dimensional subspace. Group ablation of either subspace removes the same behaviour, and insertion restores it with the same side-effect profile. The defensible mechanism is then the subspace or equivalence class, not either feature index. The explanation becomes more abstract and more stable at once.

Now suppose a rotated dictionary reconstructs equally well, gives equally coherent labels and supports the same behaviour edits. That is a serious negative control against the claim that the original axes are uniquely explanatory. It is not a negative control against the weaker claim that the represented subspace carries causally relevant information. Mechanistic language should attach to what survives admissible changes of basis. When only a particular sparse basis makes the explanation legible, call the basis a useful coordinate system. When an effect survives matching, residual controls and alternative decompositions, promote the invariant object.

This response keeps SAE research ambitious without turning every instability into defeat. Sparse coordinates may be the microscope rather than the cell. The microscope can still reveal structures and support interventions. Its optical properties must be part of the account, especially when the observed unit changes with magnification.

The causal validation battery

Causal validation begins by making the claim smaller. Name the behaviour, contrast, layer, token positions, data distribution, intervention and metric. “Feature 18492 represents deception” is too broad to test. “At the final token of these paired prompts, latent 18492 mediates part of the logit difference between a truthful answer and a strategically false answer” is testable, though still demanding.

The battery below is not a universal certification standard. It is a decision instrument for preventing proxy success from becoming a mechanistic conclusion. Each test answers a different failure mode.

1. Preserve the original behaviour before interpreting the replacement

Let the original activation be decomposed as h = W_dz + b_d + r, where r is the reconstruction residual. Compare the model tail on h with the tail on the reconstruction ĥ. Measure task outputs, not only mean-squared activation error. A small average residual can still align with a high-sensitivity downstream direction. If the SAE-replaced model changes the behaviour under study, interventions may explain the replacement rather than the original model.

Keep the residual in the intervention whenever possible. To ablate latent j, construct h^−j = r + Σ_k≠j z_kd_k + b_d. Compare this with a residual-free ablation. A large difference is diagnostic: the residual is participating in the behaviour or compensating for dictionary error.

2. Test the encoder as a detector, not a gallery

Build positive, hard-negative and counterfactual examples for the proposed meaning. Measure activation recall, precision, threshold stability and context sensitivity. Sample examples uniformly from the evaluation distribution rather than selecting only maximal activations. Test paraphrases, negations, multilingual variants where relevant, and matched texts that share surface tokens without the target property.

A feature can be semantically coherent yet operationally unusable because its recall is low. Feature absorption is one route. Seed-specific fragmentation is another. Adversarial-robustness work has also shown that small input perturbations can manipulate SAE concept readings while leaving base-model outputs largely unchanged, which is a warning against using the readings as an isolated oversight channel.[14]

3. Separate necessity from sufficiency

Necessity asks whether removing the natural contribution reduces the specified behaviour. Sufficiency asks whether inserting a controlled contribution can induce or restore it. For an output score Y, one simple pair of effect measures is:

N_j is the average ablation effect; S_j is the average insertion or steering effect; and expectations are taken over an explicitly defined prompt distribution. Normalise against the clean-to-corrupted behavioural gap when comparing tasks.

A zero necessity result does not always prove irrelevance. The computation may contain redundant or backup pathways. The feature may matter only through interactions. Conversely, a large sufficiency result does not prove natural use. White-box steering can create states that no natural prompt reaches. Recent theory and experiments formalise this concern by showing that steered residual states are generally off the manifold induced by discrete prompts.[12] Treat steering as evidence of a control handle unless natural readout is independently established.

4. Measure specificity and collateral movement

An intervention that changes the target output may also change many unrelated outputs. Record the full logit or behaviour delta, not only the desired scalar. Compare the target effect with total effect, with effects on matched control tasks, and with norm-matched random directions. Inspect neighbouring features and decoder correlations. If the candidate atom writes several properties at once, the human label may describe only the most salient consequence.

Specificity also concerns the intervention scale. Very large steering coefficients can force almost any high-dimensional model into unusual behaviour. Sweep magnitude around natural activation quantiles and report the smallest dose that produces a stable effect. A causal story that appears only at fifty times the natural latent range is a control demonstration, not a faithful account of ordinary computation.

5. Test stability without demanding false universality

Repeat across SAE seeds, widths, sparsity levels and training-data slices. Track whether a corresponding feature, feature group or causal subspace recurs. Repeat across prompt templates and positions. A behaviour-specific mediator need not receive the same feature index across dictionaries, but its effect should be recoverable by a stated matching procedure. When the result depends on one seed or one hand-selected prompt, report it as a local hypothesis.

Benchmark work reinforces the need for a portfolio. SAEBench compares interpretability, disentanglement and downstream-use metrics across more than 200 SAEs and finds that gains on common proxy metrics do not reliably transfer to practical performance.[8] A single “SAE quality” score therefore hides design choices. Select metrics that correspond to the intended decision.

6. Expose interactions and alternative explanations

Activation patching is causal in the sense that it changes an internal variable and observes an outcome, but its estimand depends on the corruption, metric and context. Best-practice work shows that methodological choices can produce different localisation results.[9] Recent causal-mediation analysis further shows that single-component patching can contain interaction effects with other components, making some mediators appear inflated or invisible.[15]

Use group ablations, conditional tests and combinatorial search when interaction is plausible. Compare the SAE explanation with neuron, attention-head, supervised-probe, PCA or task-specific subspace baselines. The serious negative control is not merely a random direction. It is a simpler competing account that could explain the same behaviour. If a supervised probe has better coverage and its orthogonalised intervention produces the same effect, sparsity has not yet earned the claim that its latent is the model’s natural unit.

7. Issue an evidence receipt

A causal result should leave a compact, replayable record. At minimum, record the base-model identifier, activation site, SAE checkpoint and training settings, feature or feature-group identifier, evaluation-set hash, prompt contrast, target metric, natural activation range, replacement fidelity, coverage, necessity, sufficiency, specificity, stability runs, residual treatment and negative controls. Record failures as well as aggregate scores.

The receipt should also state the permitted claim in one sentence and the conditions that expire it. A model-weight update, tokenizer change, different system prompt, new SAE or changed output metric can invalidate the result. This is especially important for monitors: a feature index is not a durable interface unless the model and dictionary are jointly versioned and the evaluation is rerun.

The receipt turns an interpretation from a picture into an inspectable test result. It also enables comparison between methods. A neuron circuit and an SAE circuit can be judged against the same behavioural contrast, fidelity boundary and controls rather than against method-specific aesthetics.

stateDiagram-v2
  [*] --> Claim
  state "The sparse-autoencoder bargain" as Claim
  state "Three falsifiers, not one<br/>celebration" as Test
  state "Thought experiment, the<br/>perfect compass" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

A known-world audit

Real language models do not provide ground-truth features. That makes a controlled world valuable. The accompanying NumPy artefact defines a twelve-dimensional activation space, five known features, a fixed SAE and a known downstream computation. Four features matter to the output. A fifth, context, is reconstructed but deliberately ignored by the model tail.

Alpha and beta are antipodal: alpha is +e₀ and beta is −e₀. Decoder atom d₀ points along +e₀, so its absolute cosine with both is exactly one. Its ReLU encoder activates on alpha and not beta. A separate beta-reading atom d₃ points almost along −e₀ but includes a small gamma component. This arrangement creates a geometric false positive and a specific alternative that reconstructs beta with a side effect.

def audit(feature, atom, target_output):
    h = GROUND[feature].copy()
    direction = DECODER[:, atom]
    z = encode(h)

    # Natural necessity: remove only this atom's encoded contribution.
    before = model_tail(h)
    after = model_tail(h - z[atom] * direction)
    necessity = abs((before - after)[target_output])

    # White-box steering: orient the decoder axis towards the feature.
    sign = 1.0 if direction @ h >= 0.0 else -1.0
    steered = model_tail(sign * direction) - model_tail(np.zeros(DIM))
    steering = abs(steered[target_output])
    specificity = steering / (np.abs(steered).sum() + 1e-12)

    return {
        "axis_cosine": abs(cosine(h, direction)),
        "fires": bool(z[atom] > 1e-9),
        "necessity": necessity,
        "steering": steering,
        "specificity": specificity,
    }

The script generates 20,000 sparse synthetic activations and checks the reconstruction before interpreting any latent. The SAE explains 99.30 per cent of centred activation variance in this controlled sample. That number is high enough to make the failure non-trivial, yet the code makes no claim that 99.30 per cent is a universal threshold.

The beta result is the paper’s negative control. If the audit used only absolute cosine and steering, d₀ would look ideal. The encoder and necessity columns falsify that interpretation. The context result establishes a second boundary: a direction can be genuinely represented and faithfully reconstructed while remaining irrelevant to the selected output. The alternative d₃ shows why passing necessity is not enough either. It mediates beta, but its decoder also writes gamma, reducing specificity.

The script also samples 5,000 norm-matched random directions. None equals the target steering effect. That is useful, but it does not rescue d₀ as beta’s natural detector. A strong random-direction control can establish that an intervention is special without establishing that the corresponding latent is naturally used.

A production-shaped decision: may this latent become a monitor?

Return to the synthetic evidence-writing assistant. The proposed internal monitor watches for unsupported certainty and sends high-scoring drafts to human review. This is a higher-stakes use than feature browsing because false negatives can create misplaced trust and false positives can overload reviewers.

Start with a written claim: “On draft-generation tokens in the supported-answer workflow, this latent detects unsupported certainty with at least the stated recall and adds independent warning value beyond output-level claim verification.” The claim deliberately avoids saying that the latent is the model’s concept of truthfulness.

Build evaluation cases by varying one causal feature at a time. Hold topic and wording constant while changing whether evidence supports the claim. Add hard negatives containing uncertainty vocabulary but valid evidence, and hard positives that express unsupported certainty without obvious cue words. Measure the encoder over all cases, not only the feature browser’s top activations. Compare with a supervised raw-activation probe and an output-level verifier.

Next, establish model-tail fidelity with and without the SAE. Ablate the natural latent activation while retaining the residual. Observe claim tokens, confidence language and factual-support scores. Insert the latent at natural activation quantiles and measure collateral changes in style, refusal, length and topic. Repeat across SAE seeds and widths. Run group ablations if related latents divide the behaviour. Use a label-shuffled detector and norm-matched directions as low-level controls; use the supervised probe as the serious competing baseline.

The decision is asymmetric. A latent with good precision but weak recall may still support analyst exploration. It must not become the sole monitor. A stable detector that adds independent recall can support telemetry, but the system should still verify claims against evidence at the output boundary. A behaviour-specific mediator can support a narrower mechanistic explanation. A steering direction can support a controlled experiment, subject to side-effect and recovery limits. No SAE result transfers authority to block, approve or release an output by itself.

Three release outcomes are possible. First, the latent may fail coverage while the output verifier succeeds. Keep the verifier and retain the latent only for research. Second, the latent may add independent warning value but fail necessity. Use it as ensemble telemetry and describe it as a detector, not a causal explanation. Third, the latent may pass both detector and mediator tests. Even then, route the alert through an explicit review policy with a documented threshold, reason code and outcome receipt. The mechanism evidence improves diagnosis; it does not replace governance of the action taken.

The promotion rule

Interpretability work often fails at the hand-off between research and use. A visually compelling latent enters a slide as a “model concept”, then becomes a monitoring proposal, then quietly acquires authority. The promotion rule interrupts that drift.

This rule preserves the genuine contribution of sparse features. They can expose regularities hidden in neuron coordinates, generate causal hypotheses, supply compact intervention variables and support feature-level circuits. Sparse feature circuit work demonstrates that causally implicated, human-interpretable subnetworks can improve the resolution of mechanistic analysis.[4] The rule simply refuses to infer more than the intervention supports.

It also changes SAE selection. The best dictionary for browsing may not be the best detector. The best detector may not be the most faithful circuit basis. Benchmark portfolios and task-level experiments should therefore choose the SAE for the operation, not crown one architecture with a context-free quality score. Mechanistic interpretability becomes stronger when a feature is treated as a falsifiable causal proposal rather than a discovered noun.

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: The perfect feature that did nothing
  M->>E: The sparse-autoencoder bargain
  E-->>M: compare rival explanations
  E->>D: Treat sparse features as hypotheses with handles
  D-->>Q: revise scope or act

Treat sparse features as hypotheses with handles

Superposition makes neuron-by-neuron reading unreliable, and sparse autoencoders offer a powerful response: learn a wider basis in which fewer coordinates activate at once. The response is valuable precisely because it creates inspectable hypotheses. It becomes misleading only when the learned coordinate is mistaken for the mechanism without testing how the frozen model reads and uses it.

The durable distinction is simple. Semantic examples support a label. Decoder geometry supports a direction or subspace. Encoder coverage supports a detector. Ablation and patching support a scoped necessity claim. Steering supports a control claim. Specificity, residual accounting, stability and competing baselines determine whether these pieces cohere into a mechanism.

The architecture and research decision changes here: never place an SAE latent into a consequential monitor, circuit or control path under the generic label “interpretable feature”. Attach a claim level, behaviour scope, evaluation manifest and expiry condition. Keep output evidence and deterministic safeguards outside the learned dictionary. Promote the latent only when the next causal test passes.

That discipline does not shrink mechanistic interpretability. It gives the field a way to accumulate results that can survive new seeds, new dictionaries and better theories of representation. A sparse feature need not be the model’s final ontology to be useful. It must only be honest about what it can detect, what it can change and what remains unexplained.