What an embedding buys

Suppose a model receives four words: sparrow, eagle, salmon and trout. A one-hot code gives each word its own isolated coordinate. It preserves identity perfectly, yet says nothing about which substitutions are useful. A learned embedding can place the bird words near one another and the fish words near one another, so that one downstream transformation can act on a family rather than memorising every member.

The gain is not that “birdness” now lives inside a number. The gain is computational: a shared relation has become accessible to a simple operation. Early word-vector systems made this vivid by learning continuous representations from prediction or co-occurrence, then showing that syntactic and semantic relations could be recovered more efficiently than from sparse codes.[1][2] Modern contextual models repeat the operation at every layer and token, so the point representing bank may move with its sentence.

An embedding is useful when a downstream computation can reuse a relation across cases. Nearness, direction, curvature or a separating surface matters only relative to an operation. Cosine similarity asks one question. A linear classifier asks another. A nonlinear layer can exploit distinctions that both miss. The same cloud of points therefore supports several geometries.

Three relations that are often collapsed

A point can be close to another point, linearly separable from a class, and causally connected to an output. These are different claims.

Metric geometry concerns distances and angles. It depends on a metric, often Euclidean distance or cosine similarity. Decoder geometry concerns what a chosen family of functions can recover. A linear probe asks whether one hyperplane can separate labels. Causal geometry concerns which changes to an internal state produce stable, predicted changes later in the system.

The distinctions matter because neural representations are usually identifiable only up to transformations that leave the model’s input-output function unchanged. Rotate every hidden state and rotate the next weight matrix in the opposite direction. The network behaves identically, but individual neurons and Euclidean angles change. Work on the linear representation hypothesis makes this issue explicit: even apparently simple ideas such as projection need a justified inner product, not an automatic appeal to the standard dot product.[12]

Contextual embeddings add a second warning. Their spaces can be anisotropic, with vectors occupying narrow cones rather than spreading evenly. In such a space, a high cosine similarity may be ordinary rather than diagnostic.[3] Geometry is therefore never just “what the plot looks like”. It is the plot, the metric, the decoder, the distribution and the operations that make a relation consequential.

flowchart LR
  A["The perfect probe that<br/>explained nothing"] --> B["The map found in the ruin"]
  B --> C["How training carves coordinates"]
  C --> D["The probe that finds too much"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#7B4CD8,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Change one variable: reuse

Imagine a network that must classify ten animal species. In the first version, every species appears in a unique background and the model has ample capacity. It can memorise ten unrelated templates. In the second version, backgrounds vary and later layers repeatedly need distinctions such as winged versus aquatic, warm-blooded versus cold-blooded, and large versus small. A representation that groups cases by those reusable properties now reduces downstream work.

The hidden state h becomes a working surface. Training adjusts it so that a later function can produce the required output with lower loss. In the simplest linear readout, the model computes:

Here, h is the model’s hidden state for one case, w_out is the direction to which the output is sensitive, and b is an offset. If many cases that require the same decision share a component of h, one readout direction can serve them all. This equation explains linear accessibility under one assumption: the next stage is linear in h. Remove that assumption and useful structure may be curved, distributed across interactions, or recoverable only after another nonlinear transformation.

Why clean axes sometimes appear

Several pressures can favour simple structure. A narrow bottleneck forces compression. Weight decay and low-rank constraints reward economical solutions. Repeated use of the same distinction rewards a shared coordinate. Gradient descent can discover directions that align with the downstream readout. Contrastive or predictive objectives pull some cases together and push others apart. None of these guarantees a human-named axis, but each can make one family of relations easier to express.

Clean geometry can also be an artefact of the probe or the visualisation. Projecting a 768-dimensional cloud onto two principal components discards almost everything. A linear separator may be found even when the class clusters overlap in every intuitive view. Conversely, an apparently tangled plot may contain a simple relation in another basis.

Structure emerges relative to the loss and the operations that consume it. A direction is not privileged because it looks semantic to us. It is privileged when the model’s own computations repeatedly make use of variations along it.

Capacity creates superposition, not one concept per neuron

A model often has more useful features than available dimensions. When features are sparse, several can share the same dimensions and be recovered through nonlinearities. Toy models of superposition show that optimisation can arrange features into characteristic geometric patterns, including polygons, because interference is cheaper than dedicating one neuron to each feature.[10] This is a valuable mechanism hypothesis, but its strongest evidence comes from controlled toy systems. In large models, a readable direction may combine many features, and a single feature may move across layers or bases.

Readable axes are neither necessary nor sufficient

The representation-use gap has a mirror image. A model can use a property even when the analyst’s favourite probe cannot recover it. Suppose “inside versus outside” is represented by whether a point lies within a ring. A linear probe draws one straight line and fails. A small nonlinear decoder succeeds. If the next network layer also computes a curved boundary, the property may be causally active despite poor linear decodability.

This prevents a common overcorrection. It would be wrong to replace “high probe accuracy proves use” with “low probe accuracy proves absence”. Probe failure may mean that the property is distributed, nonlinear, local to a subset of contexts, encoded in interactions between tokens, or expressed in a basis the probe family does not reach. It may also mean the property genuinely is absent. The experiment must distinguish these possibilities.

Evidence for simple linear structure is strongest when three facts coincide: the same direction decodes across varied contexts, an intervention along that direction changes the target property, and downstream effects follow a specific prediction. Work on sentiment in language models provides an example of this combined strategy, using both a broadly decodable direction and causal interventions rather than relying on classification alone.[13] Even there, the conclusion remains scoped to the tested models, layers, tasks and intervention design.

Linearity is an empirical convenience, not a metaphysical claim about concepts. It says that a family of contrasts can be handled by linear operations in a specified representation. A concept can have categorical structure, context-sensitive boundaries or several local directions. Conversely, a clean linear axis can arise from a dataset shortcut. The right question is not whether the latent space contains a perfect human taxonomy. It is whether the proposed geometry predicts how the frozen system changes under controlled counterfactuals.

What a probe actually establishes

Let z be a property such as grammatical number, sentiment or parcel fragility. Let h be the hidden state. A linear probe learns a vector w_p and predicts:

High held-out accuracy means that labels are recoverable from h by this probe family on this distribution. Alain and Bengio introduced linear classifier probes as layer-wise diagnostics, and the method remains useful precisely because the probe is trained separately from the target model.[4]

Yet four routes can produce the same high score. The model may actively use the property. The property may correlate with another feature the model uses. Upstream computation may leave a recoverable trace that downstream computation ignores. Or the probe may have enough capacity to reconstruct the task from incidental detail.

Probe accuracy is evidence of geometric availability, not causal occupancy. Availability means an analyst can recover the property. Occupancy means the model’s endogenous computation travels through a representation of that property in producing the behaviour under study.

The question must name a site and a behaviour

“Does the network represent number?” is too broad to test. Number might refer to a noun’s grammatical feature, the count of objects in an image, an arithmetic magnitude, or a token’s position. It might be expressed at one token, pooled across a sequence, or reconstructed only by several layers. It might influence agreement while playing no role in topic classification.

A disciplined claim therefore has five coordinates: the property, the system boundary, the internal site, the input distribution and the downstream behaviour. For example: “At the final subject token in layers four to six, a linearly accessible singular-plural subspace contributes to the model’s choice of the next verb on agreement templates.” This statement can be wrong in several informative ways. The subspace may decode but not contribute. It may contribute only at another token. It may work on familiar nouns and fail on nonce words. It may change verb number by damaging lexical identity.

The specificity is not bureaucratic. It determines the counterfactual. Without a named behaviour, an intervention effect cannot be interpreted. Without a named site, a failed patch cannot lower the right claim. Without a distribution, transfer and failure remain invisible. A latent claim becomes scientific when its nouns identify variables and its verbs identify interventions.

Control the instrument before interpreting the map

A probe can learn too much. Hewitt and Liang therefore proposed control tasks that pair inputs with random labels. A probe is selective when it performs well on the real property but poorly on the control, which helps distinguish structure in the representation from memorisation by the probe.[5] Minimum-description-length probing asks how much information and data are required to communicate the labels, not just whether asymptotic accuracy is high.[7]

These refinements strengthen an encoding claim. They do not turn it into a use claim. A low-capacity probe can still read a correlate that the target model ignores. Information-theoretic analyses likewise clarify what is available under a decoder, but they do not by themselves identify the downstream causal path.[6]

Decodability can survive destruction of the alleged mechanism

Consider a network with an invertible hidden layer. Because no information has been discarded, a powerful enough probe may recover almost any property of the input. Even a linear probe can succeed if the training distribution happens to align the property with a simple correlate. If the original network ignores that correlate, the probe’s success says little about mechanism.

This is why probe comparisons need explicit baselines: raw input, random features, untrained models, label permutations, matched probe capacity and distribution shift. A claim such as “layer seven contains tense” is underspecified. Compared with what? Under which decoder? With how much data? Does the direction transfer to new templates? Does removing it alter the model’s grammatical prediction?

stateDiagram-v2
  [*] --> Claim
  state "What an embedding buys" as Claim
  state "Three relations that are often<br/>collapsed" as Test
  state "Change one variable, reuse" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Patch before you proclaim

Amnesic probing removes linearly decodable information and asks whether task behaviour deteriorates, shifting attention from what is encoded to how information is used.[9] Activation patching runs a clean input and a matched corrupted input, then restores selected internal activations from the clean run into the corrupted run. Recovery of the clean behaviour localises a causal contribution under that intervention.[15] Interchange interventions go further by swapping candidate high-level variables between examples and comparing low-level and high-level counterfactual effects. Causal-abstraction work formalises when those effects align.[11]

Each method changes the question. Removal asks whether a subspace is necessary under a destructive operation. Patching asks whether a naturally occurring state can transmit an effect. Steering asks whether a direction is addressable. Interchange asks whether an interpretable variable and the network agree across a family of counterfactuals.

The strongest practical evidence comes from matched, in-distribution state swaps with predicted effects and measured collateral damage. This does not make the result infallible. It does make the causal claim sharper than a probe score.

The intervention has its own failure modes

A large effect can be misleading if the intervention damages many features. Removing a broad subspace may erase grammatical number, syntax and lexical identity together. Adding a large steering vector may push the state into a region the model never visits. Zero ablation can create an unnatural activation that resembles no valid input. Patching can mix states that are incompatible because they came from different token positions or contexts.

Recent evaluations frame this as a trade-off between completeness, changing the target property thoroughly, and selectivity, preserving non-target properties. Stronger interventions often improve the first while degrading the second.[16] Activation-patching results also depend on the corruption method and the metric used to score recovery.[14]

A probe-patch audit for real systems

The following audit treats a probe as a hypothesis generator. It asks whether the candidate geometry transfers, can be selectively changed, and carries the predicted effect through the frozen network.

Designing a counterfactual that changes one thing

The cleanest patch starts with a base case and a source case that differ in the target property while matching everything else that could drive the outcome. For grammatical number, “the pilot near the gates” and “the pilots near the gates” are better paired than unrelated sentences. For sentiment, two sentences should preserve topic, length and syntax while changing evaluation. For a visual concept, matched images should preserve pose, lighting and background where possible.

Perfect matching is rarely available. The protocol should therefore maintain a confound ledger. Record every property that differs between base and source, then probe or measure the most plausible rivals before and after patching. Use several source cases for each base case so that one idiosyncratic example cannot carry the result. Align token positions or spatial locations explicitly. When a patch crosses positions, explain why the receiving computation can interpret the transplanted state.

The intervention should also be compared with controls. Patch a random subspace of the same dimension. Patch a nearby layer. Swap states between examples that share the target property. Apply an equal-norm perturbation orthogonal to the candidate direction. A genuine target-specific effect should exceed these controls and should vary with the high-level counterfactual, rather than only with vector magnitude.

Finally, measure the result at more than the final label. Read back whether the target representation changed, whether rival representations moved, whether activation norms became unusual, and whether the output distribution changed in the predicted direction. This separates a causal chain from a generic loss of competence.

import numpy as np

def experiment(
    n: int = 10_000,
    representation_signal: float = 0.90,
    model_reliance: float = 0.08,
    seed: int = 7,
) -> dict[str, float]:
    """Compare linear decodability with causal output sensitivity."""
    if not 0.0 <= representation_signal <= 1.0:
        raise ValueError("representation_signal must be in [0, 1]")
    if not 0.0 <= model_reliance <= 1.0:
        raise ValueError("model_reliance must be in [0, 1]")

    rng = np.random.default_rng(seed)
    z = rng.choice([-1.0, 1.0], size=n)          # target property
    rival = rng.normal(size=n)                   # independent task feature
    noise = rng.normal(size=n)

    # h_property is increasingly decodable as representation_signal rises.
    h_property = (
        representation_signal * z
        + (1.0 - representation_signal) * noise
    )
    h = np.column_stack([h_property, rival])

    # A minimal linear probe. The sign of coordinate 0 is its prediction.
    probe_prediction = np.where(h[:, 0] >= 0.0, 1.0, -1.0)
    probe_accuracy = np.mean(probe_prediction == z)

    # The frozen model can rely on the property coordinate or the rival.
    logit = (
        model_reliance * h[:, 0]
        + (1.0 - model_reliance) * h[:, 1]
    )
    output = logit >= 0.0

    # Matched intervention: flip only the candidate property coordinate.
    h_patched = h.copy()
    h_patched[:, 0] *= -1.0
    patched_logit = (
        model_reliance * h_patched[:, 0]
        + (1.0 - model_reliance) * h_patched[:, 1]
    )
    patched_output = patched_logit >= 0.0
    output_flip_rate = np.mean(output != patched_output)

    return {
        "probe_accuracy": float(probe_accuracy),
        "output_flip_rate": float(output_flip_rate),
    }

for reliance in (0.0, 0.1, 0.5, 0.9):
    print(reliance, experiment(model_reliance=reliance))

Necessary, sufficient and contributory are different roles

An ablation asks whether a component is necessary under the model’s remaining ability to compensate. A patch or steering intervention can ask whether a state is sufficient to transmit some effect in a receiving context. Neither test alone establishes the full role. Redundant pathways can make an active feature look unnecessary. An unnatural injection can make an inactive direction look sufficient.

A more realistic category is contributory. The candidate subspace may shift a logit, constrain a later feature or bias a choice while several other paths cooperate. Measuring a continuous effect, such as logit difference or calibrated probability change, is often more informative than asking whether the top label flips. It reveals partial influence and opposing pathways. It also reduces the temptation to call one component “the concept”.

Mechanistic sufficiency should therefore be compositional. Identify the upstream conditions that write the state, the downstream components that read it, and the contexts in which the contribution matters. A feature that is causally active only when another gate is open is still real, but its role is conditional. The unit of explanation is often a routed interaction, not a solitary direction.

What a positive and negative result mean

A positive selective patch permits the claim that the candidate state difference contributes causally to the tested behaviour under the tested intervention and distribution. It does not prove that the property has one unique location, that the direction is globally linear, or that the model possesses the human concept in every relevant sense.

A negative patch has several interpretations. The property may be unused. The probe may target the wrong layer, basis or token. Redundant pathways may compensate. The patch may be too weak, or the matched examples may not isolate the property. A negative result therefore lowers a particular mechanism claim; it does not prove absence of every representation.