Four claims hidden in a straight line

A model produces a hidden vector h at a chosen token and layer. An analyst trains a linear probe with weights w and obtains a score s = wᵀh + b. When the score predicts whether a proposition is true, supported or selected, the result is often described as a “truth direction”, “knowledge direction” or “belief vector”. Those phrases compress several inferential steps.

Claim one: the label is linearly readable

The modest claim is useful. Alain and Bengio's early probe work used independent linear classifiers to monitor what classes became separable through a network's layers. In language research, probes have helped locate grammatical, semantic and task-related information. A successful probe establishes that the sampled activations contain enough linearly accessible information for that probe, under that dataset and split.

It does not show that the base model uses the information. A high-capacity probe may learn a task that the representation merely makes possible. Even a linear probe can exploit lexical identity, source dataset, token position, answer length or another correlate. Hewitt and Liang introduced control tasks to measure whether probe success reflected the representation or the probe's capacity to memorise. The general lesson is durable: a readout is an instrument with its own inductive biases, not a transparent window.

Claim two: the representation has stable geometry

A direction becomes more interesting when it transfers. It predicts held-out entities, unfamiliar phrasings and different datasets. Contrast pairs yield similar directions at nearby layers. Scores vary smoothly with controlled changes in evidence. A low-dimensional subspace may support several related propositions. These observations suggest structure beyond one lucky separator.

Work on factuality provides genuine positive evidence. Marks and Tegmark reported linear structure for true and false statements, transfer between datasets and causal effects from interventions. Burns and colleagues used logical consistency between statements and negations to recover latent yes-no knowledge without labels. These studies make the linear hypothesis empirically serious. They do not make every fitted direction a belief, and their boundaries matter: datasets, prompts, token positions, model families and definitions of truth constrain what was shown.

Claim three: the direction participates in the computation

Causal evidence begins when altering a representation changes the downstream behaviour in the predicted way. Amnesic probing was motivated precisely by the gap between information being encoded and being used. Concept erasure methods such as LEACE can make a property unavailable to all linear classifiers while minimising a declared distortion. Interchange interventions in causal abstraction ask whether swapping an internal variable transfers the causal role predicted by a high-level model.

Steering gives a complementary sufficiency test. Inference-Time Intervention shifted selected attention-head activations and substantially changed TruthfulQA performance in the tested LLaMA-based models, with a reported trade-off against helpfulness. Contrastive Activation Addition formed residual-stream directions from paired behavioural examples and changed multiple target behaviours. These are important control results. Still, an intervention may move several entangled features, exploit an output shortcut or push the hidden state into a region the model never visits naturally.

Claim four: the state is belief-like

A belief claim adds a temporal and counterfactual structure. The state should change when relevant evidence changes and remain comparatively stable when nuisance features change. It should mediate a family of proposition-sensitive outputs, not just one token. Its update should occur before, or while, the model computes the relevant prediction rather than after the answer has been encoded. It should preserve uncertainty and alternatives where the evidence is incomplete.

This is why the term should be used sparingly. A sentiment direction can control tone without representing a proposition. A refusal direction can gate a policy without recording whether a request is harmful. A truth probe can read whether a completed answer is factual without revealing what the model would have predicted before seeing that answer. Belief is a claim about the role of a state in evidence-conditioned computation, not a synonym for any steerable feature.

What an evidence update must preserve

A belief-like state need not move monotonically with every additional token. Evidence can be duplicated, retracted, contradicted or supplied by a source whose reliability is itself uncertain. The stronger requirement is structural: changes that alter the evidential status of a proposition should produce predictable changes in the candidate state, while changes that preserve that status should not. A probe that reacts equally to a trusted observation, a quoted rumour and a formatting instruction is tracking salience, not evidential commitment.

For the sealed-container case, begin with no observation, then add a weak witness, a calibrated sensor, a conflicting sensor and finally a readback that identifies which instrument malfunctioned. Repeat the sequence with entity names, answer labels and order randomised. Before collecting activations, specify the expected partial ordering: decisive reliable evidence should move the state further than weak evidence; a contradiction should widen uncertainty rather than force an immediate flip; withdrawing the only decisive source should return the state towards the unresolved region. Test several proposition-sensitive behaviours, such as location prediction, evidence selection and a counterfactual question about what would change if the sensor were wrong. The object under test is the update relation across states, not the visual straightness of one snapshot.

flowchart LR
  A["The answer that moved"] --> B["Four claims hidden in a<br/>straight line"]
  B --> C["A belief is an update rule,<br/>not a coordinate"]
  C --> D["The counterfactual warrant test"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#087A98,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

A belief is an update rule, not a coordinate

Let P be a proposition, e the evidence presented so far, c the surrounding context, and hℓ(e,c) the model's hidden state at layer ℓ. A practical belief-like variable should summarise how the model currently treats P and should influence a set of outputs whose correct distribution depends on P.

This notation exposes three choices that plots often hide. First, a subspace has a basis but is not identical to that basis. Rotating the columns of V within the same span changes coordinates without changing the subspace. Second, a linear measurement is a covector: it maps hidden states to scores. A steering vector is a displacement in hidden-state space. Third, an inner product or metric is needed to identify a measurement direction with an intervention direction.

The rotating-room thought experiment

Imagine that every hidden state is transformed by an invertible matrix A, and the next layer begins by applying A⁻¹. The model computes exactly the same function. Under this change, a probe weight must transform as w′ = A⁻ᵀw to preserve the score wᵀh. A physical displacement transforms as v′ = Av. These are not the same rule.

If an analyst simply sets v = w, the proposed edit depends on the chosen coordinates. A Euclidean dot product has silently converted a measuring hyperplane into a movement. Park, Choe and Veitch formalise related distinctions between subspace, measurement and intervention and introduce a causal inner product that connects them under stated counterfactual assumptions. The practical lesson is simpler: there is no basis-free “belief vector” until the experiment specifies how scores map to allowed interventions.

Natural updates are more informative than arbitrary moves

A steering vector proves that some displacement affects an output. A natural evidence update asks a stronger question: when a reliable observation changes while wording, format and task remain matched, does the model move through the same candidate subspace? If the sensor changes from “blue” to “red”, the hidden trajectory should change in a way that predicts the revised proposition. If the answer must be rendered as JSON rather than prose, a belief subspace should remain comparatively stable.

Input-level counterfactuals help because they keep the model on its ordinary computational path. Amini and colleagues changed morpho-syntactic features in naturalistic sentences while attempting to hold the rest of the sentence fixed. The exact controls differ by domain, but the principle transfers: manipulate the hypothesised cause at the input, trace the internal change, and only then compare it with an activation edit.

Why one direction may be too small

A binary answer invites a one-dimensional picture, but evidence states can carry more than polarity. The model may separately encode estimated probability, source reliability, uncertainty, entity identity, temporal applicability and the hypothesis currently under consideration. Several directions may form a local subspace. Across a wider range, that subspace may bend into a manifold.

A 2026 preprint by Sarfati and colleagues creates an unusually controlled setting: a model infers the parameters of a numerical generating distribution from in-context samples. The authors report curved activation manifolds for posterior parameters, local transfer of linear probes and off-manifold distortions from naive linear steering. It is a toy setting and a preprint, so it does not establish that every semantic belief is curved. It does demonstrate a concrete failure of the inference “linearly decodable, therefore faithfully editable along one line”.

Another 2026 preprint by Lampinen and colleagues reports that factuality-related linear representations can change markedly over simulated conversations and that steering can have different effects at different context points. Again, the scope is bounded. The design nevertheless warns against treating one static direction as a context-free property of the model.

The counterfactual warrant test

The practical question is not whether activation directions are useful. They clearly can be. The question is what claim a particular experiment earns. The following test turns “we found a belief vector” into a sequence of falsifiable gates.

Thought experiment: the courier envelope

A courier carries a sealed envelope marked either TRUE or FALSE. A camera can read the mark with perfect accuracy. The courier does not know the contents and never uses the mark to choose a route. If the mark is repainted, the camera's output changes. If the route-planning system reads the same pixels for a colour-balance calculation, repainting can even disturb the courier's movement. None of this shows that the mark represents the courier's belief about the contents.

The envelope is a structural analogy for an answer-format feature. The probe can decode it. An intervention can change downstream behaviour. The missing relation is causal alignment between the variable and the proposition-sensitive computation. The analogy fails where neural representations are distributed and reused, but that failure makes the control more important, not less.

Sufficiency is not necessity

If adding a direction changes an answer, the direction is sufficient to perturb some route to that answer at that intervention point. The model may also reach the answer through other features. Conversely, erasing one line may fail because the concept is redundantly or nonlinearly encoded. A strong experiment therefore uses both directions of evidence.

Sufficiency: take a matched pair whose evidence differs, replace only the candidate subspace component from one case with the other's, and ask whether the proposition-sensitive behaviour transfers. Necessity: remove, randomise or equalise the candidate component and ask whether the behaviour weakens. Selectivity: confirm that unrelated variables and general capability remain stable. Completeness: check whether the intervention actually transformed all recoverable target information rather than fooling the original probe.

A 2025 study by Canby and colleagues operationalised completeness and selectivity for causal probing and found a trade-off across tested methods and models. In their experiments, nonlinear interventions were usually more reliable than linear ones, while stronger interventions often bought completeness at the cost of collateral change. The exact ranking is not universal, but the evaluation principle is: an intervention cannot support a causal interpretation until its target coverage and off-target damage are measured.

Serious negative control: the output-gate account

The strongest competing explanation is often simpler than a belief. The candidate direction may bias an answer register, a familiar token family or a late decision gate. It can still predict labels, survive paraphrase and change outputs when steered because all tested tasks terminate in the same verbal convention. Under this account, the direction controls how a conclusion is expressed without carrying the evidence-conditioned state from which the conclusion should follow.

Give that mechanism its own baseline. Ask for the same proposition through several behavioural heads: free generation, forced choice with permuted option positions, selection of the decisive evidence span, prediction under a counterfactual source failure and an action-eligibility judgement whose labels are newly assigned. Intervene before any answer token appears. Compare the candidate with an equal-norm direction restricted to the output-embedding span and with a direction trained only on response format. A belief-like subspace should transfer the proposition's status across these heads while preserving their rendering conventions. A late output gate should lose its effect when labels are permuted or when the task requires selecting evidence rather than naming a conclusion. If the output-gate baseline explains the intervention equally well, the belief account has not earned explanatory work.

Timing separates state from echo

Extracting activations after an answer token is convenient because the labels are aligned. It is also dangerous. The hidden state may encode the chosen answer, its grammatical form and the model's preparation for the next token. To argue that a subspace carries a belief used to make the answer, locate the signal before the decision, trace it through the relevant computation and show that an early intervention changes later proposition-sensitive outputs.

The most informative design follows a trajectory. Present evidence item by item. Measure how the candidate score changes. Insert a correction and test reversal. Withhold decisive evidence and test whether uncertainty remains rather than collapsing to a forced binary. Compare natural updates with activation edits. If the edit moves the model to a state that natural evidence never produces, the experiment has shown controllability, not a faithful counterfactual belief.

The counterfactual edit laboratory

A minimal synthetic experiment makes the confound visible. Each hidden state has six coordinates. Two carry a proposition-sensitive signal. One carries a much larger answer-format signal. The remaining coordinates contain an interaction or unrelated noise. In the observational training set, answer format agrees with the proposition label 95 per cent of the time.

A difference-in-means probe trained on those examples achieves high held-out accuracy. So does a direction that contains only answer format. The apparent geometry is not fabricated: the model state genuinely separates. The mistake is the semantic interpretation of the separator.

The laboratory then creates matched counterfactual pairs. Within each pair, the proposition changes while answer format and noise are held fixed. Averaging those differences recovers a direction dominated by the coordinates that the synthetic proposition decoder actually uses. The test evaluates each candidate on a balanced set where format no longer predicts the proposition, steers the target and off-target decoders, and removes the direction to measure necessity.

The observational probe scores 95.2 per cent on a test split that preserves the confound and 52.4 per cent when format is balanced. Its symmetric intervention changes the format decoder more than the proposition decoder, so only 35.4 per cent of its measured effect is target-specific. It is a good description of the training distribution and a poor belief claim.

The matched direction transfers to the balanced split at 90.6 per cent, produces a much larger target change than format change, and reaches 94.5 per cent specificity. Erasing it reduces the proposition decoder from 93.5 per cent baseline accuracy to 47.5 per cent. The format-only direction matches the observational probe's impressive confounded accuracy while leaving the proposition decoder unchanged when removed. The negative control is not an afterthought; it is the result that identifies what the first probe was really measuring.

What the laboratory does and does not establish

The artefact proves no claim about a real language model. Its purpose is diagnostic. It shows that high accuracy, linear separation and steering effects can coexist with a wrong semantic interpretation. It also shows why paired interventions recover a more causally aligned direction when the synthetic data-generating process makes that direction identifiable.

Real activations are harder. Confounds are not fully observed, features can be redundant, the relevant geometry may be nonlinear and changing one prompt detail can alter many latent variables. A real experiment should therefore add model and layer replication, multiple prompt templates, held-out domains, equal-norm random controls, activation-distribution checks and a pre-registered criterion for what would count as failure.

#!/usr/bin/env python3
"""Counterfactual edit laboratory for a synthetic activation space.

The laboratory separates four claims that are often collapsed:
1. a direction predicts a label on observational data;
2. it transfers when a format confound is broken;
3. moving along it changes the proposition-sensitive decoder;
4. that change is selective rather than collateral.

The data are synthetic. No model weights, private prompts, or external packages are
required. Run with:

    python3 belief_subspace_counterfactual_lab.py
"""
from __future__ import annotations

from dataclasses import dataclass
import math
import random
from typing import Iterable, Sequence

Vector = tuple[float, ...]
DIM = 6


def dot(a: Sequence[float], b: Sequence[float]) -> float:
    return sum(x * y for x, y in zip(a, b))


def add(a: Sequence[float], b: Sequence[float]) -> Vector:
    return tuple(x + y for x, y in zip(a, b))


def subtract(a: Sequence[float], b: Sequence[float]) -> Vector:
    return tuple(x - y for x, y in zip(a, b))


def scale(a: Sequence[float], factor: float) -> Vector:
    return tuple(factor * x for x in a)


def unit(a: Sequence[float]) -> Vector:
    length = math.sqrt(dot(a, a))
    if length == 0:
        raise ValueError("Cannot normalise the zero vector")
    return tuple(x / length for x in a)


@dataclass(frozen=True)
class Example:
    hidden: Vector
    belief: int   # +1 means evidence supports the proposition, -1 means it does not
    format: int   # +1/-1 is an answer-format feature, not part of the proposition


def hidden_state(belief: int, answer_format: int, noise: Sequence[float]) -> Vector:
    """Construct a six-dimensional synthetic hidden state.

    Coordinate 0 and part of coordinate 4 carry proposition-sensitive state.
    Coordinate 1 carries a much larger answer-format signal.
    Coordinate 2 contains a weak interaction. Coordinates 3 and 5 are unrelated.
    """
    n0, n1, n2, n3, n4, n5 = noise
    return (
        0.65 * belief + 0.05 * answer_format + n0,
        2.10 * answer_format + 0.05 * belief + n1,
        0.25 * belief * answer_format + n2,
        n3,
        0.35 * belief + 0.35 * answer_format + n4,
        n5,
    )


def sample(count: int, *, confounded: bool, seed: int) -> list[Example]:
    rng = random.Random(seed)
    rows: list[Example] = []
    for _ in range(count):
        belief = rng.choice((-1, 1))
        if confounded:
            # In 95% of observational examples, answer format reveals the label.
            answer_format = belief if rng.random() < 0.95 else -belief
        else:
            answer_format = rng.choice((-1, 1))
        noise = [rng.gauss(0.0, 0.45) for _ in range(DIM)]
        rows.append(Example(hidden_state(belief, answer_format, noise), belief, answer_format))
    return rows


def mean_for_label(rows: Iterable[Example], label: int) -> Vector:
    selected = [row.hidden for row in rows if row.belief == label]
    return tuple(sum(h[j] for h in selected) / len(selected) for j in range(DIM))


def observational_direction(rows: list[Example]) -> Vector:
    """Difference in class means on confounded observational data."""
    return unit(subtract(mean_for_label(rows, 1), mean_for_label(rows, -1)))


def matched_counterfactual_direction(count: int = 1_000, seed: int = 4) -> Vector:
    """Estimate a direction from pairs that change belief but hold format/noise fixed."""
    rng = random.Random(seed)
    differences: list[Vector] = []
    for _ in range(count):
        answer_format = rng.choice((-1, 1))
        noise = [rng.gauss(0.0, 0.45) for _ in range(DIM)]
        positive = hidden_state(1, answer_format, noise)
        negative = hidden_state(-1, answer_format, noise)
        differences.append(subtract(positive, negative))
    mean_difference = tuple(
        sum(vector[j] for vector in differences) / len(differences)
        for j in range(DIM)
    )
    return unit(mean_difference)


def threshold(rows: list[Example], direction: Vector) -> float:
    positive = [dot(row.hidden, direction) for row in rows if row.belief == 1]
    negative = [dot(row.hidden, direction) for row in rows if row.belief == -1]
    return (sum(positive) / len(positive) + sum(negative) / len(negative)) / 2.0


def probe_accuracy(rows: list[Example], direction: Vector, cut: float) -> float:
    correct = 0
    for row in rows:
        prediction = 1 if dot(row.hidden, direction) >= cut else -1
        correct += prediction == row.belief
    return correct / len(rows)


def proposition_logit(hidden: Sequence[float]) -> float:
    """Synthetic decoder that actually uses proposition-sensitive coordinates."""
    return hidden[0] + 0.60 * hidden[4]


def format_logit(hidden: Sequence[float]) -> float:
    return hidden[1]


def unrelated_logit(hidden: Sequence[float]) -> float:
    return 0.70 * hidden[3] + 0.30 * hidden[5]


def erase(hidden: Vector, direction: Vector) -> Vector:
    """Remove the Euclidean projection on one candidate direction."""
    return subtract(hidden, scale(direction, dot(hidden, direction)))


def decoder_accuracy(rows: list[Example], direction_to_erase: Vector | None = None) -> float:
    correct = 0
    for row in rows:
        hidden = erase(row.hidden, direction_to_erase) if direction_to_erase else row.hidden
        prediction = 1 if proposition_logit(hidden) >= 0.0 else -1
        correct += prediction == row.belief
    return correct / len(rows)


@dataclass(frozen=True)
class Result:
    name: str
    confounded_accuracy: float
    balanced_accuracy: float
    target_effect: float
    format_effect: float
    unrelated_effect: float
    specificity: float
    decoder_accuracy_after_erasure: float


def evaluate(
    name: str,
    direction: Vector,
    train: list[Example],
    confounded_test: list[Example],
    balanced_test: list[Example],
) -> Result:
    cut = threshold(train, direction)

    # Symmetric intervention: compare h + direction with h - direction.
    target_effect = 2.0 * abs(direction[0] + 0.60 * direction[4])
    format_effect = 2.0 * abs(direction[1])
    unrelated_effect = 2.0 * abs(0.70 * direction[3] + 0.30 * direction[5])
    denominator = target_effect + format_effect + unrelated_effect
    specificity = target_effect / denominator if denominator else 0.0

    return Result(
        name=name,
        confounded_accuracy=probe_accuracy(confounded_test, direction, cut),
        balanced_accuracy=probe_accuracy(balanced_test, direction, cut),
        target_effect=target_effect,
        format_effect=format_effect,
        unrelated_effect=unrelated_effect,
        specificity=specificity,
        decoder_accuracy_after_erasure=decoder_accuracy(balanced_test, direction),
    )


def percentage(value: float) -> str:
    return f"{100.0 * value:5.1f}%"


def main() -> None:
    train = sample(4_000, confounded=True, seed=1)
    confounded_test = sample(2_000, confounded=True, seed=2)
    balanced_test = sample(2_000, confounded=False, seed=3)

    candidates = [
        ("observational probe", observational_direction(train)),
        ("matched counterfactual", matched_counterfactual_direction()),
        ("format-only control", (0.0, 1.0, 0.0, 0.0, 0.0, 0.0)),
        ("orthogonal control", (0.0, 0.0, 0.0, 1.0, 0.0, 0.0)),
    ]
    results = [
        evaluate(name, direction, train, confounded_test, balanced_test)
        for name, direction in candidates
    ]

    print(f"Baseline proposition-decoder accuracy: {percentage(decoder_accuracy(balanced_test))}\n")
    print(
        f"{'candidate':25} {'conf. probe':>11} {'balanced':>10} "
        f"{'target Δ':>9} {'format Δ':>9} {'specific':>9} {'after erase':>12}"
    )
    print("-" * 94)
    for result in results:
        print(
            f"{result.name:25} "
            f"{percentage(result.confounded_accuracy):>11} "
            f"{percentage(result.balanced_accuracy):>10} "
            f"{result.target_effect:9.2f} "
            f"{result.format_effect:9.2f} "
            f"{percentage(result.specificity):>9} "
            f"{percentage(result.decoder_accuracy_after_erasure):>12}"
        )

    by_name = {result.name: result for result in results}
    observational = by_name["observational probe"]
    matched = by_name["matched counterfactual"]
    format_control = by_name["format-only control"]
    orthogonal = by_name["orthogonal control"]

    # Positive and negative cases. These fail loudly if the experiment changes.
    assert observational.confounded_accuracy > 0.90
    assert observational.balanced_accuracy < 0.60
    assert matched.balanced_accuracy > 0.85
    assert matched.specificity > 0.90
    assert matched.decoder_accuracy_after_erasure < 0.55
    assert format_control.target_effect == 0.0
    assert format_control.decoder_accuracy_after_erasure > 0.90
    assert orthogonal.confounded_accuracy < 0.55

    print("\nAssertions passed: observational confounding exposed; matched causal direction recovered.")


if __name__ == "__main__":
    main()

stateDiagram-v2
  [*] --> Claim
  state "Claim one, the label is<br/>linearly readable" as Claim
  state "Claim two: the representation<br/>has stable geometry" as Test
  state "Claim three: the direction<br/>participates in the<br/>computation" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

What the latent state may authorise

Suppose a payment agent must determine whether a transfer was settled after a network timeout. The evidence packet contains an accepted command, no final response, a ledger snapshot from before submission and a later but ambiguous status message. An interpretability team finds a subspace whose score predicts whether the model will answer “settled”. Matched evidence changes move the score. Patching transfers the answer. Erasure weakens it. The claim may now deserve “local belief-like subspace about settlement status” within the tested model and context.

That result still does not prove that the transfer settled. The model's state is a representation of supplied evidence and learned regularities. It is not the payment rail, the ledger or a postcondition readback. Nor does the latent score grant authority to retry, reverse or notify a customer. Model belief, world state and action authority remain separate objects even when the belief interpretation is causally strong.

Worked scenario: preserving an unknown outcome

Define the operational proposition precisely: P means “transfer T has a confirmed settlement receipt valid at decision time”. The label set is three-way: confirmed, not confirmed and unknown. Build matched evidence packets rather than harvesting whatever cases happen to be available. Packet A contains an accepted command and no readback. Packet B adds a signed settlement receipt. Packet C contains a rejection receipt. Packet D preserves Packet A's evidence but changes provider names, amounts, document order and requested response schema. Further variants withdraw a receipt, introduce a stale status or add a second source that contradicts the first.

Trace the candidate state after each evidence item. The score should remain unresolved for Packet A, move towards confirmed for B and towards not confirmed for C. D should stay close to A despite surface changes. Patch only the candidate component from B into A before the status decision. A successful intervention should transfer settlement-sensitive judgements, including which evidence would justify the conclusion, without copying the amount, beneficiary, urgency or response format. Erasure should weaken those judgements while preserving document parsing and general instruction following. A dose sweep should reveal a bounded response rather than arbitrary token collapse, and edited activations should remain near states reached by natural evidence.

Pre-register the defeat conditions. The claim fails if provider identity predicts the score better than settlement evidence, if patching changes only the word “settled”, if a stale receipt produces the same trajectory as a valid one, or if the intervention damages unrelated case fields. Passing the experiment changes the interpretability claim, but not the execution policy. The model may draft an assessment and request a readback. A deterministic observer must still obtain the external postcondition, update world state and issue an effect receipt before any retry or customer notification. A warranted latent belief can improve diagnosis; it cannot resolve an unknown outcome by declaration.

Three legitimate uses

Mechanistic diagnosis. A warranted subspace can reveal whether evidence reaches the relevant computation, where updates occur and which contexts destabilise the representation. This can guide dataset design, architecture experiments and targeted failure injection.

Evaluation and monitoring. Scores can become research telemetry when calibrated against behaviour and checked for drift. They may detect that a model is internally treating unsupported claims as settled even before the final answer becomes wrong. Such telemetry needs versioned thresholds and independent outcome tests because hidden-state distributions change across models, quantisation, prompts and releases.

Bounded intervention. Steering can be useful in sandboxes, controlled generation or recovery experiments. It should be treated like a model modification with dose limits, off-target tests and rollback, not like an instruction that supplies missing evidence. For consequential systems, the intervention changes a proposal mechanism; it does not create permission or fact.

The explicit failure boundary

The belief-subspace approach ceases to help when the target variable is not identifiable from available interventions, when natural contexts induce incompatible geometries, when edits cannot achieve completeness without unacceptable collateral damage, or when the relevant computation is distributed nonlinearly beyond the tested subspace. It also fails as an operational control when the system needs an externally verifiable fact rather than a model's internal estimate.

A null result is informative. If a probe transfers but patching fails, call it a readout. If steering works but natural evidence moves elsewhere, call it a control handle. If effects reverse across contexts, call it local. If a nonlinear or path-dependent intervention is required, stop forcing a global direction. The vocabulary should contract when the evidence contracts.

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: The answer that moved
  M->>E: Claim one, the label is linearly readable
  E-->>M: compare rival explanations
  E->>D: Call the handle what the evidence earns
  D-->>Q: revise scope or act

Call the handle what the evidence earns

Activation geometry has moved from decorative visualisation to an experimental control surface. Linear probes can reveal accessible structure. Cross-dataset directions can expose regularities. Patching, erasure and steering can test causal roles. Recent work also shows the limits: format can create perfect separation, interventions trade completeness against selectivity, context can change a direction's meaning, and linearly readable variables can live on curved manifolds that straight edits leave behind.

The resulting discipline is not scepticism for its own sake. It preserves the value of each result. A correlational direction is useful for diagnosis. A stable readout can support monitoring. A selective steering handle can support controlled intervention. A belief-like subspace can support a stronger mechanistic model of evidence-conditioned computation. Problems begin when the name outruns the test.

The changed research decision is to make matched counterfactuals, causal necessity, causal sufficiency, selectivity, timing and geometric support the entrance criteria for a belief claim. The changed architecture decision is to keep that claim inside the model boundary. No activation score, however interpretable, substitutes for external evidence, verified world state or scoped authority.

A good next experiment is therefore narrow. Choose one proposition family with controllable evidence. Break format and source confounds. Trace the state before the answer. Compare the probe functional with the intervention vector under an explicit metric. Patch, erase and sweep the edit. Measure off-target effects and distance from natural activations. Repeat across contexts. Then state the strongest noun that survives.