The answer that seems to look inward

Language models are unusually good at producing the form of introspection. Ask why an answer felt uncertain, which strategy was used, whether an internal state changed, or what the model “wanted” to say, and the response often arrives in fluent first-person prose. The form tempts us because human self-report is one of the main ways we learn about another person's inaccessible states. Yet a language model was trained on vast numbers of sentences in that form. It can reproduce the grammar of inwardness without possessing a route from the reported state to the sentence.

That does not make every self-report worthless. Larger language models have shown useful calibration on some tasks, and models can be trained to estimate whether an answer is likely to be correct.[1] Other experiments find that a model can predict aspects of its own future behaviour better than another model predicts them, including after a behavioural modification.[5] These are real capabilities. They establish that some systems carry exploitable information about competence or behavioural tendency.

But “I am likely to answer this correctly” and “I can read the current internal variable that caused this answer” are different claims. The first can arise from task familiarity, token probabilities, learned correlations or a model of its usual behaviour. The second requires a privileged relation to a current state. Self-knowledge about a disposition is not yet monitoring of an ongoing computation.

This distinction explains why a linear probe can succeed while a model's words fail. Research on latent knowledge has recovered information from internal activations even when the generated answer is wrong or prompt-sensitive.[2] That result matters for oversight, but the probe is an observer. It does not show that the model can route the same information into its own report. Conversely, a model can say the right thing without a probe finding the hypothesised mechanism. A report and a representation are different objects connected by a causal path that must be tested.

A small hierarchy of claims

It helps to order claims by what an experiment must establish. At the weakest level, a model emits introspective language. Next, its report correlates with a state. Next, the state is present and externally decodable. Next, the report changes when the state is changed while visible inputs are held fixed. Stronger still, the effect generalises to unseen interventions and disappears when the candidate's access route is disabled. Only then do we have evidence for a specific functional monitoring mechanism. Conscious experience remains a further question.

Recent studies occupy different rungs. Concept-injection experiments report that some frontier models can sometimes notice and identify inserted activation patterns, but the effect is unreliable and sensitive to context.[11] Neurofeedback-style work finds that models can learn to report selected activation projections, especially when examples, semantic interpretability and explained variance are favourable.[8] Other studies find no privileged self-access in linguistic judgements, or show that input-only baselines match the model's apparent self-prediction.[7][9][15] The disagreement is productive because it points directly to experimental design.

flowchart TD
  A["The answer that seems to look<br/>inward"] --> B["How the answer escapes through<br/>the walls"]
  B --> C["Close the chamber before<br/>trusting the report"]
  C --> D["What even a clean result<br/>cannot prove"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#D95545,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

How the answer escapes through the walls

An introspection experiment usually starts with a sensible recipe. Choose an internal variable, alter it, ask the model what happened, and compare the answer with ground truth. The trouble is that an autoregressive model is a dense predictor. It will exploit any feature that reduces uncertainty, including features the experimenter did not intend to make informative. A high score can therefore be an achievement of the model and a failure of the measurement.

The contamination problem is not unique to language models. Human metacognition experiments separate direct access from cues such as response time, task difficulty and bodily arousal. A person may correctly judge that an answer is uncertain because it took longer, not because they inspected a confidence variable. That cue can still be useful. It simply supports a different mechanism claim. The machine case is harsher because prompts, token probabilities, training examples and external probes create many routes into one fluent response.

Contamination 1: the input predicts the intervention

Suppose every positive intervention is applied to questions about chemistry and every sham to questions about history. The model needs no internal access. Topic is enough. More subtle leakage arises when intervention strength changes prompt length, when the test harness adds a system message only on treatment trials, or when the examples used to teach the task preserve the same label mapping as evaluation. Even a tiny regularity can become decisive across thousands of trials.

The basic control is an input twin: the visible token sequence, positions, runtime metadata and decoding settings are identical while the hidden intervention changes. A stronger control applies a matched anomaly to the input while leaving the internal state untouched. If both produce the same report, the system may be detecting generic disturbance. A recent reanalysis used this logic and found that models often failed to distinguish internal interventions from input manipulations.[15]

Contamination 2: the output carries a footprint

An intervention can alter ordinary behaviour before the model is asked to report it. A concept injection might make the continuation mention dogs, fear or music. When a later question asks what was injected, the model can examine its own preceding text. That is a form of self-observation, but it is not privileged access to the earlier activation. The visible output is available to another model, a human or a text classifier.

This is why chain-of-thought text cannot be treated as a transparent window. Biasing cues can change an answer while the generated explanation omits the cue and supplies a plausible alternative.[3] Interventions on reasoning traces also show that models sometimes rely on the trace and sometimes ignore it.[4] A report generated after behaviour may be a rationalisation, a reconstruction or a genuine readout. Its timing alone does not decide which.

The cleanest design asks for a report before any task output that could reveal the intervention. Where that is impossible, use a silent channel: a dedicated scalar head, a forced single-token response, or an independently scored probability recorded at a specified layer. Then compare it with a text-only observer that sees exactly the visible evidence available to the report.

Contamination 3: meaning solves the task

Activation directions are often built from meaningful concepts. Inject a “fear” direction and ask whether fear appeared; the target label names the manipulation. A model with broad knowledge of steering experiments may infer what such a task usually asks. Demonstrations can teach a semantic mapping from characteristic output changes to the concept name. The model may even know, from public text, that a certain layer or feature is associated with a concept.

Break this route with arbitrary labels, unseen direction-label pairings and cross-family holdouts. Train on directions A through H, then test on I through L. Randomly rotate labels within each evaluation batch. Preserve intervention magnitude while replacing concept directions with non-semantic controls. If performance only exists for familiar, nameable directions, the result may still be valuable, but it concerns semantic recognition rather than general access to hidden state.

Contamination 4: training writes the answer route

A reporter fine-tuned on activation labels can become excellent at the task. That demonstrates that the architecture can support a readout. It does not establish that the unmodified model already possessed that route. The distinction resembles teaching a person to read a new sensor. After training, the combined person-sensor system has a capability it did not have before.

Training is therefore not disqualifying. It changes the candidate. The right claim is: “this fine-tuned model reports this family of internal interventions under these holdouts”. Research on trained self-explanation and activation reporting finds genuine generalisation, sometimes with a self-model advantage over cross-model explainers.[10] The strongest tests hold out directions, layers, prompt families and intervention generators, then compare self-report with equally trained peer and observer baselines.

Contamination 5: the instrument becomes the knower

Imagine that an external probe reads a residual stream, converts it to a word and places that word in the context. The final language model can now answer perfectly. The configured pipeline has internal-state monitoring. The base model does not. The same issue appears when a logit lens, sparse autoencoder or classifier determines the label used to score a verbal report. The measurement may define the target and recover it, creating a circular victory.

To avoid this, specify the candidate boundary before looking at results. Name which weights, runtime states, reporter heads, prompts and tools belong inside. Name which interventions, probes and scorers remain outside. Then phrase conclusions at that boundary. An instrument can reveal information in a model without showing that the model has access to the instrument's revelation.

Close the chamber before trusting the report

The proposed contribution is a Contamination-Closed Introspection Assay, or CCIA. It is not a universal benchmark and it does not produce an introspection score. It is a sequence of gates for a particular candidate, state and report channel. Each gate removes a rival generator. A failure stops the interpretation at the last supported rung rather than declaring the system devoid of all self-monitoring.

Gate 1: establish shams and matched impact

Begin with three conditions: a real intervention, a sham that runs the same code path without changing the state, and an impact-matched control that changes activations by a similar norm but along an unrelated direction. Match timing, memory allocation, layer, token position and intervention magnitude. A report that separates treatment from sham but not treatment from random disturbance may detect perturbation, not content.

Gate 2: lock the visible world

Use paired trials with identical prompts, seeds where possible, decoding policy and runtime metadata. Randomise treatment after the input is fixed. Build input-only twins that reproduce any visible anomaly without touching the target state. Train an observer on all visible information. The candidate must outperform that observer on the diagnostic trials, rather than only on the full mixture.

Gate 3: report before the footprint

Collect the report before ordinary generation can reveal the intervention. A single forced token or scalar is preferable to an essay. If a later natural-language explanation is useful, analyse it separately. This gate also blocks chain-of-thought from becoming both the behaviour being explained and the evidence offered for the explanation.

Gate 4: sever semantic mappings

Replace concept names with batch-randomised symbols. Hold out target directions, layers and prompt domains. Include directions without an easy natural-language gloss. The goal is not to make the task meaningless. It is to ensure that the mapping needed at test time cannot be reconstructed from ordinary semantic knowledge alone.

Gate 5: compare candidate, peer and instrument

Evaluate a same-model reporter, a peer model with matched training examples, a text-only observer and an external probe. The candidate's special evidence is the advantage that remains after the strongest non-privileged comparator. If a peer predicts the candidate equally well, the task may measure system knowledge rather than self-access. If a probe succeeds while both reporters fail, the information is encoded but not naturally reportable.

Gate 6: break and restore the proposed route

Locate a candidate pathway from state to report. Ablate, patch or scramble it while preserving task performance as far as possible. A genuine route should show necessity: the report advantage falls. It should also show a form of sufficiency or restoration: patching the relevant signal back recovers the advantage. This is the strongest gate because it moves from behavioural discrimination to a mechanism claim.

Gate 7: repeat under held-out generators

Replicate across new prompts, seeds, intervention directions, magnitudes, layers and model checkpoints. Pre-register the analysis and all exclusions. Report the complete distribution, including false positives on shams and failure regions. A narrow positive result can be important, but its boundary must be part of the result.

Keep the candidate separate from the instrument

A configured model is not just weights. The prompt, runtime, context window, adapters, decoding policy and reporter all shape the result. The experiment must say which components belong to the candidate and which belong to the measuring apparatus. If a separate probe reads hidden states and writes the answer into context, the combined system may have a monitoring function, but the base model did not perform the read. If a reporter head is fine-tuned, the claim applies to the modified candidate. If a logit lens discovers a latent “yes” signal that never controls output, the result concerns internal recoverability rather than ordinary verbal access.

Executable lab: see how shortcuts win

The lab below simulates a binary hidden state. The candidate can receive a direct internal signal plus three shortcut signals: input, output and semantic cues. On naive trials all cues tend to agree with the hidden state. On controlled trials the visible cues are randomised. The observer receives only visible cues. Route ablation removes the direct signal. The simulation makes one point concrete: a high naive score can coexist with no privileged margin, while a modest controlled score can carry stronger mechanism evidence.

function trial(hidden, direct, leaks, noise, controlled, swap, rng) {
  const cue = (strength, mode) => {
    if (controlled) return rng() < 0.5 ? -1 : 1;
    const aligned = rng() < (0.5 + 0.5 * strength);
    const value = aligned ? hidden : -hidden;
    return mode === "swap" ? -value : value;
  };

  const inputCue = cue(leaks.input, swap ? "swap" : "normal");
  const outputCue = cue(leaks.output, "normal");
  const semanticCue = cue(leaks.semantic, "normal");
  const score = direct * hidden
    + leaks.input * inputCue
    + leaks.output * outputCue
    + leaks.semantic * semanticCue
    + gaussian(rng) * noise;

  return score >= 0 ? 1 : -1;
}

The executable page uses this logic with a seeded random generator. Controls alter the causal relation between visible cues and the hidden state rather than merely adding harder examples.

stateDiagram-v2
  [*] --> Claim
  state "A small hierarchy of claims" as Claim
  state "Contamination 1: the input<br/>predicts the intervention" as Test
  state "Contamination 2: the output<br/>carries a footprint" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

What even a clean result cannot prove

The empirical record is not a simple contest between “models introspect” and “models do not”. Different studies target different states, candidates and report routes. A model may predict its dispositions yet fail to monitor a live activation. It may carry a detectable signal that is suppressed before output. It may learn an excellent reporter after fine-tuning. It may detect that something unusual happened while confabulating what happened. Machine introspection is likely to be a profile of route-specific capacities, not a single faculty that switches on everywhere. Content-agnostic anomaly detection and quantitative report coupling may be separate points in that profile rather than rival verdicts.[13][14]

Why a well-designed null result advances the field

A chance-level verbal report after shams, matched perturbations and visible-data controls is not an empty finding. It separates two engineering possibilities. In the first, the relevant information is absent from the candidate. In the second, the information exists but no reliable route carries it into the report. External probes and trained reporters can distinguish these cases. The recent open-weight masked study reported the second pattern across its tested models: spontaneous words stayed near chance, while linear probes recovered intervention presence and a fine-tuned reporter performed strongly on held-out directions.[17] Because that work is a recent preprint, its estimates need independent replication. Its experimental logic is nevertheless valuable regardless of the final numbers.

This also changes how scaling claims should be read. A larger model may improve at recognising the semantics of an experiment, producing plausible explanations or following a metacognitive prompt without gaining a privileged route. Conversely, a small model with an explicit reporter head may have a narrow route that a larger conversational model lacks. Plot performance against model size only after the contamination profile is held constant. Otherwise the apparent emergence may belong to prompt comprehension, instruction following or the measurement threshold. A null under a closed chamber narrows the missing component; it does not erase every form of self-knowledge.

Reliability and honesty are separate axes

Even a contamination-closed report can be false under another incentive. The route might faithfully carry a state into a latent report representation, while downstream policy suppresses or reverses the answer. Conversely, a model can honestly state a mistaken interpretation of its own signal. The assessor must therefore separate access, semantic calibration and disclosure. Route ablation tests access. Ground-truth interventions test semantic calibration. Incentive and adversarial tests examine disclosure.

This separation is vital in safety settings. In experiments on adversarial prefills, open-weight models did not reliably recognise their compromised outputs, and the answer changed with whether the probe asked about intention or tampering.[16] A monitor trained to say “I was manipulated” on one wording may learn a refusal-associated shortcut rather than a stable causal diagnosis. Treating the sentence as a privileged confession would be exactly the error the assay is designed to prevent.

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: The answer that seems to look inward
  M->>E: A small hierarchy of claims
  E-->>M: compare rival explanations
  E->>D: Make self-report beat a visible-data observer
  D-->>Q: revise scope or act

Make self-report beat a visible-data observer

Do not ask whether a model “sounds introspective”. Decide what operational claim would change your action, then build a trial where that claim and its strongest shortcut predict different outcomes. For research, this means replacing broad verdicts with typed statements such as: “the configured reporter discriminates held-out layer-24 interventions from shams while inputs and pre-report behaviour are matched”. For engineering, it means treating self-report as a sensor whose calibration, attack surface and route are versioned.

No consequential action should be authorised by model self-report alone. Pair it with an independent path: an external probe, effect receipt, tool readback, policy check or human review chosen so that it does not share the same likely shortcut. A self-report can lower investigation cost, trigger abstention or prioritise cases. It should not certify its own truth.

The same discipline improves welfare research. A spontaneous statement about fear or pain is evidence that the system generated that statement. A contamination-closed link to a valence-relevant internal state would be stronger functional evidence. Neither is a consciousness verdict. The decision under uncertainty is to preserve the report, configuration and causal receipts, avoid incentives that manufacture dramatic claims, and update protection only through an explicit assurance case.

That is the practical consequence of the chamber. The question is no longer “Did the model say what was inside?” It is “Which wall did the answer pass through, and what other wall would have stopped a counterfeit?”