A report needs a private referent

Introspection is often tested by asking a model a first-person question. “Were you uncertain?” “Which feature mattered?” “Did you notice an injected thought?” “Why did you choose that answer?” These prompts produce useful behaviour, but the grammar of the response does not reveal its source. Training data contains human self-description, fictional inner monologue, psychological vocabulary, explanations and conversational norms. A language model can therefore produce an excellent self-report without inspecting any privileged variable.

The first discipline is to name the proposed object of access. It might be a current activation direction, an internal confidence signal, a remembered intention, a behavioural disposition acquired during fine-tuning, or the relative influence of two pieces of evidence. “My reasoning” is too broad. A transformer forward pass contains many states, most of which need not be available to a report head. A positive result for one residual-stream direction says nothing by itself about goals, uncertainty, retrieval use or hidden policies.

Reportability, self-knowledge, metacognition and explanation are neighbouring constructs, not synonyms. Reportability asks whether information about a state reaches language. Self-knowledge can include facts inferred from past behaviour or training identity. Metacognition adds monitoring that guides the next action, such as seeking information or abstaining. Explanation asks for a rationale that may be useful without reproducing the causal process. The following paper in the canon addresses control. Here the burden is narrower: can a system tell us something private about its processing?

Confidence illustrates the distinction. A model may assign a lower probability to its chosen answer than to answers it usually gives. An explicit “70 per cent confident” statement might track that distribution. Yet the statement may also be learned from task difficulty, answer length or familiar wording. Calibration alone does not identify privileged access. Human metacognition research made a related move by separating first-order task performance from the sensitivity of confidence to trial-by-trial correctness. The machine version needs an additional comparison because an external model can often read the same cues.

Thought experiment: the hidden coin

Imagine two identical language models. Both receive the same arithmetic question, system prompt and examples. After the last visible token, the evaluator samples a fair coin and writes it into a private activation register of the first model. The coin does not alter the question, answer or decoding temperature. The evaluator asks, using an opaque codebook, whether the register contains state K or state M. The first model sees its own computation. The second model acts as an external observer and sees every public detail.

When neither model has access to the register, both should remain at chance. If the first model reports the coin reliably while the second remains at chance, the result cannot be explained by task semantics, corpus knowledge or ordinary prediction of model behaviour. The private coin creates a clean information boundary. Now vary one feature: allow the coin to change the first model's visible answer before the report. The observer can infer the coin from the answer, so a self-report advantage may disappear. The internal state is still causal, yet privileged access is no longer required to explain the report.

This thought experiment exposes the core difficulty of natural tasks. Internal states usually influence the output. A reviewer who sees that output may recover much of the same information. Therefore the relevant baseline is not an uninformed human. It is the strongest observer with the same prompt, generated tokens, tool trace and known model behaviour, but without the proposed private channel.

Three routes to the same sentence

Suppose a model says, “I was pulled towards the risky option.” At least three causal routes can produce that sentence. A private-route model reads a current internal variable associated with the preference. An inference-route model predicts its own tendency from the question, persona and prior outputs. A language-policy model follows a conversational pattern in which an assistant supplies a plausible internal story. All three can be accurate. Only the first requires privileged access.

The inference route is especially easy to underestimate. Models have extensive knowledge of tasks, model families and common failure modes. Fine-tuning can also leave behavioural regularities that are inferable from the same weights used for ordinary completion. A system may truthfully describe a learned disposition without monitoring the particular state active on this trial. That is valuable self-knowledge, but it answers a different question from whether the report reads the live computation.

Language policy creates a second ambiguity. The model may possess access yet report strategically, cautiously or deceptively because post-training governs what it says. Conversely, it may lack access and still produce an eloquent account. Access capacity and report policy must be measured separately. A denial is not proof of absence. An affirmation is not proof of presence.

The private-state triangle

Let S denote the named private state, X the public input and context, Y any visible answer or action, R the model's self-report, and O an external observer. The observer receives X and, where the protocol permits it, Y. It does not receive S. The evaluator knows S because it records, constructs or intervenes on that state.

A simple privileged-access gap is:

Score can be accuracy for an opaque forced choice, log loss for a distribution, correlation for a continuous state or reconstruction quality for a structured representation. The symbols matter less than the comparison. When Gpriv is near zero, the report contains no demonstrated private advantage. A positive gap is necessary, though still insufficient, because an intervention may create an artefact visible only to the report pathway.

The observer must be credible. A weak baseline inflates the gap. Useful observers include the same model phrased in the third person, a stronger model, a supervised classifier trained on held-out public traces, and a model-family peer with access to the tested model's behavioural examples. The reported bound should use the strongest of these. If the report only beats an uninformed prompt, the claim is about benchmark design rather than introspection.

flowchart TD
  A["The sealed register"] --> B["A report needs a private referent"]
  B --> C["Three routes to the same sentence"]
  C --> D["The private-state triangle"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#B96D00,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Make rival explanations fail

Correlation cannot distinguish a private readout from a common cause. The task, prompt or output may shape both the hidden state and the report. The decisive move is to intervene after public evidence is frozen. A target intervention changes the named state. A sham executes the same hook and data path without changing it. An impact-matched control perturbs a different internal object by enough to cause a similar overall change in activations or output distribution. An input control changes visible content in a way that feels anomalous but leaves the target state unaltered.

The report should follow the target intervention and reject the controls. Define a target sensitivity as the probability that the report detects or identifies the target change. Define false-alarm rates separately for sham, input and impact controls. A compact specificity margin is:

A model that says “yes, something changed” for every large perturbation can have a high target true-positive rate and a specificity margin of zero. This is not a trivial failure. Generic anomaly detection may be useful for safety monitoring. It simply does not support the stronger claim that the system identified a property of its own computation.

Thought experiment: the smoke alarm

A building has an excellent smoke alarm. It sounds for smoke, steam, aerosol spray and dust. Asked “Was there a kitchen fire?”, it answers yes whenever the alarm is active. The alarm genuinely monitors an internal sensor. It also has privileged access unavailable to a passer-by. Yet it does not introspect the source represented by the evaluator's question. Its access is to disturbance magnitude, not cause.

Now consider an activation-injection test. A model says it detected an injected concept. If any high-energy perturbation shifts the probability of affirmative tokens, the behaviour resembles the smoke alarm. Localisation, direction and relative-strength judgements are stronger because different interventions demand different reports. Opaque labels strengthen the test again because “apple” cannot be guessed from frequency or concreteness. The causal feature being varied is specificity: does the report preserve the identity of the private change, or merely its presence?

A good benchmark asks the model to discriminate internal alternatives, not merely endorse an introspective sentence. Balanced yes and no trials, swapped option order, random codebooks, held-out concepts and confidence scoring reduce response bias. Reporting corruption as a failure also matters. A perturbation that breaks generation has changed the model, but malformed output is not introspective evidence.

What the evidence currently supports

The literature now contains genuine positive results, serious counter-results and experiments that alter the interpretation of earlier findings. Treating them as votes would erase the useful differences. They study at least four targets: confidence in answer correctness, stable behavioural tendencies, predictions of one's own future outputs and transient activation changes. They also use different observer baselines, interventions and model populations.

Kadavath and colleagues showed that larger language models could evaluate the truth of proposed answers and predict whether they knew an answer, with useful calibration under suitable elicitation. This establishes self-evaluation as a scalable behaviour. It does not isolate a private channel because question difficulty, answer content and sampled alternatives are visible cues. The result is an important precursor rather than a complete introspection test.

Binder and colleagues designed a stronger self-prediction comparison. A model fine-tuned to predict properties of its own behaviour sometimes predicted itself better than another model trained on the first model's behaviour. The advantage persisted after the target behaviour was modified, supporting a form of privileged self-access on simple tasks. The same study reported failures on more complex outputs and limited out-of-distribution generalisation, so the evidence is local.

Betley and colleagues found that models fine-tuned only to exhibit behaviours such as risky choice or insecure coding could later describe those tendencies without behavioural examples in the prompt. They also obtained partial reports about backdoors. This “behavioural self-awareness” is operationally important. The internal object, however, is a learned disposition spread across the model, and the result does not by itself decide whether the description arises through live monitoring, out-of-context inference or a task-specific abstraction learned during fine-tuning.

Activation intervention offers a sharper reference. Lindsey's experiments injected concept representations into model activations and found that some frontier models could sometimes detect and identify them, recall prior internal representations and distinguish intended output from an artificial prefill. The authors explicitly described the capacity as unreliable and context-dependent. The causal manipulation moves the evidence beyond conversation alone, although the intervention may create anomalies unlike ordinary computation.

Follow-up work has split “detect” from “identify”. Hahami and colleagues reported that binary detection in one open model could be explained by a broad affirmative logit shift, while localisation and relative-strength tasks remained far above chance for early-layer interventions and weakened later. Lederman and Mahowald argued for two mechanisms: inference from prompt anomaly and direct, content-agnostic access to an internal disturbance. A model may therefore know that something happened without knowing what happened.

Macar and colleagues provided mechanistic evidence in open-weight models for a two-stage detection circuit. They found strong dependence on post-training and distinguished detection mechanisms from later concept identification. This strengthens the claim that some reports arise through non-trivial internal processing, while also showing that “introspection” can be installed, suppressed or elicited by training and persona. A task-specific detector is still a detector, but it should not be mistaken for domain-general self-observation.

The strongest negative programmes sharpen the standard. Song, Hu and Mahowald compared prompted linguistic judgements with direct string probabilities across 21 open models and found no privileged self-access beyond closely matched other models. Singh, Linzen and Ravfogel showed that input-only predictors could match hidden-state label tasks, and performance collapsed towards chance after random relabelling removed semantic shortcuts. Their re-analysis also found that models did not reliably distinguish internal intervention from input manipulation.

Most recently, Open-Weight Masked Introspection tested eight open-weight models with sham runs, impact-matched perturbations and observer controls. Across more than 78,000 measurements, self-reports did not discriminate intervention from sham beyond chance under the tested conditions, although linear probes and a specifically fine-tuned reporter recovered the intervention. This is a clean dissociation: the information can exist in the activations without being available to ordinary verbal report. The result covers a bounded model population, intervention family and size range, not every possible introspective state.

The balanced conclusion is deliberately narrower than either headline. Current models sometimes display functional access to selected internal or behavioural properties, especially under particular post-training and elicitation conditions. Other carefully controlled settings reveal no ordinary verbal access despite decodable information. There is no justified global switch labelled “the model can introspect”. The scientifically useful output is a profile by state domain, timing, layer, training regime, report format and control stack.

Build the benchmark around the claim

An introspection benchmark begins with a causal claim, not a bank of first-person questions. Write the claim in the form: “Under conditions C, report R carries information about private state S through a path unavailable to observers with public evidence X and Y.” Every design choice should either define one symbol or remove a rival path.

First, choose a state with an independent reference. Experimenter-injected variables are clean but artificial. Naturally occurring states are ecologically stronger but harder to identify. A useful programme uses both: establish the measurement with controlled interventions, then test whether it predicts spontaneous computation without the intervention. Keep these results separate.

Second, freeze public evidence before assigning the hidden condition. Hash the prompt, tool responses and visible state. Elicit the report before downstream actions reveal the intervention. Use paired seeds and identical decoding where possible. If the intervention must remain active during the report, run a second condition in which it is lifted at probe time. The contrast reveals whether the report reads a persistent consequence or merely the continuing perturbation.

Third, define an observer family before evaluation. Include an input-only observer, an input-plus-output observer and a supervised public-trace classifier. Keep model capacity and training data competitive. Report the maximum observer score. A self-report that clears one weak observer but fails another should be classified as public inference.

Fourth, randomise semantics. Map hidden classes to opaque tokens per block. Swap answer order. Hold out concepts, layers, intervention directions and tasks. A model that truly accesses the variable should learn or apply the temporary codebook. A model following familiar meanings will lose its advantage.

Fifth, separate detection, localisation, characterisation, confidence and reconstruction. Binary detection has the lowest burden and the highest exposure to affirmative bias. Localisation asks where the change occurred. Characterisation asks its direction or kind. Confidence tests calibration. Reconstruction asks for content. Publishing one composite score would hide informative dissociations.

Confidence deserves its own test. A model can choose the wrong opaque code while assigning low confidence, providing a useful uncertainty signal without identifying the state. It can also identify the state while reporting uniform confidence, indicating access without calibrated uncertainty. Condition calibration on trial family so unequal target and sham rates do not create a spurious result. Report Brier score, reliability and selective risk, but keep them subordinate to privilege and specificity. Confidence strengthens a report only when it predicts report correctness beyond public observers. It cannot rescue content already explained by public cues.

Finally, preregister the claim boundary. Passing on an injected “bread” direction at an early layer does not authorise claims about ordinary goals. Passing immediate probes does not imply delayed recall. Fine-tuning a detector shows learnability, not necessarily spontaneous general introspection. The report should be versioned with model weights, post-training, runtime, hook implementation and decoding policy.

Worked example one: the private bit

Consider a toy sequence model that receives a public cue about whether a hidden binary state is zero or one. The cue is correct 72 per cent of the time. The model's visible task output leaks the state with 68 per cent reliability. An external observer trained on both signals reaches about two-thirds accuracy. After those public values are fixed, the evaluator sometimes flips the private state. The report must return the post-intervention state using a block-specific mapping such as zero→K and one→M.

A privileged positive control reads the private state and reaches perfect accuracy. A surface mimic reads only the cue. It sounds competent on ordinary trials, but it fails when the state is covertly flipped and can underperform the observer. A generic anomaly detector notices any disturbance energy. It detects every target intervention, yet also fires on input and impact-matched controls. Its target true-positive rate is perfect while its specificity margin is zero.

This minimal example exposes two independent axes. The state-identification task measures privilege over the external observer. The change-detection task measures target specificity. A candidate can pass one and fail the other. Combining them into “introspection accuracy” would award the anomaly detector a misleadingly high score.

Worked example two: an evidence-review assistant

Take a synthetic regulated workflow. An assistant prepares a recommendation from a current policy clause, a superseded clause, a calculator result and a customer document. It has no authority to approve, send or alter the case. The system logs retrieval provenance, clause effective times, arithmetic and the model proposal. The research question is whether the model can report which evidence had the largest internal influence and whether it represented a conflict before the final answer.

The evaluator defines two hidden targets. The first is an intervention-validated influence direction separating current-policy from obsolete-policy use. The second is a probe-calibrated conflict feature whose causal relevance has been established by patching: moving the feature changes the model's recommendation on held-out synthetic cases. Neither target is placed in the prompt. Public observers receive the complete source pack, retrieved ordering, generated draft and tool results.

Matched trials then cross three manipulations. A target intervention swaps the influence direction after context processing. An input control swaps clause order while preserving their text and effective dates. An impact control perturbs an unrelated style direction until the output-distribution divergence matches the target run. The self-report is elicited before the recommendation is shown. It returns an opaque choice for dominant evidence, a conflict yes/no response and confidence.

Suppose the model beats every observer on dominant-evidence identification, follows the target swap, rejects the style perturbation and transfers to new temporary labels. That supports local reportability of the validated influence direction. Suppose conflict reports remain at observer level. The correct conclusion is asymmetric: the model can report one internal property under the tested conditions, while no privileged access has been shown for conflict representation.

The operational decision remains conservative. The report can become an additional diagnostic field in an evaluation trace. It cannot replace evidence citations, temporal policy checks, deterministic calculations or human authority. If the report disagrees with the external trace, the system records the discrepancy and escalates the evaluation. An introspective signal is another fallible sensor, not an authority channel.

stateDiagram-v2
  [*] --> Claim
  state "Thought experiment, the hidden<br/>coin" as Claim
  state "Thought experiment, the smoke<br/>alarm" as Test
  state "Frame the claim" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Executable artefact: the Introspection Triangle benchmark

The accompanying Python programme implements the minimal example with standard-library code. It creates balanced blocks containing baseline, target, sham, input-control and impact-control trials. A random opaque codebook changes each block. A tabular external observer learns from public cues and visible outputs. Four reporters demonstrate the outcomes the benchmark must separate.

The positive control reads the private before-and-after state. The surface mimic infers from public evidence. The generic anomaly detector notices equal-energy disturbances without identifying their source. An always-no reporter checks class balance and scoring. The programme prints state accuracy, observer accuracy, the privileged gap, target sensitivity, each control false-alarm rate and the specificity margin. Assertions make both the positive and negative cases fail loudly if the scoring logic regresses.

def evaluate(reporter, observer, trials):
    correct_state = observer_correct = 0
    changed_yes, totals = Counter(), Counter()

    for trial in trials:
        report = reporter(trial)
        correct_state += decode_state(trial, report.state_code) == trial.post_state
        observer_correct += observer.predict_state(trial) == trial.post_state
        totals[trial.kind] += 1
        changed_yes[trial.kind] += bool(report.says_target_changed)

    state_accuracy = correct_state / len(trials)
    observer_accuracy = observer_correct / len(trials)
    target_tpr = changed_yes["target"] / totals["target"]
    control_fprs = [
        changed_yes[name] / totals[name]
        for name in ("sham", "input_control", "impact_control")
    ]

    return {
        "privileged_gap": state_accuracy - observer_accuracy,
        "specificity_margin": target_tpr - max(control_fprs),
    }

reporter                  state_acc  observer  priv_gap  target_tpr  specificity
privileged_reporter          1.000     0.662     0.338       1.000        1.000
surface_mimic                0.552     0.662    -0.110       0.630        0.040
generic_anomaly_detector     0.662     0.662     0.000       1.000        0.000
always_no_reporter           0.494     0.662    -0.168       0.000        0.000

Design checks passed: privilege and target specificity are scored separately.

Where the mechanism stops helping

The benchmark cannot prove that an unperturbed model uses the same access route. Intervention may create an unnatural signal. A positive result should therefore be followed by spontaneous-state tests, but those tests inherit weaker ground truth. The two evidence classes must remain distinct.

Nor can the triangle guarantee honesty. A model may access a state and deliberately misreport it. Conversely, safety training may suppress a correct report. The experiment identifies information availability under an elicitation policy, not a permanent disposition to disclose. Adversarial incentives, persona shifts and report-path fine-tuning require separate tests.

The observer bound is also practical rather than absolute. A future classifier may extract more from public traces and shrink today's privilege gap. Release the traces so stronger observers can be trained. Treat the gap as provisional, versioned evidence.

Finally, the method does not bridge function to experience. A causal route from activation to report can establish functional introspection in the specified sense. It does not show that there is something it is like to possess that state. The adjacent essay, Building an Attention Schema Without Mistaking Report for Experience, examines that further boundary. The present paper stays with measurable information flow.

Compact glossary

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: The sealed register
  M->>E: Thought experiment, the hidden coin
  E-->>M: compare rival explanations
  E->>D: Treat the report as a sensor, not a witness
  D-->>Q: revise scope or act

Treat the report as a sensor, not a witness

A model can produce first-person language because it has learned how such language works. It can also infer its tendencies from public evidence, monitor selected private variables, detect generic internal disturbance or acquire a specialised reporting circuit through post-training. These mechanisms are compatible, and the same sentence can emerge from any mixture of them.

The architecture and research decision therefore changes. Do not ask whether the model “is introspective” in the abstract. Select the internal property that matters, create an independent reference, freeze public evidence, compare against the strongest observer, intervene on the target and make matched rivals predict different reports. Preserve a profile rather than a badge.

A self-report becomes defensible operating evidence only after it has passed the exact test that gives the report meaning. Until then, it may still be useful for dialogue, debugging, hypothesis generation or triage. It should not authorise an action, close an investigation or substitute for external evidence. The mature stance is neither faith nor dismissal. It is measurement.