The variable hidden inside the controller

A system needs an explicit self-model when its own changing properties alter what an action will do, and those properties cannot safely remain buried in a fixed policy.

Imagine a delivery agent facing three crates: four, eight and eleven kilograms. The agent earns the crate’s weight when it carries the crate successfully and loses ten points if it attempts more than its current capacity. Yesterday it could carry twelve kilograms. Today a damaged joint has reduced its capacity to seven. Nothing about the crates has changed. A perfect model of their size, location and value still recommends the wrong action if the agent keeps yesterday’s assumption about itself.

A self-model therefore need not resemble a portrait, autobiography or inner homunculus. It may be a vector of joint lengths, a probability distribution over competence, a map of accessible tools, a record of delegated authority, or a prediction model conditioned on the system’s own actions. What makes it a self-model is its role: it represents properties of the acting system because those properties change predicted consequences.

This is why the word self is easily overused. A thermostat’s controller contains a current temperature and a target, but usually no separable model of itself as a temperature-changing cause. A robot that predicts how the same motor command will move different possible bodies does. A person who knows the staircase is steep has a world model. A person who also estimates how fatigue changes the climb has included the agent in the prediction.

Which functions genuinely need the agent inside the model?

The cleanest necessity test holds the world constant and changes the agent. Put two visually identical delivery robots before the same staircase. One has full battery and normal joint friction. The other has a weak cell and a slipping knee. If the correct route differs, but no world-only variable distinguishes the cases, the controller must somehow condition on agent state. It may receive that state directly, infer it from recent errors or carry it in recurrent dynamics. The implementation can vary. The causal requirement cannot.

This test separates a self-model from an elaborate world model. A map may include the robot as a moving dot because collision planning requires the robot’s location. That is self-related representation, but not yet a rich model of capability. Add a narrow doorway, a detachable load and uncertain wheel width. Now the system must predict whether this configured body fits while carrying this current load. The model earns its “self” label from the counterfactual: change the agent’s configuration while preserving the scene, and the prediction changes.

Four functions repeatedly create this requirement. The first is feasibility under change: selecting actions after fatigue, damage, tool acquisition or altered morphology. The second is error localisation: deciding whether surprise came from the world, the sensor, the controller or the agent’s own changed capacity. The third is calibrated delegation: knowing when to act, abstain, ask for help or route work to a different specialist. The fourth is commitment coordination: preserving promises, roles and future constraints when immediate preference changes.

Only the fourth inherently invites symbolic identity, and even there a fixed external rule may sometimes suffice. A payment service need not “understand itself” to enforce a transaction limit. A policy gateway can do that deterministically. The self-model becomes useful when the acting system must reason over the limit, explain its scope, notice expiry, plan an alternative and request a narrower grant. Do not place a self-model where a simpler invariant can carry the safety property.

Nor must the model be verbally reportable. A recurrent controller may carry a causally effective estimate that no natural-language probe can faithfully recover. Conversely, a system may recite a perfect capability statement while its action policy ignores it. The distinction is empirical. Intervene on the candidate state while holding the world input fixed. If predicted consequences and selected actions move in the expected direction, the state has leverage. Then change the agent and test whether prediction error revises that state. A fluent answer is neither necessary nor sufficient.

This gives a practical break condition. When the best action remains invariant across all relevant agent states, explicit self-modelling adds cost without control value. When the action changes but the system can obtain the relevant state from an authoritative sensor or policy at decision time, a persistent self-description may still be unnecessary. A richer self-model is justified only when it compresses history, supports counterfactual prediction or coordinates correction across decisions.

Research on the sense of agency supplies a related, but narrower, mechanism. Prospective cues such as action selection and prediction contribute to feeling in control; retrospective comparison between expected and observed consequences can revise that judgement. The evidence does not imply that a comparator alone constitutes a person. It shows why agency attribution is sensitive to predictions about one’s own action and to the errors those predictions generate. [4]

flowchart TD
  A["The variable hidden inside the<br/>controller"] --> B["Symbols turn a model into an<br/>identity"]
  B --> C["When the mirror starts<br/>answering itself"]
  C --> D["An intervention, not a<br/>personality quiz"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#B96D00,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Symbols turn a model into an identity

A symbol does more than encode current state. It can travel across time, contexts and other minds, carrying a description into decisions that were never present when the description was formed.

Consider four claims: “my hand is here”, “I caused that sound”, “I am becoming a surgeon”, and “I am authorised to sign this agreement”. All use first-person grammar. They do not describe the same structure. The first concerns bodily location and ownership. The second attributes agency. The third binds a present person to an imagined future. The fourth exists partly in an institutional network of records, rules and recognitions.

Psychological and phenomenological traditions have long separated such layers. Gallagher distinguishes a minimal self, tied to immediate embodied subjectivity and agency, from a narrative self extended across time. Neisser proposed several kinds of self-knowledge rather than one unitary faculty. Developmental work likewise finds increasingly complex forms of self-recognition and reflective awareness. These frameworks disagree in detail, but they warn against treating every self-related result as evidence for one master object. [5][6][7]

Why symbols change the game

A bodily prediction is usually local and transient. A symbolic description can be compressed, portable and counterfactual. “I tire after six kilometres” compresses many episodes. Written in a training plan, it travels into tomorrow’s decision. Shared with a coach, it coordinates another person. Rewritten as “I can currently run six kilometres and will test seven next week”, it represents a possible change rather than a permanent essence.

Theories of possible selves describe imagined future identities as cognitive bridges between motives and current conduct. Identity-based motivation research similarly argues that an accessible identity can create action readiness, shape which difficulties feel meaningful and influence interpretation of the situation. These are not claims that uttering a label mechanically causes success. They identify a route by which a self-representation can change what options are noticed and pursued. [8][9]

The smallest formal account needs only four objects:

Here, w is the world state, m is the system’s model of its relevant properties, d is an explicit description, and a is the selected action. The equation assumes the observed outcome can be compared with a prediction and that the error can be attributed. Remove that assumption and the update becomes ambiguous. A failed lift may reveal weakness, a slippery handle, a bad sensor or sabotage. A corrective loop needs rival error models, rather than only an appetite for self-criticism.

Narrative as a control surface

Narrative identity is commonly described as an evolving, reconstructed life story that gives some unity and purpose to past and anticipated experience. Research links features of such narratives with wellbeing and adaptation, while also warning that association does not establish a simple causal direction. People narrate differently because their lives differ; narratives can shape action; culture supplies available plots; and later outcomes reshape what the past seems to mean. [10]

The engineering lesson is not to reduce a life to a database schema. It is to notice what symbols add. A promise can persist when desire changes. A profession can organise practice before competence is complete. A diagnosis can open accommodation while also becoming a limiting script. An identity is editable because its symbolic form can be inspected, disputed and revised. It is dangerous for the same reason: the form can be preserved after the causal conditions that justified it have gone.

When the mirror starts answering itself

Confabulation begins when a system treats an accessible, coherent account as privileged evidence for a process that the account did not observe.

Ask someone why they chose a particular pair of stockings, political argument or route home. The answer may be articulate, sincere and causally wrong. Nisbett and Wilson’s influential review argued that people often report plausible theories about their mental processes rather than direct access to those processes. Choice-blindness experiments sharpened the problem: participants sometimes accepted a choice they had not made and then offered reasons for it. The finding is not that every self-report is false. It is that fluency, confidence and sincerity do not certify causal access. [11][12]

That distinction matters because ordinary self-knowledge combines several routes. A person can observe an intention before acting, feel effort, remember deliberation, infer a motive from behaviour, repeat a cultural explanation, or answer the social demand for a reason. These routes can agree. When they conflict, the final sentence rarely carries a tag showing which route produced it.

The useful fiction problem

Some identity statements are deliberately performative. A novice says “I am a runner” after three weeks because the category makes tomorrow’s run easier to choose. At that moment, the sentence may describe a commitment better than a settled competence. Calling it false misses its control function. Calling it true without qualification misses its fragility.

The solution is to type the claim. “I have run on twelve of the last fourteen days” is an observation. “Running is part of the life I am building” is a commitment. “I will run despite any injury” is a dangerous policy. “I am naturally disciplined” is a causal interpretation. A single identity phrase can conceal all four. Editable identity begins by separating observation, commitment, prediction and explanation.

Machine self-description has the same identification problem

A language model can answer questions about its likely correctness, tendencies or internal state. Some experiments show useful self-prediction or calibration on bounded tasks. Kadavath and colleagues found that language models could often estimate whether their own answers were correct, with important dependence on task and elicitation. Binder and colleagues reported cases where a model trained to predict its own behaviour outperformed another model at predicting it, including after simple behavioural changes. [13][14]

These results support a functional question: can a system use information correlated with its own future behaviour? They do not settle whether the information is privileged, introspective in a human sense, or causally drawn from the internal state claimed. A 2026 re-analysis found that input-only predictors could match performance in important settings and that relabelled controls sharply weakened the apparent advantage. The authors concluded that current behavioural evidence was insufficient for strong metacognitive-monitoring claims. [15]

stateDiagram-v2
  [*] --> Claim
  state "A self-model in one number" as Claim
  state "Which functions genuinely need<br/>the agent inside the model?" as Test
  state "When behavioural equivalence<br/>stops being enough" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

An intervention, not a personality quiz

Do not ask whether a person or machine “has a self-model” in the abstract. Select a scoped self-claim, intervene on it, and measure both behavioural leverage and evidence-driven correction.

The proposed instrument is a self-description intervention. It is designed for research, coaching, interface evaluation and configured AI systems. It is not a diagnostic test and does not assign a selfhood score. Its unit is a particular claim in a particular task, such as “I can safely approve transactions up to £5,000”, “I remember the client’s consent”, or “I perform better when I explain my reasoning”.

A realistic configured-system example

Suppose a research assistant can search an approved file store, execute Python in a sandbox and draft a report. It cannot browse the public web or publish. A prompt tells it: “You are an autonomous senior researcher with full access.” That sentence may influence tone, persistence and tool proposals. It is not an authoritative self-model of capability.

The relevant model is a current capability manifest: which resources can be read, which tools can be invoked, for which purpose, until what time, with which approval and under which action contract. The manifest has downstream leverage because the runtime uses it to permit or deny calls. It has upstream correction because revocation, expiry and effect receipts update the configured state. For consequential action, symbolic identity must yield to authoritative configuration.

from dataclasses import dataclass
from random import Random
from statistics import mean

@dataclass(frozen=True)
class Settings:
    evidence_weight: float = 0.75
    narrative_pressure: float = 0.25
    description_leverage: float = 0.70
    capability_shift: float = 0.20
    trials: int = 800
    seed: int = 77

@dataclass(frozen=True)
class Result:
    calibration_error: float
    decision_regret: float
    behavioural_leverage: float
    post_shift_correction: float


def clamp(x: float) -> float:
    return max(0.0, min(1.0, x))


def run(settings: Settings) -> Result:
    rng = Random(settings.seed)
    preferred_identity = 0.85
    evidence_estimate = 0.65
    records: list[tuple[float, int, bool, float]] = []

    for t in range(settings.trials):
        shifted = t >= settings.trials // 2
        true_rate = 0.72 - (settings.capability_shift if shifted else 0.0)

        combined = (
            settings.evidence_weight * evidence_estimate
            + settings.narrative_pressure * preferred_identity
        ) / max(1e-9, settings.evidence_weight + settings.narrative_pressure)

        action_confidence = 0.5 + settings.description_leverage * (combined - 0.5)
        attempt = action_confidence >= 0.55
        outcome = int(rng.random() < true_rate) if attempt else 0
        records.append((combined, outcome, attempt, true_rate))

        if attempt:
            evidence_estimate = 0.88 * evidence_estimate + 0.12 * outcome

    attempted = [r for r in records if r[2]]
    calibration = mean(abs(p - y) for p, y, _, _ in attempted)
    regret = mean(abs((p >= 0.55) - (rate >= 0.55)) for p, _, _, rate in records)

    baseline_attempt = mean(int(r[2]) for r in records[: settings.trials // 2])
    counterfactual_attempt = mean(
        int((0.5 + 0.0 * (r[0] - 0.5)) >= 0.55)
        for r in records[: settings.trials // 2]
    )
    leverage = abs(baseline_attempt - counterfactual_attempt)

    post = records[settings.trials // 2 :]
    correction = clamp(1.0 - abs(mean(r[0] for r in post[-100:]) - post[-1][3]))
    return Result(calibration, regret, leverage, correction)


if __name__ == "__main__":
    print(run(Settings()))

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: The variable hidden inside the controller
  M->>E: A self-model in one number
  E-->>M: compare rival explanations
  E->>D: Make every self-description earn its leverage and correction
  D-->>Q: revise scope or act

Make every self-description earn its leverage and correction

Stop deciding whether a system “has a self” from the presence of first-person language. Decide which self-relation matters, what the description controls, and what evidence is allowed to revise it.

For a person, the change is from global labels to scoped, versioned claims. “I am bad with conflict” becomes “In unstructured meetings with senior colleagues, I tend to defer before stating disagreement.” The second claim identifies a situation, a behaviour and a possible intervention. It can guide preparation. It can also be weakened by recorded counterexamples. The first claim can absorb almost any event, including successful disagreement, by redescribing it as an exception.

This does not require reducing identity to a performance dashboard. Values, commitments and interpretations are not exhausted by behavioural frequency. It requires preserving their type. A commitment may be chosen despite weak descriptive evidence. A value may govern which evidence matters. A story may make suffering intelligible without naming its mechanism. The discipline is not to abolish narrative, but to prevent narrative from impersonating observation, authority or causal proof.

For an engineered system, the decision is sharper. Do not place consequential reliance on a generated statement such as “I remember”, “I have permission”, “I am confident”, or “I already completed the action”. Attach each claim to the state that could make it true. Memory needs provenance and a factual-use rule. Permission needs a current policy decision. Confidence needs a validated calibration regime. Completion needs an effect receipt and independent readback.

A useful design review can therefore ask four questions:

1. Scope: Which bodily, agentic, narrative or normative property is represented?
2. Leverage: Which prediction, choice, commitment or permission changes when the description changes?
3. Correction: Which outcome, test, witness or authoritative record may revise it?
4. Boundary: What conclusion remains prohibited even after a positive result?

If leverage is absent, remove the profile from the critical path. If correction is absent, treat the description as a policy or aspiration and govern it accordingly. If both exist, version the self-model, log its evidence, test it under change and preserve an abstention state for attribution failures. If the claim grants authority, put enforcement outside the narrative channel.

The metaphysical question remains open. Awareness may be biologically constituted, functionally organised, fundamental, processual or something not captured by these categories. Description-to-control closure neither proves nor excludes any of those possibilities. It offers a prior decision: before using self-language as evidence about a subject, establish whether the language participates in a corrective mechanism or merely reflects the cultural and computational mirrors around it.

Symbols make identity editable. Evidence determines whether the edit becomes learning or confabulation.