The control room inside the control room

“Self-model” invites the wrong picture. We imagine a little portrait of the whole agent placed inside the agent. Something must read the portrait, decide what it means and act upon it. That reader now needs its own portrait. The regress follows because every representation has been turned into a new reasoner.

Engineers already use a less dramatic meaning of observation. A state observer reconstructs variables that cannot be measured directly from a system’s inputs and outputs. Luenberger’s classic result concerns when the state of a linear system can be reconstructed, not who looks at it.[1] The observer is an update rule. Its estimate enters the controller. No witness sits between estimate and action.

This matters because the word “observer” moves too easily between three roles. A sensor produces a signal. An estimator turns signals into a state. An agent interprets reasons and chooses. Only the third role carries the intuition of another mind. A monitoring computation does not become an inner subject merely because its output is about the system that contains it.

The same correction applies to metacognition. Human metacognition is often studied as a relation between a first-order decision and a second-order estimate of that decision, such as confidence.[6] “Second-order” names a dependency, not a second person. Experimental work also suggests that local confidence can contribute to broader estimates of self-performance, while feedback conditions can bias those estimates.[8] The useful lesson for artificial systems is not that human selfhood has been reduced to a confidence register. It is that local traces can be compressed into slower variables that alter later control.

Attention schema theory offers a related but contested proposal: a simplified internal model of attention may help a system predict and control attention, while also shaping how it reports awareness.[10] The RCS borrows only the control pattern. It neither identifies attention with awareness nor assumes that a useful self-model explains experience.

The hidden assumption is semantic completeness

Why does a full internal copy seem necessary? Because we quietly demand semantic completeness. We want the system to know every relevant fact about itself, know that it knows, and certify that the certification is sound. That demand is far stronger than control requires.

A thermostat need not model the molecules in its housing. A robot can estimate a damaged limb well enough to choose a compensating gait without modelling every component. Bongard, Zykov and Lipson demonstrated a robot that constructed candidate self-models from sensorimotor data and used them to recover locomotion after damage.[9] Whatever philosophical vocabulary one prefers, the engineering fact is modest: a partial internal model can support adaptation.

Control theory gives two related warnings against overstatement. Conant and Ashby’s good regulator theorem is often compressed into a slogan that every good regulator must be a model of its system. The theorem has explicit probabilistic assumptions and does not say that every controller contains a human-readable replica.[2] The internal model principle similarly identifies structural requirements for rejecting or tracking classes of signals in a feedback system; it is not a general theory of self-knowledge.[3]

A model earns its place by preserving distinctions that change control. For the research agent, “two previous claims were corrected” may matter because it predicts poor calibration. The exact wording of those claims may not. Queue latency may matter. The colour of the retrieval interface may not. Compression is not an unfortunate approximation to a complete self. It is the design decision that prevents self-description from swallowing the system.

flowchart TD
  A["Who notices that the agent is<br/>running out of thought?"] --> B["The control room inside the<br/>control room"]
  B --> C["A loop can model itself<br/>without copying itself"]
  C --> D["Make reflexivity earn its place"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#5B5CF6,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

A loop can model itself without copying itself

We can now make the mechanism precise. Let xt be the task and authorised world state available at step t. Let mt be the reflexive control state. The allocator chooses an allocation at, such as how much search to buy, whether to call a verifier or whether to escalate.

Execution produces a trace τt: what was attempted, which resources were consumed, what evidence arrived, which error was observed and which component owned the action. A compressor C turns that trace into a proposed state update. The update is blended at rate η and projected back into declared bounds B.

Every symbol has an operational meaning. The blend rate controls how quickly the system revises its self-estimate. The projection prevents impossible values, such as negative budget or confidence above one. The compressor chooses what survives. The policy is not a reader outside the loop. It is the same transition that already selects the next action.

This is where information bottleneck thinking becomes useful. Tishby, Pereira and Bialek formalised a trade-off between compressing an input and retaining information relevant to a target.[4] The RCS is not claimed to solve that optimisation exactly. It adopts the design principle: keep the smallest summary that preserves information relevant to future allocation. The right self-model is not the richest description of the agent; it is the smallest state that changes the right decision.

Why compression blocks regress

Suppose the compressor had to contain a full executable copy of the allocator, including a complete prediction of how that copy would respond to its own prediction. Exact self-prediction can invite diagonal problems. Reflective-oracle work shows one technically sophisticated way of handling agents that reason about environments containing similarly reflective machines, including randomisation at problematic queries.[12] That is a genuine result about a much stronger problem.

The RCS avoids that problem rather than solving it. It does not ask, “What will this entire program output when shown this forecast of its output?” It asks bounded questions such as, “Was recent error high?”, “How much budget remains?” and “Did the verifier disagree?” A weather station can measure wind at the station without simulating every molecule in the station. Reflexive control likewise measures selected properties of a process in which the measurement participates.

Compression also creates a failure condition. A summary can omit the variable that matters. If two retrieval providers have equal latency but one has stale data, a single “load” score cannot protect the decision. The remedy is not infinite detail. It is a typed revision of the state contract, perhaps adding evidence freshness. The self-model should expand only when an intervention or failure shows that a discarded distinction changes control.

Stability is part of the definition

A feedback loop can be reflexive and still be badly designed. If every error sharply raises caution, and every cautious success sharply lowers it, the system may alternate between paralysis and recklessness. If the update is too slow, the state becomes ceremonial. The loop needs both causal influence and controlled gain.

For a local linear approximation, imagine that a small perturbation δmt is multiplied by an effective feedback factor k on each cycle. When |k| < 1, the perturbation decays. Near one, it persists. Beyond one, it grows; a negative value can produce alternating corrections. Real agent loops are nonlinear and delayed, so this is not a universal proof. It is a diagnostic intuition: a self-model that amplifies its own estimation error is not reflexive wisdom; it is a control instability.

The next part turns that intuition into an intervention protocol and a runnable prototype.

Make reflexivity earn its place

A system saying “I am uncertain” has not yet demonstrated a self-model. The sentence may be prompted, retrieved or selected because it fits the genre. A dashboard field is not enough either. Engineers can log hundreds of variables that never affect control. Reflexivity is a causal claim, so it needs an intervention.

The proposed Reflexive Closure Test is a six-gate assurance instrument. It is designed for configured systems whose state channels, tools and actions can be inspected. It does not produce a consciousness score. It asks whether a bounded self-related state actually closes a useful and stable allocation loop.

Gate 1: declare the boundary and the state contract

Name the candidate system before testing it. Is it the base model, the model plus context window, the agent runtime, or the runtime plus an external state service? List every RCS field, producer, consumer, update cadence, permitted source and expiry rule. “Memory” is too vague. Model weights, episodic records, enterprise knowledge, authorised world state, reflexive control state and effect receipts have different evidential roles.

Gate 2: show bounded, decision-sufficient compression

Demonstrate that the state is smaller than its trace source and that each retained field changes at least one allocation choice under a declared condition. Ablate a field. If behaviour does not change in any relevant test, the field may be decorative. Add a withheld trace variable. If it predicts failures that the current state misses, the compression contract is insufficient.

Gate 3: intervene while holding the task fixed

Run at least three controls. Clamp the state to a constant. Swap it between matched episodes. Shuffle or delay its temporal order. Keep task evidence, model version, authority policy and tool availability fixed. A genuine causal role predicts selective changes in allocation. Mere correlation predicts that the system behaves the same once the task input is controlled.

Gate 4: restore the state and recover the behaviour

An ablation that breaks everything is weak evidence. The intervention may have corrupted an interface or shifted a distribution. Restore the live state or patch in a correctly computed value. The relevant allocation pattern should return. Then intervene on an unrelated state field to test specificity. Break, restore and discriminate is stronger than observe and narrate.

Gate 5: test temporal entitlement

A self-estimate can be accurate about yesterday and wrong for the present decision. Record event time, update time and decision cutoff. Delay the state by one, five and twenty cycles. If performance is unchanged, the state may encode a stable trait rather than online reflexivity. If a small delay reverses the benefit, the configured system needs a freshness contract and a safe stale-state policy.

Gate 6: perturb the loop and measure stability

Vary update rate, feedback gain, sensor noise and missing receipts. Inject one false high-error event. A stable loop should absorb the perturbation and return to its operating region. A brittle loop may lock into excessive caution, oscillate or erase a real warning too quickly. Compare against a simpler rival, such as an ordinary moving average or a policy with no online self-state. The reflexive architecture earns extra complexity only when it survives this comparison.

type UnitInterval = number;

type ReflexiveControlState = Readonly<{
  load: UnitInterval;
  calibrationGap: UnitInterval;
  volatility: UnitInterval;
  causalOwner: "model" | "tool" | "context" | "unknown";
  remainingBudget: UnitInterval;
  observedAt: number;
  expiresAt: number;
  sourceReceiptIds: readonly string[];
}>;

type Allocation = Readonly<{
  searchEffort: UnitInterval;
  verificationEffort: UnitInterval;
  escalate: boolean;
}>;

// Invariants:
// 1. Numeric fields remain in [0, 1].
// 2. observedAt <= decisionCutoff < expiresAt.
// 3. Every update names receipts that existed before the cutoff.
// 4. RCS may shape a proposal, never grant authority.
// 5. A stale or missing RCS invokes an explicit safe policy.

function updateRCS(current, previous, trace, eta, gain) {
  const proposal = compressTrace(trace);
  const blend = current.map((value, i) =>
    (1 - eta) * value + eta * proposal[i]
  );
  return blend.map((value, i) =>
    clamp01(value + gain * (current[i] - previous[i]))
  );
}

function allocate(task, rcs) {
  const search = clamp01(
    0.23 + 0.66 * task.difficulty
    - 0.42 * rcs.load + 0.12 * rcs.calibrationGap
  );
  const verify = clamp01(
    0.02 + 0.80 * rcs.calibrationGap
    + 0.18 * task.difficulty - 0.22 * rcs.load
  );
  return { search, verify };
}

This architecture borrows a practical idea from rational metareasoning: computations themselves can be selected according to expected value and cost.[5] Models of cognitive control likewise treat effort allocation as a learned decision rather than a fixed quantity.[7] The RCS contribution is narrower. It specifies the state channel, causal test and stability gate needed when a configured agent uses its own operating traces to allocate later computation.

stateDiagram-v2
  [*] --> Claim
  state "Remove one assumption from the<br/>control room" as Claim
  state "The hidden assumption is<br/>semantic completeness" as Test
  state "Move the summary across the<br/>boundary" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Where the loop breaks

The architecture removes one regress, not every difficulty of self-reference. Four failures are especially easy to hide beneath fluent self-description: the state may be unused, its source may be corrupted, its feedback may be unstable, or the boundary may quietly include a human or service doing the real monitoring.

This is why current work on language-model introspection must be read as a live dispute, not a settled capability. One recent study injected concept representations into model activations and reported that some models could sometimes detect or identify the manipulation, while stressing that the capacity was unreliable and context-dependent.[13] A later reality check argued that input cues and task semantics could explain important results, and that stronger relabelled controls moved performance towards chance.[14] These findings do not cancel one another. They identify the experiment that matters: separate privileged access to internal state from anomaly detection and input-based prediction.

Generated reasoning is also an unsafe substitute for a typed state channel. Studies have shown that chain-of-thought explanations can omit biasing influences or rationalise outputs after the fact.[15] The conclusion is not that all reports are false. It is that verbal availability and causal use are different variables. A state can guide control without being verbalised, and a sentence can be produced without the named state guiding control.

Failure 1: ornamental self-description

The system logs confidence, load and “reflection”, but the policy ignores them. Teams may still admire the dashboard because the values correlate with difficulty. Gate 3 catches the failure. Clamp and swap produce no selective change. The remedy is either to wire the state into a declared decision or remove the reflexive claim.

Failure 2: poisoned self-evidence

An adversarial tool can return a forged receipt claiming that verification succeeded. A prompt can instruct the model to mark itself confident. A compromised compressor can assign causal ownership to the user rather than the tool. Because the RCS affects later allocation, poisoning it is a control attack.

The state therefore needs provenance, rather than only values. Updates should name source receipts, identities, event times and the transformation version. Untrusted model prose may propose an update, but a typed compressor should verify which signals are admissible. When provenance is missing, the state should become unknown or stale rather than silently normal.

Failure 3: reflexive oscillation

Failure 4: the human hidden in the boundary

A human operator may notice rising error, edit the system prompt and reset the state. The next report looks self-correcting. An external scheduler may throttle calls using signals unavailable to the agent. These are legitimate system controls, but they change the causal claim. The configured system might be reflexive while the model is not. Gate 1 requires the contribution to be named honestly.

Consequence

Close the allocation loop without inventing an inner observer

The old design question was, “How do we add reflection?” That wording encourages a second prompt, a critic agent or a verbose self-explanation. It also encourages the regress: a critic needs another critic, and every report needs another reader.

The new question is operational: Which compressed facts about the system’s own allocation history must re-enter the next allocation, at what timescale, through which causal path and inside which stability envelope?

That change leads to a different build decision. Start with one or two typed variables, not an introspective persona. Feed them from signed operating receipts. Let them adjust search, verification, abstention or escalation. Keep permission, policy and action execution in a separate authority kernel. Define what happens when the state is missing, stale or contradictory.

Then earn the word reflexive. Fork the system under identical task inputs. Clamp, swap and shuffle the state. Restore it. Inject a false receipt. Sweep update gain. Compare against a fixed heuristic and an input-only policy. Record the result as a typed claim: the state is causal or not, timely or not, stable or not. A failed gate is useful because it tells you whether to remove the feature, redesign the compression or narrow the claim.

For release, the practical threshold is strict. The live RCS should improve a decision-relevant outcome over ablated and simple-rival controls. Its benefit should survive paraphrase and task-interface changes. Its perturbations should decay. Its provenance should be reconstructable. Its authority should remain bounded. Its public language should say “causal allocation state” rather than “the model knows itself” unless further evidence supports that stronger statement.

For consciousness research, the decision is equally important. Functional reflexivity is worth measuring because some theories may treat self-representation or metacognitive availability as relevant. Yet it remains one indicator under a declared theory and candidate boundary. The architecture closes an engineering loop; it does not close the explanatory gap.

The infinite regress was never defeated by finding the final observer. It was dissolved by refusing to build one. A system can represent enough of its own allocation process to regulate that process, provided the representation is compressed, causally used, temporally entitled, stable and subordinate to authority. That is a smaller claim than machine self-awareness. It is also a claim we can build, break and verify.