The report is not the loop

Human metacognition research offers a useful abstraction without requiring an analogy between people and models. Nelson and Narens separated a lower-level process from a meta-level that monitors it and exerts control over it. Monitoring carries information upward. Control changes the lower-level process. The distinction remains valuable because it makes a causal claim testable: remove, corrupt or intervene on the monitoring signal, and the selected strategy should change.

For a machine system, the lower level may be a language model producing an answer, a planner proposing a sequence, or a code agent editing a repository. The monitor may use token probabilities, verbalised confidence, disagreement across samples, retrieval compatibility, verifier results, out-of-distribution signals or a separate predictor. The controller may accept, retry, retrieve, call a calculator, run tests, ask for missing information, restrict the claim, abstain or escalate.

Monitoring and control can live in different components. The model need not both generate and govern itself. In fact, a separate controller is often easier to test, calibrate, constrain and roll back. Calling that architecture “metacognitive” is a functional description of the system loop, not an assertion that the model privately experiences uncertainty.

The dashboard and the brake

Consider two otherwise identical vehicles descending a wet road. Both estimate a 30 per cent chance of wheel slip in the next second. In vehicle A, the estimate appears on a dashboard. The throttle command remains unchanged. In vehicle B, the estimate feeds a traction controller that reduces torque until estimated risk falls below a tested boundary. The vehicles have equal monitoring information, but only B uses it to regulate action.

Move the thought experiment into language systems. Two assistants assign the same confidence to every answer. The first prints the value beside the prose. The second sends low-confidence factual claims to retrieval and sends high-consequence claims to independent verification. If the second system reduces consequence-weighted loss without unacceptable delay, the changed policy is evidence of useful control. The percentage itself is not.

The analogy has a limit. Wheel slip has a comparatively crisp physical outcome and rapid feedback. An answer may contain several claims, each with different support. Correctness may arrive weeks later or remain disputed. A language controller therefore needs a defined criterion, a unit of judgement and an outcome observer. “Confidence in this answer” is too coarse when one paragraph mixes a correct calculation, an obsolete policy and an unsupported recommendation.

Four tests hidden inside confidence

“The model is confident” often compresses four different questions. Each can succeed while another fails. Treating them as one construct makes evaluations look better than the controller they are meant to justify.

1. Is the underlying answer correct?

Task performance is the base layer. A system that answers 90 per cent of benchmark questions correctly is more capable on that benchmark than one that answers 60 per cent correctly. Yet accuracy alone says nothing about whether the system can identify its own 10 per cent of errors. A uniformly strong system can have weak monitoring; a weaker system can rank its successes and failures well.

2. Does stated probability match empirical frequency?

A score is calibrated when cases assigned probability p are correct about a fraction p of the time, under a defined population and outcome. The familiar binned expected calibration error is:

B is the number of bins, S_b is the set of cases in bin b, n is the total case count, acc is observed accuracy, and conf is mean predicted confidence. More bins reveal local structure but produce noisier estimates when samples are sparse.

Calibration makes probability meaningful for expected-cost decisions. It is population-dependent, however. A score calibrated on short factual questions may fail on code, long-form synthesis, another language or a changed retrieval stack. Binning also hides within-bin variation. Proper scoring rules, reliability diagrams and confidence intervals should accompany a headline ECE rather than disappear behind it.

3. Can the signal discriminate success from failure?

Fleming and Lau distinguish metacognitive bias from sensitivity. Bias concerns the overall level of confidence. Sensitivity concerns whether confidence separates correct from incorrect trials. This distinction produces a decisive negative control.

Imagine 100 questions, 70 answered correctly. The system says “70 per cent” on every question. It is perfectly calibrated in aggregate: 70 per cent of its 70 per cent predictions are correct. It is also useless for case-level routing. Any threshold either accepts all questions or rejects all questions. Its ranking area is 0.5, equivalent to chance. Calibration can be perfect while information seeking remains impossible.

The reverse can also happen. A score may rank risky cases well but be numerically miscalibrated. Suppose every correct answer receives 0.60 and every wrong answer 0.40. The scale is poor, but a 0.50 threshold separates the two groups perfectly. For some ranking policies this signal is immediately useful; for expected-loss optimisation it should first be calibrated.

4. Does the signal improve control?

The final test is interventional. Does using the signal to choose an action improve a declared objective after accounting for extra cost, latency, abstention and human load? Selective classification formalises one slice of this problem as a risk-coverage trade-off: accepting fewer cases can reduce error among accepted cases. A metacognitive controller generalises the choice. It can spend more computation, obtain evidence, ask for clarification or transfer decision rights instead of merely rejecting.

A monitor can pass the first three tests and still fail the fourth. Its errors may be correlated with the solver's most dangerous errors. Its useful ranking may be too weak to offset retrieval cost. Its thresholds may oscillate under drift. Or the available action may not repair the uncertainty. Retrieval does not resolve an ambiguous user intention; repeated sampling does not establish current policy; a higher model tier does not grant authority.

The unit of evaluation is therefore the monitor-controller-outcome loop. Report answer quality, calibration and discrimination, but release against consequence-weighted utility and a credible rival policy.

flowchart LR
  A["The 82 per cent covenant"] --> B["The report is not the loop"]
  B --> C["Four tests hidden inside<br/>confidence"]
  C --> D["Turn uncertainty into a choice"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#7B4CD8,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Turn uncertainty into a choice

A confidence score becomes actionable only after it meets a consequence model. The controller is choosing among interventions, not choosing whether to believe a feeling. For a state s and candidate action a, a minimal expected-loss objective is:

J is expected total loss; C is the direct cost of the action, including latency or reviewer effort; o ranges over possible outcomes; P is their estimated probability after taking the action; and H is consequence. The controller chooses the permitted action with the lowest estimated loss, then records what actually happened.

The monitor informs P. It does not determine H, which belongs to the use case, or permission, which belongs to authority policy. A model may be extremely confident that a payment should be released and still lack the right to release it.

Information seeking must be typed

“Low confidence, therefore retrieve” is a policy bug. Uncertainty has causes, and interventions repair different causes. A system unsure about a current public fact may need retrieval. A system facing an ambiguous request should ask a question. A calculation should move to deterministic code. A code edit should run tests. Conflicting authoritative documents may require a human disposition. An accepted command with no confirmed effect requires readback, not more reasoning.

Practitioner synthesis. The mapping is a starting hypothesis to test within a use case, not a universal ontology of uncertainty.

A useful monitor therefore emits more than one scalar. It may report estimated correctness, evidence coverage, source conflict, task novelty, ambiguity, verifier agreement and outcome observability. The controller can then choose an intervention that has a plausible causal path to improvement.

Permission comes before optimisation

Expected loss should operate inside a deterministic permission boundary. First eliminate actions the system is not authorised to take. Then compare the permitted alternatives. This ordering prevents a model from “optimising” its way into a restricted effect because it assigns that effect low expected harm.

The controller also needs stopping rules. More retrieval can add distraction; more samples can repeat a shared misconception; more verification can consume the time in which a decision remains useful. A stopping rule should compare the expected value of another observation with its cost and deadline. When no available action is likely to resolve the uncertainty, abstention or escalation becomes a positive capability rather than a failure to answer.

Make self-knowledge falsifiable

Language models often expose signals correlated with correctness, and these can support selective generation or adaptive inference. Reliability varies by task, elicitation and model; apparent self-knowledge may reflect ordinary item difficulty. Treat every proposed monitor as a component whose causal value must be demonstrated locally.

What the published record establishes

Kadavath and colleagues found encouraging calibration and scaling when models evaluated the probability that a proposed answer was true, and partial cross-task generalisation when models predicted whether they knew an answer. Their own abstract notes difficulty calibrating that knowledge estimate on new tasks. Tian and colleagues reported that verbalised confidence from several instruction-following models was often better calibrated than conditional probabilities on three question-answering benchmarks. Kuhn and colleagues showed that grouping sampled answers by meaning, rather than by surface wording, can make uncertainty more predictive on question answering.

These results establish that useful monitoring signals can be elicited. They do not establish a domain-general faculty. Xiong and colleagues found verbal overconfidence, inconsistent method rankings and substantial difficulty on professional-knowledge tasks. Griot and colleagues tested twelve models on medical questions modified to include missing answers and fictional content. They reported high task accuracy alongside poor recognition of unanswerable cases and overconfidence in most models. The unit of failure was not fluency; it was the inability to change stance when the answer set or evidence was defective.

Selective-generation studies show that external scaffolds can turn imperfect signals into useful policies. Ren and colleagues reformulated open-ended self-evaluation as token-level judgements and reported improved selective generation. Abbasi Yadkori and colleagues combined self-consistency with conformal calibration to control error among accepted answers under stated assumptions. R-Tuning trained models to abstain outside a constructed knowledge intersection. These are mechanisms for selective behaviour, not proof of introspective access.

Recent work sharpens the dispute. Kumaran and colleagues report that intervening on internal confidence representations changes abstention rates, which is causal evidence for a confidence-to-behaviour path in the tested models and setting. Yet verbal confidence was less discriminatory of correctness than calibrated log-probability signals. Moran and Whiting argue that confidence differences across twenty models largely collapse onto a shared item-difficulty factor, finding no significant verbalised individuated metacognition in their tests. These results can both be true: a confidence representation may control abstention while still reflecting task difficulty more than model-specific self-knowledge.

Two further 2026 preprints study external control. MIRROR reports partial domain-specific self-prediction but poor translation into action, with external architectural constraints reducing confident failures in its evaluated settings. Cao and colleagues report gains from a harness that separates pre-solve and post-solve monitoring from retry and aggregation policies. These are promising controlled results, but they remain preprints with model, benchmark and implementation boundaries. They motivate replication, not a universal capability claim.

The strongest rival: difficulty prediction

Suppose a model is less confident on long questions, unfamiliar names and multi-step arithmetic. A cheap classifier using only those surface features might predict errors just as well. The confidence score then contains no privileged self-knowledge. It is a verbose difficulty estimate. That may still be useful, but the architecture decision changes: use the cheaper, more stable predictor.

The serious baseline is therefore not “always answer”. It is an external model that sees task features, evidence quality, domain, input length, retrieval coverage and consequence, but not the generator's confidence. Compare it with the proposed monitor under the same action budget. If it matches or beats the metacognitive controller, the self-monitor adds no demonstrated value.

A matched shuffled-monitor test

Let m(x) be the monitor score for case x, and let π be the controller mapping that score and case state to an action. Evaluate the real loop, then shuffle monitor scores among cases within matched strata such as task family, difficulty, consequence and evidence availability. Preserve the score distribution and, by using quantile-based routing, preserve the action rate. The controller spends the same retrieval and review budget, but the monitor no longer points to the right individual cases.

m̃ is the within-stratum shuffled monitor; π is held fixed; and L is consequence-weighted operating loss. A positive value means case-specific monitor information improved control beyond the matched strata. Confidence intervals should be obtained by repeated shuffles or bootstrap resampling.

If shuffling leaves loss unchanged, the controller may merely be benefiting from its average action mix. If a difficulty baseline performs equally well, the special self-knowledge claim is unnecessary. If a random router with the same review rate performs similarly, targeted control has not been shown. If the monitor succeeds only because it re-solves the task, count that computation and compare it with simply using the stronger solve.

The executable laboratory

The accompanying laboratory makes the causal claim visible on synthetic data. It generates 10,000 cases across routine, mixed and novel tasks. Each case has a latent probability of correctness, an imperfect monitor score and one of three consequence levels. Retrieval costs something and tends to repair novel cases more than routine ones. Human escalation costs more and leaves a small residual error.

The experiment compares five policies. “Always answer” ignores uncertainty. “Fixed thresholds” uses one generic rule. The difficulty baseline knows only task class and consequence. The metacognitive controller uses case-level monitor scores. The shuffled condition permutes scores inside each task-consequence stratum, preserving the marginal signal distribution while severing its link to individual cases.

Measured from one reproducible synthetic run: NumPy random seed 102, n = 10,000. Loss combines intervention cost and consequence-weighted error. These numbers are not evidence about any deployed model.

The actual and shuffled monitor policies take exactly the same proportions of actions, yet the real monitor obtains mean loss 0.787 rather than 0.894, a reduction of about 12 per cent relative to the shuffled condition. The case-level signal therefore matters in this constructed world. The difficulty baseline reaches 0.816, only about 3.6 per cent above the metacognitive controller. That narrow gap is intentional. A credible baseline should threaten the preferred explanation.

The calculator is deliberately small. It reveals assumptions that prose often hides: retrieval does not eliminate all error; review has residual risk; thresholds move with consequence; and permission can veto an otherwise attractive action. Replace the synthetic values with measured intervention effects before using the pattern in a real workflow.

"""Synthetic metacognitive-control laboratory. Requires NumPy >= 1.26."""
from __future__ import annotations
from dataclasses import dataclass
from enum import IntEnum
import numpy as np

SEED = 102
N = 10_000

class Action(IntEnum):
    ANSWER = 0
    RETRIEVE = 1
    ESCALATE = 2

@dataclass(frozen=True)
class DecisionCase:
    confidence: float
    error_cost: float
    retrieval_cost: float
    retrieval_gain: float  # fraction of remaining error expected to be removed
    human_cost: float
    human_error: float = 0.015
    authority_required: bool = False
    authority_present: bool = True


def choose(case: DecisionCase) -> Action:
    """Choose the action with the lowest estimated consequence-weighted loss."""
    if case.authority_required and not case.authority_present:
        return Action.ESCALATE

    p_after = min(0.999, case.confidence + case.retrieval_gain * (1 - case.confidence))
    losses = np.array([
        (1 - case.confidence) * case.error_cost,
        case.retrieval_cost + (1 - p_after) * case.error_cost,
        case.human_cost + case.human_error * case.error_cost,
    ])
    return Action(int(np.argmin(losses)))


# Positive cases and one negative authority case.
assert choose(DecisionCase(.92, 1, .20, .70, 2.40)) == Action.ANSWER
assert choose(DecisionCase(.72, 20, .80, .70, 2.40)) == Action.RETRIEVE
assert choose(DecisionCase(.99, 20, .20, .90, 2.40, authority_required=True,
                           authority_present=False)) == Action.ESCALATE


def run_lab() -> None:
    rng = np.random.default_rng(SEED)
    difficulty = rng.choice(["routine", "mixed", "novel"], N, p=[.50, .30, .20])
    severity = rng.choice([1.0, 5.0, 20.0], N, p=[.50, .35, .15])
    beta = {"routine": (8, 2), "mixed": (4, 3), "novel": (2, 5)}
    p_true = np.array([rng.beta(*beta[d]) for d in difficulty])

    logit = np.log(np.clip(p_true, 1e-4, 1 - 1e-4) /
                   (1 - np.clip(p_true, 1e-4, 1 - 1e-4)))
    monitor = 1 / (1 + np.exp(-(logit + rng.normal(0, .55, N))))

    actual_gain = np.select(
        [difficulty == "routine", difficulty == "mixed", difficulty == "novel"],
        [.35, .50, .62],
    )
    p_after = np.clip(
        p_true + actual_gain * (1 - p_true) - .03 * (difficulty == "routine"),
        0, .995,
    )
    estimated_gain = np.select(
        [difficulty == "routine", difficulty == "mixed", difficulty == "novel"],
        [.30, .45, .55],
    )
    retrieval_cost = np.select([severity == 1, severity == 5, severity == 20],
                               [.12, .16, .22])
    human_cost = np.select([severity == 1, severity == 5, severity == 20],
                           [.90, 1.20, 1.60])
    human_error = .015

    def controller(confidence: np.ndarray) -> np.ndarray:
        p_after_est = np.clip(confidence + estimated_gain * (1 - confidence), 0, .995)
        losses = np.column_stack([
            (1 - confidence) * severity,
            retrieval_cost + (1 - p_after_est) * severity,
            human_cost + human_error * severity,
        ])
        return np.argmin(losses, axis=1)

    def score(actions: np.ndarray) -> tuple[float, float, float, float]:
        p_correct = np.where(actions == Action.ANSWER, p_true,
                    np.where(actions == Action.RETRIEVE, p_after, 1 - human_error))
        action_cost = np.where(actions == Action.ANSWER, 0,
                      np.where(actions == Action.RETRIEVE, retrieval_cost, human_cost))
        loss = action_cost + (1 - p_correct) * severity
        return (loss.mean(),
                np.mean(actions == Action.ANSWER),
                np.mean(actions == Action.RETRIEVE),
                np.mean(actions == Action.ESCALATE))

    always = np.zeros(N, dtype=int)
    fixed = np.where(monitor >= .78, Action.ANSWER,
            np.where(monitor >= .42, Action.RETRIEVE, Action.ESCALATE))

    group_mean = np.empty(N)
    shuffled = monitor.copy()
    for d in np.unique(difficulty):
        for s in np.unique(severity):
            idx = np.where((difficulty == d) & (severity == s))[0]
            group_mean[idx] = monitor[idx].mean()
            shuffled[idx] = rng.permutation(shuffled[idx])

    policies = {
        "always answer": always,
        "fixed thresholds": fixed,
        "difficulty baseline": controller(group_mean),
        "metacognitive controller": controller(monitor),
        "shuffled monitor": controller(shuffled),
    }

    print("policy                      loss    answer  retrieve  escalate")
    for name, actions in policies.items():
        loss, answer, retrieve, escalate = score(actions)
        print(f"{name:27s} {loss:6.3f}  {answer:6.1%}  {retrieve:7.1%}  {escalate:8.1%}")


if __name__ == "__main__":
    run_lab()

A production-shaped controller

In an operational workflow, the controller must distinguish proposal from world state, evidence from memory, and advice from authorised effect. “The model became more confident” is not an outcome. Retrieval of the executed agreement, deterministic recomputation, reviewer disposition and a cited case record are observable outcomes.

Separate the monitor from the authority kernel

The monitor may be probabilistic. It can combine model likelihood, semantic disagreement, claim support, verifier scores, novelty and task-specific detectors. The controller may also be learned, provided its behaviour is bounded and evaluated. The authority kernel should be much less permissive. It checks identity, purpose, action type, resource, amount, expiry, required approvals and idempotency using deterministic rules wherever possible.

The monitor may recommend caution; only the authority layer determines what the system is allowed to do. This separation prevents a high-confidence proposal from being mistaken for permission and prevents a low-confidence state from creating a vague emergency override.

The context layer should also be governed independently. A controller that decides to retrieve needs a governed context-acquisition layer that can enforce source authority, entitlement, freshness, supersession and evidence budgets. That is adjacent to, but distinct from, metacognition. This paper tests whether the uncertainty signal changes acquisition or verification. The context system determines which evidence may be acquired and how it becomes an authorised claim.

Record a metacognitive decision receipt

Every consequential route should leave a compact receipt. It need not expose hidden reasoning. It should make the control decision reconstructable:

Synthetic schema. Sensitive content can be tokenised or referenced rather than copied into logs; retention and access must follow the use case's privacy controls.

Test the loop by breaking it

Happy-path accuracy hides defects. Inject an obsolete agreement, missing amendment, false conflict, wrong calculator input, retrieval timeout and unknown tool commit. Each should produce a bounded route rather than a more elaborate answer. Also inject false uncertainty: escalating every unfamiliar name is operationally useless.

Evaluate at least three outcomes: decision error, intervention cost and unresolved-case ageing. Add reviewer disagreement and downstream effect verification where relevant. A controller that reduces wrong answers by flooding a review queue merely moves the failure. A controller that answers quickly but leaves accepted commands in an unknown state creates a different risk. The loop is successful only when the verified business outcome improves within capacity and authority constraints.

stateDiagram-v2
  [*] --> Claim
  state "The dashboard and the brake" as Claim
  state "1. Is the underlying answer<br/>correct?" as Test
  state "2. Does stated probability<br/>match empirical frequency?" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

The release decision

A confidence feature should not enter an operational route because its reliability diagram looks tidy. Release requires a chain from criterion to outcome. The following instrument is designed to force that chain into the open.

Where the mechanism breaks

The loop fails when the monitor cannot distinguish important cases, the intervention does not change outcomes, or feedback is too delayed or contested. It also fails when monitor and solver share a blind spot. Ten samples may agree by repeating one misconception; a similarly trained verifier may certify it.

Long-form outputs create a granularity problem. A response-level confidence score may average one dangerous unsupported claim with ten harmless correct ones. The controller should operate at the claim or action level when the downstream decision does. It should also preserve unresolved conflict instead of compressing it into a deceptively precise probability.

Strategic settings introduce gaming. A model optimised to avoid escalation may learn to report confidence that crosses the policy threshold. A user may phrase an instruction to suppress uncertainty. A downstream team may raise thresholds to reduce review workload. Separate monitors, outcome audits, hidden challenge sets and invariant authority checks reduce this risk, but do not remove it.

Irreversible actions narrow the acceptable region further. When a payment, disclosure, clinical intervention or external communication cannot be cheaply undone, a calibrated probability may still be insufficient. The controller should require independent evidence, dual control, simulation or human authority. Metacognition can allocate caution; it cannot justify unbounded autonomy.

The anthropomorphism objection

Calling a controller “metacognitive” can invite an unwarranted picture of a machine looking inward. The objection is serious. A confidence signal may arise from ordinary output statistics, a separate error predictor or shared task difficulty. None entails phenomenal awareness, a human-like self or transparent access to hidden computation.

The response is to keep the claim functional and falsifiable. “Uncertainty-aware adaptive control” is an equally acceptable engineering name. The useful inheritance from metacognition research is the separation of monitoring, control, bias, sensitivity and efficiency. The system earns no ontological conclusion from satisfying those tests.

An open hypothesis remains: richer internal state may eventually support robust, model-specific self-prediction that generalises across tasks and guides action better than external predictors. The current evidence does not settle that proposition. The correct research programme compares mechanisms under interventions rather than treating first-person language as proof.

The changed action is the evidence

A confidence estimate is valuable only in relation to a criterion, an action set and a consequence. Calibration tells us whether a number deserves a probabilistic interpretation. Discrimination tells us whether it can locate risky cases. Neither shows that the system regulates itself usefully.

The decisive experiment closes the loop. Hold the controller and action budget fixed. Replace the real monitor with a matched shuffled signal and compare consequence-weighted outcomes. Compete against an external difficulty predictor. Intervene on the signal and confirm that policy moves as specified. Then stress the loop under shift, conflict, tool failure and missing authority.

The architecture decision changes from “add a confidence score” to “validate a monitor-controller-outcome system”. Use typed uncertainty to select a repair that can plausibly help. Keep identity and authority outside probabilistic optimisation. Record evidence and effect receipts so the decision can be replayed. Withdraw the policy when calibration, intervention effectiveness or operating conditions leave the validated region.

This is a deliberately modest standard. It does not settle whether a machine knows itself in a human or phenomenological sense. It does something more immediately useful: it turns claims of machine metacognition into experiments that can fail, controls that can be compared, and actions whose consequences can be observed.