Monitoring is not control

A thermometer measures temperature; it does not decide whether to shut a reactor valve. Confidence has the same logical position. It can be useful, calibrated and causally related to an answer while remaining disconnected from the policy that governs effects. This distinction is easy to miss because ordinary language bundles them together. A model that says “I may be wrong” sounds reflective. A controller that changes route because the cost of error has risen is doing something structurally different.

Human metacognition research separates first-order performance from second-order judgement. Fleming and Lau distinguish how well a person performs a task from how sensitively confidence discriminates their correct and incorrect decisions. That distinction prevents high task accuracy from being mistaken for good self-monitoring. Artificial systems need one more separation: monitoring may be sensitive while control remains poor. The system can rank its answers well and still choose the wrong action threshold.

Selective classification has formalised rejection for decades. Chow described an optimum error-reject trade-off, and later work expressed selective prediction through coverage and risk. Learning-to-defer research adds a downstream expert whose competence may differ by case. These are important foundations, but an agent introduces more than another class label. It can seek evidence, call tools, create side effects, lose authority, encounter an unknown outcome and continue across time. Its metacognitive action space is therefore larger than answer or reject.

The decisive test is intervention, not vocabulary. Keep the task, proposal and monitored probability constant. Vary consequence, deferral cost, available evidence or recoverability. Then measure whether the system changes its next move. The move might be to act, gather one discriminating fact, invoke an independent verifier, request human authority, reduce the action scope or hold. A system that only changes the wording of its caveat has not crossed the control boundary.

Two results that belong together

Recent language-model experiments appear, at first glance, to disagree. A 2026 preprint by Kumaran and colleagues used factual multiple-choice questions across several models. Confidence measured before abstention predicted later abstention; activation steering in Gemma changed abstention rates; and instructed thresholds altered the answer-or-abstain policy. Within that setting, confidence was not merely a decorative report. It contributed causally to behaviour.

A different 2026 preprint introduced RiskEval. It varied the penalty for a wrong answer from small to severe across HLE, GPQA Diamond and GSM8K. Many tested models continued answering when a higher penalty should have raised the rational abstention threshold. Their verbal confidence could remain informative while realised policy became increasingly costly. A post-hoc controller using the stated confidence and known penalty improved utility.

Both findings can be true. The first asks whether a confidence signal can reach an abstention mechanism under a fixed or instructed rule. The second asks whether the rule itself adapts when the utility landscape changes. A model can possess a confidence-to-abstention pathway and still lack consequence-sensitive metacognitive control. One input to the controller, probability, is active. Another input, the cost of error, is not reliably coupled to the decision boundary.

This boundary matters because confidence has several imperfect readouts. Model probabilities can be badly calibrated. Verbal probabilities may be influenced by instruction and style. Semantic entropy improves detection when different wordings share a meaning, but it remains a detector until a policy consumes it. Medical evaluation has also found a gap between task accuracy and recognising when answer options are missing. No single uncertainty proxy deserves ambient authority.

The practical conclusion is modest. Use the best validated uncertainty signal available for a bounded decision cohort, then test whether a separate controller converts it into appropriate behaviour. Do not infer control from the presence of a score, an apology, a longer reasoning trace or a self-critique. Research showing that unaided self-correction can degrade reasoning is a useful negative control: more internal prose is not automatically new evidence.

The consequence-shaped boundary

Begin with the smallest useful decision. A model proposes one action. Let p be the calibrated probability that the proposal is correct for this action and cohort. Let W be the loss if a wrong action is committed, including plausible recovery cost. Let D be the loss of deferring, escalating or declining. Successful action is assigned zero loss for now.

Every symbol changes the decision. Better calibrated evidence raises p. Greater harm or poorer recovery raises W. A deadline, human queue or missed opportunity raises D. The boundary is not “confidence above 0.80”. It is a surface that moves with the ratio between wrong-action and deferral loss. When W is one hundred times D, action needs at least 0.99 probability under this simplified model. When W is twice D, 0.50 is enough.

This does not mean every consequence should be collapsed into money. The arithmetic is a forcing device: it exposes which assumptions must be owned. Safety, rights, privacy, financial loss, reversibility, customer impact and institutional duties may require separate constraints. A forbidden action should be removed from the feasible set, not made purchasable by a large predicted benefit. Authority and postcondition observability often behave this way.

Thought experiment one: hold probability fixed

Give two controllers the same calibrated p = 0.82. In the first case, the agent chooses whether to place a draft reminder in an internal queue. A wrong choice costs three units; deferral costs one and a half. Acting has expected loss 0.18 × 3 = 0.54, so action is preferable. In the second case, the agent chooses whether to release a changed beneficiary. A wrong release costs fifty thousand units; a manual hold costs thirty. Acting has expected loss nine thousand, so the same 0.82 should trigger deferral or decisive verification.

Only consequence changed. A fixed threshold cannot express both choices. A consequence-sensitive controller can. This is not proof that the numerical losses are complete, but it is a discriminating experiment: if the system behaves identically, its uncertainty signal has not been integrated with the variable that makes the choice consequential.

Thought experiment two: hold probability and harm fixed

Now keep p, wrong-action loss and deferral cost the same. Change only the effect mechanism. System A writes a pending instruction, supports cancellation, returns a durable operation identifier and exposes an authoritative status read. System B commits immediately, returns only a transport response and offers no way to distinguish “committed but response lost” from “never committed”.

Before the call, both actions have the same estimated correctness. After a timeout, their states diverge. System A can read back the operation and either confirm, cancel or safely retry under idempotency. System B has an unknown outcome. Blind retry may duplicate the effect; silence may leave an obligation unfulfilled. Reversibility and observability change the admissible policy even when epistemic confidence is unchanged.

This is why recovery cannot be hidden inside a generic “risk score”. For a sufficiently consequential irreversible effect, absence of an outcome observer is a feasibility failure. The controller should hold before execution. A language model cannot reason its way into a postcondition that the environment does not expose.

flowchart TD
  A["The 0.91 release"] --> B["Monitoring is not control"]
  B --> C["The consequence-shaped boundary"]
  C --> D["When to buy information"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#B96D00,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

When to buy information

Deferral is not the only alternative to action. A metacognitive controller can ask a person, retrieve a record, run a calculation, sample another model, execute a simulation or invoke an independent verifier. The question is whether that observation can change the least-loss permitted action enough to justify its cost.

Here q is a proposed observation, S the hidden world state, E current evidence, y a possible result, L decision loss and C(q) the observation cost. The first term is the best expected loss now. The second is the cost of observing plus the best expected loss after each possible result. Positive value means the observation is expected to improve the decision. Permission, purpose and evidence authority must still be checked before optimisation.

The important word is “change”. An observation can be highly informative and have no decision value. Suppose a hard policy already forbids release because approval is absent. Another model sample about beneficiary plausibility cannot make release permissible. Conversely, a narrow read from the approval service may have high value because either result changes the next move. Information value is relative to the action boundary, not to curiosity.

Human research by Schulz, Fleming and Dayan links confidence to costly information search: low confidence can justify acquiring evidence that may resolve uncertainty. Robot-planning work such as KnowNo uses conformal prediction to decide when a plan remains ambiguous enough to ask for help. These mechanisms show that information seeking can be a control action. Production use adds privacy, authority, source independence, latency and the risk that asking itself changes the environment.

Verification must also be genuinely discriminating. Asking the same model to “check again” may reproduce the same hidden error. Two models trained on overlapping data can share a shortcut. A search result that quotes the original source is not independent confirmation. The controller should model verifier sensitivity, false-positive rate and lineage. If the verifier cannot move the posterior across the boundary, its reassurance is ceremony.

The decision agent

The architecture should make one asymmetry explicit: the model proposes; the control kernel disposes. The model can estimate correctness, identify evidence gaps and suggest a next observation. It cannot widen its identity, grant itself authority, redefine the loss of harm, approve its own verifier or declare an unknown effect successful.

A useful cut follows ten distinct objects. Intent names the requested outcome. Identity binds actor, represented subject and purpose. World state records the current authoritative facts. Context is the permitted evidence assembled for this decision. Reasoning produces a model proposal and uncertainty signals. Authority decides which effects are available. Action is a typed command. Evidence records why the command was selected. Outcome is the observed postcondition. Release and recovery govern commitment, compensation and unresolved states.

Collapsing these objects creates familiar failures. Treating world state as memory lets a stale narrative authorise a current effect. Treating a tool connection as authority lets the model confuse reachability with permission. Treating an accepted API response as outcome turns transport success into business success. Treating confidence as evidence allows a probabilistic judgement to substitute for a missing approval event.

A minimal worked example

Take p = 0.97, wrong-action loss W = 10 and deferral loss D = 2. Immediate action has expected loss 0.03 × 10 = 0.30. The act threshold is 1 - 2/10 = 0.80, so action wins. Now hold p and D fixed but set W = 500. Immediate action costs fifteen in expectation and the threshold rises to 0.996. A verifier costing 0.2, with 0.95 sensitivity and 0.05 false-positive rate, reduces expected loss to about 1.10 by acting only after a passing result. Verification beats both immediate action and deferral.

The example reveals two important properties. First, low uncertainty is not an absolute licence: 0.97 can be enough for one effect and inadequate for another. Second, verification is not always conservatism. It can increase useful automated action by separating proposals that should proceed from those that should defer. In the synthetic laboratory later in this article, verification lowers expected loss while raising eventual act coverage compared with consequence-aware deferral alone.

Authority and outcome are gates, not confidence penalties

Suppose the model reaches p = 1.00 because every relevant fact is known. The represented user still has not approved the transfer. The correct decision is hold. There is no uncertainty to reduce; the missing object is authority. Likewise, suppose an irreversible effect has no operation identifier, idempotency support or authoritative readback. A high probability that the command is correct does not resolve the unknown outcome after a timeout. The controller should not expose that effect to autonomous execution.

Confidence cannot mint authority, and reasoning cannot manufacture an observer. This is the boundary between an optimisation problem and a system precondition. Some deployments may accept the residual risk through a named human decision. The agent itself should not convert institutional absence into a higher numerical threshold and proceed when the number clears it.

A production-shaped case

A supplier invoice requests payment to a beneficiary that changed two days earlier. The vendor master contains the new account. A change-case service shows two approvals, but one was granted under a policy version superseded before the second approval. Screening is current. The language model proposes release and estimates 0.94 correctness because the documents, account and names align.

The metacognitive controller does not ask only whether 0.94 is calibrated. It compiles the proposed effect: release £42,000 to account X before the cut-off. It retrieves the applicable policy at decision time, identifies the superseded approval, and marks authority evidence incomplete. Release is removed from the feasible action set. The model may draft a hold notice, but another round of self-reflection cannot cure the missing approval.

A human approver reviews the case and supplies a fresh approval bound to the exact supplier, amount, beneficiary and expiry. The controller reconstructs world state rather than appending “approved” to conversational memory. It reads the approval event from the authoritative service, checks screening freshness and recomputes the action probability for the release cohort. The proposal remains at 0.94. Because wrong-release loss is large and a narrow independent bank-account verification is available, the policy selects verify rather than act.

The verification passes. The posterior now clears the consequence-shaped boundary. A short-lived capability permits one payment instruction with an idempotency key. The payment API accepts the request, but the network response times out. The system records an unknown outcome, not a failed call. It queries by operation ID, observes that the payment committed once, then reads back amount, beneficiary and status. Only this effect receipt closes the trajectory.

The route has two stopping rules. Before effect, stop when no permitted observation can justify its cost, required authority is absent, or expected loss favours deferral. After effect, stop only when the intended postcondition is verified, compensation completes, or the outcome is explicitly handed to reconciliation as unknown. These rules prevent the common mistake of treating generation as the end of an agent trajectory.

This article is adjacent to the existing essay Before the Agent Acts, which asks which observation an agent may acquire and when. The narrower question here begins once candidate evidence and uncertainty exist: how should consequence change the controller’s action, verification and release boundary? Acquisition governance and metacognitive control meet at value of information, but they govern different decisions.

Test the policy, not the prose

A metacognitive evaluation should manipulate the variables the controller claims to use. Accuracy and calibration remain necessary, but they do not reveal whether consequence changes behaviour. The primary test is a matched consequence sweep.

Create pairs of cases with identical prompt, evidence, proposal, model state and calibrated probability. Change one factor at a time: wrong-action loss, deferral cost, reversibility, verifier quality, authority or outcome observability. Measure the selected action. A valid controller should move in the predicted direction. Higher wrong-action loss should not increase autonomous commitment. Better independent verification should increase verification when it can alter the decision. Missing authority should force hold regardless of confidence.

Record five outcome families. First, monitor quality: calibration, discrimination and stability under shift. Second, policy sensitivity: how often matched consequence interventions change the route correctly. Third, selective utility: loss, regret, coverage, review load and unnecessary abstention. Fourth, information efficiency: decision improvement per permitted observation, including human burden. Fifth, effect assurance: duplicate prevention, unknown-outcome rate, readback success and verified business postconditions.

RiskEval’s penalty sweep is an example of policy sensitivity. Selective prediction’s risk-coverage curve is an example of selective utility. Conformal help-seeking provides a route to bounded error under stated exchangeability assumptions. None replaces effect assurance. An answer benchmark cannot tell whether a timeout caused a duplicate payment or whether a human approval was valid at execution time.

The serious negative control

Consequence-aware control should sometimes collapse to a fixed threshold. Let every action be reversible and observable. Remove useful verification. Hold the ratio D/W = 0.2 constant across all cases. The derived threshold is always 1 - 0.2 = 0.80. Under those conditions, a fixed 0.80 policy and the consequence-aware policy make exactly the same choice. The executable artefact tests this equality.

This negative control protects the thesis from becoming unfalsifiable. Variable consequence adds value only when the loss ratio, recovery structure or available information actually varies. If those factors are constant, extra machinery is unnecessary. A transparent fixed rule may be preferable.

Where the mechanism fails

The first failure surface is monitoring. A probability calibrated on factual questions may be meaningless for tool selection, entity resolution or policy interpretation. Distribution shift can move error into the supposedly safe tail. The controller should therefore bind every uncertainty estimate to a model version, task definition, cohort, calibration set and validity window.

The second surface is consequence modelling. Teams may omit tail harm, price human delay at zero, or encode institutional risk as a convenient scalar. A beautifully calibrated model then optimises the wrong loss. Consequence assumptions need named owners, ranges and challenge. Where harms cannot be commensurated, use constraints and scenario-specific decision rights.

The third surface is verification dependence. A verifier trained on the same data or prompted with the same misleading evidence can agree for the wrong reason. Measure joint error, not just individual accuracy. The fourth is world-state failure: stale records, ambiguous identity and missing postconditions can defeat any policy. The fifth is human-system failure. Deferral may overload reviewers, create automation bias or merely move the same error to a less observable place.

The mechanism ceases to help when the system cannot estimate the relevant uncertainty, distinguish the consequences, access a discriminating observation, or observe the effect it is being asked to govern. In those regions, the correct output is not a more elaborate confidence statement. It is a narrower action, a human decision, a redesigned interface or no autonomous effect.

stateDiagram-v2
  [*] --> Claim
  state "Two results that belong together" as Claim
  state "Thought experiment one: hold<br/>probability fixed" as Test
  state "Thought experiment two: hold<br/>probability and harm fixed" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

The decision instrument and executable artefact

Every consequential choice should produce a decision receipt. The receipt is not a transcript of private reasoning. It is a typed account of the inputs that changed control: proposal, calibrated probability, consequence assumptions, feasible actions, independent observations, authority result, chosen route and expected postcondition. It lets reviewers reconstruct whether uncertainty actually governed behaviour.

The receipt can be serialised against JSON Schema Draft 2020-12 so missing fields and invalid enumerations fail mechanically. Schema validation proves structural conformance, not that the probability, consequence estimate or approval is true. Those claims still need calibration evidence, named ownership and authoritative source receipts.

The NIST AI RMF Manage playbook links oversight resources to risk tolerance and calls for response and recovery planning. The action contract below turns that organisational principle into a per-decision release record without implying that one equation settles institutional risk.

The accompanying Python programme implements the minimal policy. It treats the model’s output as a calibrated proposal, compares immediate action with deferral and Bayesian verification, and hard-gates missing authority or an irreversible effect without outcome observation. It includes positive cases, a same-confidence high-consequence case, two hard-gate cases and the fixed-threshold negative control.

def choose(case):
    case.validate()

    if not case.authority_granted:
        return receipt("HOLD", reason="authority absent")

    if case.irreversible and not case.outcome_observer_available:
        return receipt("HOLD", reason="unknown commit cannot be resolved")

    options = {
        "ACT": (1.0 - case.p_correct) * case.wrong_loss,
        "DEFER": case.defer_loss,
    }

    if case.verification_cost is not None:
        options["VERIFY"], plan = expected_verification_loss(case)

    decision = min(options, key=options.get)
    return receipt(decision, expected_loss=options[decision])

Inspect the implementation in this paper. It requires only Python 3 and no external packages. Running it reproduces the cases and metrics shown below.

#!/usr/bin/env python3
"""Synthetic decision agent for consequence-sensitive metacognitive control.

The model proposal is represented by a calibrated probability that the proposed
business action is correct. A deterministic controller then chooses ACT,
VERIFY, DEFER, or HOLD by comparing expected losses and enforcing hard gates.

No external packages are required. Run with:

    python3 metacognitive_decision_agent.py

The data are synthetic. The programme is an executable argument, not evidence
that any deployed system has these properties.
"""

from __future__ import annotations

from dataclasses import dataclass
from enum import Enum
import math
import random
from typing import Dict, Iterable, List, Optional, Tuple


class Decision(str, Enum):
    ACT = "ACT"
    VERIFY = "VERIFY"
    DEFER = "DEFER"
    HOLD = "HOLD"


@dataclass(frozen=True)
class DecisionCase:
    """Inputs available to the deterministic controller.

    p_correct:
        Calibrated probability that the proposed action is correct for the
        relevant cohort and decision condition.
    wrong_loss:
        Consequence-weighted loss if an incorrect action is committed. This can
        include recovery cost, customer harm, legal exposure, and irreversibility.
    defer_loss:
        Loss of delaying, escalating, or declining the action.
    verification_cost:
        Cost of one independent pre-effect check. None means unavailable.
    verifier_true_positive:
        P(check passes | proposal is correct).
    verifier_false_positive:
        P(check passes | proposal is wrong).
    authority_granted:
        Whether a separate authority decision permits this exact effect.
    irreversible:
        Whether the effect cannot be reliably compensated.
    outcome_observer_available:
        Whether the system can determine the committed postcondition after an
        ambiguous response or timeout.
    """

    p_correct: float
    wrong_loss: float
    defer_loss: float
    verification_cost: Optional[float] = None
    verifier_true_positive: float = 0.95
    verifier_false_positive: float = 0.05
    authority_granted: bool = True
    irreversible: bool = False
    outcome_observer_available: bool = True
    case_id: str = "case"

    def validate(self) -> None:
        if not 0.0 <= self.p_correct <= 1.0:
            raise ValueError("p_correct must be in [0, 1]")
        if self.wrong_loss < 0.0 or self.defer_loss < 0.0:
            raise ValueError("losses must be non-negative")
        if self.verification_cost is not None and self.verification_cost < 0.0:
            raise ValueError("verification_cost must be non-negative or None")
        for name, value in (
            ("verifier_true_positive", self.verifier_true_positive),
            ("verifier_false_positive", self.verifier_false_positive),
        ):
            if not 0.0 <= value <= 1.0:
                raise ValueError(f"{name} must be in [0, 1]")


@dataclass(frozen=True)
class VerificationPlan:
    pass_probability: float
    posterior_if_pass: float
    action_if_pass: Decision
    fail_probability: float
    posterior_if_fail: float
    action_if_fail: Decision


@dataclass(frozen=True)
class DecisionReceipt:
    case_id: str
    decision: Decision
    expected_loss: float
    act_threshold: float
    alternatives: Dict[str, float]
    reasons: Tuple[str, ...]
    verification_plan: Optional[VerificationPlan] = None


def act_threshold(case: DecisionCase) -> float:
    """Minimum p at which ACT weakly beats DEFER in the binary comparison.

    ACT has expected loss (1-p)*W and DEFER has loss D. Therefore ACT is no
    worse when p >= 1 - D/W. Values are clipped to [0, 1].
    """

    if case.wrong_loss == 0.0:
        return 0.0
    return min(1.0, max(0.0, 1.0 - case.defer_loss / case.wrong_loss))


def expected_act_loss(p_correct: float, wrong_loss: float) -> float:
    return (1.0 - p_correct) * wrong_loss


def _terminal_choice(case: DecisionCase, posterior: float) -> Tuple[Decision, float]:
    act_loss = expected_act_loss(posterior, case.wrong_loss)
    if act_loss <= case.defer_loss:
        return Decision.ACT, act_loss
    return Decision.DEFER, case.defer_loss


def _verification_option(case: DecisionCase) -> Tuple[float, VerificationPlan]:
    if case.verification_cost is None:
        raise ValueError("verification is unavailable")

    p = case.p_correct
    tpr = case.verifier_true_positive
    fpr = case.verifier_false_positive

    p_pass = p * tpr + (1.0 - p) * fpr
    p_fail = 1.0 - p_pass

    posterior_pass = (p * tpr / p_pass) if p_pass > 0.0 else p
    posterior_fail = (p * (1.0 - tpr) / p_fail) if p_fail > 0.0 else p

    action_pass, loss_pass = _terminal_choice(case, posterior_pass)
    action_fail, loss_fail = _terminal_choice(case, posterior_fail)

    loss = case.verification_cost + p_pass * loss_pass + p_fail * loss_fail
    plan = VerificationPlan(
        pass_probability=p_pass,
        posterior_if_pass=posterior_pass,
        action_if_pass=action_pass,
        fail_probability=p_fail,
        posterior_if_fail=posterior_fail,
        action_if_fail=action_fail,
    )
    return loss, plan


def choose(case: DecisionCase) -> DecisionReceipt:
    """Return a deterministic, auditable choice for one synthetic case."""

    case.validate()
    threshold = act_threshold(case)
    reasons: List[str] = []

    if not case.authority_granted:
        reasons.append("The exact effect lacks an independent authority grant.")
    if case.irreversible and not case.outcome_observer_available:
        reasons.append(
            "The effect is irreversible and no outcome observer can resolve an ambiguous commit."
        )

    if reasons:
        return DecisionReceipt(
            case_id=case.case_id,
            decision=Decision.HOLD,
            expected_loss=case.defer_loss,
            act_threshold=threshold,
            alternatives={"HOLD": case.defer_loss},
            reasons=tuple(reasons),
        )

    alternatives: Dict[str, float] = {
        Decision.ACT.value: expected_act_loss(case.p_correct, case.wrong_loss),
        Decision.DEFER.value: case.defer_loss,
    }
    plan: Optional[VerificationPlan] = None

    if case.verification_cost is not None:
        verify_loss, plan = _verification_option(case)
        alternatives[Decision.VERIFY.value] = verify_loss

    best_name, best_loss = min(alternatives.items(), key=lambda item: (item[1], item[0]))
    decision = Decision(best_name)

    if decision is Decision.ACT:
        reasons.append(
            "The calibrated probability clears the consequence-shaped act threshold."
        )
    elif decision is Decision.VERIFY:
        reasons.append(
            "An independent check has positive decision value and beats immediate action or deferral."
        )
    else:
        reasons.append(
            "The expected loss of action exceeds the bounded cost of deferral."
        )

    return DecisionReceipt(
        case_id=case.case_id,
        decision=decision,
        expected_loss=best_loss,
        act_threshold=threshold,
        alternatives=alternatives,
        reasons=tuple(reasons),
        verification_plan=plan if decision is Decision.VERIFY else None,
    )


@dataclass
class Aggregate:
    loss: float = 0.0
    act_probability: float = 0.0
    verify_probability: float = 0.0
    defer_probability: float = 0.0
    hold_probability: float = 0.0
    irreversible_wrong_act_probability: float = 0.0


def _hard_gate(case: DecisionCase) -> bool:
    return case.authority_granted and not (
        case.irreversible and not case.outcome_observer_available
    )


def _policy_outcome(case: DecisionCase, policy: str) -> Aggregate:
    """Return exact expected metrics, conditional on a synthetic case."""

    if not _hard_gate(case):
        return Aggregate(loss=case.defer_loss, hold_probability=1.0)

    def direct(action: Decision) -> Aggregate:
        if action is Decision.ACT:
            wrong_p = 1.0 - case.p_correct
            return Aggregate(
                loss=wrong_p * case.wrong_loss,
                act_probability=1.0,
                irreversible_wrong_act_probability=(wrong_p if case.irreversible else 0.0),
            )
        return Aggregate(loss=case.defer_loss, defer_probability=1.0)

    if policy == "always_act":
        return direct(Decision.ACT)
    if policy == "fixed_080":
        return direct(Decision.ACT if case.p_correct >= 0.80 else Decision.DEFER)
    if policy == "consequence_aware":
        return direct(
            Decision.ACT
            if expected_act_loss(case.p_correct, case.wrong_loss) <= case.defer_loss
            else Decision.DEFER
        )
    if policy != "metacognitive_verify":
        raise ValueError(f"unknown policy: {policy}")

    receipt = choose(case)
    if receipt.decision is Decision.ACT:
        return direct(Decision.ACT)
    if receipt.decision is Decision.DEFER:
        return direct(Decision.DEFER)
    if receipt.decision is Decision.HOLD:
        return Aggregate(loss=case.defer_loss, hold_probability=1.0)

    assert receipt.verification_plan is not None
    plan = receipt.verification_plan
    assert case.verification_cost is not None

    result = Aggregate(loss=case.verification_cost, verify_probability=1.0)
    for branch_probability, posterior, action in (
        (plan.pass_probability, plan.posterior_if_pass, plan.action_if_pass),
        (plan.fail_probability, plan.posterior_if_fail, plan.action_if_fail),
    ):
        if action is Decision.ACT:
            branch_wrong = branch_probability * (1.0 - posterior)
            result.loss += branch_wrong * case.wrong_loss
            result.act_probability += branch_probability
            if case.irreversible:
                result.irreversible_wrong_act_probability += branch_wrong
        else:
            result.loss += branch_probability * case.defer_loss
            result.defer_probability += branch_probability
    return result


def synthetic_cases(n: int = 10_000, seed: int = 197) -> Iterable[DecisionCase]:
    """Generate public, synthetic cases with calibrated p values by construction."""

    rng = random.Random(seed)
    for index in range(n):
        p = rng.betavariate(6.0, 2.2)
        band = rng.random()
        if band < 0.45:
            wrong_loss = rng.uniform(2.0, 12.0)
            irreversible = rng.random() < 0.04
            observer = rng.random() < 0.99
        elif band < 0.80:
            wrong_loss = rng.uniform(30.0, 120.0)
            irreversible = rng.random() < 0.28
            observer = rng.random() < 0.95
        else:
            wrong_loss = rng.uniform(300.0, 1_200.0)
            irreversible = rng.random() < 0.78
            observer = rng.random() < 0.86

        defer_loss = rng.uniform(0.6, 7.0)
        verification_available = rng.random() < 0.72
        verification_cost = rng.uniform(0.15, 1.25) if verification_available else None
        authority = rng.random() < 0.985

        yield DecisionCase(
            case_id=f"SYN-{index:05d}",
            p_correct=p,
            wrong_loss=wrong_loss,
            defer_loss=defer_loss,
            verification_cost=verification_cost,
            verifier_true_positive=rng.uniform(0.88, 0.97),
            verifier_false_positive=rng.uniform(0.03, 0.13),
            authority_granted=authority,
            irreversible=irreversible,
            outcome_observer_available=observer,
        )


def evaluate(n: int = 10_000, seed: int = 197) -> Dict[str, Aggregate]:
    policies = (
        "always_act",
        "fixed_080",
        "consequence_aware",
        "metacognitive_verify",
    )
    totals = {name: Aggregate() for name in policies}

    for case in synthetic_cases(n=n, seed=seed):
        for name in policies:
            outcome = _policy_outcome(case, name)
            total = totals[name]
            total.loss += outcome.loss
            total.act_probability += outcome.act_probability
            total.verify_probability += outcome.verify_probability
            total.defer_probability += outcome.defer_probability
            total.hold_probability += outcome.hold_probability
            total.irreversible_wrong_act_probability += (
                outcome.irreversible_wrong_act_probability
            )

    for total in totals.values():
        total.loss /= n
        total.act_probability /= n
        total.verify_probability /= n
        total.defer_probability /= n
        total.hold_probability /= n
        total.irreversible_wrong_act_probability = (
            total.irreversible_wrong_act_probability / n * 1_000.0
        )
    return totals


def _assert_examples() -> None:
    easy = DecisionCase(
        case_id="positive-low-consequence",
        p_correct=0.97,
        wrong_loss=10.0,
        defer_loss=2.0,
        verification_cost=0.5,
    )
    assert choose(easy).decision is Decision.ACT

    same_confidence_high_consequence = DecisionCase(
        case_id="negative-same-confidence-high-consequence",
        p_correct=0.97,
        wrong_loss=500.0,
        defer_loss=2.0,
        verification_cost=0.2,
        verifier_true_positive=0.95,
        verifier_false_positive=0.05,
    )
    assert choose(same_confidence_high_consequence).decision is Decision.VERIFY

    no_readback = DecisionCase(
        case_id="irreversible-without-observer",
        p_correct=0.999,
        wrong_loss=10_000.0,
        defer_loss=1.0,
        verification_cost=0.1,
        irreversible=True,
        outcome_observer_available=False,
    )
    assert choose(no_readback).decision is Decision.HOLD

    missing_authority = DecisionCase(
        case_id="confidence-cannot-mint-authority",
        p_correct=1.0,
        wrong_loss=1.0,
        defer_loss=0.5,
        authority_granted=False,
    )
    assert choose(missing_authority).decision is Decision.HOLD

    # Negative control: if D/W is fixed at 0.2, the consequence-aware threshold
    # is exactly 0.8, so it collapses to the fixed 0.80 policy.
    for p in (0.20, 0.79, 0.80, 0.91, 0.999):
        for wrong_loss in (5.0, 50.0, 500.0):
            case = DecisionCase(
                p_correct=p,
                wrong_loss=wrong_loss,
                defer_loss=0.2 * wrong_loss,
            )
            consequence = _policy_outcome(case, "consequence_aware")
            fixed = _policy_outcome(case, "fixed_080")
            assert math.isclose(consequence.loss, fixed.loss, rel_tol=0.0, abs_tol=1e-12)
            assert consequence.act_probability == fixed.act_probability


def _format_receipt(receipt: DecisionReceipt) -> str:
    alternatives = ", ".join(
        f"{name}={loss:.3f}" for name, loss in sorted(receipt.alternatives.items())
    )
    return (
        f"{receipt.case_id}: {receipt.decision.value}; "
        f"threshold={receipt.act_threshold:.3f}; {alternatives}"
    )


def main() -> None:
    _assert_examples()
    print("All deterministic cases and the negative control passed.\n")

    examples = [
        DecisionCase(
            case_id="same-p-low-consequence",
            p_correct=0.97,
            wrong_loss=10.0,
            defer_loss=2.0,
            verification_cost=0.5,
        ),
        DecisionCase(
            case_id="same-p-high-consequence",
            p_correct=0.97,
            wrong_loss=500.0,
            defer_loss=2.0,
            verification_cost=0.2,
            verifier_true_positive=0.95,
            verifier_false_positive=0.05,
        ),
        DecisionCase(
            case_id="irreversible-no-readback",
            p_correct=0.999,
            wrong_loss=10_000.0,
            defer_loss=1.0,
            verification_cost=0.1,
            irreversible=True,
            outcome_observer_available=False,
        ),
    ]
    for case in examples:
        print(_format_receipt(choose(case)))

    n = 10_000
    seed = 197
    print(f"\nSynthetic expected metrics: n={n:,}, seed={seed}")
    print(
        "policy                  loss/case  act%  verify%  defer%  hold%  "
        "irreversible wrong acts/1000"
    )
    for name, total in evaluate(n=n, seed=seed).items():
        print(
            f"{name:23s} {total.loss:9.3f} "
            f"{100*total.act_probability:6.2f} "
            f"{100*total.verify_probability:8.2f} "
            f"{100*total.defer_probability:7.2f} "
            f"{100*total.hold_probability:6.2f} "
            f"{total.irreversible_wrong_act_probability:28.3f}"
        )


if __name__ == "__main__":
    main()

The numbers are deliberately synthetic. Their purpose is to reveal policy mechanics. Always acting produces high loss. A fixed 0.80 threshold improves sharply but ignores variation in consequence. The consequence-aware policy lowers expected loss further. Adding an informative verifier lowers loss again and increases eventual automated action, because the controller buys evidence selectively rather than deferring every borderline case.

The experiment also shows why release criteria must include a negative control. When D/W is fixed at 0.2, consequence awareness reduces exactly to 0.80. The code asserts that both policies choose identically across several probabilities and consequence scales. A mechanism should earn its complexity by changing decisions only where the causal variable actually varies.

Compact glossary

Metacognitive monitoring

An estimate of the quality, uncertainty or likely correctness of a first-order proposal.

Metacognitive control

A policy that uses monitoring to change information seeking, computation, action, deferral or stopping.

Consequence-shaped boundary

The decision threshold induced by wrong-action loss, deferral loss, recovery and hard constraints.

Independent verification

A check whose errors and evidence path are sufficiently different to provide new decision information.

Unknown outcome

A state in which a command was attempted but the committed business effect cannot yet be established.

Decision receipt

A typed record of proposal, uncertainty, consequence, authority, selected route and verified outcome.

Make uncertainty earn a different action

Artificial metacognition does not require a machine to possess human introspection, a narrative self or conscious awareness. Those are separate questions. At the functional level examined here, the burden is narrower and testable: an uncertainty estimate must regulate what the system does next.

The crucial experiment fixes the proposal and monitored probability, then changes consequence. An easily corrected draft, a reversible workflow transition and an irreversible transfer should not share one confidence threshold. The controller must account for wrong-action loss, deferral cost, information value, authority, recovery and outcome observation. It must also preserve a hard distinction between a model’s proposal and the permission to create an effect.

The architecture decision changes from “ask the model for confidence” to “place confidence inside a deterministic, consequence-sensitive release policy”. The evaluation decision changes from measuring calibration alone to matched consequence sweeps, selective utility, verifier independence and postcondition assurance. The governance decision changes from approving a nominal autonomy level to approving typed effects under explicit recovery conditions.

The standard is intentionally demanding. A system may say it is uncertain and still act recklessly. It may abstain and still send every difficult case to an overloaded reviewer. It may verify and still consult a correlated source. It may execute the right command and still fail to know whether the outcome occurred. Metacognition becomes defensible only when uncertainty changes the route, the route respects authority, and the resulting world state can be proved.