Solve, Verify or Escalate?

A reasoning system becomes efficient and defensible when each additional unit of effort is sent to the bottleneck it can actually reduce: candidate coverage, selection error, or missing human evidence and authority.

TLDR

The next unit of inference budget should follow the current decision bottleneck, rather than defaulting to a longer answer or another model call.
Generate again when the candidate set is weak, verify independently when a good candidate may already exist, and escalate when missing evidence, consequence or authority cannot be repaired computationally.
A practical controller compares the expected reduction in consequence-weighted loss with token cost, latency, scarce review capacity and the risk that another check merely repeats the same blind spot.
Verification is not automatically independent and human review is not automatically superior. Correlated models, weak judges, queue delay and automation bias can make both interventions worse than committing.
The operating decision changes from setting one reasoning budget to maintaining a measured portfolio policy with hard authority gates, calibrated branch outcomes and an explicit stop rule.

A model has produced a plausible recommendation. The service has enough budget for one more intervention. It can ask for another solution, run an independent check, or send the case to a person. All three consume resources. All three can improve an outcome. Yet they purchase different things.

Imagine three copies of the same case. In the first, the model has overlooked a viable interpretation, but a good checker is already available. Another diverse solution can expose the missing candidate. In the second, the correct answer is probably already present among several candidates, but the system cannot tell which one is sound. Independent verification is the useful purchase. In the third, a required document is absent and only an authorised reviewer may approve the exception. Neither more prose nor more model votes can manufacture the evidence or authority.

The next unit of effort should go to the cheapest intervention that can reduce the current consequence-weighted decision loss, subject to hard evidence and authority constraints. Solve when candidate coverage is the bottleneck. Verify when selection is the bottleneck. Escalate when the missing resource is external evidence, complementary judgement or accountable authority. Commit, abstain or request evidence when none of the three has positive value.

This is narrower than asking how much test-time compute a task deserves. The adjacent article Test-Time Compute in the Enterprise treats the size of the reasoning budget. This paper asks how to divide the next marginal unit inside that budget. It also extends the layered architecture in The Reasoning Stack by turning generation, checking and human review into competing interventions rather than a fixed sequence.

Part I

The last unit of effort

Three purchases hidden by one budget

A serving dashboard may reduce every intervention to tokens, latency and price. That accounting is necessary, but intellectually misleading. A generation call explores the answer space. A verification call observes a candidate through another instrument. A human review consumes scarce organisational attention and may carry authority that no model has. The units can be normalised into cost, yet their causal effects remain different.

Generation buys candidate coverage. Sampling another trajectory, changing a decomposition, retrieving a different evidence bundle or invoking a different solver can produce a candidate that the first attempt never reached. Self-consistency works in part because a correct result can appear through several reasoning paths and gain probability mass across samples.¹ More generation helps only when the correct or useful candidate is reachable and the system has some way to recognise it.

Verification buys discrimination. A checker asks whether a candidate satisfies constraints, agrees with evidence or survives an executable test. The checker might be a compiler, calculator, database constraint, theorem prover, process reward model, separate language model or human reviewer. Its value comes from information that separates good candidates from bad ones. Process supervision has outperformed outcome-only supervision in particular mathematical settings, but the result does not imply that every intermediate model judgement is reliable.²

Escalation buys access to something the machine path lacks. A person may obtain an unavailable document, interpret an exceptional circumstance, notice a social or organisational fact outside the data, or exercise a decision right. Human time is not literally machine compute. It belongs in the same allocation problem because it competes for the same outcome and because its queue, cost and error rate shape the policy.

The fourth action is easy to omit: stop. A low-consequence, reversible response may be worth committing even when uncertainty remains. A high-consequence case with missing evidence may be worth abstaining from rather than escalating immediately. Rational allocation includes the option to spend nothing more.

Figure 1. One budget unit, three causal destinations. The interventions are alternatives only after hard evidence and authority rules are satisfied. Geometry and labels are authored for this article; the relationship is conceptual, not measured.

Why more reasoning does not name its own value

Test-time scaling research establishes that additional inference can improve results, and that allocation should vary with problem difficulty. Snell and colleagues reported more than fourfold efficiency gains over a best-of-N baseline from compute-optimal allocation in their experimental setting.³ Rational metareasoning work expresses the same intuition as value of computation: reason only when the expected improvement exceeds the cost, reducing generated tokens while maintaining task performance in the evaluated datasets.⁴

Difficulty alone is not enough. Two equally difficult tasks can demand opposite interventions. A hard combinatorial problem with a perfect checker may reward broader search. A hard factual claim with weak evidence may reward retrieval and human investigation. A hard policy exception may remain undecidable until an authorised person acts. The useful latent variable is not just difficulty. It is the repairable bottleneck.

The bottleneck is causal. Ask what would have to change for the decision to become acceptable. If the answer is “a better candidate”, spend on solving. If it is “credible evidence that this candidate is sound”, spend on verification. If it is “a fact or decision right outside the machine boundary”, escalate. If no plausible intervention changes the expected outcome enough, stop.

Part II

A portfolio, not a ladder

A common architecture is a ladder: generate, critique, revise, judge, then send to a human if confidence remains low. The ladder is easy to draw and often wasteful. It assumes every case should buy the same interventions in the same order. A portfolio policy asks a different question after each observation: which available intervention now has the highest marginal value?

Practitioner synthesis. The four-action portfolio and the marginal-net-value instrument below combine metareasoning, selective prediction, verification and learning-to-defer ideas. They are an architecture proposal, not a published universal law.

Suppose the current state is b. It contains the candidates, evidence, known authority, posterior error estimates, remaining time and remaining budget. For an intervention i, define:

MNV(i | b) = E[L(b) − L(b′ | i)] − C_i − λ_tT_i − λ_qQ_i

L(b) is expected consequence-weighted loss in the current state. b′ is the state after observing the intervention. C_i is direct resource cost. T_i is delay, weighted by the value of time λ_t. Q_i is scarce-capacity consumption, such as an expert review slot, weighted by its shadow price λ_q. Choose the highest positive value. If every value is non-positive, stop or abstain. If authority or mandatory evidence is absent, a hard rule can remove “commit” from the legal action set before this calculation begins.

This formula is not a claim that every consequence can be priced precisely. Its discipline is comparative. It prevents a cheap token from looking useful merely because it is cheap, and it prevents a human review from looking safe merely because a person is present. An intervention earns its place through expected state change, not ceremonial reassurance.

Belief change, option creation and authority are different returns

The loss term becomes clearer when its possible improvement is separated into three returns. Epistemic value comes from better beliefs about which candidate is correct. Option value comes from creating a candidate or evidence path that did not previously exist. Authority value comes from making an action legally or organisationally available. Solve, verify and escalate can contribute to more than one return, but they do not contribute in equal proportions.

A second solution usually creates option value first. It may also provide epistemic value when agreement or disagreement is informative. A verifier usually creates epistemic value by observing a candidate through another mechanism. A human can supply epistemic value through expertise, option value by obtaining missing evidence, and authority value by making an accountable decision. Yet these functions must be recorded separately. A technical specialist who identifies the correct answer may lack approval authority. An authorised approver may sign a decision without adding any new evidence. Counting both as “human verification” hides the control boundary.

Information also has value only when the decision is sensitive to it. Suppose every plausible posterior leads to the same reversible action. A more accurate confidence estimate may be intellectually satisfying but have no immediate decision value. Conversely, a small probability shift around an irreversible action threshold can be decisive. The optimiser therefore needs the downstream policy, not just a model score. It asks whether the observation can change the permitted action, the evidence request, the review route or the recovery plan.

One useful bookkeeping expansion is to read the expected loss reduction as the sum of an epistemic gain, an option-creation gain and an authority-enablement gain, while avoiding double counting. The expansion explains why a person can be the correct route even when a verifier is more accurate, and why a deterministic check can dominate a person even in a consequential workflow. The intervention is chosen for the state transition it enables, not for its prestige in the hierarchy.

Figure 2. The reasoning portfolio frontier. The lower envelope represents the best residual loss attainable at each illustrative budget. The switching points depend on locally measured costs, branch outcomes and authority rules. Curves are illustrative and have no universal units.

Thought experiment one: the hidden unit test

A code model proposes a patch. Hold the model, repository and initial patch constant. Change only the verifier.

In world A, a deterministic test suite detects the relevant semantic regression. The system can sample patches until one passes, then inspect the diff. Generation has value because verification makes candidate quality observable. In world B, the test suite checks only syntax. A dozen patches may all pass while preserving the same silent defect. More generation creates diversity, but the selector cannot distinguish the property that matters. The bottleneck has moved from coverage to verification.

Now reverse the intervention. Hold the strong test suite constant, but suppose the generator has near-zero probability of producing a valid patch. Re-running the same tests does not help. The verifier can reject perfectly and still never create a solution. A strong checker does not compensate for an empty candidate set, and a rich candidate set does not compensate for a blind checker.

This thought experiment explains why “solve versus verify” cannot be settled by one universal ratio. Work on compute-optimal problem solving and generative verification found self-consistency more compute-efficient than a generative reward model across most practical budgets in the studied mathematical and scientific tasks, with the verifier route requiring up to eight times the inference compute to match in one comparison.⁵ The result is important and bounded: it compares specific generators, verifiers, datasets and cost accounting. A cheap executable oracle can reverse the answer.

Thought experiment two: the sealed exception

A system prepares a decision on a policy exception. Again, hold the model and prompt constant. In world A, the signed exception document is present and the policy permits a machine-generated recommendation, although a human still owns approval. Verification can check that every recommendation cites the correct clause and uses the signed facts.

In world B, the signed document does not exist. The model can write ten careful analyses of what the document might say. A verifier can confirm that the prose is internally consistent. Neither intervention reduces the decisive uncertainty. The system must request evidence or escalate to someone who can obtain it. In world C, the document exists and the answer is clear, but organisational policy reserves the decision right for a named role. Computation may prepare the case, yet cannot acquire authority.

Evidence and authority are not confidence thresholds. They are state and permission conditions. A cost optimiser may rank legal actions, but it must not price away a mandatory human decision, a missing consent or a revoked capability.

Figure 3. Candidate coverage and selector discrimination are separate control variables. The map is a conceptual diagnostic. The human, evidence and authority override remains outside the two model-centric axes.

Verification must add information, not ceremony

A model checking its own answer in a fresh prompt can be useful, but the second call is not automatically independent. Positive results exist: Ren and colleagues improved selective generation on TruthfulQA and TL;DR by reformulating self-evaluation as calibrated token-level choices in their evaluated models.⁶ The boundary is task and instrument dependent. Research on intrinsic self-correction has found settings where unsupported self-critique degraded reasoning performance.⁷ Planning experiments using the same language model for generation and verification also reported false positives and weaker results than systems with a sound external verifier.⁸ On the REVEAL benchmark, automatic verifiers particularly struggled with logical correctness and contradiction detection.⁹

The relevant quantity is conditional error correlation. Suppose one verifier falsely accepts a wrong candidate with probability 0.25. Four independent checks would reduce joint false acceptance to roughly 0.004. If a common blind spot causes all checks to share most of their errors, the residual can remain near 0.23. More calls then create an appearance of scrutiny while barely changing the posterior.

Figure 4. Repeated checks can hit a common-mode floor. Synthetic illustration with single-check false acceptance of 0.25 and a simple mixture model, ρ·α + (1−ρ)·α^k. It is a mechanism demonstration, not an empirical estimate of any model.

Published evidence. The studies in the next figure support adaptive and selective allocation in bounded benchmark settings. None tests the complete solve, verify, human-escalate and stop policy proposed here.

Figure 5. Published evidence favours conditional allocation, not one universal intervention. Numbers are sourced from the cited studies and retain their original denominators. The 2026 entries are preprints. Cross-study effect sizes are shown as an evidence rail rather than a shared scale because the experiments are not commensurate.³⁵¹⁰¹¹

A serious negative control: exact calculation

Any general reasoning policy should lose to a simpler mechanism where the simpler mechanism is decisive. Consider the question: “What is 17.5 per cent of £243,760?” A language model may produce £42,658. Another model may agree. A human may recompute it. Yet a deterministic decimal calculation can establish the result cheaply and reproducibly.

The correct intervention is verification by an exact tool, not more language generation and not human escalation. If the source amount or percentage is ambiguous, the bottleneck changes to evidence interpretation. Otherwise, spending on model diversity only adds probabilistic surface area. The negative control protects the portfolio idea from becoming an excuse for an elaborate reasoning stack.

A second negative control is a reversible, low-consequence FAQ answer. Suppose the estimated chance of error is two per cent and the consequence of an error is eight cost units. The expected loss is 0.16. If every extra check costs more than 0.16 and there is no authority constraint, committing can be rational. The system should still monitor outcomes, but it should not consume an expert queue to make a trivial reply feel ceremonially safe.

Figure 6. The portfolio should collapse to a deterministic oracle when one exists. The numeric example is exact. The intervention comparison is a worked negative control, not a benchmark result.

Part III

The argument in one passChart · flow

flowchart LR
  A["The last unit of effort"] --> B["A portfolio, not a ladder"]
  B --> C["A compute optimiser for the<br/>decision"]
  C --> D["From benchmark policy to<br/>operating control"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#087A98,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

A structural route through Solve, Verify or Escalate?, from The last unit of effort to Sources. Reading use: follow the relation, then test it against the surrounding evidence and limits.

A compute optimiser for the decision

The accompanying artefact is deliberately small. It models a binary candidate with a current probability of error. The controller may commit, buy one partially correlated second solution, buy one imperfect verifier call, or escalate to a human. Each branch updates the posterior probability of error and recursively chooses the cheapest remaining action.

The model is synthetic. It does not claim that a language model exposes a true probability of error, or that organisational consequences fit one scalar. Its purpose is to reveal logic that prose hides: an intervention can be cheap yet useless, a human can be costly yet mandatory, and the value of a second solve depends on error correlation.

A minimal worked example

Take a “hard but solvable” case. The initial candidate has an estimated error probability of 0.34 and an error consequence of 1,000 synthetic cost units. A second solver costs 0.05 units, has a standalone error rate of 0.12 and low error correlation with the first. The verifier costs 12 units, detects 55 per cent of wrong candidates and falsely rejects 15 per cent of correct candidates. Human review costs 210 units including delay and is also imperfect.

The optimiser chooses another solve. Agreement occurs with probability 0.641 and lowers the posterior error estimate to 0.087, after which the policy commits. Disagreement occurs with probability 0.359; the better candidate still has posterior error 0.209, so the policy verifies. The expected policy cost is 115.79 units. This does not mean “solving is better” in general. It means candidate diversity is unusually cheap and informative under these particular assumptions.

Figure 7. One synthetic policy path from the runnable optimiser. Probabilities and costs are assumptions supplied to the example, not measured model properties. Expected policy cost is 115.79 synthetic units.

Change the assumptions and the action changes. In the exact-calculation case, a near-perfect verifier costs 0.01 units, so the policy verifies with expected cost 2.31. In a policy exception where human authority is mandatory, “commit” is removed from the action set and the policy escalates. In the reversible FAQ, every intervention costs more than the expected error loss, so the policy commits.

The optimiser is valuable because it can recommend doing less. Many orchestration systems can add a critic, a debate or a review queue. Fewer can demonstrate that the added stage has positive marginal value after latency, correlated error and opportunity cost.

Executable artefact: a recursive portfolio policy

The Python programme below uses only the standard library. Run it with Python 3.11 or later. It prints the chosen first action for four synthetic cases and exposes the branch posterior for solve and verify. Replace every probability and cost with locally measured estimates before treating the output as an operational decision.

Open the complete runnable Python optimiser

"""A small, synthetic optimiser for solve/verify/escalate decisions.

The model is intentionally simple. It treats a current answer as a binary
candidate with probability p_wrong, then compares four actions:

* commit now;
* buy one additional, partially correlated solution;
* buy one imperfect verifier call; or
* escalate to a human reviewer.

All values are synthetic monetary-equivalent cost units. Replace them with
locally measured probabilities, latency penalties and consequence estimates.
"""

from dataclasses import dataclass
from functools import lru_cache
from math import inf
from typing import Any


@dataclass(frozen=True)
class Case:
    name: str
    p_wrong: float
    consequence: float

    solve_cost: float
    second_solver_error: float
    error_correlation: float

    verify_cost: float
    verifier_sensitivity: float       # P(reject | candidate is wrong)
    verifier_false_reject: float      # P(reject | candidate is correct)

    human_cost: float
    human_delay_cost: float
    human_floor_error: float
    human_reliance: float             # extra human error induced by a bad candidate

    authority_required: bool = False


@dataclass(frozen=True)
class Decision:
    action: str
    expected_cost: float
    branches: Any = None


def optimise(case: Case) -> Decision:
    """Return the minimum-expected-cost policy for one optional solve and verify."""

    @lru_cache(maxsize=None)
    def value(p_key: int, can_solve: bool, can_verify: bool) -> Decision:
        p_wrong = p_key / 1_000_000
        options: list[Decision] = []

        # A model may commit only when human authority is not mandatory.
        if not case.authority_required:
            options.append(Decision("commit", p_wrong * case.consequence))

        # Human review is not assumed perfect. The synthetic reliance term
        # captures the possibility that a misleading candidate affects review.
        p_human_wrong = min(
            0.5,
            case.human_floor_error + case.human_reliance * p_wrong,
        )
        options.append(
            Decision(
                "human",
                case.human_cost
                + case.human_delay_cost
                + p_human_wrong * case.consequence,
                {"p_human_wrong": p_human_wrong},
            )
        )

        if can_verify:
            sensitivity = case.verifier_sensitivity
            false_reject = case.verifier_false_reject

            p_reject = (
                p_wrong * sensitivity
                + (1.0 - p_wrong) * false_reject
            )
            p_accept = 1.0 - p_reject
            expected = case.verify_cost
            branches: list[dict[str, Any]] = []

            if p_accept > 0.0:
                p_wrong_if_accept = (
                    p_wrong * (1.0 - sensitivity) / p_accept
                )
                after_accept = value(
                    round(p_wrong_if_accept * 1_000_000),
                    can_solve,
                    False,
                )
                expected += p_accept * after_accept.expected_cost
                branches.append(
                    {
                        "signal": "accept",
                        "probability": p_accept,
                        "posterior_wrong": p_wrong_if_accept,
                        "next": after_accept.action,
                    }
                )

            if p_reject > 0.0:
                p_wrong_if_reject = p_wrong * sensitivity / p_reject
                after_reject = value(
                    round(p_wrong_if_reject * 1_000_000),
                    can_solve,
                    False,
                )
                expected += p_reject * after_reject.expected_cost
                branches.append(
                    {
                        "signal": "reject",
                        "probability": p_reject,
                        "posterior_wrong": p_wrong_if_reject,
                        "next": after_reject.action,
                    }
                )

            options.append(Decision("verify", expected, branches))

        if can_solve:
            e2 = case.second_solver_error
            rho = case.error_correlation

            # Positive correlation makes a second solution more likely to repeat
            # the first solution's error and less likely to err when the first is right.
            p_second_wrong_if_first_wrong = e2 + rho * (1.0 - e2)
            p_second_wrong_if_first_correct = e2 * (1.0 - rho)

            p_agree = (
                p_wrong * p_second_wrong_if_first_wrong
                + (1.0 - p_wrong)
                * (1.0 - p_second_wrong_if_first_correct)
            )
            p_disagree = 1.0 - p_agree
            expected = case.solve_cost
            branches = []

            if p_agree > 0.0:
                p_wrong_if_agree = (
                    p_wrong * p_second_wrong_if_first_wrong / p_agree
                )
                after_agree = value(
                    round(p_wrong_if_agree * 1_000_000),
                    False,
                    can_verify,
                )
                expected += p_agree * after_agree.expected_cost
                branches.append(
                    {
                        "signal": "agree",
                        "probability": p_agree,
                        "posterior_wrong": p_wrong_if_agree,
                        "next": after_agree.action,
                    }
                )

            if p_disagree > 0.0:
                p_first_wrong_if_disagree = (
                    p_wrong
                    * (1.0 - p_second_wrong_if_first_wrong)
                    / p_disagree
                )
                # In a binary task, disagreement means exactly one answer is wrong.
                # Keep whichever candidate has the lower posterior error.
                p_best_candidate_wrong = min(
                    p_first_wrong_if_disagree,
                    1.0 - p_first_wrong_if_disagree,
                )
                after_disagree = value(
                    round(p_best_candidate_wrong * 1_000_000),
                    False,
                    can_verify,
                )
                expected += p_disagree * after_disagree.expected_cost
                branches.append(
                    {
                        "signal": "disagree",
                        "probability": p_disagree,
                        "posterior_wrong": p_best_candidate_wrong,
                        "next": after_disagree.action,
                    }
                )

            options.append(Decision("solve", expected, branches))

        if not options:
            return Decision("no legal action", inf)
        return min(options, key=lambda option: option.expected_cost)

    return value(round(case.p_wrong * 1_000_000), True, True)


def print_decision(case: Case, decision: Decision) -> None:
    print(f"{case.name}: {decision.action}  expected cost={decision.expected_cost:.2f}")
    if isinstance(decision.branches, list):
        for branch in decision.branches:
            print(
                "  "
                f"{branch['signal']:>8}  "
                f"P={branch['probability']:.3f}  "
                f"posterior wrong={branch['posterior_wrong']:.3f}  "
                f"next={branch['next']}"
            )


CASES = [
    Case(
        name="hard but solvable",
        p_wrong=0.34,
        consequence=1_000,
        solve_cost=0.05,
        second_solver_error=0.12,
        error_correlation=0.05,
        verify_cost=12,
        verifier_sensitivity=0.55,
        verifier_false_reject=0.15,
        human_cost=180,
        human_delay_cost=30,
        human_floor_error=0.02,
        human_reliance=0.06,
    ),
    Case(
        name="cheap exact calculation",
        p_wrong=0.08,
        consequence=1_000,
        solve_cost=0.04,
        second_solver_error=0.08,
        error_correlation=0.15,
        verify_cost=0.01,
        verifier_sensitivity=0.999,
        verifier_false_reject=0.001,
        human_cost=45,
        human_delay_cost=5,
        human_floor_error=0.02,
        human_reliance=0.05,
    ),
    Case(
        name="policy exception requiring authority",
        p_wrong=0.22,
        consequence=50_000,
        solve_cost=0.20,
        second_solver_error=0.20,
        error_correlation=0.35,
        verify_cost=0.35,
        verifier_sensitivity=0.55,
        verifier_false_reject=0.18,
        human_cost=180,
        human_delay_cost=90,
        human_floor_error=0.02,
        human_reliance=0.0,
        authority_required=True,
    ),
    Case(
        name="reversible FAQ",
        p_wrong=0.02,
        consequence=8,
        solve_cost=0.50,
        second_solver_error=0.02,
        error_correlation=0.20,
        verify_cost=0.60,
        verifier_sensitivity=0.70,
        verifier_false_reject=0.08,
        human_cost=12,
        human_delay_cost=3,
        human_floor_error=0.03,
        human_reliance=0.05,
    ),
]


if __name__ == "__main__":
    for synthetic_case in CASES:
        print_decision(synthetic_case, optimise(synthetic_case))

Expected output for the supplied synthetic cases

hard but solvable: solve  expected cost=115.79
     agree  P=0.641  posterior wrong=0.087  next=commit
  disagree  P=0.359  posterior wrong=0.209  next=verify
cheap exact calculation: verify  expected cost=2.31
    accept  P=0.919  posterior wrong=0.000  next=commit
    reject  P=0.081  posterior wrong=0.989  next=solve
policy exception requiring authority: human  expected cost=1270.00
reversible FAQ: commit  expected cost=0.16

A production-shaped worked scenario

Worked scenario. The following case is synthetic and intended to expose system boundaries. It is not a description of a client deployment or a claim of operational performance.

A commercial credit team receives a request for a temporary limit exception. The system is permitted to assemble evidence, calculate ratios, identify policy clauses and draft a recommendation. It is not permitted to approve the exception. The file contains current financial statements and the standard policy, but a director approval required by the exception clause is missing.

A naive reasoning ladder spends heavily. A model drafts a recommendation, criticises it, revises it, asks a judge for confidence, then sends the polished memo to an approver. The memo may become more coherent while the decisive absence remains unchanged. Worse, fluency can hide that the approval evidence was never present.

The portfolio controller first compiles state. Intent is to prepare a reviewable exception pack, not decide credit. Identity records the requester, model service, reviewer role and eventual executor. World state records the case status and current limits. Context includes source versions, required documents and contradictions. Reasoning proposes a policy mapping. Authority restricts approval to a named human role. Action can create a refer-back request or submit a pack, but cannot approve. Evidence binds each claim to a source. Outcome confirms what changed. Release and recovery define rollback, replay and incident handling.

The controller then asks the causal question. Is a better narrative the bottleneck? No. Is selection among competing policy readings the bottleneck? Perhaps, so an independent clause checker can help. Is approval evidence missing? Yes. The immediate action is to request the document or escalate the case. Once the document arrives, generation and verification can resume. The authorised person receives a compact decision pack with unresolved facts made prominent rather than buried in prose.

Human escalation still needs a measured design. A reviewer can be anchored by the model recommendation, lack the time to inspect evidence or share the same policy misunderstanding. One control is staged independence: show the evidence and question before revealing the model conclusion, then record whether the reviewer formed an independent judgement. Another is role-specific routing: send a policy ambiguity to policy expertise, missing evidence to operations, and approval to the accountable decision owner. “Human in the loop” is too coarse to specify any of these.

Figure 8. Escalation is a typed route, not a generic fallback. The synthetic case separates model proposal, independent checks, missing-evidence handling, human authority and outcome verification. No client or deployment claim is implied.

The decision instrument

The following instrument can be used as an architecture decision record, a serving policy or an evaluation plan. It starts with the limiting state change, then demands a negative control and a measurable branch outcome.

Route	Choose it when	Evidence that it helped	Serious competing baseline	Stop or reject trigger
Solve	A usable candidate is often absent; new search, retrieval, decomposition or simulation can reach it.	Candidate recall, pass@k, oracle coverage, useful diversity, lower downstream loss.	One longer solve; deterministic algorithm; better retrieval; larger base model.	Marginal candidates repeat the same error or the selector cannot distinguish quality.
Verify	A usable candidate is likely present; a check has independent access to the property that matters.	False-accept and false-reject rates, harmful versus beneficial flips, calibration, defect localisation.	Commit first answer; majority vote; executable oracle; stronger initial solve.	The checker shares the generator’s blind spot, is more expensive than solving, or changes correct answers.
Escalate	Evidence, complementary competence, accountable judgement or authority lies outside the machine path.	Expert error by slice, value of information, queue delay, overturn quality, completed evidence, valid decision right.	Human alone; request evidence without review; deterministic policy; bounded abstention.	The reviewer is no better on the slice, is anchored, lacks time, or the queue cost exceeds expected benefit.
Commit or abstain	Residual expected loss is below intervention cost, or no legal intervention can resolve the state.	Observed outcomes, reversible recovery, stable risk-coverage curve, no missing authority.	One cheap check; sampled audit; delayed human review.	Consequence rises, evidence expires, distribution shifts, or authority changes.

Do not feed this table to the model and call the result governance. The runtime should enforce hard rules, calculate known quantities and record branch evidence. A learned router may handle the residual cases, but its false routing decisions become a new source of risk.

Figure 9. A practical phase map for the next intervention. Candidate and selector deficits are measured properties of the task slice, not model self-reports. Evidence and authority are hard overrides. Boundaries are illustrative and must be calibrated locally.

Part IV

From benchmark policy to operating control

Measure the counterfactual branches

A router learns only from branches it can observe. If every uncertain case is sent to a human, the organisation may never learn whether a cheap verifier would have sufficed. If every easy case is committed, rare high-consequence failures may remain invisible. The evaluation design must therefore recover counterfactual evidence without exposing live users to uncontrolled risk.

Start offline. On a representative, time-bounded case set, run matched branches: one longer solve, several diverse solves, deterministic checks, learned verification, human-only decision and the proposed portfolio. Blind adjudicators to the route where possible. Record not only final correctness but candidate coverage, verifier discrimination, beneficial and harmful answer changes, escalation delay, human error by slice and the consequence of residual errors.

Safe randomisation can then estimate marginal value in low-consequence or simulated settings. Exploration must be bounded by authority and action limits. For high-consequence cases, counterfactual replay, shadow review and retrospective outcome evidence may be more appropriate. The policy is not calibrated by asking the model whether it found the problem difficult. It is calibrated by observing which intervention changed the outcome on which task slice.

Price the queue, not only the token

Human review has a fixed and variable cost. A five-minute review can delay an urgent case by hours when the queue is congested. Its shadow price rises when expert capacity is scarce, so the same case may rationally route differently under different service conditions, provided hard authority rules remain unchanged.

That does not justify starving the queue. It just makes the trade explicit. A triage system needs estimates of model error and expert error at instance level, a point formalised in work on algorithmic automation and learning to defer.¹²¹³ The human route should be tested like any other component. A large meta-analysis found that human-AI combinations were, on average, worse than the better of the human or AI alone across the included experiments, with substantial heterogeneity and particular losses in decision tasks.¹⁴ This does not argue against human oversight. It argues against treating mere combination as evidence of synergy.

Measure reviewer performance before and after seeing the model answer. Record override reasons. Separate a person who supplies missing evidence from a person who verifies a calculation and from a person who owns the decision right. These are different interventions with different error structures.

Keep hard gates outside the optimiser

Economic optimisation begins only after the legal action set is defined. Required consent, mandatory human approval, segregation of duties, revoked identity and missing prerequisite evidence are not soft penalties. They remove actions. The optimiser can choose how to prepare the case or which reviewer queue to use, but it cannot decide that a cheaper unauthorised action is “worth the risk”.

This separation also improves auditability. A decision receipt should state the intent, actor identities, evidence versions, candidate set, verifier signals, selected route, authority rule, action, effect receipt and unresolved risk. It should distinguish a model proposal from a human decision and an accepted command from a verified outcome. NIST’s AI risk resources emphasise testing, evaluation, verification and validation as operational activities; a portfolio router needs the same evidence discipline as the components it selects.¹⁵

Release against matched baselines

A portfolio policy should not be compared only with a single short answer. Its serious baselines are: a stronger initial model, one longer solve, uniform best-of-N, deterministic tooling, always verify, human only and a simple rule-based cascade. Recent work illustrates why. A selective verification controller improved over always verifying on MATH-500 and reduced harmful flips, yet a longer initial solve remained the best tested cost frontier in that experiment.¹⁰ A compute-balanced routing preprint matched stronger proxies with materially fewer weighted tokens on its benchmark suite, while still exposing a gap to the pool oracle.¹¹

Release criteria should include consequence-weighted loss, coverage, calibration, false acceptance, false rejection, harmful flips, expert workload, queue delay, p95 latency, resource cost and recovery behaviour. Report each by task slice and consequence tier. Overall accuracy can improve while the policy sends exactly the wrong cases to cheap paths.

Inject failures before release. Correlate generator and verifier errors. Remove a required document. Return a stale policy. Congest the human queue. Revoke the approver’s capability. Make an action return an unknown outcome. The correct response may shift from verify to solve, from solve to escalate, or from every active route to abstention. A controller that cannot change course under these interventions is a static cascade wearing adaptive language.

Failure boundary. The optimiser ceases to be decision-worthy when outcomes are not observable, branch error rates cannot be estimated, consequences are radically misspecified, or the task distribution moves beyond the calibration envelope. In that state, its output is only a restatement of assumptions. Constrain effects, request evidence, increase independent review or abstain.

What remains an open hypothesis

Open hypothesis. On a heterogeneous workload with measurable outcomes, a bottleneck-aware portfolio across solve, verify, escalate and stop will dominate any fixed sequence at the same consequence-weighted resource budget. The existing literature supports pieces of this claim, but broad production evidence for the full four-action policy remains to be established.

A discriminating experiment would compare the portfolio with its matched baselines on a frozen task distribution, then repeat under induced evidence gaps, correlated verifiers and human queue stress. The hypothesis weakens if one simple policy remains on the efficient frontier across all slices, or if routing error consumes the gains from specialisation. That possibility must remain live.

Figure 10. The router is itself a governed, changing model of intervention value. The loop is an authored operating pattern. It requires branch-level outcome evidence, matched baselines, failure injection and independent rollback.

Compact glossary

Candidate coverage: The probability that the generated set contains a usable answer, plan or action proposal.
Selector discrimination: The ability of a verifier or decision rule to rank a good candidate above a bad one on the property that matters.
Consequence-weighted loss: Error probability combined with the severity, reversibility and timing of the resulting harm or missed value.
Marginal net value: The expected reduction in loss from one intervention, minus resource, latency and scarce-capacity costs.
Common-mode failure: An error shared by ostensibly separate generators or verifiers because they rely on the same blind spot or evidence.
Escalation: A typed hand-off to obtain evidence, competence, judgement or authority unavailable to the current machine path.

How the claim changes under pressureChart · state

stateDiagram-v2
  [*] --> Claim
  state "Three purchases hidden by one<br/>budget" as Claim
  state "Why more reasoning does not<br/>name its own value" as Test
  state "Belief change, option creation<br/>and authority are different<br/>returns" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

The revision loop begins with Three purchases hidden by one budget and keeps counterexamples inside the reasoning path. Reading use: follow the relation, then test it against the surrounding evidence and limits.

Spend where uncertainty can actually fall

A reasoning budget is not one dial. It is a portfolio of interventions with different causal powers. Generation can reveal alternatives. Verification can distinguish candidates. Human escalation can obtain evidence, exercise judgement and carry authority. Stopping can preserve resources and avoid destructive overthinking.

The architecture decision changes from “set reasoning effort to high” to “route the next unit to the bottleneck that can still move”. Put hard evidence and authority gates first. Estimate marginal value from observed branch outcomes. Penalise latency, correlated error and scarce human capacity. Keep commit and abstain available. Record a decision receipt that makes the selected route, evidence and effect reconstructable.

The research decision changes too. Do not ask whether solving, verification or human review is best in the abstract. Build task slices where each should win, include a deterministic negative control, and test whether the router discovers the difference. A portfolio policy earns deployment only when it beats one longer solve, always verify, human only and simple cascades under matched consequence-weighted resources.

The final rule is compact: solve when coverage is missing, verify when discrimination is missing, escalate when the missing resource lies outside computation, and stop when another intervention cannot justify its cost.

The evidence conversationChart · sequence

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: The last unit of effort
  M->>E: Three purchases hidden by one budget
  E-->>M: compare rival explanations
  E->>D: Sources
  D-->>Q: revise scope or act

Question, mechanism, evidence and decision exchange the burdens developed in Solve, Verify or Escalate?. Reading use: follow the relation, then test it against the surrounding evidence and limits.

Sources

Xuezhi Wang et al., Self-Consistency Improves Chain of Thought Reasoning in Language Models, ICLR.
Hunter Lightman et al., Let’s Verify Step by Step.
Charlie Snell, Jaehoon Lee, Kelvin Xu and Aviral Kumar, Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters.
C. Nicolò De Sabbata et al., Rational Metareasoning for Large Language Models.
Nishad Singhi et al., When To Solve, When To Verify: Compute-Optimal Problem Solving and Generative Verification for LLM Reasoning, COLM.
Jie Ren et al., Self-Evaluation Improves Selective Generation in Large Language Models.
Jie Huang et al., Large Language Models Cannot Self-Correct Reasoning Yet, ICLR.
Karthik Valmeekam, Matthew Marquez and Subbarao Kambhampati, Can Large Language Models Really Improve by Self-critiquing Their Own Plans?.
Alon Jacovi et al., A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for Verifiers of Reasoning Chains, ACL.
Sajib Acharjee Dip, Dawei Zhou and Liqing Zhang, Think Again or Think Longer? Selective Verification for Budget-Aware Reasoning, preprint.
Yan Zhou, Yue Ouyang, Kaiyang Zheng and Suncheng Xiang, CoBa: Cost-Effective Test-Time Scaling via Compute-Balanced Routing, preprint.
Maithra Raghu et al., The Algorithmic Automation Problem: Prediction, Triage, and Human Effort.
Hussein Mozannar and David Sontag, Consistent Estimators for Learning to Defer to an Expert, ICML.
Michelle Vaccaro, Abdullah Almaatouq and Thomas Malone, When combinations of humans and AI are useful: A systematic review and meta-analysis, Nature Human Behaviour.
National Institute of Standards and Technology, AI Risk Management Framework and associated AI Resource Center.

Evidence boundary: the 2026 studies listed above are preprints. Reported numbers belong to their stated models, datasets, budgets and token definitions. The optimiser, worked cases and unsourced figure geometries in this article are synthetic practitioner artefacts.