The bill hidden inside an answer

Modern AI systems make the cost of thinking visible. A longer generation consumes more tokens. Search expands candidate paths. A verifier scores them. A tool call takes money and time. A human review consumes scarce attention. Yet many system designs still treat these operations as one undifferentiated quantity called “reasoning”.

That conflation hides the mechanism. Inference, tool use, independent verification and human judgement do not buy the same thing. More inference explores the model’s own conditional distribution. A tool can introduce a different operation or new evidence. An independent verifier can challenge a candidate through another causal path. A person can add tacit context, responsibility and value judgement, but can also be primed, overloaded or less accurate than the machine on that instance.

Research on adaptive computation established the basic intuition that not every input deserves the same internal work. Graves’ Adaptive Computation Time let a recurrent network learn how many steps to take for each input.[3] More recent work applies value-of-computation ideas directly to language models, reducing generated reasoning while maintaining task performance in the tested settings.[5] Test-time scaling studies likewise report that the best allocation depends on prompt difficulty and the method used, rather than on a universal “more is better” rule.[6]

The hidden assumption in “think harder”

The usual instruction to spend more compute contains an unspoken assumption: each extra unit of work has a fresh chance to discover the truth. That assumption is sometimes reasonable. Sampling multiple reasoning paths can reveal alternatives, and a trained verifier can select a better candidate. But the assumption fails when the paths share the same blind spot, evidence source or objective.

Ensemble research has long separated member accuracy from diversity. Krogh and Vedelsby’s ambiguity analysis treats disagreement among predictors as relevant to ensemble behaviour.[10] The practical translation is exacting: two model calls are not two checks merely because they produce two strings. They become meaningfully distinct only to the extent that their error-generating paths differ.

Verification diversity is a causal property, not a count of reviewers. Different temperatures on one model may diversify surface paths while preserving the same training-data error. A second vendor model may still consume the same retrieved record. A human may add no independence if the model’s recommendation is displayed first and the task is reduced to clicking “approve”.

This creates the first distinction the portfolio approach needs: answer improvement is not decision improvement. A computation may make an explanation clearer, produce a more elegant plan or increase benchmark accuracy while leaving the selected action unchanged. Its operational value comes from the probability that it changes the action when change is beneficial.

flowchart TD
  A["The bill hidden inside an answer"] --> B["When more thought buys the<br/>same mistake"]
  B --> C["Price the next cognitive move"]
  C --> D["Build the verification portfolio"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#D95545,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

When more thought buys the same mistake

Consider the supplier request again. The model’s ten samples agree because each relies on the same altered supplier record retrieved from an internal index. A second model also agrees because it receives that record in its context. A reviewer agrees because the interface presents the recommendation, the record and a green confidence indicator on one screen. Four apparent layers of assurance collapse into one common cause.

Five failure modes, five different purchases

A useful portfolio starts by decomposing risk. The following five modes are not exhaustive, but they force operational precision.

A check is valuable when it covers a material failure mode that remains open. This is why a calculator can be more useful than a stronger model on an arithmetic bottleneck, while no amount of arithmetic verification can decide whether the action was authorised for this purpose.

Human involvement therefore cannot be priced as a ceremonial constant. A systematic review and meta-analysis of 106 experiments found that human–AI combinations, on average, improved on humans alone but did not outperform the better of the human or AI alone. Decision tasks showed losses on average, and effects varied with relative human and AI performance.[12] This does not argue against people. It argues against assuming that adding a person automatically creates synergy.

Research on expert deferral formalises a related point. A combined system should learn where the human and model make different errors, rather than optimise the model in isolation and append a generic confidence threshold.[11] Human-factors studies also show that highly reliable but imperfect automation can produce automation bias and complacency at the moment it fails.[13]

Delay is part of the loss function

Reasoning systems often optimise accuracy while treating latency as a service metric. In decisions, delay can alter the outcome itself. An anomaly caught after funds settle is not the same observation delivered before release. A treatment recommendation after the intervention window is a different action. A market quotation after expiry no longer supports the original decision.

The result is not a static checklist. It is a sequential policy. Cheap, high-coverage checks run first. Expensive or scarce checks are gated by unresolved risk. A human is asked a question the system has prepared, not handed an unbounded transcript. The system can also conclude that no available operation will reduce risk enough before the deadline. That is the moment to abstain, delay the action or choose a reversible alternative.

Price the next cognitive move

Decision theory supplies the cleanest starting point. Let h be everything the system currently knows, θ the unknown state of the world, a a possible action and L(a, θ) the loss incurred when action a meets state θ. The current Bayes risk is the expected loss of the best available action.

A cognitive operation c produces an observation y: another reasoning trace, a tool result, a verifier score or a human judgement. Its value is the expected reduction in risk, minus everything spent to obtain it.

This is the core of rational metareasoning: computations are actions chosen for their expected effect on the eventual decision.[1][2] Resource-rational accounts ask a related question about whether cognitive effort is adapted to its costs and environment.[4] It immediately explains why “hard question” is an incomplete routing signal. A difficult question with a large decision margin may need no extra work. An apparently easy question with a tiny margin and irreversible consequence may deserve several independent checks.

From one computation to a portfolio

The single-step equation assumes we can estimate what one operation will reveal. Real systems need to select a set or sequence whose members overlap. A compact approximation is to decompose the base error into failure modes. Let w_m be the share of error assigned to mode m, and q_i,m the chance that instrument i catches that mode under the actual configuration.

The approximation exposes an important change in language. Confidence asks, “How likely is the proposal to be right?” Coverage asks, “Which ways of being wrong have we actually tested?” Confidence belongs to the proposal; coverage belongs to the assurance portfolio.

After a set S has run, the marginal value of adding instrument i is simply J(S) − J(S ∪ {i}). A repeated reasoning sample may be valuable at the start and nearly worthless after many similar samples. A human review may be wasteful on routine arithmetic but decisive when policy exceptions dominate. A readback has low value before execution and high value after an uncertain tool outcome.

Some thoughts have option value

Selecting the largest immediate value is a useful heuristic, not a universal optimum. One cheap computation can make another computation valuable. In the supplier case, retrieving the master record does not settle the decision. It reveals a same-day contact change, which sharply raises the expected value of a callback. The retrieval bought information and also bought a better routing decision.

The reverse can happen. A detailed human review may appear valuable before a deterministic check. Once the check proves that a mandatory field is invalid, the review becomes unnecessary. This means a portfolio should model conditional branches, rather than only rank a fixed list. For a small catalogue, exhaustive subset search is transparent. For larger systems, dynamic programming, tree search or a learned metacontroller can represent the option value of intermediate observations.

The next computation changes both the decision belief and the market for later computations. Any learned allocator should therefore be evaluated on complete trajectories, including stopping, rather than rewarded only for choosing individually plausible tools.

Four numbers are not enough

Expected loss, cost, latency and coverage are necessary, but they are not sufficient. The portfolio also needs constraints that cannot be traded away. A computation may be high-value yet prohibited because it exposes personal data to an unauthorised model. A human may be the best error detector yet lack the legal authority to approve. A tool may verify the arithmetic but not the purpose binding. An irreversible action may require two-person control regardless of its average expected value.

This is where decision theory meets system architecture. The model proposal, world state, permitted context, authority, typed action and evidence receipts must remain distinct. The portfolio optimiser recommends information-gathering and verification operations. It does not grant authority. A high-value thought is not a permission to act.

Human judgement has a queue price

Human review is often entered in a design as though it were a fixed-cost oracle. It is neither fixed nor oracular. A specialist may detect a goal mismatch, recognise an unusual exception or carry accountability that no model can supply. Yet review quality depends on preparation, workload, interface order and time. Showing the model’s recommendation first can also anchor the reviewer, so a nominally independent check may inherit the proposal’s error.[11][13]

Capacity creates a portfolio externality. Imagine two reviewers with six productive hours each and cases requiring eight minutes on average. Their daily capacity is about ninety reviews. If an allocator sends eighty low-consequence cases and twenty urgent cases, ten cases must wait even before breaks, investigations or rework. The marginal cost of the eighty-first referral is therefore not eight minutes. It includes delay imposed on whichever later case loses its place. Near capacity, that shadow price can rise abruptly.

Buy human judgement where its distinctive causal contribution exceeds its queue cost. Route tacit context, value conflict, novel exceptions and accountable discretion to prepared people. Keep deterministic checks, routine reconciliation and source freshness away from them. Preserve independence by giving reviewers the raw case before the proposal when feasible. Selective routing is not a claim that people matter less. It protects the scarce conditions under which their judgement matters most.[14]

stateDiagram-v2
  [*] --> Claim
  state "The hidden assumption in<br/>“think harder”" as Claim
  state "Five failure modes, five<br/>different purchases" as Test
  state "Delay is part of the loss function" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Build the verification portfolio

The portfolio becomes useful only when it changes the configured system. The smallest implementation has four layers: a failure-mode ledger, a catalogue of cognitive instruments, a sequential allocator and an authority boundary. Each decision produces receipts so the estimated coverage can be compared with what actually failed.

A realistic configured example

Return to the supplier bank-change request. The initial proposal is not “pay”. It is a typed claim: the request appears internally consistent and the supplied account should replace the current one. The system estimates a 6 per cent base error probability. Most risk is factual provenance and policy, not arithmetic.

Keep cognition outside the authority kernel

The allocator can be probabilistic. The release boundary should not be. A model may propose the next instrument, estimate residual risk and summarise evidence. Deterministic controls still bind identity, subject, purpose, policy, current permission, action type, idempotency and postcondition verification.

The verification-portfolio optimiser

The executable lab below implements the smallest useful version of the idea. It represents base error as five failure-mode weights. Each instrument has a direct cost, latency and probability of catching each mode. The optimiser enumerates allowed subsets, rejects those outside budget or latency, estimates residual expected loss and returns the lowest-cost portfolio.

To apply the lab, replace the synthetic detection rates with estimates from failure injection, adjudicated incidents and counterfactual replay. Use configuration-specific values. “Human review” is not one row. A blind specialist, a fatigued generalist and an approver shown the model’s rationale are different instruments.

Calibrate coverage from operating evidence

Coverage values should begin as explicit priors, then be replaced by observed performance. Build a test set organised by failure mode, consequence and difficulty. Inject stale evidence, misleading instructions, arithmetic traps, policy exceptions, tool timeouts and partial effects. Run each instrument alone and in the combinations the production system actually uses. Record not only whether the final answer was correct, but which instrument detected the fault, which shared it and whether the detection arrived before the decision cutoff.

Pairwise results matter. If two verifiers each catch 60 per cent of factual errors but miss almost the same cases, their joint coverage is nowhere near the independence product. Maintain a common-cause matrix linking instruments to shared data, model lineage, prompts, objectives, operators and execution paths. Use incident replay to challenge that matrix. When a new common cause appears, update past portfolio estimates rather than treating the event as an isolated anomaly.

Human instruments require the same discipline. Sample review quality by workload, case type, time-on-task and interface condition. Compare blind review with model-primed review. Measure false blocks as well as misses, because excessive escalation creates queues that destroy later review quality. Confidence intervals should remain visible when samples are sparse. A point estimate of “78 per cent coverage” is not a property of humans or models in general; it belongs to one configured instrument under one tested distribution.

The resulting ledger is a living assurance asset. It tells the allocator what each operation has actually bought before, where evidence is thin and which dependencies invalidate multiplication. It also makes null results useful. A verifier that adds little average accuracy may still be retained if it uniquely catches a rare catastrophic mode. A costly human check may be removed from routine cases while remaining mandatory for a narrow policy boundary.

Receipts must teach the allocator what it missed

A decision receipt should preserve more than the final answer. Record the proposal, decision margin, open failure modes, evidence versions, selected instruments, their order, shared dependencies, elapsed time, detections, authority result, typed action and postcondition readback. When the outcome becomes knowable, attach it without rewriting the original packet. This separates what the system knew at the cutoff from what investigators learned later.

The obvious training set is biased. High-risk cases receive more review, so they produce more labels. Low-risk cases that pass automatically may hide systematic misses. A controller trained only on observed reviews can learn that unreviewed regions are safe simply because nobody looked. Correct this with random audits of otherwise routine cases, seeded failure injection, delayed outcome matching and counterfactual replay. Exploration should occur only where actions are reversible or safely shadowed. Hard constraints are not an exploration budget.

The learning target must also match the thesis. Reviewer agreement rewards conformity. Answer accuracy ignores cost and timing. Tool use counts activity. Instead, evaluate complete trajectories on terminal loss, direct expenditure, delay, false blocks and retained coverage of rare severe modes. Hold out common-cause scenarios rather than merely new surface examples. Version the estimates when models, prompts, tools, source systems or reviewer interfaces change. An allocator earns trust by predicting what each next check will add, then reconciling that prediction against receipts.

A positive optimiser result permits a bounded claim: under the supplied assumptions, one available portfolio has lower estimated terminal cost than the other enumerated portfolios. A negative result, where every allowed set leaves risk above tolerance, means the system should abstain, delay, seek new evidence or choose a reversible action. It does not mean the highest-scoring portfolio is safe.

from itertools import combinations
from math import prod
from typing import TypedDict

MODES = ("reasoning", "facts", "arithmetic", "policy", "execution")

class Instrument(TypedDict):
    name: str
    cost: float
    latency: float
    false_alarm: float
    coverage: dict[str, float]

def score_portfolio(
    selected: list[Instrument],
    *,
    base_error: float,
    loss_if_wrong: float,
    mode_weights: dict[str, float],
    delay_cost_per_second: float,
    false_block_cost: float,
) -> float:
    residual_share = 0.0
    for mode in MODES:
        miss_probability = prod(
            1.0 - item["coverage"].get(mode, 0.0)
            for item in selected
        )
        residual_share += mode_weights[mode] * miss_probability

    residual_loss = base_error * loss_if_wrong * residual_share
    direct_cost = sum(item["cost"] for item in selected)
    latency_cost = delay_cost_per_second * sum(item["latency"] for item in selected)
    false_block_probability = 1.0 - prod(
        1.0 - item["false_alarm"] for item in selected
    )
    false_block_loss = (1.0 - base_error) * false_block_probability * false_block_cost
    return residual_loss + direct_cost + latency_cost + false_block_loss

def optimise(instruments: list[Instrument], budget: float, max_latency: float, **kwargs):
    feasible = []
    for size in range(len(instruments) + 1):
        for group in combinations(instruments, size):
            if sum(x["cost"] for x in group) > budget:
                continue
            if sum(x["latency"] for x in group) > max_latency:
                continue
            feasible.append((score_portfolio(list(group), **kwargs), group))
    if not feasible:
        raise ValueError("No feasible portfolio")
    return min(feasible, key=lambda row: row[0])

The browser lab runs locally in this page: change the inputs, choose the available instruments and select Optimise portfolio. For the Python version, copy the block into Python 3.9 or later, supply an instrument list matching the typed schema and call optimise. Its result compares the supplied options. It cannot certify the probabilities, independence, authority or safety of the configured system.

The failure that every portfolio can miss

Imagine that the supplier master itself is compromised. Retrieval, a second model and the reviewer all consult it. The callback number has also been overwritten. The portfolio appears diverse by component type while remaining monocultural by provenance. Only a channel outside that administrative domain, such as a previously sealed contact record or bank-side confirmation, can challenge the common cause.

This edge case gives the framework its strongest boundary. The optimiser is only as good as its failure-mode ledger and joint-dependence estimates. Unknown unknowns remain. Adversaries can target whichever checks are predictable. Human competence changes with workload. Tool behaviour changes with versions and state. The correct response is not to abandon allocation, but to version the assumptions and continuously attack them.

A six-step operating protocol

1. Define the terminal decision. Name the action, alternatives, deadline, reversibility and consequence if wrong.
2. Decompose the base risk. Estimate which failure modes matter on this instance, not only the model’s aggregate confidence.
3. Catalogue configured instruments. Record cost, latency, authority, evidence provenance, mode coverage, false alarms and known common causes.
4. Select sequentially. Buy the affordable operation with the highest positive marginal decision value after prior results.
5. Preserve independence. Use blind review, distinct sources, different mechanisms and readback paths where the consequence justifies them.
6. Stop explicitly. Release only through the authority kernel, or abstain when residual risk remains above tolerance.

Operating evidence is what turns this from a neat equation into an engineering discipline. Each failure injection updates coverage estimates. Each false block prices review burden. Each incident reveals a missed mode or hidden dependence. Each human override records whether the person added information, values, authority or merely repeated the machine.