Part IThe token budget is the wrong object

Test-time compute research established a useful fact: inference quality is not fixed by model weights alone. More samples, longer search, iterative refinement and stronger selection can improve some tasks. Snell and colleagues found that prompt difficulty changed which scaling method was effective, and that adaptive allocation made test-time scaling more efficient than a uniform best-of-n policy in their mathematical reasoning experiments.¹ Wu and colleagues similarly reported Pareto-efficient combinations of smaller models and more sophisticated inference strategies on the MATH benchmark.²

Those results justify adaptive effort. They do not imply that the extra effort should always remain inside the generator. A system facing a missing fact, a disputed identity, an uncertain future effect, an incorrect calculation and a question of accountable judgement has five different problems. Longer prose is not a common cure.

Thought experiment: two identical prompts

Imagine two cases with identical wording: “Should this facility receive a temporary covenant waiver?” In case A, every contractual term is available, but the forecast contains six interacting assumptions. In case B, the arithmetic is trivial, but the current exposure is absent. Hold model, prompt, deadline and consequence constant. Change only the missing ingredient.

Extra generation can help case A by exploring alternative interpretations of the assumptions. In case B it cannot discover an exposure that is not in the model's context or weights with the required point-in-time provenance. Retrieval has high value in B and little value in A. The thought experiment isolates the causal feature: the shape of the uncertainty determines the useful computation.

A flat “thinking level” collapses that distinction. It assumes that one internal activity can substitute for evidence acquisition, causal projection, independent checking and accountable judgement. The assumption is convenient because every intervention can be billed as tokens. It is also wrong whenever the decision depends on something tokens cannot establish.

Decision loss is broader than prediction error. A false hold, a false release and an unnecessary escalation may all count as one wrong label while imposing very different consequences. Delay can destroy an otherwise correct answer. A privacy-invasive search can improve factual accuracy while worsening the total decision. The portfolio therefore needs an outcome model that distinguishes error type, reversibility, timing and downstream controls.

The unit of allocation is also larger than one model request. A trajectory may include three generations, two searches, a calculation, a verifier retry and a reviewer handoff. Pricing each call independently can make every local choice look cheap while the complete case becomes expensive. Measure cost and value at the verified task boundary, then attribute that total back to the interventions that changed it.

Part IIFive assets in the thought portfolio

The five channels are not vendors or implementation products. They are causal roles. One tool may serve more than one role, and one role may be implemented by rules, a small model, a frontier model, a database, a simulator or a person. The classification asks what change the intervention is expected to make in the decision state.

Simulation deserves separate accounting because it can produce evidence that language generation cannot. A deterministic calculator may answer whether a ratio is correct. A queue model may show that a policy breaches an operational deadline. A digital twin may compare proposed state changes before an irreversible action. These interventions can use models internally, but their value comes from an explicit transition model and observable postconditions. Their principal risk is model invalidity, not linguistic hallucination.

Verification also varies in kind. A schema validator can prove syntax, a calculator can reproduce arithmetic, a source checker can test citation support, and a learned reward model can estimate a semantic property. Combining them under one score hides different false-acceptance risks. Use deterministic checks for properties they can establish, calibrated learned checks for residual properties, and treat agreement between correlated models as weak evidence rather than independence.

Generation and verification illustrate why the channels must be priced separately. Self-consistency improves several arithmetic and commonsense benchmarks by sampling diverse reasoning paths and aggregating their answers.³ Verifier-based work instead spends compute on selecting or rejecting candidates. Cobbe and colleagues demonstrated the value of learned verifiers on GSM8K, while Lightman and colleagues found process supervision stronger than outcome supervision in their MATH experiments.⁵, ⁶

There is no universal conclusion that verification should receive more budget. Singhi and colleagues found self-consistency more compute-efficient than a generative verifier for most practical budgets in their tested models and datasets, with the verifier requiring much more compute before matching it.⁷ A later preprint by Zhou and colleagues reported that a routed mixture of cheap and strong verification matched selected high-compute baselines with substantially fewer parameter-weighted tokens on mathematical and symbolic reasoning tasks.⁸ These findings are compatible. Different verifier quality, routing policy, task distribution and cost accounting produce different marginal curves.

Thought experiment: change only the selector

Take one generator and one fixed pool of sixteen candidates. In world P, a deterministic test identifies every valid answer. In world N, a reward model has only a weak relationship with the property that matters. Everything else is unchanged.

In world P, another candidate has value because the selector can exploit the expanded pool. In world N, more candidates may increase oracle coverage while leaving delivered quality flat. The system owns better latent options but cannot recognise them. A recent preprint on open-ended benchmarks reports exactly this kind of exploitation bottleneck, with weak reward-model correlation despite improving candidate pools.⁹ Generation has option value only when the portfolio contains a credible way to exercise the option.

Retrieval has the same interaction. Lewis and colleagues introduced retrieval-augmented generation to combine parametric generation with explicit non-parametric memory and provenance-bearing sources.¹⁰ Later work on inference scaling for long-context RAG found that simply adding context was insufficient; gains depended on jointly allocating documents, in-context examples and iterative prompting under tested benchmark conditions.¹¹ Retrieval is therefore not a fixed prelude. It is an investment whose marginal return depends on evidence gaps, source quality and the model's ability to use what it receives.

Human review is also not a constant-quality terminal node. Bondi and colleagues found that the way selective prediction communicated a deferral changed human accuracy in a conservation task, including a condition where telling the person that the system deferred, without revealing its prediction, improved performance.¹² The portfolio must price the handoff design, reviewer expertise and queue state, not merely add a box labelled “human in the loop”.

flowchart TD
  A["On this page"] --> B["Part I The token budget is the<br/>wrong object"]
  B --> C["Part II Five assets in the<br/>thought portfolio"]
  C --> D["Part III The marginal-value rule"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#5B5CF6,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Part IIIThe marginal-value rule

A portfolio controller begins with a typed decision state, not a generic conversation. The state names the proposed action, candidate answer, current evidence, unresolved propositions, deadline, consequence tier, authority boundary and available interventions. It also records which observations are permitted under the acting identity and declared purpose.

Let s be that state and R(s) the expected consequence-weighted loss if the system stops now and follows its current decision policy. For an intervention i, the next increment creates a distribution of updated states s'. Its gross value is the expected fall in risk. Its net marginal value subtracts the price of resources and side effects.

This equation is a decision aid, not a claim that every consequence can be priced precisely. Its discipline comes from exposing assumptions. A team that cannot estimate incident cost may still rank consequences into bounded tiers, estimate intervals and test whether a recommendation changes across the interval. If every plausible value selects the same intervention, the decision is robust. If the recommendation flips, the uncertainty belongs in the approval record.

Estimate a distribution for each marginal gain, not only a mean. A retrieval call may usually close an evidence gap but occasionally return the wrong entity. A verifier may reject many bad candidates while missing a rare, costly failure. Conservative allocation can use a lower confidence bound on benefit and an upper confidence bound on cost. Exploration belongs in controlled evaluation traffic, not inside a high-consequence decision whose loss curve is unknown.

Observed gain also needs a counterfactual. A case that succeeded after human review does not show that review caused the success if the model's original answer was already correct. Randomised routing, matched holdouts, replay and doubly robust estimators can help in appropriate settings. Where experimentation is unsafe, use shadow mode and adjudicated counterfactual replay. Without a comparison path, the system will credit every expensive intervention for outcomes that would have happened anyway.

Hard constraints are not expensive preferences. Data entitlement, segregation of duties, mandatory verification, statutory review and human decision rights remove infeasible routes before optimisation. A high expected utility cannot purchase authority the system does not possess. This is why the portfolio controller sits inside a deterministic authority kernel rather than replacing it.

The shadow price makes scarcity explicit

A budget cap alone says when money runs out. A shadow price says what the next unit must beat. During normal operation, a batch investigation may tolerate delay and use a low time price. During an incident, the time price rises. When reviewer queues are saturated, human minutes become more expensive and the system should reserve them for cases where judgement or authority is genuinely decisive.

The shadow price should not be inferred from vendor token prices alone. It combines direct cost with the opportunity cost of latency, capacity, disclosure and delayed action. A retrieval call that costs less than a generation call may still be worse if it crosses a privacy boundary or arrives after the decision window. A human review that costs more may dominate because it is the only intervention capable of accepting residual risk.

Interventions can be complements or substitutes

The simple greedy rule is exact only under restrictive assumptions: each channel has a separable, diminishing gain curve; increments do not change the value of other channels; and estimates are calibrated. Real systems violate all three. Retrieval can make generation more valuable by introducing the missing fact. Generation can make verification more valuable by creating diverse candidates. Simulation can reveal which evidence must be retrieved next.

When interactions are material, treat allocation as a small sequential decision problem. Enumerate discrete bundles, use dynamic programming, learn a constrained policy or run a contextual bandit against verified outcomes. The metareasoning literature has long framed computation selection this way. Callaway and colleagues learned policies for deciding when to stop, how to allocate computation among options and how much planning to perform, while explicitly accounting for metareasoning overhead.¹³ Sezener and Dayan similarly modelled the value of simulations by their expected impact on the action eventually chosen.¹⁴

A second allocation level appears when many cases share one budget. A per-case optimiser may spend heavily on every marginally positive opportunity and exhaust the daily capacity before the most consequential cases arrive. The portfolio then needs a global shadow price or quota that rises with scarcity. This is a constrained scheduling problem: compare the next intervention across cases, reserve capacity for protected classes and prevent one verbose trajectory from monopolising the pool.

The controller is itself an intervention and must earn its cost. Its forecasts, routing features and optimisation code belong inside the evaluation boundary. A beautiful policy that consumes more value than it saves is a failed policy.

Part IVA minimal worked allocation

The system can buy up to six cost credits and eight latency units. The first retrieval increment is expected to remove 4.2 loss units because it obtains the missing authoritative record. A second retrieval joins a related record and removes 2.5 more. The first two verifier passes remove 2.0 and 0.7 by checking identity, arithmetic and source support. One generation increment removes 0.9 by testing an alternative interpretation. Later increments have lower or negative value.

The optimiser does not exhaust the budget mechanically. It chooses two retrieval increments, two verification increments and one generation increment. It leaves the simulation and human-review options unused because their marginal value does not clear the priced cost in this task. Expected decision loss falls from 12.0 to 1.7; priced resource costs raise the objective to 5.39. These figures come from the accompanying synthetic artefact.

This example exposes three accounting choices. First, value attaches to the changed decision state, not to an intervention's apparent sophistication. Retrieval wins because it closes the dominant gap. Second, the second verifier pass is still worth buying, but the third is absent because its marginal value would be too low. Third, unspent budget is a valid output. A system that interprets a maximum as a target will manufacture work after the decision is already good enough.

Part VA production-shaped case

Return to the synthetic covenant waiver. The proposed action is not “write a memo”. It is “prepare a decision packet for a committee that may approve, decline, condition or defer an exception”. The action contract requires current exposure, the governing covenant definition, a reproducible ratio, a stressed forecast, relevant policy conditions, unresolved contradictions and named human authority.

The case opens with a baseline proposal generated from the submitted pack. It is useful but incomplete. A portfolio controller evaluates five interventions.

Retrieval receives the first allocation because current exposure and the executed agreement are missing. The controller queries authorised systems under the case identity, records the source version and rejects a relationship manager's email as evidence of contractual terms. The retrieved agreement reveals that one adjustment in the borrower model is not permitted.

Simulation receives the next allocation because the committee needs consequences, not only a corrected ratio. A deterministic workbook or service recomputes headroom under the submitted forecast and two policy-defined stresses. The simulator exposes its assumptions and returns ranges rather than a narrative forecast.

Generation now becomes more valuable. With current facts and simulated outcomes in context, the model drafts two competing interpretations: a temporary timing breach with recoverable headroom, and a structurally weak case under the downside path. It does not choose the committee outcome. It makes the disagreement legible.

Verification checks the decision packet against the evidence contract. It recomputes ratios, resolves citation identifiers, tests that every policy claim points to the effective rule and rejects any sentence that converts an assumption into a fact. A learned verifier may assist with semantic coverage, but deterministic calculations and schema checks carry the properties they can actually prove.

Human review receives the final required allocation because approval is a decision right. The reviewer sees both interpretations, source receipts, simulation assumptions, verifier exceptions and the system's stopping reason. The interface does not present a model confidence as authority.

This sequence looks like a workflow, but the allocation is not hard-coded merely because the example follows it. A complete, current agreement may make retrieval unnecessary. A covenant with no forward-looking condition may not need simulation. A low-impact information request may not require committee review. The policy evaluates task slices and evidence contracts, then enforces any mandatory route for the typed action.

Portfolio thinking also changes capacity planning. Reviewer minutes are not an infinite safety valve. Workload-constrained deferral research shows why cost, error asymmetry and expert capacity must be modelled jointly rather than optimised one case at a time.¹⁵ In the covenant process, the system may reserve reviewers for cases whose outcome actually depends on judgement, while returning incomplete cases for evidence without consuming committee time.

The handoff itself should reduce work rather than move it. The reviewer needs a bounded question, competing options, unresolved facts, source receipts and the consequence of delay. If the person must reconstruct the whole case from a long model narrative, the system has not bought judgement efficiently. It has bought a queue.

stateDiagram-v2
  [*] --> Claim
  state "Thought experiment, two<br/>identical prompts" as Claim
  state "Generation buys alternatives" as Test
  state "Retrieval buys observations" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Executable portfolio optimiser

The artefact below enumerates discrete prefix allocations across the five channels. Each synthetic task profile supplies diminishing loss-reduction curves, direct cost, latency, human minutes and mandatory minimums. The optimiser selects the feasible bundle with the lowest residual decision loss plus priced resource cost.

Its assumptions are deliberately visible. Curves are known, increments within a channel must be bought in order, all benefits are expressed on one synthetic loss scale, and interactions are encoded only through the profile. Those assumptions are too strong for a live system, but they reveal the logic that prose can hide.

best = None
for bundle in every_prefix_combination(channels):
    if violates_authority_or_capacity(bundle):
        continue

    residual_loss = max(0, baseline_loss - bundle.expected_loss_reduction)
    objective = (
        residual_loss
        + spend_price * bundle.direct_cost
        + delay_price * bundle.latency
        + human_price * bundle.human_minutes
        + controller_overhead
    )

    if best is None or objective < best.objective:
        best = bundle

return best  # stopping is the all-zero bundle

From experiment to operating policy

A useful optimiser is not yet an operating control. The release unit is a versioned task-level allocation policy joined to evidence, outcome and cost. It names the population on which the curves were learned, the permitted interventions, the protected constraints, the maximum trajectory budget, the stopping rule and the fallback when a dependency or capacity pool is unavailable.

Learn curves on decision-relevant slices

Run each intervention at several budgets across frozen, representative cases. Score verified decisions, evidence coverage, false acceptance, postconditions and consequential errors. Keep challenge sets for incidents, novel formats and near misses separate from population samples. Otherwise a tail-heavy evaluation will overbuy expensive work for routine traffic, while an average-heavy evaluation will miss the dangerous tail.

Model routing and cascading research demonstrates that cost-quality frontiers can improve when systems select among models or escalate selectively.¹⁶, ¹⁷ The same lesson applies across intervention types, but the router needs labels for what changed the decision, not merely which model won a preference vote.

Keep the allocation estimator separate from the worker

The generator should not grant itself an unlimited budget or certify that more of its own work is valuable. Difficulty signals may come from business state, evidence completeness, independent calibration, cheap probes or observed verifier failures. Where consequence is already known, use deterministic rules. A large exposure, expired screening result or irreversible action should not depend on a model first noticing that the case matters.

Record a decision receipt

For every consequential trajectory, record the initial state, interventions considered, increments bought, expected and observed gains, resource use, rejected routes, authority constraints, stopping reason and verified outcome. This receipt is not hidden chain-of-thought. It is operating evidence for the system's externally observable choices.

Join the receipt to model, prompt, retriever, policy, simulator and verifier versions. Marginal curves can reverse after any one of them changes. A cheaper model may make verification relatively more valuable. A better retriever may reduce the need for generation. A new policy may make human review mandatory for a previously automated slice. Version change is therefore a recalibration trigger, not a routine deployment detail.

Outcome observation must extend beyond acceptance. A verifier may approve a command whose effect later times out, duplicates or lands in the wrong state. Record the postcondition and any correction, override, complaint or incident that arrives after the decision. The portfolio should learn from verified business outcome, not from the model's own claim that the task completed.

Inject failures into the allocation decision

Do not test only whether the selected intervention works. Corrupt the routing signal, remove a source, stale the simulation, bias the verifier, overload the reviewer queue and increase the shadow price during a deadline. Measure whether the controller recognises the changed state, shifts spend appropriately, preserves mandatory checks and enters a safe scarcity mode.

A scarcity policy should degrade optional generation before required verification, queue low-consequence work before bypassing authority, and switch to a fixed evaluated path when the allocation estimator is unavailable. The order is a design decision backed by consequence, not a generic preference for one component.

The budget should follow unresolved uncertainty

Test-time compute made inference spend adjustable. The next design step is to stop equating adjustable spend with longer model output. A system should buy the intervention that changes the decision state most, not the intervention that is easiest to meter.

That means generation when no adequate candidate exists, retrieval when evidence is absent, simulation when consequences are unknown, verification when error can be detected independently, and human review when judgement or authority is irreducible. Each channel needs a measured marginal curve, a failure signature and a receipt.

The architecture decision changes accordingly. Replace the global reasoning setting with a constrained, versioned allocation policy. Compare it with fixed and deterministic baselines. Let hard controls remove forbidden routes. Let stopping compete with further work. Recalibrate when tasks, models, sources, verifiers, prices or reviewer capacity change.

The result is modest but consequential: not an artificial mind that knows how long to think, but an operating system that can defend what kind of work it bought next, what uncertainty that work resolved, and why another unit was no longer worth buying.