The gate, not the ladder

A familiar diagram places a rule engine first, a small model second, a frontier model third and a person last. The arrows imply progress: each step is more capable and more expensive. That picture hides three separate choices.

First, a router selects one route before seeing its output. It needs ex-ante evidence about which route will perform well. Second, a cascade runs a route, observes its output and then decides whether to continue. It needs post-hoc evidence that the result is inadequate and that another route is likely to improve it. Third, an authority policy removes inadmissible routes regardless of predicted quality. A model may draft a payment instruction but still lack authority to release funds.

These distinctions matter because a route can be useful in one role and wasteful in another. If the input itself reveals that a specialised calculator dominates every language model, route directly to the calculator. If a generated citation can be checked only after generation, a post-hoc cascade may be useful. If policy requires human approval, no confidence score can turn model output into an authorised decision.

Routes are not ordered by one capability scale

A rule is narrow but can provide an exact, inspectable result. A small model is broad but may be weak on rare combinations. A frontier model may improve synthesis while remaining unable to observe a private ledger. A tool can see the ledger but answer only its typed question. A person can exercise judgement and authority, yet may be slower, inconsistent or unfamiliar with a specialist detail. Calling this sequence “increasing intelligence” erases the properties that determine the route.

Build the action set from the task, not from a vendor catalogue. For each route, state its preconditions, evidence access, effect, error modes, cost, latency, capacity and authority. Some routes are parallel alternatives. Others create observations needed by later routes. A retrieval call can enrich context for either model. A calculator can verify a numerical claim without replacing the surrounding explanation. A person may approve an action after a model drafts it, rather than “answering the same question better”.

A defensible cascade is therefore partially ordered by dependency and authority, not totally ordered by model size. The diagram may still flow from cheap to expensive in one workload, but that ordering is an empirical consequence of the gate objective. It is not an architectural law.

The residual-value test

Suppose stage i has produced an answer and observations z: confidence features, validator findings, retrieved evidence, tool results, policy flags and queue state. Accepting the answer has an expected consequence-weighted loss. Every permitted next route has its own expected loss and incremental operating cost. The gate should compare those futures.

The equation does not require money to be the sole unit. A latency service-level objective can be represented as a steep penalty after a deadline. A human workload limit can become a queue cost. A privacy restriction can make a route inadmissible rather than merely expensive. The purpose of a common objective is not to pretend every value is commensurable. It is to stop teams from counting token price while leaving error, delay and review labour invisible.

Only future costs belong in the gate. The cost of the small-model attempt is sunk after it has run. It matters when comparing the whole cascade with direct routing, but it should not bias the current stop-or-continue decision. At the gate, ask whether buying the next observation or answer improves the future enough to pay for itself.

Worked example 1: what did the first attempt buy?

Return to the 100-case example. Eighty cases are ordinary and twenty are hard. The junior costs £1, the senior £8, and an error carries £100 expected loss. On ordinary cases their error rates are 3 and 2 per cent. On hard cases they are 30 and 10 per cent. The calculations below use expected counts, so fractional errors are intentional.

When a reliable input feature identifies the twenty hard cases, the direct router and cascade produce the same final allocation. The cascade is £20 worse because every hard case pays for a discarded junior attempt. Its post-hoc gate has no informational role. This is a clean baseline, not a criticism of all cascading.

Now remove the input feature. Let a validator detect a contradiction in 90 per cent of junior failures and falsely flag 5 per cent of correct drafts. The junior attempt has become an experiment. Its draft creates a noisy observation that changes the posterior value of senior review. The cascade can now outperform any router limited to the original input, provided the validator's false positives, missed failures and added delay do not consume the benefit.

The distinction is the value of information. A first attempt may produce a usable answer, a diagnostic observation, or both. Measure those benefits separately. If teams credit the cheap stage for every later success but never charge it for duplicate work and false escalation, almost any ladder can be made to look efficient.

Why later stages must be measured on the residual

A frontier model may score 94 per cent on the full test set and only 71 per cent on the cases rejected by the small model. That is not a contradiction. The first gate changes the distribution. It removes easy cases and enriches the remainder for ambiguity, missing context, adversarial wording and shared blind spots.

This creates a common evaluation error. Teams compare each component on the original benchmark, order them by aggregate quality, then assume that ordering survives inside the cascade. It need not. The relevant quantity is the incremental advantage of route j over the current route, conditional on the cases and evidence at this gate. Complementary error can make a cheaper specialist more valuable than a generally stronger model. Correlated error can make a frontier escalation almost useless.

FrugalGPT provided an early language-model demonstration of complementary errors: on its evaluated datasets, inexpensive APIs sometimes answered cases that a more expensive API missed, allowing a learned cascade to improve cost and, in some settings, accuracy. Its authors also state the boundary clearly: the cascade required labelled data and similar train-test distributions.⁵ RouterBench later supplied a broader multi-model outcome dataset and an explicit framework for comparing routers.⁹

flowchart LR
  A["The gate, not the ladder"] --> B["What can justify escalation"]
  B --> C["The full cost of another attempt"]
  C --> D["Build and test the cascade"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#7B4CD8,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

What can justify escalation

A gate is an estimator with operational consequences. It predicts whether another route will improve the outcome enough to justify its cost. Four evidence classes are useful, but none is sufficient everywhere.

Ex-ante evidence: route before paying for a weak attempt

Input type, language, length, required modality, known domain, requested action and available tools can predict comparative performance before any model runs. When this evidence is strong, direct routing avoids duplicate work. RouteLLM learns such a strong-versus-weak choice from preference data. Its results also expose the distribution problem: routers trained on general preference data performed near random on some out-of-domain benchmarks until the training set received small amounts of task-relevant augmentation.⁷

Ex-ante routing is especially appropriate for eligibility. A calculator should receive an arithmetic expression directly. A sovereign-data constraint can select a local model before content leaves a boundary. A policy-controlled case can go straight to a person. In these examples, running a small language model first produces no decision-relevant evidence.

Post-hoc evidence: the first attempt must reveal something new

A cascade becomes attractive when the output exposes information unavailable from the input: a failed schema, unsupported claim, contradiction, low retrieval coverage, tool error, test failure or disagreement among independent checks. The 2025 unified analysis of routing and cascading makes the dependency explicit. Routing benefits from accurate ex-ante quality estimation; cascading benefits when post-hoc estimation adds substantial information. If post-hoc evidence adds little, direct routing is preferable.⁸

Why model confidence is an unsafe shortcut

Classification confidence can be calibrated against empirical correctness, although modern neural networks are often miscalibrated and simple post-hoc calibration can deteriorate under dataset shift.²³ Generative output creates an additional problem: it supplies a sequence of token probabilities, not one probability that the answer is correct.

Gupta and colleagues show that summing token log probabilities tends to penalise longer generations, while averaging can over-correct and favour them. Token-level quantiles and learned post-hoc rules improved deferral across their FLAN-T5 experiments, but no single quantile worked across all tasks.⁶ A fluent answer can also be confidently unsupported. A low-likelihood wording can be correct. The gate should predict comparative outcome improvement, not merely recycle the model's ease of generation.

Calibrate the gain, not just the answer

The natural training label for a gate between routes i and j is paired: how much consequence-weighted loss did route j remove relative to route i on this case? Correct-versus-incorrect labels are sufficient only when all errors have equal cost and the downstream route is an oracle. Neither assumption is generally safe.

Create paired outcomes by running both routes on a development sample, scoring them against trusted references or blinded review, and preserving disagreements. For open generation, separate factual support, task completion, policy adherence and presentation. A single model judge can be useful for triage, but correlated preferences may teach the gate to favour the same style that one route produces. Anchor the gate with human-labelled or mechanically verifiable cases and report judge disagreement.

Then plot predicted gain against realised gain by decile and by consequence band. A well-ranked but poorly calibrated estimator can still support a budgeted router, yet its numeric threshold should not be interpreted as an error probability. Conversely, a globally calibrated score can be operationally useless if it fails to rank the residual cases where the routes differ. The release metrics should include:

• paired reduction in expected loss and its uncertainty;
• severe misses among accepted cases, not only average error;
• coverage and route share at every gate;
• regret against the best available route in hindsight;
• cost-quality curves across thresholds, including direct routing; and
• queue exceedance, deadline miss and fallback use under load.

Keep train, threshold-selection and test periods separate. A gate fitted to a model pair is part of that pair's release lineage. Replacing the small model, changing retrieval, shortening prompts or updating a policy can change both the residual cohort and the meaning of the score, even when the gate code is untouched.

Tools and people are comparative routes, not oracles

A deterministic tool can dominate a model on an eligible subproblem and be useless elsewhere. A database lookup may verify an identifier but not interpret an ambiguous policy. A solver may prove a constraint result but fail on a mistranscribed input. Tool eligibility, input validity and postcondition verification therefore belong in the gate.

People have expertise, context and accountable authority, but they also have heterogeneous skill, fatigue and limited capacity. Learning-to-defer research treats the human as another fallible decision-maker whose comparative competence matters.¹⁰ Work on cost-sensitive deferral extends the problem to asymmetric error costs, multiple experts and workload limits.¹¹ A human route without named skills, capacity and decision rights is not a control. It is an unmeasured queue.

The full cost of another attempt

Token price is visible, so it often becomes the optimisation target. A cascade can reduce token spend while increasing end-to-end cost. Sequential calls add latency. A verifier consumes compute. Repeated prompts duplicate input. A human queue misses deadlines. A wrong low-consequence answer and a wrong irreversible action should not carry the same loss.

This framing produces two useful effects. First, it can justify spending more compute on a rare, high-consequence case. Second, it can reject a seemingly accurate cascade whose queueing or switching cost erases its benefit. The relevant objective is the outcome of the whole decision process, not the unit price of one model call.

Thresholds are portfolio and load decisions

A local threshold affects every downstream resource. Tightening the small-model gate raises tool traffic. Tool failures raise frontier traffic. Frontier abstention raises human traffic. Human delay can then increase the loss of cases that would otherwise benefit from review. Optimising each gate independently can therefore produce a globally inferior cascade.

Use either joint optimisation or an explicit operating budget for the whole portfolio. At minimum, sweep threshold combinations rather than one threshold at a time. Recalculate under peak arrivals, reviewer absence, model rate limits and tool outages. If the chosen policy depends on a narrow capacity margin, the fallback threshold and its quality cost must be approved before the incident, not improvised during it.

This also clarifies why “human in the loop” cannot be a residual slogan. Name the queue, service discipline, skills, maximum work in progress, priority policy and deadline. A person who receives a low-quality case bundle without the evidence that triggered escalation may spend more time reconstructing context than deciding the case. That reconstruction is part of switching cost.

Worked example: a governed service-case cascade

Consider a synthetic operation processing 100,000 service cases each month. Some inputs are structured and low consequence. Others contain free text, missing evidence or requests that could affect customer rights. The architecture uses five routes:

1. Rules validate mandatory fields and resolve unambiguous structured states.
2. A small model classifies intent and drafts routine responses.
3. A deterministic tool verifies identifiers, balances or policy predicates where an exact interface exists.
4. A frontier model handles residual synthesis when the evidence package is sufficient and the action remains advisory.
5. A person decides cases with reserved authority, unresolved contradiction or high residual consequence.

The synthetic gate accepts 35 per cent at rules, 42 per cent at the small model, 12 per cent after a tool, 8 per cent after frontier revision and 3 per cent after human review. Those percentages are not targets. They are outputs from case mix, thresholds and authority policy. A change in product, evidence quality or reviewer capacity should move them.

Suppose the small model costs £0.03 per attempted case, the frontier model £0.65, and human review £8 before queueing. Human review dominates direct service cost even at 3 per cent of volume. Yet removing it can increase expected loss sharply if those 3,000 cases carry concentrated consequence. The correct optimisation asks whether the human route is applied to the residual where it has comparative and authority advantage, not whether it is expensive.

Accepted share is not the same as attempted share. All 100,000 cases receive the rule check. The small model attempts the 65,000 not accepted by rules. Of its 23,000 residual cases, 15,000 are eligible for exact tool verification; 12,000 clear. The 8,000 ineligible cases and 3,000 inconclusive tool cases reach the frontier model. It accepts 8,000, leaving 3,000 for people.

The table reveals two decisions hidden by a route-share chart. First, the operation buys 194,000 route attempts to complete 100,000 cases. Direct routing could be cheaper if input features reliably identify tool, frontier or human cases. Second, human review is about 72 per cent of service cost. A one-point reduction in human share saves more direct cost than a large percentage reduction in small-model price, but only if those cases can move without increasing expected harm or violating authority.

Add consequence and the ranking may reverse. Suppose one missed high-severity case carries £20,000 expected recovery cost while an ordinary drafting defect carries £20. A route that prevents ten severe misses can justify substantial review spend even if average accuracy barely moves. This is why the evaluator uses consequence-weighted loss and why severe cases need their own acceptance ceiling.

The operating ledger for each gate records more than a threshold. It stores the current case definition, upstream versions, estimator calibration, downstream comparative gain, route cost, capacity, admissibility rule, owner and expiry. The evidence must be replayable. NIST's AI risk guidance similarly treats measurement, monitoring, human oversight and documented response as continuing activities rather than one-time model tests.¹²

stateDiagram-v2
  [*] --> Claim
  state "Routes are not ordered by one<br/>capability scale" as Claim
  state "The residual-value test" as Test
  state "Worked example 1: what did the<br/>first attempt buy?" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Build and test the cascade

A cascade is a policy over observations, routes and constrained resources. Treat it like an experimental system. Reproduce every component baseline, preserve the paired cases that drive escalation, and test how errors propagate when the gate, downstream route or queue is wrong.

python paper-084-cascade-evaluator.py --case both --cases 20000 --seed 84
python paper-084-cascade-evaluator.py --case positive --csv cascade-results.csv

# Expected checks:
# Positive: informative cascade meets the risk ceiling and beats always-frontier.
# Negative: optimisation closes the gate; forced escalation is dominated.

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: The gate, not the ladder
  M->>E: Routes are not ordered by one capability scale
  E-->>M: compare rival explanations
  E->>D: Escalate only when the residual earns it
  D-->>Q: revise scope or act

Escalate only when the residual earns it

The largest model is not a universal destination, the human is not an infinite oracle, and confidence is not authority. A defensible cascade treats every continuation as an option bought with compute, time, capacity and risk. The option has value only when evidence at the gate predicts that a particular admissible route will reduce consequence-weighted loss on the residual population.

The architecture decision changes in one precise way: design the gate before adding the tier. Name the action, define the residual, measure paired comparative gain, price the full marginal cost, enforce authority mechanically and run a negative control that removes the claimed advantage. If the gate cannot survive that test, route directly, use a fixed mechanism or abstain.

The enduring unit of inference economics is therefore neither the token nor the model call. It is the verified reduction in expected decision loss purchased by the next observation or action. A cascade that can show that reduction is an engineered allocation system. One that cannot is merely a more expensive diagram.