Calibrated Cascade Routing in Production: Small Models First, Confident Escalation

The toll of routing everything to the frontier

The same cost problem appears across bank AI programmes: a frontier-model demonstration succeeds, but the production inference bill grows faster than demand. Every request, from a balance question to a multi-step dispute, has been sent to the same model at the same unit cost and latency. The architecture has no mechanism for matching model spend to request difficulty.

Picture a contact-centre agent embedded in a financial institution. It reads an incoming customer request, classifies the intent, retrieves permitted account or product information, drafts a response and, in a minority of cases, escalates to a human reviewer. The worked composite in this article assumes 2.4 million interactions a month across chat and voice-transcript follow-up.

At an assumed 0.028 USD per 1,000 output tokens and 610 output tokens per interaction, routing all 2.4 million interactions to that tier costs about 40,992 USD a month in raw output-token charges, before retries, input tokens, tools or hosting overhead. The same composite uses p50 latency of 1,850 milliseconds and p95 of 4,100 milliseconds. Those waits matter in an assisted-service workflow even when they do not determine the whole call duration.

Most of those requests may not need a frontier model. A large proportion are narrow, bounded and repetitive: balance enquiries, standing-order questions, card-block requests, address updates and document checks. A smaller proportion are genuinely hard: ambiguous disputes, multi-account fraud patterns, mixed intents and policy edge cases. Sending every case through the same route is not only a modelling choice; it is an architecture choice.

It is a routing decision made explicitly, per request, with a measured and monitored confidence signal deciding whether a small model's answer can be trusted or whether the request needs to go further up the chain. That decision, done properly, is where most of the savings live, and it is also where most naive implementations quietly break.

All volumes, prices, thresholds, performance values and incidents below form one modelled composite. They make the arithmetic and control choices inspectable; they are not reported measurements from a disclosed institution. Production decisions require current provider prices, locally measured token distributions and adjudicated outcomes.

The cascade principle

A cascade is a sequence of models ordered by capability and cost, where a request is first attempted by the least costly route that has demonstrated acceptable risk for that cohort. Requests then defer to a stronger route when the acceptance condition is not met. In the composite, 72% of requests fall into high-frequency, low-ambiguity classes, 20% need more reasoning within one domain and 8% form the difficult tail. This distribution is an assumption to test, not a universal law of enterprise traffic.

The reference design has three tiers. Tier 0 is a distilled 1-to-3-billion-parameter model calibrated on the local task. Tier 1 is a mid-sized model used when Tier 0 defers. Tier 2 is a frontier route reserved for the difficult tail, while materially sensitive cases can bypass model confidence and go to a deterministic or human gate. Tier 0's purpose is bounded coverage, not universal correctness.

The economics only work if the gating decision at each tier is trustworthy. If Tier 0 escalates too eagerly, the cascade collapses into "send everything to Tier 2 anyway" with extra latency bolted on for the Tier 0 attempt. If Tier 0 escalates too rarely, wrong answers reach customers or back-office queues with a false stamp of confidence, and the bank absorbs the cost of the error downstream, often at far higher expense than any inference bill. This is the crux of the whole design: the gate has to be calibrated, not merely present.

The eligibility gate matters more than the ordering. Customer-specific balances, payment instructions, personalised decisions and action responses stay out of a shared semantic cache. Eligible entries carry tenant and entitlement scope, policy version, source version and expiry. A similarity score cannot substitute for those exact controls.

Calibration: why raw confidence cannot be trusted

A common cascade error is using a small model's raw softmax probability as the escalation signal, assuming that 0.94 means 94 percent of comparable predictions are correct. It does not unless calibration data demonstrates that relationship. Neural classifiers, including fine-tuned language models, can be overconfident. In the worked example, outputs near 0.97 are wrong about 15 percent of the time. Self-reported certainty is weaker still because it can shift with wording rather than task evidence. The exact gap is illustrative; the required control is an observed reliability curve on local held-out data.

Calibration is the discipline of correcting this gap so that a stated confidence of 0.80 genuinely corresponds to an 80% chance of correctness, measured against held-out labelled data. Three techniques cover most production needs.

Temperature scaling

Temperature scaling divides the model's logits by a single learned scalar, T, before softmax. It does not change the ranking of class outputs; it changes the sharpness of the probability distribution. It is comparatively cheap to fit against held-out labels by minimising negative log-likelihood. In the composite, a 6,200-ticket calibration set moves average confidence on correct answers from 0.96 to 0.89 and on incorrect answers from 0.91 to 0.61. The values are illustrative; the required evidence is an out-of-sample reliability curve and uncertainty by cohort.

Platt scaling

Platt scaling fits a logistic regression on top of a raw score, mapping it to a probability with a slope and intercept. It can suit a binary score whose error is not well described by uniform logit sharpening. For example, a fraud-flag classifier may be materially overconfident on the “no fraud indicators found” class even when its other class is acceptable. That pattern should be demonstrated on held-out local data before choosing the calibrator.

Isotonic regression

Isotonic regression is a non-parametric method that fits a monotonic step function from raw scores to calibrated probabilities. It can fit shapes a logistic curve misses, but its flexibility increases the data requirement and overfitting risk. Ten thousand labels may be a useful planning assumption for one problem, not a general minimum. Compare methods out of sample with reliability diagrams, a proper scoring rule and cohort error rather than choosing from the training fit.

Expected calibration error as the metric that matters

None of this is useful without measurement. Expected calibration error, ECE, bins predictions by confidence, compares mean confidence with observed accuracy in each bin and weights the absolute gaps by bin population. An ECE of 0.02 means a two-point weighted average bin gap under that particular binning; it does not prove every cohort or high-confidence tail is calibrated. The composite starts Tier 0 at ECE 0.114 before calibration.

After temperature scaling, the composite ECE is 0.031. That measure belongs beside accuracy, reliability plots, proper scoring rules and cohort error. A rising ECE can warn that a score-based gate is changing even while aggregate accuracy appears stable.

It is worth being explicit that calibration and accuracy are different properties and can move independently. A model can become more accurate after a fine-tune while simultaneously becoming worse calibrated, because the fine-tune sharpens its output distribution without correcting the mapping between confidence and correctness. This exact scenario recurs in the failure modes section below, and it is the single most common cause of cascade degradation I have seen across bank deployments.

Escalation policy design

Calibration gives you a trustworthy probability. The escalation policy is the separate decision of what threshold on that probability triggers a handoff to the next tier, and this decision is not a single global number. Different intents carry different costs of error. A wrong answer on a balance enquiry is embarrassing and quickly corrected. A wrong answer on a sanctions screening flag or a large-value payment instruction carries regulatory and financial consequences that are orders of magnitude more expensive than the inference cost saved by staying at Tier 0. Because of this, every deployment I have built uses per-intent thresholds rather than one number applied uniformly.

In the payments back-office system, the calibrated confidence threshold for escalating from Tier 0 to Tier 1 sits at 0.78 for routine account maintenance intents, but at 0.92 for anything touching payment instruction amendments, and at 0.97, effectively "escalate almost everything", for sanctions-adjacent language regardless of the model's stated confidence, because the cost asymmetry there is severe enough that a small residual error rate is not acceptable at any inference cost saving. Thresholds are tuned against a held-out calibration set of real historical tickets, typically 8,000 to 15,000 examples per intent family, refreshed quarterly.

For each candidate threshold, we compute the resulting escalation rate, the accuracy of the Tier 0 answers that would be accepted at that threshold, and the downstream cost of the errors that would slip through, then select the threshold that minimises total expected cost, which is inference cost plus the expected cost of accepted errors, not the threshold that maximises accuracy or minimises escalation in isolation.

The boundary case deserves attention. A request at calibrated confidence 0.77 against a threshold of 0.78 is not operationally far from one at 0.79, yet the routing action changes. The composite uses a ±0.03 review band: the request defers to Tier 1 while the Tier 0 answer is retained for comparison. Agreement between two models is not ground truth. It makes the case a candidate for human adjudication; only the adjudicated outcome may enter a labelled calibration or training set.

Monitoring in production has to track more than aggregate accuracy. The composite computes live ECE per intent family on a rolling sample of human-reviewed outcomes. A two-percent sample is shown for capacity planning, not prescribed as adequate. A sustained ECE increase of 0.02 opens a recalibration investigation; it does not automatically retrain from the metric alone, because label delay, cohort mix and bin occupancy may explain the movement.

Semantic caching on top of the cascade

Even a well-tuned cascade repeats stable, non-personal work when wording differs. “Where is the nearest branch?” and “show branch locations” may share an approved informational response after entitlement and location handling. Semantic caching embeds an eligible request and searches a scoped index for an entry above threshold. It should not reuse customer-specific facts or executable instructions merely because their wording is similar.

The similarity threshold needs local tuning, but eligibility comes first. “Increase my transfer limit to 5,000” and “increase it to 50,000” may sit close in embedding space; both are excluded from response caching. The composite uses 0.94 for stable policy explanations and 0.89 for low-consequence public information. Neither value transfers safely to another embedding model, corpus or distance function.

Cache hit economics are worth stating plainly. In the composite, embedding and lookup cost 0.00002 USD. Tier 0 costs 0.0006 USD per 1,000 output tokens and a cacheable response averages 180 output tokens, or 0.000108 USD. On those stated components, a cache hit is about 5.4 times cheaper, and latency falls from 85 milliseconds p50 to under 20 milliseconds. The worked route uses a 31% cache hit rate; a real rate depends on intent mix, safety exclusions and invalidation policy.

Staleness is the counterweight to this saving. A cached product rate becomes wrong when the source changes. The reference control combines a time-to-live set by content class with an invalidation hook from the authoritative product system. Four hours for rates and 30 days for stable location content are scenario settings; the source's change process and consequence should set the real values.

Without active invalidation, stale answers remain possible until TTL expiry. That exposure window should be measured directly and treated as a release constraint, not inferred from attractive aggregate cache economics.

Failure modes in production

A cascade with calibrated gating and semantic caching is not a system to deploy once and leave alone. The following failure patterns provide concrete tests for the first operating year.

Overconfidence drift after a small-model fine-tune

The symptom appears gradually: escalation rate from Tier 0 to Tier 1 drops over several weeks without any change to the traffic mix, and a few weeks after that, downstream error reports start ticking up on intent categories that had previously been stable. The cause is that a routine fine-tune of the Tier 0 model, done to improve raw accuracy on newly labelled tickets, sharpens the model's output distribution and pushes confidence scores upward across the board, but the calibration mapping fitted before the fine-tune is now stale and no longer corrects for the new confidence distribution.

In the failure scenario, a fine-tune drops escalation from 22% to 14% while accepted-case accuracy falls from 96% to 89%. The fix is procedural: re-estimate calibration after every behaviour-changing update and invalidate old routing thresholds until the new mapping passes evaluation.

Cache poisoning from stale embeddings

The symptom here is subtler: a small number of customers report being given information that was correct weeks ago but is now wrong, clustered around a specific product or rate change, while the overall accuracy metrics look fine because the affected volume is a small percentage of total traffic. The cause is a gap between the event that changes ground truth, such as a rate update, and the invalidation hook that is supposed to purge the relevant cache entries, often because the hook was scoped too narrowly to the exact product code and missed a related product variant that shares the same rate.

In the cache-failure scenario, an invalidation event is scoped to one product code and misses a linked account type. The control ties invalidation to the underlying authoritative field and supplements events with a sampled consistency sweep. The sample rate should be chosen from change frequency and harm, not copied from the illustrative 1% daily sweep.

Threshold miscalibration under distribution shift

The symptom is a sudden and sustained increase in escalation rate, or occasionally a sudden drop, that correlates with an external event rather than any change to the models or thresholds themselves. The cause is that the calibration set used to tune thresholds was drawn from a traffic distribution that has since shifted, commonly after a product launch, a regulatory change that alters the shape of customer queries, or a seasonal pattern such as a spike in dispute volume after a widely publicised card fraud campaign.

In the distribution-shift scenario, a new instant-payment rail moves escalation on payment-timing queries to 61% against a 24% comparison cohort. The remedy is to treat calibration sets as living artefacts and collect labelled pilot cases before full rollout rather than wait for organic volume.

Cascading latency when escalation rate spikes

The symptom is system-wide latency degradation even though unloaded service time at each model is unchanged. The cause is capacity planning around a 19% Tier 1 and 5% Tier 2 deferral pattern while stressed demand moves them to 40% and 15%. Upper tiers then queue under contention.

In the modelled stress case, a fraud campaign pushes Tier 1 escalation to 47% and Tier 1 p95 latency from 610 to 2,100 milliseconds through queuing. The response combines upper-tier headroom with a pre-approved degraded mode. Lowering the Tier 0 acceptance threshold would reduce escalation, but it would also admit lower-confidence answers; it is permitted only for reversible, low-consequence cohorts inside a separately tested floor. High-consequence cases fail closed or queue for human review. Raising the acceptance threshold would increase escalation and worsen this capacity problem.

Worked example: a global financial institution contact centre

The worked contact-centre model assumes 2.4 million customer interactions a month across chat and voice-transcript follow-up. Every interaction initially reaches one frontier-tier model. At an assumed $0.028 per 1,000 output tokens and 610 output tokens per interaction, monthly inference cost is about $41,000. The scenario uses p50 latency of 1,850 milliseconds, p95 of 4,100 milliseconds and a target p50 of 1,200 milliseconds. These are model inputs for comparing architectures, not reported client measurements.

The modelled design introduces a 1.3-billion-parameter Tier 0 trained on 340,000 anonymised tickets, a 13-billion-parameter Tier 1 and the existing frontier route as Tier 2. Before calibration, the scenario assigns an implausibly high 58% Tier 0 deferral rate because raw scores cluster near the top of the range. This version saves little because more than half the non-cached traffic reaches an upper tier.

After temperature scaling on 12,500 held-out tickets and per-intent threshold tuning, Tier 0 defers 19% of the traffic it sees and Tier 1 defers 5% of its received traffic to Tier 2. The scenario's eligibility-controlled cache absorbs 31% of total traffic first. Of 2.4 million monthly interactions, 744,000 are cache hits and 1,656,000 reach Tier 0. Tier 0 resolves 1,341,360 and defers 314,640. Tier 1 resolves 298,908 and defers 15,732 to Tier 2. The counts use the same conditional denominators as the stated rates.

Using 610 output tokens for every model call, the stated output-token rates produce about 606 USD at Tier 0, 729 USD at Tier 1 and 269 USD at Tier 2. Adding about 15 USD for the stated cache lookups gives a raw monthly total near 1,619 USD, roughly 96% below the 40,992 USD all-frontier output-token baseline. This comparison excludes input tokens, reserved capacity, retries, tooling and hosting on both sides; a fully loaded business case must add them consistently. The modelled p50 is 190 milliseconds and p95 is 980 milliseconds. A 3,000-case human review yields 94.6% quality against a 95.1% reference; that difference is acceptable only if its interval clears a pre-agreed non-inferiority margin.

The worked monitoring trace starts Tier 0 ECE at 0.028 and keeps it between 0.025 and 0.035 through two modelled retunes. A fine-tune then pushes ECE to 0.061 and triggers recalibration. These are scenario values for testing the alert logic, not a disclosed production time series.

The routing plane as a decision system

A cascade is a sequence of abstention decisions. Each stage either answers inside a defined envelope or yields to a stronger route. Calibration, consequence and capacity are therefore part of the product contract. The cheapest model should not go first merely because it is cheap. It should go first only where its accepted error envelope fits the action.

Confidence and consequence are separate axes. A highly confident model can still be wrong on a case with an unacceptable downside. A low-confidence case may be harmless enough to handle with an undo window. Route on both.

Calibrate on the population that reaches each gate

The second model does not see the original population. It sees cases rejected by the first stage. That set is usually harder and differently distributed. Calibration must therefore be measured at each gate, after every upstream filter.

Run the cascade as a capacity market

A route can be statistically sound and operationally unsafe. Human queues saturate. Strong-model latency spikes. Provider quotas bind. A production router needs a capacity policy before those conditions arrive.

A fallback is a product state, not an exception handler. Define which requests pause, which narrow and which continue. Queue age belongs beside model quality on the release dashboard. A perfect escalation policy fails when no qualified reviewer can respond.

Guo et al.’s calibration study remains a useful baseline. Selective prediction is framed directly in work on risk-coverage trade-offs. The NIST AI RMF connects measurement to governance. The NIST Generative AI Profile broadens the risk catalogue. The Federal Reserve’s 2026 revised model-risk guidance is relevant as a source of risk-based validation practice, while explicitly excluding generative and agentic AI from its formal scope.

Three release questions follow. Is each score calibrated on the cases that reach its gate? Does every route have a consequence ceiling? Can the system enter a safe state when capacity disappears? If any answer is unknown, the cascade is not yet production-ready. Acceptance is a testable policy decision, not a feeling about a score.

The route and its threshold form one governed decision. Changing either one changes the accepted risk.

Notes for practitioners

Build the calibration pipeline before the cascade, not after. It is tempting to wire up three model tiers, get something running end to end, and treat calibration as a tuning exercise to layer on later. Every time I have seen this order reversed, the initial escalation rate has come out far too high or far too low, and the team spends weeks debugging what looks like a routing bug but is actually an uncalibrated confidence signal.

Track expected calibration error as a first-class production metric, on a dashboard next to latency and cost, not buried in a model evaluation notebook that gets checked at release time. A rising ECE is an earlier and more specific warning than a falling accuracy figure, because accuracy can hold steady for weeks while calibration quietly drifts underneath it, particularly after any fine-tune of the small model.

Set thresholds per intent, against the true cost of an error for that intent, and resist the pressure to ship a single global threshold because it is simpler to explain. The cost asymmetry between a wrong balance figure and a wrong sanctions flag is large enough that a uniform threshold is wrong for one of them by construction.

Treat the calibration set as something that needs active maintenance tied to the product roadmap, not a static artefact refreshed on a fixed calendar. Any planned product launch, rate change, or new customer path through the contact centre should trigger a review of whether the existing calibration set still represents the traffic it is about to see.

Size upper-tier capacity against the worst observed escalation rate, not the average, and agree a circuit-breaker policy with the business before an incident forces an improvised one. A cascade that saves cost on an average day but falls over during a fraud campaign or a product launch has simply moved the risk from the inference bill to the incident log.

Finally, treat semantic caching as a layer with its own failure modes, not a free win bolted on top of the cascade. A cache with a threshold set too loose or an invalidation hook scoped too narrowly will serve wrong answers with the same apparent confidence as right ones, and because a cache hit never reaches a model, none of the confidence gating built into the cascade will ever catch it. The safeguard has to live in the cache layer itself: tight per-intent similarity thresholds, TTLs matched to how often the underlying data actually changes, and a background sweep that checks the cache against source systems rather than trusting the invalidation hooks to catch everything.