A compact model costs less per call. On the hard slice it retries, invokes tools, reaches a reviewer and still leaks failures. A stronger model costs more per call yet can be cheaper per verified outcome.

Token arithmetic is necessary and insufficient. Input and output tokens, cached context, model tier, test-time compute, tool calls, serial and parallel branches, retries, validators, fallbacks, batching, tail latency, human review and recovery interact with the request population. The route, not the isolated call, is the economic unit.

Inference economics designs that route. It divides eligible requests by difficulty, consequence and service need, then assigns the least costly path that clears a predeclared quality, risk, coverage and latency contract. The optimisation target is constrained: reduce decision-relevant cost and delay for verified outcomes without silently excluding difficult work or exporting burden to reviewers.

This paper assumes that an operating ledger can already distinguish attempts, accepted results and verified outcomes. It goes deeper into how live marginal and step-cost architecture changes that ledger. It does not repeat Paper 023's lifecycle total-cost boundary, although engineering and control overhead must be amortised when judging whether route complexity pays.

The original construct is an inference route contract. It records population slices, constraints, context and cache policy, deterministic and model stages, tool and verifier topology, routing and abstention, test-time compute, retry and fallback, human review, stage meters, a terminal outcome receipt, routing evaluation, drift triggers, replacement test, owner and expiry.

01 · Route contractFix the population and constraints before comparing prices

The eligible population defines which requests may enter the route and what counts as coverage. Slice it using observable pre-decision features that matter to economics: difficulty, consequence, context length, freshness need, tool requirement, language, service tier and reversibility. A slice is useful only if it can be identified reliably at routing time.

Each slice receives a quality and risk threshold, coverage minimum and latency objective. Quality means the final accepted and verified outcome appropriate to the workflow, not a benchmark score imported from another population. Risk includes harmful escapes, rights breaches and recovery. Coverage prevents a cheap route from appearing efficient by dropping cases it cannot handle.

The contract then specifies context assembly, cache scope and invalidation; deterministic checks; model tiers; tools; candidate count; verifier; routing and abstention; retry and fallback; human review; and failure recovery. For every stage it records the meter, error state, latency contribution and owner. The terminal receipt links these stage events to an accepted, rejected, escalated or failed outcome.

Routing research shows that cascades and preference-based routers can change cost-quality trade-offs in evaluated settings.12 Their reported results remain tied to particular tasks, models and evaluation choices. An enterprise route needs in-domain receipts before it claims the same frontier.

The strong rival: simplicity can be economically superior

Routers, cascades, caches and verifiers add engineering, monitoring, evaluation and incident surfaces. At modest volume or with a homogeneous population, one capable model may be cheaper and safer overall even when its unit call price is higher. Route complexity must earn its keep.

Progress begins with a simple reference path and terminal receipts. Add a branch only when observed slices show material dispersion in cost, quality or latency and when volume can amortise the added control burden. If route labels are weak, misrouting harm is hard to detect, model behaviour changes rapidly or traffic cannot support evaluation, a clever optimiser can destroy the evidence needed to operate it.

Hidden technical debt grows through system dependencies and interacting changes.7 Production readiness still requires tests, monitoring and ownership when a route is economically motivated.8 Simplicity is not a lack of sophistication; it is a candidate architecture on the same constrained frontier.

02 · Constrained frontierEliminate routes that are cheaper only because they fail differently

A route is dominated when another route delivers at least as much verified quality and coverage with no greater decision-relevant cost or latency, and improves at least one of those dimensions. A route outside a hard risk, coverage or service boundary is infeasible, not cheap.

Cost includes call, tool, cache, verifier, retry, fallback, review, recovery and amortised route overhead. Latency includes the tail relevant to service, not just median model response. Quality and coverage are measured on the eligible in-domain population with slice results visible. A frontier drawn from average benchmarks can conceal hard-case failures and selection.

The token-price decoy occurs when a compact model sits below a larger model on call price but above it on total cost per verified outcome. Retries, tool misuse, human review or undetected escapes reverse the ranking. The ledger reports the reversal rather than hiding it inside a blended average.

03 · Slice before routingGive easy, uncertain and consequential requests different paths

A route selector should use signals available before the decision outcome. Stable structured cases may admit deterministic validation or a compact model. Ambiguous context, unfamiliar combinations or high consequence may justify a stronger model, more candidates, a specialist tool or human review. Some requests should abstain immediately because the route lacks lawful data, validated scope or safe fallback.

Difficulty labels are hypotheses until verified against outcomes. A router trained on preference data or proxy confidence may misclassify exactly the cases that matter. Measure routing confusion by slice: easy sent deep, hard sent cheap, eligible sent to abstention and out-of-scope admitted. Weight these errors by consequence and recovery, not only by classification accuracy.

Selection effects can counterfeit savings. If the compact path silently rejects long or unusual cases, its average quality may rise and cost may fall while coverage worsens. The terminal receipt keeps every eligible request in the denominator and assigns a state: verified, rejected by policy, safely abstained, escalated, recovered or escaped.

A measured cascade begins with an inexpensive stage only when that stage can identify completion or escalation reliably. Deterministic checks can validate syntax, rights, mandatory fields or known rules. A compact model can handle a stable semantic slice. A stronger model or human receives cases that cross a consequence or uncertainty boundary. Overhead and error at each hand-off remain in the route cost.

Allocate deeper compute selectively

Test-time compute can create additional candidates, search, refinement or verification. Research shows that its effectiveness varies by problem difficulty and method.3 The economic question is therefore not whether more compute helps on average. It is which slice benefits enough to justify extra spend and latency while still meeting coverage and service.

A candidate budget belongs in the route contract. Begin with the smallest budget that clears the slice threshold, then measure marginal outcome gain, verifier reliability and tail latency. Stop when additional candidates are dominated or when the selection mechanism cannot identify the better output. Applying the maximum budget to routine requests can spend heavily for negligible terminal improvement.

04 · Topology and servingSerial and parallel designs trade spend, recovery and tail latency

A serial route pays for later stages only when earlier stages escalate. It can save model spend on easy cases but accumulates latency across hand-offs, retries and tool waits. A parallel route pays for branches at once and may reduce elapsed time or enable comparison, but it spends on candidates that are discarded. The right topology depends on slice probability, branch cost, correlation of failure and the service tail.

The cascade-overhead control asks whether the router, verifier and observability cost more than they save. On a small homogeneous population, nearly every request may follow the same branch, making the selector redundant. Engineering and control overhead divided by low volume can dominate call savings. A simple capable route may also be easier to qualify and recover.

Serving mechanisms change the feasible frontier. PagedAttention targets memory management and throughput in particular serving conditions.4 Scheduling and batching shape throughput and latency together,5 while speculative decoding changes latency and compute under stated assumptions.6 None supplies a universal saving. Workload, hardware, model and service evidence determine whether the mechanism matters here.

Track first-response latency, terminal latency and tail latency separately. A quick first call followed by slow repair can look responsive while the verified outcome misses service. Parallelism can improve the median yet worsen congestion and the tail at high concurrency. The route receipt links topology to the actual service outcome.

05 · Context and cacheA high hit rate can be an economic failure

Caching can avoid repeated context assembly or inference, but a hit is valuable only when the content remains fresh, authorised and scoped to the request. The cache key must preserve tenant, jurisdiction, policy, model and relevant version boundaries. Invalidation follows the information obligation, not a convenient time-to-live alone.

The cache-contamination control tests whether high hits coincide with stale evidence, cross-scope retrieval, missed policy change, lower verified outcomes or incidents. A cheap cached response that triggers remediation or rights harm is not a saving. Track prevented computation, validation burden, invalidation misses, recovery and the outcome distribution of hits versus misses.

Long prompts and retrieved context can increase call cost and distract the model. Context assembly should select necessary evidence, preserve provenance and identify freshness. Cache policy and retrieval policy therefore belong in the same contract. A model upgrade can invalidate previous response caches even when source data have not changed.

06 · Worked counter-caseCompare simple, cascade and selective deep-compute routes

A regulated-finance evidence workflow receives stable routine requests, ambiguous multi-source cases and a smaller consequential escalation slice. Every route must preserve authorised retrieval, evidence lineage, coverage and human appeal. The terminal receipt records accepted evidence completeness, downstream verification, latency, retries and reviewer time.

The simple reference path sends all eligible cases to one capable model with deterministic pre-checks and a fixed verifier. It has clear operations and no routing error. The measured cascade sends routine cases through deterministic validation and a compact model, escalating failed verification or uncertain cases to the reference model. It retains all requests in the coverage denominator.

The selective deep-compute design applies multiple candidates or additional verification only to ambiguous consequential cases. It does not spend deeper compute on the routine slice. A human reviews unresolved high-consequence cases; the route abstains when rights, data or validated scope fail. Each design includes tool, review, recovery and amortised control overhead.

The decision compares call cost, total route cost, cost per verified outcome, quality by slice, coverage and terminal tail latency. If the cascade saves calls but its router and verifier fail on hard cases, it is infeasible. If selective compute improves the consequential slice but breaches latency, it must narrow or change topology. The chosen route is the non-dominated feasible design on in-domain receipts.

07 · Negative controlsProve that the saving survives overhead and selection

Three negative controls prevent the architecture from grading itself. The token-price decoy compares the compact call with the reference route after retries, tools, review and escaped failure. The cascade-overhead control adds engineering, evaluation, monitoring and latency before claiming a saving. The cache-contamination control separates valid hits from stale, cross-scope and route-invalid reuse.

Run controls by slice and preserve the eligible denominator. A cost reduction that disappears when hard cases, abstentions or reviewer work return to the sample is not an efficiency gain. A quality improvement measured only on the branch selected by the new router cannot establish that the policy improved the original population.

Sensitivity tests vary the dominant uncertain drivers: difficult-case share, retry and review, call price, candidate budget, valid cache hits, router error, verification coverage, overhead volume and tail latency. The route choice should state where it reverses. If a narrow assumption decides the architecture, buy evidence on that assumption before scaling complexity.

Build receipts that survive route changes

A useful receipt begins before the first model call. It records eligibility, slice features, policy and route version, service tier, cache decision and the information available to the selector. Every later stage appends start and end time, meter, output status, validation result and reason for transition. This event chain allows finance and engineering to reconstruct why a request became expensive rather than merely observing the final invoice.

The terminal state is categorical and complete. A request can be verified, safely abstained, rejected by policy, escalated, recovered after failure, awaiting a delayed label or escaped. Intermediate generation is not an outcome. If the workflow verifies results only after downstream use, the receipt remains open and the economic view reports mature and pending cohorts separately.

Receipt identity must persist across retries and branches. Otherwise, three model calls look like three successful requests and parallel candidates inflate volume. The root request identifier links stage calls, tool operations, human review and incident recovery. The system also records the discarded candidates so parallel spend cannot disappear from the chosen output's lineage.

Comparison requires a policy version, not only a model name. Changing a cache, prompt, router threshold, verifier or fallback can alter coverage and outcome even when the primary model is unchanged. A phased evaluation therefore compares complete route policies on equivalent slices. Vendor benchmarks and model leaderboards may motivate a candidate, but they cannot replace this operating comparison.

Measure the tail as an economic event

Latency becomes costly when it changes the service outcome. Record queue time, context assembly, first call, tools, verification, retry, human wait and recovery separately. Concurrent load can move a route from inside to outside the service boundary without changing its nominal call price. Test ordinary, peak and failure conditions, then attach the relevant capacity or reservation cost.

Parallel work can reduce elapsed time for an individual request while increasing contention for the population. Serial escalation can conserve capacity while creating a long tail for the hard slice. Report both the request-level and system-level effects. A design that meets an isolated latency test but worsens the production queue is not on the feasible frontier.

Keep human operations inside the route

Human review is neither free verification nor generic oversight. Specify the cases, skills, evidence, decision right, expected handling time, availability and rework. Review burden often depends on error complexity rather than request count. A compact model that sends many confusing cases to specialists can transfer cost and delay even when the review rate appears modest.

Measure whether reviewers can detect the failures the automated verifier misses. If not, the route has a residual escape rather than a priced control. Retain abstention and escalation capacity for consequential uncertainty. Where reviewer decisions themselves require outcome evaluation, the terminal receipt follows the combined policy rather than assigning unquestioned correctness to the human.

Separate priced recovery from unpriced harm

Operational recovery can often be measured: repeat processing, incident investigation, customer contact, rollback, requalification and backlog clearance. Include those observed burdens. Severe or rights-related harm may not support a defensible expected currency value. Keep it as a constraint or separate scenario account rather than inserting a speculative penalty that makes the arithmetic look complete.

This distinction improves rather than weakens the economic case. Decision-makers can see that a route saves operating cost only within a bounded risk envelope. If the envelope fails, the optimisation stops. A cheap route does not receive permission to trade an unpriced boundary for a favourable average.

Design for replacement and simplification

The contract defines a reference evaluation set, route-independent outcome receipt and interface tests. When a model, provider or serving mechanism changes, the team can rerun the same slice and terminal contract. This prevents a vendor tier name or nominal context window from becoming the economic specification.

Replacement testing also reveals when architecture can shrink. A newer single model may absorb a cascade, a deterministic check may replace a model stage, or an improved source system may remove retrieval work. The owner compares the new simple route with the current frontier after migration and requalification overhead. Past investment in complexity is sunk; it does not justify keeping dominated branches.

Ownership follows the terminal contract. Product owns the eligible population and service consequence; engineering owns topology, meters and recovery; evaluation owns slice evidence and verifier limits; risk owns hard boundaries and safe failure; finance owns reconciliation of priced stages and overhead. One accountable route owner resolves conflicts and sets expiry. Without that integration, each team can report a local success while the combined route becomes slower, costlier or less safe. The receipt makes the disagreement inspectable and gives the operating committee a concrete decision: retain, reroute, narrow, simplify or abstain. The owner records why every remaining branch still earns its burden.

08 · Decision instrumentInference Route Economics Lab

The lab uses synthetic user-entered prices. It is a decision aid, not a vendor quotation or benchmark forecast. Describe a candidate route and its in-domain receipt. The calculation separates call cost, total route cost and cost per verified outcome, then checks whether complexity fits the population and constraints.

The numeric receipt is deliberately incomplete where evidence is weak. Escaped failure and incident severity should not be hidden in an invented expected value. If material risk is hard to detect, the route fails or abstains until a safe evaluation and fallback exist. Sensitivity identifies the input most likely to change the recommendation.

09 · Operating cadenceObserve, reroute, validate, reconcile and simplify

Start with a reference route, in-domain slice labels and terminal receipts. Establish the quality, risk, coverage and tail-latency contract before changing architecture. Record call and stage meters with the same route version used to evaluate outcomes.

During a shadow or phased comparison, inspect routing confusion, retries, verifier failures, review, abstention and escapes by slice. Compare the full eligible population under each policy. Include overhead at the volume expected during the decision period, not an imagined mature scale.

When a branch proves non-dominated, introduce it behind an explicit routing and rollback rule. Monitor population mix, cache validity, provider or model change, tool behaviour, tail congestion and reviewer load. A drift trigger returns the affected slice to the reference path until the route is requalified.

Reconcile call spend, total route cost and verified-outcome cost at different cadences. FinOps guidance emphasises granular and volatile AI spend alongside allocation, forecasting and value-aligned unit economics.9 Cost telemetry becomes decision evidence only when it joins terminal outcomes and service.

Risk and quality remain constraints through change. NIST connects context, metrics, monitoring, response and safe failure,1011 while ISO/IEC 42001 frames roles, objectives, monitoring and improvement.12 Replace a model or vendor by re-running the route contract, not by assuming tier labels carry behaviour.

Simplify when dispersion disappears, volume falls, overhead rises or a single model dominates the cascade on terminal receipts. Removing a router or cache can be an optimisation. The economic cadence is not perpetual elaboration; it keeps the smallest route that satisfies the current outcome contract.

10 · ConclusionRoute economics begins where token arithmetic ends

Inference is an operating route through context, cache, models, tools, candidates, verifiers, retries, fallbacks and people. The cheapest component call can sit inside the most expensive outcome route. A stronger call can be economical when it prevents work and failure elsewhere.

The disciplined design slices the population, declares constraints, measures terminal receipts and compares feasible routes on total cost, verified outcomes, coverage and tail latency. Deterministic paths, compact models, cascades and deeper compute are levers, not goals. Their engineering and control surface must be amortised and their selection effects exposed.

The result is a constrained frontier that can change as demand, models, prices, tools and risks change. By keeping the outcome contract stable and the architecture replaceable, leaders can reroute and simplify without confusing a lower token bill with better economics.

Evidence spineReferences

1. Chen, L., Zaharia, M. and Zou, J. (2024), ‘FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance’, Transactions on Machine Learning Research. arXiv:2305.05176.
2. Ong, I. et al. (2025), ‘RouteLLM: Learning to Route LLMs with Preference Data’, ICLR 2025. arXiv:2406.18665.
3. Snell, C. et al. (2025), ‘Scaling LLM Test-Time Compute Optimally Can Be More Effective than Scaling Parameters for Reasoning’, ICLR 2025. arXiv:2408.03314.
4. Kwon, W. et al. (2023), ‘Efficient Memory Management for Large Language Model Serving with PagedAttention’, Proceedings of SOSP 2023. doi:10.1145/3600006.3613165.
5. Agrawal, A. et al. (2024), ‘Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve’. arXiv:2403.02310.
6. Leviathan, Y., Kalman, M. and Matias, Y. (2023), ‘Fast Inference from Transformers via Speculative Decoding’, ICML 2023. arXiv:2211.17192.
7. Sculley, D. et al. (2015), ‘Hidden Technical Debt in Machine Learning Systems’, Google Research. Google Research publication.
8. Breck, E. et al. (2017), ‘The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction’, Google Research. Google Research publication.
9. FinOps Foundation, ‘FinOps for AI’. FinOps technology category.
10. National Institute of Standards and Technology (2023), ‘AI Risk Management Framework: Core’. NIST AI RMF Core.
11. Tabassi, E. (2023), Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1. doi:10.6028/NIST.AI.100-1.
12. International Organization for Standardization (2023), ‘ISO/IEC 42001:2023, Information technology, Artificial intelligence, Management system’. ISO/IEC 42001 overview.