Model Selection and Routing Without Benchmark Theatre

A model should enter a route only when it wins on the real task under hard sovereignty and latency constraints, and a router should ship only when its conditional advantage survives routing error, overhead and the best fixed admissible baseline.

TLDR

A public benchmark rank is only a prior. Select models on the losses they produce for your own task distribution, under the constraints that make a route legally and operationally admissible.
Routing creates value only when models cross over by request type and observable request features predict those crossovers well enough to capture the oracle advantage.
Evaluate every candidate on the same requests, compare against the best fixed admissible policy, and include router latency, cost, failures and constraint violations in one decision loss.
A single locally validated model is the stronger design when differences are small, the useful signal is hidden, labels arrive late, or the router adds another unsafe failure point.
Release a router only when paired evidence shows positive net value, hard constraints remain unbroken, and a monitored fixed-policy fallback can replace it immediately.

Five envelopes and one winning model

A selection panel has three candidate language models and one clean slide. The slide gives each model a public benchmark average, a list price and a median response time. Model Atlas is first on the benchmark, so the room begins discussing volume discounts. Then five sealed request envelopes arrive from the intended service.

The first contains a public document that needs a short summary in 450 milliseconds. The second contains identifying information and a structured extraction that must remain on premises. The third asks for policy reasoning over material approved only for a particular region. The fourth is a code repair whose result will be executed only after tests pass. The fifth is a rare, high-consequence case for which a fluent wrong answer is far more costly than a slow answer.

Nothing about Atlas's public score tells the panel whether it is eligible for the second envelope, whether its extra quality matters on the first, whether its queueing tail violates the third, or whether the fourth should use a cheaper model because a deterministic verifier catches mistakes. A single average has combined requests whose losses, permissions and recovery paths are different.

The panel can still buy Atlas. It might even route many requests there. What it cannot justify is the inference from highest general score to best operating policy. That inference is benchmark theatre: the substitution of visible model prestige for evidence about the decision the system must actually make.

The core answer is simple. Model selection is a constrained task-level policy problem, not a model beauty contest. Public evaluations help form a candidate set. Local paired evidence determines which models belong in the set, which requests each may receive, and whether a router adds value over one fixed admissible model.

There is a second trap. Suppose the three models genuinely have complementary strengths. A hindsight oracle that sees all their outputs could choose the best answer on every request. That does not prove a deployable router can do so. The router acts before the answer exists. It sees only request features, current system state and its own estimates. If those signals do not predict the crossovers, the oracle advantage is inaccessible.

Figure 1. A leaderboard cannot see the request envelope. The benchmark podium is illustrative. The request classes are a worked scenario, not a deployment claim. The decision changes with task success, tail latency, consequence and sovereignty, none of which is recoverable from one aggregate rank.

Part I

The leaderboard loses the task

A benchmark result is an observation about a model, prompts, data, scoring method and execution configuration. Change any of those and the result may change. This is not an argument against benchmarks. It is an argument for reading them as measurements with a declared scope.

Broad evaluation efforts such as HELM made this scope problem visible by evaluating across scenarios and metrics rather than publishing one undifferentiated notion of model quality. RouterBench then assembled more than 405,000 model outcomes to support systematic router comparisons. Both are useful precisely because they expose variation. Neither can know the loss function, entitlements, workload mixture or recovery design of a future service.

The first act of selection is therefore to name the decision unit. “Choose an LLM” is too vague. “Choose a policy for each customer-email extraction request after redaction, with field-level correctness, 700-millisecond p95 latency and no external data transfer” is testable. The unit might be one response, one tool call, one completed case or one verified business outcome. It should be the smallest unit whose success matters to the user and whose cost can be attributed.

Next define loss. Accuracy alone is rarely enough. A useful abstraction is:

L(x,m) = wq·lq(x,m) + wc·c(x,m) + wl·t(x,m) + wf·f(x,m) + I[m ∉ A(x)]

x is a request and m is a candidate model. lq is task-quality loss, c is cost, t is latency or deadline loss, and f is failure or recovery loss. The weights express the declared consequence of each term.

A(x) is the set of models admissible for that request. The indicator I is zero for an eligible route and effectively infinite for a forbidden one. A prohibited transfer is not a small quality penalty that a better answer can compensate.

The terms should be operational. Quality might mean exact extraction, unit tests passed, preference under a blinded rubric, or a human review outcome. Latency should include queueing and retries, not a vendor's median. Cost should include input and output tokens, router inference, tool calls, failed attempts, review and recovery. Failure should distinguish a harmless abstention from an unobserved wrong action.

The model with the highest success rate can still have the highest decision loss. That is not a paradox. It means the additional successes occurred where they were worth less than the added cost, delay or exposure, while the remaining failures occurred where they were expensive.

Worked example: two tasks, two models

Consider two synthetic models. Local is quick and cheap. Regional is slower and more costly but substantially stronger on policy reasoning. Their outcomes below are illustrative and measured over the same held-out requests.

Paired task-level results for two synthetic models
Task	Model	Success	Mean cost units	p95 latency
Short public summary	Local	97%	1.0	180 ms
Short public summary	Regional	95%	4.0	510 ms
Policy reasoning	Local	72%	1.2	210 ms
Policy reasoning	Regional	93%	4.5	590 ms

If summaries are nearly all the traffic and errors are cheap, Local may be the best fixed policy. If policy reasoning becomes half the workload, Regional may win despite its price. A rule that sends summaries to Local and reasoning to Regional reaches a 95 per cent weighted success rate in a balanced mix, but only because the task label predicts the crossover. Remove that label and the route cannot recover the result.

This small example also exposes a common comparison error. Evaluating Local on one dataset and Regional on another does not estimate the difference between them. Paired evidence does. Give every candidate the same requests, prompts, tools, context and scoring protocol. Then each request becomes its own control, reducing the noise created by workload composition.

Aggregate preference is not task success

Human preference arenas provide valuable evidence about how people compare open-ended answers under a particular interaction format. Chatbot Arena, for example, uses pairwise human votes to estimate model rankings. A service outcome can demand something else: a correct field, a citation that supports a claim, a tool effect, a safe refusal, or a verified state change.

Arena preference can enter the source pack as a general prior. It should not silently become the production target. The same caution applies to an LLM judge. A judge may make evaluation scalable, but its position bias, style preference and correlated errors become part of the measuring instrument. Paper 80 in this canon treats that instrument directly. Here the operational rule is enough: use deterministic checks where possible, human anchors where necessary, and judge disagreement as evidence of uncertainty rather than truth.

CheckList showed why aggregate held-out accuracy can conceal specific behavioural failures. Model cards likewise ask authors to state intended use, evaluation conditions and subgroup performance. The selection analogue is a task-by-condition matrix. It should contain easy cases, hard cases, safety cases, malformed inputs, relevant shifts and requests on which a cheaper model ought to be sufficient. A router evaluated only on hard benchmark items can look efficient while sending every easy mathematics or coding request to its strongest model.

Part II

Constraint first, score second

The redacted twin thought experiment varies one feature. Two requests contain the same question and require the same answer. In one, the names and account identifiers have been replaced with synthetic placeholders. In the other, the original identifiers remain. The semantic task, estimated difficulty and expected answer quality are unchanged. Only the data classification changes.

A score-first router may send both to the same high-performing external model. A constraint-first system computes the eligible set before comparing model utility. The unredacted request may permit only an on-premises model. The redacted request may permit regional or external processing, subject to the organisation's rules. Sovereignty changes feasibility before intelligence enters the comparison.

This is a structural point, not a claim that every personal-data request must stay on premises. Applicable law, contracts, transfer mechanisms and organisational policy determine the boundary. The European Union's GDPR Article 44 governs transfers of personal data to third countries, while supervisory guidance such as the UK ICO's international transfer guidance explains the transfer regime in its jurisdiction. Architecture must translate the applicable obligations into explicit route constraints, with legal and privacy owners validating that translation.

Figure 2. Capability-cost-sovereignty frontier. Required signature figure. Positions are synthetic and illustrative, not provider measurements or prices. Cost is in scenario units per request; success is a paired task outcome. Sovereignty is represented as a filter on the candidate set rather than a compensable score.

The admissibility compiler

In implementation, the eligibility test should not be a prompt asking a model whether a route is permitted. It is a deterministic compilation of request and system facts. Inputs may include data class, user and workload identity, purpose, geography, contractual retention terms, modality, required context length, approved tools, current model version and service health. The output is a typed candidate set or an explicit refusal.

Hard constraints deserve their own test suite. Create requests at every boundary: a document before and after redaction, a user before and after entitlement expiry, a region before and after a policy change, and a model endpoint before and after a retention setting changes. The expected route should change for the intended reason only.

Figure 3. The redacted twin. This is a thought experiment. The task wording and expected answer are held constant while the data classification changes. The exact eligible sets are illustrative; an organisation must derive its own from applicable obligations and approved controls.

Part III

The argument in one passChart · flow

flowchart LR
  A["Five envelopes and one winning<br/>model"] --> B["The leaderboard loses the task"]
  B --> C["Constraint first, score second"]
  C --> D["Routing requires predictable<br/>crossovers"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#7B4CD8,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

A structural route through Model Selection and Routing Without Benchmark Theatre, from Five envelopes and one winning model to The route earns its place. Reading use: follow the relation, then test it against the surrounding evidence and limits.

Routing requires predictable crossovers

Once hard constraints have filtered the candidates, routing asks a narrower question: can a request-level policy choose among the survivors better than one fixed policy?

Three conditions are necessary. First, models must cross over. If one admissible model has lower loss for every meaningful cohort, use it. Second, the request must reveal a signal correlated with the crossover: task type, language, modality, consequence, document structure, observed difficulty, available verifier or another pre-inference feature. Third, the value captured by the signal must exceed misroutes and overhead.

Published work demonstrates both possibility and danger. RouteLLM learned strong-versus-weak routing from preference data and reported more than twofold cost reductions in some benchmark settings without reducing measured response quality. Hybrid LLM predicted whether a request required a larger model and reported substantial reductions in large-model calls in its experiments. FrugalGPT showed how cascades can trade cost against quality. These results establish that conditional selection can work in bounded settings.

They do not establish that any router, workload or model pool will work. LLMRouterBench re-evaluated ten representative baselines across more than 400,000 instances, 21 datasets and 33 models. It confirmed model complementarity, yet found many methods clustered together and several unable to reliably beat a simple best-single baseline. The authors also reported a persistent gap to the hindsight oracle, driven largely by failures to identify the right model.

A separate robustness study, How Robust Are Router-LLMs?, found routers that relied on category shortcuts: simple mathematics and coding requests were often sent to the strongest model, while harmful prompts could be sent to weaker models. The lesson is not that learned routers are useless. It is that the router must be evaluated as a fallible model whose errors have task-specific consequences.

The blind usher

Imagine two orchestras behind identical curtains. Orchestra A plays Bach better; Orchestra B plays Ellington better. An usher must direct each audience member before the programme is announced. A hindsight evaluator hears both performances and shows a large oracle gain from perfect assignment. The usher, however, sees identical tickets and has no predictive clue. Its best policy is simply to send everyone to the orchestra with the lower average disappointment.

This thought experiment holds model complementarity constant and removes router information. It shows why “different models win different examples” is insufficient. Complementarity creates possible headroom. Observability determines whether a route can capture it.

Let the best fixed admissible policy have expected loss Rfixed. Let a hindsight oracle choose the lowest-loss eligible model after seeing all outcomes, with loss Roracle. The maximum available headroom is:

H = Rfixed − Roracle

H is not expected router gain. It is an upper bound created by model crossovers on the observed workload.

M = Rrouter,models − Roracle

M is router miss cost: the loss created because the deployable policy cannot reproduce hindsight choices.

Net route value = H − M − O

O is route overhead, including classifier cost, latency, operational complexity and additional failure exposure. Lower decision loss is better, so a route earns release only when the net term is positive with appropriate uncertainty.

Oracle headroom is a finite budget. A complicated router cannot recover more than that budget, and every error or extra stage spends some of it. If the best fixed model and the oracle differ by only 0.2 loss units, a 0.3-unit route overhead makes the project uneconomic before training begins.

Figure 4. Oracle headroom as a finite budget. This relationship is a practitioner synthesis. Bar lengths are illustrative. The diagnostic separates model complementarity (H) from router quality (M) and route overhead (O), so an impressive oracle result cannot masquerade as a deployable gain.

Part IV

Turn the portfolio into an experiment

A route should begin as a replay, not a traffic switch. The experiment preserves the real workload shape while preventing model choice from changing what gets measured.

1. Freeze the task contract

Define the request unit, eligible data, context builder, prompt version, tools, output schema, verifier and outcome window. For a coding task, “success” may require compilation, tests and a bounded diff. For extraction, it may require field accuracy and abstention on absent values. For advice drafting, it may require evidence support and human disposition. The contract should state what happens when no model qualifies.

2. Compile hard eligibility

Create a request-by-model matrix before scoring quality. Mark each cell eligible, ineligible or conditionally eligible with the responsible control and evidence. Conditions might include redaction, a regional endpoint, an approved retention mode, a particular model version, a maximum context class or a deterministic postcondition check. Test that matrix independently from the router.

3. Use public evidence to curate, not crown

Public benchmarks, model cards, release notes and prices help remove obviously unsuitable candidates. They reduce search cost. They should also inform challenge cases: multilingual names, long contexts, function calls, structured output and known safety boundaries. Do not add models merely to make the portfolio look diverse. LLMRouterBench reports diminishing returns from larger ensembles compared with careful curation. Every extra candidate expands evaluation, capacity, security and change-management work.

4. Run paired outcomes

Send every replay request to every eligible candidate in shadow. Preserve model version, endpoint, parameters, system prompt, context and tool configuration. Record raw output separately from evaluation. Use request IDs to align outcomes. When stochasticity matters, repeat a declared sample of requests rather than pretending one draw is a stable model property.

Scoring should match the task. Exact validators, unit tests, calculators and database readbacks are stronger than stylistic judgement. Human review remains necessary where the target is genuinely evaluative. A model judge can assist, but it should be calibrated against blinded human anchors and tested for model identity, response order and verbosity effects.

The Measure function of the NIST AI Risk Management Framework similarly treats measurement as contextual: methods should be fit for purpose, limits should be documented, and results should support course correction. A router release gate turns that principle into a concrete comparison among operating policies.

5. Establish four baselines

The minimum comparison set is: the cheapest fixed admissible policy, the strongest fixed admissible policy, the best fixed admissible policy by total loss, and a simple rule route. Add the hindsight oracle as an upper bound, not a deployable competitor. If a learned router cannot beat the best fixed policy and a transparent rule, it has not earned its complexity.

6. Split by time and by failure

A random split can leak repeated templates and stable task mixtures. Keep a later time window for model and workload shift. Hold out organisations, languages, document families or repositories when those boundaries matter. Add deliberate failure cases: unavailable endpoints, stale model metadata, a changed data class, missing task features, router timeout, malformed outputs and a queue spike. A route that wins only when every signal is present is not yet an operating policy.

7. Compare paired loss with uncertainty

For each request, subtract router loss from best-fixed loss. Bootstrap or otherwise estimate uncertainty over those paired differences using a resampling method suitable for the dependence structure. Report the mean difference, interval, constraint violations and cohort results. Do not hide a catastrophic high-consequence slice behind a positive average.

Figure 5. Paired replay before policy learning. This is a practitioner experiment design. The fan-out is a shadow evaluation over eligible models, not a recommendation to invoke all models in live service. Pairing controls workload composition and makes the best-fixed comparison credible.

A decision table, not one composite score

The loss function supports optimisation, but release review should retain its components. A route that saves cost by increasing severe failures should not pass because a chosen weight made the arithmetic favourable. Publish a compact scorecard with denominators and thresholds.

Minimum scorecard for a model route
Evidence	Question answered	Required comparison	Release concern
Task outcome	Did the completed task meet its contract?	Paired success and severity by cohort	Average hides high-consequence failure
Admissibility	Was the chosen model permitted for this request?	Zero violations, including fallback	A weighted score cannot excuse a breach
Latency	Did the full path meet its deadline?	p50, p95, p99 and timeout rate	Median conceals queues and retries
Cost	What did the verified outcome consume?	Tokens, route, tools, retries and review	Per-call price omits failure and recovery
Headroom	Is there enough model crossover to exploit?	Best fixed versus hindsight oracle	Small headroom cannot fund complexity
Router capture	Can visible signals recover the headroom?	Router versus oracle and best fixed	Category shortcuts can look intelligent
Shift	Does the policy survive changed traffic?	Future window and held-out groups	Model or task mix may invalidate the route

Figure 6. When routing helps. The surface is explicitly illustrative, not measured. Routing becomes plausible as model crossovers and pre-inference signal quality rise; latency, operating cost, label noise and consequence move the release boundary against it.

A production-shaped replay

The companion artefact builds a synthetic portfolio with three models and 4,000 requests. The names, prices, latencies, quality probabilities and task mix are invented. The point is the evaluation structure.

The workload contains public summaries, on-premises-only extraction, regional-only policy reasoning and public code repair. Each request has a consequence class, deadline, latent difficulty and noisy observed difficulty. All candidate policies receive the same materialised model outcomes. The evaluator enforces sovereignty before selection, charges failures according to consequence, includes deadline loss, and compares fixed policies, a task-aware router, a hindsight oracle and a shuffled-signal negative control.

Synthetic paired replay over 4,000 requests
Policy	Success	Severe failures	Mean cost	p95 latency	Mean loss
Fixed local-small	73.22%	503	0.95	175 ms	15.18
Fixed regional-mid	84.92%	295	4.40	596 ms	12.62
Fixed external-frontier	88.65%	295	8.65	1,549 ms	18.30
Task router, total	85.08%	295	4.50	1,210 ms	12.27
Hindsight oracle	93.38%	194	1.96	508 ms	6.70
Shuffled-signal router	84.08%	311	4.52	1,127 ms	13.09

The fixed external-first policy has the highest non-oracle success rate, yet the worst decision loss. It pays more, misses latency targets more often and still falls back for restricted tasks. Regional-mid is the best fixed admissible baseline at 12.62 loss units. The task router lowers loss to 12.27, an improvement of 0.344 units after its own cost and latency.

The bootstrap interval for the paired improvement is 0.158 to 0.536 units in this seeded run. Oracle headroom is 5.920 units; router misses consume 5.410 and overhead consumes 0.166, leaving 0.344. This is a narrow but positive case. The proper interpretation is not “routing is always better”. It is “this policy captured a small, measurable portion of available headroom under these assumptions”.

The negative control scrambles task and difficulty signals while preserving the model pool and request outcomes. It loses 0.478 units against the best fixed baseline, with a 95 per cent interval from minus 0.734 to minus 0.208. The failure matters because it shows the positive route depends on informative request features. Merely distributing traffic among models does not create value.

A negative control protects against portfolio theatre. Other useful controls include a random router with the same model proportions, a task-only rule, a difficulty-only rule, a router trained on permuted labels and a constant route with identical overhead. Each isolates a possible explanation for the measured gain.

Figure 7. Production-shaped synthetic replay. All values come from the seeded companion script, not a live service. Denominator: 4,000 requests. Route shares are 53.5 per cent local, 29.6 per cent regional and 16.9 per cent external. Decision loss combines synthetic cost, latency and consequence-weighted failure.

Part V

How the claim changes under pressureChart · state

stateDiagram-v2
  [*] --> Claim
  state "Worked example, two tasks, two<br/>models" as Claim
  state "Aggregate preference is not<br/>task success" as Test
  state "The admissibility compiler" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

The revision loop begins with Worked example, two tasks, two models and keeps counterexamples inside the reasoning path. Reading use: follow the relation, then test it against the surrounding evidence and limits.

Keep the route honest in operation

A replay establishes an initial policy. Operation can invalidate it within hours. Model versions change. Prices change. An endpoint slows. The task mix moves. A new redaction rule changes eligibility. A verifier becomes stricter. A route therefore needs evidence, expiry and recovery, not just a classifier endpoint.

Separate the route decision from serving convenience

A model router predicts which candidate should handle the task. A load balancer chooses an instance. A queueing controller may revise the choice because current latency makes the preferred model miss its deadline. These are related but distinct decisions. Combining them can be efficient, but the receipt should preserve why a model was selected, why an instance was chosen and whether a capacity override occurred.

Latency estimates must be fresh. A model that was on the frontier during replay can leave it under load. Use observed end-to-end latency by task and payload band, not a static model label. If a fallback is permitted, evaluate the fallback path, including the time already spent before escalation.

The router itself crosses a trust boundary

A semantic router may need to inspect the request to classify it. Sending a sensitive request to an external routing model before rejecting external generation has already crossed the boundary. Constraint compilation must therefore precede any router that cannot safely see the payload. Safer options include local features, typed metadata supplied by the application, an on-premises classifier, or a coarse deterministic gate followed by a richer router inside the permitted zone.

Privacy-preserving model collaboration is an active research area. PAPILLON, for example, studies collaboration between local and API models while jointly considering response quality and privacy. Its existence reinforces the need to measure both. A local preprocessing stage is not automatically private if its output still reveals sensitive information.

Observe the outcome, not merely the dispatch

The route log should record more than model name. For a consequential request, keep a decision receipt containing the task contract, model and router versions, eligible set, chosen route, predicted loss or confidence, policy reason, timing and cost. Link it to an outcome record produced by the verifier or downstream system. Without that link, a team can optimise routing clicks while remaining blind to completed work.

The control anatomy is broader than “prompt plus model”. Intent defines the task. Identity binds the requester and workload. World state includes current model versions, endpoint health and policy. Context contains the permitted evidence. Reasoning produces a route proposal. Authority constrains eligible models. Action dispatches the request. Evidence records the decision and output. Outcome verifies the task. Release and recovery determine canarying, rollback and incident response.

Do not call every persistent fact memory. The route uses workflow state for the current request, enterprise knowledge for task evidence, world state for endpoint and policy facts, and an evidence ledger for what was selected and observed. Conflating these stores makes freshness and authority difficult to test.

Correct the partial-feedback trap

After release, the system normally observes only the chosen model's outcome. It no longer sees the counterfactual outcomes needed to know whether another model would have been better. Retraining directly on this selective feedback can reinforce early choices.

Maintain a safe exploration or shadow policy where consequence and consent permit it. Sample paired canaries from eligible candidates, preserve a fixed holdout, and continue scoring the best-fixed baseline on current traffic. In high-consequence settings, exploration may be inappropriate; use offline replay and human-approved change windows instead. The method must respect the same authority boundaries as the live route.

Know the failure boundary

Routing ceases to help when model crossovers are too small, the distinguishing signal is unavailable before inference, labels arrive too late, evaluation is unreliable, traffic is too low to estimate cohorts, or model churn invalidates the policy faster than it can be validated. It also ceases to help when routing overhead consumes headroom or when a route introduces an unacceptable privacy, security or availability dependency.

For irreversible or regulated actions, model routing does not grant authority. The selected model may draft, classify or propose. Deterministic controls, accountable approval, typed action contracts and postcondition verification still govern effects. A superior model score cannot replace permission, confirmation or evidence that the intended state changed exactly once.

Model route evidence record

Decision unit: Name the completed task and its verifier, not merely the prompt category.
Hard eligibility: List identity, purpose, data class, geography, retention, modality, tool and version constraints. Require zero violations.
Candidate evidence: Record public priors, model cards and the paired local evaluation that justified inclusion.
Fixed baseline: State the best admissible fixed policy, including deterministic constraint fallback.
Headroom: Report best-fixed loss minus hindsight-oracle loss with task and cohort denominators.
Router capture: Report miss cost, route overhead, net paired improvement and uncertainty.
Negative controls: Include shuffled signals, simple rules and a constant route with comparable overhead.
Failure boundary: Name the slices, shifts and unavailable signals that invalidate the policy.
Operation: Define canary, outcome monitoring, route receipt, expiry, owner and fixed-policy rollback.

This record is a decision instrument, not a universal standard. It makes the claim falsifiable: a reviewer can see what won, against which baseline, under which constraints, for how long, and what evidence would withdraw the route.

Figure 8. Model route evidence record. This decision instrument is practitioner synthesis. A verified task outcome sits inside paired evidence, policy comparison, hard authority and recovery. Performance cannot compensate for a broken outer control.

Executable artefact: paired router evaluator

The standard-library Python programme below reproduces the production-shaped scenario. It generates synthetic requests, materialises the same model outcomes for every policy, enforces hard sovereignty constraints, compares fixed policies and a hindsight oracle, estimates a paired bootstrap interval, and runs a shuffled-signal negative control.

Assumptions. Model names, quality probabilities, costs, latencies, workload shares and loss weights are synthetic. Cost is expressed in scenario decision units, not currency. The run is seeded. Python 3.10 or later is sufficient; no third-party packages are required.

Save the code as paper-78-router-evaluator.py and run:

python paper-78-router-evaluator.py

The positive case should report the task router below the best fixed policy in mean loss, with a positive paired interval. The negative case should report the shuffled router above the fixed baseline, proving that informative request signals caused the gain.

View the complete runnable Python programme

#!/usr/bin/env python3
"""Paired replay evaluator for task-level model selection and routing.

All prices, latencies, model names and task outcomes are synthetic. The script
creates the same requests for every candidate policy, enforces hard sovereignty
constraints, compares against admissible fixed-model baselines, computes an
oracle upper bound, and runs a shuffled-signal negative control.

Python 3.10+; standard library only.
"""

from __future__ import annotations

from dataclasses import dataclass
from math import ceil
import random
from statistics import mean
from typing import Callable, Dict, Iterable, List, Mapping, Sequence, Tuple

SEED = 78078
N_REQUESTS = 4000
BOOTSTRAPS = 1200


@dataclass(frozen=True)
class Model:
    name: str
    zone: str
    base_cost: float
    base_latency_ms: int
    quality_by_task: Mapping[str, float]


@dataclass(frozen=True)
class Request:
    request_id: int
    task: str
    sensitivity: str
    consequence: str
    slo_ms: int
    difficulty: float
    observed_difficulty: float


@dataclass(frozen=True)
class Outcome:
    success: bool
    cost: float
    latency_ms: int


@dataclass
class Evaluation:
    name: str
    losses: List[float]
    successes: int
    severe_failures: int
    costs: List[float]
    latencies: List[int]
    violations: int
    route_counts: Dict[str, int]

    @property
    def mean_loss(self) -> float:
        return mean(self.losses)

    @property
    def success_rate(self) -> float:
        return self.successes / len(self.losses)

    @property
    def average_cost(self) -> float:
        return mean(self.costs)

    @property
    def p95_latency(self) -> int:
        ordered = sorted(self.latencies)
        return ordered[min(len(ordered) - 1, ceil(0.95 * len(ordered)) - 1)]


MODELS: Tuple[Model, ...] = (
    Model(
        "local-small",
        "on_prem",
        base_cost=0.8,
        base_latency_ms=95,
        quality_by_task={
            "public_summary": 0.955,
            "restricted_extraction": 0.935,
            "policy_reasoning": 0.735,
            "code_repair": 0.675,
        },
    ),
    Model(
        "regional-mid",
        "approved_region",
        base_cost=4.5,
        base_latency_ms=330,
        quality_by_task={
            "public_summary": 0.968,
            "restricted_extraction": 0.962,
            "policy_reasoning": 0.912,
            "code_repair": 0.842,
        },
    ),
    Model(
        "external-frontier",
        "global_api",
        base_cost=11.0,
        base_latency_ms=890,
        quality_by_task={
            "public_summary": 0.974,
            "restricted_extraction": 0.979,
            "policy_reasoning": 0.958,
            "code_repair": 0.953,
        },
    ),
)
MODEL_BY_NAME = {model.name: model for model in MODELS}

TASKS: Tuple[Tuple[str, float, str, str, int], ...] = (
    # task, population share, sensitivity, consequence, p95 target
    ("public_summary", 0.38, "public", "low", 450),
    ("restricted_extraction", 0.22, "on_prem_only", "high", 350),
    ("policy_reasoning", 0.24, "regional_only", "high", 900),
    ("code_repair", 0.16, "public", "medium", 1500),
)

ALLOWED_ZONES = {
    "public": {"on_prem", "approved_region", "global_api"},
    "regional_only": {"on_prem", "approved_region"},
    "on_prem_only": {"on_prem"},
}

FAILURE_PENALTY = {"low": 6.0, "medium": 24.0, "high": 95.0}
ROUTER_COST_OVERHEAD = 0.16
ROUTER_LATENCY_OVERHEAD_MS = 14


def weighted_choice(rng: random.Random) -> Tuple[str, str, str, int]:
    draw = rng.random()
    cumulative = 0.0
    for task, share, sensitivity, consequence, slo_ms in TASKS:
        cumulative += share
        if draw <= cumulative:
            return task, sensitivity, consequence, slo_ms
    task, _, sensitivity, consequence, slo_ms = TASKS[-1]
    return task, sensitivity, consequence, slo_ms


def make_requests(n: int, seed: int) -> List[Request]:
    rng = random.Random(seed)
    requests: List[Request] = []
    for request_id in range(n):
        task, sensitivity, consequence, slo_ms = weighted_choice(rng)
        difficulty = min(1.0, max(0.0, rng.betavariate(2.1, 2.6)))
        noise = rng.gauss(0.0, 0.13)
        observed = min(1.0, max(0.0, difficulty + noise))
        requests.append(
            Request(
                request_id=request_id,
                task=task,
                sensitivity=sensitivity,
                consequence=consequence,
                slo_ms=slo_ms,
                difficulty=difficulty,
                observed_difficulty=observed,
            )
        )
    return requests


def model_success_probability(model: Model, request: Request) -> float:
    base = model.quality_by_task[request.task]
    # Difficulty hurts weaker models more. This creates conditional crossovers
    # without making task labels alone a perfect routing signal.
    sensitivity = {
        "local-small": 0.28,
        "regional-mid": 0.16,
        "external-frontier": 0.08,
    }[model.name]
    return min(0.995, max(0.02, base - sensitivity * request.difficulty))


def materialise_outcomes(
    requests: Sequence[Request], seed: int
) -> Dict[Tuple[int, str], Outcome]:
    rng = random.Random(seed)
    outcomes: Dict[Tuple[int, str], Outcome] = {}
    for request in requests:
        length_factor = 0.75 + 0.85 * rng.random()
        for model in MODELS:
            success_p = model_success_probability(model, request)
            success = rng.random() < success_p
            jitter = max(0.55, rng.lognormvariate(0.0, 0.24))
            latency = int(model.base_latency_ms * (0.72 + request.difficulty) * jitter)
            cost = model.base_cost * length_factor * (0.72 + 0.65 * request.difficulty)
            outcomes[(request.request_id, model.name)] = Outcome(
                success=success,
                cost=cost,
                latency_ms=latency,
            )
    return outcomes


def is_eligible(model: Model, request: Request) -> bool:
    return model.zone in ALLOWED_ZONES[request.sensitivity]


def loss_for(
    request: Request,
    model: Model,
    outcome: Outcome,
    *,
    extra_cost: float = 0.0,
    extra_latency_ms: int = 0,
) -> float:
    if not is_eligible(model, request):
        return 10_000.0
    latency = outcome.latency_ms + extra_latency_ms
    lateness = max(0.0, latency - request.slo_ms) / request.slo_ms
    failure = 0.0 if outcome.success else FAILURE_PENALTY[request.consequence]
    # Cost is deliberately in the same synthetic decision-loss units as the
    # stated failure and latency penalties. These are scenario weights, not prices.
    return outcome.cost + extra_cost + failure + 4.0 * lateness


def strongest_eligible(request: Request) -> str:
    for name in ("external-frontier", "regional-mid", "local-small"):
        if is_eligible(MODEL_BY_NAME[name], request):
            return name
    raise RuntimeError("No eligible model")


def primary_with_constraint_fallback(primary: str) -> Callable[[Request], str]:
    def route(request: Request) -> str:
        if is_eligible(MODEL_BY_NAME[primary], request):
            return primary
        return strongest_eligible(request)

    return route


def predicted_success(
    model: Model,
    request: Request,
    signal: float | None = None,
    task_signal: str | None = None,
) -> float:
    observed = request.observed_difficulty if signal is None else signal
    base = model.quality_by_task[request.task if task_signal is None else task_signal]
    sensitivity = {
        "local-small": 0.28,
        "regional-mid": 0.16,
        "external-frontier": 0.08,
    }[model.name]
    return min(0.995, max(0.02, base - sensitivity * observed))


def risk_aware_router(
    request: Request,
    signal: float | None = None,
    task_signal: str | None = None,
) -> str:
    required_success = {"low": 0.76, "medium": 0.88, "high": 0.91}[request.consequence]
    eligible = [model for model in MODELS if is_eligible(model, request)]
    candidates = [
        model
        for model in eligible
        if predicted_success(model, request, signal, task_signal) >= required_success
    ]
    if candidates:
        return min(candidates, key=lambda model: (model.base_cost, model.base_latency_ms)).name
    # When no route is predicted to meet the target, choose the highest predicted
    # success and retain the failure for review rather than pretending cost won.
    return max(
        eligible,
        key=lambda model: (
            predicted_success(model, request, signal, task_signal), -model.base_cost
        ),
    ).name


def oracle_router(
    request: Request, outcomes: Mapping[Tuple[int, str], Outcome]
) -> str:
    eligible = [model for model in MODELS if is_eligible(model, request)]
    return min(
        eligible,
        key=lambda model: loss_for(
            request, model, outcomes[(request.request_id, model.name)]
        ),
    ).name


def evaluate(
    name: str,
    requests: Sequence[Request],
    outcomes: Mapping[Tuple[int, str], Outcome],
    route: Callable[[Request], str],
    *,
    router_overhead: bool = False,
) -> Evaluation:
    losses: List[float] = []
    costs: List[float] = []
    latencies: List[int] = []
    route_counts: Dict[str, int] = {model.name: 0 for model in MODELS}
    successes = severe_failures = violations = 0

    for request in requests:
        model_name = route(request)
        model = MODEL_BY_NAME[model_name]
        outcome = outcomes[(request.request_id, model_name)]
        extra_cost = ROUTER_COST_OVERHEAD if router_overhead else 0.0
        extra_latency = ROUTER_LATENCY_OVERHEAD_MS if router_overhead else 0
        if not is_eligible(model, request):
            violations += 1
        if outcome.success:
            successes += 1
        elif request.consequence == "high":
            severe_failures += 1
        route_counts[model_name] += 1
        costs.append(outcome.cost + extra_cost)
        latencies.append(outcome.latency_ms + extra_latency)
        losses.append(
            loss_for(
                request,
                model,
                outcome,
                extra_cost=extra_cost,
                extra_latency_ms=extra_latency,
            )
        )

    return Evaluation(
        name=name,
        losses=losses,
        successes=successes,
        severe_failures=severe_failures,
        costs=costs,
        latencies=latencies,
        violations=violations,
        route_counts=route_counts,
    )


def paired_bootstrap_ci(
    baseline: Sequence[float], candidate: Sequence[float], seed: int
) -> Tuple[float, float, float]:
    if len(baseline) != len(candidate):
        raise ValueError("Paired samples must have equal length")
    rng = random.Random(seed)
    differences = [b - c for b, c in zip(baseline, candidate)]
    observed = mean(differences)
    n = len(differences)
    samples: List[float] = []
    for _ in range(BOOTSTRAPS):
        samples.append(mean(differences[rng.randrange(n)] for _ in range(n)))
    samples.sort()
    lo = samples[int(0.025 * len(samples))]
    hi = samples[int(0.975 * len(samples))]
    return observed, lo, hi


def pct(value: float) -> str:
    return f"{100.0 * value:6.2f}%"


def route_mix(evaluation: Evaluation) -> str:
    n = len(evaluation.losses)
    return ", ".join(
        f"{name} {100.0 * count / n:.1f}%"
        for name, count in evaluation.route_counts.items()
        if count
    )


def print_table(evaluations: Iterable[Evaluation]) -> None:
    print(
        f"{'policy':34} {'success':>9} {'severe':>8} {'cost':>9} "
        f"{'p95 ms':>8} {'loss':>9} {'viol.':>7}"
    )
    print("-" * 92)
    for result in evaluations:
        print(
            f"{result.name:34} {pct(result.success_rate):>9} "
            f"{result.severe_failures:8d} {result.average_cost:9.2f} "
            f"{result.p95_latency:8d} {result.mean_loss:9.2f} "
            f"{result.violations:7d}"
        )


def main() -> None:
    requests = make_requests(N_REQUESTS, SEED)
    outcomes = materialise_outcomes(requests, SEED + 1)

    fixed = [
        evaluate(
            f"fixed {model.name}",
            requests,
            outcomes,
            primary_with_constraint_fallback(model.name),
        )
        for model in MODELS
    ]
    best_fixed = min(fixed, key=lambda result: result.mean_loss)

    router_no_overhead = evaluate(
        "task router, model loss only",
        requests,
        outcomes,
        risk_aware_router,
    )
    router = evaluate(
        "task router, total",
        requests,
        outcomes,
        risk_aware_router,
        router_overhead=True,
    )
    oracle = evaluate(
        "hindsight oracle",
        requests,
        outcomes,
        lambda request: oracle_router(request, outcomes),
    )

    shuffled_profiles = [
        (request.task, request.observed_difficulty) for request in requests
    ]
    random.Random(SEED + 2).shuffle(shuffled_profiles)
    profile_by_id = {
        request.request_id: shuffled_profiles[index]
        for index, request in enumerate(requests)
    }
    shuffled = evaluate(
        "negative control: shuffled task signal",
        requests,
        outcomes,
        lambda request: risk_aware_router(
            request,
            signal=profile_by_id[request.request_id][1],
            task_signal=profile_by_id[request.request_id][0],
        ),
        router_overhead=True,
    )

    print("Synthetic paired replay: positive case")
    print("All candidate policies see the same 4,000 requests and realised outcomes.\n")
    print_table([*fixed, router, oracle])
    print(f"\nBest admissible fixed baseline: {best_fixed.name}")
    print(f"Router mix: {route_mix(router)}")

    headroom = best_fixed.mean_loss - oracle.mean_loss
    miss_cost = router_no_overhead.mean_loss - oracle.mean_loss
    overhead = router.mean_loss - router_no_overhead.mean_loss
    net = headroom - miss_cost - overhead
    print("\nHeadroom test (lower decision loss is better)")
    print(f"  oracle headroom H     = {headroom:7.3f}")
    print(f"  router miss cost M    = {miss_cost:7.3f}")
    print(f"  routing overhead O    = {overhead:7.3f}")
    print(f"  net H - M - O         = {net:7.3f}")

    observed, lo, hi = paired_bootstrap_ci(
        best_fixed.losses, router.losses, SEED + 3
    )
    verdict = "PASS" if lo > 0 else "FAIL"
    print(
        f"  paired loss improvement = {observed:7.3f} "
        f"(95% bootstrap CI {lo:7.3f} to {hi:7.3f}) -> {verdict}"
    )

    print("\nNegative control: scramble the task and difficulty signals")
    print_table([best_fixed, shuffled])
    observed_n, lo_n, hi_n = paired_bootstrap_ci(
        best_fixed.losses, shuffled.losses, SEED + 4
    )
    verdict_n = "PASS" if lo_n > 0 else "FAIL"
    print(f"Shuffled router mix: {route_mix(shuffled)}")
    print(
        f"  paired loss improvement = {observed_n:7.3f} "
        f"(95% bootstrap CI {lo_n:7.3f} to {hi_n:7.3f}) -> {verdict_n}"
    )
    print(
        "\nInterpretation: ship a router only when the positive case beats the best "
        "fixed admissible policy after overhead, while the negative control shows "
        "that the gain depends on informative task-level signals rather than portfolio theatre."
    )


if __name__ == "__main__":
    main()

Expected seeded output

Synthetic paired replay: positive case
All candidate policies see the same 4,000 requests and realised outcomes.

policy                               success   severe      cost   p95 ms      loss   viol.
--------------------------------------------------------------------------------------------
fixed local-small                     73.22%      503      0.95      175     15.18       0
fixed regional-mid                    84.92%      295      4.40      596     12.62       0
fixed external-frontier               88.65%      295      8.65     1549     18.30       0
task router, total                    85.08%      295      4.50     1210     12.27       0
hindsight oracle                      93.38%      194      1.96      508      6.70       0

Headroom test (lower decision loss is better)
  oracle headroom H     =   5.920
  router miss cost M    =   5.410
  routing overhead O    =   0.166
  net H - M - O         =   0.344
  paired loss improvement = 0.344 (95% bootstrap CI 0.158 to 0.536) -> PASS

Negative control: scramble the task and difficulty signals
  paired loss improvement = -0.478 (95% bootstrap CI -0.734 to -0.208) -> FAIL

Compact glossary

Admissible model: A candidate that satisfies the request's hard identity, purpose, data, geography, tool, version and service constraints.
Best fixed policy: The lowest-loss policy that uses one primary model, plus mandatory deterministic constraint fallback, for the evaluated workload.
Hindsight oracle: An undeployable upper bound that chooses the best eligible model after observing every candidate outcome for each request.
Oracle headroom: The loss difference between the best fixed policy and the hindsight oracle. It measures possible value from model crossovers.
Router miss cost: The portion of oracle headroom lost because pre-inference signals fail to predict which model will produce the lowest task loss.

The evidence conversationChart · sequence

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: Five envelopes and one winning model
  M->>E: Worked example, two tasks, two models
  E-->>M: compare rival explanations
  E->>D: The route earns its place
  D-->>Q: revise scope or act

Question, mechanism, evidence and decision exchange the burdens developed in Model Selection and Routing Without Benchmark Theatre. Reading use: follow the relation, then test it against the surrounding evidence and limits.

The route earns its place

The selection panel should not replace one leaderboard with a larger dashboard. It should replace the model contest with a policy experiment.

Begin with the real task and its consequence. Compile the eligible model set from identity, purpose, data and operating state. Use public benchmarks to find candidates, then run those candidates on the same local requests. Measure completed outcomes, latency, cost, failure and recovery. Establish the best fixed admissible policy. Compute oracle headroom. Only then ask a router to capture it.

The decisive comparison is paired and modest: does the proposed route reduce task-level decision loss against the best fixed admissible policy after its own misses and overhead? A positive average is insufficient if a hard constraint breaks, a severe cohort regresses, or the gain disappears in the next time window. The policy needs a receipt, an expiry and a fixed fallback.

Sometimes the result will be a learned route. Sometimes a transparent rule will win. Often the answer will be one well-chosen model behind deterministic gates. That is not a failure of sophistication. It is the purpose of the method.

Choose the smallest operating policy whose advantage survives contact with the task. The benchmark may nominate models. Evidence decides the route.

References and source notes

Li, H. et al. LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routing. Primary research and benchmark.
Ong, I. et al. RouteLLM: Learning to Route LLMs with Preference Data. Primary research and open implementation.
Hu, Q. J. et al. RouterBench: A Benchmark for Multi-LLM Routing System. Primary research and public outcome dataset.
Ding, D. et al. Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing. Primary research.
Chen, L., Zaharia, M. and Zou, J. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. Primary research.
Kassem, A. M., Schölkopf, B. and Jin, Z. How Robust Are Router-LLMs? Analysis of the Fragility of LLM Routing Capabilities. Primary negative and limitation evidence.
Liang, P. et al. Holistic Evaluation of Language Models. Primary evaluation framework.
Zheng, L. et al. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. Primary evaluation platform.
Ribeiro, M. T. et al. Beyond Accuracy: Behavioral Testing of NLP Models with CheckList. Primary evaluation method.
Mitchell, M. et al. Model Cards for Model Reporting. Primary documentation proposal.
Siyan, L. et al. PAPILLON: Privacy Preservation from Internet-Based and Local Language Model Ensembles. Primary privacy and collaboration research.
National Institute of Standards and Technology. AI Risk Management Framework and Measure function. Official risk and measurement guidance.
European Union. General Data Protection Regulation, Article 44. Official legal text.
UK Information Commissioner's Office. International transfers guidance. Official regulatory guidance.

Evidence boundary. Published results are attributed to their source and remain bounded by their datasets, model pools and metrics. The headroom inequality, admissibility compiler, evidence record, worked scenarios and release rule are practitioner synthesis. All numeric route results in this article are synthetic and reproducible with the companion programme.