Evaluation Is a Measurement System

TLDR

Evaluation is a measurement system that links an abstract capability to observed cases, labels, splits, metrics and a decision. A model score is only the final reading from that chain.
Leakage, weak labels and convenient test sets can produce a precise number for the wrong construct. Reproducibility cannot rescue a measurement whose target or causal time is mis-specified.
Metric choice belongs to the operating decision. Prevalence, error cost, threshold, abstention and subgroup exposure determine whether a higher aggregate score is useful.
A valid offline test can still fail after release because populations, policies, interfaces and label processes move. Evaluation must therefore continue through shadow, canary and outcome evidence.
Release the narrowest claim supported by the weakest link in the evidence chain, and attach every pass threshold to an owner, expiry condition and response.

The model that won without improving anything

Two fraud models are tested on one hundred thousand historical card payments. Model A reaches 96.2 per cent accuracy. Model B reaches 96.7 per cent. The team selects B, describes the gain as material, and prepares a release.

Then an analyst asks for the confusion matrices at the operating threshold. Both models classify nearly every payment as legitimate because fraud is rare. B gains most of its aggregate accuracy by recognising additional easy legitimate cases. At the threshold permitted by the investigation team, it misses more high-value fraud and creates more low-value alerts. Its probability estimates are also less calibrated. The larger headline score describes a smaller operational capability.

Nothing in the arithmetic was false. The test ran correctly. The data file was stable. The mistake came earlier: the team treated a metric as if it were the thing being measured. Accuracy was a reading produced by a chain of choices about the construct, cases, labels, time, aggregation and decision. Change any link and the meaning of 96.7 changes.

Evaluation deserves the same engineering attention as a sensor. A thermometer needs a defined quantity, calibration, operating range, uncertainty and maintenance. A machine-learning evaluation needs a target construct, sampling frame, label process, temporal boundary, comparison design, metric, threshold and route to consequence. It also needs a statement of what the result cannot support.

The practical answer is direct. Do not ask whether a model is good. Ask which bounded claim a particular evaluation distinguishes from its strongest rival. That question turns a leaderboard into a measurement design.

A score has a chain behind it

Suppose the claim is “the model detects customers likely to miss a payment early enough for a supportive intervention.” The evaluation dataset contains accounts, a binary default label and model scores. It is tempting to move straight to area under the ROC curve. Yet the claim contains several untested terms.

Does “miss a payment” mean one day late, thirty days past due, a write-off, or entry into a collections process? Is the outcome observed equally for every account? Does “early enough” exclude information created after a reminder or forbearance offer? Does the intervention itself change the outcome? Does the model need calibrated risk, a ranking, or a decision at a fixed capacity? Which customers bear the cost of a false alarm?

Each answer changes the measurement. The score is downstream of all of them.

Figure 1. A score is the end of a measurement chain

Reading. The chain can fail at the construct, sampling frame, labels, split, metric or decision threshold. Practitioner synthesis.

The chain begins with a construct: the capability or property the team wants to know about. It passes through an operational definition, case population, observation process, labels, split design, model run, metric and decision rule. Evidence can weaken at any point. A strong model result cannot repair a label that measures a different event. A careful confidence interval cannot repair a test set that received future information.

This chain also explains why more test cases do not automatically improve validity. A million biased labels can estimate the wrong quantity very precisely. Precision concerns variation under the measurement procedure. Validity concerns whether the procedure supports the intended interpretation. They are related and distinct.

Evaluation is an intervention on a claim

A useful test creates different expected observations under rival explanations. Model B may beat A because its representation captures a stable signal. It may also win because it uses a post-outcome feature, sees duplicates, benefits from a favourable subgroup mixture, or receives more tuning against the test set. An evaluation that permits all explanations measures performance without locating mechanism.

The test becomes more discriminating when one causal feature is varied while others remain fixed. Remove the suspect feature. Move the temporal split. Match the tuning budget. Freeze the threshold. Reweight the population. Preserve the model and vary label authority. These interventions transform an attractive score into evidence about why the score exists.

Construct before metric

A construct is an idea that cannot be read directly from one field. Fairness, reasoning, relevance, hallucination, toxicity, creditworthiness and usefulness are constructs. Even apparent facts such as “correct answer” can become ambiguous when several answers satisfy the request or when the source is contested.

Construct validity asks whether the observations and scoring procedure support the interpretation. It has several practical faces.

Content coverage asks whether the cases span the relevant behaviours. Convergent evidence asks whether different credible measurements agree. Discriminant evidence asks whether the test stays separate from adjacent properties. Consequential validity asks what follows when the measurement is used. Transport validity asks whether the interpretation survives the environment in which the claim will travel.

Consider an agent benchmark that scores whether a calendar event exists at the end of a task. The construct is safe task completion. The final state alone misses whether the agent invited the wrong person, leaked a private note, created duplicates and then deleted them, or ignored a required confirmation. The benchmark measures one outcome property and leaves trajectory assurance mostly unobserved.

This does not make the benchmark useless. It narrows the claim. The result can support “the final calendar contained the target event under these cases.” It cannot support “the agent schedules meetings safely.” Claims should contract to the measurement rather than stretching the measurement to the claim.

Thought experiment: swap the name, keep the test

Take a benchmark called “strategic reasoning.” Its cases are short multiple-choice puzzles with unambiguous answers. Rename it “symbolic puzzle selection” without changing a case or score. The numerical ranking stays fixed, yet the inferred capability shrinks.

Now add a planning environment where actions reveal information and mistakes alter later options. Some models reverse order. The original test captured a useful component, but the name encouraged a larger interpretation than the cases carried.

The thought experiment varies only the interpretive bridge. It shows that evaluation contains an argument: observed behaviour under conditions $C$ is evidence for construct $K$ . The bridge must be defended, not smuggled into the benchmark title.

The argument in one passChart · flow

flowchart LR
  A["The model that won without<br/>improving anything"] --> B["A score has a chain behind it"]
  B --> C["Construct before metric"]
  C --> D["Causal time and leakage"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#087A98,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

A structural route through Evaluation Is a Measurement System, from The model that won without improving… to Sources. Reading use: follow the relation, then test it against the surrounding evidence and limits.

Causal time and leakage

Prediction has a time of use. Every input must be available, permitted and sufficiently fresh at that time. Data leakage occurs when the evaluation supplies information that would not exist in the intended decision context or when information crosses a boundary between training, tuning and testing.

The familiar example is a feature recorded after the outcome. Less obvious leakage arrives through preprocessing fitted on the full dataset, duplicates across splits, target-derived encodings, human notes written after a decision, entity histories that include the future, and repeated tuning against a supposedly untouched test set.

Figure 2. Leakage is information crossing the evaluation clock

Reading. Features created after the decision point make a retrospective test look predictive. Times are schematic.

The cleanest control is an as-of reconstruction. For every case, declare a decision time $t_{d}$ . A feature $x_{j}$ is eligible only if its event time, publication time and entitlement state make it observable at $t_{d}$ . When a feature summarises history, rebuild it from events available by that cut-off. Store the query and snapshot identity, not merely the resulting number.

Thought experiment: the helpful collections flag

A mortgage model predicts default at application. One feature says whether the account entered collections. It is extraordinarily predictive. The field exists in the warehouse row and passes a random split. Remove only the time relationship: collections happens after missed payments, which happen after the application decision.

The model may still be useful for retrospective case classification. It is invalid for application-time prediction. The same column changes status when the claimed decision time changes. Leakage therefore belongs to a claim and a clock, not to a feature name alone.

Random splitting often hides this problem because past and future cases share a distribution and entities. A temporal holdout better tests forward transfer. A grouped split prevents the same customer, device, document or repository from appearing on both sides. Neither is universally sufficient. A valid design follows the unit of generalisation.

Test-set leakage through human attention

An evaluation set can leak without a data join. Repeatedly inspecting errors, changing prompts and rerunning the same benchmark turns the test into training feedback. The final score no longer estimates untouched performance. Versioned private holdouts, rolling refreshes and explicit tuning budgets reduce this pressure. They do not remove the need to document how many decisions the benchmark influenced.

The correct response is not permanent secrecy. It is separation of roles. Development cases teach. Validation cases select. Test cases estimate a bounded claim after choices freeze. Operational monitoring asks whether that claim still holds.

Labels have authority

A label is an observation produced by a process. Some labels are direct outcomes: a payment cleared, a component failed, an appeal succeeded. Others are adjudications: reviewers examined evidence and resolved a case. Others are proxies: a complaint code stands in for dissatisfaction, a click stands in for relevance, a past intervention stands in for risk. Some labels are predictions from an older model.

These sources do not carry equal authority.

Figure 4. Label authority is not binary

Reading. Labels range from direct outcomes to proxies and inherited model decisions. Distance from the target construct should remain visible. Practitioner synthesis.

A useful label record includes the target definition, observer, evidence available, timestamp, adjudication route, uncertainty, policy version and whether the label was influenced by a model. If labels come from several routes, stratify performance by route. A model can appear accurate because it learns which cases receive careful review.

Inter-rater agreement helps characterise one part of the process. It does not prove truth. Two reviewers can agree because they share the same ambiguous guidance. Disagreement can reveal a boundary case rather than noise to be averaged away. Adjudication should preserve the original votes and reasons so that uncertainty remains inspectable.

Selective labels alter what can be known

In lending, the repayment outcome is observed for approved applicants and not for rejected applicants. In healthcare, a diagnosis may be observed only for patients sent for a test. In fraud, manual review may concentrate on high scores. The historical decision controls which labels exist.

An ordinary test on observed labels estimates performance inside that selection process. It may not support claims about the full population. Inverse-propensity methods, natural experiments, exploration policies and bounds can help under assumptions. The assumptions must remain visible. No metric repairs missing counterfactual outcomes for free.

Weak supervision can be productive when direct labels are scarce. Multiple noisy rules, heuristics or sources may be combined into probabilistic labels. The evaluation should then separate uncertainty in the label model from uncertainty in the predictor. Testing against labels generated by the same rules used for training creates circular evidence.

Metrics meet consequence

For binary classification, a threshold turns a score into four counts: true positives $T P$ , false positives $F P$ , false negatives $F N$ and true negatives $T N$ . Accuracy is

$\frac{T P + T N}{T P + F P + F N + T N} .$

It weights every case equally and inherits the class prevalence. Recall $T P / (T P + F N)$ asks how many positives were found. Precision $T P / (T P + F P)$ asks how many positive predictions were correct. Specificity and false-alarm rate describe the negative class. ROC and precision-recall curves vary the threshold, but they still require a choice about the relevant operating region.

Figure 3. The best metric depends on the cost geometry

Reading. Two models with similar aggregate accuracy can occupy different operational loss positions. Values are synthetic.

A decision should use a loss or utility model. For a threshold $τ$ , a simple expected loss is

$L (τ) = c_{F N} F N (τ) + c_{F P} F P (τ) + c_{R} R (τ),$

where $c_{F N}$ and $c_{F P}$ are error costs and $c_{R}$ is the cost of review, delay or another downstream action. Costs may vary by amount, customer, time and recoverability. Writing them as constants is an approximation that must be tested.

Calibration matters when a probability determines pricing, triage or evidence gathering. A model is calibrated over a group when events predicted at probability $p$ occur about proportion $p$ . Aggregate calibration can conceal subgroup miscalibration. Brier score, log loss and calibration plots reveal different aspects; none replaces a decision analysis.

Abstention changes the metric space

Some systems can defer uncertain cases. Then coverage becomes a first-class quantity. Accuracy can rise by abstaining on hard cases, but operational burden and unequal deferral may rise too. Plot risk against coverage. State who receives deferred cases, how long they wait, and what happens when review capacity is exhausted.

For generated text, one scalar is even less sufficient. Factual support, task completion, harmfulness, style and refusal may conflict. Human and model judges introduce their own measurement errors. Use item-level rubrics, blind comparisons, judge calibration, disagreement analysis and direct property tests where possible. A fluent judge explanation is not evidence of judge validity.

Shift and the vanishing test set

An offline test estimates behaviour under a distribution. Release changes that distribution. Users adapt to the interface. Operators route unusual cases differently. Policy alters labels. Attackers search for weaknesses. Upstream systems change formats. The original test set slowly becomes a historical sample of a retired environment.

Covariate shift changes input frequencies. Label shift changes outcome prevalence. Concept shift changes the relationship between inputs and targets. Policy shift changes what actions and labels are observed. Supervision drift changes the label process itself. These shifts require different detectors and responses.

Figure 5. A stable score can conceal a moving population

Reading. Equal overall performance can coexist with opposite subgroup changes when mixture weights move. Synthetic populations.

Comparing feature histograms can reveal obvious movement and miss semantic change. A code field may retain its distribution while its business meaning changes. A document template may preserve words and move a decisive clause. A tool API can return the same schema with different default behaviour. Monitoring should combine statistical signals, contract tests, sampled adjudication and outcome evidence.

Aggregates hide moving mixtures

Suppose accuracy remains 90 per cent for two quarters. In the first quarter, experienced users dominate and receive 95 per cent accuracy; new users receive 75 per cent. In the second, new-user accuracy improves to 85 per cent while experienced-user accuracy falls to 88 per cent, and the population mix changes. The aggregate can remain stable while the experience reverses.

Slice by variables tied to mechanism and consequence: population, time, source, interface, language, task difficulty, authority route and value at risk. Avoid an uncontrolled fishing expedition across thousands of slices. Predeclare important groups, add discovery methods for unknown clusters, and correct the certainty of claims when many comparisons are made.

Figure 6. Aggregate success can reverse inside subgroups

Reading. The four panels separate population, time, interface and consequence slices. Bars are illustrative.

Small groups produce uncertain estimates. Show denominators and intervals. Pooling can stabilise estimates but may hide the very difference being investigated. Hierarchical models offer partial pooling when assumptions fit. The objective is not a colourful dashboard. It is to identify where the release claim expires.

Comparison needs an uncertainty model

The same test cases usually pass through both candidate models. Their errors are paired, so independent-sample formulas waste information or misstate uncertainty. For binary outcomes, inspect the cases on which A and B disagree and use a paired procedure such as McNemar’s test where its assumptions fit. For continuous loss, bootstrap cases as the deployment sampling unit, preserving groups and time blocks when observations are dependent.

Do not treat repeated stochastic runs as extra independent customers. Variation across seeds, prompts or decoding samples is a separate source from variation across cases. A hierarchical analysis can represent case and run effects. At minimum, report performance distributions across both dimensions and show how often the release decision reverses.

A confidence interval is conditional on the sampling and analysis design. It does not include uncertainty from a wrong construct, future shift or unobserved labels. State what was resampled and why that unit matches the claim. Cluster by customer, document, repository or episode when several rows share a causal unit.

Model selection creates optimism. Trying fifty configurations and reporting the winner on the same validation set selects favourable noise. Nested validation, correction for multiple comparisons, a locked final holdout and an explicit experimentation ledger reduce this effect. The relevant number is not only the winner’s score; it is the score after the search procedure that produced the winner.

Practical significance should precede statistical significance. With enough cases, a tiny difference becomes detectable. Ask whether the interval excludes the minimum change needed to alter a decision after cost, calibration and risk. A release may rationally retain the simpler model even when a complex candidate wins by a statistically credible but operationally irrelevant margin.

Evaluators are instruments too

Generated outputs often lack one automatic ground truth, so teams use human reviewers or language-model judges. Their evaluations need evaluation.

Define a rubric with observable properties and examples at boundaries. Blind reviewers to model identity and treatment where possible. Randomise answer order in pairwise comparison. Include repeated items to measure within-rater stability, known anchors to estimate calibration, and adversarial cases where verbosity, citation style or confidence conflicts with correctness.

For a model judge, record judge model, prompt, decoding, context and position. Measure agreement with adjudicated labels by task type, not only overall. Inspect false approval because a permissive judge can make every candidate look strong. Inspect false rejection where safe refusal or concise truth is penalised. Run position swaps and style controls. A judge that changes preference when candidates swap sides carries order bias.

Do not let the candidate author the decisive evidence the judge sees. If factual support is the property, provide source passages or run entailment checks against retrieved evidence. If tool correctness is the property, expose structured calls and environment outcomes. Asking a judge whether an answer “seems grounded” tests rhetoric.

Human disagreement should remain data. Adjudicate consequential cases and preserve minority reasons. Some disagreements reveal an underspecified task or incompatible values. Collapsing them into a majority label can manufacture clarity. The right remediation may be a narrower claim, richer outcome representation or human decision rather than a more accurate predictor of the majority.

Evaluation assets need governance

Test cases influence design and eventually become stale. Give each suite a purpose, owner, provenance, inclusion rule, known gaps, version and retirement condition. Separate public benchmarks, development sets, private release gates, incident regressions and live sampled reviews. They answer different questions.

Protect sensitive cases. Redact or synthesise where possible, bind access to purpose, and prevent evaluation logs from becoming a secondary data leak. If a test contains adversarial payloads, keep it isolated from systems that might execute them. If labels contain reviewer judgement, record the policy version that framed it.

When an incident becomes a regression case, preserve the failure mechanism without exposing private details. A synthetic analogue can reproduce the state transition. Verify that it actually fails the old version and passes the repaired one. A regression suite full of cases that never discriminated anything becomes ceremonial weight.

Precommit the decision, not every discovery

Before running a release comparison, record the primary claim, population, metric, threshold, minimum useful difference, exclusions and analysis. This prevents the result from choosing its own success criterion. It also makes a null or negative result interpretable.

Exploratory analysis remains valuable. It can discover new slices, failure modes and hypotheses. Label it exploratory and test important discoveries on fresh cases. The distinction is temporal: evidence that suggested a hypothesis should not be presented as an independent confirmation of it.

Define stop rules for experiments. A severe control violation may end a canary immediately. Futility bounds can stop a comparison unlikely to reach the useful difference. Safety and statistical rules should be specified with the operational owner so analysts are not pressured to continue until a favourable result appears.

Preserve all planned outcomes, including inconclusive ones. Selective reporting creates a distorted evidence base inside an organisation just as it does in public research. A compact evaluation ledger can show attempted comparisons, versions, decisions and reasons. It turns failure into reusable information and makes repeated testing against the same cases visible.

The purpose of precommitment is not bureaucracy. It protects the meaning of the measurement while leaving room for discovery. The lighter the claim, the lighter the record can be. The more consequence attached to the score, the more important it is to know what would have counted as failure before the number appeared.

An evaluation decision also needs a named owner. Analysts own calculation integrity, domain experts own construct interpretation, engineers own executable controls, and accountable leaders own residual risk. These roles can overlap, but the receipt should show who accepted each bridge. Ambiguous ownership lets a technically correct metric drift into a policy decision no one explicitly made.

Finally, set an evidence expiry. A new label policy, source migration, interface, population shift or model dependency can retire the result. Expiry does not mean the old test was false. It means the conditions that gave it meaning no longer match the proposed use.

How the claim changes under pressureChart · state

stateDiagram-v2
  [*] --> Claim
  state "Figure 1. A score is the end<br/>of a measurement chain" as Claim
  state "Evaluation is an intervention<br/>on a claim" as Test
  state "Thought experiment: swap the<br/>name, keep the test" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

The revision loop begins with Figure 1. A score is the end of a… and keeps counterexamples inside the reasoning path. Reading use: follow the relation, then test it against the surrounding evidence and limits.

Build an evaluation audit

The executable artefact below audits a deliberately small classification dataset. Each case carries a decision time, feature timestamps, an outcome, a score and a label-authority class. The positive case uses only information available by the decision and labels backed by direct or adjudicated evidence. The negative case adds a collections flag created after the decision and a label inherited from a legacy model.

. It uses only the Python standard library and includes positive and negative cases.

#!/usr/bin/env python3
"""An evaluation audit that treats time, labels and consequences as test inputs."""

from __future__ import annotations

from dataclasses import dataclass
from statistics import mean


@dataclass(frozen=True)
class Case:
    case_id: str
    decision_time: int
    outcome: int
    score: float
    label_authority: str
    feature_times: dict[str, int]


AUTHORITY = {"direct": 3, "adjudicated": 2, "proxy": 1, "legacy_model": 0}


def audit(cases: list[Case], threshold: float = 0.5) -> dict:
    leakage = {
        case.case_id: sorted(name for name, timestamp in case.feature_times.items() if timestamp > case.decision_time)
        for case in cases
    }
    leakage = {case_id: names for case_id, names in leakage.items() if names}

    predictions = [case.score >= threshold for case in cases]
    outcomes = [bool(case.outcome) for case in cases]
    tp = sum(pred and actual for pred, actual in zip(predictions, outcomes))
    fp = sum(pred and not actual for pred, actual in zip(predictions, outcomes))
    fn = sum(not pred and actual for pred, actual in zip(predictions, outcomes))
    tn = sum(not pred and not actual for pred, actual in zip(predictions, outcomes))
    recall = tp / (tp + fn) if tp + fn else 0.0
    false_alarm_rate = fp / (fp + tn) if fp + tn else 0.0
    expected_cost = (5 * fn + fp) / len(cases)
    weakest_label = min(AUTHORITY[case.label_authority] for case in cases)

    return {
        "case_count": len(cases),
        "leakage": leakage,
        "recall": round(recall, 3),
        "false_alarm_rate": round(false_alarm_rate, 3),
        "expected_cost_per_case": round(expected_cost, 3),
        "mean_score": round(mean(case.score for case in cases), 3),
        "weakest_label_authority": weakest_label,
        "releaseable": not leakage and weakest_label >= AUTHORITY["adjudicated"],
    }


def run() -> None:
    valid = [
        Case("A", 10, 1, 0.90, "direct", {"application": 4, "history": 8}),
        Case("B", 10, 0, 0.20, "adjudicated", {"application": 3, "history": 9}),
        Case("C", 10, 1, 0.70, "direct", {"application": 5, "history": 7}),
        Case("D", 10, 0, 0.10, "adjudicated", {"application": 4, "history": 8}),
    ]
    leaked = valid + [
        Case("E", 10, 1, 0.99, "legacy_model", {"application": 4, "collections_flag": 15})
    ]

    positive = audit(valid)
    negative = audit(leaked)
    print({"valid": positive, "leaked": negative})
    assert positive["releaseable"] is True
    assert negative["releaseable"] is False
    assert negative["leakage"] == {"E": ["collections_flag"]}


if __name__ == "__main__":
    run()

The audit reports leakage, recall, false-alarm rate, a consequence-weighted cost and the weakest label authority. Its releaseable result is intentionally conservative. A strong confusion matrix cannot override temporal leakage or a circular label source.

The code is not a universal validator. Real systems need event-time semantics, source lineage, group-aware metrics, uncertainty and a larger policy model. Its contribution is structural: validity conditions are executable inputs rather than prose written after the score.

The evaluation evidence chain

Figure 7. Evaluation evidence should narrow a claim

Reading. Each link carries a stronger condition: code ran, metric was valid, task transferred, and the decision improved. Practitioner synthesis.

A result earns broader authority one link at a time.

Execution evidence: the intended code, data and model version ran reproducibly.
Metric evidence: calculations, denominators and uncertainty are correct.
Construct evidence: cases and labels distinguish the claimed capability from rivals.
Transport evidence: important shifts, subgroups and interfaces were tested.
Outcome evidence: the resulting decision improves the bounded operational objective without unacceptable harm.

Failure at a lower link limits higher claims. Outcome evidence can reveal that an offline construct was incomplete. Construct evidence can show why a metric is misleading. The chain is therefore revisable, not a one-way compliance checklist.

A release instrument

Figure 8. The release gate asks what the test can support

Reading. A defensible release moves only when the construct, split, labels, uncertainty and consequence thresholds are explicit.

Gate	Question	Required record	Authority if passed
Construct	What property is named, and what rival can mimic success?	Construct map and negative controls	Test may speak about the bounded property
Time	Was every input available and permitted at decision time?	As-of reconstruction and lineage	Test may speak about prospective use
Labels	Who observed the target, with what evidence and selection?	Label authority and missingness analysis	Test may speak about the labelled population
Comparison	Were budget, information and tuning held fairly?	Matched baseline manifest	Test may attribute an incremental gain
Decision	Which threshold, capacity and consequence matter?	Utility, calibration and deferral analysis	Test may support a release decision
Transport	Where does the result stop travelling?	Shift, subgroup and interface stress tests	Claim receives an explicit boundary
Operation	What signal expires the evidence?	Owner, monitor, threshold and response	Claim remains conditionally active

From offline score to operating evidence

Consider a document-triage system that routes incoming complaints. The model proposes a category and urgency. A retrieval component supplies policy evidence. A workflow sends high-urgency cases to a specialist team. Success seems to be category accuracy on a labelled archive.

The measurement design separates six questions. Does the archive represent current complaint channels? Were categories assigned before or after investigation? Can the model detect rare harm themes, not merely common product words? Does routing improve response time? Do false alarms overwhelm specialists? Does the interface encourage reviewers to accept suggestions?

Offline, the team runs temporal and channel holdouts, direct label adjudication, calibration and matched baselines. In shadow, it observes proposed routes without changing work. In canary, it releases to a bounded queue with dual review and readback. Operationally, it monitors queue time, overrides, missed high-harm cases, appeals and downstream resolution. The evaluation stack expands as contact with consequence increases.

The model can pass one layer and fail another. Category accuracy may improve while response time worsens because the new taxonomy fragments queues. That is not a paradox. The model metric and business outcome measure different links. The release decision should follow the intended outcome while retaining component evidence for diagnosis.

Failure boundary

Evaluation cannot prove universal safety or future performance. It samples behaviours under a design. Rare interacting failures may lie outside the cases. Strategic users may adapt. Human review may change under workload. Outcomes may arrive late or remain partly unobservable.

The answer is bounded assurance. State the tested environments, unsupported populations, maximum consequence, residual uncertainty and recovery path. Use failure injection to explore interactions. Preserve an unknown-outcome state when effects cannot be verified. Withdraw authority when monitoring crosses a threshold.

The strongest objection is that this level of evaluation slows experimentation. Early research cannot build a production evidence chain for every idea. Correct. The depth should match the claim. Exploratory work may report a demonstration or hypothesis. A public benchmark may report task performance. A consequential release needs stronger evidence. The error is not limited evidence; it is expansive language attached to limited evidence.

Make the number carry only what it measured

Model B’s 96.7 per cent accuracy was a valid calculation and an invalid release argument. Once the construct, threshold and consequence were restored, the apparent improvement disappeared. That outcome is useful. The evaluation prevented a score from acquiring authority it had not earned.

The changed engineering decision is to version the measurement system alongside the model. Store construct definitions, case selection, causal cut-offs, label provenance, split rules, metrics, thresholds, subgroup plans, uncertainty, owners and expiry. Treat changes to any of them as evaluation changes. Re-run the appropriate evidence, not merely the metric script.

Make one person accountable for each active claim. Ownership is not authorship of the benchmark; it is responsibility for noticing when the evidence no longer supports production authority. The claim record should name its last observation, next review, expiry signal and withdrawal action. This turns evaluation from a launch ceremony into a maintained measurement system.

A score should narrow uncertainty about a decision. If it cannot say which rival it ruled out, which population it covers and which consequence it improves, it is an observation waiting for an argument. Build that argument before release. Keep it alive after release. Let the weakest verified link determine the strength of the claim.

The evidence conversationChart · sequence

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: The model that won without improving anything
  M->>E: Figure 1. A score is the end of a measurement chain
  E-->>M: compare rival explanations
  E->>D: Sources
  D-->>Q: revise scope or act

Question, mechanism, evidence and decision exchange the burdens developed in Evaluation Is a Measurement System. Reading use: follow the relation, then test it against the surrounding evidence and limits.

Sources

Primary Jacobs and Wallach, Measurement and Fairness, 2021.
Primary D’Amour et al., Underspecification Presents Challenges for Credibility in Modern Machine Learning, 2022.
Primary Kapoor and Narayanan, Leakage and the Reproducibility Crisis in Machine-Learning-Based Science, 2023.
Primary Davis and Goadrich, The Relationship Between Precision-Recall and ROC Curves, 2006.
Primary Sugiyama et al., Covariate Shift Adaptation by Importance Weighted Cross Validation, 2007.
Primary Mitchell et al., Model Cards for Model Reporting, 2019.
Primary Sculley et al., Hidden Technical Debt in Machine Learning Systems, 2015.
Official NIST, AI Risk Management Framework, current official framework.