The model that passed after the problem disappeared

A natural-language inference item contains a premise and a hypothesis. The intended task is to decide whether the premise entails, contradicts or is neutral towards the hypothesis. Remove the premise and the task should become impossible, because the relation has lost one of its terms.

Yet Gururangan and colleagues trained a simple classifier on the hypothesis alone and reported about 67 per cent accuracy on SNLI and 53 per cent on MultiNLI. Workers who wrote hypotheses had left label-correlated traces such as patterns of negation and vagueness. The result did not show that every high-scoring system ignored the premise. It showed something more foundational: the benchmark permitted a route to success that bypassed the capability its score was commonly taken to represent.

This is the construct-validity problem. A benchmark observes responses to a finite operation. Researchers and buyers then interpret those responses as evidence for something broader: reasoning, safety, planning, truthfulness, clinical competence or general language understanding. The operation and the construct are not identical. The distance between them must be crossed by an argument.

Benchmark construct validity is the strength of the causal case that the named capability, rather than a cheaper rival route, generated the score and will support the intended use. A dataset can be clean, difficult and reproducible while that case remains weak. Conversely, a small benchmark can be highly informative when its items are sampled deliberately and its interventions separate competing explanations.

This paper develops an audit for that argument. It is narrower than What Would Count as Intelligence?, which asks what a broad intelligence claim requires, and it precedes the multi-level operational evidence in The Evaluation Stack for Agentic Systems. The question here is precise: what must be true before a benchmark result may carry the name of a capability?

Suppose a model scores 84 per cent on a benchmark labelled “reasoning”. Four statements can follow, each stronger than the previous one:

1. the configured system received credit on 84 per cent of these scored items;
2. it can solve similar items drawn through the same process;
3. the score is mainly caused by the target reasoning process rather than nuisance skills or exposure;
4. the result predicts useful reasoning in the intended environment.

The first statement is close to observation. The others are inferences. Reliability, label quality and statistical uncertainty matter at every step, but they do not complete the inferential chain. A perfectly repeatable ruler can still measure the wrong dimension.

Cronbach and Meehl introduced construct validation for cases where a test score is interpreted as an attribute not exhausted by the test operation. Messick later placed the emphasis on the interpretation and use of scores, not on a permanent seal attached to an instrument. The practical consequence for AI is severe: “validated benchmark” is incomplete language unless the claimed construct, candidate system, conditions and intended decision are stated.

The construct-evidence chain

A benchmark can be represented as a chain of design commitments. First comes a definition of the phenomenon. That definition is operationalised as a task family. Items sample that family. An interface presents the items to a bounded system. A scorer converts responses into credit. An aggregation rule produces a reported score. Finally, someone turns that score into a claim or decision.

Every arrow can fail. A broad label such as “agentic capability” may combine intent recognition, planning, tool use, state tracking and recovery without saying how they interact. A task family may cover only tidy exam questions. Items may be selected for convenience. The interface may add a strict output grammar that weak models fail independently of the target skill. The scorer may reward verbosity or familiar phrasing. The aggregate may hide catastrophic failures in a small subgroup. The final user may interpret the number as readiness for a setting never tested.

Bean and colleagues found that 61.2 per cent of the reviewed benchmarks defined their target as a composite phenomenon, while 21.1 per cent required specific output formats that could themselves be challenging. Exact matching appeared in at least part of 81.3 per cent of benchmarks. None of those choices is automatically wrong. They become threats when the benchmark name suppresses the auxiliary demands that actually move the score.

A small causal model

Let the observed benchmark score be:

C is the target capability. B covers background knowledge and other abilities. I is proficiency with the prompt, interface and output format. E is exposure to items, near-duplicates or benchmark-specific optimisation. J is the scoring process, including human or model judges. ε represents sampling and execution noise.

The benchmark is informative about C when changes in C predict changes in S, while plausible changes in the nuisance terms do not dominate the result. One score cannot identify those contributions. The evaluator needs multiple measurements or interventions whose outcomes differ under competing explanations.

Construct validity is therefore not a correlation between a label and a number. It is a tested network of predicted relationships. A reasoning benchmark should converge with independent reasoning measures, distinguish reasoning from recall and format compliance, change when the underlying relation changes, remain stable under irrelevant paraphrase, and predict performance on tasks for which the reasoning claim matters.

Reliability is necessary and insufficient

A benchmark that changes rank when punctuation, prompt wording or runtime settings change has an instrument problem. Biderman and colleagues document how language-model evaluation is sensitive to setup choices and difficult to reproduce across harnesses. BetterBench assessed 24 benchmarks against 46 lifecycle practices and found large quality differences, with statistical significance and easy replication commonly absent. These are serious failures because an unstable measure cannot sustain a delicate construct inference.

But high reliability can coexist with invalid interpretation. A public answer key produces perfectly repeatable scores. A cue-only classifier may exploit the same annotation habit every time. A multiple-choice exam can consistently measure recall while being advertised as clinical judgement. Reliability asks whether the instrument repeats; validity asks what the repeated result means.

The measured subject is a configured system

A model name is not a sufficient system boundary. The same weights can produce materially different benchmark results under a different system prompt, number of demonstrations, decoding policy, tool set, retrieval corpus, context window, retry budget or answer extractor. Agent benchmarks add environment version, browser state, credentials, action limits and recovery policy. Human review can be part of the configured subject or an external evaluator, but it cannot be left ambiguous.

This matters because construct claims are often transferred across boundaries without evidence. A base model’s multiple-choice score is used to describe a retrieval-augmented service. A tool-enabled agent inherits the model’s safety result even though its action space has changed. A prompt update keeps the same product name while altering which instructions dominate. The benchmark may still be repeatable, yet it is now measuring a different candidate system.

A validity card should therefore begin with a versioned subject manifest: model identifier, prompt hash, tool contracts, accessible data, resource limits, sampling settings and evaluator. It should also state where learning or adaptation can occur. If a human studies failures and edits the prompt between runs, the engineering process has adapted. If a memory service writes during the run, the service boundary includes that store. Capability belongs to the tested configuration, not automatically to every product that contains the same model.

Modern models are powerful searchers over regularity. If a benchmark offers a predictive feature cheaper than the intended capability, a training process has reason to use it. Geirhos and colleagues call these decision rules shortcuts: they work under standard test conditions and fail when the conditions change. Shortcut use is not moral cheating. It is a mismatch between the optimisation pressure and the evaluator’s causal story.

Four substitutions to look for

Construct underrepresentation occurs when the benchmark samples too little of the named capability. A “planning” benchmark made only of short deterministic mazes does not cover replanning after observation error, uncertain effects or irreversible actions. Bean and colleagues found that 27.0 per cent of reviewed benchmarks used convenience sampling and 38.2 per cent reused data from previous benchmarks or human exams.

Construct-irrelevant variance occurs when success depends on something the claim does not intend to measure. A strict JSON response may turn reasoning errors into formatting errors. A multilingual test may mix domain knowledge with translation difficulty. A code benchmark may confound algorithm design with dependency setup. The auxiliary demand can depress a capable system or favour a specialised one.

Shortcut access occurs when a surface feature, item identity or training overlap predicts the answer. The NLI hypothesis-only result is one example. Public benchmark reuse creates another. Recht and colleagues reconstructed test sets for CIFAR-10 and ImageNet and observed accuracy drops, including roughly 11 to 14 percentage points on ImageNet, even though relative improvements largely transferred. The original score still measured something, but its absolute interpretation did not survive a near-replication of the sampling process.

Scoring substitution occurs when the metric rewards a proxy for the desired response. Exact match may treat a semantically correct free-text answer as wrong. A model judge may prefer verbosity, position or its own style. Aggregate accuracy may value every error equally even when the intended use contains asymmetric harm. The scorer becomes part of the construct whether the benchmark author acknowledges it or not.

Thought experiment: the arithmetic test with a perfect margin

Build a 1,000-item arithmetic benchmark. Every question requires two-digit addition. During item generation, place the correct answer in the longest option 95 per cent of the time. System A computes addition but has a 12 per cent execution error. System B selects the longest option. The leaderboard shows B at 95 per cent and A at 88 per cent.

Now vary one causal feature at a time. First, shuffle option lengths while preserving every arithmetic problem and answer. A’s performance should remain near 88 per cent; B should fall towards chance. Second, keep the original options but change one operand, making a different option correct. A should usually change its answer; B should remain anchored to length. Third, replace public items with newly generated additions. A should transfer; a lookup system should not.

The initial ranking was computed correctly. What failed was the capability interpretation. The decisive tests are not merely harder items. They are interventions chosen because rival mechanisms predict different outcomes.

Negative controls that can win

A weak negative control is a random guesser. It confirms that the task is not trivial but says little about the intended construct. A serious negative control is designed to reproduce the preferred result through the wrong mechanism.

For NLI, hypothesis-only classification is serious because it removes the relation while retaining annotation artefacts. For visual question answering, a question-only system tests language priors. For retrieval, a popularity baseline tests whether common entities dominate. For tool use, a script that memorises interface paths tests whether the environment is fixed. For a public benchmark, an item-identity lookup tests exposure directly.

The control should be allowed to look impressive. If it wins, the benchmark has learned something about itself. If it loses only on transformations that preserve the intended construct, the preferred mechanism earns discriminating evidence. The point is not to humiliate a model. It is to make two causal stories disagree.

Bad labels and bad constructs are different failures

MMLU-Redux manually re-annotated 5,700 questions across 57 subjects and estimated that 6.49 per cent of MMLU questions contained errors. That is an instrument-quality problem: ambiguous questions and incorrect keys corrupt the observed score. Correcting them improves reliability and fairness of comparison.

But perfect labels would not establish that a multiple-choice academic test measures broad reasoning, professional judgement or safe deployment. That is a construct problem. The distinction matters because the remedies differ. Label review, deterministic scoring and reproducible harnesses repair the instrument. Task decomposition, representative sampling, intervention design and outcome prediction repair the validity argument.

GSM-Symbolic illustrates the value of controlled generation. Its authors changed numerical values and inserted apparently relevant but unnecessary clauses into grade-school mathematics problems, reporting substantial performance variation and, in some settings, large declines. The strongest inference is not that a single paper settled whether language models “really reason”. It is that a benchmark becomes more diagnostic when it can vary one structural feature while holding others fixed.

Contamination has more than one channel

Direct item contamination is the clearest case: an evaluation question, answer or close paraphrase appears in training, retrieval or demonstrations. Family contamination is subtler. A model may have seen thousands of problems generated from the same templates, public solution styles or benchmark-specific conventions, making fresh item identities insufficient. Adaptive contamination occurs at community level when repeated leaderboard optimisation selects prompts, models and methods for one fixed test even without literal training overlap.

These channels require different controls. String and embedding searches can find some direct overlap, but absence of a match is not proof of novelty. Procedural generation can create fresh values while preserving familiar templates. A sealed or time-stamped test reduces direct exposure, while cross-family transfer and hidden transformation sets test whether competence extends beyond benchmark conventions. Repeated public use should be treated like repeated access to a validation set: the score may remain useful for regression, but its status as independent evidence decays.

Freshness is a relation between an item and the complete system history. It cannot be established from the evaluator’s unfamiliarity alone. When training provenance is unavailable, the right output is not “uncontaminated”; it is an uncertainty statement, a stronger fresh-set design, and a claim that remains conditional on unknown exposure.

A construct audit begins before data collection. The evaluator writes down the target property, the intended use, and at least two rival explanations for success. Each explanation must predict a different pattern under a controlled change. The audit then asks two complementary questions:

Tₙ changes something the construct says should not matter, such as option order, entity names or harmless paraphrase. T𝚌 changes the target relation, such as reversing a premise, removing necessary evidence or changing a governing rule. A credible system should be stable in the first case and appropriately sensitive in the second.

Thought experiment: one fire drill, two definitions of planning

A robot is evaluated in a simulated office. The benchmark calls a run successful when the robot reaches an exit after an alarm. Every map has one open corridor and no moving people. A reactive wall follower scores 100 per cent. A planner that builds a map scores 96 per cent because localisation occasionally fails.

Now change one feature at a time. Add a blocked corridor after the robot commits to a route. If planning includes replanning, the system should reconstruct a path. Keep the map unchanged but move exit signs. A system relying on visual signage should shift; a map-based planner should not. Add a second exit with different smoke risk. A shortest-path system and a risk-aware planner should diverge. Finally, ask the system to explain its route without acting. Verbal success may correlate with planning but cannot replace action evidence.

The original benchmark validly measured exit-reaching in a narrow environment. It did not isolate planning. Broadening the claim requires a task family in which planning, reactive control, sign following and memorised routes make different predictions.

Six sources of validity evidence

Content evidence asks whether the sampled items cover the relevant task space. A blueprint should enumerate sub-capabilities, difficulty factors, operating conditions and exclusions before item writing. Stratified sampling can be more informative than a much larger convenience set.

Response-process evidence asks what route produced the response. Process traces are not automatically faithful, but interventions can test whether retrieved evidence, tool calls or intermediate states are causally used. Remove a cited source, swap it with a contradiction, or patch a stored state and observe whether behaviour changes as predicted.

Internal-structure evidence asks whether score relationships match the construct model. If “agent competence” is claimed to contain planning, tool use and recovery, item correlations and failure clusters should support that structure rather than one dominant format factor. A single total score should not be reported when the dimensions behave independently.

Convergent and discriminant evidence asks whether the benchmark agrees with independent measures of the same property and separates from measures of different properties. Two benchmarks sharing the same public source, prompt style and judge may correlate because they share artefacts. Independence of method matters.

Predictive and ecological evidence asks whether scores forecast the intended outcome under representative conditions. A coding benchmark can be compared with repository-level defect repair. A safety benchmark can be compared with incident-relevant behaviours in a sandbox. Hutchinson and colleagues warn that ML evaluations often centre a narrow set of decontextualised predictive behaviours while neglecting broader application concerns.

Consequential evidence asks how the score will be used and who bears error. A rank used for research exploration has a different burden from a threshold used to approve customer-facing action. The same instrument may support one decision and not the other.

Statistics belong inside the validity argument

A benchmark rank without uncertainty can turn sampling noise into a capability story. Report item-level denominators, repeated runs where stochasticity matters, confidence intervals and paired comparisons. Analyse whether the result survives plausible task weights, subgroup prevalence and scorer variation. The Benchmark Lottery demonstrates that rank can change when benchmark tasks change; a robust claim should therefore report a frontier or profile when reasonable mixtures cross.

Statistical power also constrains what a non-result means. If two systems differ by one point on 100 items, the benchmark may be unable to distinguish them. If a shortcut intervention contains only ten cases, passing it is weak evidence. Conversely, a tiny but carefully selected set can expose a mechanism when every item is a discriminating experiment.

More examples reduce variance only within the sampled design. They do not repair a missing construct, a contaminated item source or a scorer that rewards the wrong thing.

Claim language is an audit output

The final benchmark product should include a sentence the evidence is permitted to support. “Model A is better at reasoning” is usually too broad. A defensible alternative might be: “Under this prompt and tool-free configuration, Model A achieved higher paired accuracy than Model B on fresh, English, short-form deductive items, retained its advantage under option reordering, and changed answers appropriately on relation-reversal counterfactuals.”

That sentence is longer because it carries the system boundary, task family, comparison and discriminating evidence. It also reveals what has not been shown: other languages, long-horizon reasoning, tool use, uncertain premises and operational decisions. The omissions are not editorial weakness. They are the scientific boundary of the result.

When a benchmark is used for selection, translate the claim into a decision rule. A router may choose the lowest-cost model that passes format, fresh-set and counterfactual gates for one workload. A research paper may report a profile rather than a winner. A release committee may require predictive evidence from a shadow workflow. The score is an input; the bounded claim is the audit’s real output.

Worked example: a cue-rich classification benchmark

The companion lab creates 200 balanced binary items. The intended label is a semantic feature. On the public test, an unintended cue matches the label on 95 per cent of items. Three systems compete.

The cue reader predicts from the surface cue. The target rule uses the semantic feature but has a deterministic 12 per cent error. The public lookup has memorised the public item-to-label mapping and falls back to the cue on unseen items.

On the public set, the lookup scores 100 per cent, the cue reader 95 per cent, and the target rule 88 per cent. A conventional leaderboard ranks the intended mechanism last. The audit then applies three tests. A cue-neutral set preserves labels while breaking cue correlation. A fresh set changes item identities. A counterfactual reverses the semantic feature while holding cue and identity fixed.

The audit does not claim that the target rule has a general semantic faculty. It authorises a narrow statement: within this synthetic task family, its performance is invariant to the tested cue, transfers to fresh identities and responds to the tested construct change. The result is deliberately weaker than the label “understands”.

Worked example: evidence-bound case resolution

Consider a document-intelligence system proposed for resolving cases from long files. The attractive benchmark asks 500 clean questions over selected documents and uses exact-match final answers. The team calls the score “case-resolution capability”. That label silently bundles several burdens: finding relevant evidence, linking entities, applying the rule effective at the event time, resolving contradictions, recognising missing evidence, and producing an authorised recommendation.

A production-shaped benchmark should preserve those burdens rather than replace them with question answering. Sample complete cases across document quality, entity ambiguity, temporal changes and contradiction patterns. Score claims against evidence spans. Test whether the recommendation changes when a decisive document is superseded. Include cases where abstention is correct. Keep execution authority outside the benchmark unless action is genuinely part of the construct.

One aggregate accuracy is not enough. Retrieval recall is a diagnostic, not the outcome. Citation presence is weaker than citation support. A fluent rationale cannot compensate for applying an expired rule. A correct final recommendation reached from unauthorised evidence may still be unusable. The benchmark should mirror the dependency structure of the decision, with hard gates where one failure invalidates the whole case.

A useful scorecard might report evidence recall and claim precision at the inner layer, policy applicability and abstention calibration at the middle layer, and reviewer-confirmed decision quality at the outer layer. The release claim is bounded by the weakest required gate. This is less convenient than one number, but it tells an architect what to improve and a reviewer what remains unproven.

The executable construct audit

The artefact below uses only the Python standard library. It creates the synthetic public, cue-neutral, fresh and counterfactual sets; evaluates all three systems; computes a Wilson interval for public accuracy; applies explicit claim gates; and asserts one positive and two negative cases. Save it as benchmark_construct_audit.py and run python3 benchmark_construct_audit.py.

"""Synthetic benchmark construct-validity audit.

Python 3.10+. Standard library only.
All items and thresholds are synthetic teaching fixtures.

The audit compares three systems that can look strong on one public test:
1. CueReader uses an unintended answer-correlated cue.
2. TargetRule uses the intended semantic relation, with deterministic 12% noise.
3. PublicLookup has memorised the public item-to-label mapping.

A model may support the named capability claim only when it passes a fresh set,
resists a nuisance-only cue intervention, responds to a construct-changing
counterfactual, and has no direct benchmark exposure.
"""

from __future__ import annotations

from dataclasses import asdict, dataclass
from math import sqrt
import json
from typing import Iterable, Protocol


@dataclass(frozen=True)
class Item:
    item_id: str
    semantic: int  # the intended construct-relevant feature
    cue: int       # an unintended surface feature
    label: int


class System(Protocol):
    name: str
    benchmark_exposed: bool

    def predict(self, item: Item) -> int: ...


class CueReader:
    name = "cue_reader"
    benchmark_exposed = False

    def predict(self, item: Item) -> int:
        return item.cue


class TargetRule:
    name = "target_rule"
    benchmark_exposed = False

    @staticmethod
    def _noisy(item_id: str) -> bool:
        # Exactly 24 of 200 items are noisy: 12% deterministic error.
        index = int(item_id.split("-")[-1])
        return index % 25 < 3

    def predict(self, item: Item) -> int:
        return 1 - item.semantic if self._noisy(item.item_id) else item.semantic


class PublicLookup:
    name = "public_lookup"
    benchmark_exposed = True

    def __init__(self, public_items: Iterable[Item]) -> None:
        self.lookup = {item.item_id: item.label for item in public_items}

    def predict(self, item: Item) -> int:
        # Unknown fresh items fall back to the visible cue.
        return self.lookup.get(item.item_id, item.cue)


def make_sets(n: int = 200) -> dict[str, list[Item]]:
    standard: list[Item] = []
    cue_neutral: list[Item] = []
    fresh: list[Item] = []
    counterfactual: list[Item] = []

    for i in range(n):
        y = i % 2
        public_id = f"public-{i:03d}"
        fresh_id = f"fresh-{i:03d}"

        # The public test leaks a cue on 95% of items.
        standard_cue = 1 - y if i % 20 == 0 else y
        # This cue is balanced and independent of the label.
        neutral_cue = (i // 2) % 2

        standard.append(Item(public_id, semantic=y, cue=standard_cue, label=y))
        cue_neutral.append(Item(public_id, semantic=y, cue=neutral_cue, label=y))
        fresh.append(Item(fresh_id, semantic=y, cue=neutral_cue, label=y))

        # Hold the cue and identity fixed, but reverse the construct-relevant fact.
        counterfactual.append(
            Item(public_id, semantic=1 - y, cue=standard_cue, label=1 - y)
        )

    return {
        "standard": standard,
        "cue_neutral": cue_neutral,
        "fresh": fresh,
        "counterfactual": counterfactual,
    }


def accuracy(system: System, items: list[Item]) -> float:
    return sum(system.predict(item) == item.label for item in items) / len(items)


def flip_rate(system: System, left: list[Item], right: list[Item]) -> float:
    pairs = zip(left, right, strict=True)
    return sum(system.predict(a) != system.predict(b) for a, b in pairs) / len(left)


def wilson_interval(successes: int, n: int, z: float = 1.96) -> tuple[float, float]:
    p = successes / n
    denominator = 1 + z * z / n
    centre = (p + z * z / (2 * n)) / denominator
    half = z * sqrt((p * (1 - p) + z * z / (4 * n)) / n) / denominator
    return centre - half, centre + half


@dataclass(frozen=True)
class AuditResult:
    system: str
    standard_accuracy: float
    standard_95pc_interval: tuple[float, float]
    cue_neutral_accuracy: float
    fresh_accuracy: float
    counterfactual_flip_rate: float
    nuisance_drop: float
    benchmark_exposed: bool
    capability_claim: str


def audit(system: System, sets: dict[str, list[Item]]) -> AuditResult:
    standard = accuracy(system, sets["standard"])
    neutral = accuracy(system, sets["cue_neutral"])
    fresh = accuracy(system, sets["fresh"])
    sensitivity = flip_rate(system, sets["standard"], sets["counterfactual"])
    hits = round(standard * len(sets["standard"]))

    checks = {
        "fresh": fresh >= 0.75,
        "nuisance_invariance": standard - neutral <= 0.10,
        "construct_sensitivity": sensitivity >= 0.80,
        "no_direct_exposure": not system.benchmark_exposed,
    }
    failed = [name for name, passed in checks.items() if not passed]
    claim = "eligible within tested scope" if not failed else "blocked: " + ", ".join(failed)

    return AuditResult(
        system=system.name,
        standard_accuracy=round(standard, 3),
        standard_95pc_interval=tuple(round(v, 3) for v in wilson_interval(hits, len(sets["standard"]))),
        cue_neutral_accuracy=round(neutral, 3),
        fresh_accuracy=round(fresh, 3),
        counterfactual_flip_rate=round(sensitivity, 3),
        nuisance_drop=round(standard - neutral, 3),
        benchmark_exposed=system.benchmark_exposed,
        capability_claim=claim,
    )


def main() -> None:
    sets = make_sets()
    systems: list[System] = [CueReader(), TargetRule(), PublicLookup(sets["standard"])]
    results = [audit(system, sets) for system in systems]

    # Positive case: intended rule survives all discriminating tests.
    target = next(result for result in results if result.system == "target_rule")
    assert target.capability_claim == "eligible within tested scope"
    assert target.standard_accuracy == target.cue_neutral_accuracy == target.fresh_accuracy == 0.88
    assert target.counterfactual_flip_rate == 1.0

    # Negative controls: high public scores do not authorise the capability claim.
    cue = next(result for result in results if result.system == "cue_reader")
    lookup = next(result for result in results if result.system == "public_lookup")
    assert cue.standard_accuracy == 0.95 and cue.cue_neutral_accuracy == 0.5
    assert lookup.standard_accuracy == 1.0 and lookup.fresh_accuracy == 0.5
    assert cue.capability_claim.startswith("blocked")
    assert lookup.capability_claim.startswith("blocked")

    print(json.dumps([asdict(result) for result in results], indent=2))


if __name__ == "__main__":
    main()

A ten-question benchmark construct audit

1. What is the construct? Define the capability positively, list its components, and state what does not count. “Reasoning” without a theory of the required relation is not yet a construct.
2. What decision will use the score? Research comparison, model routing, procurement and deployment approval require different evidence and loss functions.
3. What is the candidate system? Record model, prompt, tools, retrieval, memory, allowed retries, resource limits and human help. A configuration change changes the measured subject.
4. How was the task space sampled? Publish a blueprint, strata, exclusions, prevalence assumptions and item-generation process. Show where the benchmark underrepresents the claimed environment.
5. Which nuisance abilities move the score? Test knowledge, language, format, latency, tool setup and judge dependence separately where they are not part of the target.
6. Which wrong mechanisms can succeed? Build cue-only, lookup, popularity, fixed-policy or question-only baselines that are capable of winning visibly.
7. What should stay invariant and what should change? Pre-register nuisance transformations and construct counterfactuals, then compare the observed pattern with each rival mechanism.
8. Is the scorer measuring the desired response? Validate exact match, execution tests, human rubrics or model judges against independently adjudicated cases. Preserve disagreement rather than hiding it in one mean.
9. How uncertain and fragile is the ranking? Report intervals, paired tests, repeated runs, subgroup results, task-weight sensitivity and implementation versions.
10. Does the result predict the intended outcome? Connect benchmark performance to a fresh simulation, shadow workflow or bounded field outcome. If that bridge is absent, keep the claim at task performance.

What would weaken this argument?

The intervention-heavy approach adds design and execution cost. It would be less necessary if simple static benchmark scores consistently predicted independently measured operational outcomes across model families, prompts, time and task mixtures. Stable rank under fresh sampling, nuisance transformations and alternate scorers would show that the cheap proxy already captures the useful variance.

Evidence of that kind should be welcomed. The proposal is not that every benchmark needs a laboratory of infinite counterfactuals. It is that the burden of validation rises with the breadth and consequence of the claim. A narrow internal regression test may need only reliability and change detection. A claim about general reasoning, safety or autonomous work needs much more.

The failure boundary

No audit can enumerate every shortcut. Generative systems may exploit regularities the evaluator has not imagined. Fresh items can still be near-duplicates in representation space. Process traces may be post-hoc. Operational outcomes can be confounded by human adaptation and workflow design. The construct itself may be contested or value-laden.

Construct validity therefore never becomes an all-clear certificate. It is a living assurance claim with evidence, known defeaters, scope and expiry. Weidinger and colleagues argue that generative-AI evaluation needs iterative refinement and stronger institutions, not one-off static tests. Dynamic benchmarks such as Dynabench address some saturation and adaptivity problems, but dynamic difficulty does not automatically guarantee construct clarity. Harder is not the same as more valid.

The practical endpoint is modest. An audit should leave the evaluator able to say which causal routes were tested, which remain open, what population and conditions were sampled, and what decision the evidence can support today.

A leaderboard is a compact record of performance under an evaluation procedure. It becomes dangerous only when the name above the column claims more than the procedure has earned.

The repair is not to abandon benchmarks. Shared tests make progress visible, expose regressions and focus research. The repair is to treat every broad label as a causal hypothesis. Define the construct. Sample its task space. Build controls that can succeed for the wrong reason. Demand stability under nuisance changes and sensitivity to construct changes. Test fresh items. Validate the scorer. Report uncertainty. Connect the score to the decision it is meant to inform.

The architecture decision changed by this argument is simple: do not select or release a system from aggregate rank alone. Require a construct-validity card beside the score, and let failed gates narrow the claim. A model that scores 95 per cent through a cue may still be useful for that dataset. A model that scores 88 per cent through the intended relation may be the stronger candidate. A perfect public lookup remains a negative control, not evidence of capability.

The most defensible benchmark is not the one that makes the largest claim. It is the one that shows exactly why its score deserves the smaller claim it makes.