A score is an effect, not a property

Imagine a thermometer that also changes with room lighting. Its reading may be repeatable. It may even correlate with temperature in the laboratory. Yet a warm-coloured lamp can move the needle while the air remains unchanged. The instrument is not useless. Its reading simply cannot carry the intended interpretation until the extra cause is understood.

Benchmarks pose the same problem at a higher level. “Reasoning”, “factuality”, “planning”, “coding ability” and “safety” are not columns directly present in a dataset. They are constructs: theoretical properties invoked to explain patterned performance. A benchmark supplies tasks, an interface and a scoring rule. Researchers then infer from the observed result to the construct.

Cronbach and Meehl framed construct validation around a deceptively simple question: what constructs account for variance in test performance?^[1] Messick later centred validity on the inferences drawn from responses and performances, rather than only on a test form.^[2] Borsboom and colleagues offered a more explicitly causal alternative: a test measures an attribute when the attribute exists and variation in it causes variation in the measurement outcome.^[3] These traditions disagree about the best theory of validity, but they converge on one practical warning. A score does not interpret itself.

Five layers that are often collapsed

Consider the claim “Model X has strong mathematical reasoning”. At least five layers sit between the phrase and the score. Collapsing them creates an illusion of direct measurement.

A benchmark may represent one layer well and another badly. A clean exact-match metric does not rescue an unrepresentative task family. A realistic task does not rescue a judge that prefers verbose answers. A broad dataset does not rescue an undefined capability claim. Validity is therefore claim-specific and use-specific. The same result can validly support “the configured system achieved 82 per cent under protocol P” while failing to support “the base model can reason”.

Reliability is not the rescue

Suppose the panel repeats ReasonGrid ten times with deterministic decoding and obtains 91 every time. The result is highly repeatable. That is useful because random score movement would make interpretation almost impossible. Yet repeatability does not tell us which cause is being repeated. The same prompt, cue structure, exposed items and judge can reproduce the same causal mixture with great precision.

A bathroom scale can return the same number whenever a person wears a heavy coat. Its reliability improves our confidence that the device is responding consistently. It does not establish that the reading isolates body mass. In benchmark terms, reliability mainly constrains uncontrolled variation in U. It does not remove contributions from S, I, E or J.

The distinction matters because three questions are often compressed into one. Does the candidate obtain a similar score when the identical protocol is repeated? Does its rank remain similar when items are resampled from the same construction process? Does the score support the intended capability interpretation under changes that should be irrelevant? The first concerns repeatability, the second sampling robustness, and the third construct validity. Evidence for one does not automatically answer the others.

High reliability can make a confounded measurement look more authoritative. The remedy is not to value reliability less. It is to demand reliability at each condition and then examine the pattern across conditions.

flowchart LR
  A["A score is an effect, not a<br/>property"] --> B["Reveal the route, not just the<br/>arrival"]
  B --> C["Make the capability travel"]
  C --> D["Build an evidence case instead<br/>of a leaderboard"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#007D68,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Reveal the route, not just the arrival

A benchmark usually records whether the candidate arrived at an accepted answer. Capability claims often concern the route: infer a rule, construct a plan, retrieve a fact, revise a belief, or repair a programme. Several routes can reach the same answer. Some instantiate the target demand. Others bypass it.

This does not make every shortcut illegitimate. Using a lookup table is an excellent strategy when the target is rapid retrieval. Eliminating options is part of competent multiple-choice test taking. Calling a tool can be central to a system-level coding task. The error occurs when a route that the claim excludes is allowed to dominate the score.

Four ways the route can hide

Item defects can change the target after the fact. Gema and colleagues manually re-annotated 5,700 MMLU questions across all 57 subjects and estimated that 6.49 per cent of the full benchmark contained errors. Re-evaluation on their corrected subset changed reported performance and rankings.^[10] A wrong key is not model weakness. An ambiguous question may reward guessing style rather than knowledge. Dataset quality is therefore part of the measurement bridge, not clerical housekeeping.

Meaning-preserving interfaces can move scores. Sclar and colleagues varied prompt formatting in few-shot evaluations. Across their tested open models and tasks, they reported roughly ten accuracy points of variation on average and a maximum spread of 76 points for one LLaMA-2-13B setting.^[9] The exact magnitude should not be transferred to every model or prompt regime. The causal lesson is narrower: a single prompt format can conceal substantial interface dependence, and model rankings need not be stable across formats.

Prior exposure can imitate generalisation. Li and colleagues used public web sources as a proxy for potential benchmark exposure across more than fifteen models and six multiple-choice benchmarks. Their method reported varying contamination estimates and, in some settings, measurable score inflation.^[11] Their proxy cannot prove which examples entered a particular proprietary training run, so the finding is evidence of risk rather than a universal contamination verdict. Still, a public static benchmark invites a generator that partly relies on familiarity.

The evaluator can become a second model under test. In an analysis exceeding 150,000 judgement instances, Shi and colleagues found systematic position bias across LLM judges, with variation across judges, tasks and candidate quality gaps.^[12] If swapping answer order changes the winner, the score includes judge behaviour. An apparently open-ended capability benchmark can therefore reduce to a compound contest between generator style and evaluator preference.

The Lumen tile trap

Suppose a benchmark claims to measure rule induction. Each tile has a colour, a shape and a number of dots. The true rule is simple: a tile belongs to class Lumen when it has an even number of dots. In the training examples, every even tile happens to be blue and every odd tile coral. Colour is easier to detect than parity.

On the canonical test, blue still predicts the label. A parity-based candidate and a colour-based candidate both score perfectly. Now reverse the colour association while preserving dot parity, item count and class balance. The parity candidate continues. The colour candidate fails. The second test does not prove that the first candidate possesses a general faculty of reasoning. It establishes a smaller negative result: the named colour shortcut is insufficient to explain its performance.

The Lumen example also shows why benchmark repair cannot stop at “add harder questions”. Harder items may amplify reading load, token length or evaluator uncertainty while leaving the original shortcut intact. The design goal is not difficulty for its own sake. It is causal separation.

Use positive and negative controls

A good intervention programme asks for two different responses. A negative control changes something outside the construct and predicts stability. A positive control changes a property inside the construct and predicts movement. Measuring only invariance is dangerous because a candidate can look stable when the test is too easy, the score is saturated or the manipulation is too weak.

Return to the Lumen tiles. Reversing colour while preserving dot parity is a negative control for the parity construct. It should not reduce a genuine parity strategy. Now add a positive control: instead of counting visible dots directly, ask whether the total remains even after one dot is added and another removed according to a written instruction. This introduces an extra transformation that belongs to the intended rule-composition demand. Performance should change in a predictable direction as the transformation becomes harder.

The four possible patterns are informative. Stability on the negative control and sensitivity to the positive control supports selective responsiveness. Movement on both suggests that the nuisance variable still matters or the negative control was not construct-preserving. Stability on both suggests a ceiling, memorisation or a control that failed to engage the target. Movement only on the negative control suggests that the score is carried by the wrong property.

Construct evidence requires selective sensitivity, not maximum invariance. Before running the test, write down which interventions should leave the score unchanged, which should lower it, and how much uncertainty would make the pattern undecidable. This turns post-hoc storytelling into a risky prediction.

Make the capability travel

A capability claim becomes more credible when performance travels across changes that should be irrelevant to the construct and responds to changes that should matter. This sounds obvious, but it changes benchmark design. Instead of expanding one item pool and averaging harder, we create controlled variation around the score.

The proposal here is a Construct Transport Audit. It is a practitioner synthesis, not an established psychometric standard. Its smallest form is a two-by-two design that varies two causal properties: whether a named shortcut is available and whether the interface is familiar.

Read the four cells as interventions

Call the scores A, B, C and D, following Figure 5. Assume that higher is better and all values use the same scale. The average shortcut effect compares cells where the shortcut is available with cells where it is blocked:

The average interface effect compares the familiar interface with the altered interface:

The interaction asks whether the interface penalty changes when the shortcut disappears:

Finally, define the transport floor as the lower score across the two cells where the named shortcut is blocked:

For Cedar, the shortcut effect is five points, the interface effect is three, the interaction is two, and the transport floor is 78. For Quartz, the corresponding values are 25, 20, 26 and 47. Quartz still has evidence of competence under the canonical protocol. What disappears is the broad inference that its higher score reflects more of the target capability. The transport floor is not a latent ability estimate; it is a scoped lower bound inside the tested intervention space.

The equations are valid only if the manipulations do what their labels claim. If the altered interface is harder to parse, ΔI mixes nuisance with genuine difficulty. If shortcut blocking changes class balance or introduces rare vocabulary, ΔS is confounded. The design therefore needs item-level review, counterbalancing and uncertainty intervals. In a serious study, bootstrap by item family rather than treating near-duplicate variants as independent observations.

Executable lab: the Construct Transport Audit

The lab below accepts four observed scores for two candidates. It computes the shortcut effect, interface effect, interaction and transport floor. The default values reproduce Cedar and Quartz. Replace them with scores from a balanced four-cell evaluation.

How to use the artefact

Start with one named shortcut and one interface variation. Build matched item families, then run the same configured candidate under all four conditions. A positive result means the candidate meets your floor and stays within your nuisance tolerance in this bounded audit. It permits language such as “performance remained stable when cue X was blocked and format Y was changed”. It does not permit “the system possesses general reasoning”.

A negative result does not prove the target capability is absent. It says the current score cannot separate that capability from the manipulated route, or that the candidate cannot express the capability through the tested interface. Redesign the task, narrow the claim, modify the system, or decide that the dependency is acceptable in the target environment.

The audit cannot establish internal mechanism, real-world safety, consciousness, broad domain transfer or absence of every shortcut. It is strongest as a falsification instrument for specific rival explanations. Repeat it across independent task families before making a wider claim.

from dataclasses import dataclass
from typing import Literal

Grade = Literal[
    "transport-supported",
    "promising-but-confounded",
    "insufficient-evidence",
]

@dataclass(frozen=True)
class AuditResult:
    canonical: float
    shortcut_effect: float
    interface_effect: float
    interaction: float
    transport_floor: float
    grade: Grade


def audit(
    a: float,
    b: float,
    c: float,
    d: float,
    *,
    minimum_floor: float,
    nuisance_tolerance: float,
) -> AuditResult:
    """Audit four comparable scores where higher is better.

    A: shortcut available, familiar interface
    B: shortcut available, altered interface
    C: shortcut blocked, familiar interface
    D: shortcut blocked, altered interface
    """
    values = (a, b, c, d, minimum_floor, nuisance_tolerance)
    if any(value < 0 or value > 100 for value in values):
        raise ValueError("All scores and thresholds must be in [0, 100].")

    shortcut = ((a + b) - (c + d)) / 2
    interface = ((a + c) - (b + d)) / 2
    interaction = (a - b) - (c - d)
    floor = min(c, d)

    nuisance_ok = (
        abs(shortcut) <= nuisance_tolerance
        and abs(interface) <= nuisance_tolerance
    )
    if floor >= minimum_floor and nuisance_ok:
        grade: Grade = "transport-supported"
    elif floor >= minimum_floor:
        grade = "promising-but-confounded"
    else:
        grade = "insufficient-evidence"

    return AuditResult(a, shortcut, interface, interaction, floor, grade)


cedar = audit(86, 82, 80, 78, minimum_floor=70, nuisance_tolerance=8)
quartz = audit(92, 59, 54, 47, minimum_floor=70, nuisance_tolerance=8)
print(cedar)
print(quartz)

stateDiagram-v2
  [*] --> Claim
  state "Five layers that are often<br/>collapsed" as Claim
  state "The causal inventory" as Test
  state "Reliability is not the rescue" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Build an evidence case instead of a leaderboard

A leaderboard answers a narrow question: who scored more under the published protocol? A capability decision needs a chain of warrants from construct definition to task design, candidate boundary, scoring behaviour and intended use. The chain can be concise, but it must be inspectable.

Recent work shows why this remains unfinished. Bean and colleagues systematically reviewed 445 LLM benchmark papers and found recurring gaps across construct definition, task operationalisation, metric choice and claim interpretation. Only about half of reviewed benchmarks justified why they were valid measures of an important phenomenon, and nearly every paper had a weakness in at least one reviewed area.^[13] Their review has limitations, including venue selection and a constrained review process, but its scale makes the pattern difficult to dismiss.

HELM demonstrated another useful move: evaluate many models across a shared set of scenarios and several metrics rather than letting accuracy erase calibration, robustness, fairness, bias, toxicity and efficiency.^[8] Breadth improves transparency, yet a broader dashboard still needs a construct argument. More columns do not automatically identify which causal route carried each result.

The capability claim packet

The following protocol turns a benchmark result into a scoped evidence case. It is intentionally small enough to use before a research release, procurement decision or system change.

1. Write the claim before choosing the benchmark. Name the property, subject, conditions and decision. “The configured assistant resolves routine refund cases under policy P” is testable. “The model reasons well” is not yet specific enough.
2. Declare the candidate boundary. Version weights, prompt, context, tools, state, decoding, retry policy and judge. Do not compare configured systems while labelling only the base model.
3. Map the construct. List relations that competent performance must preserve, plausible strategies that count, and routes that would be false friends. This creates the intervention targets.
4. Sample several task families. Include convergent tasks that express the construct differently and discriminant tasks that reward adjacent but distinct abilities. A capability should travel across methods without becoming indistinguishable from everything else.
5. Intervene on nuisance routes. Paraphrase, counterbalance cues, alter response format, introduce matched novel items, randomise judge order and block known leakage paths. Predefine which changes should leave performance stable.
6. Audit scoring and uncertainty. Inspect labels, parsers, judge consistency, ceiling effects and clustered items. Report intervals and per-family failures, not only a pooled mean.
7. State the permitted inference. Record what the result supports, what it rules out, what rival remains and which operational decision follows. Version the packet with the benchmark and configured system.

Do not average away the break

An average of 80 can describe two very different systems. One scores between 77 and 83 across every task family. Another scores 100 on familiar factual items and 60 on the planning cases that carry most operational loss. The mean is numerically correct in both cases. Its decision meaning is not.

Report the score as a profile before reducing it to a scalar: task-family performance, transport floor, interface spread, named shortcut effects, calibration, abstention and resource use. Then state the aggregation rule. Equal weighting is a choice, not a neutral default. Use-weighted aggregation is appropriate when frequencies and costs are defensible, but preserve hard minimums for failure modes that cannot be compensated by excellence elsewhere.

This is why a transport floor matters beside an average. The floor asks whether a candidate remains usable at a tested boundary. It should not dominate every decision, because a rare and artificial condition can produce an irrelevant minimum. The answer is to predeclare which conditions represent plausible operating variation and which are diagnostic stress tests.

A capability profile should preserve the failure that changes the decision. Aggregation may summarise evidence after that break remains visible; it should not erase it.

Use language that matches the evidence

This language may appear conservative, but it increases practical usefulness. A release board can act on a bounded statement. An engineer can reproduce a protocol. A researcher can design the next intervention. An inflated label such as “general reasoning” hides the exact condition under which the claim should fail.

The strongest boundary

Construct transport can still be fooled. A candidate may use one broad shortcut that survives every paraphrase and cue reversal. Several task families may share the same source corpus. A judge ensemble may inherit the same preference. Stability can also arise from a ceiling, where all variants are too easy to reveal differences. Invariance across weak interventions is not evidence of independence from stronger rivals.

The response is not an infinite benchmark. Add interventions according to the cost and scope of the decision. Use discriminant tasks that should not correlate if the construct is specific. Add fresh or private items when exposure matters. Where mechanism matters, combine behavioural transport with causal ablations, patches or tool restrictions. Where deployment matters, sample the operational environment and monitor post-release drift.

There is also a legitimate alternative goal: task-specific engineering. A team may care only whether a configured system passes a fixed conformance suite. In that case, broad capability language is unnecessary. The benchmark functions as an acceptance test or regression monitor. It can be valuable without being promoted into a measurement of general intelligence.

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: A score is an effect, not a property
  M->>E: Five layers that are often collapsed
  E-->>M: compare rival explanations
  E->>D: Make the intended capability carry the score
  D-->>Q: revise scope or act

Make the intended capability carry the score

The usual decision is “choose the model with the highest benchmark score”. After a construct-validity audit, the decision becomes more precise: choose the configured system whose performance remains above the required floor when plausible shortcuts are blocked, meaning-preserving interfaces change, and the task portfolio reflects the intended use.

This shifts what a research team publishes. The headline number remains, but it is accompanied by a candidate-system manifest, score spread, named rival generators, transport results, item-quality review, judge checks and the exact inference permitted. A null or fragile result becomes useful. It tells the community which causal route still carries the score.

It also shifts procurement and release authority. A one-point leaderboard gain should not outweigh a twenty-point interface collapse. A model-level claim should not be inferred from a tool-rich agent result. A fixed public test should not authorise a consequential deployment without a representative operational sample.

A benchmark is most valuable when it stops being a trophy and becomes an instrument. Its purpose is not to compress every property into one rank. Its purpose is to make a specific inference vulnerable to the right counterexample. The score earns a capability interpretation only when the intended capability, rather than the convenient test, carries the result.