The rule inside the score

Learning selects a rule from many rules that fit the observations. Generalisation depends on which one was selected and where it must travel.

A test set answers a conditional question

Let a model receive an input x, produce a prediction f(x), and incur loss when that prediction disagrees with a label y. Training minimises average loss on observed examples. A conventional test set asks whether performance remains similar on unseen examples drawn by roughly the same process.

Under independent and identically distributed sampling, this is a powerful question. The test result estimates expected loss for the population represented by that process. Classical learning theory studies how sample size, model capacity and the learning algorithm constrain the gap between empirical and expected risk. Those tools remain useful.

But the guarantee is conditional. It depends on the test population, the sampling unit, the label process and the loss. Change any of those, and a new question appears. “Unseen” does not mean “unlike training”. It often means only “not previously sampled”.

Three transfers are often folded into one score. The first is from a finite sample to the source population that could have produced it. The second is from that source population to the environments in which the system will be used. The third is from predictive performance to the consequences of a decision policy. A random hold-out is designed mainly for the first transfer. It provides no automatic bridge for the other two.

This distinction matters even when the statistics are impeccable. Suppose a test set contains ten thousand independently sampled parcels from the original warehouse. Its confidence interval may be narrow, and the estimate may be unbiased for that warehouse. None of this tells us whether sticker policy, lighting or camera placement can change. Precision about the wrong population is still precision.

Call the difference sample novelty versus environment novelty. Sample novelty asks whether the model has seen this particular row. Environment novelty asks whether the relation carrying the prediction has been challenged. The two can coincide, but they need not. A new photograph can be environmentally familiar. An edited copy of an old photograph can be environmentally novel if the edit breaks a suspected shortcut while preserving the target.

Capacity does not tell the learner which rule you meant

A small model can overfit, and a large model can generalise. Model size alone does not settle the matter. Deep networks can fit random labels, showing that successful optimisation does not by itself explain why the learned rule works beyond the sample.^[1] Modern test error can also follow non-classical curves as capacity increases, so a simple “larger means more overfit” story is unreliable.^[2]

The important question is selection among fitting solutions. Data geometry, architecture, optimisation, regularisation and preprocessing all bias that selection. A model may prefer a visually simple texture, a frequent token, a site identifier or a proxy variable because it is easier to exploit than the structure a human considers relevant.

Shortcut learning names this pattern: a decision rule performs well under standard conditions but fails when the shortcut no longer tracks the intended target.^[3] Underspecification names a related pipeline-level problem: several models satisfy the observed requirements yet behave differently in deployment-relevant conditions.^[4] These are not exotic corner cases. They are the normal consequence of evidence that leaves multiple rules alive.

flowchart TD
  A["Opening case"] --> B["The rule inside the score"]
  B --> C["Generalisation is transport<br/>across change"]
  C --> D["Make the hidden rule visible"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#D95545,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Generalisation is transport across change

A model does not generalise “in general”. It transports performance from one set of conditions to another, under assumptions about what may change and what should remain stable.

Name the environments before measuring the gap

Let e denote an environment: a site, time period, acquisition process, demographic mixture, interface, policy regime or intervention. Each environment induces a distribution over inputs and labels. The risk of model f in environment e is:

X is the observed input, Y the target, L the cost of a prediction and P_e the data-generating distribution in environment e. A random split estimates risk for an environment similar to the source. A broader task might care about average risk across a deployment mixture, worst-group risk, maximum risk over named environments or cost after an action policy.

The task is partly encoded in the risk functional. Average accuracy says frequent cases matter in proportion to their frequency. Worst-group accuracy says no declared group may be sacrificed by aggregation. A cost-sensitive loss says some errors are more consequential than others. None is universally correct. Each states a decision.

Take two environments. Model A has losses of 4 per cent and 24 per cent. Model B has losses of 12 per cent and 13 per cent. If the first environment supplies 95 per cent of cases, Model A has lower average loss. If either environment must remain safe, Model B has lower maximum loss. The models have not changed. The preferred model changes because the decision rule changed.

This small example exposes a common reporting error. Teams select a model using average validation accuracy, then describe the result as robust because they also display subgroup metrics. Display is not selection. If the release decision refuses to sacrifice an environment, the selection objective or gate must encode that refusal.

The task has shape before it has data

A useful task description has at least five geometric features. It has a centre, the environments expected most often. It has an extent, the range of authorised variation. It has seams, where subgroups or sites meet. It has cliffs, where small input changes create large decision costs. It has moving ground, where labels or behaviour change after deployment.

Dataset design samples this shape. A random split samples densely near the centre. Leave-one-site-out evaluation probes a seam. A temporal hold-out follows moving ground. Counterfactual interventions approach a suspected cliff from both sides. No single split covers all of them, and adding every possible test is neither feasible nor necessary. The point is to align the sampled geometry with the decision.

The shape also limits what can be learned. If every training environment couples the same shortcut to the target, there is no observational evidence that the coupling is dispensable. An algorithm may still avoid the shortcut because of prior structure, pretraining or regularisation, but the dataset did not identify that choice. The claim should then acknowledge reliance on those priors.

The unit of independence is part of the task

Consider medical images from repeated visits, audio clips from the same speaker, satellite tiles from adjacent coordinates or documents generated from the same template. If related observations are split across training and test sets, the nominally unseen examples may share identity, context or acquisition artefacts with training data. The split is independent at the row level but not at the decision-unit level.

This is more than “data leakage” in the narrow sense of copying the answer. It is a boundary error. The project has not decided what must be new at deployment. If the future decision concerns a new patient, patient identity should define the split. If it concerns a new hospital, hospital should define at least one stress environment. If it concerns future behaviour, time should move in one direction.

A realistic task-shape audit

Consider a crop-disease classifier intended for field use. The raw input is a leaf photograph. The target is a disease state confirmed by an agreed diagnostic process. Training data arrive from three farms, each using a different camera and collection tray. Disease prevalence also differs by farm.

A random image split allows the model to use tray colour, camera processing or farm-specific background as proxies. A plant-level split prevents near-duplicate leaves from crossing the boundary. A farm-held-out split asks whether performance travels to a new collection environment. A time-held-out split tests seasonal change. A counterfactual edit that replaces the tray while preserving the leaf tests a particular shortcut. These tests do not compete. They correspond to different deployment changes.

A configured evaluation might therefore contain four datasets: an independent in-environment test set, a held-out farm, a later season and a synthetic tray intervention reviewed for label preservation. It would report average accuracy, per-farm accuracy, calibration and abstention. The release question becomes concrete: does the system meet the minimum risk across the environments it is authorised to face?

The audit should also follow errors into action. A false negative may delay field treatment; a false positive may trigger unnecessary spraying. If uncertain cases are routed to an agronomist, the relevant outcome is not raw classifier accuracy alone. It is residual error after abstention, reviewer workload and the time required to obtain a decision. Generalisation of a score can differ from generalisation of the configured policy.

Now imagine the new farm uses a camera outside the authorised range. Monitoring detects a change in colour response and increases abstention. This is not evidence that the model has generalised to the new device. It is evidence that the system recognised the edge of its envelope and failed safely. Detection and recovery are legitimate responses to unknown shift; confident prediction is not the only form of competence.

Benchmarks such as WILDS were created because real applications face shifts across hospitals, locations, time and other settings that ordinary datasets underrepresent.^[5] Replicated test sets for major image benchmarks have also produced material accuracy drops even when collection procedures were carefully matched.^[6] These findings do not imply that every benchmark is worthless. They show that the benchmark’s sampling route is part of the result.

Make the hidden rule visible

A stress test is useful when it separates rival rules. Random corruption is not enough. The change must preserve what should matter and alter what should not.

Cross the target with the suspected nuisance

A useful stress test begins with rival explanations. Write the intended rule and at least one shortcut rule in terms that could produce predictions. Then ask what change would make them disagree. This reverses the usual sequence, in which a team adds generic noise and hopes that robustness means something.

The intervention needs a preservation argument. Identify the target property that should remain unchanged, the nuisance property that will change, the method used to create or collect the case, and the evidence that the label is still valid. For images, a background edit can alter shadows or occlusion. For text, paraphrasing can change pragmatics. For tabular data, reweighting can create combinations that are impossible in the real process. A stress set that corrupts the target tests tolerance to damage, not invariance to nuisance.

Suppose the target is object shape and the suspected nuisance is background colour. The minimum revealing design contains all four combinations: target A with background 1, target A with background 2, target B with background 1, and target B with background 2. If training contains mostly the diagonal combinations, the off-diagonal cases become decisive.

This simple cross does three jobs. It shows whether the model has observed each combination. It measures the dependence of predictions on the nuisance. It also reveals an evaluation gap before a model is trained. If the project cannot construct or collect the off-diagonal cases, it should lower the strength of its generalisation claim.

Use paired cases when possible. Start from one instance, change only the suspected nuisance, and compare the prediction. Paired analysis reduces unrelated variation and supports a clearer causal interpretation. Then complement it with naturally collected environments, because synthetic edits can introduce artefacts of their own. Agreement between controlled and natural tests is stronger than either alone.

Finally, predefine the expected signature. A target-sensitive model should keep its class when only the nuisance changes. A shortcut-sensitive model should follow the nuisance. A model that abstains may be behaving appropriately if the edit moves the input outside its valid support. Without this prediction ledger, surprising results are too easy to explain after the fact.

The smallest useful numerical example

Take 100 training examples. In 95, the shortcut agrees with the label. The stable feature is noisier and agrees in 80. A simple learner assigns weight 1.8 to the shortcut and 0.4 to the stable feature. On an identically sampled test set, it scores near the shortcut’s reliability.

Now create a shift set in which the stable feature is unchanged but the shortcut agrees only 10 per cent of the time. A prediction score of 0.4 × stable + 1.8 × shortcut is dominated by the shortcut and reverses on many cases. A stable-only score of 1.0 × stable keeps roughly the same accuracy. The exact values depend on noise and optimisation. The causal lesson does not: the original split rewarded a dependency that the deployment change broke.

Notice the assumption. We treated the stable feature’s relation to the label as invariant. If the shift also damages that feature, the stable-only model may fail too. Stress testing is not an oracle. It is a way to make assumptions executable.

Build a task-shape contract

Before choosing a robustness method, write down the task’s change-set. A change-set is the collection of environment changes the system is expected or authorised to face. It need not contain every imaginable future. It should contain the plausible, consequential and testable changes that matter to the decision.

The following instrument turns that idea into a project artefact. It is deliberately stricter than a dataset card and narrower than a full safety case. Its purpose is to connect task definition, split design, stress tests and release criteria.

Complete the contract before the final model exists. Otherwise, the observed strengths of a favoured model tend to redefine the task. A system that performs poorly on one site may suddenly be described as “not intended” for that site, even when the site was part of the original need. Versioning the contract makes scope changes visible and reviewable.

The contract should also name unknowns. Some changes are plausible but not yet testable. Record them as unsupported environments rather than assigning a speculative score. This creates a research queue: collect data, build a simulator, design a proxy or constrain use until evidence exists.

Algorithms help only after the task is shaped

Invariant Risk Minimisation proposes learning a representation for which one classifier is optimal across training environments.^[7] Group distributionally robust optimisation instead minimises the worst loss over predefined groups.^[8] Both encode valuable ideas: seek relationships that persist, and do not let average performance hide a failing group.

Neither removes the need to define environments and groups. An invariance penalty cannot discover every relevant shift from one homogeneous environment. Group robustness cannot protect an unlabelled or absent group. Practical IRM formulations can fail to recover intuitive invariances, and theoretical critiques identify conditions under which the objective can select undesirable solutions.^[9] Broad domain-generalisation comparisons have also found that carefully tuned empirical risk minimisation remains a strong baseline and many specialised methods do not consistently outperform it.^[10]

The decision is therefore not “ERM or invariance”. It is a sequence. Declare the change-set. Construct environments that make rival rules disagree. Fit a strong baseline. Compare algorithms using a selection rule aligned to deployment risk. Preserve a hidden challenge set so repeated tuning does not turn the stress test into another familiar benchmark.

Data interventions often dominate objective changes. Rebalancing groups, collecting minority combinations, removing a proxy field or changing acquisition can make the intended rule easier to identify. Algorithmic penalties are most credible when they operate on environments whose differences are meaningful. A mathematically elegant penalty over arbitrary partitions may optimise stability that has no deployment interpretation.

Model inspection can help generate hypotheses, but it should not close the case. Feature attribution, probes and saliency may suggest a shortcut. The release evidence should include behavioural interventions that alter the suspected feature, because readable internal patterns need not be causally used. Conversely, a failed explanation method does not prove the model is robust. The test remains tied to predictions under change.

stateDiagram-v2
  [*] --> Claim
  state "A test set answers a<br/>conditional question" as Claim
  state "Capacity does not tell the<br/>learner which rule you meant" as Test
  state "Name the environments before<br/>measuring the gap" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

The train-test-shift lab

The lab below makes the hidden rule observable. It creates one stable feature, one environment-specific shortcut and three evaluation worlds.

What the experiment tests

Each synthetic example has a binary label. The stable feature is a noisy measurement of that label and keeps the same relationship across environments. The shortcut feature is cleaner in training but its reliability can change. Ordinary ERM fits both features using only the training environment. A stress-selection procedure fits several candidate feature sets and regularisation strengths, then selects the candidate with the best worst accuracy across declared stress environments.

The experiment encodes a strong assumption: the stable feature truly remains stable and the specified stress environments resemble the changes we care about. A positive result permits a narrow conclusion. It supports lower shortcut dependence under those synthetic shifts. It does not prove causal correctness, general robustness or readiness for an unrelated real system.

There are three candidate feature families: stable only, shortcut only and both. Each is fitted with several regularisation strengths. Stress-selection chooses the candidate with the highest minimum accuracy across the familiar, neutral and reversed environments. This is a miniature form of model selection under a maximum named-environment risk rule.

The stable-only candidate is not privileged by code. It wins only when its worst-environment result exceeds the alternatives. Increase stable noise far enough and no candidate performs well. Make the shift mild and the combined model may remain competitive. These outcomes matter because they prevent the story from becoming “always discard the easy feature”. The correct rule is to select evidence that survives the declared changes.

How to read a positive or negative result

When ERM remains strong on the familiar test but collapses on the reversed shortcut, the experiment demonstrates an evaluation blind spot. When stress-selection chooses the stable feature and preserves shifted accuracy, it demonstrates that the declared environment variation was sufficient to change model selection.

A negative result has several meanings. The stable feature may be too noisy. The candidate model family may not contain a stable solution. The stress set may be too small or may not isolate the shortcut. The alleged invariant may be false. The proper response is diagnosis, not relabelling the failed model as generally incapable.

The same logic scales to real systems. Replace synthetic shortcut reliability with environments defined by site, time, device, subgroup or a validated counterfactual generator. Replace feature subsets with candidate architectures, data strategies or checkpoints. Preserve the selection discipline: choose using the risk functional in the task-shape contract, then evaluate once on a protected challenge set.

A practical run has six receipts. The data manifest states which unit is independent and how environments were assigned. The model manifest records architecture, preprocessing and training configuration. The stress manifest records every intervention and its label-preservation review. The selection receipt shows the metric used to choose the candidate. The effect receipt stores per-environment results with uncertainty. The release receipt names thresholds, unsupported environments and monitoring triggers.

These receipts prevent a common collapse of meaning. Without them, “test accuracy” can refer to the random hold-out in one report, the stress set in another and a post-threshold operational metric in a third. Typed evidence keeps the journey attached to the number.

from dataclasses import dataclass
import numpy as np

@dataclass(frozen=True)
class Env:
    name: str
    shortcut_p: float
    n: int
    seed: int

def make_env(spec, stable_noise=1.15, shortcut_noise=0.20):
    rng = np.random.default_rng(spec.seed)
    y = rng.choice([-1.0, 1.0], size=spec.n)
    stable = y + rng.normal(0, stable_noise, spec.n)
    agrees = rng.random(spec.n) < spec.shortcut_p
    shortcut = np.where(agrees, y, -y) + rng.normal(0, shortcut_noise, spec.n)
    return np.column_stack([stable, shortcut]), y

def fit_logistic(x, y, l2=0.01, steps=1200, lr=0.08):
    xb = np.column_stack([np.ones(len(x)), x])
    w = np.zeros(xb.shape[1])
    for _ in range(steps):
        margin = np.clip(y * (xb @ w), -40, 40)
        grad = xb.T @ (-y / (1 + np.exp(margin))) / len(xb)
        grad[1:] += l2 * w[1:]
        w -= lr * grad
    return w

def accuracy(w, x, y):
    xb = np.column_stack([np.ones(len(x)), x])
    return np.mean(np.where(xb @ w >= 0, 1.0, -1.0) == y)

# Complete runnable version, candidate search and typed checks:
# paper33_train_test_shift_lab.py

Run python paper33_train_test_shift_lab.py with NumPy installed. The complete file searches stable-only, shortcut-only and combined candidates across regularisation settings.

The stress test can become another shortcut

Repeatedly tuning against one public shift set turns that set into part of the training loop. Teams learn its quirks, choose augmentations that target it and report progress as though the evaluation remained independent. This is adaptive overfitting at the benchmark level.

The remedy is not permanent secrecy. It is layered evidence. Keep some stress generators visible so teams can reason about failure. Preserve a protected challenge set for model selection audits. Add prospective data after release. Rotate tests when the system, environment or decision policy changes. A mature programme evaluates the process that produces models, not only one frozen checkpoint.

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: Opening case
  M->>E: A test set answers a conditional question
  E-->>M: compare rival explanations
  E->>D: Define the task terrain before celebrating the score
  D-->>Q: revise scope or act

Define the task terrain before celebrating the score

Stop asking whether a model generalises before asking where its performance must travel.

The standard workflow begins with a dataset, chooses a random split, trains several models and reports the best test score. The revised workflow begins with the decision. It names the unit that must be new, the target meaning, the changes the system may face and the relationships expected to survive. Only then does it design splits and stress tests.

This does not abolish ordinary generalisation testing. The random hold-out remains the right instrument for estimating performance on new samples from the source process. The change is one of scope. Keep that estimate, then add only the routes needed by the intended use. A narrow, well-supported envelope is more useful than a broad claim whose assumptions are invisible.

This changes model selection. A candidate is no longer preferred merely because it wins on the familiar distribution. It must satisfy the risk rule across the declared generalisation envelope. A simpler model may be chosen because its performance is stable. A larger model may still win if it demonstrates both higher average performance and acceptable shift behaviour. The decision follows evidence, not a prior preference for simplicity or scale.

It also changes data collection. When a shortcut is suspected, the first response should not always be a new loss function. Collect the missing combinations. Add environments that vary the nuisance while preserving the target. Hold out the unit that will truly be new. Record the configured pipeline that creates the model’s input.

Finally, it changes language. Say “the model generalised from these farms to this held-out farm”, “remained calibrated across these devices” or “survived this validated background intervention”. Do not convert those statements into “the model generalises” without qualification.

A benchmark score is a receipt for one journey through the task terrain. The responsible decision is to define the journey before celebrating the arrival.