Three failures behind one score

A generalisation claim always contains an unspoken phrase: “to data that differ in these ways, but not those ways”. A random holdout makes that phrase narrow. It usually asks how a trained predictor performs on another sample from approximately the same data-generating procedure. That is a useful question, but it is not the only one a deployed system must answer.

Underfitting begins before the sample is considered. The chosen features, model family or optimisation procedure cannot represent or reach a rule that captures the stable relation between input and target. A straight line fitted to a strongly curved signal is the familiar picture. More observations locate the wrong straight line more precisely; they do not turn it into the missing curve. In the language of repeated experiments, the centre of the learned predictors remains displaced from the target.

Overfitting concerns sensitivity. A learner uses accidental detail in the finite sample, so a new sample from the same population produces a materially different fitted function. Training error can be tiny while expected error is large. Yet the training gap is only a symptom. A model may interpolate its sample and still generalise well, while another may retain non-zero training error and still be unstable.

Shortcut learning concerns the environment that makes a feature predictive. A shortcut can be learned consistently across samples and seeds. It may therefore have low measured variance and excellent in-distribution accuracy. The rule fails when a plausible intervention changes the nuisance feature without changing what the prediction is supposed to mean. Stability inside one environment is not invariance across environments.

Thought experiment: change only the chalk

Take two copies of the red chalk dataset. In the first, preserve every pixel. In the second, remove the chalk by masking only the marked colour range while leaving the crack texture untouched. Keep the labels, model class, training set size, optimiser and random seeds fixed. If accuracy collapses only in the second experiment, the intervention identifies reliance on chalk more directly than another random split could.

This experiment does not prove that every remaining feature is causal or robust. It answers one narrower question: would the learned decision survive removal of a known nuisance association? That modest scope matters. Generalisation is never certified by one clever stress test. It is accumulated through tests whose interventions correspond to credible changes in acquisition, population, time, policy or operating process.

The first practical distinction is therefore simple: ask whether the proposed remedy changes representational reach, repeated-sample stability, or environmental invariance. Adding parameters changes reach and perhaps optimisation. Adding independent observations changes sampling uncertainty. Rebalancing or intervening on a nuisance feature changes what rules remain successful. Treating them as interchangeable spends evidence without gaining diagnosis.

Bias and variance are repeated-experiment quantities

A single fitted model does not possess a visible blob called bias or variance. Both quantities are defined by imagining the learning procedure repeated. Draw a new training sample, rerun the optimiser, and observe how the prediction changes at the same input. The geometry is a cloud of possible fitted functions. Bias describes where the cloud is centred. Variance describes how widely it spreads.

For the cleanest case, suppose an outcome follows Y = f(X) + ε. Here X is the input, f(X) is the conditional mean we want to predict, and ε is zero-mean irreducible noise with variance σ². A learning algorithm receives a random training set S and returns a predictor h_S. Under squared loss, expected prediction error at a fixed input decomposes as:

The first term is noise that no predictor of Y from X can remove. The second is squared bias: move the average learned prediction away from f(X), and it grows. The third is variance: spread the learned predictions around their average, and it grows. Integrating these quantities over inputs gives population-level terms. If optimisation introduces randomness, the law of total variance can further separate variation due to training samples from variation due to initialisation, minibatch order or other optimiser choices.

Bias is not unfairness, and variance is not uncertainty attached to one prediction. Those meanings exist elsewhere, but here the terms describe a learning procedure under repetition. The decomposition is exact for squared error under its assumptions. Classification error and cross-entropy do not inherit this exact three-term arithmetic, so practitioners should estimate analogous sensitivities rather than force a regression identity onto every metric. Neal and colleagues make both the squared-loss boundary and the sampling-versus-optimisation distinction explicit.

Underfitting is systematic, overfitting is conditional

A high-bias learner makes a similar mistake each time. A high-variance learner makes different mistakes. This is why a larger sample usually attacks variance more directly than bias: repeated datasets become more alike in the information they provide. Conversely, a richer representation can reduce bias by allowing the centre of the fit to move towards the target. It may also increase variance, decrease it, or leave it unchanged, depending on the algorithm and regime.

The familiar U-shaped curve is therefore a possible empirical pattern, not the definition of generalisation. It arises when increasing effective complexity reduces bias but raises variance enough to create an optimum in between. The decomposition itself does not require those terms to move in opposite directions. In neural-network width experiments, Neal et al. observed settings where both estimated bias and variance fell as width increased. The useful question is not whether a model lies on the left or right of a textbook drawing. It is which repeated intervention changes the error.

Thought experiment: add data without changing capacity

Freeze the features, model family, regularisation, optimiser and test distribution. Train once with twenty-four independent examples and again with two hundred and fifty-six. If the average fitted curve barely moves but the fan of fitted curves narrows, the change is chiefly variance reduction. If the centre remains far from the target, more examples have revealed a stable limitation rather than repaired it. If both centre and spread change, the algorithm's effective behaviour depended on the data regime, which is itself a result worth reporting.

This experiment varies one causal feature, sample size, while preserving the model. A capacity sweep performs the complementary intervention. Together they distinguish “too little information for this learner” from “the learner cannot express or reach the needed relation”. Neither test asks whether the relation survives a changed environment. That third axis arrives later.

Minimal worked example: one function, fifteen learning regimes

The executable lab below draws noisy observations from a fixed one-dimensional function, fits Legendre polynomials of increasing degree, and repeats each experiment eighty times. The reported expected test mean squared error is the sum of measured squared bias, prediction variance and known noise variance. The units are squared target units.

With twenty-four examples, degree one is stably wrong, degree three improves the centre, and degree thirteen reacts violently to sample detail. With two hundred and fifty-six examples, the same degree-thirteen family becomes both accurate and stable in this experiment. Raw capacity did not change between the last two rows; the data regime changed the geometry of the selected fits.

flowchart TD
  A["The red chalk model"] --> B["Three failures behind one score"]
  B --> C["Bias and variance are<br/>repeated-experiment quantities"]
  C --> D["Interpolation bends the<br/>capacity axis"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#D95545,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Interpolation bends the capacity axis

The textbook story often stops when a model becomes flexible enough to fit every training example. Beyond that point, zero training error is taken as evidence that the model has memorised noise. Modern practice made this stopping rule untenable. Large neural networks can fit random labels, as Zhang and colleagues demonstrated, yet the same architectures can generalise well on genuine labels. Expressive capacity explains what a model can fit. It does not by itself identify which interpolating solution the training algorithm selects.

Belkin and colleagues described a double-descent pattern: test risk may rise near the interpolation threshold and then fall again as capacity increases further. Nakkiran and colleagues reported related patterns across model size, training time and data regimes, motivating the idea of effective model complexity rather than raw parameter count. These are empirical and theoretical patterns under particular tasks and algorithms, not a replacement curve guaranteed to appear everywhere.

Three meanings of complexity

Raw capacity describes the size of the available function class, perhaps through parameter count, depth, degree or another proxy. Effective complexity describes which behaviours the entire training procedure can realise at a given stage and data regime. Selected-solution complexity describes the particular interpolating rule found by optimisation, including any explicit or implicit preference for small norms, smoothness, margins or sparse features.

Confusing these quantities creates contradictory advice. A parameter-rich model can implement both a jagged memoriser and a smooth interpolant. Gradient-based optimisation, initialisation, architecture and regularisation influence which one is reached. In linear regression, Bartlett and colleagues gave conditions under which a minimum-norm interpolating predictor can fit noisy observations and still approach good prediction risk. The conditions involve the covariance spectrum and effective rank. Benign overfitting is therefore conditional structure, not permission to ignore validation.

This also changes how to read zero training error. It establishes that the optimisation procedure found an interpolating solution. It does not establish poor generalisation, good generalisation, semantic correctness or robustness to shift. Those are separate empirical claims. Overfitting should name the observed failure of unseen risk relative to an appropriate target distribution, not act as a synonym for “more parameters than examples”.

A serious competing baseline

More elaborate robustness machinery is not automatically better than carefully tuned empirical risk minimisation. In the DomainBed study, Gulrajani and Lopez-Paz compared multiple domain-generalisation algorithms under a common testbed and found strong empirical risk minimisation competitive or best across the evaluated datasets. This is an important negative result for any story that equates a specialised method name with reliable transfer.

The correct negative control is a transparent baseline trained and selected with the same information budget. If a proposed method wins only because it sees environment labels, uses a larger architecture, receives more tuning trials or is selected on the final stress set, the causal claim has changed. The comparison must hold those privileges fixed or state them as part of the cost.

Interpolation research and domain-generalisation results point in the same methodological direction. Do not infer mechanism from the silhouette of one curve. Manipulate capacity, algorithm and environment separately, then measure which manipulation moves the risk that matters.

Shortcut learning adds an environment axis

Bias and variance are usually defined while one data distribution remains fixed. Deployment often violates that boundary. Cameras change, hospitals serve different populations, templates are redesigned, policies alter who enters a queue, and time changes the prevalence of events. Even when the target meaning remains stable, correlations around it can move.

Let P_e(X,Y) denote the input-label distribution in environment e. For a predictor h and loss ℓ, its risk in that environment is:

An ordinary random holdout estimates R₀ for the observed environment, subject to sampling and measurement error. A deployment claim concerns a set of plausible environments E. One useful diagnostic, introduced here as a decision quantity rather than a standard theorem, is the shift penalty:

The symbols matter. E is not “the future” or “all possible worlds”. It is a documented set of interventions judged credible for the system boundary. The outer maximum makes the diagnostic a non-negative penalty: if every specified environment improves on the baseline risk, the penalty is zero rather than a misleading negative benefit. Enlarging E makes the claim stronger and usually harder to satisfy. Omitting a foreseeable acquisition or population change makes the claim weaker, even if the in-distribution score is unchanged.

The equivalence class hidden by an IID split

Suppose every cracked panel has red chalk and every sound panel does not. Shape, texture and chalk all support the same labels within the observed distribution. The data do not tell the learner which relation the operator intends to remain stable. Geirhos and colleagues use a related star-and-moon example: location can solve both training and IID test images, while a controlled location change exposes the shortcut.

This is an identification problem. Many decision rules lie in the equivalence class that performs well on the benchmark. An environmental intervention cuts through that class by making previously equivalent rules disagree. D'Amour and colleagues call a pipeline underspecified when it can return distinct predictors with equivalent held-out performance but materially different deployment behaviour.

Production-shaped worked example: document triage

Consider a synthetic service-operation classifier that routes incoming forms to a specialist queue. The intended evidence lies in the request text, declared product and account state. Historical data also contain template identifiers, scanner borders, branch codes and fax headers. Those acquisition features are cheap to learn and highly correlated with the queue because each branch used a preferred template.

A random document split reports 96 per cent accuracy. Ten seeds all lie within half a percentage point, so ordinary variance looks small. A source-held-out split tells a different story: performance on a newly onboarded digital channel falls to 71 per cent. Masking template identifiers recovers some cases but harms others. The next experiment should therefore vary one feature family at a time, not declare every source field forbidden.

The investigation can create four matched views of the same underlying form: original image, normalised scan border, redacted template code, and content-preserving re-render. If the route changes while the substantive fields remain fixed, the change is direct operating evidence that acquisition details influence the prediction. A text-only baseline and a template-only baseline help identify whether the full model earns value beyond either source.

A shortcut is defined relative to the target contract and plausible change, not by aesthetic dislike of a feature. Suppose the template code is itself the legally governed form type, validated at ingestion and guaranteed stable by an upstream contract. Removing it then changes the task rather than testing robustness. This is a negative control for shortcut rhetoric: a feature can be easy, non-semantic to a human reader and still be legitimate evidence.

Real shifts and the limit of stress testing

Distribution shift is not merely a toy concern. The WILDS benchmark assembled ten datasets with shifts across hospitals, camera traps, time and location, and reported substantial gaps between in-distribution and out-of-distribution performance for standard training. Recht and colleagues recreated CIFAR-10 and ImageNet test sets through closely matched collection processes and observed absolute accuracy drops across broad model families, even while relative model improvements transferred.

These results establish neither that every model will fail nor that one robustness algorithm will solve the problem. They show that the test distribution is part of the measurement system. The environment set must be selected from the deployment process, threat model and consequence analysis. A medical site shift, a new document channel and a seasonal retail change are not interchangeable merely because all are labelled OOD.

The explicit failure boundary is finite and important: no collection of stress tests proves robustness to unspecified future environments. Tests can support a bounded claim over named variations. They can reveal known shortcuts and compare candidate rules. They cannot guarantee invariance to changes nobody represented, nor can they resolve a target definition that itself changes across environments.

Build a generalisation lab

A useful evaluation programme should make rival explanations disagree. If every candidate is measured on one random holdout, a high score is compatible with low bias, low variance, a stable shortcut, leakage, benchmark adaptation or a genuinely invariant rule. The evaluation has ranked outputs without identifying why they work.

The generalisation lab uses three orthogonal interventions. The capacity intervention varies representational and effective complexity while holding the data protocol fixed. The sample intervention varies training observations and optimiser randomness while holding the model specification fixed. The environment intervention changes one deployment-relevant association while preserving the target contract. Together they form an experimental coordinate system.

1. Specify the claim before splitting the data

Write down the target, unit of prediction, loss, operating threshold, consequence of error and the variations deployment is expected to introduce. Name what is held fixed. For the document example, a claim might cover new scanner models and approved template revisions while excluding a change in the legal definition of the queue. This turns “robust” into a bounded statement that can be tested.

2. Establish honest baselines

Include a constant or prevalence baseline, a transparent low-capacity model, and feature-family baselines where shortcuts are plausible. Train and select them with the same access to labels, groups and tuning trials as the preferred method. A complex model should earn its extra variance, latency and interpretive debt through decision-relevant gains.

3. Sweep capacity and data separately

Record training and validation risk across a capacity proxy, but repeat the sweep at several sample sizes. A high-error plateau at all sample sizes suggests representational or target limitations. A narrow optimum that moves as data grow suggests sample-sensitive effective complexity. An interpolation peak or second descent should be reported rather than forced into a U-shaped narrative.

4. Repeat samples and optimiser seeds

Bootstrap resampling is useful when independent datasets are unavailable, although it does not reproduce every data-collection process. Separate resampling from optimiser seeds: multiple seeds on one dataset estimate one source of variation, while new samples estimate another. Report distributions and confidence intervals, not only the winning seed. For consequential thresholds, measure how often repeated runs cross the decision boundary.

5. Design matched environment interventions

Start from a causal story about acquisition or population. Change one nuisance association while preserving label semantics and as much input content as possible. Use time-held-out, site-held-out or source-held-out sets when those partitions correspond to real deployment changes. Keep a negative-control environment where the challenged association remains valid, so an unnecessary invariance demand can be detected.

6. Keep the final stress set out of model selection

Repeatedly tuning on a shift benchmark turns it into another training environment. Use development interventions for diagnosis and reserve at least one final, governed stress set for release evidence. Record every access, selection criterion and discarded run. Generalisation evidence has a lineage, and adaptive reuse weakens it.

7. Decide through an envelope, not a champion score

For each candidate, retain the vector of IID risk, resampling spread, optimiser spread, group risks, environment penalties and threshold crossings. A single weighted score is convenient but hides trade-offs. Release thresholds should come from consequence and recovery, not the desire to produce a winner.

Run the executable artefact

The script below uses only NumPy and synthetic data. Its regression half measures bias and variance over repeated samples for a grid of polynomial degrees and sample sizes. Its classification half creates a core feature and a stronger shortcut feature, then reverses only the shortcut correlation.

The positive case is the correlation flip: the combined model falls from about 0.95 IID accuracy to about 0.05, while the core-only model remains near 0.80. The negative control keeps the shortcut contract stable, where the combined model remains near 0.95. The artefact demonstrates diagnosis, not a universal preference for lower-capacity or core-only models.

Run: save as generalisation_lab.py, install NumPy, then execute python generalisation_lab.py. Fixed seeds make the displayed output reproducible with a compatible NumPy version.

"""Generalisation lab: separate capacity, sampling, and environment effects.

Assumptions
-----------
* Synthetic data only.
* Squared-loss bias/variance arithmetic is used only for regression.
* The shortcut experiment is a bounded correlation intervention, not proof of
  universal robustness.

Expected behaviour
------------------
1. With 24 samples, high-degree polynomial fits have low training error but
   high prediction variance.
2. With 256 samples, the same high-capacity family becomes much more stable.
3. A classifier using a highly predictive shortcut succeeds IID and fails when
   that correlation reverses.
4. A core-only classifier is less accurate IID but remains stable under the
   correlation intervention.
5. In the negative-control environment, where the shortcut contract remains
   stable, the shortcut model continues to perform well.
"""

from __future__ import annotations

from dataclasses import dataclass
from typing import Iterable

import numpy as np


@dataclass(frozen=True)
class RegressionResult:
    n: int
    degree: int
    train_mse: float
    bias2: float
    variance: float
    expected_test_mse: float


@dataclass(frozen=True)
class ShortcutResult:
    model: str
    environment: str
    mean_accuracy: float
    sd_accuracy: float


def true_function(x: np.ndarray) -> np.ndarray:
    """Target conditional mean for the regression experiment."""
    return np.sin(np.pi * x) + 0.30 * np.cos(3.0 * np.pi * x)


def ridge_fit(design: np.ndarray, target: np.ndarray, strength: float) -> np.ndarray:
    """Closed-form ridge fit with an unpenalised intercept."""
    penalty = np.eye(design.shape[1]) * strength
    penalty[0, 0] = 0.0
    return np.linalg.solve(design.T @ design + penalty, design.T @ target)


def run_regression_lab(
    seed: int = 7,
    repeats: int = 80,
    sample_sizes: Iterable[int] = (24, 64, 256),
    degrees: Iterable[int] = (1, 3, 5, 9, 13),
    noise_sd: float = 0.25,
    ridge_strength: float = 1e-4,
) -> list[RegressionResult]:
    """Measure training fit, squared bias, and variance across repeated samples."""
    rng = np.random.default_rng(seed)
    grid = np.linspace(-1.0, 1.0, 500)
    target_on_grid = true_function(grid)
    results: list[RegressionResult] = []

    for n in sample_sizes:
        for degree in degrees:
            grid_design = np.polynomial.legendre.legvander(grid, degree)
            predictions: list[np.ndarray] = []
            training_losses: list[float] = []

            for _ in range(repeats):
                x = rng.uniform(-1.0, 1.0, n)
                y = true_function(x) + rng.normal(0.0, noise_sd, n)
                design = np.polynomial.legendre.legvander(x, degree)
                weights = ridge_fit(design, y, ridge_strength)
                predictions.append(grid_design @ weights)
                training_losses.append(float(np.mean((design @ weights - y) ** 2)))

            prediction_array = np.stack(predictions)
            mean_prediction = prediction_array.mean(axis=0)
            bias2 = float(np.mean((mean_prediction - target_on_grid) ** 2))
            variance = float(np.mean(np.var(prediction_array, axis=0, ddof=1)))
            expected_test_mse = bias2 + variance + noise_sd**2
            results.append(
                RegressionResult(
                    n=n,
                    degree=degree,
                    train_mse=float(np.mean(training_losses)),
                    bias2=bias2,
                    variance=variance,
                    expected_test_mse=expected_test_mse,
                )
            )

    return results


def make_shortcut_data(
    rng: np.random.Generator,
    n: int,
    shortcut_match_probability: float,
) -> tuple[np.ndarray, np.ndarray]:
    """Create a stable core feature and an environment-dependent shortcut."""
    y = rng.choice(np.array([-1.0, 1.0]), size=n)
    core = 0.85 * y + rng.normal(0.0, 1.0, n)
    matches = rng.random(n) < shortcut_match_probability
    shortcut_sign = np.where(matches, y, -y)
    shortcut = shortcut_sign + rng.normal(0.0, 0.12, n)
    design = np.column_stack([np.ones(n), core, shortcut])
    return design, y


def classification_accuracy(
    design: np.ndarray,
    target: np.ndarray,
    weights: np.ndarray,
) -> float:
    prediction = np.where(design @ weights >= 0.0, 1.0, -1.0)
    return float(np.mean(prediction == target))


def run_shortcut_lab(
    seed: int = 19,
    repeats: int = 100,
    train_size: int = 600,
    test_size: int = 3_000,
) -> list[ShortcutResult]:
    """Compare IID, correlation-flip, and stable-contract environments."""
    rng = np.random.default_rng(seed)
    measurements: dict[tuple[str, str], list[float]] = {}
    environments = {
        "iid": 0.95,
        "correlation_flip": 0.05,
        "stable_contract": 0.95,  # Negative control: no environmental change.
    }

    for _ in range(repeats):
        train_x, train_y = make_shortcut_data(rng, train_size, 0.95)
        both_weights = ridge_fit(train_x, train_y, strength=1.0)
        core_weights = ridge_fit(train_x[:, :2], train_y, strength=1.0)

        for environment, match_probability in environments.items():
            test_x, test_y = make_shortcut_data(rng, test_size, match_probability)
            measurements.setdefault(("core+shortcut", environment), []).append(
                classification_accuracy(test_x, test_y, both_weights)
            )
            measurements.setdefault(("core_only", environment), []).append(
                classification_accuracy(test_x[:, :2], test_y, core_weights)
            )

    results: list[ShortcutResult] = []
    for (model, environment), values in sorted(measurements.items()):
        array = np.asarray(values)
        results.append(
            ShortcutResult(
                model=model,
                environment=environment,
                mean_accuracy=float(array.mean()),
                sd_accuracy=float(array.std(ddof=1)),
            )
        )
    return results


def print_regression_results(results: list[RegressionResult]) -> None:
    print("REGRESSION: capacity x sample size")
    print(" n  degree  train_mse   bias2  variance  expected_test_mse")
    for row in results:
        print(
            f"{row.n:3d} {row.degree:7d}  {row.train_mse:9.4f} "
            f"{row.bias2:7.4f} {row.variance:9.4f} {row.expected_test_mse:18.4f}"
        )


def print_shortcut_results(results: list[ShortcutResult]) -> None:
    print("\nSHORTCUT: environment intervention")
    print(" model           environment          mean_accuracy  sd")
    for row in results:
        print(
            f" {row.model:15s} {row.environment:20s} "
            f"{row.mean_accuracy:13.4f} {row.sd_accuracy:7.4f}"
        )


def main() -> None:
    regression_results = run_regression_lab()
    shortcut_results = run_shortcut_lab()
    print_regression_results(regression_results)
    print_shortcut_results(shortcut_results)


if __name__ == "__main__":
    main()

REGRESSION: capacity x sample size
 n  degree  train_mse   bias2  variance  expected_test_mse
 24       1     0.2830  0.2440    0.0329             0.3394
 24       3     0.0904  0.0507    0.0282             0.1414
 24       5     0.0784  0.0415    0.1304             0.2344
 24       9     0.0349  0.3886    4.0455             4.4966
 24      13     0.0254  0.0346    5.6459             5.7430
 64       1     0.2987  0.2428    0.0119             0.3173
 64       3     0.1079  0.0491    0.0092             0.1208
 64       5     0.0897  0.0368    0.0092             0.1085
 64       9     0.0550  0.0093    0.0802             0.1520
 64      13     0.0508  0.0129    0.4816             0.5569
256       1     0.3018  0.2426    0.0027             0.3078
256       3     0.1075  0.0491    0.0021             0.1137
256       5     0.0962  0.0365    0.0021             0.1011
256       9     0.0641  0.0049    0.0032             0.0707
256      13     0.0589  0.0001    0.0042             0.0668

SHORTCUT: environment intervention
 model           environment          mean_accuracy  sd
 core+shortcut   correlation_flip            0.0505  0.0047
 core+shortcut   iid                         0.9497  0.0038
 core+shortcut   stable_contract             0.9503  0.0036
 core_only       correlation_flip            0.8026  0.0071
 core_only       iid                         0.8020  0.0069
 core_only       stable_contract             0.8015  0.0069

stateDiagram-v2
  [*] --> Claim
  state "Figure 1. One benchmark score,<br/>three escape routes" as Claim
  state "Thought experiment, change<br/>only the chalk" as Test
  state "Figure 2. Bias moves the<br/>centre; variance widens the<br/>fan" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Use the decision instrument

The lab produces evidence, but release still requires judgement. The table below maps common fingerprints to the next discriminating experiment. It is deliberately diagnostic rather than prescriptive. A medical threshold, a search ranking and a low-consequence content tagger can observe the same error pattern while making different release decisions.

The matrix should be read in order of evidence, not as a menu of preferred techniques. High bias and a shortcut can coexist. A model may fail to express the intended signal and still exploit a nuisance correlation. Likewise, an invariant feature can be estimated with high variance when data are scarce. Mixed fingerprints call for sequential isolation, beginning with the cheapest intervention that can falsify the leading explanation.

There is no universal percentage for these gates. The tolerance follows consequence, reversibility and recovery. A model that drafts a reversible category suggestion may tolerate a wider envelope than a model that automatically withholds a service. The important engineering move is to make the envelope explicit before results tempt the team to relax it.

The release artefact should preserve the whole experimental vector. Store the data-version identifiers, split logic, intervention definitions, seeds, capacity settings, baselines, confidence intervals, group denominators and selection rule. A headline metric can be derived later. The reverse reconstruction is usually impossible.