TLDR

1. A training run is an observation, not an experiment. The experiment is the counterfactual comparison that makes one claimed cause change while declared nuisance factors are blocked, randomised or sampled.
2. A seed names one route through stochastic training. It helps reconstruct and pair runs, but it neither removes variance nor guarantees identical results across software releases, hardware and platforms.
3. Controls, ablations and tuning budgets define the causal question. A frozen-recipe ablation and a budget-matched ablation answer different questions and should never share one interpretation.
4. Even a clean benchmark experiment has a hard boundary: it estimates behaviour over the sampled data, training process and environment, not deployment value under untested shift, people or policy.
5. Choose models from distributions of paired effects, negative controls and locked decision rules. Reject best-seed evidence and any result whose run ledger cannot reconstruct what was compared.

Seed 17 wins the architecture review

Worked scenario

A team replaces the activation in a document classifier and reports an accuracy gain from 92.1 to 92.6 per cent. Both models used seed 17. The test set was fixed. The training script is in version control. The new model therefore looks reproducible, controlled and better.

Then a second engineer asks what seed 17 controlled. It fixed the initial weights and the data-loader generator, but the baseline had been tuned six months earlier with twelve trials. The new model received forty trials. Its image augmentation library was also upgraded. Early stopping selected the best validation checkpoint, and the team looked at the test score after each promising run. The visible comparison changed one activation; the actual comparison changed an activation, search budget, software path and selection history.

Nothing here requires misconduct. Each choice is ordinary. Together they make the 0.5-point gain causally ambiguous. A rerun may reproduce the number and still fail to support the explanation. Reproducibility can preserve a confounded result perfectly.

The consequence is practical. The unit of evidence is no longer “model B scored 92.6”. It is a set of controlled differences with manifests, denominators, uncertainty, negative controls and a boundary statement. That standard is stricter than a notebook rerun, yet cheaper than repeatedly debating results whose causal meaning was never designed.

The neighbouring article The Eval Pipeline is the Product treats continuous evaluation after models, prompts and systems begin changing in service. This paper asks the narrower upstream question: when does a model-training comparison support the cause named in the research claim?

Part I

The score is a causal mixture

Imagine pouring water from several coloured bottles into one glass. The final colour records their mixture, but it does not reveal how much came from each bottle. A deep-learning score behaves similarly. Architecture, sampled data, initial weights, mini-batch order, augmentation, optimiser state, hyperparameter search, stopping rule, numerical kernels and evaluator all contribute to the observed value.

Published work has repeatedly shown that these contributors can be large enough to reverse a ranking. Reimers and Gurevych trained 50,000 LSTM systems and found that random seed choice could move named-entity-recognition F1 by about one absolute point for two systems, enough to change the apparent research conclusion.^[1] Henderson and colleagues documented similarly consequential variance and reporting problems in deep reinforcement learning.^[2] These results do not establish a universal variance level. They establish that a single run can be an unreliable estimator in important model families.

Bouthillier and colleagues widened the frame. Their benchmark model includes variation from data sampling, augmentation, initialisation and hyperparameter choice, then examines the error rate of declaring one algorithm better than another.^[3] The important lesson is structural: variance belongs to the experimental object whenever the claim is meant to survive that variance. Fixing a factor removes it from the estimated population. Sampling it includes it.

Name the effect before the run

The useful formal question is modest. Let Y(a, u) be the test metric produced when treatment a is applied under nuisance state u. Treatment might be ReLU rather than an identity activation. The nuisance state may include the sampled train split, initialisation, data order and augmentation draw. The target is an average contrast over a declared distribution of nuisance states:

If U contains only initialisation seeds while the data split stays fixed, Δ addresses training stochasticity on that split. It does not estimate uncertainty from sampling a different dataset. If U also samples grouped data splits, the claim broadens accordingly. If hardware is fixed, the result says nothing about portability across accelerators. The experiment’s scope is encoded by what it varies, not by the breadth of its prose.

Pairing means evaluating both treatments under the same block u, then analysing the difference inside each block. This follows the older experimental-design logic of blocking known nuisance variables before randomising the remainder. NIST’s statistical handbook describes blocking as a way to isolate important nuisance effects so that treatment comparisons occur within comparable conditions.^[4] Deep learning gives the block unusual names, but the logic is unchanged.

A seed is a coordinate, not an antidote

A random seed selects a pseudorandom sequence. It can make a particular software path recoverable and can couple two arms to the same data order or augmentation draw. It does not turn a stochastic procedure into a universal constant. Nor does setting one seed necessarily govern every library, process and device involved.

Current PyTorch guidance is explicit that complete reproducibility is not guaranteed across releases, commits or platforms, and that deterministic algorithms may need to be requested separately.^[5] This matters because “we used the same seed” often bundles three different claims: the run can be reconstructed on the same stack; the compared arms received matched randomness; and the result generalises over randomness. Only the first two are plausible consequences of careful seeding. The third requires sampling more than one seed.

Seed selection after observing performance is hyperparameter selection. Reporting the best of fifty seeds estimates a search procedure that spent fifty trials, not the expected performance of one training run. Dodge and colleagues showed that conclusions about model quality can change with the hyperparameter-search budget, and proposed reporting validation performance as a function of that budget.^[6] The same principle applies to random restarts.

Fine-tuning studies make the distinction more concrete. Dodge and colleagues separately varied weight initialisation and data order, then showed that early stopping changes how those sources appear in the selected result.^[12] A ledger should therefore record seed roles, not merely one undifferentiated integer.

Thought experiment: the two runners

Two runners test a new shoe. In the first design, one runner wears the old shoe on a calm morning and another wears the new shoe in afternoon rain. Ten repetitions provide ten measurements, but weather and person remain entangled with shoe. In the second design, each runner completes both conditions on matched days, with order randomised. Fewer observations may yield a cleaner shoe effect because the comparison happens within person and weather block.

Now translate person into data split and weather into initialisation. Independent random seeds in each arm estimate two noisy means. Matched blocks estimate paired differences and can remove shared nuisance variation. The analogy breaks where training paths interact with architecture: the same initial tensor may not have the same functional meaning in different graphs, and two architectures may consume random numbers differently. Pair what is structurally shareable, especially data partitions and evaluation cases, then record where coupling ends.

Part II

flowchart LR
  A["Seed 17 wins the architecture<br/>review"] --> B["The score is a causal mixture"]
  B --> C["Build the counterfactual<br/>before the model"]
  C --> D["Ablations are two different<br/>experiments"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#7B4CD8,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Build the counterfactual before the model

A useful experiment starts with a sentence that can be false. “Our architecture is better” is too loose. Better at what metric, under what resource budget, over which data-generating process, and compared with which alternative? A sharper claim is: “At equal parameter count and training budget, replacing the identity activation with ReLU raises held-out accuracy on noisy XOR data across sampled splits and training seeds.”

That sentence determines the design. Parameter count and budget must be matched because the claim conditions on them. Data splits and seeds must be sampled because the claim ranges over them. The metric and test set must be locked because they define the outcome. The identity activation is the control because it preserves the parameterised graph while removing the non-linearity.

The best experimental specification is a dependency map from each phrase in the claim to an observable field in the run ledger. If “equal compute” appears in the prose, log steps, examples, accelerator time and search trials. If “across customers” appears, split by customer rather than by individual rows. If “stable” appears, define whether stability means low variance, no catastrophic failures, calibration preservation or ranking consistency.

The unit of randomisation decides what is independent

Suppose one trained model predicts 10,000 test examples. Those examples help estimate its test error, but they are not 10,000 independent training runs. Suppose ten checkpoints come from one trajectory. They are not ten independent models. Treating correlated observations as independent creates pseudoreplication and intervals that look much tighter than the experiment earned.

The independent unit follows the stochastic process being generalised. For initialisation variability, it is usually a training run. For data-sampling variability, it may be a dataset split or site. For multi-dataset claims, datasets may be the higher-level units, with runs nested inside them. A production-shaped study often needs this hierarchy because hundreds of seeds on one narrow benchmark cannot substitute for several genuinely different populations.

No universal seed count solves the design. Begin with a pilot that estimates the variance of paired differences. Then ask how precisely the decision must distinguish the smallest effect worth acting on. A 0.2-point gain may justify a large study when inference cost falls materially, while even a two-point gain may be irrelevant if it violates latency or subgroup constraints. Report the count, interval and decision threshold together.

Worked example: one non-linearity, one question

Worked scenario

The executable scaffold creates a noisy XOR dataset from four Gaussian clusters. Both arms use the same two linear layers, hidden width, parameter count, optimiser, learning rate, epochs and mini-batch order. The control inserts an identity operation between the layers, so the composition remains linear. The treatment inserts ReLU, allowing a non-linear boundary.

Each block generates a train, validation and test split from declared seeds. Both arms receive the same split, initial parameter draws and epoch-level data order. The result is analysed as a set of paired test-accuracy differences. The experiment also runs a sham case in which ReLU is compared with the identical ReLU configuration. That negative control should return a difference of exactly zero on the same software and platform manifest.

On the recorded ten-block run, the identity control averaged 46.05 per cent test accuracy and the ReLU treatment averaged 98.2 per cent. The mean paired difference was 52.15 percentage points; the bootstrap interval over the ten declared blocks was 49.4 to 55.1 points. These are synthetic teaching results, not evidence about a public benchmark. Their purpose is to make the experimental contract inspectable.

The large effect does not make controls optional. The sham arm averaged the same score as its identical control and produced a zero paired difference in every block. If the sham had moved, the first conclusion would concern the scaffold, not ReLU. This ordering is important: validate that the experiment can preserve no effect before trusting its ability to measure an effect.

Controls that answer different accusations

A baseline answers whether the proposed method improves on an alternative. A negative control asks whether the experimental machinery reports an effect where none should exist. A positive control asks whether the setup can detect a known effect. These roles are distinct.

Useful deep-learning negative controls include an inert treatment flag, labels permuted independently of inputs, a component replaced by a parameter-matched no-op, or an evaluator applied to duplicate predictions under different names. The right control depends on the suspected failure. Permuted labels can reveal leakage or memorisation, but they do not test whether two search budgets were equal. A sham component can test orchestration, but not data contamination.

A strong control sometimes reproduces the attractive score for the wrong reason. Lucic and colleagues compared several GAN methods under broad optimisation and random restarts, finding that most could reach similar scores and no tested method consistently beat the non-saturating GAN baseline.^[7] The point is not that algorithmic improvements never matter. It is that weak baselines can make tuning effort look like mechanism.

Controls should therefore be selected from competing explanations, not from convenience. Before running, list the stories that could generate the desired result: greater capacity, more compute, favourable preprocessing, test leakage, fragile seed, easier optimisation or genuine structural advantage. Give each serious story either a matched control, a logged covariate, an intervention or an explicit limitation.

When a control fails, stop interpreting the treatment

A non-zero sham result is not a nuisance to subtract from the headline effect. It says that treatment labels, manifests, caching, evaluation or numerical execution can move the result even when the declared model does not change. The correct response is to block release, trace that pathway and rerun the affected blocks. Subtracting the sham assumes its distortion is stable and additive, which is usually another untested claim.

A failed positive control creates the opposite problem. If an intentionally destructive augmentation, corrupted label slice or known weak model is not detected, the evaluator may be insensitive, the sample too small or the metric misaligned. The treatment’s null result then supports “the instrument did not discriminate” as strongly as “the treatment had no effect”. Sensitivity must be established before absence becomes interpretable.

Controls can also disagree across levels. The training sham may pass while a duplicated prediction file receives different scores after post-processing. That localises the defect to the evaluator rather than the optimiser. Controls are calibration experiments for the evidence pipeline. Their value lies in identifying where interpretation must stop, not in decorating a successful result.

Part III

Ablations are two different experiments

An ablation sounds simple: remove component C and observe the damage. Yet removal changes more than one thing. Parameter count may fall. Optimisation may become easier or harder. The best learning rate may move. Training time may shrink. Other components may compensate. The interpretation depends on which of these changes the question permits.

The first estimand is frozen-recipe contribution. Remove C while preserving the surrounding recipe as far as structurally possible. This asks what C contributes to the configured system the team actually built. It is appropriate for tracing dependencies and diagnosing whether the headline system relies on the component under its current training procedure.

The second is budget-matched capability without C. Remove C, then retune the remaining system under the same declared search and compute budget. This asks whether the component is necessary for the best system achievable without it. Compensation is allowed because operational model selection would use it. This is often the more relevant architecture decision.

The two estimates can disagree without contradiction. A component may add two points in the frozen recipe, while a retuned ablation recovers all but 0.2 points. The first result says the current weights and hyperparameters rely on C. The second says the broader design space barely does. Publishing only the larger number silently changes “current contribution” into “architectural necessity”.

Thought experiment: remove the stabiliser

Consider a deep network trained with a normalisation layer. Remove the layer and keep the learning rate, schedule and initialisation unchanged. Training destabilises and accuracy falls sharply. One explanation is that normalisation carries useful representation. Another is that the original optimiser settings were selected for the normalised geometry.

Now change one causal feature: allow both arms the same twenty-trial tuning budget. The ablated network recovers most of the loss at a smaller learning rate. The evidence for “normalisation is required” weakens; the evidence for “normalisation makes this recipe easier to optimise” remains. Change another feature: equalise wall-clock rather than trial count. The normalised model may explore more settings because each run finishes sooner. Resource equality itself has several valid definitions.

This thought experiment shows why an ablation table needs a question above it. Without the estimand, rows of check marks and scores are descriptive surgery. With it, each removal becomes an intervention whose collateral changes are either controlled or intentionally included.

Tuning is part of the treatment unless designed otherwise

Hyperparameter optimisation creates a second stochastic process around training. It samples configurations, observes validation scores and selects a winner. Comparing one carefully tuned proposal with a copied baseline estimates the combined value of method plus tuning asymmetry. Sometimes that is the intended product question. It is not a clean mechanism comparison.

Dodge and colleagues’ budget curves make this visible: the best validation result depends on how many trials or how much time the search receives.^[6] Their finding joins a wider reproducibility programme. Pineau and colleagues describe the NeurIPS 2019 programme’s code policy, challenge and reproducibility checklist as institutional attempts to improve how results are conducted and communicated.^[8] Current NeurIPS guidance still asks authors to provide a reasonable route for reproducing or verifying experimental results.^[9]

A fair search design should state the optimisation algorithm, candidate space, budget, stopping rule, validation metric and whether failed trials count. Equal trial counts are not equal compute when methods have different cost. Equal accelerator-hours may favour methods that train quickly. Equal monetary cost may be the operationally relevant constraint. Choose the resource that matches the claim, then report the alternatives needed to interpret it.

Never tune on the locked test set. Repeatedly checking test performance during development converts the test into a validation signal, even if no gradient touches it. Use validation data for selection, preserve a final test evaluation, and record every access. When data is scarce, nested cross-validation or a separate external evaluation may be preferable, but the same principle remains: the evidence used to choose cannot independently judge the choice.

Worked example: augmentation for unseen document issuers

Worked scenario

A document-image model routes incoming pages either to automatic extraction or to human review. The proposed treatment adds an augmentation policy intended to improve recall on scans from issuers absent during training. The business constraint holds false routing below a fixed threshold because a misplaced page can contaminate a case.

The claim is not “augmentation improves accuracy”. It is: “Under equal training and search cost, the augmentation policy increases recall at a prespecified false-routing ceiling for previously unseen issuers, without worsening the worst named document cohort beyond tolerance.” That wording forces several design decisions.

Split by issuer, not page, so templates from one issuer cannot appear in both train and test. Sample several issuer-group partitions because the claim ranges over possible unseen issuers. Inside each partition, pair baseline and treatment across initialisation and data-order seeds. Give both arms equal accelerator-hour search budgets, with the same validation objective. Lock the operating threshold from validation data before the final test.

Add three controls. A no-op augmentation tag checks orchestration. A deliberately destructive augmentation is a positive control for the evaluation’s ability to detect harm. An incumbent non-neural classifier establishes whether the deep model is earning its complexity. Report recall and false routing by issuer cohort, calibration near the threshold, run failures, compute and the distribution of paired differences.

Then stress the mechanism. If improvement comes from invariance to blur, a blur-specific slice should benefit more than clean scans. If the same gain appears on untouched images while the augmentation is disabled at training, the proposed explanation weakens. If gains vanish after budget-matched tuning, the original result was partly search allocation. A production-shaped experiment links each architectural story to a pattern of results that a rival story would not predict.

Part IV

stateDiagram-v2
  [*] --> Claim
  state "Name the effect before the run" as Claim
  state "A seed is a coordinate, not an<br/>antidote" as Test
  state "Thought experiment, the two<br/>runners" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Turn runs into an evidence package

A folder of checkpoints is not an experiment record. The record must reconstruct why each run existed, what changed, which shared block it belonged to, how it was selected and which claim it informed. That is the role of the run ledger.

The minimum ledger row contains a candidate-system version, treatment arm, data and training seeds, dataset snapshot or generation rule, split identifiers, hyperparameters, code revision, dependency versions, device and numerical settings, stopping reason, validation result, locked test result, runtime, failure state and artifact hashes. A single experiment manifest then binds the rows to the hypothesis, estimand, controls, search policy and decision rule.

Neural training code can be perfectly available yet scientifically incomplete. Availability answers whether an artefact can be inspected. Repeatability asks whether the same manifest can produce the same or acceptably similar result on the stated stack. Replication asks whether the claim survives a materially independent implementation or sample. Vocabulary varies across communities, so name the operation rather than relying on a badge word alone.

The included scaffold writes trial_ledger.csv and summary.json. It separates data seeds from training seeds, hashes each arm manifest, requests deterministic PyTorch algorithms, reports paired differences and runs both a positive and sham case. The script’s assertions are release gates: the lower interval bound for the known positive effect must exceed 0.20, and the sham mean must equal zero.

Report the distribution that answers the claim

For paired designs, plot and summarise paired differences. Raw score distributions remain useful because they reveal variance and ceiling effects, but the treatment contrast is the primary object. Report mean or median according to the decision, an interval, all run counts, failed runs and the exact aggregation rule. A standard deviation without the run-level values can hide multimodality and catastrophic failures.

Statistical significance is not a substitute for experimental significance. Dror and colleagues show that test choice depends on the task, setup and metric, and document frequent misuse or omission in NLP research.^[10] Begin with the estimand and dependence structure, then choose an interval or test that respects them. A tiny p-value can accompany a trivial effect; a wide interval can make an apparently large effect undecidable.

Report search results, not only the chosen winner. The number of attempted configurations, early-stopped runs and total resource budget tells the reader what selection process produced the model. Lucic’s GAN study and Dodge’s budget analysis both show why this matters.^[7][6] If a production team would always run a five-model ensemble or choose the best of ten restarts, evaluate that policy directly and charge its full cost.

Preserve adverse examples. A mean gain alongside one severe subgroup collapse is not a clean win. D’Amour and colleagues call attention to underspecification: pipelines can return predictors with similar held-out performance that behave differently in deployment domains.^[11] Stress tests and subgroup slices therefore belong beside the primary metric whenever the release claim depends on behaviour beyond the in-distribution average.

A null result needs a resolution statement

“No statistically significant difference” merges at least three outcomes. The treatment may be practically equivalent to the control, the estimate may be too imprecise to decide, or the experiment may be measuring the wrong contrast. These lead to different actions. Equivalence requires a predeclared band around zero that represents effects too small to matter, not merely a p-value above a conventional threshold.

Suppose the material threshold is half a percentage point. An interval wholly inside minus 0.5 to plus 0.5 supports practical similarity within the declared scope. An interval from plus 0.8 to plus 2.0 supports a useful gain. An interval from minus 0.7 to plus 1.4 crosses meaningful harm and benefit, so the evidence is indeterminate even when its centre is positive. Reporting only the mean erases this decision geometry.

Sample size should follow that geometry. Use pilot paired differences to estimate dispersion, then plan enough independent blocks to resolve the smallest effect that would change the decision. Preserve the stopping rule. Adding blocks only while the interval is unfavourable, or stopping as soon as it excludes zero, makes the sampling path part of selection. When sequential monitoring is intended, declare and analyse it as sequential monitoring rather than pretending the final block count was fixed.

The experiment decision card

Before allocating compute, complete the following card. Its purpose is not paperwork. Each field prevents a common route by which a descriptive score acquires a causal interpretation it did not earn.

A simple release rule might read: accept the treatment only if the paired mean improvement exceeds 0.5 points, the 95 per cent interval excludes zero, no named cohort declines by more than 0.3 points, the sham control stays inside ±0.05 points, and compute rises by less than 10 per cent. The numbers are local choices, not universal standards. Their value is that they convert post-hoc enthusiasm into an ex-ante decision.

When the result misses the rule, the available decisions are broader than “reject the paper”. Narrow the claim, collect more blocks, repair the control, change the metric, investigate an interaction or conclude that the evidence is inconclusive. An honest indeterminate result is useful because it tells the next experiment exactly which uncertainty remains.

Where this mechanism stops helping

The first boundary is deterministic computation. A convex model trained by an exact deterministic solver on a fixed dataset may have no meaningful seed variance. Repeating identical runs adds little. The experiment still needs a comparator, held-out evaluation and scope statement, but a large seed sweep would be ritual rather than evidence. This counterexample weakens any universal rule that every machine-learning result requires the same number of restarts.

The second boundary is external validity. A rigorous benchmark comparison cannot establish customer benefit, operational resilience, fairness under a new population or safe human review. Those outcomes require representative data, simulations, shadow use or controlled production evidence. A clean internal experiment is necessary for many claims and sufficient for fewer.

The third boundary is interacting change. If the treatment redesigns architecture, optimiser, data curriculum and inference policy together, a single-factor explanation is unavailable. The package can still be evaluated as a configured system under equal resources, but component-level causal language should stop. Factorial or sequential experiments may later decompose interactions.

The fourth boundary is adaptive research behaviour. Teams learn from every run. Once results alter the hypothesis, search space or data handling, the experiment has entered an adaptive sequence. Preserve that history and reserve fresh evidence for the final claim. Pretending the last comparison was planned in isolation understates selection.

Finally, the method would become less necessary in a domain where rankings remain invariant across seeds, data samples, tuning budgets, implementations and stress conditions. That observation would justify a cheaper protocol for that domain. Until such stability is demonstrated, the burden belongs to the comparison, not to the reader’s faith in a familiar benchmark.

#!/usr/bin/env python3
"""Reproducible paired deep-learning experiment scaffold.

The experiment uses only synthetic data and compares two arms under paired
sources of variation. It writes a trial ledger and a machine-readable summary.

Positive case
-------------
A two-layer network with ReLU is compared with the same parameterised network
using the identity activation on a noisy XOR task. Parameter counts,
initialisation draws, batch order, optimiser and training budget are matched.
The expected paired effect is positive because the identity composition is
linear while ReLU can represent the XOR boundary.

Negative case
-------------
The same ReLU configuration is compared with itself under the same block seeds.
The expected paired difference is exactly zero. A non-zero result indicates a
plumbing, manifest or nondeterminism defect in the scaffold.

Requirements: Python 3.10+, numpy, torch.
Run: python deep_learning_experiment_scaffold.py --out experiment_output
"""
from __future__ import annotations

import argparse
import csv
import hashlib
import json
import platform
import random
import sys
from dataclasses import asdict, dataclass
from pathlib import Path
from typing import Iterable, Literal

import numpy as np
import torch
from torch import nn

Activation = Literal["identity", "relu"]


@dataclass(frozen=True)
class ExperimentConfig:
    blocks: int = 10
    train_per_quadrant: int = 60
    validation_per_quadrant: int = 25
    test_per_quadrant: int = 50
    noise: float = 0.42
    hidden_width: int = 16
    epochs: int = 45
    batch_size: int = 120
    learning_rate: float = 0.03
    bootstrap_draws: int = 10_000
    confidence_level: float = 0.95
    base_seed: int = 48_000


@dataclass(frozen=True)
class TrialRecord:
    case: str
    block: int
    data_seed: int
    train_seed: int
    arm: str
    activation: str
    validation_accuracy: float
    test_accuracy: float
    manifest_hash: str


class TinyClassifier(nn.Module):
    """Same parameterised graph in both arms; only activation changes."""

    def __init__(self, hidden_width: int, activation: Activation) -> None:
        super().__init__()
        self.fc1 = nn.Linear(2, hidden_width)
        self.fc2 = nn.Linear(hidden_width, 1)
        self.activation_name = activation
        self.activation: nn.Module = nn.ReLU() if activation == "relu" else nn.Identity()

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return self.fc2(self.activation(self.fc1(x))).squeeze(-1)


def set_global_determinism(seed: int) -> None:
    """Make same-manifest reruns deterministic on the same software/platform."""
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.set_num_threads(1)
    torch.use_deterministic_algorithms(True)


def make_xor_split(seed: int, per_quadrant: int, noise: float) -> tuple[torch.Tensor, torch.Tensor]:
    rng = np.random.default_rng(seed)
    centres = np.array([[-1.0, -1.0], [-1.0, 1.0], [1.0, -1.0], [1.0, 1.0]], dtype=np.float32)
    labels = np.array([0.0, 1.0, 1.0, 0.0], dtype=np.float32)
    xs: list[np.ndarray] = []
    ys: list[np.ndarray] = []
    for centre, label in zip(centres, labels, strict=True):
        points = rng.normal(loc=centre, scale=noise, size=(per_quadrant, 2)).astype(np.float32)
        xs.append(points)
        ys.append(np.full(per_quadrant, label, dtype=np.float32))
    x = np.concatenate(xs, axis=0)
    y = np.concatenate(ys, axis=0)
    order = rng.permutation(len(y))
    return torch.from_numpy(x[order]), torch.from_numpy(y[order])


def accuracy(model: nn.Module, x: torch.Tensor, y: torch.Tensor) -> float:
    model.eval()
    with torch.inference_mode():
        pred = (torch.sigmoid(model(x)) >= 0.5).to(y.dtype)
    return float((pred == y).float().mean().item())


def train_one(
    activation: Activation,
    cfg: ExperimentConfig,
    data_seed: int,
    train_seed: int,
) -> tuple[float, float]:
    set_global_determinism(train_seed)
    x_train, y_train = make_xor_split(data_seed + 11, cfg.train_per_quadrant, cfg.noise)
    x_val, y_val = make_xor_split(data_seed + 23, cfg.validation_per_quadrant, cfg.noise)
    x_test, y_test = make_xor_split(data_seed + 37, cfg.test_per_quadrant, cfg.noise)

    model = TinyClassifier(cfg.hidden_width, activation)
    optimiser = torch.optim.Adam(model.parameters(), lr=cfg.learning_rate)
    loss_fn = nn.BCEWithLogitsLoss()

    for epoch in range(cfg.epochs):
        model.train()
        generator = torch.Generator().manual_seed(train_seed + 10_000 + epoch)
        order = torch.randperm(len(y_train), generator=generator)
        for start in range(0, len(order), cfg.batch_size):
            idx = order[start : start + cfg.batch_size]
            optimiser.zero_grad(set_to_none=True)
            loss = loss_fn(model(x_train[idx]), y_train[idx])
            loss.backward()
            optimiser.step()

    return accuracy(model, x_val, y_val), accuracy(model, x_test, y_test)


def manifest_hash(cfg: ExperimentConfig, case: str, block: int, arm: str, activation: str) -> str:
    manifest = {
        "config": asdict(cfg),
        "case": case,
        "block": block,
        "arm": arm,
        "activation": activation,
        "python": platform.python_version(),
        "numpy": np.__version__,
        "torch": torch.__version__,
        "platform": platform.platform(),
    }
    payload = json.dumps(manifest, sort_keys=True, separators=(",", ":")).encode("utf-8")
    return hashlib.sha256(payload).hexdigest()[:16]


def bootstrap_interval(
    values: np.ndarray,
    draws: int,
    confidence_level: float,
    seed: int,
) -> tuple[float, float]:
    rng = np.random.default_rng(seed)
    n = len(values)
    samples = rng.choice(values, size=(draws, n), replace=True).mean(axis=1)
    alpha = (1.0 - confidence_level) / 2.0
    low, high = np.quantile(samples, [alpha, 1.0 - alpha])
    return float(low), float(high)


def summarise_case(
    case: str,
    records: Iterable[TrialRecord],
    cfg: ExperimentConfig,
) -> dict[str, object]:
    rows = [r for r in records if r.case == case]
    by_block: dict[int, dict[str, float]] = {}
    for row in rows:
        by_block.setdefault(row.block, {})[row.arm] = row.test_accuracy
    paired = np.array(
        [by_block[b]["treatment"] - by_block[b]["control"] for b in sorted(by_block)],
        dtype=np.float64,
    )
    low, high = bootstrap_interval(
        paired,
        draws=cfg.bootstrap_draws,
        confidence_level=cfg.confidence_level,
        seed=cfg.base_seed + (1 if case == "positive" else 2),
    )
    control = np.array([by_block[b]["control"] for b in sorted(by_block)], dtype=np.float64)
    treatment = np.array([by_block[b]["treatment"] for b in sorted(by_block)], dtype=np.float64)
    return {
        "case": case,
        "blocks": len(paired),
        "control_mean_test_accuracy": round(float(control.mean()), 6),
        "treatment_mean_test_accuracy": round(float(treatment.mean()), 6),
        "paired_mean_difference": round(float(paired.mean()), 6),
        "paired_difference_std": round(float(paired.std(ddof=1)), 6),
        "bootstrap_confidence_level": cfg.confidence_level,
        "bootstrap_interval": [round(low, 6), round(high, 6)],
        "paired_differences": [round(float(v), 6) for v in paired],
    }


def run(cfg: ExperimentConfig) -> tuple[list[TrialRecord], dict[str, object]]:
    records: list[TrialRecord] = []
    cases: dict[str, tuple[Activation, Activation]] = {
        "positive": ("identity", "relu"),
        "negative_sham": ("relu", "relu"),
    }
    for case, (control_activation, treatment_activation) in cases.items():
        for block in range(cfg.blocks):
            data_seed = cfg.base_seed + 100 * block
            train_seed = cfg.base_seed + 100 * block + 7
            for arm, activation in (
                ("control", control_activation),
                ("treatment", treatment_activation),
            ):
                val_acc, test_acc = train_one(
                    activation=activation,
                    cfg=cfg,
                    data_seed=data_seed,
                    train_seed=train_seed,
                )
                records.append(
                    TrialRecord(
                        case=case,
                        block=block,
                        data_seed=data_seed,
                        train_seed=train_seed,
                        arm=arm,
                        activation=activation,
                        validation_accuracy=round(val_acc, 6),
                        test_accuracy=round(test_acc, 6),
                        manifest_hash=manifest_hash(cfg, case, block, arm, activation),
                    )
                )

    summaries = {
        case: summarise_case(case, records, cfg)
        for case in cases
    }
    positive = summaries["positive"]
    sham = summaries["negative_sham"]

    # Release assertions encode the claim and the negative control.
    pos_low = float(positive["bootstrap_interval"][0])  # type: ignore[index]
    sham_mean = float(sham["paired_mean_difference"])  # type: ignore[arg-type]
    assert pos_low > 0.20, f"Positive case failed: lower interval bound was {pos_low:.3f}"
    assert abs(sham_mean) < 1e-12, f"Sham control failed: mean difference was {sham_mean:.6f}"

    summary: dict[str, object] = {
        "assumptions": [
            "Synthetic noisy XOR data; no claim about external datasets.",
            "The causal treatment is activation choice in an otherwise matched graph.",
            "Blocks pair data generation, initialisation and batch-order seeds.",
            "Bootstrap uncertainty describes these declared blocks only.",
            "Determinism is asserted for the same software and platform manifest.",
        ],
        "environment": {
            "python": platform.python_version(),
            "numpy": np.__version__,
            "torch": torch.__version__,
            "platform": platform.platform(),
        },
        "config": asdict(cfg),
        "results": summaries,
        "release_assertions": {
            "positive_lower_bound_gt_0_20": True,
            "sham_mean_difference_eq_0": True,
        },
    }
    return records, summary


def write_outputs(out_dir: Path, records: list[TrialRecord], summary: dict[str, object]) -> None:
    out_dir.mkdir(parents=True, exist_ok=True)
    rows_path = out_dir / "trial_ledger.csv"
    with rows_path.open("w", newline="", encoding="utf-8") as handle:
        writer = csv.DictWriter(handle, fieldnames=list(asdict(records[0]).keys()))
        writer.writeheader()
        for record in records:
            writer.writerow(asdict(record))
    summary_path = out_dir / "summary.json"
    summary_path.write_text(json.dumps(summary, indent=2, sort_keys=True) + "\n", encoding="utf-8")


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--out", type=Path, default=Path("experiment_output"))
    parser.add_argument("--blocks", type=int, default=ExperimentConfig.blocks)
    args = parser.parse_args()
    cfg = ExperimentConfig(blocks=args.blocks)
    records, summary = run(cfg)
    write_outputs(args.out, records, summary)
    print(json.dumps(summary["results"], indent=2, sort_keys=True))
    print(f"\nWrote {len(records)} trial rows to {args.out.resolve()}")
    return 0


if __name__ == "__main__":
    sys.exit(main())

{
  "config": {
    "base_seed": 48000,
    "batch_size": 120,
    "blocks": 10,
    "bootstrap_draws": 10000,
    "confidence_level": 0.95,
    "epochs": 45,
    "hidden_width": 16,
    "learning_rate": 0.03,
    "noise": 0.42,
    "test_per_quadrant": 50,
    "train_per_quadrant": 60,
    "validation_per_quadrant": 25
  },
  "environment": {
    "numpy": "2.3.5",
    "platform": "Linux-6.18.35-x86_64-with-glibc2.41",
    "python": "3.13.5",
    "torch": "2.10.0+cpu"
  },
  "release_assertions": {
    "positive_lower_bound_gt_0_20": true,
    "sham_mean_difference_eq_0": true
  },
  "results": {
    "negative_sham": {
      "blocks": 10,
      "bootstrap_confidence_level": 0.95,
      "bootstrap_interval": [
        0.0,
        0.0
      ],
      "case": "negative_sham",
      "control_mean_test_accuracy": 0.982,
      "paired_difference_std": 0.0,
      "paired_differences": [
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0
      ],
      "paired_mean_difference": 0.0,
      "treatment_mean_test_accuracy": 0.982
    },
    "positive": {
      "blocks": 10,
      "bootstrap_confidence_level": 0.95,
      "bootstrap_interval": [
        0.494,
        0.551
      ],
      "case": "positive",
      "control_mean_test_accuracy": 0.4605,
      "paired_difference_std": 0.048422,
      "paired_differences": [
        0.555,
        0.495,
        0.605,
        0.505,
        0.485,
        0.595,
        0.495,
        0.485,
        0.465,
        0.53
      ],
      "paired_mean_difference": 0.5215,
      "treatment_mean_test_accuracy": 0.982
    }
  }
}

Compact glossary

Treatment

The deliberately changed factor whose effect the experiment seeks to estimate.

Nuisance factor

A source of variation that can move the outcome without being the target explanation.

Block

A matched nuisance state within which control and treatment are compared.

Estimand

The precise quantity the design intends to estimate, including its target population.

Negative control

A condition expected to show no effect, used to reveal leakage, orchestration defects or bias.

Pseudoreplication

Treating correlated observations as independent evidence and thereby overstating precision.

Frozen-recipe ablation

Removal of a component while preserving the configured training recipe.

Budget-matched ablation

Removal followed by retuning under the same declared search or compute budget.

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: Seed 17 wins the architecture review
  M->>E: Name the effect before the run
  E-->>M: compare rival explanations
  E->>D: Change the question before you run the model
  D-->>Q: revise scope or act

Change the question before you run the model

A deep-learning experiment begins when the team stops asking “what score did this run get?” and asks “which causal contrast will this collection of runs identify?” That change moves seeds, controls, ablations, search and reporting out of the appendix. They become the mechanism that gives the score meaning.

The architecture decision also changes. Do not select the model with the highest observed endpoint. Select the configured procedure whose paired effect clears the material threshold, survives negative controls, remains acceptable across named slices, fits the resource envelope and can be reconstructed from its ledger. Where those conditions fail, narrow the claim or collect the missing evidence.

The durable intuition is simple: a score is an outcome; an experiment is a designed difference. Deep learning makes that difference expensive because the training pipeline contains many stochastic and adaptive paths. It also makes disciplined design unusually valuable. A small, well-blocked comparison can carry more causal weight than a large pile of unrelated runs.

The next experiment should therefore be planned from its final decision backwards. Write the claim, estimand, competing story, controls and release rule. Only then choose the seeds and launch the jobs. Compute can train models. Design is what turns their variation into knowledge.