Let a training set be S = {(xᵢ, yᵢ)}, sampled from an unknown process. Training usually minimises empirical risk, the average loss on those observed pairs. The quantity we care about is population risk, the expected loss on future pairs from the relevant operating distribution. The gap between them is not a moral defect called overfitting. It is the cost of selecting a predictor from finite evidence.

The adjacent article Optimisation, Approximation and Guarantees asks whether the trained objective bridges to the decision that matters. The narrower question here begins when that objective permits several credible fits: which explanation should the learning procedure prefer?

Classical capacity stories say that a larger hypothesis class can fit more accidental patterns. That remains useful, but modern neural networks complicate it. Large networks can fit random labels, showing that architectural capacity alone does not explain why they generalise on natural labels. Interpolating models can also enter a second regime where test error falls again, and minimum-norm interpolators can achieve low risk under specific data spectra. These results do not abolish regularisation. They remove the lazy equation “zero training error equals bad explanation”.

The relevant object is the combination of model, data, optimisation and stopping rule. Gradient descent does not inspect every parameter vector equally. Augmentation changes the empirical distribution. Dropout trains a stochastic family of subnetworks. Weight decay modifies the update. Early stopping changes how far the optimiser travels. A regulariser is part of the learning algorithm, not decorative insurance attached afterwards.

A useful unifying idea comes from algorithmic stability. Informally, a learning algorithm is stable when replacing or removing one training example changes its loss on another example only slightly. Bousquet and Elisseeff formalised several versions of this idea and connected them to generalisation bounds. Hardt, Recht and Singer later analysed stochastic gradient methods through the same lens, showing how fewer iterations can promote stability under their assumptions.

This does not prove that every successful deep model is stable in one universal metric. It gives a disciplined causal question: what perturbation should leave the learned function mostly unchanged? A practical answer needs four fields.

The stability contract is the paper’s original decision instrument. It turns “regularise more” into a falsifiable statement. The intervention should reduce sensitivity to declared nuisance variation, preserve sensitivity to valid signal, and improve the operating metric under a matched test. If it merely lowers a norm or smooths a chart, the causal burden remains unmet.

Imagine a tiny building dataset with two standardised inputs. xt is a calibrated thermometer reading. xs is a numerical encoding of the sensor’s serial batch. The target y is the independently measured room temperature. In the training building, older sensors happen to be installed in colder rooms, so xs and xt are almost the same direction.

Two linear predictors can now tell nearly indistinguishable training stories. Predictor A uses ŷ = xt. Predictor B uses ŷ = 0.5xt + 0.5xs. Because the two inputs are highly correlated in the sample, both have low empirical error. Predictor B even has the smaller Euclidean coefficient norm: about 0.71 rather than 1. A plain L2 penalty can therefore prefer the mixed explanation, despite the serial batch being a historical accident rather than a cause of room temperature.

Move the model to a second building where new and old devices are distributed across all rooms. Predictor A continues to follow temperature. Predictor B now moves when serial batch changes, even if the physical reading and true temperature do not. The failure was not visible in the original loss because the sample never varied the two candidate causes independently.

The decisive test is an intervention on one feature. Hold xt and y fixed, then permute valid serial batches across records. Predictor A is invariant. Predictor B changes by half the serial perturbation. That function-level sensitivity reveals the unsupported explanation more directly than its parameter norm. A targeted augmentation could train on these permutations, or the data collection plan could deliberately include warm rooms with old sensors and cold rooms with new ones.

Now change one fact. Suppose serial batch identifies a documented calibration offset that the raw reading has not yet corrected. Randomising the batch would erase valid information. The same permutation moves from a useful nuisance test to a wrong-invariance attack. The regulariser becomes justified only after the role of the feature is settled inside the task boundary.

This small case explains why the four levers cannot be ranked in the abstract. Weight decay sees coefficient geometry and may choose the wrong mixture. Noise on the thermometer may damage the trusted signal. Early stopping may first learn the dominant correlated direction, which also mixes both inputs. A semantically targeted augmentation can separate them, but only if the serial truly is irrelevant. Regularisation works through a declared model of variation, not through an automatic preference for simpler-looking parameters.

A constant predictor is perfectly stable under sample replacement, input noise and augmentation. It is also useless whenever the target varies. A shortcut can be stable under the perturbations chosen by the team while failing under an untested change. Stability can rule out a fragile explanation; it cannot establish that the surviving explanation is correct.

A complete evaluation therefore asks three separate questions. Does the predictor achieve acceptable loss on representative operating cases? Does it remain steady under changes declared irrelevant? Does it remain appropriately sensitive, or equivariant, when a target-relevant feature changes? The last question is the natural negative image of invariance. A credit-risk score should be insensitive to document compression artefacts, yet responsive to a verified change in income or indebtedness. Passing only the first half can produce a smooth system that ignores evidence.

Calibration adds another independent burden. Two predictors can have the same classification stability while attaching different confidence to their answers. The more confident one may create larger downstream harm when both are wrong. For this reason, the consequence field in the contract names the decision metric and tolerance rather than treating unchanged class labels as sufficient.

The paper’s claim is consequently narrower than “stable models generalise”. Regularisation is useful when it helps select a predictor whose invariances and sensitivities agree with the task, and when that selection improves independent outcome evidence. Stability is the mechanism under test; correctness, calibration and consequence remain release conditions.

For ordinary gradient descent, adding an L2 penalty produces an update that shrinks every parameter before following the data gradient. If η is the learning rate and λ the decay strength, the update is approximately:

Small coefficients can be a useful proxy for a stable function when features share a meaningful scale. The proxy becomes unreliable when parameterisations have symmetries. Two ReLU networks can represent the same function while carrying very different layer norms. Dinh and colleagues showed that equivalent reparameterisations can make minima arbitrarily sharper without changing generalisation. The same caution applies to raw weight magnitude: parameter geometry is evidence about the function only after scale and symmetry are controlled.

Optimiser choice also matters. With standard stochastic gradient descent, L2 regularisation and multiplicative weight decay align closely. In adaptive optimisers, adding an L2 term to the loss is not generally the same as decaying weights directly because coordinates are rescaled by the optimiser. AdamW was introduced to decouple the two operations. A configuration record should therefore state the actual update rule, not merely “L2 used”.

Injecting small random noise changes the objective from performance at one point to expected performance in a neighbourhood. For squared error and sufficiently small input noise, a Taylor expansion adds a term related to derivatives of the learned mapping. Bishop showed the connection to Tikhonov regularisation. The intuitive result is simple: a predictor that changes violently around each observation pays more expected loss.

Dropout applies the idea to internal pathways. Each update sees a thinned network, so a useful representation cannot rely completely on one fragile coalition of units. The original dropout paper interpreted testing as an efficient approximation to averaging many thinned networks. Wager, Wang and Liang connected dropout to adaptive regularisation in generalised linear models. These accounts overlap, but they are not identical guarantees for arbitrary deep architectures.

The failure mode is equally clear. Noise is helpful only when the sampled neighbourhood mostly preserves the target. Too much input noise blurs a narrow feature. Too much activation noise can reduce capacity or destabilise optimisation. In sequential control, stochastic policies can also change the data the system encounters. Noise regularises the neighbourhood it samples, whether or not that neighbourhood is meaningful.

Augmentation replaces each observed example with an orbit of transformed examples. A crop, translation, colour change, paraphrase or simulated sensor perturbation says that the label should survive that transformation. The objective becomes an expectation over transformations T sampled from an augmentation policy:

Mixup extends this idea by training on convex combinations of examples and labels. AutoAugment searches over augmentation policies. These methods can improve benchmark performance, but their success does not make augmentation semantics-free. An automated search optimises a validation objective inside a particular data regime. It cannot decide whether a rotation, crop or colour change is legally, clinically or operationally irrelevant.

Train two image classifiers on photographs of gloves. The first task predicts material: leather or fabric. The second predicts handedness: left or right. Apply the same horizontal flip to both datasets. In the material task, the flip plausibly preserves the label. In the handedness task, it reverses the label. Only one causal feature changed, but the augmentation moved from useful invariance to systematic label noise.

Now make the case less obvious. Suppose a document classifier uses page orientation. Rotating a scan by two degrees may represent scanner noise; rotating it by ninety degrees may hide a layout cue that distinguishes a form from a letter. The correct policy is task-conditional and often subgroup-conditional. Augmentation is a claim about the world, not a free multiplication of data.

Early stopping regularises through time. In linear least squares, gradient descent decomposes along the singular directions of the feature matrix. Directions with large singular values are learned quickly. Weak directions accumulate slowly and often contain more noise. Stopping after a finite number of steps acts like a frequency-dependent filter rather than a uniform penalty.

Prechelt’s empirical study made validation-based stopping systematic and exposed the trade between training time and generalisation. Stability analyses later supplied formal interpretations under smoother assumptions. Recent work on discrete gradient descent for least squares shows an equivalence between finite-time solutions and a generalised ridge problem under broad learning-rate schedules, while also identifying conditions where early stopping is not beneficial.

The caveat matters. Neural networks learn features as well as coefficients, so a fixed spectral story is incomplete. Late training can sometimes improve generalisation, as grokking illustrates. A noisy validation set can select the wrong checkpoint, and repeated tuning can overfit the validation split. Early stopping is a hypothesis about learning order, not a law that later knowledge is always worse.

These interventions can substitute for, amplify or obstruct one another. Strong augmentation may already control the variance that weight decay was meant to suppress. Decay can change optimisation speed, so a stopping rule tuned without it may stop at a different functional state. Dropout changes gradient noise and effective learning rate. Batch normalisation introduces scale symmetries that alter how decay acts. Golatkar, Achille and Soatto found that weight decay and augmentation can matter most during early learning, with much smaller effects when introduced near convergence.

The practical implication is methodological. Compare each intervention against a matched baseline, then run removal and timing ablations. A result from “all regularisers on” cannot identify which assumption earned the gain. It can also hide a harmful component behind a stronger helpful one.

The executable lab makes the argument small enough to inspect. It samples eighteen noisy observations from a smooth public synthetic function and fits a degree-twelve polynomial in a Legendre basis. The unregularised model trains for 8,000 gradient steps. Four alternatives change one mechanism: stop after 300 steps, apply weight decay, jitter the inputs, or combine mild versions.

The test set is a dense grid from the known synthetic target. This is not a benchmark and does not estimate real-world performance. It is a controlled specimen: every method sees the same observations, basis and evaluation distribution. The positive case asks whether the interventions suppress high-frequency sample noise. The negative case adds a narrow but genuine bump around x = 0.25, then applies aggressive jitter while keeping labels fixed. That augmentation now encodes a false invariance.

A regularisation argument is weak if its baseline is designed to fail. Modern theory supplies a stronger control: some interpolating solutions generalise well because the optimiser selects a favourable minimum-norm or spectrally aligned solution. In such regimes, explicit decay may add little, and stopping early may interrupt useful learning. The question is no longer whether the model reached zero training error. It is which interpolator was selected and how its error aligns with the data covariance.

This observation weakens any universal claim that regularisation improves generalisation by reducing capacity. It does not weaken the stability-contract claim. A benign interpolator can itself be stable under the relevant perturbations, or can allocate fitting error to low-signal directions whose population cost is small. The empirical test remains the same: perturb the data, inputs and training path, then measure function change where consequence lives.

Consider a composite document-classification scenario. A model distinguishes payslips from bank statements. In the training set, most payslips came from scanner A and most statements from scanner B. A faint corner watermark therefore predicts the label. The text and layout also carry valid evidence, but they require a more complex representation.

The team adds crops, contrast jitter and weight decay. Average validation accuracy rises. On documents from a newly introduced scanner, performance improves because the watermark is no longer dependable. The intervention appears successful. A subgroup review then finds that a small payroll provider prints a compact code in the same corner. Aggressive cropping removes a valid issuer cue and worsens that provider’s error.

The correct conclusion is not that cropping failed. It is that the original augmentation policy bundled two assumptions: scanner watermarks are nuisance, and all corner content is nuisance. The first was supported; the second was false for one subgroup. A production regulariser should be evaluated as a targeted intervention on a named shortcut, not as a generic accuracy enhancer.

A production training recipe often begins as inheritance: use weight decay because the reference configuration used it, use augmentation because the benchmark did, and stop when validation loss rises. That can be a sensible starting point. It is not yet an explanation. The release decision should record why each intervention belongs to this task and what observation would remove it.

The tuning procedure itself needs a contract. Trying fifty augmentation strengths and reporting the best validation score gives that family fifty chances to fit quirks in the validation set. A baseline granted only three trials is not a matched competitor. Equalise search budgets, preserve an untouched test or temporal holdout, and report the dispersion across seeds and selected hyperparameters. Otherwise the apparent regularisation gain may be selection overfitting performed one level above the model.

Selection evidence should also follow consequence. When the release criterion is worst-group recall, choosing the checkpoint on average loss can silently optimise the wrong objective. The monitored metric, stopping patience and tie-break rule are part of the learning algorithm. A regularised model is the output of the entire selection procedure, not merely a set of weights trained with a penalty.

The proposed explanation would lose force if matched experiments repeatedly found that improved unseen performance arrived without improved stability under any task-relevant perturbation, while more unstable predictors won under independent shifts and subgroup tests. It would also weaken if a claimed regulariser helped only by fixing optimisation, such as preventing exploding activations, with no evidence of a selection effect after training quality was matched.

That possibility should remain open. Regularisation is a family name covering penalties, stochastic training, data design and stopping rules. Some methods can improve conditioning, calibration, margin, ensemble averaging or representation learning through mechanisms not captured by one stability measure. The contract is therefore an experimental frame, not a complete theory of deep-network generalisation.

The argument changes three practical decisions. First, data augmentation policy becomes part of the task specification and data contract, not merely a training script. Domain owners should approve transformations that encode semantic invariance, especially for regulated or safety-relevant labels.

Second, training lineage should record regulariser type, strength, timing and optimiser semantics. “Weight decay 0.01” is incomplete without exclusions, schedule and update rule. “Early stopped” is incomplete without monitored metric, patience, checkpoint selection policy and the number of tuning attempts against the validation set.

Third, evaluation should preserve perturbation suites and subgroup slices as release evidence. A model update can keep headline accuracy while changing which explanation it relies on. Re-running the stability contract makes that change visible before production drift does.

Regularisation does not rescue a model by making it modest. It chooses which variations the learner should distrust. Weight decay distrusts some parameter directions. Noise distrusts local sensitivity. Augmentation distrusts changes inside a proposed transformation orbit. Early stopping distrusts directions that require more optimisation time to acquire.

The decision is defensible only when that distrust has a domain meaning and an empirical test. Start with the nuisance, not the technique. Declare what should remain invariant, where the claim stops, and which rare signal must survive. Then compare matched learners, inject the proposed perturbation, and include a case where the same intervention ought to fail.

The changed architecture decision is straightforward: regularisation settings belong in the evidence-bearing task contract, alongside data semantics, optimiser behaviour, subgroup outcomes and release conditions. A stable explanation is not the smoothest one available. It is the one that remains steady under irrelevant change and remains responsive to the change that matters.