The paper begins with stochastic gradient descent. A mini-batch estimates a training-set gradient and suits parallel hardware. Yet each layer receives activations produced by all earlier layers, whose parameters change after every update. Ioffe and Szegedy call the resulting evolution of internal activation distributions “internal covariate shift”. They propose that later layers must repeatedly adapt to those changes and that sigmoid units may be pushed into saturated regions where derivatives are small.

This diagnosis motivates stabilising the inputs to each nonlinearity. Full whitening would centre features, scale them to unit variance and remove correlations. Doing that over the full training set after every parameter update is expensive, and differentiating an inverse covariance square root is awkward. The proposed simplifications are decisive: normalise each scalar feature independently and estimate its moments from the current mini-batch.

The normalisation is not an occasional data-cleaning step. It is inserted into the model. For batch B = {x₁,…,xₘ}, the layer calculates μᴮ, σ²ᴮ, x̂ᵢ and yᵢ. Because μᴮ and σ²ᴮ depend on every xᵢ, backpropagation includes those paths. Learned γ and β then allow the system to recover any necessary scale and offset.

The experiments range from a small sigmoid network on MNIST to an Inception variant on ImageNet. The ImageNet story includes more than adding a BN operator. To exploit the mechanism, the authors raise the learning rate, remove Dropout, reduce L2 regularisation, accelerate learning-rate decay, remove local-response normalisation, shuffle more thoroughly and reduce photometric distortion. The headline step reduction belongs to that configured bundle.

Later work would question the diagnosis. Networks can benefit from Batch Normalization even when measured distributional change is not reduced in the way “internal covariate shift” suggests. A later explanatory proposal emphasises a smoother optimisation landscape. That evidence does not erase the 2015 contribution. It sharpens citation: the algorithm and reported experiments are primary results; the proposed cause should be described as the authors’ account and tested separately.

The paper also has no direct consciousness claim. Normalising an activation is a computational transformation. It may affect training, robustness or representation geometry, but it does not establish awareness, subjective experience or agency. “Internal” in the title means inside a network, not inside a mind.

Write a network as loss ℓ = F₂(F₁(u,Θ₁),Θ₂). From the perspective of F₂, x = F₁(u,Θ₁) is its input. Updating Θ₁ changes x even if the original dataset u is fixed. Parameters Θ₂ are therefore adjusted against a stream whose mean, variance and higher-order structure may evolve during training.

For a sigmoid g(z), the derivative g(z)(1−g(z)) becomes small when |z| is large. If changing earlier layers moves pre-activations into those tails, gradients flowing backwards through the sigmoid shrink. Small learning rates and careful initialisation can limit the damage; the paper seeks a structural treatment that keeps inputs in a more responsive range.

The analogy to ordinary covariate shift needs care. Classical covariate shift concerns a change in the input distribution between training and testing under particular conditional assumptions. Here the change occurs during optimisation inside one model. The authors extend the phrase to sub-networks. That naming supplies an intuition, not an equivalence theorem.

The desired target is stronger than stable mean and variance. Fixing all aspects of a layer-input distribution would make a downstream optimisation problem stationary. Batch Normalization only standardises first and second marginal moments within each batch. Correlations, shapes and dependencies can still change. The method takes a tractable step towards the proposed goal.

Why might even that help? Centring prevents a feature’s offset from drifting far into a saturating nonlinearity. Scaling reduces sensitivity to arbitrary parameter magnitudes. Batch dependence adds noise. Learned γ and β preserve expressiveness. Later analyses suggest that reparameterisation can make gradients and losses behave more predictably. These mechanisms can coexist; the primary evidence does not uniquely select one.

A fair reconstruction therefore distinguishes observation from cause. Figure 1 in the paper shows more stable activation percentiles for a typical sigmoid in the BN network and faster MNIST training. That is consistent with the proposed account. It is not a controlled mediation analysis showing that distribution stability alone caused the accuracy change.

Consider x = u + b and an external centring step x̂ = x − E[x]. Suppose an optimiser changes b by Δb while ignoring the dependence of E[x] on b. Recompute the mean afterwards: u + b + Δb − E[u + b + Δb] equals u + b − E[u + b]. The output is unchanged. The parameter can grow while the loss stands still.

Scaling can magnify the mismatch. If a whitening transform is treated as an occasional repair, each parameter step targets co-ordinates that the repair immediately changes. The gradient is not the derivative of the actual composed system. This is not a small implementation detail: it changes the optimisation problem being solved.

The remedy is to write x̂ = Norm(x,X) and differentiate with respect to both the focal example x and the set X used to calculate statistics. In a mini-batch, every activation contributes to μᴮ and σ²ᴮ. Backpropagation then assigns direct and indirect effects correctly.

Full whitening would use the covariance matrix and its inverse square root. It would also couple feature dimensions and require costly, sometimes ill-behaved matrix derivatives. Batch Normalization chooses independent scalar normalisation. It does not decorrelate features, but it is cheap, differentiable and compatible with stochastic batches.

The choice of mini-batch statistics solves another problem. Population moments over the entire training set would be expensive after every update because activations themselves change with network parameters. A batch supplies current estimates at the same cadence as SGD. The price is sampling noise and cross-example dependence.

This reasoning also explains why the learned bias before BN is redundant. Subtracting the batch mean removes a constant bias. Algorithm 2 replaces the pre-normalisation bias with β after standardisation. β has an observable effect because it is applied after centring.

First calculate the mini-batch mean μᴮ = (1/m)Σxᵢ. Then calculate the biased mini-batch variance σ²ᴮ = (1/m)Σ(xᵢ−μᴮ)². The paper uses denominator m during training, not m−1. Next standardise each example: x̂ᵢ = (xᵢ−μᴮ)/√(σ²ᴮ+ε).

If ε were zero and variance positive, the x̂ values would have batch mean zero and biased variance one. Positive ε makes the resulting variance slightly below one and prevents an undefined division when all batch values coincide. Implementations and tests must state ε; omitting it changes the arithmetic at small variance.

Finally output yᵢ = γx̂ᵢ+β. The scale γ and shift β are learned. Without them, every BN layer would permanently force its output to standardised units, potentially limiting what the following nonlinearity can represent. If γ = √(Var[x]) and β = E[x] under matching moments, the original values can be recovered.

Identity recovery is an expressiveness argument, not a promise that training will choose identity. The network can learn to widen, narrow or shift a feature as the task demands. For a sigmoid, β can locate the working point and γ can control how much of the curve is used.

A batch with m = 1 has zero variance for every scalar feature, so training BN cannot obtain useful within-batch standardisation. Very small batches produce noisy moment estimates. The original method is therefore tied to batch formation, though later normalisation families change the axes over which moments are computed.

For vector layers, the operation is applied independently to each feature. It centres and scales marginals; it does not whiten cross-feature covariance. That distinction keeps the algorithm cheap and differentiable but limits claims about complete distributional stabilisation.

Let dL/dyᵢ arrive from the network above. The direct affine paths give dL/dγ = Σ(dL/dyᵢ)x̂ᵢ and dL/dβ = ΣdL/dyᵢ. Gradients also travel through x̂ to the inverse standard deviation, through σ²ᴮ to every centred deviation, and through μᴮ to every input.

The resulting dL/dxᵢ contains three kinds of term: the example’s own normalised gradient, a correction for the batch-average gradient, and a correction correlated with x̂. A compact formula is (γ/√(σ²+ε))/m times [m gᵢ − Σgⱼ − x̂ᵢΣ(gⱼx̂ⱼ)], where gᵢ is dL/dyᵢ. This form makes the shared dependencies visible.

If one training example changes, it can alter μᴮ, σ²ᴮ and therefore every x̂ in the batch. The same image paired with different companions can produce a slightly different training activation. The primary suggests this stochasticity regularises the model and reports that Dropout could sometimes be removed or weakened.

Regularisation is an empirical effect, not an invariant property of every batch regime. Large batches reduce moment noise; correlated or poorly shuffled batches can bias it. The authors found more thorough within-shard shuffling improved validation accuracy by about one percentage point, consistent with the idea that varied companions produce more useful randomness.

Coupling also affects evaluation design. Comparing a single training example under two configurations while silently changing its batch can confound the effect. Reproducible probes should hold batch membership fixed or switch the model to inference mode, depending on the question.

The many-to-many derivative is why a naïve per-example numerical check is valuable. Perturb one xⱼ, recompute the whole batch, and compare the change in total loss with the analytic gradient. A formula that treats μ and σ² as constants will fail that test.

During training, each mini-batch supplies μᴮ and σ²ᴮ. Those stochastic statistics match the SGD rhythm and permit differentiation through current activations. During inference, mean and variance are estimated from the training population, commonly through moving averages accumulated during training. The resulting transformation is deterministic for one input.

The variance estimator requires attention. Mini-batch training uses the biased denominator m. To estimate population variance from batch variances, the paper gives Var[x] = m/(m−1) Eᴮ[σ²ᴮ] for batches sampled in the stated way. Modern library details can differ, so reproducing a result requires implementation-specific momentum and variance conventions.

Once population μ and Var are fixed, BN followed by γ and β is a linear transformation of x: y = γ(x−μ)/√(Var+ε)+β. It can be folded into adjacent linear weights for inference. The conceptual train–serve boundary remains even if the deployed graph no longer shows a separate operator.

Using batch statistics at inference would make a patient’s score, a photograph’s class or a control action depend on other cases in the same serving batch. It would also fail for batch size one. Conversely, accidentally using stale inference statistics during training removes the intended stochastic coupling and can shift activations.

The separation introduces a possible mismatch. If deployment data differ from the population used to accumulate moments, frozen statistics may centre and scale poorly. Updating them changes system state and needs governance. The original paper assumes training-derived population estimates; domain shift remains a separate problem.

For evaluation, “model mode” is therefore part of the protocol. Validation curves should use the intended inference transform. A reported accuracy without stating whether moment estimates were frozen, recomputed or adapted leaves an important degree of freedom unresolved.

A fully connected activation dimension has one scalar per example, so its BN batch contains m values. A convolutional feature map contains values at many spatial positions for each example. Normalising every location with unrelated parameters would undermine the assumption that the same feature has the same meaning across location.

The paper therefore jointly normalises all activations in a feature map over the mini-batch and spatial positions. For a batch of m and a feature map of size p×q, the effective collection has m·p·q values. At inference, one population mean and variance and one learned γ and β are applied across that map.

This increases the number of samples used for moment estimation, though spatial activations are correlated and therefore not equivalent to independent examples. It also means padding, resolution and feature-map geometry can influence statistics. The operation’s axes are part of its definition.

Placement matters. The primary inserts BN before the nonlinearity, replacing z = g(Wu+b) with z = g(BN(Wu)). The pre-BN bias is omitted because centring removes it. Later architectures sometimes use different ordering, but claims about the 2015 experiments should retain the paper’s placement.

Convolutional BN demonstrates a broader lesson from Paper 0052. Architecture encodes a prior: a feature’s scale and offset are shared over position. Normalisation is not neutral to representation; it chooses which activation instances belong to one statistical population.

For sequence, graph or spatial models, the same question recurs. Which axes are exchangeable enough to share moments, and which must remain distinct? Batch Normalization answers for the architectures in the paper; other normalisation schemes answer differently.

Ignoring ε and sign complications, BN(Wu) equals BN(aWu) for positive scalar a. Scaling all incoming weights changes the pre-activation mean and standard deviation by the same factor, which cancels in x̂. The forward function becomes insensitive to that parameter scale.

The gradient with respect to scaled weights changes inversely: differentiating with respect to aW introduces a factor 1/a. Larger weight norm therefore yields a smaller parameter gradient, providing a form of self-stabilisation. The layer Jacobian with respect to u is unchanged under the stated scale transform.

Epsilon weakens exact scale invariance when variance is comparable to ε. γ, β, weight decay, optimiser state and finite precision add further context. A paper-faithful laboratory can test the idealised property and separately display the epsilon deviation rather than stating universal equality.

The authors go further with a conjecture. If consecutive normalised vectors were Gaussian and uncorrelated, and the local transform were linear, unit covariance on both sides would imply JJᵀ = I and singular values one. They immediately note that real transformations are not linear and normalised values are not guaranteed Gaussian or independent. The claim is a heuristic for gradient flow, not a proved property of trained BN networks.

Empirically, BN permitted higher initial learning rates and less sensitivity to initialisation. It also reduced the need for Dropout in these experiments. Those benefits interact: noise can regularise, scale reparameterisation can alter effective step sizes, and centring can keep sigmoids responsive.

A clean causal question changes one mechanism at a time. The production-minded question may instead ask which whole recipe trains fastest at target accuracy. The ImageNet section primarily answers the second after first reporting a BN-only baseline.

The BN-Baseline comparison is the cleanest operator addition: the Inception variant is kept otherwise constant and BN is inserted before each nonlinearity. It reaches 72.2% in 13.3 million steps, less than half the baseline’s 31 million, and attains a slightly higher maximum single-crop validation accuracy of 72.7% versus 72.2%.

BN-x5 exploits the new regime. Its initial learning rate is five times larger, Dropout is removed, L2 weight regularisation is reduced fivefold, learning-rate decay is six times faster, local-response normalisation is removed, shuffling is improved and photometric distortions are reduced. It reaches 72.2% in 2.1 million steps and a maximum 73.0%. The paper describes this as fourteen times fewer steps.

BN-x30 uses an initial learning rate thirty times the baseline. It starts somewhat more slowly than BN-x5 but reaches 74.8% after six million steps. The original Inception configuration diverged to machine infinity when given the fivefold rate, showing that BN changed which hyperparameter regime was usable.

A sigmoid variant reaches 69.8%, while original Inception with sigmoid remained near chance. This supports the claim that the mechanism made saturating nonlinearities trainable in that architecture. It does not show that sigmoid outperformed ReLU; the ReLU variants remained stronger.

The ensemble result uses six modified BN-x30 networks, multi-crop inference and an arithmetic average of class probabilities. It reports 4.9% top-5 validation error and 4.82% test error. The single BN-Inception model has 7.82% top-5 error with one crop and 5.82% with 144 crops. Those are different systems and compute budgets.

The abstract’s phrase about exceeding estimated human accuracy belongs to the ImageNet top-5 protocol and the cited estimate. It is not a claim that the system sees as humans do or possesses general visual understanding. Benchmark categories, crops and ensemble voting define the comparison.

The forward transformation is directly specified. Given x, ε, γ and β, its output is calculable. The backpropagation paths follow from the chain rule. Training and inference modes are explicit. These are the strongest parts of the paper because they can be reproduced independently of the authors’ narrative.

The experiment tables establish outcomes for named networks, datasets, crops, ensembles and training schedules. They show faster step-wise convergence, accessible high learning rates, successful sigmoid training and improved ImageNet error. They do not alone reveal which internal quantity mediates those outcomes.

The proposed internal-covariate-shift account is supported by activation-percentile plots and intuition about downstream adaptation. Later controlled work found that deliberately adding distributional change did not remove BN’s benefit and argued that the reparameterised loss landscape became smoother and gradients more predictive. That is evidence against treating reduction of shift as the sole or necessary explanation.

Later results also do not license a single replacement slogan. “Smoothness” can refer to several measurements, and behaviour changes with architecture, batch size, optimiser and objective. The responsible conclusion is narrower: causal mechanism should be separately tested; the operation’s success does not depend on our repeating its original diagnosis as settled fact.

The paper’s regularisation claim is contextual. Batch composition makes training activations stochastic, and Dropout was removable or reducible in the reported systems. Small or non-i.i.d. batches may behave differently. One should measure held-out performance, sensitivity to batch construction and train–inference discrepancy.

Finally, “exceeding human raters” is an evaluation statement about top-5 classification under a benchmark, not an ontological statement about perception. The method neither represents a theory of mind nor tests consciousness. It belongs in this canon because normalisation changes the conditions under which internal representations can be learned, not because it makes those representations experienced.

Paper 0052 asked what makes a representation useful: smoothness, reusable factors, hierarchy, sparsity, coherence and disentangling. Paper 0053 moves from those desiderata to optimisation mechanics. It changes the statistical co-ordinates in which a layer learns, then lets the layer recover useful units through γ and β.

The link to earlier canon is equally direct. Backpropagation in Paper 0048 must now traverse group statistics. Momentum in Paper 0044 operates on gradients altered by the reparameterisation. The ImageNet systems in Papers 0050 and 0051 supplied the depth, data and benchmark setting in which improved conditioning and higher learning rates became consequential.

Paper 0054, He, Zhang, Ren and Sun’s residual-learning paper, will change the path rather than the scale. Identity shortcuts let deep networks learn residual functions and create direct routes for information and gradients. Batch normalisation remains inside those residual blocks, so Paper 0053 supplies an immediate prerequisite.