Batch normalisation, layer normalisation and their relatives alter activations or weights using statistics or norms. They are credited with stabilising training, permitting larger learning rates, accelerating convergence and sometimes improving generalisation. Those effects arrive together in practice, so replacing the layer is not one question. It is at least two: can optimisation remain stable, and can held-out performance remain competitive?

The authors target residual networks because their identity skip already gives depth a special structure. A block computes x_l+1 = x_l + F_l(x_l). If F were zero, information and gradients could follow the identity route. This makes a zero or near-zero residual branch less destructive than a zeroed plain layer. Yet ordinary He or Xavier initialisation makes F nonzero, and thousands of additions can accumulate.

The paper’s name expands to fixed-update initialisation. Its goal is not merely fixed activation variance. It reasons from the change in the whole network function after a stochastic-gradient step, then chooses branch scale so the update stays comparable as depth grows. This top-down target distinguishes Fixup from heuristics that divide each branch output by a constant without tracing how many branch updates add together.

The method removes normalisation statistics entirely. It does not compute mini-batch means, channel variances or weight norms during initialisation or training. That makes it attractive where batch-dependent behaviour, memory, communication or sequence-length interactions are problematic. The primary nevertheless evaluates scientific claims about trainability and accuracy, not every operational advantage a later system might obtain.

A successful replacement also needs fair controls. Fixup keeps the learning rate used by normalised baselines in its depth study, then adds explicit regularisers where generalisation requires them. The paper does not claim that initialisation alone reproduces every implicit regularisation effect of BatchNorm. Indeed, its ImageNet diagnosis depends on Fixup fitting the training set too well and overfitting more.

The contribution is therefore a separation experiment. Stable high-rate optimisation can be achieved without activation normalisation when initial branch scale and parameterisation control the update. Generalisation remains a second design problem. Preserving that separation prevents the headline “without normalization” from becoming the false conclusion “normalization never matters”.

Consider x_l+1 = x_l + F_l(x_l) at random initialisation. Under the paper’s simplifying condition that the branch has conditional mean zero, total variance decomposes into the input variance plus the expected conditional variance of F. The second term is nonnegative. Thus variance cannot decline as blocks accumulate.

Preventing decline is useful: the identity path resists the vanishing-signal problem of a long plain composition. But ordinary residual initialisation can make each branch output comparable in scale to its input. Repeated addition then grows variance rapidly: exponentially under the paper’s standard-initialisation analysis. The skip is not a universal stabiliser; it changes the direction of the danger.

The authors connect activation scale to gradients using positive homogeneity. ReLU, linear maps and certain other operations satisfy f(αx) = αf(x) for positive α. Under stated assumptions, the input-gradient norm of cross-entropy has a lower bound related to the loss above the prediction entropy. When logits grow, the loss and gradients cannot all remain benign.

This is an initialisation analysis, not a theorem that every trained ResNet’s variance follows one curve. It fixes inputs and considers randomness in weights; it uses sequential blocks composed of positively homogeneous functions and a zero-mean symmetric final layer. Biases, normalisation and later optimisation alter the conditions. The simplified theory guides a recipe rather than exhaustively modelling training.

Figure 2 in the paper uses three paths through a residual network to explain accumulated updates. A parameter change in an early branch can influence output along many skip-and-branch routes. More importantly, branch updates can be highly correlated because every branch responds to the same loss signal and identity-connected features. Treating them as independent noise underestimates their sum.

This is why merely keeping forward activations finite is not sufficient. Two initialisations can have acceptable variance yet produce different functional change after one update. Fixup chooses the update of the network function as the governing quantity. The scale of a weight is a means; controlled learning dynamics are the intended end.

The paper defines the change Δf(x) produced by one stochastic-gradient step and studies its scale at initialisation. This differs from tracking only parameter norms. A tiny parameter movement can have a large functional effect if many paths amplify it, while a larger movement can matter little along an inactive route. The function is the common currency across layers.

Suppose L branches each change output by roughly η and those changes align. Their sum is roughly Lη. At depth 10,000, a learning rate safe for a shallow model can become catastrophic. If each branch instead contributes η/L, their correlated sum remains η. This is the fixed-update budget in its simplest form.

Why assume correlation rather than square-root cancellation? Residual branches are trained on the same example against the same output loss and are connected by identity routes. Appendix B argues that updates contributed by different branches accumulate in sync. An independence assumption would predict √L scaling and recommend too little shrinkage for the aligned case.

The Θ notation hides constants and architecture details. It describes how scale changes with L, not an exact numerical equality for every neuron. Fixup translates that asymptotic target into a practical exponent for branches of m weight layers. The laboratory later makes the simplified proportionalities explicit rather than presenting them as a literal simulator of a trained network.

The first-step focus is deliberate. Initialisation chooses the point from which optimisation begins, so its strongest guarantee concerns the initial scale. Scalar multipliers, biases, learning-rate schedules, regularisation and learnt weights then shape later dynamics. Stable first steps are necessary for very deep training at a high rate, but not sufficient for convergence or generalisation.

A top-down budget also clarifies why all-zero residual weights are not ideal. They make branch output zero, but symmetry and ReLU conventions can stop useful internal gradients. Fixup zeroes only the last branch layer while giving earlier layers scaled random features. The branch initially contributes nothing, yet its final layer can learn immediately from nonzero upstream activations.

A branch’s first useful update depends on products of weights along its internal path. If the last of m layers starts at zero, the branch output is zero but the gradient of that last layer depends on the preceding m−1 layers. Scaling each of those weights by s multiplies their combined path by s^m−1.

Set s = L^{−1/(2m−2)}. Raising this to m−1 gives L^{−(m−1)/(2m−2)} = L^−1/2. The functional change produced by updating the zero final layer contains this path scale twice in the simplified analysis: once in its activation and once in its gradient: so the branch update is proportional to L⁻¹. Summing L aligned branches returns a depth-independent order.

Parentheses prevent a common mistake. The exponent is negative one divided by 2m−2, not −1/2 times m minus 2. When m = 2, it is −1/2. When m = 3, it is −1/4. Longer branches need less shrinkage per weight because more scaled weights multiply along the path.

L counts residual branches, not every convolution and not nominal model depth printed in an architecture name. Translating a paper formula into code requires identifying which blocks contribute residual branches and how many weight layers m sit inside each. Bottleneck blocks and basic blocks therefore receive different exponents even at the same L.

The rule scales only weight layers inside residual branches. The classification layer and last branch layers follow the zero-initialisation rule instead. Shortcut projections and non-residual components require architecture-aware handling. A blanket multiplication of every parameter by the exponent is not Fixup.

The derivation assumes branches of a common structure for clarity. Real networks can have stages, changes of width and different branch types. The paper turns the theory into a template and validates particular ResNet and Transformer implementations. Extending the rule to heterogeneous graphs is an engineering judgement that should be tested rather than treated as a consequence already proved.

Rule one initialises the classification layer and the last weight layer of every residual branch to zero. Every block therefore begins as its shortcut, and initial logits from the classifier are zero. The network does not have to unlearn arbitrary residual corrections before developing useful ones.

Rule two applies a standard method such as He initialisation to every other layer, then multiplies only weights inside residual branches by L^{−1/(2m−2)}. The paper calls this rule essential: it is sufficient and necessary in its experiments for training extremely deep residual networks. It instantiates the fixed-update derivation.

Rule three adds a scalar multiplier, initially one, in every residual branch and scalar biases, initially zero, before each convolution, linear layer and element-wise activation. These are scalar controls rather than per-channel normalisation parameters. They restore flexible shifts and branch scale without measuring activation statistics.

The multiplier changes later effective learning dynamics. With weight decay and learning-rate decay, weight norms and multiplier growth can interact. The authors report that one multiplier per branch mimics useful weight-norm dynamics and avoids searching for a different learning-rate schedule. This is an empirical rationale, not a theorem that the multiplier duplicates all effects of a normalisation scale parameter.

Bias placement matters. “Before each convolution, linear, and element-wise activation” means a scalar shift appears at several points in a branch. A conventional convolutional bias alone is not the same parameterisation. The paper removes ordinary vector biases and introduces O(K) scalar parameters across K layers, contrasting this with the larger parameter counts of normalisation schemes.

Zeroing every layer would be different. In a ReLU branch it can block gradients or preserve symmetry, so useful internal features do not form. Fixup retains random, depth-scaled earlier layers and zeroes only the last. Those earlier activations give the last layer something to learn from immediately; once it moves, gradients reach deeper branch weights.

The study uses a Wide-ResNet architecture with width one, weight decay 5×10⁻⁴ and the default BatchNorm learning rate of 0.1. It compares Fixup with BatchNorm, LSUV and an approach that rescales each residual block output by 1/√2. Batch size is 128 through 1,000 layers and 64 at 10,000 because of computational demands.

The budget is deliberately one epoch. Figure 3 asks whether the network starts learning at the aggressive rate as depth increases. Fixup tracks BatchNorm’s first-epoch accuracy even at 10,000 layers. LSUV and 1/√2 scaling fail to train at that same rate beyond 100 layers under the plotted protocol.

This does not mean a 10,000-layer Fixup network reaches a superior converged test result. It means gradients and functional steps remain usable long enough to improve accuracy during the first pass through the data. Final optimisation cost, memory, inference latency and generalisation are separate questions.

A smaller learning rate could stabilise other initialisations, and the paper acknowledges this in a footnote. But lowering the rate can reduce convergence speed. Holding the 0.1 rate fixed turns the study into a stronger comparison of initial scaling: can the method tolerate the rate that works with BatchNorm?

The baselines test alternative intuitions. The 1/√2 output factor controls forward variance locally but does not account for the number and length of branches in the functional update. LSUV post-processes an orthogonal initialisation to obtain unit variance layer by layer. Their failure at extreme depth supports the paper’s focus on correlated update accumulation rather than activation variance alone.

The graph is evidence for the initial regime, not a universal law. Results depend on ReLU residual architecture, optimiser, dataset, rate, width and epoch. Fixup’s theoretical recipe travels farther than this one chart, but the chart’s exact scope should accompany any use of “10,000 layers”.

On the standard ResNet-110 CIFAR-10 protocol, BatchNorm records 6.61% test error. Xavier initialisation without normalisation records 7.78% and cannot use the large rate; Fixup uses the large rate and records 7.24%. The latter is a roughly seven per cent relative improvement over 7.78, while remaining behind BatchNorm.

The authors attribute the residual gap primarily to regularisation. Their evidence is behavioural: Fixup converges quickly and, on ImageNet, attains lower training error than normalised models but worse test error. If optimisation were the remaining problem, one would expect trouble fitting the training set. Instead, the unnormalised model fits too well.

ResNet-50 makes the contrast concrete. BatchNorm records 23.6% ImageNet test error; Xavier without normalisation 31.5%; Fixup alone 27.6%. Adding Mixup improves Fixup to 24.0%, close to GroupNorm plus Mixup at 23.9 and BatchNorm plus Mixup at 23.3. These rows differ in both normalisation and regularisation and should not be reduced to one causal switch.

For ResNet-101, BatchNorm records 22.0%, BatchNorm plus Mixup 20.8%, GroupNorm plus Mixup 21.4 and Fixup plus Mixup 21.4. The paper describes properly regularised Fixup as on par with GroupNorm. It does not beat the displayed BatchNorm-plus-Mixup row.

Tuning is explicit. The best Mixup coefficients selected by cross-validation are 0.2 for BatchNorm, 0.1 for GroupNorm and 0.7 for Fixup. Under strong regularisation, the learning rates for Fixup’s scalar multipliers and biases are reduced tenfold. Giving every method one shared coefficient would look symmetric but would not be the paper’s performance protocol.

Appendix C adds WideResNet experiments with Mixup and Cutout. On CIFAR-10, Fixup plus both regularisers records 2.3%, versus 2.5 for the displayed BatchNorm-plus-regulariser baseline. On SVHN both record 1.4%. These results support the separation: normalisation-free optimisation plus explicit regularisation can compete, but the regularisers are load-bearing parts of the claim.

The translation experiments use the fairseq implementation of the Transformer. The authors replace LayerNorm with the Fixup parameterisation rather than merely deleting LayerNorm and hoping that a conventional initialisation survives. Residual branches remain the relevant unit: their weights are scaled according to the total number of branches and the number of weight layers within each branch.

On IWSLT German-to-English, both systems receive an explicit dropout search over 0.3, 0.4, 0.5 and 0.6. Cross-validation chooses 0.5 for both. After 200,000 updates, LayerNorm scores 34.2 BLEU and Fixup scores 34.5. On WMT 2016 English-to-German, the chosen dropout is 0.4; after the same update budget both score 29.3 BLEU.

BLEU is higher-is-better, unlike the image tables’ error rates. That change of direction sounds trivial, yet it matters when assembling evidence across domains. The safe statement is that Fixup matches the reported LayerNorm baselines on these two translation protocols, with a small numerical advantage on IWSLT and a tie on WMT.

These results widen the empirical scope but do not show that normalisation is irrelevant to language modelling. The paper studies two supervised translation settings, one architecture family and one training budget. Later Transformer variants, larger models, different optimisers and generative objectives are outside its test. Nor is dropout removed: explicit regularisation remains at 0.5 or 0.4.

The transfer is informative because it preserves the structural reason for Fixup. A Transformer layer also adds learned residual branches to a stream. If each branch makes a depth-independent functional move, the aggregate update can grow with the number of layers. Allocating a depth-normalised update budget can therefore help even though the branch operators are attention and feed-forward networks rather than stacks of convolutions.

Still, the exact theoretical assumptions are cleaner than a full Transformer. The proof sketch relies on simplified positively homogeneous scalar networks and omits biases. Attention includes softmax operations and several interacting projections. The translation experiment therefore supplies empirical support where the derivation is not a literal model of every operator.

The comparison also clarifies what “without normalisation” means. Fixup removes activation normalisation and its running or per-example statistics, but it does not remove every stabilising choice. The architecture still has shortcuts; the optimiser still has a schedule; dropout still regularises; scalar biases and multipliers still adapt. A mechanism claim should name that entire support structure.

Rule two is the load-bearing intervention. The authors report that scaling residual weights according to depth is both sufficient and necessary in their very-deep training experiments. Rules one and three improve performance and make the full recipe competitive, but the prescribed branch scaling is what prevents the first functional update from growing with depth.

A factor ten smaller than the prescribed scale is not a safer version of the same method. It slows optimisation because the early branch pathways transmit weaker updates than the fixed-budget calculation intends. A factor ten larger is not merely faster: it can destabilise training. This Goldilocks behaviour is consistent with a target update scale, not with the vague advice “make residual weights tiny”.

Initialising all residual-branch weights to zero also performs poorly. Although the network begins as an identity map, internal features are absent and symmetries or zero derivatives can obstruct learning. Fixup’s asymmetric choice: random scaled inner weights, zero final weight: retains a nonzero feature basis while keeping each branch’s initial output exactly zero.

Removing the scalar biases degrades results in the ablation. Those biases provide shifts that activation normalisation would otherwise make easy to express. The branch multiplier begins at one and can grow during training, changing the effective scale of a branch as optimisation proceeds. Neither feature is captured by the exponent alone.

The theoretical argument is intentionally idealised. It uses positively homogeneous activations, simplified scalar weights, independence-style scaling estimates and an assumption that gradients across residual branches are initially aligned enough to add coherently. At a zero ReLU input the derivative convention also matters. The proof motivates the initial exponent; it does not solve the nonlinear trajectory after thousands of updates.

That boundary is productive rather than embarrassing. A useful theory identifies the dangerous quantity: the network’s functional update: and predicts a controllable intervention. Experiments then test the intervention in real convolutions and Transformers. The paper does not require the simplified network to be a faithful simulator of every hidden activation.

Fixup also belongs in a taxonomy. BatchNorm normalises using batch statistics and learns affine corrections. LayerNorm uses statistics within an example. GroupNorm partitions channels. Fixup computes none of these activation statistics; it changes initial weights and adds scalar parameters. Weight normalisation reparameterises individual weights and LSUV calibrates activation variance after initialisation. These mechanisms can share outcomes without being equivalent.

The engineering trade changes accordingly. Avoiding activation statistics removes dependence on batch composition and reduces the need for normalisation state. But it transfers responsibility to correct architectural accounting: one must count residual branches L, identify m for each branch type, zero the correct final layers and place scalar controls correctly. A silent miscount can preserve syntactic validity while defeating the derivation.

At the operator level, the contribution is exact enough to implement: count residual branches, count weight layers in a branch, apply the exponent, zero final branch layers and the classifier, and add scalar multipliers and biases. The executable laboratory below exposes the central arithmetic. Those are claims about parameterisation and the scale of an initial update.

At the theoretical level, the paper proposes a mechanism: residual branches produce correlated changes, so their functional updates add more like L than √L. Setting the effective path gain to L^−1/2 makes each branch spend about η/L and the whole depth spend about η. This is an explanatory model under stated assumptions.

At the experimental level, very deep image networks begin learning at a high rate; ordinary-depth image networks become competitive when regularised; Transformers match LayerNorm baselines on two translation tasks. These observations support trainability and portability. They do not directly measure semantic reasoning, causal understanding or autonomous planning.

A residual network can nevertheless be part of an agent. Stable optimisation may let a perception module grow deeper, preserve features across layers and learn useful representations. Better credit assignment can enable downstream capabilities. But “enables a component” is not “constitutes the property”. A camera’s sharper lens can improve navigation without itself choosing a destination.

The consciousness boundary is firmer. Loss reduction, high BLEU, low classification error and stable gradients are behavioural and mechanistic measurements. None is an assay for phenomenal awareness. Fixup does not offer a theory of subjective experience, a marker of sentience or a comparison between conscious and unconscious processing.

Anthropomorphic language is especially risky here because “the network decides how much each branch contributes” sounds natural. Literally, gradient descent changes weights and scalar multipliers according to an objective and data. Calling this a decision is a metaphor unless a separate agency criterion is specified and tested.

The right bridge to consciousness research is methodological. The paper shows how to isolate a failure mode, derive an intervention, stress-test it at an extreme, ablate its parts and delimit generalisation. Those habits are valuable for evaluating claims about intelligent or conscious systems. The result itself remains an optimisation result.

That separation protects both sides. It prevents a striking engineering feat: training 10,000 layers for an epoch: from being inflated into evidence about minds. It also preserves the genuine importance of the feat: a clean account of why signal preservation is insufficient and why the scale of learning-induced change deserves first-class analysis.