Consider a feed-forward network before meaningful learning has begun. Each layer forms weighted sums, applies a nonlinearity and hands its activations upward. Back-propagation traverses the same stack in reverse, multiplying an output error by derivatives and transposed weight matrices. If a typical layer transmits only half the relevant magnitude, five such layers transmit roughly one thirty-second. If it transmits twice as much, five layers amplify by thirty-two. The issue is not a single catastrophic gate. It is the product of many ordinary ones.

Glorot and Bengio’s contribution is methodological as much as algebraic. They do not infer trainability from final error alone. They monitor activation distributions, back-propagated gradients, weight gradients and layer Jacobians during training. These views separate several mechanisms that the phrase “vanishing gradients” can blur. A unit may saturate because its input moved into a flat tail. A backward signal may contract at every layer even while weight-gradient variances initially look similar. A network may eventually escape a plateau, yet consume most of its training budget doing so.

The paper arrives after unsupervised layer-wise pretraining had repeatedly made deep networks easier to optimise. Rather than treating pretraining as magic, the authors ask which numerical conditions it may repair. Their answer is not that one scalar variance explains all deep learning. It is that scale is a controllable source of difficulty and that layer-by-layer statistics make the failure visible.

Shapeset-3×2 is the cleanest causal instrument. Each 32×32 image contains one or two objects drawn from three categories: triangle, parallelogram and ellipse. Scale, rotation, translation, grey level, geometry, overlap and relative position vary. Singles and unordered pairs with repetition yield nine target configurations. Because examples can be generated without limit, a persistent optimisation failure cannot be blamed simply on exhausting a small training set.

The finite datasets broaden the observation. The paper uses MNIST with 50,000 training, 10,000 validation and 10,000 test digits; CIFAR-10 with 50,000 labelled training images, 10,000 of those reserved for validation, and 10,000 test images; and a historical Small-ImageNet construction of 37×37 greyscale images with 90,000 training, 10,000 validation and 10,000 test cases in ten coarse classes. “Small-ImageNet” here is the authors’ 2010 dataset, not a claim about a modern ImageNet protocol.

The classifiers have one to five fully connected hidden layers, 1,000 units per hidden layer and a softmax output. Training uses stochastic back-propagation with mini-batches of ten and negative conditional log-likelihood. Learning rate and depth are selected separately by validation performance after five million updates. That separate tuning matters: comparisons do not punish one activation with another activation’s learning rate. It also means the study is not a single-setting ablation.

Three nonlinearities define the main families: logistic sigmoid, hyperbolic tangent and softsign. Biases begin at zero. The baseline weights use a zero-centred uniform interval whose half-width is `1/√fan-in`; the proposed rule enters later. The best Shapeset depth is five for all families except sigmoid, whose best depth is four: already an empirical hint that one extra layer can worsen a scale pathology.

Imagine a lower layer whose sigmoid outputs cluster around one half. The next layer sums many signed weights times positive activations. Although the weights have zero mean in expectation, any realised unit receives a finite random imbalance. Repeating that transformation through depth can move activation means. In the paper’s Shapeset trace, the top hidden layer rapidly moves towards sigmoid output zero, where the derivative is also near zero. Backward information then struggles to reach lower layers.

The distinction is geometric. For `tanh(x)`, the output and input are both zero at the operating centre, and the derivative there is one. For softsign, zero is likewise centred and locally responsive. For logistic sigmoid, output zero is not the centre corresponding to input zero; it is a limiting saturated state reached only at strongly negative input. Telling every activation to become “near zero” is therefore helpful for one family and harmful for another.

The reported four-hidden-layer sigmoid network eventually escapes. Around epoch 100 the top layer slowly desaturates, after lower layers have changed enough to send a more useful transformation upward. A five-hidden-layer model in the study never escapes. This makes a training plateau a dynamic episode, not necessarily proof of a permanent local minimum. The network may have a route out, but its derivatives make that route painfully slow.

The authors frame this as a hypothesis supported by traces, not a closed causal theorem. They note that sigmoid networks initialised through unsupervised pretraining do not exhibit the same behaviour. Pretraining may place activations and weights in a region where upper units avoid the saturated boundary. Yet many parameter dependencies develop during learning, so the initial independent-weight story cannot by itself explain everything that follows.

Hyperbolic tangent is centred: `tanh(0)=0`, and its derivative at zero is one. The baseline random start therefore places many units in a locally linear, responsive region. Yet the paper’s activation percentiles show a different problem developing. Under standard initialisation, the first hidden tanh layer saturates first, then the second, and so on. Values gather near −1 and +1, where slopes are small. Avoiding one saturated boundary has not prevented training from driving layers towards two others.

Softsign, `x/(1+|x|)`, shares tanh’s range and zero centre. Its difference lies in the tails. Tanh closes exponentially on its asymptotes; softsign approaches them polynomially. A large-magnitude softsign input therefore keeps more derivative than a tanh input of similar magnitude. In the reported traces, the softsign layers move towards larger weights together rather than saturating in a clear bottom-to-top sequence. Their final activation modes concentrate around the knees: roughly between −0.6 and −0.8 and between 0.6 and 0.8: rather than almost entirely at the extremes or at zero.

This is a useful correction to a simplistic rule. Perfect linearity would pass gradients but compute no layered nonlinear representation. Complete saturation supplies nonlinear states but little local sensitivity. Training needs a distribution of operating points that bends enough to form features while retaining slopes that carry credit. The paper studies this balance empirically through full distributions, not merely through their means.

Softsign proves more robust to whether the baseline or proposed initialisation is used in these experiments. “More robust” does not mean every softsign result improves under normalisation. On MNIST, Table 1 gives 1.64% for standard softsign and 1.72% for normalised softsign. The paper’s evidence supports an interaction among activation family, scale and dataset, not a global ranking of nonlinearities.

The main experiments minimise negative conditional log-likelihood with a softmax output. The paper contrasts this with the quadratic cost traditionally used in feed-forward classifiers. In a two-weight slice through a small tanh network, the quadratic surface contains more severe flat regions. Negative log-likelihood provides a steeper, more usable landscape in the shown slice.

This observation matters because back-propagation begins with the derivative of the loss. Even a perfectly scaled hidden stack cannot recover a learning signal that the output criterion has already weakened. Conversely, a strong output gradient can still be diminished by poorly scaled hidden Jacobians. Loss and initialisation occupy different links in the same causal chain.

The authors are careful about priority: better classification behaviour from log-likelihood was already known. Its inclusion serves diagnostic completeness. A study of deep training that changes only the hidden-layer distribution while leaving a severe output plateau intact may misattribute the failure. By choosing an appropriate classification loss, the authors reduce one source of flatness before comparing activation and scale effects.

Figure 5 in the primary is a local two-dimensional surface, not a global map of a high-dimensional objective. A smoother slice does not prove the absence of saddles, poor basins or plateaux elsewhere. It shows a concrete mechanism by which a particular loss can supply weak local gradients. The companion therefore uses the surface as an explanatory probe, not a certificate of convexity.

Let a layer receive independent, equally variable input features. Let its weights be independently drawn with zero mean, and suppose the activation operates close enough to linear that its derivative is approximately one. A weighted sum then adds `fan-in` independent contributions. The output variance is the input variance multiplied by `fan-in × Var(W)`.

Across layers these factors multiply. If layer `i` has incoming width `n_i`, the paper writes the variance at a later representation as the starting variance times the product of `n_i Var(W_i)` terms. Forward preservation asks each term to equal one. A layer with fan-in 512 would therefore like weight variance `1/512` under this linear account.

The baseline interval is `U[−1/√n, +1/√n]`. Because a centred uniform distribution on `[−a,a]` has variance `a²/3`, the baseline variance is `1/(3n)`, not `1/n`. Its forward multiplier is exactly one third when `n` is the fan-in. Five equal-width layers yield `(1/3)⁵ ≈ 0.00412`: less than half of one per cent of the starting activation variance under the idealised calculation.

This derivation is a microscope, not the whole organism. Real inputs are correlated, nonlinear derivatives are not always one, and training couples weights to activations. But a mechanism can be informative without being globally exact. The baseline already creates a strong contraction before any learnt dependence exists; preventing that avoidable handicap is a rational design target.

Back-propagation distributes the cost derivative from each receiving unit to its inputs. Under the same independence and linearity assumptions, the relevant multiplier is `fan-out × Var(W)`. Backward preservation therefore wants weight variance `1/fan-out`, whereas forward preservation wants `1/fan-in`.

For a 512-to-1,024 layer, the forward condition requests `1/512`; the backward condition requests `1/1024`. Choosing the first preserves activation variance but doubles the backward variance. Choosing the second preserves the backward variance but halves the forward variance. The disagreement is structural: it comes from the rectangular shape of the layer.

The paper adds a subtle observation. At initialisation, the variance of gradients on individual weights can be roughly equal across layers even while gradients on unit inputs shrink as they travel downwards. A weight gradient contains both the activation arriving from below and the error signal arriving from above. Their variance products can cancel the apparent depth dependence under equal widths. Looking only at weight gradients would therefore miss a transport failure in the hidden signals.

During learning the story changes. Weights cease to be independent of activations; units enter nonlinear regions; layers specialise. In the reported tanh network, initially similar weight-gradient magnitudes diverge, becoming larger in lower layers, especially under standard initialisation. The proposed normalisation keeps their scales more alike for longer. That reduction in inter-layer discrepancy may improve conditioning, but the paper does not reduce all learning dynamics to a static variance equation.

Forward flow asks for `fan-in × Var(W)=1`. Backward flow asks for `fan-out × Var(W)=1`. Glorot and Bengio choose `Var(W)=2/(fan-in+fan-out)`. This is the reciprocal of the arithmetic mean of the two widths: neither direction receives absolute priority.

To implement that target with a centred uniform distribution, set the half-width to `√6/√(fan-in+fan-out)`. The factor is not decorative. A uniform variable on `[−a,a]` has variance `a²/3`; substituting the proposed half-width yields `(6/(fan-in+fan-out))/3 = 2/(fan-in+fan-out)` exactly. For equal widths `n`, the rule reduces to variance `1/n`, so forward and backward multipliers both become one.

For unequal widths, the compromise exposes the remaining trade-off. In a 512-to-1,024 layer, the proposed variance is `2/1536=1/768`. The forward multiplier is `512/768=2/3`; the backward multiplier is `1024/768=4/3`. Their arithmetic relationship is balanced around the two goals, but neither is one. The method prevents either side’s width from being ignored; it does not abolish rectangularity.

The later nickname “Xavier initialisation” can make the equation sound like a universal modern constant. Historically, this paper calls it normalised initialisation and derives it for a near-linear, symmetric-activation account. Rectifiers, convolutional kernels, residual branches and attention blocks change the relevant variance calculus. The durable insight is the method: identify what should be preserved in both directions, derive the per-layer scale, and test the result through distributions rather than folklore.

For layer `i`, the paper monitors `J_i = ∂z_{i+1}/∂z_i`. A singular value describes how strongly an infinitesimal perturbation along one input direction is stretched or contracted. Values far below one erase that direction; values far above one amplify it. A chain of Jacobians compounds these effects just as the variance products do, but with directional structure exposed.

When adjacent layers have the same dimension, the paper associates the average singular value with an average ratio of mapped infinitesimal volumes and activation variance. On Shapeset-3×2, this average ratio is reported around 0.5 under the baseline initialisation and around 0.8 under the proposed normalisation. The latter is closer to one, matching the goal of more faithful layer-to-layer transport.

“Around one” must be read carefully. An average of one could hide one direction stretched tenfold and many directions nearly erased. The paper does not claim that every singular value is one, nor does it establish the later, stronger notion often called dynamical isometry. Its Jacobian statistic is an investigative instrument: it connects what activations do on the way up with what gradients can do on the way down.

During training, the authors explicitly stop trusting the simple independence calculation as a full account. Activations depend on weights, derivatives change with operating regions, and the network is no longer random in the original sense. They continue to plot distributions. Under standard initialisation, back-propagated gradients begin smaller in lower layers, then that trend reverses quickly. Under normalised initialisation the initial decrease is absent, and weight-gradient scales remain better aligned during learning.

This observational discipline is the paper’s most transferable lesson. Final error says whether a run worked. Layer-wise distributions suggest why, when and where it struggled. A model can reach a respectable endpoint after wasting most of its computation in a saturated plateau; a diagnostic trace makes that hidden cost visible.

Table 1 reports final test error for five-hidden-layer networks. On Shapeset, tanh falls from 27.15% under the baseline rule to 15.60% under normalised initialisation. On CIFAR-10 it moves from 55.90% to 52.92%; on Small-ImageNet from 70.58% to 68.57%; and on MNIST from 1.76% to 1.64%. The direction is consistent for tanh, though the size of the change is not.

Softsign begins much stronger on Shapeset: 16.27%. Normalisation changes that to 16.06%. On CIFAR-10 it moves from 55.78% to 53.80%, and on Small-ImageNet from 69.14% to 68.13%. But on MNIST it moves in the wrong direction, from 1.64% to 1.72%. The exception blocks the slogan “normalisation always improves error”. The supported claim is conditional: a gentler nonlinearity is less sensitive to the initial scale, while tanh benefits markedly when layer-to-layer magnitude is better preserved.

The sigmoid row is poor across the four displayed datasets: 82.61% Shapeset, 2.21% MNIST, 57.28% CIFAR-10 and 70.66% Small-ImageNet: and the table does not present a sigmoid-normalised row. It supports the observed difficulty of sigmoid with small random initialisation, not a numerical estimate of how equation 16 would rescue it.

On 100,000 Shapeset examples, an optimised RBF SVM baseline reaches 59.47% error, while a five-layer normalised tanh network reaches 50.47%. That comparison is contextual and uses a fixed finite subset, unlike the main online curves. Denoising-autoencoder pretraining also appears as a reference curve. The authors’ conclusion is calibrated: preserving activation and gradient flow removes a good part of the discrepancy between purely supervised and unsupervised-pretrained deep networks. It does not erase every gap or explain every trace.

The paper demonstrates numerical trainability, classification behaviour and diagnostic associations. None of these measurements tests whether the network understands its classes, chooses its own goals, acts autonomously or has subjective experience. Better conditioned credit assignment is a property of an optimisation process. Mental predicates require their own operational evidence.