The tower that failed before training began

A team proposes a 96-block network. Every block has the same width and the same two-layer branch. The optimiser, data and loss are fixed. Five launch configurations differ only in activation, normalisation and the strength of the residual addition.

The first uses tanh with Xavier-style weights and no skip path. By the last block, its state has about two per cent of the input's root-mean-square magnitude. The second uses ReLU with He scaling. Its backward signal remains usable, but its forward magnitude wanders over an eleven-fold range. The third adds an identity skip and seems safer. It is not. With the residual branch added at full strength, state and gradient magnitudes grow by roughly fourteen orders of magnitude. The fourth changes one number, scaling each branch by 1/√96, and both rivers remain near their starting scale. The fifth also normalises the branch input and narrows the drift further.

These are synthetic measurements from the executable suite later in the article, not benchmark results. Their purpose is causal. The skip connection does not rescue a badly scaled branch, and a fashionable activation does not make a deep composition trainable by itself. Depth creates repeated transport. Every block may preserve, attenuate, amplify, rotate or couple the state. Backpropagation repeats the same problem in reverse.

Depth is a transport problem

A deep network is a composition of local transformations. That simple fact produces the central difficulty. Suppose a scalar signal crosses 96 stages, and each stage passes on 99 per cent of its magnitude. No stage looks broken, yet the endpoint retains only about 38 per cent. At 80 per cent per stage, the endpoint is roughly five ten-billionths of the start. Reverse the error and use a local gain of 1.05: the final magnitude exceeds the start by more than one hundred times.

Real networks do not multiply one scalar. They transform vectors, and direction matters as much as size. A layer can preserve the overall norm while crushing the particular direction that carries the useful distinction. The appropriate local object is therefore a Jacobian, the matrix of partial derivatives that describes how a small change in one layer changes the next.

Backpropagation multiplies transposed Jacobians in the reverse order. The forward representation and the backward credit signal therefore share a transport system. A design can keep activations finite while making gradients useless, or preserve an average gradient norm while producing a few destructive outlier directions. Trainability requires a corridor, not a single scalar target. Useful diagnostics include state RMS, gradient RMS, singular-value spread, correlation collapse, saturation, dead units and the ratio between each residual update and the stream it modifies.

A recent primary preprint sharpens the directional part of this account. Everett analyses effective Jacobian rank in Transformer feed-forward blocks and reports that skip connections, normalisation placement and branch-to-skip scale jointly determine how many gradient directions survive depth at initialisation. In its studied networks, initial Jacobian rank predicts which configurations train on CIFAR-10. The result is recent and not yet a general law, but it exposes a failure that an RMS-only dashboard can miss: a river may keep its volume while losing most of its channels.

Thought experiment: the relay with one adjustable dial

Place 96 relay operators in separate rooms. Each receives a vector, applies the same rule and passes the result on. The operators cannot see the whole chain. First set every local gain to 0.99. The final message is quieter but recognisable. Set it to 1.01. The final message is 2.6 times stronger. Nothing dramatic happened in any room; depth converted a one per cent local difference into a system-level change.

Now vary only the orientation. Every operator preserves vector length but rotates the signal so that the component carrying the answer shrinks by one per cent. A norm monitor reports perfect health while the relevant information disappears. This is why variance preservation is a useful commissioning check, not a complete theory of learning. It must be paired with directional or task-sensitive tests.

Three controls shape the river

Activation sets the local gate

An activation function decides which pre-activations pass, how strongly they pass and how their local derivative behaves. Sigmoid and tanh compress large magnitudes into bounded outputs. That can be useful, but their derivatives approach zero in the saturated tails. ReLU keeps a unit slope on the positive side and removes positive saturation, while setting the negative side and its derivative to zero. GELU and SiLU use smooth, input-dependent gates rather than a hard sign threshold.

The relevant question is not which curve looks most expressive. It is which distribution of pre-activations the network will present to that curve. Glorot and Bengio connected saturating non-linearities, initialisation and poor gradient propagation in deep feed-forward networks. He and colleagues derived a variance-aware initialisation for rectifiers because ReLU discards roughly half of a symmetric input at initialisation. GELU and Swish later showed that smooth, non-monotonic gates can improve particular deep models, but their published gains are architecture- and task-bound rather than a universal ordering.

An activation and its initialisation form one mechanism. Replacing tanh with ReLU while retaining a scale designed for another derivative distribution changes forward variance and backward gain at once. Replacing ReLU with GELU inside an already tuned residual stack may require a learning-rate or branch-scale retest. “Use GELU” is therefore an incomplete architecture decision.

Thought experiment: change only the gate

Feed the same vector of pre-activations into three sealed blocks. Every weight, input and downstream loss is held fixed. The first block uses tanh, the second ReLU, the third GELU. Now shift the pre-activation mean from zero to minus three without changing its variance. Tanh moves into a saturated negative tail, ReLU closes almost every gate, and GELU passes small negative values with small but non-zero derivatives.

The causal feature is not smoothness by itself. It is where the operating distribution sits on the curve. Add a normaliser before the gate and the three blocks move back towards their central regions. Move the normaliser after the block and the local derivative pattern does not receive that protection. This one-feature intervention shows why activation and normalisation placement cannot be evaluated independently.

Normalisation chooses the ruler and who shares it

Normalisation is often described as keeping numbers small. That misses two architectural decisions. First, it defines a coordinate transformation: which mean or magnitude is removed, and which scale becomes irrelevant. Second, it defines a coupling boundary: which examples, tokens, channels or features share the statistics used for that transformation.

Batch Normalisation estimates moments across examples in a mini-batch for each feature or channel. This can stabilise and accelerate convolutional training, but it makes one example's transformation depend on its neighbours, its batch size and the training or inference mode. Layer Normalisation instead computes moments across features within one example, so it has no cross-example batch dependence. RMSNorm removes only root-mean-square scale, leaving the mean untouched. These are different invariances, not interchangeable implementations of a generic “norm” layer.

The original Batch Normalisation paper motivated the method through changing internal distributions. Later experiments by Santurkar and colleagues found that its optimisation benefit could not be explained simply by reduced internal covariate shift and instead documented smoother loss and gradient behaviour. That result is a warning against turning an implementation into a settled causal story. Normalisation can help through several coupled effects, including reparameterisation, scale invariance, smoother optimisation and noise from estimated statistics.

Placement matters just as much as type. In a post-normalised residual block, the state must cross the normaliser after each addition. In a pre-normalised block, the identity stream bypasses the branch normaliser. Xiong and colleagues analysed Transformer initialisation and showed why Post-LN can produce large gradients near the output and require careful warm-up, while Pre-LN has better-behaved gradients at initialisation. This is not proof that Pre-LN always yields better final representations. It shows that moving the same operation changes the gradient topology.

Residual flow protects a route, but it also accumulates branches

A residual block adds a learned change to an existing state. The simplest form is h_l+1 = h_l + F_l(h_l). This creates an identity term in the local Jacobian. If the branch is initially small, information and gradients have a direct route even before the branch has learned anything useful.

The identity term changes multiplication from a product of arbitrary Jacobians to a product of matrices near I. That is the durable intuition behind residual learning. He and colleagues showed that identity skip mappings and pre-activation support direct forward and backward propagation across many blocks. ReZero makes the point stark by attaching a zero-initialised scalar to each residual branch, so the network begins as the identity and learns how much branch to admit.

Yet addition has a second face. If each branch contributes an independent change with variance v, adding 96 full-strength branches can make state variance grow with depth. Correlated branches can grow faster. The residual stream may dominate normalisers, overflow low-precision ranges or turn later layers into small corrections to a massive accumulated state. A residual connection protects a path; residual scaling protects the path from the branches it collects.

Fixup, DeepNorm and DeepScaleLM take different routes to this goal. Fixup rescales and zero-initialises parts of residual branches so deep residual networks can train without normalisation. DeepNorm modifies residual scaling and initialisation to bound updates in very deep Transformers. DeepScaleLM derives moment-preserving scales through Transformer components and reports training models with 1,000 layers. These results support the causal burden while warning against a single recipe: the invariant is controlled transport, not a mandatory layer order.

Worked example: one local derivative, three global systems

Strip the block to one scalar so that its failure cannot hide inside an average. Let the branch have local derivative j = 0.2 at every stage and let depth L = 96. A plain stack has total derivative j^L, about 7.9 × 10⁻⁶⁸. The branch is locally responsive, yet almost no credit reaches its first stage. This is the multiplicative depth problem in its cleanest form.

Add an identity route and the derivative becomes (1 + αj)^L. With full branch scale α = 1, the same positive derivative produces about 4.0 × 10⁷. Reverse only the branch orientation so that j = −0.2, and the result becomes about 5.0 × 10⁻¹⁰. The identity term prevents an individual block from having derivative 0.2, but it does not force the product to remain near one. The sign and alignment of the learned change still matter.

Now choose α = 1/√96. The two aligned extremes become approximately 6.96 and 0.138. That is a much narrower range, although it is not perfect preservation. The example deliberately assumes every branch Jacobian points in the same scalar direction, a worst-case correlation that a high-dimensional random initialisation may not realise. Its value is diagnostic: depth scaling buys a budget for accumulated change; it does not prove that branch directions will cancel.

A real commissioning test must therefore inspect more than mean variance. Compare perturbations aligned with important features, estimate Jacobian spectral spread on small specimens, and monitor whether the residual-to-stream ratio grows with depth. Two models can report the same state RMS while one preserves many directions and the other channels nearly everything through a narrow subspace. The first may support distributed credit; the second may train only its final layers.

flowchart TD
  A["The tower that failed before<br/>training began"] --> B["Depth is a transport problem"]
  B --> C["Three controls shape the river"]
  C --> D["Ablation, five deep rivers and<br/>one shallow control"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#5B5CF6,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Ablation: five deep rivers and one shallow control

The companion artefact builds synthetic networks from the same two-linear-layer branch, W2(activation(W1(x))). It sends 128 Gaussian examples through width-64 stacks, then sends a random unit-RMS gradient backwards. It records forward state RMS and backward state-gradient RMS at every block. The deep configurations use 96 blocks; the negative control uses eight.

This is deliberately narrower than training a classifier. No dataset structure, optimiser trajectory or generalisation claim can hide the mechanism. The suite asks whether the initial architecture can transport a generic state and a generic sensitivity. Its positive case is a pre-RMS-normalised residual stack with branch scale 1/√96. Its negative case uses the same residual branch at scale one.

The unscaled residual result is the most important. A common story says the identity path solves vanishing gradients. Locally, it does create a derivative term of one. Globally, it also adds every branch to the same stream. In this run, the branch contributions compound into an unusable state and an equally unusable gradient. The identity route exists, but it is submerged.

The tanh result contains a subtler lesson. Nearly every local derivative in the suite exceeds 0.1 because the state collapses towards the near-linear centre of tanh rather than saturating in its tails. A “live derivative fraction” monitor would look healthy while the state magnitude disappears. No single health metric is sufficient. The ReLU plain stack gives the opposite warning: its gradient ratio is acceptable, yet its forward magnitude travels over an eleven-fold range.

What the instrument reveals, and what it deliberately leaves open

The positive case does not “win” a model comparison. It demonstrates that one complete block equation preserves two generic transport quantities under one controlled initialisation. A task could still prefer the plain stack because its shallower effective computation regularises better, or reject RMSNorm because centring carries useful inductive bias. The ablation also uses fixed random weights. It cannot reveal whether optimisation later creates attention outliers, whether features become correlated, or whether a stable stack learns a shortcut.

The suite is valuable because those limitations are explicit. It removes training dynamics to expose a structural precondition, then hands the surviving designs to a matched learning experiment. The next gate should add a small public task, at least three seeds and layerwise update measurements. A configuration fails early if it cannot transport at target depth. It advances, without being certified, if it stays inside a declared corridor. The artefact is a falsification instrument for architecture, not a proxy leaderboard.

The serious counterexample: none of the three controls is individually necessary

The preferred explanation would be too strong if it claimed that deep learning requires ReLU, a normaliser and a residual connection. The shallow negative control already weakens that claim. At eight layers, the plain He-initialised ReLU stack stays inside a tight corridor. Depth is the intervention that makes the architectural details decisive.

Published results go further. Xiao and colleagues trained vanilla convolutional networks with up to 10,000 layers using an orthogonal initialisation designed for dynamical isometry. Fixup trained deep residual networks without normalisation. Normalizer-Free Networks combined scaled weight standardisation, residual design and adaptive gradient clipping to achieve strong large-scale image results without BatchNorm. A 2025 preprint replaces Transformer normalisers with a learned Dynamic Tanh operation and reports matched or improved results across several studied vision and language settings. These are not exceptions to the transport account. They are competing implementations of it.

The causal claim is therefore narrower: activation, normalisation and residual design are common ways to shape signal transport, but any parameterisation that keeps the relevant Jacobians and states usable can substitute. An observation that would weaken even this narrower thesis would be reliable deep training while forward states, task-relevant perturbations and backward sensitivities all collapse or explode throughout most of the stack. Current successful alternatives instead work by preserving or controlling some transport quantity.

Commission the block equation before the full run

Architecture reviews often list components: GELU, RMSNorm, residual connections. A trainability review should instead write the full state transition, including order and scale. The difference between Norm(x + F(x)), x + F(Norm(x)) and x + Norm(F(Norm(x))) is not cosmetic. Each expression creates a different forward variance recursion and a different backward route.

Precision makes that distinction operational. A slowly growing residual stream may look harmless in float64 diagnostics yet consume the exponent or mantissa headroom of bfloat16, float16 or quantised accumulators. Normalisation can restore scale before one operation while leaving an earlier matrix multiplication exposed to outliers. Loss scaling protects some backward underflow but cannot reconstruct a direction already rounded away. The commissioned transport corridor must therefore be defined in the arithmetic used by the intended run. Record maxima and quantiles alongside RMS, and repeat the sweep with the actual kernels when compiler fusion or reduced-precision accumulation changes the block equation in practice.

BatchNorm would couple examples and make behaviour sensitive to micro-batch composition, so the team chooses a per-token feature normaliser. RMSNorm is a plausible starting point when scale control is required without centring. The feed-forward branch uses a smooth gate such as GELU or SiLU, but that choice remains provisional until its activation and derivative distributions are measured. The residual branch is depth-scaled or zero-initialised according to a validated parameterisation. The final output normalisation and low-precision accumulator policy are explicit.

Before allocating the long run, the team executes three matched pilots: 12, 48 and 96 layers. It records per-layer state RMS, maximum magnitude, gradient RMS, update-to-weight ratio, residual-to-stream ratio and the fraction of examples near precision limits. It repeats the sweep with the normaliser moved, the residual scale doubled and halved, and the activation replaced by ReLU. A shallow plain network acts as a negative control. A small task run then checks that the configuration that transports signals also reduces loss and uses deeper layers.

The decision is evidence-shaped. If every architecture is stable at 12 layers, the test has not discriminated. If only the pre-normalised configurations survive at 96 layers, placement has earned its complexity. If a normaliser-free parameterisation matches the corridor and converges, normalisation has not earned mandatory status. If the stable configuration's deep layers receive tiny parameter updates, the system may be trainable yet effectively shallow.

The depth commissioning record

• Write the exact block. Record activation, normalisation axis and placement, residual scale or gate, initialisation, precision and depth. Component names without order are insufficient.
• Measure both rivers. Sweep forward state RMS and maximum magnitude, then backward state-gradient RMS. Add task-relevant perturbations where generic noise could miss a crushed direction.
• Inspect local gates. Record saturation, dead-unit fraction, derivative quantiles and branch-output RMS. A healthy average can conceal a dead tail or rare outlier.
• Vary one causal feature. Hold weights, data and seed constant while moving the normaliser, changing activation, or altering residual scale. Do not compare whole model families and call the result an ablation.
• Stress the intended regime. Use target depth, micro-batch size, sequence length, precision, distributed statistics and activation checkpointing. A desktop toy can hide the coupling that fails in the real run.
• Keep a serious baseline. Include a shallower plain model, an orthogonal or Fixup-style initialisation, or another transport-preserving rival. The preferred stack must earn its operational cost.
• Cross the training boundary. Run a small matched optimisation test. Require loss reduction, useful updates in early and late layers, reproducibility across seeds and no growing precision debt.
• Version the evidence. Store the block equation, seeds, curves, thresholds and failure notes with the model configuration. Changing depth or precision invalidates the old commissioning result.

A compact mechanism selection matrix

Executable artefact: the transport ablation suite

Save the source below as activation_normalisation_residual_ablation.py and run python3 activation_normalisation_residual_ablation.py. It requires NumPy and uses only synthetic data. The assertions encode the article's discriminating claims: the unscaled residual must explode, the scaled pre-normalised residual must remain bounded, and the shallow plain control must remain benign.

#!/usr/bin/env python3
"""Signal-transport ablation suite for
'Activation, Normalisation and Residual Flow'.

Python 3.10+ and NumPy 1.24+.

The suite does not claim to predict final task accuracy. It measures two
initialisation-time transport diagnostics through synthetic deep networks:

1. forward state RMS across depth;
2. backward state-gradient RMS across depth.

Positive case: a depth-scaled pre-normalised residual stack.
Negative case: the same residual branch with an unscaled skip addition.
Negative control: a shallow He-initialised plain ReLU stack.
"""
from __future__ import annotations

from dataclasses import dataclass
import json
import math
from typing import Callable

import numpy as np

Array = np.ndarray


def rms(x: Array) -> float:
    """Root-mean-square magnitude over every element."""
    return float(np.sqrt(np.mean(np.square(x))))


def rmsnorm(x: Array, eps: float = 1e-8) -> tuple[Array, Array]:
    """Per-example RMSNorm without learnable gain."""
    scale = np.sqrt(np.mean(x * x, axis=1, keepdims=True) + eps)
    return x / scale, scale


def rmsnorm_backward(grad_y: Array, x: Array, scale: Array) -> Array:
    """Exact vector-Jacobian product for y = x / RMS(x)."""
    correction = x * np.mean(grad_y * x, axis=1, keepdims=True) / (scale**3)
    return grad_y / scale - correction


def activate(z: Array, name: str) -> Array:
    if name == "relu":
        return np.maximum(z, 0.0)
    if name == "tanh":
        return np.tanh(z)
    raise ValueError(f"unsupported activation: {name}")


def activation_derivative(z: Array, name: str) -> Array:
    if name == "relu":
        return (z > 0.0).astype(z.dtype)
    if name == "tanh":
        t = np.tanh(z)
        return 1.0 - t * t
    raise ValueError(f"unsupported activation: {name}")


@dataclass(frozen=True)
class Configuration:
    name: str
    depth: int
    activation: str
    residual: bool
    residual_scale: float = 0.0
    pre_rmsnorm: bool = False


@dataclass
class Result:
    configuration: Configuration
    forward_rms: list[float]
    gradient_rms: list[float]
    live_derivative_fraction: list[float]

    @property
    def forward_drift(self) -> float:
        floor = max(min(self.forward_rms), 1e-30)
        return max(self.forward_rms) / floor

    @property
    def input_to_output_gradient_ratio(self) -> float:
        return self.gradient_rms[0] / self.gradient_rms[-1]

    def summary(self) -> dict[str, float | int | str | bool]:
        return {
            "configuration": self.configuration.name,
            "depth": self.configuration.depth,
            "forward_end_rms": self.forward_rms[-1],
            "forward_drift": self.forward_drift,
            "input_to_output_gradient_ratio": self.input_to_output_gradient_ratio,
            "mean_live_derivative_fraction": float(
                np.mean(self.live_derivative_fraction)
            ),
            "pre_rmsnorm": self.configuration.pre_rmsnorm,
            "residual_scale": self.configuration.residual_scale,
        }


def initialise_pair(
    rng: np.random.Generator, width: int, activation: str
) -> tuple[Array, Array]:
    """Initialise a two-linear-layer branch W2(phi(W1 x))."""
    if activation == "relu":
        first_std = math.sqrt(2.0 / width)  # He scaling before ReLU
    elif activation == "tanh":
        first_std = math.sqrt(1.0 / width)  # Xavier-style scaling
    else:
        raise ValueError(activation)
    second_std = math.sqrt(1.0 / width)
    w1 = rng.standard_normal((width, width)) * first_std
    w2 = rng.standard_normal((width, width)) * second_std
    return w1, w2


def run_configuration(
    config: Configuration,
    *,
    width: int = 64,
    batch: int = 128,
    seed: int = 7,
) -> Result:
    """Run one deterministic forward/backward transport experiment."""
    rng = np.random.default_rng(seed)
    x = rng.standard_normal((batch, width))
    forward_rms = [rms(x)]
    caches: list[tuple[Array, Array, Array | None, Array, Array, Array, Array]] = []
    live: list[float] = []

    for _ in range(config.depth):
        x_in = x
        if config.pre_rmsnorm:
            branch_in, scale = rmsnorm(x_in)
        else:
            branch_in, scale = x_in, None

        w1, w2 = initialise_pair(rng, width, config.activation)
        z = branch_in @ w1
        a = activate(z, config.activation)
        branch = a @ w2

        if config.residual:
            x = x_in + config.residual_scale * branch
        else:
            x = branch

        caches.append((x_in, branch_in, scale, z, a, w1, w2))
        forward_rms.append(rms(x))
        if config.activation == "relu":
            live.append(float(np.mean(z > 0.0)))
        else:
            live.append(
                float(np.mean(np.abs(activation_derivative(z, config.activation)) > 0.1))
            )

    grad = rng.standard_normal((batch, width))
    grad /= rms(grad)  # terminal gradient RMS is exactly one
    reversed_gradient_rms = [rms(grad)]

    for x_in, _branch_in, scale, z, _a, w1, w2 in reversed(caches):
        if config.residual:
            branch_grad = config.residual_scale * grad
            skip_grad = grad
        else:
            branch_grad = grad
            skip_grad = np.zeros_like(grad)

        grad_a = branch_grad @ w2.T
        grad_z = grad_a * activation_derivative(z, config.activation)
        grad_branch_in = grad_z @ w1.T

        if config.pre_rmsnorm:
            assert scale is not None
            grad_branch = rmsnorm_backward(grad_branch_in, x_in, scale)
        else:
            grad_branch = grad_branch_in

        grad = skip_grad + grad_branch
        reversed_gradient_rms.append(rms(grad))

    gradient_rms = list(reversed(reversed_gradient_rms))
    return Result(config, forward_rms, gradient_rms, live)


def build_suite(depth: int = 96) -> list[Configuration]:
    scaled = 1.0 / math.sqrt(depth)
    return [
        Configuration("tanh_xavier_plain", depth, "tanh", False),
        Configuration("relu_he_plain", depth, "relu", False),
        Configuration("relu_residual_unscaled", depth, "relu", True, 1.0),
        Configuration("relu_residual_scaled", depth, "relu", True, scaled),
        Configuration(
            "relu_prenorm_scaled", depth, "relu", True, scaled, pre_rmsnorm=True
        ),
        Configuration("relu_he_plain_shallow", 8, "relu", False),
    ]


def assert_expected_relationships(results: list[Result]) -> None:
    by_name = {result.configuration.name: result for result in results}

    # Negative case: an identity skip does not rescue an over-strong branch.
    exploding = by_name["relu_residual_unscaled"]
    assert exploding.forward_drift > 1e8
    assert exploding.input_to_output_gradient_ratio > 1e8

    # Positive case: depth-aware scaling plus pre-normalisation keeps both rivers bounded.
    stable = by_name["relu_prenorm_scaled"]
    assert stable.forward_drift < 3.0
    assert 0.5 < stable.input_to_output_gradient_ratio < 2.0

    # Negative control: depth is the intervention. A shallow plain stack is already benign.
    shallow = by_name["relu_he_plain_shallow"]
    assert shallow.forward_drift < 3.0
    assert 0.5 < shallow.input_to_output_gradient_ratio < 2.0

    # Activation-only pathology: the tanh/Xavier plain stack loses most forward magnitude.
    tanh_plain = by_name["tanh_xavier_plain"]
    assert tanh_plain.forward_rms[-1] < 0.1


def sampled_curves(result: Result) -> dict[str, list[float] | list[int] | str]:
    depth = result.configuration.depth
    points = sorted({0, depth // 12, depth // 6, depth // 3, depth // 2,
                     (2 * depth) // 3, (5 * depth) // 6, depth})
    return {
        "configuration": result.configuration.name,
        "layers": points,
        "log10_forward_rms": [
            math.log10(max(result.forward_rms[index], 1e-30)) for index in points
        ],
        "log10_gradient_rms": [
            math.log10(max(result.gradient_rms[index], 1e-30)) for index in points
        ],
    }


def main() -> None:
    results = [run_configuration(config) for config in build_suite()]
    assert_expected_relationships(results)

    payload = {
        "assumptions": {
            "data": "128 synthetic Gaussian examples",
            "width": 64,
            "deep_blocks": 96,
            "branch": "W2(activation(W1(x)))",
            "terminal_gradient_rms": 1.0,
            "seed": 7,
            "interpretation": (
                "initialisation-time transport diagnostic, not a claim about final accuracy"
            ),
        },
        "summary": [result.summary() for result in results],
        "sampled_curves": [sampled_curves(result) for result in results],
        "checks": "PASS",
    }
    print(json.dumps(payload, indent=2))


if __name__ == "__main__":
    main()

tanh_xavier_plain          end=0.0198  drift=61.4x       grad-ratio=0.0250
relu_he_plain              end=0.439   drift=11.5x       grad-ratio=0.571
relu_residual_unscaled     end=1.52e14 drift=1.54e14x    grad-ratio=2.01e14
relu_residual_scaled       end=1.56    drift=1.58x       grad-ratio=1.63
relu_prenorm_scaled        end=1.37    drift=1.39x       grad-ratio=1.45
relu_he_plain_shallow      end=0.864   drift=1.33x       grad-ratio=0.935
checks: PASS

The suite is intentionally small enough to inspect. It omits learned normaliser gains, biases, attention, convolution, data structure and optimiser updates. Extending it is useful only if each addition answers a new causal question. A larger simulation that entangles ten mechanisms would be less informative.

stateDiagram-v2
  [*] --> Claim
  state "Thought experiment: the relay<br/>with one adjustable dial" as Claim
  state "Activation sets the local gate" as Test
  state "Thought experiment, change<br/>only the gate" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Compact glossary

Forward state

The representation passed from shallow to deeper layers during inference or training.

Backward sensitivity

The derivative of the loss with respect to an intermediate state, carrying credit towards earlier parameters.

Transport corridor

A bounded regime in which relevant states and sensitivities remain numerically and directionally usable across depth.

Dynamical isometry

A condition in which input-output Jacobian singular values are concentrated near one, preserving perturbation magnitudes.

Residual branch

The learned change added to an identity stream, often written αF(N(h)).

Normalisation axis

The tensor dimensions over which statistics are computed, which determines both invariance and coupling.