By 2015, depth was central to high-performing visual recognition. Additional convolutional stages could build features over larger effective receptive fields and compose increasingly abstract transformations. But simply stacking more layers was not a reliable recipe. Gradients could vanish or explode, and even after those instabilities were mitigated, a second phenomenon remained: the deeper system’s training error could rise.

He and colleagues call this the degradation problem. It is essential to notice the word training. If a deeper network reached lower training error but worse validation error, ordinary overfitting would be a plausible diagnosis. Their motivating curves show a deeper plain network performing worse on the training set itself. The optimisation procedure has failed to find even a solution that emulates the shallower model.

The existence argument is simple but powerful. Suppose an 18-layer network implements some useful function. A 34-layer architecture with compatible shapes could reproduce it if its extra layers implemented identity. The deeper hypothesis class is not missing the shallower function. Therefore its larger training error is not evidence that the target became unrepresentable. It indicates that the chosen parameterisation and optimiser do not reach the obvious embedded solution.

This argument does not say that an arbitrary added layer can literally be set to identity under every architecture. Nonlinearities, dimension changes and normalisation affect the construction. It says that identity is the natural baseline against which added transformations should be learnt. If ordinary stacked layers find that baseline difficult, the architecture can put it into the computation explicitly.

The paper’s first figure makes the issue visible on CIFAR-10: the 56-layer plain network has higher training and test error than the 20-layer plain network. The authors also show a related ImageNet comparison between 18- and 34-layer plain networks. Those experiments motivate residual learning; they are not yet evidence that any depth, dataset or optimiser will behave the same way.

“Degradation” is also easy to confuse with numerical decay of a signal. It does not mean activations monotonically shrink with depth. It names a performance pattern: a constructed deeper model is harder to optimise into a good solution. Residual connections may help gradients, but the primary diagnosis is broader than one derivative pathology.

Write the residual branch as F(x,{Wᵢ}), where the Wᵢ are its learnt weights. For the two-layer example in the paper, F may take the form W₂σ(W₁x), with rectified linear units and batch normalisation arranged around the convolutions. The shortcut carries x around that branch. Addition reunites the routes before another nonlinearity in the original design.

If the best mapping is close to identity, the residual branch need only approach zero. That may be easier than making several nonlinear layers jointly reproduce x. The paper presents this as a hypothesis supported by optimisation results, not as a universal theorem about all learning problems. Its language is deliberately comparative: the residual function should be easier to optimise than the original unreferenced function.

The reference point matters even when H is not identity. Suppose the desired output is x plus a small feature refinement. A plain block must encode the entire input-preserving transformation through its weights. A residual block gets input preservation for free and dedicates its learnt path to the refinement. When the correction should be large, the residual branch can still express it.

Addition is not concatenation. Concatenating x and F(x) would increase the feature dimension and require a later operator to decide how to mix the components. Elementwise addition keeps the output shape fixed and imposes a particular compositional form. This design choice creates the direct algebraic route that makes the block distinctive.

The shortcut is also not a frozen copy of a whole earlier representation stored elsewhere. It is an edge in the computational graph. During the forward pass it transmits x; during backpropagation the derivative of the identity route contributes directly. Both routes are evaluated at the same block and their signals add.

Residual does not mean the statistical residual of a fitted regression model. It means the difference between a desired underlying mapping and the reference identity. The terminology helps only if the reference is kept visible. F is not required to be small in every trained block, and the paper does not impose a penalty that forces it to vanish.

For equal dimensions, equation 1 uses y = F(x,{Wᵢ}) + x. The shortcut performs identity and introduces neither an additional parameter nor meaningful computation. This property lets the authors compare plain and residual networks with closely matched parameter counts and complexity: the performance difference is not purchased by a large new branch.

When dimensions differ, equation 2 becomes y = F(x,{Wᵢ}) + Wₛx. The matrix or convolution Wₛ aligns the shortcut with the residual output. In the ImageNet implementation, a projection shortcut uses a 1×1 convolution, with stride when spatial downsampling is required. The reference route is now learnt rather than literal identity.

The paper compares three shortcut options. Option A uses identity shortcuts throughout and zero-padding for increased dimensions, so it adds no parameters. Option B uses projections only when dimensions increase; other shortcuts remain identity. Option C uses projections for all shortcuts. Option C introduces more parameters and is not the default basis for claiming that shortcut benefits come free.

These options are evidence about mechanism as well as engineering. On ImageNet, all three outperform the plain counterpart, while the all-projection option is modestly better than the strictly parameter-free option. The authors infer that degradation is unlikely to be solved merely by adding parameters. Projection can help, but the identity architecture already removes much of the optimisation difficulty.

Zero-padding and projection do not preserve exactly the same information geometry. Padding copies existing channels and fills new ones with zeros; projection can rotate, scale or mix channels. Describing both simply as “skip connections” hides a consequential intervention. Any reproduction should report which option is used at each stage boundary.

A dimension change is also where casual statements about derivative one need correction. An identity shortcut contributes an identity Jacobian. A projected shortcut contributes Wₛ. That route can still be direct, but its gradient is transformed. The residual framework provides an additive path; it does not guarantee that every such path is unit-scaled.

The ImageNet networks begin with a 7×7 convolution and max pooling, then move through four spatial stages. The plain 34-layer network stacks 3×3 convolutions. Its residual counterpart uses the same principal convolutions and feature-map sizes, adding shortcuts around paired layers. Downsampling occurs at stage transitions, where stride and dimension alignment matter.

For 18- and 34-layer networks, the authors use a basic two-layer residual block. For 50 layers and beyond, they introduce a bottleneck design: a 1×1 convolution reduces channel dimension, a 3×3 convolution operates in the narrower space, and another 1×1 convolution restores the output dimension. The shortcut bypasses this three-layer residual branch.

“Bottleneck” does not mean an information bottleneck objective. It names a computational shape. Expensive 3×3 operations are performed on fewer channels, while the surrounding 1×1 convolutions change dimensionality. This makes 50-, 101- and 152-layer models feasible at lower floating-point complexity than a naive wide stack.

The paper reports that the 152-layer ResNet has about 11.3 billion floating-point operations, lower than VGG-16/19 at roughly 15.3/19.6 billion, despite being far deeper. Depth counted as weighted layers therefore cannot be read as a direct proxy for computation. Kernel size, channel width and stage resolution determine cost.

Batch normalisation follows each convolution and precedes activation in this original architecture. The network ends with global average pooling and a 1,000-way fully connected softmax. Training uses SGD with momentum, weight decay, scale augmentation and standard image crops. These choices are part of the experimental system; a shortcut diagram alone is not a complete reproduction.

The paper counts depth according to weighted layers, not every operation in the graph. A 152-layer label does not imply 152 residual blocks. Naming conventions later varied across software libraries, so trustworthy reporting should state the block counts or architecture table rather than rely on a model family name.

In a scalar block y = F(x) + x, differentiation gives dy/dx = F′(x) + 1. If F′ is small, the local derivative stays near one. By contrast, a plain block y = F(x) has derivative F′ alone. Multiplying many small derivatives through a deep plain chain can attenuate a signal rapidly.

For vectors, the identity matrix is added to the residual Jacobian. Across multiple blocks, derivatives still compose, and residual terms can amplify, rotate or cancel directions. Nonlinearities and batch-normalisation statistics complicate the exact graph. The shortcut creates a useful route; it is not a proof that gradients can never vanish or explode.

The forward interpretation and gradient interpretation reinforce each other. If several residual functions are near zero, representations can propagate with modest alteration. Their derivatives can likewise remain near identity. Optimisation may then adjust local corrections without requiring every layer to coordinate an entire end-to-end mapping from scratch.

Yet “information highway” language should be kept precise. An identity edge carries the tensor supplied to it. After addition and activation, the next block receives a transformed value. Stage transitions may use projection. The full network also begins and ends with non-residual operations. Directness is local and compositional, not an untouched wire from pixels to logits.

The primary paper does not present the later algebraic analysis of pre-activation identity mappings. Its original blocks apply activation after addition. Subsequent work by the same authors rearranged activation and normalisation to create cleaner identity propagation. A historical reconstruction should not silently import that later block into the 2015 experiment.

Optimisation benefit is established empirically through lower training error and convergence behaviour. The derivative equation supplies a mechanistic intuition consistent with that evidence. It should not be inflated into a guarantee over arbitrary residual weights, data, losses and optimisers.

In the plain pair, the 34-layer network has higher validation error than the 18-layer network. The training curves show that this is associated with higher training error, fitting the degradation diagnosis. The result is not merely a late validation fluctuation; the deeper plain system is harder to optimise throughout much of training.

In the residual pair, the relationship reverses. The 34-layer ResNet achieves lower training error and lower validation error than the 18-layer ResNet. The deeper model can now use its added transformations rather than suffer from them. This within-family depth comparison is central to the paper’s claim that residual learning eases optimisation.

The 34-layer residual model also substantially improves over the 34-layer plain model. Tables 2 and 3 report ten-crop top-1 validation error of 25.03% for ResNet-34 versus 28.54% for plain-34, and top-5 error of 7.76% versus 10.02%. These are model-specific validation figures under the described protocol, not the later test-set ensemble number.

The 18-layer models provide a useful nuance. Their final accuracy is fairly similar, although the residual model converges faster early in training. This pattern supports an optimisation advantage that becomes more important with depth rather than a claim that every residual network must beat every plain peer by a fixed amount.

Because the paired architectures have comparable convolutional construction, the experiment is more causally informative than comparing ResNet-152 with VGG-19. Cross-family comparisons differ in depth, block design, width, normalisation, training details and computation. They matter for practical performance but identify less about the shortcut alone.

The training protocol still matters. Batch normalisation is used after each convolution, learning rates follow a schedule, and image augmentation shapes the data stream. Residual learning is evaluated inside that system. It is reasonable to attribute the paired contrast to the architectural reformulation, but not to pretend that the block operates independently of its optimisation context.

Table 3 reports single-model ImageNet validation results using ten-crop testing. ResNet-50, ResNet-101 and ResNet-152 progressively reduce error. ResNet-152 records 21.43% top-1 and 5.71% top-5 validation error under that protocol. These numbers demonstrate the scale of one deep residual model, but they are not the competition test-set result.

Table 4 broadens comparison with prior systems under the paper’s best-results protocol: fully convolutional evaluation whose scores are averaged at multiple image scales. ResNet-152 records 19.38% top-1 and 4.49% top-5 validation error. Table 5 then moves to ensembles: six residual models achieve 3.57% top-5 error on the ImageNet test set.

The authors state that this ensemble won the ILSVRC 2015 classification task. “ResNet achieved 3.57%” is defensible only when the ensemble and test protocol remain attached. Attributing it to a single 152-layer network quietly changes the experimental unit and overstates what one forward system delivered.

The paper compares the 152-layer network with VGG nets in both depth and complexity. It is eight times deeper than VGG-19 by the stated layer count yet has lower computational complexity. That is a carefully delimited architectural comparison, not a general law that residual networks always cost less than non-residual ones.

Performance at scale also reflects ensemble diversity, multi-scale evaluation and crop aggregation. Those techniques do not undermine the representation; they specify the route from model scores to the headline. A scientific companion should make them visible because deployment latency and reproducibility depend on them.

The classification result sits alongside first-place competition outcomes for detection, localisation and segmentation. Some of those tasks use the residual model as a feature extractor inside additional systems. Success across tasks strengthens the claim that the learned representation transfers, while still requiring task-specific heads, data and evaluation protocols.

The CIFAR models use 3×3 convolutions, three stages of feature-map sizes, global average pooling and a final classifier. Their depth follows 6n+2. With n = 3, 5, 7, 9 and 18, the networks have 20, 32, 44, 56 and 110 layers. A much larger n produces the 1,202-layer experiment.

Across the main range, the residual networks show favourable depth behaviour. The 110-layer model obtains lower training error than shallower peers and achieves 6.43% test error in the reported run. The authors repeat it five times and report 6.61% ± 0.16, indicating that a single run should not be treated as a noise-free constant.

They also report a 1,202-layer network with 19.4 million parameters. It fits training data very well, but its test result is worse than the 110-layer network. The authors attribute this gap to overfitting and note that stronger regularisation could help. The experiment therefore separates the degradation problem from ordinary overfitting: residual learning addresses the former, but it does not abolish the latter.

A response-magnitude analysis supports the residual interpretation. The paper examines standard deviations of layer responses and reports generally smaller responses for residual functions than for plain counterparts, suggesting that the residual branches often learn perturbations around identity. The distributions overlap and the measurement is descriptive; it is not a proof that small residuals cause the optimisation gain.

The CIFAR construction uses option A shortcuts and deliberately simple regularisation. This makes the depth experiment comparatively austere, but comparisons with modern training recipes would change many factors. Its scientific value lies in testing the 2015 hypothesis across a wide depth range under one declared setup.

The 1,202-layer result is especially useful for guarding against mythology. “Residual networks enable arbitrary depth” is too strong. The architecture allowed this enormous model to reach low training error, demonstrating that degradation was controlled, while the test set still penalised excess capacity. Optimisation and statistical learning remain separate problems.

Object detection adds localisation, region proposals and task-specific training to image classification features. A stronger classification backbone can improve those components, but the detector’s outcome is not produced by the backbone alone. The appendix therefore describes baselines, region-proposal choices and competition enhancements rather than presenting a naked ResNet score.

On PASCAL VOC, replacing VGG-16 with ResNet-101 in Faster R-CNN improves mean average precision under the stated train/test splits. On COCO, the authors describe a 28% relative improvement in the detection metric from the deeper representation. “Relative” matters: it describes the proportional gain over the baseline measure, not a 28 percentage-point jump.

The paper also develops the idea of using convolutional features across the whole image rather than treating later residual stages only as per-region classifiers. Detection experiments vary how features and proposals are shared. These details make the appendix more than a single transfer table; it shows how the representation is integrated into a structured vision pipeline.

First-place results across ImageNet detection and localisation and COCO detection and segmentation demonstrate breadth. They do not establish that residual learning is sufficient for those tasks. Losses, annotations, proposal mechanisms, multi-scale testing and competition ensembles remain causal contributors.

Transfer evidence nonetheless strengthens the paper’s central architectural claim. If residual learning only exploited an idiosyncrasy of one classification head, replacing the feature extractor in detectors might not help. Improvements across tasks are consistent with the deeper network learning generally useful visual features.

It is still possible for benchmark success and mechanism to diverge. A deeper backbone brings different capacity and receptive fields in addition to the residual parameterisation. The matched ImageNet study is the main evidence about optimisation; the detection study is evidence about downstream utility. Combining them yields a stronger, properly layered case.

The strongest directly supported mechanism is architectural: an output is the sum of a learnt residual function and a shortcut reference. The strongest empirical optimisation evidence is comparative: deeper residual networks fit better than matched deeper plain networks in the reported settings. The strongest performance evidence is protocol-bound: specific single models and ensembles reach specific errors on ImageNet and transfer tasks.

A broader claim that skip connections make all very deep networks easy to train requires evidence beyond this paper. Architectures can use different normalisation, activation placement, attention, recurrence or optimisation. Residual ideas generalise widely, but each generalisation changes the object being studied.

Agency claims are even more distant. A residual block offers no goal, policy, action selection or environment model by itself. It can be used inside systems that act, but the mathematical operation y = F(x) + x does not distinguish an agent from a classifier. Agency must be evaluated at the level where objectives and action consequences exist.

Consciousness claims are unsupported. A more trainable hierarchy may enable complex representations, and direct paths may help preserve information, but neither property measures experience. Calling a shortcut “memory”, “selfhood” or “awareness” substitutes metaphor for an operational test. The paper reports images, labels, losses and detection metrics.

The paper’s enduring conceptual contribution is nonetheless profound. It changes the default question from “what mapping should these layers learn?” to “what correction should they make to what is already available?” That idea has influenced architectures far beyond convolutional vision. Its portability is an engineering and mathematical legacy, not evidence of a mind emerging from addition.

Good historical reading keeps later success from erasing original uncertainty. The authors write that they hypothesise residual mappings are easier to optimise and then supply experiments. They do not claim a general proof. Preserving that structure: problem, reformulation, matched test, scaling, transfer: makes the result stronger because every claim retains its warrant.

Paper 0053 placed batch statistics inside a differentiable network, stabilising and re-scaling the co-ordinates in which layers learn. Paper 0054 changes the graph itself. A residual branch learns a correction while a shortcut carries its reference. In the original ImageNet ResNets, batch normalisation remains inside the residual branch, so these papers are complementary rather than rival explanations of trainability.

The earlier backpropagation paper in Canon 0048 supplies the chain rule through both branches. The ImageNet systems in Papers 0050 and 0051 supply the data, convolutional hierarchy and benchmark context. Residual learning answers the next engineering question: how can that hierarchy become far deeper without asking every new stack to reconstruct what it should often preserve?

Paper 0055 returns to regularisation with Nitish Srivastava and colleagues’ dropout paper. Instead of preserving an identity path, dropout randomly removes units during training and averages the resulting implicit ensemble at test time. The sequence is deliberate: after batch normalisation and residual learning made deep systems easier to optimise, the canon isolates a different problem: how to reduce co-adaptation and overfitting.