After every split can scale, every parameter update must travel

XGBoost made greedy tree construction practical by matching its statistics to sparse columns, caches and disks. Deep networks pose a different systems problem inside the objective itself. Their parameters interact through many layers or many time steps. A small move near the input can be magnified, damped or twisted before it changes the loss. The optimiser sees only the gradient at its current point, yet it must choose a direction and scale that remain useful as those interactions unfold.

By 2013, Hessian-free optimisation had shown that some deep autoencoders and recurrent neural networks could be trained from random starts when local curvature was treated seriously. That success encouraged a story in which difficult deep objectives demanded sophisticated second-order machinery. Sutskever and colleagues do not deny the value of Hessian-free methods. They ask how much of the observed advantage instead came from two more elementary choices: where the network was placed at time zero, and how its first-order updates remembered the recent past.

The paper’s answer is experimental and mechanistic. A well-designed random initialization keeps units and recurrent dynamics in a trainable regime. Momentum accumulates velocity where successive gradients agree and damps directions where gradients reverse. A slowly increasing momentum coefficient makes this memory stronger during the long transient phase, while a late reduction allows finer convergence. With those ingredients, stochastic first-order training reaches errors previously associated with Hessian-free optimisation on the authors’ selected tasks.

This is not a claim that optimization difficulty disappears. It is a claim about attribution. If a poorly scaled network blocks useful signals at the start, changing the update rule alone cannot rescue the missing information. If a well-scaled network repeatedly discards directional agreement, its gradients may be individually correct yet collectively inefficient. Initialization defines the landscape the optimiser first encounters; momentum defines how evidence from successive encounters is combined.

A velocity turns repeated local advice into a longer-term direction

Let θ denote the parameters and f(θ) the objective. Plain gradient descent moves by −ε∇f(θ), where ε is the learning rate. Classical momentum adds a velocity v. At each step it retains a fraction μ of the previous velocity, subtracts the current gradient contribution and then moves the parameters by the new velocity.

The memory is directional. Suppose a shallow valley is long from west to east and narrow from north to south. Along the shallow axis, gradients tend to point east over many consecutive steps. Their contributions accumulate in v, increasing speed. Along the steep axis, one step overshoots north and the next gradient points south. Alternating contributions cancel inside the velocity, reducing zigzagging.

Momentum is therefore not merely “use a bigger step”. A larger learning rate multiplies the current gradient in every direction. Momentum distinguishes directions by temporal persistence. It may produce a large effective step where evidence agrees and a smaller one where evidence reverses. The distinction matters most in ill-conditioned objectives, where eigen-directions have very different curvature.

The coefficient μ controls memory length. Expanding the recurrence shows that v is an exponentially weighted sum of past negative gradients. With μ=.9, a contribution survives k steps with weight .9^k. With μ=.99, it survives much longer. This helps sustained motion but makes an unsuitable velocity harder to stop. Momentum is acceleration and inertia at once.

Correct the destination implied by velocity, not only the place already left

Nesterov accelerated gradient uses the same state variables but evaluates the gradient at θ+μv, the point reached by the retained part of the old velocity before the new correction. The parameters still receive the resulting velocity.

The difference looks small because both methods add μv and a gradient correction. Geometrically it changes the timing. Classical momentum asks what the slope was at the current point and then combines that answer with inherited motion. NAG first imagines taking the inherited motion and asks what the slope will be there. If the velocity is about to climb the opposite wall of a ravine, the look-ahead gradient can turn it before the full step is committed.

The paper illustrates this with a two-dimensional quadratic. Classical momentum builds large oscillations along the high-curvature vertical direction. NAG nearly removes them while preserving fast progress along the shallow direction. One NAG iteration can be only slightly better, but the corrected velocity becomes the next iteration’s memory. Small differences compound.

NAG should not be mystified as knowledge of the future. It evaluates an ordinary gradient at a deterministically extrapolated point. The benefit depends on that extrapolation being informative. If μv is far outside the locally useful region, or if stochastic noise dominates, look-ahead can be inaccurate. The mechanism is anticipatory correction, not clairvoyance.

Curvature eigen-directions reveal what the algorithms remember

To analyse local behaviour, the paper studies a positive-definite quadratic q(x)=xᵀAx/2+bᵀx. Because A is symmetric, it has orthogonal eigenvectors. Rotating coordinates into that basis separates the objective into independent one-dimensional quadratics. Each coordinate has curvature λ, one eigenvalue of A.

This rotation is not a change to the optimisation problem. Classical momentum and NAG are invariant to orthonormal reparameterisation: run them in the eigenbasis and rotate the iterates back, and the path is the same. The benefit is explanatory. Instead of one tangled n-dimensional recurrence, one can inspect n independent two-state systems, each tracking position and velocity under its own curvature.

For NAG, expanding the look-ahead gradient along a quadratic eigen-direction gives v′=μ(1−λε)v−ελy, ignoring the linear shift after centring at the optimum. Classical momentum uses v′=μv−ελy. The effective momentum of NAG therefore falls as curvature λ rises. Along flat directions it is close to μ, preserving acceleration. Along steep directions the factor is smaller, providing automatic braking.

Theorem 2.1 formalises this coordinate-wise relation. Its scope matters. It is exact for the specified quadratic and clarifies local behaviour when a smooth objective is approximated by a Hessian. A deep network’s Hessian changes along the path, may have negative directions and is observed through stochastic gradients. The theorem explains a mechanism; it does not prove global accelerated convergence for the networks in Sections 3 and 4.

Train fast while direction persists, then lower inertia for the final estimate

Classical convergence analysis often studies the final neighbourhood of an optimum. There, precise local rates and stochastic variance dominate. The paper argues that deep learning spends a large and practically important fraction of its budget in an earlier transient phase. The parameters are far from a final basin, loss-reducing directions can persist over many updates and the main problem is reaching a good region rather than estimating its exact bottom.

Momentum is especially useful in this phase because persistence is observable across successive gradients. The autoencoder experiments do not jump immediately to μ=.999. Equation 5 raises μ slowly from an initially modest value towards a selected maximum. The schedule lengthens memory as the trajectory establishes a direction. For the last 1,000 of 750,000 updates, the authors reduce μ to .9 without reducing the learning rate. Table 2 reports lower training error after this low-momentum fine-tuning.

This creates two regimes. The first is optimization as rapid transport: exploit coherent direction, cross long shallow regions and tolerate some oscillation. The second is optimization as estimation: reduce inertia so the stochastic trajectory can settle more precisely. A single constant μ must compromise between them. A schedule assigns different behaviour to different phases.

The schedule is evidence-informed but not automatic. For each μ maximum the paper chooses the learning rate that achieves the lowest final training error from a six-value set. The final reduction is tested on three autoencoders. Those choices show that μ and ε interact; they do not establish equation 5 as a universal schedule. Modern warm-up, decay, restart and adaptive methods inherit the broader lesson that optimiser state should be governed across time.

A random start must keep units responsive and representations diverse

The autoencoder experiments use Martens’s sparse initialization. Each non-output unit receives connections from 15 randomly selected units in the preceding layer; those weights are drawn from a unit Gaussian and biases begin at zero. The intention is to make total input depend less directly on layer width and to avoid early saturation. Sparse incoming sets also make units receive more distinct mixtures than dense draws that average many previous outputs.

For tanh units the authors simulate sigmoid-like behaviour by setting biases to .5 and rescaling weights by .25, but the reported deep autoencoders use standard sigmoid nonlinearities. The details matter because a scale that is safe for one activation or fan-in can be poor for another. Initialization is a distribution over functions and signal paths, not merely a tradition of “small random numbers”.

Table 3 changes only the scale multiplier for the Curves autoencoder. Multipliers .25 and .5 both end at squared error 16; the default multiplier 1 reaches .074; multiplier 2 reaches .083; multiplier 4 deteriorates to .35. The result is not monotonic. Too little scale can make signals and gradients weak; too much can saturate sigmoids or create unstable curvature. A broad trainable region exists around the designed scale, but the extremes fail.

The paper compares multiple random initializations in the appendix and rejects a simplistic claim that any random start works. It also avoids claiming that sparse initialization is the final answer. Its point is diagnostic: conclusions about the power of an optimizer are conditional on whether the initialized model exposes a usable objective. Optimizer comparisons that give one method a viable start and another an arbitrary one confound geometry with update rule.

Momentum closes a training-error gap under extensive tuning

The three deep autoencoder problems come from Hinton and Salakhutdinov: Curves, MNIST and Faces. Networks contain seven to eleven layers and reconstruct their inputs through a low-dimensional bottleneck. Each trial uses 750,000 parameter updates on minibatches of 200. The paper reports training squared error because it is investigating optimization; it explicitly sets aside regularization and generalisation.

With no momentum, Table 1 reports errors .48 on Curves, 2.1 on MNIST and 36.4 on Faces. The best NAG entries are .074, .73 and 7.7. Larger μ maxima generally improve results, and NAG usually beats classical momentum at .995 and .999. The improvements are substantial enough to challenge the view that only a second-order method can optimize these models.

Comparison with Hessian-free results requires care. The table lists one HF column without L2 regularization and a separate published HF column. The methods do not have identical update counts, network nonlinearities or regularization. Chapelle and Erhan’s SGDC baseline uses 1.7 million steps and tanh networks. The authors call attention to such differences rather than presenting a single clean race.

The most defensible conclusion is conditional: given the paper’s sparse initialization, schedule and per-configuration learning-rate search, momentum-based stochastic gradient can attain strong training errors on these three benchmarks. The experiment does not show that NAG always dominates CM, that either always matches HF in wall-clock cost, or that lower training error generalizes better.

Memory requires dynamics near a useful edge, not arbitrary recurrence

The recurrent experiments use standard tanh RNNs with 100 hidden units on synthetic addition, multiplication and memorization tasks. The challenge is long-term dependence: an early input must influence an output many time steps later. Repeated multiplication by the recurrent Jacobian can shrink information and gradients towards zero or expand them uncontrollably.

Drawing on echo-state networks, the authors set sparse hidden-to-hidden connectivity with fan-in 15 and spectral radius 1.1. A radius below one tends to forget old input; far above one can create oscillatory, chaotic dynamics and exploding gradients. Slightly above one aims to retain information without severe explosion. The paper also centres inputs and outputs and tunes input-to-hidden scale by task, warning that irrelevant distractors make excessively large input weights harmful.

Every trial runs 50,000 parameter updates with minibatches of 100 sequences. Momentum begins at .9 for 1,000 updates and then takes a selected value up to .995. Results average four random seeds. Table 5 uses task-level error measures rather than the optimized squared error or cross-entropy: fraction of incorrect time steps, failed memorization cases or outputs exceeding an error tolerance.

NAG at .995 reaches .00025 on addition and .00005 on 20-bit memorization. CM at .98 is best among displayed settings on multiplication with .025, while NAG at .995 gives .63 on 5-bit memorization. These counterexamples are important. NAG tolerates large μ well and is often stronger, but it does not dominate every task. The paper’s larger achievement is that ordinary RNNs trained by carefully configured first-order momentum can solve dependencies once considered nearly inaccessible.

Conjugate gradients carry a second kind of velocity

Hessian-free optimisation builds a local quadratic model and approximately solves it with conjugate gradients. Although called a second-order method, each inner CG step is itself first-order on that quadratic. CG accumulates information about curvature-aligned directions during its inner iterations, preferring directions whose eigenvalues cluster favourably.

The connection to momentum becomes sharper through hot starts. An HF run can initialize the new CG solve from a scaled version of the previous solution instead of zero. That old solution carries useful direction across outer updates, much as velocity carries it across first-order steps. If CG is terminated after one step, the paper says HF becomes equivalent to NAG except that curvature determines the learning-rate-like factor.

This does not collapse HF and NAG into the same algorithm. HF constructs and trusts a changing quadratic model, uses curvature-vector products and performs multiple inner iterations. Momentum updates from stochastic gradients and a compact velocity state. Their failure modes and costs differ. The comparison instead identifies a shared principle: optimization accelerates when information about persistent low-curvature directions survives long enough to guide later motion.

The authors modify HF to be more momentum-like by removing weight decay late in training, producing the HF dagger result in Table 1. They also compare CG and NAG on a local quadratic and note mixed evidence. HF may benefit when the quadratic remains faithful over the inner solve; NAG may benefit when repeated recomputation tracks a changing true objective. Approximation fidelity is part of the trade-off.

Read every headline result with its design choices attached

The primary’s empirical case has a useful internal structure. The autoencoder study varies momentum type and maximum, selects learning rate for each setting, tests a late μ reduction and separately changes initialization scale. The RNN study fixes a documented initialization, varies momentum type and level, reports four seeds and uses task-interpretable errors. Theoretical analysis then supplies a quadratic mechanism that explains why NAG can damp steep directions.

That triangulation is stronger than one benchmark table. At the same time, each element has limits. The theory is local and quadratic. The autoencoder evidence concerns training squared error without regularization. The RNN tasks are synthetic, and Table 5 contains wins for both NAG and CM. Hyperparameter search is finite and configuration-specific. The experiments demonstrate feasibility under careful design, not an invariant ordering of methods.

The historical context also matters. Sigmoid and tanh networks, Hessian-free optimisation and hand-designed synthetic dependencies occupied a different part of the 2013 landscape than modern residual architectures, normalization layers, ReLU-family activations, adaptive optimisers and very large batches. Later systems should not be judged by reproducing the table values. The durable contribution is the co-design lens: initialize signal dynamics, choose memory, schedule that memory and measure the whole trajectory.

Momentum laboratory: rotate the ravine, look ahead and audit every step

The laboratory below runs gradient descent, classical momentum and NAG on the same two-dimensional positive-definite quadratic. Profiles alter curvature, rotation, learning rate, momentum and schedule. The displayed paths are generated from the defining equations rather than illustrative animation.

Four images to carry away

From scalable structures to trainable trajectories

Paper 0043 treated optimization as algorithm–system co-design: gradients and Hessians became tree scores that could move through sparse columns, caches and disks. Paper 0044 turns from data movement to parameter movement. It asks why a mathematically simple first-order update succeeds or fails inside a deep composition. In both papers, the headline algorithm is inseparable from the pathway that makes its statistics usable.

The paper also joins several earlier canon threads. Robbins–Monro supplied stochastic approximation and the need for gain schedules. Kiefer–Wolfowitz showed how noisy direction estimates depend on coupled scales. Friedman’s gradient boosting made function-space descent explicit. Adam later stored coordinate-wise first and second moments, while AdamW separated decay from its adaptive geometry. Sutskever et al. clarify the intermediate step: one global velocity already becomes curvature-sensitive when Nesterov look-ahead is analysed in the Hessian eigenbasis.

For the consciousness-and-agency arc, momentum is a disciplined example of memory altering action. The velocity is not a record for its own sake. It compresses recent directional evidence into a state variable that changes the next observation point. The analogy should stop there: an optimizer has no experience or intention. But it provides a clean formal instance of history-dependent agency, in which the same present gradient yields a different action because the system carries a different past.

The forward question is how increasingly complex learning systems manage state, scale and geometry without hand-tuning every phase. Later optimisers, normalization schemes and architectural changes distribute that burden across more mechanisms. Paper 0044 establishes the causal vocabulary needed to judge them: starting dynamics, directional persistence, curvature, schedule, experimental protocol and evidence boundary.

Complete the square of motion, not only the objective

How to audit an initialization-and-momentum claim

1. Specify the starting distribution. Record fan-in, scale, bias, activation, recurrent spectrum and random seeds.
2. Separate optimizer from schedule. State ε, μ, warm-up, decay, fine-tuning and how each was selected.
3. Inspect trajectories, not only endpoints. Plot loss, gradient norms, velocity norms, oscillation and saturation across the transient and final phases.
4. Match the comparison budget. Count updates, gradient evaluations, curvature products, minibatch sizes, tuning trials and wall-clock cost.
5. Distinguish optimization from prediction. Report training objective and held-out performance separately, with regularization held explicit.
6. Stress the geometry. Test shallow and steep directions, rotations, stochasticity, abrupt changes and initialization scale.
7. Preserve counterexamples. If CM beats NAG on one task or large μ fails under one start, keep that result in the conclusion.