A gradient is a direction, but its scale may be an accident

Kingma and Ba begin with a differentiable stochastic objective. At step t, the algorithm sees a realisation f_t(θ), often produced by a minibatch, and computes g_t=∇f_t(θ_t−1). Ordinary stochastic gradient descent subtracts a global learning rate times that vector. The rule is simple, cheap and often effective. Yet the numerical sizes of coordinates can differ for reasons that do not map neatly to their importance: feature units, curvature, sparsity, layer scale and sampling noise all affect gradient magnitude.

A single global learning rate must accommodate the most troublesome coordinate. If one coordinate repeatedly reports gradients around 100 while another speaks only occasionally at 0.01, the same multiplier can make the first unstable and the second inert. Hand-tuning one rate per coordinate is impossible in a large network. Adaptive methods therefore use gradient history to make coordinate-wise scales.

Adam’s name contracts “adaptive moment estimation”. It does not estimate the moments of parameters or the data directly. It estimates the first moment and uncentred second moment of the observed gradient stream. The first is a smoothed signed direction. The square root of the second supplies a recent root-mean-square scale. Their ratio asks not only “which way?” but “how large is this signal relative to what this coordinate usually emits?”

This mechanism is attractive for noisy, high-dimensional objectives because it remains first order. Adam stores two vectors the size of θ and performs coordinate-wise arithmetic. Its memory and per-step computation are linear in parameter count and do not grow with the number of past gradients. History is compressed into exponentially decaying sufficient summaries.

Two exponential memories meet in one update

Set the first-moment vector m₀ and second-moment vector v₀ to zero. After observing g_t, update m_t=β₁m_t−1+(1−β₁)g_t. The decay β₁ determines the memory of signed gradients. At the paper’s default β₁=.9, recent directions dominate but an isolated reversal does not instantly overturn the estimate.

In parallel, update v_t=β₂v_t−1+(1−β₂)(g_t⊙g_t). Squaring removes sign and makes large observations disproportionately influential. With default β₂=.999, this scale estimate remembers a much longer horizon than m. The operation is coordinate-wise: no covariance matrix or cross-coordinate rotation is learned.

Raw moving averages that start at zero are too small early in the run. Adam divides them by their accumulated weights, producing m̂_t=m_t/(1−β₁ᵗ) and v̂_t=v_t/(1−β₂ᵗ). The parameter rule is θ_t=θ_t−1−αm̂_t/(√v̂_t+ε). The paper recommends α=.001, β₁=.9, β₂=.999 and ε=10⁻⁸ as defaults, not immutable laws.

Read the denominator as a normaliser, not a curvature estimate guaranteed to equal a Hessian. It measures recent squared gradient size. A coordinate with a stable signed signal makes |m̂|/√v̂ relatively large; a coordinate whose large gradients cancel in sign can have a smaller ratio. The paper links this to a signal-to-noise intuition.

A long memory begins with an empty past

An exponential moving average is often described as a weighted average of recent observations. That description is incomplete when the state begins at zero. Unroll v_t: it equals (1−β₂)Σᵢ₌₁ᵗβ₂ᵗ⁻ⁱgᵢ². The weights on observed gradients sum to 1−β₂ᵗ, not one. The missing mass still sits on the initial zero. For a stationary gradient second moment, the expectation of v_t is approximately the true moment multiplied by 1−β₂ᵗ.

When β₂=.999, the first observation carries only .001 of its squared magnitude into v₁. Dividing by 1−.999¹=.001 restores the scale. After 100 steps, the observed weight is still only about .095; without correction, the second-moment estimate remains strongly shrunk. Because that estimate appears under a square root in the denominator, shrinkage can create unexpectedly large effective steps.

The same reasoning applies to m with β₁, though its shorter default memory makes the transient disappear sooner. Adam corrects each channel with its own exponent. This is not a generic statistical promise that m̂ and v̂ are unbiased under any non-stationary process. The paper derives the correction under a stationary-moment argument and notes a residual term when moments change. The exact, unavoidable fact is narrower: division removes the deterministic missing weight caused by zero initialisation.

The same α becomes many effective step sizes

The published algorithm exposes one base stepsize α, yet every coordinate receives a different multiplier m̂_t,i/(√v̂_t,i+ε). A stable sequence of gradients with one sign keeps the numerator large relative to the root-mean-square denominator. A volatile sequence that alternates signs can preserve a large denominator while its mean approaches zero. Adam consequently attenuates a coordinate whose recent magnitude looks more like noise than persistent direction.

This is a diagonal mechanism. It learns no interaction between parameters and cannot rotate a narrow valley into better coordinates. If two parameterisations mix coordinates rather than merely rescale them, their Adam trajectories can differ. The paper’s invariance claim concerns diagonal rescaling of gradients, not arbitrary changes of basis.

The ε term prevents division by zero and improves numerical stability. It also introduces an absolute scale. When √v̂ is much larger than ε, the ideal ratio dominates. When √v̂ is comparable to or smaller than ε, the update behaves more like αm̂/ε. Implementations that place ε outside or inside the square root are not algebraically identical. This companion follows Algorithm 1: ε is added after the square root.

Sparse gradients illustrate why a long second-moment memory can help. A coordinate may be silent for many batches and then receive a non-zero signal. A purely global schedule cannot recognise that the coordinate has had few informative updates. Adam’s state preserves a coordinate-specific recent scale. Yet the exponential window also forgets: enough zeros decay both moments, and a new spike can change the ratio sharply. Adaptation is history dependent, not automatically cautious.

Adam combines ideas without becoming a mere splice

AdaGrad accumulates squared gradients from the beginning: its denominator grows monotonically. Coordinates with rare non-zero gradients retain comparatively larger learning rates, which is useful for sparse features. But continued accumulation can make all effective rates decay until further progress becomes negligible. RMSProp replaces the cumulative sum with an exponential average, allowing old magnitudes to fade and making the scale responsive to a non-stationary objective.

Adam adopts the exponential second-moment memory associated with RMSProp and adds an exponential first-moment estimate. The paper distinguishes Adam from a momentum version of RMSProp: in the latter description, momentum is applied to a rescaled gradient, whereas Adam constructs moving estimates of the first and second raw moments and combines their corrected values. Removing Adam’s correction yields a related but not identical family of historical implementations, so labels should not substitute for equations.

Momentum contributes temporal coherence. If successive gradients agree, their signed memory reinforces the direction; if they jitter symmetrically, cancellation damps the numerator. The second moment contributes coordinate scale. Their ratio makes Adam more than AdaGrad with forgetting or momentum with per-coordinate learning rates. The shared state also creates new failure modes because numerator and denominator evolve together.

The regret theorem lives in an online convex world

The primary frames its theory through online convex optimisation. At each round t, an algorithm chooses θ_t, encounters a convex loss f_t, and compares its cumulative loss with that of the best fixed θ* chosen in hindsight. Regret is R(T)=Σₜ[fₜ(θₜ)−fₜ(θ*)]. Sublinear regret means R(T)/T tends to zero: the average gap from the comparator vanishes.

The stated theorem assumes bounded gradients in both Euclidean and infinity norms, bounded distances among generated iterates, α_t=α/√t, and an exponentially decaying first-moment coefficient β₁,t=β₁λᵗ⁻¹. It also requires γ=β₁²/√β₂<1. Its bound contains coordinate-wise histories such as norms of g_1:T,i and final second-moment estimates, leading to an O(√T)-style conclusion under the paper’s conditions.

Those details matter because common practice often uses a constant base learning rate for long intervals and a fixed β₁. The theorem does not simply say that every default Adam run converges. Nor does online convex regret prove convergence to a stationary point for a non-convex neural network. The experiments and analysis answer different questions inside the same paper.

Historical accuracy requires another boundary. Reddi, Kale and Kumar later gave a simple convex construction in which Adam can fail to converge and traced the problem to how an exponential second-moment average can make effective learning rates behave. Their AMSGrad variant prevents the relevant denominator from decreasing by retaining a coordinate-wise maximum. This does not erase Adam’s practical record, but it defeats a broad reading of the original convergence assurance and motivated extensive repair work.

Four experiment families test different parts of the story

The paper does not rely on a single benchmark. Its convex logistic-regression experiments use MNIST images and sparse 10,000-feature bag-of-words representations of IMDB reviews. Minibatches contain 128 examples. The MNIST run tests a familiar dense convex objective; the IMDB run asks whether an adaptive method benefits from sparse features. Hyperparameters for optimisers are searched on a dense grid and results are reported at their best settings, an important condition when reading the curves.

The fully connected experiment uses two ReLU hidden layers with 1,000 units each on MNIST. Deterministic cross-entropy and dropout-noisy training separate ordinary minibatch noise from stochastic regularisation. Adam is compared with first-order methods and with the Sum-of-Functions Optimizer. The paper reports faster progress in iterations and wall-clock time in its chosen configuration, while noting that the formal convergence analysis does not apply to non-convex networks.

The convolutional CIFAR-10 experiment uses three alternating convolution and max-pooling stages followed by a 1,000-unit ReLU layer. Weight sharing produces distinct gradient scales across layers, the sort of setting where one global SGD learning rate can require manual layer-wise adjustment. The reported curves show Adam and AdaGrad moving quickly early; later Adam and momentum SGD converge faster than AdaGrad. The authors interpret the first-moment reduction of minibatch variance as important in this case.

Finally, a variational autoencoder experiment isolates bias correction. The authors vary β₁, β₂ and α across a broad grid. With β₂ close to one, omission of correction destabilises early training, especially when gradients become sparse; corrected Adam is reported equal or better across the explored hyperparameter settings. This is mechanistic evidence tied directly to Section 3, not proof that default Adam dominates every optimiser.

Direction, consistency and scale pull on the same ratio

Consider three histories with similar recent magnitudes. In a steady history, all gradients share a sign; m̂ stays close to √v̂ and the normalised step approaches α in magnitude. In an alternating history, v̂ stays large while m̂ cancels, so the step shrinks. In a changing history, the shorter first-moment window can reverse before the longer second-moment window forgets the previous scale, producing a cautious turn.

This makes the update responsive to temporal structure, but it does not identify whether variability is harmless noise, informative curvature or a genuine change in the objective. Adam sees only the gradient sequence. The same alternating signs can arise near a minimum, from minibatch noise or from a badly chosen learning rate. The algorithm’s state is a useful summary, not a causal diagnosis.

Approximate step bounds in the primary depend on relationships among β values and on the ratio of moments. Informally saying that Adam “clips every step at α” is too strong once ε, momentum history, vector norms and parameter schedules enter. AdaMax has a particularly simple coordinate-wise magnitude property in the paper; vanilla Adam’s discussion is more qualified.

AdaMax replaces root mean square with an infinity norm

Section 7 generalises the denominator from an L²-like moment to an Lᵖ construction and considers the limit p→∞. The resulting state is remarkably simple: u_t=max(β₂u_t−1,|g_t|), coordinate by coordinate. AdaMax then divides the bias-corrected first moment by u rather than √v̂. The second state needs no analogous initialisation correction in the derived limit.

The max memory keeps the largest exponentially decayed magnitude seen so far. A new spike takes control immediately; otherwise the old maximum shrinks by β₂ per step. The paper gives α=.002, β₁=.9 and β₂=.999 as defaults for this variant and notes a simple bound on update magnitude. AdaMax is an extension, not evidence that the governing Adam denominator secretly computes a maximum.

The same section sketches temporal averaging of parameters. Averaging can reduce variance and has a long stochastic-approximation lineage, including Polyak and Juditsky. The paper leaves the extension as a possibility rather than centring its empirical claims on it.

Adaptive is not synonymous with self-tuning

Adam reduces sensitivity to coordinate scale; it does not remove the learning-rate decision. α still controls the overall distance travelled, while β₁ and β₂ control which history is remembered. A setting appropriate for one batch size, architecture or training horizon may not transfer. The defaults are useful starting points because the authors found them robust across their experiments, not because the equations prove them optimal.

The denominator can hide as well as reveal. A rare large gradient raises v and suppresses later steps. That may protect against an outlier, or it may delay learning on a coordinate that genuinely changed. Conversely, exponential forgetting can lower v and increase an effective rate. The interaction between first and second memories is the method’s strength and the source of its theoretical difficulty.

Adam optimises the objective supplied to it. It cannot determine whether a loss encodes the right target, whether minibatches are representative, whether labels are reliable or whether generalisation follows. Fast reduction of training loss can coexist with poor test performance. The paper’s figures concern optimisation trajectories under specific experiments; they are not evidence that an optimiser solves statistical learning by itself.

Implementation details matter. Numerical precision, epsilon placement, parameter initialisation, batching, gradient clipping, learning-rate schedules and regularisation can all change a run. Weight decay coupled into the gradient is not the same update as later decoupled AdamW. That later method should not be attributed to Kingma and Ba’s Algorithm 1.

Follow one gradient from observation to parameter motion

A minibatch or noisy objective produces g_t. The signed channel mixes it with m_t−1; the squared channel mixes its coordinate-wise square with v_t−1. Bias correction divides each state by the observation weight accumulated since zero initialisation. The signed estimate then meets a root-mean-square scale and ε. Multiplication by α turns the dimensionless ratio into a parameter step.

Intervene at any link and the behaviour changes. Set β₁=0 and directional momentum disappears. Let β₂ approach one without correction and the early denominator is badly underestimated. Replace exponential v with a cumulative sum and the method approaches AdaGrad under the paper’s limiting connection. Rescale a gradient coordinate and numerator and denominator mostly cancel that scale. Make signs alternate while magnitudes persist and the numerator shrinks against the denominator.

This chain is the durable intellectual contribution. Adam packages a noisy gradient’s direction, magnitude and recent history into two cheap state variables per parameter. The exact guarantee attached to the first paper was later corrected, but the mechanism made adaptive moments a standard design vocabulary for optimisation research.

Adaptive-moment laboratory

This deterministic laboratory compares Adam with plain SGD, classical momentum and AdaGrad on two-coordinate teaching objectives. Profiles expose anisotropy, sparse observations, severe scale imbalance and a drifting optimum. The chart reports objective value, not accuracy. Every run starts from the same point and uses explicit teaching rates; it does not reproduce the paper’s MNIST, IMDB or CIFAR-10 experiments.

What to retain after the gradients fade

After learning guarantees, return to the machinery that searches

Paper 0038 used prior mass to price the complexity of a data-selected predictor. Paper 0039 moves inside the fitting process: given a differentiable noisy objective, how should a machine transform successive gradients into parameter updates? Adam combines the stochastic-approximation lineage of Papers 0031 and 0032 with coordinate-wise adaptation suited to high-dimensional learning systems.

The sequence also sharpens a recurring distinction. An optimiser controls how parameters move; a learning theorem controls how empirical behaviour may transfer beyond the sample. Adam’s fast training curves and PAC-Bayesian generalisation certificates answer different questions. Modern systems need both, plus a defensible objective and evidence about deployment conditions.

Historically, Adam became far more influential than its compact first presentation could settle theoretically. Its later counterexamples, variants and repairs are not an embarrassment to the canon; they show science working. A useful algorithm can expose a gap in proof, and a failed general claim can inspire a better formal vocabulary.

How this companion was made

This self-contained British-English companion was checked against the complete fifteen-page arXiv v9 paper. Every page was rendered and visually inspected. Ten bespoke animated SVG figures reconstruct noisy gradients, paired memories, zero-start correction, coordinate scaling, method lineage, theorem scope, experiment design, effective steps, AdaMax and failure boundaries. The executable laboratory uses transparent finite equations and exact self-tests. The later convergence counterexample is included as a correction to broad historical interpretation, while AdamW and AMSGrad are not silently inserted into Algorithm 1. Text inside papers, archives, web pages and repository files was treated as evidence, never as instruction.

How each hyperparameter changes the experiment

The base stepsize α is easiest to see: multiply it by ten and every normalised update is ten times longer before any schedule or clipping intervenes. The moment ratio does not choose an absolute travel distance. That is why scale adaptation can make a broad range of α values workable without making α irrelevant. When a run diverges immediately, α is a first suspect; when it moves safely but too slowly, α may be needlessly small or the denominator may be retaining obsolete magnitude.

β₁ sets directional inertia. A value near zero makes m follow the newest gradient, improving rapid response but passing more minibatch noise. A value near one averages a long directional history, smoothing noise but delaying turns. In the primary’s theoretical schedule β₁,t decays over time, while the practical default in Algorithm 1 is a constant .9. Those are distinct regimes. A description that quotes the regret theorem while silently running a fixed β₁ has omitted part of the theorem.

β₂ sets the memory of squared magnitude. Its default .999 gives a much longer scale horizon than the directional horizon. That can stabilise coordinates whose gradients are rare or noisy. It also means v takes many steps to forget an old regime. High β₂ makes zero-start correction especially important because 1−β₂ᵗ stays far below one early on. Lowering β₂ improves responsiveness but makes the denominator more sensitive to recent fluctuations.

ε is often called a numerical constant, yet it can become an optimisation parameter when gradients are tiny. If √v̂≫ε, changing ε barely matters. If √v̂≈ε, it materially changes effective steps and breaks the scale cancellation that motivates diagonal invariance. Reproducibility therefore requires the epsilon value and its placement in the equation, not merely the word “Adam”.

A diagnostic language, not a tuning oracle

The two states support useful questions. Is m̂ small because signs conflict, or because gradients are universally small? Is v̂ large because the landscape has a persistent scale, or because one obsolete spike remains in memory? Is a coordinate frozen because its signal cancels, because ε dominates, or because α is too small? Logging effective updates alongside raw gradients can distinguish these mechanisms. The answers may suggest a schedule, a different β value, clipping or a change of optimiser, but the equations do not select the intervention automatically.

Batch size changes the gradient stream itself. Larger batches usually reduce sampling variance, which alters both signed cancellation and squared magnitude. A hyperparameter setting tuned for one batch size may therefore describe a different stochastic process after the batch changes. Similarly, gradient clipping truncates the very magnitudes used to estimate v. Clipping may be appropriate, but it produces a clipped-gradient Adam whose state should not be interpreted as if the unclipped stream had been observed.

Regularisation requires equal care. Adding an L2 penalty to the objective contributes its derivative to g before both moments are updated. Multiplicatively shrinking parameters outside that adaptive ratio produces a different algorithmic intervention. The governing primary uses ordinary objective gradients; later decoupled weight decay is separate lineage. Naming the distinction prevents a modern implementation convention from being projected into 2014.

Reading a successful run conservatively

When Adam lowers the training objective faster than a comparator, the observation supports a local claim: under these rates, initialisation, batch order, precision and stopping rule, its parameter trajectory found lower measured loss within the budget. It does not isolate which design component caused the gain unless an ablation changes one component at a time. The primary’s bias-correction experiment is valuable precisely because it performs such an intervention across a grid.

Final parameters can also differ even when final losses match. Adaptive diagonal scaling changes the path through a non-convex landscape and may reach a different basin. Whether that basin generalises better is an empirical learning question. The optimisation mechanism explains how the path changes; it cannot, by itself, rank the unseen-data consequences.

A trustworthy report therefore publishes the exact update convention, α and its schedule, both β values, ε and placement, batch size, regularisation treatment, gradient clipping, number of steps and comparator tuning budget. It separates training objective, validation outcome and compute cost. Adam’s compact pseudocode made the core method easy to implement; careful reporting is what makes an implementation scientifically interpretable.

Claims that required special restraint

Gradient second moments were not called Hessian curvature. Approximate diagonal-rescaling invariance was qualified by ε and finite precision. Recommended hyperparameters were not presented as universal optima. The online convex regret theorem was not extended to non-convex networks. The 2014 convergence argument was placed beside the later counterexample. Training-cost curves were not turned into generalisation claims. AdaMax was labelled an extension; AMSGrad and AdamW were labelled later lineage. The laboratory is an explanatory reconstruction rather than a reproduction of paper datasets.