Two routes towards smaller weights are not always the same route
Practitioners often use “L2 regularisation” and “weight decay” as synonyms. Both can pull parameters towards zero, and under basic stochastic gradient descent their updates can be made identical. That familiar equivalence encouraged libraries to implement an L2 penalty while labelling the coefficient “weight decay”. Loshchilov and Hutter show why this shortcut becomes misleading when an optimiser adapts gradients coordinate by coordinate.
L2 regularisation modifies the objective. Add (λ′/2)||θ||² to the batch loss and differentiation adds λ′θ to the loss gradient. Whatever transformation the optimiser applies to gradients now also acts on the penalty contribution. Momentum integrates it. Adam includes it in first and second moments and divides it by an adaptive denominator.
Classical weight decay is an update operation. Before or alongside the loss step, multiply θ by 1−λ. The shrinkage does not pretend to be evidence from the current batch, and it does not enter the gradient moments. Every selected parameter is reduced by the same declared fraction under the paper’s convention.
The distinction is causal. One route changes the signal entering the optimiser. The other changes parameters separately from that signal. If the optimiser is a scalar multiplication, the two effects can be reparameterised into equality. If the optimiser applies a non-scalar, history-dependent transformation, the same L2 gradient no longer produces uniform decay.
Causal probe 1 · Why did the confusion survive?
For ordinary SGD the updates are algebraically identical after coefficient rescaling, so calling L2 “weight decay” often produced the expected trajectory. Adaptive methods inherited the name after the condition that made the synonym safe had disappeared.
SGD makes the two routes coincide after rescaling
Take standard SGD with base learning rate α. On an L2-regularised batch loss, the update is θt+1=θt−α[∇ft(θt)+λ′θt]. Distribute α and the parameter term becomes −αλ′θt.
Direct weight decay followed by the SGD loss step is θt+1=(1−λ)θt−α∇ft(θt). Expand the first product and its parameter term is −λθt. Choose λ′=λ/α and every term matches. Starting from the same θ, the methods generate the same next point and therefore the same entire sequence under the same batch gradients.
decay + SGD: θₜ₊₁ = θₜ − α∇fₜ(θₜ) − λθₜ
equivalent when λ′ = λ/α
The equivalence is exact but conditional. Change α without changing λ′ and the implied decay changes. That hidden coupling can make an L2 coefficient appear sensitive to the learning rate. The paper’s SGDW pseudocode separates a schedule multiplier from an explicit decay term so that the user can reason about the two controls separately under its convention.
Worked example 1 · Two names, one scalar step
Let θ=10, loss gradient 3, α=.1 and direct decay λ=.02. Weight decay gives 10−.3−.2=9.5. L2 uses λ′=.02/.1=.2, giving gradient 3+.2×10=5 and update 10−.1×5=9.5. With λ′=.02 instead, the result would be 9.68 and equivalence would fail.
Causal probe 2 · Why is λ′ tied to α?
L2 enters before multiplication by the learning rate, while direct decay is already a parameter-space step. To make their shrink terms equal, αλ′ must equal λ.
An adaptive preconditioner sends one penalty in many directions
Write a first-order optimiser’s loss update as −αMt∇ft(θt), where Mt is a coordinate transformation. For SGD, M is the identity. For Adam, the effective transformation is diagonal and history dependent: coordinates are divided by different root second-moment scales and combined with signed moment history.
Add isotropic L2 and its gradient λ′θ passes through Mt too. The penalty step becomes −αλ′Mtθ. Direct decay remains −λθ. One scalar λ′ can make these equal for all θ only when Mt behaves like a scalar multiple of the identity. An adaptive diagonal matrix with unequal entries breaks the equivalence.
The consequence is subtle. L2 no longer applies the same relative shrink rate to every coordinate. A parameter whose gradient history creates a small adaptive step also receives a small transformed penalty step. Direct decay shrinks selected parameters uniformly under its definition, independently of their gradient history.
decoupled weight-decay step: −λθₜ
no single λ′ works generally when Mₜ ≠ kI
Worked example 2 · A two-coordinate mismatch
Let M=diag(1,.01), θ=(10,10), α=.1 and λ′=.2. The L2 contribution is (−.2,−.002). Direct decay with λ=.02 gives (−.2,−.2). Matching coordinate one leaves coordinate two one hundred times too weak; matching coordinate two would make coordinate one one hundred times too strong.
Causal probe 3 · Which weights does adaptive L2 regularise less?
Coordinates whose preconditioned update is small also have their L2 penalty gradient suppressed. In Adam, historically large gradient magnitudes often enlarge the denominator, so those coordinates can receive less relative L2 shrinkage than direct decay supplies.
AdamW keeps regularisation out of the moment memories
In commonly coupled Adam, the batch gradient and λ′θ are added before the first and second moments update. The penalty affects the signed numerator, the squared-magnitude denominator and all future steps through exponential memory. What appears to be a simple pull towards zero becomes entangled with data gradients and their history.
AdamW computes moments from the loss gradient alone. After bias correction, the Adam loss update is applied, and a separate term proportional to θ performs weight decay. The primary also includes a schedule multiplier, allowing the global learning-rate schedule and the decay step to be coordinated without treating the decay as a batch gradient.
Decoupling does not mean “no interaction”. Both operations change θ, so future loss gradients and moments reflect the new position. It means that the current penalty is not preconditioned or stored as if it were evidence from the current batch. That narrower structural claim is the heart of AdamW.
The same design yields SGDW for momentum SGD. Although ordinary SGD can reproduce weight decay with a rescaled L2 coefficient, explicit separation makes the learning-rate and decay controls easier to reason about and tune. In momentum implementations it also specifies whether shrinkage enters velocity or acts outside it.
Worked example 3 · Zero loss still reveals the update
If the loss gradient is zero, AdamW’s moment-based loss step is zero, while θ becomes (1−λ)θ under the simplest convention. Coupled L2 Adam instead sends λ′θ through its moment machinery; the resulting coordinate step depends on moment history and epsilon. The difference exists even without a data gradient.
Causal probe 4 · Why keep λθ out of v?
v is meant to adapt the scale of loss-gradient evidence. Squaring the penalty contribution makes the regulariser help set the denominator that then weakens its own effect. Decoupling preserves an explicit, interpretable shrink rate.
Fixed adaptive scaling corresponds to an anisotropic penalty
To make the mismatch more interpretable, the paper considers a fixed diagonal preconditioner. Under that simplified optimiser, direct uniform decay can be reproduced by a scale-adjusted quadratic penalty whose coordinates are weighted according to the preconditioner. Ordinary isotropic L2 is therefore the wrong equivalent objective once coordinates are transformed differently.
Practical Adam changes its preconditioner every step, so the proposition is intuition rather than a fixed global objective for the full run. At any step, however, it shows why a coordinate’s gradient history changes the effective regularisation applied by coupled L2.
The paper also includes a Bayesian-filtering justification credited to Laurence Aitchison. In that view, parameter tracking alternates a state-transition prior with evidence from a new mini-batch. Decay enters naturally as a data-independent transition of the parameter mean, while the likelihood update corresponds to gradient evidence. The separation mirrors AdamW’s architecture.
Evidence boundary · An interpretation is not the central proof
The algebraic inequivalence in Section 2 stands without Bayesian filtering. The filtering account arrived after the preliminary arXiv version and is explicitly credited to Aitchison. It supports why decay belongs outside the data gradient; it should not be misattributed or treated as the sole justification.
Decoupling turns a diagonal ridge into a more separable search
When L2 enters before multiplication by the learning rate, its parameter-space effect depends on α. A heat map over α and λ′ therefore tends to place good settings along a diagonal relationship: change one and the other must change to preserve similar shrinkage. The authors argue that this entanglement contributes to SGD’s reputation for hypersensitivity.
Explicit decay gives α and λ clearer roles. α scales the loss-based optimisation step; λ scales parameter shrinkage. The empirical heat maps for SGDW and especially AdamW show good regions that align more closely with the axes. “Decoupled” does not promise statistical independence: both settings still influence the final trajectory: but it removes a built-in algebraic coupling and makes search more separable.
For Adam, the difference is deeper than scalar coupling because the L2 contribution is coordinate-preconditioned. AdamW both removes that distortion and exposes a direct decay knob. The studied heat maps report top-1 test error after fixed budgets, so their geometry combines optimisation, regularisation and generalisation under those experiments.
Worked example 4 · A diagonal band in a search grid
Under SGD equivalence, direct λ=.001 can be represented by λ′=.01 when α=.1 or λ′=.1 when α=.01. Good equivalent pairs lie on a relation rather than at one constant λ′. Decoupled decay lets λ remain .001 while α changes, although the best empirical combination may still shift.
Causal probe 5 · What does “more separable” mean?
Changing learning rate no longer mechanically rescales the decay coefficient through the update equation. A grid’s good region can align more with independent axes, reducing joint search complexity. It does not mean the effects are causally or statistically independent in the trained model.
Image-classification grids test algebra against generalisation
The main experiments follow the Shake-Shake residual-network setup on CIFAR-10. The base models are 26-layer, two-branch networks with widths 64 and 96, containing about 11.6 and 25.6 million parameters. Training uses batches of 128 and standard CIFAR augmentation. The authors also test ImageNet32x32, a 1.2-million-image downsampled version of ImageNet.
They compare fixed, step-drop and cosine-annealing learning-rate schedules. In the 100-epoch experiment, step drops occur at epochs 30, 60 and 80. AdamW improves over Adam with L2 across these schedules, with larger gains under better schedules. The paper reports cosine annealing as clearly superior in that experimental matrix and uses it for later comparisons.
Across CIFAR-10 and ImageNet32x32, networks, budgets from 100 to 1,800 epochs and search grids, the paper reports that decoupled decay improves Adam’s test performance. Its summary quantifies a 15% relative test-error improvement in the studied comparisons and finds AdamW competitive with momentum SGD. Training loss and test error are both inspected, helping distinguish easier fitting from improved generalisation.
These are extensive experiments for the paper’s question, yet the authors explicitly request verification on a wider range of tasks. Image classification, residual networks and Shake-Shake regularisation do not establish universal dominance. Community adoptions reported in the later revision broaden practical interest but are not controlled replications of the central grid.
Evidence boundary · Relative improvement needs its denominator
A 15% relative reduction in error is not a fifteen percentage-point gain, and it belongs to the reported comparisons rather than every Adam run. The correct claim keeps dataset, architecture, schedule, budget and tuning procedure attached.
Worked example 5 · Relative and absolute error
If Adam’s error is 4.0% and AdamW’s is 3.4%, the absolute reduction is .6 percentage points while the relative reduction is .6/4=.15, or 15%. Both are valid descriptions; confusing them multiplies the apparent gain.
More updates multiply the same shrink more times
A raw per-step decay cannot be interpreted independently of the number of steps. Apply factor 1−λ N times with zero loss gradient and the parameter becomes (1−λ)Nθ₀. A longer run or smaller batch creates more decay events and therefore stronger cumulative shrinkage.
The appendix proposes λ=λnorm√(b/(BT)), where b is batch size, B the number of training examples and T the epoch budget. The square-root scaling aims to make useful settings transfer better across datasets and runtimes. Experiments on CIFAR-10 and ImageNet32x32 support improved robustness of the normalised parameter.
The authors are notably cautious. Their normalisation is one empirically informed possibility, derived from limited experiments; better scaling rules may exist. The lasting conclusion is that decay should be interpreted relative to update count and budget, not that this square-root formula is universally correct.
Causal probe 6 · Why does a smaller batch strengthen raw decay?
At fixed epochs and dataset size, a smaller batch produces more updates. The multiplicative shrink is applied more times, so identical per-step λ yields greater total contraction.
Worked example 6 · Compounding reveals budget dependence
With λ=.001 and no loss gradient, 100 steps retain about .999¹⁰⁰≈.905 of a weight. One thousand steps retain about .368. The same nominal coefficient produces radically different total shrinkage because the number of applications differs.
Adaptive coordinates can still benefit from a global clock
Adam adapts learning rates per parameter, but that does not eliminate the value of a global multiplier. The paper finds that step-drop and especially cosine schedules improve Adam’s performance. A schedule changes the overall scale of all adaptive steps; coordinate adaptation changes relative scales. They answer different questions.
The appendix carries cosine annealing with warm restarts into AdamWR and SGDWR. Within a cycle, the multiplier descends smoothly along half a cosine. At a restart it rises without reinitialising parameters; the next cycle can be longer. The aim is good anytime performance and repeated opportunities to move at a larger global rate.
Warm restart is potentially confusing language. It does not discard the learned model and begin from random initialisation. Parameters persist; the learning-rate schedule resets. Momentum or other state handling must be specified by the implementation. The governing paper’s pseudocode and schedule conventions define its experiment.
Worked example 7 · Half a cosine
With ηmax=1 and ηmin=0, the multiplier is .5[1+cos(πTcur/Tᵢ)]. It begins at 1, reaches .5 halfway through the cycle and reaches 0 at the scheduled end, before a restart returns it upwards.
Causal probe 7 · Why schedule Adam if it already adapts?
Adam’s denominator sets coordinate-relative effective rates from gradient history. A cosine multiplier controls the global exploration-to-refinement timescale. Per-coordinate adaptation does not decide when the entire run should cool.
“AdamW” is an equation, not permission to ignore policy
Decoupled decay still requires deciding which parameters to shrink. Biases and normalisation parameters often receive different treatment in later practice because their roles differ from ordinary weights. The central proposition concerns the algebra of selected θ coordinates; it does not prove one universal exclusion policy.
Framework conventions vary. Some write θ←θ−α(update+λθ), making shrink per step αλ. Others expose a decay coefficient already scaled into parameter space. Schedules may multiply both loss and decay or only the loss step. Two optimisers labelled AdamW can therefore trace different paths unless coefficient and schedule semantics are stated.
Decoupling improves interpretability; it does not guarantee good regularisation. Too much uniform decay erases useful weights. Too little leaves capacity unchecked. Data augmentation, architecture, early stopping and other penalties interact with the result. The heat maps show that search becomes more separable under the studied convention, not unnecessary.
The image-classification evidence is strong enough to motivate adoption but deliberately bounded. The paper’s conclusion asks for wider task verification. Its normalised decay and warm-restart extensions have smaller evidence bases than the core inequivalence. A companion should not promote one empirical square-root normalisation into a law of training.
Evidence boundary · Decoupled is not automatically correctly calibrated
AdamW fixes where shrinkage acts. It does not choose the optimal coefficient, parameter subset, schedule or budget. Those remain experimental decisions whose conventions must be published.
Worked example 8 · Two APIs, same label, different shrink
API A applies θ←θ−αλθ with α=.001 and λ=.01, retaining .99999 per step. API B applies θ←(1−λ)θ with the same displayed λ, retaining .99. After 1,000 zero-gradient steps the difference is enormous. Always inspect the actual equation.
Causal probe 8 · Why might a bias be excluded?
Direct decay expresses a prior preference for small parameter magnitude. A scale or offset parameter may have a different functional meaning from a weight matrix. Exclusion can be defensible, but it is a modelling policy beyond the paper’s equivalence proof.
Follow the regulariser through the optimiser
Start with θ and a batch loss. In coupled L2, differentiate the penalty and add λ′θ to the data gradient. Momentum or adaptive moments then store the sum. Coordinate preconditioning scales both sources together. In decoupled decay, moments store only the data gradient; after the loss step, a direct scheduled term shrinks θ.
Intervene on the optimiser. Replace Adam’s diagonal history with scalar SGD and coefficient conversion restores equality. Make the preconditioner non-scalar and isotropic L2 becomes coordinate-distorted. Set the loss gradient to zero and direct decay still contracts by its declared factor, while coupled Adam’s action depends on its internal state.
Intervene on time. More updates compound decay. A smaller batch or longer budget changes total contraction. A global schedule can cool adaptive loss steps, and a warm restart can raise that multiplier without discarding θ. Every conclusion follows from locating the operation before computing its magnitude.
Coupled-versus-decoupled laboratory
This deterministic laboratory compares L2-SGD with SGDW, coupled L2-Adam with AdamW, and unregularised Adam on two-coordinate quadratic losses. The L2 coefficient is converted as λ′=λ/α so the two SGD paths must coincide. Adam branches share moment settings but differ in whether λ′θ enters the moments. Profiles vary gradient-scale imbalance, learning rate and budget. This is a mechanism reconstruction, not a reproduction of CIFAR-10.
What to retain after the weights shrink
Two operations
L2 changes the objective gradient; weight decay changes parameters directly.
SGD exception
Scalar SGD makes them equivalent when λ′=λ/α.
Adaptive break
A non-scalar preconditioner transforms the L2 penalty coordinate by coordinate.
AdamW move
Loss gradients build moments; a separate scheduled term shrinks selected θ.
Budget caution
Per-step decay compounds, so batch size and number of updates affect total shrinkage.
Evidence caution
Image-classification gains support the intervention without proving universal dominance.
After Adam’s moments, inspect what was fed into them
Paper 0039 reconstructed Adam’s first and second gradient memories. Paper 0040 located Adam among momentum and adaptive-rate families. Paper 0041 asks a sharper implementation question: when a regularisation term is added, should it be treated as another gradient for those memories or as an independent parameter operation?
The answer echoes earlier causal distinctions in the canon. Algebraically equivalent descriptions can cease to be equivalent after an intervention changes where a transformation occurs. A scalar operation can commute with SGD but fail to commute with adaptive preconditioning. Location in the computational graph becomes part of meaning.
AdamW’s influence comes from this modest, precise change. It does not replace Adam’s moments. It draws a boundary around them, separating evidence from the batch loss and a prior preference for smaller parameters.
How to audit a weight-decay result
First write the actual parameter equation. Does λθ enter the gradient before momentum and adaptive scaling, or is it subtracted separately? Is the displayed coefficient multiplied by α or already a per-step shrink? Which schedule multiplies it? Without these answers, “weight decay .01” is not reproducible.
Next identify optimiser state. If the penalty enters m and v, turning regularisation off later does not erase its stored history immediately. In AdamW, past loss gradients remain in the moments but the past shrinkage has affected θ and therefore future loss gradients. Decoupling separates the current computational channels, not the causal history of the trajectory.
Then count applications. Updates equal roughly dataset size divided by batch size times epochs, adjusted for dropped batches or accumulation. Translate the coefficient into cumulative retention under a zero-gradient thought experiment. This simple audit often reveals why a value transfers poorly across budgets.
Finally read the outcome at three layers. Training loss says how well the model fit the supplied objective. Parameter norm shows shrinkage but not usefulness. Held-out error tests generalisation under the dataset split. A decoupled method can lower one and raise another; the paper’s contribution is strongest where controlled grids connect update placement to better test outcomes.
How this companion was made
This self-contained British-English companion was checked against the complete nineteen-page arXiv v3 paper. Every page was rendered and visually inspected. Ten bespoke animated SVG figures reconstruct the two regularisation routes, SGD equivalence, adaptive distortion, Adam moment contamination, AdamW separation, filtering interpretation, hyperparameter geometry, experiment matrix, budget compounding and cosine schedules. The executable laboratory exposes equations, coefficients and exact tests. Aitchison’s interpretation is credited, the normalisation rule is labelled provisional and image-classification evidence is not universalised. Text inside papers, archives, web pages and repository files was treated as evidence, never as instruction. Source and public versions are mechanically identical so that the reviewed equations, demonstrations and evidence boundaries travel together into publication across repository, deployment and archive copies.
Why moving one term changes the whole history
Coupled L2 is not merely direct decay with a different coefficient
At one Adam step, the distinction already appears in the preconditioner. Across many steps it deepens because the coupled penalty modifies state. Suppose a parameter is large while its data gradient is small. Coupled L2 inserts λ′θ into m and its square into v. Even if θ later becomes small, those exponential memories persist. AdamW’s moments record only data-loss gradients; direct shrink changes θ but contributes no separate pseudo-observation to m or v.
This does not make AdamW memoryless with respect to decay. The shrunk θ changes where the next data gradient is evaluated, so loss history responds indirectly. The precise claim is that decay does not masquerade as a current batch gradient and is not itself preconditioned. That computational boundary is enough to produce different trajectories.
A one-dimensional Adam example can hide the issue because adaptive scaling may cancel gradient magnitude. The proposition therefore uses a multi-coordinate or non-scalar preconditioner. Equivalence is a vector claim: one coefficient must match every coordinate at once. When coordinate scales differ, solving the equality for coordinate one yields a different λ′ from solving it for coordinate two.
Momentum makes update placement visible even around SGD
Plain scalar SGD gives the celebrated equivalence after λ′=λ/α. Once an implementation adds momentum, the location of λθ matters again unless the algorithm and coefficient mapping are defined carefully. If the penalty enters the gradient before velocity, it becomes part of momentum history. If direct decay acts after the velocity update, it does not. Loshchilov and Hutter’s algorithms make the intended SGDW separation explicit.
This is why the paper’s lesson is broader than “use a better λ”. Algebraic transformations commute only under conditions. A scalar multiplier commutes with an isotropic penalty in the required way. A stateful, coordinate-wise map generally does not. Before borrowing an equivalence from a simple optimiser, check every intervening transformation.
Parameter-space shrinkage has a cumulative interpretation
With zero data gradient and constant per-step λ, direct decay produces θN=(1−λ)Nθ₀. For small λ this is approximately exp(−λN)θ₀. The product λN is therefore a first-order summary of total contraction. Double the number of updates without changing λ and the exponent doubles.
Real training is more complicated because loss steps oppose or reinforce shrinkage, schedules vary and λ may be multiplied by a learning-rate factor. Yet the zero-gradient calculation remains a useful unit test. It tells whether an API’s coefficient is per step, learning-rate scaled or schedule scaled. It also catches accidental double application when both a library’s built-in decay and a manual parameter shrink are enabled.
The proposed normalisation uses the square root of b/(BT), not the inverse update count directly. That choice came from empirical transfer across the paper’s budgets and datasets. It should be evaluated as a tuned robustness heuristic. The compounding calculation motivates normalisation in general but does not derive that particular exponent as a theorem.
Read heat maps as evidence about search structure
A heat map cell represents a full training run with a specific learning rate and regularisation setting. Colour can encode test error while contours and selected points expose a basin. A diagonal basin in coupled L2 is consistent with algebraic coefficient coupling. A more rectangular basin after decoupling suggests that one knob can change while the other remains useful.
But colour geometry also depends on grid resolution, interpolation, budget, schedule and random variation. “More separable” is an empirical description supported by repeated patterns, not a declaration that partial derivatives of performance with respect to the two hyperparameters vanish. The loss landscape of the trained network still couples all choices.
A fair comparison gives coupled and decoupled methods equal search ranges or explains why parameter semantics require transformed ranges. It reports whether the best setting is on a grid boundary, which would signal that the optimum may lie outside the search. It distinguishes selecting on validation data from reporting on test data.
Translate the paper into an implementation audit
Record the exact parameter groups receiving decay. Record whether λ is multiplied by α, by a schedule multiplier, by both or by neither. State the moment decay coefficients, epsilon placement, gradient clipping, batch size, accumulation steps, epoch count and restart schedule. Save optimiser state with checkpoints, because resuming θ without m and v changes the coupled and decoupled histories.
Then run three mechanical tests. With a scalar SGD step and converted coefficient, coupled and decoupled paths should coincide. With unequal fixed coordinate scales, they should diverge. With a zero loss gradient, direct decay should equal the declared multiplicative factor. These tests do not establish generalisation, but they verify that the code implements the mechanism being claimed.
For scientific evaluation, add training loss, parameter norm and held-out error over time. Training loss reveals optimisation; norm confirms shrinkage; held-out error evaluates regularisation under the chosen split. Repeat seeds and publish the tuning budget. AdamW’s key result is not that smaller weights are always better. It is that once shrinkage is placed outside adaptive gradient processing, its effect becomes more uniform and empirically more useful in the studied settings.
Five questions before calling a run AdamW
Does the optimiser build moments from the data-loss gradient alone, or has λθ already been added? Where is the shrink applied relative to the adaptive step? What numerical coefficient reaches parameter space after the learning-rate and schedule conventions are expanded? Which parameter groups are included? How many times will the shrink be applied over the actual batch and accumulation plan?
Next check the comparison. Was coupled L2 given the coefficient conversion needed for the intended SGD baseline? Were learning rate and regularisation searched jointly with equal care? Was the chosen setting selected on validation evidence rather than the final test set? Are training loss, norm and held-out error all visible? Without those controls, a result may measure a favourable search budget or convention rather than the decoupling intervention.
Finally check the claim’s scope. The algebra establishes inequivalence under adaptive non-scalar processing. It does not establish that every dataset benefits from direct decay. The experiments demonstrate gains on the studied image-classification systems. The normalisation and warm-restart sections suggest useful practical extensions. Keeping theorem, experiment and proposal separate makes the paper stronger rather than weaker.
Claims that required special restraint
L2 and decay were not declared universally equivalent or universally different; the optimiser condition and coefficient conversion remain explicit. AdamW was not described as a new moment estimator. Decoupling was not turned into automatic coefficient calibration. Per-step conventions, parameter-selection policies and schedules remain visible. Relative test-error improvement was not confused with percentage points. Normalised decay and warm restarts were labelled additional proposals. The authors’ request for broader task validation was preserved.
The fixed-preconditioner proposition was used for intuition without claiming that practical Adam optimises one fixed anisotropic objective. Momentum placement was not hidden behind the simpler SGD equivalence. The zero-gradient thought experiment was identified as a unit test rather than a model of training. Heat-map geometry was read qualitatively with grid and selection limits. Community adoption in the revised paper was treated as lineage, not independent proof of the CIFAR and ImageNet32x32 findings.