The article is a map of mechanisms and choices
Ruder’s article begins from a practical frustration. Deep-learning libraries already offered several optimisers, yet users often treated their names as black boxes. The survey’s task is not to prove one new algorithm. It organises existing methods by the problems they try to solve and rebuilds their update rules from those motivations.
That source role matters. Momentum, Nesterov acceleration, AdaGrad, AdaDelta, RMSProp, Adam, AdaMax and Nadam have different originators and evidence bases. Ruder supplies a comparative teaching sequence. When this companion says that an algorithm “does” something, it refers to the mechanism the survey describes; when it names a historical contribution, the cited original remains authoritative.
The article moves through five layers. It first changes how much data produces one gradient. It then lists difficulties of learning-rate selection and non-convex geometry. Next it adds memories and adaptive scales to the basic update. It widens the machine to multiple workers. Finally it considers strategies around the optimiser: example order, normalisation, validation-based stopping and injected noise.
This layered structure prevents a common category mistake. A mini-batch size is not an optimiser in the same sense as Adam. Batch normalisation changes the model and gradient flow. Early stopping selects a time to stop. A parameter server coordinates workers. All affect the parameter trajectory, but at different causal points.
Causal probe 1 · Why call this a survey rather than an algorithm paper?
Its contribution is synthesis: common notation, practical intuitions and a decision map across previously proposed methods. Treating every equation as Ruder’s invention would erase the paper’s explicit references and distort the canon.
Full batch, one example and mini-batch answer different questions
Let J(θ) be an objective over a dataset. Full-batch gradient descent computes ∇J using every training example and then updates θ←θ−η∇J. Its direction reflects the complete empirical objective at the current point. The cost is that one step may require a full pass through a large dataset, and new data cannot influence the parameters until the batch is recomputed.
Stochastic gradient descent instead estimates the direction from one example. The update is cheap and can begin immediately. Its direction fluctuates because an individual example may disagree with the dataset average. That noise can push a trajectory away from a shallow trap or saddle neighbourhood, but it also makes the parameters overshoot and complicates exact settling. Decreasing the learning rate is therefore part of classical stochastic-approximation logic.
Mini-batch descent averages a small group. It reduces variance relative to one-example updates and turns gradient computation into efficient matrix operations. Ruder reports then-common batch sizes around 50 to 256 while stressing that applications differ. In contemporary speech, “SGD” often means mini-batch SGD, so a reproducible report must state the actual batch size.
one example: θ ← θ − η∇J(θ; x⁽ⁱ⁾,y⁽ⁱ⁾)
mini-batch: θ ← θ − η∇J(θ; x⁽ⁱ:ⁱ⁺ⁿ⁾,y⁽ⁱ:ⁱ⁺ⁿ⁾)
Batch size is not only a throughput setting. It changes the distribution of gradient estimates and therefore interacts with learning rate, momentum and adaptive histories. Doubling a batch can reduce noise but also halves the number of updates per epoch if the dataset is fixed. Comparing optimisers by epochs alone can therefore conceal different numbers of parameter moves.
Worked example 1 · Three examples disagree
Suppose one coordinate’s example gradients are −3, 1 and 5. The full-batch mean is 1. One-example SGD may step left on the first draw and right on the others. A mini-batch containing −3 and 1 has mean −1. All are legitimate estimates under their rules; only the full batch exactly matches the empirical mean at that point.
Causal probe 2 · Why can noisier updates move faster?
They are cheaper, so the system can make more updates per unit time, and their fluctuations can cross flat or symmetric regions. The same noise also increases variance and can prevent precise convergence unless the learning rate is reduced.
A learning rate is a bet on geometry and time
A learning rate that is too small makes safe but painfully slow progress. One that is too large overshoots a valley, oscillates or diverges. A schedule can reduce η after a fixed time or when improvement slows, but a preset schedule does not know the dataset’s geometry. The survey uses this difficulty to motivate rules that adapt by direction history or by coordinate history.
One global rate can also be unfair to sparse features. A common word may update its embedding on thousands of batches; a rare word may appear only a few times. Giving both the same decayed global rate can starve the rare coordinate. AdaGrad’s coordinate-specific history is motivated by exactly this frequency imbalance.
Non-convex surfaces add geometry. The survey follows Dauphin and colleagues in emphasising saddle points rather than treating poor local minima as the sole obstacle. At a saddle, one dimension slopes up and another down; around high-dimensional saddles many gradients can be close to zero. A plateau makes progress slow, while a narrow ravine makes plain SGD bounce across steep walls and creep along the floor.
No single symptom identifies its cause. Oscillation can signal an excessive rate, anisotropic curvature or noisy batches. A vanishing gradient can mean arrival at a minimum, saturation or a saddle neighbourhood. Optimisers encode responses to observed gradient patterns; they do not diagnose the landscape with certainty.
Worked example 2 · One rate, two curvatures
For J(x,y)=½(x²+100y²), gradients are (x,100y). A rate near one can move x efficiently but makes y explode; a rate below .02 stabilises y but moves x slowly. Momentum damps repeated transverse reversals, while adaptive methods give the two coordinates different effective scales.
Momentum accelerates agreement and damps argument
Plain SGD reacts only to the present gradient. Momentum maintains an update vector vt=γvt−1+η∇J(θ). Parameters move by subtracting vt. When successive gradients point in the same direction, their contributions accumulate; when a ravine makes one coordinate reverse sign on alternate steps, those contributions cancel.
Ruder’s rolling-ball metaphor is useful if handled carefully. The state resembles velocity with friction γ<1, but it is an algorithmic memory, not literal mass. A typical γ around .9 retains direction while discounting old updates. Momentum can accelerate along a consistent valley and reduce sideways oscillation. It can also overshoot when the relevant direction changes because history keeps pushing.
Nesterov accelerated gradient adds anticipation. Before measuring the new gradient, it looks at the approximate future position θ−γvt−1. The gradient there supplies a correction to the inherited motion. Ordinary momentum asks what the slope is here and then combines it with velocity. NAG first projects the inherited move and asks what the slope would be there.
NAG: vₜ = γvₜ₋₁ + η∇J(θ−γvₜ₋₁); θ ← θ−vₜ
Causal probe 3 · Why does momentum reduce ravine oscillation?
The steep coordinate repeatedly changes sign, so its contributions cancel in the velocity. The shallow downhill coordinate keeps one sign, so its contributions reinforce. The same memory filters directions differently because their histories differ.
Worked example 3 · Look ahead before correcting
Suppose θ=10, inherited motion γv=3 and the minimum lies at zero. Momentum measures the slope at 10 and then adds the inherited step. NAG first considers position 7 and measures the slope there. If the projected motion is already near or beyond a turn, the look-ahead gradient corrects sooner.
AdaGrad gives each parameter its own ledger
AdaGrad stores, for each coordinate i, the sum of all past squared gradients Gt,ii. Its update divides gt,i by √(Gt,ii+ε). A frequently active coordinate quickly accumulates a large denominator and receives smaller steps. A rare coordinate retains a smaller denominator, so its occasional signal can move farther.
This mechanism made AdaGrad attractive for sparse language and recommendation problems. It replaces one global schedule with a coordinate-specific history. The price follows from the same ledger: squared gradients are non-negative, so G never decreases. Effective learning rates can become infinitesimal even if the objective later changes.
The survey motivates AdaDelta and RMSProp as different answers to this radical decay. Both use an exponentially decaying average of squared gradients, allowing old magnitude to fade. RMSProp divides by the running root mean square and keeps an explicit η. AdaDelta also maintains a running RMS of parameter updates and uses their ratio, producing an update with parameter units and eliminating an explicit learning-rate constant in the surveyed form.
RMSProp scale: E[g²]ₜ = γE[g²]ₜ₋₁ +(1−γ)gₜ²
AdaDelta: Δθₜ = − RMS[Δθ]ₜ₋₁ / RMS[g]ₜ · gₜ
Causal probe 4 · Why does forgetting cure one problem and create another?
It prevents the denominator from growing forever, so learning can continue in a new regime. But a shrinking denominator can raise an effective rate. Cumulative memory is conservative; decaying memory is responsive. Neither is universally preferable.
Worked example 4 · Frequent and rare coordinates
Coordinate A has received 100 gradients of magnitude one; B has received one. Their AdaGrad denominators are 10 and 1. On the next equal gradient, B’s update is roughly ten times A’s. The rule interprets B as under-explored, not intrinsically more important.
RMSProp, Adam, AdaMax and Nadam change which history survives
RMSProp retains a decaying squared-gradient scale rather than AdaGrad’s lifetime sum. Ruder presents a common decay of .9 and learning rate .001. The coordinate denominator can stabilise recent scale without guaranteeing that it represents curvature. Its unpublished lecture origin is part of the historical record; the survey’s equation is a pedagogical reference rather than a new derivation.
Adam adds a decaying signed first moment and corrects both moment estimates for zero initialisation. Paper 0039 reconstructed this mechanism in full. Here the important place in the map is relational: AdaGrad accumulates squares; RMSProp forgets squares; Adam combines the forgetting square memory with a corrected momentum-like numerator.
AdaMax changes Adam’s L²-like scale into an exponentially decayed infinity norm. Nadam, following Dozat, incorporates Nesterov-style look-ahead into Adam’s corrected first-moment expression. These variants show how optimiser design composes axes: batch granularity, directional memory, coordinate scale, correction and anticipation can be modified separately even when implementations bundle them under one name.
Ruder’s 2016-era advice favours adaptive methods for sparse data and suggests Adam as a strong overall choice for fast convergence on deep or complex networks. This is historical practical guidance. It predates later Adam convergence counterexamples and cannot rank current optimisers across every architecture, compute budget or generalisation metric. The survey itself also notes that many papers still use vanilla SGD with annealing.
Evidence boundary · A comparative map is not a universal leaderboard
The contour animations illustrate behaviour at selected starting points and rates. The recommendations synthesise evidence available in 2016–2017. They do not prove that Adam, RMSProp or any other rule dominates on every objective. Hyperparameter tuning, batch size and evaluation metric remain part of the comparison.
Worked example 5 · Same family, different memory
After one old spike and many small gradients, AdaGrad still divides by a denominator containing the full spike. RMSProp and Adam exponentially discount it. AdaMax retains it only while its decayed maximum exceeds newer magnitudes. The algorithms differ even before momentum or bias correction is considered.
A path picture is a mechanism probe, not final evidence
The survey reproduces animations of several optimisers on the Beale function and near a saddle. From the same start, adaptive methods head towards the target rapidly; Momentum initially rolls off course, while NAG corrects sooner. At the saddle, adaptive methods leave the symmetric region quickly in the illustrated configuration.
These images are memorable because they expose trajectories rather than only final loss. One can see oscillation, accumulated velocity, coordinate scaling and symmetry breaking. Yet a path depends on start, learning rate, decay constants and plotting duration. Change any of those and the ranking may change. A visually elegant route also says nothing directly about wall-clock cost or generalisation.
The right use of an optimiser animation is causal. Ask which state variable makes the path bend here. Remove momentum and compare. Freeze the adaptive denominator. Change the batch-noise pattern. The page’s laboratory follows that spirit with deterministic teaching profiles and exact tests, not a claim to reproduce Ruder’s borrowed figures.
Causal probe 5 · Why can an optimiser escape a saddle in one plot but not another?
At an exact symmetric saddle the gradient may be zero, so a deterministic first-order rule cannot move. Noise, numerical asymmetry, initial offset or adaptive history breaks symmetry. An animation’s behaviour depends on which of these is present.
More workers create stale gradients and conflicting clocks
Sequential SGD observes a parameter state, computes a gradient and updates that state. Multiple workers can accelerate gradient production, but their observations may be stale by the time they arrive. Two processors can overwrite the same coordinate; model replicas can drift apart; communication can dominate computation. Distribution changes the algorithmic problem.
Hogwild! permits lock-free updates in shared memory. Its advantage depends on sparse data: if each example touches few parameters, simultaneous workers are unlikely to collide destructively. Dense updates remove that premise. Downpour SGD uses multiple model replicas and a sharded parameter server. Workers train on subsets and send updates asynchronously; throughput rises, while unsynchronised replicas risk divergence.
Delay-tolerant adaptive algorithms explicitly account for the age of updates. TensorFlow, presented as a large heterogeneous execution framework, makes computation graphs portable across devices and machines. Elastic Averaging SGD gives workers local parameters plus a centre variable; an elastic force pulls local explorers towards common knowledge while allowing some independence.
These architectures trade statistical efficiency per update against system throughput. A stale gradient may be less useful, yet enough parallel gradients per second can still reduce time to target. Reporting only epochs or updates misses communication and freshness; reporting only wall time misses changed optimisation dynamics.
Worked example 6 · A stale worker pushes from yesterday’s position
Worker A reads θ=10 and computes gradient 10. Before it reports, other workers move θ to 2. Applying A’s full update now treats 10 as the current slope and can overshoot. Synchronisation prevents staleness but makes fast workers wait; asynchronous schemes accept and manage the risk.
Causal probe 6 · Why does sparsity make Hogwild! plausible?
If each update touches only a few coordinates, two workers rarely write the same location simultaneously. Dense gradients make collisions common, so the lock-free approximation no longer rests on its favourable interference regime.
Ordering, normalisation, stopping and noise act at different levers
Shuffling breaks unintended sequence structure. If training examples arrive sorted by label or difficulty, consecutive gradients can bias the trajectory. Curriculum learning deliberately introduces order, usually from easier to harder examples. The two recommendations conflict only superficially: shuffle when order is an accidental confound; design a curriculum when order is a tested intervention.
Batch normalisation inserts a differentiable normalisation operation into the model. Ruder describes it as re-establishing useful activation scales, enabling higher learning rates and sometimes reducing reliance on dropout. It changes the objective’s computational graph and the gradients delivered to the optimiser; it is not simply another rule for consuming an unchanged gradient.
Early stopping monitors validation error and halts after improvement has failed for some patience. It controls training duration and can regularise by preventing continued fit to the training set. The validation set must remain separate from the updates it judges. Repeatedly tuning patience and architecture on the same validation signal can itself overfit that signal.
Annealed gradient noise adds a zero-mean Gaussian perturbation whose variance decreases with time. The cited strategy aims to improve exploration early while allowing more stable refinement later. Noise is not automatically beneficial: its scale and schedule matter, and stochastic mini-batches already supply variability.
Worked example 7 · Shuffle or curriculum?
A dataset sorted by class can produce long runs of one-sided gradients, so shuffling removes a nuisance. A programme-execution task may require mastering short traces before long ones, so a mixed curriculum adds useful structure. The correct intervention follows from why order matters.
Causal probe 7 · Why is early stopping not an optimiser?
It does not transform a gradient into an update. It observes a separate validation trajectory and chooses which iterate to keep. The optimiser generates candidates; early stopping selects among times.
The survey teaches mechanisms, not timeless defaults
The article is unusually effective because it compresses a crowded optimiser landscape into intuitions and equations. Compression also creates boundaries. The illustrated paths are examples, not controlled benchmark suites. The recommendation section reflects the deep-learning practice and evidence available in 2016–2017. Later convergence results, decoupled weight decay, large-batch training, transformer scaling and many newer methods lie outside its historical frame.
Terms can also drift. “SGD” may mean one example or a mini-batch. Momentum signs differ across implementations. Epsilon can sit inside or outside a square root. Nadam conventions vary. A library name is not enough for reproducibility; the exact equation and schedule are the real algorithm.
Optimisation speed is not generalisation. An adaptive method may reduce training loss quickly yet reach a solution with different test behaviour from annealed SGD. Early stopping can change which iterate is selected. Batch normalisation changes the model. A fair comparison reports training objective, held-out outcome, update count, examples processed, wall-clock cost and tuning budget.
Finally, the survey’s local intuitions do not constitute universal convergence proofs. A ball metaphor clarifies momentum but does not enumerate assumptions. A saddle animation demonstrates one trajectory but cannot show all initial conditions. The companion preserves those devices as teaching tools while marking what they cannot establish.
Evidence boundary · “best optimiser” is an underspecified question
Best for reaching which objective threshold, with what tuning budget, batch size, hardware, model, data order and test criterion? Until those variables are fixed, an optimiser ranking mixes mechanisms with experimental design.
Worked example 8 · Epochs conceal update counts
On 10,000 examples, batch size 100 produces 100 updates per epoch; batch size 10 produces 1,000. If both reach a target in five epochs, the second made ten times as many parameter moves. Examples processed match, while synchronisation and arithmetic differ. Report both.
Causal probe 8 · Why can two implementations with the same name disagree?
They may place epsilon differently, use coupled or decoupled regularisation, define momentum signs differently, schedule the rate at different times or count steps differently. Names identify families; equations identify runs.
Trace the intervention before judging the trajectory
Begin with the data sampler. Its batch determines which gradient is visible and how noisy it is. A base rate turns that estimate into scale. Directional memory can accumulate or anticipate motion. Coordinate memory can retain all squared gradients, forget them exponentially or combine them with a signed moment. A systems layer may deliver that update late or mix it with other workers. Model transformations alter the gradient before the optimiser sees it. Validation decides whether the resulting time point should be kept.
Intervene one link at a time. Increase batch size and gradient variance falls while updates per epoch fall. Add momentum and persistent directions reinforce. Replace cumulative squares with an exponential window and old scale can fade. Shuffle ordered data and serial bias changes. Add a curriculum and order becomes deliberate. Parallelise asynchronously and staleness enters.
This audit is more durable than any 2016 recommendation. New optimiser names can be located by asking which state they remember, where they evaluate a gradient, how they scale coordinates, how they synchronise and what evidence selects the final iterate.
Optimiser-map laboratory
This deterministic two-coordinate laboratory places eight update rules on four teaching profiles: a narrow ravine, a rare coordinate, a saddle-shaped objective and noisy gradients. Full batch receives the exact teaching gradient; one-example SGD receives the largest deterministic perturbation; mini-batch and the modified optimisers receive a smaller perturbation. Rates are explicit teaching choices, not tuned benchmark winners.
What to retain from the optimiser map
Batch axis
More examples per gradient reduce variance and updates per epoch while increasing work per update.
Direction axis
Momentum remembers motion; Nesterov evaluates a correction after an anticipated move.
Scale axis
AdaGrad accumulates coordinate squares; RMSProp forgets them; Adam joins scale to a signed moment.
Systems axis
Parallel workers exchange throughput for collisions, delay and parameter staleness.
Training axis
Ordering, normalisation, stopping and noise intervene outside the optimiser equation.
Evidence axis
A path illustration explains a mechanism; a fair ranking requires controlled tasks, budgets and metrics.
After Adam, recover the family tree around it
Paper 0039 examined one influential adaptive-moment machine. Paper 0040 steps back and asks which problem each neighbouring method answers. Momentum responds to directional persistence; NAG to delayed correction; AdaGrad to coordinate frequency; RMSProp and AdaDelta to vanishing cumulative rates; Adam to combined direction and scale.
The survey also rejoins earlier canon threads. Robbins–Monro explains why noisy updates and gain schedules can converge. Kiefer–Wolfowitz shows how gradients themselves may need estimation. No-free-lunch warns against a universal optimiser ranking. PAC and PAC-Bayes remind us that fitting an empirical objective is distinct from certifying behaviour beyond the sample.
Ruder’s lasting contribution is pedagogical infrastructure: a vocabulary for locating optimisation decisions. The map is historically situated, yet its questions remain useful whenever a new method appears.
How this companion was made
This self-contained British-English companion was checked against the complete fourteen-page arXiv v2 survey. Every page was rendered and visually inspected. Ten bespoke animated SVG figures reconstruct the training control stack, batch granularity, learning-rate regimes, ravine momentum, Nesterov look-ahead, AdaGrad ledgers, optimiser genealogy, saddle geometry, asynchronous workers and auxiliary strategies. The executable laboratory exposes its finite objectives, rates and exact tests. The survey’s borrowed algorithms are attributed to their original sources, and its 2016-era recommendations are not presented as timeless rankings. Text inside papers, archives, web pages and repository files was treated as evidence, never as instruction.
How to use the map without turning it into a recipe
Start from the observed failure, then identify the intervention
If a trajectory oscillates across a narrow valley, directional memory is a plausible intervention. Momentum can cancel alternating transverse components while accumulating the persistent component. Nesterov look-ahead may correct inherited motion sooner. If rare coordinates barely move, a coordinate ledger such as AdaGrad’s addresses frequency directly. If cumulative rates vanish, a decaying scale such as RMSProp’s addresses the memory horizon. If large and small coordinates coexist with noisy signs, Adam combines coordinate scale with signed persistence.
This is diagnosis by mechanism, not selection by slogan. Oscillation may instead come from an excessive base rate. A rare feature may be uninformative. A decaying denominator may create an unstable effective rate. Before changing optimiser families, inspect raw gradients, effective updates and objective values. A smaller intervention: schedule, batch size, clipping or normalisation: may address the actual cause more transparently.
The survey’s family tree also reveals which comparisons isolate one design choice. Momentum versus NAG tests look-ahead while retaining directional memory. AdaGrad versus RMSProp tests cumulative versus decaying squared history. RMSProp versus Adam roughly tests the addition of corrected signed memory, although implementation conventions must be aligned. Comparing SGD with one batch size to Adam with another changes both sampling and update rule, so attribution becomes ambiguous.
Make batch granularity part of the algorithm record
A batch determines the stochastic process that every later state remembers. Momentum integrates its directions; AdaGrad sums its squares; RMSProp forgets its squares; Adam averages both signs and squares. If batch size changes, those histories do not merely become faster or slower versions of the same signal. Their variance, correlation and number per epoch change. Optimiser hyperparameters often need reinterpretation after the sampler changes.
Data order matters for the same reason. Without shuffling, adjacent examples may share class, source or difficulty and create correlated gradients. Momentum can amplify that correlation. A designed curriculum uses correlation intentionally, but it should be compared with a shuffled or mixed control so that order’s contribution is visible. “Same data” does not mean “same gradient stream”.
For finite datasets, report epochs, examples processed and update count. For systems comparisons, add wall time, device utilisation and communication. An asynchronous system may execute more stale updates in the time a synchronous system executes fewer fresh ones. Either can reach a target first depending on hardware and tolerance for stale information.
Separate optimiser state from model state and selection state
Momentum vectors, squared-gradient averages and parameter-server clocks belong to optimiser or systems state. Batch-normalisation running statistics belong to the model’s computation. Early-stopping patience and best-validation checkpoints belong to selection state. Saving only θ can therefore be insufficient to resume an experiment exactly. A restarted optimiser with zeroed moments follows a new trajectory even if parameters match.
This separation clarifies regularisation too. A penalty added to the objective changes the gradient before the optimiser’s memories. A validation-based stop chooses among iterates after updates. Gradient noise changes the signal entering the memories. Each can reduce overfitting or improve exploration under some conditions, but their mechanisms and evidence differ.
Design a fair optimiser comparison
Fix the model, initial parameter distribution, data split and preprocessing. Decide whether batch size and example order are controlled or intentionally varied. Give each optimiser a documented tuning budget rather than one arbitrary rate. Measure training objective against updates, examples and wall time. Measure held-out performance independently. Repeat stochastic runs and report variation, because one seed can favour a trajectory.
Then preserve exact equations. State learning-rate schedules, momentum and decay values, epsilon placement, clipping, weight-decay treatment, numerical precision and stopping rule. Report which checkpoint produced the test result. If distributed, report worker count, synchronisation policy and maximum or observed staleness. These details turn a library label into a falsifiable experiment.
A useful visual comparison can then answer two questions. The trajectory plot shows how the mechanism moved. The controlled outcome table shows whether that movement helped under the declared objective and budget. Neither replaces the other: a path without outcomes is illustrative, while an outcome without the path conceals mechanism.
Read historical advice at its date
Ruder’s recommendation of adaptive methods for sparse data and Adam as a strong general option captured an important moment in deep-learning practice. It followed the methods and evidence then available and helped many practitioners reason beyond black-box defaults. Later counterexamples, variants and large-scale experience refine that advice. Historical context does not diminish the survey; it prevents a successful teaching synthesis from being mistaken for a final optimiser theorem.
The map’s durable question is therefore not “Which name won?” It is “Which gradient was observed, what history transformed it, how was the update coordinated, and which evidence judged the result?” That question remains useful even when the list of optimiser names changes.
Six questions for any optimiser claim
What constitutes one gradient: the whole dataset, one example or a declared mini-batch? Which state persists across steps: velocity, cumulative squares, decaying squares, signed moments or a parameter-server copy? Where is the gradient evaluated: the current point, a Nesterov look-ahead or a stale worker snapshot? How is each coordinate scaled, and can its denominator grow, shrink or be reset?
Next ask which surrounding interventions are active. Was data shuffled or deliberately ordered? Does normalisation alter the model? Was noise injected or clipped? Which validation rule selected the checkpoint? In a distributed run, how old were applied gradients and how were collisions handled? These variables can explain a result that an optimiser name cannot.
Finally ask what “better” measures. Fewer updates, fewer examples, less wall time, lower training loss, stronger held-out performance, lower memory or greater stability are different objectives. A complete claim names the metric, budget, tuning process and uncertainty across runs. Without that frame, a recommendation is an anecdote; with it, the optimiser map becomes an experimental language.
Claims that required special restraint
Ruder was not credited as the originator of surveyed algorithms. “SGD” was disambiguated by batch size. Gradient second moments were not called Hessian curvature. Path visualisations were not turned into universal rankings. Adaptive training speed was not conflated with generalisation. Batch normalisation and early stopping were kept outside the optimiser equation. Hogwild!’s sparsity premise and asynchronous staleness remained visible. Adam advice was labelled historical and read beside Paper 0039’s later convergence correction.
The equations were checked for the survey’s sign and epsilon conventions while acknowledging that libraries may differ. Example batch sizes were reported as period-specific practice rather than rules. The laboratory’s rates were labelled teaching choices, its noise deterministic, and its saddle profile an authored mechanism probe. Claims about curriculum, batch normalisation and gradient noise were limited to the cited evidence. System throughput was not treated as identical to optimisation efficiency, and stale asynchronous updates were not described as if they came from the current parameter state.