The map changes with the coordinates

A physical hill has a height that does not depend on whether a surveyor measures east-west distance in metres or feet. A parameter loss surface lacks that privilege. The vertical quantity is fixed only after choosing a loss and data sample. The horizontal geometry comes from a parametrisation and a metric. Change either, and Euclidean distance, gradient components and Hessian eigenvalues change.

Worked example: one predictor, a family of minima

Take the smallest deep linear model that can expose the problem. One input passes through two scalar weights:

Here x is fixed at one, the target is one, and a and b are the two weights. Every point satisfying ab = 1 predicts exactly the same value and has zero loss. The balanced point is a = b = 1. A rescaled point is a = c, b = 1/c, for any positive c.

At one of these minima, the Hessian is:

The zero eigenvalue follows the curve of equivalent factorizations. The other is c² + c⁻². It equals 2 at c = 1, about 100.01 at c = 10, and grows without bound as the internal scale becomes unbalanced. Prediction, loss and function are unchanged. Euclidean sharpness is not.

Dinh and colleagues established a broader version of this objection for rectified networks: parameter rescalings can produce observationally equivalent minima with arbitrarily different Hessian sharpness under common definitions. Their result does not make curvature useless. It makes unqualified curvature claims ill-typed. A metric or normalisation must say which parameter changes count as comparable. [1]

Thought experiment: robustness to which perturbation?

Return to the equivalent pair (a,b)=(1,1) and (10,0.1). Add an isotropic Gaussian perturbation with standard deviation 0.01 to each stored parameter. The two perturbation distributions are identical in coordinate units, yet they are not equivalent interventions on the represented function. At the rescaled point, a small change to b is multiplied by a large a; at the balanced point, the same coordinate disturbance has a different functional effect. Calling one solution less robust silently privileges the storage coordinates.

A useful flatness measure therefore begins by declaring a perturbation distribution q and the consequence being protected:

If q models an actual failure mechanism, such as weight quantisation, analogue device noise or a bounded update error, the quantity has operational meaning. If q is an arbitrary Euclidean ball, the result is a property of that ball and parametrisation. Function-space perturbations, path-normalised perturbations or metrics induced by output distributions can reduce some coordinate dependence, but each introduces its own modelling choice.

Flatness becomes informative when the perturbation has a named cause and a named protected outcome. It does not become an intrinsic certificate merely because it is averaged, normalised or plotted smoothly. This is the bridge missing from many casual statements that a wide minimum “must” generalise.

What survives a change of map

The objective value at a represented function survives an invertible change of coordinates. A Euclidean gradient vector does not; it transforms with the Jacobian. At a critical point, the Hessian transforms as JᵀHJ. A step of length 0.01 therefore has no coordinate-free behavioural meaning. This is why natural-gradient methods begin from a metric over distributions or model behaviour, and why filter-wise normalisation was introduced for comparative landscape plots. Li and colleagues showed that raw random-direction plots can reverse apparent sharpness when weight scales differ, while normalised directions make the chosen comparison more stable. [2]

The mountain analogy still earns a narrow role. Once coordinates, metric and objective are frozen, contours can reveal anisotropy, negative curvature and locally safe step scales. The analogy fails when a visual property of that chart is promoted into an intrinsic property of the learned function.

flowchart TD
  A["The valley that moved while<br/>the model stayed still"] --> B["The map changes with the<br/>coordinates"]
  B --> C["Four local forces shape a<br/>trajectory"]
  C --> D["What a landscape plot can<br/>establish"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#B96D00,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Four local forces shape a trajectory

Near a parameter state θ, a twice-differentiable loss can be approximated by:

Here g is the gradient, H the Hessian and Δ a small displacement. The gradient gives first-order change. Hessian eigenvectors identify locally independent curvature directions; eigenvalues give signed curvature along them. Positive values bend upward, negative values expose a locally descending direction at a stationary point, and values near zero indicate weak second-order information. This approximation is a microscope, not a route map.

Curvature is a spectrum, not a badge

Write a displacement in the Hessian eigenbasis as Δ = Σ_i z_iv_i. The quadratic term becomes:

That expansion explains why no single curvature statistic can answer every question. The largest eigenvalue gives the worst local upward bend and helps bound a simple gradient-descent step. The most negative eigenvalue detects the strongest local descent direction at a stationary point. The trace aggregates signed curvature and can hide a small set of negative directions inside a positive majority. A condition number compares sharp and shallow positive directions, but becomes unstable when eigenvalues are zero or negative. The spectral bulk and isolated outliers describe a different structure again.

Near-zero eigenvalues deserve special caution. They may represent a genuine broad set of low-loss functions, a symmetry that changes parameters without changing predictions, an unused unit, an underdetermined data direction, or a point where second derivatives are simply uninformative. Counting them as evidence of a “flat basin” conflates distinct mechanisms. The two-weight example has an exact zero eigenvalue because the factorisation is non-identifiable, not because the predictor has learned a robust semantic rule.

The matrix itself must also be named. The true loss Hessian includes terms that can create negative curvature. A generalised Gauss-Newton matrix or Fisher information matrix is positive semi-definite under common constructions and answers a different local sensitivity question. Substituting one for another may be computationally sensible, but it changes the claim. A curvature report should name the matrix, spectrum statistic, estimation method, data split and parameter metric.

Curvature turns a step size into a stability claim

For a one-dimensional quadratic with curvature λ, gradient descent multiplies the distance from the minimum by 1 - ηλ each step. The dimensionless product ηλ matters. Values between zero and two contract; values above two diverge. In many dimensions, the largest positive eigenvalue constrains the safe step while small positive eigenvalues set the slow directions. A long narrow ravine therefore produces zigzagging: the sharp wall limits the step, while the shallow floor controls the wait.

Real neural training can depart from this monotone quadratic regime. Cohen and colleagues observed full-batch gradient descent across several studied architectures and tasks enter an “edge of stability”: the largest Hessian eigenvalue rose towards roughly 2/η, short-term loss oscillated, and long-term loss continued to fall. They explicitly restricted the claim to the studied full-batch setting and did not present the eigenvalue as a generalisation measure. [7] The useful lesson is precise: a learning rate and a curvature measurement form one coupled operating condition.

Step scale, batch scale and noise are not one temperature

It is tempting to compress learning rate and batch size into one “noise temperature”. That shorthand can be useful inside a stated diffusion approximation, but it drops mechanisms that matter in actual training. Mini-batch noise is state-dependent and anisotropic. Sampling without replacement correlates updates. Momentum filters noise across time. Adaptive preconditioning changes its direction and scale coordinate by coordinate. Clipping truncates large events. A learning-rate schedule changes both deterministic motion and the effective perturbation.

Keskar and colleagues reported a generalisation gap between large- and small-batch training in their experiments and associated it with sharp and flat minima. [13] The result was influential, but it does not license “small batch is better” as a universal mechanism. Weight scale, number of updates, learning-rate tuning and normalisation can alter the comparison. Jastrzębski and colleagues later measured how the sharpest directions visited by stochastic gradient descent depended jointly on learning rate and batch size, and used directional interventions to argue that motion along the top Hessian eigendirection affects the sharpness encountered. [11]

A discriminating experiment varies batch size while matching processed examples and tuning effort, then separately varies the learning rate to preserve or deliberately change the stability scale. It measures the covariance projected onto selected Hessian subspaces rather than relying only on batch count. The conclusion can then say whether altered noise geometry, altered deterministic step scale, or their interaction best predicts the trajectory.

Thought experiment: the stationary point that is not a destination

Place gradient descent exactly at the origin of f(x,y)=½x²-½y²+¼y⁴. The gradient is zero, so deterministic descent never moves. Now change only the initial y coordinate from zero to 0.001. Negative curvature amplifies the displacement and the trajectory moves towards one of two lower minima. Repeat with g(x,y)=½x²+¼y⁴. The gradient is equally tiny, but the Hessian has no negative eigenvalue; the same perturbation slowly returns towards the origin.

This negative control matters. A stalled loss curve, small gradient norm and nearly flat slice are compatible with a saddle, a shallow minimum, numerical underflow, a symmetry direction or a poor scale. Dauphin and colleagues made the influential case that saddles and plateaux can dominate high-dimensional non-convex difficulty, drawing on random-function models and early neural experiments. [4] Later theory showed that, under smoothness and Hessian-Lipschitz assumptions, perturbed gradient descent can reach an approximate second-order stationary point with only polylogarithmic dimension overhead relative to first-order stationarity. [5] Those are conditional results, not proof that every neural plateau is a strict saddle.

Second-order stationarity is not a success criterion

An approximate second-order stationary point has a small gradient and no substantially negative Hessian eigenvalue under a chosen tolerance. It may still have high loss, poor test behaviour, unacceptable calibration or the wrong business consequences. Conversely, a trajectory can deliver a useful model without ever satisfying a strict local stationarity test because training stops early, the data stream changes or regularisation remains active.

This boundary prevents an optimisation explanation from swallowing the learning problem. Escaping a saddle establishes local progress on the specified objective. It does not establish that the objective represents the desired prediction, decision or social outcome. The companion article Optimisation, Approximation and Guarantees develops that separate objective-to-decision bridge. Here, the narrower point is that even perfect local geometric diagnosis remains inside the optimisation layer.

Noise has direction, not just magnitude

A mini-batch gradient can be written as the full-data gradient plus an error. Its covariance is:

The matrix Σ records where stochastic updates vary. Two batches with the same scalar variance can inject noise into different eigenspaces. In selected models, stochastic-gradient variance can align with negative curvature and aid saddle escape; that is a structural assumption, not a generic property of adding Gaussian noise. [12] Ghorbani and colleagues measured Hessian spectra at neural-network scale and found, in their studied non-batch-normalised networks, isolated high-curvature directions accompanied by strong gradient concentration; batch normalisation substantially altered both patterns. They also showed that the largest eigenvalue alone did not explain the full optimisation difficulty. [6]

Noise, curvature and preconditioning must be inspected in a common basis. Batch size changes the covariance, learning rate scales its effect, momentum correlates it through time, and adaptive methods rescale coordinates. “SGD is noisy” is therefore less informative than a report of which directions receive useful signal, destabilising variance or almost nothing.

What a landscape plot can establish

A two-dimensional surface is a restriction of a high-dimensional objective to a selected plane:

The centre θ*, directions δ and η, their normalisation, the plotted radius and the data used to evaluate loss define the picture. It is evidence about that plane. It is not a photograph of the full parameter space.

Thought experiment: two directions out of a million

Suppose the Hessian has 999,999 positive eigenvalues and one negative eigenvalue. A random two-dimensional plane is unlikely to align strongly with the exceptional direction. The slice can look like a bowl even though the centre is a saddle. Select the most positive and most negative Hessian eigenvectors instead, and the saddle appears immediately. Nothing about the objective changed. Only the observation directions did.

Böttcher and Wheeler formalised this failure for random projections. In one constructed high-dimensional example, only about 0.5 per cent of 10,000 random two-dimensional projections displayed the underlying saddle. That percentage belongs to their example, not to neural networks generally. The mechanism is the point: expected projected curvature is dominated by aggregate curvature and can erase a minority of negative directions. [3]

Three legitimate uses of a slice

First, a normalised local slice can compare nearby sensitivity under a fixed measurement protocol. Second, a trajectory plane can show whether a particular optimiser crossed a barrier between recorded states. Third, a Hessian-aligned slice can display selected extreme curvature directions. Each claim should name its construction and radius.

Problems begin when absence is interpreted as proof. A smooth random slice cannot establish global convexity. A barrier on the straight segment between two solutions cannot establish disconnected basins. A flat-looking neighbourhood cannot establish robustness to functionally meaningful perturbations. A low-loss curve cannot establish that ordinary training would discover or traverse it.

What an honest negative visual result says

Suppose twenty filter-normalised random slices around a checkpoint show no barrier and no visible negative curvature. The defensible result is that none appeared in those twenty sampled planes over the declared radius and data split. It is not evidence that the full neighbourhood is convex. Likewise, if a top-and-bottom Hessian slice reveals a saddle, it establishes signed local curvature at the centre, not that the optimiser encountered that point or that the negative direction remains useful after a large step.

A useful visual record therefore stores the direction vectors or their generation seeds, normalisation rule, centre checkpoint, radius, grid resolution, batch or dataset, model mode, treatment of batch-normalisation statistics and numerical precision. Without that provenance, a landscape image is difficult to reproduce and impossible to compare. The caption is part of the measurement apparatus.

Connectivity is a path claim

Garipov and colleagues trained simple curved paths between independently trained network solutions on several image-classification settings, keeping loss or accuracy nearly constant along the path even when the direct line crossed a high-loss barrier. [8] The result changed the useful picture from isolated wells to connected low-loss regions. It did not prove universal connectivity, nor did it make straight-line interpolation irrelevant. It showed that connectivity depends on the admissible path family and on active optimisation of that path.

A related caution comes from replication. Goodfellow, Vinyals and Saxe reported simple loss behaviour along the line from initialisation to the trained solution in networks of their period. [10] Frankle revisited the experiment on VGG and ResNet settings and found that larger models often stayed at high loss until near the endpoint, while barriers emerged soon after training began. [9] The disagreement is productive: a single chosen path is a probe whose result can change with architecture, task and training phase.

Compare training protocols through matched interventions

An optimiser is not merely a speed control. It chooses a path through an underdetermined problem, and that path can select among many low-loss functions. Yet a fair causal claim requires more than running AdamW and momentum SGD once and plotting their final neighbourhoods.

Production-shaped worked scenario

Consider a synthetic but realistic document classifier with the same architecture, initialisation distribution and public training corpus. Protocol A uses AdamW. Protocol B uses momentum SGD. The question is whether the optimiser changes robustness to a declared document-format shift, not merely which reaches lower training loss sooner.

A weak comparison gives both protocols 20 epochs and tunes A across forty trials but B across five. A reaches lower loss and appears sharper in an unnormalised plot. None of those observations isolates the update rule. Epochs hide different batch counts, tuning budgets differ, weight decay has different executable semantics, and the plot is scale-sensitive.

A stronger experiment fixes processed examples, total gradient evaluations, precision, augmentation, regularisation at the level of the update equations, checkpoint-selection rule and tuning budget. It repeats seeds, measures target shift performance, estimates extreme Hessian eigenvalues with Hessian-vector products, records gradient-noise projections, and applies the same normalised slice protocol. If A and B still select different robustness profiles at matched training loss, the optimiser-induced selection claim gains support.

The optimiser state is part of the trajectory

Two runs can arrive at the same parameter vector and still take different next steps. Momentum carries a velocity. Adam carries first- and second-moment estimates. A loss scaler, trust ratio, gradient accumulator or adaptive clipping threshold adds further state. The learning-rate scheduler has a phase. The data loader has a position and random-number state. None is visible in a static parameter-space plot.

This matters for causal comparison and recovery. Restarting from weights alone does not necessarily continue the same dynamical system. Resetting Adam moments can alter transient behaviour; changing batch order changes stochastic forcing; restoring a checkpoint under a different precision path changes rounding and overflow events. For long training runs, a reproducible trajectory record therefore includes optimiser tensors, schedule counters, sampler state, random seeds, precision configuration and the exact objective implementation.

Regularisation semantics also belong in the protocol. Classical L2 regularisation adds a parameter-dependent term to the objective. Decoupled weight decay applies a separate shrinkage update. They coincide only under restricted update rules. Comparing “the same weight decay value” across SGD and AdamW can therefore compare different algorithms and different objectives at once.

The proper unit of comparison is the training protocol as an executable state machine. The optimiser family is one component. The architecture and data define what can be represented and observed; the protocol defines which part of that space is reachable under a finite budget.

All numbers in this worked scenario are synthetic. They demonstrate a discriminating design and must not be read as a benchmark result.

The landscape claim audit

The practical instrument below converts a visual claim into a sequence of tests. It deliberately separates local geometry from selected-function behaviour.

A pre-registered comparison record

Before training, write down the causal contrast in one sentence: “Holding architecture, data order distribution, processed examples, precision, tuning budget and endpoint selection fixed, change only the update rule.” Then record allowed hyperparameter ranges, the primary endpoint, seed count, failure criteria and which quantities are descriptive. If a setting diverges, retain it as evidence about the tested range rather than silently replacing it.

During training, capture loss and target metrics against examples, gradient evaluations, wall time and energy where available. Sample gradient norms, update norms, top and bottom Hessian eigenvalues, and projections of stochastic-gradient covariance at predeclared checkpoints. After training, run equivalent-reparametrisation and slice-direction controls before interpreting sharpness. This record makes post-hoc storytelling harder and turns a surprising trajectory into a reusable experiment.

Optimiser choice as a selection mechanism

When many parameter states fit the data, the update rule helps determine which are reachable from a given initialisation under a finite budget. Learning rate, momentum, preconditioning, clipping, batch order and stopping all contribute. The optimiser is therefore part of the effective hypothesis selection process, not an interchangeable courier.

This idea has firm special cases, such as implicit bias results for linear models, and growing empirical support in richer systems. A recent position paper by Pascanu and colleagues argues that optimiser design should be studied as a way to induce qualitative properties such as sparsity, representation structure or resistance to forgetting, rather than only convergence speed. [14]

stateDiagram-v2
  [*] --> Claim
  state "Worked example: one predictor,<br/>a family of minima" as Claim
  state "Thought experiment: robustness<br/>to which perturbation?" as Test
  state "What survives a change of map" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Executable curvature and trajectory lab

The browser artefact below operates only on synthetic objectives. It exposes three claims that prose can blur. The reparametrisation control holds the represented predictor fixed while changing Hessian sharpness. The optimiser comparison holds an anisotropic quadratic and gradient-evaluation budget fixed while changing the update rule. The saddle test compares the same perturbation on a strict saddle and a flat minimum.

#!/usr/bin/env python3
from __future__ import annotations
import math
from dataclasses import dataclass

@dataclass(frozen=True)
class Config:
    optimiser: str = "gd"
    objective: str = "ravine"
    learning_rate: float = 0.04
    steps: int = 120
    momentum: float = 0.9
    kappa: float = 20.0
    perturbation: float = 1e-3

def eigvals2(a: float, b: float, d: float) -> tuple[float, float]:
    trace = a + d
    disc = math.sqrt((a - d) ** 2 + 4.0 * b * b)
    return (0.5 * (trace - disc), 0.5 * (trace + disc))

def equivalent_minimum(c: float) -> dict[str, float]:
    a, b = c, 1.0 / c
    lo, hi = eigvals2(b * b, 1.0, a * a)
    return {"product": a * b, "loss": 0.5 * (a * b - 1.0) ** 2,
            "lambda_min": lo, "lambda_max": hi}

def objective(name: str, x: float, y: float, kappa: float):
    if name == "ravine":
        return 0.5 * (x*x + kappa*y*y), (x, kappa*y)
    if name == "saddle":
        return 0.5*x*x - 0.5*y*y + 0.25*y**4, (x, -y + y**3)
    if name == "plateau":
        return 0.5*x*x + 0.25*y**4, (x, y**3)
    raise ValueError(name)

def run(cfg: Config):
    x, y = ((0.0, cfg.perturbation) if cfg.objective != "ravine" else (4.0, 4.0))
    vx = vy = mx = my = sx = sy = 0.0
    rows = []
    for t in range(cfg.steps + 1):
        loss, (gx, gy) = objective(cfg.objective, x, y, cfg.kappa)
        rows.append((t, x, y, loss, math.hypot(gx, gy)))
        if t == cfg.steps:
            break
        if cfg.optimiser == "gd":
            dx, dy = gx, gy
        elif cfg.optimiser == "momentum":
            vx, vy = cfg.momentum*vx + gx, cfg.momentum*vy + gy
            dx, dy = vx, vy
        elif cfg.optimiser == "adam":
            b1, b2, n = cfg.momentum, 0.999, t + 1
            mx, my = b1*mx + (1-b1)*gx, b1*my + (1-b1)*gy
            sx, sy = b2*sx + (1-b2)*gx*gx, b2*sy + (1-b2)*gy*gy
            dx = (mx/(1-b1**n)) / (math.sqrt(sx/(1-b2**n)) + 1e-8)
            dy = (my/(1-b1**n)) / (math.sqrt(sy/(1-b2**n)) + 1e-8)
        else:
            raise ValueError(cfg.optimiser)
        x -= cfg.learning_rate * dx
        y -= cfg.learning_rate * dy
    return rows

def self_check():
    p, q = equivalent_minimum(1.0), equivalent_minimum(10.0)
    assert math.isclose(p["product"], q["product"])
    assert math.isclose(p["loss"], q["loss"])
    assert q["lambda_max"] > 50 * p["lambda_max"]

    exact = run(Config(objective="saddle", learning_rate=0.1,
                       steps=80, perturbation=0.0))
    perturbed = run(Config(objective="saddle", learning_rate=0.1,
                           steps=80, perturbation=1e-3))
    control = run(Config(objective="plateau", learning_rate=0.1,
                         steps=80, perturbation=1e-3))
    assert exact[-1][3] == 0.0
    assert perturbed[-1][3] < -0.20
    assert control[-1][3] >= 0.0
    print("checks: passed")

if __name__ == "__main__":
    self_check()

.

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: The valley that moved while the model stayed still
  M->>E: Worked example: one predictor, a family of minima
  E-->>M: compare rival explanations
  E->>D: Replace the mountain with an instrument panel
  D-->>Q: revise scope or act

Replace the mountain with an instrument panel

The mountain myth fails because it turns a configured measurement into a thing. It encourages researchers to call a random slice “the landscape”, a small gradient “a trap”, a top eigenvalue “flatness”, and an optimiser name “the cause”. The repair is not to abandon geometry. It is to make every geometric claim carry its measurement contract.

The changed decision is to select experiments before selecting stories. Test coordinate dependence with an equivalent reparametrisation. Test a suspected saddle against a flat control. Inspect random, trajectory and Hessian-aligned directions for different purposes. Couple step size to measured curvature. Compare update rules under matched information, tuning and compute. Then evaluate the property of the selected function that actually matters.

Sometimes the result will justify a local geometric explanation. Sometimes it will reveal that batch construction, normalisation, regularisation or stopping supplied the causal difference. Sometimes the strongest honest conclusion will be that a plot is descriptive. That restraint is productive: it tells the next experiment exactly which feature to vary.

A loss surface remains a powerful instrument when its coordinates, scope and intervention are visible. The goal is no longer to find the prettiest valley. It is to discover which local signals changed the trajectory, which protocol selected the endpoint, and which claims survive when the map is redrawn.