TLDR

1. A complex differentiable model is trainable because composition converts one global error into local sensitivity calculations that can be reused across the whole programme.
2. The computation graph is the executable record of those dependencies: reverse mode walks it backwards, multiplying along paths and adding where paths meet.
3. Design models as small typed transformations, include preprocessing and loss construction in the inspected graph, and test every custom derivative against an independent numerical check.
4. A correct graph cannot rescue a hard threshold, hidden side effect, stale branch, saturated derivative or mathematically faithful gradient that is numerically useless.
5. Choose reverse mode for many inputs and few outputs, forward mode for the opposite shape, and make graph breaks explicit architecture decisions.

The compositional bargain

A function is a rule that maps an admissible input to an output. The useful move is to resist treating a model as one enormous rule. Let f turn an input x into an intermediate z, and let g turn z into an output y. Their composition is written g ∘ f, meaning that f acts first and g receives its result.

This naming seems cosmetic until a correction must travel backwards. For scalar quantities, the chain rule says that the sensitivity of y to x is the sensitivity of y to z, multiplied by the sensitivity of z to x.

The equation is familiar. Its engineering meaning is easier to miss. Each primitive needs only a local derivative rule. Multiplication knows how its output changes with either input. Addition passes sensitivity unchanged. The hyperbolic tangent uses 1 − tanh²(z). The squared error uses twice the prediction error. None of these primitives needs to understand the model around it.

Automatic differentiation grew from precisely this decomposition. Wengert's 1964 programme evaluated derivatives by breaking an algebraic function into elementary expressions rather than expanding one symbolic derivative.[1] Modern systems differ in tracing, intermediate representations and compiler strategy, but the causal economy is the same: global sensitivity is assembled from local rules whose scope is small enough to implement, test and reuse.

A minimal worked example

Return to the opening model. Its forward computation is:

Start at the loss. The local derivative ∂L/∂y is 2(y − t) = 0.523188. The derivative ∂y/∂z is 1 − y² = 0.419974. Their product gives ∂L/∂z = 0.219726. Because ∂z/∂w = x = 1.5, the weight gradient is 0.329589. Because ∂z/∂b = 1, the bias gradient is 0.219726.

A small step opposite those positive gradients will lower the loss locally. Yet the gradients do not prove that the weight is the worldly cause of the error. They describe the slope of this chosen function at this chosen point. The distinction matters whenever inputs are correlated, features are proxies, or the objective omits a real consequence.

Figure 1 adds a useful warning. The loss is zero along an entire line of weight and bias pairs. The graph can tell us a local descent direction from the current point. It cannot choose a unique explanation without more data, regularisation or constraints. Differentiability provides navigability, not identifiability.

What inspectability actually requires

Breaking a model into functions does not automatically make it inspectable. A useful boundary states the admissible input type and shape, the output it promises, the parameters it reads, any state it mutates, and the derivative rule it exposes. A function that silently reads a global variable or changes behaviour with hidden state may be easy to call while remaining difficult to reason about. The graph records declared data dependencies; it cannot reveal a dependency that the programme concealed from the representation.

For vector-valued functions, the local contract is a linear map rather than a table of every partial derivative. A primitive can say how an input tangent is pushed forwards, or how an output adjoint is pulled backwards, without materialising its full Jacobian. This is why matrix multiplication, convolution and attention can participate in large graphs: their derivative rules exploit structure rather than constructing vast dense matrices.

Inspectability means that a reviewer can reconstruct both the value path and the sensitivity path at the chosen boundary. That requires names for consequential intermediates, stable shape information, visible parameter reuse and declared graph breaks. It does not require exposing every compiler instruction. The right level is the smallest representation that preserves the claim being reviewed. For a custom normalisation layer, that may be one primitive plus a verified VJP. For a model-wide investigation, it may be a staged intermediate representation with selected activation and gradient probes.

flowchart TD
  A["One wrong number, four<br/>possible causes"] --> B["The compositional bargain"]
  B --> C["The graph carries two journeys"]
  C --> D["Where gradients fail without<br/>being false"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#B96D00,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

The graph carries two journeys

A computation graph is a directed record of dependencies. Nodes hold inputs, intermediate values or primitive operations. An edge says that one result was needed to produce another. For a straight-line differentiable programme, the graph is usually acyclic. Loops are represented by unrolling, a loop primitive, a fixed-point operator or a compiler-level control-flow structure.

The forward journey computes values. The reverse journey computes sensitivities. During the forward pass, a tracing system records enough information to replay local derivative rules. PyTorch describes this conceptually as a directed acyclic graph of operations created while tensors execute.[9] JAX can instead stage a typed functional intermediate representation called a jaxpr, which can be inspected and transformed.[12] TensorFlow can turn Python functions into graph data structures that may be saved, optimised and executed away from Python.[14]

These are implementation choices, not rival definitions of the mathematics. An eager tape, a static graph and compiler intermediate representation can all support the same derivative calculation. The essential object is the executed dependency structure plus valid local linearisation rules.

Three techniques are often blurred. Finite differences perturb a complete programme and approximate a slope. Symbolic differentiation rewrites an expression into another expression, which can duplicate subexpressions and grow unwieldy. Automatic differentiation evaluates the original primitives while propagating exact chain-rule information to floating-point accuracy.[15] Backpropagation is the neural-network use of reverse-mode differentiation to move a loss signal through layered parameters; the optimiser that applies those gradients is a separate mechanism.[3] Keeping these objects distinct clarifies failures: an autodiff defect concerns the derivative programme, while a poor optimiser step can occur even when every derivative is correct.

The reverse recurrence

Suppose each node v_j is produced by a primitive function of its parent nodes. Let the final scalar be L. Define the adjoint of a node as v̄_i = ∂L/∂v_i. The reverse rule is:

This recurrence is the whole reverse pass in compressed form. Seed the output with L̄ = 1, because ∂L/∂L = 1. Visit nodes in reverse topological order. Each node sends its incoming adjoint through its local derivative rule. If two downstream branches depend on the same node, their contributions accumulate.

A common implementation error in hand-built engines is to overwrite a gradient at the shared node instead of adding to it. The resulting forward values are perfect. Only the backward semantics are broken. This is why the graph must be more than a visual diagram. Its fan-out and topological order carry executable meaning.

Forward mode, reverse mode and the shape of the question

For a vector function f: ℝⁿ → ℝᵐ, the Jacobian J_f has m rows and n columns. Forward mode computes a Jacobian-vector product, J_fv. It asks how one chosen input direction v changes all outputs. Reverse mode computes a vector-Jacobian product, uᵀJ_f. It asks how one chosen combination of outputs changes all inputs.

Training usually maps many parameters to one scalar loss. The Jacobian is very wide, so reverse mode obtains the full gradient in one backward sweep. JAX's official explanation makes the same dimensional argument and notes the associated memory cost: reverse mode stores or reconstructs information from the forward computation.[11] Forward mode is attractive when inputs are few and outputs are many, such as sensitivity of a large simulated field to a small number of control variables.

The famous cheap-gradient idea gives useful intuition: for scalar-output arithmetic circuits, all first derivatives can be computed within a constant factor of evaluating the original function.[2] It does not promise that every real tensor programme is automatically cheap. Pathological transformations can create asymptotic slowdowns, and practical performance depends on sparsity, mutation, compiler optimisation and memory movement.[7]

Modern work sharpens the mechanism. Reverse mode can be decomposed into linearising primitives, separating residual values from the linear part, and transposing that linear programme.[6] This is more than a theoretical reformulation. It explains why local linear rules compose, why the backward pass runs in reverse order, and why checkpointing is a choice about which forward residuals to retain.

A 2025 result makes the compiler consequence concrete. Dangel and colleagues collapse Taylor-mode automatic differentiation for linear differential operators by pushing a sum through the computation graph. Their PyTorch experiments report lower time and memory for the studied Laplacian and biharmonic calculations than nested first-order differentiation.[16] The boundary matters: the rewrite uses linearity, and the implementation does not cover arbitrary operators. The broader lesson is exact. A computation graph is an optimisation surface only when a proposed rewrite preserves its mathematical contract.

Where gradients fail without being false

An autodiff engine can produce the exact derivative of the programme it saw and still mislead the system designer. Three questions must remain separate. Did the graph capture the executed computation? Were the local derivative rules mathematically valid? Does the derivative answer the decision question we care about?

A gradient can be computationally correct, numerically unusable and semantically irrelevant at the same time. Each failure requires a different remedy.

Graph completeness

A graph is complete only relative to a claimed function. If feature normalisation occurs in untracked code, a parameter is detached, an external simulator is called without a derivative contract, or an in-place mutation changes a saved value, the backward graph may describe a different programme from the one the team imagines.

Dynamic graphs reduce some mismatches because they record operations that actually ran. They do not guarantee that every consequential effect is represented. File writes, network calls, database changes, random sampling and human decisions may sit outside the differentiable trace. A graph break is not automatically a defect. It is a boundary that needs a declared treatment: stop the gradient, supply a validated custom rule, use an estimator, redesign the objective, or evaluate the component separately.

Training and inference also diverge easily. Dropout is disabled, a branch changes, quantisation is introduced, a retrieval result enters, or a policy threshold is applied. The training gradient remains exact for the training programme. It says nothing direct about an inference-only branch unless that branch is included in evaluation.

Local-rule validity

Framework primitives come with derivative conventions. At non-differentiable points, systems may choose a subgradient, a continuity-based value, an arbitrary convention or an error. PyTorch documents such choices explicitly for non-differentiable functions.[9] Custom primitives are more dangerous because a user can return the correct forward value with a wrong backward rule.

Independent gradient checking is therefore a release control, not a classroom exercise. Central finite differences compare the autodiff result with two nearby function evaluations. PyTorch's gradcheck follows this analytical-versus-numerical pattern.[10] The comparison is imperfect near discontinuities and sensitive to precision, but it can expose a faulty local rule that ordinary forward tests will never see.

Large steps compare different parts of the curve and suffer truncation error. Tiny steps subtract nearly equal floating-point numbers and suffer cancellation. A failed check may reveal a wrong derivative, a non-smooth point, an unstable function or a poor step size. The diagnostic must inspect all four possibilities.

Numerical usefulness

A correct chain of derivatives can shrink towards zero or grow without bound. Repeated Jacobian multiplication explains vanishing and exploding gradients in long compositions. Bengio, Simard and Frasconi showed why learning long dependencies becomes increasingly difficult under gradient descent.[4] The graph has done its job. The architecture has created a poor signal path.

Saturation is the local version. In the opening example, tanh has derivative 0.419974 at z = 1. At z = 8, that derivative is effectively zero. The model can be far from the desired behaviour while an upstream parameter receives almost no correction. Residual connections, normalisation, activation choices and objective design address such problems in later layers of the canon. The present lesson is narrower: inspect gradient magnitude along the path, not only the final parameter gradient.

Semantic relevance

The gradient ∂L/∂amount in a fraud score is a property of the score function. It is not evidence that increasing a customer's transaction amount causes fraud in the world. Nor does ∂L/∂a protected attribute establish an admissible intervention. Gradients answer local counterfactual questions inside the model, holding the rest of the programme fixed. Causal claims require an intervention model and assumptions about the data-generating process.

The objective can also be wrong. A differentiable proxy may reward accuracy while ignoring calibration, recovery cost, fairness or a hard authority boundary. Automatic differentiation efficiently optimises whatever objective it is given; it does not validate the objective.

Negative control: two knobs and a cheap function

Suppose a programme has only two scalar inputs, takes microseconds to run and has no custom operations. Central differences need four extra evaluations to estimate both derivatives. They may be easier to implement and sufficiently accurate. A symbolic algebra system may produce a compact exact derivative.

This case weakens any claim that a computation graph is always necessary. It is not. The graph earns its cost when inputs are numerous, intermediates are shared, derivative transformations must compose, higher-order products are required, or the team needs inspectable boundaries and compiler optimisation. The correct baseline for a small function is often a simpler method.

Memory is part of the mechanism

Reverse mode needs forward values for local rules. Saving everything can make memory grow with depth. Checkpointing discards selected intermediates and recomputes them during the backward pass. JAX documents this directly as a memory-versus-computation trade.[13] Chen and colleagues demonstrated systematic sublinear-memory schedules for deep networks by analysing the computation graph.[5]

This yields an architectural rule: the graph is simultaneously a derivative plan and a lifetime plan for intermediate state. A model that fits in inference memory may fail in training because reverse mode retains activations. A checkpoint policy that saves memory may increase latency or repeat non-deterministic work. Those choices deserve explicit tests.

From toy function to reviewed model

The foundational mechanism becomes operational when the graph includes the whole optimised objective while preserving boundaries that should remain deterministic or accountable. Consider a synthetic transaction-risk model. It transforms amount, device novelty and counterparty novelty into a probability. A weighted log loss trains the score. A separate policy decides whether the transaction enters review.

The model uses:

For amount £480, device novelty 0.7 and counterparty novelty 0.4, the transformed amount is 1.757858, the score is 0.820501, the probability is 0.694343 and the weighted loss is 1.459159. The local derivative ∂L/∂s is −1.222630. The amount sensitivity is −0.001476 loss units per pound, or approximately −0.1476 per £100 near this point, while all other inputs are held fixed.

That derivative helps optimise the score. It should not directly authorise a hold. The policy might require p ≥ 0.8 plus deterministic eligibility checks. The threshold is intentionally outside the training gradient. Its consequences should be selected through calibration, cost analysis, fairness tests, operational capacity and accountable decision rights.

This separation prevents two opposite errors. One is forcing a non-differentiable policy into the training path and then inventing a gradient convention that has no decision meaning. The other is allowing a smooth model score to bypass deterministic eligibility, capacity or authority controls merely because it is differentiable.

A release trace for the production-shaped case

A review begins by fixing the claimed function: the feature transformations, score, label convention, weighting and scalar objective for one version of the model. The team then captures the traced graph or intermediate representation for representative input shapes. Branches that depend on data need cases on both sides. Random operations need an explicit key or seed policy. Any custom operation receives a separate forward specification and derivative specification.

The next evidence concerns signal quality rather than mere existence. Sampled batches should show finite gradients at the objective, intermediate score and each parameter group. Histograms can expose a branch that never receives updates, a saturated activation or a scale difference that dominates the optimiser. Selected examples should be checked in float64 against finite differences or an analytic case. The deliberately wrong derivative from the executable artefact belongs in the same test suite, because it proves that the checker can reject a plausible forward implementation.

Finally, the team profiles the actual mode and residual plan. It records peak saved-state memory, recomputation cost and behaviour under checkpointing. It separately evaluates the threshold and eligibility policy against calibration, capacity and consequence criteria. The resulting release record contains two linked claims: the model can learn the stated smooth objective, and the deterministic policy can govern the resulting score. Neither claim silently substitutes for the other.

from __future__ import annotations
from math import tanh
from typing import Callable, Iterable

class Value:
 """A scalar reverse-mode automatic differentiation node."""

 def __init__(self, data: float,
 _children: Iterable["Value"] = (),
 op: str = "", label: str = "") -> None:
 self.data = float(data)
 self.grad = 0.0
 self._prev = tuple(_children)
 self._backward: Callable[[], None] = lambda: None
 self.op, self.label = op, label

 @staticmethod
 def _coerce(other: float | "Value") -> "Value":
 return other if isinstance(other, Value) else Value(float(other))

 def __add__(self, other: float | "Value") -> "Value":
 other = self._coerce(other)
 out = Value(self.data + other.data, (self, other), "+")
 def _backward() -> None:
 self.grad += out.grad
 other.grad += out.grad
 out._backward = _backward
 return out

 def __radd__(self, other: float | "Value") -> "Value":
 return self + other

 def __mul__(self, other: float | "Value") -> "Value":
 other = self._coerce(other)
 out = Value(self.data * other.data, (self, other), "*")
 def _backward() -> None:
 self.grad += other.data * out.grad
 other.grad += self.data * out.grad
 out._backward = _backward
 return out

 def __rmul__(self, other: float | "Value") -> "Value":
 return self * other

 def __neg__(self) -> "Value":
 return self * -1.0

 def __sub__(self, other: float | "Value") -> "Value":
 return self + (-self._coerce(other))

 def tanh(self) -> "Value":
 y = tanh(self.data)
 out = Value(y, (self,), "tanh")
 def _backward() -> None:
 self.grad += (1.0 - y * y) * out.grad
 out._backward = _backward
 return out

 def backward(self) -> None:
 order: list[Value] = []
 seen: set[int] = set()
 def visit(node: Value) -> None:
 if id(node) in seen:
 return
 seen.add(id(node))
 for parent in node._prev:
 visit(parent)
 order.append(node)
 visit(self)
 self.grad = 1.0
 for node in reversed(order):
 node._backward()

def central_difference(fn: Callable[[float], float],
 x: float, h: float = 1e-6) -> float:
 if h <= 0:
 raise ValueError("h must be positive")
 return (fn(x + h) - fn(x - h)) / (2.0 * h)

def bad_square(x: Value) -> Value:
 # Correct x² value, deliberately wrong derivative: 3x rather than 2x.
 out = Value(x.data * x.data, (x,), "bad_square")
 def _backward() -> None:
 x.grad += 3.0 * x.data * out.grad
 out._backward = _backward
 return out

def main() -> None:
 x, w = Value(1.5, label="x"), Value(0.8, label="w")
 b, target = Value(-0.2, label="b"), Value(0.5, label="target")
 prediction = (w * x + b).tanh()
 loss = (prediction - target) * (prediction - target)
 loss.backward()

 numeric_w = central_difference(
 lambda candidate: (tanh(candidate * 1.5 - 0.2) - 0.5) ** 2,
 0.8)
 good = abs(w.grad - numeric_w) < 1e-7

 probe = Value(2.0, label="probe")
 bad_square(probe).backward()
 numeric_bad = central_difference(lambda v: v * v, 2.0)
 caught = abs(probe.grad - numeric_bad) >= 1e-7

 print(f"loss={loss.data:.9f}")
 print(f"dL/dw autodiff={w.grad:.9f} finite_difference={numeric_w:.9f}")
 print(f"positive_case={'PASS' if good else 'FAIL'}")
 print(f"bad_square autodiff={probe.grad:.6f} finite_difference={numeric_bad:.6f}")
 print(f"negative_case={'PASS (fault detected)' if caught else 'FAIL'}")

if __name__ == "__main__":
 main()

loss=0.068431502
dL/dw autodiff=0.329588500 finite_difference=0.329588500
positive_case=PASS
bad_square autodiff=6.000000 finite_difference=4.000000
negative_case=PASS (fault detected)

stateDiagram-v2
  [*] --> Claim
  state "A minimal worked example" as Claim
  state "What inspectability actually<br/>requires" as Test
  state "Thought experiment, the sealed<br/>module" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Compact glossary

Function

A rule mapping admissible inputs to outputs. Its derivative describes local sensitivity, not necessarily real-world causation.

Composition

Feeding one function's output into another. Composition creates the chain of local derivative contracts.

Primitive

An operation for which the system knows evaluation and transformation rules, such as addition, multiplication or matrix multiplication.

Computation graph

A representation of executed operations and dependencies. It may be an eager tape, static graph or compiler intermediate form.

Primal

The ordinary forward value of a computation, distinguished from tangent or adjoint information used for derivatives.

Adjoint

The accumulated sensitivity of a scalar output to an intermediate value, often written v̄ = ∂L/∂v.

JVP

A Jacobian-vector product that pushes one input direction forward through all outputs.

VJP

A vector-Jacobian product that pulls one output direction backwards to all inputs.

Graph break

A boundary where the tracked differentiable programme stops. It requires an explicit treatment rather than an assumed gradient.

Checkpointing

Recomputing selected forward intermediates during reverse mode to trade more computation for less stored state.

Review the path, not the presence of a gradient

The durable intuition is simple. A complex model does not need one gigantic derivative formula. It needs a decomposition into small transformations, each able to report how local changes pass through. The computation graph records the dependencies that make those reports composable. Reverse mode then multiplies along paths and adds where paths meet.

That mechanism explains the extraordinary reach of gradient-based learning. It also establishes a stricter review standard. The existence of a backward pass is evidence that software executed derivative rules; it is not evidence that the intended system was represented, the rules were valid, the signal was useful or the objective deserved optimisation.

The architecture decision changed by this argument is concrete. Build differentiable components as small, typed, side-effect-aware functions. Inspect the traced graph or intermediate representation. Keep preprocessing and objective construction inside the reviewed path. Put discrete authority outside the gradient unless a defensible surrogate and decision process connect them. Select forward, reverse or mixed mode from the shape of the derivative question. Profile saved state and checkpoint deliberately.

Finally, release the gradient as a chain of claims. The objective has meaning. The graph is complete. Local rules are valid. Signals are finite and informative. The mode and memory plan fit. An independent positive check agrees, and a negative test proves the check can fail.

The changed decision is to stop approving “autodiff enabled” as an implementation detail and start approving a gradient path as operating evidence. That is the point at which composition becomes more than convenient notation. It becomes a disciplined interface between mathematical intent, executable computation and model change.