Part one: the recurrent corridor

A feed-forward model sees an input and produces an output. A recurrent model also receives a summary of what it did one step earlier. Jeffrey Elman described this arrangement as a way to give a network a dynamic memory, representing time through its effect on ongoing processing rather than by allocating a separate input position to every moment.^[1]

The basic update is compact:

x_t is the current input, h_t−1 is the previous hidden state, W_h and W_x are learned transformations, b is a bias, and φ is usually a nonlinear activation such as tanh.

The equation appears to put the entire past inside h_t−1. That wording is dangerously generous. The hidden state is finite, continuously rewritten and useful only through the distinctions it preserves. A recurrent network has no independent archive of earlier states unless the surrounding system stores them. It carries one evolving compression.

Unroll the recurrence and the apparent loop becomes a deep computation with shared parameters. The hidden state at step 64 depends on the state at step 63, which depends on step 62, and so on. Backpropagation through time applies the chain rule to this unrolled graph. Werbos presented BPTT as an extension of backpropagation to dynamic systems; the central operation is the same derivative accounting used in ordinary networks, repeated along time.^[2]

A minimal scalar recurrence

Strip the system down to one number and remove new inputs after the first step:

Here a is the recurrent gain. The local sensitivity of the next state to the previous state is a(1 − tanh²(a h_t−1)). Near zero, tanh is almost linear, so the multiplier is roughly a. Far from zero, tanh saturates and its derivative approaches zero. A weight of one does not therefore guarantee a derivative of one.

Suppose the effective local multiplier remains 0.9. One step retains 90 per cent of a small perturbation. After sixty-four steps, the retained fraction is 0.9⁶⁴, about 0.00118. The transition looked gentle locally. Composition made it severe.

This is the first durable intuition: temporal depth turns small local losses into large global amnesia. The same multiplication applies to the influence of an early state on a later state and to one important route by which gradient travels backwards.

flowchart LR
  A["Opening case: the instruction<br/>at the start of the stream"] --> B["Part one, the recurrent corridor"]
  B --> C["Part two, why the past vanishes"]
  C --> D["Part three, what gates<br/>actually buy"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#087A98,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Part two: why the past vanishes

For a loss L_t at time t and an earlier hidden state h_k, the chain rule gives a product of local Jacobians:

J_j = ∂h_j/∂h_j−1. In a tanh RNN it combines the recurrent matrix with the activation derivatives at step j.

A multidimensional Jacobian can contract some directions and expand others. The relevant question is not whether every matrix entry is small. It is how the product acts on the direction that carries the task-relevant perturbation. Repeated singular values below one shrink that direction. Repeated values above one expand it. Rotations can move a useful direction into a contracting one even when each individual step appears well behaved.

Bengio, Simard and Frasconi analysed why gradient-based learning becomes harder as temporal contingencies span longer intervals.^[3] Pascanu, Mikolov and Bengio later described vanishing and exploding gradients from analytical, geometric and dynamical perspectives, and proposed gradient norm clipping as a practical response to explosion.^[6] Clipping limits the size of an update after a gradient becomes large. It does not reconstruct a gradient that has already decayed towards zero.

Thought experiment 1: move the cue, change nothing else

Train two systems on the same binary decision. Both receive the same cue, the same distractor distribution, the same final query, the same labels and the same number of training examples. In system A, the cue arrives four steps before the query. In system B, it arrives sixty-four steps before. Only the gap changes.

If B fails while A succeeds, “the model lacks capacity” is incomplete. Both systems can represent a one-bit answer. The discriminating variable is the length and condition of the causal path. The experiment should therefore record sensitivity to the cue and gradient magnitude by lag, rather than reporting final accuracy alone.

The half-life is useful, and incomplete

If a direct path multiplies by a constant q at every step, its half-life is:

As q approaches one from below, half-life grows sharply. Moving from 0.90 to 0.99 does not improve retention by eleven per cent. It changes the half-life from about 6.6 steps to about 69. This nonlinearity explains why a small gate-bias change can alter trainability over long delays.

The measure has boundaries. A real Jacobian varies by time, input and hidden direction. Several paths can add or cancel. A gate can depend on the state it is carrying. The full derivative may therefore differ from a simple product of displayed gate values. Half-life is an instrument for a declared path, not a certificate that the whole model remembers.

Three distinct ways to lose the past

Contraction reduces the magnitude of an old perturbation. Interference mixes later inputs into the same finite state, so an old distinction becomes unrecoverable even if norms remain large. Credit truncation stops the training graph before the relevant cause. Truncated BPTT can make a state observable at inference while preventing the distant loss from teaching the write decision that created it.

These failures require different remedies. Scaling or orthogonalising the recurrent transition addresses contraction. Gates can reduce interference by admitting some candidates and carrying old state around others. Longer unrolls, auxiliary losses or different supervision address missing credit. Calling all three “vanishing gradient” hides the design choice.

Negative control: when long memory is unnecessary

Consider a first-order temperature process whose next value depends on the current reading and current control input. Give a model the present state explicitly. A linear autoregression, a small feed-forward window or a plain RNN can be an adequate baseline. An LSTM may fit too, but success would not show that gating earned its cost.

This negative control matters because short tasks can make every recurrent architecture look like a memory system. If the target is Markov under the supplied state, long temporal credit is not the causal requirement. A memory claim needs a delay intervention that changes performance while holding the task semantics fixed.

Part three: what gates actually buy

Long short-term memory was introduced to address insufficient, decaying error flow in recurrent learning.^[4] Its enduring idea is easier to see by ignoring the name and following the state path.

The candidate g_t proposes new content. The input gate i_t controls how much enters. The forget gate f_t scales the old cell state. The output gate o_t controls how much of the cell is exposed through the hidden state. Current PyTorch LSTM semantics implement this familiar four-part update.^[15]

The cell update is additive. Old state has a direct route into new state instead of being forced through a complete nonlinear rewrite. Holding the gates fixed, the direct derivative from c_t−1 to c_t is f_t. Across a delay, that route contains the product of forget gates. This is the adjustable near-identity path.

The forget gate was not decorative complexity. Gers, Schmidhuber and Cummins introduced it to let continual LSTM systems reset internal state when sequence boundaries were not externally supplied.^[5] A carry path without a revision mechanism can preserve obsolete information indefinitely. Forgetting is therefore part of useful memory, not its opposite.

Thought experiment 2: perfect retention, one superseding fact

Imagine a recurrent unit whose state transition is exactly the identity whenever no input arrives. It stores a maintenance flag for a thousand steps with no decay. Now insert a verified “maintenance complete” event at step 400. Keep every other property fixed.

If the unit has no selective update mechanism, perfect retention becomes stale lock. It either keeps the obsolete flag or must transform the entire state in a way that risks other stored distinctions. A gate can lower carry for the affected dimensions, admit the replacement and restore high carry afterwards. The causal benefit is not merely longer memory. It is conditional control over the timescale of each state component.

Gate values are time constants

A constant forget value near one produces a long direct half-life. A value near zero resets quickly. Tallec and Ollivier formalised a connection between learnable gates and robustness to time transformations, then proposed chrono initialisation to align gate biases with expected temporal dependencies.^[13] The practical lesson is modest but consequential: gate bias is an architectural prior over duration.

A forget-gate bias of zero produces an initial sigmoid value of 0.5, whose direct half-life is one step. Learning can move it, but the distant signal needed to request that move may already be weak. Jozefowicz, Zaremba and Sutskever evaluated more than ten thousand recurrent architectures and reported that adding a bias of one to the LSTM forget gate closed a performance gap between their LSTM and GRU settings.^[9] That result is task- and implementation-bounded, but it supports the mechanism: the initial timescale can change trainability.

GRU: carry and candidate in one state

The gated recurrent unit uses one exposed state rather than a separate cell and hidden state. In a common formulation:

The update gate z_t interpolates between the old state and a new candidate n_t. A reset gate controls how the previous state contributes while forming that candidate. Cho and colleagues introduced the GRU in an RNN encoder-decoder for statistical machine translation.^[7] PyTorch’s documented GRU equations use the old-state coefficient z_t, so the direct carry route again contains a product of gate values.^[15]

It is tempting to turn LSTM and GRU into a universal ranking. The evidence does not support one. Chung and colleagues found gated units better than tanh units on their polyphonic music and speech tasks, with GRU comparable to LSTM.^[8] Greff and colleagues compared eight LSTM variants across speech, handwriting and music, reporting that none significantly improved on the standard LSTM and that the forget gate and output activation were especially important in their study.^[10] These are useful bounded results, not a law that one cell wins every sequence problem.

The strongest objection: gates are not the only path

A plain recurrent system can preserve long-range signal when its transition is designed around stable propagation. Le, Jaitly and Hinton showed that a ReLU RNN initialised with an identity or scaled identity recurrent matrix could be comparable to LSTM on their four benchmarks.^[11] Arjovsky, Shah and Bengio constrained recurrence to a unitary transformation so eigenvalues had magnitude one, targeting vanishing and exploding gradients on long-memory tasks.^[12]

More recently, Orvieto and colleagues showed that carefully designed linear, diagonal recurrent units with suitable parameterisation, initialisation and normalisation could match deep state-space models on the Long Range Arena benchmark while retaining recurrent inference.^[14] This is a serious counterexample to any story in which gates uniquely “solve” recurrence.

Feng and colleagues pushed the same challenge in a different direction. By removing selected hidden-state dependencies from LSTM and GRU gates, they derived minimal cells whose associative updates can be parallelised during training and reported competitive results across their task suite.^[16] The result does not establish a universal winner. It shows that useful gate semantics can survive substantial algebraic simplification.

The counterexample sharpens the thesis. Orthogonal or near-identity transitions preserve magnitude, but a system must also forget, select and revise. Gates offer one learnable way to vary timescale with content. Stable linear recurrence offers another design point, often paired with input-dependent selection elsewhere. The causal object is the controlled state-transition path, not the historical cell family.

Part four: a temporal memory acceptance test

Benchmark accuracy compresses several temporal behaviours into one number. A recurrent model can succeed by exploiting a short cue, fail only under distractors, retain obsolete state, or answer from the current input while ignoring the supposed memory. A useful test must intervene on the path.

The proposed acceptance test has four verbs. Write asks whether a relevant event changes the state. Retain asks whether its causal influence survives the required delay and routine interference. Revise asks whether a superseding event can replace it within an acceptable number of steps. Read asks whether the state changes the correct output only when the query requires it.

Retention curve R(Δ)

Intervene on an early cue and measure the change in a later output as lag Δ increases. Report the required P50, P95 and maximum horizon, not one convenient delay.

Interference susceptibility I

Hold the cue and lag fixed, then vary distractor count and amplitude. Measure how quickly causal margin or accuracy degrades.

Revision latency V

Insert a superseding event and count the steps until output reflects the new state. Excessive retention appears as a stale-state failure.

Read selectivity Q

Present matched queries that do and do not require the stored fact. A state that changes every output is entanglement, not selective use.

Controls that make the memory claim causal

A decoder probe can sometimes recover an old cue from hidden state even when the model’s own output ignores it. Decodability is therefore a weak pass condition. The decisive test keeps the current input fixed, changes only the earlier cue, and measures the late output. A stronger state-swap test takes two matched sequences with opposite cues, exchanges their recurrent states immediately before the query, and asks whether the answers exchange too. Stored information counts as functional memory only when intervening on its temporal carrier changes the decision.

Two shortcut controls should accompany that intervention. First, erase or permute the early cue across sequences. Performance should fall specifically on queries that require it. If performance survives, a later feature or label artefact may be carrying the answer. Second, train a current-window baseline that sees only the last few events. Matching performance means the declared long dependency was not necessary under the supplied representation, even if a recurrent probe can reconstruct the cue.

The learning path needs its own ablation. Train otherwise matched models with backward horizons shorter than, equal to and longer than the required delay. A sharp improvement when the horizon crosses the causal gap supports distant credit assignment as the mechanism. No change has several possible explanations: local supervision may already teach the write, a shortcut may bypass the old cue, or the optimiser may still fail to use the added path. Inspect cue intervention, state sensitivity and early-step gradient together before choosing among them.

These controls turn one benchmark score into a discriminating experiment. They can falsify the memory story while leaving accuracy unchanged, which is exactly why they belong in an acceptance test rather than an interpretability appendix.

Run the RNN, LSTM and GRU challenge

The browser lab below isolates the direct retention mechanism. Each trial writes a cue of +1 or −1, passes it through a selected number of noisy steps, and asks for the cue’s sign at the end. The plain RNN uses a tanh recurrence. The LSTM-like and GRU-like paths use simplified fixed-gate carry equations. Freezing the gates makes the causal variable visible and keeps architecture search, parameter count and optimiser tuning out of the comparison.

This is a mechanism demonstration, not evidence that a trained LSTM or GRU will achieve the displayed score on a real task. The positive case uses a long carry factor and low distractor admission. The negative control shortens the dependency. The failure preset gives every path the wrong timescale.

The short case is the negative control. Every path recovers the cue, so gating has not earned a claim. The long noisy case isolates the value of a longer, cleaner carry path. The misgated case is equally important: adding gates while retaining a 0.90 timescale does not rescue the task.

The identical LSTM-like and GRU-like rows are deliberate. Once their gates are frozen and candidate details are removed, both reduce to the same scalar carry equation. At that level, the architecture label contains no extra causal information. Their learned behaviour differs because their gate networks, candidate construction and state exposure differ, which must be tested rather than assumed.

#!/usr/bin/env python3
from __future__ import annotations
import argparse, math, random, statistics
from dataclasses import dataclass

@dataclass(frozen=True)
class Config:
    name: str
    carry: float
    noise_admission: float
    recurrence: str

def step(config: Config, state: float, x: float) -> float:
    if config.recurrence == "rnn":
        return math.tanh(config.carry * state + config.noise_admission * x)
    if config.recurrence in {"lstm", "gru"}:
        return config.carry * state + config.noise_admission * math.tanh(x)
    raise ValueError(config.recurrence)

def run(config: Config, delay: int, trials: int,
        noise_sd: float, seed: int) -> dict[str, float]:
    rng = random.Random(seed)
    correct, margins, sensitivities = 0, [], []
    eps = 1e-5
    for _ in range(trials):
        cue = 1.0 if rng.random() >= 0.5 else -1.0
        distractors = [rng.gauss(0.0, noise_sd) for _ in range(delay)]

        def final_state(cue_value: float) -> float:
            state = math.tanh(cue_value)
            for distractor in distractors:
                state = step(config, state, distractor)
            return state

        final = final_state(cue)
        correct += int((final >= 0) == (cue >= 0))
        margins.append(cue * final)
        derivative = (final_state(cue + eps) - final_state(cue - eps)) / (2 * eps)
        sensitivities.append(abs(derivative))

    return {
        "accuracy": correct / trials,
        "mean_margin": statistics.fmean(margins),
        "median_sensitivity": statistics.median(sensitivities),
    }

def half_life(q: float) -> float:
    return math.log(0.5) / math.log(q) if 0 < q < 1 else math.inf

def suite(delay: int, noise: float, trials: int,
          seed: int, misgated: bool):
    gated_carry = 0.90 if misgated else 0.99
    gated_noise = 0.10 if misgated else 0.02
    configs = [
        Config("plain RNN", 0.90, 0.12, "rnn"),
        Config("LSTM-like path", gated_carry, gated_noise, "lstm"),
        Config("GRU-like path", gated_carry, gated_noise, "gru"),
    ]
    return [(config, run(config, delay, trials, noise, seed))
            for config in configs]

def print_suite(title: str, results) -> None:
    print("\n" + title)
    print(f"{'model':18} {'carry':>7} {'half-life':>10} "
          f"{'accuracy':>10} {'margin':>10} {'sensitivity':>12}")
    for config, result in results:
        print(f"{config.name:18} {config.carry:7.3f} "
              f"{half_life(config.carry):10.2f} "
              f"{result['accuracy']:10.3f} "
              f"{result['mean_margin']:10.3f} "
              f"{result['median_sensitivity']:12.6f}")

def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--trials", type=int, default=5000)
    parser.add_argument("--seed", type=int, default=46)
    args = parser.parse_args()
    print_suite("Positive long-delay case: selective carry, 64 noisy steps",
                suite(64, 0.35, args.trials, args.seed, False))
    print_suite("Negative control: short dependency, four quiet steps",
                suite(4, 0.0, args.trials, args.seed, False))
    print_suite("Failure case: gates choose the wrong timescale",
                suite(64, 0.35, args.trials, args.seed, True))

if __name__ == "__main__":
    main()

A production-shaped worked scenario

The temporal contract exposes four different tests. The maintenance event must produce a measurable write. Across the observed event-rate distribution, its influence must remain above threshold for thirty minutes. A completion event must reduce the old influence and establish the new state within one event. Queries unrelated to maintenance should be invariant to the stored flag.

This distinction prevents a category error. A recurrent hidden state is a model proposal about relevant history. It may be useful for classification or forecasting. It should not silently become the sole record that maintenance exists, has ended or may authorise an effect. The operational system still needs typed events, current world state, evidence and recovery outside the model.

Choose by temporal contract

Before selecting a recurrent cell, write the dependency as an engineering contract. State the distribution of required lags in domain time and event count. Name what writes state, what should be ignored, what supersedes it, when it is read, how training credit reaches the write, and whether the state is merely predictive or operationally authoritative.

A fair RNN, LSTM and GRU comparison must also match more than hidden width. Under common implementations, LSTM computes four gate or candidate blocks per step, GRU computes three, and a plain RNN computes one. Equal width therefore means unequal parameter count and work. Equal parameter count means unequal state width. Report both, include tuning budget, and judge the temporal acceptance envelope against latency and memory cost.