The alarm that arrives before the question

Imagine a monitoring system reading one sensor value every millisecond. Most readings are ordinary. A rare control packet marks a reading as the start of an incident. Ten minutes later, an operator asks one of two questions: “What was the most recent marked value?” or “What was the value exactly 317 milliseconds before the mark?”

The first question has a tiny sufficient memory. The system can keep one number and overwrite it only when a mark appears. The second question is different. The future query can point to an arbitrary earlier position, so the system needs either an addressable history or a compression scheme rich enough to reconstruct that position. Sequence length is identical. The required memory geometry is not.

This is the central answer: a modern selective state-space model, or SSM, is powerful because it learns which parts of a stream should alter a compact state. Attention is powerful because a query can address many earlier token representations directly. Selectivity narrows the gap, sometimes dramatically, but it does not make a fixed-size state and a growing archive equivalent for every task.

That distinction changes how long-sequence systems should be designed and evaluated. A benchmark that asks for a running statistic may flatter recurrence. A benchmark that asks for arbitrary key-value recall may flatter addressable memory. A production workload often contains both, plus an external evidence store that neither architecture should replace.

Two memory geometries

A causal attention layer represents each earlier position with a key and value. At the current position, a query scores those keys and mixes the associated values. During autoregressive inference, the implementation normally retains a key-value cache for prior positions. The cache is not raw text, and attention weights are not explanations, yet the useful structural fact remains: earlier positions stay separately addressable inside the active context.

A recurrent model takes another route. It folds the next input into a latent state and discards the previous state after the update. The state can be large and distributed across layers, channels and heads, but its shape is fixed by the architecture. History survives only through the transformation it has already caused.

The recurrent mechanism

A classical state-space system separates a hidden state from its observations. Deep sequence SSMs inherit that idea from control and signal processing, then learn the parameters inside a neural network. In a discrete form, one channel can be written as:

Here x_t is the new input, h_t is the latent state, Ā_t controls how the previous state persists or changes, B̄_t writes the input into state, and C_t reads from state to produce the output. The bars indicate parameters after a continuous system has been discretised for sampled data. Increasing retention in Ā_t extends influence through time. Increasing a component of B̄_t makes the current input alter that component more strongly. Changing C_t changes which stored mixture becomes visible.

Dense causal attention computes a different object:

The current query q_t scores each earlier key k_i; the normalised weight α_t,i controls how much of value v_i enters the output. A later query can therefore form a new retrieval pattern over the same stored positions. The Transformer made this mechanism the centre of a parallel sequence architecture. [2]

The archive analogy has limits. Keys and values are learned projections, not database rows. Several layers repeatedly rewrite token representations. Finite precision, context limits and attention diffusion still cause failures. Yet the analogy identifies the contract relevant to recall: the query can discriminate among past positions before they are mixed into one state.

Consider ten tokens encoding five key-value pairs: A:4, B:9, C:7, D:2 and E:6. The final token asks for C. An attention head can form a query from C, score the five key positions and route the value 7 forward. A recurrent layer can also solve the task, but it must have already transformed the five pairs into a state from which C:7 remains recoverable. If the same state must answer any of a thousand possible keys, the number of distinctions it must preserve rises even though the output is still one value. This is why candidate count can be more diagnostic than raw sequence length.

There is also a temporal difference in where selection occurs. Attention can postpone much of the choice until the query arrives. A recurrence performs irreversible mixing at every step, although later layers and input-dependent reads can partly compensate. Deferred selection favours addressability; early selection favours compression. Neither is universally superior. If the query distribution is narrow and stable, early selection may remove enormous waste. If the query is open-ended, early selection creates a harder prediction problem about future relevance.

Selection turns compression into control

Early structured SSMs were usually linear and time invariant inside each layer. The same transition and input maps applied at every position. This regularity was valuable because one recurrence could also be expressed as a convolution, enabling parallel training. HiPPO gave the state an online approximation meaning by projecting accumulated history onto polynomial bases. S4 then imposed useful structure on the state matrix so long filters could be computed efficiently and stably. [3] [4]

Time invariance also creates a blind spot. Suppose ordinary readings and alarm readings have identical numeric scale, while a separate marker determines which one matters. A fixed convolution can weight positions by distance, but the correct positions vary with content. The model needs the current input to alter how information is written, retained or exposed.

Mamba's central move was to make the step size Δ, input map B and read map C functions of the input. The underlying A remains learned, while the discretised transition changes through the input-dependent step. The result is a linear time-varying recurrence whose behaviour can differ at each token. The Mamba paper describes this as selection: context-aware control over what enters or propagates through state. [6]

A minimal worked example

Let each input contain a value v_t and a binary marker m_t. The task is to output the most recent marked value. A one-dimensional selective recurrence can solve it exactly:

When the marker is zero, the old state passes through. When it is one, the new value replaces it. Sequence length does not increase the required state. This toy update is simpler than a Mamba block, but it exposes the causal advantage of selection: irrelevant inputs do not have to compete inside memory.

Selectivity also provides a learned forgetting policy. A boundary token can encourage reset. A salient event can slow decay along useful channels. A read map can expose different aspects of the same compressed state at different positions. Recent analysis gives more formal support: input selectivity can counteract sensitivity decay, improve approximation of discontinuous functions and participate in analytical solutions to associative-recall constructions. Those results clarify capability; they do not imply that every trained model will discover the construction. [13]

Selection should therefore be understood as a routing policy inside a constrained memory, not as extra storage appearing for free. Opening a write path protects the current input only by using some of the representational budget. Slowing decay for one feature can allow another feature to interfere. A learned reset can remove stale context, but a mistaken reset destroys information with no token-level archive to revisit. The useful question is not whether a gate exists; it is whether training makes the gate align with the future loss under shifts in content, spacing and sequence length.

This gives a sharper ablation. Keep the recurrent state size and all surrounding layers fixed. Compare a time-invariant SSM, a selective SSM and a selective SSM whose input-dependent parameters are shuffled across positions. If selectivity is causal, the true selective model should retain its advantage on variable-spacing tasks, while the shuffled control should lose it. Such an intervention is more informative than comparing two end-to-end models that also differ in width, optimiser and convolution.

Parallel training, recurrent decoding

A naive recurrence is sequential, which appears hostile to GPU training. Earlier linear time-invariant SSMs avoided that problem by converting the recurrence into a global convolution. Input-dependent parameters break the fixed convolution kernel. Mamba recovered parallelism with a hardware-aware scan, kernel fusion and recomputation, keeping expanded state in faster on-chip memory rather than materialising it across the full sequence in high-bandwidth memory. [6]

This is a useful warning against comparing architectures by big-O notation alone. FlashAttention retains exact attention while reducing costly memory traffic through tiling. Mamba's scan is also designed around the memory hierarchy. A theoretically linear kernel may lose to a highly optimised quadratic kernel at practical lengths, dimensions or batch sizes. Measure wall-clock behaviour on the target workload and device. [7]

Mamba-2 reframed a family of recurrent and attention-like computations through structured state-space duality, then used the connection to design a faster state-space layer. Mamba-3, released in 2026, revisited discretisation, complex-valued state dynamics and multi-input multi-output updates. Its authors report stronger retrieval and state tracking, including synthetic tasks that earlier real-valued transitions struggled to solve, while targeting inference efficiency. [8] [15]

These developments make “SSM versus attention” less like a contest between sealed species. Both can be expressed through structured sequence transformations, and hybrids can interleave them. The operational distinction still matters because the recurrent execution path and the addressable-cache path expose different memory, latency and recall behaviour.

Training and decoding must also be reported separately. A selective scan can parallelise a full known sequence during training or prefill, then switch to a one-step recurrent update during generation. Dense attention can use highly parallel matrix multiplication during prefill, then repeatedly read a growing cache during decoding. A model that wins prefill may lose long generation, and a model that wins single-stream decoding may underutilise hardware at low batch size. End-to-end service cost depends on prompt length, output length, concurrency and cache residency, not on one kernel headline.

flowchart LR
  A["The alarm that arrives before<br/>the question"] --> B["Two memory geometries"]
  B --> C["Selection turns compression<br/>into control"]
  C --> D["A compact state is a bet on<br/>what the future will ask"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#007D68,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

A compact state is a bet on what the future will ask

A fixed-size state can encode a great deal. Real numbers can represent enormous combinatorial spaces in ideal mathematics. Deep networks distribute memory across many channels and layers. Selectivity can protect important content, and a query can influence later reads. It would therefore be wrong to say that recurrence cannot recall.

The practical constraint is narrower and more useful. Finite-precision state has limited capacity. Every update must decide, explicitly or implicitly, which distinctions among prior sequences still matter. If two histories produce nearly the same state, a later computation cannot reliably answer a question that requires distinguishing them. Compression succeeds when the state is close to a sufficient statistic for the future task.

Running sum is the clean negative control. One scalar state exactly accumulates an arbitrarily long integer stream within the chosen numeric range. Keeping every token separately adds no information needed by the final sum. The “long-range” label alone tells us nothing about the right mechanism.

Arbitrary lookup reverses the situation. Suppose a stream contains many unrelated key-value pairs, then the final query names one key chosen after all pairs have arrived. A full archive can compare the query with each retained key. A fixed state must preserve enough separable information about all candidate pairs before knowing which one will be requested. State size, precision, depth and training all affect performance, but the workload has created a recall-capacity pressure that a running sum does not.

This is not a proof that a particular finite network must fail at a particular length. A state with thousands of channels can store many associations, distributed encodings can be efficient, and several layers can implement algorithms that a one-layer toy cannot. The claim is comparative: when the number of independently retrievable facts grows while state size and precision stay fixed, more histories must share nearby states. Robust retrieval then requires stronger coding, cleaner optimisation or more depth. Addressable memory pays a growing storage cost to avoid forcing all those facts through the same bottleneck.

Precision belongs in the argument. With ideal real numbers, a single scalar can encode an arbitrary finite bit string in its digits. A physical implementation has bounded precision, noise, nonlinearities and finite margins. A memory scheme that works only by separating exponentially close states will be brittle under quantisation and training error. Capacity discussions should therefore state numeric precision and robustness, not rely on infinite-precision constructions.

What experiments currently establish

Published evidence

The Mamba paper showed that input-selective SSMs could solve selective-copy and induction-style synthetic tasks used to expose weaknesses in earlier time-invariant models. It also reported competitive language modelling and strong efficiency results at the tested scales and hardware. These results established that recurrent compression need not be confined to smooth signals. They did not establish universal superiority over attention. [6]

Research on the recall-throughput trade-off found that recurrent alternatives with fixed state can trail attention on recall-intensive tasks, and proposed combining linear attention with a small sliding attention window. The authors relate recall to recurrent state size and report a Pareto frontier rather than a single winner. [9]

A mechanistic study of associative recall found that Transformers and a Based model solved the tested task fully, Mamba came close, and other SSM-style models failed. Causal interventions suggested that the successful architectures reached the answer through different internal mechanisms: Transformer-style induction heads stored associations through token interactions, while Mamba depended on a mechanism involving its short convolution and final state. This matters because equal accuracy can conceal unequal generalisation behaviour. [12]

The objection is serious: some apparent recall deficits are optimisation deficits. Mimetic initialisation makes state-space layers easier to train on copying and associative recall. Later work found modern recurrent models unusually sensitive to learning rate on these tasks. A failed run can therefore understate architectural capacity, while a carefully tuned run can still leave a state-size boundary. [11] [14]

Mamba-3 strengthens the caution against freezing the argument around one architecture version. Its more expressive dynamics target state-tracking failures in prior linear models and improve the reported performance-efficiency frontier. The correct conclusion is conditional: fixed-state sequence models are improving, but their evaluation must separate representational capacity, learned selectivity, optimisation stability and hardware execution.

Benchmark choice can otherwise reverse the story. Long Range Arena was valuable because it compared efficient sequence models across text, images, mathematical structure and pathfinding at lengths from roughly one thousand to sixteen thousand. S4's strong results demonstrated that attention was not the only route to long-range dependence. Yet success on classification over a long input does not by itself establish arbitrary token recall. The output may depend on a compressible global feature. Conversely, an associative-recall benchmark isolates lookup but can underrepresent natural-language abstraction. A credible evaluation portfolio needs both. [5]

Mechanistic evidence also deserves its own ledger. Behavioural accuracy says whether an answer was correct. Intervention evidence asks which internal components were necessary. Systems evidence measures memory movement and latency. These ledgers can disagree without contradiction. A model may reach equal accuracy through a mechanism that generalises differently, or show elegant asymptotics through a kernel that is slow on the available device.

A serious failure boundary

The selective-memory argument ceases to help when the task cannot be compressed into the available state without losing distinctions that later queries require. It also ceases to predict deployment performance when input-output projections, feed-forward layers, tokenisation, training data or kernels dominate the result. A nominally constant-memory decoder can still be impractical if its recurrent state is large, poorly utilised by hardware, or replicated across many layers and batches.

Conversely, attention's addressability does not guarantee retrieval. A query may attend diffusely, keys may collide, positional behaviour may fail outside training, and the active context may omit the required evidence. Exact evidence duties should be handled by an external source and claim-level verification, not by assuming that either hidden state or KV cache is an authoritative record.

An executable stress test for the memory contract

Executable artefact

The included Python programme does not train Mamba or a Transformer. That would entangle architecture, optimiser, data, parameter count and kernel quality. Instead, it isolates the causal distinction with three transparent memory contracts:

1. a scalar running sum, which is a sufficient statistic;
2. a scalar selective update that retains the last marked value;
3. arbitrary lookup using either a fixed number of key-value slots or a full growing archive.

Each sequence contains marked and unmarked synthetic events. The random-access query is chosen uniformly from marked events after ingestion. A fixed-slot state keeps only the most recent marked pairs. The archive keeps all of them. The benchmark uses 2,000 trials per length, marker probability 0.125 and seed 86.

The result is deliberately unsurprising. That is its value. The positive controls prevent a false conclusion that fixed memory simply fails on long sequences. The arbitrary-lookup case prevents the opposite conclusion that selectivity removes all capacity pressure. Increasing slots moves the failure point; it does not change the qualitative geometry.

Run the artefact

python selective_memory_stress_test.py

Synthetic memory-contract stress test | trials=2000 | mark_probability=0.125 | seed=86
    length   mean marked   last marked  running sum     random slots=8    random slots=32   random archive
        32           4.1         1.000        1.000              0.999              1.000            1.000
       128          16.1         1.000        1.000              0.527              1.000            1.000
       512          63.8         1.000        1.000              0.140              0.522            1.000
      2048         255.9         1.000        1.000              0.032              0.121            1.000

def last_marked_state(events):
    state = None
    for event in events:
        if event.marked:          # input-dependent write gate
            state = event.value
    return state

class FixedSlots:
    def __init__(self, capacity):
        self.capacity = capacity
        self.slots = []

    def write(self, event):
        if not event.marked:
            return
        self.slots.append((event.key, event.value))
        if len(self.slots) > self.capacity:
            self.slots.pop(0)     # compression forces an eviction

    def query(self, key):
        for stored_key, value in reversed(self.slots):
            if stored_key == key:
                return value
        return None

class FullArchive:
    def __init__(self):
        self.data = {}

    def write(self, event):
        if event.marked:
            self.data[event.key] = event.value

    def query(self, key):
        return self.data.get(key)

From simulator to a fair neural benchmark

A proper architecture comparison should hold parameter count, training tokens, optimiser search budget, precision and data pipeline as closely as possible. It should report at least four task families: a sufficient-statistic control, selective copy, associative or random-access recall, and state tracking. Test both interpolation and length extrapolation. Record prefill throughput, decode throughput, peak memory, recurrent state or KV size, and accuracy by query distance.

Two extra controls are essential. First, vary recurrent state size and attention window size to draw a frontier rather than compare one arbitrary point. Second, allocate comparable hyperparameter search, because recent evidence shows recurrent recall can be unusually sensitive to initialisation and learning rate. A benchmark without these controls may measure tuning effort more than memory mechanism.

The practical answer is often a memory portfolio

Pure architectures are useful scientific probes. Production systems have no obligation to preserve their purity. Jamba interleaves attention and Mamba layers; its published configuration used one attention layer for seven Mamba layers, selected for that model's quality and resource objectives rather than as a universal ratio. Based combines linear attention with a small sliding softmax-attention window. These designs retain compact recurrent processing for much of the sequence while reserving addressability where it earns its cost. [10] [9]

A third component is frequently more important than either mixer: explicit retrieval from an authoritative store. Model state is useful for ongoing computation. A recent attention window is useful for local interaction. Neither should be the sole record for a transaction, policy clause, source document or irreversible action.

The components should receive different tests. Recurrent state is tested for stability, reset behaviour and length extrapolation. The attention window is tested for recall by distance, distractor count and adversarial similarity. Retrieval is tested for authority, freshness, entitlement and source coverage. Combining them without separate evidence produces a system whose aggregate score hides the location of failure.

A production-shaped worked scenario

Worked scenario

Consider a synthetic payment-monitoring service. It processes an open-ended event stream. For each account it needs rolling velocity features, recent interaction patterns and the ability to reconstruct exact events when an investigator opens a case.

The rolling features are compressible: counts, decayed totals, last-seen states and learned stream summaries fit a recurrent update. Recent interactions may require flexible comparison across a bounded local window, such as several attempts that differ in merchant, device and amount. Exact case evidence belongs in an event store with identity, valid time, provenance and access control. The model may retrieve those events when a question arrives; it should not claim that its hidden state is the evidence.

The portfolio also improves recovery. Recurrent state can be checkpointed and rebuilt from events. The attention window can be reconstructed from recent records. The evidence store remains authoritative when model versions change. This separation prevents a hidden-state corruption from silently rewriting history.

A decision instrument for sequence memory

Practitioner synthesis

Begin with the query contract, not with a favourite architecture. The following questions turn a vague “long context” requirement into testable choices.

What would weaken this paper's thesis?

Open hypothesis

The memory-geometry thesis would weaken if fixed-size recurrent models, at realistic precision and matched compute, learned robust arbitrary recall across growing candidate sets without increasing state, external memory or effective depth. It would also weaken operationally if exact or sparse attention achieved equal quality with negligible incremental memory and latency on the target deployment.

Neither possibility should be dismissed. State-space research continues to expand the expressivity of recurrence, while attention kernels and sparsity schemes continue to improve. The thesis is therefore a design hypothesis with a discriminating test: vary the number of independently retrievable items while holding the sufficient-statistic complexity fixed. Watch whether quality tracks sequence length, relevant-item count, state size or cache size.

stateDiagram-v2
  [*] --> Claim
  state "The recurrent mechanism" as Claim
  state "A minimal worked example" as Test
  state "Parallel training, recurrent<br/>decoding" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Compact glossary

State-space model

Here, a structured neural sequence layer that updates a latent recurrent state. The broader term has other meanings in control, statistics and reinforcement learning.

Selectivity

Input-dependent control over how sequence information is written, retained, propagated or read.

KV cache

Stored key and value projections for earlier positions used during autoregressive attention decoding.

Sufficient statistic

A compressed state that preserves all information needed for a specified future inference, under stated assumptions.

Associative recall

A task where the model must retrieve a value associated with a key presented earlier in context.

State-space duality

A mathematical framework connecting structured SSM computations with forms of attention and structured matrices.

Choose the memory contract before the mixer

The important question is not whether state-space models have “long memory”. It is which distinctions from the past must remain available to which future queries, at what cost and with what evidence duty.

A recurrent state is an evolving claim about what history still matters. Selectivity makes that claim conditional on the input, allowing rare events, boundaries and useful timescales to control compression. This is a genuine architectural advance over applying one fixed filter everywhere. It explains why modern SSMs can handle discrete, information-dense sequences far better than earlier linear time-invariant layers.

Attention offers another contract. It preserves a set of separately addressable representations and lets each query decide how to combine them. That flexibility costs memory during decoding and compute during dense prefill, though hardware-aware kernels and sparse patterns can move the frontier.

The architecture decision should now be made by stress-testing the memory obligation. Include a sufficient-statistic negative control, a selective-copy task, arbitrary recall, state tracking and length extrapolation. Sweep state size and attention window. Match tuning effort. Measure the actual device. Where exact evidence matters, route back to an authoritative store.

The likely outcome is not a permanent winner. It is a defensible memory portfolio: recurrent state for what can be compressed, addressable attention for what must remain flexibly retrievable, and external evidence for what must remain true, attributable and recoverable.