One state, many writers

The original Transformer placed a residual connection around each attention and feed-forward sublayer. In its published post-normalisation form, a sublayer returned LayerNorm(x + Sublayer(x)). Many interpretability derivations use the pre-normalised layout, where normalisation happens before the sublayer and the identity path remains an explicit addition. The location of normalisation matters for training dynamics and for how literally we may decompose the stream, so the equations below state their architectural assumption rather than treating every transformer as identical.

Let R_l be the residual tensor entering block l, with n token positions and width d. A common pre-normalised decoder block can be written as:

A_l and M_l are writes. They have the same shape as the state they modify. Expanding the additions gives R_L = R_0 + Σ(A_l + M_l). This is useful, but it must be read carefully. The writes are additive at the point of insertion; they are not statistically independent votes. Every later write is a nonlinear function of the state created by earlier writes. A small early change can alter attention patterns, gate different feed-forward features and change normalisation scales.

The word shared has two precise meanings. First, all blocks operate in a compatible d-dimensional address space, so an early module can write a direction that a later module reads. Second, the state spans token positions, so attention can move information between positions while feed-forward modules transform the information available at each position. The residual stream is therefore a tensor-shaped bulletin board, not one global vector floating above the sequence.

Thought experiment one: remove the desk

Imagine replacing every residual addition with pure overwrite: R_{l+1} = F_l(R_l). A block that wants to add one grammatical feature must also reproduce every unrelated feature later computation may need. With the additive path, it can write only the difference it has learned. The causal feature varied is the identity route; attention and feed-forward capacity remain unchanged. We should expect optimisation and information preservation to become harder, although a sufficiently capable overwrite network could still learn the task.

This is why the workspace metaphor is useful. It makes a design burden visible. Each module can specialise because it need not emit a complete replacement representation. It can make a bounded update in a common space. Residual connections were introduced more broadly to ease the training of deep networks; in transformers, the same additive topology also creates a tractable communication surface for mechanistic analysis.

The anatomy of a write

Calling a block a “reader” or “writer” can sound metaphorical until we follow the matrices. For one attention head, queries, keys and values are linear projections of the normalised residual state. The query-key interaction determines a pattern over source positions. The values are mixed according to that pattern and projected back into residual width by the output matrix.

The attention pattern answers where information is drawn from. The product W_V W_O helps determine what transformation is carried from source to destination. The write A is then added to the residual stream. Two heads can look at the same token and write opposing directions. Conversely, two different attention patterns can produce similar writes if the source value vectors are similar. An attention map is evidence about routing weights, not a complete explanation of the message or its causal use.

The feed-forward module has no cross-position mixing in the standard architecture. At each position it reads the current vector, activates an expanded set of intermediate features and projects them back to residual width:

Here φ is the activation or gate. A useful empirical interpretation treats rows of the input projection as pattern detectors and corresponding output directions as value-like writes. Work on feed-forward layers found human-interpretable correlations and showed their outputs being refined through residual connections. That does not license the claim that every neuron is a clean fact slot. Features may be distributed, superposed or meaningful only in combination.

How the final readout turns state into a prediction

At the last token position, a decoder usually normalises the final residual vector and multiplies it by an unembedding matrix. If vectors are written as rows, the vocabulary logits are z_t = Norm_f(R_{L,t})W_U + b_U. Each column of W_U defines a direction that favours one vocabulary token. The difference between two logits is therefore associated with one direction, the difference between their unembedding columns.

This geometry explains the appeal of direct logit attribution. If a module adds write w and the final normalisation is treated as fixed, its direct contribution to the A-versus-B logit difference is the dot product between w and the A-minus-B unembedding direction. Contributions from several writes then add. The calculation is fast, legible and often useful for identifying modules that directly promote or suppress an answer.

It is still a local accounting device rather than a full causal trace. A write may have almost no direct projection onto the output direction yet change what a later head attends to. Another write may directly favour A while also triggering an MLP that suppresses A more strongly. Layer or RMS normalisation couples coordinates through scale, so the exact output effect is not always the raw dot product. Direct effect asks what a write says to the output; total effect asks what downstream computation does after hearing it.

Consider two modules. The first writes +3 units towards Maria. The second reads that enlarged margin as evidence of a repeated-name shortcut and writes -4 units towards Maria. Direct attribution calls the first module pro-Maria and the second anti-Maria. Ablating the first may nevertheless increase the final Maria logit if the suppressor no longer activates. Neither calculation is wrong; they answer different causal questions. A serious analysis reports both the immediate residual contribution and the recomputed downstream effect.

Intermediate lenses add another distinction. The plain logit lens applies the final unembedding to an earlier residual state. It assumes that early and late representations are aligned well enough for the final readout to be meaningful. The tuned lens learns a layer-specific affine translator and has been shown to produce more reliable latent predictions across several autoregressive models. Even then, a readable prediction is an observation about decodability. Patching or another intervention is needed to show that the model actually uses the decoded state.

Worked example one: four coordinates, two writes

Take a minimal answer-position state r₀ = [1, 0, 0, 0]. The first coordinate indicates that candidate A was selected. An attention-like module reads that feature and writes Δ₁ = [0, 0, 1, 0], placing “selected A” into a downstream-readable coordinate. A feed-forward module then reads the third coordinate and writes Δ₂ = [0, 0, 0, 2], where the fourth coordinate is evidence for output A. The final state is r₂ = [1, 0, 1, 2].

If the unembedding reads the fourth coordinate as A’s logit, both writes matter, but in different ways. The first creates a latent intermediate; the second makes it directly output-relevant. Reading the final logit contribution alone would highlight the second write and miss the route that made it possible. Looking only at the attention pattern would highlight the first module’s source and miss the transformation. The residual trace preserves the sequence.

Thought experiment two: rotate the control room

Suppose every residual vector is multiplied by the same orthonormal rotation U. Every reader is adjusted by U^T and every writer by U. The input-output function can remain unchanged while every coordinate acquires a different numerical meaning. Experiments with rotated residual and computation bases have shown that transformers can train and function without depending on one privileged residual coordinate system in the strong sense a neuron-by-neuron story would require.

The causal feature varied is the basis, while the represented linear subspaces and computation are preserved. A named coordinate is therefore weak evidence for a named concept. Directions, subspaces and distributed features can be more stable units, but even a causally effective subspace may be misleading if an intervention activates a dormant alternative route. Interpretability must test the route, not merely find a vector that changes the answer.

flowchart LR
  A["The name that moves without moving"] --> B["One state, many writers"]
  B --> C["The anatomy of a write"]
  C --> D["From viewing to intervention"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#7B4CD8,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

From viewing to intervention

A residual trace becomes explanatory when it supports a counterfactual. Run a model on a clean prompt that produces behaviour A and on a matched corrupted prompt that produces behaviour B. Cache the residual states. During a new corrupted run, replace one state at layer l and position t with the clean value, then let every downstream operation recompute. If the defined metric moves towards A, the patched state carried information that was causally usable under that intervention.

For the Maria-Daniel example, a useful metric is the logit difference logit(Maria) - logit(Daniel). It is continuous and, apart from final normalisation, closely aligned with the additive output geometry. Accuracy alone would hide partial rescue. Probability change can be distorted by unrelated logits because softmax couples the whole vocabulary. The metric must be fixed before the scan, along with the prompt contrast and patch direction.

Here m is the selected behaviour metric. A rescue near one means that this particular replacement recovered the clean contrast. It does not mean the patched location “contains 100 per cent of the concept”, nor that the same route is used across all prompts. The intervention inserts a whole vector, potentially carrying many correlated features. Its effect depends on the surrounding corrupted state and the downstream response.

Worked example two: a production-shaped tracing study

A credible study uses a distribution, not one sentence. Construct several hundred paired prompts with matched token counts and varied names, templates and distractors. In each pair, swap only the grammatical role that determines the correct indirect object. Record clean and corrupted logit differences, tokenisation, confidence and final accuracy. Exclude pairs where the base model does not reliably separate the two answers, because failed behaviour cannot support a clean mechanism contrast.

Begin with a coarse layer-by-position scan of resid_pre, resid_mid and resid_post. This locates where role information is available to downstream computation. Then decompose only the high-effect cells into attention-head outputs and feed-forward writes. Next use path-specific patching to test whether a proposed upstream writer affects the result through the hypothesised downstream reader. Finally, ablate or resample the proposed circuit and assess faithfulness, completeness, minimality and behaviour on held-out templates.

This staged design follows the architecture. The residual scan asks where the useful state flows. Component patching asks who contributed it. Path patching asks which reader-writer edge mattered. Feature analysis asks what basis best describes the transmitted content. Reversing that order invites stories: one can always find an attention head, neuron or sparse feature that correlates with a familiar word.

Why the same patch can support several stories

Activation patching is a causal intervention, but its interpretation remains conditional. Clean-to-corrupt “denoising” and corrupt-to-clean “noising” answer different counterfactuals. Replacing a full residual vector may restore several features at once. Replacing a narrow subspace may place the model off its normal activation manifold. Zero, mean and resample ablations create different downstream situations. A later block may compensate for a removed component, producing the self-repair behaviour described as the Hydra effect.

One especially serious failure mode occurs when a patched subspace changes the output by activating a dormant path that the unmodified model did not use. The intervention is behaviourally effective yet the proposed feature story is false. This is why successful steering, patching or editing cannot by itself establish a faithful mechanism. Route-specific tests, matched resampling, reverse-direction interventions and an explicit account of downstream compensation are needed.

Keep four claims separate

A patching study can accidentally slide through four progressively stronger claims. The first is causal localisation: replacing this activation changed the selected metric for these inputs. The second is content identification: the changed activation represents a named variable such as grammatical role. The third is pathway explanation: a particular upstream module wrote that variable and a particular downstream module read it. The fourth is distributional mechanism: the same route explains the behaviour across the intended task distribution.

Whole-residual rescue supports the first claim. To earn the second, vary the candidate variable while holding plausible confounds fixed, then test whether the same subspace tracks and controls that variable across counterexamples. To earn the third, isolate the writer-reader path and show that bypassing either endpoint removes the predicted effect without relying on an alternative route. To earn the fourth, pre-register held-out prompt families, names, syntax, languages or domains and report where the circuit stops generalising.

The Maria-Daniel task illustrates the gap. A residual patch at the final token may restore the correct name because it carries person identity, grammatical role, lexical frequency and a template-specific position signal together. A linear probe might decode “indirect object” from the same vector. Yet the model could actually use a simpler repeated-name detector, with the role probe reading a correlated by-product. A counterexample with three names, passive voice or an intervening clause can separate these explanations.

Causal scrubbing offers one way to formalise such a hypothesis. State which input distinctions each internal node is allowed to depend on, then resample activations while preserving the distinctions the interpretation claims are relevant. If behaviour survives the scrubbing, the abstraction gains support. If it fails, either the proposed variables are incomplete or the route is wrong. The important shift is from naming an activation after inspection to specifying an executable model of what information each node must preserve.

Finally, causal localisation is asymmetric. Restoring a clean state inside a corrupted run tests whether clean information can rescue that context. Corrupting one state inside a clean run tests whether the clean computation depends on that state under the resulting disturbance. Self-repair, saturation and nonlinear routing can make the magnitudes differ. Reporting only the more dramatic direction selects an answer after seeing the result.

Where the workspace metaphor breaks

The metaphor earns its keep only if it prevents category errors. The residual stream is transient activation state inside a forward pass. The context window is the sequence supplied to the model. A key-value cache stores projected keys and values from earlier generated tokens to avoid recomputation. External memory stores information across runs. These objects can interact, but calling all of them “memory” hides their authority, lifetime and failure modes.

The stream is also not a passive ledger. Layer normalisation changes the scale and, in some layouts, the state itself. Attention readers can alter their routing after a patch. Feed-forward gates can switch sharply. Later writes can cancel earlier ones. Superposition allows many features to occupy overlapping directions. The additive equation describes how updates enter; it does not promise that every update remains separately recoverable.

Pre-normalised and post-normalised models also give a patch different semantics. In a pre-normalised block, replacing R_l inserts a state on an explicit identity path and every later module sees a normalised version of that hybrid. In the original post-normalised layout, the sum is immediately normalised before becoming the next state. A patch taken before normalisation, after normalisation or after the whole block therefore changes different objects. Hook names such as resid_pre and resid_post must be tied to the implementation’s actual equations.

This matters when comparing models. A large rescue at “layer 12” in two architectures need not indicate the same computational stage. Depth fractions, block boundaries and norm placement differ. Even within one family, tied embeddings, parallel branches, residual scaling and final normalisation can change direct-logit accounting. Mechanistic results should name tensor shape, position convention, normalisation state and patch timing, not only a layer number.

Architecture variants further narrow the claim. Post-normalised blocks do not expose the same unbroken additive identity path as the pre-normalised equations above. Residual scaling, parallel attention and MLP branches, gated residuals, mixture-of-experts routing, recurrent hybrids and emerging multi-stream or hyper-connection designs alter who shares which state. “The residual stream” should always name a concrete tensor and update rule in the model being studied.

A serious negative control: the terminal shortcut

Build a model that solves the Maria-Daniel task by encoding the answer directly in the final token embedding and letting the last feed-forward module read it. Middle attention heads may still produce structured patterns, and their residual outputs may correlate with the answer, but the terminal shortcut makes them unnecessary. A coarse residual scan should show no selective rescue through the proposed mid-layer route. If the analysis nevertheless “discovers” a rich attention circuit, the method is fitting an attractive story to epiphenomenal activity.

This control varies the mechanism while preserving behaviour. It is stronger than patching an arbitrary irrelevant neuron. A method that distinguishes the genuine routed model from the shortcut model has earned some construct validity. One that returns the same explanation for both has not.

Two meanings of workspace

A recent primary study used a Jacobian-based lens to identify a middle-layer subspace whose contents could be verbalised, deliberately modulated and used across downstream computations. The authors describe functional parallels with global workspace theory and explicitly treat philosophical implications as unsettled. That work offers a stronger, empirically testable meaning of “workspace”: a privileged subset of representations with broad accessibility.

The present thesis is narrower. Standard residual architecture creates a shared communication substrate even if no privileged, verbalizable subspace exists. The architectural claim is about read and write topology. The functional-global-workspace claim is about which contents become broadly accessible and controllable. Evidence for the second may build on the first, but the first cannot establish the second. Neither, on its own, establishes subjective experience.

A tracing laboratory you can run

The accompanying notebook implements a deterministic residual-routing model in NumPy. It is deliberately smaller than a transformer so every state and write can be inspected. Four token positions share six features. An attention-like selector reads candidate identity at one position and writes selected identity at the answer position. An MLP-like compiler turns that state into output evidence. A final writer changes both logits equally, proving that absolute logit movement need not change the answer contrast.

The clean run selects A and has logit difference +2.00. The corrupted run selects B and has -2.00. Patching the clean residual at the answer position immediately after selection restores +2.00. Patching only directions orthogonal to the selected-identity subspace, or patching the distractor position, leaves the corrupted result unchanged. Assertions turn these expectations into executable positive and negative controls.

Inspect the implementation in this paperInspect the implementation in this paper

def forward(candidate):
    r = embed(candidate)
    states = [r.copy()]
    for writer in (selector_write, compiler_write, suppressor_write):
        r = r + writer(r)
        states.append(r.copy())
    return r[3, 4:6], states


def forward_with_patch(base_candidate, source_states,
                       patch_layer, patch_position, projection=None):
    r = embed(base_candidate)
    if patch_layer == 0:
        r = patch(r, source_states[0], patch_position, projection)

    for layer, writer in enumerate(WRITERS, start=1):
        r = r + writer(r)
        if layer == patch_layer:
            r = patch(r, source_states[layer], patch_position, projection)
    return r[3, 4:6]

# Expected checks
clean logit difference:    +2.00
corrupt logit difference:  -2.00
positive patch:            +2.00
orthogonal negative patch: -2.00
irrelevant-position patch: -2.00

Scaling the laboratory to a real model

A full layer-by-position scan costs one or more forward passes per patch location. For a model with L layers and n positions, naive scanning is roughly proportional to Ln patched runs for each prompt pair and hook type. Batched patching, cached clean activations and gradient-based attribution approximations can reduce exploratory cost, but approximations should be checked against exact patches on a representative subset. Speed does not repair a mismatched counterfactual.

Store activations at the smallest precision that preserves the effect under study, and record the transformation applied before storage. Quantisation, device casts and folded normalisation can change exact vectors. When privacy matters, residual activations should be treated as sensitive derived data because they may encode prompt content and model-internal properties. The trace store needs retention, access and deletion rules separate from ordinary application logs.

For each experiment, save a compact receipt: model and tokenizer hashes, prompt-pair hash, generation settings, hook name, tensor index, source activation identity, patch rule, metric code version and output logits. A result without these fields is difficult to replay after a model upgrade. The objective is not to retain every activation forever; it is to make consequential mechanism claims reconstructable.

Real-model analysis should end with an intervention budget. Coarse residual scans are suitable for locating a route. Fine feature or path tests are expensive and should be reserved for hypotheses that would change a decision, such as removing a shortcut, validating a safety monitor or determining whether an edit affects one capability or many. Interpretability earns operational value when its next experiment is chosen by decision consequence, not by visual novelty.

stateDiagram-v2
  [*] --> Claim
  state "Thought experiment one, remove<br/>the desk" as Claim
  state "How the final readout turns<br/>state into a prediction" as Test
  state "Worked example one: four<br/>coordinates, two writes" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Compact glossary

Residual stream

The evolving activation tensor to which transformer sublayers add their outputs. In a decoder it usually has one d-dimensional vector per token position.

Write

An attention or feed-forward output projected to residual width and added to the current state.

Activation patching

Replacing an internal activation in one run with an activation from another run, then recomputing downstream operations to measure a counterfactual effect.

Logit difference

The difference between two selected output logits, often used as a continuous task metric.

Decodability

The ability of a probe or lens to recover information from a state. Decodability does not by itself show that the model uses that information.

Self-repair

A downstream compensatory response that partially restores behaviour after a component is ablated or replaced.