TLDR

1. Unmasked self-attention is permutation equivariant: shuffling tokens shuffles its token-level outputs unless another mechanism supplies order.
2. Absolute embeddings label locations, relative biases label displacements, RoPE rotates query-key subspaces, and ALiBi adds a directional distance prior.
3. A position formula that accepts larger indices does not guarantee useful behaviour at lengths, offsets or distance distributions absent from training.
4. Causal masks are a serious negative control because deep autoregressive Transformers can infer useful position even without an explicit positional encoding.
5. Choose and validate positional geometry through permutation, global-shift, distance-stretch, direction and out-of-range interventions tied to the deployed task.

Order begins by breaking a symmetry

Attention starts with a useful freedom. For each token, it constructs a query, a key and a value. A query asks what information would help this token; keys advertise what each token can match; values carry the information to mix. Because every token can be processed in parallel, the mechanism does not have to walk left to right as a recurrent network does. That parallelism helped make the original Transformer compelling.[1]

The freedom has a price. Pairwise content matching alone does not make an ordered sequence. Suppose the input matrix X contains one row per token. One attention head computes:

Nothing in that expression names “first”, “previous” or “seven places away”. A token can match another because their projected content aligns. If two inputs contain the same rows in a different order, the same pairwise relations are present in a correspondingly shuffled matrix.

The permutation proof in one line

Let P be a permutation matrix that reorders token rows. Replacing X with PX permutes queries, keys and values in the same way. Pairwise scores become P(QKᵀ)Pᵀ. Row-wise softmax preserves that coordinated permutation, so the final output is P·Attention(X). In plain language, shuffle the inputs and the token-level outputs shuffle with them. This property is permutation equivariance.

Equivariance is not the same as invariance. A pooled classifier can become invariant because it discards token locations after the equivariant layers. A token-level model retains a shuffled set of outputs. Neither arrangement, by itself, says which order is the intended sentence.

Thought experiment one: two worlds with the same pairwise content

Imagine a bidirectional encoder receives three unique vectors, A, B and C. In world one, the correct label depends only on whether all three are present. In world two, the label depends on whether A precedes B. Keep the tokens, projections and attention weights otherwise identical.

World one needs no order. Adding a rich positional system may consume capacity or encourage accidental location shortcuts. World two is unsolvable for a pooled, position-free encoder because every permutation presents the same content set. The need for positional information belongs to the task symmetry, not to the word “Transformer”. This distinction prevents a common mistake: treating positional encoding as compulsory decoration rather than an inductive bias whose value must be earned.

The serious negative control: a causal mask already carries order

The clean permutation argument applies to unmasked self-attention. Autoregressive decoders add a triangular causal mask: token i may attend only to tokens at positions j ≤ i. That mask changes with row index. The first token sees one predecessor including itself; the tenth sees ten. Even when no explicit positional vector is added, the pattern of available context differs by location.

Haviv and colleagues trained causal language models without explicit positional encodings and found them competitive with standard counterparts across their tested datasets, sizes and lengths. Probes recovered implicit absolute position, and the authors proposed predecessor count through the causal mask as a source.[7] Later work sharpened the architectural point: a deep cascade of set-like operations can become sequence-sensitive in the autoregressive setting, although a single layer lacks the same capacity.[13]

flowchart TD
  A["The sentence that disappears<br/>when the cards are shuffled"] --> B["Order begins by breaking a<br/>symmetry"]
  B --> C["Four ways to create geometry"]
  C --> D["Why formula length is not<br/>usable length"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#B96D00,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Four ways to create geometry

Position can enter before attention, inside its score, through a transformation of queries and keys, or through the mask. These placements are not cosmetic. They change what content and position can interact with, which symmetries are exact, and how the model encounters unseen distances.

1. Absolute coordinates: tell every token where it is

The original Transformer added a positional vector pᵢ to each token embedding xᵢ before the first layer. Its fixed sinusoidal version used sine and cosine pairs at geometrically spaced frequencies. The paper also tested learned positional embeddings and reported nearly identical translation results in that setting.[1]

A useful intuition is a bank of clocks. One hand turns quickly, another slowly. The joint pattern identifies a coordinate within a large range. Nearby positions have nearby phases in slow dimensions and more distinct phases in fast ones. Because sine and cosine shifts can be expressed through linear combinations, relative offsets are recoverable in principle.

The analogy has limits. A learned attention head sees content-plus-position after arbitrary projections and nonlinear layers, not a clean clock face. Cross terms can entangle token identity with absolute location. A global shift from positions 0 to 511 into positions 8,000 to 8,511 may therefore change behaviour even though all pairwise distances are preserved. Learned lookup embeddings have an even sharper boundary: positions outside the table are undefined unless the table is resized, interpolated or otherwise extended.

2. Relative representations: label the relation, not the address

Shaw, Uszkoreit and Vaswani modified self-attention so a pair could carry a learned representation of its relative displacement. On their WMT 2014 translation experiments, the relative method improved by 1.3 BLEU for English to German and 0.3 BLEU for English to French compared with their absolute baseline; adding both gave no further improvement in that experiment.[2] Those numbers are historical results from one architecture and task family, not a universal ranking.

The enduring idea is more general. When the score depends on i−j, shifting every token by the same constant leaves that term unchanged. This is translation equivariance over sequence coordinates. Transformer-XL used a relative formulation partly to preserve temporal coherence when reusing hidden states across segments.[3] T5 simplified relative information into learned scalar biases, grouping large distances into buckets so many far offsets share a parameter.[4]

Relative geometry suits tasks where “two tokens earlier” means roughly the same thing anywhere in the sequence. It can be a poor fit when absolute zones matter, such as a form whose header, signature block and appendix carry different roles. Bucketing also introduces deliberate resolution loss. Once two distances enter the same bucket, the bias cannot distinguish them, though content and later layers still may.

3. RoPE: turn index into angle, then compare rotated content

Rotary position embedding, or RoPE, applies a position-dependent rotation to pairs of query and key coordinates before their dot product. Each two-dimensional pair uses a different angular frequency. The rotation is norm preserving, so it changes orientation rather than vector length. Most importantly, the product of two rotations depends on their angle difference.[5]

This identity explains RoPE's elegant geometry. A global shift adds the same angle to query and key, which cancels in their relative rotation. Different frequency bands give the head access to fine and coarse phase relationships. Yet periodic components also create aliasing-like effects, and extrapolation can place relative phases outside the distribution the model learned to use.

One popular explanation says RoPE succeeds because attention naturally decays with distance. That statement is too strong. Barbero and colleagues showed mathematically that, for a chosen offset, a compatible query can make the RoPE score maximal; their mechanistic study found a trained model using high frequencies for robust positional patterns and preferring low frequencies elsewhere. They argued that monotonic distance decay is unlikely to be RoPE's core mechanism.[11] RoPE enables offset-selective structure; it does not impose a simple universal recency curve.

Minimal worked example: one quarter-turn separates identical content

Take a two-dimensional query q=(1,0) and two identical keys k₁=k₂=(1,0). Without positional information, both dot products equal 1. The head cannot prefer one identical key over the other.

Use a rotary frequency of a quarter-turn per token. Put the query at position 2, key one at position 1 and key two at position 0. Their relative rotations are minus one quarter-turn and minus two quarter-turns. The first score is cos(π/2)=0; the second is cos(π)=−1. Identical content now receives different scores because the offsets differ.

Shift every position by 100. The offsets remain −1 and −2, so the scores remain 0 and −1. Stretch the gap instead, moving the query one more step while leaving the keys fixed. The phases change. The intervention exposes exactly what RoPE preserves: common translation disappears, relative displacement remains causally active.

4. ALiBi: make distance a prior in the logit

Attention with linear biases, or ALiBi, leaves token embeddings untouched and adds a head-specific linear penalty to causal attention logits. If query i looks back to key j, a head with slope mₕ subtracts mₕ(i−j). Several slopes create heads with different effective ranges.

Press, Smith and Lewis reported that a 1.3 billion parameter model trained on length 1,024 extrapolated to 2,048 with the same perplexity as a sinusoidal model trained at 2,048, while their ALiBi training used 11 per cent less time and memory in that setup.[6] The mechanism is transparent: a linear function is defined at any distance and directly encourages recency.

That prior can also be wrong. A distant governing clause, an early variable declaration or a far-away table header may remain decisive. A steep slope forces content similarity to overcome a growing positional penalty. ALiBi trades a strong, extrapolatable locality bias for reduced neutrality about long-range evidence. The trade can help language modelling while harming tasks where arbitrary-distance retrieval or exact global structure matters.

Why formula length is not usable length

A sinusoid, rotation or linear bias can be evaluated at position one million. That is a mathematical property of the function. It says nothing by itself about whether a trained network will use the resulting phases, offsets and logits correctly. A formula may extrapolate while the learned procedure does not. One recurring failure arises when deployed positional relations leave the support of the relations that shaped that procedure.

Thought experiment two: move the same room, then lengthen the corridor

Train a model on documents whose relevant sentence always appears between positions 100 and 300. First intervention: prepend 5,000 neutral tokens to every document, shifting the entire relevant region while preserving all internal distances. Second intervention: insert 5,000 neutral tokens between a question and its supporting sentence, preserving their absolute neighbourhoods on each side while increasing their separation.

An absolute scheme may struggle with the first intervention because the coordinate origin changed. A relative scheme can pass the shift test yet fail the corridor test because the distance is unseen, bucketed, heavily penalised or mapped to unfamiliar rotary phases. These are different causal questions. Calling both “long context” hides the mechanism.

Now change direction. Place the evidence after the question in a bidirectional encoder, then before it. A symmetric distance function treats |i−j| the same; a signed relative function distinguishes past from future; a causal decoder forbids the future case entirely. Direction is a third axis, independent of coordinate and distance.

What the evidence establishes, and what it does not

Research results vary because the task, mask, architecture, training range and metric vary. ALiBi's original evidence concerned language-model perplexity under a two-times length extrapolation in a particular setup.[6] Kazemnejad and colleagues compared five schemes on decoder-only Transformers trained from scratch over reasoning and mathematical tasks. In that battery, NoPE often outperformed explicit absolute, T5-style, ALiBi and rotary methods.[8] The result is an important counterexample to universal claims, but it does not prove that NoPE dominates large pretrained models or every sequence task.

Ruoss and colleagues linked failures to out-of-distribution positional encodings and randomised ordered position subsets during training. Across 6,000 trained models on 15 algorithmic tasks, their method improved unseen-length test accuracy by 12.0 percentage points on average.[9] This supports a causal lesson broader than the specific method: expose the model during training to the positional relations it must later handle, rather than assuming a smooth formula forces a systematic algorithm.

Chen, Varoquaux and Suchanek studied bidirectional masked language models and identified locality and symmetry patterns correlated with sentence-representation performance. Their swap probes also found models robust to local swaps yet fragile to global swaps, revealing that common encodings may encode a coarse neighbourhood prior without reliably representing semantic-role changes.[10] Their scope matters: the study deliberately separated BERT-style encoders from causal decoders because masks alter the positional problem.

Long-context benchmarks reinforce the distinction between accepted input length and usable evidence. RULER configured retrieval, multi-hop tracing and aggregation tasks across length and complexity. In its evaluation of 17 long-context models, almost all degraded as length increased; among models claiming at least 32K context, only half met the benchmark's satisfactory threshold at 32K.[12] Position is only one cause, but a declared window cannot substitute for task-level measurement.

Three debts hidden by a single “context length”

Coordinate debt appears when a model depends on absolute regions. A global shift changes predictions even though content and relative relations are preserved. Learned absolute tables expose this clearly, but sinusoidal additive schemes can also entangle content with origin through projections.

Distance debt appears when the model encounters offsets unlike those in training. Relative buckets may collapse them, ALiBi may over-penalise them, and RoPE may rotate features into phase relationships the network has not calibrated. A scheme can be exactly translation equivariant and still accumulate distance debt.

Procedure debt appears when the model learned a finite-range shortcut rather than the intended algorithm. For addition, copying or bracket matching, the positional system may let the model memorise training-size patterns. Task-aligned positions can change the problem dramatically. Position coupling, for example, assigns the same identifier to digits with the same significance, directly exposing arithmetic structure rather than raw token index.[14]

The failure boundary

Positional geometry cannot rescue missing content, insufficient attention capacity, poor optimisation, broken tokenisation or a task whose real structure is not a line. A source-code repository contains scope, syntax trees, files and dependency edges. A document contains pages, tables, headings and cross-references. An image has two-dimensional translation and orientation. Flattening each into one token index and then perfecting a one-dimensional encoding may preserve the wrong relations.

Nor does relative geometry always deserve preference. Absolute location can be causal: the first row of a table may define a schema; a signature at the end of a contract has a role; a sensor sample's wall-clock time can matter beyond pairwise lag. The mechanism ceases to help when its invariance erases a distinction the task needs.

stateDiagram-v2
  [*] --> Claim
  state "Figure 1. A token sequence<br/>becomes four positional<br/>geometries" as Claim
  state "The permutation proof in one line" as Test
  state "Figure 2. Content attention<br/>follows the cards" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Choose by intervention, not fashion

A production decision should start with the relations the model must use, then test candidate schemes under controlled changes. Model size, training data and optimiser can overwhelm small architectural differences in aggregate benchmarks. An intervention holds content constant while changing one positional property. That makes the failure interpretable.

A production-shaped worked scenario: the distant exception

The naive evaluation asks whether the model accepts 24,000 tokens and retrieves the exception once. The positional evaluation asks a sequence of harder questions. Does the answer change if the whole pack is preceded by an unrelated appendix, preserving the rule-exception distance? Does it change if neutral material is inserted between rule and exception? Does moving the exception earlier alter its authority even when metadata says it is still the superseding clause? Can the model handle two exceptions at equal distance on opposite sides in a bidirectional setting? Does a missing exception produce abstention rather than a confident general-rule answer?

An ALiBi head with a steep slope may require exceptionally strong content similarity to recover the distant exception. RoPE computes a relative phase for the offset, but the trained model may not have learned reliable use at that phase range. T5-style buckets may treat 18,000 and 22,000 tokens as the same positional class, which may be acceptable if exact distance is irrelevant. An absolute scheme may use early versus late document zones, useful for headers and appendices but risky if layout changes.

The architecture decision may therefore be hybrid. Use relative geometry for textual dependency, structured metadata for clause identity and effective time, retrieval to reduce irrelevant distance, and deterministic supersession logic after evidence extraction. Position encoding should carry only the positional burden that remains after better representations and system boundaries are chosen.

Run the encoding lab

The laboratory below holds token content fixed and changes only positions. Two keys share identical content, so a position-free score gives them equal weight. The global-shift slider adds the same constant to every position. The gap slider inserts synthetic neutral distance before the query without adding content, isolating the geometry rather than modelling a full language task.

Expected positive case: RoPE and ALiBi should show near-zero global-shift error because their pairwise scores depend on displacement. Expected negative case: NoPE should assign equal weights to the duplicate keys. Sinusoidal additive attention generally changes under a global shift because content and absolute phases mix before projection.

import numpy as np

D = 4
TOKENS = np.array([
    [1.0, 0.2, 0.4, -0.1],
    [0.6, 0.8, -0.2, 0.3],
    [0.1, 0.7, 0.5, -0.4],
    [0.6, 0.8, -0.2, 0.3],  # identical content to token 1
    [0.3, -0.1, 0.9, 0.7],  # query
])
BASE_POS = np.array([0, 1, 3, 4, 6], dtype=float)
WQ = np.array([[0.7, -0.2, 0.1, 0.3],
               [0.1, 0.8, -0.4, 0.2],
               [-0.3, 0.2, 0.9, 0.1],
               [0.2, 0.1, 0.3, 0.7]])
WK = np.array([[0.6, 0.1, -0.2, 0.4],
               [-0.2, 0.7, 0.3, 0.1],
               [0.2, -0.3, 0.8, 0.2],
               [0.4, 0.2, 0.1, 0.6]])


def softmax(x):
    z = x - np.max(x)
    e = np.exp(z)
    return e / e.sum()


def sinusoid(pos):
    freq = np.array([1.0, 1.0 / 100.0])
    out = np.empty(D)
    out[0::2] = np.sin(pos * freq)
    out[1::2] = np.cos(pos * freq)
    return out


def rope(vec, pos):
    freq = np.array([1.0, 1.0 / 100.0])
    out = vec.copy()
    for r, omega in enumerate(freq):
        a, b = vec[2*r], vec[2*r + 1]
        c, s = np.cos(pos * omega), np.sin(pos * omega)
        out[2*r] = c*a - s*b
        out[2*r + 1] = s*a + c*b
    return out


def weights(scheme, shift=0.0, gap=0.0):
    pos = BASE_POS + shift
    pos[-1] += gap
    qx, keys = TOKENS[-1], TOKENS[:-1]
    qpos, kpos = pos[-1], pos[:-1]

    if scheme == "sin":
        q = (qx + sinusoid(qpos)) @ WQ
        k = np.stack([(x + sinusoid(p)) @ WK for x, p in zip(keys, kpos)])
        scores = k @ q / np.sqrt(D)
    else:
        q = qx @ WQ
        k = keys @ WK
        if scheme == "rope":
            q = rope(q, qpos)
            k = np.stack([rope(v, p) for v, p in zip(k, kpos)])
        scores = k @ q / np.sqrt(D)
        if scheme == "alibi":
            scores -= 0.12 * (qpos - kpos)
    return softmax(scores)


for scheme in ("none", "sin", "rope", "alibi"):
    base = weights(scheme, shift=0)
    shifted = weights(scheme, shift=17)
    print(f"{scheme:>6} shift L1 = {np.abs(base-shifted).sum():.6f}")

assert np.allclose(weights("rope", 0), weights("rope", 17), atol=1e-10)
assert np.allclose(weights("alibi", 0), weights("alibi", 17), atol=1e-10)
assert not np.allclose(weights("sin", 0), weights("sin", 17), atol=1e-4)
assert np.isclose(weights("none")[1], weights("none")[3], atol=1e-12)
assert not np.isclose(weights("rope")[1], weights("rope")[3], atol=1e-4)
print("All positional intervention checks passed.")

The positional audit

Use the following tests before comparing headline benchmark scores. Construct matched examples, keep content and labels fixed where possible, and report both average performance and the change caused by each intervention. A scheme earns adoption when its failures align with task irrelevance, not when it wins an unrelated league table.

What to log during training and release

Track the distribution of absolute positions, signed offsets and attention distances sampled during training. For bucketed methods, report bucket occupancy; for RoPE, report the target length and any frequency scaling; for ALiBi, preserve head slopes; for absolute tables, preserve maximum index and resizing method. These are part of model lineage because changing them changes the model's geometry even when weights elsewhere remain constant.

At release, plot task quality against length and intervention type, not length alone. Separate retrieval, aggregation, exact ordering and missing-evidence cases. Test at several offsets inside and beyond the training range. Include a content-only or NoPE control where architecture permits. A model that degrades gracefully and abstains can be preferable to one with a larger accepted window and brittle positional shortcuts.