For a full self-attention layer without positional information, permuting the rows of the input permutes the output rows in the same way. In compact notation, if P is a permutation matrix and A is the attention operation, then:

Vaswani and colleagues introduced fixed sinusoidal encodings because the Transformer removed recurrence and convolution, both of which had carried order through their computation. They also tested learned absolute embeddings and reported similar translation results in their setting.[1] Shaw, Uszkoreit and Vaswani then moved positional information into pairwise attention relations, improving the studied translation tasks over the absolute baseline.[2] These are more than two implementations of one label. They preserve different relations.

Preservation is a commuting test

There is a compact way to express the issue without turning the paper into abstract algebra. Let T_c shift every sequence index by the same integer c. Let S(i, j) be the positional contribution to the score between two tokens. A shift-invariant comparator satisfies S(i + c, j + c) = S(i, j). The intervention can be performed before or after computing the relation and the result agrees. This is the practical meaning of a commuting test.

Do not confuse invariance with equivariance. If the output is a sequence of labels, shifting the input may be expected to shift the labels to corresponding positions. That is equivariance. If the output is a single answer such as “Alice trusts Bob”, shifting the sentence may be expected to leave the answer unchanged. That is invariance. The same positional mechanism can support both, depending on the downstream readout. What matters is whether the internal relation transforms predictably.

RoPE obtains its relative law because its rotations form a representation of integer translation: applying position i and then an offset c is equivalent to applying position i + c. Relative biases satisfy the same shift test by using j − i directly. Learned absolute tables do not generally satisfy it because the row at i + c has no enforced relation to the row at i. Fixed sinusoids do possess a structured shift operation at the carrier level, yet addition to content and learned projection can prevent that structure from surviving into the score.

The preservation test must be applied where the claim is made. Carrier algebra does not automatically survive the comparator. Comparator algebra does not automatically survive nonlinear layers and task heads. This is why the three-layer contract is more informative than naming an encoding family.

flowchart LR
  A["When a location tag becomes a<br/>causal assumption"] --> B["Three ways of keeping order"]
  B --> C["Why the clocks fail beyond the<br/>horizon"]
  C --> D["Build and test the positional<br/>contract"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#007D68,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Absolute position: name the slot

The most direct method gives each index i a vector p_i and adds it to the token representation x_i. The layer receives z_i = x_i + p_i. A learned table can assign a distinctive address to each trained slot. This is useful when position 0, position 1 or a fixed grid cell has a stable role.

But addition mixes what and where before queries and keys are projected. Expanding a dot product produces content-content, content-position, position-content and position-position terms. Unless the learned projections acquire a special structure, shifting every token changes more than an address. It changes the cross-terms through which content is compared. Absolute addition offers positional identity, not guaranteed translation equivariance.

Fixed sinusoidal encodings replace a finite table with paired sine and cosine functions at geometrically spaced wavelengths. Each position is read on several clocks, from fast-turning to slow-turning. For a fixed offset k, the code at i + k can be obtained by a linear transformation of the code at i.[1] This gives the model a useful algebraic route to relative offsets. It does not force the full attention comparator to use that route, because content and position remain added before arbitrary projections.

Relative position: name the displacement

A relative method asks a different question. Instead of “where is token j?”, it asks “how far and in which direction is j from i?” Shaw and colleagues inserted learned relative representations into the attention compatibility and value computations.[2] Later designs used scalar biases, buckets or disentangled content-position terms. Transformer-XL combined a relative scheme with segment recurrence.[3] T5 used learned relative position biases with distance buckets.[4] DeBERTa separated content and position vectors inside attention.[5]

A relative comparator therefore preserves a relation while discarding an origin. That can be exactly right for syntax, music intervals and local causal dependencies. It can also be wrong for sequence headers, absolute time or tasks where the first occurrence has a special status. Distance buckets introduce another contract: retain fine distinctions nearby, then compress distant positions. This is not a defect if the task only needs “far away”. It is a failure if 513 and 4,096 tokens apart must remain different.

What each representation chooses to forget

A useful encoding is selective. Learned absolute position can remember that an item occupied slot 7 even when its neighbours move, but it does not supply a principled relation between slot 7 and an unseen slot 70,000. A relative bias remembers that one token is three places before another, but discards whether the pair began at slot 7 or slot 700. A clipped relative bias forgets the difference between every distance beyond its threshold. ALiBi retains a monotone ordering of distances in its bias but reduces position to one scalar preference for nearness. RoPE retains directional phase differences across several scales while discarding the common origin.

This gives a sharper design vocabulary than “has position” or “does not have position”. Ask whether the model must distinguish direction from magnitude, exact distance from distance class, or local order from absolute calendar time. A relative function based only on |j − i| cannot distinguish before from after. A signed bias can. A periodic phase can distinguish many offsets but may assign similar scores again later. A learned address can distinguish every trained slot while offering no compositional rule for shifting them.

The distinction also explains hybrid designs. DeBERTa uses relative interactions through its layers while reintroducing absolute position for masked-token decoding.[5] The architecture is saying that relational context should dominate representation building, while absolute location can still help a particular prediction interface. Hybridisation is justified when separate computations genuinely need separate invariants. It is not justified merely by concatenating every available positional signal.

Rotary position: compare phase

Rotary Position Embedding, or RoPE, takes an elegant middle route. It assigns an absolute rotation to each query and key, but their dot product depends on the difference between the two rotations. In each two-dimensional subspace, position i rotates a vector by iθ. Across the model dimension, different pairs use different frequencies.[6]

Length extrapolation sounds like one question: can the model accept more tokens? It is at least four. Can the positional mechanism produce representations for new indices? Do attention scores remain numerically and statistically stable? Can the model retrieve relevant content from the larger field? Can it execute the required computation over a longer structure? A “yes” to the first does not entail the others.

Defined is not familiar

A learned absolute table has an obvious boundary. Row 4,097 does not exist if training allocated 4,096 rows. A sinusoidal or rotary formula can generate position 4,097, but that only solves addressability. The trained projections may see angles, cross-terms or score magnitudes outside their experience. A clipped relative bias continues to run but may map every sufficiently distant token to the same bucket. ALiBi continues its line, but the penalty grows with distance and therefore encodes a stronger recency prior as the window expands.

Extrapolation is not the ability to calculate a coordinate; it is the ability to preserve a task-relevant relation under a distribution shift in geometry. That distinction explains why methods can look excellent on language-model perplexity yet fail on algorithmic tasks, or retrieve a passkey while mishandling dependencies among several distant items.

YaRN adjusted frequency scaling and attention magnitude to extend RoPE-based models more efficiently than earlier approaches in its reported settings.[10] LongRoPE searched non-uniform interpolation factors and used progressive extension to report contexts up to two million tokens on selected models and tasks.[11] These methods show that the frequency spectrum is an engineering surface, not sacred geometry. They also reinforce a boundary: extending the positional substrate is one component of long-context capability, alongside data, optimisation, attention computation and evaluation.

A numerical view of compressed resolution

Take a RoPE model pretrained to length 4,096 and extend it to 32,768 by linear interpolation. The scale factor is eight. Deployment positions 8,000 and 8,008 are presented to the positional mechanism as 1,000 and 1,001. Their old-scale separation is one. Deployment positions 8,000 and 8,001 are presented as 1,000 and 1,000.125. They remain distinct in continuous arithmetic, but the angular difference in every rotary plane is one eighth of what the pretrained model associated with neighbouring tokens.

That compression has two opposed effects. It keeps all angles inside the range experienced during pretraining, reducing the shock of novel phase magnitudes. It also changes the scale at which local relations appear. High-frequency planes now turn more slowly per deployment token. A model adapted on long sequences may relearn how to interpret the compressed steps; a model used without adaptation may not. Non-uniform methods such as YaRN and LongRoPE treat frequencies differently precisely because a single scale factor need not preserve both local resolution and long-range stability.[10][11]

Now reverse the intervention. Keep the old angular step and directly extrapolate to 32,768. Local resolution remains familiar, but the fastest planes complete many additional cycles and the slowest planes enter regions not represented in training. Neither policy is free. Interpolation changes the meaning of a unit; extrapolation changes the range of phases. A configured benchmark should test both consequences rather than treating scale as a tuning constant.

The useful disagreement around NoPE

A decoder-only Transformer with a causal mask has an asymmetry even without explicit position vectors. Token i can attend to a prefix of length i, so depth and attention patterns can reveal order indirectly. Kazemnejad and colleagues found NoPE outperforming several explicit schemes on their battery of length-generalisation tasks and argued that it often learned T5-like relative patterns.[8] Later arithmetic work showed that positional descriptions and data formatting can radically alter extrapolation, and that strong results are not robust across task formulations.[12][13]

stateDiagram-v2
  [*] --> Claim
  state "The shuffled telegram" as Claim
  state "Preservation is a commuting test" as Test
  state "“Only Alice trusts Bob”" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

The engineering question is usually phrased as “Which positional encoding is best?” That question has no task-independent answer. A better process begins by writing the transformations that should preserve the output, the distinctions that must remain visible and the deployment horizon over which both claims must hold.

The five tests

The smallest useful assessment can be synthetic. It need not claim to reproduce a large language model. It should expose the positional logic that prose tends to hide.

A controlled experiment that can lower the thesis

The proposed contract becomes scientifically useful only if it can fail. Build four synthetic task families with the same vocabulary and model size. In the absolute-anchor task, the label depends on the token at a named slot. In the offset-copy task, the label depends on a token exactly k positions back. In the ordered-update task, several amendments must be applied in sequence. In the periodic-marker task, the correct output depends on phase within a repeating structure. These tasks require different preserved relations by construction.

Train each positional scheme on lengths 8 to 32 using identical architectures, parameter budgets, batches and random seeds. Evaluate ordinary held-out examples, globally shifted examples within length 32, and length extensions at 64, 96 and 128. Record task accuracy, shift consistency, attention-logit range, distance-confusion matrices and failure by relative offset. Then repeat with position interpolation and with a retrieval system that moves relevant evidence into a short window. Retrieval is a separate configured-system condition, not an invisible rescue.

The contract hypothesis predicts an interaction, not a universal ranking. Absolute position should have an advantage when named slots are causal. Relative or rotary methods should resist origin shifts on offset tasks. A monotone distance bias may help local copying and hurt remote ordered updates. Rotary methods may handle moderate displacement richly while showing periodic confusions at selected ranges. If one method wins every task and intervention consistently, the task-contract account is weakened. If in-range perplexity predicts all out-of-range outcomes as well as the contract measures do, the proposed decision instrument adds little.

The smallest publishable result is therefore a matrix of task requirement by intervention, with confidence intervals over seeds and a declared null policy. This design also exposes the rival explanation. If a scheme wins only because it optimises faster, matching training loss or compute should reduce the difference. If the advantage remains specifically under the transformation it preserves, the symmetry account gains support.

The lab tests a fixed pair of content vectors. Learned absolute position adds deterministic synthetic table rows within the training horizon and becomes undefined beyond it. Sinusoidal absolute position adds position vectors before comparison, exposing origin-dependent cross-terms. Relative bias clips distance at sixteen. RoPE rotates query and key pairs at several frequencies. ALiBi applies a monotone distance penalty. NoPE leaves every distance identical in this comparator.

import argparse
import math
import numpy as np

Q = np.array([0.8, -0.3, 0.5, 0.1, -0.4, 0.7, 0.2, -0.6])
K = np.array([-0.2, 0.9, 0.4, -0.5, 0.3, 0.1, -0.8, 0.6])


def deterministic_row(position: int, dim: int = 8) -> np.ndarray:
    """Synthetic learned table row. Defined only where the caller permits it."""
    return np.array([
        math.sin((position + 1) * (j + 2) * 1.713)
        + 0.5 * math.cos((position + 3) * (j + 1) * 0.917)
        for j in range(dim)
    ]) * 0.35


def sinusoid(position: int, dim: int = 8) -> np.ndarray:
    out = np.zeros(dim)
    for pair in range(dim // 2):
        theta = 10000 ** (-2 * pair / dim)
        out[2 * pair] = math.sin(position * theta)
        out[2 * pair + 1] = math.cos(position * theta)
    return out


def rotate(vector: np.ndarray, position: int) -> np.ndarray:
    out = vector.copy()
    dim = len(vector)
    for pair in range(dim // 2):
        theta = 10000 ** (-2 * pair / dim)
        angle = position * theta
        c, s = math.cos(angle), math.sin(angle)
        a, b = vector[2 * pair], vector[2 * pair + 1]
        out[2 * pair] = a * c - b * s
        out[2 * pair + 1] = a * s + b * c
    return out


def score(scheme: str, i: int, j: int, train: int) -> float | None:
    delta = j - i
    if scheme == "learned":
        if i >= train or j >= train:
            return None
        return float((Q + deterministic_row(i)) @ (K + deterministic_row(j)))
    if scheme == "sinusoidal":
        return float((Q + sinusoid(i)) @ (K + sinusoid(j)))
    if scheme == "relative":
        return float(Q @ K - 0.12 * min(abs(delta), 16))
    if scheme == "rope":
        return float(rotate(Q, i) @ rotate(K, j))
    if scheme == "alibi":
        return float(Q @ K - 0.07 * abs(delta))
    if scheme == "nope":
        return float(Q @ K)
    raise ValueError(f"Unknown scheme: {scheme}")


def analyse(scheme: str, train: int, test: int, shift: int) -> dict:
    base = score(scheme, 2, 7, train)
    moved = score(scheme, 2 + shift, 7 + shift, train)
    shift_error = None if base is None or moved is None else abs(base - moved)

    values = [score(scheme, 0, d, train) for d in range(1, test)]
    out_of_table = sum(v is None for v in values) / len(values)
    finite = [(d + 1, v) for d, v in enumerate(values) if v is not None]

    collisions = 0
    for a in range(len(finite)):
        for b in range(a + 1, len(finite)):
            if abs(finite[a][1] - finite[b][1]) < 1e-3:
                collisions += 1

    monotonic_breaks = sum(
        finite[n + 1][1] > finite[n][1] + 1e-9
        for n in range(len(finite) - 1)
    )
    return {
        "shift_error": shift_error,
        "out_of_table": out_of_table,
        "collisions": collisions,
        "monotonic_breaks": monotonic_breaks,
    }


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--scheme", default="rope",
                        choices=["learned", "sinusoidal", "relative", "rope", "alibi", "nope"])
    parser.add_argument("--train", type=int, default=32)
    parser.add_argument("--test", type=int, default=96)
    parser.add_argument("--shift", type=int, default=17)
    args = parser.parse_args()
    print(analyse(args.scheme, args.train, args.test, args.shift))

A configured selection protocol

Suppose a team is building a legal-document system. Clauses retain meaning when the document is preceded by a cover sheet, so global shift should usually be irrelevant. Signed order matters because an amendment follows the clause it changes. Exact absolute page location matters only for citations shown to a reviewer, not for semantic matching. The longest expected document is 80,000 tokens, and the system retrieves candidate sections before a bounded reasoning pass.

The contract suggests relative or rotary comparison inside the model, explicit document coordinates in a separate evidence record, and retrieval for horizon control. It also requires an amendment-chain task, not only passkey retrieval. A monotone recency bias may help local drafting but could suppress foundational definitions, so ALiBi would need a long-distance counterexample test. Learned absolute embeddings are unnecessary for semantics but may remain useful in a layout encoder. The answer is a configured architecture, not a universal winning encoding.