Order without numbered seats

Before asking how RoPE works, ask what would be missing without it. A standard attention score starts with a query vector at one token and a key vector at another. Their dot product says how compatible their contents are. It does not, by itself, say which token came first.

The symmetry that hides sequence order

Suppose the input contains three token vectors. Apply the same query, key and value projections to each. If the tokens are permuted, the matrix of pairwise scores is permuted in the same way, and the outputs follow that permutation. The layer is not confused. It is simply respecting a symmetry: the operation has no privileged first, second or third slot.

Bare self-attention is a set operation with a permutation-equivariant output. A causal mask changes that statement because each position can attend to a different number of predecessors. Recurrence, convolution, padding boundaries and data regularities can also leak order. But the content-only dot product contains no ordinal coordinate. The original Transformer therefore added positional encodings to token embeddings because its architecture had neither recurrence nor convolution.[1]

Why an absolute label is not always the right object

The simplest remedy is a table: position 0 receives one learned vector, position 1 another, and so on. A table can memorise useful regularities, but it treats each location as a separate name. Positions outside the table are undefined, and a global shift changes every label even when the relational structure of the sequence is unchanged.

The original sinusoidal encoding replaced the table with sine and cosine functions whose wavelengths form a geometric progression. For a fixed offset, the encoding at the shifted position is a linear transformation of the original encoding.[1] That fact already hints at rotation: sine and cosine at one phase become sine and cosine at another phase through a two-dimensional rotation matrix.

flowchart LR
  A["Order without numbered seats"] --> B["The identity hidden in a rotation"]
  B --> C["The frequency bank sets the ruler"]
  C --> D["Test the geometry before<br/>extending it"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#087A98,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

The identity hidden in a rotation

A rotation seems like an odd place to store a token index. The reason becomes clear in two dimensions, where a vector can be read as a complex number and a position step can be read as multiplication by a unit phase.

One feature pair as a clock hand

Take two coordinates, x and y, from a projected query. Treat them as the complex number z = x + iy. Multiplying by e^iθ rotates the pair through angle θ without changing its length. At position m, multiply by e^imθ. Moving from position m to m + 1 then applies the same rotation once more.

This is more than a compact code. Repeated position steps compose correctly: rotating by a steps and then b steps is the same as rotating once by a + b steps. Negative steps use the inverse rotation. The integers have been represented as actions on a plane.

Rotate both sides, then compare

Now let a query content vector q sit at position m, and a key content vector k sit at position n. Rotate each by its own position. Their dot product is

This is the load-bearing identity: R_m^TR_n = R_n−m. Shift both tokens by the same constant c, and the score is unchanged because (n + c) − (m + c) = n − m. RoPE encodes an absolute phase on each vector, yet exposes relative position in their interaction. That is the apparent paradox resolved.

Direction survives because sine changes sign

The compact matrix identity also reveals what a single rotary pair can distinguish. Write the displacement as δ = n − m. Expanding the two-dimensional score gives a content-alignment term multiplied by cos(δθ), plus an oriented term multiplied by sin(δθ). The first term is unchanged when the order is reversed. The second changes sign.

Swap the query and key positions while holding their content fixed. The displacement changes from δ to −δ. Cosine stays the same, but sine reverses, so the score can distinguish “before” from “after” whenever the oriented content term is non-zero. If the two content vectors are parallel, that term vanishes and this one pair cannot express direction for that comparison. Other pairs and learned projections may still do so.

Rotation also preserves the length of each query and key pair. Position therefore changes their orientation without injecting a larger norm merely because an index is larger. This does not freeze the distribution of complete attention logits. Learned projections can place different content in different frequency pairs, and the sum across pairs can still change substantially with distance. RoPE preserves local vector norm while making compatibility position-sensitive.

From one clock to many frequencies

A real attention head has many coordinates. RoPE pairs them, assigning a different θ_i to each two-dimensional plane. The full operator is block diagonal: many small rotations running in parallel. Common implementations use a geometric schedule such as θ_i = B^−2i/d, where d is the head dimension, i indexes a coordinate pair, and B is a frequency base.

The fast pairs change sharply between adjacent tokens. Slow pairs preserve a gradual phase over thousands of tokens. Together they map one integer position to a point on a product of circles, sometimes called a torus. Multiple frequencies do not abolish periodicity; they make exact recurrence harder and provide several positional scales at once.

Why rotation is more than an added sinusoid

Sinusoidal absolute encoding and RoPE use the same elementary geometry but place it at different points in the computation. Add a positional vector to a token before the query and key projections, then expand the dot product. Four families of terms appear: content with content, content with position, position with content, and position with position. The model can learn useful relative relations from those terms, and a fixed shift of the sinusoidal code is indeed a linear transformation.[1] The relative law is nevertheless not isolated in the score by default.

RoPE first forms content-dependent queries and keys, then applies the position operator. The equality R_m^TR_n = R_n−m is therefore built into the bilinear comparison itself. Conventional implementations leave values unrotated, so position modifies which message is selected rather than rewriting the payload carried by the value vector. The distinction is not fixed versus learned position. It is where the desired invariance enters the causal path.

This boundary matters when diagnosing a model. A wrong rotary offset can corrupt routing even if token representations look sensible. A value-path defect can corrupt the retrieved message even when every rotary identity test passes. The two failures may produce the same wrong answer, but they require different interventions.

Where the Fourier comparison is exact, and where it stops

For each frequency θ, the function χ_θ(m) = e^imθ converts addition of integer positions into multiplication of phases: χ_θ(m + n) = χ_θ(m)χ_θ(n). In harmonic analysis, such functions are characters of the translation group. This is the precise Fourier structure inside RoPE.

But RoPE is not, in ordinary use, the discrete Fourier transform of the token sequence. It does not sum every token against an orthogonal frequency grid and return spectral coefficients. It modulates coordinate pairs in queries and keys with a finite, usually non-uniform bank of phases. Recent work has analysed RoPE attention through non-uniform Fourier transforms, but that is a model of its frequency behaviour, not permission to call every rotary layer a full Fourier transform.[13]

The frequency bank sets the ruler

The relative-position identity is exact for every chosen frequency. The usefulness of the resulting geometry is not. A finite bank decides which offsets look distinct, which look similar, and which phase combinations the model ever sees during training.

A positional kernel you can inspect

Strip away learned content for a moment. Put equal weight on every frequency pair and compare position 0 with offset Δ. A simple normalised similarity is

This is not the trained attention score. Learned queries and keys weight phases differently and mix content with position. K is a diagnostic lens: it reveals the geometry supplied to learning before the model decides how to use it.

Identifiability is a margin, not a yes-or-no property

For a task, the useful question is not whether every integer has a mathematically unique phase signature. It is whether the offsets the task must separate remain far enough apart after projection, scaling and numerical rounding. A parser may need to distinguish distances 1, 2 and 3 with high confidence. A document retriever may care more about separating 8,000 from 24,000 than about one-token precision at either location.

The equal-weight kernel K is a first warning system, not a collision certificate. Two offsets can have the same average cosine while distributing phase error across frequency pairs differently. Learned queries and keys can amplify one pair, suppress another, or use the sine channel that K omits. Conversely, two signatures that look distinct in exact arithmetic may become operationally close after low-precision trigonometry and a narrow rotary subspace.

A better Gate B test defines the required offset set, computes pairwise phase distances under the deployed scaling and precision, then records the smallest separation margin. Inject the most confusable pair into a task whose content is otherwise identical. If behaviour changes only when the positional margin changes, the geometry is causally implicated. Task-relative separation, not abstract uniqueness, is the operative standard.

Fast resolution, slow context

A frequency θ has wavelength λ = 2π/θ tokens. The fastest conventional pair has a wavelength near 6.28 tokens. Slower pairs have progressively larger wavelengths. In a 128-dimensional head with base 10,000, the slowest pair has a wavelength of about 54,410 tokens. At a 4,096-token training length, the fastest pair has turned roughly 652 times, while the slowest has completed only about 0.075 of a cycle.

That spread is useful. It is also uneven training exposure. Some channels repeatedly display their periodic behaviour. Others reveal only a small arc. The model may learn local phase relations confidently while seeing too little evidence to infer how the slowest channels behave across a full turn.

The near-collision that preserves the law

With only four frequency pairs, the geometry is much easier to fool. Use an eight-dimensional head with base 10,000. The wavelengths are approximately 6.28, 62.8, 628 and 6,283 tokens. At offset 6,283, several dials return close to their starting directions at once. The equal-weight phase similarity is about 0.996.

The relative identity has not failed. R_m^TR_n is still exactly R_n−m. The problem is that R₆₂₈₃ happens to resemble the identity on this small frequency bank. Exact relational algebra and useful positional identifiability are different properties.

Three meanings of “it knows position”

The phrase hides three separate claims:

Algebraic relativity asks whether global shifts cancel and displacements compose correctly. RoPE answers yes by construction. Geometric identifiability asks whether task-relevant offsets produce distinguishable operators at the available dimension and precision. That depends on the frequency bank. Statistical familiarity asks whether training has taught the model to use the relevant phase combinations, distances and score distributions. That depends on data, optimisation and context curriculum.

This distinction resolves an apparent contradiction in the literature. RoPE has an exact relative-position formula, yet empirical studies find that it does not automatically generalise to longer downstream sequences.[8] Both statements can be true because the first concerns algebra and the second concerns learned behaviour outside the training regime.

RoPE is one answer, not the definition of position

Learned relative representations can add vectors indexed by displacement.[2] T5 adds learned relative-position biases to attention logits.[3] ALiBi adds a head-specific linear penalty proportional to distance, creating an explicit recency prior without sinusoidal phases.[5] Causal language models without explicit positional encodings can still acquire implicit position signals, plausibly from the causal mask and learned computation.[7] A broader review shows how these choices distribute position across inputs, scores and architectural constraints.[6]

stateDiagram-v2
  [*] --> Claim
  state "The symmetry that hides<br/>sequence order" as Claim
  state "Why an absolute label is not<br/>always the right object" as Test
  state "One feature pair as a clock hand" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Test the geometry before extending it

A configuration file can permit 32,000 tokens while the model still fails to use evidence at token 28,000. Extending the index range changes phase geometry, training distribution and attention competition at the same time.

Stretching the context changes two things at once

Suppose a model was trained to length 4,096 and must now serve 32,768 tokens. Raw extrapolation keeps the angular step per token unchanged. Every frequency travels eight times farther than during training, exposing the model to new phase combinations and potentially different attention-logit behaviour.

Position interpolation instead maps target index m to m/8. The longest target sequence stays inside the old phase range, which reduces phase novelty. But adjacent target tokens now differ by only one eighth of the former angle. Interpolation trades extrapolation risk for compressed local positional resolution. Fine-tuning can teach the model to work with that altered ruler.

Position Interpolation reported extensions of RoPE-based models with relatively short fine-tuning by linearly down-scaling indices.[9] YaRN introduced frequency-aware interpolation and attention scaling, with later revisions continuing to refine the method.[10] LongRoPE searched non-uniform scaling factors and used progressive extension.[11] LongRoPE2 combined rescaling with mixed-length training and argued that insufficient exposure of higher RoPE dimensions contributes to persistent out-of-distribution behaviour.[12]

These are important results, not universal guarantees. Effective context depends on the model, training data, attention architecture, task and evaluation. Official libraries now expose several RoPE variants, including linear, dynamic, YaRN, LongRoPE and model-specific scaling.[15] A recognised configuration name confirms that code can run. It does not prove that evidence at the declared limit can influence the answer correctly.

The same formula can be wired into different systems

Before blaming a frequency schedule, verify that the implementation realises the schedule you think it does. Libraries differ in how they pair coordinates. Some rotate adjacent coordinates; others split the head into two halves and pair corresponding entries. These conventions are equivalent only when the weights, rotation helper and checkpoint agree. Mixing them preserves vector norms while silently changing every oriented interaction.

Position identifiers create a second boundary. Padding tokens, packed sequences and a key-value cache can make the physical array index differ from the logical token position. In autoregressive decoding, a new query must use the position that follows the cached keys. Resetting its identifier to zero does not test a new origin. It moves only one side of the comparison and changes every displacement. A legitimate origin shift must be applied consistently to both query and key positions.

Three further choices matter: how many head dimensions are rotary, where scaling is applied, and in which precision phases are generated. Rotating only part of a head leaves a content-only subspace beside the positional one. Scaling queries but not cached keys breaks the relative identity. Computing large angles in limited precision can introduce avoidable phase error even when the symbolic formula is correct. Precomputation and cache reuse should therefore be tested as part of the mechanism, not treated as neutral plumbing.

Record the frequency base, rotary dimension, pairing convention, position-id policy, scaling rule, cache offset and phase precision with the result. That manifest turns “we use RoPE” into a reproducible mechanism claim. Without it, two systems can share a configuration label while implementing materially different positional operators.

A three-layer positional competence audit

The practical contribution of this paper is an audit that refuses to collapse three questions into one score.

Gate A, algebraic relativity: verify the operator identity numerically across random vectors, shifts and data types. A failure here indicates an implementation or indexing defect. Gate B, geometric identifiability: inspect adjacent resolution, distant phase similarities, wavelength coverage and numerical precision inside the target range. A failure here indicates an unsuitable ruler, not necessarily a failed model. Gate C, statistical familiarity: run controlled behavioural tests across positions, lengths and phase interventions. A failure here may arise from training, attention competition or the rest of the architecture even when the first two gates pass.

Run the geometry audit

The interactive lab below computes the bare frequency geometry. It tests the exact relative law with deterministic pseudo-random vectors, plots K(Δ), finds a high-similarity distant offset, counts slow channels that complete less than one cycle during training, and displays the trade-off introduced by interpolation.

How to interpret the result

from __future__ import annotations

import argparse
import math
from dataclasses import dataclass

import numpy as np


@dataclass(frozen=True)
class AuditResult:
    relative_law_error: float
    adjacent_similarity: float
    distant_similarity: float
    distant_offset: int
    slow_pairs: int
    pair_count: int
    angular_step_ratio: float


def frequencies(head_dim: int, base: float) -> np.ndarray:
    if head_dim < 2 or head_dim % 2:
        raise ValueError("head_dim must be a positive even integer")
    if base <= 1:
        raise ValueError("base must be greater than 1")
    pair_index = np.arange(head_dim // 2, dtype=np.float64)
    return base ** (-2.0 * pair_index / head_dim)


def rotate(x: np.ndarray, position: float, theta: np.ndarray) -> np.ndarray:
    pairs = x.reshape(-1, 2)
    angle = position * theta
    c, s = np.cos(angle), np.sin(angle)
    out = np.empty_like(pairs)
    out[:, 0] = pairs[:, 0] * c - pairs[:, 1] * s
    out[:, 1] = pairs[:, 0] * s + pairs[:, 1] * c
    return out.reshape(-1)


def kernel(offsets: np.ndarray, theta: np.ndarray, scale: float) -> np.ndarray:
    return np.cos(offsets[:, None] * scale * theta[None, :]).mean(axis=1)


def audit(
    head_dim: int = 128,
    base: float = 10_000.0,
    train_length: int = 4096,
    target_length: int = 32768,
    interpolate: bool = False,
    search_start: int | None = None,
) -> AuditResult:
    if train_length < 2 or target_length < 2:
        raise ValueError("train_length and target_length must both be at least 2")
    if search_start is not None and search_start < 2:
        raise ValueError("search_start must be at least 2")

    theta = frequencies(head_dim, base)
    scale = min(1.0, train_length / target_length) if interpolate else 1.0

    rng = np.random.default_rng(7)
    q, k = rng.normal(size=head_dim), rng.normal(size=head_dim)
    tests = [(3, 11), (17, 29), (101, 503), (train_length // 3, train_length - 1)]
    errors = []
    for m, n in tests:
        direct = np.dot(rotate(q, m * scale, theta), rotate(k, n * scale, theta))
        relative = np.dot(q, rotate(k, (n - m) * scale, theta))
        errors.append(abs(direct - relative))

    stop = min(target_length, 100_000)
    start = min(max(2, search_start if search_start is not None else train_length), stop)
    offsets = np.arange(start, stop + 1, dtype=np.float64)
    similarities = kernel(offsets, theta, scale)
    winner = int(np.argmax(similarities))

    training_cycles = train_length * theta / (2.0 * math.pi)
    return AuditResult(
        relative_law_error=float(max(errors)),
        adjacent_similarity=float(kernel(np.array([1.0]), theta, scale)[0]),
        distant_similarity=float(similarities[winner]),
        distant_offset=int(offsets[winner]),
        slow_pairs=int(np.sum(training_cycles < 1.0)),
        pair_count=len(theta),
        angular_step_ratio=scale,
    )


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--head-dim", type=int, default=128)
    parser.add_argument("--base", type=float, default=10_000.0)
    parser.add_argument("--train-length", type=int, default=4096)
    parser.add_argument("--target-length", type=int, default=32768)
    parser.add_argument("--interpolate", action="store_true")
    args = parser.parse_args()
    print(audit(**vars(args)))

Save the code as positional_geometry_audit.py, install NumPy, and run python positional_geometry_audit.py. Add --interpolate to compare linear interpolation. A positive result on the relative-law test means the rotary implementation obeys its intended algebra within numerical tolerance. A negative result points to an indexing, pairing, scaling or precision defect.

The code cannot establish that a model understands order, retrieves distant evidence or reasons across long documents. It deliberately excludes learned weights, tokenisation, attention masks, values, training data and downstream tasks. Its purpose is to expose the positional mechanism that prose often hides, then tell you which behavioural experiment is still required.

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: Order without numbered seats
  M->>E: The symmetry that hides sequence order
  E-->>M: compare rival explanations
  E->>D: Test whether rotation preserves the relation
  D-->>Q: revise scope or act

Test whether rotation preserves the relation

Do not choose or scale a positional encoding because it supports a large number in a configuration field. Choose it because its invariance, resolution and training regime match the task.

For a task whose meaning should survive a global shift, demand and test a relative law. For tasks that depend on exact local offsets, inspect what interpolation does to adjacent angular separation. For long-range retrieval, search for positional near-returns, but also control attention dilution. For extrapolation, measure performance by both absolute location and relative displacement so that origin failures do not hide inside aggregate accuracy.

The engineering decision is to separate “the formula is defined” from “the geometry is discriminative” and from “the model has learned to use it”. RoPE gives an elegant answer to the first question. The frequency bank and numerical implementation shape the second. Only controlled training and intervention answer the third.

This also changes how alternatives are compared. A learned relative bias, ALiBi, RoPE and NoPE may achieve similar short-context scores while preserving different relations. Compare them by the intervention that matters: global shift, local displacement, length stretch, sparse retrieval, multidimensional structure or streaming state. Output matching on the original benchmark is not enough.

Periodic geometry succeeds because it reuses a small set of transformations across an unbounded index domain. Its limit is the same fact viewed from the other side: periodic signals revisit old directions. The right question is therefore not whether rotation “contains position” in the abstract. It is whether the chosen rotations preserve the relation your system must act upon, at the scales where it must act.