The three windows hidden in one number

“Context window” is used as though it names one property. In practice it compresses at least three. The first is an admission limit: the service accepts a sequence up to a stated token count. The second is a positional competence range: positional signals and learned sequence dynamics remain stable enough for the model to process locations beyond those seen in its original training. The third is a decision-sufficient range: on a named task and risk slice, the model still finds and uses the evidence needed for an acceptable outcome.

Only the first is normally visible in an API specification. The second depends on architecture, position encoding, extension method and training. The third depends on all of those plus the task, prompt shape, evidence distribution, distractors, output criterion and evaluator. There is no task-independent “usable context length” waiting to be read from a model card.

Context is an input, not a durable memory

A long prompt is also routinely described as “memory”. That metaphor obscures system boundaries. The context is a temporary input assembled for one inference. Episodic memory records prior events. Workflow state records progress. Enterprise knowledge stores reusable claims. World state describes what is currently true. An evidence ledger records what supported a decision. Concatenating any of these into a model call does not merge them into one trustworthy memory system.

This distinction matters because the commercial-credit failure above cannot be repaired by adding more conversation history. The relevant issue is not whether the model remembers an earlier turn. It is whether the current call contains the applicable documents, whether their relationships survive assembly, and whether the model uses them. A larger window can reduce truncation while increasing the amount of irrelevant, stale or conflicting material competing for influence.

Thought experiment: move one clause, change nothing else

Take a fixed document pack and a fixed question. Place the decisive exception at 5%, 50% and 95% of the token sequence. Hold the words, token count, model, sampling settings and answer rubric constant. If success changes, position has causal force. This intervention is stronger than comparing three naturally different documents because only one feature moves.

Liu and colleagues' “Lost in the Middle” experiments used this logic on key-value retrieval and multi-document question answering. They reported that several models performed best when relevant information appeared near the beginning or end and worse in the middle. The exact curve is model and task dependent, but the experiment established a durable evaluation principle: randomise evidence position or a long-context score can be an artefact of document order.

What position extension repairs

Imagine that a model learned to navigate a city whose street numbers stop at 4,096. Asking it to accept street 32,000 creates two separate problems. The coordinate system must assign a meaningful location to that street, and the navigator must have learned behaviours that remain useful there. Position extension addresses the first problem directly. Continued long-context training can help with the second. Neither establishes that the navigator can compare every relevant address under heavy traffic.

Rotary position as a changing phase

Transformers first construct token representations, then need some way to encode order. Rotary Position Embedding, or RoPE, rotates pairs of query and key dimensions by angles that depend on token position. The useful intuition is phase. Two tokens acquire position-dependent orientations, so their attention score can reflect relative displacement rather than merely content.

The difficulty appears when positions extend far beyond those used in training. High-frequency components can rotate through unfamiliar phases, while low-frequency components may not distinguish distant locations well enough. Position Interpolation offers a clean repair: compress a longer coordinate range back into the range the model has seen. If the original length is L, the target length is L′, and the scale is s = L′/L, an extended position m is represented approximately as m/s.

YaRN makes the repair more selective. It interpolates rotary frequencies differently and adds attention scaling, because dimensions operating at different wavelengths need different treatment. ALiBi takes another route, adding distance-dependent biases to attention rather than encoding absolute positions in token representations. These methods are genuine engineering advances. They can preserve perplexity, stabilise fine-tuning and make previously impossible sequence lengths tractable.

They do not create a universal long-context faculty. A study of effective long-context training found that broad downstream evaluation and the long-context data mixture matter alongside the extension technique. A position method changes how distance is represented; it does not specify what evidence deserves attention, how claims combine, or when an answer should be withheld.

Thought experiment: keep the clause fixed, multiply its rivals

Return the decisive exception to the middle of the policy pack. Leave its wording and position unchanged. Now add ten, one hundred and one thousand clauses that share its vocabulary but apply to other products, dates or jurisdictions. Any fall in success cannot be blamed on positional coordinates because the evidence has not moved. The intervention changes competition.

A one-head toy makes the pressure visible. Suppose the relevant token receives attention logit z_e, each of D similar distractors receives z_d, and the relevance margin is Δ = z_e - z_d. Ignoring all other tokens, the relevant token's attention share is:

The counterexample that the argument must survive

The strongest answer to “long context does not work” is that some systems have demonstrated striking literal retrieval at enormous lengths. The Gemini 1.5 technical report reported greater than 99% retrieval on its needle-in-a-haystack evaluations up to at least ten million tokens. That result matters. It shows that long-range addressability can be far stronger than early context-window scepticism implied.

It does not erase the distinction. A uniquely marked needle asks whether one item can be recovered. The opening case asks whether two clauses can be found, joined through supersession and applied against a general rule. Excellent literal retrieval is a positive capability and a necessary control; it is not a certificate for every evidence geometry. A serious evaluation keeps the success case and asks what additional causal feature creates the failure.

flowchart LR
  A["The clause was inside the window"] --> B["The three windows hidden in<br/>one number"]
  B --> C["What position extension repairs"]
  C --> D["Context rot as a failure surface"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#087A98,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Context rot as a failure surface

“Context rot” is useful only if it stops being a vague diagnosis. Here it means a decline in task performance as admitted context grows, before the hard token limit is reached. The decline may come from position sensitivity, distractor competition, weak association, dispersed dependencies, contradictory evidence, output drift or a base reasoning limitation. These mechanisms can coexist, but they call for different remedies.

Several evaluation programmes expose the plurality. RULER extended simple needle retrieval with multiple needles, variable queries, tracing and aggregation; its authors found that performance could fall sharply as length and complexity increased. HELMET reported that needle-in-a-haystack scores were poor predictors of performance on broader long-context categories. NoLiMa removed obvious lexical overlap between questions and target passages, making retrieval depend more on latent association; in its 32K tests, eleven evaluated models fell below half of their short-context baseline.

Tasks that require relationships amplify the distinction. LooGLE found long-dependency understanding difficult and reported limited gains from position extension alone on its evaluations. LongBench v2 assembled 503 difficult questions across contexts from 8,000 to 2,000,000 words. In its historical evaluation, the best direct-answering result was 50.1%, while a reasoning model reached 57.7%. These are benchmark-specific numbers, not timeless rankings, but they demonstrate that fitting text and solving over text remain separate achievements.

Two July 2026 preprints sharpen the evaluation design further. PredicateLongBench varies computation, adversarial decoys, search-space size, quantifier complexity and context structure. WildTrace studies naturally dispersed evidence trails across 214 long-form sources. Because both are recent preprints, they are used here as test-design proposals rather than settled model rankings.

Five mechanisms can produce the same wrong answer

A final error does not identify its cause. First, coordinate failure occurs when positional behaviour itself becomes unstable outside the trained range. A model may lose local coherence, mishandle ordering or show a general perplexity rise. Position extension and continued training are plausible interventions here.

Second, selection failure occurs when the answer-bearing passage never receives enough influence. Position, lexical ambiguity and distractor count can all change selection. Retrieval, reranking, section markers and evidence staging target this mechanism more directly than another round of position scaling.

Third, association failure appears when the question and evidence do not share a convenient surface form. A user asks for “the silver heron”, while the operative record uses only a registry number. The system must resolve identity before it can retrieve the answer. Entity linking, iterative search or typed relations may be required.

Fourth, composition failure arises after the right passages have been located. The model must join several claims, respect directionality, resolve a conflict or apply an exception. Copying each clause on request does not prove that the combined inference is available. A graph traversal, calculation, verifier or structured intermediate representation can isolate the missing operation.

Fifth, answer-control failure occurs when the model has enough evidence yet produces an unsupported conclusion, ignores absence or follows an instruction embedded in source text. Citation checks, constrained outputs, abstention criteria and deterministic authority controls address this stage. Calling all five failures “the model forgot” conceals the intervention that should be tested next.

Measure the evidence chain, not only the sentence

Two systems can return the same correct answer for different reasons. One identifies the controlling exception and supersession notice. Another follows the general rule and happens to land on the same outcome in an easy case. Final-answer accuracy treats them as equal, so the shortcut survives until the conditions change.

A context evaluation should therefore record at least four linked measurements. Evidence recall asks whether every necessary source item was selected. Support precision asks whether cited items actually support the claim rather than merely sharing words. Relation completeness asks whether required links, such as identity, chronology, exception or causation, were represented. Decision validity checks the answer and any proposed effect. An abstention score belongs beside them because unsupported confidence is often more costly than a declared gap.

Counterfactual omission is especially revealing. Remove one necessary clause while leaving the rest of the pack intact. A grounded system should change its answer, reduce confidence or abstain. If the answer remains unchanged, either the removed evidence was not causally used or the task contains another sufficient route. Add a semantically similar but inapplicable clause as a second control. A system that switches to the decoy has a discrimination problem rather than a capacity problem.

These measurements turn the commercial-credit scenario into a diagnosable trace. The system might retrieve both documents yet miss supersession, proving that retrieval recall is not the blocker. It might cite the exception but apply it to the wrong product, exposing identity resolution. It might construct the right decision while proposing an unauthorised action, exposing an authority boundary beyond context. The same wrong output can therefore demand a different architecture.

A minimal worked example: four tasks in the same pack

The accompanying stress harness constructs synthetic packs of 32, 128 and 512 blocks. It places evidence near the beginning, middle or end; varies distractor similarity; and asks four kinds of question. Literal retrieval names the key directly. Associative retrieval names an alias that must be resolved. Multi-hop retrieval follows a three-link chain. The absent case has no valid answer and rewards abstention.

An oracle reader consumes every block marked as evidence. It passes all 324 cases, showing that the fixtures contain enough information. A deliberately bounded lexical reader ranks once, reads only a square-root-sized budget and cannot launch a new search after learning an intermediate identifier. Its measured results are shown below. They are properties of this transparent toy, not estimates of any language model.

The literal row is the negative control for an overbroad theory of decay. It remains the strongest answer-bearing task even at 512 blocks. The absent row verifies that apparent accuracy is not bought by emitting any code seen in the pack. The associative and multi-hop rows show another failure source: a one-shot reader may find the first clue yet fail to recover what that clue makes newly relevant.

Thought experiment: a perfect index card

Suppose an oracle extracts the relevant paragraph from a very long document and copies it immediately before the question. Keep that paragraph and question fixed, then add unrelated text earlier in the prompt. Retrieval has been made perfect by construction. If answer quality still falls, the failure lies downstream of locating the evidence.

A 2025 study accepted at Findings of EMNLP, “Context Length Alone Hurts LLM Performance Despite Perfect Retrieval”, reports degradation under related controls across five models. One study should not carry the argument alone. Its value is experimental: it identifies a way to separate retrieval failure from context-length effects on reasoning or answer production. A release test should include this condition while treating the reported magnitudes as bounded to the evaluated models and tasks.

A production-shaped worked scenario

Return to the credit-renewal decision. The evidence has four roles: the customer event establishes unresolved beneficial ownership; the product term permits automatic renewal by default; the exception suspends that permission; and the supersession notice keeps the exception effective after a handbook replacement. A correct answer needs identity resolution, temporal applicability, conflict handling and a final policy consequence.

Dumping the whole policy estate into one call preserves maximum recall in principle. It also mixes obsolete versions, examples, nearby products and clauses with different authority. A context compiler instead retrieves candidate evidence, resolves document identity, filters by effective time and entitlement, then emits a typed packet. The model still reasons probabilistically, but it reasons over an inspectable evidence set whose omissions can be tested.

A context stress test that can fail

A useful test suite changes one causal feature at a time, then combines features to reveal interactions. It begins with a solvable fixture and a negative control. It does not start with a model leaderboard because a leaderboard hides the acceptance criterion of the product.

Build the factorial surface

For each real task family, create a public or synthetic corpus whose evidence and answer are known. Vary at least five axes: admitted length, evidence position, distractor similarity, number of required facts and relationship depth. Add source conflict, temporal supersession and evidence absence when they occur in the product. Repeat with multiple seeds or document orders.

Evaluate the final answer, the cited evidence and the decision to abstain. For consequential workflows, add an independent rule or human check on the proposed action. The score that matters is the weakest required slice, not the average over easy positions and literal needles.

The executable artefact

Inspect the implementation in this paper and Review the recorded evidence in this paper. It requires Python 3.10 or later and only the standard library. The oracle proves that each answer-bearing fixture is solvable. The bounded reader exposes how a declared read budget and one-shot lexical ranking interact with evidence geometry.

# Positive and negative controls
python context_stress_test.py --smoke

# Full 324-case factorial grid
python context_stress_test.py --grid --output context_stress_results.csv

# The bounded reader declares its limitation.
budget = min(n, max(8, round(math.sqrt(n))))
score = lexical_overlap + edge_prior + stable_jitter
selected = highest_scoring_blocks[:budget]
answer = parse_without_a_second_retrieval_round(selected)

The expected smoke output contains five passes: two oracle fixture checks, a literal edge success, an associative middle failure and correct abstention when evidence is absent. The grid writes one row per case with the expected answer, oracle answer, bounded answer and correctness flags. The point is not to make the toy look intelligent. It is to make every success and failure attributable.

The context fitness decision

The operational question is no longer “does the model support 128K?” It is “what is the largest context slice this workflow can defend?” Define the validated context length as a constrained release boundary:

The minimum over required slices prevents excellent passkey scores from hiding a weak multi-hop region. The lower confidence bound prevents a small sample from masquerading as certainty. Cost matters because a longer prefill can raise latency and spend even when quality remains flat. For an irreversible action, success should include verified outcome or mandatory human approval rather than fluent answer accuracy alone.

A compact release worksheet

Architecture follows the evidence geometry

Use direct long context when the corpus is bounded, entitlement is uniform, evidence is dense and the validated curve remains flat. Use retrieval when only a small portion is likely to matter and source recall can be measured. Use iterative retrieval when one fact reveals the key for the next. Use a context compiler when identity, effective time, authority or source relationships determine applicability. Use deterministic calculation or policy code when the answer is a rule execution rather than an interpretation.

Preserve source identifiers outside the prompt so that a model proposal can cite the exact evidence packet. For consequential actions, bind that packet to a decision receipt and verify the effect separately. The window is context, not authority. A token appearing in the prompt does not acquire permission to instruct a tool, and a model finding a clause does not authorise the resulting business action.

This model-level argument sits below the broader architectural essays “Context Is the New Compute”, “Governed Context Architecture: Reconstruction, Provenance and Active Memory for Long-Horizon Agents” and “Context Compilers for Policy-Filtered Retrieval”. Those pieces ask how a system assembles evidence. This paper supplies the narrower release rule: no assembly strategy earns trust until its task-specific context surface has been measured.

The window you can defend

A larger context window solves a real problem: it lets a system admit more of a sequence without truncation. Position extension solves another: it keeps positional machinery useful beyond its original range. Neither number answers the product question. A product needs to know whether a model can locate, combine and apply the relevant evidence across the conditions that matter.

The durable decision is therefore simple. Keep the provider maximum as an infrastructure limit. Measure a context surface over length, position, distractor similarity, evidence geometry, absence and consequence. Define L* from the weakest required slice, with uncertainty and cost included. Beyond that boundary, retrieve, stage, compile, verify or escalate rather than treating unused capacity as usable cognition.

The defensible context window is the one your task can survive, not the one your endpoint can accept. That change turns a marketing number into an engineering claim that can fail, improve and accumulate evidence.