Context is the New Compute

Here is the uncomfortable pattern I keep seeing: a team ships an agent on a frontier model, it underperforms, and the instinct is to blame the model or rewrite the prompt. Then someone finally inspects what the model actually received in its context window on the failing cases, and finds a mess. Truncated documents. Three contradictory policy versions. Tool outputs from four steps ago crowding out the current question. The model didn't fail. It answered the question it was actually asked, which was not the question anyone intended.

After several years of running LLM systems in production at a global financial institution, my working position is blunt: for agentic systems, context engineering is the core discipline. Models are increasingly interchangeable commodities behind a gateway. The context assembly pipeline (what the model sees, in what order, at what fidelity, under what budget) is where your system's intelligence actually lives, and it's the part you own.

The context window is a budget, not a bucket

The single most useful mental shift: stop thinking of the context window as space to fill and start treating it as a budget to allocate: with an owner, line items, and enforcement.

Every production agent we run has an explicit context budget, partitioned roughly like this:

• System contract (instructions, role, output schema): fixed allocation, ruthlessly compressed, versioned like code.
• Task state: the current request plus working memory the agent genuinely needs now: not the full transcript of everything that happened.
• Retrieved knowledge: the largest and most variable line item, capped hard.
• Tool schemas: only the tools plausibly relevant to this state of the workflow, not the full registry.
• Headroom: reserved space for reasoning and output. Squeeze this and quality degrades in ways that look mysteriously random.

Two facts make the budget framing non-negotiable. First, long-context degradation is real: advertised window sizes are not effective reasoning sizes, and mid-context information reliably gets less attention than the edges. Filling the window is often actively worse than a disciplined subset. Second, context is your marginal cost. In agentic systems, input tokens dominate spend: an agent that drags its full history through fifteen tool-call steps pays for that history fifteen times. When we put hard budgets and aggressive state-pruning on one of our heaviest workflows, cost per case dropped by more than half and answer quality improved. That result surprised the team. It shouldn't have.

Retrieval is a systems problem wearing a search costume

"We do RAG" tells me almost nothing. The gap between naive vector search and a production retrieval layer is the gap between a demo and a system.

What production retrieval actually involves:

Query understanding before search. The user's message is rarely the right retrieval query. Real cases need decomposition (one question becomes three lookups), rewriting against domain vocabulary, and routing, because not everything lives in one index. "What's the fee for X" is a structured lookup; "explain our position on Y" is a document search. Sending both to the same vector index is how you get confident nonsense.

Hybrid retrieval as the default. Pure embedding search fails exactly where banks are unforgiving: exact product codes, policy reference numbers, negations, dates. Lexical plus semantic plus a reranker is the boring, correct baseline. Embeddings-only is a prototype smell.

Knowledge architecture over retrieval tricks. This is the unfashionable part that dominates outcomes. Chunking that respects document structure instead of character counts. Metadata (effective dates, jurisdiction, product line, audience) attached at ingestion and filtered at query time. Supersession handling, so the 2023 policy cannot outrank the 2025 one that replaced it. In a regulated environment, retrieving a stale document isn't a relevance bug; it's a compliance event. No reranker fixes a corpus that doesn't know which of its documents are still true.

I'll say it plainly: most "RAG quality" problems are content governance problems. Teams tune embedding models for weeks when the actual issue is that nobody owns the corpus.

Context failure modes: a field guide

Once you start attributing agent failures to context rather than to "the model being dumb," patterns emerge. The ones I see most, roughly in order of frequency:

Context poisoning. One wrong item (a hallucinated intermediate conclusion, a stale document, a bad tool output) enters working memory early and every subsequent step builds on it. The final answer is coherent, well-reasoned, and wrong. Defence: validate at the boundaries. Tool outputs get schema-checked and sanity-checked before entering context, not after they've contaminated three steps of reasoning.

Context distraction. The relevant fact is present but buried under twelve marginally-related chunks, and the model anchors on the wrong one. This is the direct cost of "retrieve top-20 and hope." Defence: retrieve wide, rerank hard, admit few. Precision into the window beats recall into the window, almost always.

Context clash. Two retrieved sources genuinely disagree (old terms vs. new terms, policy vs. procedure) and the model silently picks one. The failure isn't the pick; it's the silence. Defence: detect contradiction in the assembly pipeline and either resolve it by metadata (recency, authority) or surface it explicitly: "sources conflict, escalating."

Context erosion. In long agentic runs, summarisation-compaction slowly launders away the constraint that mattered (the customer's vulnerability flag, the "do not contact" marker) because it looked incidental at compression time. Defence: certain fields are structural, carried outside the summarised narrative, and physically cannot be compacted away.

Instruction drift. Thirty steps in, the system contract from the top of the run has faded relative to the churn of recent tool outputs. The agent starts freelancing. Defence: re-assert the contract near the end of the assembled context on every step. Yes, it costs tokens. It's the cheapest reliability you will ever buy.

The meta-point: every one of these is diagnosable only if you log the assembled context per step. If you can't replay exactly what the model saw, you are debugging by folklore.

Treat the context pipeline as a versioned, tested artifact

The operational consequence of everything above: the context assembly pipeline is production software and deserves production discipline.

• Version it. A change to chunking, reranking, budget allocation, or assembly order is a release, with an owner and a rollback path. Some of our worst regressions came from "harmless" retrieval config tweaks that no one thought to gate.
• Eval it in layers. Retrieval quality (is the right evidence in the window?) is measured separately from generation quality (did the model use it correctly?). When an end-to-end eval fails, layered metrics tell you which half of the system to fix. Without them, teams reflexively fix the prompt: usually the wrong half.
• Watch it drift. Corpora grow, query mixes shift, budgets silently saturate. Context health (retrieval hit rates, contradiction rates, budget utilisation, truncation frequency) belongs on the same dashboard as latency and cost.

Where the leverage is

Model capability is now something you rent, and everyone rents from the same three or four landlords. Your prompts will be rewritten quarterly and your framework will be replaced twice. What compounds (the asset that gets more valuable every quarter you invest in it) is the machinery that reliably puts the right, current, non-contradictory evidence in front of a model under a budget, and can prove it did so.

That machinery is context engineering. It's not the garnish around the model. In an agentic system, it is the system.

A context compiler, not a prompt builder

The practical architecture resembles a compiler. Raw requests and enterprise records are not concatenated. They are resolved against policy, normalized into typed evidence and assembled under a declared budget.

A useful context object is inspectable before it is intelligent. An engineer should be able to read the manifest and explain why every item was admitted.

The context risk matrix

Relevance and authority are separate axes. Highly relevant material can still be unsuitable for a decision. A customer email may be central to a complaint while remaining unverified evidence.

This gate creates a disciplined distinction. Retrieval finds candidates; context assembly decides what may influence the model. The two steps should be evaluated separately.

Context needs a product model, not a shared folder

Enterprise context crosses several ownership domains. The product team owns the task definition. A policy owner controls the source that states what is permitted. A data owner controls customer and case fields. The agent platform controls assembly. Security controls entitlement. None can delegate its responsibility to the vector index.

A context product makes those responsibilities explicit. It has a declared consumer, decision type, source catalogue, admission contract, service level and quality measures. It also has an owner who can withdraw a source when validity is uncertain. The output is not simply text. It is a versioned package of evidence and instructions prepared for one reasoning step.

The context contract should be specific enough to reject a technically relevant source. A vulnerable-customer decision may require the active customer-contact policy, live channel preference, current case facts and a verified vulnerability marker. It may prohibit unverified profile inferences and unrelated historical notes. A generic “retrieve customer context” instruction cannot express that boundary.

Service levels also differ. Product descriptions can tolerate a slower update cycle than fraud alerts or account restrictions. The source catalogue should declare maximum age, supersession behaviour and failure policy by data class. Freshness is a business property, not a timestamp threshold chosen by the retrieval team.

Ownership becomes visible when a source fails. The corpus team can confirm ingestion. The information owner decides whether the content remains authoritative. Security decides whether a principal may receive it. The context service decides whether the remaining evidence is sufficient to proceed. This separation prevents a retrieval success from being mistaken for a valid decision basis.

Build the manifest before the prompt

The context manifest is the control record for one assembled window. It lists admitted items, rejected candidates, source versions, authority labels, token allocations, transformations and unresolved conflicts. The rendered prompt is derived from it. The manifest should exist even if the model provider never sees it.

This structure solves a persistent debugging problem. When an answer is wrong, teams often cannot tell whether the relevant evidence was absent, truncated, misplaced, contradicted or ignored. The manifest answers the first four questions before anyone changes the prompt.

Transformations need provenance. If a long policy is summarized, the manifest records the source passages, summarizer version and resulting digest. If structured case fields are converted into prose, it records the field names and record version. If personally identifiable data is masked, it records the policy that required masking without exposing the original value to the model log.

An item can be admissible yet poorly placed. System constraints, case facts, tool results and reference material compete for attention. Assembly rules should define order and repetition for each item type. Critical invariants may appear in a structured state block and be restated near the proposed action. Supporting material can remain in a lower-priority evidence section.

Prompt text is a rendering; the manifest is the governed artefact. That distinction allows a team to change formatting or model provider without losing the evidence model.

Evaluate four links in the context chain

End-to-end answer accuracy is necessary but diagnostically weak. A context system has at least four quality links: candidate discovery, admission and assembly, model use, and business outcome. Each needs its own test.

The test collection should include clean cases, conflicting sources, missing mandatory evidence, superseded documents, entitlement denials and long histories. It should preserve the expected context manifest as well as the expected answer. That allows a retrieval or assembly change to be tested before model behaviour adds noise.

Production evaluation should sample manifests by route and outcome. A high-performing common path can hide a weak rare path, especially where a specific jurisdiction or product has limited source coverage. Failure analysis should slice by source owner, corpus version, decision type and context-budget pressure.

This layered design stops a common waste pattern. If discovery never found the active policy, prompt tuning cannot repair the case. If the right policy entered but the model ignored a qualifying clause, changing embeddings is irrelevant. Measure the handoff between layers before optimizing a component.

Context economics is about reuse and rework

Token cost is only one part of context economics. A cheap context package that causes repeat calls, unnecessary tool use or human correction can be expensive at the case level. A larger package may be justified if it reduces ambiguity on a material decision. The useful denominator is accepted outcome, not input token.

Cost attribution should follow the manifest. It can show tokens by component, retrieval calls by source, transformations performed, cache reuse and repeated assembly after a failure. This makes the largest avoidable line item visible. It may be duplicated policy text, verbose tool output, an irrelevant transcript or an overly broad tool catalogue.

Caching requires the same discipline as retrieval. Stable system contracts and public reference material are strong candidates. Customer state and entitlement are not. Cache keys should include the source version, policy version, jurisdiction and any field that changes meaning. A hit is safe only when the admission contract would produce the same item now.

The design should distinguish three kinds of reuse. Byte reuse avoids repeated transport. Transformation reuse avoids summarizing or normalizing the same valid source. Decision-context reuse reuses an assembled package. The third has the highest risk because purpose, principal or case state may have changed.

Debug context as a causal system

A context incident should begin with the assembled manifest, not with a fresh conversation asking the model what went wrong. The investigator compares expected and actual evidence, checks transformations, reconstructs placement and then tests whether the model used the evidence.

This causal map creates different corrective actions. Missing source ownership calls for information governance. A filtered entitlement may expose a role design problem. Lost qualifying language calls for transformation changes. A valid context package with an unacceptable outcome calls for model, workflow or policy work.

The correction should be replayed against both the failed case and nearby cases. Adding one source can increase distraction elsewhere. Raising a token ceiling can move critical text into a weaker position. Replacing a summary can change latency and cost. Context changes need regression testing because they alter the model's effective environment.

The model call is the last link in a context supply chain. Production maturity comes from controlling the supply chain, preserving its manifest and assigning failures to the boundary that produced them.

Preserve structure until prose is necessary

Context pipelines often turn every input into paragraphs too early. Structured case records, policy metadata, tool results and permissions become fluent text, then the model is expected to recover the original distinctions. That conversion loses types, optionality and authority.

Keep structured information structured through admission and assembly. A case fact should carry its field name, value, source record, effective time and sensitivity. A policy clause should carry its document hierarchy, jurisdiction and validity. A tool result should carry its schema version and status. Render prose only at the boundary where the selected model needs it.

This improves conflict handling. Two prose statements may appear equally credible. Two typed items can expose that one is an authoritative account status and the other is an unverified customer statement. The assembler can admit both while labelling their relationship. The model is asked to reconcile a declared conflict rather than discover one hidden in wording.

Conflict policy should be deterministic where precedence is known. An active regulation outranks an internal guide. A current product term outranks its superseded version. An authoritative system field outranks a derived memory record. Where precedence is not known, the system should preserve both sources and route the uncertainty rather than invent a hierarchy.

Negative information needs special care. “No vulnerability recorded” may mean an authoritative absence, an unchecked field or a failed lookup. Those states should not collapse into the same sentence. Typed context can distinguish false, unknown, not_applicable and unavailable. That distinction often changes the safe workflow.

Tool schemas also belong in the context product. Expose only tools reachable from the current workflow state. Include parameter meaning, preconditions and side effects, not merely a function signature. A model choosing between draft_refund and execute_refund needs to see their different authority and consequences.

Prose is an interface format, not an evidence model. The closer the context remains to its source structure, the easier it is to validate, budget and reconstruct.

The same principle applies to conversation history. Preserve material facts and decisions as typed state with links to source turns. Summaries can carry narrative continuity, but they should not be the only copy of a consent restriction, promised action or unresolved dispute. A summary model should not decide which legal or operational state survives.

Model adapters can then render the same manifest differently for different providers. One model may work best with XML-like sections, another with compact JSON and another with carefully ordered prose. The underlying evidence, authority and budget remain constant. This makes model comparison fairer because the business context contract is not rewritten for each route.

Structured manifests also support selective disclosure. The assembler can remove prohibited fields before rendering, mask identifiers for a particular model route, and prove which rule caused the transformation. It does not need to scan a large prose blob and hope every sensitive fragment is found.

The practical design goal is not to eliminate natural language. Policies, correspondence and explanations are naturally textual. The goal is to avoid laundering structured control state into ambiguous narrative. Let models interpret language; do not ask language to carry controls that already have a type.

Primary references

• Liu et al., Lost in the Middle, on position-dependent use of long context.
• NIST, AI Risk Management Framework, for governed measurement and lifecycle controls.
• OWASP, Top 10 for LLM Applications, including prompt injection and vector or embedding weaknesses.
• Google Cloud, Vertex AI RAG Engine overview, for managed retrieval architecture and corpus operations.
• AWS, Data architecture in the Generative AI Lens, for data quality and governance guidance.

The competitive asset is not a larger window. It is a controlled method for deciding what deserves to enter it.