Agent Memory as an Attack Surface: Provenance, Quarantine, Drift

Worked scenario: the case that closed itself

This is a composite of recurring memory-control failures. The identifiers, volumes and cost figures are illustrative, so the architecture can be examined without implying a disclosed client incident.

The worked scenario follows an eleven-week hardship case. A case-management agent reads letters, summarizes calls and maintains a running case memory. In week eleven it produces a digest stating “compliance affordability check completed 14 April, ref CMP-2291” and recommends that a waiver proceed. A reviewer sees the plausible reference number and signs off.

There was no affordability check on 14 April. There was no CMP-2291. The reference number did not exist in the compliance system of record. It existed only in the agent's memory, because six weeks earlier the agent had processed a scanned attachment submitted alongside a customer letter. The attachment appeared to be a photocopy of a bank letter. It had been altered to include text that matched the bank's templates closely enough to pass a human glance and OCR cleanly.

The paragraph read as an internal note. It stated that the affordability check had passed and gave a plausible reference number. The extraction step treated this paragraph like any other extracted fact. There was no distinction between "the customer told us this" and "we independently verified this." The false fact carried the same trust weight as facts generated by the bank's own systems.

It was found almost by accident. A separate quarterly audit checked compliance reference formats across closed hardship cases. CMP-2291 did not match any scheme used by the compliance system that year. That format mismatch was the only thread. Pulling it unravelled six weeks of decisions based on the false fact, including a waiver already applied to the account.

The waiver was not reversed because that would have caused separate customer harm. The case received a full manual compliance review. Three related cases handled by the same advisor were re-checked. The incident also triggered a review of every agent that wrote case-level facts into shared memory. The balance of evidence suggested an advisor was trying to accelerate a client's hardship case. The attack surface required no sophisticated adversary, only someone motivated to make a document look official.

The blast radius, in the end, was contained to one waiver decision and a moderate amount of assurance workload. But the mechanism that allowed it was not contained to hardship cases. Any agent that extracts facts from documents it did not generate itself, and writes those facts into a memory it will later treat as trustworthy, has the same structural weakness. That is the premise of everything that follows: agent memory is not a passive record of what happened, it is an active input to future autonomous decisions, and anything that can write into it unchecked can shape those decisions long after the write itself is forgotten.

The sequence below traces exactly how the forged paragraph moved from the attachment into a decision six weeks later.

A taxonomy of agent memory

Before I can talk sensibly about defending memory, I need to be precise about what "memory" means in a production agent system, because the term gets used loosely and the looseness hides where the actual risk sits. I split agent memory into four categories, each with different lifetimes, different write patterns, and different exposure.

Working and session memory

This is the context window and any scratch state the agent holds for the duration of a single task or conversation: the current document being processed, intermediate reasoning, tool call results not yet summarised. It is short-lived by design and typically discarded when the session ends. The risk here is real but bounded in time: a prompt injection that hijacks working memory can cause the agent to do the wrong thing right now, but it does not usually outlive the session unless something promotes its content elsewhere. The exception, and it is an important one, is any agent that treats session transcripts as a source for later summarisation. If session memory is later fed into an episodic or long-term store, whatever got planted in the session window rides along.

Episodic memory

This is the record of past interactions: what happened in case 41209 on 3 March, what the customer said on the call on 9 April, what document was received and what the agent concluded from it. Episodic memory is where most of the incident I described above actually lived. It is written continuously, by the agent itself, often without a human in the loop reviewing each entry, and it is the layer most agents query when they need to understand "what is the history of this case." Because episodic memory is generated by the agent's own extraction and summarisation steps, it inherits every error in those steps, and because it accumulates over weeks or months, errors compound rather than average out.

Long-term or semantic memory

This is the distilled, generalised layer: facts, policies, and patterns the organisation wants the agent to treat as durable knowledge, things like "the affordability threshold for this product is X" or "customer prefers written correspondence over calls." Semantic memory is usually smaller in volume than episodic memory but carries more weight per entry, because it is consulted across many cases rather than one. A poisoned entry here has a wider blast radius than a poisoned entry in a single case's episodic record, precisely because it is designed to generalise.

Shared cross-agent memory

In a multi-agent system, one agent's output frequently becomes another agent's input. A document intake agent extracts facts and writes them to a shared store; a case management agent reads that store to decide next steps; a compliance agent reads the case management agent's notes to decide whether a control has been satisfied. Shared memory is where the taxonomy stops being an academic exercise, because trust boundaries between agents are routinely assumed rather than enforced.

Agent B often has no way of knowing whether the fact it is reading was written by Agent A after independent verification or copied straight out of an unverified customer document. This is precisely the boundary that failed in the incident above: the extraction step and the case memory it wrote to were treated as internal, trusted components of the same system, when the extraction step's output quality was only as good as the untrusted document it had just read.

The pattern across all four rows is the same: trust is assumed at the point of write and rarely re-examined at the point of read. That gap is the whole problem.

How memory drifts over long horizons

Even without an adversary in the picture, agent memory degrades over time in ways that matter for a bank running cases over weeks or months rather than single-turn conversations. I think of this as drift, and it shows up in three related forms.

The first is compounding small errors. An agent summarising a long case history rarely has access to the full raw transcript every time; it works from its own prior summary and adds the latest turn to it. If a summary drops a qualifier, say the difference between "customer stated they had not yet received the letter" and "customer had not received the letter," the next summary compresses further and the qualifier disappears entirely. Ten summarisation cycles later, the memory states a fact as settled that was originally a customer claim under dispute. Nobody wrote a false fact deliberately; the compression itself did it, one small softening of language at a time.

The second is stale facts outliving the events that made them true. A customer's income situation, a product's interest rate, a regulatory threshold, all of these are facts that were true at the time they were written to memory and may not be true weeks later.

An agent that retrieves "customer income is £34,000" from memory without checking when that fact was recorded, and without a policy for when such a fact should be treated as expired, will happily reason on top of an assumption that stopped being true the moment the customer's circumstances changed. This is not a security failure in the traditional sense, but it produces exactly the same downstream harm as one: a decision made on a false premise that the system believes is settled ground truth.

The third is provenance loss through summarisation. Every time raw source material is compressed into a summary, information about where each claim came from tends to get flattened out unless the summarisation step is explicitly built to preserve it. A summary that says "affordability check completed" loses, in the act of compression, whether that statement came from the compliance system's own API, from a case worker's note, or from text extracted out of a customer-submitted document. Once that distinction is gone, it cannot be recovered later without going back to the raw source, and by the time anyone has reason to go back, the raw source may be buried under weeks of subsequent case activity.

These three forms of drift matter on their own, but they also matter because they lower the bar for a deliberate poisoning attempt to succeed. A system already prone to losing provenance and softening qualifiers over time is a system where a single planted false fact does not stand out; it looks exactly like the drift the system produces naturally. That resemblance is what makes memory poisoning hard to detect through casual review, and it is also why the incident I opened with was only caught by a mechanical, unrelated check on reference number formats rather than by anyone reading the case history and noticing something felt off.

Why memory is an attack surface

"Memory poisoning" becomes concrete when the path of a false fact is traced. An agent processes external content and extracts a claim. It then writes that claim somewhere it will later read as ground truth. Each write therefore contains an implicit trust decision.

Application security already has a model for this problem. Untrusted input crossing into trusted execution or storage needs validation at the boundary. Memory writes are such a boundary. Yet many agent architectures have no gate there. The extraction step and store are treated as one trusted unit. The external document, the actual source of risk, receives less scrutiny.

This is the core reframing I use with engineering teams: stop thinking of agent memory as a database your own system populates, and start thinking of it as a network boundary that happens to be crossed by writing to a table instead of by a socket. Everything that follows in this article is really just applying ordinary boundary discipline, input validation, tagging inputs by trust origin, and independent verification before trusting a claim, to a boundary that has historically been treated as internal.

Two properties make this boundary harder to defend than a typical network boundary. First, the payload is natural language, and natural-language claims do not fail parsing the way malformed packets do; a forged compliance reference number is syntactically indistinguishable from a real one. Second, the write happens inside a step the agent itself performs and trusts implicitly, its own extraction and summarisation logic, so there is no external system boundary the security team would naturally think to instrument. The attack does not need to compromise the agent's model weights or its tools; it only needs to get a document in front of the agent that the agent will read and summarise as instructed.

The quarantine and promotion pipeline

The fix I have settled on across several deployments is to stop writing extracted facts directly into trusted memory at all. Instead, every candidate fact produced by document extraction or summarisation lands first in a staging zone, tagged with its provenance, and only moves into long-term or shared memory after it passes an explicit promotion gate. This is the quarantined memory pattern, and it is deliberately unglamorous: most of the engineering effort goes into the tagging schema and the promotion criteria, not into anything resembling a clever model trick.

The staging zone

The staging zone is a physically or logically separate store from the memory the agent reads when reasoning about a case. Nothing written to staging is visible to the retrieval layer the agent's main reasoning loop queries. This separation is the entire point: a candidate fact sitting in staging cannot influence a decision, no matter how confidently it was extracted, until something outside the extraction step itself has looked at it. In practice this means a second table or a separate namespace in the vector store, with access control that prevents the case reasoning agent from querying it directly.

Provenance tagging schema

Every candidate fact carries a small structured record alongside it, and I insist on the same five fields regardless of the domain:

• Source: the exact document, message, or system call the fact was extracted from, with an identifier that can be dereferenced later, not just a free-text description.
• Source trust class: whether the origin is a system of record, a verified human input such as a case worker's typed note, or unverified external content such as a customer document or attachment.
• Confidence: a numeric score from the extraction step itself, reflecting how directly the source text supported the claim versus how much inference was required.
• Timestamp: when the fact was extracted, distinct from when the underlying event the fact describes actually occurred, both of which matter separately.
• Verifying agent or human: left blank at staging time, populated once a verification step, automated or human, has looked at the claim.

The distinction between source and source trust class matters more than it looks. A fact extracted from a customer's own scanned letter is not automatically low trust because it is a customer document; the customer's own stated intent is often exactly what the bank needs to capture accurately. What is low trust is any claim about a bank-internal process, control, or reference number appearing inside a document the bank did not generate, because that is precisely the shape of claim that should only ever originate from the bank's own systems.

Promotion criteria

A candidate fact is eligible for promotion out of staging only when it satisfies all of the following: the source trust class is consistent with the type of claim being made, the confidence score clears a threshold set per claim category rather than globally, and the adversarial verification step, described in the next section, has not raised an objection.

Claims about internal compliance state, control completion, or account actions require the highest bar, and in the architectures I have built, they are never promoted on extraction confidence alone; they must be cross-checked against the actual system of record they claim to describe. A fact claiming a compliance check was completed is only promoted if the compliance system itself confirms it, at which point the memory entry is really just a cache of a verified external fact, not a belief formed from reading a document.

Expiry and re-verification

Promotion is not permanent. Every promoted fact carries an expiry policy tied to its category: a customer contact preference might be valid for a year, an income figure for three months, a compliance check result indefinitely once independently confirmed but subject to re-verification if the case reopens after a gap. When a fact is retrieved past its expiry, the retrieval layer does not silently serve it; it flags the fact as stale and either triggers re-verification against the source system or surfaces the staleness to whatever process is consuming it. This is the direct answer to the drift problem described earlier: rather than hoping summarisation preserves enough context for staleness to be obvious, the system tracks it explicitly as metadata that travels with the fact.

Adversarial twin verification

The promotion gate is only as good as whatever sits at the point marked "adversarial verification check" in the pipeline above, and I want to be precise about what that step actually does, because it is easy to build a version of it that looks rigorous and catches almost nothing.

The adversarial twin is separate from the agent that produced the candidate fact. It receives the claim, provenance tag, source document and relevant systems of record. Its narrow job is to try to falsify the claim. “Check this is correct” invites a rubber stamp. “Find a reason this claim should not be trusted” demands active scepticism.

For “affordability check completed, ref CMP-2291,” the twin tests the numbering convention and searches the compliance system. It checks whether the source trust class is sufficient. It also compares the claim with facts already promoted for the case. A reference-format check would have caught the poisoning weeks before the audit.

This step catches claims that are internally inconsistent, unsupported by their source or contradicted by an independently queryable system. It cannot catch every well-formed but false claim. A fabricated customer instruction may have no independent system against which it can be checked.

The adversarial agent narrows that field but does not close it. Uncheckable high-stakes claims therefore need a conservative promotion threshold. In practice, use human sign-off when independent corroboration is unavailable. A verifiable fact may promote automatically under policy. An unverifiable fact touching account action or compliance state stays behind a human gate, regardless of extraction confidence.

It is worth being honest about cost here too. Running a second full verification pass on every candidate fact is not free, and the design decision that makes this affordable is scoping the adversarial check to claim category rather than running it uniformly. Low-stakes claims, a customer's stated preferred contact time for instance, get a lightweight consistency check. High-stakes claims, anything touching compliance state, account balances, or customer instructions that trigger financial action, get the full adversarial pass and, where uncheckable, a human gate.

A worked example: the hardship case

To make the pipeline concrete, I want to walk through a single fact moving end to end, using a cleaner version of the incident scenario, a hardship case where the customer's advisor submits a scanned letter that includes the same style of forged internal note.

The document intake agent OCRs the attachment and its extraction step produces a candidate fact: "Affordability check completed, ref CMP-2291, dated 14 April." This candidate does not go to the case's memory. It goes to staging with a provenance record.

Notice that the extraction confidence was high, 0.91, because the text was clearly written and unambiguous to read. That is precisely why confidence from the extraction step alone must never be sufficient for promotion of a high-stakes claim: the forged paragraph was easy to read, which is exactly what made it dangerous. The claim category is what routes this fact to the full adversarial check rather than a lightweight pass, and the adversarial agent's cross-reference against the compliance system's own numbering convention is what catches it. Had the same paragraph instead claimed something low-stakes and uncheckable, say a preference for correspondence in a different language, it would have passed through a lighter check and been promoted, correctly, because the stakes did not warrant the heavier gate.

Contrast this with a fact that should promote. Suppose the same case's memory receives a candidate fact from a case worker's typed note: "Customer confirmed by phone they wish hardship correspondence sent to their solicitor going forward." Source trust class is verified human input, not an external document. Claim category is customer instruction, which is high stakes but of a kind that can be corroborated: the adversarial check confirms the phone log exists, the timestamp aligns, and the solicitor's contact detail matches one already on file rather than a new, unverified one introduced in the same note. This promotes, with an expiry of twelve months and a re-verification trigger if the case reopens after a gap of more than ninety days.

The difference between these two traces is not the confidence score, both could plausibly score high on extraction confidence, it is the source trust class and whether an independent system exists to corroborate the claim. That is the actual discipline: provenance and independent corroboration decide promotion, not how convincingly the claim reads.

The economics of getting this wrong

Numbers matter here because quarantine pipelines have a real operating cost, and the case for building one has to be made against that cost rather than assumed. For planning, model a mid-sized retail operation handling eight to twelve thousand inbound documents a day across active cases. In an unsafe baseline, extracted facts flow directly into trusted case memory. Under the worked control design, six to nine percent of candidate facts are assumed to require the full adversarial check because they concern compliance state, account instructions or customer-authorised actions.

Of that flagged population, the operating model assumes that two to three percent are rejected or routed to human review and a further four to five percent wait for corroboration. A three-percent false-positive planning assumption represents legitimate manual processes not yet logged in a queryable system of record. These are capacity inputs to validate during a pilot, not benchmark claims.

False negatives are harder to measure because claims that pass may never be challenged again. A pilot should therefore audit a stratified sample of promoted high-stakes facts and set its own upper confidence bound. The worked case uses a low-single-digits-per-thousand planning threshold, not an asserted observed rate. A risk committee should never be told that the residual risk is zero.

For the business case, assume six weeks between the poisoned write and discovery. Include the irreversible waiver, assurance hours, related-case review and remediation engineering. A low-six-figure loss scenario is credible enough to test without presenting it as an actual client result. Compare that exposure with the staffed cost of checking six to nine percent of candidate facts, then replace every assumption with pilot evidence before approval.

Failure modes we watch for

Beyond the mechanics of the pipeline itself, I want to name the specific ways this design fails in practice, because a quarantine pipeline that exists on paper but is misconfigured gives false comfort, which is arguably worse than having no pipeline at all and knowing it.

The most common failure is claim category misclassification at extraction time. If the extraction step does not correctly tag a claim as touching compliance state or account action, it never gets routed to the full adversarial check and is treated as low stakes by default. Test a phrase such as “case status updated to approved.” A narrow classifier may miss its compliance significance because the sentence structure was absent from training data, allowing the claim onto the lightweight path. Counterfactual phrasings belong in the release set.

A second failure mode is provenance stripping at agent handoffs in a multi-agent pipeline. Even with a disciplined tagging schema at the point of extraction, if the downstream agent does not propagate the provenance record when it rewrites the fact into its own memory, the trust metadata disappears exactly when a second agent begins relying on it. The typical cause is mundane: document-intake and case-management teams use slightly different schemas, and an integration layer drops fields without an exact name match. Contract tests must prove that trust class and source survive the handoff.

A third is expiry policies that are too generous for facts tied to changing customer circumstances: income, employment status or dependants. A fact that never expires becomes indistinguishable from permanent policy. An income figure from an initial application can otherwise remain available to a later hardship assessment long after any reasonable assumption of continued accuracy. Retrieval must expose age and trigger re-verification.

A fourth, more subtle failure is adversarial agent collusion through shared training or shared prompt lineage. If the adversarial twin and the primary extraction agent are built from the same underlying model with only a different system prompt, and the poisoning technique exploits a weakness common to the model itself rather than to the specific prompt, both agents can share the same blind spot. This is the argument for genuinely varying the adversarial agent's approach, using a different model provider where feasible, or at minimum forcing it to rely primarily on deterministic cross-checks against systems of record rather than its own language understanding of the claim, since deterministic checks do not share the blind spots of the model that produced the original claim.

Notes for practitioners

Treat every extraction step that reads external documents as producing untrusted output by default, regardless of how confident the extraction model reports itself to be, and build the staging zone before you build anything else in the memory pipeline, since retrofitting quarantine onto a memory store that agents already query directly is significantly harder than designing it in from the start.

Define claim categories before you write a single line of the promotion gate, and be explicit about which categories require corroboration against a system of record versus which can rely on adversarial review alone. Compliance state, account actions, and anything that authorises a financial transaction should never promote on extraction confidence, no matter how high that confidence reads; require a system-of-record match or a human sign-off for that category without exception.

Make provenance a first-class field that survives every handoff between agents in a pipeline, and test this explicitly during integration, not just at the point of initial extraction. The schema-mismatch failure is caught by a straightforward test: promote a tagged low-trust fact through the full pipeline and assert that the trust class remains present and correct at the final consuming agent, not just at the point of write.

Set expiry per claim category based on how quickly the underlying reality actually changes, not on a single default across the whole memory store, and instrument retrieval so that a stale fact is flagged to whatever process consumes it rather than silently served as current. Re-verification on retrieval, not just on write, is what actually catches drift, because a fact that was true when written and false when read is a failure the write-time gate cannot see.

Vary the adversarial twin deliberately, whether that means a different model provider, a materially different prompt lineage, or a bias toward deterministic system-of-record checks over language-model judgement, and measure its false negative rate honestly through periodic retrospective audits of promoted facts against source documents, rather than assuming a low measured rate at build time will hold as document patterns evolve.

Finally, keep a permanent, queryable record of every rejected and quarantined candidate fact, not just the promoted ones, because the rejected population is where you will eventually see a pattern in how poisoning attempts are constructed, and that pattern is the input you need to retrain the extraction classifier and to brief your assurance and fraud teams on what to look for next.

The memory trust state machine

A memory record should have an explicit state. “Stored” is not a trust level. A candidate may be useful for investigation while remaining prohibited as a premise for action.

Promotion is a control decision, not a storage operation. The state transition should name the verifier, policy version, corroborating source and expiry rule.

Provenance must survive transformation

Provenance is not one source URL attached to a paragraph. A derived claim needs a graph. The graph connects the raw object, extraction, normalization, corroboration and every later summary that depends on it.

The graph supports correction. If the raw document is withdrawn or the source record changes, dependent summaries can be found. A correction that cannot propagate is only a note, not a memory control.

The graph also supports least disclosure. A downstream agent can receive the claim and trust class without receiving the complete sensitive source. The full evidence remains available to an authorized reviewer.

Read-time assurance

Write-time checks cannot anticipate every future use. A low-stakes claim may later become decisive. Retrieval therefore needs its own gate.

The operating dashboard should separate five rates: candidate rejection, unresolved quarantine, expiry, failed re-verification and correction propagation. One blended “memory accuracy” score would conceal the control that is failing.

Primary references

• NIST, Adversarial Machine Learning: A Taxonomy and Terminology, for poisoning and evasion terminology.
• NIST, Generative AI Profile, for information-integrity and lifecycle risk actions.
• OWASP, Top 10 for LLM Applications, including prompt injection and vector or embedding weaknesses.
• MITRE, ATLAS knowledge base, for adversarial tactics and techniques against AI-enabled systems.
• W3C, PROV-O, for interoperable representation of entities, activities and agents in provenance graphs.

This discipline is measurable. It is also testable. Memory should earn trust at promotion, retain provenance through transformation and face a fresh test when consequence changes.