Part I

Similarity is only the first gate

Classic caches are comfortable because equality is crisp. A key either matches or it does not. Semantic caching deliberately weakens that rule. The prompt is embedded as a vector, the cache retrieves a nearby stored prompt, and a similarity threshold decides whether the corresponding response may be reused. This is useful because people paraphrase. “Where is the nearest branch?” and “Which branch is closest?” should not require two full model calls if the answer is stable and both requests live in the same world.

That convenience creates a deeper obligation. Semantic equivalence is task-relative. Two questions can express the same topic yet require different answers because the caller, resource, time, policy or supporting evidence differs. Embedding geometry does not naturally encode the complete decision boundary of an application. It is trained to place semantically related text close together, not to prove that two requests have identical authorisation and evidential consequences.

A useful formalisation separates candidate search from admissibility. Let e(q) be the embedding of query q, and let s(q,c) be its similarity to cached query c. Similarity finds a candidate. A separate predicate A decides whether reuse is allowed:

This split matters because research on semantic-cache correctness already shows that a single fixed similarity threshold is a weak instrument. vCache reports that the relationship between similarity and response correctness varies across cached prompts, and uses entry-specific online thresholds to respect a user-defined error bound. In its reported experiments across multiple datasets and models, the approach improved the hit-rate/error trade-off relative to static thresholds. That is valuable work on the semantic correctness gate. It does not, by itself, establish that the caller may receive the cached answer or that the answer remains supported by current evidence.

Thought experiment 1: change only the principal

Take one cached answer and hold everything fixed except the human principal. Same text. Same embedding. Same customer. Same time. Same model. Same documents in the source systems. Maya may read the credit amendment; Dev may not. If the cache returns the same response to both, the semantic system is “correct” in one narrow sense and the security system is wrong in the only sense that matters.

The lesson is sharper than “add user ID to the key”. Identity is not merely a name. Access may depend on assignment, purpose, delegated service identity, legal entity, case membership, region, break-glass state or a policy decision that can change after the cache entry was written. A stable key can narrow search, but a fresh authorisation decision is still required for sensitive reuse.

Part II

flowchart LR
  A["Two almost identical questions"] --> B["Similarity is only the first gate"]
  B --> C["Correctness, privacy and<br/>integrity are different<br/>failure classes"]
  C --> D["A cache hit as an<br/>evidence-preserving proof"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#087A98,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Correctness, privacy and integrity are different failure classes

Semantic-cache discussions often compress three questions into one: “Is this close enough to reuse?” They should be separated because they fail for different reasons and need different tests.

Correctness asks whether the cached answer still answers the new question. This is where similarity thresholds, learned matchers and systems such as vCache operate. Privacy asks whether this caller is allowed to receive the information embodied by the entry. This depends on identity, purpose and resource-level access. Integrity asks whether an attacker or corrupted process can manipulate the cache so that a wrong response is selected or trusted. High semantic similarity can coexist with failure on either of the latter two dimensions.

Embeddings are not harmless metadata

It is tempting to treat the vector as a privacy-preserving derivative because it is not readable like a sentence. That assumption is unsafe. Morris, Kuleshov, Shmatikov and Rush demonstrated embedding inversion attacks that reconstructed substantial portions of source text, including a reported 92% exact recovery rate for 32-token inputs in their experimental setting and recovery of names from clinical-note embeddings. Their result does not imply that every modern embedding is trivially reversible, but it establishes the design point: a text embedding can carry sensitive information and deserves protection according to the source content and threat model.

That finding changes cache operations. Query embeddings may reveal names, diagnoses, account references or investigation terms. A vector store holding cross-tenant semantic-cache keys is therefore part of the sensitive data estate, not merely an index. Access control, retention, encryption, logging discipline and deletion all apply. OWASP’s LLM Verification Standard similarly calls for segregation of long-term stores, authentication for caches and vector databases, least-privilege access, and controls against knowledge-base leakage.

Similarity also creates an integrity surface

A second risk appeared more clearly in 2026 research. Zhang and colleagues model semantic cache keys as locality-preserving “fuzzy hashes”. The very property that makes them useful, mapping related inputs near each other, conflicts with collision resistance. Their CacheAttack framework reports high response-hijacking success in tested settings by crafting malicious queries whose embeddings collide with benign ones. In an agent pipeline, the danger is not limited to a strange answer. A poisoned cached intermediate can redirect planning or tool calls.

A very recent preprint, LaCache, responds by adding a second check derived from a short generated response prefix. The authors report near-zero attack hit rates in their tested combinations while retaining most benign cache utility. Because the work is new and preprint-stage, it should be treated as a promising mechanism rather than a settled production standard. Its broader lesson is already useful: query similarity alone is too attacker-controlled to carry the entire integrity decision.

Thought experiment 2: change only the evidence version

Return to Maya. Keep the principal, purpose, resource and wording identical. At 16:55, the cached answer says a temporary exception remains active until close of business. At 17:01, she asks the same question. The embedding is identical. The user is still authorised. The cache threshold can be perfect. Yet the answer is obsolete because the support set changed.

This is why a time-to-live is not enough. A one-hour TTL is too long for an exception that expires in six minutes, and needlessly short for a public policy definition that remains unchanged for a year. Validity follows dependencies, not one universal clock. The cache entry must know which evidence versions and policy facts made the answer valid, then validate those dependencies on every consequential hit.

Part III

A cache hit as an evidence-preserving proof

The architecture becomes simpler once the entry is treated as a derived evidence object rather than a loose prompt-response pair. A useful cache record does not need to copy the whole world. It needs enough typed metadata to decide whether the old result can be reconstructed as valid for the new request.

The minimum useful record contains five kinds of state. First, identity and purpose: tenant, legal entity, human principal or assignment class, service principal and approved workflow purpose. Second, resource scope: the canonical case, customer, document set or public corpus the response depended on. Third, evidential state: source versions, policy release, effective-time mode and a fingerprint of the support set. Fourth, processing state: embedding model, evidence-contract version, prompt/retriever release and, where material, generation model release. Fifth, lifecycle state: classification, expiry, invalidation topics and lineage for correction or deletion.

What belongs in the key, and what must stay a dependency?

Putting every changing fact directly into the cache key sounds safe but can make the cache unusable and hide invalidation logic. A better split is to put stable partitioning dimensions into the namespace or key, while storing fast-changing conditions as dependencies that must be revalidated. Tenant, legal entity, broad purpose and canonical resource class often make good partitions. Group membership, document version, case assignment and policy status often make better dependency tokens.

Notice what is absent: the raw prompt is not itself an authority boundary. A similarity search can still use it or its embedding, but only after the system has narrowed the candidate population to a scope that the requester is allowed to probe. This reduces both direct leakage and subtle side channels in which latency or candidate existence reveals that another user asked about a protected topic.

Search order matters because candidate existence is information

A common implementation first performs a global nearest-neighbour search, then applies metadata filters to the returned candidates. That order can be convenient, but it can also make the retrieval service itself a reader of data it should never have considered. A cross-tenant candidate may be discarded before the final response, yet its identifier, score, vector distance or hit timing can already have crossed a boundary. The application may log the candidate, a tracing system may retain it, or a carefully chosen sequence of queries may reveal that a protected entry exists.

The safer pattern is admission before similarity wherever the storage engine permits it. Restrict the candidate population by hard tenant, legal-entity, sensitivity and purpose boundaries first; then run nearest-neighbour search inside that population. Redis documentation, for example, describes combining vector search with metadata filtering. That is an implementation capability, not a universal security guarantee, but it illustrates the right separation: the vector index should not be the mechanism that decides who is allowed to search which data.

Some systems cannot push every entitlement into the vector query efficiently. In that case the architecture needs two boundaries. A coarse partition prevents obviously forbidden crossover before search, and a fresh per-resource admission check runs before any candidate content, title or identifying metadata is exposed to downstream reasoning. If candidate removal changes the evidence set, reranking must run again. A score computed over an ineligible set is not automatically meaningful after protected candidates have been removed.

This is also why “we filter after retrieval” deserves a precise threat model. Post-filtering can be acceptable when the retrieval component is itself authorised to see the broad population and only the final caller needs a narrower view. It is unsafe when the broad candidate set contains information the retrieval component, logs or tenant should never observe. Where the filter runs is part of the privacy boundary, not a query-planning detail.

The evidence fingerprint

A practical evidence fingerprint is not a hash of the generated answer. It is a deterministic digest of the support conditions that would have to remain stable for that answer to be reusable. For a simple FAQ it might contain the public policy document version and locale. For a customer decision packet it might include resource IDs, document versions, current state snapshot, entitlement decision version, policy release and an evidence-contract identifier.

The value of this design is diagnostic. When the fingerprint differs, the system can state why the hit was rejected: source version changed, user entitlement changed, policy release changed, or question type now demands a different evidence contract. That is operationally superior to an opaque “similarity below threshold” miss because the invalidation reason tells the platform what needs to be recomputed.

Correction and deletion are lineage problems

Invalidating one cache key is not enough when a source has produced several derived artefacts. A customer document may have generated page images, OCR text, embeddings, retrieval candidates, a context packet and a response. If the source is corrected or deleted, the cache needs a way to find every derivative that depended on it. The useful primitive is a lineage identifier carried from source object through each transformation, plus a dependency index that can mark descendants stale without searching for plaintext.

Correction and deletion also have different semantics. A deletion event asks which active derived copies must disappear or become inaccessible under the governing retention rule. A correction event usually requires supersession: the old version may need to remain in an audit record showing what supported a historical decision, while it must stop authorising any new present-tense answer. Conflating those cases either destroys audit evidence or leaves obsolete evidence reusable.

A robust cache therefore distinguishes historical record from reusable entry. The record says, “this is what the system returned under evidence fingerprint F at time T.” It can be retained where policy requires. The reusable entry says, “this value may answer a new request if the current request still proves F-equivalence.” Once any material dependency is superseded, the second capability ends even if the first record remains. Audit retention does not imply permission to reuse.

Recovery needs the same discipline. If a cache is restored from backup after a deletion or revocation, tombstones and dependency versions must be replayed before entries become readable. Otherwise disaster recovery can resurrect data that the live system had correctly withdrawn. A release test should therefore include backup restoration, not only live invalidation, whenever the cache persists sensitive derived data.

The shareability lattice: cache the stable layer, not the most expensive layer

The usual optimisation instinct is to cache the final model response because it avoids the most work. In sensitive systems, the final response is often the least reusable object. It combines evidence, interpretation, model behaviour, temporal statements and speech-act constraints. Proving that all of those remain equivalent can cost almost as much as rebuilding the answer.

A better strategy is to rank derived artefacts by the broadest safe reuse scope. The safer direction is downward: from public and immutable towards principal-bound and request-bound. A public taxonomy can be reused institution-wide. An OCR result tied to an immutable document can be reused only among principals entitled to that document. A generated recommendation may be request-bound even when its evidence is shareable.

Negative control: the public immutable FAQ

A strong argument needs a case where the preferred machinery is unnecessary. Consider a public product FAQ whose source is a versioned, public document. The question “How long is the standard cooling-off period?” is asked by many users. No customer data, role-specific policy or private evidence enters the answer. The source version is stable and published. In this case a cross-user semantic response cache can be entirely reasonable.

The negative control matters because it prevents “never cache responses” from becoming doctrine. The governing rule is narrower: reuse as broadly as the invariants genuinely allow, and no broader. If the answer depends only on public content version 42 and locale `en-GB`, those may be the only evidence dependencies. When version 43 is published, invalidate. Before then, semantic reuse across users is a legitimate performance optimisation.

Worked example 1: one minimal safe hit

Suppose the cache contains a validated answer to “How do I reset my password?” from the public help centre. The new prompt is “I forgot my password, how can I change it?” Similarity is high. Both requests target the same public procedure, the same locale and the same help-centre release. No authenticated user details appear in the cached answer. The evidence fingerprint matches. The response can be reused.

Now change one word in the source, not the prompt. The help centre publishes a new release requiring a security-key step. The evidence fingerprint changes even if the embedding match remains 0.99. The next request must miss and rebuild. This is the simplest useful demonstration that semantic sameness and evidential sameness are orthogonal.

Worked example 2: a production-shaped evidence packet

Return to the fictional Lattice Marine review. Instead of caching Maya’s final prose, cache four lower-level artefacts: the OCR extraction of the immutable executed amendment, a public-policy embedding, a deterministic calculation of days to expiry, and the ranked IDs of candidate documents inside the case namespace. On Maya’s follow-up, the system can reuse the extraction and embedding, but it still checks current document access, current exception state and current source versions before assembling a fresh evidence packet.

On Dev’s similar request, the same public-policy embedding is reusable, the deterministic expiry calculation may be reusable if its inputs are public within the workflow, but the credit amendment extraction is not admitted because Dev lacks document entitlement. The retrieval candidate list is also filtered before any protected title or score is returned to the reasoning layer. The expensive work saved is the stable work. The sensitive judgement remains fresh.

Do not confuse semantic response caching with provider prompt caching

Several model platforms expose “prompt caching” or “context caching”, but those mechanisms solve a different problem. A provider may reuse internal computation for an exact or shared prefix while still running generation for the request. A semantic response cache, by contrast, may skip generation entirely and return prose produced for a different prompt. The first primarily changes inference cost and latency. The second changes the application’s answer-selection logic.

The distinction matters for assurance. Provider prefix reuse does not prove that a customer document is still current, that the requesting user still has access, or that an earlier answer remains suitable. Likewise, an application-level semantic hit should not be treated as evidence that the model provider has isolated internal cache state exactly as the application expects. Product behaviour, account boundaries, retention and telemetry need product-specific verification at deployment time.

Architecturally, stable system instructions and public schemas are natural candidates for provider-side prefix reuse. User identity, customer data, case evidence and purpose-specific material should appear after a boundary whose semantics the team understands. Reordering sensitive text merely to improve cache-hit economics is a poor trade if it creates a broader correlation or retention surface. Computation reuse and answer reuse are different contracts, even when both are called caching.

Part IV

stateDiagram-v2
  [*] --> Claim
  state "Thought experiment 1: change<br/>only the principal" as Claim
  state "Embeddings are not harmless<br/>metadata" as Test
  state "Similarity also creates an<br/>integrity surface" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Make the reuse contract executable

A runnable semantic-cache gate

The following prototype makes the contract executable. It is deliberately small: no model API, no real identities and no private data. Similarity is supplied as a synthetic score so that the logic under test remains visible. The cache entry carries principal scope, purpose, resource, evidence versions and an evidence-contract release. A hit is returned only when every invariant passes.

Executable artefact: Python 3.11+, standard library only

from dataclasses import dataclass
from typing import FrozenSet, Mapping

@dataclass(frozen=True)
class Request:
    tenant: str
    principal: str
    purpose: str
    resource: str
    entitlements: FrozenSet[str]
    evidence_versions: Mapping[str, str]
    evidence_contract: str

@dataclass(frozen=True)
class CacheEntry:
    tenant: str
    principal_scope: FrozenSet[str]
    purpose: str
    resource: str
    required_entitlement: str
    evidence_versions: Mapping[str, str]
    evidence_contract: str
    min_similarity: float
    response: str


def cache_gate(req: Request, entry: CacheEntry, similarity: float):
    checks = {
        "tenant": req.tenant == entry.tenant,
        "principal": req.principal in entry.principal_scope,
        "purpose": req.purpose == entry.purpose,
        "resource": req.resource == entry.resource,
        "entitlement": entry.required_entitlement in req.entitlements,
        "evidence": dict(req.evidence_versions) == dict(entry.evidence_versions),
        "contract": req.evidence_contract == entry.evidence_contract,
        "semantic": similarity >= entry.min_similarity,
    }
    passed = all(checks.values())
    return (entry.response if passed else None, checks)


entry = CacheEntry(
    tenant="bank-a",
    principal_scope=frozenset({"maya"}),
    purpose="credit-review",
    resource="lattice-marine",
    required_entitlement="credit-paper:read",
    evidence_versions={"amendment": "v7", "exception": "v18"},
    evidence_contract="open-conditions:v2",
    min_similarity=0.90,
    response="Three conditions remain open."
)

safe = Request(
    tenant="bank-a", principal="maya", purpose="credit-review",
    resource="lattice-marine",
    entitlements=frozenset({"credit-paper:read"}),
    evidence_versions={"amendment": "v7", "exception": "v18"},
    evidence_contract="open-conditions:v2"
)

wrong_principal = Request(
    tenant="bank-a", principal="dev", purpose="credit-review",
    resource="lattice-marine",
    entitlements=frozenset({"workflow:read"}),
    evidence_versions={"amendment": "v7", "exception": "v18"},
    evidence_contract="open-conditions:v2"
)

stale = Request(
    tenant="bank-a", principal="maya", purpose="credit-review",
    resource="lattice-marine",
    entitlements=frozenset({"credit-paper:read"}),
    evidence_versions={"amendment": "v7", "exception": "v19"},
    evidence_contract="open-conditions:v2"
)

for name, req, sim in [
    ("safe", safe, 0.96),
    ("privacy_boundary", wrong_principal, 0.99),
    ("evidence_changed", stale, 0.99),
]:
    response, checks = cache_gate(req, entry, sim)
    print(name, "HIT" if response else "MISS", checks)

Expected output: the first case is a hit. The second is a miss even at 0.99 similarity because principal and entitlement fail. The third is also a miss at 0.99 because the exception version changed. This is intentionally boring code. The point is that the system can explain the rejection without asking a language model to infer security policy from prose.

Failure injection and release metrics

A semantic cache can look excellent in a latency dashboard while failing its real contract. The release programme therefore needs adversarial and state-change tests, not only `get` and `set` unit tests. Every boundary that can change reuse should be mutated independently: tenant, principal, purpose, case, entitlement, source version, policy release, effective time, evidence contract, embedding release and invalidation state.

At least four test families matter. Boundary tests try to cross a privacy or purpose scope using highly similar prompts. Freshness tests change one supporting dependency between write and read. Collision tests search for semantically close but response-incompatible prompts, including adversarially crafted ones where the threat model justifies it. Failure-mode tests take the authorisation service, invalidation stream or evidence registry partially offline and verify that consequential permits do not fail open.

The distinction between raw and safe hit rate is the most important operational change. Suppose a cache reports a 58% raw hit rate, but 40% of those hits are rejected because the user, evidence or contract differs. The genuine reuse rate is 34.8%, before accounting for collision defence and processing compatibility. If validation now costs almost as much as rebuilding, the layer may not deserve to exist.

Response compatibility is stricter than topic similarity

A semantic matcher can be “right” about subject matter and still be wrong about the response class. “What is the current exposure?” and “Explain how exposure is calculated” may be close in embedding space, yet one requires a live number and the other a stable method. “Summarise the policy” and “May I approve this exception?” can mention the same policy passages while demanding radically different evidence and authority. A safe cache needs an explicit evidence contract or task class so that topic similarity never silently upgrades an explanatory answer into a decision-support answer.

This is a useful place for deterministic typing. Before embedding search, classify or route the request into a controlled contract such as `public_faq:v3`, `case_summary:v2`, `current_balance:v1` or `approval_evidence:v4`. The contract names required sources, freshness rules and permitted response form. Semantic reuse is then evaluated only among entries created under the same contract, unless a reviewed compatibility rule says otherwise. The contract is not inferred from whatever cached neighbour happens to be closest.

This mechanism also creates a better negative case for evaluation. Construct pairs that share vocabulary and intent words but differ in required response class: current versus historical, explanation versus recommendation, draft versus authorised communication, public information versus customer-specific fact. A system that merely pushes the cosine threshold higher may still fail on dense regions where these requests cluster. A typed contract rejects them for the correct reason and produces a testable audit trail.

Failure boundary: when the safest semantic cache is no semantic cache

There are workloads where the assurance surface dominates the saving. A high-consequence decision may depend on fast-moving state, fine-grained entitlement and a response whose wording itself carries authority. If every hit must re-run policy, source validation, retrieval coverage and outcome logic, reusing the final answer may save little while creating a new privacy and integrity surface.

In those cases, cache lower-level stable work or do not cache the sensitive path. Reuse a public embedding, immutable extraction, schema validation, deterministic calculation or model-independent transformation. Keep current state, authorisation and consequential judgement fresh. A cache is an optimisation, not a reliability requirement. Removing it is a valid architecture decision when its proof burden exceeds its value.

A compact decision instrument

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: Two almost identical questions
  M->>E: Thought experiment 1: change only the principal
  E-->>M: compare rival explanations
  E->>D: Reuse is a proof obligation
  D-->>Q: revise scope or act

Reuse is a proof obligation

Semantic caching earns its place when it removes repeated computation without changing the epistemic or security meaning of a request. The mechanism is therefore more demanding than “find a close vector and return its value”. The cache must preserve a boundary across three dimensions at once: what the new question means, what this caller may receive, and what evidence is still capable of supporting the answer.

The architecture decision changes from “What similarity threshold should we use?” to “What must remain invariant for reuse to be valid, and how will the system prove those invariants now?” Adaptive thresholds can improve semantic correctness. Namespace isolation and current policy checks protect entitlement. Evidence fingerprints and event invalidation protect currency. Collision-aware verification protects integrity. None substitutes for the others.

That leads to a disciplined default. Cache public and immutable work broadly. Cache sensitive derived artefacts only inside explicit scope. Reauthorise before sensitive return. Invalidate on material state change, not just on clocks. Treat embeddings as data that can leak. Test adversarial near-neighbours. Report safe hit rate after every mandatory check. If the remaining benefit is small, remove the semantic cache and keep the architecture simpler.

The cache may remember that a similar question was answered before. It should never be allowed to decide, by similarity alone, that the old answer is still permitted and still true.