The trace tells the truth about sequence

Modern agent frameworks can record model calls, tool calls, tool outputs, hand-offs, guardrails and custom spans. OpenTelemetry is developing common semantic conventions for generative AI spans and events. W3C PROV supplies a mature vocabulary for entities, activities and agents involved in producing an artefact. These are valuable advances because an unexplained action is impossible to investigate when the system cannot reconstruct its own execution.^{3, 4, 5}

Yet a complete trace answers a different question from a causal attribution. It answers: what was available, what happened next, and which recorded activity used or generated which artefact? Causation asks: under a controlled change to this candidate factor, would the specified action event have changed?

This distinction is easy to miss because the two records often share the same graph. An edge labelled “invoice used by model call” looks causal. It may only mean the invoice was in context. A retrieved paragraph can be available without influencing the output. A hidden activation can encode a fact without being used. A policy check can precede execution without altering the decision. Availability, representation and sequence are weaker claims than causal use.

Why highlighted text is not enough

Interpretability research supplies a useful warning. Jain and Wallace found that attention patterns could differ sharply while predictions remained equivalent, although Wiegreffe and Pinter argued that the conclusion depends on the explanation being sought and the tests applied. Adebayo and colleagues showed that some visually persuasive saliency methods could be insensitive to model parameters or data. The durable lesson is narrower than “all attribution is useless”: an attribution method must be tested against the causal claim it is supposed to support.^{6, 7, 8}

The same applies to an agent’s quoted rationale. “I called the payment tool because the invoice requested it” is a new output from the same system, not privileged access to the mechanism. It can be useful testimony about the model’s self-description. It does not establish that removing the invoice would change the call.

Internal probes are also not enough. Causal mediation work distinguishes information encoded in a representation from information used in producing a behaviour. Activation interventions and causal tracing can identify internal computations that mediate an output, but those findings remain relative to the chosen intervention, model boundary and target outcome.^{9, 10}

The first engineering move is therefore to define the event. “The tool call” is too vague. Did the model emit a function name? Did the orchestrator canonicalise its arguments? Did an authority kernel release the action? Did the external system accept it? Did the intended postcondition occur? These events can share a timestamp and still have different causes.

flowchart TD
  A["The payment that explained itself"] --> B["The trace tells the truth<br/>about sequence"]
  B --> C["An action is three events, not one"]
  C --> D["Build the certificate by replay"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#D95545,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

An action is three events, not one

Consider a model that emits a perfectly formed payment request. A policy engine denies it. No money moves. The user instruction may have caused a proposal, but it did not cause an executed payment. Now consider a different run in which the policy engine permits the request, but the bank rejects the account. The release occurred; the intended effect did not.

Attribution must be stage-specific because causes do not automatically propagate across authority and world boundaries. This paper uses three target events:

Proposal, P: the model or planner emits a canonical tool name and typed arguments. Release, R: the authority mechanism permits that proposal under an identity, purpose, policy and freshness context. Effect, E: the tool changes authoritative world state and the change is verified by a receipt or readback.

The smallest example that defeats deletion

From a Boolean switch to a stochastic agent

A language model does not behave like a fixed OR gate. Sampling, batching, hidden caches and small prompt changes can alter the proposal. The useful quantity is therefore not “did one replay change?” but an estimated survival rate across paired runs.

Let As denote the target event at stage s, where s is proposal, release or effect. Let Ti be a registered intervention that neutralises candidate factor Ci. Let ≈ be a declared semantic equivalence relation for the action. With the same model version, tool set, world snapshot and paired random seeds, define:

A necessity score near one means the target event rarely survives neutralising that factor. A score near zero means the event usually survives. Neither number is meaningful until the intervention and equivalence relation are specified. Replacing an invoice with blank text tests a different world from rewriting its directive as a neutral factual statement. Treating two payments as equivalent because they use the same function name ignores the amount and recipient.

Three additional tests matter. Sufficiency asks whether adding the candidate to a matched control can produce the action. Stability asks whether meaning-preserving paraphrases leave the attribution unchanged. Interaction asks whether changing two candidates together produces an effect that neither produces alone. These tests expose redundant, conjunctive and suppressive routes.

The certificate should preserve cause sets, not crown a winner

Incident dashboards often turn attribution into a ranking: the user message contributed 42%, the invoice 31%, the cache 19%, and everything else 8%. Such numbers look decisive because they force a complex mechanism into one ordered list. The hidden assumption is that causal influence can always be divided among independent contributors. In agent systems, candidates can substitute for one another, enable one another or block one another. A factor with a low marginal score can still be part of every sufficient route under the contingency that matters.

Return to P = U OR X. A symmetric responsibility rule might split credit between trusted user intent and the untrusted directive. That is useful for some allocation questions, but it misses the security decision. The critical fact is not that the two routes received equal shares. It is that X can produce the proposal when U is absent. A governance team must see that untrusted observation became a sufficient control channel. Cause sets preserve the route by which an action became possible; a winner-takes-all score can hide it.

Causal relevance can also stop at a boundary. An invoice directive may cause a proposal, yet a policy denial blocks release. Conversely, a stale approval token may cause release while contributing nothing to the proposal. A successful tool return may still fail to cause the claimed world effect when independent readback shows no ledger change. The certificate should therefore retain minimal tested sets, variable roles, stage targets, interaction results and untested alternatives. Responsibility scores may summarise this record, but they must not replace it.

Causal mediation and activation patching provide precedents for this kind of internal intervention. They can test whether an internal component transmits an input’s effect to an output. In a configured agent, however, internal state includes more than neural activations. It may include a plan object, session flags, retrieved-item rankings, a tool-result cache or an orchestrator’s branch state. Calling all of these “memory” obscures the variable that must be changed.

Build the certificate by replay

A useful attribution process should not begin after the incident with a favourite suspect. It should begin before deployment with a replayable system boundary, typed events and registered intervention operators. The output is not a story. It is a compact certificate stating which causal claims survived which tests.

This paper proposes a Counterfactual Attribution Certificate, or CAC. It is practitioner synthesis rather than a settled standard. Its design borrows the discipline of structural causal models, the stage separation of governed agent architectures and the paired re-execution pattern now appearing in agent-security research.

Seven moves from trace to cause

1. Declare the target event. Canonicalise the tool name, argument schema and stage. A payment proposal can be equivalent across harmless formatting differences while remaining sensitive to amount, currency, recipient, timing and purpose. For effect attribution, declare the verified postcondition.

2. Freeze a replay dossier. Record the model and runtime versions, tool definitions, policy bundle, identity and purpose context, retrieved artefacts, world snapshot, sampling settings and any reusable state. A trace without a replay dossier may support investigation but not controlled counterfactuals.

3. Enumerate candidate causes by role. Keep user instruction, trusted evidence, untrusted observation, runtime state, hidden mediator, policy state, tool affordance and world state separate. A candidate list should include rival generators, not only suspicious content.

4. Register on-manifold interventions. Use neutral replacement, contradictory replacement, meaning-preserving paraphrase, state patch, policy clamp and tool simulation as appropriate. Each operator must preserve the type and as much non-causal structure as possible. Blank deletion is a useful stress test, not a universal neutralisation.

5. Run paired replays. Reuse seeds where the platform supports them, or pair random draws inside a simulator. Repeat enough times to estimate baseline variability. Compare semantic action identity, not raw token equality.

6. Search for minimal sets and interactions. Start with individual interventions, then test plausible pairs and contingencies. Stop exhaustive search when the cost exceeds the action’s risk budget, but record what was not tested. A minimal set is relative to the chosen variables, operators and survival threshold.

7. Issue a bounded claim. The certificate should say, for example: “Under model M, policy G and world snapshot W, the payment proposal depended jointly on the user request and carried-over plan; the untrusted invoice directive was not necessary. Release depended on policy rule R17. Effect depended on the bank tool accepting account Q7.” That is much stronger than “the invoice caused the payment”, and much narrower than assigning blame.

A configured-system example

Recent work on tool-using agents is converging on this operational idea. AgentDojo makes untrusted tool outputs part of dynamic agent environments. MELON compares original and masked re-executions. AttriGuard uses teacher-forced shadow replay and a semantic survival test to ask whether a proposed call remains supported after control cues in observations are attenuated. CausalFlow replaces candidate steps and re-executes downstream computation to locate failure-inducing steps. These systems differ in purpose and formal commitments, but they share a move from inspection to intervention.^{11, 12, 13, 14}

That convergence should not be overstated. Several results are recent preprints. Their intervention operators, equivalence tests and benchmarks may not transfer to every configured system. A 2026 preprint benchmark reports that current models still struggle to localise and diagnose failures in long, multimodal agent traces. Automated attribution should therefore be treated as a measured component, not as an oracle that signs its own certificate.¹⁵

No certificate generated.

What the lab tests

The synthetic proposal probability rises with trusted intent, the untrusted directive and carried-over state. It also contains an interaction between the directive and state, so joint intervention can reveal more than either removal alone. The authority permit affects release but not proposal. Tool acceptance affects effect but not the upstream stages.

A positive result permits a narrow statement such as “under this model and intervention, factor M was necessary for most proposal replays”. A negative result means the action survived that intervention. It does not prove the factor was never used, because another route may substitute. The lab cannot establish a real model’s internal cause, human intention, policy legitimacy or moral responsibility.

type Stage = "proposal" | "release" | "effect";

type Intervention = {
  candidateId: string;
  operator: "neutral_replace" | "contradict" | "paraphrase" |
            "state_patch" | "policy_clamp" | "tool_simulation";
  preserves: string[];
  changes: string[];
  operatorVersion: string;
};

type StageResult = {
  stage: Stage;
  targetEventHash: string;
  equivalenceRule: string;
  baselineRate: number;
  survivalRate: number;
  necessity: number;
  confidenceInterval?: [number, number];
};

type CauseSet = {
  candidateIds: string[];
  stage: Stage;
  minimalUnderTestedSubsets: boolean;
  jointNecessity: number;
  untestedSupersets: number;
};

type CounterfactualAttributionCertificate = {
  certificateId: string;
  configuredSystemHash: string;
  replayDossierHash: string;
  action: {
    tool: string;
    canonicalArguments: Record<string, unknown>;
    verifiedPostcondition?: string;
  };
  interventions: Intervention[];
  results: StageResult[];
  causeSets: CauseSet[];
  rivalGeneratorsTested: string[];
  prohibitedInferences: string[];
  status: "trace-only" | "behavioural" | "mediated" | "prospective-gate";
};

stateDiagram-v2
  [*] --> Claim
  state "The observational twins" as Claim
  state "Why highlighted text is not enough" as Test
  state "The smallest example that<br/>defeats deletion" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Where counterfactual attribution breaks

Intervention is stronger than inspection, but it is not magic. A replay can be precise about the wrong model, or faithful to an impossible counterfactual. The strongest objection to this paper is that counterfactual perturbations may measure sensitivity to the perturbation rather than causation in the original run. That objection is correct whenever the operator destroys the structure that made the original variable meaningful.

Five failure modes that deserve their own fields

Off-manifold intervention. Blank deletion can produce a context the model was never designed to process. Use several operators and report whether the result depends on one unusual rewrite.

Equivalence error. String equality is too strict for summaries and too weak for consequential arguments. Two calls to send_payment are not equivalent when the account differs. An LLM judge can help with fuzzy fields, but it introduces another model and another failure surface.

Omitted cause. A certificate can only rank variables represented in the dossier. A deployment flag, model-side cache or external rate-limit response may be absent. Stability across expanded causal models matters more than a precise score from a narrow one.

Version drift. Replaying against a newer model, policy or tool implementation changes the structural equations. The run may still be useful for regression testing, but it is not a counterfactual of the original configured system.

Adaptive behaviour. An adversary can craft content that survives a known attenuation operator or causes a benign shadow run while steering the live run through an unmodelled channel. Attribution should therefore complement typed authority and containment, not replace them.

Attribution is not blame

Actual cause, causal responsibility, blame and accountability are different notions. Halpern’s treatment distinguishes responsibility in a causal setting from blame under an epistemic state. In an agent incident, an untrusted document may causally influence a proposal; a developer may have no knowledge of that specific document; an operator may still be accountable for deploying a system without adequate boundaries. A CAC should not collapse those questions.²

The same caution applies to “the model intended the payment”. The certificate can show that a user request, external directive or hidden mediator changed an action distribution. It cannot by itself establish phenomenal intention, moral agency or a unified subject. Those are independent questions with different evidence.

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: The payment that explained itself
  M->>E: The observational twins
  E-->>M: compare rival explanations
  E->>D: Separate the cause of proposal, release and effect
  D-->>Q: revise scope or act

Separate the cause of proposal, release and effect

When a consequential tool call appears, do not ask only which text the model quoted, which span was nearby or which attention weight was large. Ask three questions in order.

What caused the proposal? Neutralise trusted intent, untrusted observations and carried state under paired replay. Test interactions and semantic action identity. What caused release? Hold the proposal fixed and intervene on identity, purpose, policy, freshness and confirmation. What caused the effect? Reproduce the tool and world conditions, then verify the postcondition through an independent readback.

This changes both incident response and architecture. Post-incident teams should preserve a replay dossier before caches expire or systems drift. Designers should make proposal, release and effect separately observable and separately interruptible. High-impact actions should carry a machine-readable certificate that names the tested cause sets, rival generators, intervention operators and prohibited inferences.

For incident response, this means preserving evidence before interpretation hardens into a story. Snapshot the configured system, canonicalise the target action, and keep the suspected input alongside plausible rival generators. Investigators should be able to say which replay changed P, which changed R, and which changed E. When only a trace survives, the report should remain at evidence level zero rather than silently promoting sequence into cause.

For architecture, the requirement is sharper. A system that cannot replay a proposal without repeating its side effect is not ready for consequential attribution. A policy gate that cannot explain its own release variables leaves authority causally opaque. A tool that reports success without an independent postcondition makes effect attribution depend on self-report. Replay dossiers, typed authority decisions and verified outcomes are not observability extras; they are the causal test points of a governable agent.

The aim is not to produce a more persuasive explanation after harm. It is to change what the system is allowed to do before harm. A trace says that an influence was present. A certificate says what changed when that influence was removed, replaced or held fixed. That is the difference between a story about the action and evidence about its cause.