The result that vanished

The first instinct after an AI incident is to rerun the prompt. That instinct treats the model call as the experiment. In a compound system, it is only one moving part.

The visible request may be unchanged while the effective execution differs. A prompt template can add instructions. Retrieval can select another document. A tool can return a new world state. A policy service can narrow authority. A cache can reuse an old answer. The orchestrator can reorder parallel calls. The provider can serve a different model revision behind a stable alias. Even local numerical software may not produce identical results across releases, devices or backends, despite identical seeds.[9]

Machine-learning research has long treated code, data, hyperparameters and reporting discipline as reproducibility concerns.[1] That remains necessary. It is no longer sufficient for a tool-using agent whose result depends on mutable services and delegated authority. The object that must be reproduced is the configured execution, not the text sent to a model.

The same visible request can cross six distinct decision surfaces. Context assembly decides what the model is allowed to see. The model proposes an interpretation or action. Tool calls expose a time-specific world state. Policy decides whether the proposal is permitted for this subject and purpose. The effect adapter attempts a mutation. Readback establishes what actually happened. A transcript usually preserves fragments of the first three. Yet a dispute may concern the last three. Replaying the words while omitting authority, effects and readback reconstructs a conversation, not the consequential event.

Three goals should therefore be named separately. Repeatability asks whether the same configured run yields the same relevant result in the same environment. Reproducibility asks whether another controlled environment can reconstruct that result within a stated tolerance. Counterfactual replay asks whether changing one declared factor changes it. The first two establish a usable baseline. The third turns that baseline into a causal instrument. A system can be repeatable yet causally opaque, or variable at the token level yet reproducible at the level of a typed decision.

There is a second problem. Two models can reach the same benchmark score while behaving differently under deployment conditions. Research on underspecification shows that many predictors can fit the development objective yet diverge elsewhere.[2] Reproducing the headline metric therefore does not reproduce the particular mechanism or failure surface that produced an incident. Reinforcement-learning studies have likewise shown how implementation details, seeds and reporting choices can change apparent conclusions.[3]

The hidden assumption behind “run it again” is that all unobserved differences are noise. Sometimes they are the cause. A useful reproducibility system must turn hidden differences into named variables.

Counterfactual replay joins two older ideas. Record-and-replay systems preserve the inputs and nondeterminism crossing a chosen execution boundary.[4][5] Causal inference asks what changes under a defined intervention while other relevant conditions are held fixed.[7] The new instrument applies both ideas to a compound AI run.

The null result matters as much as the dramatic flip. If a baseline cannot be reconstructed, the incident is not evidence that the model is inherently irreproducible. It is evidence that the configured boundary was not preserved at the chosen resolution. If a controlled model swap leaves the decision unchanged, that does not prove the model was irrelevant in every possible run. It lowers the model revision’s explanatory value for this specimen. Replay works by shrinking the claim until it matches the intervention. That discipline is the opposite of an incident narrative that starts with a favoured culprit and searches the trace for confirming fragments.

flowchart TD
  A["The result that vanished"] --> B["Turn a run into a causal specimen"]
  B --> C["The replay loom"]
  C --> D["Build the instrument, then<br/>break it"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#D95545,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Turn a run into a causal specimen

A transcript is evidence of what was said. A trace is evidence of what was called. Neither necessarily contains enough state to recreate the execution or isolate a cause.

It helps to separate four artefacts that are often collapsed under the word “log”. The distinctions are operational. Each supports a different claim.

W3C PROV gives a general vocabulary for entities, activities, derivations and responsibility.[8] OpenTelemetry supplies common semantic attributes for traces and spans, including generative-AI operations, model requests, messages and tool exchanges.[10] SLSA treats provenance as verifiable information about where, when and how an artefact was produced.[12] These standards are valuable foundations. Provenance describes lineage; replay additionally requires executable capture and a controlled substitute for live dependencies.

A manifest is more than a bag of identifiers. It is an executable claim about what lay inside and outside the causal boundary. For every dependency, it should answer four questions. Which immutable identity was invoked? Which content crossed the boundary, in what order? At what valid and observed times was that content true? Under which subject, purpose and authority could it influence the decision? A model revision without decoding and runtime is incomplete. A retrieval snapshot without result order and document digests is incomplete. A policy identifier without the evaluated subject and purpose is incomplete. The manifest becomes useful when another runner can resolve each authorised digest into the same typed input or an explicitly declared substitute.

Names are not identities. A label such as latest, production or supplier-index is a lookup instruction whose answer can change. Preserve the resolved content identity as well as the alias that selected it. Time also needs two meanings. Valid time says when a fact applied in the world; observation time says when the system learned it. Replaying yesterday’s decision with today’s corrected fact answers a useful counterfactual, but it does not reconstruct yesterday. Bitemporal capture prevents the investigation from silently replacing the historical information set with present knowledge. Order matters too: the same documents or messages can produce another context when ranking, truncation or parallel completion order changes.

Observation and effect must remain separate. A tool response can be replayed as recorded evidence. A tool mutation must not be blindly repeated. Preserve the request, authority decision, effect receipt and postcondition readback as four records. During replay, route the same typed request to a non-effectful adapter. That design preserves the decision path while preventing a diagnostic run from becoming a second business action.

The smallest useful causal model

Let one configured run produce outcome Y. A practical decomposition is:

This notation does not claim those variables are independent. It exposes the opposite problem. A normal rerun can change several terms at once. The observed difference, Y′ − Y, then cannot identify which term mattered.

A causal specimen is the manifest, captured boundary state and executable substitutes needed to reconstruct one decision within a declared tolerance. A replay cut is a checkpoint after which every inbound dependency is recorded, deterministically regenerated or sandboxed. A counterfactual replay certificate records the baseline digest, intervention, unchanged-field digest, resulting outcome and prohibited inference.

For one factor X, the replay effect is the paired difference:

For stochastic systems, one pair is fragile. Use matched seeds or captured random draws across a set of runs, then compare distributions. Pairing reduces variance because each comparison shares the same stochastic path. It does not make a hosted model deterministic, and it does not rescue an incomplete boundary.

A replay report often says “only the model changed” without proving it. Canonicalise the manifest, remove the declared intervention fields, and hash what remains. The baseline and replay should have the same unchanged digest. This catches accidental template edits, regenerated retrieval sets, tool-schema changes and cache drift. It is not cryptographic proof that a provider honoured a hidden model revision. It is proof about the fields the boundary can observe and control.

The replay loom

A usable replay system is designed before an incident. After the effect, missing inputs are usually unrecoverable.

The method begins by choosing the decision you may need to explain. “Reproduce the run” is too vague. Reproduce which property: exact tokens, tool calls, structured proposal, authority decision, external effect, or verified postcondition? Exact token equality may be impossible across hardware or provider revisions. A typed action and its readback may still be reproducible within a meaningful tolerance.

Set that tolerance before seeing the replay. For free text, exact strings may be irrelevant while extracted entities, cited evidence and prohibited claims matter. For a ranking, the top item, ordering distance or membership of an approved set may be the invariant. For a typed action, the action type, subject, amount, currency and idempotency key may require exact equality even when explanatory prose varies. For an external effect, the only meaningful test may be an independently observed postcondition. A tolerance is part of the hypothesis, not a convenience chosen after two runs disagree.

The baseline gate comes next. Re-execute the specimen without intervention. If it does not reproduce the declared property, stop. Do not compare the failed baseline with a modified run and call the difference an effect. Diagnose missing state, unresolved aliases, uncontrolled scheduling, numerical drift or an over-strict tolerance first. This ordering feels conservative because it produces more unresolved incidents. It also prevents a common error: attributing to the intervention a difference that the replay apparatus produces by itself.

A baseline may pass at one layer and fail at another. The generated wording can differ while the same citations, typed proposal and authority decision remain stable. Conversely, a byte-identical proposal can lead to a different outcome if entitlement changed before effect or if the external system rejected an idempotency key. Report this as a vector of replay results rather than one Boolean: content similarity, evidence set, proposal equality, policy equality, effect receipt and readback. The decision under investigation determines which coordinates are load-bearing. This avoids spending engineering effort on exact tokens when the real risk is a changed payment, and it avoids declaring success from a matching sentence when the authorised consequence diverged.

1. Declare the consequence and comparison metric. Decide whether success means bitwise equality, the same typed proposal, the same policy decision or the same verified outcome.
2. Choose a replay boundary. Place every mutable dependency inside the recorded specimen or behind a deterministic adapter.
3. Assign immutable identities. Record content digests for model revision, prompts, retrieval snapshots, tool schemas and responses, policy bundles, runtime images and caches.
4. Create replay cuts. Capture state before retrieval, before authority and before any irreversible effect. The last cut must permit a dry-run substitute.
5. Reproduce the baseline. If the declared property misses tolerance, improve capture before attribution.
6. Apply one declared intervention. Replace a typed field or coherent bundle and prove the remainder stayed fixed.
7. Compare paired outcomes. Record effect size, decision change, uncertainty and the exact scope of the conclusion.

stateDiagram-v2
  [*] --> Claim
  state "Figure 1 The counterfactual<br/>replay loom" as Claim
  state "The copied prompt" as Test
  state "Figure 2 Anatomy of a causal<br/>specimen" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Build the instrument, then break it

The replay harness below is deliberately small. Its purpose is to expose invariants that disappear inside a large observability platform.

It defines immutable specifications for model, context, tool observation, policy and runtime. It creates a canonical manifest digest, executes the synthetic Cedar decision and rejects a replay when declared intervention fields do not match actual changes. Replays are dry-run only. External tools must be recorded or sandboxed.

#!/usr/bin/env python3
"""Typed counterfactual replay harness for Paper 189.

The programme creates a synthetic baseline execution, seals a reproducibility
manifest, applies explicit one-factor interventions, verifies that undeclared
fields did not change, and compares paired outcomes. It performs no network or
external system calls.
"""

from __future__ import annotations

import copy
import hashlib
import json
import platform
import random
from dataclasses import asdict, dataclass, is_dataclass
from typing import Any, Final, Iterable, Mapping

DECISION_THRESHOLD: Final[float] = 0.55
MODEL_BIAS: Final[dict[str, float]] = {"r1": 0.020, "r2": -0.018}
CONTEXT_BIAS: Final[dict[str, float]] = {"c1": 0.012, "c2": 0.018}


def canonical_json(value: Any) -> str:
    """Return stable JSON suitable for content hashing."""
    if is_dataclass(value):
        value = asdict(value)
    return json.dumps(
        value,
        sort_keys=True,
        separators=(",", ":"),
        ensure_ascii=False,
        allow_nan=False,
    )


def digest(value: Any) -> str:
    """Return a SHA-256 digest of a canonical value."""
    return hashlib.sha256(canonical_json(value).encode("utf-8")).hexdigest()


@dataclass(frozen=True)
class ModelSpec:
    provider: str
    family: str
    revision: str
    decoding: str = "sample"


@dataclass(frozen=True)
class ContextSpec:
    template_revision: str
    retrieval_snapshot: str
    ordered_document_digests: tuple[str, ...]


@dataclass(frozen=True)
class ToolObservation:
    tool_name: str
    schema_revision: str
    response_digest: str
    supplier_confidence: float


@dataclass(frozen=True)
class PolicySpec:
    policy_revision: str
    approval_threshold: float
    subject: str
    purpose: str


@dataclass(frozen=True)
class RuntimeSpec:
    code_revision: str
    python_version: str
    platform: str
    seed: int


@dataclass(frozen=True)
class RunInputs:
    model: ModelSpec
    context: ContextSpec
    tool: ToolObservation
    policy: PolicySpec
    runtime: RuntimeSpec


@dataclass(frozen=True)
class RunOutcome:
    score: float
    decision: str
    proposal_digest: str
    effect_mode: str


@dataclass(frozen=True)
class RunManifest:
    run_id: str
    inputs: RunInputs
    outcome: RunOutcome
    manifest_digest: str


@dataclass(frozen=True)
class Intervention:
    """A declared replacement of one or more leaf fields."""

    name: str
    replacements: Mapping[str, Any]


@dataclass(frozen=True)
class ReplayResult:
    intervention: str
    baseline_reproduced: bool
    declared_changes: tuple[str, ...]
    actual_changes: tuple[str, ...]
    unchanged_digest_equal: bool
    baseline_score: float
    replay_score: float
    score_delta: float
    baseline_decision: str
    replay_decision: str
    decision_changed: bool
    certificate_digest: str


def cedar_execute(inputs: RunInputs) -> RunOutcome:
    """Execute the synthetic Cedar decision in dry-run mode.

    Assumptions encoded by this toy system:
    - model and context revisions contribute fixed additive biases;
    - the recorded supplier confidence enters linearly;
    - a seeded stochastic term represents chance within the configured run;
    - the authority threshold belongs to policy, not to the model;
    - the effect surface is sandboxed, so no external state changes.
    """
    try:
        model_bias = MODEL_BIAS[inputs.model.revision]
        context_bias = CONTEXT_BIAS[inputs.context.template_revision]
    except KeyError as exc:
        raise ValueError(f"Unknown configured revision: {exc.args[0]}") from exc

    confidence = inputs.tool.supplier_confidence
    if not 0.0 <= confidence <= 1.0:
        raise ValueError("supplier_confidence must be between 0 and 1")

    jitter = random.Random(inputs.runtime.seed).uniform(-0.028, 0.028)
    score = (
        0.500
        + model_bias
        + context_bias
        + 0.20 * (confidence - 0.50)
        + jitter
    )
    decision = "APPROVE" if score >= inputs.policy.approval_threshold else "REVIEW"
    proposal = {
        "score": round(score, 12),
        "decision": decision,
        "subject": inputs.policy.subject,
        "purpose": inputs.policy.purpose,
    }
    return RunOutcome(
        score=score,
        decision=decision,
        proposal_digest=digest(proposal),
        effect_mode="sandbox-only",
    )


def make_manifest(run_id: str, inputs: RunInputs) -> RunManifest:
    """Execute a run and seal its manifest with a content digest."""
    outcome = cedar_execute(inputs)
    unsigned = {"run_id": run_id, "inputs": asdict(inputs), "outcome": asdict(outcome)}
    return RunManifest(
        run_id=run_id,
        inputs=inputs,
        outcome=outcome,
        manifest_digest=digest(unsigned),
    )


def flatten(value: Any, prefix: str = "") -> dict[str, Any]:
    """Flatten dataclasses, mappings and sequences into leaf paths."""
    if is_dataclass(value):
        value = asdict(value)
    if isinstance(value, Mapping):
        result: dict[str, Any] = {}
        for key in sorted(value):
            path = f"{prefix}.{key}" if prefix else str(key)
            result.update(flatten(value[key], path))
        return result
    if isinstance(value, (list, tuple)):
        return {
            f"{prefix}.{index}": item
            for index, item in enumerate(value)
        }
    return {prefix: value}


def _replace_path(tree: dict[str, Any], path: str, replacement: Any) -> None:
    parts = path.split(".")
    node: Any = tree
    for part in parts[:-1]:
        if not isinstance(node, dict) or part not in node:
            raise KeyError(f"Intervention path does not exist: {path}")
        node = node[part]
    leaf = parts[-1]
    if not isinstance(node, dict) or leaf not in node:
        raise KeyError(f"Intervention path does not exist: {path}")
    node[leaf] = replacement


def _inputs_from_dict(data: Mapping[str, Any]) -> RunInputs:
    return RunInputs(
        model=ModelSpec(**data["model"]),
        context=ContextSpec(
            template_revision=data["context"]["template_revision"],
            retrieval_snapshot=data["context"]["retrieval_snapshot"],
            ordered_document_digests=tuple(data["context"]["ordered_document_digests"]),
        ),
        tool=ToolObservation(**data["tool"]),
        policy=PolicySpec(**data["policy"]),
        runtime=RuntimeSpec(**data["runtime"]),
    )


def intervention_record(
    baseline: RunInputs,
    intervention: Intervention,
) -> tuple[RunInputs, tuple[str, ...], tuple[str, ...], bool]:
    """Apply an intervention and prove no undeclared leaf changed."""
    candidate_tree = copy.deepcopy(asdict(baseline))
    declared = tuple(sorted(intervention.replacements))
    for path, replacement in intervention.replacements.items():
        _replace_path(candidate_tree, path.removeprefix("inputs."), replacement)

    candidate = _inputs_from_dict(candidate_tree)
    before = flatten(baseline, "inputs")
    after = flatten(candidate, "inputs")
    actual = tuple(sorted(path for path in before if before[path] != after[path]))

    undeclared_actual = set(actual) - set(declared)
    declared_missing = {
        path for path in declared if path not in before or before[path] == after.get(path)
    }
    if undeclared_actual or declared_missing:
        raise ValueError(
            "Invalid intervention. "
            f"Undeclared changes={sorted(undeclared_actual)}; "
            f"declared but unchanged or missing={sorted(declared_missing)}"
        )

    unchanged_before = {k: v for k, v in before.items() if k not in declared}
    unchanged_after = {k: v for k, v in after.items() if k not in declared}
    unchanged_equal = digest(unchanged_before) == digest(unchanged_after)
    return candidate, declared, actual, unchanged_equal


def replay(
    baseline_manifest: RunManifest,
    intervention: Intervention,
    *,
    tolerance: float = 1e-12,
) -> ReplayResult:
    """Reproduce the baseline, intervene and issue a bounded certificate."""
    baseline_again = cedar_execute(baseline_manifest.inputs)
    baseline_reproduced = (
        abs(baseline_again.score - baseline_manifest.outcome.score) <= tolerance
        and baseline_again.decision == baseline_manifest.outcome.decision
        and baseline_again.proposal_digest == baseline_manifest.outcome.proposal_digest
    )
    if not baseline_reproduced:
        raise RuntimeError(
            "Baseline replay failed. The specimen or tolerance is inadequate, "
            "so no causal comparison is permitted."
        )

    candidate, declared, actual, unchanged_equal = intervention_record(
        baseline_manifest.inputs, intervention
    )
    replay_outcome = cedar_execute(candidate)
    certificate_body = {
        "baseline_manifest_digest": baseline_manifest.manifest_digest,
        "intervention": intervention.name,
        "declared_changes": declared,
        "actual_changes": actual,
        "unchanged_digest_equal": unchanged_equal,
        "baseline_outcome": asdict(baseline_manifest.outcome),
        "replay_outcome": asdict(replay_outcome),
        "tolerance": tolerance,
    }
    return ReplayResult(
        intervention=intervention.name,
        baseline_reproduced=baseline_reproduced,
        declared_changes=declared,
        actual_changes=actual,
        unchanged_digest_equal=unchanged_equal,
        baseline_score=baseline_manifest.outcome.score,
        replay_score=replay_outcome.score,
        score_delta=replay_outcome.score - baseline_manifest.outcome.score,
        baseline_decision=baseline_manifest.outcome.decision,
        replay_decision=replay_outcome.decision,
        decision_changed=replay_outcome.decision != baseline_manifest.outcome.decision,
        certificate_digest=digest(certificate_body),
    )


def baseline_inputs(seed: int = 41) -> RunInputs:
    """Return the fully configured synthetic Cedar baseline."""
    return RunInputs(
        model=ModelSpec(
            provider="synthetic-local",
            family="cedar-decider",
            revision="r1",
        ),
        context=ContextSpec(
            template_revision="c1",
            retrieval_snapshot="supplier-register-snapshot-A",
            ordered_document_digests=("doc:7aa1", "doc:42bd", "doc:918f"),
        ),
        tool=ToolObservation(
            tool_name="supplier-risk",
            schema_revision="v3",
            response_digest="tool:62d430",
            supplier_confidence=0.70,
        ),
        policy=PolicySpec(
            policy_revision="procurement-17",
            approval_threshold=DECISION_THRESHOLD,
            subject="supplier-074",
            purpose="purchase-order-review",
        ),
        runtime=RuntimeSpec(
            code_revision="paper-189-demo-1",
            python_version=platform.python_version(),
            platform="portable-standard-library",
            seed=seed,
        ),
    )


def paired_seed_sweep(seeds: Iterable[int]) -> dict[str, Any]:
    """Estimate the distributional model effect under paired chance."""
    rows: list[dict[str, Any]] = []
    for seed in seeds:
        r1 = cedar_execute(baseline_inputs(seed))
        r2_inputs, _, _, _ = intervention_record(
            baseline_inputs(seed),
            Intervention(
                name="model-only-r2",
                replacements={"inputs.model.revision": "r2"},
            ),
        )
        r2 = cedar_execute(r2_inputs)
        rows.append(
            {
                "seed": seed,
                "r1_score": round(r1.score, 6),
                "r2_score": round(r2.score, 6),
                "paired_delta": round(r2.score - r1.score, 6),
                "r1_decision": r1.decision,
                "r2_decision": r2.decision,
            }
        )

    count = len(rows)
    if count == 0:
        raise ValueError("paired_seed_sweep requires at least one seed")
    return {
        "number_of_paired_seeds": count,
        "r1_approval_rate": sum(r["r1_decision"] == "APPROVE" for r in rows) / count,
        "r2_approval_rate": sum(r["r2_decision"] == "APPROVE" for r in rows) / count,
        "mean_paired_score_delta": sum(r["paired_delta"] for r in rows) / count,
        "first_twenty_pairs": rows[:20],
    }


def main() -> None:
    baseline = make_manifest("cedar-baseline-001", baseline_inputs())
    model_only = replay(
        baseline,
        Intervention(
            name="replace model r1 with r2",
            replacements={"inputs.model.revision": "r2"},
        ),
    )
    tool_only = replay(
        baseline,
        Intervention(
            name="replace recorded supplier confidence",
            replacements={"inputs.tool.supplier_confidence": 0.56},
        ),
    )

    report = {
        "baseline": {
            "score": round(baseline.outcome.score, 6),
            "decision": baseline.outcome.decision,
            "manifest_digest": baseline.manifest_digest,
        },
        "model_only_counterfactual": asdict(model_only),
        "tool_only_counterfactual": asdict(tool_only),
        "paired_seed_sweep": paired_seed_sweep(range(1, 101)),
        "interpretation": {
            "permitted": (
                "Within this synthetic configured system, the decision depends "
                "on both model revision and recorded supplier confidence."
            ),
            "not_established": [
                "provider-internal execution identity",
                "faithfulness of any generated reasoning trace",
                "population-level performance outside this paired experiment",
                "absence of hidden variables outside the manifest boundary",
            ],
        },
    }
    print(json.dumps(report, indent=2, sort_keys=True))


if __name__ == "__main__":
    main()

What the manifest must contain

The minimal fields depend on the decision. A text-only evaluation may not need effect receipts. A financial action does. The figure shows which claims become available as the manifest deepens.

The manifest creates an information-governance tension. A hash can prove that two authorised readers received the same bytes, but it cannot replay bytes that were never retained. Retaining every prompt, document and tool response can violate purpose limitation, confidentiality or deletion duties. The answer is not indiscriminate logging. Use a replay enclave with field-level classification, envelope encryption, scoped resolution rights and expiry rules tied to the decision’s risk. Preserve typed facts when raw text is unnecessary. Preserve a salted digest when equality is sufficient. Record that content was deliberately unavailable when neither storage nor resolution is permitted. An honest hole in the specimen is safer than a secret copy disguised as observability.

Revocation changes what replay means. An investigator may no longer be entitled to resolve a document that legitimately influenced the original run. Historical reconstruction can still prove that its digest participated without exposing its content. Where content is essential, replay can execute inside an authorised service and return a bounded comparison rather than the source. Reproducibility does not override information governance; it must operate through it.

Operationally, separate the evidence ledger from the replay payload store. The ledger carries typed identities, hashes, policy decisions, receipts and retention status. The payload store holds encrypted content only when authorised reconstruction requires it. A replay runner receives short-lived capability grants for the exact specimen and intervention, not broad access to the production corpus. It writes a new certificate rather than modifying the historical manifest. This separation makes deletion, legal hold, access review and reproducibility compatible at the system boundary. It also prevents an observability platform from becoming a privileged shadow database whose contents are easier to query than the governed source systems.

The strongest ways replay can lie

Replay feels objective because it produces a runnable result. That confidence is dangerous. The instrument can be faithful to the wrong boundary.

Hidden aliasing. A model name, retrieval index name or tool endpoint may resolve differently while the manifest appears unchanged. Capture immutable revision identifiers or acknowledge that the provider remains outside the controlled boundary.

Correlated interventions. Changing a model package may also change its tokenizer, safety template or tool-call parser. A “model-only” intervention is then a bundle. Either treat the bundle as the cause or separate its components.

Inconsistent snapshots. Distributed agents can observe state at different logical times. A collection of local snapshots may never have coexisted. Distributed snapshot algorithms exist because a consistent global state is not a trivial timestamped dump.[6]

Observer effects. Recorders alter scheduling, latency and resource pressure. Practical replay systems identify nondeterminism entering their boundary, yet even mature designs can fail to reproduce bugs that disappear under recording.[5]

Off-manifold counterfactuals. Swapping a new model into an old internal state may create a combination that could never occur in production. Intervene at supported interfaces, not arbitrary hidden tensors, unless the question is specifically about internal mechanism.

Counterfactual replay is a local diagnostic instrument, not a universal proof of cause. It can establish dependence under a declared, controlled boundary. It cannot by itself show that the dependence generalises to future traffic, that a model used a particular internal rationale, or that no unrecorded variable mattered. For population-level release decisions, combine replay with shadow traffic, controlled experiments and distributional evaluation.

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: The result that vanished
  M->>E: Figure 1 The counterfactual replay loom
  E-->>M: compare rival explanations
  E->>D: Capture the causal specimen before the incident
  D-->>Q: revise scope or act

Capture the causal specimen before the incident

Do not ask after an incident whether the prompt can be rerun. Ask before release whether the consequential execution can be reconstructed, safely forked and perturbed.

That shift changes architecture. Model aliases become insufficient identities. Retrieval needs snapshot semantics. Tool calls need request and response digests. Policy must be versioned separately from model proposals. Caches become part of causal state. Effects require typed receipts and independent readback. The replay path must terminate in a sandbox.

It also changes incident language. A different output after a rerun is not “model nondeterminism” unless model-side randomness is the only changed factor. A matching output is not proof of the same mechanism. An intervention that flips one decision does not prove a release-wide effect. Reproducibility should narrow claims, not manufacture certainty.

Release governance should make that narrowing visible. A consequential change is not ready merely because aggregate tests pass. It should state which historical specimens were replayed, which variables were intervened upon, which tolerances held, which effects were sandboxed and which dependencies remained outside control. Incident review should attach the replay certificate to the claim it supports: “this case depended on retrieval snapshot R under model M and policy P”, not “the AI failed”. Over time, the organisation accumulates a library of causal specimens that can be rerun against proposed changes before those changes encounter live authority.

This creates three distinct responsibilities. Product teams define the decision-relevant invariants and replay cuts because they know what consequence matters. Platform teams provide content-addressed capture, deterministic adapters, sandboxed effects and certificate generation. Independent reviewers challenge intervention validity and the unresolved exterior. None of the three should silently substitute for the others. A platform can prove hashes without knowing whether the tolerance is meaningful. A product owner can name the right outcome without proving the inputs stayed fixed. A reviewer can expose a rival explanation but cannot recover state that was never captured. Treating replay as shared operating evidence, rather than a specialist debugging feature, aligns those responsibilities before an incident.

NIST’s generative-AI profile calls for post-deployment monitoring, incident response, change management and provenance tracking.[11] OpenTelemetry can standardise trace vocabulary. Reproducibility checklists can improve experimental reporting. Those practices become materially stronger when a consequential run also carries a replayable causal specimen.

If the system cannot provide those things, it may still be useful. It is not yet causally auditable. After a disputed outcome, the honest conclusion may be “unresolved because the execution boundary was not preserved”. That sentence is less satisfying than a confident blame assignment. It is also more reliable.

The first implementation need not capture every possible future question. Start with one consequential decision, one replay cut and the five or six dependencies most likely to change its answer. Inject known faults, confirm that the specimen localises them, and record where it abstains. Expand the boundary only when a new incident exposes a missing causal variable. This produces an evidence-led capture design rather than a limitless logging programme. The goal is not perfect historical simulation. It is a controlled path from a disputed result to the narrowest conclusion that the preserved system can honestly support.

The practical decision is to treat replayability as a property of the configured system and its authority boundary, not as a debugging feature added to the model after failure.