Reconstruct the world the decision could see

Reproducibility has several established meanings. The ACM’s artifact terminology distinguishes results obtained with the original artefacts from results obtained independently, while machine-learning practice often uses reproducibility for obtaining similar results with the same code and data. Pineau and colleagues made that practical burden visible through code submission, checklists and programme-level evidence in the NeurIPS reproducibility initiative. Terminology varies, so a consequential decision system should state the promised operation rather than rely on the word alone: byte-identical execution, equivalent decision, robust aggregate result or independent replication.

Record-and-replay computing shows the narrowest version clearly. ReVirt captured enough nondeterministic input to replay a virtual machine at instruction-level fidelity, and the rr debugger records an execution so developers can return to the same failure repeatedly. These systems succeed because their boundary is explicit. They capture the influences the replayed machine cannot regenerate deterministically. A database row and a model identifier are not an equivalent boundary for a compound AI decision. Retrieval, external services, policy selection, human input, permissions and downstream effects all cross it.

Four cuts through one decision

The first cut is valid time: when a proposition applied in the represented world. The second is recorded or knowledge time: when the system possessed that version. Temporal-database research distinguishes these dimensions because a correction can be valid for an earlier period while becoming known only later. The credit-score correction in the opening case is exactly such an event. It may improve today’s belief about January, but it cannot be inserted into the original system’s knowledge without changing history.

The third cut is the decision instant and workflow state. A recommendation may depend on whether a document was pending, an account was already restricted, or a human had acknowledged an exception. The fourth is governed applicability: which policy, model release, prompt, retrieval index, tool schema and permission set were effective for that case. Publication time, deployment time and effective time can differ. “Latest” silently chooses among them.

W3C PROV provides a useful vocabulary for entities, activities, agents, use, generation and derivation. OpenLineage similarly attaches facets to runs, jobs and datasets. These standards help connect evidence to transformations. They do not by themselves determine valid-time semantics, policy applicability, the exact context exposed to a model, or the authority carried into an effect. Provenance answers “where did this come from?” A replay envelope must additionally answer “was this version knowable and admissible for this decision, and what could the configured system do with it?”

flowchart LR
  A["The decision that changes when<br/>nothing “important” changes"] --> B["Reconstruct the world the<br/>decision could see"]
  B --> C["Treat replay as a ladder of claims"]
  C --> D["Build the replay envelope"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#7B4CD8,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Treat replay as a ladder of claims

The word replay invites a picture of one mechanism. In practice it covers at least five claims. At the bottom, a trace rendering re-displays stored messages. It can aid investigation without proving that the trace is complete. A historical reconstruction rebuilds the admitted evidence and state at a cut-off. A decision reperformance executes the pinned system against that reconstruction and compares its decision receipt with the original.

Above reperformance is counterfactual simulation. One or more declared components change: policy P18 replaces P17, a human override is removed, a verifier is added, or a threshold moves. The result is a prediction of the configured decision mechanism under the intervention. At the top is causal outcome estimation: what customer behaviour, loss, safety event or operational consequence would have followed the alternative action. That branch was not observed. It must be identified from experiments, quasi-experiments, logged-policy propensities, structural assumptions or another defensible causal design.

This ladder prevents a common category error. Suppose the pinned historical system exactly reproduces APPROVE. That supports a claim about faithful reconstruction and deterministic or distributional equivalence. Suppose P18 on the same historical evidence produces DECLINE. That supports a mechanism claim: the policy change crosses a decision boundary. Neither result tells us whether declining would have reduced default loss, displaced borrowing elsewhere, harmed a viable business or changed customer behaviour. The post-action world responds to the action.

What must be the same?

Even within reperformance, “the same result” needs a declared equivalence rule. Execution equivalence asks for the same instruction, token, tool-call or state-transition sequence. It is appropriate when debugging a race, investigating an exploit or proving that a captured executable trace is faithful. It is often unnecessarily strict for an AI decision. A harmless punctuation change in an explanation can break byte equality while leaving the decision, evidence and authority unchanged.

Decision equivalence compares a typed receipt: action, policy rule, material reason codes, evidence references, uncertainty class, escalation state and authority disposition. The generated prose may vary, but the propositions that authorised or blocked the action must match. For the overdraft case, APPROVE with score and ratio reason codes is equivalent only if the same P17 clauses and historical evidence support it. APPROVE reached through a different exception path is a divergence even when the final label matches.

Distributional equivalence is needed when a stochastic component cannot or should not be reduced to one seed. Pinning a seed may fail across changed numerical kernels, serving stacks or provider releases. Capturing the sampled answer permits trace rendering, but feeding that answer forward does not test whether the model would generate an equivalent proposal again. A stronger test runs the pinned configuration repeatedly and compares predeclared behavioural quantities: action rate, abstention rate, reason-code distribution, calibration and boundary-case errors. The tolerances must be set before results are seen.

The comparison should follow consequence. Exact token order may matter when a parser consumes free text. It matters less when a deterministic validator converts proposals into a typed action and rejects unsupported fields. Conversely, two fluent explanations are not equivalent if one omits the decisive evidence or changes a caveat. Replay should compare the smallest representation that still preserves the decision’s safety and evidential invariants.

Human judgement creates another boundary. A system can reconstruct the screen, evidence packet, policy and authority presented to an approver. It can preserve the approver’s recorded decision and rationale. It cannot re-execute the person’s private cognition. A later reviewer may test whether the packet was sufficient, whether policy was followed and how the automated recommendation changes under an intervention. Describing that as a replay of the human mind would exceed the evidence. The manifest should mark human judgement as an observed contribution, not a deterministic component.

The unobserved branch

Logged decision systems observe outcomes under actions actually taken. A fraud service that holds a payment observes the investigation, release delay and any subsequent disposition under HOLD. It does not observe the loss that would have occurred under ALLOW for that same payment. Replaying the classifier can say whether another threshold would have selected ALLOW. It cannot manufacture the missing outcome.

Offline policy evaluation offers a disciplined bridge when actions were logged with known probabilities. For a target policy π′, a simple inverse-propensity estimate is:

Here n is the number of logged decisions, x_i is the context, a_i the action actually taken, y_i the observed outcome, and p_i the historical policy’s probability of taking that action. The indicator keeps cases where the target policy would choose the logged action. The weight corrects for how often the historical policy exposed that action. Li and colleagues established an unbiased replay method for contextual-bandit evaluation under its stated setting. Doubly robust estimation and counterfactual risk minimisation improve the toolkit, but still require logged propensities, overlap and controlled variance.

If the historical policy never allowed an action for a region of cases, its probability is zero there. No amount of rerunning identifies the target policy’s value in that region from those logs. If propensities are tiny, a few cases receive huge weights and uncertainty expands. The right output is therefore an estimate with assumptions and intervals, not a cinematic branch labelled “what would have happened”. Bottou and colleagues frame this as counterfactual reasoning in systems that interact with their environment. The central lesson transfers: intervention changes the data-generating process.

Build the replay envelope

A replayable decision needs more than an application log. It needs a manifest that binds six planes into one integrity-checked object. The first plane is the evidence cut: stable source versions, hashes or snapshots, plus valid and recorded times. The second is world and workflow state: case status, prior actions, pending dependencies and the state versions read by tools. The third is governed logic: policy versions, effective intervals, threshold tables and exception rules.

The fourth plane is configured-system lineage: model weights or service release, prompt, decoding settings, retrieval index, embedding model, tool schemas, code and feature transformations. The fifth is nondeterminism: random seeds where useful, sampled outputs, concurrency order, external responses and human contributions. The sixth is authority and effect: who requested, who proposed, who approved, which capability token permitted action, what idempotency key was used, and what postcondition was observed. The replay envelope is complete only relative to a declared claim. Exact token reproduction needs a different capture boundary from equivalent policy reperformance.

Event sourcing is helpful because state can be rebuilt by replaying an ordered event history. Fowler also notes the hard parts: external queries may need their responses retained, and code changes complicate replay. Those caveats become decisive in AI systems. An event saying “retrieval completed” is not enough if the index was rebuilt and the retrieved passages were not captured. A tool-call record is not enough if the tool’s response was mutable. A model alias is not enough if the provider changed the underlying release. A human-review event is not enough if the evidence displayed to the reviewer and the final rationale were not retained.

The manifest should therefore prefer immutable identifiers and content digests over names. It can reference an Iceberg snapshot, object-store version, signed policy artefact, model digest, prompt commit, tool schema hash and effect receipt. Where a source exposes only current state, the manifest records the captured response and its limitation. Where retention rules require deletion, it may preserve a non-sensitive integrity and deletion receipt rather than the payload. A pointer to expired history is not replayability.

A replay manifest that can fail closed

The manifest is not an archive inventory. It is an executable contract. Each required field has a verifier, and missing material changes the outcome from “replayed” to “insufficient history”. A simplified manifest for the opening case includes the decision and knowledge cut-offs, source-version digests, P17’s signed artefact, the decision-service commit, prompt and rules versions, the seed, the recommender’s scoped authority, effect mode, expected action and original decision receipt.

The accompanying Python laboratory uses synthetic facts with valid and recorded times. It reconstructs the evidence known on 15 January, verifies its digest, loads P17 and reproduces APPROVE with effects disabled. It then runs two deliberately different branches. A naive present-day rerun consumes the corrected score and P18, producing DECLINE but labelling the result temporal leakage. A valid policy counterfactual keeps the historical evidence and replaces only P17 with P18, also producing DECLINE but labelling the result a simulated decision, not an observed outcome.

#!/usr/bin/env python3
"""Replay-manifest laboratory for a synthetic consequential decision.

Python 3.10+, standard library only.

The laboratory demonstrates four different operations:
1. historical reconstruction under the evidence and policy known at decision time;
2. deterministic reperformance of the original decision;
3. a naive present-day rerun that leaks later evidence and policy;
4. a policy counterfactual that changes one cause while holding the historical
   evidence cut fixed.

It also contains negative cases. A replay is blocked when a required policy
version is absent or when the captured evidence has been altered.
"""

from __future__ import annotations

from dataclasses import asdict, dataclass
from datetime import datetime, timezone
from hashlib import sha256
import json
from typing import Any, Iterable, Mapping, Sequence

UTC = timezone.utc


class ReplayError(RuntimeError):
    """Raised when the manifest is insufficient or fails integrity checks."""


def instant(value: str) -> datetime:
    """Parse an ISO 8601 instant that uses Z or an explicit UTC offset."""
    return datetime.fromisoformat(value.replace("Z", "+00:00")).astimezone(UTC)


def canonical_json(value: Any) -> str:
    """Serialise deterministically so the same evidence has the same digest."""
    return json.dumps(value, sort_keys=True, separators=(",", ":"), default=str)


def digest(value: Any) -> str:
    return sha256(canonical_json(value).encode("utf-8")).hexdigest()


@dataclass(frozen=True)
class FactVersion:
    subject: str
    predicate: str
    value: float
    valid_from: str
    recorded_at: str
    source_version: str

    def valid_at(self, at: datetime) -> bool:
        return instant(self.valid_from) <= at

    def known_by(self, cutoff: datetime) -> bool:
        return instant(self.recorded_at) <= cutoff


@dataclass(frozen=True)
class Policy:
    policy_id: str
    effective_from: str
    minimum_score: int
    maximum_dti: float

    def effective_at(self, at: datetime) -> bool:
        return instant(self.effective_from) <= at


@dataclass(frozen=True)
class DecisionReceipt:
    decision_id: str
    decision_at: str
    action: str
    policy_id: str
    evidence_digest: str
    system_digest: str
    authority_scope: str
    effect_mode: str


@dataclass(frozen=True)
class ReplayManifest:
    decision_id: str
    decision_at: str
    knowledge_cutoff: str
    valid_time_target: str
    policy_id: str
    system_version: str
    prompt_version: str
    rules_version: str
    random_seed: int
    authority_scope: str
    effect_mode: str
    evidence_digest: str
    expected_action: str

    def system_digest(self) -> str:
        return digest(
            {
                "system_version": self.system_version,
                "prompt_version": self.prompt_version,
                "rules_version": self.rules_version,
                "random_seed": self.random_seed,
            }
        )


FACT_HISTORY: tuple[FactVersion, ...] = (
    FactVersion(
        subject="case-042",
        predicate="credit_score",
        value=705,
        valid_from="2025-01-01T00:00:00Z",
        recorded_at="2025-01-02T09:00:00Z",
        source_version="bureau-response-v1",
    ),
    # A later correction is retroactive in valid time but was not knowable at
    # the original decision cut-off.
    FactVersion(
        subject="case-042",
        predicate="credit_score",
        value=680,
        valid_from="2025-01-01T00:00:00Z",
        recorded_at="2025-01-20T08:30:00Z",
        source_version="bureau-correction-v2",
    ),
    FactVersion(
        subject="case-042",
        predicate="monthly_income",
        value=6000,
        valid_from="2024-12-01T00:00:00Z",
        recorded_at="2025-01-10T11:00:00Z",
        source_version="verified-income-v3",
    ),
    FactVersion(
        subject="case-042",
        predicate="monthly_obligations",
        value=2160,
        valid_from="2025-01-01T00:00:00Z",
        recorded_at="2025-01-10T11:02:00Z",
        source_version="obligations-v5",
    ),
)

POLICIES: tuple[Policy, ...] = (
    Policy(
        policy_id="P17",
        effective_from="2024-10-01T00:00:00Z",
        minimum_score=700,
        maximum_dti=0.40,
    ),
    Policy(
        policy_id="P18",
        effective_from="2025-02-01T00:00:00Z",
        minimum_score=720,
        maximum_dti=0.35,
    ),
)

ORIGINAL_MANIFEST = ReplayManifest(
    decision_id="decision-042",
    decision_at="2025-01-15T10:00:00Z",
    knowledge_cutoff="2025-01-15T10:00:00Z",
    valid_time_target="2025-01-15T10:00:00Z",
    policy_id="P17",
    system_version="decision-service@8f2c1d7",
    prompt_version="explanation-prompt@3",
    rules_version="eligibility-rules@17",
    random_seed=4317,
    authority_scope="recommend_only",
    effect_mode="disabled_in_replay",
    evidence_digest="",  # filled after the historical evidence cut is built
    expected_action="APPROVE",
)


def select_evidence(
    history: Iterable[FactVersion], *, valid_at: datetime, known_by: datetime
) -> dict[str, FactVersion]:
    """Return the latest version of each predicate that was knowable by cutoff."""
    selected: dict[str, FactVersion] = {}
    for fact in history:
        if not fact.valid_at(valid_at) or not fact.known_by(known_by):
            continue
        previous = selected.get(fact.predicate)
        if previous is None or instant(fact.recorded_at) > instant(previous.recorded_at):
            selected[fact.predicate] = fact
    required = {"credit_score", "monthly_income", "monthly_obligations"}
    missing = sorted(required - selected.keys())
    if missing:
        raise ReplayError(f"missing historical evidence: {', '.join(missing)}")
    return selected


def evidence_payload(evidence: Mapping[str, FactVersion]) -> list[dict[str, Any]]:
    return [asdict(evidence[key]) for key in sorted(evidence)]


def find_policy(policies: Sequence[Policy], policy_id: str) -> Policy:
    for policy in policies:
        if policy.policy_id == policy_id:
            return policy
    raise ReplayError(f"required policy version {policy_id!r} is unavailable")


def latest_effective_policy(policies: Sequence[Policy], at: datetime) -> Policy:
    candidates = [p for p in policies if p.effective_at(at)]
    if not candidates:
        raise ReplayError("no policy is effective at the requested time")
    return max(candidates, key=lambda p: instant(p.effective_from))


def evaluate(evidence: Mapping[str, FactVersion], policy: Policy) -> tuple[str, dict[str, float]]:
    score = int(evidence["credit_score"].value)
    income = evidence["monthly_income"].value
    obligations = evidence["monthly_obligations"].value
    if income <= 0:
        raise ReplayError("monthly income must be positive")
    dti = obligations / income
    action = (
        "APPROVE"
        if score >= policy.minimum_score and dti <= policy.maximum_dti
        else "DECLINE"
    )
    return action, {"credit_score": score, "dti": round(dti, 4)}


def materialise_manifest(base: ReplayManifest, history: Iterable[FactVersion]) -> ReplayManifest:
    evidence = select_evidence(
        history,
        valid_at=instant(base.valid_time_target),
        known_by=instant(base.knowledge_cutoff),
    )
    payload_digest = digest(evidence_payload(evidence))
    return ReplayManifest(**{**asdict(base), "evidence_digest": payload_digest})


def historical_replay(
    manifest: ReplayManifest,
    *,
    history: Iterable[FactVersion] = FACT_HISTORY,
    policies: Sequence[Policy] = POLICIES,
) -> DecisionReceipt:
    """Reconstruct and reperform the historical decision with effects disabled."""
    if manifest.effect_mode != "disabled_in_replay":
        raise ReplayError("replay must not carry live effect authority")

    evidence = select_evidence(
        history,
        valid_at=instant(manifest.valid_time_target),
        known_by=instant(manifest.knowledge_cutoff),
    )
    actual_digest = digest(evidence_payload(evidence))
    if actual_digest != manifest.evidence_digest:
        raise ReplayError("evidence digest mismatch: snapshot is missing or altered")

    policy = find_policy(policies, manifest.policy_id)
    action, _ = evaluate(evidence, policy)
    if action != manifest.expected_action:
        raise ReplayError(
            f"reperformance diverged: expected {manifest.expected_action}, got {action}"
        )

    return DecisionReceipt(
        decision_id=manifest.decision_id,
        decision_at=manifest.decision_at,
        action=action,
        policy_id=policy.policy_id,
        evidence_digest=actual_digest,
        system_digest=manifest.system_digest(),
        authority_scope=manifest.authority_scope,
        effect_mode=manifest.effect_mode,
    )


def naive_present_day_rerun(
    *,
    at: str = "2025-03-01T00:00:00Z",
    history: Iterable[FactVersion] = FACT_HISTORY,
    policies: Sequence[Policy] = POLICIES,
) -> tuple[str, str, dict[str, float]]:
    """Deliberately wrong audit method: use all evidence and current policy."""
    now = instant(at)
    evidence = select_evidence(history, valid_at=now, known_by=now)
    policy = latest_effective_policy(policies, now)
    action, metrics = evaluate(evidence, policy)
    return action, policy.policy_id, metrics


def policy_counterfactual(
    manifest: ReplayManifest,
    alternative_policy_id: str,
    *,
    history: Iterable[FactVersion] = FACT_HISTORY,
    policies: Sequence[Policy] = POLICIES,
) -> tuple[str, str, dict[str, float]]:
    """Hold historical evidence fixed and vary only the policy version."""
    evidence = select_evidence(
        history,
        valid_at=instant(manifest.valid_time_target),
        known_by=instant(manifest.knowledge_cutoff),
    )
    if digest(evidence_payload(evidence)) != manifest.evidence_digest:
        raise ReplayError("cannot simulate from an unverified historical evidence cut")
    alternative = find_policy(policies, alternative_policy_id)
    action, metrics = evaluate(evidence, alternative)
    return action, alternative.policy_id, metrics


def demonstrate() -> None:
    manifest = materialise_manifest(ORIGINAL_MANIFEST, FACT_HISTORY)
    historical = historical_replay(manifest)
    present_action, present_policy, present_metrics = naive_present_day_rerun()
    cf_action, cf_policy, cf_metrics = policy_counterfactual(manifest, "P18")

    print("POSITIVE: historical replay")
    print(f"  action={historical.action} policy={historical.policy_id}")
    print(f"  evidence_digest={historical.evidence_digest[:12]}... verified")
    print(f"  effects={historical.effect_mode}")

    print("\nCONTROL: naive present-day rerun")
    print(
        f"  action={present_action} policy={present_policy} metrics={present_metrics}"
    )
    print("  label=TEMPORAL_LEAKAGE_NOT_REPRODUCTION")

    print("\nCOUNTERFACTUAL: alternative policy on historical evidence")
    print(f"  action={cf_action} policy={cf_policy} metrics={cf_metrics}")
    print("  label=SIMULATED_DECISION_NOT_OBSERVED_OUTCOME")

    print("\nNEGATIVE: missing policy version")
    try:
        historical_replay(manifest, policies=(POLICIES[1],))
    except ReplayError as exc:
        print(f"  blocked={exc}")
    else:  # pragma: no cover - a failed guard should fail the demonstration
        raise AssertionError("replay should have been blocked")

    print("\nNEGATIVE: altered historical evidence")
    tampered = tuple(
        FactVersion(**{**asdict(f), "value": 706})
        if f.source_version == "bureau-response-v1"
        else f
        for f in FACT_HISTORY
    )
    try:
        historical_replay(manifest, history=tampered)
    except ReplayError as exc:
        print(f"  blocked={exc}")
    else:  # pragma: no cover
        raise AssertionError("replay should have been blocked")

    assert historical.action == "APPROVE"
    assert (present_action, present_policy) == ("DECLINE", "P18")
    assert (cf_action, cf_policy) == ("DECLINE", "P18")


if __name__ == "__main__":
    demonstrate()

Negative control: when the full envelope earns nothing

Consider a sealed, deterministic tax calculation. The input is an immutable public fixture, the function is pure, the runtime has fixed arithmetic semantics, there are no external calls, no human judgement, no permissions and no effects. The output is entirely determined by the input and code digest. Capturing a retrieval index, authority chain and effect receipt would add ceremony without explanatory value. Input, executable and expected output are sufficient for the stated claim.

This is a serious negative control because it prevents replay engineering from becoming a universal data-retention argument. Capture only influences that can change the claim being tested. As interaction, mutable context and consequence enter, the boundary must expand. The replay envelope is not justified by “AI” as a label. It is justified by openness of the decision system to changing state, policy, stochastic components, people and the world.

The failure boundary

A replay design cannot reconstruct information that was never captured and is no longer recoverable. It cannot infer the exact response of a current-only API from a request log. It cannot know which passages a mutable index returned from an index name. It cannot recover a provider’s aliased model release after the alias moved, recreate an undocumented human conversation, or establish that an effect committed when the downstream result remained unknown. It also cannot preserve deleted personal data contrary to lawful retention limits.

In these cases, the honest result is not a best-effort animation. It is a bounded failure: historical evidence unavailable, system lineage unresolved, authority incomplete, outcome unknown, or causal support absent. A replay platform’s first assurance property is its willingness to refuse a stronger claim than its history supports.

stateDiagram-v2
  [*] --> Claim
  state "Four cuts through one decision" as Claim
  state "Thought experiment, the sealed<br/>envelope" as Test
  state "What must be the same?" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Test a production-shaped decision

The minimal laboratory proves the distinction with three facts and two policies. A production-shaped case adds orchestration. The overdraft renewal gathers bureau evidence, verified income and obligations; computes the ratio deterministically; retrieves the policy effective for the customer and product; asks a model to draft an evidence-linked recommendation; runs deterministic eligibility checks; presents the packet to a relationship manager; records approval; and sends an authorised command to the lending system.

The decision receipt binds the case-state version, evidence digest, P17 release, configured-system digest, proposer identity, human approver, authority scope and effect idempotency key. The effect receipt records that the limit change committed once and that a readback observed the intended state. A later bureau correction creates a new fact version and opens an impact review. It does not overwrite the evidence packet or the original receipt.

The assurance team now asks four separate questions. First, can the historical packet and P17 reproduce the service’s recommendation? Second, did the human see that packet and approve within authority? Third, did the command produce the intended limit exactly once? Fourth, what changes under P18 or corrected evidence? Each question has a different receipt. Combining them into one “replay passed” flag would hide where the evidence is strong and where it stops.

Historical reperformance returns APPROVE and the same reason codes. The authority check confirms that the service could recommend but not approve, and that the named manager held approval rights. Effect verification confirms one committed limit change. A P18 branch on the historical packet returns DECLINE, isolating the policy effect. A corrected-evidence branch under P17 also returns DECLINE, isolating the evidence effect. A branch that uses P18, the corrected score and a newer model is retained only as a current operational comparison because its difference is confounded.

Suppose the team then claims that deploying P18 earlier would have prevented a loss. The case replay cannot support that statement. The borrower’s eventual performance occurred after APPROVE and may have been affected by the available credit. Under DECLINE, the business might have reduced spending, borrowed elsewhere or failed for another reason. The team needs a portfolio-level causal design, with treatment assignment or logged propensities, adequate overlap, outcome definitions, interference analysis and uncertainty. The individual replay remains useful as a mechanism test and candidate-generator for that study.

A release gate for replayability

The practical decision is made before an incident. Select representative cases across consequence, topology and exception paths. For each, declare the strongest replay claim the system needs to support. Run the gate with effects disabled and with at least one deliberately missing or altered artefact. A release should not pass because the happy path can be animated. It should pass because the system reconstructs valid cases, isolates interventions and refuses unsupported ones.

Two measures are especially revealing. Reconstruction yield is the proportion of sampled decisions for which the declared historical packet, configured system and authority chain can be resolved. claim calibration is the proportion of outputs whose label does not exceed the evidence actually present. Neither should be averaged into one score. A high yield with overclaimed counterfactual outcomes is unsafe evidence practice; perfect claim calibration with near-zero yield exposes an unusable history design.

Hidden technical debt research warns that machine-learning behaviour arises from data dependencies, configuration, feedback loops and external change, not from model code alone. Replayability turns that systems insight into a release property. The system does not need to freeze the entire enterprise forever. It needs to preserve the smallest sufficient decision world, under lawful retention and explicit claim boundaries, so that important decisions can be examined without allowing the present to rewrite the past.