A checkpoint saves a perspective

A process checkpoint can preserve registers, local variables, call stacks and a position in a workflow history. Database recovery can use write-ahead logs to repeat committed history and undo incomplete transactions. Durable workflow engines can rebuild orchestration state by replaying an event history, provided orchestration code remains deterministic.[3][7][8] These are powerful guarantees. None automatically proves what an external bank, ticketing system, messaging platform or human operator did during an uncertain interval.

Distributed systems research has long distinguished a collection of local states from a consistent global state. Chandy and Lamport showed how a global snapshot can be recorded without stopping the underlying computation, while preserving causal plausibility.[1] Rollback-recovery work then examined coordinated checkpoints, message logging and the hazards of restoring components to local states that never coexisted.[2] The familiar failure is an orphan message: a receiver remembers a message whose sender, after rollback, no longer remembers sending it.

The agentic version is an orphan effect. The external system remembers a transfer, booking, disclosure or cancellation that the recovered workflow no longer remembers causing. This is worse than a missing log line. It can change money, rights, inventory, obligations or a person’s options. A local checkpoint is therefore evidence about an executor, not a complete account of the world it touched.

The hidden assumption inside “resume”

When engineers say “resume from the last successful step”, they usually assume that step boundaries align with causal boundaries. The assumption holds inside a single transactional store when the step and its effects commit atomically. It weakens when the step spans a database and a message broker. It fails when the step reaches a system that neither shares the transaction nor exposes reliable readback.

The transactional outbox pattern addresses one slice of this problem. A service writes its domain change and an outgoing event to the same local transaction, then change-data capture publishes the event.[15] That prevents the service from committing its database update while forgetting to publish the corresponding event. It does not make a downstream payment, email or physical operation atomic with the originating database. The outbox closes one boundary, not every boundary.

HTTP idempotency closes another slice. RFC 9110 defines an idempotent method by the intended effect of repeated identical requests, and explicitly links idempotency to safe automatic retry after a communication failure.[4] The emerging IETF Idempotency-Key field aims to extend that property to otherwise non-idempotent methods such as POST, but it remains an Internet-Draft, and a client cannot assume a server honours the key without an explicit contract.[5] Even a respected key needs a payload fingerprint, a scope and a retention period. Reusing the same key with different content creates a new ambiguity.

flowchart LR
  A["The payment that already happened"] --> B["The checkpoint that lies"]
  B --> C["Five streams, one cut"]
  C --> D["A protocol that earns resumption"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#087A98,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

From distributed snapshots to governed effects

A classical consistent cut is a set of event prefixes closed under causal precedence. Informally, if the cut contains an event, it must also contain the events that caused it. It may contain a sent message that has not yet been received, because that message can be in transit. It may not contain a receive without the corresponding send.

For consequential execution, define five ordered event streams. I records intent versions. A records authority decisions and their validity. S records observed world-state versions and preconditions. E records action attempts and external effects. R records receipts, independent readbacks and verification. A candidate cut is a tuple of prefixes:

The formal line is useful only after we specify the predecessors. An external effect must point to the exact intent version, policy decision, action identifier, payload hash and precondition version that authorised and shaped it. A receipt must point to the effect or external reference it evidences. A recovery decision must point to the cut it used and the fresh authority under which it acts.

The decisive mechanism is causal closure plus typed correspondence. Causal closure prevents an effect from appearing without its causes. Typed correspondence prevents a valid receipt for one action from closing the cut for a different action. A trace identifier is useful for correlation, but correlation is not correspondence. W3C Trace Context and OpenTelemetry propagate trace and span identifiers across services.[10][11] They do not, by themselves, prove that a policy allowed an action, that a payload matched the authorised intent, or that an observed postcondition is authoritative. That limitation is an inference from the scope of those specifications, not a defect in tracing.

What each stream must remember

The intent stream should not preserve a loose instruction such as “pay the supplier”. It should preserve the version of the commitment that the action is meant to realise: subject, purpose, target, amount or tolerance, relevant constraints and the identity of the approving human or upstream process. When the plan changes, the system creates a new intent version. It does not edit the old one in place. Otherwise a receipt from yesterday can appear to satisfy a materially different request today.

The authority stream records a decision, rather than only the credential that transported it. A bearer token can be valid while the requested purpose, subject or amount is outside policy. Conversely, a token may expire after a valid action was committed. The recovery record therefore needs the policy decision identifier, actor, subject, purpose, permitted action type, limits, policy version, check time and validity interval. The credential may be referenced or hashed, but the decision that bound it to the action must remain independently intelligible.

The world-state stream records the facts on which the action depended. “Invoice approved” is insufficient unless the system knows which invoice version, which approving source and how fresh the observation had to be. A precondition can be as precise as “invoice N-144 remains approved, unpaid and at version 6; account A7 remains at version 41; payee fingerprint equals P9”. When the external resource supports conditional writes, these versions become active guards. When it does not, they still reveal whether recovery is acting on a world that has moved.

The effect stream separates attempt, acceptance and commitment. An HTTP 202 may mean only that work entered a queue. A transport acknowledgement means bytes arrived, not that the business effect happened. A provider may expose a durable external reference before final settlement. The action contract must say which transition counts as the governed effect. It must also define the idempotency scope. Is the action ID unique per account, per provider, per day or globally? How long does the provider retain it? Does the provider reject a repeated key with a different payload, or silently return the first result? A recovery system cannot be more precise than the external effect contract it was given.

The receipt stream is intentionally plural. A caller-side response, provider receipt, broker-delivered completion event, ledger readback and downstream postcondition are different evidence types. They may agree, lag or conflict. A payment API response can establish provider acceptance while an account-statement readback establishes settlement. A warehouse command receipt can establish controller acceptance while a sensor reading establishes physical movement. Recovery policy chooses the minimum evidence needed for the consequence and records which evidence was unavailable.

These streams may live in one event store, but they should not collapse into one generic “memory”. Their schemas answer different questions: what was wanted, what was permitted, what was believed about the world, what was attempted, and what became true. The recovery algorithm earns its decision by joining those questions through stable identifiers and versions. A single append-only log can preserve order; only typed relations preserve meaning.

Unknown is a first-class state

Many APIs expose success and failure, then treat timeout as a flavour of failure. For recovery, timeout is a third logical state: unknown. It means the caller cannot distinguish “not applied” from “applied but unacknowledged”. RFC 9110 permits automatic retry of an idempotent request precisely because the intended effect remains the same even if the first request succeeded.[4] For a non-idempotent effect, the correct transition from unknown is not immediately to retry. It is to query, correlate, read back, or quarantine.

Suppose each payment attempt commits remotely, but its acknowledgement is independently lost with probability p. A naive client retries until it receives an acknowledgement. Under those deliberately simple assumptions, the number of commits is geometric. The expected number of extra commits per logical payment is:

At p = 0.02, ten thousand logical payments produce about 204 extra commits in expectation. The equation is not a production estimate. It assumes every retry commits, every loss is independent and the provider performs no deduplication. Remove any assumption and the number changes. The mechanism does not: a small unknown-outcome rate becomes a material duplicate-effect rate when blind retry turns uncertainty into another side effect.

The six recovery moves

A recovery mechanism should begin by refusing to do the thing operators often expect it to do: continue. It first freezes new consequential effects for the affected action lineage. Reads, evidence collection and bounded health checks may continue under a separate recovery capability. This prevents the investigation from racing the workflow it is investigating.

Next, it reconstructs the candidate cut. The process joins the durable workflow history, policy decision record, precondition snapshot, action-attempt record, external provider status, effect receipts and independent readback. Missing data remains missing. The reconciler does not infer “not done” from “not logged”.

Third, it separates old authority from new authority. OAuth token introspection offers a standard mechanism for checking whether a token is currently active and for retrieving its authorisation context.[9] That can support a recovery-time check, but a currently inactive token does not retroactively invalidate an effect that was properly authorised while the token was active. Conversely, an old allow decision does not authorise a fresh retry after expiry or revocation. The recovery service therefore asks for a capability scoped to one of four verbs: observe, continue, compensate or quarantine.

Fourth, it resolves the external effect. The preferred path is an authoritative query keyed by the stable action ID. The second-best path is a provider contract that guarantees idempotent replay for the same action ID and payload. A receipt delivered through a broker can help, but it must be matched to the action. A trace can locate candidates. Neither substitute for authoritative effect status.

Fifth, it chooses a typed decision. Execute means no prior effect exists, current authority permits a new effect, and preconditions remain true. Adopt means the prior effect exists and matches its authorised lineage. Verify means more evidence can resolve the outcome without causing another effect. Reauthorise means no new effect may occur until a fresh decision is obtained. Compensate means an existing effect must be counteracted under a defined compensator. Quarantine means the evidence cannot support a safe automated choice.

Finally, it emits a recovery receipt. That receipt names the cut, the evidence used, the decision, the code and policy versions, and the unresolved limitations. Recovery is complete only when the reason for continuation is itself durable and reviewable.

The Recovery Cut Certificate

The proposed implementation artefact is a Recovery Cut Certificate, or RCC. It is not another unstructured log bundle. It is a typed, immutable record used by the recovery decision. The smallest useful RCC binds one intent version to one action identity, one authority-at-effect decision, one precondition version, one external outcome and one recovery decision.

The certificate should not place every field in one security domain. Policy evidence may be signed by an authority service. External receipts may come from providers. Independent readback may come through a separate connector. The RCC is a composition envelope whose signatures and digests let a reviewer test correspondence. It can be stored beside, not inside, the model’s conversational history. World state, workflow state and evidence are different stores with different authority.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://rajeshrmahapatra.com/schema/recovery-cut-certificate.json",
  "title": "RecoveryCutCertificate",
  "type": "object",
  "required": [
    "execution_id", "step_id", "intent", "authority_at_effect",
    "precondition", "action", "effect", "recovery_decision"
  ],
  "properties": {
    "execution_id": {"type": "string", "minLength": 1},
    "step_id": {"type": "string", "minLength": 1},
    "intent": {
      "type": "object",
      "required": ["version", "subject", "purpose"],
      "properties": {
        "version": {"type": "integer", "minimum": 1},
        "subject": {"type": "string"},
        "purpose": {"type": "string"},
        "digest": {"type": "string", "pattern": "^[a-f0-9]{64}$"}
      }
    },
    "authority_at_effect": {
      "type": "object",
      "required": ["decision_id", "outcome", "checked_at", "valid_until"],
      "properties": {
        "decision_id": {"type": "string"},
        "outcome": {"const": "allow"},
        "checked_at": {"type": "string", "format": "date-time"},
        "valid_until": {"type": "string", "format": "date-time"},
        "policy_version": {"type": "string"}
      }
    },
    "precondition": {
      "type": "object",
      "required": ["resource", "version", "observed_at"],
      "properties": {
        "resource": {"type": "string"},
        "version": {"type": "string"},
        "observed_at": {"type": "string", "format": "date-time"}
      }
    },
    "action": {
      "type": "object",
      "required": ["action_id", "type", "payload_hash", "idempotency_scope"],
      "properties": {
        "action_id": {"type": "string"},
        "type": {"type": "string"},
        "payload_hash": {"type": "string", "pattern": "^[a-f0-9]{64}$"},
        "idempotency_scope": {"enum": ["provider", "application", "none"]}
      }
    },
    "effect": {
      "type": "object",
      "required": ["status"],
      "properties": {
        "status": {"enum": ["not_started", "unknown", "committed", "rejected"]},
        "external_reference": {"type": "string"},
        "effect_receipt_digest": {"type": "string"},
        "readback_digest": {"type": "string"},
        "verified_at": {"type": "string", "format": "date-time"}
      }
    },
    "recovery_authority": {
      "type": "object",
      "properties": {
        "decision_id": {"type": "string"},
        "permits": {"enum": ["observe", "continue", "compensate", "quarantine"]}
      }
    },
    "recovery_decision": {
      "enum": ["execute", "adopt", "verify", "reauthorise", "compensate", "quarantine"]
    }
  },
  "allOf": [
    {
      "if": {"properties": {"effect": {"properties": {"status": {"const": "committed"}}}}},
      "then": {"properties": {"effect": {"required": ["external_reference"]}}}
    }
  ]
}

The interruption harness

The Python harness below injects crashes after authority, after send, after remote commit and after receipt. It then reconstructs a cut and emits a typed recovery decision. Run it with Python 3.10 or later. A positive result means the tested scenario preserved the encoded invariants: no duplicate remote commit in the simulated idempotent case, no fresh effect under expired recovery authority, and no adoption of a mismatched payload. A negative result identifies a concrete history that violates one of those properties.

The harness cannot establish that a real provider is honest, that a production policy is correct, or that all concurrency schedules were explored. It is a small executable specification. Its value is that it makes crash points, external truth and recovery choices explicit enough to test.

from __future__ import annotations

from dataclasses import dataclass, field
from datetime import datetime, timedelta, timezone
from enum import Enum
from hashlib import sha256
from typing import Dict, List, Optional, Tuple


class CrashPoint(str, Enum):
    NONE = "none"
    AFTER_AUTHORITY = "after_authority"
    AFTER_SEND = "after_send"
    AFTER_REMOTE_COMMIT = "after_remote_commit"
    AFTER_RECEIPT = "after_receipt"


class EffectStatus(str, Enum):
    NOT_STARTED = "not_started"
    UNKNOWN = "unknown"
    COMMITTED = "committed"
    REJECTED = "rejected"


class RecoveryDecision(str, Enum):
    EXECUTE = "execute"
    ADOPT = "adopt_existing_effect"
    REAUTHORISE = "reauthorise"
    VERIFY = "verify"
    COMPENSATE = "compensate"
    QUARANTINE = "quarantine"


@dataclass(frozen=True)
class Intent:
    execution_id: str
    version: int
    subject: str
    purpose: str
    amount_pence: int
    payee: str


@dataclass(frozen=True)
class Authority:
    decision_id: str
    subject: str
    purpose: str
    maximum_pence: int
    valid_until: datetime

    def permits(self, intent: Intent, at: datetime) -> bool:
        return (
            at <= self.valid_until
            and intent.subject == self.subject
            and intent.purpose == self.purpose
            and intent.amount_pence <= self.maximum_pence
        )


@dataclass(frozen=True)
class TypedAction:
    action_id: str
    intent_version: int
    payee: str
    amount_pence: int
    expected_account_version: int

    @property
    def payload_hash(self) -> str:
        canonical = (
            f"{self.intent_version}|{self.payee}|{self.amount_pence}|"
            f"{self.expected_account_version}"
        )
        return sha256(canonical.encode()).hexdigest()


@dataclass
class JournalEvent:
    kind: str
    data: dict


@dataclass
class Journal:
    events: List[JournalEvent] = field(default_factory=list)

    def append(self, kind: str, **data: object) -> None:
        self.events.append(JournalEvent(kind, dict(data)))

    def first(self, kind: str) -> Optional[JournalEvent]:
        return next((e for e in self.events if e.kind == kind), None)


class InjectedCrash(RuntimeError):
    pass


class RemoteLedger:
    """Synthetic external system with optional idempotency and readback."""

    def __init__(self, supports_idempotency: bool = True, supports_readback: bool = True):
        self.supports_idempotency = supports_idempotency
        self.supports_readback = supports_readback
        self.commits: List[dict] = []
        self.by_action_id: Dict[str, dict] = {}

    def commit(self, action: TypedAction) -> dict:
        if self.supports_idempotency and action.action_id in self.by_action_id:
            return self.by_action_id[action.action_id]

        receipt = {
            "external_reference": f"EXT-{len(self.commits) + 1:04d}",
            "action_id": action.action_id,
            "payload_hash": action.payload_hash,
            "amount_pence": action.amount_pence,
            "payee": action.payee,
        }
        self.commits.append(receipt)
        self.by_action_id[action.action_id] = receipt
        return receipt

    def query(self, action_id: str) -> Optional[dict]:
        if not self.supports_readback:
            raise RuntimeError("readback unsupported")
        return self.by_action_id.get(action_id)


@dataclass
class RecoveryCut:
    intent_recorded: bool
    authority_recorded: bool
    precondition_recorded: bool
    attempt_recorded: bool
    effect_status: EffectStatus
    receipt_recorded: bool
    payload_matches: bool

    @property
    def causally_closed(self) -> bool:
        if self.effect_status == EffectStatus.COMMITTED:
            return (
                self.intent_recorded
                and self.authority_recorded
                and self.precondition_recorded
                and self.attempt_recorded
                and self.payload_matches
            )
        return True


def execute(
    intent: Intent,
    authority: Authority,
    action: TypedAction,
    journal: Journal,
    remote: RemoteLedger,
    now: datetime,
    crash_at: CrashPoint = CrashPoint.NONE,
) -> None:
    journal.append("intent", version=intent.version, execution_id=intent.execution_id)

    if not authority.permits(intent, now):
        journal.append("authority_denied", decision_id=authority.decision_id)
        return
    journal.append(
        "authority_allowed",
        decision_id=authority.decision_id,
        checked_at=now.isoformat(),
        valid_until=authority.valid_until.isoformat(),
    )
    if crash_at == CrashPoint.AFTER_AUTHORITY:
        raise InjectedCrash(crash_at.value)

    journal.append(
        "precondition",
        expected_account_version=action.expected_account_version,
        payload_hash=action.payload_hash,
    )
    journal.append("attempt", action_id=action.action_id, payload_hash=action.payload_hash)
    if crash_at == CrashPoint.AFTER_SEND:
        raise InjectedCrash(crash_at.value)

    receipt = remote.commit(action)
    if crash_at == CrashPoint.AFTER_REMOTE_COMMIT:
        raise InjectedCrash(crash_at.value)

    journal.append("effect_receipt", **receipt)
    if crash_at == CrashPoint.AFTER_RECEIPT:
        raise InjectedCrash(crash_at.value)

    journal.append("verified", external_reference=receipt["external_reference"])


def reconstruct_cut(journal: Journal, remote: RemoteLedger, action: TypedAction) -> RecoveryCut:
    intent_recorded = journal.first("intent") is not None
    authority_recorded = journal.first("authority_allowed") is not None
    precondition_recorded = journal.first("precondition") is not None
    attempt_recorded = journal.first("attempt") is not None
    receipt = journal.first("effect_receipt")

    remote_receipt: Optional[dict] = None
    if remote.supports_readback:
        remote_receipt = remote.query(action.action_id)

    if receipt or remote_receipt:
        observed = receipt.data if receipt else remote_receipt
        effect_status = EffectStatus.COMMITTED
        payload_matches = observed["payload_hash"] == action.payload_hash
    elif attempt_recorded:
        effect_status = EffectStatus.UNKNOWN
        payload_matches = True
    else:
        effect_status = EffectStatus.NOT_STARTED
        payload_matches = True

    return RecoveryCut(
        intent_recorded=intent_recorded,
        authority_recorded=authority_recorded,
        precondition_recorded=precondition_recorded,
        attempt_recorded=attempt_recorded,
        effect_status=effect_status,
        receipt_recorded=receipt is not None,
        payload_matches=payload_matches,
    )


def decide_recovery(
    cut: RecoveryCut,
    intent: Intent,
    recovery_authority: Authority,
    action: TypedAction,
    remote: RemoteLedger,
    now: datetime,
) -> RecoveryDecision:
    if not cut.causally_closed or not cut.payload_matches:
        return RecoveryDecision.QUARANTINE

    if cut.effect_status == EffectStatus.COMMITTED:
        # Closing evidence for an already-authorised effect is not a fresh effect.
        return RecoveryDecision.ADOPT

    if not recovery_authority.permits(intent, now):
        return RecoveryDecision.REAUTHORISE

    if cut.effect_status == EffectStatus.NOT_STARTED:
        return RecoveryDecision.EXECUTE

    # An attempt exists but its outcome is unknown.
    if remote.supports_readback:
        return RecoveryDecision.VERIFY
    if remote.supports_idempotency:
        return RecoveryDecision.EXECUTE
    return RecoveryDecision.QUARANTINE


def run_scenario(
    crash_at: CrashPoint,
    *,
    idempotent: bool,
    readable: bool,
    revoke_before_recovery: bool,
) -> Tuple[RecoveryDecision, int, RecoveryCut]:
    t0 = datetime(2030, 1, 1, 9, 0, tzinfo=timezone.utc)
    intent = Intent("run-190", 3, "acct-7", "supplier-payment", 125_000, "Northwind")
    execution_auth = Authority(
        "allow-17", "acct-7", "supplier-payment", 200_000, t0 + timedelta(minutes=10)
    )
    recovery_auth = Authority(
        "allow-18",
        "acct-7",
        "supplier-payment",
        200_000,
        t0 + (timedelta(minutes=1) if revoke_before_recovery else timedelta(hours=1)),
    )
    action = TypedAction("pay-run-190-v3", 3, "Northwind", 125_000, 41)
    journal = Journal()
    remote = RemoteLedger(idempotent, readable)

    try:
        execute(intent, execution_auth, action, journal, remote, t0, crash_at)
    except InjectedCrash:
        pass

    recovery_time = t0 + timedelta(minutes=15)
    cut = reconstruct_cut(journal, remote, action)
    decision = decide_recovery(
        cut, intent, recovery_auth, action, remote, recovery_time
    )
    return decision, len(remote.commits), cut


if __name__ == "__main__":
    matrix = [
        (CrashPoint.AFTER_AUTHORITY, True, True, False),
        (CrashPoint.AFTER_REMOTE_COMMIT, True, True, False),
        (CrashPoint.AFTER_REMOTE_COMMIT, False, False, False),
        (CrashPoint.AFTER_SEND, False, False, True),
    ]
    for crash, idem, readback, revoked in matrix:
        decision, commits, cut = run_scenario(
            crash,
            idempotent=idem,
            readable=readback,
            revoke_before_recovery=revoked,
        )
        print(
            f"{crash.value:26} idempotent={idem:<5} readback={readback:<5} "
            f"revoked={revoked:<5} -> {decision.value:24} "
            f"commits={commits} effect={cut.effect_status.value}"
        )

stateDiagram-v2
  [*] --> Claim
  state "A checkpoint saves a perspective" as Claim
  state "The hidden assumption inside<br/>“resume”" as Test
  state "From distributed snapshots to<br/>governed effects" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Compensation is not time reversal

Sagas decompose long-lived transactions into steps with compensating transactions for committed work when later steps fail.[6] The pattern is indispensable, but the word “compensate” can suggest a symmetry that rarely exists. Refunding a payment does not erase the recipient’s temporary access to funds. Sending a correction does not retract a disclosed secret. Rebooking does not restore a missed medical appointment. A compensator creates another effect with its own authority, cost, delay and evidence.

The RCC should therefore record semantic reversibility rather than a binary “can undo”. A useful scale distinguishes exact inverse, business offset, best-effort repair and irreversible effect. The more irreversible the action, the stronger the pre-effect controls and the lower the tolerance for an unresolved outcome. Compensation reduces consequence; it does not rewrite history.

Exactly once is always scoped

Durable workflow systems provide valuable execution guarantees. Temporal reconstructs workflow state from event history and checks deterministic replay.[7] Azure Durable Task uses event sourcing to maintain orchestration state and likewise constrains orchestrator code to deterministic behaviour.[8] AWS distinguishes Standard workflows, described with exactly-once workflow execution, from Express workflows with at-least-once execution.[12] Kafka documents exactly-once processing within specific transaction and connector conditions, while warning that connector support can determine whether the guarantee is possible.[13]

These claims are not interchangeable, and none should be stretched beyond its declared boundary. “Exactly once” may refer to a state-machine transition, a transaction over a log and offsets, a connector’s interaction with a sink, or a provider’s idempotent interpretation of an action ID. An execution guarantee is meaningful only with its system boundary, failure model and external effect contract attached. A workflow engine can avoid starting one of its own states twice while an activity invoked by that state still faces an unknown remote outcome. The end-to-end argument remains relevant: some correctness properties can be completed only by the application endpoints that know what the business effect means.[14]

Four boundaries the certificate cannot remove

First, a provider may not expose authoritative readback. A sent email, a voice instruction, a physical actuator or a human conversation may create an effect that cannot be queried by action ID. The RCC can show that evidence is insufficient. It cannot manufacture observability.

Second, the provider may lie or be internally inconsistent. A receipt and a readback from the same compromised system are correlated witnesses. High-consequence actions may require an independent downstream observation, such as a ledger movement, inventory count or recipient confirmation.

Third, policy can be temporally complex. A later rule may forbid displaying data needed to investigate an earlier effect. Recovery authority must be separately scoped so that evidence collection does not become an unbounded bypass of current policy.

Fourth, concurrency can change the object under repair. A compensation based on account version 41 may be unsafe after legitimate actions advance the account to version 47. Recovery must either use conditional writes against current versions or reason about commutative effects. The RCC records the old precondition; it does not freeze the world.

Lineage-driven fault injection offers a useful research precedent: reason backwards from desired outcomes to identify combinations of faults that could prevent them.[16] The RCC supplies a domain-specific lineage for consequential actions. A failure injector can delete receipts, delay readbacks, revoke authority, fork workers, mutate payloads and reorder messages. The test is not whether the workflow eventually reports success. It is whether every reported success has one authorised effect and enough evidence to justify resumption.

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: The payment that already happened
  M->>E: A checkpoint saves a perspective
  E-->>M: compare rival explanations
  E->>D: Resume only from a causally closed cut
  D-->>Q: revise scope or act

The familiar architecture decision is whether to use checkpoints, event sourcing, durable workflows or retries. The changed decision is more specific: before selecting a recovery mechanism, decide where external truth, authority and evidence can be joined into a causally closed cut. If the join does not exist, no orchestration feature can make resumption safe by itself.

For each consequential action, require one stable action ID and payload fingerprint. Bind them to an immutable intent version, an authority decision valid at effect time and a versioned precondition. Record the attempt before release. Resolve the external outcome through authoritative readback or a documented idempotency contract. Verify the postcondition through a path suited to the consequence. Issue a recovery receipt that makes the continuation decision reviewable.

Then design explicit failure outcomes. A proved absence plus fresh authority permits execution. A matching prior effect permits adoption. An unknown effect with reliable readback permits verification. An absent effect with expired authority requires reauthorisation. A mismatched or unresolvable effect requires compensation or quarantine. Do not let a generic retry policy choose among these states.

The smallest deployable version is not a new platform. It is a typed action journal, an authority decision ID, a provider action key, an outcome query, a postcondition check and a recovery decision table. Add the RCC when those fields span several systems. Add systematic failure injection before claiming recoverability.