A checkpoint is one witness, not the world

Checkpointing solves a real problem. A long-running computation should not return to the beginning whenever a worker, machine or network fails. A checkpoint records enough durable state to reconstruct progress from a useful boundary. In distributed systems, the deeper problem is obtaining a consistent cut across participants whose events have only a partial order. Chandy and Lamport’s distributed snapshot formulation established how a system can record a meaningful global state without stopping all computation.

Modern stream processors inherit that lineage. Apache Flink describes a snapshot as source positions plus the operator state produced up to those positions, and uses a Chandy-Lamport variant for asynchronous barrier snapshotting. Its documentation also makes a decisive qualification: exactly-once managed state is not automatically exactly-once end to end. Replayable sources and transactional or idempotent sinks are required for that stronger claim. The checkpoint is powerful precisely because its system boundary is explicit.

Agent and workflow runtimes often blur that boundary. Their checkpoint may contain messages, model outputs, a plan cursor, tool results, retrieved context and a compact state object. None of those fields is necessarily authoritative about the live bank account, ticketing system, inbox, code repository or approval service. A serialised belief that an effect is pending does not make the effect pending. A remembered credential does not make it valid.

Three different kinds of state

The refund case becomes easier when “state” is split into three stores. Workflow state records the runtime’s progress through a procedure. World state records what currently exists in authoritative external systems. Evidence state records why the runtime believes an action was proposed, admitted, attempted and observed. They may be correlated, but they are not interchangeable.

This distinction also prevents the word “memory” from swallowing unlike objects. The conversation that led to the refund may be episodic memory. The customer’s current balance is world state. The approval policy is enterprise knowledge. The action receipt is operating evidence. The node cursor is workflow state. Persisting all five in one database does not erase their different authority, freshness or recovery semantics.

Thought experiment: two identical checkpoints

Imagine copying one checkpoint into two sealed envelopes. In world A, the refund API rejected the request before doing anything. In world B, it committed the refund and lost the response. Open either envelope and the bytes are identical. The only causal feature changed is the external effect.

If a recovery algorithm reads only the checkpoint, it must make the same decision in both worlds. Yet no single decision is safe: retry is necessary in A and duplicative in B. Therefore no checkpoint-only policy can solve the lost-response interval. It needs another observation, such as a stable operation lookup, a transactional sink, or a postcondition query. This is an information limit, not a prompting problem.

The practical consequence is severe. A checkpoint schema should never claim more authority than its producer possesses. A field named payment_status inside the workflow is a recorded belief unless the payment system itself owns that field or provides a verifiable receipt. Better names expose the boundary: last_observed_payment_status, observed_at, operation_id and source_revision.

Checkpoint granularity and consistency debt

More frequent checkpoints reduce recomputation, but they do not automatically reduce ambiguity. Saving immediately before a tool call makes the intended request recoverable. Saving immediately after the response makes the acknowledged result recoverable. A failure can still occur after the target commits and before the second save. The dangerous interval becomes shorter, not logically closed.

Checkpoint frequency is therefore an economic parameter, while effect reconciliation is a correctness mechanism. For expensive pure computation, frequent saves trade storage and latency for less repeated work. For consequential effects, the design question is where an operation identity, admission and receipt become durable. A one-millisecond ambiguity can duplicate a payment just as surely as a one-minute ambiguity.

Granularity also determines how much old reasoning is replayed. Persisting after every model token would be wasteful and would still say nothing about an external commit. Persisting only at the end of a mission makes recovery coarse and may repeat costly proposals. A useful boundary lies before a consequential intent is admitted, after the effect owner has made the outcome queryable, and after independent postcondition verification. Those are semantic boundaries, not arbitrary timer intervals.

Replay stops at the effect boundary

Replay is often described as running a workflow again. That phrase hides its useful discipline. A durable runtime normally reconstructs decisions from recorded history rather than repeating every piece of work. Temporal’s official workflow documentation explains that code is re-executed against an event history, with recorded activity results reused. Microsoft’s Durable Task documentation similarly requires orchestrator code to be deterministic because it may replay several times.

Inside that deterministic region, replay is a controlled reconstruction. A recorded timer does not wait again. A recorded random choice does not change. A completed tool result can be memoised. The runtime can rebuild local variables and reach the same logical position without re-performing the external work represented by those events.

The membrane is crossed when code changes a system whose commit record is not atomically coupled to the workflow history. Sending an email, transferring money, deleting an object, submitting an order or updating a third-party case creates a fact outside the replay log. The runtime can remember an attempted call. It cannot infer commitment from silence.

Idempotency is about intent, not resemblance

HTTP semantics call a method idempotent when several identical requests have the same intended server effect as one request. That definition explains why retries can be safe, but it does not manufacture safety for a business operation. Two byte-identical requests may represent two intended purchases. Two syntactically different requests may represent one intended refund regenerated by a language model.

A durable operation identity carries the missing intent. The caller chooses an identifier before crossing the effect boundary and preserves it across retries. The server binds that identifier to the request’s semantic parameters, stores the outcome atomically with the mutation, and returns a semantically equivalent response on later uses. The AWS Builders’ Library describes this caller-provided request-token pattern and warns that reusing a token with different parameters should fail validation.

For an agent, deriving the key from regenerated text is unsafe. A restored model may phrase the same instruction differently or choose a fresh tool-call identifier. The identifier must come from deterministic case state, such as refund:{case_id}:{decision_version}, and the gateway must reject parameter drift. The idempotency record also needs a retention policy longer than the possible retry horizon.

The unknown outcome is a real state

Suppose the external system offers neither idempotent submission nor query by operation identity. The request times out after leaving the client. Four outcomes are conceptually possible: absent, pending, committed or failed. The client can observe only silence. Labelling that silence “failed” converts an epistemic gap into a duplicate. Labelling it “succeeded” converts the same gap into a possible omission.

Unknown outcome must be represented as a first-class state with its own release path. Recovery may query a business postcondition, search an authoritative ledger, wait for an asynchronous event, invoke manual reconciliation or escalate to the system owner. It must not silently collapse uncertainty into a retry policy.

This is where “exactly once” claims need a denominator and boundary. Kafka can couple offsets and output records inside its transaction domain. Flink can make managed state reflect each input exactly once. Neither fact proves that an arbitrary external bank, email provider or browser surface changed exactly once. The guarantee is compositional only when every boundary supplies the required transaction, idempotency or readback semantics.

Negative control: when ordinary replay is enough

Consider a workflow that calculates compound interest from immutable inputs and writes only its result into the same transactional database as its checkpoint. Change no external system, hold inputs fixed, and make the write atomic with progress. Here deterministic replay is sufficient. The restored computation can run again because repeated calculation has no external side effect and the final write belongs to one transaction domain.

This negative control matters because it prevents an over-engineered conclusion. The three-witness protocol is not needed for every node. It is needed where a step crosses an effect, freshness or authority boundary. Pure transformations, validated calculations and memoised model proposals can remain cheap to replay.

Three designs that shrink the ambiguity

The first design keeps progress and effect in one atomic domain. A database transaction can update business state and the workflow cursor together. A transactional outbox can commit the intended message beside local state, after which a relay delivers it with deduplication. This is the cleanest option when the organisation controls both schema and write path. Its boundary simply moves to the relay and consumer, which still need a delivery and idempotency contract.

The second design uses an idempotent command plus authoritative query. The caller creates o, the target atomically stores o, request parameters and outcome, and repeated submissions of o return the same semantic result. Recovery asks GET /operations/o before mutation. This pattern fits payment, provisioning and ticket APIs when one service owns the business effect. It fails if keys are retained for less time than checkpoints can resume, or if the query reports transport acceptance rather than the intended postcondition.

The third design separates reservation from confirmation. A try-confirm-cancel or gated-settlement protocol first reserves a bounded effect, then confirms it only after dependencies and authority are satisfied. Expiry can safely release an unconfirmed reservation. This reduces irreversible exposure, but adds timeout policy, resource locking and orphan cleanup. It is useful when the target can offer a provisional state and the business can tolerate delayed confirmation.

These designs are alternatives to blind retry, not interchangeable labels. Atomic coupling offers the strongest local proof. Idempotent command and readback compose across a service boundary. Reservation creates a reversible intermediate state. Compensation addresses an effect that already settled. Architecture should select the cheapest mechanism whose evidence matches the consequence.

Thought experiment: 16:59 and 17:01

Restore the same pre-action checkpoint twice. In the first world the clock reads 16:59, one minute before the approval expires. In the second it reads 17:01. Hold the plan, user, evidence and tool request constant. Only current authority changes.

A recovery system that treats the checkpointed approval as executable power acts in both worlds. That is historical authorisation masquerading as current authorisation. A safer design asks whether the exact operation was already admitted while the approval was current. If yes, the system may reconcile or complete that admitted operation according to its contract. If no, it must obtain fresh authority.

This distinction separates two events that are often compressed into one “tool call”. Admission decides whether an exact effect may enter execution; settlement decides whether the admitted effect became the verified outcome. Approval expiry after admission does not erase an already committed refund. It does prevent a restored workflow from inventing a new refund.

flowchart TD
  A["The refund that already happened"] --> B["A checkpoint is one witness,<br/>not the world"]
  B --> C["Replay stops at the effect<br/>boundary"]
  C --> D["The recovery consistency cut"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#5B5CF6,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

The recovery consistency cut

A useful checkpoint cut says which events belong before recovery and which remain after it. For effectful workflows, that cut must cross more than the execution log. The proposed recovery consistency cut joins three observations for one stable operation identity:

1. Progress, P: the durable workflow prefix, including checkpoint version and deterministic results.
2. Effect, E: the authoritative status of the external operation and its business postcondition.
3. Authority, A: current permission or an immutable record that the exact operation was admitted while permission was valid.

Let o be the stable operation identity, c the restored checkpoint and t recovery time. The recovery decision can be written as:

QE returns an effect state such as absent, pending, committed, failed or unknown. QA returns current, durably admitted, expired, revoked or consumed elsewhere. Changing the checkpoint changes what computation may be reconstructed. Changing the effect query changes whether action is still required. Changing the authority query changes whether a new action may be initiated. None of the three terms subsumes the others.

The safety obligation is more concrete than the slogan “exactly once”:

The first line controls duplication. The second prevents a checkpoint from conferring power. The third prevents optimistic completion based only on local narrative. A system may implement stronger transactional guarantees, but it should still be able to produce evidence for these three propositions.

A recovery protocol in seven moves

The cut becomes operational through a protocol. It is deliberately narrower than a universal transaction manager, so it can wrap APIs, durable workflow activities, human approvals and agent tools.

1. Fence the resumer. Claim the checkpoint version with compare-and-swap, a lease or a workflow-run token. Two workers must not independently externalise the same recovery decision.
2. Reconstruct deterministic progress. Replay only code whose choices are derived from recorded history. Reject nondeterministic changes rather than silently taking a new path.
3. Recover the stable operation identity. It must have been chosen before first execution and bound to the semantic request, actor, subject, purpose and decision version.
4. Read effect state first. Query the target service or an authoritative ledger. A committed effect moves to postcondition verification; it does not call the mutating endpoint again.
5. Resolve authority. If the effect is absent, find a durable admission for that exact operation. Without one, evaluate current identity, policy, purpose, freshness and approval.
6. Execute or reconcile. Execute only the same admitted operation through an idempotent or transactional gateway. If effect state is unknown, enter a reconciliation queue instead.
7. Write a recovery receipt. Record the checkpoint version, resume claimant, effect observation, authority decision, action taken, postcondition readback and remaining uncertainty.

The ordering matters. Checking authority before readback may prompt an unnecessary new approval for an effect that already happened. Executing before fencing allows competing resumers. Marking complete before postcondition verification confuses an accepted command with a verified business outcome.

What a recovery receipt must prove

A normal execution receipt says what a system attempted or accepted. A recovery receipt must additionally explain why acting after interruption was safe. At minimum it should bind:

• workflow and checkpoint identifiers, including the version claimed by the resumer;
• stable operation identity and a digest of its semantic parameters;
• requesting actor, represented subject and declared purpose;
• authority decision or prior admission, with policy version and relevant expiry;
• effect readback source, observed status and observation time;
• action decision: complete, execute, compensate, reauthorise, reconcile or abandon;
• effect receipt and independent postcondition verification; and
• unresolved uncertainty, owner and release condition.

This joins the ten control objects that an AI system otherwise tends to mix: intent, identity, world state, context, reasoning, authority, action, evidence, outcome, and release and recovery. The model may propose the next move. A deterministic authority kernel and outcome observer decide whether that move can safely cross the cut.

Freshness is not merely a timestamp check. The policy version may have changed. The customer may have withdrawn consent. A sanctions status may have changed. The underlying case may have been closed by another operator. Recovery therefore compiles current decision context rather than replaying old context as though time stopped with the worker.

Recovery state should be typed, not narrated

An agent can produce a fluent explanation that it “already handled” a step, but prose is not a recovery protocol. Use a closed state set with guarded transitions: NOT_ADMITTED, ADMITTED, EFFECT_PENDING, EFFECT_COMMITTED, OUTCOME_VERIFIED, UNKNOWN_OUTCOME, NEEDS_REAUTHORISATION and ABANDONED. Each transition names the evidence that permits it.

For example, EFFECT_PENDING → EFFECT_COMMITTED requires a target-owned receipt or readback. NOT_ADMITTED → ADMITTED requires current policy and identity. UNKNOWN_OUTCOME → EFFECT_PENDING is forbidden merely because a retry timer fired. A human disposition or new observation must resolve the uncertainty first. Invalid transitions should fail closed and become visible operating events.

The model may summarise evidence, propose a branch or identify a likely postcondition. The deterministic kernel validates the typed contract, queries authoritative sources and writes the transition. This keeps natural-language reasoning useful without allowing narrative confidence to rewrite the execution ledger.

From one refund to a production-shaped mission

Worked scenario, synthetic

Consider a fee-remediation mission containing 4,000 customer cases. Each case has evidence gathering, eligibility calculation, human review above a threshold, refund execution, customer notification and ledger reconciliation. The cases run in parallel, but each refund is consequential and each notification must describe the verified outcome.

The mission graph checkpoint records dependencies and progress. It is not allowed to hold ambient payment authority. For case 1842, the reviewer approves a £50 refund under policy version 12. The authority kernel creates an immutable admission bound to refund:1842:v3, the customer account, amount, purpose and decision version. The payment gateway accepts that operation identity and exposes a query endpoint returning absent, pending, committed or rejected.

The worker crashes after the gateway commits but before the workflow writes completion. On resume, a compare-and-swap claim fences case 1842. The runtime reconstructs the graph and queries the gateway. It finds the committed receipt, reads the account ledger to verify the £50 credit, writes a recovery receipt, and unlocks the notification node. It does not ask the model whether the refund “probably happened”, and it does not submit the refund again.

Case 1843 failed earlier, before admission. Its reviewer approval has since expired because policy version 13 changed the evidence threshold. The restored checkpoint still contains the old approval reference. The authority kernel finds no durable admission and rejects initiation under current policy. The case moves to NEEDS_REAUTHORISATION. Same mission, same recovery machinery, different causal facts.

Compensation is another effect, not time travel

Sagas divide a long transaction into local transactions with compensating actions. The pattern remains valuable when one atomic transaction cannot span participants. Its limit is semantic: sending a second email does not unsend the first; refunding a debit does not erase exposure, timing or customer experience; deleting a created record may be forbidden after downstream use.

Recovery therefore classifies effects. A reversible internal reservation may be compensated automatically. A reversible but externally visible effect may need review. An irreversible effect demands stronger admission, idempotency and readback before release. Compensation itself needs a stable identity, authority and receipt. Otherwise the recovery mechanism can duplicate the attempted repair.

Recent transactional-agent proposals make a related point by separating tool execution from settlement and tracking which effects must settle together. The general lesson does not require adopting any one framework: checkpoint, retry, lock and compensation each solve a different failure dimension, so their names should not be used as substitutes for a complete effect contract.

Concurrent recovery and stale workers

A crash is rarely clean. The original worker may be slow rather than dead. A queue may redeliver. An operator may click resume while an automated retry starts. An expired lease holder may wake and continue. Idempotency limits duplicated effects, but it does not prevent conflicting decisions or stale writes around the effect.

Use a monotonic checkpoint version, lease epoch or fencing token. Every state transition and effect admission carries that epoch. A downstream gateway rejects an older epoch when a newer resumer has taken ownership. Compare-and-swap ensures only one resumer consumes an interrupt. Fencing reaches farther: it protects against a stale worker that still has network access after losing the local race.

Least authority applies here. NIST’s zero-trust guidance treats authentication and authorisation as discrete functions before access to a resource. For recovery, the useful translation is that being the former workflow owner, possessing a network route or deserialising an old credential creates no implicit trust. Resume ownership and effect authority are re-established through explicit, bounded decisions.

Human interruption is a protocol event

Human review is often implemented as “pause, save, wait for input”. That description omits identity, scope and consumption. A review response should be a signed or attributable decision object bound to the case version, proposed operation, amount, policy purpose and expiry. The workflow checkpoint stores a reference and observed result, not a reusable bearer of ambient power.

Resume input also needs consume-once semantics. Two browser submissions, a queue redelivery and an operator retry must not create three approvals or fork the case accidentally. If a reviewer changes the amount, that is a new intent and operation version rather than a retry of the old one. If an operator deliberately explores an alternative branch, the fork must receive a new identity and no inherited claim over effects already settled on the original branch.

This makes human-in-the-loop recovery less mysterious. The person does not “wake the agent”. The person contributes a bounded decision to an evidence ledger. The runtime then re-evaluates whether that decision is applicable to the present case and whether another actor has already consumed it.

Recoverability decision instrument

Before allowing automatic resume for an action, classify it using the following matrix. The decision is made per effect contract, not per workflow name.

Prove recovery by interruption

Happy-path tests establish almost nothing about recovery. The dangerous states live between durable records: after admission but before dispatch, after external commit but before receipt, after receipt but before checkpoint, after lease loss but before worker termination, and after authority expiry but before a delayed retry arrives.

An interruption harness places a crash or timeout at each boundary, restarts from durable evidence, and observes both workflow state and external effects. The expected property is not merely “the run finishes”. It is that every completed case has one defensible effect, every absent effect was initiated under valid or prior admitted authority, every unknown outcome remains unreleased, and every losing resumer is fenced.

The accompanying standard-library Python artefact models four cases. The negative case generates a new attempt identity after a crash and produces two refunds. The positive case uses one stable identity, discovers the committed effect by readback and finishes after the original approval expires without invoking it again. A pre-admission crash followed by expiry stops for reauthorisation. A compare-and-swap race admits one resumer.

Executable artefact: interruption harness

Assumptions. The example uses an in-memory checkpoint store, authority kernel and refund gateway to expose the protocol rather than infrastructure. The safe gateway atomically indexes effects by operation identity and permits authoritative readback. The unsafe gateway treats each attempt as a new effect. Production implementations must provide durable storage, authentication, audit retention and failure handling for their own boundaries.

Expected output. The program exits only if all assertions pass, then prints the following report:

{
  "checkpoint_only_negative": {
    "external_effects": 2,
    "workflow_phase": "COMPLETE"
  },
  "consume_once_claim": {
    "accepted_resumers": 1,
    "rejected_resumers": 1
  },
  "effect_readback_positive": {
    "approval_consumed": true,
    "external_effects": 1,
    "resumed_after_expiry": true,
    "workflow_phase": "COMPLETE"
  },
  "expired_authority_boundary": {
    "external_effects": 0,
    "reason": "approval has expired",
    "workflow_phase": "NEEDS_REAUTHORISATION"
  }
}

#!/usr/bin/env python3
"""Synthetic interruption harness for recoverable execution.

The harness contrasts checkpoint-only replay with a recovery protocol that uses:
1. a stable operation identifier;
2. durable, operation-bound admission of authority;
3. effect readback rather than blind re-execution; and
4. a compare-and-swap resume claim.

It uses only the Python standard library and mutates no external systems.
Run: python3 recovery_interruption_harness.py
Expected: all assertions pass and a JSON report is printed.
"""

from __future__ import annotations

from dataclasses import dataclass, replace
from enum import Enum
import json
from typing import Dict, Optional


class Phase(str, Enum):
    APPROVED = "APPROVED"
    EFFECT_PENDING = "EFFECT_PENDING"
    COMPLETE = "COMPLETE"
    NEEDS_REAUTHORISATION = "NEEDS_REAUTHORISATION"


@dataclass(frozen=True)
class Checkpoint:
    case_id: str
    phase: Phase
    operation_id: str
    approval_id: str
    amount_pence: int
    version: int = 0
    resume_owner: Optional[str] = None
    reason: Optional[str] = None


@dataclass
class Approval:
    approval_id: str
    expires_at: int
    consumed_by: Optional[str] = None


@dataclass(frozen=True)
class Admission:
    operation_id: str
    approval_id: str
    admitted_at: int
    amount_pence: int


@dataclass(frozen=True)
class EffectReceipt:
    operation_id: str
    amount_pence: int
    sequence: int


class CheckpointStore:
    """A tiny optimistic-locking store used to fence concurrent resumers."""

    def __init__(self, checkpoint: Checkpoint) -> None:
        self._checkpoint = checkpoint

    def read(self) -> Checkpoint:
        return replace(self._checkpoint)

    def write(self, checkpoint: Checkpoint, expected_version: int) -> Checkpoint:
        if self._checkpoint.version != expected_version:
            raise RuntimeError("stale checkpoint version")
        self._checkpoint = replace(checkpoint, version=expected_version + 1)
        return self.read()

    def claim_resume(self, owner: str, expected_version: int) -> bool:
        if self._checkpoint.version != expected_version:
            return False
        if self._checkpoint.resume_owner is not None:
            return False
        self._checkpoint = replace(
            self._checkpoint,
            resume_owner=owner,
            version=expected_version + 1,
        )
        return True


class AuthorityKernel:
    """Consumes a current approval once and binds it to one exact operation."""

    def __init__(self, approvals: Dict[str, Approval]) -> None:
        self.approvals = approvals
        self.admissions: Dict[str, Admission] = {}

    def admit(
        self,
        approval_id: str,
        operation_id: str,
        amount_pence: int,
        now: int,
    ) -> Admission:
        prior = self.admissions.get(operation_id)
        if prior is not None:
            if (
                prior.approval_id != approval_id
                or prior.amount_pence != amount_pence
            ):
                raise RuntimeError("operation identifier reused with different intent")
            return prior

        approval = self.approvals[approval_id]
        if approval.expires_at < now:
            raise PermissionError("approval has expired")
        if approval.consumed_by not in (None, operation_id):
            raise PermissionError("approval was consumed by another operation")

        approval.consumed_by = operation_id
        admission = Admission(
            operation_id=operation_id,
            approval_id=approval_id,
            admitted_at=now,
            amount_pence=amount_pence,
        )
        self.admissions[operation_id] = admission
        return admission

    def read_admission(self, operation_id: str) -> Optional[Admission]:
        return self.admissions.get(operation_id)


class RefundGateway:
    """Idempotent effect service with authoritative readback by operation ID."""

    def __init__(self) -> None:
        self._effects: Dict[str, EffectReceipt] = {}
        self._sequence = 0

    def refund(self, admission: Admission) -> EffectReceipt:
        prior = self._effects.get(admission.operation_id)
        if prior is not None:
            if prior.amount_pence != admission.amount_pence:
                raise RuntimeError("operation identifier reused with different amount")
            return prior

        self._sequence += 1
        receipt = EffectReceipt(
            operation_id=admission.operation_id,
            amount_pence=admission.amount_pence,
            sequence=self._sequence,
        )
        self._effects[admission.operation_id] = receipt
        return receipt

    def unsafe_refund(self, attempt_id: str, amount_pence: int) -> EffectReceipt:
        """A non-idempotent API: every attempt identifier creates another effect."""
        self._sequence += 1
        receipt = EffectReceipt(attempt_id, amount_pence, self._sequence)
        self._effects[attempt_id] = receipt
        return receipt

    def read(self, operation_id: str) -> Optional[EffectReceipt]:
        return self._effects.get(operation_id)

    @property
    def effect_count(self) -> int:
        return len(self._effects)


class SimulatedCrash(RuntimeError):
    pass


def unsafe_checkpoint_only_case() -> dict:
    """Negative case: a crash after the effect causes a second refund on replay."""
    gateway = RefundGateway()
    checkpoint = Checkpoint(
        case_id="case-unsafe",
        phase=Phase.APPROVED,
        operation_id="refund:case-unsafe",
        approval_id="approval-unsafe",
        amount_pence=5_000,
    )

    gateway.unsafe_refund("attempt-1", checkpoint.amount_pence)
    # The process dies before recording completion. The restored checkpoint still
    # says APPROVED, so a naive resumer generates a fresh request identifier.
    gateway.unsafe_refund("attempt-2", checkpoint.amount_pence)
    checkpoint = replace(checkpoint, phase=Phase.COMPLETE)

    assert gateway.effect_count == 2
    return {
        "external_effects": gateway.effect_count,
        "workflow_phase": checkpoint.phase.value,
    }


def effect_readback_positive_case() -> dict:
    """Positive case: a committed effect is discovered, not executed again."""
    approval = Approval("approval-safe", expires_at=10)
    authority = AuthorityKernel({approval.approval_id: approval})
    gateway = RefundGateway()
    store = CheckpointStore(
        Checkpoint(
            case_id="case-safe",
            phase=Phase.APPROVED,
            operation_id="refund:case-safe:v1",
            approval_id=approval.approval_id,
            amount_pence=5_000,
        )
    )

    checkpoint = store.read()
    admission = authority.admit(
        checkpoint.approval_id,
        checkpoint.operation_id,
        checkpoint.amount_pence,
        now=5,
    )
    checkpoint = store.write(
        replace(checkpoint, phase=Phase.EFFECT_PENDING),
        expected_version=checkpoint.version,
    )
    gateway.refund(admission)

    try:
        raise SimulatedCrash("effect committed; completion checkpoint not written")
    except SimulatedCrash:
        pass

    # Resume after the original approval has expired. The resumer does not invoke
    # the approval again. It reads the effect by the same stable operation ID.
    resumed_at = 20
    checkpoint = store.read()
    receipt = gateway.read(checkpoint.operation_id)
    assert receipt is not None
    checkpoint = store.write(
        replace(checkpoint, phase=Phase.COMPLETE, reason="effect confirmed by readback"),
        expected_version=checkpoint.version,
    )

    assert resumed_at > approval.expires_at
    assert approval.consumed_by == checkpoint.operation_id
    assert gateway.effect_count == 1
    return {
        "approval_consumed": approval.consumed_by is not None,
        "external_effects": gateway.effect_count,
        "resumed_after_expiry": resumed_at > approval.expires_at,
        "workflow_phase": checkpoint.phase.value,
    }


def expired_authority_boundary_case() -> dict:
    """Boundary case: no durable admission exists, so an expired approval stops."""
    approval = Approval("approval-expired", expires_at=10)
    authority = AuthorityKernel({approval.approval_id: approval})
    gateway = RefundGateway()
    store = CheckpointStore(
        Checkpoint(
            case_id="case-expired",
            phase=Phase.APPROVED,
            operation_id="refund:case-expired:v1",
            approval_id=approval.approval_id,
            amount_pence=7_500,
        )
    )

    # The process crashes before authority is admitted and before any effect.
    checkpoint = store.read()
    try:
        authority.admit(
            checkpoint.approval_id,
            checkpoint.operation_id,
            checkpoint.amount_pence,
            now=20,
        )
    except PermissionError as error:
        checkpoint = store.write(
            replace(
                checkpoint,
                phase=Phase.NEEDS_REAUTHORISATION,
                reason=str(error),
            ),
            expected_version=checkpoint.version,
        )

    assert gateway.effect_count == 0
    assert checkpoint.phase is Phase.NEEDS_REAUTHORISATION
    return {
        "external_effects": gateway.effect_count,
        "reason": checkpoint.reason,
        "workflow_phase": checkpoint.phase.value,
    }


def consume_once_claim_case() -> dict:
    """Two resumers race; optimistic fencing admits only one owner."""
    store = CheckpointStore(
        Checkpoint(
            case_id="case-race",
            phase=Phase.EFFECT_PENDING,
            operation_id="refund:case-race:v1",
            approval_id="approval-race",
            amount_pence=2_500,
        )
    )
    version_seen_by_both = store.read().version
    claims = [
        store.claim_resume("worker-a", expected_version=version_seen_by_both),
        store.claim_resume("worker-b", expected_version=version_seen_by_both),
    ]
    assert claims.count(True) == 1
    return {
        "accepted_resumers": claims.count(True),
        "rejected_resumers": claims.count(False),
    }


def run() -> dict:
    report = {
        "checkpoint_only_negative": unsafe_checkpoint_only_case(),
        "consume_once_claim": consume_once_claim_case(),
        "effect_readback_positive": effect_readback_positive_case(),
        "expired_authority_boundary": expired_authority_boundary_case(),
    }
    return report


if __name__ == "__main__":
    print(json.dumps(run(), indent=2, sort_keys=True))

Evidence that should survive the game day

Operational evidence needs enough structure to answer five questions without reconstructing a story from logs: Who initiated the case? Which exact effect was admitted? What did the target system do? Why did the resumer choose its branch? What verified business outcome followed?

Measure duplicate-effect count, unresolved unknown outcomes, stale-resumer rejection, reauthorisation frequency, time to authoritative readback, postcondition mismatch and compensation failure. Report denominators and consequence classes. “Ninety-nine per cent recovered” is uninformative if the remaining one per cent contains all irreversible actions.

Run the harness against upgrades as well as failures. Replay-based runtimes require deterministic code evolution. Tool schemas change. Idempotency retention windows expire. Policy engines alter decisions. A checkpoint that restored correctly last month may no longer be semantically valid against today’s action contract. Recovery compatibility belongs in the release manifest.

stateDiagram-v2
  [*] --> Claim
  state "Three different kinds of state" as Claim
  state "Thought experiment: two<br/>identical checkpoints" as Test
  state "Checkpoint granularity and<br/>consistency debt" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Compact glossary

Checkpoint

A durable representation sufficient to reconstruct workflow progress from a declared boundary.

Replay

Re-execution against recorded history to rebuild deterministic state without repeating recorded work.

Admission

An immutable decision that one exact, typed operation may enter execution under valid authority.

Effect receipt

Evidence from the effect owner that a stable operation was accepted, committed, rejected or remains pending.

Readback

A non-mutating query of authoritative effect state or the intended business postcondition.

Unknown outcome

A state in which the caller cannot determine whether an external effect committed after an ambiguous response.

Fencing token

A monotonic ownership value that lets downstream components reject stale workers.

Recovery receipt

A joined record of progress, effect observation, authority and the recovery decision.

Resume only what can still be defended

Checkpointing makes long work survivable. Replay makes reconstructed computation coherent. Neither one guarantees that the world outside the runtime agrees, or that old permission still authorises new action. The refund case fails because one local checkpoint is asked to speak for three systems.

The architecture decision changes from “which engine can persist an agent?” to “where is the recovery consistency cut for each consequential effect?” For every cut, require a stable operation identity, explicit admission, authoritative effect readback, current-state reconciliation, a fenced resumer and a recovery receipt. Use ordinary replay for pure or atomically local work. Use idempotent or transactional gateways where external effects can be designed. Stop at unknown outcomes where they cannot.

The research decision changes too. Evaluate resume semantics by injecting interruption at the seams, not by showing that a workflow eventually continues. A recovery mechanism earns trust when it can demonstrate bounded effects, non-resurrected authority and verified outcomes under the failures it claims to survive.