The attempt is not the process

A worker attempt is one bounded execution of code: one model call, one document extraction, one API invocation or one function process. A workflow is the longer-lived business object that survives attempts. An effect is the change left in an external system. These three objects often share one log stream in a demonstration, which makes them look like one thing. They are not.

Language-model workers add visible variability. In Published evidence tau-bench, agents were evaluated across repeated tool-using conversations, and the authors introduced pass^k to expose consistency across trials rather than celebrating one successful run. That benchmark does not establish the failure rate of any particular production system. It establishes a useful design premise: repeated executions of the same nominal task can diverge, so progress cannot safely reside only inside one execution trace.

What a retry can and cannot buy

If one attempt succeeds with probability p, and failures are independent and transient, the probability that at least one of n attempts succeeds is:

The equation is useful because its assumptions are easy to violate. Invalid input remains invalid. A revoked permission remains revoked. Ten clients retrying an overloaded dependency can deepen the overload. Google SRE documents how retries can amplify cascading failure, while the AWS Builders' Library recommends bounded backoff and jitter to avoid synchronised waves.^{8, 6} A retry policy is therefore a hypothesis about the failure class, not a universal reflex.

Give one layer the retry budget

Retries become especially dangerous when every layer believes it is being helpful. Suppose an activity calls a client library, which calls a service gateway, which calls another service, and each of four layers permits three attempts. One workflow attempt can create as many as 3 × 3 × 3 × 3 = 81 downstream calls before the workflow itself decides to retry. Add a fifth retrying layer and the worst-case fan-out becomes 243. The arithmetic is simple; the production symptom is often misdiagnosed as an inexplicable traffic spike.

Assign retry ownership to the layer that understands both the business deadline and the effect semantics. A low-level client may perform one transport retry when it can prove no request bytes were accepted, but it should not silently consume the whole case budget. The workflow should know how many attempts have occurred, how much time remains and whether the next attempt is still permitted by cost, authority and downstream health.

Carry a deadline or remaining budget through the call chain. Each activity must reserve enough time to report a clean failure before the outer workflow expires. When the budget is exhausted, transition to a known state such as DEPENDENCY_UNAVAILABLE or MANUAL_REVIEW_REQUIRED. Do not let a worker continue in the background after the business process has already decided that its result is too late to use.

Hedged requests, where a second copy races a slow first copy, illustrate the same boundary. They can reduce tail latency for replay-safe reads, as the systems literature on tail latency explains.¹³ They are unsafe for an unprotected write because the “winning” response says nothing about whether the losing request also committed. Redundant work is a latency technique only when duplicate execution has controlled semantics.

The dangerous interval after a write

There are three relevant crash points around an external write. A worker can stop before the remote system commits, after the commit but before the response arrives, or after the response has been durably recorded. Ordinary timeout handling collapses the first two into the same local observation: “no response”. Their correct recovery actions are opposite. The first needs another attempt. The second may duplicate an effect.

Unknown outcome is a legitimate workflow state. It should trigger a query by operation identity, a postcondition check or an operator route. AWS's guidance on idempotent APIs uses the same underlying dilemma: a caller that receives no response cannot infer whether a side effect occurred, so the service needs a stable client request identity or the caller must reconcile.⁷ Temporal likewise warns that an activity can execute more than once even when its completion is observed once by the workflow.³

A negative control: when durability adds little

Consider a pure function that converts a public JSON document into a canonical ordering and writes no external state. It finishes in 200 milliseconds and the caller can recompute it cheaply. A queue with ordinary retry is enough. A durable event history, compensation graph and long-running state machine would add operational surface without changing the outcome.

This negative control matters. Durable orchestration earns its cost when progress outlives a process or when an attempt can leave consequential partial effects. It is not a decorative layer for every model call.

flowchart LR
  A["The credit that happened twice"] --> B["The attempt is not the process"]
  B --> C["The deterministic shell"]
  C --> D["Build the effect protocol"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#087A98,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

The deterministic shell

Durable-execution systems usually rely on one of two implementation families. A declarative state machine persists each transition. A code-first engine persists an event history and replays orchestration code to reconstruct local state. Both separate orchestration from activities that touch the outside world.

Temporal documents replay as re-executing workflow code against recorded events, reusing prior activity results instead of redoing the external work. Microsoft's Durable Task documentation describes the same event-sourcing constraint: orchestrator code must be deterministic because it can replay multiple times.^{2, 9} “Deterministic” here does not mean the model gives the same answer. It means that once the workflow has accepted and recorded an answer, replay reaches the same next command.

That distinction produces a practical architecture rule: randomness, model inference, network calls, clocks and database reads belong in activities; branching on their recorded results belongs in the workflow. A fresh worker may execute the next activity, but it does not invent a new past.

A state transition must carry a contract

A box labelled “call agent” is not enough. Each material step needs an input identity, a precondition, a completion condition and an explicit failure route. For an external effect it also needs an idempotency strategy, a readback rule and, where possible, compensation.

Five invariants make the shell durable

The surface syntax of an orchestrator can vary, but five invariants should survive a change of language, cloud or framework.

First, one logical case has one stable identity. Every attempt, callback, approval and effect carries that identity or a derived step identity. A new worker does not create a new case merely because the old process disappeared.

Second, accepted facts are monotonic. Once the workflow records that evidence set E was used, proposal P was accepted or effect R was observed, replay may read that fact but must not silently replace it with a fresh model answer. A later correction is a new event with provenance, not an edit that erases the old basis.

Third, state transitions use compare-and-set semantics. Two workers may race after a lease expires or a message is delivered twice. Only one should advance ANALYSED to CREDIT_REQUESTED. The loser observes that the expected prior state has changed and stops. This is why a durable workflow needs optimistic concurrency, a transactional state update or an equivalent fencing mechanism rather than “last write wins”.

Fourth, activity completion and business completion are separate. A worker returning “credit issued” is an activity result. The business postcondition is an independently observed ledger entry with the expected identity, amount and beneficiary. The workflow advances on the latter when consequence is material.

Fifth, every non-terminal state has an owner and a clock. WAITING_FOR_APPROVAL names an approver route and expiry. OUTCOME_UNKNOWN names a reconciliation owner and deadline. COMPENSATION_FAILED remains open until an operator accepts or repairs it. Durability without ownership can preserve a stalled case forever.

These invariants also clarify what to persist. The history needs case and step identities, accepted inputs and outputs, transition versions, deadlines, authority decisions and effect receipts. It does not need an unbounded transcript of hidden reasoning or every token emitted by a model. Persist what is necessary to reconstruct the next authorised command and explain the resulting effect.

Timeouts are state transitions

A timeout is often implemented as an exception in worker code. In a durable workflow it is better understood as a transition caused by the absence of an expected event before a deadline. That shift matters because the process can stop consuming compute while it waits, and a different worker can handle the timeout route later.

Use separate budgets where the distinction changes recovery: time waiting for capacity, time allowed for one attempt, time between progress heartbeats and total business deadline. A worker that has not started is different from one that started and may have changed an external system. A long-running extraction that reports progress is different from a silent process at the same wall-clock age.

Temporal's long-running activity guidance uses heartbeats as resumable progress markers and explicitly notes that the last checkpoint may be processed again, so the boundary still needs idempotency.⁴ The general lesson is broader than one runtime: a checkpoint narrows recomputation, but it does not turn the interval after the checkpoint into an atomic transaction.

Duplicate workers and late completion

Durable systems commonly provide at-least-once activity execution because a scheduler cannot always distinguish a dead worker from a slow one. A timeout may release the task to worker B while worker A is still running. If both can write, an idempotency key protects the business effect, but the workflow state also needs a fencing rule so the late worker cannot overwrite a newer result.

A monotonically increasing attempt or lease token is one pattern. The workflow issues token 12 to worker A, later expires it and issues token 13 to worker B. A state update or effect adapter accepts only the highest valid token. Merely checking a timestamp inside worker A is weaker because pauses, clock skew and network delay can let stale work arrive after the lease has changed.

Cancellation has the same shape. Sending “cancel” expresses intent; it does not prove the worker stopped or the external system rolled back. The workflow should record CANCEL_REQUESTED, wait for an acknowledgement or effect readback, and define what happens to a result that arrives after cancellation. Safe reads can be discarded. A committed write may require compensation. A human approval received after case cancellation should be retained as an event but denied authority to advance the case.

Late results are normal distributed-system events, not exceptional mysteries. Design their acceptance rules before deployment, especially for long model calls, browser automation and external batch jobs that cannot be interrupted promptly.

Thought experiment 2: the reviewer who replies tomorrow

A model proposes a payment exception that requires human approval. In design A, a worker process remains open, polling a mailbox every minute. In design B, the workflow stores WAITING_FOR_APPROVAL, an approval request identity, an expiry time and the model proposal hash, then releases the worker.

Kill both compute hosts overnight. Design A loses its in-memory correlation unless separate recovery code reconstructs it. Design B starts no special recovery path. The persisted state already says what is awaited. When a signed approval event arrives, any worker can validate the approver, deadline and proposal version before advancing.

Now vary one detail: the reviewer replies after the approval expired. A raw callback token may still wake the process. A durable business state can reject the event because authority was time-bounded. Waiting is not inactivity; it is an explicit state with identity, ownership and expiry.

Idempotency first, compensation second

Idempotency means repeating the same logical request does not create an additional effect. It is strongest when the receiving system enforces a uniqueness constraint on a caller-supplied operation key and returns the original result for a semantically identical retry. A cache in the caller is weaker because it can be lost exactly when it is needed.

Idempotency also has a semantic boundary. Reusing the same key with a different amount or beneficiary must fail, not silently return a prior result. The operation identity binds an intent, not merely a network packet. Store a request hash or canonical business parameters with the key so “same key, different intent” becomes an explicit conflict.

Some effects cannot be made idempotent at the target. Others succeed once but later need to be mitigated because a subsequent step fails. The saga formulation introduced compensating transactions for long-lived work that cannot remain inside one atomic transaction.¹ A compensation is not time travel. It is a new authorised action in a changed world. Releasing a reservation may fail. Reversing a credit may require review. A notification cannot be unread.

Represent compensation as states such as COMPENSATION_REQUIRED, COMPENSATING, COMPENSATED and COMPENSATION_FAILED. Do not hide it in a catch block that logs and moves on. The business case remains open until the compensating effect is observed or an accountable owner accepts the residual state.

Compensations also have an order. If a workflow reserves inventory, issues a credit and then schedules delivery, a failure after the third step cannot be repaired by running every inverse in arbitrary sequence. Later effects may depend on earlier ones, so compensations usually run in reverse dependency order and each one has its own preconditions. A delivery already collected by a courier may no longer be cancellable; the workflow must choose a different remedy.

Do not assume compensation restores the original world. Time has passed, prices may have changed, a customer may have acted on a notification and another process may have consumed the reserved resource. The defensible claim is narrower: compensation moves the system into an explicitly acceptable state under current evidence and authority. That state may be “financially neutral”, “customer made whole” or “manual remediation opened”, rather than a byte-for-byte reversal.

For this reason, store both the original effect receipt and the compensation receipt. An audit trail that shows only the net balance hides the operational path and makes it impossible to distinguish an ordinary completion from a failure that required recovery. Recovery is part of the business history, not noise to be erased after the balance looks right.

The hard boundary of the shell

Durability solves loss of progress and ambiguity about execution. It does not establish that the model's proposal was correct, that evidence was sufficient, that the policy was fair or that the tool was authorised. If a wrong proposal is recorded, replay faithfully preserves the wrong proposal. If a target API offers neither idempotency nor a reliable query by operation identity, the orchestrator cannot manufacture exactly-once effects from outside.

The shell converts hidden execution failure into explicit state; it does not convert weak judgement into truth. Quality evaluation, authority checks, source verification and human decision rights remain separate controls.

Build the effect protocol

The most useful durable workflow design starts at the consequence and works backwards. Ask what observable state would prove the effect happened exactly once for this intent. Then define the request identity, the readback, the timeout route and the evidence record. Only after that should the model prompt or agent loop be attached.

Success, failure and unknown are different outcomes

Many APIs expose a binary interface: return value or exception. A consequential workflow needs at least three execution outcomes. Success means the postcondition was observed. Known failure means the target rejected the request or the workflow knows the effect did not occur. Unknown means the caller cannot yet distinguish commit from non-commit.

Worked example: a resumable provisional credit

The executable artefact below uses only Python's standard library and SQLite. A stochastic analysis worker fails once, then returns a proposal. The orchestrator checkpoints the accepted proposal and deliberately crashes. A new process opens the same database and resumes from ANALYSED rather than calling the model again.

The credit adapter then commits an effect and deliberately loses its acknowledgement. In the positive case, a stable idempotency key makes the retry return the first effect. In the negative case, the key is removed and the same blind retry creates two credits. The artefact is synthetic, but the positive and negative cases isolate the exact mechanism.

#!/usr/bin/env python3
"""A minimal durable shell around a stochastic worker.

Standard-library only. The positive demo proves that a workflow can resume after
an orchestrator crash and can survive an ambiguous external write when the
write uses a stable idempotency key. The negative demo removes that key and
shows the duplicate effect created by a blind retry.
"""

from __future__ import annotations

import argparse
import json
import random
import sqlite3
import sys
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Callable, TypeVar

T = TypeVar("T")


class TransientFailure(RuntimeError):
    """The worker failed before producing a usable result."""


class AmbiguousOutcome(RuntimeError):
    """The external system may have committed, but its acknowledgement was lost."""


class SimulatedProcessCrash(RuntimeError):
    """The orchestrator process stopped after a durable checkpoint."""


@dataclass(frozen=True)
class RetryPolicy:
    max_attempts: int = 3


class Store:
    def __init__(self, path: Path) -> None:
        self.path = path
        self.db = sqlite3.connect(path)
        self.db.row_factory = sqlite3.Row
        self.db.executescript(
            """
            PRAGMA journal_mode=WAL;
            CREATE TABLE IF NOT EXISTS workflows (
                workflow_id TEXT PRIMARY KEY,
                state TEXT NOT NULL,
                analysis_json TEXT,
                last_error TEXT
            );
            CREATE TABLE IF NOT EXISTS events (
                seq INTEGER PRIMARY KEY AUTOINCREMENT,
                workflow_id TEXT NOT NULL,
                kind TEXT NOT NULL,
                step TEXT NOT NULL,
                payload_json TEXT NOT NULL
            );
            CREATE TABLE IF NOT EXISTS effects (
                effect_id INTEGER PRIMARY KEY AUTOINCREMENT,
                workflow_id TEXT NOT NULL,
                idempotency_key TEXT,
                effect_type TEXT NOT NULL,
                amount INTEGER NOT NULL
            );
            CREATE UNIQUE INDEX IF NOT EXISTS one_effect_per_key
                ON effects(idempotency_key)
                WHERE idempotency_key IS NOT NULL;
            CREATE TABLE IF NOT EXISTS injected_faults (
                workflow_id TEXT NOT NULL,
                fault_name TEXT NOT NULL,
                PRIMARY KEY (workflow_id, fault_name)
            );
            """
        )
        self.db.commit()

    def close(self) -> None:
        self.db.close()

    def ensure_workflow(self, workflow_id: str) -> None:
        self.db.execute(
            "INSERT OR IGNORE INTO workflows(workflow_id, state) VALUES (?, 'NEW')",
            (workflow_id,),
        )
        self.db.commit()

    def workflow(self, workflow_id: str) -> sqlite3.Row:
        row = self.db.execute(
            "SELECT * FROM workflows WHERE workflow_id = ?", (workflow_id,)
        ).fetchone()
        if row is None:
            raise KeyError(workflow_id)
        return row

    def transition(
        self,
        workflow_id: str,
        state: str,
        *,
        analysis: dict[str, Any] | None = None,
        error: str | None = None,
    ) -> None:
        analysis_json = json.dumps(analysis, sort_keys=True) if analysis else None
        self.db.execute(
            """
            UPDATE workflows
               SET state = ?,
                   analysis_json = COALESCE(?, analysis_json),
                   last_error = ?
             WHERE workflow_id = ?
            """,
            (state, analysis_json, error, workflow_id),
        )
        self.event(workflow_id, "transition", state, {"error": error})
        self.db.commit()

    def event(
        self, workflow_id: str, kind: str, step: str, payload: dict[str, Any]
    ) -> None:
        self.db.execute(
            "INSERT INTO events(workflow_id, kind, step, payload_json) VALUES (?, ?, ?, ?)",
            (workflow_id, kind, step, json.dumps(payload, sort_keys=True)),
        )

    def fault_once(self, workflow_id: str, fault_name: str) -> bool:
        """Return True once, durably, for this workflow and fault name."""
        try:
            self.db.execute(
                "INSERT INTO injected_faults(workflow_id, fault_name) VALUES (?, ?)",
                (workflow_id, fault_name),
            )
            self.db.commit()
            return True
        except sqlite3.IntegrityError:
            return False

    def effects(self, workflow_id: str) -> list[sqlite3.Row]:
        return list(
            self.db.execute(
                "SELECT * FROM effects WHERE workflow_id = ? ORDER BY effect_id",
                (workflow_id,),
            )
        )

    def event_log(self, workflow_id: str) -> list[sqlite3.Row]:
        return list(
            self.db.execute(
                "SELECT * FROM events WHERE workflow_id = ? ORDER BY seq",
                (workflow_id,),
            )
        )


class SyntheticWorkers:
    """Failure-prone work and an external effect adapter."""

    def __init__(self, store: Store, *, enforce_idempotency: bool) -> None:
        self.store = store
        self.enforce_idempotency = enforce_idempotency

    def analyse_case(self, workflow_id: str, attempt: int) -> dict[str, Any]:
        # The first attempt fails to make retry behaviour visible. Later output is
        # seeded for a reproducible demo, but it still represents a fallible worker.
        if attempt == 1:
            raise TransientFailure("model endpoint returned 503")
        rng = random.Random(f"{workflow_id}:{attempt}")
        confidence = round(0.80 + rng.random() * 0.10, 3)
        return {
            "proposal": "ISSUE_CREDIT" if confidence >= 0.75 else "HUMAN_REVIEW",
            "confidence": confidence,
            "worker_attempt": attempt,
        }

    def issue_credit(self, workflow_id: str, amount: int) -> dict[str, Any]:
        key = f"{workflow_id}:issue-credit:v1" if self.enforce_idempotency else None

        if key is not None:
            existing = self.store.db.execute(
                "SELECT * FROM effects WHERE idempotency_key = ?", (key,)
            ).fetchone()
            if existing is not None:
                return {
                    "effect_id": existing["effect_id"],
                    "deduplicated": True,
                    "idempotency_key": key,
                }

        cursor = self.store.db.execute(
            """
            INSERT INTO effects(workflow_id, idempotency_key, effect_type, amount)
            VALUES (?, ?, 'PROVISIONAL_CREDIT', ?)
            """,
            (workflow_id, key, amount),
        )
        effect_id = int(cursor.lastrowid)
        self.store.db.commit()  # The external effect is now real.

        # The acknowledgement disappears after commit. The caller sees a timeout
        # and cannot infer whether the write happened from the exception alone.
        if self.store.fault_once(workflow_id, "credit_ack_lost"):
            raise AmbiguousOutcome(
                f"credit may have committed; acknowledgement for effect {effect_id} was lost"
            )

        return {
            "effect_id": effect_id,
            "deduplicated": False,
            "idempotency_key": key,
        }


class Orchestrator:
    def __init__(self, store: Store, workers: SyntheticWorkers) -> None:
        self.store = store
        self.workers = workers

    def retry(
        self,
        workflow_id: str,
        step: str,
        policy: RetryPolicy,
        fn: Callable[[int], T],
    ) -> T:
        last_error: Exception | None = None
        for attempt in range(1, policy.max_attempts + 1):
            self.store.event(workflow_id, "attempt", step, {"attempt": attempt})
            self.store.db.commit()
            try:
                result = fn(attempt)
                self.store.event(
                    workflow_id,
                    "attempt_succeeded",
                    step,
                    {"attempt": attempt},
                )
                self.store.db.commit()
                return result
            except (TransientFailure, AmbiguousOutcome) as exc:
                last_error = exc
                self.store.event(
                    workflow_id,
                    "attempt_failed",
                    step,
                    {"attempt": attempt, "error": str(exc)},
                )
                self.store.db.commit()
        assert last_error is not None
        raise last_error

    def run(self, workflow_id: str, *, crash_after_analysis: bool = False) -> None:
        self.store.ensure_workflow(workflow_id)
        state = self.store.workflow(workflow_id)["state"]

        if state == "NEW":
            analysis = self.retry(
                workflow_id,
                "ANALYSE",
                RetryPolicy(max_attempts=3),
                lambda attempt: self.workers.analyse_case(workflow_id, attempt),
            )
            self.store.transition(workflow_id, "ANALYSED", analysis=analysis)
            if crash_after_analysis:
                raise SimulatedProcessCrash("process stopped after ANALYSED checkpoint")
            state = "ANALYSED"

        if state == "ANALYSED":
            row = self.store.workflow(workflow_id)
            analysis = json.loads(row["analysis_json"])
            if analysis["proposal"] != "ISSUE_CREDIT":
                self.store.transition(workflow_id, "WAITING_FOR_HUMAN")
                return

            receipt = self.retry(
                workflow_id,
                "ISSUE_CREDIT",
                RetryPolicy(max_attempts=2),
                lambda _attempt: self.workers.issue_credit(workflow_id, amount=2500),
            )
            self.store.event(workflow_id, "effect_receipt", "ISSUE_CREDIT", receipt)
            self.store.transition(workflow_id, "CREDIT_OBSERVED")
            state = "CREDIT_OBSERVED"

        if state == "CREDIT_OBSERVED":
            # Notification is represented as a durable event in this small demo.
            self.store.event(
                workflow_id,
                "notification",
                "NOTIFY",
                {"template": "credit-issued", "recipient": "synthetic-user"},
            )
            self.store.transition(workflow_id, "COMPLETED")


def reset(path: Path) -> None:
    if path.exists():
        path.unlink()
    wal = Path(f"{path}-wal")
    shm = Path(f"{path}-shm")
    for extra in (wal, shm):
        if extra.exists():
            extra.unlink()


def print_summary(store: Store, workflow_id: str) -> None:
    row = store.workflow(workflow_id)
    effects = store.effects(workflow_id)
    print(f"workflow_state={row['state']}")
    print(f"external_effect_count={len(effects)}")
    for effect in effects:
        print(
            "effect",
            dict(effect),
        )
    print("event_kinds=" + ",".join(event["kind"] for event in store.event_log(workflow_id)))


def positive_demo(path: Path) -> int:
    reset(path)
    workflow_id = "case-positive"

    store = Store(path)
    workers = SyntheticWorkers(store, enforce_idempotency=True)
    orchestrator = Orchestrator(store, workers)
    try:
        orchestrator.run(workflow_id, crash_after_analysis=True)
    except SimulatedProcessCrash as exc:
        print(f"simulated_crash={exc}")
    finally:
        store.close()

    # A fresh process reconstructs progress from SQLite and resumes at ANALYSED.
    store = Store(path)
    workers = SyntheticWorkers(store, enforce_idempotency=True)
    Orchestrator(store, workers).run(workflow_id)
    print_summary(store, workflow_id)
    assert store.workflow(workflow_id)["state"] == "COMPLETED"
    assert len(store.effects(workflow_id)) == 1
    store.close()
    print("positive_assertion=PASS (one observed effect after crash and retry)")
    return 0


def negative_demo(path: Path) -> int:
    reset(path)
    workflow_id = "case-negative"
    store = Store(path)
    workers = SyntheticWorkers(store, enforce_idempotency=False)
    Orchestrator(store, workers).run(workflow_id)
    print_summary(store, workflow_id)
    duplicate = len(store.effects(workflow_id)) != 1
    store.close()
    print(
        "negative_assertion="
        + ("EXPOSED (blind retry created duplicate effects)" if duplicate else "NOT EXPOSED")
    )
    return 0 if duplicate else 1


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser()
    parser.add_argument("demo", choices=("positive", "negative", "both"))
    parser.add_argument(
        "--db",
        type=Path,
        default=Path("durable_demo.sqlite3"),
        help="SQLite path used for checkpoints, events and synthetic effects",
    )
    return parser.parse_args()


def main() -> int:
    args = parse_args()
    if args.demo == "positive":
        return positive_demo(args.db)
    if args.demo == "negative":
        return negative_demo(args.db)

    positive_code = positive_demo(args.db)
    negative_path = args.db.with_name(args.db.stem + "-negative" + args.db.suffix)
    negative_code = negative_demo(negative_path)
    return max(positive_code, negative_code)


if __name__ == "__main__":
    try:
        raise SystemExit(main())
    except KeyboardInterrupt:
        print("interrupted", file=sys.stderr)
        raise SystemExit(130)

simulated_crash=process stopped after ANALYSED checkpoint
workflow_state=COMPLETED
external_effect_count=1
positive_assertion=PASS (one observed effect after crash and retry)
workflow_state=COMPLETED
external_effect_count=2
negative_assertion=EXPOSED (blind retry created duplicate effects)

Durable state outlives code releases

A workflow waiting for a person, batch window or external document can remain open across several deployments. If new orchestration code produces a different command sequence when old history replays, recovery can fail even though the new code works for fresh cases. Temporal's versioning and replay-testing guidance makes this operational burden explicit.⁵

Version three things separately: the workflow definition, the durable state schema and each activity contract. A compatible release can add fields with defaults or change worker internals while preserving the same logical activity identity. An incompatible release needs a version marker, parallel worker path or migration that is tested against captured histories.

Canarying a durable workflow means more than sending new traffic to new code. It also means resuming old histories, rolling back with open cases, delivering late callbacks and proving that idempotency keys remain stable across releases.

Observe business progress, not just attempts

Infrastructure telemetry answers whether a worker started, consumed CPU or returned an exception. Workflow observability answers whether the case advanced, which business precondition failed, whether an effect is unknown and who owns the next action. Both are needed, but they should not be confused.

Keep an event for each attempt, then a separate accepted-result event. Keep a model proposal receipt, then a policy or human decision receipt. Keep a tool request, then an effect receipt and readback. This lets an operator see that three attempts produced one accepted activity result and one external effect.

The event history should avoid private payload sprawl. Store typed references, hashes, policy versions, effect identifiers and bounded summaries. Large evidence remains in its governed source. The workflow history is a reconstruction spine, not a second document lake.

Retention follows the longest credible replay or dispute window, not the shortest infrastructure log policy. Idempotency records may need to outlive the workflow if a late message can still arrive. Conversely, personal content should not be copied into every event merely to simplify debugging. A durable design therefore pairs long-lived identifiers and receipts with revocable, access-controlled references to sensitive evidence.

This separation makes deletion and correction possible. The workflow can retain that evidence item E17 supported a decision while the governed source applies its own retention or legal-hold rules. Where later reconstruction requires the original content, that obligation must be stated explicitly rather than assumed from a hash that proves integrity but cannot recreate meaning.

stateDiagram-v2
  [*] --> Claim
  state "What a retry can and cannot buy" as Claim
  state "Give one layer the retry budget" as Test
  state "The dangerous interval after a<br/>write" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

A production-shaped case

Consider a synthetic retail-bank fee dispute. The customer asks for a charge to be reviewed. The process must identify the account, gather transaction evidence, determine policy eligibility, decide whether human approval is required, issue a provisional credit when authorised and notify the customer only after the ledger change is observed.

The language model has a bounded role: classify the narrative, identify missing information and propose a disposition with cited evidence. It does not own case progress, approval validity or ledger semantics. Those remain in the workflow.

Walk the failure paths before the happy path

If the model endpoint returns 503, the workflow retries within a model-call budget and can route to a smaller fallback model only if that alternative has been evaluated for the same task. If the model returns a malformed proposal, schema repair may retry, but a policy contradiction should terminate the activity as a semantic failure rather than loop.

If the worker crashes after the proposal is checkpointed, the next worker reuses the accepted proposal. If a reviewer responds after expiry, the workflow records the late event but does not revive authority. If the ledger commits and the acknowledgement disappears, dependent notification remains blocked in CREDIT_OUTCOME_UNKNOWN until readback resolves it.

If a new software release arrives while the case waits for approval, the proposal hash and workflow version determine which path may resume. If compensation is later required, it receives a new operation identity and authority decision. The original credit is never deleted from history.

The workflow admission test

A dedicated durable orchestrator is justified when several of the following properties coincide. The test is intentionally mechanism-first. It does not prescribe a vendor.

A single safe read or cheap pure transformation can stay inside ordinary request handling. A one-step idempotent write may need only a queue, a transaction and a deduplication table. A multi-step case with human waits, policy versions and ambiguous effects needs a first-class process state even if every worker is conventional code.

The strongest competing baseline is therefore not “no reliability”. It is a simple database-backed state machine plus a queue. That baseline is often excellent. Choose a durable-execution platform when its replay, timers, visibility, versioning and operational tooling reduce more risk than the platform introduces. The mechanism is mandatory when the problem demands it; the product is optional.

The cost of remembering

Durability moves complexity rather than removing it. Event histories consume storage and can become slow to replay. Fine-grained checkpoints increase write volume. Stable idempotency records need retention rules long enough to cover late retries. Open workflows complicate schema retirement because yesterday's case may resume on tomorrow's runtime. Operators need tools to inspect, pause, repair and terminate individual cases without editing database rows by hand.

There is also a latency and cost trade. Recording every small step can dominate a fast path, while placing too much work inside one activity recreates the original recovery problem. The useful boundary is a business fact whose recomputation is costly, whose result affects later authority or whose external effect must not repeat. Batch low-consequence internal computations; checkpoint decisions, waits and side-effect boundaries.

Practitioner synthesis The operating model should budget for history growth, replay tests, stuck-case queues, compensation handling and idempotency-record retention. A team that funds only the happy-path worker has not funded durable execution, even if a managed service persists the state machine.

Release evidence for a durable workflow

Before release, kill every worker after each material transition and prove the case resumes. Inject an acknowledgement loss after an external commit and prove one effect remains. Deliver the same callback twice, then after expiry. Exhaust retries under overload and show that backoff does not create a retry storm. Resume old event histories on the proposed release. Fail the compensation and show that the case remains visibly owned.

These tests are more persuasive than a success-path demonstration because they exercise the causal claim. The workflow is durable only if process loss, duplicate delivery, unknown outcome and version change produce bounded, inspectable states rather than improvised operator archaeology.

Make progress the deterministic object

A stochastic worker can be useful without being dependable enough to own a long-running process. Its job is to perform one bounded piece of work and return a result that the workflow may accept, reject or escalate. The workflow's job is different: preserve the case identity, record what has become true, schedule the next attempt and prevent repeated execution from becoming repeated consequence.

The architecture decision changes when reliability is framed as process semantics rather than model quality. Do not begin by asking which agent framework should keep a loop alive. Begin with the external effect, its operation identity, its postcondition and the state required when its outcome is unknown. Then add deadlines, bounded retries, checkpoints and compensation around that contract.

A better model may reduce the number of failed activities. It cannot eliminate network partitions, process death, late approvals, duplicate delivery or code releases that overlap open cases. Those are properties of the system around the model. The defensible design is therefore modest and strong: let stochastic workers propose and compute, while a durable, inspectable workflow remembers what the business process has actually earned the right to do next.