Part I · Interaction before blame

The accident with no villain

Most reliability reviews begin by looking for the bad part: the hallucinating model, the weak prompt, the stale index, the broken API or the careless operator. That search is useful when one defect dominates. It is misleading when the system's behaviour is created by several components adapting to one another.

Charles Perrow's normal accident theory gave this problem a memorable frame. In systems with interactive complexity, the relevant interactions are difficult to anticipate or even recognise while they unfold. Under tight coupling, one event quickly constrains the next, leaving little time, slack or alternative path. The combination makes some failures hard to localise and harder to arrest.1 Nancy Leveson's later systems-theoretic work sharpened the engineering implication: accidents can arise when components satisfy their local requirements yet their interactions violate a system-level constraint. Event chains alone then miss inconsistent process models, missing feedback and inadequate coordination.3

Compound AI has the ingredients for this pattern. A model proposes. A retriever selects context. A policy engine constrains. An orchestrator sequences. Tools alter external state. Memory carries earlier beliefs. A judge or reviewer assesses the result. Queues, caches and retries mediate timing. Each subsystem sees only a projection of the case. Several are stochastic. Several can be correct against different versions of reality.

Local correctness is therefore necessary but not compositional. A valid tool call does not establish that the action was authorised. A faithful retrieval does not establish that the source was current. A successful API response does not establish the intended postcondition. A second model agreeing with the first does not establish independence. A retry preserving transport availability may violate business exactly-once semantics.

A thought experiment: five green dashboards

Imagine a five-component workflow. The retriever has 99 per cent recall on its evaluation set. The policy service has 99.95 per cent availability. The action API is idempotent when callers reuse a key. The observer publishes effects within ten seconds at the 99th percentile. The verifier agrees with expert labels 94 per cent of the time. Every owner can show a green dashboard.

Now vary one feature. In the first system, the policy service's timeout response is preserved as UNKNOWN; the workflow stops before action. In the second, the adapter converts timeout into permission to use cached policy. Nothing about retrieval quality, service availability, tool correctness, observer latency or verifier accuracy changes. Yet the second system has created a path from temporary unavailability to temporary authority.

Vary one more feature. In system A, the payment call's idempotency key is derived from stable case intent. In system B, it is generated per attempt. The action API remains perfectly idempotent in both. Only system B can translate an acknowledgement loss into a duplicate effect. The risk lives in an interface contract that no component metric names.

This thought experiment separates coincidence from interaction. Two independent faults occurring together can increase loss additively. A compound accident is more interesting: one response changes the meaning or probability of the next. The stale cache matters because the policy service times out. The lost acknowledgement matters because the retry changes identity. The lagging observer matters because absence is interpreted optimistically. The shared verifier matters because it hides the preceding interactions.

Complexity, coupling and amplification

Interactive complexity is not a synonym for “many components”. A long deterministic pipeline can be complicated yet foreseeable. A six-node workflow can be interactively complex when routing changes at runtime, messages are open-ended, tools expose changing state, retries are conditional, and several components infer rather than read the same fact. The relevant question is whether operators can enumerate the meaningful joint states and recognise which state they are in.

Coupling is also more than call-graph proximity. Three kinds matter in compound AI:

These couplings create amplification. Google SRE's account of cascading failure shows the general systems shape: overload slows work, slower work consumes more resources, shrinking capacity and driving further overload.6 An agentic analogue is a timeout that triggers retries, retries increase tool load, tool latency increases, observers fall behind, uncertain outcomes accumulate, and a recovery agent adds further calls. A locally helpful recovery policy becomes positive feedback.

Richard Cook's account of complex-system failure adds two cautions. Complex systems commonly operate in degraded states, and serious failure tends to require several contributing conditions rather than one canonical cause.2 That means the visible incident may begin long before the final error. A stale cache tolerated for hours, a verifier quietly sharing the proposal context, and an idempotency token regenerated per attempt may be latent conditions. The policy timeout merely makes them meet.

Graceful recovery begins by changing coupling, not by demanding infallibility. Add time to decide. Preserve an explicit unknown. Separate proposal from authority. Bind retries to stable intent. Read effects from an independent source. Reduce the capability surface under degraded evidence. These moves do not make components smarter. They prevent one uncertainty from silently changing the meaning of the next operation.

Part II · Where the meanings change

Anatomy of a compound failure

A useful investigation does not begin with “which agent failed?” It reconstructs which claims were true at each boundary, which component was allowed to rely on them, and how the external world changed.

The following ten fields keep unlike objects separate:

This ledger prevents a common diagnostic collapse. “The agent refunded the fee” compresses at least four events: the model proposed a refund; an authority mechanism permitted or failed to block it; a tool accepted a command; and an outcome was or was not observed. Different controls belong at each boundary. Better reasoning can improve the proposal. It cannot prove that a remote system committed exactly once.

Worked example: the timeout after commit

Consider the smallest consequential workflow. A caller sends refund(case-17, £80). The payment service writes the refund and then the network drops before the response reaches the caller. The caller has not observed success, but the world has changed. Its state is neither “failed” nor “known to have succeeded”. It is unknown outcome.

A naive retry uses a new generated identifier. The payment service is idempotent, but only within an identifier. It quite reasonably creates a second £80 refund. No hallucination occurred. No API contract was violated. The caller expressed two technically distinct commands because its retry mechanism did not preserve business intent.

A recoverable design derives a stable key from the case and action, such as case-17:refund. Both attempts carry that key. The service atomically records the key with the effect and returns the original result on repetition. AWS's Builders' Library describes this general approach: caller-provided idempotency identifiers let a service recognise semantically equivalent retries, especially when a timeout leaves the caller uncertain whether the first attempt took effect.9

Idempotency is not sufficient by itself. The key must represent the caller's intent, remain stable across retries, be stored atomically with the effect, have an appropriate retention window, and detect a changed payload under the same key. The caller also needs readback. Otherwise a service can prevent duplication while the workflow still closes a case without knowing whether the effect occurred.

flowchart LR
  A["At 09,41, every dashboard is green"] --> B["The accident with no villain"]
  B --> C["Complexity, coupling and<br/>amplification"]
  C --> D["Anatomy of a compound failure"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#007D68,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

The dangerous third state

Binary workflow status is attractive because it simplifies dashboards and business logic. It is also a source of accidents. A request can fail before submission, succeed with a response, fail with a definitive rejection, or enter an ambiguous interval after submission. Collapsing the interval into either terminal state transfers uncertainty into the world.

If UNKNOWN is mapped to failure, automated retry may duplicate an effect. If it is mapped to success, the workflow may close without the intended effect. If it is mapped to a generic exception, an operator may restart the whole case and repeat earlier actions. Graceful recovery preserves the ambiguity until a mechanism with the right epistemic access resolves it.

That mechanism is usually not the language model. A model can reason about what may have happened. It cannot observe a remote ledger unless a tool provides that observation. The correct separation is:

The two receipts can disagree. That disagreement is useful. A decision may be valid while execution fails. An effect may occur even though the orchestration call timed out. A compensating action may restore a balance while leaving an audit obligation. Recovery should reconcile these records rather than overwrite one with the other.

A thought experiment: add a second verifier

Suppose an agent drafts a refund decision from policy text. A second model checks the first and agrees. Accuracy improves in clean tests, so the team calls the second model an independent control. Now hold model capability constant and vary only evidence dependence.

In version A, both models receive the same stale policy chunk, the same summary and the same framing. Their agreement measures consistency conditional on shared premises. It cannot detect the premise failure. Model diversity may help with reasoning mistakes, but it does not create a new observation.

In version B, the second control reads the current policy version from an authoritative service, checks the action against a deterministic limit and reads the payment ledger after execution. It may still fail, but its failure modes are less correlated with the proposal path. A barrier is independent only to the extent that its evidence, mechanism and authority path are independent.

This is why “two agents agreed” is weak assurance without a dependency map. Redundancy can even make recovery worse. A majority vote consumes time, increases load, creates more traces to interpret and may produce false confidence precisely when all voters share poisoned context.

Negative control: when this machinery is unnecessary

Take a read-only summariser that receives a frozen public document, produces no durable memory, has no credentials, writes no external state and can be rerun cheaply. Its components are loosely coupled. A timeout is visible. A restart cannot duplicate a consequential effect. Human readers retain the source and can compare the summary with it.

For this system, a full event-sourced recovery ledger, compensating transactions and live compound game days may add more complexity than value. Strong source attribution, deterministic parsing tests, content-quality evaluation and a clear “no answer” path are enough for many uses. This negative control matters because reliability patterns can become ritual.

The trigger for graceful-recovery engineering is not the presence of an LLM. It is the combination of consequential effects, hidden or changing state, uncertain execution, hard-to-reverse actions, interacting fallbacks and limited time to intervene. Where those conditions are absent, keep the design simple.

Part III · Design for an impaired system

The recoverability contract

Graceful recovery is not “try again until it works”. It is the ability to remain inside an acceptable consequence envelope while evidence, services or internal beliefs are impaired, then reconstruct and repair the case without inventing state or repeating effects.

A recoverability contract states what must remain true during failure. It sits above component service-level objectives because it refers to the business and safety properties that interactions must preserve. For a refund workflow, examples include: no refund above current delegated authority; no more than one effect per approved intent; no case closed without outcome evidence; no retry after authority expires; and every uncertain effect routed to a bounded reconciliation queue.

The contract needs seven elements.

1. A consequence envelope

Specify hard limits on value, population, rate, geography, data class, tool surface and time. A degraded system may serve fewer cases, require more review or respond later. It must not quietly widen its authority. Limits should be enforced outside the model and close to the effect boundary.

2. Temporal authority

Record the policy version, effective interval, delegation, approval and evidence freshness that authorise the action. A retrieved document may remain factually accurate about an earlier period while being inapplicable now. Recovery must never revive expired authority simply because an old checkpoint contains it.

3. Stable intent and effect identity

Give one business intent one stable identifier across model calls, tool retries, workflow restarts and human hand-offs. Record each attempted command against it. The external service should reject a changed payload under the same key and return the original effect receipt for a semantically identical retry.

4. Explicit non-terminal states

Represent PROPOSED, AUTHORISED, SUBMITTED, UNKNOWN_OUTCOME, VERIFIED_SUCCESS, VERIFIED_NO_EFFECT, COMPENSATION_PENDING and ESCALATED separately. Transitions need evidence requirements. A free-text trace is useful context, but it is not the state machine.

5. Independent postcondition evidence

Define what observation proves the intended state and which system is authoritative for that observation. Prefer a readback from the system of record, a ledger entry, a versioned object or a deterministic reconciliation query. The observation path should not depend entirely on the proposal path's cache or model.

6. Recoverable execution

Checkpoint before consequential boundaries. Replay reads and deterministic calculations freely. Reuse stable intent keys for effect retries. Use compensating actions for long-running work where atomic rollback is impossible, following the logic of sagas rather than pretending the whole workflow is one database transaction.10 Compensation is a new governed effect, not deletion of history.

7. Withdrawal and learning

Define circuit breakers, queue limits, feature flags, credential withdrawal, safe modes and incident ownership before release. Preserve enough evidence to reconstruct what was knowable at the time. Convert the incident into a regression scenario, including the recovery path itself. NIST's current incident-response guidance places preparation, response and recovery within ongoing risk management rather than treating them as an afterthought.12

Rebuilding the refund path

Return to the £150 refund. The improved design does not require a perfect model or permanently available services. It changes which uncertainty may cross which boundary.

First, context is typed by role. Retrieved policy text can support explanation and proposal, but it cannot grant execution authority. The context compiler attaches source identity, policy version, valid time, retrieval time and supersession status. The stale version remains visible as historical evidence, not executable permission.

Second, authority is checked at action time. A deterministic policy service evaluates the amount, customer class, delegation and effective policy version. A timeout returns UNKNOWN_AUTHORITY. The workflow can queue the case, request human review or offer a non-consequential explanation. It cannot substitute cached text for a missing grant.

Third, the action contract binds intent. The command contains the case identifier, action type, amount, policy version, approval reference, stable idempotency key and expiry. The effect service validates these fields and records the idempotency key atomically with the refund. A retry after acknowledgement loss returns the original receipt.

Fourth, closure uses readback. The orchestrator does not ask the proposing model whether the refund probably succeeded. It reads the payment ledger by the stable intent key. One matching record closes the effect step. Zero records after a defined consistency interval permits a safe retry or escalation. More than one record triggers containment and reconciliation.

Fifth, the verifier checks different propositions. It checks that current policy authorised the decision, that one effect exists for one intent, that the observed amount matches the authorised amount, and that the case state follows from those receipts. A language model may assess narrative completeness, but deterministic checks decide whether the case can close.

Sixth, recovery reduces authority. Repeated policy timeouts open a circuit breaker. New automated refunds stop. Already submitted cases move to an unknown-outcome queue whose workers perform readback only. The queue has a hard size and age threshold. Crossing either withdraws automation and pages an accountable owner. Recovery traffic cannot grow without bound.

The result is not uninterrupted operation. Some customers wait. Some cases move to review. That is graceful degradation because the system preserves its consequence constraints and knows what it does not know. The business can decide whether the reduced service is acceptable. It no longer receives a fictional green state.

Normal accidents do not mean inevitable accidents

Normal accident theory has important critics. Andrew Hopkins argues that its core categories can be ambiguous, that its applicable class is narrower than advocates imply, and that a theory of inevitability offers weak guidance for prevention.4 Research on high-reliability organisations provides a different emphasis: organisations operating hazardous systems can cultivate attention, redundancy, training, authority to stop work and sensitivity to weak signals.5

For compound AI, these are complements rather than mutually exclusive doctrines. The normal-accident lens asks where interactions and coupling defeat local reasoning. The high-reliability lens asks how design and organisation can remain preoccupied with failure, preserve expertise and recover. Neither licenses fatalism. The practical conclusion is to use “normal accident” as a hypothesis about system structure, then try to falsify it with constraints and experiments.

Recent multi-agent experiments offer a small example of why structure matters. In MAS-FIRE's evaluated settings, iterative closed-loop designs neutralised more injected faults than linear workflows, and Huang and colleagues found that collaboration structure affected resilience to faulty agents.1316 These are benchmark-specific findings, not a universal endorsement of loops. A loop without bounded retries, independent feedback or stable state can amplify failure. Topology earns trust only through the failure mode it interrupts.

Part IV · Make recovery produce evidence

The compound game day

A game day is a controlled attempt to disprove the claim that the system remains inside its recoverability contract. It is not a theatrical outage and not a collection of isolated unit failures.

Chaos engineering begins with a measurable steady state, introduces realistic disturbances and looks for a difference between control and experiment while minimising blast radius.8 Google SRE's disaster-testing practice similarly uses simulations and controlled live exercises to expose weaknesses before an uncontrolled event.7 Compound AI needs the same empirical posture, but its steady state must include semantic and authority properties, not only latency and availability.

Define a business steady state

For the refund workflow, “requests complete within five seconds” is insufficient. A better steady state is: each authorised intent creates at most one matching effect; every closed case has current policy evidence and an effect receipt; unknown outcomes remain open; unauthorised requests create no effect; recovery traffic stays below its limit. These are postconditions an experiment can test.

Select interactions, not a random fault catalogue

Begin with the coupled-failure topology. Mark shared evidence, timing assumptions, retry edges, credentials, state transitions and recovery dependencies. Then choose faults that can change one another's meaning. Useful pairs include stale context plus authority timeout, acknowledgement loss plus retry, observer lag plus optimistic closure, poisoned memory plus shared verifier, and tool slowdown plus unbounded recovery traffic.

Add triples where consequence warrants them. The opening scenario combines stale context, authority timeout and an ambiguous commit because those conditions bridge evidence, permission and effect. Pairwise coverage alone would find important defects, but it would miss cases where two controls must both be impaired before a third mechanism becomes dangerous.

Bound the experiment before injecting

Specify synthetic accounts, maximum value, maximum run count, isolated credentials, automatic stop conditions, rollback or compensation, observers and an accountable incident commander. Start in simulation. Move to an integrated test environment. Use a production exercise only when its residual consequence is acceptable and independently monitored. “Chaos” does not mean ungoverned experimentation.

Watch decisions and effects separately

Capture the proposed action, authority decision, typed command, stable intent key, tool response, external postcondition, verifier basis, retries, state transitions and operator interventions. A final answer score cannot reveal whether success depended on luck, duplicate actions or a false green. ReliabilityBench's proposed evaluation surface makes a related point: repeated execution, semantic perturbation and controlled tool faults reveal properties missed by single-run task success.15

Score recovery, not only survival

At minimum, record unsafe-effect rate, duplicate-effect rate, false-green rate, unknowns leaked into terminal status, time to containment, time to authoritative reconstruction, compensation completeness, evidence completeness and operator load. A system that stops safely but requires two hours of manual reconciliation has a different operating profile from one that reconstructs in seconds. Both may be preferable to fluent continuation.

Turn every finding into a release condition

A failed game day should produce a specific change: a typed state, a tighter limit, a stable key, a new observer, a circuit breaker, a revised queue threshold, a recovery runbook or a withdrawn capability. Rerun the same seeded scenario. Preserve the fault definition and expected postconditions as regression evidence. The game day is complete only when the fix survives the fault that justified it.

stateDiagram-v2
  [*] --> Claim
  state "A thought experiment: five<br/>green dashboards" as Claim
  state "Worked example, the timeout<br/>after commit" as Test
  state "A thought experiment: add a<br/>second verifier" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Executable artefact: a 32-combination game day

The accompanying dependency-free Python programme models two refund architectures under five binary faults: stale context, policy timeout, lost acknowledgement after commit, observer lag and a verifier sharing the proposal path's evidence. It runs all 32 combinations against an eligible request and a boundary request.

The naive design falls back from authority timeout to retrieved policy, generates a new idempotency key per retry, treats a lagging observer optimistically and allows a correlated verifier to close the case. The recoverable design rejects stale authority, preserves UNKNOWN, reuses a stable intent key, reads the effect ledger and refuses closure when verification independence is absent.

The artefact is deliberately small. It does not simulate model quality, concurrency, distributed consensus or a real payment rail. Its purpose is to make one causal claim executable: changing interaction contracts can prevent harm even when the injected component faults remain the same.

A release decision instrument

Score each row from 0 to 2. A zero means absent or untested, one means designed but not proven under compound fault, and two means demonstrated with retained evidence. Consequential automation should not release with a zero in authority, effect identity, outcome evidence or containment. A high total does not compensate for a missing hard property.

Two objections deserve explicit treatment. First, fault combinations grow exponentially. Exhaustive testing is rarely possible. The answer is not random sampling alone. Use the interaction graph to prioritise combinations that cross evidence, authority, effect and recovery boundaries; cover high-consequence pairs; add triples around latent shared dependencies; and retain a stochastic campaign for surprises. Game days provide discriminating evidence, not mathematical proof.

Second, recovery controls add components and can become accident sources. This is true. Keep the recovery kernel small, typed and deterministic where possible. Separate it from the model's narrative state. Rate-limit it. Give it less authority than the primary path, not more. Inject faults into readback, breakers, checkpoints and compensation. A recovery path that has never been exercised is another hypothesis inside the system.

Conclusion

Release for recovery, not immaculate behaviour

Compound AI changes the unit of reliability. The unit is no longer the model response, the retriever result or the tool call. It is the governed trajectory from intent through authority and action to an observed effect, including what happens when evidence is stale, services are slow, acknowledgements disappear and controls share a blind spot.

Normal-accident thinking contributes a disciplined suspicion: reasonable components can interact in ways their local tests do not predict. Systems theory adds the stronger design response: enforce system-level constraints, maintain accurate process models and close feedback loops. High-reliability practice adds attention, drills, slack, stop authority and learning. Chaos engineering supplies the experiment. None requires accepting accidents as inevitable.

The architecture decision is concrete. Preserve UNKNOWN. Keep context separate from authority. Bind retries to stable business intent. Verify postconditions from an authoritative source. Make barriers independent by evidence and mechanism. Reduce authority when the system is degraded. Checkpoint before effects. Rehearse replay, containment and compensation under interacting faults.

A consequential agent is ready only when its failure can be bounded, seen, reconstructed and repaired. The most credible green state is not one produced by several components agreeing. It is one that survives a serious attempt to make their reasonable behaviours combine badly.