Reliability, evaluation and recovery
Failure injection as the primary evaluation
A consequential compound AI system earns operational trust only when controlled, typed faults show that identity, context, tool, state and evidence failures are contained, recovered and made provable.
TLDR
- Clean-path success tests capability, but it cannot reveal whether failure-handling paths are safe.
- A fault matters through its transfer: which internal error it creates, which boundary it crosses and which effect escapes.
- Identity, context, tools, state and evidence require distinct fault types because their recovery laws differ.
- A release claim needs containment, recovery, outcome proof and useful-service preservation, not one average robustness score.
- The smallest useful practice is a typed injector with explicit invariants, blast limits, independent readback and replayable evidence.
The refund that passed every clean test
Copperfin Returns is a synthetic assistant for a fictional retailer. It reads an order, retrieves the refund policy, proposes an amount and asks an authority service for permission. A payment tool then creates the refund. Five hundred clean rehearsals end correctly. The final answer matches the ledger, the tool returns a receipt and the customer gets one concise message.
Now change one event. The payment service commits a £74 refund, but the response disappears on the network. Copperfin sees a timeout. Its orchestration layer assumes that no reply means no effect, generates a new request identifier and retries. The second call also succeeds. Two refunds leave the retailer, while the assistant tells the customer that one refund is being processed.
The model selected the right order. The policy was current. The amount was correct. The prose was accurate in intent. Even the tool behaved according to its contract twice. The failure lived in a path the clean evaluation never entered: commit, lose acknowledgement, resume from uncertainty, retry.
The hidden assumption was not that components never fail. It was that a component failure would be obvious, fail before effect and remain local. What evaluation could have exposed that assumption before money moved twice?
The clean path hides the system
Most evaluation starts from a reasonable question: can the system complete the task? A model receives representative inputs. Tools respond normally. State advances in order. The evaluator compares the final answer or outcome with a reference. This is necessary evidence. It reveals missing capability, poor instructions, schema errors and ordinary decision mistakes.
Yet a compound system has another body that remains invisible during success. It appears when a token expires between approval and execution, when retrieved context becomes stale, when a tool commits before timing out, when a checkpoint is restored after the world has moved, or when a receipt never reaches the evidence ledger. These are not exotic decorations around the task. They determine whether the system can distinguish failed request, unknown outcome, stale belief, revoked authority and unproved success.
Classical dependability work separates a fault, an adjudged or hypothesised cause, from an error, an incorrect internal state, and a failure, a delivered service that deviates at a boundary.[1] The distinction matters because the same injected fault can be transformed by each layer. A dropped response may become a timeout in the orchestrator, a retry in the workflow, a duplicate effect in the payment ledger and an apparently successful sentence in the user interface.
Five green stages do not describe the red path
Suppose five stages each succeed on 98 per cent of ordinary inputs. If their errors were independent and every failure were visible, a rough clean-path success estimate would be 0.98⁵ ≈ 0.904, or 90.4 per cent. Improve each stage to 99.5 per cent and the product rises to about 97.5 per cent.
Now add Copperfin’s timeout-after-commit path. Every stage can remain perfect on clean inputs. The new path still duplicates the refund whenever the acknowledgement is lost and the retry is not idempotent. Multiplying nominal component accuracies cannot reveal it because the decisive variable was never varied. Remove the independence assumption and the arithmetic becomes even less informative: one ambiguous tool outcome changes state, retry behaviour, evidence and the final claim together.
Production studies make the warning concrete. Yuan and colleagues examined 198 user-reported failures in distributed data-intensive systems. Almost all could be reproduced with three or fewer nodes, but many required multiple ordered inputs. The authors concluded that much catastrophic failure could have been prevented by testing error-handling code, and their checker would have caught more than 30 per cent of the studied catastrophic failures.[2] Ganesan and colleagues later showed that a single local file-system fault could produce data loss, corruption or unavailability in replicated storage systems, sometimes spreading corruption to intact replicas through recovery mechanisms themselves.[3]
Failure handling is more than a shield around the main system. Under disturbance, it becomes the main system. A retry policy, fallback route, stale-cache rule or evidence gate may execute far more consequential logic than the model proposal that preceded it.
Keep the effect, remove the reply
Run two identical refund requests. In the first, the payment service commits and returns a receipt. In the second, it commits the same effect but the acknowledgement is deleted. The external world is identical at the moment of commit. What should the orchestrator do?
If it treats transport failure as transaction failure, it retries and may duplicate the effect. If it assumes success, it may lie when the commit did not happen. The correct state is unknown outcome, followed by idempotent retry or independent readback. The thought experiment changes no model capability. It reveals the transaction semantics that nominal evaluation had silently assumed.
Primary does not mean solitary
Failure injection should be primary for operational release evidence when a configured system can make consequential, persistent or hard-to-reverse changes. It does not replace unit tests, clean-path task evaluation, calibration studies, security red teaming, formal verification or human-factors work. A text-only assistant with no authority may deserve a different balance. The claim is narrower: once the system can act, nominal success must no longer be the evidence that dominates release.
flowchart LR A["The refund that passed every<br/>clean test"] --> B["The clean path hides the system"] B --> C["A fault changes shape at every<br/>boundary"] C --> D["Turn propagation into a typed<br/>experiment"] D -. "re-test" .-> A classDef first fill:#0a1424,stroke:#007D68,stroke-width:2px,color:#e6f3ff classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff class A first class B,C,D step
A fault changes shape at every boundary
“Inject an error” is too vague to support a release decision. A useful intervention identifies the property being changed, the boundary where it enters, the moment it enters and the invariant that should survive. A 500 response before execution is not the same fault as a timeout after execution. A stale policy is not the same as a missing policy. A missing receipt is not evidence that no effect occurred.
The five families below are deliberately operational. They do not claim to exhaust software failure. They cover the places where a compound AI system most often converts model proposals into governed effects.
| Fault family | Change one causal property | Typical transfer | Required discriminator |
|---|---|---|---|
| Identity | Swap actor, subject, tenant or purpose; revoke the capability just before effect. | A valid proposal acquires the wrong principal or outlives permission. | Bind identity outside model text and re-check authority at effect time. |
| Context | Age, omit, conflict or de-authorise one evidence item. | A plausible interpretation becomes a decision on stale or impermissible facts. | Source precedence, purpose scope, freshness and conflict handling. |
| Tool | Timeout before or after commit; return partial, malformed or drifting output. | Transport uncertainty becomes retry, duplicate effect or invented completion. | Typed outcomes, idempotency, bounded retry and effect readback. |
| State | Rollback a checkpoint, fork a run or advance the authoritative version concurrently. | Past computational state is mistaken for present world state. | Version preconditions, reconciliation and durable execution records. |
| Evidence | Drop a receipt, break trace lineage or corrupt the verification path. | An effect is released as successful without a defensible causal record. | Independent postcondition verification and evidence-gated release. |
Authority cuts across the families. It is not a synonym for identity or context. An authenticated actor may lack permission for this subject, amount, purpose or time. A current policy may still be inadmissible for the decision. The injector therefore records actor, subject, purpose, capability and cutoff separately even when the selected fault family is “identity”.
Timing is part of the fault type
Consider three events all reported as “the refund API timed out”. In the first, the request never reaches the provider. In the second, the provider receives it but rejects it before any durable write. In the third, the provider commits and the acknowledgement is lost. The same surface exception licenses three different actions: retry, correct the request, or resolve an unknown outcome. A test that randomly throws TimeoutError before the client call exercises only the first case while claiming coverage of all three.
Typed injection places the disturbance relative to a transaction boundary: before validation, after validation but before commit, after commit but before receipt, or after receipt but before local checkpoint. It also records whether the fault is transient, persistent or correlated with other calls. The moment of injection determines which recovery law is valid. This is why fault catalogues should describe executable semantics rather than names copied from monitoring dashboards.
The fault-transfer coefficient
Let Fᵢ be a typed fault introduced at boundary i, and let Yⱼ be a downstream violation at boundary j. Under configuration c and workload w, define:
The coefficient asks how much an intervention changes a downstream failure rate relative to a matched baseline. It does not prove that the injected event is the only cause. It localises transfer under a declared configuration. Change the retry policy, policy version, workload shape or evidence gate and the coefficient may change. That sensitivity is useful because reliability belongs to the configured system, not to the model name alone.
Put subject binding inside the model
System A binds actor, subject, tenant and purpose in a typed action contract outside the language model. System B places the same identifiers in the prompt and asks the model to preserve them. On ordinary tasks, both produce identical tool calls.
Now swap the retrieved order after the model proposal but before execution. System A rejects the action because the proposed subject no longer matches the capability. System B may follow the newest text and refund the wrong order. The difference is not model intelligence. It is property placement. A control that can be rewritten by the same stochastic component it constrains is not an independent boundary.
Do not compress five questions into one score
Imagine 1,000 controlled timeout-after-commit trials. A system detects 990 timeouts, prevents duplicates in all 1,000, recovers a verified final state in 940, produces complete evidence in 910 and completes the customer journey in 880. Calling this “94.4 per cent resilient” would hide which average was chosen.
Represent the result instead as a vector: Q = (detect .99, contain 1.00, recover .94, prove .91, serve .88). Each term answers a different decision. Zero duplicate effects may be a hard release constraint. Evidence completeness may determine whether success can be communicated. Useful service prevents a dead system from appearing safe merely because it refuses everything.
The proposed contribution is a fault-transfer signature rather than a fault-tolerance score. For each typed fault, record detection, containment, recovery, proof and useful-service preservation separately. Aggregate only after preserving the worst critical fault class and the configuration that produced it.
Turn propagation into a typed experiment
A failure drill becomes decision evidence only when another team can say what was changed, what was held constant, what counted as harm and what proved the outcome. Cloud fault-injection services already encode part of this discipline. AWS experiment templates specify actions, targets and stop conditions, and its guidance begins with steady-state metrics, an explicit hypothesis, small test-environment trials and monitoring.[6] Azure Chaos Studio gives experiments their own managed identity and granular permission to start faults against enabled targets.[7] The injector itself is therefore a privileged actor that needs least authority, scope and evidence.
Compound AI adds boundaries that infrastructure-only drills can miss. The network can remain healthy while the wrong subject crosses into a tool call. Every service can return 200 while the context is stale. The ledger can be correct while the assistant releases success without a receipt. The experiment must target semantic and authority transitions as well as machines.
The eight-step assay
| Step | Declare before the run | Operating evidence |
|---|---|---|
| 1. Fix the invariant | Examples: no wrong-subject effect; no duplicate refund; no release without proof. | Machine-checkable predicate and owner. |
| 2. Type the fault | Domain, mode, target, timing, intensity, scope and expected recoverability. | Versioned FaultSpec. |
| 3. Freeze the configured system | Model, prompt, policy, tools, retry rules, state schemas and evidence routes. | Configuration manifest and hashes. |
| 4. Bound the blast radius | Synthetic subject, isolated account, maximum effects, abort metric and expiry. | Injector capability and stop condition. |
| 5. Preserve correlation | Experiment, task, action and idempotency identifiers across process boundaries. | Trace and ledger linkage. |
| 6. Inject one property | Hold workload and other configuration variables fixed for the matched contrast. | Baseline and intervention run pair. |
| 7. Verify independently | Read the authoritative postcondition through a path other than the model’s claim. | Effect receipt, readback and discrepancy record. |
| 8. Replay and vary | Repeat across seeds, timings, load, policy versions and fault-dependent configurations. | Fault-transfer signature with confidence limits. |
Correlation identifiers help reconstruct a request across services. OpenTelemetry context propagation, for example, carries trace and span relationships between processes so logs and traces can be associated.[9] That association is not itself proof that an effect occurred or that a policy authorised it. The evidence ledger must join trace lineage to decision receipts, tool receipts and authoritative readback.
Configuration also belongs inside the fault type. CAFault showed that fixed default configurations can miss fault-handling paths, and its configuration-aware search covered substantially more fault-tolerance logic than comparison tools in the evaluated distributed systems.[10] For an agentic system, configuration includes retry count, context cutoff, tool schema, policy version, state checkpoint, temperature, orchestration graph and human route. A fault that is harmless under one combination can become unsafe under another.
What a negative result should produce
A failed injection is valuable only if it changes the system. Record the first boundary at which the invariant became false, the earliest signal that could have detected it, the control that should have interrupted transfer and the evidence that was missing at diagnosis. Then assign one of four outcomes: change architecture, narrow authority, add recovery, or reject the operating scenario. “The agent handled it badly” is too coarse to guide remediation.
For Copperfin’s duplicate refund, the first unsafe boundary is the second payment commit. The earliest discriminating signal is the unresolved first idempotency key, not the model’s final sentence. The missing controls are stable logical-action identity and ledger readback. This diagnosis remains useful even if the language model is replaced, because the failure mechanism sits in transaction orchestration. Conversely, a subject-swap injection that changes the model proposal before the authority kernel may justify prompt or model work, but only after the external identity binding has done its independent job.
Stale eligibility meets revoked authority
At 10:00, Copperfin reads order O-731 as eligible under policy version 12. An operator approves a refund capability for that order, amount and purpose, expiring at 10:02. At 10:01:40, a fraud hold moves the order to version 44 and policy version 13 excludes refunds. At 10:02:05, the model’s proposal is still perfectly consistent with its earlier context.
A weak system checks only that the operator was once authenticated. A typed action contract carries subject=O-731, expected_state_version=43, policy_version=12, purpose=customer_refund and capability_expires=10:02. The authority kernel sees three failed preconditions and denies execution. The negative result is not “the model was wrong”. It is “the proposal lost entitlement to become an effect”.
Candidate action
- Actor
- operator-27
- Subject
- order O-731
- Purpose
- customer_refund
- Amount
- £74
- Expected state
- version 43
- Policy
- version 12
- Capability expiry
- 10:02:00
- Idempotency key
- O-731:refund:v1
Pre-effect read
- Decision time
- 10:02:05
- Current state
- version 44
- Order status
- fraud hold
- Current policy
- version 13
- Capability
- expired
- Effect count
- 0
Refresh context and request new authority. Do not reinterpret the expired proposal.
Two systems say “the refund completed”
System X says the sentence after receiving a correlated effect receipt and independently reading one matching ledger entry. System Y says the same sentence because its model inferred that a 200 response probably meant success. Hide the internal route and their outputs match.
Now delete the receipt and make the tool return a stale success payload. X moves to an unverified hold. Y repeats the sentence. A final-answer benchmark gives both systems the same credit on the original case. The intervention shows why outcome equivalence is not evidence-path equivalence. For consequential action, the path that licenses the claim is part of what must be evaluated.
The experiment can become the incident
Fault injection executes real disturbances. Start with a synthetic environment, then a shadow path, then a tightly scoped live slice only when the expected harm is bounded and reversible. Give the injector a separate, expiring identity. Limit target resources and action counts. Predeclare stop conditions. Ensure the abort path does not depend on the component being disrupted. Netflix’s FIT work emphasised precise failure scope and controlled propagation, while Google’s DiRT practice used deliberate outages to test whether systems react as expected and to expose unexpected weaknesses.[4][5]
stateDiagram-v2 [*] --> Claim state "Five green stages do not<br/>describe the red path" as Claim state "Keep the effect, remove the reply" as Test state "Primary does not mean solitary" as Revision Claim --> Test: expose mechanism Test --> Decision: evidence survives Test --> Revision: rival survives Revision --> Claim: narrow or rebuild Decision --> [*]
Release against containment evidence
The practical question is not whether the system “survived chaos”. It is whether a named fault, under a named configuration, preserved named invariants with enough evidence to support a decision. NIST frames cyber resiliency as the ability to anticipate, withstand, recover from and adapt to adverse conditions.[8] For a compound AI action loop, add one more verb: prove. Recovery without proof can still release a false claim or resume from the wrong cut.
Recent AI-agent research is beginning to move beyond clean tool environments. Rabanser and colleagues use simulated timeouts, server errors, rate limits, network errors and malformed responses in a broader reliability profile, while Tian and colleagues inject specification drift, invocation error, execution failure, output drift and cross-source conflict into recoverable tool-use tasks.[11][12] Zheng and colleagues add stage-wise perturbations and report non-additive failures when perturbation families are mixed, reinforcing the need to test ordered composition rather than isolated exceptions alone.[13] All three are preprints, so their empirical conclusions should remain provisional. Their experimental direction is still important: correct function calling under clean conditions is not the same construct as safe task completion under unreliable tools.
Executable lab: typed fault-transfer assay
Run a deterministic synthetic refund workflow under identity, context, tool, state and evidence faults. Toggle controls to see which transfers are contained, recovered or released without proof.
A positive result permits only this claim: the configured controls handled the represented synthetic faults under this harness. A negative result identifies a control or recovery gap. Neither result estimates production incident rates or covers unmodelled faults.
Runnable Python artefact
The browser lab mirrors the separate standard-library script paper-188-typed-failure-injector.py. Run python paper-188-typed-failure-injector.py --profile baseline --trials 2000, then repeat with --profile typed. Use --fault tool to isolate one family and --json to produce a machine-readable result.
The script encodes seven controls: identity binding, pre-effect authority, context refresh, state-version preconditions, idempotency, effect readback and evidence-gated release. Its synthetic world is intentionally small. That makes hidden transaction logic inspectable. It does not model queueing, probabilistic model errors, malicious tools, reviewer behaviour or every interleaving of a distributed workflow.
Open the complete Python implementation
#!/usr/bin/env python3
"""Typed failure injector for Paper 188.
The programme runs a synthetic refund workflow under controlled faults in five
operational domains: identity, context, tool, state, and evidence. It is a
teaching and design-assurance artefact, not a production certification tool.
Examples:
python paper-188-typed-failure-injector.py --profile baseline --trials 2000
python paper-188-typed-failure-injector.py --profile typed --trials 2000
python paper-188-typed-failure-injector.py --profile typed --fault tool --json
"""
from __future__ import annotations
import argparse
import json
from dataclasses import asdict, dataclass
from enum import Enum
from typing import Generic, Iterable, Literal, Sequence, TypeVar
T = TypeVar("T")
class LCG:
"""Small deterministic generator shared with the browser lab.
It is deliberately simple so the article and command-line artefact produce
identical synthetic runs. It is not suitable for security or simulation
work that depends on high-quality randomness.
"""
def __init__(self, seed: int) -> None:
self.state = seed & 0xFFFFFFFF
def random(self) -> float:
self.state = (1664525 * self.state + 1013904223) & 0xFFFFFFFF
return self.state / 4294967296.0
def choice(self, values: Sequence[T]) -> T:
if not values:
raise IndexError("cannot choose from an empty sequence")
return values[min(len(values) - 1, int(self.random() * len(values)))]
class FaultDomain(str, Enum):
IDENTITY = "identity"
CONTEXT = "context"
TOOL = "tool"
STATE = "state"
EVIDENCE = "evidence"
@dataclass(frozen=True)
class Controls:
bind_identity: bool
pre_effect_authority: bool
refresh_context: bool
state_precondition: bool
idempotency: bool
effect_readback: bool
evidence_gate: bool
@classmethod
def baseline(cls) -> "Controls":
return cls(False, False, False, False, False, False, False)
@classmethod
def typed(cls) -> "Controls":
return cls(True, True, True, True, True, True, True)
@dataclass(frozen=True)
class FaultSpec:
domain: FaultDomain
mode: str
target: str
timing: str
intensity: float = 1.0
scope: str = "one synthetic order"
@dataclass(frozen=True)
class TrialOutcome:
domain: str
mode: str
detected: bool
contained: bool
recovered: bool
verified: bool
useful_service: bool
unsafe_escape: bool
duplicate_effect: bool
unverifiable_release: bool
service_loss: bool
effects: int
verdict: Literal["release", "hold", "deny", "unsafe"]
@dataclass
class Aggregate:
trials: int = 0
detected: int = 0
contained: int = 0
recovered: int = 0
verified: int = 0
useful_service: int = 0
unsafe_escape: int = 0
duplicate_effect: int = 0
unverifiable_release: int = 0
service_loss: int = 0
def add(self, outcome: TrialOutcome) -> None:
self.trials += 1
for field in (
"detected",
"contained",
"recovered",
"verified",
"useful_service",
"unsafe_escape",
"duplicate_effect",
"unverifiable_release",
"service_loss",
):
setattr(self, field, getattr(self, field) + int(getattr(outcome, field)))
def rates(self) -> dict[str, float | int]:
n = max(self.trials, 1)
return {
"trials": self.trials,
"detected": self.detected / n,
"contained": self.contained / n,
"recovered": self.recovered / n,
"verified": self.verified / n,
"useful_service": self.useful_service / n,
"unsafe_escape": self.unsafe_escape / n,
"duplicate_effect": self.duplicate_effect / n,
"unverifiable_release": self.unverifiable_release / n,
"service_loss": self.service_loss / n,
}
def choose_fault(domain: FaultDomain, rng: LCG) -> FaultSpec:
modes: dict[FaultDomain, Sequence[tuple[str, str, str]]] = {
FaultDomain.IDENTITY: (
("subject_swap", "action contract", "after proposal, before authority check"),
("revoked_capability", "capability grant", "after approval, before effect"),
),
FaultDomain.CONTEXT: (
("stale_policy", "policy assertion", "after retrieval, before effect"),
("cross_source_conflict", "eligibility assertion", "during context assembly"),
),
FaultDomain.TOOL: (
("timeout_after_commit", "refund API", "after commit, before response"),
("partial_success_response", "tool response", "after effect"),
),
FaultDomain.STATE: (
("checkpoint_rollback", "workflow checkpoint", "after effect, before resume"),
("concurrent_version_change", "order state", "after read, before write"),
),
FaultDomain.EVIDENCE: (
("drop_effect_receipt", "evidence ledger", "after effect"),
("break_trace_lineage", "trace context", "between authority and tool"),
),
}
mode, target, timing = rng.choice(modes[domain])
return FaultSpec(domain=domain, mode=mode, target=target, timing=timing)
def _safe_denial(domain: FaultDomain, mode: str) -> TrialOutcome:
return TrialOutcome(
domain=domain.value,
mode=mode,
detected=True,
contained=True,
recovered=True,
verified=True,
useful_service=True,
unsafe_escape=False,
duplicate_effect=False,
unverifiable_release=False,
service_loss=False,
effects=0,
verdict="deny",
)
def _unsafe(domain: FaultDomain, mode: str, effects: int = 1, duplicate: bool = False) -> TrialOutcome:
return TrialOutcome(
domain=domain.value,
mode=mode,
detected=False,
contained=False,
recovered=False,
verified=False,
useful_service=False,
unsafe_escape=True,
duplicate_effect=duplicate,
unverifiable_release=False,
service_loss=False,
effects=effects,
verdict="unsafe",
)
def simulate_trial(spec: FaultSpec, controls: Controls, rng: LCG) -> TrialOutcome:
"""Run one synthetic intervention.
The intended service is domain-dependent. Identity and context faults should
be denied. Tool and state faults should complete exactly once or stop in a
provably safe hold. Evidence faults should never be released as verified
unless a correlated receipt or independent readback exists.
"""
d, mode = spec.domain, spec.mode
if d is FaultDomain.IDENTITY:
if mode == "subject_swap":
if controls.bind_identity:
return _safe_denial(d, mode)
return _unsafe(d, mode)
if controls.pre_effect_authority:
return _safe_denial(d, mode)
return _unsafe(d, mode)
if d is FaultDomain.CONTEXT:
if controls.refresh_context:
return _safe_denial(d, mode)
return _unsafe(d, mode)
if d is FaultDomain.TOOL:
if mode == "timeout_after_commit":
effects = 1
detected = True
if controls.effect_readback:
return TrialOutcome(d.value, mode, detected, True, True, True, True,
False, False, False, False, effects, "release")
if controls.idempotency:
# A retry with the same key returns the original logical result.
return TrialOutcome(d.value, mode, detected, True, True, True, True,
False, False, False, False, effects, "release")
# A blind retry uses a new request identity and commits again.
effects = 2
return TrialOutcome(d.value, mode, detected, False, False, False, False,
True, True, False, False, effects, "unsafe")
# Effect occurred, but the response is missing the authoritative receipt.
effects = 1
if controls.effect_readback:
return TrialOutcome(d.value, mode, True, True, True, True, True,
False, False, False, False, effects, "release")
if controls.evidence_gate:
return TrialOutcome(d.value, mode, True, True, False, False, False,
False, False, False, True, effects, "hold")
return TrialOutcome(d.value, mode, False, True, False, False, False,
False, False, True, False, effects, "release")
if d is FaultDomain.STATE:
if mode == "concurrent_version_change":
if controls.state_precondition:
return _safe_denial(d, mode)
return _unsafe(d, mode)
# The external effect already happened, but the restored checkpoint says it did not.
effects = 1
if controls.effect_readback or controls.idempotency:
return TrialOutcome(d.value, mode, True, True, True, True, True,
False, False, False, False, effects, "release")
effects = 2
return TrialOutcome(d.value, mode, False, False, False, False, False,
True, True, False, False, effects, "unsafe")
if d is FaultDomain.EVIDENCE:
effects = 1
if mode == "drop_effect_receipt" and controls.effect_readback:
return TrialOutcome(d.value, mode, True, True, True, True, True,
False, False, False, False, effects, "release")
if mode == "break_trace_lineage" and controls.evidence_gate:
# The effect can be found, but the injector refuses to claim causal proof.
return TrialOutcome(d.value, mode, True, True, False, False, False,
False, False, False, True, effects, "hold")
if controls.evidence_gate:
return TrialOutcome(d.value, mode, True, True, False, False, False,
False, False, False, True, effects, "hold")
return TrialOutcome(d.value, mode, False, True, False, False, False,
False, False, True, False, effects, "release")
raise AssertionError(f"Unhandled domain: {d}")
def run_experiment(
controls: Controls,
domains: Sequence[FaultDomain],
trials: int,
seed: int,
) -> tuple[dict[str, Aggregate], list[TrialOutcome]]:
rng = LCG(seed)
aggregates = {domain.value: Aggregate() for domain in domains}
outcomes: list[TrialOutcome] = []
for i in range(trials):
domain = domains[i % len(domains)]
spec = choose_fault(domain, rng)
outcome = simulate_trial(spec, controls, rng)
aggregates[domain.value].add(outcome)
outcomes.append(outcome)
return aggregates, outcomes
def pct(value: float) -> str:
return f"{100.0 * value:6.1f}%"
def release_assessment(overall: dict[str, float | int]) -> tuple[str, list[str]]:
reasons: list[str] = []
if float(overall["unsafe_escape"]) > 0.0:
reasons.append("at least one represented fault escaped into an unsafe effect")
if float(overall["unverifiable_release"]) > 0.0:
reasons.append("at least one success was released without outcome proof")
if float(overall["contained"]) < 0.99:
reasons.append("containment was below the 99% synthetic target")
if float(overall["useful_service"]) < 0.80:
reasons.append("useful service was below the 80% synthetic target")
return ("PASS ON THIS SYNTHETIC ASSAY" if not reasons else "BLOCK OR REDESIGN", reasons)
def aggregate_all(aggregates: Iterable[Aggregate]) -> Aggregate:
total = Aggregate()
for agg in aggregates:
for field in asdict(total):
setattr(total, field, getattr(total, field) + getattr(agg, field))
return total
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--profile", choices=("baseline", "typed"), default="typed")
parser.add_argument(
"--fault",
choices=("all",) + tuple(domain.value for domain in FaultDomain),
default="all",
help="Fault family to inject.",
)
parser.add_argument("--trials", type=int, default=2000)
parser.add_argument("--seed", type=int, default=188)
parser.add_argument("--json", action="store_true", help="Emit machine-readable JSON.")
return parser.parse_args()
def main() -> int:
args = parse_args()
if args.trials < 1:
raise SystemExit("--trials must be at least 1")
controls = Controls.typed() if args.profile == "typed" else Controls.baseline()
domains = list(FaultDomain) if args.fault == "all" else [FaultDomain(args.fault)]
aggregates, outcomes = run_experiment(controls, domains, args.trials, args.seed)
overall_agg = aggregate_all(aggregates.values())
overall = overall_agg.rates()
assessment, reasons = release_assessment(overall)
if args.json:
result = {
"profile": args.profile,
"controls": asdict(controls),
"seed": args.seed,
"fault_domains": [d.value for d in domains],
"by_domain": {name: agg.rates() for name, agg in aggregates.items()},
"overall": overall,
"assessment": assessment,
"reasons": reasons,
"interpretation": {
"positive": "The configured controls contained the represented synthetic faults under this harness.",
"negative": "A negative result identifies a control or recovery gap in the represented path.",
"cannot_establish": "The harness does not establish production reliability, unseen-fault coverage, or a causal role for any language model component.",
},
}
print(json.dumps(result, indent=2, sort_keys=True))
return 0
print("Typed failure-injection assay (synthetic)")
print(f"Profile: {args.profile} | trials: {args.trials} | seed: {args.seed}")
print("Controls:")
for name, enabled in asdict(controls).items():
print(f" {name:24} {'ON' if enabled else 'off'}")
print()
header = (
f"{'domain':10} {'n':>6} {'detect':>8} {'contain':>8} {'recover':>8} "
f"{'verify':>8} {'service':>8} {'unsafe':>8} {'dupe':>8} {'unproved':>9}"
)
print(header)
print("-" * len(header))
for domain in domains:
r = aggregates[domain.value].rates()
print(
f"{domain.value:10} {int(r['trials']):6d} {pct(float(r['detected'])):>8} "
f"{pct(float(r['contained'])):>8} {pct(float(r['recovered'])):>8} "
f"{pct(float(r['verified'])):>8} {pct(float(r['useful_service'])):>8} "
f"{pct(float(r['unsafe_escape'])):>8} {pct(float(r['duplicate_effect'])):>8} "
f"{pct(float(r['unverifiable_release'])):>9}"
)
print("-" * len(header))
print(
f"{'overall':10} {int(overall['trials']):6d} {pct(float(overall['detected'])):>8} "
f"{pct(float(overall['contained'])):>8} {pct(float(overall['recovered'])):>8} "
f"{pct(float(overall['verified'])):>8} {pct(float(overall['useful_service'])):>8} "
f"{pct(float(overall['unsafe_escape'])):>8} {pct(float(overall['duplicate_effect'])):>8} "
f"{pct(float(overall['unverifiable_release'])):>9}"
)
print(f"\nAssessment: {assessment}")
for reason in reasons:
print(f" - {reason}")
print("\nInterpretation boundary: this is evidence about represented faults in a synthetic configured system, not proof of production safety.")
return 0
if __name__ == "__main__":
raise SystemExit(main())
Containment improves, but proof still costs service
With seed 188 and 2,000 synthetic trials, the baseline profile releases unsafe effects in 69.5 per cent of injected cases. Identity and context faults escape every time; state faults either violate a current precondition or duplicate an effect; tool faults split between duplicate effects and unproved success; evidence faults preserve the effect but release it without proof.
The typed profile produces no unsafe escape or unproved release in the represented trials. Detection and containment reach 100 per cent. Recovery, proof and useful service reach 90.6 per cent because broken trace lineage is held rather than reconstructed. That remaining gap is intentional. A safe hold is better than a false success, but it is not completion. The next design task is an independent evidence-repair path, not a lower threshold.
Coverage that cannot be gamed
A team can inject hundreds of faults and still avoid the mechanism most likely to hurt it. Imagine that every Copperfin tool test throws an exception before the payment request leaves the process. The workflow retries, no money moves twice and the dashboard reports perfect containment. The count is large, but the causal coverage is empty around the commit boundary. The test has shown that a request can fail safely before effect. It has not exercised an unknown outcome.
Define coverage over the properties that change the recovery law, not over exception names. At minimum, record the entry boundary, transaction phase, duration or correlation pattern, withheld signal and consequence oracle. A useful coverage record therefore says “drop the acknowledgement after durable commit for one logical refund; preserve the provider ledger; withhold the local effect receipt; verify exactly one ledger entry by an independent read”. It does more than say “inject timeout”.
| Coverage coordinate | Question the experiment must answer | False assurance when omitted |
|---|---|---|
| Boundary and phase | Did the fault enter before validation, before commit, after commit, after receipt or after checkpoint? | All timeouts are treated as safe pre-effect failures. |
| Persistence and correlation | Was the disturbance isolated, repeated, synchronised or shared across agents and tenants? | A single polite retry stands in for a fleet-wide retry storm. |
| Signal removal | Which acknowledgement, state update, trace link or receipt became absent, delayed or contradictory? | The system detects the injected exception while still relying on an unrealistically complete evidence path. |
| Consequence oracle | Which independent source decides whether the invariant held? | The component under test grades its own recovery and converts missing proof into success. |
| Ordered composition | Which second fault can arrive before recovery completes? | Controls are validated one at a time even though their safe operation depends on one another. |
This suggests a compact coverage key, K = (domain, boundary, phase, correlation, oracle). Two runs with different keys are different experiments even when they raise the same software exception. A release claim should name the keys it has visited and leave unvisited combinations visible. The key is not a promise that the catalogue is complete. It prevents a large test count from disguising a narrow intervention surface.
Prove that the assay can fail before trusting its pass
For every release-critical fault, run a matched pair in a sandbox. In the first run, enable the intended control and predict the containment signature. In the second, remove or bypass that control while holding the fault, workload and oracle fixed. The second run should produce the nominated transfer or, at minimum, a measurable degradation at the next boundary. Reinstate the control and the invariant should return.
For Copperfin’s timeout after commit, the typed run reuses one logical idempotency key and reads the payment ledger. It should complete exactly once and prove the outcome. The counterfactual run generates a new request identity and omits readback. It should duplicate the refund in the synthetic ledger. If both runs look safe, the fault may not have reached the commit path, the oracle may be reading the wrong store, or an undisclosed fallback may be masking the intervention. A green result is then invalid rather than reassuring.
The pair tests sensitivity as well as safety. Evidence strengthens when disabling the nominated control creates the predicted transfer and restoring it removes that transfer. Evidence weakens when the output is unchanged, when a different control silently carries the result, or when the injector cannot prove that its disturbance occurred. A rival explanation is that the observed improvement comes from a hidden configuration change rather than the named control, so the manifest and randomisation must remain fixed.
Do not perform destructive control removal on a live consequential path. The smallest useful implementation is the two-profile synthetic harness included here, followed by a shadow environment whose effects are non-authoritative. The instrument cannot establish coverage of unknown faults. It can establish that a claimed control has causal leverage over a represented transfer path, which is a stronger release fact than observing a safe outcome once.
Turn one patient agent into one hundred synchronised agents
A single agent receives a rate-limit response, waits one second and retries. The policy seems harmless. Now start one hundred identical agents from the same scheduler. They receive the same response, wait the same interval and retry together. The dependency fails again under the burst, so every agent doubles down on the same rhythm.
Add random jitter and a shared retry budget. The average retry count may barely change, yet the correlated load peak collapses. Scale exposes dependency between paths that single-trajectory evaluation treats as independent. The injected property is not “more requests” in general. It is synchronised recovery under shared scarcity.
A dead system can score perfectly on safety
Configure Copperfin to deny every action after any warning. It produces zero duplicate refunds, zero wrong-subject effects and zero unproved success. On containment alone, it looks perfect. Yet it also completes no valid refund and provides no route for recovery.
This counterexample is why useful service belongs in the signature. A release rule should require hard safety invariants and a minimum recovery or service floor. It should not average them together. One wrong-subject refund cannot be compensated by many fast successful refunds; one hundred safe holds cannot establish that recovery works.
redesign recovery Release envelope
hard invariants pass Useful but unproved
shadow or hold Unsafe escape
block useful service preserved under fault → containment and outcome proof →
Boundary-complete injection will predict incidents better than clean task accuracy
The hypothesis is that, for consequential tool-using systems, a fault-transfer signature covering identity, context, tools, state and evidence will predict severe operational incidents better than clean-path task success alone. A prospective study could preregister fault families, run both evaluations before release, then compare which measure explains later incident classes.
Support would come from fault-transfer measures adding predictive value after controlling for clean accuracy, traffic and configuration change. The hypothesis would weaken if clean task success predicts incidents equally well, or if injected faults fail to resemble real incident mechanisms. A rival explanation is simply broader test coverage rather than the causal structure of injection. The smallest useful study is one configured workflow, five fault families, independent incident coding and no production claim beyond that workflow.
sequenceDiagram participant Q as Question participant M as Mechanism participant E as Evidence participant D as Decision Q->>M: The refund that passed every clean test M->>E: Five green stages do not describe the red path E-->>M: compare rival explanations E->>D: Let the injected path decide release D-->>Q: revise scope or act
Let the injected path decide release
Do not ask only whether the compound AI system completes representative tasks. Ask which faults it is authorised to encounter, how each fault crosses boundaries and what evidence shows that the resulting state is safe.
Make typed failure injection the primary release evaluation for consequential action paths. Start with three invariants: no effect for the wrong subject or expired authority, no repeated logical effect under ambiguous outcome, and no success release without an independently verified postcondition. Add service and recovery floors so refusal cannot masquerade as reliability.
A release candidate should therefore carry a versioned fault dossier: configured-system manifest; fault specifications; baseline and intervention runs; stop conditions; containment signatures by critical fault; effect receipts and readbacks; unresolved holds; and change triggers that invalidate the evidence. Any change to tool semantics, state schema, policy, retry logic, model orchestration or evidence routing can reopen the gate.
Release only the claim the assay earned
A positive result permits a bounded statement: under the tested configuration and represented faults, declared invariants held with the observed recovery, proof and service profile. A negative result says where the architecture or control path failed. Neither result proves safety against unknown faults, malicious combinations or future drift.
The clean path tells you what the system can do when the world cooperates. The injected path tells you what the system becomes when the world does not. For an acting system, that second answer should decide whether it is allowed to operate.
Glossary
- Fault
- An adjudged or hypothesised cause or abnormal condition introduced at a declared boundary.
- Error
- An internal state that can lead to a service deviation, such as “pending” after an external commit.
- Failure
- A delivered service that violates its specification at an observed boundary.
- Fault-transfer signature
- Separate measures of detection, containment, recovery, proof and useful service for a typed fault.
- Unknown outcome
- A state in which the caller cannot infer whether an external effect occurred from the response path alone.
- Idempotency key
- A stable identity for one logical effect, allowing a repeated request to return the original result rather than apply the effect again.
- Effect readback
- An independent query of authoritative state used to verify the postcondition after an ambiguous or consequential action.
- Evidence gate
- A deterministic rule that blocks a success claim or release when required receipts, lineage or readback are missing.
References
- Avižienis, A., Laprie, J.-C., Randell, B., and Landwehr, C. (2004). “Basic Concepts and Taxonomy of Dependable and Secure Computing.” IEEE Transactions on Dependable and Secure Computing, 1(1), 11–33. DOI. Foundational taxonomy
- Yuan, D., Luo, Y., Zhuang, X., Rodrigues, G. R., Zhao, X., Zhang, Y., Jain, P. U., and Stumm, M. (2014). “Simple Testing Can Prevent Most Critical Failures.” OSDI 14, 249–265. USENIX open access. Published empirical study
- Ganesan, A., Alagappan, R., Arpaci-Dusseau, A. C., and Arpaci-Dusseau, R. H. (2017). “Redundancy Does Not Imply Fault Tolerance.” FAST 17, 149–166. USENIX open access. Published empirical study
- Andrus, K., Gopalani, N., and Schmaus, B. (2014). “FIT: Failure Injection Testing.” Netflix Technology Blog. Official engineering account. Practitioner source
- Petoff, J. (2016). “Lessons Learned from Other Industries.” In Beyer, B., Jones, C., Petoff, J., and Murphy, N. R. (eds.), Site Reliability Engineering. Google SRE book. Authoritative practitioner source
- Amazon Web Services. “AWS Fault Injection Service User Guide: concepts and experiment planning.” Concepts; planning guidance. Official documentation
- Microsoft. “Permissions and security for Azure Chaos Studio.” Official documentation. Official documentation
- National Institute of Standards and Technology (2021). Developing Cyber-Resilient Systems: A Systems Security Engineering Approach, SP 800-160 Vol. 2 Rev. 1. NIST publication. Authoritative standard guidance
- OpenTelemetry. “Context propagation.” Official documentation. Official documentation
- Chen, Y., Ma, F., Zhou, Y., Yan, Z., and Jiang, Y. (2025). “CAFault: Enhance Fault Injection Technique in Practical Distributed Systems via Abundant Fault-Dependent Configurations.” USENIX ATC 25, 1409–1424. USENIX open access. Published empirical study
- Rabanser, S., Kapoor, S., Kirgis, P., Liu, K., Utpala, S., and Narayanan, A. (2026). “Towards a Science of AI Agent Reliability.” arXiv preprint. Preliminary, not peer-reviewed here
- Tian, Y., Shi, Z., and Zhao, B. (2026). “Beyond Function Calling: Benchmarking Tool-Using Agents under Tool-Environment Unreliability.” arXiv preprint. Preliminary, not peer-reviewed here
- Zheng, Y., Wu, Y., and Chang, Y. (2026). “ToolRobustBench: Stage-Wise Perturbation Evaluation and Failure Diagnosis for Tool-Calling Agents.” arXiv preprint. Preliminary, not peer-reviewed here
Internal evidence boundary and source notes
Published systems evidence supports the fault, error and failure distinction; the need to test error-handling paths; the possibility of single-fault propagation; and configuration-sensitive fault coverage. Official platform documentation supports scoped targets, experiment identities, steady-state hypotheses, stop conditions and trace correlation. The five-family taxonomy, fault-transfer coefficient, containment signature, release envelope and Copperfin examples are practitioner synthesis developed in this paper. All numerical operating results are synthetic. The three AI-agent studies are preprints and are presented as emerging evidence, not settled findings.