Part IWhat the happy path cannot tell you

A successful ordinary run establishes that one trajectory through one environment produced an acceptable result. It does not establish that the system can recognise corrupted context, preserve subject identity across delegation, distinguish a timeout from a non-commit, resume from current rather than remembered state, or reject a verifier that cannot observe the effect. Those properties remain dormant on the happy path.

The classical dependability vocabulary is useful because it separates three moments that are often collapsed into “the model failed”. A fault is the hypothesised cause: a stale policy, swapped identity, malformed tool response or lost acknowledgement. An error is the resulting invalid state inside the system: an obsolete rule treated as current, an action bound to the wrong subject, or a write whose outcome is unknown. A failure occurs when the delivered service departs from its contract: the wrong record changes, a duplicate effect occurs, or the system claims success without evidence. This fault-error-failure chain is central to the taxonomy of dependable computing developed by Avižienis and colleagues, and it stops the evaluator from scoring every anomaly as the same event.

The outcome can look right while the contract is wrong

The address example exposes an uncomfortable case: the visible business value can be correct while the execution contract is broken. If the requested address appears after two writes, a text-only evaluator sees no defect. Yet the duplicate notification may confuse the customer, the duplicate audit record may misstate the sequence of events, and another tool may charge twice rather than overwrite one field. Exactly-once semantics, authority boundaries and outcome support are properties of the trajectory and the world, not of the final sentence.

Traditional fault-injection work deliberately perturbs hardware, software or communication to evaluate dependability. Chaos engineering carries the same experimental stance into distributed operations: define a measurable steady state, compare control and experimental groups, introduce plausible disturbances and try to falsify the belief that the system will continue to deliver acceptable service. Lineage-driven fault injection adds another useful idea: reason backwards from the outcome and its dependencies rather than spraying arbitrary faults across a vast state space.

Thought experiment: the twins at the commit boundary

Imagine two assistants that use the same model, temperature, prompt, tool schema and test requests. Both score 98 per cent on a nominal benchmark. Assistant A calls a write endpoint with a new random request identifier after every timeout. Assistant B carries one idempotency key through retries and performs an authoritative readback before announcing completion.

On a reliable network, the twins are observationally equivalent. Increase only the probability of a timeout after commit. Their behaviour separates immediately. A begins to duplicate effects. B either reconstructs the completed write or reports an unresolved outcome. The fault does not merely lower accuracy; it reveals a latent architectural difference that ordinary sampling cannot identify. More nominal cases may tighten confidence around the wrong quantity.

The primary claim is about barriers, not chaos theatre

Random breakage can produce an impressive dashboard without answering a precise question. A useful experiment names the barrier under test. A stale-policy injection tests freshness and supersession controls. A subject-swap injection tests identity binding through delegation. A timeout-after-commit tests idempotency, effect receipts and readback. A stale checkpoint tests state reconstruction and optimistic concurrency. A missing readback tests whether the workflow preserves an unknown outcome rather than converting uncertainty into success.

Each experiment should therefore begin with a falsifiable sentence: “If fault F activates at interface I, barrier B will detect it before checkpoint J, prevent prohibited effect E, and produce terminal state S.” The claim can fail in several informative ways. The fault may not activate. The detector may remain silent. Detection may arrive after the effect. The barrier may stop the task but corrupt state while doing so. Recovery may loop. The verifier may share the same corrupted source. Each failure points to a different engineering change.

flowchart LR
  A["The green tick that proves<br/>almost nothing"] --> B["Part I What the happy path<br/>cannot tell you"]
  B --> C["Part II Make propagation the<br/>measured object"]
  C --> D["Part III Build a typed<br/>injector, not a bag of chaos"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#087A98,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Part IIMake propagation the measured object

Most evaluation tables place tasks on rows and models on columns. Failure injection needs a different geometry. Put fault families on rows and decision checkpoints on columns. The cell asks whether a known perturbation made that checkpoint invalid, suspect or unchanged. This fault-transfer view shows where an error first appears, how far it travels and where a barrier changes its course.

For an agentic workflow, useful checkpoints are not arbitrary log spans. They correspond to the system’s decision contract: intent, identity, world state, context, reasoning, authority, action, evidence, outcome, and release or recovery. A fault may begin in one place and first become observable elsewhere. A stale policy enters through context but becomes consequential when authority is granted. A malformed tool result enters through action but may be caught only when evidence is checked. A poisoned memory enters before the current trace yet changes the plan and later tool choice.

The unit of reliability evidence is therefore not the final answer and not the raw trace. It is the change in validity across named checkpoints, conditioned on a verified fault activation. This preserves causality. It also creates an actionable map: reinforce the edge where invalidity crosses, rather than retrain a model simply because the last sentence was wrong.

Metrics that keep denominators honest

Six measures cover the minimum useful experiment. Let a fault family be f. Activation rate records whether the scheduled perturbation actually fired. Detection rate asks whether a named control identified the resulting anomaly. Containment rate asks whether prohibited or duplicate effects were prevented. Recovery rate applies only to trials where a valid recovery path exists. Escape rate counts invalid outcomes that the system accepted or represented as successful. Detection latency measures how many checkpoints, events or milliseconds elapsed between activation and recognition.

The symbols are intentionally simple. When A falls, the harness is not reaching the intended branch and the other rates can mislead. When D rises but C does not, observability has improved without reducing harm. When C is high and R is low, the system stops safely but loses service. When E is non-zero for a high-consequence family, an average success score should not wash it away.

A recent AgentChaos preprint makes trigger verification explicit. Its authors intercept language-model API responses, record whether the configured perturbation actually occurred and compute faulted performance on triggered tasks. Across their evaluated systems, benchmarks and 65 configurations, pass@1 fell by as much as roughly 50 percentage points in one setting, while their tested diagnosis methods remained below 56 per cent for fault-step accuracy. The exact values belong to that study, but the denominator lesson is general: a scheduled injection that never fires is not a robustness trial.

ToolBench-X, another recent preprint, preserves at least one valid recovery path in each injected instance. That design choice matters because it separates inability to recover from impossibility of success. ToolMaze similarly distinguishes explicit from implicit tool failures and transient from permanent ones; its reported models suffered their sharpest degradation under implicit semantic corruption. These studies are early and should not be treated as settled benchmarks. They nevertheless point towards a better evaluation object: diagnosis and recovery under controlled, solvable disturbance.

Fault intensity is a curve, not a switch

One successful recovery from one timeout does not justify a general claim. The same fault can vary by position, duration, burst length, semantic subtlety and correlation with other dependencies. A timeout before any durable work is different from a timeout after commit. A schema error that raises an exception is different from a valid-looking response with one decisive field omitted. A stale value that is one minute old may be harmless in one workflow and disqualifying in another.

Build an intensity curve for each important family. For transient failures, vary probability and burst length. For semantic corruption, vary how much of the payload remains plausible. For stale context, vary age and whether a superseding rule changes the decision. For identity faults, vary where the subject binding is lost. For state faults, vary the number and consequence of intervening events. The goal is not to find a theatrical breaking point. It is to locate the region where the barrier changes from reliable containment to intermittent escape.

Repeated stochastic runs then need uncertainty, not just percentages. Report the number of scheduled and triggered trials, confidence intervals for escape and recovery, and the distribution of detection latency. Use paired control and injected runs on the same task where possible, because task difficulty otherwise masquerades as fault impact. Preserve the clean-run result too: a system with very low nominal success can appear insensitive simply because little performance remains to lose.

Frequency and consequence should remain separate. A rare post-commit ambiguity can deserve a stronger gate than a common harmless retry. Conversely, a high-frequency benign timeout may dominate cost and service availability without creating unsafe effects. The experiment should therefore estimate response by fault intensity, while the release decision combines that response with exposure, consequence and reversibility. This keeps engineering evidence distinct from the business decision that consumes it.

The fault-transfer matrix

For repeated stochastic trials, define a transfer entry as Pij = Pr(checkpoint j is invalid | fault i activated). Rows are fault types; columns are checkpoints. Values near zero show masking, irrelevance or effective containment. Values near one show a reliable propagation route. An additional column should record whether the terminal outcome was acceptable, unknown or unacceptable.

The matrix is not a causal graph by itself. Shared hidden variables, model variance and environment changes can affect several cells. Paired runs, deterministic fixtures where possible, stable seeds, repeated trials and intervention on one feature at a time improve interpretation. The matrix becomes especially useful when compared across releases. A new model may improve nominal success while increasing transfer from stale context to authority. A new retry policy may improve availability while increasing duplicate effects. Those regressions are visible only when the fault family remains stable.

Thought experiment: the verifier that drinks from the same well

Suppose an agent retrieves a policy paragraph from a cache, uses it to authorise an action, then asks a verifier to check whether the action followed policy. The verifier reads the same cached paragraph. Inject a stale policy into the cache. Both components agree. The trace contains planning, checking and a high-confidence approval, yet independence is illusory.

Now vary one feature. The verifier resolves the policy from an authoritative store with a supersession check and effective-time rule. Disagreement appears before execution. The extra model call in the first design added ceremony; the independent evidence path in the second added assurance. Verification strength depends less on the label “verifier” than on whether its evidence and failure modes are sufficiently independent.

This is why fault injection should target shared dependencies, not only individual model outputs. Corrupt the source used by both proposer and checker. Delay the identity provider. Replay the same stale checkpoint to every agent. Remove one required field from a tool response while preserving HTTP success. The experiment asks whether diversity is real or merely multiple components repeating one damaged premise.

Part IIIBuild a typed injector, not a bag of chaos

A fault catalogue becomes useful when every entry carries an operational contract. A string such as timeout=true is too weak. The experiment needs to know where the timeout occurs, whether the underlying operation committed, what activation predicate proves it fired, which detector should respond, what terminal state counts as safe and how cleanup will restore the environment.

The type system does not need to be elaborate. It must make five distinctions explicit: surface, timing, semantics, expected barrier and oracle. Surface identifies context, identity, tool, state, verification or another governed boundary. Timing distinguishes before call, during execution, after commit and during readback. Semantics distinguishes omission, corruption, delay, duplication, replay and unauthorised substitution. The expected barrier names the control under test. The oracle defines acceptable world state, not merely acceptable prose.

Five surfaces that expose different architecture

Context faults change what the system is allowed to know or treat as current. Useful injections include a superseded policy, a contradictory source, a missing entitlement or a retrieved passage whose provenance no longer authorises use. The detector is usually not “the model noticed”. It is a context compiler that checks source identity, effective time, version, entitlement and claim support.

Identity faults change who requested, proposed, approved or receives an action. Swap the subject after authentication, drop the delegating principal, substitute a service account or reuse a capability outside its intended scope. The expected barrier is a mechanical binding between intent, authenticated subject, delegated actor and typed effect. Identity faults deserve isolated synthetic environments because even one escaped trial can alter the wrong record.

Tool faults change invocation or execution. Return a timeout before commit, a timeout after commit, a syntactically valid partial payload, a stale value, a schema drift, throttling or an explicit rejection. These are not interchangeable. Retry is appropriate for some transient pre-commit failures and dangerous for an unknown post-commit outcome. The injection type must therefore carry commit semantics.

State faults change the relationship between remembered workflow and current reality. Resume from a stale checkpoint, replay an event, advance an entity version between plan and action, expire an approval or remove a prerequisite. A robust workflow reconstructs from authoritative evidence, checks versions and avoids reviving authority that has expired since the checkpoint.

Verification faults damage the evidence used to declare completion. Make readback unavailable, corrupt an effect receipt, let the verifier read the same stale snapshot as the proposer or return a confident judgement without the required field. The safe response is often not failure but unknown outcome, followed by reconciliation. Converting absent evidence into success is an escape even when the hidden world happens to be correct.

Worked example: an executable address-change laboratory

The companion artefact implements a tiny synthetic world with two customer records, a current policy, an audit trail, notifications and an idempotency store. It runs a weak workflow and a bounded workflow against the same five injections. No language model is required. That is deliberate: the experiment isolates system controls before stochastic reasoning is added.

The weak workflow trusts the supplied policy and subject, writes without compare-and-set, retries a timeout without preserving action identity and treats a missing readback as success. The bounded workflow binds the authenticated subject, refreshes superseded context, uses an idempotency key, enforces the expected record version and refuses to claim success when authoritative readback is absent.

The artefact’s property checks include both positive and negative cases. The control path must succeed for both workflows. The bounded timeout-after-commit case must produce exactly one effect and a verified terminal state. The weak subject-swap case must escape and alter the wrong synthetic record, proving that the harness can expose the defect. The bounded missing-readback case must preserve a review-required outcome rather than manufacture certainty.

class Surface(str, Enum):
    CONTEXT = "context"
    IDENTITY = "identity"
    TOOL = "tool"
    STATE = "state"
    VERIFICATION = "verification"

@dataclass(frozen=True)
class Injection:
    name: str
    surface: Surface
    activation_point: str
    mutation: str
    expected_detector: str
    safe_response: str
    recoverable: bool
    severity: int

# Positive property: a post-commit timeout is recovered exactly once.
timeout = index[("bounded", "tool.timeout_after_commit")]
assert timeout.status == "SUCCESS"
assert timeout.effects == 1
assert timeout.recovered and timeout.contained and not timeout.escaped

# Negative property: the weak identity binding lets the fault escape.
identity_escape = index[("weak", "identity.subject_swap")]
assert identity_escape.escaped

Run the file with python failure_injection_lab.py. The expected summary is deterministic. In this constructed suite, the weak workflow detects none of the five faults and lets all five escape. The bounded workflow detects all five, contains all five and recovers both cases defined as recoverable. Those numbers validate the code path, not an external system.

weak:    detection 0.00 | containment 0.00 | recovery 0.00 | escape 1.00
bounded: detection 1.00 | containment 1.00 | recovery 1.00 | escape 0.00

fault                               IDEN CONT AUTH ACTI EVID OUTC
context.stale_policy                  .    X    .    .    .    .
identity.subject_swap                 X    .    .    .    .    .
tool.timeout_after_commit             .    .    .    .    .    .
state.intervening_update              .    .    .    X    .    .
verification.readback_unavailable     .    .    .    .    ?    .

#!/usr/bin/env python3
"""Typed fault-injection laboratory for a synthetic address-change agent.

The programme uses only the Python standard library. It compares a weak
workflow with a bounded workflow across context, identity, tool, state and
verification faults. All data and thresholds are synthetic.
"""

from __future__ import annotations

import argparse
import copy
import json
from dataclasses import asdict, dataclass, field
from enum import Enum
from typing import Any, Dict, Iterable, List, Optional, Tuple


class Surface(str, Enum):
    CONTEXT = "context"
    IDENTITY = "identity"
    TOOL = "tool"
    STATE = "state"
    VERIFICATION = "verification"


CHECKPOINTS = ("identity", "context", "authority", "action", "evidence", "outcome")


@dataclass(frozen=True)
class Injection:
    name: str
    surface: Surface
    activation_point: str
    mutation: str
    expected_detector: str
    safe_response: str
    recoverable: bool
    severity: int


CATALOGUE: Tuple[Injection, ...] = (
    Injection(
        name="context.stale_policy",
        surface=Surface.CONTEXT,
        activation_point="context_compile",
        mutation="Replace current policy P-4 with superseded policy P-3.",
        expected_detector="policy_version_gate",
        safe_response="Refresh context and re-authorise before action.",
        recoverable=True,
        severity=4,
    ),
    Injection(
        name="identity.subject_swap",
        surface=Surface.IDENTITY,
        activation_point="identity_bind",
        mutation="Change the action subject from C-100 to C-200 after authentication.",
        expected_detector="subject_binding_gate",
        safe_response="Reject the action and require a new authenticated intent.",
        recoverable=False,
        severity=5,
    ),
    Injection(
        name="tool.timeout_after_commit",
        surface=Surface.TOOL,
        activation_point="tool_commit",
        mutation="Commit the write, then return a timeout to the caller.",
        expected_detector="idempotency_plus_readback",
        safe_response="Retry with the same key, then read back authoritative state.",
        recoverable=True,
        severity=5,
    ),
    Injection(
        name="state.intervening_update",
        surface=Surface.STATE,
        activation_point="before_write",
        mutation="Advance the record version after planning but before execution.",
        expected_detector="compare_and_set_gate",
        safe_response="Stop, reconstruct state and ask whether the new intent still applies.",
        recoverable=False,
        severity=4,
    ),
    Injection(
        name="verification.readback_unavailable",
        surface=Surface.VERIFICATION,
        activation_point="readback",
        mutation="Return no authoritative observation after the write.",
        expected_detector="unknown_outcome_gate",
        safe_response="Report an unknown outcome and route to reconciliation.",
        recoverable=False,
        severity=3,
    ),
)


@dataclass
class Record:
    address: str
    version: int


@dataclass
class World:
    records: Dict[str, Record] = field(
        default_factory=lambda: {
            "C-100": Record("4 Old Street", 7),
            "C-200": Record("9 Other Road", 3),
        }
    )
    current_policy: str = "P-4"
    notifications: List[str] = field(default_factory=list)
    audit: List[Dict[str, Any]] = field(default_factory=list)
    idempotency: Dict[str, Dict[str, Any]] = field(default_factory=dict)

    def snapshot(self) -> Dict[str, Any]:
        return {
            "records": {key: asdict(value) for key, value in self.records.items()},
            "notifications": list(self.notifications),
            "audit": copy.deepcopy(self.audit),
            "idempotency": copy.deepcopy(self.idempotency),
        }


@dataclass(frozen=True)
class Request:
    actor: str = "U-100"
    authenticated_subject: str = "C-100"
    requested_subject: str = "C-100"
    new_address: str = "22 New Lane"
    supplied_policy: str = "P-4"
    expected_version: int = 7
    request_id: str = "REQ-148-001"


@dataclass
class Event:
    checkpoint: str
    valid: bool
    detail: str


@dataclass
class Result:
    system: str
    injection: str
    scheduled: bool
    triggered: bool
    detected: bool
    contained: bool
    recovered: bool
    escaped: bool
    status: str
    effects: int
    detection_step: Optional[int]
    events: List[Event]
    transfer: Dict[str, int]
    final_world: Dict[str, Any]

    def public_dict(self) -> Dict[str, Any]:
        data = asdict(self)
        data["events"] = [asdict(event) for event in self.events]
        return data


class CommitTimeout(RuntimeError):
    """Raised after a synthetic write has already committed."""


class AddressTool:
    def __init__(self, world: World, injection: Optional[Injection]):
        self.world = world
        self.injection = injection
        self.timeout_fired = False

    def update(
        self,
        *,
        subject: str,
        address: str,
        expected_version: Optional[int],
        idempotency_key: Optional[str],
        enforce_compare_and_set: bool,
    ) -> Dict[str, Any]:
        if idempotency_key and idempotency_key in self.world.idempotency:
            return copy.deepcopy(self.world.idempotency[idempotency_key])

        record = self.world.records[subject]
        if enforce_compare_and_set and expected_version != record.version:
            return {
                "ok": False,
                "reason": "VERSION_CONFLICT",
                "observed_version": record.version,
            }

        record.address = address
        record.version += 1
        receipt = {
            "ok": True,
            "subject": subject,
            "address": address,
            "version": record.version,
        }
        self.world.audit.append({"kind": "address_write", **receipt})
        self.world.notifications.append(f"address_changed:{subject}:{record.version}")
        if idempotency_key:
            self.world.idempotency[idempotency_key] = copy.deepcopy(receipt)

        if (
            self.injection
            and self.injection.name == "tool.timeout_after_commit"
            and not self.timeout_fired
        ):
            self.timeout_fired = True
            raise CommitTimeout("synthetic timeout after durable commit")
        return receipt

    def read(self, subject: str) -> Optional[Dict[str, Any]]:
        if self.injection and self.injection.name == "verification.readback_unavailable":
            return None
        record = self.world.records[subject]
        return {"subject": subject, "address": record.address, "version": record.version}


def _mark(events: List[Event], checkpoint: str, valid: bool, detail: str) -> None:
    events.append(Event(checkpoint, valid, detail))


def _transfer(events: Iterable[Event]) -> Dict[str, int]:
    """0 = valid, 1 = suspect/unknown, 2 = invalid at the checkpoint."""
    output = {checkpoint: 0 for checkpoint in CHECKPOINTS}
    for event in events:
        if event.checkpoint not in output:
            continue
        if event.valid:
            continue
        output[event.checkpoint] = 1 if "unknown" in event.detail.lower() else 2
    return output


def _summarise(
    *,
    system: str,
    injection: Optional[Injection],
    world: World,
    events: List[Event],
    status: str,
    detected: bool,
    recovered: bool,
    safe_subject: str = "C-100",
) -> Result:
    triggered = injection is not None
    wrong_subject_changed = world.records["C-200"].address != "9 Other Road"
    duplicate_effect = len(world.notifications) > 1
    lost_update = bool(
        injection
        and injection.name == "state.intervening_update"
        and status == "SUCCESS"
        and world.records["C-100"].address == "22 New Lane"
    )
    latest: Dict[str, bool] = {}
    for event in events:
        latest[event.checkpoint] = event.valid
    unsupported_success = status == "SUCCESS" and any(
        not latest.get(checkpoint, True)
        for checkpoint in ("identity", "context", "authority", "action", "evidence", "outcome")
    )
    escaped = bool(
        wrong_subject_changed or duplicate_effect or lost_update or unsupported_success
    ) and status == "SUCCESS"
    contained = not escaped and not wrong_subject_changed and not duplicate_effect and not lost_update
    effects = len(world.notifications)
    detection_step = next(
        (index for index, event in enumerate(events, start=1) if not event.valid), None
    )
    return Result(
        system=system,
        injection=injection.name if injection else "control.none",
        scheduled=triggered,
        triggered=triggered,
        detected=detected,
        contained=contained,
        recovered=recovered,
        escaped=escaped,
        status=status,
        effects=effects,
        detection_step=detection_step,
        events=events,
        transfer=_transfer(events),
        final_world=world.snapshot(),
    )


def run_weak(injection: Optional[Injection]) -> Result:
    world = World()
    request = Request()
    events: List[Event] = []
    tool = AddressTool(world, injection)

    target = request.requested_subject
    if injection and injection.name == "identity.subject_swap":
        target = "C-200"
    identity_valid = target == request.authenticated_subject
    _mark(events, "identity", identity_valid, f"actor bound to {target}")

    policy = request.supplied_policy
    if injection and injection.name == "context.stale_policy":
        policy = "P-3"
    context_valid = policy == world.current_policy
    _mark(events, "context", context_valid, f"policy {policy} accepted without freshness check")
    _mark(events, "authority", identity_valid and context_valid, "authority inferred from supplied fields")

    if injection and injection.name == "state.intervening_update":
        world.records[target].address = "18 Intervening Avenue"
        world.records[target].version += 1

    try:
        receipt = tool.update(
            subject=target,
            address=request.new_address,
            expected_version=None,
            idempotency_key=None,
            enforce_compare_and_set=False,
        )
    except CommitTimeout:
        # The weak worker treats timeout as non-commit and repeats the side effect.
        receipt = tool.update(
            subject=target,
            address=request.new_address,
            expected_version=None,
            idempotency_key=None,
            enforce_compare_and_set=False,
        )
    action_valid = identity_valid and context_valid and len(world.notifications) == 1
    _mark(events, "action", action_valid, f"tool receipt {receipt}")

    observation = tool.read(target)
    evidence_valid = observation is not None and observation["address"] == request.new_address
    # Weak verification treats a missing readback as success if a receipt exists.
    weak_evidence_claim = evidence_valid or receipt.get("ok", False)
    detail = "readback matches" if evidence_valid else "unknown readback treated as success"
    _mark(events, "evidence", evidence_valid, detail)
    _mark(events, "outcome", action_valid and weak_evidence_claim, "workflow returned SUCCESS")

    return _summarise(
        system="weak",
        injection=injection,
        world=world,
        events=events,
        status="SUCCESS",
        detected=False,
        recovered=False,
    )


def run_bounded(injection: Optional[Injection]) -> Result:
    world = World()
    request = Request()
    events: List[Event] = []
    tool = AddressTool(world, injection)

    target = request.requested_subject
    if injection and injection.name == "identity.subject_swap":
        target = "C-200"
    identity_valid = target == request.authenticated_subject
    _mark(events, "identity", identity_valid, f"authenticated subject={request.authenticated_subject}; target={target}")
    if not identity_valid:
        _mark(events, "outcome", True, "rejected before effect")
        return _summarise(
            system="bounded",
            injection=injection,
            world=world,
            events=events,
            status="REJECTED",
            detected=True,
            recovered=False,
        )

    supplied_policy = request.supplied_policy
    if injection and injection.name == "context.stale_policy":
        supplied_policy = "P-3"
    context_valid = supplied_policy == world.current_policy
    _mark(events, "context", context_valid, f"supplied={supplied_policy}; authoritative={world.current_policy}")
    if not context_valid:
        # Refreshing authoritative context is the tested recovery path.
        supplied_policy = world.current_policy
        _mark(events, "context", True, f"refreshed to {supplied_policy}")

    _mark(events, "authority", True, "identity and current policy checked mechanically")

    if injection and injection.name == "state.intervening_update":
        world.records[target].address = "18 Intervening Avenue"
        world.records[target].version += 1

    idempotency_key = f"address:{request.request_id}:{target}"
    try:
        receipt = tool.update(
            subject=target,
            address=request.new_address,
            expected_version=request.expected_version,
            idempotency_key=idempotency_key,
            enforce_compare_and_set=True,
        )
    except CommitTimeout:
        receipt = tool.update(
            subject=target,
            address=request.new_address,
            expected_version=request.expected_version,
            idempotency_key=idempotency_key,
            enforce_compare_and_set=True,
        )

    if not receipt.get("ok"):
        _mark(events, "action", False, f"blocked: {receipt['reason']}")
        _mark(events, "outcome", True, "stopped for state reconstruction")
        return _summarise(
            system="bounded",
            injection=injection,
            world=world,
            events=events,
            status="REVIEW_REQUIRED",
            detected=True,
            recovered=False,
        )

    action_valid = len(world.notifications) == 1
    _mark(events, "action", action_valid, f"idempotent receipt {receipt}")

    observation = tool.read(target)
    if observation is None:
        _mark(events, "evidence", False, "unknown outcome: authoritative readback unavailable")
        _mark(events, "outcome", True, "routed to reconciliation without claiming success")
        return _summarise(
            system="bounded",
            injection=injection,
            world=world,
            events=events,
            status="REVIEW_REQUIRED",
            detected=True,
            recovered=False,
        )

    evidence_valid = (
        observation["subject"] == target
        and observation["address"] == request.new_address
        and observation["version"] == receipt["version"]
    )
    _mark(events, "evidence", evidence_valid, f"authoritative readback {observation}")
    status = "SUCCESS" if evidence_valid and action_valid else "REVIEW_REQUIRED"
    _mark(events, "outcome", status == "SUCCESS", f"workflow returned {status}")

    return _summarise(
        system="bounded",
        injection=injection,
        world=world,
        events=events,
        status=status,
        detected=bool(injection),
        recovered=bool(
            injection
            and injection.recoverable
            and status == "SUCCESS"
            and not any(not event.valid for event in events if event.checkpoint in {"action", "outcome"})
        ),
    )


def run_suite() -> List[Result]:
    results: List[Result] = [run_weak(None), run_bounded(None)]
    for injection in CATALOGUE:
        results.extend((run_weak(injection), run_bounded(injection)))
    return results


def metrics(results: Iterable[Result]) -> Dict[str, Dict[str, float]]:
    grouped: Dict[str, List[Result]] = {}
    for result in results:
        if result.injection == "control.none":
            continue
        grouped.setdefault(result.system, []).append(result)

    output: Dict[str, Dict[str, float]] = {}
    for system, rows in grouped.items():
        triggered = sum(row.triggered for row in rows)
        recoverable = sum(
            row.triggered
            and next(item for item in CATALOGUE if item.name == row.injection).recoverable
            for row in rows
        )
        output[system] = {
            "activation_rate": triggered / len(rows),
            "detection_rate": sum(row.detected for row in rows) / triggered,
            "containment_rate": sum(row.contained for row in rows) / triggered,
            "recovery_rate": (
                sum(row.recovered for row in rows) / recoverable if recoverable else 0.0
            ),
            "escape_rate": sum(row.escaped for row in rows) / triggered,
        }
    return output


def render_heatmap(results: Iterable[Result]) -> str:
    rows = [row for row in results if row.system == "bounded" and row.triggered]
    header = "fault".ljust(36) + " ".join(name[:4].upper().rjust(4) for name in CHECKPOINTS)
    lines = [header, "-" * len(header)]
    symbol = {0: " . ", 1: " ? ", 2: " X "}
    for row in rows:
        cells = " ".join(symbol[row.transfer[name]].rjust(4) for name in CHECKPOINTS)
        lines.append(row.injection.ljust(36) + cells)
    lines.append("Legend: . valid, ? unknown/suspect, X invalid and detected")
    return "\n".join(lines)


def assert_properties(results: List[Result]) -> None:
    index = {(row.system, row.injection): row for row in results}
    assert index[("weak", "control.none")].status == "SUCCESS"
    assert index[("bounded", "control.none")].status == "SUCCESS"

    timeout = index[("bounded", "tool.timeout_after_commit")]
    assert timeout.status == "SUCCESS"
    assert timeout.effects == 1
    assert timeout.recovered and timeout.contained and not timeout.escaped

    identity_escape = index[("weak", "identity.subject_swap")]
    assert identity_escape.escaped
    assert identity_escape.final_world["records"]["C-200"]["address"] == "22 New Lane"

    unknown = index[("bounded", "verification.readback_unavailable")]
    assert unknown.status == "REVIEW_REQUIRED"
    assert unknown.contained and not unknown.escaped


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--json", action="store_true", help="print full JSON results")
    args = parser.parse_args()

    results = run_suite()
    assert_properties(results)
    if args.json:
        print(json.dumps({
            "catalogue": [asdict(item) for item in CATALOGUE],
            "metrics": metrics(results),
            "results": [row.public_dict() for row in results],
        }, indent=2, default=str))
        return

    print("Typed failure-injection lab: synthetic address-change workflow")
    print(json.dumps(metrics(results), indent=2))
    print()
    print(render_heatmap(results))
    print("\nAll property checks passed.")


if __name__ == "__main__":
    main()

The serious negative control

A primary evaluation should face a case where its preferred mechanism is unnecessary. Consider a pure, deterministic, stateless parser that accepts a small formal language, has no external effects and can be exhaustively checked against its grammar and properties. System-level fault injection may add little beyond ordinary mutation and property testing. There is no long-lived context, delegated identity, external tool, persisted state or outcome ambiguity to propagate through.

This negative control protects the thesis from becoming “break everything because chaos is good”. Failure injection earns primacy as the number of hidden interfaces, persistent states, asynchronous effects and recovery claims increases. When the full behavioural space is already exhaustively specified and checked, simpler verification can dominate.

A second boundary concerns impossible tasks. If every independent address service is unavailable, task completion is not a valid recovery oracle. The correct outcome may be a durable pending state, explicit uncertainty and human escalation. A benchmark that scores only completion will punish safe behaviour and reward fabrication. ToolBench-X’s choice to retain at least one valid recovery path is useful for comparing recovery skill; separate unsolvable trials are still necessary to test abstention and safe stop.

Part IVTurn faults into release evidence

A laboratory becomes operationally useful when its results change release scope. The release question is not “Did chaos pass?” It is “Which consequence-bearing barriers have direct evidence, under what faults and intensities, with what residual escape rate and recovery debt?” A model update, prompt change, tool migration, policy revision or workflow refactor should rerun the relevant fault families because each can alter propagation.

NIST’s AI Risk Management Framework playbook treats measurement as purpose-dependent and calls for stress testing, tests related to incidents and near misses, documentation of conditions under which a system fails safely, and, where appropriate, chaos-engineering approaches. Google’s SRE literature similarly uses disaster-recovery exercises and controlled fault injection to test preparedness rather than infer it from written plans. The operational lesson is modest: controls that have never faced their triggering condition remain assumptions.

Production-shaped worked scenario: one request, five interventions

Return to the address-change assistant, now shaped like an enterprise workflow. A user session supplies authenticated identity and intent. A context compiler retrieves the current servicing policy and entitlement. A planner proposes a typed action with subject, field, value, reason and expected record version. An authority kernel checks permission and risk. The tool executes with an idempotency key. An outcome observer reads the record back and emits an effect receipt. A human queue receives unresolved cases.

The first release candidate passes 2,000 nominal synthetic and historical-replay cases. That supports capability, but the reliability gate begins only now. Five paired interventions run against isolated data. The stale-policy case should refresh before authority. The subject-swap case should reject before action. The timeout-after-commit case should reconstruct one effect. The intervening-update case should stop on a version conflict. The missing-readback case should preserve an unknown outcome and create one reconciliation item.

Suppose four barriers behave correctly, but the missing-readback case still returns “Address updated successfully” because the orchestration layer treats any successful tool receipt as final. The underlying write usually did occur. The team argues that the message is therefore harmless. The gate should still block the success claim. The defect is not address accuracy; it is unsupported certainty. A future tool may acknowledge receipt before durable commit, or a failover may lose the write. The workflow must reserve “verified” for observed postconditions.

The remedy is small and architectural. Introduce an explicit outcome state with values such as VERIFIED, REJECTED, NO_EFFECT, UNKNOWN and RECONCILIATION_REQUIRED. Only the outcome observer can issue VERIFIED. The model can describe the state but cannot promote it. The same injection then becomes a permanent regression test.

From single faults to compound game days

Single-fault experiments establish local barrier behaviour. Compound experiments test assumptions about independence and recovery order. Combine stale context with verifier dependence. Combine a timeout-after-commit with an expired idempotency record. Combine an identity-provider slowdown with a queued action whose approval expires. Combine a tool schema change with a model that fills missing fields from conversation memory.

Recent multi-agent research provides early evidence that topology changes resilience. An ICML 2025 study injected faulty agent responses across several collaboration structures and observed materially different performance drops; its hierarchical arrangement was more resilient in the tested tasks than two alternatives. MAS-FIRE, a 2026 preprint, reports that iterative closed-loop designs neutralised a substantial share of faults that collapsed linear workflows in its experiments. These results should not be universalised, but they support an important evaluation rule: fault tolerance belongs to the interaction structure, not just the strongest model in the structure.

Security injection remains related but distinct. AgentDojo and InjecAgent expose tool-using agents to indirect prompt injection and untrusted data. They test hostile manipulation rather than ordinary operational failure. A mature programme shares infrastructure across reliability and security, including trace capture, typed mutations, activation proof and effect oracles, while preserving different threat assumptions and release criteria.

Contain the experiment before asking it to test containment

Failure injection can itself cause harm. Begin in a deterministic component harness with synthetic data. Move to an isolated end-to-end environment. Use shadow or read-only paths before reversible state changes. Production experiments require explicit ownership, small blast radius, stop conditions, observability, cleanup, protected subjects and an incident route. Some faults, especially identity substitution and irreversible financial effects, may never be appropriate in a live environment.

The experiment’s containment plan should be as typed as the fault. Name the maximum number of affected requests, allowed effects, time window, abort metric, restoration action and accountable operator. A fault that cannot be reliably withdrawn is not a controlled injection. A tool that cannot distinguish test identity from customer identity should remain outside live trials.

The release instrument

The table below converts results into a decision. Thresholds must come from consequence, reversibility, exposure and local policy, not from a universal percentage. A high-volume read-only summariser and a low-volume payment authoriser should not share one floor. The invariant is structural: activation must be proven, escape must be visible, and high-consequence faults must not disappear inside an average.

Retest triggers should follow the architecture. Change the model and rerun semantic corruption and tool-selection faults. Change retrieval and rerun freshness, contradiction and entitlement faults. Change orchestration and rerun checkpoint, retry and timeout cases. Change identity or permissioning and rerun principal propagation and scope faults. Change a verifier and rerun blind-source and false-positive controls. Change a tool contract and rerun schema, commit and readback cases.

Incident learning should feed the same catalogue. Every material incident or near miss asks two questions: which typed injection would have reproduced the causal chain, and why did the existing suite not contain it? The resulting case is more valuable than a generic post-incident increase in benchmark size. It turns operating evidence into a regression barrier.