The 99 per cent system that fails

Suppose a request must pass through ten stages. Each stage succeeds 99 per cent of the time. If every stage is required, every request takes the same route, and failures are independent, end-to-end success is not 99 per cent. It is 0.99 raised to the tenth power, about 90.4 per cent. At twenty stages it falls to about 81.8 per cent.

The multiplication is useful, but only inside a narrow box. It assumes the same route for every task. It assumes a stage has one stable error rate. It assumes errors at one stage do not make errors at another more likely. It also treats any local failure as terminal, even when a verifier, abstention gate or recovery procedure catches it.

Topology decides how errors combine

A serial pipeline behaves like an AND requirement for success: every required stage must work. A redundant pair behaves like an OR requirement only if either result is acceptable and a sound selector can recognise it. A router creates a mixture over paths. A voter introduces a new decision component whose errors must also be counted. A feedback loop makes later failure probabilities depend on earlier outputs and recovery attempts.

These topologies cannot be reduced to one average accuracy. Consider two 95 per cent solvers. In a serial decomposition where both must be correct, success is 90.25 per cent under independence. In an ideal parallel design where either correct answer is sufficient and a perfect selector identifies it, success is 99.75 per cent. In a two-agent agreement gate, disagreement may trigger abstention, so reliability depends on whether abstention counts as safe success, service failure or acceptable deferral. The parts are unchanged; the top event and composition rule change the answer.

Fan-out creates a second kind of composition. A request that waits for twenty parallel services does not multiply correctness in the same way, but its deadline can be dominated by the slowest branch. Dean and Barroso showed why tail latency becomes a system property as services scale.⁵ In a compound AI workflow, a late retriever, tool or verifier can turn an otherwise correct trajectory into a timeout top event. Reliability must therefore specify whether “success” includes timeliness, rather than only semantic correctness.

Accuracy and reliability answer different questions

A component score asks how often a part produces an acceptable output on a test distribution. System reliability asks how often a defined top event is avoided for a defined workload and operating period. The top event might be a wrong answer, an unauthorised tool call, a missed service-level deadline, a duplicate payment, or a failure to escalate.

This distinction is inherited from reliability engineering. Fault-tree analysis starts with a specified undesired event and works backwards through the combinations that can cause it.² NIST similarly treats risk as involving both probability and consequence; reliability supplies only part of that decision.¹¹ A system can be reliable for low-stakes drafting and unreliable for irreversible execution because the top events and tolerances differ.

The top event also fixes what counts as containment. If the top event is “a wrong recommendation reaches a person”, abstention contains the defect. If the top event is “the customer receives no answer within thirty seconds”, abstention may itself be a failure. If the top event is “an unauthorised payment settles”, a correct explanatory response is irrelevant unless the authority and effect paths are controlled. Reliability numbers are meaningless without their event definition, workload and time horizon.

A practical review should therefore reject statements such as “the agent is 97 per cent reliable” unless they are completed. Reliable against which top event? On which task classes? With which route mixture? Over how many retries? Under which tool and state versions? With what deadline and recovery policy? Precision here is not bureaucracy. It prevents a local benchmark from being mistaken for a system guarantee.

The hidden assumption is independence

Imagine two reviewers that each miss a defect on ten per cent of cases. If their misses are independent, both miss together on one per cent. That number makes two-reviewer designs look attractive. Yet the calculation silently assumes the reviewers fail for unrelated reasons.

Agreement is an especially weak proxy for independence. Two reviewers can agree because the answer is well supported, because the task is easy, because both copied the same evidence, or because both share a misconception. The observable event “A agrees with B” does not identify which generator produced the agreement. A reliability review needs the paired outcome table: both correct, A only correct, B only correct, and both wrong. Joint wrong cases carry the information that marginal accuracy hides.

Suppose 1,000 matched cases yield 100 errors for each reviewer. Under independence, about ten joint errors are expected. If eighty cases are jointly wrong, the redundancy is mostly nominal. Yet even this comparison can overstate implementation dependence because difficult cases cluster. Stratify by task family, ambiguity, evidence quality and route, then repeat the estimate. What remains after conditioning is the residual dependence the architecture must explain.

The classic warning comes from software diversity. Knight and Leveson tested independently developed program versions and challenged the assumption that development independence guarantees failure independence.³ Reliability engineering names the broader pattern common-cause failure: several components fail because a shared cause reaches them through a coupling mechanism.⁴

The two-part definition matters. “Same model family” is a possible coupling mechanism, not a complete cause. It becomes operationally relevant when a particular stimulus, representation or training-induced blind spot reaches several components through that shared family. Conversely, two different models can still co-fail because they receive the same misleading evidence. Reliability work should record both the initiating condition and the path by which it reaches multiple stages.

This prevents diversity theatre. Changing role names, temperatures or system prompts may increase surface variation while leaving the decisive dependency untouched. Genuine diversity can come from a different model family, but also from a different sensor, evidence source, representation, algorithm, authority boundary or verification method. The design question is not “how many agents?” It is “how many causally distinct ways remain for this top event to occur?”

Four forms of coupling in compound AI

Compound AI introduces at least four distinct ways for errors to travel together.

Routing is therefore not plumbing around the reliability model. It is part of the model. A router that sends uncertain cases to a longer path increases the number of opportunities for failure precisely where the input is already hard. This can create selection bias: the complex route looks bad partly because it receives difficult work, while the difficult work looks manageable because easy cases are removed. Route-conditioned measurement is needed before blaming a component.

flowchart TD
  A["The 99 per cent system that fails"] --> B["The hidden assumption is<br/>independence"]
  B --> C["Model the trajectory, the<br/>shock and the escape"]
  C --> D["Turn the model into an<br/>operating instrument"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#B96D00,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Model the trajectory, the shock and the escape

The practical model needs three layers. A path says which stages a task actually visits. A shock represents a shared condition that changes several stage error probabilities. Containment represents detection, abstention, recovery or compensation that stops a local defect from becoming the top event.

The path, coupling and containment equation

For a task class x, let r be a route and z a state of shared shocks. Let u_i be the probability that stage i produces an uncontained defect after the route, shock state and prior history are known. A useful first model is:

The equation does not abolish dependence. It moves dependence into explicit variables. If failures remain dependent after conditioning, the product must be replaced by a joint model, Bayesian network, copula, empirical replay or simulation. Conditional independence is a testable modelling choice, not a gift from notation.

For the smallest useful implementation, decompose escaped defect probability as u_i = e_i(1 − d_ir_i). Here e_i is the chance of a local defect, d_i is the chance the defect is detected, and r_i is the chance recovery succeeds given detection. This factorisation assumes recovery is attempted only after detection and that the two probabilities are adequately represented by their product. Real systems may need separate branches for safe rejection, human escalation, compensation and unrecoverable partial effects.

The decomposition changes debugging. A high e_i calls for a better component or narrower task. A low d_i calls for observability and independent checking. A low r_i calls for reversible actions, idempotency or a tested runbook. Collapsing all three into “accuracy” makes teams improve the wrong layer.

History h_i is included because compound systems are not memoryless. A model proposal can change the next prompt. A tool error can trigger a retry. A verifier can add a critique that steers the second attempt into a new failure mode. The route may branch after each observation. For short acyclic workflows, enumeration can be exact. For loops, partial failures and changing state, simulation or a dynamic probabilistic model is usually clearer.

A realistic configured example

Consider a synthetic service assistant with three routes. Fifty-five per cent of requests only require an answer. Thirty-five per cent request a refund and cross an effect boundary. Ten per cent are exceptions that should be handed to a person. The refund path uses identity, retrieval, proposal, authority, execution and readback. The answer path stops before authority and execution.

This example exposes another hidden variable: the outcome may be unknown rather than simply failed. Treating timeout as “no effect” and retrying is a modelling error. Treating it as “effect definitely happened” is also wrong. The route needs an explicit unknown-outcome state, a bounded recovery path and evidence that closes the uncertainty. Google’s reliability guidance likewise warns that overload and retry behaviour can create cascading failure rather than local recovery.¹³

Turn the model into an operating instrument

A reliability model is useful only when its inputs come from operating evidence. Component benchmark scores are priors, not final parameters. Route probabilities come from traces. Local defect rates come from task-stratified evaluation. Coupling comes from paired-error analysis and interventions. Containment rates come from injected defects that the system must detect and recover from. NIST’s Generative AI Profile likewise treats measurement, monitoring and risk response as lifecycle activities rather than a one-off benchmark.¹²

Build the first model before deployment with synthetic tasks and an operational twin, but mark every parameter by provenance. “Measured in matched replay”, “estimated from fault injection”, “borrowed from a component benchmark” and “expert assumption” are not interchangeable. Run sensitivity analysis over the assumed values. If the release decision flips across a plausible range, the missing measurement is decision-critical and should be collected before authority expands.

Do not pool traffic too early. A global two per cent error rate can conceal a 0.2 per cent rate on routine answers and a 20 per cent rate on rare irreversible exceptions. Define task classes by the causal demands of the route: read-only versus mutating, reversible versus irreversible, single-source versus evidence synthesis, and ordinary versus exceptional authority. The classes should predict route and consequence, rather than only mirror product categories.

Build the ledger before the calculator

Start with one top event. “The assistant is unreliable” is not testable. “A refund is issued for the wrong amount without being blocked or corrected before settlement” is testable. Then record each task class, route, stage, shared dependency, detector, recovery and effect boundary. The route identifier must be logged before the outcome is known, or analysis will select on success.

A minimal ledger has one row per observed or simulated trajectory. It records task class, selected route, component versions, context and state versions, shock indicators, local defects, detector decisions, recovery actions, effect receipt and verified outcome. The ledger should preserve “not observed” separately from “observed and absent”. Missing readback is not a successful effect. A missing detector signal is not evidence that no defect occurred.

Estimate route weights from the intended workload and from stress workloads. Estimate local escaped-defect rates with confidence intervals rather than point estimates alone. Estimate joint failures for components that are meant to provide redundancy. For rare severe top events, direct frequency may be unavailable; combine structured fault trees, conservative bounds, targeted injection and explicit uncertainty rather than inventing precision.

The calculator is a decision instrument, not a certification engine. A positive result means one design has lower estimated top-event probability under the supplied assumptions and workload. A negative result means the proposed control did not reduce risk under that model, or that its benefit was offset elsewhere. Neither result establishes production reliability without representative traces, calibrated fault injection and independent outcome verification.

Use it comparatively. Clone the configuration, change one mechanism, and keep the workload fixed. Replace the second reviewer with an independent evidence path. Add a typed authority gate. Move low-confidence tasks to human review. Add readback and idempotency. The useful output is not a decimal alone, but the difference in top-event probability and the cause whose contribution changed.

Then reverse the question. Hold the architecture fixed and alter the workload. Increase the share of long routes, raise the common-shock probability, or lower recovery effectiveness during peak load. This reveals cliffs that an average-case estimate hides. A design that looks acceptable only when every uncertain parameter takes its optimistic value does not have a reliability margin.

from __future__ import annotations

from dataclasses import dataclass
from random import Random
from typing import Dict, List


@dataclass(frozen=True)
class Component:
    error: float
    detect: float
    recover: float


@dataclass(frozen=True)
class Route:
    name: str
    weight: float
    steps: List[str]


@dataclass(frozen=True)
class Shock:
    name: str
    probability: float
    affects: List[str]
    fail_probability: float
    detect: float
    recover: float


def contained(rng: Random, detect: float, recover: float) -> bool:
    return rng.random() < detect and rng.random() < recover


def choose_route(rng: Random, routes: List[Route]) -> Route:
    draw = rng.random()
    cumulative = 0.0
    for route in routes:
        cumulative += route.weight
        if draw <= cumulative:
            return route
    return routes[-1]


def simulate(
    components: Dict[str, Component],
    routes: List[Route],
    shocks: List[Shock],
    trials: int = 100_000,
    seed: int = 185,
) -> dict:
    if abs(sum(route.weight for route in routes) - 1.0) > 1e-9:
        raise ValueError("Route weights must sum to one")

    rng = Random(seed)
    failures = 0
    escaped_causes: Dict[str, int] = {}

    for _ in range(trials):
        route = choose_route(rng, routes)
        active_shocks = [s for s in shocks if rng.random() < s.probability]
        top_event = False

        for step in route.steps:
            component = components[step]
            causes: List[tuple[str, float, float, float]] = []

            if rng.random() < component.error:
                causes.append((f"local:{step}", 1.0, component.detect, component.recover))

            for shock in active_shocks:
                if step in shock.affects and rng.random() < shock.fail_probability:
                    causes.append((f"shock:{shock.name}", 1.0, shock.detect, shock.recover))

            for cause, _, detect, recover in causes:
                if not contained(rng, detect, recover):
                    failures += 1
                    escaped_causes[cause] = escaped_causes.get(cause, 0) + 1
                    top_event = True
                    break

            if top_event:
                break

    success = 1.0 - failures / trials
    dominant = max(escaped_causes, key=escaped_causes.get, default="none")
    return {"success": success, "failure": 1.0 - success, "dominant": dominant}

Use fault injection to estimate containment

End-to-end success alone cannot reveal why a system survived. Inject a stale retrieval result, a malformed tool response, a permission revocation, a partial timeout and an inconsistent readback. For each injection, record whether the defect was detected, whether execution was blocked, whether recovery succeeded and whether the verified outcome matched the intent.

Recent multi-agent failure taxonomies distinguish specification and system-design failures, inter-agent misalignment, and task verification or termination failures.⁹ Preliminary MAS-FIRE work extends this direction with systematic fault injection and reports topology-dependent differences in fault containment.¹⁰ A recent agent-reliability agenda likewise argues that a single success score should be decomposed into consistency, robustness, predictability and safety.¹⁵ These proposals are promising but should be treated as early evidence until independently replicated across tasks and architectures.

A useful campaign varies both the fault and the observation point. Inject a bad retrieval result before planning, then inject the same defect after planning as a corrupted tool argument. If only one is caught, the location matters. Inject two defects with the same semantic content but different encodings. If both bypass the verifier, the vulnerability may be conceptual rather than lexical. Inject the same fault under the short and long routes. If containment changes, topology or budget is part of the cause.

Measure graceful outcomes, not just binary completion. The system may block safely, ask for clarification, escalate with a complete case packet, compensate an effect, or stop with unknown outcome. These states have different operational costs and consequences. They should not all be merged into “failure”, but none should be silently counted as success.

The strongest rival explanation

Paired failures do not automatically prove architectural coupling. Hard inputs can make every competent component fail. A stale policy may be correlated with unusual cases. Route changes may coincide with a new customer population. The rival explanation is unobserved task difficulty, not shared implementation.

Test the rival by conditioning on task features and intervening on the suspected coupling. If two reviewers remain correlated after matching on task difficulty, then replace one model, prompt, evidence source or tool path. If correlation falls, the changed dependency was causal. If it does not, the shared difficulty or an unmeasured cause remains. This is why a dependency graph should be paired with controlled substitutions, not treated as proof.

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: The 99 per cent system that fails
  M->>E: Topology decides how errors combine
  E-->>M: compare rival explanations
  E->>D: Appendices
  D-->>Q: revise scope or act

Appendices