Part I: The second-opinion illusion

Two climbers, one map

Multi-model designs often count visible differences: provider, parameter count, architecture, system prompt or persona. Those differences can matter. They can alter priors, decoding behaviour and the set of mistakes a system tends to make. But they do not answer the release question: what could make both paths accept this particular wrong action?

Failure-path independence is the absence of a shared causal route that can carry a decision-relevant defect through both checks. It is narrower than statistical independence over all outputs, and more operational than saying two systems are “diverse”. It asks about the failures that pass a gate, under the actual evidence, task framing, interaction order and authority design.

The lesson predates language models. N-version programming asked separate teams to implement the same specification, then used agreement among versions as protection against software faults. A landmark experiment found that independently developed programs still failed together more often than a simple independence assumption predicted.[1] The accompanying theory identified a reason: some inputs are intrinsically harder, so observing one version fail changes the probability that another fails.[2][3]

This does not prove that diversity is useless. It proves that a component count cannot be substituted for a dependence claim. Diversity may reduce errors while still leaving the residual risk several times larger than an independence calculation suggests.

The hidden common ancestors

Suppose a proposer and verifier produce separate outputs. Draw arrows backwards from each accepted answer. Where do the arrows meet? They may meet at a document retriever, an OCR service, the same policy summary, the same generated test cases, the same benchmark annotation, the same user framing or the same ambiguous requirement. They may also meet in subtler places: related training data, similar post-training preferences or a shared tendency to reward fluent rationales.

A shared ancestor is not automatically dangerous. Both paths may use the same authoritative exchange rate or signed policy because there is only one correct source. The risk appears when the shared node can be wrong, incomplete or adversarially shaped, and neither branch contains an intervention capable of detecting that defect.

Recent work on language models makes the same problem visible at a larger scale. An analysis spanning hundreds of models found that error patterns were correlated, especially among related model families, and that models often converged on the same wrong answer more frequently than a random baseline would suggest.[5] A separate panel study reports that nine model judges can behave like only a few effective independent votes, although this result is preliminary and evaluation-specific.[6]

These findings do not license a universal correlation coefficient for “LLMs”. Dependence changes with task, prompt, evidence, answer space and model family. They do support a design warning: the appearance of a panel is not evidence of a portfolio.

Four ways to look separate while remaining coupled

A verifier can be independent in one sense and dependent in another. Model independence means that a different learned system produces the second judgement. This can diversify learned shortcuts, but it says nothing about shared inputs. Evidence independence means that the second path obtains at least one decisive fact without inheriting the proposer’s representation. This can expose missing or corrupted context, but it may still apply the same mistaken rule.

Procedural independence means that the verifier decomposes the claim differently or uses another oracle. A numerical reconciliation, constraint solver or counterexample search can reject an answer that another generative critique would endorse. Authority independence means that the check has a separately governed ability to block, abstain or escalate. Without it, independent knowledge may exist but never alter the outcome.

These dimensions should not be collapsed. A different model with no authority is an advisory opinion. A deterministic check over the same corrupted extract is procedurally different but evidentially coupled. A human controller with an independent source may still be anchored if the interface presents the model’s conclusion first. The required combination depends on the failure being controlled.

That is why “independent verifier” is incomplete as an architecture label. It needs an object: independent of which cause, for which claim and at which boundary? The more consequential the action, the less acceptable it is to answer with provider names.

flowchart LR
  A["Part I, The second-opinion<br/>illusion"] --> B["Part II, Dependence hides in<br/>averages"]
  B --> C["Part III, Build another causal<br/>path"]
  C --> D["Part IV, Test the path"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#7B4CD8,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Part II: Dependence hides in averages

The wrong answer key

Average accuracy asks how often a system is right over all cases. Verification asks a different question: when the proposer is wrong, how often does the gate still open? A reviewer can be excellent on easy cases and useless exactly where it is needed.

Change only the distribution

A compact diagnostic is the error-dependence amplification:

In the example, D is about 5.98. More revealingly, after the proposer fails, the probability that the verifier also fails is 46.6 per cent, not 7.8 per cent. That conditional quantity belongs in the assurance case.

Do not optimise the diagnostic

D is useful because it challenges an independence baseline. It is not a target to minimise in isolation. Suppose a new verifier becomes dramatically better on easy cases but remains unchanged on hard cases. Its marginal error falls, so the product in the denominator shrinks. D can rise even while the absolute joint failure rate falls. A team reporting only amplification might reject a genuine improvement.

The reverse can also mislead. A verifier that rejects nearly everything may achieve a low conditional acceptance rate, but it has not become a useful collaborator. It has transferred the workload to humans. Report at least four quantities together: proposer error, verifier abstention or rejection burden, undetected joint failure, and conditional miss on proposer failures. Add latency and review cost when the control is operational.

Stratification matters for the same reason. A single conditional miss rate can hide a path that catches arithmetic errors but misses every entitlement error. Group cases by plausible mechanism, not only by business category. “Missing evidence”, “stale world state”, “ambiguous rule”, “tool execution fault” and “adversarial instruction” are more useful strata than a single aggregate called quality.

A dependence metric is an instrument for finding shared causes, not a badge for certifying a pair of models. When it changes, inspect which cases moved and which path property changed.

Agreement is not the same as joint failure

Free-form systems complicate the binary picture. Two reviewers can both be wrong in different ways and therefore disagree, which may still trigger abstention. Conversely, they can give the same wrong answer, or one can accept the other’s wrong proposal despite holding a different internal belief. A dependence test should therefore match the protocol’s actual failure event: same wrong answer, incorrect approval, failure to identify a violated constraint, or unauthorised release.

The large LLM error-correlation study is useful because it examined not only whether models erred, but whether they converged on the same wrong option.[5] That is closer to a majority-vote failure. For a release gate, however, the operational event is simpler: did the verifier block the proposer’s wrong action? Measure the event that reaches authority, not a convenient proxy for disagreement.

The sealed-envelope verifier

Multi-agent debate and self-consistency can improve performance on some reasoning and factuality tasks.[7][8] That is an important positive result. It means repeated sampling, critique and aggregation can locate a better answer. It does not establish that agreement is independent assurance. A method can improve mean accuracy while preserving a high conditional miss rate on a particular failure class.

Evidence on self-correction sharpens the point. Broadly prompted self-feedback is unreliable outside tasks with special structure, while reliable external feedback improves correction.[9] The design question is therefore not “one model or many?” It is “which additional observation or operation can disconfirm the proposal?”

Part III: Build another causal path

Different along the dimensions that can break the error

A verification path has more anatomy than a model endpoint. It begins before inference and ends after the claimed effect. The following dimensions should be recorded separately, because averaging them into one “diversity score” would manufacture precision.

Evidence lineage

Did both paths receive the same retrieved chunks, document extraction, sensor feed or human summary? An independent path may query the authoritative source directly or use a separately generated sample.

Representation and extraction

Did both consume the same OCR, parser, schema mapping or feature transform? Re-running the same model on the same malformed representation cannot recover omitted state.

Question and objective

Did the verifier answer “does this rationale sound right?”, or test a narrower invariant such as amount equality, policy entitlement, conservation, type safety or temporal validity?

Computation or oracle

Did both paths use open-ended generation, or did one invoke a compiler, theorem checker, database constraint, simulator, signed ledger, unit test or domain calculation?

Model and training lineage

Provider, architecture, data lineage and post-training still matter. They are one layer of diversity, not a certificate for the whole path.

Interaction order

Did the verifier commit before seeing the proposal, or inherit its decomposition and confidence? Blind-first work can preserve evidence value.

Authority and receipts

Can the verifier actually block release? Does the system preserve the proposal, check result, source identifiers, policy decision and effect readback?

The verifier should be designed backwards from a plausible shared failure, not forwards from an available second model. Ask what defect must be caught, which observation exposes it and which tool can make that observation with different failure causes.

Verify the claim in slices

A consequential proposal is rarely one indivisible statement. “Release this payment” contains several claims: the requester has the right identity; the supplier and bank account are authorised; the invoice corresponds to delivered goods; the net amount is correct; required approvals exist; the evidence was valid at the decision cut-off; and the action is still permitted now. Asking a second model whether the whole proposal is correct encourages it to repeat the proposer’s narrative.

Decomposition lets each claim meet an appropriate check. Identity can be validated by the identity provider. Amount equality can be computed. Approval thresholds can be evaluated by a policy engine. Document completeness can be checked against a signed manifest. Temporal claims can be replayed against bitemporal records. A model may still be needed to interpret an unusual clause, but its interpretation becomes one typed input rather than the final authority.

This is not defence by bureaucracy. It reduces duplicate cognition. The verifier does not need to recreate every judgement. It needs enough orthogonal coverage to make the dangerous errors observable. One path can verify the arithmetic while another checks entitlement, and the authority kernel can require both receipts only when irreversibility warrants the cost.

The method also clarifies disagreement. If the proposal and amount oracle disagree, the system knows which claim failed. If two broad model rationales disagree, a human first has to discover what they are disagreeing about. Claim decomposition converts vague dissent into a recoverable state transition.

Swap the oracle, rather than only the speaker

This thought experiment blocks two simplistic rules. “Always use another provider” is too weak, because shared evidence and task structure can dominate. “Always use deterministic tools” is also too strong, because tools faithfully enforce their inputs. The design criterion is whether the path introduces a disconfirming constraint that the proposer’s error cannot easily survive.

The payment check, rebuilt

The configured example separates vocabulary that is often collapsed. The model produces a proposal. The ledger supplies world state. The policy engine holds authority. The payment instruction is a typed action. The mismatch, policy decision and payment readback are evidence and receipts. None of these should be called “memory”.

The verification path dossier

The following instrument turns “independent reviewer” into an auditable claim. For each consequential action class, record the proposer path, the verifier path, the suspected common cause, the intervention that should expose it and the empirical test. Leave cells blank when evidence is absent. Missingness is more honest than a reassuring score.

The smallest useful dossier can fit on one page. Start with one high-consequence failure class, such as omitted document state or misapplied amount threshold. Define the proposer failure, define what counts as a verifier miss, create a matched test set with injected instances, and preserve both paths’ source and configuration identifiers. This is enough to move from a naming claim to a causal test.

When unanimity is the attack

An adversary does not need to defeat several models separately if it can shape their common input. A poisoned retrieval item, malicious document annotation or compromised policy summary can fan out to every reviewer. More agents may then amplify the malicious source by producing multiple mutually reinforcing rationales.

A recent N-version coding-agent preprint reports both sides of the story: diverse implementations often reduced observed failure, yet coincident failures exceeded an independence prediction and clustered around ambiguous or difficult items.[11] Because this is a preprint on a particular benchmark, its numerical results should not be generalised. Its experimental shape is valuable: retain the benefit of diversity, then measure the dependence that remains.

stateDiagram-v2
  [*] --> Claim
  state "Two climbers, one map" as Claim
  state "The hidden common ancestors" as Test
  state "Four ways to look separate<br/>while remaining coupled" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Part IV: Test the path

A small dependence lab

The interactive lab below implements the difficulty-mixture mechanism. It assumes proposer and verifier failures are independent within each difficulty stratum, then shows how a mixed workload and a shared-source fault create marginal dependence. The numbers are synthetic. Their purpose is to expose logic that prose can hide.

How to use the lab

Begin with the default mixture. The 90 per cent easy workload hides a hard stratum that creates nearly sixfold joint-risk amplification. Lower the verifier’s hard-case miss rate while leaving proposer rates unchanged. This simulates a check that is specifically better on proposer failure cases. The conditional miss and actual joint risk fall.

Next, set both hard-case rates low but introduce a shared-source fault. This represents a missing page, poisoned retrieval item or common incorrect label that defeats both paths before their own reasoning begins. The amplification rises again. The repair is not necessarily a stronger model. It is an independent source, representation or invariant that can detect the common fault.

The lab encodes three assumptions. Cases belong to an easy or hard stratum. Errors are independent inside each stratum except for the explicit shared-source event. The shared event makes both paths fail. Real data can violate all three assumptions, so the production audit should use observed rows, stratified by meaningful failure classes.

The executable audit

The Python artefact below accepts case-level records. It computes marginal error, joint error, naive independent joint error, amplification, the verifier’s miss rate conditional on proposer failure and per-stratum summaries. Replace the synthetic rows with authorised test data. A browser button above downloads the same code as verification_dependence_audit.py.

from __future__ import annotations

from dataclasses import dataclass, asdict
from typing import Iterable, Sequence
import json


@dataclass(frozen=True)
class Case:
    case_id: str
    stratum: str
    proposer_wrong: bool
    verifier_wrong: bool
    verifier_accepts: bool


@dataclass(frozen=True)
class Metrics:
    n: int
    proposer_error: float
    verifier_error: float
    joint_error: float
    naive_joint: float
    amplification: float | None
    conditional_miss: float | None


def _rate(values: Sequence[bool]) -> float:
    if not values:
        raise ValueError("A rate requires at least one observation")
    return sum(values) / len(values)


def audit(cases: Iterable[Case]) -> dict[str, object]:
    rows = list(cases)
    if not rows:
        raise ValueError("Provide at least one case")
    if len({row.case_id for row in rows}) != len(rows):
        raise ValueError("case_id values must be unique")

    p = _rate([row.proposer_wrong for row in rows])
    v = _rate([row.verifier_wrong for row in rows])
    joint = _rate([
        row.proposer_wrong and row.verifier_wrong for row in rows
    ])
    naive = p * v
    proposer_failures = [row for row in rows if row.proposer_wrong]
    miss = (
        _rate([row.verifier_accepts for row in proposer_failures])
        if proposer_failures else None
    )
    amplification = joint / naive if naive > 0 else None

    overall = Metrics(
        n=len(rows), proposer_error=p, verifier_error=v,
        joint_error=joint, naive_joint=naive,
        amplification=amplification, conditional_miss=miss,
    )

    by_stratum: dict[str, dict[str, object]] = {}
    for name in sorted({row.stratum for row in rows}):
        subset = [row for row in rows if row.stratum == name]
        by_stratum[name] = audit_without_strata(subset)

    return {"overall": asdict(overall), "by_stratum": by_stratum}


def audit_without_strata(rows: Sequence[Case]) -> dict[str, object]:
    p = _rate([row.proposer_wrong for row in rows])
    v = _rate([row.verifier_wrong for row in rows])
    joint = _rate([row.proposer_wrong and row.verifier_wrong for row in rows])
    naive = p * v
    failures = [row for row in rows if row.proposer_wrong]
    return asdict(Metrics(
        n=len(rows), proposer_error=p, verifier_error=v,
        joint_error=joint, naive_joint=naive,
        amplification=(joint / naive if naive > 0 else None),
        conditional_miss=(
            _rate([row.verifier_accepts for row in failures])
            if failures else None
        ),
    ))


if __name__ == "__main__":
    sample = [
        Case("e1", "easy", False, False, False),
        Case("e2", "easy", False, False, False),
        Case("e3", "easy", True, False, False),
        Case("h1", "hard", True, True, True),
        Case("h2", "hard", True, True, True),
        Case("h3", "hard", False, True, False),
    ]
    print(json.dumps(audit(sample), indent=2))

Code caption. The audit tests whether verifier failures coincide with proposer failures more often than marginal rates imply, while preserving the release-relevant acceptance event and failure strata.

The verifier_wrong field measures the verifier’s own answer error. The verifier_accepts field measures the gate failure that matters when the proposer is wrong. They may differ. For example, a verifier can be uncertain rather than wrong yet still accept because the policy threshold is too permissive.

Read the result as a bounded claim

A low observed conditional miss does not mean the verifier is independent in general. It means that, among the proposer failures represented in the tested configuration, the gate usually blocked. Record the denominator. One miss among ten proposer failures and ten among one hundred have the same point estimate but support different confidence. Zero observed misses is especially easy to overread when failures are rare.

A negative result is often more useful than a flattering average. If both paths miss injected stale-state cases, the dossier points to a shared temporal source or absent readback. Repair that path, then rerun the same cases. The experiment becomes an engineering loop rather than a one-time benchmark.

For live monitoring, preserve enough information to distinguish drift from mix change. An increase in joint failure may mean that the verifier degraded, but it may also mean that more difficult cases entered the workload. Track failure strata, path versions and action types. Where labels arrive slowly, use near misses, human overrides and replayed incidents as leading evidence, while keeping them separate from confirmed failures.

Do not let the test become a route to automatic release. A positive result permits the organisation to credit a specific control in a specific assurance argument. It does not remove the need for authority limits, reversibility, incident response or periodic failure injection. Evidence for independence is versioned operating evidence, not a permanent property of an endpoint.

What would strengthen or weaken the proposal?

The strongest boundary

NIST guidance for critical AI systems emphasises validation against intended use and misuse, and the Generative AI Profile calls for independent evaluation and assessment.[12][13] The path framework adds an engineering question to that governance requirement: independent in what causal sense, for which failure, under which configuration and with what receipt?

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: Part I, The second-opinion illusion
  M->>E: Two climbers, one map
  E-->>M: compare rival explanations
  E->>D: Spend verification on a different failure path
  D-->>Q: revise scope or act

Spend verification on a different failure path

Stop asking, “How many models agreed?” Ask, “What new fact or constraint entered because the second path existed?”

If the answer is only another rationale over the same evidence, treat the result as ensemble performance, not independent assurance. It may still improve average quality. It should not silently lower the risk estimate for a consequential release.

If the second path uses independently obtained state, a different representation, an orthogonal invariant or an executable oracle, and if it demonstrates a low conditional miss rate on the proposer’s actual failure classes, then it has earned a bounded verification claim. The claim belongs to that configured path, action class and test distribution. It does not travel automatically to another task or model version.

The decision is to buy disconfirmation, not duplication. Spend the next unit of verification effort on the path that can make the proposal fail for a different reason. Sometimes that will be another model. Often it will be a ledger query, a compiler, a simulator, a held-out source, a blind-first human judgement or a deterministic authority rule.

Two green verdicts can still be useful. One green verdict and one well-designed contradiction test are usually more informative.