The meeting that manufactured confidence

Imagine three agents reviewing an exception to a lending policy. The first is called a credit analyst, the second a risk challenger and the third a policy specialist. Their system prompts differ. Their names, tone and order of speaking differ. Yet all three use the same base model, receive the same retrieved paragraphs and inherit the same mistaken entity match. They debate for four rounds and reach a polished unanimous answer.

The answer is wrong for one simple reason: the disagreement was theatrical. The agents did not possess independent evidence or independent failure modes. They were three conversational surfaces over one correlated error. More turns added reasons, not information.

Now alter one feature. Keep the question, the models and the debate protocol unchanged, but give the policy specialist an independently resolved customer identifier and the current policy version. It initially dissents. The team can improve only if that evidence survives the conversation, is legible to the adjudicator and outweighs the majority's fluent but obsolete rationale. If the dissenter merely yields, debate makes the group worse than a private vote.

This case exposes the paper's answer. Debate is not a source of accuracy by itself. It is an information-conversion procedure. Its opportunity is bounded by correct information already present, or newly acquired, somewhere in the group. Under a fixed information set, its realised value is the number of wrong initial decisions it rescues minus the number of correct initial decisions it corrupts. The relevant baseline is therefore not one hurried answer. It is the best non-interactive method available under the same model, token, tool, latency and evidence budget.

This distinction changes architecture. Agent count is no longer a proxy for epistemic strength. Persona diversity is no longer accepted as independence. Consensus is no longer accepted as validation. A debate system earns its place only when a benchmark shows that interaction converts recoverable disagreement more efficiently than voting, verification or additional evidence.

Disagreement is not diversity

A group can disagree for weak reasons and agree for weak reasons. One model sampled at different temperatures may produce different wording around the same latent shortcut. Conversely, two systems may produce the same answer from genuinely independent evidence. Observable disagreement is therefore neither necessary nor sufficient for useful diversity.

The ensemble literature provides the durable intuition. A collection improves when its members are individually competent and their errors are not too correlated. Breiman's random-forest analysis made strength and correlation joint determinants of generalisation. Earlier ambiguity decompositions showed how individual error and ensemble diversity trade against each other. Those results do not transfer mechanically to language-model conversations, but they identify the right object: diversity matters through the joint distribution of errors, not through surface variety.

Thought experiment 1: three identical votes

Suppose three agents vote two to one for option A. In the first world, each independently inspected a different sensor and one sensor was noisy. In the second, all agents copied the same incorrect database row, but one happened to hedge. In the third, the minority agent holds a verified document while the majority inferred from convention. The vote tally is identical. The epistemic situation is not.

Only the third world creates a strong case for structured cross-examination. The first may need simple majority voting because independent noise averages out. The second needs source correction, not conversation. A debate policy that reacts to the 2-1 split alone cannot distinguish these worlds.

Error-relevant diversity

For decision engineering, diversity should be defined by what it can rescue. Let an agent's error indicator be E_i=1 when agent i is wrong and E_i=0 when it is correct. Pairwise error correlation describes whether two agents fail on the same cases. The double-fault rate, P(E_i=1,E_j=1), is often more operationally legible because it directly counts shared misses.

Neither measure is enough on its own. An average correlation can hide one highly redundant cluster and one valuable specialist. A better local question is: when the current set is wrong, how often is this candidate agent correct? For candidate j and current team S, define marginal rescue coverage as P(E_j = 0 ∧ E_i = 1 for every i ∈ S). This quantity asks whether the new member contributes answers unavailable to the existing team.

Different system prompts can increase linguistic diversity without changing marginal rescue coverage. Heterogeneous models may still share training corpora, benchmark contamination or a common retriever. Different tools can still read one stale source. Independence must be tested on labelled cases, especially the high-consequence slice, rather than inferred from names or architecture diagrams.

Interaction itself can destroy the property it needs. In a human estimation experiment, mild social influence reduced diversity without improving collective error. Language models are not human participants, so this is not direct evidence about agents. It is a useful warning about mechanism: once initial estimates are exposed, later agreement no longer tells us how many independent signals existed.

Preserve an independence window. Collect each agent's answer, confidence, evidence and decisive reason before any exchange. Freeze that record. Only then allow rebuttal. This makes initial diversity measurable and prevents the final transcript from overwriting the evidence needed to judge whether debate helped.

The rescue and corruption mechanism

Debate can be evaluated without pretending that every token in the transcript is faithful reasoning. Begin with four observable events on a labelled dataset. Let Y be the correct answer, X_i agent i's private answer, M the private majority answer and F the final answer after debate.

The private-vote accuracy is V=P(M=Y). The oracle coverage is O=P(∃ i:X_i=Y), the fraction of cases where at least one participant initially knows the answer. For an odd-sized binary team, G=O-V is the recoverable gap: cases the majority misses even though a correct minority exists.

Now measure selection rescue, R=P(M≠Y, ∃i:X_i=Y, F=Y), novel success, S=P(∀i:X_i≠Y, F=Y), and corruption, C=P(M=Y,F≠Y). The general accounting identity is:

D = V + R + S - C.

For fixed-information debate, where interaction cannot create a correct answer when every private answer is wrong, S=0. The identity reduces to D=V+R-C. More eloquent transcripts do not appear. More agreement does not appear. Selection improves only when rescue exceeds corruption. Conversion efficiency, η=R/G, asks what share of recoverable cases debate captures. Corruption rate, κ=C/V, asks how often interaction destroys a correct private majority. When G=0, there is no correct minority to select; any gain must be attributed separately to synthesis, new evidence or extra computation.

Three routes to a better final answer

The accounting above describes selection: at least one participant has a correct complete answer and interaction helps the system choose it. Selection is the cleanest debate claim because the initial records reveal the opportunity. A correct minority can point to a decisive premise, expose a contradiction or demonstrate a calculation. The adjudicator's job is to recognise the stronger case without allowing popularity to substitute for evidence.

A second route is synthesis. No private answer is wholly correct, but different agents hold compatible partial structures. One identifies the applicable rule, another resolves the entity and a third notices a temporal exception. Their combination produces an answer unavailable in any initial message. Synthesis is real, but it should not be hidden inside selection rescue. It belongs to S=P(O=0,F=Y), the probability that everyone initially misses while the final system succeeds. Then compare debate with a single model given the same combined evidence and additional compute. Otherwise the gain may belong to context assembly rather than interaction.

The third route is acquisition. During the exchange, an agent retrieves a new document, executes code, asks a clarifying question or observes an environment change. The final answer improves because the information set changed. This can be an excellent system design, but the causal treatment is the tool or evidence call. A fair baseline gives a non-debate controller the same acquisition opportunity.

These routes imply different architectures. Selection needs private commitments and reliable adjudication. Synthesis needs typed intermediate claims and compatibility checks. Acquisition needs tool authority, source provenance and postcondition verification. Calling all three “debate” obscures the mechanism and makes results hard to transfer. A benchmark should name which route produced each corrected case.

A minimal worked example

Consider three agents on eight binary cases. Each agent is correct on four cases, so individual accuracy is identical. Private majority voting solves four cases. An oracle that selects any correct initial answer could solve seven. Debate rescues cases 5 and 6, fails to rescue case 7 and corrupts case 2.

The final gain is one case, not three. A report that counts only successful corrections would overstate the value by ignoring corruption. A report that compares the final result with one agent would also misattribute gains already available from private voting.

The oracle is diagnostic, not deployable. It tells us how much correct information the group contains before interaction. A large oracle gap creates an opportunity for adjudication. A small gap means the agents mostly know and miss the same things. In that regime, debate has little raw material. Add a new model, tool, source or decomposition that changes the joint errors before adding rounds.

Thought experiment 2: hold the answers constant, change the interaction

Freeze the private answers and evidence from the eight-case example. Run three protocols. Protocol A reveals only the vote. Protocol B allows unrestricted persuasion. Protocol C requires each challenge to name a disputed claim, attach evidence and state what observation would reverse the position.

Protocol A cannot exploit a correct minority beyond voting. Protocol B may rescue case 5, but it may also corrupt case 2 because confidence and verbosity become unmeasured channels. Protocol C gives the adjudicator a falsifiable object. The answer distribution has not changed; only the conversion mechanism has. This is why evaluating final accuracy alone is insufficient. We need rescue and corruption by protocol, plus evidence quality and cost.

A recent formal analysis models certain debate updates as a martingale: without a directional mechanism that favours correct signals, exchanging beliefs has no expected tendency towards truth. The assumptions are restrictive, but the negative control is valuable. It asks researchers to identify the asymmetry that makes correct evidence more persuasive than error. “The agents discussed it” is not such an asymmetry.

flowchart LR
  A["The meeting that manufactured<br/>confidence"] --> B["Disagreement is not diversity"]
  B --> C["The rescue and corruption<br/>mechanism"]
  C --> D["The baseline problem"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#7B4CD8,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

The baseline problem

A foundational debate proposal asked whether opposing agents could help a weaker judge recover information that would otherwise be hidden. Controlled studies have since shown promising cases. Debate has helped non-expert judges on reading-comprehension tasks with information asymmetry, and human-written debates have outperformed one-sided consultancy in some settings. Multi-agent language-model studies have also reported gains on mathematics, factuality and strategic reasoning.

These results establish possibility, not a universal architecture rule. The positive studies also differ in what debate means. Du and colleagues used repeated proposal and response among model instances. ReConcile combined heterogeneous models, discussion and confidence-weighted voting. Scalable-oversight experiments constructed a judge who lacked information available to the debaters. A result in one setting does not automatically support free-form peer discussion in another.

The negative and mixed studies vary just as much. The “Debate or Vote” analysis found that majority voting explained most gains in its tested configurations. Controlled logical-reasoning work found that initial competence and group diversity dominated several presentational choices, while excessive disagreement and majority pressure could hurt. A recent team study reported that groups often failed to exploit an answer already known by their strongest member, with compromise rather than ignorance as the central loss. These are different failure modes, not one verdict.

It would therefore be misleading to average every protocol into a single effect called “multi-agent debate”. The transferable question is mechanistic: where did a correct signal originate, how did interaction change its influence and what competing use of compute was denied? A multi-agent system usually spends more calls, tokens and wall-clock time than a single pass. The extra budget can be used in several ways: sample independent answers, retrieve another source, execute a calculation, run a verifier, ask one model to critique itself or invoke a stronger model. The causal question is whether interaction adds value beyond the best alternative use of the same budget.

A broader field review on this site surveys recent agent research as a whole. The present article narrows its observations about shared blind spots and matched compute into one causal test: whether interaction converts correct minority information better than a non-interactive alternative.

Self-consistency is the first serious baseline. It samples multiple reasoning paths and aggregates answers without allowing them to influence one another. Simple multi-agent sampling and voting can also improve with more instances. These methods capture variance reduction and answer search while preserving independence. If debate only matches them, the conversation layer has not earned causal credit.

Heterogeneous private voting is the second baseline. ReConcile compared diverse model families, same-model agents and paraphrased agents, then combined discussion with confidence-weighted voting. Its results suggest that model diversity can matter. Yet a clean attribution still requires a private heterogeneous vote before discussion. Otherwise model complementarity and conversational conversion remain entangled.

Verification is the third baseline. Many reasoning tasks have executable checks, source-grounded claim tests, constraints or domain rules. A deterministic verifier can be cheaper and less socially fragile than free-form debate. For questions without a complete verifier, targeted retrieval or a process checker may still settle the disputed step.

A study comparing many debate protocols with majority voting found that voting accounted for most observed gains across its benchmarks. Another matched-budget study reported that competitive and consensus-seeking debate underperformed single-agent baselines, while a deliberately collaborative truth-seeking protocol recovered gains. These findings are not contradictory. They imply that protocol-specific conversion must be demonstrated; generic interaction should receive no prior credit.

Why “same number of agents” is not matched compute

Three agents debating for three rounds may generate nine messages plus a judge call. Three agents voting generate three messages. Equal agent count therefore conceals unequal inference. Matching only tokens also misses parallelism, context length, tool cost and model tier. The baseline contract should match the resources that matter to the deployment: total model spend, maximum sequential latency, permitted evidence access and model quality.

There is no single universally fair match. A latency-constrained service may compare equal wall-clock budgets. An offline investigation may compare equal monetary cost. A safety evaluation may compare equal access to privileged evidence. The chosen match must be declared before results are inspected.

The judge is part of the treatment

Debate does not end when the agents stop talking. A final judge, human or model, converts the transcript into an answer. Position bias, verbosity preference, model self-preference and confidence can change that decision. Work on divergent multi-agent debate has reported that model judges may be unfair when comparing different model families. Scalable-oversight studies show that debate can help weak judges in deliberately asymmetric information settings, but that benefit depends on adversarial evidence being accessible and legible.

Report judge performance separately. Give the same judge shuffled transcripts, anonymised agent identities and controlled answer lengths. Include a “private evidence only” arm. If the judge succeeds without dialogue, the benefit belongs to evidence presentation rather than debate. If shuffling speakers changes outcomes, the protocol has a position channel. If the judge follows the longest answer, the system is measuring rhetoric.

A benchmark that can reject debate

A useful debate benchmark should be designed to make the preferred system lose. That requires more than an accuracy column. It needs a case-level record of private answers, evidence, interaction and outcome, plus interventions that isolate the supposed mechanism.

The proposed benchmark has six stages.

1. Construct a labelled case set with consequential slices. Record difficulty, ambiguity, source freshness, tool dependence and error severity. Separate ordinary cases from cases where a plausible majority can share a blind spot.
2. Freeze private commitments. Every participant produces an answer, calibrated confidence, evidence references and one decisive claim before seeing any peer output.
3. Measure the opportunity. Calculate individual accuracy, majority accuracy, oracle coverage, pairwise double faults, marginal rescue coverage and recoverable gap.
4. Run matched arms. Compare private vote, self-consistency, heterogeneous vote, critique, retrieval or verification, and debate under declared cost and latency budgets.
5. Intervene on the mechanism. Shuffle speaker order, hide confidence, cap message length, remove decisive evidence, swap the judge and inject a correct minority. A causal claim should predict which intervention changes rescue.
6. Account for outcomes. Report rescue and corruption by severity, abstention, cost, latency and evidence quality. A one-point gain that introduces a rare catastrophic corruption may fail the operating decision.

The benchmark's primary estimand is not “Does debate beat one answer?” It is: At a matched budget, how much of the recoverable gap does interaction convert, and what correct mass does it destroy?

Pre-registered rejection conditions

Reject the debate topology for the tested scope when any of four conditions holds. First, the oracle gap is too small to support the required improvement. Second, debate does not beat the best matched non-interactive arm across repeated runs. Third, corruption exceeds the agreed severity-weighted tolerance. Fourth, gains disappear when agent names, order or verbosity are controlled, indicating a rhetorical rather than evidential mechanism.

These conditions are intentionally harsh. Architecture should not be awarded for producing an impressive transcript. It should be selected because a discriminating test shows that the interaction channel changes decisions in the intended direction.

Statistical discipline for stochastic teams

The labelled case, not the message, is the primary unit of analysis. Ten rounds on one problem do not create ten independent observations. Run every arm on the same cases, repeat stochastic seeds and use paired intervals for differences in accuracy, rescue and corruption. A paired bootstrap over cases is often more informative than separate confidence intervals because it preserves which system won on which item.

Report distributions, not only means. Debate may improve average accuracy while creating a long tail of costly corruption. Break results down by task difficulty, initial vote pattern, evidence source, model combination and consequence class. For each slice, publish the number of cases and uncertainty. Small apparent gains on rare subsets are especially vulnerable to selection after results are seen.

Protocol choice, judge choice, stopping rule and baseline budget should be fixed before the final evaluation set is opened. Development can explore freely on a separate set. The release result should then answer a pre-declared decision: for example, “Adopt conditional debate only if severity-weighted net rescue exceeds two percentage points and the lower confidence bound remains positive against the verifier baseline.” A threshold converts a benchmark from a demonstration into an operating test.

Executable artefact: correlated-error debate simulator

The simulator below makes one relationship visible. It generates exchangeable agent errors with chosen individual accuracy and positive intraclass correlation. It then applies an abstract debate layer that can rescue a wrong majority when at least one agent is correct, or corrupt a correct majority. The output separates majority gain, oracle opportunity, rescue and corruption.

The positive preset represents competent agents with mostly independent errors and a protocol that converts evidence well. The negative preset holds individual accuracy constant but increases shared error and weakens conversion. Expected behaviour: the positive case should beat voting; the correlated case should expose a small oracle gap and may lose after corruption.

The simulator shows why adding agents can disappoint even when each member is competent. With independent errors, majority voting can rise well above individual accuracy. As correlation increases, the vote approaches the individual member. Oracle coverage also contracts because the whole team misses the same cases. Debate cannot recover information that is absent.

The rescue slider represents every mechanism the abstract model omits: evidence quality, challenge design, judge competence and protocol incentives. Raising it without evidence is wishful thinking. In an empirical benchmark, rescue is measured, not assumed. The corruption slider prevents a one-sided story by representing conformity, persuasive error and judge failure.

stateDiagram-v2
  [*] --> Claim
  state "Thought experiment 1: three<br/>identical votes" as Claim
  state "Figure 1. The same 2-1 vote<br/>can encode three different<br/>evidence structures" as Test
  state "Error-relevant diversity" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

A production-shaped decision

Consider a synthetic policy-exception triage system. Each case contains an application, identity-resolved customer data, a policy corpus with effective dates and a calculated affordability result. The system recommends “standard”, “exception review” or “insufficient evidence”. A human retains decision authority. The risk is asymmetric: wrongly clearing a true exception is more serious than sending an ordinary case for review.

The team evaluates 2,000 synthetic and expert-labelled cases, including 300 high-consequence exceptions. The numbers below are a worked scenario, not a deployment result. All model arms use the same model family unless explicitly heterogeneous. Costs are normalised to one strong single pass.

The persona debate beats private voting by 0.2 percentage points, but it loses to the cheaper verifier. That result should stop a default multi-agent rollout. The error analysis shows that most shared misses arise from stale policy retrieval and effective-date mistakes. Conversation is an indirect treatment. A bitemporal retriever and deterministic policy check directly address the failure.

Debate remains useful on a narrow residual slice. Some cases contain two valid documents, conflicting identity evidence or a policy clause whose application depends on a factual distinction. Here heterogeneous evidence roles create an oracle gap: one agent checks identity, one checks temporal entitlement and one recomputes the rule. They commit privately. Debate opens only when answers differ and the consequence threshold is crossed.

The exchange is typed. A challenge names the disputed proposition, the evidence receipt, the applicable policy version and the observation that would resolve it. The final adjudicator can accept one claim, request a tool call, abstain or route to a human. The language model proposes; deterministic controls enforce evidence access, identity and effect limits.

The production design is therefore conditional debate, not a permanent debating society. Preserve private evidence, invoke interaction on cases with recoverable disagreement, verify decisive claims and stop when the marginal information value falls below cost or consequence thresholds.

Authority, stopping and the decision receipt

Interaction should not expand authority. Each agent receives the minimum read permissions and tool capabilities required for its evidence role. The adjudicator may recommend a disposition, but it cannot silently inherit the ability to approve, communicate or execute it. A human or deterministic workflow retains the decision right defined for the case.

Every debate instance should end with a compact decision receipt rather than the transcript alone. The receipt records private answers, the disputed propositions, evidence introduced after the independence window, verified and unresolved claims, the selected disposition, abstention reason, cost and the identity of the accountable reviewer. This structure supports replay without treating the model's prose as authoritative state.

Stopping is an explicit control. End when a decisive claim is verified, when all remaining disagreement is value judgement reserved for a person, when the maximum sequential budget is reached or when another round repeats existing evidence. Do not use consensus as the stopping condition. Consensus can arrive because the correct minority was persuaded away. The system should also preserve an “unknown outcome” state when a tool call or external effect cannot be verified, rather than allowing continued discussion to manufacture closure.

The debate eligibility instrument

Four gates turn the analysis into an architecture decision. The opportunity gate asks whether the oracle gap is material and comes from complementary failures. The conversion gate asks whether rescue exceeds corruption under matched compute. The evidence gate asks whether disputed claims can be checked rather than merely argued. The operating gate asks whether the gain survives latency, cost, privacy, auditability and severity-weighted risk.

Where the mechanism breaks

The proposed account has clear boundaries. It is strongest for tasks with labelled or adjudicable outcomes, separable private answers and a meaningful notion of correct evidence. It becomes harder for open-ended design, negotiation, scientific hypothesis generation and value-laden decisions where no stable answer key exists. In those domains, debate may increase option coverage or reveal assumptions even when “accuracy” is undefined. The benchmark must then measure novelty, constraint satisfaction, downstream utility and human decision quality rather than force a truth label.

Pairwise correlation is also an incomplete description of language-model dependence. Errors can cluster by topic, prompt, language, evidence source and temporal state. Teams may be independent on easy cases and perfectly correlated on rare consequential ones. Report slice-level joint errors and worst-group performance. A global average can conceal the exact regime that motivated the architecture.

Oracle coverage assumes one initial answer is correct. Some interactions genuinely create new solutions by composing partial insights, executing a new tool call or reformulating the problem. Those cases should be recorded as synthesis or evidence acquisition, not rescue. Add a fifth event: “no initial agent correct, final answer correct”. Then test whether the same gain appears when one agent receives equivalent extra compute without dialogue.

Finally, correctness is not the only harm. Debate can expose private context across agents, multiply prompt-injection surfaces, create long unreviewable transcripts and blur who introduced a claim. A higher benchmark score does not authorise broader action. Identity, permissions, typed tools, evidence lineage and human decision rights remain separate controls.

What would weaken the thesis?

The paper's thesis would be weakened by repeated evidence that same-model, same-context, unconstrained debate reliably beats compute-matched private sampling, independent critique and verification across tasks, judges and model families, while maintaining low corruption. That result would suggest interaction generates a broadly useful internal search dynamic even without measurable initial independence.

Current evidence does not justify assuming that result. Some protocols and information-asymmetry settings show real gains. Other controlled studies attribute most gains to voting, expose consensus failures or show teams suppressing their strongest member. The defensible position is conditional: measure the opportunity, isolate the conversion mechanism and select the simplest winning treatment.