Abstract
Calling the same language model three times and asking the outputs to debate can improve an answer. The mechanism is less obvious than the label. The improvement may come from seeing more samples, using more tokens, adding critique prompts or selecting among candidates. It may come from independent information. It may also disappear when a strong single-agent method receives the same budget.
This paper tests a falsifiable thesis: multi-agent gains largely disappear when compute, prompts and sampling are matched to strong single-agent baselines, except where participants access genuinely different evidence, tools, models or governed roles. The word “largely” refers to a pattern across comparable tasks, not a claim that every reported result disappears. The thesis fails if dialogue itself delivers stable gains over matched self-consistency, sequential critique and ensemble baselines across task families and independent implementations.
The evidence does not justify one universal verdict. Early debate studies report improvements in reasoning and factuality. Controlled comparisons find that debate does not reliably beat self-consistency or ensembling and can be sensitive to protocol choices. Later work frames debate as test-time scaling and identifies conditional advantages by difficulty, capability and task. Scalable-oversight experiments find benefits under some information asymmetries, while broader tasks give mixed results. Role-based systems show useful software and workflow decompositions, but often compare a multi-call system with a weaker or cheaper baseline.
The practical conclusion is narrower than “use one agent” or “use many”. Add another agent only when it contributes a named source of non-redundant evidence, capability or control. Otherwise compare the design with the best use of the same budget. In a bank, agent count is not a governance mechanism. Tool permissions, information boundaries, independent verification and human authority must be explicit.
1. Research question and falsifiable thesis
The research question is:
Under equal inference and interaction budgets, when does multi-agent collaboration provide value beyond repeated sampling, sequential critique or model ensembling?
The thesis has two parts. First, homogeneous debate often functions as a test-time sampling and revision procedure. Its gain should therefore be compared with a single controller that can generate several independent candidates, critique them and select one under the same budget. Second, collaboration has a stronger basis when agents hold different usable information: distinct documents, tools, model inductive biases, task expertise or permissions.
The thesis would be falsified by a preregistered body of studies showing that, after model calls, tokens, prompts, context, tools and selection rules are matched, multi-agent dialogue consistently improves correctness, calibration or control performance across tasks. It would also be weakened if homogeneous agents develop stable, complementary hypotheses without explicit diversity and outperform equally diverse non-dialogue ensembles.
Three observations would support it:
- gains shrink when the single-agent baseline receives equal samples and revision rounds;
- gains grow when agents obtain independent evidence or use different models and tools;
- communication that collapses diversity too early removes the gain.
Improving Factuality and Reasoning through Multiagent Debate provides an influential positive result. Should we be going MAD? reports that studied debate strategies do not reliably outperform self-consistency and ensembling, although tuning matters. Revisiting Multi-Agent Debate as Test-Time Scaling finds conditional rather than uniform effectiveness. Demystifying Multi-Agent Debate reports that vanilla debate can underperform majority vote and studies diversity and confidence interventions. Together, these sources make baseline design the central question.
2. Search and inclusion method
The structured search was completed on 22 July 2026. Discovery sources were arXiv, ACL Anthology, official conference proceedings and author repositories linked from canonical papers. Query families were: “large language model multi-agent debate”, “multi-agent debate self-consistency matched compute”, “LLM agent collaboration role team”, “heterogeneous model ensemble”, “scalable oversight debate”, “multi-agent communication sparse” and “multi-agent software development evaluation”.
Inclusion required one of the following:
- a direct empirical comparison of multi-agent discussion, debate or collaboration;
- a strong single-agent or ensemble baseline that can explain the gain;
- a framework paper with evaluated role, tool or information separation;
- a benchmark for social or task-oriented agent interaction;
- an information-asymmetric oversight experiment.
Surveys, opinion pieces and product announcements were excluded from the evidence table. Framework papers without a reproducible evaluation were used only to describe architecture. Duplicate versions were collapsed to the latest canonical record. The stopping rule required positive, negative and conditional debate results; strong sampling baselines; role-based systems; heterogeneous aggregation, and evidence from at least five independent groups.
The included studies cannot support a pooled accuracy difference. They use arithmetic, factuality, general knowledge, coding, social interaction, software workflows and extractive question answering. Agent count, model, prompt, round count, judge and token budget vary. The synthesis therefore compares effect direction and experimental control.
| Evidence cluster | Representative sources | Question it can answer | Important confounder |
|---|---|---|---|
| homogeneous debate | Du et al.; Should we be going MAD? | does dialogue improve selected tasks? | more calls, tokens and tuning |
| test-time sampling | self-consistency; More Agents Is All You Need | do independent samples improve selection? | selection oracle and sample count |
| heterogeneous aggregation | LLM-Blender; Mixture-of-Agents | can different models contribute complementary outputs? | larger total compute and model access |
| information-asymmetric debate | scalable oversight; persuasive debaters | can a judge use adversarially presented evidence? | simulated judge and assigned positions |
| role-based teams | CAMEL, AutoGen, MetaGPT, ChatDev | can roles and tools organise work? | end-to-end baseline and evaluator strength |
| social environments | SOTOPIA, AgentVerse | how do agents interact in open-ended settings? | construct validity of simulated social behaviour |
3. Four mechanisms hidden behind “multi-agent”
Multi-agent systems are often grouped by interface rather than causal mechanism. A transcript with speaker names may implement at least four different ideas.
The first mechanism is sampling. Independent calls produce different reasoning paths. A majority or judge can select a better answer. Self-Consistency Improves Chain of Thought Reasoning demonstrates the value of sampling multiple reasoning paths without assigning them social roles. More Agents Is All You Need studies the scaling effect of multiple agent samples. A debate result must separate dialogue from this base effect.
The second mechanism is revision. One output is exposed to criticism and rewritten. The critic can be called another agent, but a single orchestrator can also run proposal, critique and revision prompts. The relevant comparison is not one unreviewed answer.
The third mechanism is diversity. Different models may make different errors. Agents may receive separate documents or use specialised tools. LLM-Blender ranks and fuses outputs from multiple language models. Mixture-of-Agents uses outputs of several models across layers. Their gains are evidence for aggregation, not necessarily for debate.
The fourth mechanism is decomposition. CAMEL, AutoGen, MetaGPT and ChatDev organise agents by role and workflow. These systems explore a valuable software pattern. To establish a multi-agent advantage, evaluation must compare them with a single controller using the same role prompts, tools, context and budget.
A label is not a mechanism. Architecture review should name which mechanism is expected to help and which baseline could reproduce it more simply.
4. The matched-budget baseline
A fair comparison defines a budget before running the systems. Monetary cost alone is insufficient because provider prices change and latency or tool risk can dominate. A useful budget vector includes model calls, input and output tokens, wall-clock time, tool actions, retrieved evidence, context capacity and human judging effort.
Prompt content must also be normalised. Giving agents distinct expert prompts while the baseline receives a generic prompt tests role prompting plus collaboration. That can be a legitimate system comparison, but it does not isolate dialogue. A matched single controller can generate the same role perspectives sequentially before selection.
Context differs by topology. In an all-to-all debate, each agent may read every prior response. A single baseline should receive the same candidate evidence, although not necessarily the speaker labels. Selection rules matter too. If the multi-agent system uses a strong final judge and the baseline uses majority vote, the judge may explain the gain.
Repeated trials are necessary because language-model sampling is stochastic. Reporting the best prompt or protocol after extensive tuning creates unequal optimisation effort. Protocol hyperparameters should be tuned on a development set with equal search budgets. Final tasks and seeds should remain held out.
| Budget dimension | What to match | Why a mismatch misleads |
|---|---|---|
| model invocations | total calls by model class | more samples increase chance of a correct candidate |
| generated tokens | total output and critique tokens | longer deliberation provides more computation |
| input context | retrieved and peer content | more evidence can explain improvement |
| tool operations | searches, executions and database reads | external computation may dominate reasoning |
| latency | serial and parallel wall time | parallel agents trade infrastructure for response time |
| tuning effort | prompt and protocol search | a tuned debate can beat an untuned baseline |
| selector | judge, vote or verifier | better selection can be the real method |
The correct report can show both system performance and mechanism performance. A buyer may reasonably choose a more expensive system because it works better. A research claim that “debate causes the gain” requires the stricter comparison.
5. What the early debate evidence shows
The 2023 multiagent debate paper by Du et al. reports that multiple model instances proposing and debating answers can improve factuality and reasoning on studied tasks. The contribution was conceptually important: model outputs could become inputs to other model instances, allowing disagreement and revision rather than independent voting.
The result does not imply that conversational exchange is always the active ingredient. More agents create more initial samples. Multiple rounds create more inference. A final answer after exposure to peers receives information unavailable to a one-shot baseline. Those features should be unpacked.
ReConcile proposes a round-table style framework in which diverse model agents discuss and seek consensus. ChatEval applies multi-agent debate to language-model evaluation. These studies expand the use of interaction beyond arithmetic or factual questions. They also introduce another layer: the system may use models both as problem solvers and judges.
When the judge has no independent oracle, consensus can be wrong. Models trained on overlapping data can share a misconception. Seeing a confident peer can cause a correct minority to revise towards an incorrect majority. A transcript can become more coherent while evidential diversity falls.
The key observable is not how many messages were exchanged. It is whether the correct hypothesis or decisive evidence existed, whether communication preserved it and whether the selector recognised it. Debate can fail because no agent produced the right candidate, because a correct candidate was suppressed or because the final judge chose poorly.
6. Controlled and contrary results
Should we be going MAD? directly compares debate strategies with self-consistency and multiple reasoning paths. The authors report that the tested debate systems do not reliably outperform those alternatives, while protocol tuning can matter. This is strong contrary evidence to a broad claim that debate is inherently superior.
Revisiting Multi-Agent Debate as Test-Time Scaling treats debate as one form of additional inference-time computation. Its reported results are conditional. On mathematical reasoning, advantages over self-agent scaling are limited overall and vary with difficulty and model capability. On safety tasks, collaborative refinement can increase vulnerability under some settings, while diversity can help under others.
Demystifying Multi-Agent Debate reports that vanilla debate can underperform simple majority vote and develops a theoretical and empirical account based on initial diversity and confidence. The paper’s interventions aim to preserve diverse candidates and weight updates more rationally. The contribution supports a mechanism claim: independent candidate coverage and calibrated evidence matter more than the theatrical form of debate.
These sources do not prove that dialogue has no value. They show that its value is conditional and comparison-sensitive. A difficult task may benefit from iterative challenge. A weak model may improve after seeing a correct peer. A judge may combine partial evidence across answers. The same protocol can also spread a shared error.
An enterprise architecture should not assume that a panel of identical model instances constitutes independent assurance. Correlated errors remain correlated. A second role prompt may change style more than evidence. Independence should be demonstrated through error overlap, source overlap or intervention, not inferred from speaker names.
7. Information diversity is the scarce resource
Agent diversity is useful only when it changes the information available to the decision. Different names or personas do not guarantee different beliefs. Even different models can be highly correlated if they share training sources and prompts.
Let each agent begin with an evidence set. Collaboration can help when the union contains decisive evidence that no single participant receives and the communication process transmits it without excessive distortion. It can also help when one model recognises an error pattern another misses. If every evidence set is identical, the remaining diversity comes from stochastic sampling and model differences.
The following measures would make the idea testable:
- source overlap between agents;
- disagreement before communication;
- error correlation on held-out tasks;
- fraction of final answers using evidence introduced by another agent;
- survival rate of correct minority candidates;
- marginal gain from each additional agent under fixed total budget.
Model heterogeneity can increase diversity, as LLM-Blender and Mixture-of-Agents suggest, but it introduces selection and operational cost. Information heterogeneity can be designed more directly. One agent can inspect transaction evidence, another policy text and a third system logs. That decomposition should preserve provenance so the final decision can cite each contribution.
Tool heterogeneity can also be genuine. A code executor can verify arithmetic. A retrieval service can locate authoritative documents. A rules engine can apply deterministic eligibility logic. Calling all of these “agents” is optional. What matters is that each component adds a distinct, testable signal.
8. Debate under information asymmetry
Debate has a different purpose when the judge cannot inspect all evidence directly. The goal is not simply to improve a model’s answer. It is to help a weaker judge identify the better claim.
On Scalable Oversight with Weak LLMs Judging Strong LLMs compares debate, consultancy and direct question answering across several task types and information asymmetries. The reported result is not uniform. Debate performs well relative to consultancy in the studied setups, while comparison with direct answering varies by task.
Debating with More Persuasive LLMs Leads to More Truthful Answers studies whether stronger debaters can help judges select truthful answers in an information-asymmetric setting. This line of work is closer to adversarial evidence presentation than to a cooperative team of identical solvers.
The mechanism depends on contestability. Opposing participants must be able to expose unsupported claims. The judge must be able to verify cited exhibits. If agents can invent evidence and the judge cannot check it, rhetorical strength can dominate truth. If both agents see the same compressed summary, the information asymmetry has not been addressed.
This matters in banking review. A proposer and challenger can surface evidence for and against a model-risk approval, but they should not argue from hidden chain of thought. They should cite controlled artefacts: test results, policy clauses, data lineage and unresolved exceptions. The human decision maker needs access to those artefacts.
9. Role-based collaboration is workflow design
Role-based agent frameworks replace an undifferentiated loop with participants such as planner, developer, reviewer and manager. CAMEL explores role-playing between language-model agents. AutoGen provides a framework for conversational agents with tools and human participation. MetaGPT encodes standardised operating procedures into a software team. ChatDev organises a virtual software company through communicative roles. AgentVerse supports collaborative and emergent agent behaviours.
The architectural value is legibility. Each role can have a smaller prompt, tool set and output contract. A reviewer can receive the proposal without the proposer’s writable tools. A planner can decompose work without production access. This is a control benefit even when it does not improve benchmark accuracy.
The same workflow can often run through one model instance called several times with different contexts. Whether to call the roles “agents” is secondary. The research question is whether parallel or interactive role separation changes outcomes after budget and tools are matched.
Role prompts can improve performance by making acceptance criteria salient. They can also create false confidence. A “security reviewer” model does not possess independent security expertise merely because the system prompt assigns the title. Its evidence should be tested against known defect suites and external tools.
Permission separation is more concrete. A reviewer without write access cannot accidentally deploy the code it inspects. An executor that accepts only an approved artefact cannot improvise a new one. Use roles to create enforceable information and authority boundaries, not organisational theatre.
| Role property | Prompt-only version | Enforced version | Evidence of independence |
|---|---|---|---|
| expertise | “act as a risk expert” | specialised model, retrieval or validated tool | held-out error profile and source use |
| information | same context with a new persona | separately scoped evidence store | source-overlap measurement |
| authority | instruction not to execute | distinct credentials and gateway | permission test and audit log |
| review | asks model to criticise itself | separate context, criteria and read-only tools | blind defect-detection study |
| approval | model says “approved” | named human or policy service signs decision | immutable approval record |
10. Software teams and the evaluator problem
Software development is a natural test bed for agent collaboration because outputs can be executed. MetaGPT and ChatDev model familiar roles. AutoGen enables tool-using conversations. Magentic-One presents a generalist multi-agent system with an orchestrator and specialist agents. The field has moved from toy dialogue towards repository work and computer use.
Execution supplies useful oracles: compilation, tests, static analysis and repository state. It does not make evaluation complete. A patch can pass existing tests while introducing a security issue or maintenance burden. A multi-agent team may generate more test attempts and tool calls than its baseline. A role-labelled review may repeat the same model’s blind spot.
To isolate collaboration, compare at least four conditions:
- one model, one pass;
- one controller, matched samples and sequential roles;
- several agents with communication;
- several agents with separate tools or evidence.
All should receive the same issue, repository snapshot, tool permissions and total budget. A common external test suite should judge outputs. Human maintainability review should be blinded to condition. Error categories should include incorrect patch, regression, security defect, unnecessary change and failure to abstain.
The comparison can reveal different mechanisms. If B matches C, role decomposition matters but communication does not. If D exceeds B and C, tool or information diversity matters. If C improves only with more tokens, the result is test-time scaling. If no condition reliably passes hidden tests, the bottleneck is capability rather than coordination.
For enterprise use, repository agents should operate on isolated branches with limited credentials. Review and deployment remain separate gates. A conversational consensus among coding agents cannot approve production release.
11. Failure modes of collective reasoning
Multi-agent designs add failure modes that do not exist in one isolated response.
Correlated error occurs when agents share the same misconception. Premature convergence occurs when later agents copy an early answer rather than preserving independent hypotheses. Authority bias occurs when a designated expert is followed without stronger evidence. Rhetorical capture occurs when fluent argument defeats a correct but less persuasive answer. Context dilution occurs when repeated messages crowd out the task and primary sources. Responsibility diffusion occurs when no component owns verification.
Communication can propagate both corrections and errors. An incorrect first message can become context for every later participant. Agent count then increases confidence without increasing independent evidence. The transcript may look like multiple confirmations even though every confirmation derives from one claim.
Adversarial settings add collusion and manipulation. An agent with untrusted retrieved content can transmit an injected instruction to peers. A compromised tool response can contaminate a shared memory. Permission should not propagate with information. One agent’s ability to read a document does not grant another agent the authority to execute its embedded instruction.
Failure diagnosis needs lineage. Every material claim should identify its source agent and underlying artefact. The final selector should know whether three agents independently verified a fact or merely repeated it. A citation graph is more informative than a speaker count.
| Failure | Observable signal | Naïve interpretation | Better control |
|---|---|---|---|
| correlated error | same wrong claim and overlapping evidence | “three agents agree” | independent sources or external verifier |
| premature convergence | disagreement falls before new evidence appears | consensus means confidence | delay peer exposure; preserve initial answers |
| rhetorical capture | selection follows style or length | best argument won | blind claims; verify cited evidence |
| context dilution | primary facts disappear in summaries | more discussion is more reasoning | bounded typed messages and source pointers |
| tool contagion | one agent relays untrusted instruction | peer recommendation is safe | provenance and policy at every hop |
| responsibility diffusion | no explicit verification owner | the team checked it | named verifier and stopping criterion |
12. A baseline-normalised evaluation protocol
The paper proposes a protocol that can test collaboration rather than agent count. It is a design, not a reported experiment.
Begin with a held-out task population across closed-answer reasoning, document synthesis, code repair and policy-constrained tool use. Define common evidence, tools and outcome criteria. Select at least two model families and preregister budgets.
For each task, run:
- independent sampling with majority or verifier selection;
- sequential propose, critique and revise with one controller;
- homogeneous multi-agent debate;
- heterogeneous model collaboration;
- information-partitioned collaboration;
- tool-specialised collaboration.
The sampling count, total generated tokens, retrieval calls, tool operations and selector should be matched where the mechanism permits. Parallel wall time should be reported separately. Every condition should use the same final adjudicator.
The primary outcome is task-specific correctness or safe resolution. Secondary outcomes include calibration, evidence coverage, correct-minority survival, cost, latency, tool risk, abstention and control failures. Pairwise task-level analysis is preferable because it shows which tasks change outcome.
An effect-direction plot can classify each comparison without pretending measures are commensurable:
| Mechanism contrast | Supports collaboration if | Supports simpler baseline if | Ambiguous when |
|---|---|---|---|
| debate vs matched self-consistency | debate wins under equal samples and selector | self-consistency matches or exceeds | budgets or tuning differ |
| dialogue vs sequential critique | peer exchange adds stable gain | one controller matches it | prompts expose different evidence |
| heterogeneous vs homogeneous | model diversity reduces correlated error | added model access has no gain | stronger model dominates ensemble |
| partitioned vs shared information | union and exchange improve outcome | shared full context works as well | partitions also reduce distraction |
| tool-specialised vs common tools | specialisation improves safe use | one controller routes tools equally well | permissions differ |
The protocol should include negative tasks where one agent receives a persuasive false statement and another receives verifiable contrary evidence. This tests whether the topology preserves a correct minority. It should also include tasks whose evidence is fully shared. If collaboration helps only the partitioned condition, information diversity is the likely mechanism.
Evaluation, assurance and counterevidence
13. From conversation to typed evidence exchange
Free-form conversation is an easy research interface. It is a weak production contract. Messages mix claims, evidence, instructions, confidence and social cues. A structured exchange can preserve what matters.
A claim message can contain:
- task and subject scope;
- proposition;
- source identifiers and quoted locations;
- transformation or calculation applied;
- confidence with calibration reference;
- known counterevidence;
- requested verification;
- expiry and supersession status;
- sender role and tool provenance.
The message is not a chain of thought. It is an auditable claim package. A recipient can test the source and respond with a contradiction or verification result. The shared board can deduplicate repeated claims and expose when consensus depends on one source.
Typed exchange also limits prompt injection. Retrieved text remains attached as data with provenance. It cannot silently become a peer instruction. A policy service can reject a message that requests an action beyond the sender’s role. The final executor accepts approved action objects, not prose consensus.
Free-form debate may remain useful for generating hypotheses. Before a hypothesis influences an external decision, it should be converted into a typed claim with evidence. This creates a clean boundary between exploratory language and governed action.
14. Bank and regulated-enterprise implications
Multi-agent patterns can support regulated work when roles reflect real control boundaries. A financial-crime investigator, policy retriever and evidence verifier can contribute different artefacts. A maker-checker workflow can separate proposal from approval. A model-risk review can preserve a challenger’s unresolved objection.
They can also create an illusion of control. Two instances of the same model with the same context are not independent lines of defence. A “compliance agent” has no regulatory authority. A majority vote cannot override a deterministic policy or a named approver.
Information partitioning must respect data minimisation. An agent that needs policy text does not automatically need customer identifiers. A transaction-analysis component may receive pseudonymised features. A reviewer can inspect cited evidence without receiving write credentials. The system should record every transfer between scopes.
Human approval should occur at the consequence boundary. The human receives the proposal, source evidence, policy criteria, disagreements, confidence and unresolved questions. The interface should not hide minority objections behind a consensus score. If one verifier identifies a possible sanctions match, the issue remains visible until resolved.
Model risk validation should test the configured team. Material changes include model versions, prompts, number of participants, order, communication schema, selection rule, tool access and stopping condition. Adding an agent changes cost, error correlation and attack surface. It is not a minor prompt edit.
Operational resilience needs ownership. If agents disagree indefinitely, a timeout must route the case. If one model provider fails, the system must know whether remaining roles can continue safely. If the shared memory is corrupted, the team should rebuild from versioned source evidence rather than trust the transcript.
15. Worked scenario: transaction-monitoring alert review
Consider a worked scenario in which an agent team assists an analyst with a transaction-monitoring alert. It is not a report of an actual deployment. The team may collect evidence and draft a recommendation. The human analyst retains disposition authority.
The evidence collector receives the alert’s transaction scope and retrieves ledger entries. The policy agent receives current typology and escalation rules but no unnecessary customer attributes. The counterevidence agent looks for benign explanations such as payroll, known counterparties or documented business activity. A verifier recomputes totals and checks source lineage.
Each role posts typed claims. “Rapid movement of funds” is not accepted as a free-form impression. The message identifies transactions, timestamps, accounts and the rule whose condition may be met. The counterevidence claim cites the customer file and its effective date. The verifier flags that one supporting document expired before the alert period.
A homogeneous debate could produce a similar narrative, but the worked design adds genuine differences: scoped sources, separate tools and an independent calculation. The value comes from evidence structure, not simulated personalities.
The human disposition record names the evidence accepted and the exception resolved. If more information is required, the workflow requests it. No agent can convert discussion into a regulatory filing or customer restriction without the required authority path.
This scenario also reveals the cost trade-off. Four model roles may be unnecessary for straightforward alerts. A deterministic retrieval and rules path with one drafting model may be cheaper and easier to validate. The team should activate only when information conflict or complexity justifies it.
16. Design and operating implications
16.1 A simple information model
The value of another agent can be described without assuming a collective mind. Let a task have several plausible hypotheses and a set of observations that can distinguish them. Each participant begins with some observations and a procedure for turning them into a claim. The team gains only if an additional participant increases the probability that decisive evidence reaches a competent selector.
There are four gates. Coverage asks whether any participant finds the relevant evidence. Independence asks whether participants make errors for different reasons. Transmission asks whether the evidence survives communication. Selection asks whether the final decision recognises it. Failure at any gate can erase the value of extra participants.
If every agent receives the same context and uses the same model, coverage can still increase through stochastic sampling. Independence is limited by shared weights and training. Transmission may add little because peers repeat similar claims. Selection becomes the main mechanism. This resembles self-consistency with a more elaborate interaction pattern.
If agents receive disjoint sources, coverage can increase materially. Transmission then matters because no one participant initially has the complete case. A structured synthesis agent may be justified. Yet partitioning can also hide contradictions. The system should allow agents to request specific missing evidence rather than rely on a fixed split.
Independence should be measured by joint errors, not anecdotes. Two models that reach different wording but fail the same tasks are not diverse in the relevant sense. Pairwise disagreement is also insufficient because agents can disagree randomly. Useful diversity means that one participant is correct on cases where another is wrong and that the selector can identify the better contribution.
Transmission has bandwidth and distortion. A full transcript preserves more words but may bury sources. A summary reduces cost but can omit minority evidence. Typed claims preserve selected fields and lineage, but the schema may exclude an unforeseen insight. The right format depends on task uncertainty.
Selection is often the weakest gate. A model judge can prefer fluent consensus. Majority vote fails when errors are correlated or when only one participant sees the decisive source. External execution, rules or a human expert can provide a stronger oracle. In their absence, the team should expose disagreement instead of manufacturing certainty.
The information model yields a practical test. Before adding a role, state which gate it improves and how that improvement will be observed. A “critic” may improve selection by spotting unsupported claims. A search specialist may improve coverage. A different model may improve independence. If the design cannot name the gate, the extra role is likely ornamental.
16.2 Why self-consistency is a demanding baseline
Self-consistency samples several reasoning paths and aggregates answers. It can be implemented without messages between samples, which preserves initial independence. For closed-answer tasks, majority vote is simple. For open tasks, a verifier or ranker can compare candidates.
This baseline is demanding because many proposed benefits of debate are available already. Multiple samples explore alternatives. A final judge compares them. A critique-revision stage can be added while retaining one controller. The debate must improve the information or selection process beyond those operations.
Matching only call count can still be unfair. Debate participants often receive prior answers as additional input. A self-consistency baseline can pass the candidate set to the same judge. Debate may use longer role instructions, so the baseline should receive equivalent task criteria. If debate uses retrieval between rounds, the baseline needs the same retrieval budget.
Independence can favour self-consistency. Debate exposes participants to earlier outputs and may cause convergence. A rigorous design captures all initial answers before communication. It then measures whether correct candidates survive. If initial majority vote is correct and later debate becomes wrong, communication harmed selection. If an initially wrong majority changes after one agent introduces verifiable evidence, dialogue helped.
The baseline also has limits. Majority vote cannot combine complementary partial proofs. A single candidate may contain evidence that another lacks. A structured judge can fuse them, but that begins to resemble collaboration. The distinction should be based on information flow, not the number of process IDs.
For consequential tasks, self-consistency does not provide independent control. Ten samples from one model cannot approve a payment. The baseline is about epistemic performance, not authority. A policy gateway and human maker-checker remain outside the comparison.
16.3 The judge is part of the method
Many multi-agent papers focus on debaters and treat the final judge as a neutral endpoint. The judge is an estimator with its own error distribution. It may share the base model, training data and stylistic preferences of the participants. It may be swayed by length, order, confidence or consensus.
A judge can operate in several modes. Majority vote uses no semantic comparison. A model ranker chooses among answers. A synthesiser creates a new answer. A deterministic verifier executes code or checks a rule. A human expert inspects evidence. Each supports a different claim.
Order effects can be tested by permuting anonymised candidate answers. Style effects can be tested by normalising length and removing role labels. Consensus bias can be tested by duplicating one candidate without adding evidence. Evidence sensitivity can be tested by placing a correct source in a minority answer and measuring whether the judge uses it.
The judge should not see hidden condition labels. Otherwise it may prefer the answer presented as a “debate consensus”. It should receive the same evidence representation across baseline and treatment. If the judge generates a synthesis, that generation cost belongs to every condition.
Calibration matters when the judge can abstain. A selector that chooses the best candidate 70 per cent of the time may still be useful if it recognises the remaining uncertainty and routes it. A selector that always emits high confidence creates a poor control even with higher average accuracy.
Banking adds authority. A model judge may classify whether evidence appears to meet criteria. It cannot make a reserved human decision merely because the multi-agent panel agreed. The word “judge” should not obscure the distinction between computational selection and legal or policy authority.
16.4 Cost, latency and diminishing returns
An agent team consumes resources through model inference, context transfer, tool use, orchestration and adjudication. Parallel calls can reduce wall-clock latency while increasing peak capacity. Serial debate increases both time and context. Messages produced early are repeatedly reread by later calls.
Cost should be reported per attempted task, not per successful task alone. Failed debates can consume their entire round budget. A single-agent baseline may abstain earlier. Tool costs and human review also belong in the account.
Marginal value usually falls with participant count when errors are correlated. The first additional independent sample can change a decision. The tenth similar answer may add only confidence. Dynamic stopping can reduce waste: stop when an external verifier confirms the result, when no new evidence appears or when the decision remains unresolved after a bounded number of rounds.
Dynamic routing should be compared with always-on teams. Straightforward tasks can use one agent and deterministic checks. Ambiguous or high-consequence cases can activate a challenger or specialist. This creates a selective collaboration policy.
The routing classifier becomes part of evaluation. If it fails to identify difficult cases, the team activates too late. If it sends every task to the full panel, cost rises without selectivity. Metrics should include routing recall for cases that benefit from collaboration and avoided cost on cases that do not.
Latency has operational consequences. An interactive customer flow may not tolerate three serial debates. A batch model-risk review might. Parallel agents require rate-limit and capacity planning. A provider outage can remove several roles at once if all use the same service, which weakens operational diversity.
The economically relevant metric is incremental safe resolution per added unit of compute and human effort. Accuracy alone cannot determine whether the team is a good design.
16.5 Security and adversarial collaboration
Multi-agent systems enlarge the number of channels through which untrusted content can move. One agent retrieves a document containing an injected instruction. It summarises the instruction to a peer. The peer treats the message as trusted because it came from an internal role. The original provenance disappears.
Every claim should retain the trust class of its underlying content. Internal transport does not upgrade external data. If an agent quotes a web page, the receiving agent should see that the source is untrusted and that the quotation carries no authority.
Tool permissions should remain local to the role. A research agent can request that an executor perform an action, but the request passes through the same policy gateway as any other proposal. A peer message cannot delegate credentials. The executor should accept a typed action with subject, purpose, preconditions and approval, not a conversational imperative.
Consensus can be attacked. A compromised participant can flood the shared context, repeat a false claim or imitate confidence. Rate limits and message schemas constrain these behaviours. Source-based deduplication prevents one claim from appearing independent when repeated by several agents.
The selector is another target. An attacker can craft text for a model judge rather than the original task. Blinded, structured candidate representations reduce stylistic manipulation. Deterministic verification should be used where available.
Monitoring needs graph context. Logs should show which message caused a tool proposal, which source supplied the claim and which policy allowed execution. A flat transcript is hard to query. A provenance graph can reveal that five later claims all descend from one poisoned document.
Security evaluation should introduce compromised or injection-affected participants and measure containment. The desired result is not necessarily task completion. It may be isolation, abstention and escalation without unauthorised effect.
16.6 Human factors and automation bias
A panel of agents can create an impression of independent review. The human sees a polished majority and may assume several experts reached the conclusion. If the agents share a model and evidence, the display exaggerates independence.
Interfaces should communicate provenance honestly. Show how many distinct sources and model families contributed. Mark repeated claims. Preserve a minority objection when it cites unresolved evidence. Do not use avatars or job titles as proxies for expertise.
Human workload can increase when transcripts are long. A concise evidence matrix is easier to review than multiple rounds of prose. The matrix should separate facts, interpretations, policy criteria and missing information. The reviewer can open the underlying source where needed.
The human should know why a case was escalated. “Agents disagreed” is not enough. The packet should identify the contested proposition, each supporting source, the verification attempted and the decision authority required. This supports accountable judgement.
Automation bias can be studied by varying whether participants are described as one model, several identical models or distinct experts while holding content fixed. If reviewer decisions change, presentation is affecting trust. Training and interface design should address that bias.
The human can also contribute genuinely different evidence. A domain expert may recognise an exception absent from the model context. The system should let the reviewer add a source or correction, record it and rerun only affected checks. Human input should not disappear into an untraceable chat history.
16.7 An operating model for selective collaboration
A practical operating model begins with one bounded controller and deterministic services. It activates additional roles only for a stated reason.
At intake, a router classifies task type, consequence, evidence diversity and uncertainty. Low-consequence tasks with a strong verifier use the simplest path. Cases with conflicting sources activate a challenger. Cases requiring distinct tools activate specialists. High-consequence decisions route to a human regardless of agent agreement.
Each role has an owner, input contract, output schema, tool scope and stopping rule. The shared evidence board is versioned. Claims expire when a source is superseded. The selector sees provenance and dissent. External execution is a separate service.
Offline evaluation determines which task strata benefit from collaboration. Online monitoring checks whether those strata remain stable. Drift can appear as changing source quality, model errors or routing rates. A sudden rise in full-team activation may signal upstream data problems rather than harder work.
Incidents should be replayable from observable events. The organisation must know which models, prompts, sources and tools were active. It should be able to disable one role without corrupting open cases. Cases in progress need a controlled handoff to the simpler path or a human.
Vendor concentration should be visible. A team that uses four endpoints from one provider may not be operationally independent. Model diversity can improve error coverage but raises data-transfer, contracting and validation complexity. The benefit must be measured.
Governance forums should review the selective policy, not every conversational pattern. They approve permitted task families, evidence sources, role permissions, consequence boundaries and evaluation thresholds. Prompt changes remain controlled implementation changes within that envelope.
16.8 A novice’s test for a proposed agent team
Start with the claimed benefit. If the answer is “different perspectives”, ask what makes the perspectives different. Is it another sample, another prompt, another model, another document, another tool or another permission? Each has a different baseline.
Next ask whether a single controller can perform the same steps sequentially. If yes, compare it. A multi-process architecture may still improve latency or maintainability, but those are systems benefits rather than collective reasoning.
Then inspect the final decision. Who selects it? What evidence can that selector verify? Can a correct minority survive? Does agreement come from independent sources or repeated messages? If no external check exists, uncertainty should remain visible.
Finally inspect authority. Can any agent change a customer, account or production system? Which rule permits the change? Does another agent’s approval have legal meaning? Usually it does not. Machine agreement should never be mistaken for delegated authority.
The four-question version is concise:
- What unique evidence or capability does each role add?
- What matched simpler baseline will test that value?
- How will the selector verify rather than count claims?
- What independent control governs external action?
If a design has precise answers, a team may be justified. If the answers are speaker names and conversation rounds, the architecture needs more work.
16.9 Open research questions
The largest gap is matched-budget reproduction across model generations. Debate protocols often use proprietary models that change. Repeating standardised comparisons across open and closed model families would show whether reported mechanisms persist.
Another gap is source-level diversity measurement. Studies often vary personas or model temperature without measuring whether agents find different evidence. Retrieval experiments can control source partitions and overlap. They can test when partial information creates a true collaboration advantage.
Selector research needs stronger external validity. Model judges can be tested on adversarially styled candidates, duplicated claims, minority evidence and known ground truth. Human judges can be studied for automation bias and review burden.
Long-horizon tasks need cost-aware communication. A debate that helps a ten-token arithmetic answer may not scale to a repository or investigation. Researchers should track message growth, repeated evidence and error propagation over time.
Safety tasks need a consequence model. Collaboration might reduce an average error while introducing a rare unauthorised action through permission contagion. Evaluation should separate epistemic error, policy breach, data exposure and side effect.
Finally, studies should report when a simpler architecture wins. Negative results are design knowledge. They identify task regimes where sampling, deterministic verification or one well-equipped controller is sufficient.
16.10 Baseline recipes for common claims
Different claims require different baselines. A generic “single agent” condition is rarely enough.
For a claim about reasoning diversity, use independent self-consistency with the same number of initial samples. Preserve those samples before any communication. Give the final selector the same candidate set. Debate adds value only if interaction changes the result beyond selection.
For a claim about critical review, use one controller that generates a proposal, starts a fresh context with the proposal and criteria, produces a critique, then revises. Match the number of calls and tokens. A separate agent process may provide isolation, but the reasoning claim depends on performance.
For a claim about role expertise, give the sequential baseline the same role instructions and retrieval sources. Then test whether concurrent or interactive roles improve the outcome. If the specialist uses a different model or fine-tune, include that model in the baseline.
For a claim about tool specialisation, allow one controller to route the same tools through a permission broker. Compare tool errors, task outcome and policy adherence. Multiple agents may reduce context complexity, but different credentials change the control surface and must be reported.
For a claim about independent assurance, prompt separation is insufficient. Use different validation methods. One component might execute tests, another check policy, and a human exercise reserved judgement. The baseline is not another eloquent answer; it is the absence of that independent signal.
For a claim about latency, compare serial and parallel schedules under the same work. Report peak concurrency and total inference. Parallel teams can be faster while more expensive. That is a valid systems result.
For a claim about resilience, introduce component failure. Remove one agent, corrupt one message or make one provider unavailable. Measure whether the team degrades safely. Speaker multiplicity without service or model diversity may provide no resilience.
These recipes improve decision quality because they permit a precise result. A role architecture can be easier to maintain even if it has no accuracy gain. A heterogeneous ensemble can improve correctness while increasing latency. A reviewer role can reduce prohibited actions while leaving task accuracy unchanged. Each is more useful than a single aggregate winner.
16.11 Reading a multi-agent result without overclaiming
A careful reading begins with the unit of comparison. Is the paper comparing models, complete systems or prompt protocols? A complete system may include stronger models, more calls and more tools. Its result supports a system claim, not a dialogue mechanism.
Next inspect task structure. Closed-answer mathematics favours voting and automatic checking. Open-ended writing favours model judges and may hide factual defects. Software tasks have executable tests but incomplete specifications. Social simulations measure behaviour inside constructed goals. Transfer between these regimes is not automatic.
Then inspect the resource table. If no table exists, reconstruct calls, rounds, samples and selectors from the method. A three-agent, three-round protocol can require many times the inference of a one-shot baseline. That does not make the result invalid. It changes the procurement question from “which reasoning method is better?” to “is the additional result worth the additional resources?”
Inspect independence. Did agents use the same model, prompt and evidence? Were initial answers captured before sharing? Did the study measure correlated error? If not, “diversity” may mean sampling variance.
Inspect the selector. A strong proprietary judge may explain the gain. An answer chosen by test execution has a different evidence basis from one chosen by another language model. Human ratings need a rubric and agreement report.
Finally inspect failure cases. Did discussion correct a wrong majority, or did it persuade a correct minority to conform? Did agents cite sources? Did any interaction increase unsafe behaviour? The distribution of reversals often reveals more than average accuracy.
For enterprise readers, translate the result into a bounded statement: “This configured protocol improved this outcome on these tasks under this budget and evaluator.” Then identify the missing local evidence. That sentence is less dramatic and more actionable.
16.12 A deployment decision record
A decision to use a team should preserve the causal case. The record names the task population, the simpler baseline, the additional roles, the unique contribution expected from each role and the evidence that the contribution occurred. It also records added model calls, tools, latency, data transfers and human work.
The acceptance threshold should be consequence-aware. A research assistant may justify collaboration through better source coverage. A customer-facing workflow needs evidence that the team does not increase unsafe or inconsistent actions. A maker-checker design needs permission tests, not only answer accuracy.
The record should identify conditions under which the team falls back to a simpler path. A provider outage may remove model diversity. A missing source may make a specialist ineffective. A high queue may make serial debate breach service targets. These are operating conditions, not footnotes.
Monitoring then tests the original rationale. If the challenger rarely contributes a unique verified claim, its role may be redundant. If one source supplies nearly every accepted answer, information partitioning may be wrong. If human reviewers override consensus on one task class, routing or evaluation needs revision.
The review date should follow material change rather than a ceremonial annual cycle. Model updates, new tools, altered prompts, a changed selector or a new customer consequence can invalidate the evidence. The team remains a configured system whose parts interact.
This record prevents architecture drift. New agents are not added because a framework makes them easy to instantiate. They are added when a bounded evaluation shows an incremental contribution and an owner accepts the associated cost and risk.
Implementation and operating detail
17. Evidence table
The evidence table separates reported contribution from this paper’s inference.
| Primary source | Reported focus | Limitation for the thesis | Synthesis inference |
|---|---|---|---|
| Du et al. debate | multi-agent debate for factuality and reasoning | added sampling and rounds complicate mechanism attribution | positive evidence that peer outputs can aid revision |
| Should we be going MAD? | comparison of debate strategies and prompting baselines | task and model coverage remain bounded | strong contrary evidence to inherent debate superiority |
| Revisiting MAD | debate as conditional test-time scaling | findings vary by task and model regime | effectiveness is conditional, not architectural law |
| Demystifying MAD | diversity and confidence mechanisms | recent preprint needing further independent replication | candidate diversity and update rules matter |
| Self-consistency | multiple reasoning paths and answer aggregation | not an interactive protocol | essential strong sampling baseline |
| More Agents Is All You Need | performance scaling through multiple agents | “agent” samples do not isolate dialogue | count can proxy test-time sampling |
| ReConcile | diverse round-table collaboration | model and protocol cost complicate comparison | heterogeneity and consensus are separable mechanisms |
| ChatEval | multi-agent debate for evaluation | model judges share biases and lack ground truth | evaluation needs independent validity evidence |
| LLM-Blender | ranking and fusing outputs from multiple models | uses several models and a learned fusion process | aggregation can exploit model diversity without debate |
| Mixture-of-Agents | layered aggregation of multiple model outputs | added model access and compute are central | heterogeneous evidence can be valuable |
| Scalable oversight | debate and consultancy with weaker judges | simulated judge and task asymmetry constrain transfer | debate may help under defined information asymmetry |
| Persuasive debaters | debater strength and truthful judge decisions | specialised experimental game | verifiable exhibits and judge design are decisive |
| CAMEL | role-playing agent communication | role assignment does not isolate multi-agent gain | prompts can structure decomposition |
| AutoGen | framework for tool and human-integrated conversations | framework flexibility is not a controlled effect | useful orchestration substrate, not evidence of superiority |
| MetaGPT | software roles and operating procedures | system-level baseline normalisation is difficult | structured artefacts can reduce workflow ambiguity |
| ChatDev | communicative software-development agents | generated projects and evaluator define validity | role workflows need hidden tests and human review |
| AgentVerse | collaborative and emergent agent framework | heterogeneous tasks limit one mechanism claim | interaction form should be evaluated per task |
| DyLAN | dynamic selection and communication network | selection phase and compute complicate attribution | task-adaptive team selection can reduce redundancy |
| Magentic-One | orchestrated generalist multi-agent system | broad system evaluation does not isolate agent count | orchestrator and specialists form a testable system |
| SOTOPIA | interactive social intelligence evaluation | simulated social goals differ from enterprise decisions | conversation quality is its own construct |
The strongest evidence for the thesis comes from direct debate comparisons and test-time scaling analyses. The strongest evidence against a simple version comes from information-asymmetric oversight and heterogeneous aggregation, where interaction or combination can expose evidence unavailable to one weak judge or model.
18. Quality assessment and evidence gaps
Internal validity varies. Closed-answer tasks allow objective adjudication but may reward sampling. Open-ended software and social tasks are more realistic yet harder to judge. Model-as-judge evaluation can share biases with the systems under test. Proprietary model updates complicate reproduction.
External validity to banking is limited. Public tasks rarely reproduce identity, confidentiality, regulatory consequence and operational handoff. Role-based frameworks demonstrate organisation but not bank control effectiveness. The paper therefore treats bank implications as architecture inference.
Leakage and task familiarity remain concerns. If several agents use the same model, they may share memorised answers. A debate can make those answers more verbose without adding independent knowledge. Studies should use fresh tasks, contamination analysis and source-grounded questions where possible.
Statistical uncertainty is not always reported in a form that supports protocol ranking. Multi-agent runs add variance through multiple samples and interactions. Paired task confidence intervals and repeated seeds are more informative than one aggregate number. Hyperparameter search should be included in the resource account.
Conflicts of interest are typical of systems research: authors introduce a framework and evaluate it. Open code and prompts help but do not remove design bias. Independent reproduction under matched budgets is especially important.
19. Limitations and non-claims
This synthesis is not a registered systematic review. The fast-moving evidence base includes recent preprints. It does not reproduce experiments or calculate a pooled effect. Different definitions of agent, debate, collaboration and cost make broad aggregation inappropriate.
The paper does not establish that one agent is safer than several, that debate never helps or that role-based systems lack value. It does not claim that model heterogeneity guarantees independent errors. It does not claim that a human judge can always identify truth after debate. It does not treat a consensus transcript as regulatory evidence.
The main thesis is deliberately vulnerable to future evidence. Strong, preregistered studies may find stable dialogue effects after matched sampling and revision. New training methods may make agents preserve correct minority views. Better judges may use interaction more effectively. Such results would narrow or overturn the present conclusion.
20. Decision-grade conclusion
The multi-agent question is not how many model instances appear in a diagram. It is what new information, capability or control each one contributes. Homogeneous debate can improve an answer, but its baseline must receive the same opportunities to sample, critique and select.
The literature now contains positive, negative and conditional results. That is useful. It shifts the design question from “does debate work?” to “which mechanism works under which information and budget conditions?” The clearest surviving cases involve genuine asymmetry: different evidence, models, tools or authority.
For regulated enterprises, collaboration should be built around typed evidence, provenance and enforceable roles. Preserve independent proposals before peer exposure. Make contradictions visible. Use an external verifier where possible. Keep execution behind policy and human authority. A second voice is not a second line of defence unless it can fail differently and prove what it saw.