The strongest member is the baseline

A weak test asks whether an AI-assisted person performs better than an unaided person. That is often useful, but it establishes augmentation only. If the agent alone is better than both, the organisation may have added cost, delay and ambiguity while calling the result collaboration.

A preregistered meta-analysis by Vaccaro, Almaatouq and Malone assembled 106 experiments and 370 effect sizes that reported human-only, AI-only and combined performance. The combined systems were better than humans alone on average, yet worse than the better solo member. The pooled effect for strong synergy was negative, with substantial variation across tasks and study designs.¹ Decision tasks showed average losses; creation tasks looked more promising, though the estimated creation gain was uncertain. When humans were stronger alone, combinations tended to gain. When AI was stronger alone, combinations tended to lose.

This is not evidence that collaboration is futile. It is evidence that the usual interface, one person receives one complete AI answer and chooses whether to accept it, often fails to turn two competence profiles into a better decision. Explanations and confidence displays did not emerge as significant moderators in that meta-analysis. The design problem lies deeper than adding persuasive context to a recommendation.

Human-group research points in the same direction. Woolley and colleagues found a group-level performance factor across varied tasks, associated more with interaction patterns such as social sensitivity and turn-taking equality than with the maximum individual intelligence in the group.² Hong and Page showed, under specific mathematical conditions, how diverse problem solvers can outperform a group selected only for individual ability.³ Neither result says that diversity always wins. Both say that composition and process can carry information that individual rank misses.

Bahrami and colleagues supply a sharper boundary. Pairs of people made difficult visual discriminations, exchanged confidence and reached a joint answer. Dyads with similar perceptual sensitivity could outperform their better member; dyads with markedly different sensitivity could perform worse because confidence reports were not commensurable.⁴ The weaker observer could sound as certain as the stronger observer without providing equally diagnostic evidence. Communication was present in both conditions. What changed was whether the interaction could translate private reliability into appropriate influence.

This result prevents a sentimental reading of diversity. Different members are useful when their differences reveal task-relevant information and the protocol can interpret that information. Difference in vocabulary, model family or professional role may matter, but none is a substitute for measured local competence. A team can be socially balanced and epistemically redundant; it can also be unequal in general ability yet complementary on a narrow set of cases. The evaluation must discover which arrangement exists.

The outcome measure must also match the consequence. Accuracy treats all mistakes alike. A decision system may justify human participation even when average accuracy falls slightly if the human reliably prevents a rare catastrophic error. Conversely, a tiny accuracy gain may not pay for hours of specialist review, missed cut-offs or inconsistent accountability. Let utility include the cost of each error class, time, attention, compute, reversibility and unresolved risk. Then define the strong-synergy margin as:

U is the declared utility under the evaluation conditions. A positive margin means the whole beats both solo baselines. It does not by itself explain why.

Compute this margin by task slice as well as in aggregate. A human may dominate cases with contradictory testimony, an agent may dominate routine pattern matching, and both may fail when a source is stale. An overall average can conceal these regions, especially when common easy cases swamp rare consequential ones. Pre-register slices from the causal structure of the task, such as evidence availability, novelty, consequence, customer vulnerability and reversibility, rather than inventing them after seeing a favourable result.

The relevant competitor may then be a selective router rather than one global winner. A simple rule that sends familiar, well-supported cases to the agent and novel or contradictory cases to a specialist can outperform a conversational team. Collective intelligence must earn its interaction tax against the cheapest mechanism that exploits the same competence map. That comparison changes architecture: sometimes the right design is deliberation; sometimes it is routing; sometimes it is a single strong member with abstention.

Negative control: the dominant specialist

Suppose one agent is better on every consequential task slice, its failures are detectable by a calibrated abstention rule, and humans add no independent evidence. A mandatory human vote is then a negative control for the collaboration thesis. It should not improve performance. It may instead introduce delay and uncalibrated override.

A human decision may still be required because law, policy or institutional legitimacy assigns authority to a person. That is a valid governance design. It should be described as accountable human authority, not as evidence that the mixed team is more intelligent. Performance complementarity and decision rights answer different questions.

flowchart LR
  A["Three votes, one mistake"] --> B["On this page"]
  B --> C["The strongest member is the<br/>baseline"]
  C --> D["Disagreement is the raw material"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#087A98,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Disagreement is the raw material

Two members can have the same individual accuracy and radically different collective value. The difference is where their errors land. If they fail on the same cases, the second judgement repeats the first. If each catches cases the other misses, disagreement contains recoverable information.

The same issue appears when an organisation calls three prompts to one model, deploys three agents on the same foundation model and corpus, or asks three people to assess a recommendation after showing them the same polished answer. Surface diversity can conceal a shared latent cause. Different role names do not guarantee different information.

For exchangeable estimates with equal error variance σ² and average pairwise error correlation ρ, the variance of their mean is:

n is the number of estimates, σ² is each estimate's error variance, ρ is their average pairwise correlation, and ē is the mean error. When ρ is zero, averaging gains the usual factor of n. When ρ is one, additional members do not reduce variance.

This equation is an intuition pump, not a universal model of team cognition. Human-agent decisions may be categorical, members may differ in competence, interaction changes later responses, and shared errors may concentrate on high-cost cases. It nevertheless exposes the quantity that headcount hides: the effective number of independent error paths.

The four-cell ledger

For a binary task, record the joint correctness of one human and one agent before either sees the other's answer. Four probabilities describe the opportunity:

• p11: both are correct;
• p10: only the human is correct;
• p01: only the agent is correct;
• p00: both are wrong.

The disagreement mass is D = p10 + p01. It is the fraction of cases where selecting the right member could improve the final answer. The oracle ceiling, if a perfect arbitrator always chose the correct member on disagreement, is p11 + D = 1 − p00. Cases in p00 require new evidence, a third mechanism or abstention; voting cannot repair them.

Build the ledger at the level where a decision can change. A single global table may show healthy disagreement while every high-consequence case sits in p00. Conversely, a small global gain may be operationally valuable if the human-only cells contain precisely the rare failures the agent cannot detect. Weighting after aggregation hides this structure. Record the joint cell, task slice, error severity and decisive evidence for every case, then compute utility from that case-level record.

Correlation is descriptive; provenance is causal. Two agents may show low historical error correlation yet become coupled after a retrieval change gives both the same index. Two people may appear independent but share one policy interpretation or copy from the same case note. The ledger should therefore sit beside a dependency map: model lineage, training overlap where known, retrieval collections, tools, prompts, upstream data and prior recommendations seen. The purpose is not to prove metaphysical independence. It is to identify common causes that a failure injection can perturb.

The arbitration threshold

Assume the team chooses between the human and agent answers whenever they disagree. Let q be the probability that the protocol selects the correct answer on a disagreement. The team accuracy is:

Human accuracy is p11 + p10; agent accuracy is p11 + p01. Therefore the team beats the stronger member only when:

If the human and agent win equally often on disagreements, the threshold is 0.5. Any arbitrator better than chance can add value. If the agent is correct on nine out of ten disagreements, the threshold rises to 0.9. The human must recognise almost every rare agent failure without disturbing the many cases where the agent is right.

In the specimen above, the agent wins 16 of the 28 disagreements. The threshold is 16/28, or 57.1 per cent. An arbitration protocol that chooses correctly on 70 per cent of disagreements reaches 87.6 per cent overall, 3.6 percentage points above the agent. A random tie-breaker reaches 82 per cent and destroys two points of agent performance.

The ledger converts a vague claim about complementary strengths into a testable design constraint. High disagreement is not automatically good. It may indicate that one member is poor. Low disagreement is not automatically bad. It may indicate both are excellent. The useful quantity is the combination of disagreement mass, who wins those disagreements, how costly each error is, and whether the protocol can select the locally competent member.

Diversity creates complementarity potential; arbitration converts that potential into performance. A team can fail at either stage. It can recruit replicas and have no useful disagreement. Or it can preserve different views, then let authority, fluency or a misleading confidence score choose the wrong one.

Influence can preserve or destroy evidence

Disagreement must be measured before collaboration changes it. Once a person has seen an agent's answer, their revised judgement is no longer an independent sample. Once several agents share one scratchpad, later outputs reflect both the task and the earlier agents' framing. This does not make deliberation bad. It means sequence becomes part of the causal mechanism.

Social influence has two faces. Lorenz and colleagues showed that even mild exposure to others' estimates could reduce opinion diversity and undermine the wisdom of crowds in estimation tasks.⁵ Becker, Brackbill and Centola later showed that decentralised networks can improve individual and collective estimates, while centralised influence makes the result vulnerable to the performance of the central individual.⁶ The distinction is not influence versus isolation. It is whether the network corrects errors without allowing one source to become an unexamined centre of gravity.

Small-group structure offers one route. Navajas and colleagues asked 5,180 people to estimate quantities individually, deliberate in groups of five and produce group consensus. Averaging a small number of independent group consensuses outperformed the initial crowd aggregation.⁷ Deliberation occurred inside groups; diversity survived between groups. Bernstein, Shore and Lazer found a related benefit for intermittent interaction on a complex problem-solving task: periods of independent work preserved exploration, while occasional connection spread useful discoveries.⁸

Adaptive networks add another possibility. Almaatouq and colleagues studied networks that changed who observed whom in response to performance feedback. Under the tested conditions, adapting social connections helped groups respond to biased and changing information.⁹ For human-agent systems, the analogue is not unrestricted self-organisation. It is a governed router that updates influence weights from validated outcomes, keeps the change inspectable and withdraws trust when the evidence expires.

These human-group results do not transfer mechanically to agents. Models can share pretraining, retrieval, prompts and tool implementations at a scale that human group studies did not examine. Yet the design principle transfers: interaction should move evidence, not merely copy conclusions. Ask an agent to provide the observation that discriminates between hypotheses, the failed test that changed its view, or the source that another member lacks. A second paraphrase of the same recommendation adds social pressure without adding information.

More explanation is not a reliable cure. Bansal and colleagues found that explanations in three AI-assisted decision tasks did not necessarily improve complementarity and could increase acceptance of AI recommendations whether those recommendations were correct or incorrect.¹⁰ A fluent rationale may improve the apparent coherence of the team while worsening q, because it makes wrong advice harder to reject. The 2024 meta-analysis similarly did not find explanations or confidence displays to be significant moderators across its study set.¹

The danger is visible in generative work as well. Doshi and Hauser found that access to generative AI could improve the judged quality of an individual's short story, especially for less creative writers, while making the collection of stories more similar to one another.¹³ Better members did not automatically produce a better population of ideas. A shared generator can raise each local score while narrowing the search space the group explores.

Creation tasks therefore need a different collective benchmark. There may be no single correct answer and no simple four-cell ledger. Measure quality and usefulness alongside novelty, coverage of the design space, duplication, downstream selection and the value of the best discovered candidate. Preserve independent ideation before critique, then test whether interaction improves the frontier rather than merely moving everyone towards the same polished centre.

Confidence can help when it discriminates correct from incorrect outputs. Recent work by Li and Steyvers separates base accuracy from metacognitive sensitivity, the ability of confidence to rank one's own correct answers above one's errors. In their model and behavioural experiment, higher AI metacognitive sensitivity improved assisted decisions; under some conditions, a less accurate but more self-discriminating assistant produced greater team value.¹⁵ This is promising, but it does not license a generic confidence badge. The confidence signal, task slice and human use of it must all be measured.

The explicit failure boundary

The mechanism breaks in four important conditions. First, D = 0: members never disagree, so no selector can create value from their combination. Second, both members share a systematic blind spot concentrated in p00; new evidence is required. Third, one member wins nearly every disagreement, pushing the arbitration threshold towards perfection. Fourth, the interaction protocol alters initial judgements before their error structure can be observed.

A recent preprint derives related error-correlation limits for confidence-based human-AI aggregation and explicitly excludes interactive deliberation that can generate new answers.¹⁶ That boundary matters. The four-cell model governs selection between existing answers. A team may exceed its oracle ceiling by retrieving a missing document, running a calculation or forming a new hypothesis. In that richer setting, the unit of analysis becomes the trajectory: which intervention produced new evidence, and whether the gain survives a simpler retrieval or tool-use baseline.

When shared error dominates, add an evidence source before adding another voice. A second model family, independent sensor, alternative retrieval index, counterfactual test or specialist may create new information. Another agent role wrapped around the same model and corpus usually does not.

Turn a collection into a tested decision system

Consider a synthetic payment-recall triage service. Its job is to assemble evidence and recommend whether a case should proceed, pause for missing information or escalate to an authorised specialist. This is a worked scenario, not a deployment report.

The agent is strong at scanning transaction history, matching beneficiary changes, detecting cross-account patterns and retrieving the applicable procedure. The human investigator is stronger at interpreting contradictory customer narratives, recognising unusual commercial context and judging when the evidence is insufficient. A deterministic service is stronger than either at identity checks, permission, cut-off calculations, idempotency and postcondition readback.

A naive workflow gives the agent's complete recommendation to the investigator, asks for approval, then runs a tool. This arrangement makes the human responsible while depriving the evaluation of an independent human judgement. It also treats authority as if it were another vote.

The collective workflow separates ten objects:

The investigator first records an assessment from the evidence packet without the model recommendation. The agent independently produces a proposal and a list of decisive claims. Agreement on low-consequence cases can follow a bounded route. Disagreement opens an evidence-oriented packet: what source differs, which fact would reverse each judgement, and whether a deterministic check can settle the issue. The human does not vote against the model in the abstract; the human resolves a specified conflict under declared authority.

The collective benchmark

Evaluate the configured team on the same held-out cases as every solo baseline. Record each member's initial answer before exposure, the evidence available at that moment, the final answer, who influenced it, action cost and observed outcome. Randomise interface order where feasible. Segment by case type, consequence, distribution shift and missing evidence.

The minimum baseline set is: best human; best agent; the configured team; a random disagreement selector; a simple rule that always follows the member with higher validation accuracy; a selective-routing baseline that sends only uncertain cases to the other member; and an oracle disagreement selector that defines the ceiling available without new evidence. A complicated team that cannot beat simple routing has not earned its coordination cost.

Match resource budgets carefully. If the team gets two model calls, a specialist and twice the decision time, compare it with a solo system allowed to spend the same marginal budget on retrieval, verification or a stronger model. Otherwise the experiment attributes the value of extra resources to social composition. Keep the target labels hidden from every adaptive component, and freeze the aggregation rule before the final holdout.

Estimate uncertainty at the level where observations are independent. Repeated decisions from one reviewer, one customer or one case family are clustered, so a naive binomial interval can be too narrow. Report case and reviewer counts, bootstrap or model the relevant hierarchy, and show the confidence interval for the strong-synergy margin rather than only for each accuracy. For delayed outcomes, distinguish proxy labels from verified business outcomes and state how much of the evaluation remains unresolved.

People and agents also learn during a trial. A human may discover the model's strengths, become fatigued or begin to mimic it; a routing agent may adapt its weights. Plot performance and disagreement over time. Improvement can be genuine team learning, while falling disagreement can signal either successful coordination or loss of independent judgement. Outcome-linked evidence, not smooth convergence, separates the two.

Wilder, Horvitz and Kamar argue for training and selecting AI systems by team utility rather than standalone accuracy, particularly where machines can target cases that humans find difficult.¹¹ Bayesian work by Steyvers and colleagues likewise models complementarity through both accuracies and the dependence between human and machine judgements.¹² A medical-diagnosis preprint offers an encouraging open-ended example: hybrid collectives assembled from 40,762 physician differential diagnoses and five language models across 2,133 cases outperformed the examined individual and homogeneous collective baselines.¹⁴ Its preprint status and domain-specific aggregation method limit the claim, but the result illustrates what genuine error complementarity can look like.

function evaluate({p11, p10, p01, p00, q}) {
  const total = p11 + p10 + p01 + p00;
  if (Math.abs(total - 1) > 1e-9) throw new Error("Cells must sum to 1");
  const human = p11 + p10;
  const agent = p11 + p01;
  const disagreement = p10 + p01;
  const team = p11 + q * disagreement;
  const bestSolo = Math.max(human, agent);
  const requiredQ = disagreement === 0
    ? Infinity
    : Math.max(p10, p01) / disagreement;
  return {
    human, agent, team,
    strongMargin: team - bestSolo,
    requiredQ,
    oracle: 1 - p00
  };
}

const positive = evaluate({
  p11: 0.68, p10: 0.12, p01: 0.16, p00: 0.04, q: 0.70
});
console.assert(Math.abs(positive.strongMargin - 0.036) < 1e-9);

const negative = evaluate({
  p11: 0.77, p10: 0.03, p01: 0.05, p00: 0.15, q: 0.55
});
console.assert(negative.strongMargin < 0);

A release decision instrument

The benchmark must test the protocol, not only its members. Run ablations that remove independent commitment, hide confidence, swap output order, replace one agent with another call to the same model, change the retrieval source and inject stale evidence. These interventions identify whether the apparent gain depends on complementarity or on an accidental interface pattern.

NIST's AI Risk Management Framework treats AI risk as socio-technical and asks organisations to measure systems under conditions similar to deployment, including human roles and contextual factors.¹⁷ The collective benchmark makes that principle concrete. The evaluated object is the end-to-end arrangement of people, models, evidence, interface, policy and authority.

The release rule should be statistical and operational. Require the lower confidence bound of consequence-weighted Δstrong to remain above zero on the target distribution, or above a declared minimum improvement that pays for coordination. Set non-compensable gates for severe harm, privacy, unauthorised action and unknown outcomes. Monitor the four-cell ledger after release because a model update, policy change or staff turnover can alter both disagreement and arbitration.

Failure injection should target the hypothesised mechanism. Give both agents the same stale source and check whether the system recognises that agreement is dependent. Swap the order of recommendations. Corrupt a confidence signal while leaving the answer unchanged. Remove the human-only evidence source. Introduce an unfamiliar but valid case where both models share a training shortcut. Disable the outcome readback after an approved action. Each fault asks a different question: did independence vanish, did arbitration become status-driven, did the team know that both members were outside scope, or did decision quality become disconnected from verified effect?

A release should also survive member replacement. Replace the named human with another trained reviewer and the agent version with a behaviourally equivalent candidate. If the gain depends on one unusually skilled person, one undocumented prompt quirk or one evaluator who knows the test set, the system has demonstrated a partnership anecdote rather than a reusable collective capability. The operating claim must name its population, training, evidence access and expiry conditions.

One observation would weaken the central argument: a repeatable class of tasks where teams beat the strongest solo member despite high shared error, no preserved independent evidence and arbitration no better than chance. Such a result would imply that another mechanism, perhaps interactive generation of new hypotheses, motivational effects or richer tool access, explains the gain. The correct response would be to widen the model and test that mechanism directly, not to protect the three-condition account.

The architecture decision changes from “add a human and another agent” to “buy an independent evidence path, measure the disagreement, and prove the arbitration”. Headcount follows only when those three quantities justify it.