The hidden quantity is answer mass

A language model does not choose an answer in one indivisible act. It assigns conditional probability to the next token, then the next, creating a distribution over complete sequences. A visible response contains a reasoning path, formatting choices and a final answer. Two sequences can differ almost everywhere yet imply the same decision.

Greedy decoding chooses the highest-probability token at each step. Even an idealised decoder that found the globally most probable complete sequence would still optimise a trajectory. The user usually cares about an answer class. Those objectives coincide only when each answer has one relevant surface form or when the highest-probability path belongs to the answer with the greatest total mass.

The self-consistency method introduced by Wang and colleagues samples diverse chain-of-thought paths and selects the answer that appears most consistently after those paths are marginalised. Their experiments reported substantial gains over greedy chain-of-thought decoding on several arithmetic and commonsense benchmarks. The mechanism matters more than the headline: the method changes the unit of selection from one sequence to an empirical answer distribution.

Self-consistency is Monte Carlo marginalisation

Let the model produce n samples, each containing a path and a final output. A normaliser g maps the output to an answer class. The empirical support for answer a is:

This estimator is simple, but each symbol hides an engineering obligation. Samples must vary enough to explore alternatives. The normaliser must preserve semantic equivalence without merging materially different answers. The number of samples must be large enough for the observed frequencies to stabilise. The output class must be meaningful for the task.

Minimal worked example

Suppose we draw eleven paths from the illustrative distribution. One possible sample contains six answers of 117, three of 126 and two of 156. Plurality vote returns 117. Another draw may contain four, four and three, leaving a tie. A third may over-sample the 126 path. Self-consistency does not guarantee the true answer on every draw. It is a stochastic estimator whose error falls only under assumptions about the distribution and dependence between samples.

The first practical implication follows immediately: evaluate the whole sample-and-select procedure, not the base model's pass@1 and not an oracle that always spots a correct candidate. The model, sampling policy, normaliser, selector, stopping rule and tie policy together define the system.

When a minority answer can still be the mode

The familiar majority-vote intuition comes from a binary setting. If every independent voter is correct with probability greater than one half, an odd-sized majority becomes more reliable as the number of voters grows. Language-model answers are usually multi-class. The correct answer may receive 40 per cent of the mass while two different wrong answers receive 32 and 28 per cent. It is a minority of all samples but still the plurality class. Conversely, a binary correct probability of 40 per cent is usually made worse by majority voting because the wrong class owns the mode.

This distinction changes what should be measured. Per-sample accuracy is insufficient. The experiment needs the entire answer-frequency profile for each question: the rank of the correct class, the gap between the leading two classes, the number of semantically distinct wrong modes and how those quantities change with the sampling policy. Averaging them over a benchmark can hide a dangerous subset where one systematic wrong answer dominates.

Nor is the vote share a calibrated probability that the answer is true. It estimates how often the configured generator emits that class under repeated draws. A 90 per cent vote can arise because the evidence is decisive, because the prompt strongly anchors one interpretation, or because every sample inherits the same shortcut. Agreement is evidence about the generator's distribution before it is evidence about the world.

A practical system should therefore store two margins. The sampling margin is the difference between the leading answer classes. The verification margin is the difference after applying independent checks or scores. A large sampling margin with a weak verification margin is a warning, not a release condition. It says the model is stable while the evidence remains unresolved.

flowchart TD
  A["The answer hidden behind the<br/>second-best paths"] --> B["The hidden quantity is answer mass"]
  B --> C["A crowd of one needs three gates"]
  C --> D["Coverage is not selection"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#5B5CF6,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

A crowd of one needs three gates

Calling repeated outputs a “crowd” can obscure the main risk. Every answer may come from one model, one prompt, one context order and one set of learned shortcuts. The count can rise while the information barely changes. Three gates determine whether sampling produces evidence or merely repetition.

Gate one: the useful answer must be reachable with effective diversity

If one draw has probability p of producing at least one acceptable candidate and draws are independent, the probability of seeing an acceptable candidate by n attempts is 1 − (1 − p)ⁿ. This is a coverage result. It says nothing about selecting that candidate. It also collapses when attempts are dependent.

Independence is not a cosmetic assumption. Same-model samples share parameters, training data, prompt wording and evidence order. Increasing temperature can create surface variety while preserving the same hidden mistake. Using several models may diversify phrasing yet retain correlated errors. A 2025 study across more than 350 language models found substantial error correlation, including agreement on wrong answers in 60 per cent of jointly wrong cases on one leaderboard dataset. That result concerns cross-model populations, not self-consistency draws, but it is a warning against treating vote count as independent testimony.

A useful approximation for equally correlated measurements is:

Diversity is an intervention, not a temperature setting

Temperature changes the sharpness of token sampling. It can expose lower-probability paths, but it can also increase arithmetic slips, malformed schemas and irrelevant detours. A rise in lexical variety does not establish a rise in useful diversity. The intervention has earned its cost only when it changes decision-relevant features while preserving enough competence for the selector to recover them.

Useful diversity can come from several places. Different random seeds explore local alternatives. Different decompositions alter which intermediate dependency is solved first. Context permutations test whether evidence order controls the answer. Independent retrieval queries expose different source subsets. Different model families may reduce some shared errors, although cross-model correlation remains possible. Tool-generated calculations create a qualitatively different evidence channel. Each intervention changes a different causal feature and should be evaluated separately.

A matched ablation is more informative than one large “ensemble” comparison. Hold the task, evidence and sample budget fixed. Compare: seed-only variation; temperature variation; decomposition variation; evidence-order variation; model-family variation; and a deterministic calculation or retrieval baseline. For each condition, measure unique strings, unique answer classes, unique cited evidence, pairwise semantic distance, oracle coverage and selected accuracy. The desired intervention raises coverage or selection accuracy without merely inflating surface distance.

One particularly revealing test is context permutation under answer aggregation. Shuffle equally authoritative passages while preserving their contents. If the vote changes sharply, samples are not independent views of stable evidence. They are repeated responses to a positional perturbation. That is precisely the kind of shared mechanism that a large vote count can conceal.

Gate two: equivalent answers must be recoverable

Closed-form arithmetic makes aggregation look easier than it is. The strings “117”, “117 cases” and “one hundred and seventeen” can be normalised safely. Code candidates may be grouped by execution outputs on discriminating tests. A classification task can map synonyms to one label. Open-ended analysis has no such obvious equivalence relation.

A brittle exact-match vote can split one correct semantic answer across many phrasings while a repeated misconception keeps one identical string. Universal Self-Consistency was proposed partly to address this limitation by asking a model to select the most consistent free-form response. That broadens the method, but it moves risk into the selecting model. Semantic similarity can also merge answers that differ in scope, time, exception or authority.

Gate three: the selector must discriminate

When the correct class is the mode, majority vote can work. When correct candidates are rare but mechanically checkable, an executable verifier can work. When one wrong mode dominates, a vote returns it more confidently. When a learned reward model shares the proposer's blind spot, best-of-N may select polished error.

This gate separates two questions that are often blended: did the model ever generate a correct candidate? and did the system choose it? The first is reachability or coverage. The second is precision under selection. Both are necessary for operational success.

Six selectors make six different claims

The phrase “select the best answer” is too coarse for architecture. An exact vote claims that literal frequency is meaningful. A canonicalised vote claims that a specified equivalence relation preserves every decision-relevant difference. Execution clustering claims that matching behaviour on available tests is a useful proxy for semantic equivalence. An outcome verifier claims it can recognise a successful final result. A process verifier claims intermediate steps predict eventual correctness. Human adjudication claims an accountable reviewer has sufficient evidence and time to resolve the case.

These claims are not interchangeable. Exact vote is cheap and transparent but fragments paraphrases. Semantic clustering can recover paraphrases but may merge an exception with a general rule. Execution is strong when tests are discriminating, yet two wrong programs can agree on weak test inputs. An outcome verifier can choose a rare correct candidate, but its score may reward polished form or familiar reasoning. A process verifier can guide search before completion, but local plausibility need not imply a correct global answer. Human review adds contextual judgement, though reviewers can anchor on a model's majority and inherit its framing.

The correct baseline is therefore selector-specific. Compare exact vote with a deterministic normaliser, not only with another language model. Compare a learned semantic selector with blinded human equivalence judgements and adversarial near-neighbours. Compare code execution with mutation tests that expose under-specified cases. Compare best-of-N with random selection from the same candidate set to measure how much value the verifier actually adds. Compare process-guided search with independent full-path sampling at equal token and latency budgets.

This paper stops at that architecture boundary. The next question is whether outcome and process verifiers detect different failure classes and distort different incentives. Here the narrower point is enough: a sampling claim is incomplete until it names the selector and the proposition that selector can legitimately establish.

A stronger empirical boundary now exists for long contexts. Byerly and Khashabi report that self-consistency can degrade performance on long-context tasks when repeated samples amplify persistent positional bias. The finding matters because it shows that agreement can be a symptom of a shared retrieval failure rather than evidence of a robust answer.

Coverage is not selection

Repeated sampling is easiest to understand in a domain with a decisive checker. A candidate program either passes a sufficiently discriminating test suite or it does not. A formal proof either checks or fails. More samples can raise the probability that at least one candidate passes, and the checker converts coverage into usable success.

Without such a checker, additional candidates can expose the correct answer while making the choice harder. Brown and colleagues measured this gap on a 128-problem MATH subset with Llama-3-8B-Instruct. Coverage, defined by an oracle that knows whether any sample is correct, rose from 79.8 per cent at 100 samples to 95.3 per cent at 10,000. Majority-vote and reward-model selection methods moved only from 38.7 to 39.8 per cent over the same range. The model increasingly contained the answer, but the system could not recover it.

Coverage is an upper bound only under a valid oracle. Even mechanical verification can be incomplete. Unit tests may miss a defect, pass a brittle patch or depend on a flaky environment. A proof checker establishes formal validity relative to an encoding and axioms, not whether the formal statement represents the user's intent. Verification shifts the boundary; it does not dissolve it.

Sampling is not yet search

Independent sampling spends one complete trajectory on each attempt. Self-consistency adds an equivalence relation and an aggregator. Best-of-N adds a scorer that ranks complete candidates. Search makes sequential allocation decisions: expand this prefix, prune that one, backtrack from another. These distinctions determine where compute goes and which errors can be corrected.

Tree of Thoughts formalised one influential version of this idea: maintain several intermediate “thoughts”, evaluate progress and explore with breadth-first or depth-first search. Later test-time-compute work compared best-of-N, beam search and lookahead with process reward models, showing that the best allocation depends on question difficulty and budget. No topology is universally superior because the selector's reliability changes along the trajectory.

Sampling estimates a distribution; search actively reshapes which parts of that distribution receive compute. Search is justified when partial states expose useful signals, later errors depend strongly on earlier choices, and backtracking can recover. Independent sampling remains a strong baseline when complete trajectories are cheap and prefix scores are untrustworthy.

A production-shaped case: evidence-bound policy interpretation

A naive design samples twelve free-form memos and asks another model to choose the most persuasive. That design confounds reasoning, style and authority. A better design fixes the evidence bundle, asks each candidate to emit the same schema, and separates four fields: disposition, clause identifiers, conditions and unresolved facts. The system canonicalises only the disposition labels. It does not merge conditions or citations.

Candidate diversity is induced by varying decomposition order and requiring independent calculation of effective dates, not by merely raising temperature. A deterministic checker rejects candidates citing absent clauses or using superseded versions. Remaining candidates are grouped by disposition. Agreement is treated as a routing signal. A separate evidence checker tests whether each cited clause entails the proposed condition. If the leading class lacks a sufficient margin, contains contradictory conditions or depends on missing facts, the system abstains.

This design makes the three gates observable. Reachability is measured as oracle coverage on a human-adjudicated test set. Recoverability is audited with pairs that should and should not canonicalise together. Discrimination is measured as end-to-end selected accuracy, calibration by vote margin and error under injected shared biases such as moving the decisive clause to the middle of a long context.

Evaluation design for the policy case

The release experiment should use a frozen set of synthetic or public policy cases with adjudicated dispositions, clause-level support and explicit unresolved facts. Partition by policy family, effective period and document length so that near-duplicate clauses do not leak between development and test sets. Preserve a hidden challenge slice containing superseded clauses, contradictory passages, missing evidence, negation and decisive text placed near the middle of a long context.

Report pass@1, oracle coverage@N and selected accuracy@N as separate curves. Add normalisation false-merge and false-split rates, because a selector can appear better by collapsing distinctions the evaluator forgot to preserve. Record the effective number of answer classes, citation diversity, vote margin, verifier margin, abstention rate, conditional error after release, tokens, latency and cost. For operational relevance, weight errors by consequence rather than treating every disposition as equally reversible.

The baseline ladder should be deliberately uncomfortable: deterministic rules for cases they fully cover; one greedy model proposal; N samples with exact vote; N samples with the typed normaliser; N samples with independent evidence verification; and a human review condition with candidates hidden or shuffled to test anchoring. Search is added only if prefix-level decisions can be defined and validated. The preferred design must beat the simplest adequate baseline on the same cases and budget.

Failure injection carries the causal burden. Duplicate one misleading clause across the context, move the decisive clause between positions, corrupt one citation identifier, split equivalent answers across surface forms, and force all samples through the same wrong intermediate calculation. A robust system should reveal which gate failed: coverage, equivalence or discrimination. If selected accuracy rises only when an oracle normaliser or oracle verifier is used, the research result is a diagnosis, not a deployable gain.

A release threshold can then be expressed in decision terms: sampling is retained only where it reduces weighted error beyond pass@1, survives the shared-bias controls, keeps normalisation error below its bound and provides enough marginal value to justify latency. Cases outside that envelope route to another mechanism. This makes “use self-consistency” a falsifiable design decision rather than a default prompt pattern.

Spend attempts where they change the decision

A fixed sample count is easy to implement and difficult to justify. Easy questions may stabilise after three attempts. Ambiguous questions may consume forty without producing useful diversity. Some hard questions have a rare correct path worth searching for. Others are outside the model's support and should trigger retrieval, calculation or human review.

Adaptive consistency uses observed agreement to stop early and has reported large reductions in sample budgets on its studied reasoning and coding tasks. The general idea is sound: allocate compute according to uncertainty and marginal gain. Yet agreement alone is an unsafe stopping rule when shared bias is plausible. A robust controller combines agreement with evidence checks, diversity diagnostics and task consequence.

A four-question decision instrument

Stopping is part of the estimator

A controller can process samples in batches. After each batch it records the leading answer class, vote margin, entropy, number of unique classes, verifier score, evidence coverage and estimated cost of another batch. It stops when a release threshold is met, when marginal improvement is small, or when a failure trigger fires.

Useful failure triggers include a stable disagreement between vote and verifier, repeated citations to the same weak evidence, a high lexical-diversity score with no change in decision features, contradictory conditions within the leading class, and long-context sensitivity to evidence order. These signals do not prove error. They show that extra samples are not supplying the kind of evidence the selector assumes.

The release decision should preserve uncertainty rather than laundering it through a plurality. A sample-and-select system should emit the chosen answer, the candidate distribution after canonicalisation, the selection method, the stopping reason, failed checks and unresolved alternatives. For consequential use, this becomes part of a decision receipt, not an invisible internal trace.

Marginal value is a loss calculation

The next sample has value only through a possible change in action. Let the current candidate set imply an expected decision loss, including false release, unnecessary abstention and delayed service. Estimate the loss again after one more batch, then subtract inference cost and latency cost. Continue only when the expected reduction is positive and the uncertainty in that estimate is acceptable. The calculation can be rough, but its terms should be explicit.

This prevents an accuracy-only optimiser from spending heavily on reversible, low-consequence questions while under-protecting rare high-consequence ones. A one-point gain may be valuable when it avoids an irreversible payment or a harmful policy disposition. The same gain may be wasteful for a draft that a person will edit. Consequence also changes the selector: a high vote may release a low-risk suggestion, while the same vote merely prioritises human review for a consequential decision.

Adaptive budgets should be learned from held-out behaviour, not improvised from one live answer's confidence. Fit stopping thresholds on representative cases, test them under drift and cap total spend. Include a hard exit when evidence is missing or authority is absent. Inference compute cannot repair an information deficit or create permission to act.

from collections import Counter
from random import Random


def trial(p=0.62, n=15, rho=0.05, fracture=0.05, seed=0):
    """Return coverage, exact-vote success and semantic-vote success."""
    rng = Random(seed)
    shared = rng.random() < rho
    shared_outcome = rng.random() < p if shared else None
    labels = []

    for _ in range(n):
        correct = shared_outcome if shared else rng.random() < p
        if correct:
            # C1..C4 are semantically equivalent but literally distinct.
            label = f"C{rng.randrange(1, 5)}" if rng.random() < fracture else "C"
        else:
            label = "W"
        labels.append(label)

    coverage = any(label.startswith("C") for label in labels)
    counts = Counter(labels)
    top_count = max(counts.values())
    tied = [label for label, count in counts.items() if count == top_count]
    exact_winner = rng.choice(tied)
    exact_success = exact_winner.startswith("C")

    correct_votes = sum(label.startswith("C") for label in labels)
    semantic_success = correct_votes > n // 2
    return coverage, exact_success, semantic_success


def estimate(config, trials=20_000):
    rows = [trial(**config, seed=i) for i in range(trials)]
    return tuple(sum(row[j] for row in rows) / trials for j in range(3))


positive = dict(p=0.62, n=15, rho=0.05, fracture=0.05)
correlated_negative = dict(p=0.62, n=15, rho=0.90, fracture=0.60)
fracture_negative = dict(p=0.62, n=15, rho=0.00, fracture=0.80)

for name, config in [
    ("positive", positive),
    ("correlated", correlated_negative),
    ("fracture", fracture_negative),
]:
    coverage, exact_vote, semantic_vote = estimate(config)
    print(name, {"coverage": coverage,
                 "exact_vote": exact_vote,
                 "semantic_vote": semantic_vote})

Explicit failure boundary

Sampling ceases to help when the correct answer has negligible support, when attempts share the same causal error, when the answer cannot be canonicalised without losing decision-relevant distinctions, when the verifier is weaker than the proposer on the relevant failure modes, or when the cost and latency of another attempt exceed the expected reduction in decision loss.

There is a sharper boundary for open-ended work: no plurality rule can establish truth when quality is plural, context-dependent or value-laden. Multiple drafts may support exploration, comparison and critique, but their agreement is not a ground-truth oracle. In those settings, sampling should expose alternatives to an accountable evaluator, not manufacture a false singular answer.

The boundary must also be checked by slice, not only in aggregate. Sampling may help short arithmetic cases and harm long evidence cases in the same product. It may improve common classes while concentrating error on rare ones. Report gains by context length, answer frequency, consequence, evidence position and disagreement pattern. A global average cannot authorise sampling for a slice where the correct class is systematically non-modal or where the selector's false confidence is most costly.