Part I

The judge is a sensor with nuisance responses

Model-based evaluation has a legitimate attraction. Human comparison is expensive, slow and difficult to repeat at the scale of prompt, model and retrieval changes. Early systems such as G-Eval showed that a carefully prompted strong model could align better with human ratings than earlier automatic metrics on selected summarisation and dialogue tasks.^[3] MT-Bench and Chatbot Arena likewise demonstrated practical value while explicitly documenting position, verbosity, self-enhancement and reasoning limitations.^[1] The evidence therefore supports neither worship nor dismissal. It supports instrument design.

A thermometer is useful because its response to temperature has been characterised, its scale is traceable and known disturbances are controlled. It is not trusted because it emits a number. The metrology vocabulary distinguishes calibration, measurement uncertainty and the conditions under which a result applies.^[15] An LLM judge needs the same discipline, although its internal transformation is more complex and its behaviour may change with the prompt, model version and case distribution.

What the score actually contains

Consider a pairwise comparison. Let response A and response B differ in task quality, but also in which one appears first, how long each is, how polished the prose sounds and whether its style resembles the judge’s own family. A simple diagnostic model is:

Here Jᵢ is the latent preference expressed by the judge on case i. Qᵢ is the rubric-relevant quality difference. Pᵢ, Lᵢ, Sᵢ and Fᵢ represent position, length, style and family affinity. The coefficients show how strongly each feature moves the verdict. εᵢ gathers idiosyncratic variation and omitted effects.

This equation is a diagnostic abstraction, not a claim that a language model literally performs linear regression. Its value is causal. Repeating a call and averaging can reduce the contribution of εᵢ. It does not force the four nuisance coefficients towards zero. Repeatability is the stability of a reading; validity is whether the reading responds to the construct the decision names. A judge can be highly repeatable and consistently wrong for the same reason.

The estimand comes before the judge prompt

A team often begins by writing “You are an expert evaluator” and only later asks what the resulting number means. Reverse that order. Define the estimand first. It might be the proportion of drafts that trained reviewers would accept under a named rubric, the probability that candidate A is preferred to candidate B on a stated population, or the rate of claim-level support failures. These are different quantities. They need different units, samples and reference processes.

Pairwise judgement can reduce the burden of inventing an absolute scale, but it introduces order, tie and transitivity questions. A pointwise score avoids a visible opponent, yet a rating of seven is not automatically twice as good as a rating of 3.5, nor is the distance from six to seven necessarily comparable with the distance from eight to nine. Treating ordinal categories as interval measurements can create false arithmetic before model error is considered.

Criterion decomposition helps only when the criteria retain the decision. “Accurate”, “complete”, “relevant” and “well written” are not independent switches. A response may be complete because it states an exception, and that exception may make it longer. The evaluation specification should state which criterion dominates when they conflict. In the complaint case, an unsupported remedy is a hard defect; warmth cannot compensate for it. The judge should return a typed record with criterion label, cited response spans, an uncertainty state and a reason code. Aggregation belongs in a deterministic rule whose weights and vetoes are inspectable.

A judge prompt is part of the measurement procedure, not a natural-language description of an already existing scale. Changing examples, criterion order, labels or requested rationale can change the instrument. Preserve the prompt, parser, model version, decoding and case presentation as one manifest. Otherwise two teams can report the same metric name while measuring different things.

Minimal worked example: one pair, four readings

Suppose A is factually correct and concise. B is equally clear but contains one unsupported claim and is 60 per cent longer. The judge chooses B when B appears first. Swap the order, and it chooses A. Restore the original order but remove B’s unsupported sentence, and it still chooses B. Compress B to the same length as A without changing meaning, and it chooses A. The four readings reveal at least two mechanisms: position and length influence the verdict, while the factual defect is underweighted.

The example does not prove that every long answer is overvalued. It specifies what must be measured. A judge may legitimately prefer a longer response when the added material satisfies a completeness requirement. The relevant counterfactual holds rubric-relevant meaning fixed while changing presentation. Without that control, “length bias” can be a careless name for useful sensitivity to evidence.

Thought experiment: the silent swap

Give the judge the same two responses twice. In the first trial A is listed first; in the second B is listed first. No wording, rubric or evidence changes. If the verdict changes, the judge has violated an invariance the evaluation claim normally assumes. Work on positional bias has shown that rankings can be manipulated by changing response order, and that balanced-position methods can mitigate the problem rather than make it imaginary.^[2]

Running both orders is therefore not merely “asking twice”. It is an intervention on one suspected cause. The resulting flip rate has direct operational meaning. A five per cent flip rate says that one in twenty pair decisions is order-contingent under this test distribution. Whether that is tolerable depends on the decision, but hiding it inside a single agreement statistic is not defensible.

Part II

flowchart LR
  A["The release that improved by<br/>getting worse"] --> B["The judge is a sensor with<br/>nuisance responses"]
  B --> C["Counterfactuals expose what<br/>the judge is rewarding"]
  C --> D["Calibration needs a human<br/>anchor without pretending<br/>humans are perfect"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#007D68,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Counterfactuals expose what the judge is rewarding

The nuisance features matter because they are rarely distributed evenly. A new model may systematically produce longer answers. A vendor’s models may share a recognisable rhetorical signature. One candidate may always occupy the first slot because of an evaluation script. When the nuisance correlates with system identity, a small per-case response becomes a ranking bias.

Thought experiment: the same answer in a different suit

Begin with one answer whose factual content and instruction compliance have already been checked. Produce two semantically equivalent variants. The plain version uses short paragraphs. The dressed version adds headings, a summary, polished transitions and confident closure. Ask the judge which is more accurate. If the dressed version wins repeatedly, the judge is treating style as evidence for a criterion style cannot establish.

Now change the criterion to “which response is easier for a hurried reader to navigate?” The same sensitivity may become legitimate. A feature is a bias only relative to the construct and intervention being claimed. This is why generic accusations such as “LLMs prefer verbosity” are less useful than an audit that names the rubric, population, counterfactual and observed effect.

Length-controlled AlpacaEval made this logic operational by estimating the counterfactual preference if candidate and baseline outputs had equal length. In its reported setting, controlling length improved robustness and increased correlation with Chatbot Arena rankings.^[6] Other work constructed intentionally flawed answers and found that factual errors could be rated more favourably than brevity or grammatical defects, exposing a style-over-substance failure.^[7] These findings do not supply a universal coefficient. They demonstrate interventions worth running locally.

Build a bias atlas instead of a blacklist

A useful audit does not label the judge “biased” and stop. It maps the direction, magnitude and conditions of each response. CoBBLEr tested six proposed cognitive biases across multiple evaluators and reported bias indications in roughly 40 per cent of comparisons in its benchmark, alongside limited overlap between machine and human rankings.^[4] That result describes a designed stress test, not the expected failure rate of every deployment. Its value is the catalogue of perturbations.

For each nuisance, retain four records: the intervention used, the affected slices, the effect size and the decision threshold. Position may matter only on close pairs. Verbosity may dominate broad helpfulness rubrics but disappear under claim-level factual checks. Family affinity may appear after a candidate model change. A bias atlas makes these conditional patterns visible and prevents a mitigation for one regime from being treated as a permanent cure.

Mitigation itself needs a control. Swapping positions twice and averaging can reduce first-position preference, but it doubles calls and may create inconsistent pairs. Style normalisation can remove the very clarity difference the rubric intends to measure. Regression adjustment can extrapolate beyond the calibration support. Every mitigation should therefore be evaluated on held-out human-labelled cases and compared with the unmitigated procedure. A correction earns its place by improving the target estimate, not by making the diagnostic plot look flatter.

Self-preference is more than a vendor label

A judge can favour outputs associated with itself or its model family. Controlled work found non-trivial self-recognition and a relationship between elicited self-recognition and self-preference, with experiments designed to test plausible confounders.^[5] The operational risk is broader than a model literally reading an author name. Token choice, sentence rhythm, refusal style and preferred explanatory structure can act as a signature.

Blinding model names is necessary but insufficient. The stronger intervention is to paraphrase or style-normalise matched subsets, compare same-family and cross-family errors, and avoid using the generating model as the sole judge of its own candidate. A release process should also inspect whether a judge upgrade changes which candidate families gain. Version pinning preserves reproducibility; overlap calibration explains whether old and new scales remain comparable.

Why nine judges may carry only two opinions

A common response is to create a panel. Majority voting helps when errors are sufficiently independent and each voter is better than chance. Language-model judges may violate the independence premise because they share training data, evaluation conventions, prompt-induced shortcuts and difficulties. They can agree because the answer is right, or because the same polished error attracts all of them.

A recent preprint tested nine frontier judges from seven model families and estimated that their errors provided roughly two independent votes’ worth of information on the studied tasks. The best single judge matched or exceeded the full panel under the reported conditions.^[14] The result is recent and should not be generalised beyond its experiments. Its decision-relevant lesson is durable: count error diversity, not model logos.

For a panel, record a binary error vector for every judge on the human-anchor set. Estimate pairwise error correlations by slice. A simple Kish-style approximation is n_eff = n / [1 + (n - 1)ρ̄], where n is the nominal number of judges and ρ̄ is their average error correlation. It is a diagnostic, not a complete theory of majority accuracy. When ρ̄ rises, adding another similar judge contributes little independent evidence.

Evaluation changes when candidates can learn the judge

So far the candidate systems have been passive. In a development loop, they are selected, prompted or trained against the judge. Any stable preference can then become a target. A system that discovers that long, enumerated answers score well may produce more enumeration. A model trained on judge rewards may learn the judge’s preferred uncertainty language without becoming better calibrated. The metric stops observing a fixed population and begins shaping it.

This is an evaluation form of Goodhart pressure: once the reading controls selection, candidate behaviour moves towards whatever the instrument rewards, including its defects. Protect a human-only holdout that is never used for prompt tuning. Rotate adversarial cases. Measure whether gains persist after style normalisation and on deterministic outcomes. Keep judge rationales out of the candidate context when they would expose the scoring shortcut. If the candidate has been directly optimised against a judge, treat that judge as in-sample evidence and require an independent oracle for release.

The same issue affects panels. Training against the majority may produce outputs that exploit the shared attractor more effectively than any one model. Diversity must include different evidence channels, such as execution results, source checks and people, rather than only more generative critiques. An evaluator that participates in optimisation needs a stronger independence test than an evaluator used once.

The serious negative control

There are cases where an LLM judge is unnecessary, and that is useful evidence. If the criterion is whether a JSON object validates, a cited passage contains the claimed date, a calculation equals the reference result or a mandatory disclosure appears, code can usually decide more directly. Give the model judge these cases as a negative control. If a deterministic oracle and the judge agree almost perfectly, the model adds cost and uncertainty without adding judgement.

There are also cases where nuisance effects are small under a tightly specified rubric. Strong judges have achieved substantial human agreement in bounded evaluations, and fact-oriented tasks may behave differently from broad stylistic preference.^[1][3][16] This competing explanation matters. The right conclusion is not that model judging is inherently invalid. It is that the burden of proof lies with the local measurement system, not with a benchmark reputation.

Part III

Calibration needs a human anchor without pretending humans are perfect

Calibration links readings to a reference process. For an LLM judge, the reference is usually a sample adjudicated by people under a rubric. That phrase contains three constraints. It is a sample, so sampling error and coverage matter. It is adjudicated, so disagreements need a procedure. It is under a rubric, so the labels are answers to a defined question, not context-free truth.

Human preference platforms such as Chatbot Arena demonstrate how anonymous pairwise comparisons can produce valuable evidence at scale.^[10] A production anchor may require domain experts instead of general users, particularly when one omitted exception changes acceptability. The anchor should include routine cases, close calls, known incidents, model-family variation, terse and verbose outputs, and cases where the correct state is “indeterminate”. Random sampling alone may underrepresent the difficult slices that govern release risk.

Agreement alone hides the direction of harm

An overall agreement rate combines different errors. Imagine a judge that agrees with reviewers on 90 per cent of cases because 90 per cent are acceptable, yet accepts every response. It has perfect sensitivity and zero specificity. It is useless for detecting the failures the release gate exists to stop. Prevalence also changes agreement: the same judge can look better when easy positive cases dominate the sample.

Report the confusion matrix before a single agreement score. For pairwise preferences, show accuracy on human-A and human-B cases, tie handling and order consistency. For ordinal ratings, show adjacent versus material disagreements and a cost matrix that reflects consequence. Rejecting a safe answer and accepting an unsupported financial promise need not carry the same operational weight. Calibration by consequence can be more informative than one symmetric average.

Inter-rater agreement among humans belongs beside judge-human agreement. Low human agreement may reveal poor instructions, missing evidence or genuine plurality. High human agreement on a narrow expert rubric can support a stronger anchor, though it still does not prove the rubric measures the outcome the organisation cares about. Construct validity travels through the whole chain: business claim, rubric, reviewer decision, judge reading and release action.

Rating indeterminacy is data, not annotation failure

Some prompts admit several acceptable responses. Reviewers may reasonably disagree about whether a caveat is necessary, whether a tone is proportionate or whether two incomplete answers deserve the same rating. Forcing one label can make the reference less truthful. Research on rating indeterminacy showed that forced-choice validation could mis-rank judge systems and reported a case where the selected judge was 31 per cent worse than the true top judge under response-set validation.^[12]

A defensible anchor therefore records one of three structures. Use a single label when policy or evidence settles the matter. Use a response set when several ratings are defensible. Use an unresolved state when the rubric, source or policy owner cannot yet decide. Disagreement is sometimes a property of the task that the evaluation system must preserve. Collapsing it into majority truth can train the judge to imitate arbitrary closure.

From raw judge rate to calibrated estimate

Suppose the judge labels each system response as acceptable or unacceptable. On the human-anchor set, estimate sensitivity q₁ = P(judge accepts | human accepts) and specificity q₀ = P(judge rejects | human rejects). On the large test set, let p̂ be the raw fraction the judge accepts. A classical misclassification correction, recently developed with uncertainty treatment for LLM judging, estimates the human-anchored acceptance rate as:^[13]

The numerator removes false acceptance implied by the estimated specificity. The denominator rescales the remaining signal by the judge’s ability to distinguish the two classes. When both sensitivity and specificity rise, correction becomes more stable. When q̂₀ + q̂₁ approaches one, the denominator approaches zero. The judge is then near chance as a discriminator, and a corrected point estimate can become violently unstable.

The denominator is a failure boundary, not a numerical inconvenience. Do not clip an absurd corrected value and continue. Withdraw the judge from that slice, gather a larger or better anchor, improve the rubric, or use a different oracle. Confidence intervals must reflect uncertainty in both the large judged sample and the smaller calibration sample. A narrow interval around the raw rate is misleading when the sensitivity and specificity estimates are uncertain.

Calibrate by slice, then keep watching

One overall confusion matrix can conceal the cases that matter. Estimate performance separately for close comparisons, safety refusals, long outputs, minority languages, candidate families, tool-using responses and high-consequence categories. A judge that is useful on ordinary summaries may be near chance on mathematical proofs or code. LLMBar and JudgeBench were created precisely because easy, natural pairs can make a judge look stronger than hard pairs with objective distinctions.^[8][9]

Calibration is also perishable. A new judge model, system prompt, decoding setting, candidate model, rubric or case mix can change the error matrix. Pin the full judge manifest. On change, run an overlap set through old and new configurations. Keep a stratified human sample after release. Trigger recalibration when disagreement, counterfactual flip rates or candidate-family residuals move beyond their control bands.

There is a deeper frontier limit. Theory on scalable evaluation shows that when the judge is no more accurate than the evaluated model, debiasing cannot provide arbitrarily large savings in ground-truth labels; under the analysed assumptions, the ideal gain is bounded by roughly a factor of two and practical gains may be smaller.^[11] This does not eliminate model-assisted evaluation. It blocks the fantasy that a peer-level judge can replace independent evidence at the capability frontier.

Part IV

stateDiagram-v2
  [*] --> Claim
  state "Response A, compact and<br/>supportable" as Claim
  state "Response B, polished and<br/>unsupported" as Test
  state "What the score actually contains" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Build the judge into an evaluation circuit

A safe evaluation design assigns each claim to the cheapest oracle that can establish it. Deterministic checks handle syntax, arithmetic, citation resolution, required fields, forbidden transitions and policy thresholds. Retrieval or tool traces establish what evidence was available. A model judge handles bounded qualitative criteria such as clarity, relevance or completeness only after the rubric defines acceptable variation. Human adjudication anchors and audits the whole circuit.

Production-shaped worked scenario: a complaint-answer release

Consider a synthetic system that drafts answers to duplicate-payment complaints. The release claim is narrow: given an approved case summary and policy extract, the system should produce a clear draft whose material claims are supported, whose mandatory disclosure appears and whose uncertain remedy is not presented as decided. The system does not send the answer or decide redress.

The evaluation circuit begins with code. A schema validator checks required fields. A citation resolver verifies that every source identifier exists. A claim-evidence matcher can flag unsupported statements for review, but deterministic policy facts such as the maximum stated review window are checked directly against the supplied policy record. A phrase and semantic test searches for promises of reversal when no authorised decision exists. These checks define hard failures.

The LLM judge then receives only the remaining qualitative questions, each separately: Is the uncertainty explained without obscuring the next step? Does the draft acknowledge the complaint without inventing an outcome? Is the response concise enough for the channel while retaining the mandatory information? The judge returns criterion-level labels and reasons linked to exact spans. It never emits one holistic “quality 8.7” score.

A stratified human anchor covers ordinary cases, close pairs, long drafts, terse drafts, model families, vulnerable-customer language and every hard-failure mutation. Counterfactual packs swap pair order, equalise length and rewrite style. The release gate uses hard-oracle pass rates first, then calibrated judge estimates by slice, then unresolved high-consequence cases. The model judge supplies scalable observations; the release authority remains in the typed decision rule and accountable approval.

Operate the instrument as a changing dependency

The judge needs an owner, service boundary and evidence ledger. Store a hash of the prompt and rubric, the provider or weight version, decoding parameters, parser version, calibration-set version, slice results and the release decisions that consumed them. Preserve sampled inputs and outputs under appropriate privacy controls so a later disagreement can be reconstructed without logging an unrestricted copy of production content.

Define an unknown-outcome state for failed judge calls. A timeout is not a negative label, and a malformed response is not evidence that the candidate failed. Retries need idempotent case identifiers and a limit, because selective retry can change the sample. Monitor refusal, parse failure, latency and cost separately from quality. A judge that silently drops the hardest cases can improve its apparent agreement while reducing coverage.

Incidents should create permanent tests. When a human reviewer finds that the judge rewarded an unsupported promise, add the original case, a position swap, a style-neutral variant and a nearby negative control. Re-run the pack on every judge change. This converts a one-off disagreement into operating evidence about a mechanism, which is more valuable than merely lowering a dashboard score.

Executable artefact: the human-anchored judge lab

The accompanying Python programme uses synthetic pairwise cases and the standard library. Each case has a human-rubric quality difference plus position, length, style, family-affinity and shared-trap features. Stable per-case noise lets the programme change one feature at a time. It audits a low-bias positive judge and a polished-error negative judge, estimates sensitivity and specificity, applies the corrected-rate formula, bootstraps an interval and computes the effective votes of a correlated nine-judge panel.

Run it with python human_anchored_judge_lab.py. The expected output below is seeded. It does not predict the behaviour of any named model. Replace the simulator with recorded judge calls and human labels to use the same audit structure on a real evaluation.

HUMAN-ANCHORED JUDGE CALIBRATION LAB
Synthetic pairwise cases: 480 (160 calibration, 320 evaluation)
The human label is a rubric-bound anchor, not metaphysical ground truth.

JUDGE: calibrated_sensor
  agreement with human anchor : 0.921
  position-swap flip rate     : 0.015
  length-neutral flip rate    : 0.008
  style-neutral flip rate     : 0.006
  family-neutral flip rate    : 0.006
  sensitivity / specificity  : 0.933 / 0.887
  raw judge A-win rate        : 0.559
  corrected A-win rate        : 0.545
  bootstrap 95% interval     : [0.446, 0.624]
  human A-win rate (audit)    : 0.566
  release gate                : PASS

JUDGE: polished_bias
  agreement with human anchor : 0.694
  position-swap flip rate     : 0.233
  length-neutral flip rate    : 0.087
  style-neutral flip rate     : 0.138
  family-neutral flip rate    : 0.040
  sensitivity / specificity  : 0.775 / 0.620
  raw judge A-win rate        : 0.578
  corrected A-win rate        : 0.501
  bootstrap 95% interval     : [0.251, 0.720]
  human A-win rate (audit)    : 0.566
  release gate                : FAIL
    - accuracy below 0.82
    - position flip above 0.05
    - length flip above 0.06
    - style flip above 0.06
    - sensitivity below 0.80
    - specificity below 0.80
    - correction unstable or CI too wide

CORRELATED PANEL DIAGNOSTIC
  nominal judges             : 9
  effective independent votes: 1.81
  mean single-judge accuracy : 0.814
  best single-judge accuracy : 0.841
  majority-panel accuracy    : 0.856

Interpretation: replication reduces idiosyncratic noise; it does not
remove a nuisance response or a mistake shared across the panel.

#!/usr/bin/env python3
"""Human-anchored calibration lab for a synthetic LLM judge.

The programme models an evaluator as a noisy sensor of a latent human
preference plus nuisance sensitivities to position, length, style and
model-family affinity. It demonstrates:

* a positive case whose nuisance sensitivities remain below release gates;
* a negative case that looks useful in aggregate but fails counterfactual tests;
* human-anchor calibration through sensitivity and specificity;
* bias correction of a raw judge-positive rate;
* a panel diagnostic showing how correlated errors reduce effective votes.

It uses synthetic data only and the Python standard library.
"""

from __future__ import annotations

from dataclasses import dataclass, replace
import math
import random
import statistics
from typing import Iterable, Sequence

SEED = 80


@dataclass(frozen=True)
class PairCase:
    """One pairwise evaluation case.

    Positive deltas favour response A. human_a is the human-adjudicated
    preference after applying the task rubric, not an assertion of universal
    truth.
    """

    case_id: int
    quality_delta: float
    length_delta: float
    style_delta: float
    family_delta: float
    shared_trap: float
    a_first: bool
    human_a: bool


@dataclass(frozen=True)
class Judge:
    name: str
    signal: float
    position_bias: float
    length_bias: float
    style_bias: float
    self_bias: float
    shared_trap_bias: float
    noise_sd: float


@dataclass(frozen=True)
class Audit:
    name: str
    accuracy: float
    position_flip: float
    length_flip: float
    style_flip: float
    family_flip: float
    q1: float
    q0: float
    raw_rate: float
    corrected_rate: float
    corrected_ci: tuple[float, float]
    human_rate: float
    passed: bool
    failures: tuple[str, ...]


def logistic(x: float) -> float:
    return 1.0 / (1.0 + math.exp(-max(-30.0, min(30.0, x))))


def make_cases(n: int, rng: random.Random) -> list[PairCase]:
    """Create a distribution with ordinary cases and polished-error traps."""
    cases: list[PairCase] = []
    for i in range(n):
        # Candidate A has a modest average quality advantage, but many pairs
        # remain close enough that nuisance cues can reverse the verdict.
        quality = rng.gauss(0.16, 0.85)
        human_a = quality > 0.0

        length = max(-1.8, min(1.8, rng.gauss(0.18, 0.82)))
        style = max(-1.8, min(1.8, 0.45 * length + rng.gauss(0.0, 0.74)))
        family = max(-1.5, min(1.5, rng.gauss(0.0, 0.72)))

        # About one case in six is constructed so that surface polish points
        # towards the human-rejected answer. This exposes shared shortcuts.
        trap = 1.0 if rng.random() < 0.17 else 0.0
        if trap:
            if human_a:
                length = -abs(length) - 0.35
                style = -abs(style) - 0.35
            else:
                length = abs(length) + 0.35
                style = abs(style) + 0.35

        cases.append(
            PairCase(
                case_id=i,
                quality_delta=quality,
                length_delta=length,
                style_delta=style,
                family_delta=family,
                shared_trap=trap,
                a_first=(rng.random() < 0.5),
                human_a=human_a,
            )
        )
    return cases


def case_noise(case_id: int, judge_name: str, seed: int = SEED) -> float:
    """Stable per-case noise so counterfactuals change one feature only."""
    stable = sum((idx + 1) * ord(ch) for idx, ch in enumerate(judge_name))
    rng = random.Random(seed * 1_000_003 + case_id * 9_973 + stable)
    return rng.gauss(0.0, 1.0)


def judge_case(judge: Judge, case: PairCase) -> bool:
    position = 1.0 if case.a_first else -1.0
    # A trap pushes the evaluator towards the polished response. Its sign is
    # carried by style_delta, so the same mechanism can favour A or B.
    trap_direction = 1.0 if case.style_delta >= 0 else -1.0
    score = (
        judge.signal * case.quality_delta
        + judge.position_bias * position
        + judge.length_bias * case.length_delta
        + judge.style_bias * case.style_delta
        + judge.self_bias * case.family_delta
        + judge.shared_trap_bias * case.shared_trap * trap_direction
        + judge.noise_sd * case_noise(case.case_id, judge.name)
    )
    return score >= 0.0


def predictions(judge: Judge, cases: Sequence[PairCase]) -> list[bool]:
    return [judge_case(judge, case) for case in cases]


def rate(values: Iterable[bool]) -> float:
    data = list(values)
    return sum(data) / len(data) if data else float("nan")


def flip_rate(left: Sequence[bool], right: Sequence[bool]) -> float:
    return sum(a != b for a, b in zip(left, right)) / len(left)


def sensitivity_specificity(
    human: Sequence[bool], judged: Sequence[bool]
) -> tuple[float, float]:
    positives = [j for h, j in zip(human, judged) if h]
    negatives = [not j for h, j in zip(human, judged) if not h]
    return rate(positives), rate(negatives)


def corrected_prevalence(raw: float, q1: float, q0: float) -> float:
    denominator = q0 + q1 - 1.0
    if denominator <= 0.05:
        raise ValueError("judge is too close to chance for stable correction")
    return max(0.0, min(1.0, (raw + q0 - 1.0) / denominator))


def bootstrap_ci(
    calibration: Sequence[PairCase],
    evaluation: Sequence[PairCase],
    judge: Judge,
    repeats: int = 500,
) -> tuple[float, float]:
    """Percentile interval reflecting calibration and test resampling."""
    rng = random.Random(SEED + 10_000 + sum(ord(c) for c in judge.name))
    estimates: list[float] = []
    for _ in range(repeats):
        cal = [calibration[rng.randrange(len(calibration))] for _ in calibration]
        test = [evaluation[rng.randrange(len(evaluation))] for _ in evaluation]
        cal_h = [case.human_a for case in cal]
        cal_j = predictions(judge, cal)
        q1, q0 = sensitivity_specificity(cal_h, cal_j)
        raw = rate(predictions(judge, test))
        try:
            estimates.append(corrected_prevalence(raw, q1, q0))
        except ValueError:
            continue
    estimates.sort()
    if len(estimates) < repeats // 2:
        return (float("nan"), float("nan"))
    lo = estimates[int(0.025 * (len(estimates) - 1))]
    hi = estimates[int(0.975 * (len(estimates) - 1))]
    return lo, hi


def audit_judge(
    judge: Judge,
    calibration: Sequence[PairCase],
    evaluation: Sequence[PairCase],
) -> Audit:
    all_cases = list(calibration) + list(evaluation)
    base = predictions(judge, all_cases)
    human = [case.human_a for case in all_cases]

    swapped = predictions(judge, [replace(case, a_first=not case.a_first) for case in all_cases])
    no_length = predictions(judge, [replace(case, length_delta=0.0) for case in all_cases])
    no_style = predictions(judge, [replace(case, style_delta=0.0) for case in all_cases])
    no_family = predictions(judge, [replace(case, family_delta=0.0) for case in all_cases])

    cal_h = [case.human_a for case in calibration]
    cal_j = predictions(judge, calibration)
    q1, q0 = sensitivity_specificity(cal_h, cal_j)

    test_j = predictions(judge, evaluation)
    raw = rate(test_j)
    human_rate = rate(case.human_a for case in evaluation)
    try:
        corrected = corrected_prevalence(raw, q1, q0)
        ci = bootstrap_ci(calibration, evaluation, judge)
    except ValueError:
        corrected = float("nan")
        ci = (float("nan"), float("nan"))

    metrics = {
        "accuracy below 0.82": rate(h == j for h, j in zip(human, base)) < 0.82,
        "position flip above 0.05": flip_rate(base, swapped) > 0.05,
        "length flip above 0.06": flip_rate(base, no_length) > 0.06,
        "style flip above 0.06": flip_rate(base, no_style) > 0.06,
        "family flip above 0.05": flip_rate(base, no_family) > 0.05,
        "sensitivity below 0.80": q1 < 0.80,
        "specificity below 0.80": q0 < 0.80,
        "correction unstable or CI too wide": (
            math.isnan(corrected) or math.isnan(ci[0]) or (ci[1] - ci[0]) > 0.20
        ),
    }
    failures = tuple(name for name, failed in metrics.items() if failed)

    return Audit(
        name=judge.name,
        accuracy=rate(h == j for h, j in zip(human, base)),
        position_flip=flip_rate(base, swapped),
        length_flip=flip_rate(base, no_length),
        style_flip=flip_rate(base, no_style),
        family_flip=flip_rate(base, no_family),
        q1=q1,
        q0=q0,
        raw_rate=raw,
        corrected_rate=corrected,
        corrected_ci=ci,
        human_rate=human_rate,
        passed=not failures,
        failures=failures,
    )


def pearson(xs: Sequence[float], ys: Sequence[float]) -> float:
    mx, my = statistics.mean(xs), statistics.mean(ys)
    numerator = sum((x - mx) * (y - my) for x, y in zip(xs, ys))
    dx = math.sqrt(sum((x - mx) ** 2 for x in xs))
    dy = math.sqrt(sum((y - my) ** 2 for y in ys))
    return numerator / (dx * dy) if dx and dy else 0.0


def panel_diagnostic(cases: Sequence[PairCase]) -> tuple[float, float, float, float]:
    """Return mean judge accuracy, best accuracy, panel accuracy, effective votes."""
    panel: list[Judge] = []
    # Different names create independent noise, while common nuisance weights
    # create correlated errors on the same polished-error traps.
    for idx in range(9):
        panel.append(
            Judge(
                name=f"panel_{idx + 1}",
                signal=1.38 + 0.035 * (idx % 3),
                position_bias=0.13 + 0.015 * (idx % 2),
                length_bias=0.23 + 0.018 * (idx % 3),
                style_bias=0.27 + 0.02 * (idx % 4),
                self_bias=0.08,
                shared_trap_bias=0.38,
                noise_sd=0.45,
            )
        )

    human = [case.human_a for case in cases]
    votes = [predictions(judge, cases) for judge in panel]
    accuracies = [rate(v == h for v, h in zip(row, human)) for row in votes]
    majority = [sum(row[i] for row in votes) >= 5 for i in range(len(cases))]
    panel_accuracy = rate(v == h for v, h in zip(majority, human))

    errors = [[float(v != h) for v, h in zip(row, human)] for row in votes]
    correlations: list[float] = []
    for i in range(len(errors)):
        for j in range(i + 1, len(errors)):
            correlations.append(pearson(errors[i], errors[j]))
    rho = max(0.0, statistics.mean(correlations))
    n = len(panel)
    effective_votes = n / (1.0 + (n - 1.0) * rho)
    return statistics.mean(accuracies), max(accuracies), panel_accuracy, effective_votes


def fmt(value: float) -> str:
    return "n/a" if math.isnan(value) else f"{value:.3f}"


def print_audit(audit: Audit) -> None:
    print(f"\nJUDGE: {audit.name}")
    print(f"  agreement with human anchor : {audit.accuracy:.3f}")
    print(f"  position-swap flip rate     : {audit.position_flip:.3f}")
    print(f"  length-neutral flip rate    : {audit.length_flip:.3f}")
    print(f"  style-neutral flip rate     : {audit.style_flip:.3f}")
    print(f"  family-neutral flip rate    : {audit.family_flip:.3f}")
    print(f"  sensitivity / specificity  : {audit.q1:.3f} / {audit.q0:.3f}")
    print(f"  raw judge A-win rate        : {audit.raw_rate:.3f}")
    print(f"  corrected A-win rate        : {fmt(audit.corrected_rate)}")
    print(
        "  bootstrap 95% interval     : "
        f"[{fmt(audit.corrected_ci[0])}, {fmt(audit.corrected_ci[1])}]"
    )
    print(f"  human A-win rate (audit)    : {audit.human_rate:.3f}")
    print(f"  release gate                : {'PASS' if audit.passed else 'FAIL'}")
    if audit.failures:
        for failure in audit.failures:
            print(f"    - {failure}")


def main() -> None:
    rng = random.Random(SEED)
    cases = make_cases(480, rng)
    calibration = cases[:160]
    evaluation = cases[160:]

    positive = Judge(
        name="calibrated_sensor",
        signal=1.65,
        position_bias=0.025,
        length_bias=0.025,
        style_bias=0.030,
        self_bias=0.020,
        shared_trap_bias=0.020,
        noise_sd=0.30,
    )
    negative = Judge(
        name="polished_bias",
        signal=1.10,
        position_bias=0.34,
        length_bias=0.32,
        style_bias=0.45,
        self_bias=0.22,
        shared_trap_bias=0.56,
        noise_sd=0.33,
    )

    print("HUMAN-ANCHORED JUDGE CALIBRATION LAB")
    print("Synthetic pairwise cases: 480 (160 calibration, 320 evaluation)")
    print("The human label is a rubric-bound anchor, not metaphysical ground truth.")
    print_audit(audit_judge(positive, calibration, evaluation))
    print_audit(audit_judge(negative, calibration, evaluation))

    mean_acc, best_acc, panel_acc, effective = panel_diagnostic(evaluation)
    print("\nCORRELATED PANEL DIAGNOSTIC")
    print("  nominal judges             : 9")
    print(f"  effective independent votes: {effective:.2f}")
    print(f"  mean single-judge accuracy : {mean_acc:.3f}")
    print(f"  best single-judge accuracy : {best_acc:.3f}")
    print(f"  majority-panel accuracy    : {panel_acc:.3f}")
    print("\nInterpretation: replication reduces idiosyncratic noise; it does not")
    print("remove a nuisance response or a mistake shared across the panel.")


if __name__ == "__main__":
    main()

The judge instrument card

Before a judge score enters a release gate, complete one record. The card prevents a model name and a headline agreement rate from standing in for a measurement design.

Measurement claim

The exact property, unit and population the judge reading is intended to estimate.

Rubric and exclusions

Criterion definitions, acceptable plurality, prohibited inference and cases routed to indeterminate.

Judge manifest

Model and version, system and user prompts, decoding, output schema, pair order policy and date-free immutable identifier.

Human anchor

Sampling plan, reviewer competence, independent labels, adjudication rule, response-set handling and ownership.

Nuisance audit

Position swap, length control, style rewrite, author blinding, family residuals and adversarial judge-targeting cases.

Error profile

Sensitivity, specificity, agreement and uncertainty by consequence, difficulty, language, family and other material slices.

Panel independence

Error correlation, effective vote count and evidence that added judges contribute a different failure mechanism.

Decision rule

Which thresholds cause ship, narrow, collect evidence, human review or stop, with a named owner for each outcome.

Expiry and drift

Changes that invalidate calibration, recurring human sample, control bands, recalibration trigger and rollback path.

The explicit failure boundary

LLM judging ceases to help when no stable rubric exists, the anchor does not cover the deployment population, sensitivity plus specificity approaches one, counterfactual flip rates exceed the decision’s tolerance, or candidate outputs can adversarially target the judge. It should not hold final authority over irreversible, high-consequence decisions. It is also weak at the capability frontier when the evaluated system can exploit failures the peer-level judge shares.

In those conditions, use a deterministic oracle, an environment outcome, independent expert adjudication, stronger external evidence or a narrower claim. A panel is not an escape unless its errors are demonstrably diverse. A human sample is not an escape unless its reviewers, rubric and adjudication are fit for the decision. When the instrument cannot be calibrated, narrow the claim rather than widening the confidence language.

For the broader operating model around golden sets, mutation tests, CI gates and evaluation ownership, see The Eval Pipeline is the Product. For component, trajectory, human-system and production evidence, see The Evaluation Stack for Agentic Systems. This paper narrows the question to whether the judge itself deserves to function as an instrument.

Compact glossary

Construct

The property the evaluation claims to measure, such as factual support or clarity.

Nuisance response

A change in the reading caused by a feature outside the declared construct.

Human anchor

A bounded sample labelled through a specified reviewer and adjudication process.

Sensitivity

The probability that the judge accepts a case the human anchor accepts.

Specificity

The probability that the judge rejects a case the human anchor rejects.

Counterfactual audit

A test that changes one suspected nuisance while preserving rubric-relevant content.

Rating indeterminacy

A condition where several labels are defensible under the available evidence and rubric.

Effective votes

The approximate independent information in a panel after accounting for correlated errors.

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: The release that improved by getting worse
  M->>E: Response A, compact and supportable
  E-->>M: compare rival explanations
  E->>D: Release the measurement claim, not the judge score
  D-->>Q: revise scope or act

Release the measurement claim, not the judge score

A model judge can turn expensive qualitative review into a scalable evaluation signal. That is a real capability. It can also turn presentation preferences and shared blind spots into stable rankings. The difference is not the prestige of the judge model or the size of the test run. It is whether the evaluation system has established what moves the reading.

The durable intuition is simple: place the judge inside a calibration chamber. Swap position. Neutralise length. Rewrite style. Blind authorship. Compare errors across families. Anchor a stratified sample in a human process that can express disagreement. Estimate sensitivity and specificity by slice. Correct the raw rate and carry calibration uncertainty into the release decision. Test panel independence before counting votes.

The decision that changes is architectural: model judgement is evidence about a release claim, never the release authority itself. Deterministic controls establish deterministic facts. Human and environmental evidence settle cases the judge cannot. The judge earns a bounded role only where its nuisance responses, error matrix, expiry conditions and escalation path are visible.

That discipline may produce a less impressive scorecard. It produces something more valuable: a measurement result whose meaning survives the question, “What would have changed this verdict besides quality?”