Ask a team to show you their LLM system and they'll show you prompts, agents, maybe a slick trace viewer. Ask them how they know it works and the room gets quiet, or someone says "we tested it a lot before launch," which means: a few people typed things at it in April.
From regulated-system delivery, I hold a strong view: the evaluation pipeline is one of the most valuable artefacts the team will build. Prompts, models, retrieval and guardrails all change. The harness is what makes a replacement testable against the claims already approved.
Why prompts are worthless and harnesses aren't
Over a year, a model may be upgraded, a corpus may grow, guardrails may be retuned and business rules may change. A hosted provider may also change served behaviour according to its version policy. The exact cadence varies; each event can alter a release claim.
Each change asks the same question: does the evidence still support the approved use? An evaluation pipeline supplies reproducible cases, traces and thresholds. It does not reduce fitness to one number, and it does not make a weak oracle objective.
This is why I tell teams: a brilliant prompt is an asset that depreciates to zero at the next model release. The harness is the asset that appreciates: every incident, every human override, every edge case discovered in production gets folded back in as a test case, and the harness gets more discriminating every quarter. It's the only component with compounding returns. Prompt engineering without an eval harness isn't engineering; it's superstition with version control.
Golden datasets: where the real work lives
The heart of the pipeline is the golden dataset, and building a good one is genuinely hard, which is why most teams don't.
What a production-grade golden set looks like:
- Built with the people who own the ground truth. Operations specialists, policy owners and complaints teams know what “correct” means in context. Their workshops formalise business judgement; the time required depends on use-case ambiguity and review capacity.
- Deliberately hostile. A planning set may reserve roughly a third for ordinary paths and the rest for ambiguity, adversarial phrasing, vulnerable-customer scenarios, conflicting sources and requests outside scope. That ratio is illustrative. Refusal and escalation cases deserve explicit coverage because a confident answer outside mandate can be worse than a bounded failure.
- Stratified and sized honestly. Enough cases per category that a regression in one slice is statistically visible, not drowned in the aggregate. A single overall score across 100 cases tells you nearly nothing; 60 cases in a critical category tells you something.
- Versioned and owned like code. Dataset changes go through review. Products change and policies are superseded, so the set has a named owner and event-driven refresh rules. Age alone does not prove staleness; source changes, coverage drift and failed production cases do.
The feedback loop is the multiplier: every incident, reviewer override and substantiated complaint is a candidate case. Over time, the suite can become a valuable operational record, provided privacy, selection bias and label quality are controlled. This is a design expectation, not a claim about an undisclosed eighteen-month dataset.
LLM-as-judge: useful, and riskier than it looks
Human review cannot cover every output in a large release, so model-assisted judging can help triage scale. It is not unavoidable, and it must not grade criteria a deterministic check can resolve. Its documented failure modes include:
- Self-preference and style bias. Judges rate fluent, confident, well-formatted answers higher: largely independent of correctness. A regression that makes answers wronger but smoother can raise your judge scores. Position bias in pairwise comparisons and length bias are equally real.
- The judge drifts too. Your judge is a model on someone else's release schedule. When it changes, your scores shift and your historical baselines quietly stop meaning anything. Pin judge versions; treat a judge upgrade as a change requiring recalibration.
- Criteria mush. Ask a judge "is this response good, 1–10?" and you get astrology. The score means nothing and moves for reasons nobody can explain.
What makes LLM-as-judge actually work:
- Decompose. Never one holistic score. Separate binary checks: is every factual claim supported by the retrieved sources? Does the response comply with policy X? Was scope respected? Binary, criterion-specific judgments are dramatically more reliable than scalar vibes.
- Calibrate against humans, permanently. A held-back set of human-graded cases, with judge-human agreement measured continuously. Agreement below threshold means the judge is broken and its scores are void. An uncalibrated judge is not measurement; it's a random number generator with a professional tone.
- Keep humans on a sample. A fixed percentage of production outputs, human-scored on a cycle, forever. This is your ground-truth anchor and your early warning that the judge and reality have parted ways.
- Prefer deterministic checks wherever possible. Schema validity, presence of required disclosures, banned-phrase screens, citation-to-source matching: cheap, perfectly reliable, and they should carry as much of the load as they can before any judge gets involved.
Regression gates in ci: where evals grow teeth
An eval suite that runs "sometimes, before big releases" is a dashboard. An eval suite wired into CI as a blocking gate is a control. Only the second one changes engineering behaviour.
The mechanics that matter:
- Every behaviour-changing component triggers an impact check. Prompt edit, model route, retrieval configuration, guardrail or corpus update can alter behaviour. The selected test pack should follow the affected claim rather than a pseudo-precise promise that one line causes a twenty-point regression.
- Tiered execution for speed. Run a small critical set on frequent changes, the relevant full packs before merge and extended adversarial tests on a scheduled or release trigger. Set service levels from engineering flow and consequence. A slow gate creates bypass pressure, but no universal four-hour boundary defines failure.
- Category-level thresholds, not aggregates. The gate fails if any critical slice regresses (refusals, compliance, grounding) even when the overall average improves. Averages are where regressions go to hide.
- Handle nondeterminism honestly. Run flaky-adjacent cases multiple times, gate on pass rates, and set thresholds with statistical headroom: otherwise the team learns to re-run the gate until it's green, which is the same as having no gate.
- Failures are cases, not just numbers. When the gate fails, show which cases flipped, their traces and affected claims. A percentage without denominators and examples invites argument; concrete failures invite diagnosis.
In a regulated environment this gate does double duty: it's the enforcement mechanism for thresholds agreed with model risk, and every gate run is an evidence artifact. When a validator asks how you ensure changes don't degrade the system, the answer is a CI link, not a paragraph of assurances.
The reframe
There is no defensible universal percentage for evaluation effort. A third of delivery capacity can be a useful planning stress test for a high-consequence system, but the allocation should follow the cost of labels, traceability, change frequency and consequence.
The budget should also distinguish creation from operation. Building a case library, adjudication rubric and trace schema is an initial investment. Running regression packs, resolving disputed labels and sampling live outcomes is recurring work. Treating the second category as temporary project cost leaves the control unfunded after launch. Evaluation has an operating model, not only a build plan. Its capacity should appear in service ownership, release calendars and incident routines.
Evidence quality matters more than raw test volume. Ten thousand near-duplicate prompts do not replace a small set of high-consequence recovery cases with reliable oracles. Track coverage by claim, failure mechanism, cohort and consequence. Retire tests that no longer influence a decision, but retain their lineage so a later incident can explain what changed and why.
That sounds expensive until you see the alternative: systems nobody dares change, model upgrades deferred for quarters out of fear, every release a held breath. The eval pipeline is what converts an LLM system from something you made work once into something you can keep changing safely. In production, that ability is the entire game.
Prompts are the demo. The harness is the product.
Write the evaluation specification before the system
An evaluation specification is a contract between the business claim and observable behaviour. It defines what the system is allowed to do, where it must abstain and which failures stop a release. Writing it early changes the architecture. Teams discover missing evidence, ambiguous policy and unsafe tool boundaries before code makes those gaps expensive.
The specification begins with claims. “Answers policy questions accurately” is too broad. “Answers current UK retail-mortgage policy questions with a resolvable approved source, or abstains” is testable. The claim names the domain, jurisdiction, authority standard and safe alternative. A narrow claim creates a useful test; a broad aspiration creates a debate.
Every claim should connect to a failure model. What would make the answer wrong? The source may be stale. Retrieval may select a draft. The model may omit an exception. The interface may hide uncertainty. A tool may write to the wrong record. The reviewer may approve without reading. The evaluation stack should expose these mechanisms separately.
The intended population needs explicit exclusions. A system may cover personal current accounts but not small-business accounts. It may work in English but not accept translated source material. It may draft letters but not send them. Exclusions should appear in product routing and test selection. An exclusion that exists only in documentation is not a boundary.
Design a portfolio, not one benchmark
One static dataset cannot answer every release question. A portfolio contains stable regression cases, changing production samples, adversarial cases, counterfactual pairs and recovery scenarios. Each has a different purpose.
Stable cases protect known behaviour. They should be difficult enough to detect regression and small enough to inspect. Production samples reveal new demand and hidden cohorts. Adversarial cases test deliberate attempts to cross boundaries. Counterfactual pairs test whether a decisive attribute changes the outcome while irrelevant attributes do not. Recovery cases test partial failure, timeout and rollback.
| Test family | Primary use | Refresh policy | Main weakness |
|---|---|---|---|
| Stable regression | Detect known failure recurrence | Add after defects; remove rarely | Can become overfitted and predictable |
| Representative sample | Estimate current task behaviour | Refresh by demand and time | Requires careful sampling and labels |
| Counterfactual pair | Test policy and fairness sensitivity | Add with new decision rules | Pair construction can encode assumptions |
| Adversarial case | Test hostile or manipulative paths | Refresh with threat intelligence | Pass rate may not estimate normal risk |
| Recovery exercise | Test safe state and compensation | Re-run after dependency changes | Expensive to automate fully |
| Human-system study | Test review and escalation | Repeat after interface or staffing change | Small studies need cautious interpretation |
Coverage is not the number of cases. It is the relationship between risks, cohorts, system paths and evidence. A thousand near-duplicate prompts may cover less than twenty carefully chosen trajectories. Count distinct failure mechanisms before counting test rows.
The coverage map should reveal blank cells. A tool may be tested on routine cases but not on a partial write. A policy may be tested in one language but not another. A high-consequence cohort may have too few adjudicated outcomes. Blank cells become explicit decisions. The team can fill them, narrow scope or accept a documented limitation.
Govern the oracle
An oracle decides whether the observed behaviour is acceptable. Some oracles are deterministic. A source identifier either resolves or it does not. A payment amount either respects the ceiling or it does not. Other judgements require trained reviewers. Tone, completeness and policy interpretation may have legitimate disagreement.
Model judges can reduce manual effort. They should be treated as measurement instruments with known error, not as neutral authorities. Evaluate the judge against expert labels. Measure disagreement by case type. Keep a human sample. Prevent the judge from seeing information that would not be available to the system when that would bias the result.
An oracle also needs an uncertainty state. Reviewers may find that a policy is ambiguous or the evidence is incomplete. Forcing a binary label hides that defect. Record “indeterminate” with a reason. Route those cases to policy owners. A recurring indeterminate class is evidence that the operating rule, not the model, needs clarification.
Inter-rater disagreement is useful. It identifies vague rubrics, hard cases and contested business policy. Do not average it away immediately. Inspect the disagreement. Revise the rubric when reviewers interpreted it differently. Escalate policy disagreement to the accountable owner. Preserve reasonable plurality where several responses are acceptable.
Test the test with controlled defects
An evaluation suite can pass because it is weak. Mutation testing challenges the suite by introducing controlled defects. Remove a citation. Swap an approved policy for an obsolete one. Raise a tool ceiling. Reverse a routing condition. Hide an escalation reason. The relevant test should fail.
If a mutation survives, the suite does not observe that risk. Add instrumentation or a test. This is especially valuable for process claims. Final-answer scoring may not notice that the system used a prohibited source before producing a correct answer.
Mutation testing should remain bounded. Do not inject unsafe behaviour into production. Use isolated test environments and synthetic or approved data. Preserve the mutation definition, expected detector and actual result. A test earns confidence when it catches a defect designed to challenge its claim.
Connect offline evidence to production
Offline tests control cases and expected outcomes. Production reveals demand, dependency failures, reviewer behaviour and rare combinations. The two systems need a deliberate exchange.
Not every production record should enter a test set. Apply purpose, privacy, retention and access rules. Remove unnecessary personal data. Preserve the behaviour needed to reproduce the defect. Label the source and selection reason. Production-derived cases can otherwise create a hidden data lake with unclear rights.
Production confirmation should mirror release claims. If the release promises cited policy answers, monitor citation resolution, source authority and sampled correctness. If it promises human oversight, monitor queue capacity, evidence engagement and correction. If it promises bounded tool use, monitor denied calls, ceilings and compensation. Monitor the claim in the language used to approve it.
Operate evaluation as a product
The evaluation product needs users, service levels and ownership. Engineers need fast local checks. Release teams need deterministic gates. Validators need reproducible evidence. Product owners need cohort and outcome views. Incident teams need traces. Executives need a small set of risk-linked decisions.
Separate speed lanes. A small pre-merge pack should run quickly and catch common defects. A broader nightly pack can cover more models, cohorts and adversarial cases. A release pack should bind results to a candidate manifest. Periodic human studies can test matters that automation cannot establish.
Test failures need defect management. Assign severity from consequence, not inconvenience. Link the failure to the claim, case, trace and owning component. Preserve the first failing evidence. Retest the corrected build. Add a stable regression case when appropriate. Close only when the risk owner accepts the result.
Version tests and rubrics. A rising score may reflect an easier dataset, a looser rubric or a better system. Keep those causes distinguishable. When a rubric changes, run an overlap set under old and new rules. Explain the effect before comparing trends.
The evaluation backlog should be prioritised like a product backlog. High-consequence coverage gaps come first. Repeated manual effort may justify automation. Flaky tests need repair because teams learn to ignore unreliable gates. Low-value vanity metrics should be retired. The strongest evaluation stack is not the largest; it is the one that changes the right decisions consistently.
Research benchmarks remain useful reference points. HELM’s scenario-based framework makes explicit the need to assess multiple scenarios and metrics. GAIA and AgentBench illustrate evaluation across tools and environments. They cannot supply the institution’s policy, user population or consequence thresholds. Local evaluation is where those parts enter.
Build the first evaluation backlog
Begin with the five decisions most likely to block or narrow release. For each one, write the claim, consequence and safe alternative. Ask which observed behaviour would falsify the claim. This creates the first test families without pretending the team already knows every failure.
Create a small golden set with domain experts. Include routine cases only to anchor the baseline. Spend most review time on exceptions, conflicting evidence, missing data, policy boundaries and cases that should abstain. Record why each case matters. A row with no named risk is a candidate for removal.
Build deterministic oracles first. Check schemas, citations, permissions, tool arguments, ceilings and prohibited transitions with code. Reserve expert judgement for matters that require it. Define the rubric with examples of acceptable variation. Record an indeterminate state rather than forcing agreement.
Add trace capture before scaling the dataset. A failing answer without retrieval, routing and tool events is expensive to diagnose. A correct answer reached through a prohibited path is dangerous to miss. Keep trace access controlled and minimise sensitive content.
Turn the highest-consequence failure into a hard gate. Add cohort thresholds for the remaining material claims. Run the gate on a fixed candidate manifest. Make the output a decision record that names failed cases and owners, not a coloured scorecard.
Connect production sampling after the first controlled release. Review cases selected by consequence, uncertainty, route change and random sampling. Separate product demand shifts from system regressions. Add confirmed failures to the stable set after privacy and duplication review.
Retire tests deliberately. Remove duplicates, obsolete policies and cases with broken provenance. Preserve the reason and history. A test suite can become stale in the same way as a retrieval corpus.
Finally, review the backlog with product, engineering, security, operations and validation. Ask which release claim has the weakest evidence. Fund that gap before adding another broad metric. An evaluation roadmap should reduce uncertainty about decisions, not maximise the number of tests executed.
A pipeline that answers release questions
An evaluation suite becomes useful when each result changes a decision. Tests should map to an action: ship, narrow, investigate, roll back or collect more evidence. A large scorecard with no decision owner is reporting overhead.
| Evaluation layer | Question answered | Useful artefact | Owner |
|---|---|---|---|
| Component | Does retrieval, routing or a tool work alone? | Unit results and trace samples | Engineering |
| Task | Does the whole system complete a bounded job? | Golden-set score and error taxonomy | Product and evaluation |
| Safety | Can misuse or hostile input cross a boundary? | Adversarial cases and control traces | Security |
| Human system | Can a reviewer detect and correct errors? | Review study and escalation data | Operations |
| Production | Has behaviour changed after release? | Sentinel trends and incident links | Service owner |
Treat change as an evaluation event
The model is only one source of change. Prompts, retrieval data, chunking, tools, policies and user mix also shift behaviour. The pipeline should therefore listen to the whole system manifest.
| Change | Minimum response | Common mistake |
|---|---|---|
| Model or decoding | Full task, safety and calibration regression | Comparing only headline quality |
| Prompt or policy | Affected workflow and refusal tests | Calling text changes “non-code” |
| Corpus or chunker | Retrieval recall, provenance and stale-data tests | Testing generation without retrieval |
| Tool or permission | Tool contract, abuse and blast-radius tests | Reusing a model-only benchmark |
| User or case mix | Cohort drift and human-outcome review | Assuming the launch sample still represents demand |
Evaluation debt compounds like software debt. A missing oracle becomes a manual argument at every release. A missing trace becomes guesswork during an incident. A missing cohort label hides who pays for an error.
The NIST AI RMF frames measurement as one part of a governed lifecycle. The NIST Generative AI Profile offers a cross-sector risk profile. The NIST AI Resource Center collects testing, evaluation, verification and validation material. For security cases, the OWASP Top 10 for Agentic Applications gives a current threat taxonomy. The Federal Reserve’s 2026 revised model-risk guidance provides useful risk-based validation disciplines for covered models, but it explicitly excludes generative and agentic AI from its formal scope. Those disciplines are therefore an internal-governance analogue, not a claim of regulatory coverage. The claim, test, threshold, owner and release must remain connected.