The strong rivalDiscovery should outrun specification

Generative systems exhibit open-ended behaviour, and customers invent uses the team did not predict. A detailed pre-release contract can freeze learning, encourage tuning to the suite and create false completeness. Discovery, rapid experiments and production observation may reveal more than a static specification. This rival is right about unknown unknowns.

The answer is not to omit the contract. It is to include known unknowns, an exploration budget, renewable challenge sets, shadow operation, production sampling and conditions for changing the specification itself. A release boundary protects customers while the team learns. When a new failure family appears, it becomes a clause, test and indicator rather than proof that all prior evaluation was futile.

The contract fails when thresholds are written after results, public benchmarks are contaminated, easy cases dominate averages, human rubrics are vague or commercial pressure waives gates without an owner. It also fails if teams optimise the product for a golden set and stop looking. The specification must discriminate between acceptable and unacceptable products, then change when reality supplies better evidence.

Evaluation Product ContractEight clauses make the promise executable

The clauses are linked. A required citation behaviour needs evidence cases with current and contradictory sources, a measure of proposition support, a threshold and a release consequence. A prohibited customer denial needs a test across rare consequence classes and a hard block that cannot be averaged away. A new tool permission may leave the model unchanged while invalidating authority, recovery and privacy tests.

Write the evaluation before the release claim. If the team first sees an average and then chooses a threshold, the decision is vulnerable to motivated adjustment. A range may be appropriate where sampling uncertainty or human judgement varies. A hard gate is appropriate where one observed failure proves the product crossed a prohibited boundary. State which evidence would reverse the decision.

The contract is not complete. It records known blind spots and how discovery continues. Exploration can examine open-ended behaviour below protected action authority. Production sampling can find new failure families. The specification owner decides whether the finding changes the product promise, operating envelope, test portfolio or threshold and records why.

Four evidence levelsPreserve what each test can establish

Component capability tests the model or service under controlled conditions. It can compare scenario performance, calibration, robustness, latency and cost. HELM widened model evaluation across scenarios and metrics while explicitly documenting gaps. This is valuable input evidence. It does not include the product's retrieval, permissions, interface, users or operating process unless the scenario contains them.

End-to-end product behaviour tests the deployed configuration. It checks whether retrieval provides current evidence, prompts preserve the policy boundary, tools enforce authority, the interface reveals uncertainty, handoff reaches the right person and recovery repairs downstream state. A component can remain identical while a prompt, connector or interface creates a regression.

User and team performance tests the human-AI configuration. Measure task outcome, error detection, appropriate reliance, time, workload, disagreement, accessibility and handoff quality with the intended roles. Offline product success can survive while people verify every output, create hidden rework or defer to a fluent error. Human ratings are measurements with a task, population, protocol and uncertainty.

Realised production outcomes test whether the product creates the promised value and acceptable impact under live demand. Link adoption, handling time, rework, customer effect, incidents, distributional results, cost and latency to eligible cases. A faster draft is not captured value if downstream review grows. A favourable average is not acceptance if a protected subgroup or rare severe class fails.

The release matrix keeps levels separate. A team may conditionally release a bounded proposal-only feature after component and product tests, then use shadow operation to collect team evidence. It should not call production outcomes proven before they exist. Conversely, live success cannot excuse an observed prohibited behaviour whose affected population lacks a safe appeal.

Evaluation trace ledgerGive every claim a discriminating test

The ledger starts with the exact product claim and intended audience. “Answers from current policy” implies more than link availability. A discriminating test includes current and superseded versions, contradictory sources, missing provenance and retrieval failures. It passes only when behaviour differs correctly between acceptable and unacceptable cases.

Population and provenance make the result interpretable. Record languages, geographies, task difficulties, data-quality conditions, user roles and versions. Document where test cases came from, who labelled them, which public data may have contaminated training and how the set is maintained. A small challenge set can reveal a prohibited failure without estimating a population rate; do not pretend it does both.

Thresholds come before results. Use acceptable ranges where sampling and judgement vary, distributional slices where aggregate success can hide harm and hard gates where any observed instance crosses a product boundary. The owner can revise the specification when new evidence warrants it, but the ledger preserves the earlier rule and reason. Quietly moving the threshold destroys the release trace.

Blind spots are first-class fields. A test suite can never prove absence of all failures. State unrepresented cohorts, inaccessible supplier behaviour, delayed outcomes and untested interactions. Pair each with exploration, monitoring, manual review or a narrower operating envelope. Product copy and support should not imply coverage the ledger lacks.

Live indicators extend the test. A source-version mismatch, rising override, latency tail, appeal pattern, cost spike or subgroup outcome can trigger re-evaluation. The signal should map to a claim and decision, not merely a generic dashboard. If a production indicator breaches a hard gate, rollback should not wait for the next evaluation report.

Case portfolioMake difficult behaviour visible before it becomes an incident

Representative cases estimate performance in the intended envelope. Boundary cases sit near exclusions, permission limits or ambiguous task definitions. Adversarial cases probe manipulation and prohibited behaviour. Counterfactuals change one decision-relevant attribute while preserving the rest. Abstention cases verify that the product refuses safely and preserves evidence. Recovery cases test interruption, correction, rollback, appeal and downstream remedy.

Coverage is a reasoned portfolio, not a promise of exhaustiveness. Use model cards and dataset documentation to expose intended use, factors, composition and maintenance. CheckList demonstrates the value of capability-oriented behavioural tests beyond held-out accuracy. Dynamic benchmark approaches show how humans and models can surface new failures. Adopt the principle without assuming any one suite fits the product.

Prevent easy cases from dominating. Report ranges and slices by consequence, cohort, difficulty, language, tool path and data quality. Rare severe cases may not support stable rate estimates, yet one observed unsupported denial can still fail a prohibited gate. The release decision preserves that fact instead of averaging it into thousands of successful summaries.

Challenge sets expire. Teams can overfit prompts, retrieval and workflow to familiar examples even without changing the model. Renew cases from production disputes, new policies, red-team work and independent review. Protect a holdout where possible and record when a test influenced development. Contamination risk changes how much the result can establish.

Change-trigger webRegression includes everything around the model

Version the model, prompt, retrieval index and policy, tools and permissions, interface, workflow and environment. A prompt can change abstention. Retrieval can serve stale evidence. A connector can expand action authority. An interface can hide uncertainty. A workflow can remove the reviewer who caught errors. Demand can move the product outside its tested population.

Each change maps to contract clauses and claims. A copy change that only fixes spelling needs little evaluation. A new confidence label needs reliance and accessibility tests. A supplier model update may require capability, product and human-team regression. A new customer geography can require population and legal review even when code is unchanged.

Automate repeatable suites, but keep adjudication and exploration. Automated tests are valuable for deterministic rules, known failure patterns and fast regressions. Human review remains necessary where context, consequence or rubric judgement matters. Production sampling searches for cases the suite does not know. The revalidation record explains why the chosen depth fits the change.

Human and automated evaluatorsA judge is another measured component

Human evaluation starts with the task, audience and acceptable evidence. “Helpful” is ambiguous if assessors do not know whether the output supports a novice, specialist or affected customer. Write observable rubric anchors, train assessors on boundary examples, record disagreement and adjudication, and report uncertainty. Assessors need relevant expertise and enough independence to identify a product failure rather than reward conformity to expected style.

Human ratings are not unquestionable truth. Fatigue, incentives, cultural assumptions, interface order and access to evidence can shape judgement. Compare assessors where disagreement matters, investigate systematic differences and preserve affected-party input without assuming every preference should become a release threshold. A protocol makes judgement inspectable, not infallible.

An LLM judge can scale triage, pairwise comparison or rubric application. Research on MT-Bench and Chatbot Arena documented useful scalability alongside position, verbosity and self-enhancement biases. Validate the judge against the relevant human protocol and failure population. Include concise correct answers, fluent contradictions, policy exceptions and adversarial styles. Monitor drift when the judge or candidate model changes.

Never make an unvalidated automated judge the sole authority for a consequential gate. It may agree with style while missing policy contradiction. Use deterministic checks where behaviour is rule-like, expert adjudication where context is material and a validated judge where its error is understood. The evaluation trace identifies which authority decided each result.

Worked release matrixKeep hard gates, ranges and trade-offs visible

A bounded decision-support product passes its range-based task criteria across required cohorts, meets shadow handling-time and rework limits, stays inside latency and cost constraints and supports accessible completion. One challenge case still produces an unsupported customer denial. The prohibited behaviour has a zero-observed hard gate. The correct route is block or redesign, not an average score.

“Zero observed” does not estimate a zero population rate. It states a release rule for this test portfolio. The evidence record keeps sample design, uncertainty and blind spots visible. Production controls must still prevent the action, sample outcomes and provide appeal. If the team cannot test every route through the protected action, the operating envelope remains narrower.

Trade-offs belong only where compensation is legitimate. The named owner may accept a small latency increase for better source checking or choose shadow operation while business outcomes are measured. They cannot trade a prohibited privacy exposure for higher task accuracy without changing the product, authority and applicable decision process. The matrix labels who owns each trade-off and which rows are non-compensable.

Release readiness includes rollback. Define the live indicator, detection window, safe degraded mode, decision owner and evidence preserved. A high-severity unsupported action, source-version mismatch or material subgroup failure can trigger immediate containment. Lesser drift can trigger targeted re-evaluation or a shadow cohort. Commercial pressure should not create an unnamed waiver path around the matrix.

Interactive instrumentEvaluation Contract Compiler

The Compiler joins the eight clauses, four evidence levels, hard gates and change state into one release route. It never generates a maturity or quality score and never lets a strong average compensate for a failed prohibited-behaviour gate.

Five release routesMake evidence change the product decision

Release within the evidenced operating envelope when the bounded promise, required behaviours, hard gates, cohorts, human protocol, constraints, shadow results, live indicators and rollback all support operation. Publish the versioned envelope and exclusions. Continue sampling and change-triggered regression; release is a maintained decision.

Run in shadow and collect outcome evidence when pre-release component and product evidence passes but the team or production outcome is not yet established. Prevent external effect, compare workflow performance and define the outcome window. Shadow is not indefinite production by another name; it has an owner and decision date.

Narrow the product promise or redesign the workflow when the model is capable but retrieval, interface, permissions, user burden, cost, latency, accessibility or a population gap makes the current product unacceptable. Lower authority, exclude a cohort or redesign the verification and handoff path, then evaluate the changed product.

Build a discriminating evaluation before deciding when claims, populations, rubrics, case provenance, evaluator validity, thresholds or version trace are missing. A generic benchmark or style-sensitive judge cannot settle the release. Write the rule before collecting the deciding result and include a case that separates acceptable from unacceptable behaviour.

Block or roll back because a hard gate failed when prohibited behaviour, privacy, security, severe subgroup harm or an explicit rollback signal breaches the contract. Strong averages, speed and cost cannot compensate. Contain the affected action, preserve the trace and return only after the product or boundary changes and the gate passes.

Unknown unknowns do not justify release without gates. They justify protected exploration, production sampling, feedback and challenge-set renewal. When a new family appears, add the clause, provenance, case, threshold, live indicator and owner. Decide whether existing users or outcomes were affected and whether rollback or remedy is required.

Evaluation is complete only when it can change a decision. A claim without a discriminating test remains an aspiration. A test without a threshold is a measurement. A threshold without an owner is a suggestion. A hard gate without an enforced consequence is theatre. The product specification connects the whole chain and keeps it current as reality changes.

Operate the evaluation contract

At each release, archive the exact model, prompt, retrieval, tool, interface and workflow versions with test data provenance, assessor protocol, automated-evaluator version, results, uncertainty, blind spots and decision. This is not a dump of every experiment. It is the evidence receipt for the product people will actually use.

After launch, join live indicators to the claim ledger. Review handling time, rework, appeals, source mismatches, override, cost, latency, incidents and realised outcomes by relevant cohort. Protect privacy and avoid covert worker surveillance. A rising verification burden may reverse an offline success even when users still complete the task.

Retirement is a valid evaluation outcome. If the product can no longer meet a hard gate, its evidence population disappears or the cost of verification exceeds captured value, withdraw the promise or reduce it to a safer function. Past benchmark leadership does not keep a present product fit for service.

Operational specificationDesign evidence that can survive a release meeting

Turn the product promise into observable cases

Start with a claim a user or buyer can recognise. “The assistant prepares a policy-grounded response for an authorised service agent” identifies an output, source condition and user. Add the affected customer, the action the agent retains and what the product must never do. “High quality” does not reveal whether success means a complete draft, supported claims, correct policy, usable handoff or a customer outcome.

Decompose the claim into behaviour before choosing metrics. The assistant retrieves the current policy, distinguishes mandatory from optional terms, attaches provenance to material claims, exposes missing customer information, abstains when the policy is unresolved and never sends externally. A test can now introduce an old policy, contradictory clause, missing field, restricted customer, tool failure and send permission. The product must respond differently across those conditions.

Test the actual state path. A golden answer copied into a scoring tool omits retrieval, authentication, permissions, interface state and downstream writes. Run cases from user request through data access, model call, source display, human action and any external effect. Observe retries, caches and partial failures. A system that produces the right text after using unauthorised data has not passed the product contract.

Use counterfactuals carefully. Change a decision-relevant field while holding other conditions stable, then check whether output and route change as specified. Swap a protected attribute only when the causal and policy interpretation is legitimate. A superficial pair can conceal real dependencies or imply a fairness conclusion it cannot establish. Document why the transformation is valid and involve domain and affected-party expertise.

Specify measures without building a compensating score

Task measures can include completion, proposition support, field accuracy, correct abstention and recovery success. Interaction measures can include error discovery, override, time, workload and accessibility. Service measures include latency distributions, availability and queue age. Business measures include eligible outcome, rework and captured value. Risk measures track prohibited actions, privacy, security, incidents and distributional effects. Each answers a different product question.

Do not add the measures into a quality score. A weighted aggregate lets a team hide a severe failure by improving cheap dimensions. Preserve hard gates first, then show acceptable ranges, confidence or judgement uncertainty, cohorts and explicit trade-offs. A release owner can decide that latency within one band is acceptable for better evidence handling. They cannot decide that more fluent answers compensate for unauthorised customer messages.

Sampling design should match the inference. A representative set supports an estimate for a defined population when selection and outcomes permit it. A curated challenge set supports finding whether known dangerous behaviour remains possible. A red-team case explores an attack path. A small specialist review can resolve a policy boundary. Report what each can establish rather than placing every result beside a percentage.

Sample size is not a universal number. Base it on decision consequence, expected variation, cohort needs, desired precision and feasible outcome collection. Rare severe events often require targeted cases and enforced controls because an affordable sample cannot estimate the tail well. Human ratings need enough repeated and overlapping judgements to understand assessor variability where it affects the release decision.

Make shadow operation an evidence phase

Shadow operation runs the product in the real workflow without granting the consequential effect. It can compare proposed actions with actual decisions, measure latency and handling, and reveal context or permission failures. Define which users know the shadow is present, how data are protected, what outcome becomes available and when the phase ends. A silent experiment on workers or customers can create its own ethical and legal problems.

A shadow comparison must account for selection. If only easy cases enter the shadow lane, its success cannot support the full queue. If reviewers spend extra time because they know an experiment is observed, workload may not transport to routine operation. Record eligibility, exclusions, missing outcomes and workflow changes. Use qualitative evidence to explain why metrics moved, then test the explanation.

Production reversal occurs when offline and shadow task success persists but the operating outcome worsens. Service agents may verify every source, copy data between tools or correct output before the measure sees it. Handling time and rework rise while model accuracy looks stable. The trace ledger connects the product claim to these live indicators so the decision becomes narrow or redesign, not a request for another benchmark.

Business outcome evidence needs an eligible denominator and a counterfactual appropriate to the decision. Adoption among volunteers does not prove value for all eligible cases. A before-after comparison can confuse seasonal demand or policy change with the product. Use experiments or credible observational designs where feasible, and state the limitation where not. Product evaluation should inform a business claim without manufacturing causal certainty.

Keep gates enforceable under pressure

A hard gate needs machine and organisational enforcement. The protected action can be disabled when the gate fails, and the release process records who can change that state. A waiver requires the named risk owner, bounded exposure, evidence, expiry and a new decision. If sales or launch leadership can routinely bypass the rule through an informal message, the gate is descriptive theatre.

Finance belongs in the contract because evaluation itself and product operation consume resources. Measure inference, retrieval, review, adjudication, incident and remedy cost within the operating envelope. A cheap component may create expensive verification. A slower route can capture more value if it prevents rework, but the trade-off must stay below hard service and consequence limits.

Accessibility is a required behaviour, not a final audit. Test representative tasks with keyboard, screen reader, zoom, reduced motion and relevant cognitive or language needs. Confirm that evidence, uncertainty, errors, actions and recovery remain operable. A product that achieves task accuracy but excludes an intended user cohort has failed its envelope or needs a narrower promise.

Privacy and security cases test data and authority paths. Introduce purpose boundaries, cross-tenant context, prompt injection, unauthorised tools, expired identity, sensitive output and logging failure. Some checks can be automated; others need adversarial and architecture review. Passing a model safety benchmark cannot substitute for the deployed access and data controls.

Renew the specification when reality teaches

An unknown unknown enters through a dispute, support ticket, incident, production sample or unexpected user strategy. First contain any consequential effect and preserve the versioned trace. Determine whether the case was inside the promised envelope, an unrecognised boundary or a new use. Add an evaluation clause and challenge family that would have discriminated the failure, then decide whether to narrow, redesign or rollback.

Do not merely add the exact incident to a golden set. Generalise the failure mechanism and create varied cases that resist memorisation. If a stale jurisdiction caused advice, vary source dates, jurisdiction fields, conflicts and retrieval outages. If an interface hid an abstention, vary screen size, role, time pressure and action authority. Keep an independent holdout or fresh-case process to detect test-suite gaming.

Retire obsolete tests with a recorded reason. A product boundary may disappear, a policy may change or a deterministic control may make an old model test irrelevant. Retention for history differs from running every case on every change. Targeted regression keeps the suite usable while the trace ledger shows which claims remain covered and which require renewed evidence.

The specification itself is versioned. A new customer group, product promise or authority is not a silent threshold adjustment. It creates a new envelope, affected-party analysis, evidence design and release decision. Teams can learn quickly because they know what may change, who owns it and which protected behaviours remain non-negotiable while discovery continues.