TLDR
- A compound AI system needs evidence at model, retrieval, tool, trajectory, decision and operational-outcome layers. Each layer answers a different question and can pass while another fails.
- Final task success is necessary for many claims and insufficient for assurance. A lucky trajectory can reach the right state through stale evidence, an unauthorised action or an effect that cannot be reproduced.
- Component tests localise defects; trajectory tests expose propagation; decision tests connect evidence to consequence; outcome tests reveal whether the workflow improved the real process.
- Evaluation should be organised by claims and failure modes, not by available dashboards. Every metric needs a denominator, threshold, owner, expiry and response that can change release authority.
- Ship an evaluation manifest with every version, then use shadow, canary, failure injection, readback and recovery evidence to keep the release claim alive.
The correct address reached by the wrong route
An agent receives a request to update a customer’s postal address. At the end of the test, the core system contains the correct address. The benchmark records success.
The trajectory tells another story. Retrieval supplied an old policy page. The model skipped a required identity check. A tool call timed out, so the agent retried without an idempotency key. The first call had already succeeded. The second created a duplicate case and triggered a downstream letter. A reviewer later removed the duplicate, leaving the final address correct.
Outcome-only evaluation awards a point. Component-only evaluation might award several more: the language model extracted the address, the API schema was valid, and the final read returned the requested value. Yet the system was not safe to release for unsupervised updates. Success concealed an authority violation and an ambiguous effect.
This is why compound systems need an evaluation stack. Evidence must follow the route from model behaviour to retrieved context, tool interaction, trajectory, decision and verified business outcome. The layers should connect, not collapse. A final failure needs local evidence for diagnosis. A final success needs intermediate evidence to rule out luck, policy breach and hidden harm.
The stack changes release practice. Teams stop asking one benchmark to certify the system. They define claims, map failure modes to observable layers, inject faults, record thresholds and connect failure to hold, rollback, escalation or redesign. Evaluation becomes an operating control rather than a score report.
Six layers, six questions
The evaluation stack is nested because higher-level behaviour contains lower-level behaviour while adding interactions and consequence.
Figure 1. The evaluation stack nests evidence without collapsing it
Model layer: can the model perform the bounded operation?
Model tests isolate generation, classification, extraction, ranking or reasoning under controlled context. They measure properties such as accuracy, calibration, refusal, structured output, robustness and latency. They help compare model versions and prompt variants.
The model layer should not receive credit for capabilities supplied by retrieval, tools or workflow. If a calculator produces the number, evaluate whether the model chose and used the calculator correctly; do not describe the language model as arithmetically accurate. If external documents contain the answer, separate retrieval coverage from answer synthesis.
Model benchmarks remain valuable. They are fast, repeatable and diagnostic. Their claim ends at the controlled boundary. A strong score does not show that the application retrieves permitted evidence, calls the right tool or produces a useful outcome.
Retrieval and context layer: did the system acquire decision-relevant evidence?
Retrieval evaluation asks whether relevant, authoritative, permitted and fresh evidence entered context. Recall at , precision, ranking metrics and answer support are useful, but the object is more than a document list.
A context compiler may filter by identity, purpose, effective date and evidence status. Evaluation should test those controls. It should detect stale policy, superseded documents, contradictory sources, missing metadata, malicious instructions and context overflow. The retrieved set must be evaluated against the decision, not only topical similarity.
Useful records include query version, index version, filters, candidate set, selected evidence, publication time and exclusion reasons. Without them, a wrong answer cannot be separated from a missing document or a synthesis error.
Tool layer: did each interaction satisfy its contract?
Tool evaluation covers selection, arguments, identity, permission, timeout, idempotency, error interpretation and readback. A syntactically valid function call can be semantically wrong. The tool may operate on the wrong account, use stale state or return an accepted command whose effect remains unknown.
Test positive, negative and ambiguous cases. The agent should handle validation errors, partial results, rate limits, timeouts, duplicate requests, stale versions and malicious tool output. Tool mocks are useful for deterministic coverage; staging and sandbox environments reveal integration behaviour mocks cannot reproduce.
Trajectory layer: did the sequence stay inside an acceptable envelope?
Trajectory evaluation inspects plans, observations, state transitions, retries, confirmations, policy decisions and recovery. It asks whether a good outcome was reached through permitted states and whether an early deviation predicted later failure.
The unit is not raw chain-of-thought. It is an event trace of observable decisions and effects: model proposal, evidence reference, tool request, policy result, state update and verification. Private reasoning need not be exposed for the trajectory to be inspectable.
Decision layer: was available evidence converted into the right bounded action?
Decision evaluation connects prediction to consequence. It checks thresholds, abstention, information seeking, human review, authority and alternatives. A model can be accurate while the decision rule is poor. A weaker model with better calibration and deferral can create better outcomes.
The decision layer needs explicit actors: who proposed, vetoed, approved and executed. It also needs a counterfactual baseline: what would the ordinary policy, human process or simpler system have done?
Operational layer: did the process improve and remain recoverable?
Operational evidence includes verified business outcome, service quality, harm, workload, appeals, cost, drift, incidents and recovery. Outcomes may arrive later than technical metrics. Some are partly observed or influenced by the release itself.
The layer should not collapse to return on investment. It retains safety, fairness, privacy, reliability and contestability as separate constraints. A faster process that creates unreviewed harm is not an unqualified success.
Connect the layers with claims
An evaluation programme should begin with claims, not a catalogue of metrics. For each claim, name the mechanism, rival, boundary and consequence. Then identify which layer can observe each part.
Suppose the claim is “the complaint-triage agent reduces time to specialist review without increasing missed high-harm cases.” Model classification is one mechanism. Retrieval of current routing policy is another. Tool and trajectory controls determine whether routing occurs correctly. Decision evaluation measures prioritisation and abstention. Operational evaluation measures time and missed harm.
No single layer supports the full sentence. Together they can form an evidence graph.
Figure 2. A failed outcome is the start of localisation
The fault tree starts from a bad outcome and branches. Was the evidence missing? Was the proposal wrong despite good evidence? Did a tool create the wrong effect? Did policy approve an impermissible action? Did human review fail? Local tests distinguish these explanations.
Claims need typed support edges
An evidence item may support, limit, contradict or expire a claim. Store that relation. A model score supports component behaviour under a test distribution. A canary outcome supports a bounded operating claim. A drift alert limits transfer. An incident contradicts a safety subclaim until resolved.
This prevents evidence accumulation from becoming a folder of reports. The system can answer which claims still have current support and which release decisions depend on them.
Thought experiment: preserve the outcome, change the path
Create two trajectories that end with the same correct address. Path A verifies identity, checks policy, performs one idempotent update and reads back the record. Path B skips identity, retries after timeout and relies on a later reviewer to repair duplication. An outcome benchmark treats them equally.
Now preserve Path B and change only the final state to wrong. The benchmark detects failure but still cannot locate the cause. Trajectory evidence distinguishes both experiments. It reveals that final success and process assurance are orthogonal dimensions.
Figure 3. Final success can hide an unsafe trajectory
The lower path is a negative control for outcome-only evaluation: it succeeds visibly while withholding the proposed mechanism of safe execution. A release test needs such lucky-success cases, not only obvious failures.
flowchart LR A["The correct address reached by<br/>the wrong route"] --> B["Six layers, six questions"] B --> C["Connect the layers with claims"] C --> D["Trajectory is more than a<br/>transcript"] D -. "re-test" .-> A classDef first fill:#0a1424,stroke:#007D68,stroke-width:2px,color:#e6f3ff classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff class A first class B,C,D step
Trajectory is more than a transcript
A transcript records messages. A trajectory record represents stateful events with semantics. It distinguishes an observation from an instruction, a proposal from an authorised decision, and an accepted command from a verified effect.
A useful event schema includes:
- event identifier and parent;
- actor, subject and purpose;
- effective time and observed world-state version;
- input evidence identifiers;
- proposal or action contract;
- policy decision and reason;
- tool request, response and idempotency key;
- effect receipt or unknown-outcome state;
- postcondition readback; and
- cost, latency and error classification.
The schema supports deterministic checks. Every consequential action must have fresh authority. Every accepted command must resolve to success, failure or unknown. Every unknown must block incompatible follow-up work. Every verified effect must match intent.
Trajectory metrics require semantics
Path length alone is not quality. A short path can skip a control. A long path can gather necessary evidence. Useful measures include unnecessary actions, repeated failures, invalid transitions, unsupported claims, unverified effects, recovery time and distance from an allowed-state envelope.
Define state-machine properties. For example:
proposed -> authorised -> executed -> verified is
permitted. proposed -> executed is not. A timeout moves
executing -> unknown, not back to proposed.
Property tests can explore transitions more reliably than a judge
reading prose.
Process reward and judge limits
Step-level judges can provide dense feedback. They can also reward familiar reasoning style, miss hidden tool state and share model errors. Calibrate them against human or deterministic labels on a sampled set. Measure false approval and false rejection by step type. Use judges to prioritise inspection, not to invent ground truth.
When hidden reasoning is unavailable, evaluate observable choices. Did the agent retrieve the required source? Did it ask for missing information? Did it call the permitted tool? Did it stop after an unknown outcome? These properties often carry more operational meaning than a narrative trace.
Decision and outcome evidence
A decision layer converts uncertain evidence into action. Evaluation must include the policy that performs this conversion.
Let be an action, the available state and the consequence-weighted utility. A policy selects among act, ask, defer, refuse, simulate and escalate. The model score is one input. A good decision may decline the model’s top proposal because authority is missing or harm is asymmetric.
Abstention is a decision, not an error bucket
Measure risk against coverage. Inspect who is deferred, how long review takes and whether reviewers have the evidence needed. A system can improve automatic accuracy by sending every difficult case to an overloaded queue. The model metric rises while operational service degrades.
Human review needs its own baseline. Compare unaided, model-first, evidence-first and independent-first interfaces. Measure anchoring, override quality and decision time. “Human approved” is an event, not proof that the joint decision was correct.
Outcomes arrive on different clocks
Figure 6. Operational outcomes arrive later than model scores
Model quality can be measured immediately. Control violations appear during a trajectory. Customer harm, financial loss, appeals or long-term behaviour may emerge later. A release should use leading indicators without pretending they are final outcomes.
For the address update, immediate indicators include identity-check completion, duplicate commands and readback. Later outcomes include returned mail, complaints and correction requests. Both clocks matter. An early control can prevent harm before enough outcome cases accumulate.
Counterfactual baseline
Outcome improvement is comparative. Compare with the existing workflow, a deterministic policy, human-only process or simpler model. Adjust for changing case mix and policy where possible. A before-and-after chart without a credible counterfactual can attribute seasonal or organisational change to the agent.
Randomised or stepped rollouts provide strong evidence when permissible. Matched controls, interrupted time series and causal models may help under assumptions. Preserve those assumptions and sensitivity analyses. Operational evaluation inherits the difficulty of real-world causal inference.
Do not average away a control failure
Teams often want one release score. Weighted averages are convenient and dangerous when some properties are mandatory. A model-quality gain can numerically compensate for an authority violation even though the organisation would never make that trade.
Separate constraints from objectives. Constraints define unacceptable states: unauthorised action, privacy breach, unresolved duplicate effect and missing mandatory evidence. Objectives rank acceptable options: accuracy, latency, cost and user effort. A release must satisfy constraints before optimisation scores matter.
Within an objective, aggregation still needs a declared denominator and weighting. Case averages give frequent events more influence. Consequence-weighted averages give costly cases more influence. Worst-slice measures protect known groups but can be noisy. Report a compact vector and the decision rule rather than hiding trade-offs in a decimal.
Assurance cases provide another structure. A top claim is decomposed into subclaims, supported by evidence and challenged by defeaters. A test result does not simply add points; it supports a particular edge under conditions. A drift event, incident or expired policy becomes a defeater that can withdraw the claim.
The evaluation manifest can generate part of this case. It should not declare itself sufficient. Reviewers need to see assumptions, missing evidence and unresolved counterarguments. The goal is a living argument whose authority changes with the system.
Unknown outcomes deserve their own metric
Compound systems interact with effects that may not return a clean success or failure. A network timeout after an accepted command produces uncertainty. Counting the request as failure can trigger a dangerous retry. Counting it as success can conceal a missed action.
Track unknown-outcome rate, time to resolution, duplicate prevention, reconciliation completeness and residual harm. Inject ambiguity deliberately. The executor should query an effect receipt, idempotency record or live postcondition before proceeding. If resolution is impossible, the case moves to manual reconciliation.
Unknown is not an embarrassing gap to impute away. It is an operational state with constrained transitions. Evaluation should penalise workflows that convert ambiguity into confident narrative.
Failure injection localises the stack
Ordinary test cases sample expected work. Failure injection asks how faults propagate. Inject one fault at a time, then combinations that reflect real coupling.
At the model layer, produce malformed structured output or calibrated uncertainty. At retrieval, remove the decisive source, insert a superseded policy or add hostile content. At the tool layer, return timeout, partial success, stale version and duplicate acceptance. At trajectory, reorder events or lose checkpoint state. At decision, corrupt confirmation scope. At operation, delay outcome labels or overload review.
Figure 4. Evidence coverage is a matrix, not one score
The coverage matrix maps failure modes to observable layers. Empty cells show where a fault can pass without detection. A stale document may be detected during retrieval, contradicted by a policy gate, noticed in review or observed only after a wrong outcome. Strong designs provide early detection and later backstops.
Fault propagation has a transfer function
For injected fault , measure:
- probability the layer detects it;
- probability it reaches each downstream layer;
- consequence if it reaches a decision;
- time to containment;
- recovery completeness; and
- whether the case becomes a regression test.
This is more informative than a pass count. A low-frequency fault with high propagation and irreversible harm may dominate release. A common harmless formatting fault may be accepted or repaired automatically.
Thought experiment: hide the cause from the evaluator
Create two failing runs with the same wrong final answer. In one, retrieval omits the source. In the other, retrieval is correct and the model contradicts it. Give the evaluator only the final answer; it cannot distinguish the failures. Add retrieved evidence; it can. Now replace the evidence with a citation count but no source identity; localisation becomes ambiguous again.
The experiment varies observability. It shows why traces need typed evidence rather than decorative telemetry. More logs do not guarantee diagnosis. The right state must be recorded.
A typed evaluation manifest
The executable artefact defines six layers and a manifest containing checks. A complete positive case covers every layer. A negative case has the correct final state but only outcome and tool evidence; the tool check reports an unauthorised call. The evaluator holds the release because layers are missing and a high-consequence check failed.
. It uses only the Python standard library and includes positive and negative cases.
#!/usr/bin/env python3
"""A typed evaluation manifest spanning component to operational outcome."""
from __future__ import annotations
from dataclasses import dataclass
from enum import Enum
class Layer(str, Enum):
MODEL = "model"
RETRIEVAL = "retrieval"
TOOL = "tool"
TRAJECTORY = "trajectory"
DECISION = "decision"
OUTCOME = "outcome"
@dataclass(frozen=True)
class Check:
name: str
layer: Layer
passed: bool
consequence: int
evidence_id: str
@dataclass(frozen=True)
class Manifest:
claim: str
checks: tuple[Check, ...]
maximum_failed_consequence: int = 1
def evaluate(self) -> dict:
covered = {check.layer for check in self.checks}
missing = [layer.value for layer in Layer if layer not in covered]
failed = [check for check in self.checks if not check.passed]
severe = [check.name for check in failed if check.consequence > self.maximum_failed_consequence]
release = not missing and not severe
return {
"claim": self.claim,
"covered_layers": sorted(layer.value for layer in covered),
"missing_layers": missing,
"failed_checks": [check.name for check in failed],
"severe_failures": severe,
"decision": "release" if release else "hold",
}
def run() -> None:
complete = Manifest(
claim="The bounded workflow produces a verified customer-address update",
checks=tuple(
Check(f"{layer.value}-check", layer, True, 2, f"ev-{index}")
for index, layer in enumerate(Layer, start=1)
),
)
lucky_success = Manifest(
claim="The workflow succeeded once",
checks=(
Check("final-record-looks-correct", Layer.OUTCOME, True, 3, "ev-final"),
Check("tool-call-was-unauthorised", Layer.TOOL, False, 4, "ev-tool"),
),
)
positive = complete.evaluate()
negative = lucky_success.evaluate()
print({"complete_stack": positive, "lucky_success": negative})
assert positive["decision"] == "release"
assert negative["decision"] == "hold"
assert "model" in negative["missing_layers"]
assert "tool-call-was-unauthorised" in negative["severe_failures"]
if __name__ == "__main__":
run()
The code makes two principles executable. Coverage and result are different. A passed outcome cannot fill missing layers. Second, consequence changes authority. A severe failed check blocks release even if other checks pass.
Make test evidence reproducible
Every check should be rerunnable from a manifest or have a reason why it cannot be. Record model, prompt, context compiler, index, tool, policy, workflow, environment, seed and evaluator versions. Store case identifiers and expected properties. Preserve raw structured outcomes where privacy permits, not only charts.
Stochastic systems need repeated runs, but repetition must respect the sampling unit. Ten generations on one case characterise run variance, not population coverage. Sample cases and runs separately. Report the distribution of cost and latency as well as quality; long-tail trajectories can break service even when the mean is acceptable.
Control external state. A live website, mutable API or changing search index can make replay misleading. Snapshot public evidence where licensing and policy permit, or store hashes and effective references. For tool integration, use deterministic simulators for regression and periodic live conformance tests for reality.
Expected outputs should express properties where exact text is unnecessary. A support test may require every consequential claim to link to an accepted evidence node. An action test may require one idempotency key and a verified postcondition. Property assertions survive harmless wording changes better than golden strings.
Evaluation data can contaminate the system
Benchmark cases used repeatedly become development data. Models may have trained on public tasks. Prompt authors learn quirks. Private release sets can leak through logs or judge explanations. Maintain provenance and exposure history for evaluation cases.
Use rotating holdouts and incident-derived cases. Generate metamorphic variants that preserve the expected property while changing surface form. Keep a final hidden set for release, but do not rely on secrecy alone. Mechanistic property tests and live monitoring remain necessary.
Sensitive evaluation data needs purpose-bound access. A red-team corpus may contain harmful payloads or personal information. Isolate execution, redact reports and prevent test content from entering persistent model memory. Evaluation is itself part of the attack surface.
Review the evaluation implementation
Metric code can contain bugs. Labels can misalign after a join. Failed tool calls can be dropped from denominators. A judge parser can treat missing output as a pass. Validate the evaluator with hand-computed cases, mutation tests and schema checks.
Reconcile counts from source events to final metrics. For every exclusion, record a reason. Report missing and invalid cases rather than silently filtering them. An evaluator should fail closed when a mandatory field is absent.
Independent review is especially important when one team both builds and approves a consequential system. Independence does not require organisational hostility. It requires the reviewer to access raw evidence, reproduce key results and raise defeaters without depending on the author’s interpretation.
A production manifest needs more fields: version, claim, test population, metric, denominator, threshold, confidence, owner, evidence location, expiry, response and exceptions. It should be immutable per release and superseded through a new version, not edited after outcomes arrive.
Figure 7. An evaluation manifest binds claim to evidence
The manifest binds one release to its evidence. Every field should answer a practical question. What is claimed? Which layer observes it? Which cases were run? What threshold was agreed before results? Who owns failure? When does evidence expire?
Example manifest fragment
claim: "Address update is applied once to the verified customer"
release: "address-agent/2.4.1"
checks:
- layer: retrieval
property: "effective policy selected as of request time"
threshold: "100% across policy-version boundary suite"
response: "hold release"
- layer: tool
property: "timeout resolves by idempotent readback"
threshold: "0 duplicate effects in 10,000 injected cases"
response: "withdraw execution capability"
- layer: outcome
property: "verified address matches confirmed intent"
threshold: ">= 99.9%; all mismatches reviewed"
response: "stop canary and reconcile effects"Thresholds need justification. “Zero failures” can be appropriate for a bounded deterministic property and misleading for a stochastic population estimate. Record sample size and uncertainty. A test that observed zero incidents in one hundred cases does not establish a one-in-a-million rate.
stateDiagram-v2 [*] --> Claim state "Figure 1. The evaluation stack<br/>nests evidence without<br/>collapsing it" as Claim state "Model layer: can the model<br/>perform the bounded operation?" as Test state "Retrieval and context layer:<br/>did the system acquire<br/>decision-relevant evidence?" as Revision Claim --> Test: expose mechanism Test --> Decision: evidence survives Test --> Revision: rival survives Revision --> Claim: narrow or rebuild Decision --> [*]
Release through progressive contact
Evaluation changes as the system approaches live consequence.
Figure 5. Release gates consume different evidence
Component tests run quickly and isolate logic. Integration tests verify contracts across real components. Shadow tests observe live inputs without changing outcomes. Canaries apply bounded effects to a limited population with rollback and enhanced review. Operational monitoring keeps claims conditional on current evidence.
Each stage consumes evidence from the previous stage and adds contact with reality. Passing a canary does not delete component tests. A later failure should be reproduced at the earliest feasible layer. This shortens diagnosis and prevents recurrence.
Shadow limitations
Shadow systems do not experience every effect of deployment. Users and operators may behave differently when recommendations matter. Latency may change when tools execute. Labels may depend on actions. Treat shadow evidence as a test of live input and proposed behaviour, not a full outcome trial.
Canary requirements
A canary needs bounded population, consequence limits, explicit owner, stop thresholds, effect inventory, rollback or compensation and independent outcome verification. Randomly selecting one per cent of traffic is insufficient when rare high-risk cases concentrate outside or inside that slice.
Use risk-stratified inclusion and exclusion. Some actions may remain proposal-only until stronger evidence exists. Canary authority should be narrower than target authority.
Recovery completes the evaluation loop
A rollback that restores a software version may not restore business state. The system may have sent messages, changed records, created orders or influenced human decisions. Evaluation should inventory effects and test compensation or escalation for each.
Run game days with bounded synthetic or sandbox effects. Verify that the incident route can withdraw capabilities, stop new work, identify affected cases, reconcile unknown outcomes, apply compensation and notify accountable owners. Measure time to containment and time to verified recovery separately.
Post-incident review updates more than a runbook. It creates a regression case at the earliest layer able to detect the mechanism, a propagation test across the stack and an operational monitor if recurrence remains possible. It also revises the assurance claim and residual risk.
Recovery evidence matters before release. A system whose failures cannot be observed or repaired should receive less authority even if its success rate is high. Irreversibility changes the threshold.
Monitoring must change something
A metric without a response is observation, not control. Every operational signal should map to continue, investigate, reduce authority, rollback, compensate or escalate. Alert fatigue indicates that thresholds or ownership are broken.
Version the monitoring logic. A release can remain unchanged while its evaluation policy changes. Preserve which policy judged each event so counterfactual replay can reconstruct the decision.
The stack completeness record
Figure 8. The stack is complete only when failure changes authority
| Layer | Primary question | Representative evidence | Release-changing failure |
|---|---|---|---|
| Model | Can the bounded operation be performed under controlled context? | Accuracy, calibration, robustness, structured-output tests | Material regression or unsupported capability |
| Retrieval | Did permitted, fresh and decision-relevant evidence enter context? | Coverage, authority, freshness, contradiction and injection suites | Mandatory evidence missing or untrusted instruction promoted |
| Tool | Was the correct effect requested and resolved exactly once? | Schema, identity, timeout, idempotency and readback tests | Wrong entity, unauthorised call or unknown outcome mishandled |
| Trajectory | Did the sequence remain in the allowed state envelope? | Transition properties, fault propagation and recovery traces | Control bypass or unrecoverable state |
| Decision | Did evidence and consequence produce the right bounded action? | Utility, abstention, human review and authority tests | Irreversible action below evidence threshold |
| Outcome | Did the live process improve without unacceptable harm? | Verified effects, service, harm, appeal, cost and recovery | Outcome threshold crossed or effect cannot be reconciled |
Completeness does not mean equal test volume at every layer. Allocate depth by consequence and uncertainty. A read-only drafting assistant needs strong privacy and factual-support tests but no payment postcondition. A funds-transfer agent needs deep tool, authority, unknown-outcome and recovery evidence.
Evidence debt
When a component changes, identify which evidence expires. A model update may invalidate output, decision and trajectory results. An index update may invalidate retrieval and answer support. A tool API change may invalidate integration and recovery. A policy change may invalidate the whole decision route even when code remains fixed.
Build a dependency graph from release assets to checks and claims. Re-run the affected subgraph. Blanket retesting wastes time; selective retesting without dependency evidence misses coupling.
Sample operation without losing the denominator
Full adjudication of every live case may be impossible. Use risk-stratified sampling: a random base sample estimates ordinary performance, while oversamples target high consequence, new regimes, disagreement, low confidence and control anomalies. Keep inclusion probabilities so weighted estimates can recover the population where assumptions hold.
Do not report the oversample as prevalence. A review queue enriched for suspected failures will look worse than operation by design. Conversely, a convenience sample of easy completed cases will look better. Link every reviewed case to the live denominator and sampling route.
Delayed outcomes require cohorts. Freeze which cases entered each window and track maturity. Recent cohorts may have more unknown outcomes, so comparing raw rates with mature cohorts creates false improvement. Report pending, resolved and unobservable states separately.
Review capacity is part of the evaluation system. When alerts exceed capacity, selection changes and feedback weakens. Monitor queue age, reviewer agreement and unreviewed high-risk cases. If oversight cannot execute the promised control, reduce automated authority rather than treating the review plan as evidence.
Publish a compact operating ledger beside the release: active claims, latest evidence, open defeaters, expired checks and current authority. This gives operators one place to see whether the system is still inside its evaluated boundary without collapsing the underlying layers into a green badge.
The ledger should preserve disagreement. A retrieval check may be healthy while live outcomes deteriorate, or outcome evidence may be immature while tool controls remain strong. Showing those states separately prevents a composite score from averaging away the exact layer that should limit authority. Operators need the weakest relevant link and its consequence, not a ceremonial percentage.
Keep the underlying receipts addressable so a reviewer can move from claim to case without reconstructing the release by hand.
Where the stack ends
No finite stack proves that an open system will never fail. Case coverage is incomplete. Rare interactions and strategic behaviour remain. Outcomes can be contested or delayed. Evaluation reduces uncertainty and constrains authority; it does not certify the future.
The strongest objection is operational burden. Six layers, manifests and fault injection can become process theatre. The remedy is claim-driven minimalism. If a metric cannot change a claim, locate a fault or alter authority, remove it. If a simpler deterministic control prevents the failure, prefer it over another evaluator.
Another objection is metric conflict. Component quality may rise while outcome quality falls. This is not a reason to select one master metric. It is evidence that the causal bridge is incomplete. Keep the layers separate long enough to diagnose the conflict.
The correct address at the opening was real. So were the skipped identity check, duplicate effect and hidden repair. A one-point benchmark preserved only the part of the story easiest to count.
The evaluation stack exists to stop a visible success from erasing its route. Test components for localisation, trajectories for control, decisions for consequence and outcomes for real value. Bind them with a manifest. Let failures withdraw authority. When the final state is right for the right reasons, the system has earned a stronger claim. When it is right by accident, the stack should be capable of saying no.
sequenceDiagram participant Q as Question participant M as Mechanism participant E as Evidence participant D as Decision Q->>M: The correct address reached by the wrong route M->>E: Figure 1. The evaluation stack nests evidence without collapsing it E-->>M: compare rival explanations E->>D: Sources D-->>Q: revise scope or act
Sources
- Primary Liang et al., Holistic Evaluation of Language Models, 2022.
- Primary Liu et al., AgentBench: Evaluating LLMs as Agents, 2023.
- Primary Jimenez et al., SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, 2023.
- Primary Es et al., RAGAS: Automated Evaluation of Retrieval Augmented Generation, 2023.
- Primary Lightman et al., Let’s Verify Step by Step, 2023.
- Primary Mitchell et al., Model Cards for Model Reporting, 2019.
- Official NIST, AI Risk Management Framework, official framework.
- Official OpenTelemetry, Specification, current official observability specification.