The validation object is a configured decision system

The phrase foundation model was chosen to mark both breadth and incompleteness. The original Stanford report describes such models as adaptable intermediary assets whose defects may propagate into downstream applications. It also argues that societal effects become more tractable when examined in specific systems used by people.¹ That observation has a direct validation consequence: evidence about the base model cannot settle what an adapted, retrieved, tooled and governed application will do.

A compound AI system combines one or more models with ordinary software and organisational controls.² It may include a prompt compiler, retrieval service, reranker, calculator, policy engine, tool gateway, case store, human approval queue and outcome observer. The useful boundary is neither every dependency in the enterprise nor the model endpoint alone. It is the smallest boundary that contains the mechanisms capable of changing the decision or effect under review.

Validation is conditional, not a permanent property

A model is often discussed as though validation were a durable label attached to an artefact. In practice, every defensible conclusion is conditional. It holds for a version, population, purpose, environment, threshold and period of evidence. A supplier may show that a model performs well on a public benchmark. A local team may show that a prompt produces grounded drafts on sampled cases. Neither statement establishes that the surrounding workflow will use the right customer, current policy, permitted tool or final approval.

For example, “the model gives correct credit advice” is not a usable validation claim. A bounded claim might state that release 2.4.1, used by authorised credit staff for supported renewal cases, produces no recommendation when mandatory evidence is absent, binds every material claim to a current source, and cannot save a memo unless the final version has been approved. The validator can construct counterexamples for that statement. The committee can also see exactly which adjacent uses remain unapproved.

The equation is a boundary reminder, not a claim that the system is fully known. It makes four distinctions visible. First, an output and an effect are separate. A proposed transfer and a posted transfer are not the same event. Second, current world state differs from remembered belief. Third, the release version includes more than a model name. Fourth, a human reviewer is part of the mechanism only to the extent that the queue, information, time and authority make review effective.

Four units that should not be collapsed

A model is a computational component. A configured system is the connected implementation that uses it. A use envelope states users, decisions, populations, data, exclusions, environments, action rights and limits. A release is a reproducible set of versions and policies that can be tested, monitored and rolled back. Validation can challenge a claim only when all four are named.

This resolves a common inventory dispute. The foundation model can remain one inventoried model or third-party component. The deployed application can simultaneously be an inventoried AI system with dependencies and controls. The answer need not be a metaphysical decision about whether the workflow “is a model”. It is an evidence decision: which boundary must be represented so that a material failure cannot hide between registers?

Composition creates failure modes that component tests cannot infer

Suppose a retriever selects the right document in 96 per cent of a test set, a model answers correctly in 94 per cent of cases when given the right passage, and a policy gate applies its rules correctly in 99 per cent of unit tests. Multiplying those rates would already be questionable because the tests use different populations and conditions. More importantly, the product says nothing about failures created by the seams: an effective date parsed in the wrong locale, a missing source identifier, a retry that duplicates an action, or a gate that checks the draft before a later model rewrite.

System error is not the sum of component error. It includes interaction terms. Components may fail on the same hard cases, so errors are correlated. Routing changes which population reaches each model. Retrieval alters the evidence distribution. A policy engine may be correct about the state it receives while that state is stale. A human may be formally present but unable to review the volume. Each local statement can be true while the system claim is false.

Minimal worked example: the date that crosses an interface

Consider a synthetic benefits assistant. The authoritative policy says that a rule becomes effective on 3 April. The document store exports the date as 03/04/2026. The retrieval component is evaluated only on whether it returns the correct paragraph, which it does. The language model explains the paragraph accurately under a day-month assumption. The downstream rules service runs in a month-day locale and interprets the date as 4 March. Every component passes its own local test, yet the assembled system applies the rule one month early.

The remedy is not another general benchmark for the model. The relevant evidence is a contract test at the interface: dates must be transported in ISO 8601 format, the effective-time semantics must be explicit, and a property test must reject ambiguous values. An end-to-end scenario then confirms that a request on 20 March uses the old rule and a request on 10 April uses the new one. The failure existed in representation and composition, not in paragraph retrieval or language understanding.

Ordering is part of the modelled behaviour

A workflow can use the correct components in the wrong sequence. A safety classifier that runs before a tool result arrives cannot inspect the result. An approval recorded before the final draft is regenerated does not approve the final draft. A permission checked when a task starts may expire before an action executes. These are not merely implementation defects around the “real” model. They change which function the system computes.

The evaluator can become another unvalidated model

Open-ended system tests often use a language model to grade groundedness, policy adherence or answer quality. That can increase coverage, but the judge is a measurement instrument, not an oracle. Research has documented position, verbosity, self-preference and reasoning limitations in LLM judges.⁸ A validation programme should therefore define the judging rubric, blind or randomise avoidable biases, calibrate against adjudicated examples, report disagreement, and retain deterministic checks where the property is mechanical.

Replacing one uncertain component with an uncalibrated evaluator does not create independent evidence. The judge may share training data, preferences or blind spots with the system under test. Independence is strengthened by heterogeneous evidence: property tests, reference calculations, source checks, fault injection, human adjudication and operational telemetry, rather than by adding more model opinions.

A serious negative control: when model-level validation is enough

Consider a fixed logistic-regression model that estimates the probability of equipment failure from six validated sensor fields. It runs in a batch job, produces no external action, has no retrieval, free text, memory, model routing or changing prompt, and feeds a dashboard whose users follow a separately controlled procedure. Its input schema and transformation code are stable and tested. Here the system boundary still matters, but a conventional review of data, methodology, implementation, performance, limitations and monitoring may cover nearly all material uncertainty.

This negative control prevents the method from becoming governance maximalism. The need for compound validation rises with compositional uncertainty, consequence, statefulness, environmental openness and authority. It does not rise merely because a system has a fashionable label.

flowchart LR
  A["The approval that disappears<br/>when one wire moves"] --> B["The validation object is a<br/>configured decision system"]
  B --> C["Composition creates failure<br/>modes that component tests<br/>cannot infer"]
  C --> D["Validation must follow claims<br/>through the whole lifecycle"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#007D68,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Validation must follow claims through the whole lifecycle

NIST's AI Risk Management Framework is intentionally system-oriented and voluntary, covering the design, development, use and evaluation of AI products, services and systems.³ Its generative-AI profile notes that risks vary by lifecycle stage and scope, and that third-party models, data and libraries complicate attribution.⁴ The PRA's model-risk principles stress inventory, tiering, governance, independent validation, monitoring and model interdependencies for firms in scope.⁵ These sources support a direction, but a validator still needs an operational unit of work.

That unit should be a falsifiable claim. “The system is safe” is too broad to test. “For applications within the documented small-business population, the system does not write a credit recommendation unless required data and policy evidence are present, calculations match the reference engine, and an authorised reviewer approves the final version” can be challenged. It names population, behaviour, preconditions, authority and effect.

A validation programme should build evidence vertically from components to outcomes and horizontally across the lifecycle. The vertical dimension catches local and interaction defects. The horizontal dimension catches changes, drift, incidents and expired assumptions. Neither can substitute for the other.

Seven claims in a validation dossier

1. Purpose and use-envelope claim. Intended users, decisions, populations, data classes, exclusions, environments, action rights and prohibited uses are explicit. The validator tests whether the stated purpose matches the actual workflow and incentives.
2. Boundary and lineage claim. Models, prompts, retrieval indexes, rules, tools, schemas, identities, human roles and operational dependencies are inventoried and pinned to a reproducible release. Supplier evidence is linked without being mistaken for local validation.
3. Component-fitness claim. Each material component meets its local requirement under relevant slices, boundary conditions and degraded inputs. This includes deterministic calculations, access-control decisions and data-quality checks as well as model evaluations.
4. Interaction and trajectory claim. Interfaces preserve type, identity, time, provenance and semantics. Observable trajectories obey required ordering, do not bypass controls, and retain enough evidence to reconstruct the recommendation and effect.
5. End-to-end outcome claim. Representative, severe, rare and adversarial scenarios test business properties, not merely fluent answers. The test set includes abstention, escalation, incomplete evidence, conflicting evidence and unknown outcomes.
6. Authority and recovery claim. Stochastic proposals cannot exceed mechanically enforced permissions. Human review has information, capacity and genuine veto power. Effects are verified, duplicate actions are prevented, and interruption, rollback and reconciliation are practised.
7. Operations and change claim. Monitoring detects movement in evidence, population, policy, tools, workflow and outcomes. Defined changes trigger targeted or full revalidation. Residual risks, conditions, expiry and the accountable acceptance decision remain visible.

These claims are deliberately broader than a model card. Model cards and datasheets improve documentation of models and datasets, while system cards and internal audit frameworks extend attention to interactions, use and organisational process.⁹¹⁰ The validation dossier differs by making a release decision explicit: claim, evidence, counter-evidence, owner, expiry, restriction and trigger.

Supplier evidence must be translated into local claims

A foundation-model provider may supply evaluation results, a model or system card, security information, change notices and restrictions. Those materials are necessary inputs to third-party risk and component assessment. They are not substitutes for local evidence because the provider does not control the institution's data selection, prompts, retrieval corpus, tool permissions, reviewer capacity or business effect.

It helps to separate three evidence statuses. Inherited evidence concerns a property that transfers without material change, such as the cryptographic identity of a pinned model artefact. Translated evidence concerns a supplier result whose relevance must be justified for the local population and configuration. Local evidence is generated from the configured system and use envelope, including interface contracts, severe scenarios, authority tests, operational drills and outcome monitoring.

The burden is asymmetric. Strong local controls can contain some model limitations, but a supplier benchmark cannot prove those controls exist. Conversely, a system-level pass should not hide a weak component that succeeds only because a narrow test population avoids its limitation. The dossier should preserve both levels and show the bridge between them. When supplier transparency is insufficient, the response is a tighter envelope, stronger containment, more conservative monitoring or no release, rather than an invented assurance claim.

Failure injection is the primary test of composition

Ordinary test sets ask whether the system succeeds under sampled cases. Failure injection asks how faults propagate. Replace an authoritative document with a superseded one. Return a tool timeout after the external effect may already have occurred. Corrupt a source identifier while preserving fluent text. Exhaust the human-review queue. Revoke an identity between proposal and execution. Make the evaluator disagree with the reference adjudication. The validation question becomes: which control contains the fault, what evidence records the event, and can the system return to a known state?

Behaviour-driven evaluation research similarly argues that aggregate benchmarks can miss operational failure modes in compound systems and proposes explicit test specifications tied to scenarios.⁷ That result should not be generalised beyond its evaluated tasks, but its design principle is durable: state the expected behaviour under a named condition, then generate or curate cases that can falsify it.

The validation boundary is also an observability requirement

A validator cannot establish what happened if the system does not retain decision-relevant events. That does not mean recording private chain-of-thought. It means recording source identifiers and versions, structured claims where needed, tool proposals and results, policy decisions, approvals tied to the final artefact, external effect receipts, errors, retries and the release manifest. The log should be minimised and protected, but sufficient for replay and attribution.

If a consequential system cannot be versioned, observed, replayed or bounded, independent validation should block release rather than manufacture confidence. This is a failure boundary of the method. A pre-release programme cannot prove safe behaviour across an open world. It can establish bounded claims, expose known defeaters, set runtime controls and define when authority must be withdrawn.

A production-shaped credit case

The following case is a synthetic composite. It is not evidence from a named deployment. A commercial-credit assistant prepares a draft memo for small-business renewals. It gathers authorised customer and facility data, calls a deterministic financial-ratio service, retrieves the effective lending policy, asks a language model to organise evidence and identify gaps, and presents a draft to a credit officer. The officer retains the decision. The system may save the approved memo, but it cannot approve, decline or change a facility.

The intended use seems modest, yet its behaviour spans several mechanisms. A correct model can cite a superseded policy. A correct ratio service can receive the wrong accounting period. A correct policy gate can inspect a draft that changes later. An attentive officer can miss a discrepancy if the interface hides provenance. A successful save call can time out before confirmation, creating an unknown outcome and a duplicate on retry.

Define the use envelope before testing

The release is limited to renewals below a stated exposure threshold, legal entities with one supported accounting standard, English-language source documents, a named policy family and staff in the credit function. It excludes new-to-bank lending, distressed cases, guarantor analysis, exceptions requiring committee authority and any automated customer communication. Required inputs, unsupported document types, review time and escalation routes are explicit.

This envelope does two jobs. It creates a valid test population and prevents an approval from leaking into adjacent uses. A high score on ordinary renewals cannot authorise distressed restructures. A memo-writing approval cannot authorise a lending decision. Scope is a control only when the runtime can recognise and reject cases outside it.

What the validator actually tests

Component tests establish extraction quality by document type, ratio-service agreement with a reference implementation, policy-retrieval recall for effective and superseded versions, model claim support, abstention and draft completeness. Interface tests verify customer identity, accounting period, units, policy effective time, source provenance and stable identifiers. The model's supplier report informs capability and limitations, but local prompts, retrieved evidence and workflow receive their own evidence.

End-to-end scenarios then vary one material feature at a time: missing accounts, conflicting revenue figures, a policy updated during an open case, an unsupported entity, an unavailable ratio service, malicious instructions embedded in a document, an officer rejecting the draft, and a save request with an ambiguous outcome. The expected behaviour is specified before running the test. For example, a missing mandatory account should produce an evidence-gap state and no recommendation, not a plausible estimate.

Failure injection reveals two blocking defects in the first synthetic release. A policy-index refresh can briefly serve a mixed set of old and new chunks without exposing the index version. Separately, the save tool times out after committing, while the workflow retries with a new idempotency key. The first defect defeats policy provenance. The second can duplicate a memo and makes the outcome unknowable from the task trace. Neither appears in model accuracy.

The release decision is therefore conditional on an atomic index publication with versioned readback, and blocked from save access until the tool supports stable idempotency and postcondition verification. The drafting capability may still proceed in a sandbox without external writes. Validation changes authority at the narrowest causal boundary rather than approving or rejecting “AI” as one indivisible object.

A proportionate decision instrument

Validation intensity should be driven by two dimensions: consequence if the system is wrong, and compositional uncertainty about how behaviour arises. A high-consequence deterministic calculator needs strong mathematical and implementation validation but may not need conversational red teaming. A low-consequence research assistant with many tools may need broad interaction testing but limited action controls. A high-consequence, stateful agent needs both.

The validation dossier is a living release argument

A test report lists activities and results. A dossier explains why those results support a bounded release decision. Each claim links to evidence and to defeaters: observations that would weaken or invalidate it. The dossier also records who owns remediation, who accepts residual risk, when the evidence expires and which changes reopen the claim.

This matters because compound systems evolve asynchronously. A provider changes a model alias. An embedding update changes retrieval neighbourhoods. A tool adds an optional field. A policy exception is superseded. A human queue grows. A monitoring judge is replaced. If approval is represented only by a document signed at one date, the live configuration can drift away while the approval appears intact.

Executable artefact: validation dossier gate

The following client-side tool evaluates a small JSON dossier. It does not certify a real system. Its purpose is to expose release logic that prose often hides: required claims, critical failures, unresolved conditions and accountable acceptance. The positive case should return RELEASE. The negative case should return BLOCK because the release is unpinned, an interface test fails, rollback is unproved and residual risk lacks acceptance.

function evaluateDossier(dossier) {
  const required = [
    "use_envelope", "pinned_boundary", "component_fitness",
    "interfaces", "end_to_end", "authority_recovery",
    "operations_change"
  ];

  const blockers = [];
  const conditions = [];
  for (const id of required) {
    const claim = dossier.claims.find(item => item.id === id);
    if (!claim) blockers.push(`Missing claim: ${id}`);
    else if (!claim.evidence?.length) blockers.push(`No evidence: ${id}`);
    else if (claim.status === "fail") blockers.push(`Failed claim: ${id}`);
    else if (claim.status === "conditional") conditions.push(id);
    if (claim?.defeaters?.some(item => item.severity === "critical" && item.open)) {
      blockers.push(`Open critical defeater: ${id}`);
    }
  }

  if (!dossier.release?.pinned) blockers.push("Release is not pinned");
  if (!dossier.accountable_owner) blockers.push("No accountable owner");
  if (!dossier.rollback_tested) blockers.push("Rollback not demonstrated");
  if (!dossier.residual_risk_accepted) blockers.push("Residual risk not accepted");

  return blockers.length ? "BLOCK" : conditions.length ? "CONDITIONAL" : "RELEASE";
}

How independent challenge changes the dossier

The development team supplies design evidence, test results and known limitations. Independent validation reconstructs the system boundary, challenges data and scenarios, reruns material tests, probes interactions, assesses evaluators, and decides whether the evidence supports each claim. It should be able to narrow the use envelope, impose conditions, require remediation or recommend that authority remain disabled.

The accountable business owner does not outsource the decision to validation. The owner accepts or rejects residual exposure within delegated authority, with the dossier showing what has and has not been established. Internal audit may then assess whether the framework, independence, records and decisions operate as intended. These roles remain distinct even when a smaller organisation combines people across them.

The release gate is not a score threshold. A severe unresolved defect cannot be averaged away by many passing tests. Conversely, a minor documentation condition need not block a low-consequence sandbox. The dossier must support non-compensatory logic where the property demands it: unbounded authority, unknown identity, absent effect verification or an unversioned policy can each be a veto.

What should trigger revalidation

Triggers should follow causal change: a model or decoding update; prompt or system-instruction change; embedding, index or source change; tool schema or permission change; workflow, routing or retry change; policy or threshold change; new population, language or use; material drift; incident; evaluator replacement; weakened human capacity; or a new external dependency. A calendar review remains useful, but it should catch silent deterioration, not substitute for event-driven control.

Targeted revalidation is legitimate when lineage and dependencies are precise. A formatting-only interface change may need contract and regression tests. A new model, action right or policy family may reopen most claims. Where change impact cannot be determined, the uncertainty itself justifies broader testing.

stateDiagram-v2
  [*] --> Claim
  state "Validation is conditional, not<br/>a permanent property" as Claim
  state "Four units that should not be<br/>collapsed" as Test
  state "Thought experiment 1: remove<br/>only the action right" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Compact glossary

Configured system

The models, context, tools, policy, workflow, people and operational dependencies connected to produce a decision or effect.

Use envelope

The approved users, decisions, populations, data, environments, action rights, exclusions and limits for one system release.

Release manifest

A reproducible record of material versions, configurations, policies, schemas, permissions and evidence applicable to a release.

Defeater

An observation or condition that weakens, limits or invalidates a claim, such as a failed slice, bypass path or unversioned dependency.

Effect receipt

Evidence from an authoritative system that a proposed external action produced, did not produce, or has an unknown outcome.

Revalidation trigger

A change or event that requires specified validation claims and tests to be reopened.

Research and source notes

1. Rishi Bommasani et al., “On the Opportunities and Risks of Foundation Models”. Primary research report defining foundation models as broad but incomplete intermediary assets.
2. Matei Zaharia et al., “The Shift from Models to Compound AI Systems”. Research-lab synthesis on systems combining models with retrieval, tools and programmatic components.
3. NIST, Artificial Intelligence Risk Management Framework (AI RMF 1.0). Official voluntary framework covering AI products, services and systems.
4. NIST, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. Official profile on generative-AI risks, value chains, testing, monitoring and incident considerations.
5. Prudential Regulation Authority, SS1/23: Model risk management principles for banks, current 2026 version. Official supervisory statement for firms in scope.
6. Federal Reserve, OCC and FDIC, Supervisory Guidance on Model Risk Management, SR 26-2. Official 2026 guidance; generative and agentic AI are expressly outside its formal scope.
7. Pranav Bhagat et al., “Evaluating Compound AI Systems through Behaviors, Not Benchmarks”. Peer-reviewed behaviour-driven evaluation method for information-seeking compound systems.
8. Lianmin Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena”. Primary study documenting utility and biases of model-based evaluators.
9. Margaret Mitchell et al., “Model Cards for Model Reporting”; Timnit Gebru et al., “Datasheets for Datasets”. Primary documentation proposals for model and dataset transparency.
10. Meta AI, “System Cards, a new resource for understanding how AI systems work”. Practitioner documentation approach that makes system interactions and use visible; not independent validation evidence.
11. Boming Xia et al., “An AI System Evaluation Framework for Advancing AI Safety”. Peer-reviewed taxonomy distinguishing component and system evaluation across the lifecycle.
12. D. Sculley et al., “Hidden Technical Debt in Machine Learning Systems”; Eric Breck et al., “The ML Test Score”. Foundational engineering analyses of dependencies, testing and monitoring beyond offline model experiments.

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: The approval that disappears when one wire moves
  M->>E: Validation is conditional, not a permanent property
  E-->>M: compare rival explanations
  E->>D: Validate the release that can cause the consequence
  D-->>Q: revise scope or act

Validate the release that can cause the consequence

A foundation model can be evaluated without knowing every application that will use it. A consequential application cannot be validated by pretending those possible configurations are equivalent. The moment context selects evidence, tools expose capabilities, workflow orders checks, policy grants authority, people review, and operations produce effects, the relevant object has changed.

The practical decision is to approve a bounded release claim, not a model in the abstract. Inventory the full causal boundary. Pin the configuration. State the use envelope. Test components, seams, trajectories, decisions, effects and recovery. Inject failures. Calibrate evaluators. Preserve defeaters. Tie every material change to a revalidation path. Keep action rights narrower than the evidence supports.

This does not discard established model-risk practice. It keeps its most durable disciplines: clear purpose, effective challenge, independent validation, limitations, monitoring, governance and accountable use. It changes their unit of application. For a compound system, the dossier must show how those disciplines survive composition.

The decisive question for a risk committee is therefore not “Has the foundation model been validated?” It is: for this version, population, evidence path, authority boundary and operating environment, what behaviour has been established, what could defeat that claim, and how quickly can the institution withdraw or recover when reality disagrees?