A familiar delivery failure occurs when a strong demonstration reaches model risk, compliance and operational-resilience review without an evidence file. Questions about intended use, data, failure modes, permissions, fallback and monitoring then become late design work. The schedule impact follows the size of those gaps and the institution's review capacity.
None of those questions are unreasonable. Most of them are versions of a single question: "How do you know, and can you prove it?" Everything in this article is about being able to answer that question before you're asked, because in a regulated environment, the evidence is the product. The model is just an ingredient.
Model risk management: classify the system before choosing controls
Banks have long used inventories, tiering, independent validation and ongoing monitoring for model risk. The regulatory classification of an LLM system is not identical everywhere. In the United States, the Federal Reserve’s SR 26-2 superseded SR 11-7 in April 2026. It expressly excludes generative and agentic AI from that guidance’s model definition. It also says a bank’s wider risk-management and governance practices should guide controls for tools outside scope. UK and other supervisory regimes use their own definitions. The practical step is therefore to document the classification, obtain risk approval and govern the complete system in proportion to its use and consequence.
What MRM means in practice for an LLM system:
- Govern the system, not only the model. The relevant unit includes model, prompts, retrieval, tools, guardrails and interface as a versioned whole. A prompt change is a system change; whether it is material depends on the behaviour and evidence it can affect.
- Independent validation needs materials that don't exist unless you build them. Validators will ask for intended-use definitions, known limitations, evaluation methodology, and evidence of performance on your data: not vendor benchmark tables. A vendor's model card is an input to your documentation, not a substitute for it.
- Tier honestly. Risk scales with use, autonomy and consequence. A drafting assistant with competent review differs from an agent that acts on customer accounts. Narrowing autonomy can reduce exposure, but only when the human step and fallback are operationally real.
The teams that succeed treat MRM as a design partner from week two. The teams that suffer treat it as a gate at the end.
Evals as release gates, not research artifacts
In a bank, evaluations should change release decisions. The suite is executable evidence for fit-for-purpose claims. A prompt, model, retrieval or guardrail change enters the relevant impact check and cannot reach production while a material gate remains failed or unevaluated.
The structure that has held up for us:
- Golden datasets per use case, built with the business specialists who own the process and covering edge cases, vulnerable-customer scenarios, adversarial phrasing and out-of-scope requests. “Twenty per cent” can be a planning placeholder, not a target. Coverage follows the failure model.
- Layered metrics. Retrieval correctness, factual grounding, policy compliance, tone, and format measured separately. A single end-to-end score hides which component regressed.
- Thresholds agreed with risk, in writing, before launch. "Grounded-answer rate ≥ X, harmful-output rate ≤ Y on the adversarial set": signed. When the gate fails at 2am before a release, you want the decision to already have been made by the threshold, not litigated by whoever's on the call.
One hard-won warning: a golden dataset can decay. Products change, policies are superseded and customer language shifts. Use event-driven review plus a scheduled coverage check, and treat incidents and overrides as candidate cases. Six months without a content change does not itself prove staleness; changed sources, uncovered demand and failed cases do.
Guardrails: defence in depth, because every layer fails sometimes
No single guardrail works. Production safety is a stack, and each layer exists because some incident (ours or someone else's) proved the layer above it insufficient:
- Input controls: prompt-injection screening, PII handling, topic scoping before the model ever sees the request.
- Constrained generation: schema-enforced outputs wherever possible. Structure is the cheapest guardrail there is: a system that must emit valid JSON against a schema has far fewer ways to go rogue than one emitting prose.
- Output checks: grounding verification against retrieved sources, policy classifiers, PII egress scanning.
- Action controls: for agentic systems, this is the layer that matters most: tool entitlements, risk-tiered approvals, rate and value limits. A wrong statement is an incident; a wrong action is a regulatory event. Budget your paranoia accordingly.
- Human review, mandatory for whatever your tiering says is high-consequence.
Guardrails need their own evaluations because they can miss attacks or over-trigger on legitimate work. Material interventions should create categorised, privacy-controlled events. Those events support tuning and incident evidence; logging everything without purpose, access limits and retention controls creates another risk.
Rollback: decided in advance, executed in minutes
Everything is versioned together (model route, prompts, retrieval config, guardrail config) as one deployable unit, because rolling back a prompt against a mismatched retrieval index just creates a second incident. Beyond that, three requirements:
- Progressive exposure. Where privacy and operational design permit it, use shadow comparison before a small canary and graded ramp. Shadow evidence exposes the production distribution before outputs are served, but it does not test user reliance, action effects or every live dependency.
- Rollback triggers written down before launch. Specific metrics, specific thresholds, named decision-owner. In the middle of an incident, nobody thinks clearly; the runbook thinks for you.
- Degraded modes, not just off-switches. The mature version of rollback is a ladder: full service → reduced autonomy (human approval on everything) → template-only responses → graceful hand-off to human channels. Killing the service entirely is itself an operational-resilience event if the business now depends on it. Build the intermediate rungs.
Monitoring drift: the system changes under you even when you change nothing
LLM systems can drift through inputs, served model behaviour and knowledge. Traditional application monitoring remains necessary for latency, failures and dependencies, but it does not by itself measure grounding, policy compliance or task quality.
A practical stack combines scheduled canary cases, distribution monitoring, grounding samples, human adjudication and probes for served-model changes. A provider may change behaviour within its published or contracted version policy. The control goal is to detect a material change through your own evidence rather than assume either immutability or inevitable silent change.
Ongoing monitoring appears across many risk-management regimes, but scope and terminology differ. Operational and validation views can share governed evidence while exposing the detail each audience is authorised to see.
Compliance sign-off: engineer the evidence path
Final sign-off is materially easier when evidence is produced by the delivery process rather than reconstructed at the end. It is not the only variable: risk appetite, legal interpretation, third-party terms and operational readiness can still govern the decision.
The practices that reduce avoidable delay are early second-line engagement, a standing artefact trail, a living system file and claims traceable to evaluation runs, approved records or monitoring evidence. They improve flow; they do not promise that quarters will always become weeks.
A claim you can't demonstrate on demand is a liability, so make fewer claims and instrument every one of them.
The real timeline
Prototype-to-production time varies from a bounded internal assistant to a high-consequence acting system. Six to eighteen months may be a planning range for some programmes, not a benchmark. Evaluation, evidence, controls, monitoring and governance engineering often dominate the critical path, and reusable machinery can reduce repeated work without weakening review.
A global programme also needs a jurisdiction map before it needs another generic checklist. Record which legal entity owns the use, where data originates and is processed, which customers or employees are affected, which third parties participate and which internal policies apply. Link each requirement to a control owner and evidence identifier. This prevents a statement written for one country or business line from silently becoming the approval basis for another.
The map should separate binding obligations, supervisory guidance, internal policy and prudent engineering practice. All four can shape the design, but they have different authorities and change processes. Do not present a useful control pattern as though a regulator prescribed that exact implementation. Record the source, interpretation date, accountable legal or compliance reviewer and the system decision it changes. That discipline makes the architecture easier to update when guidance changes.
That machinery is the moat. The model never was.
Start with the decision system, not the model endpoint
The regulated object is the decision system that people will rely on. That system includes retrieval, prompts, policies, tools, identities, user interfaces, human queues and downstream records. The model may be changed without changing the use. A small prompt edit may change the use materially. Classification therefore starts with purpose and consequence.
A useful intake record answers nine questions. What business decision is supported? Who is affected? Which data enters the system? Which external effects can occur? Which steps remain deterministic? Where can a person intervene? What is the maximum plausible harm? Which legal entities operate the service? Which evidence will demonstrate control?
The record should be short enough to review. It should also be precise enough to determine ownership. If the intended use cannot be stated without words such as “assist” or “enhance,” the boundary is not yet clear. Replace those words with observable actions. “Drafts a complaint response for mandatory case-handler review” is classifiable. “Enhances complaint handling” is not.
This sequence prevents a common governance error. Teams often classify a foundation model once and reuse that classification for every application. The same model may summarise an internal meeting, prepare a customer notice and propose a payment action. Those uses have different consequences. Risk follows the use, integration and authority, not the model brand.
Build one control architecture across four lines of evidence
The control architecture has four linked evidence lines. Product evidence shows that the use case solves the stated problem. Technical evidence shows that the system behaves inside its tested envelope. Operational evidence shows that people and fallback routes work under real conditions. Governance evidence shows who accepted the remaining risk and which changes require renewed review.
| Evidence line | Core question | Typical artefacts | Independent challenge |
|---|---|---|---|
| Product | Is the use bounded, useful and suitable? | Intended use, process map, baseline and affected-party analysis | Business risk and product review |
| Technical | Does the complete system meet its claims? | Version manifest, evaluations, threat model and trace samples | Validation, security and privacy review |
| Operational | Can people run and recover the service? | Queue study, fallback test, support model and incident exercise | Operations and resilience review |
| Governance | Is authority explicit and current? | Classification, owners, exceptions and approval record | Compliance, legal and accountable executive |
No line substitutes for another. Strong task accuracy does not prove that access control works. A signed policy does not prove that escalation capacity exists. A penetration test does not prove that a reviewer understands the output. A release case is credible only when these evidence lines meet at the same versioned system.
The system manifest is the joining key. It should identify the model and decoding settings, prompts, retrieval index, source snapshot, chunking policy, tools, permission policy, guardrails, interface version and escalation policy. The manifest need not expose secrets to every reviewer. It must let authorised teams reconstruct what ran.
This is also where current supervisory scope needs care. The Federal Reserve’s 2026 guidance excludes generative and agentic AI from its formal model definition. It still points banks toward appropriate governance for tools outside that definition. The current PRA SS1/23 uses its own scope and proportionality. A global institution should record its classification by legal entity. It should not assume that one inventory label answers every jurisdiction.
Treat change as a portfolio of evidence impacts
Teams lose time when every change receives the same review. They also create risk when apparently small changes bypass review. A better method classifies the evidence impact rather than the file type.
A change is material when it can alter a release claim, a control boundary or an affected-party outcome. A provider version change may be material. A prompt edit may be material. A front-end label may be material if it changes how people interpret confidence. A code refactor may be non-material when behaviour and evidence remain unchanged.
The change record should show the old and new state, impacted claims, selected test packs, result identifiers and approval route. A phrase such as “prompt-only change” describes implementation, not risk. The deciding question is whether the change can alter behaviour or the meaning people take from it.
Release packs should be modular. Retrieval changes should call retrieval, grounding and stale-source tests. Tool changes should call contract, permission and rollback tests. Model changes should call task, calibration, safety and cohort regressions. Interface changes should call oversight and accessibility tests. This mapping reduces review effort while keeping the decision defensible.
Place third-party services inside the operating boundary
Using a hosted model does not automatically make a system unsuitable. It does create dependencies that need explicit treatment. The boundary includes the provider’s service, region, retention terms, subprocessors, version policy, outage behaviour and exit route.
The European Union’s Digital Operational Resilience Act gives financial entities a direct framework for ICT risk and third-party dependencies. It should be read with the institution’s sector and jurisdiction requirements. NIST SP 800-218A adds secure-development practices for generative AI and foundation-model systems.
Due diligence should connect to runtime enforcement. If only de-identified data is approved, the input path needs a tested de-identification control. If one region is approved, routing should enforce that region. If data retention must be disabled, the configuration and contract need aligned evidence. If a provider becomes unavailable, the system should narrow, queue or stop according to consequence.
Exit planning belongs in the first release, not the renewal discussion. Preserve provider-neutral evaluation cases. Keep prompts and system policies under institutional version control. Define how logs, embeddings and fine-tuned artefacts are returned or destroyed. Test one migration path before dependency becomes urgent. An exit plan that has never been exercised is a negotiation position, not an operational control.
Design the incident record before the incident
Probabilistic systems can fail without producing an exception. The incident path must therefore begin with behavioural signals as well as service errors. A policy breach, unsupported claim, unusual tool sequence or missed escalation may all open an investigation.
The first action should preserve evidence. Capture the system manifest, input and authorised trace, retrieved source identifiers, tool calls, policy decisions, reviewer actions and external effects. Apply the institution’s privacy and retention rules. Avoid copying sensitive material into an unrestricted incident document.
Containment should be granular. Disable one tool, one route or one case class when possible. Broader shutdown remains appropriate when the boundary is unknown. The safest containment action is the narrowest one whose effectiveness can be demonstrated.
Closure requires more than a corrected prompt. Add a regression case that reproduces the failure. Identify why earlier evaluation or monitoring missed it. Update the relevant threshold, control or operating instruction. Verify the change on the complete system. Link the closure decision to the new evidence. This turns an incident into an improvement of the release case rather than a temporary patch.
A practical evidence programme
The first programme milestone should be an agreed evidence map, not a polished demonstration. Bring the business owner, engineering lead, operations owner, security, privacy, compliance and validation together around one workflow. Walk through the decision and its possible effects. Record disagreements about scope rather than hiding them in meeting notes.
Next, establish a deterministic baseline. Measure the current process on the same cases the proposed system will face. Capture quality, time, escalation and known failure. The baseline prevents the model from being compared with an imaginary perfect process. It also identifies steps that should remain rules or standard software.
Build the system manifest and test harness alongside the first vertical slice. The slice should include retrieval, one bounded output, one realistic review path and a safe failure state. Use approved or synthetic data. Preserve traces from the beginning. Instrumentation added after the pilot will miss the decisions needed for validation.
Run risk discovery before broad user testing. Threat-model indirect prompts, excess permission, data leakage, stale sources, unsupported claims and partial tool failure. Translate each material scenario into a control and an evaluation case. Assign an owner for cases that cannot yet be automated.
Then test the human system. Give representative reviewers realistic work. Observe whether they inspect evidence, understand uncertainty and use escalation correctly. Measure queue arrival and service time. Confirm that the review route still works during absence, peak demand and dependency outage.
Prepare the release meeting around decisions. Show intended use, exclusions, residual risks, failed cases, open limitations, fallback and change policy. Avoid a tour of architecture slides. Reviewers should know exactly which claim they are accepting and which evidence supports it.
After release, compare production with the approved envelope. Sample outcomes. Review drift and access events. Exercise recovery. Feed defects into stable tests. The programme is mature when each material change can reuse this evidence machinery without weakening scrutiny.
The release decision, made inspectable
A regulated release needs more than a successful benchmark run. It needs a chain of evidence that another team can reproduce. The chain starts with an approved use case. It ends with a named owner accepting a bounded residual risk. Each link should have a durable artefact.
The loop matters. A release is not a certificate that remains true. It is a claim tied to a model version, prompts, tools, policies, data boundaries and operating conditions. Change any material element and the evidence must be reconsidered.
| Release question | Minimum evidence | Accountable role | Stop condition |
|---|---|---|---|
| Is the use permitted? | Purpose, affected users, legal basis and prohibited uses | Business owner | Purpose is vague or broader than testing |
| Does the system work? | Golden cases, edge cases and failure analysis | Product and evaluation leads | Critical cohort misses its threshold |
| Can it cause harm? | Threat model, abuse tests and tool-boundary tests | Security lead | Unbounded write or disclosure path |
| Can people intervene? | Escalation tests, queue capacity and override evidence | Operations owner | Human route is nominal but unusable |
| Can the release be reconstructed? | Version manifest, approvals and evaluation snapshot | Release owner | Evidence cannot be tied to the deployed build |
Control strength should follow consequence
Not every model-assisted step needs a committee. Every step does need a stated consequence class. Classification prevents teams from applying expensive controls everywhere while missing the few actions that create real exposure.
| Consequence class | Typical example | Appropriate control | Production evidence |
|---|---|---|---|
| Informational | Summarise an internal policy | Citation and sampled review | Grounding and omission results |
| Reversible | Draft a case note | Undo window and audit log | Rollback test and reviewer sampling |
| Material | Recommend a customer outcome | Mandatory competent review | Override quality and cohort performance |
| Irreversible | Execute payment or legal notice | Deterministic control or dual authorisation | End-to-end control test |
The human step must be operationally real. Measure queue age, evidence engagement, disagreement and escalation quality. A button labelled “approve” does not establish oversight. The EU AI Act makes effective human oversight an explicit concern for high-risk systems. The ICO guidance on AI explanations also treats context and affected audiences as central.
For the wider control system, the NIST AI Risk Management Framework supplies a lifecycle vocabulary. Its Generative AI Profile adds risks specific to generative systems. Security work can be joined to the NIST Secure Software Development Framework. These sources do not replace sector rules. They help teams connect risk, evidence, ownership and change control without pretending one benchmark settles the release.