Executive briefThe committee approved three different releases
TLDR
- A release pack is a signed manifest for one capability, build, use envelope, authority, population, environment and exposure.
- Every evidence item needs scope, method, result, limitation, provenance, freshness, owner and invalidation metadata.
- A Release Delta Graph proves which evidence remains valid, which must be refreshed and what a silent provider change invalidates.
- Hashes and signatures establish integrity, not behavioural fitness; accountable challengers still judge relevance, representativeness and residual risk.
- Runtime evidence can invalidate a release before its scheduled review and must connect directly to containment, reconciliation, remedy and re-release.
The approval slide named release 7.4. The evaluation notebook tested a managed endpoint whose provider alias had already moved. The deployment ticket referred to prompt policy 7.3, while the production receipt recorded a new retrieval index and tool permission. Every document existed. None proved the system that entered production. The committee had approved three incompatible release identities and operations had deployed a fourth.
An evidence pack is not a folder of artefacts and not the assurance argument itself. Paper 106 defined the claim, argument, evidence and defeater logic of an AI assurance case. This paper deals with the release instantiation: the versioned artefacts, decisions, integrity links, open findings, exposure conditions and operational receipts that make one bounded claim actionable. Policy compilations, regulatory crosswalks, model-risk tests, human-oversight evidence and decision-observability hooks may enter the pack, but the pack does not recreate those systems.
Completeness is a weak property. A generic supplier report may be present but cover another region. A benchmark may have passed but omit the critical population. A rollback command may work while completed external actions remain unreconciled. A training record may be current while queue volume has doubled. A pack with every required heading can still be unfit for release because the evidence is stale, irrelevant, contradictory or attached to the wrong artifact.
This paper proposes the fourteen-section AI Release Evidence Manifest, an Evidence Descriptor for every item, a Release Delta Graph and a Pack Integrity Rule. Structural checks should be machine-verifiable. Sufficiency remains a challengeable judgement about whether the evidence supports the stated release claim within its use envelope and expiry conditions.
The strongest rival: a lightweight release record
A low-risk wording change inside a reversible drafting tool may need automated regression, version control and a concise delta record rather than a fourteen-section dossier. Heavy packs for every experiment encourage copied evidence, false precision and delayed learning. Evidence depth, independence and release authority should scale with consequence, change novelty, exposure and recoverability.
Proportionality does not mean omission by convenience. The lightweight path still names the artifact, use envelope, predecessor, tests, owner, deployment receipt and invalidation trigger. Reuse is legitimate only when version, population, environment, method, provider scope and freshness carry forward. The choice is between proportionate proof and disproportionate paperwork, not between governance and speed.
| Shortcut | Why it fails | Decision-relevant replacement |
|---|---|---|
| All required files are attached | Files may concern another version, scope or population | Evidence Descriptors resolve each item to the release claim |
| The supplier supplied assurance | Generic provider evidence may omit endpoint, region and configuration | Match provider scope and supplement use-specific evidence |
| The benchmark passed | Average performance can hide critical failures and subgroup regression | Production-like cases, baselines, uncertainty and controlling failures |
| Rollback is documented | Code restoration may not reverse external actions or state | Rehearsed rollback, reconciliation, affected-case trace and redress |
| Training is complete | Attendance does not prove intervention under real workload | Representative competence, queue capacity and exercised escalation |
AI Release Evidence ManifestFourteen sections bind proof to one decision
1. Release identity
Record the capability, build, model or managed endpoint, prompt, policy, retrieval corpus, tools, configuration, region and dependency versions. Include immutable identifiers where available and state which elements remain externally managed.
2. Use envelope
Name intended and prohibited uses, populations, decision authority, environment, exposure, completion condition, reversibility and review date. The envelope prevents evidence for a bounded drafting aid from licensing an external-action system.
3. Decision record
Identify release authority, consequence-recovery tier, top assurance claim, decision, conditions, dissent and accepted residual risk. Conditions must be executable constraints or monitored obligations, not aspirations hidden in minutes.
4. System and provider map
Show components, data flows, human roles, direct providers, material fourth parties and shared cloud, identity, model or data dependencies. A provider name is not a sufficient description of an operating dependency.
5. Artifact provenance
Link sources, builds, dependencies, integrity digests, signatures, software and model materials, attestation subjects and reproducibility limits. Provenance establishes origin and process; it does not prove behaviour or business suitability.
6. Data and context evidence
State provenance, rights, legal basis where applicable, quality, coverage, representativeness, freshness, entitlements, retention and deletion. Separate training data, evaluation data, retrieval content and live user context.
7. Model and behaviour evidence
Pin model role, configuration, prompts, policies, structured constraints, abstention, limitations and change history. A family label or provider alias does not establish the actual model serving the release.
8. Evaluation evidence
Include production-like cases, critical failures, relevant subgroups, adversarial tests, baselines, uncertainty, statistical limits and unresolved findings. Average success never cancels a release-blocking failure in a consequential pathway.
9. Security and misuse evidence
Connect the threat model to abuse cases, prompt and context attacks, authority, tool calls, supply chain, secrets, identity, isolation and containment. Test the deployed configuration, not a model endpoint in isolation.
10. Human and organisational readiness
Name owners, competence, workload, intervention point, authority, runbooks, support and escalation capacity. Demonstrate behaviour in representative cases rather than equating training attendance with readiness.
11. Operational readiness
Record capacity, latency, cost, telemetry, SLOs, route and decision receipts, alerts, incident process and evidence retention. Monitoring must expose conditions that invalidate the release claim, not only service availability.
12. Recovery and continuity
Define safe state, rollback, state reconciliation, affected-case identification, correction, redress, fallback and provider exit. Recovery proof includes committed external effects, not merely redeploying old code.
13. Open items and exceptions
For every gap, record consequence, compensating control, affected population, owner, deadline, approval, expiry and revocation condition. A finding remains open even when the file containing it is marked complete.
14. Release lineage
Link the predecessor, evidence delta, model and policy changes, approval, deployment receipt, canary exposure and next revalidation trigger. Lineage distinguishes a new release from an unauthorised mutation of the last one.
The manifest should be machine-readable enough to reject missing identifiers, broken hashes, expired evidence and unowned exceptions before an approval meeting. NIST OSCAL demonstrates how control implementations, assessments, results and plans of action can be represented in structured formats. SPDX 3.0 and CycloneDX 1.7 can describe software, model, data, service and dependency materials. SLSA 1.2 and in-toto attestations can strengthen build and subject provenance. None of these formats decides whether the evidence is relevant to the release claim.
Keep negative evidence in the same manifest. Failed adversarial cases, unresolved subgroup regressions, incident observations and challenger dissent are not embarrassing attachments to be buried. They determine conditions, exposure, exception expiry or rejection. A release record that retains only passed tests corrupts the decision context and makes later revalidation unreliable.
Evidence DescriptorMake every item answerable and invalidatable
An item begins with an identifier, title and content hash. It names its producer, accountable owner and independent reviewer. It declares system, version, population, environment and time scope, then links to the exact claim or release condition supported. Method, result, limitations, provenance, access, retention, freshness, invalidation trigger, unresolved finding and disposition complete the record.
The descriptor prevents a familiar substitution. A penetration report may prove a particular test method against one build. It does not establish evaluation quality, human readiness or the safety of a later tool permission. A model card can explain intended use and limitations, and a dataset datasheet can describe motivation, composition and maintenance. Both are valuable evidence inputs. Neither becomes release proof until scope and method match the deployed system and decision.
Independence is also item-specific. Automated regression can be produced by the delivery team for a low-risk prompt change. A high-consequence change may require a separate validation function to challenge the test design and disposition. Independence does not mean ignorance of the system. It means sufficient organisational distance, competence, access and authority to question the producer's claim without release pressure deciding the answer.
| Descriptor field | Challenge question | Invalidation example |
|---|---|---|
| Version and scope | Does this item concern the deployed model, policy, region and population? | Provider alias changes its underlying model |
| Method and result | Was the method capable of observing the controlling failure? | Average benchmark omits high-consequence cases |
| Limitations | What was not tested, observed or inferred? | Canary labels are delayed and coverage is partial |
| Freshness | Which environmental or operational changes expire the evidence? | Volume doubles and intervention queues breach the window |
| Finding and disposition | Does the disposition constrain release, exposure or authority? | Open subgroup regression lacks an owner or expiry |
Release Delta GraphUnchanged files do not prove unchanged behaviour
Every candidate begins from an approved predecessor or explicitly declares that no reliable predecessor exists. For each manifest section, the graph labels evidence as reused, refreshed, added, retired or invalidated. Each edge carries a reason, approver and scope. Reuse means the receiving claim, artifact, population, environment, provider scope, method, freshness and retention requirement still match. Similar filenames and unchanged hashes are not enough.
A prompt wording change may reuse system mapping, data rights, threat model and recovery evidence if those dependencies did not move and their validity periods remain open. The regression test and prompt record are refreshed, the changed prompt is added, and the predecessor prompt is retired. That is valid evidence reuse. The delta stays lightweight because the change and consequence are bounded.
A managed-model alias is the opposite case. Source code and endpoint name may be unchanged while the provider changes the underlying model, safety configuration or routing. Model and behaviour evidence becomes invalid. Evaluation, security, human-readiness, cost, latency and possibly data-handling evidence require review. The graph turns an apparently empty code diff into a material release delta.
Retrieval changes propagate differently. A corpus update can alter coverage, freshness, entitlements, provenance, deletion, prompt-injection exposure and population outcomes. New tool authority expands the system from recommendation to external commitment, invalidating the previous use envelope, threat model, human intervention design and recovery proof. The delta graph should traverse semantic and authority dependencies, not only technical files.
| Gate | Reuse is valid when | Refresh or invalidate when |
|---|---|---|
| Release identity | Artifact and managed dependencies resolve identically | Alias, prompt, policy, tool, region or configuration changes |
| Population and environment | Exposure, users, data and operating context remain in scope | New population, channel, language, volume or decision authority |
| Method relevance | The method still observes the controlling failure | Failure path, interface, baseline or acceptance criterion changes |
| Freshness and provider scope | Validity window is open and provider evidence matches the service | Evidence expires, provider changes or external conditions move |
| Operational outcome | Runtime observations support the same claim and detection remains adequate | Incident, alert, drift, queue or customer challenge contradicts it |
Integrity and decision semanticsMatching hashes are necessary and insufficient
The integrity chain starts with subjects that can be resolved. A content digest can identify a prompt file, model artefact, policy bundle or retrieval snapshot. SLSA provenance can describe a build and its source. An in-toto statement can bind a predicate to a subject. SPDX and CycloneDX can inventory packages, models, data, services and dependencies. Signatures establish who attested to what. The manifest records which attestation the release authority relied upon.
Then the deployment process emits a receipt containing the approved release identifier, actual artifact and dependency identifiers, policy, region, exposure state and time. Runtime route and decision receipts show what was actually invoked. The gate compares them. If the committee approved model build A but the gateway routed to model B, the pack fails even when both builds have individually valid provenance.
Integrity does not imply fitness. A perfectly signed evaluation can use the wrong population. A complete bill of materials can omit a behavioural regression. A valid supplier report can be irrelevant to the deployed endpoint. Structural verification should prevent substitution and missing metadata, freeing human challengers to test the harder questions of applicability, method, inference and residual risk.
Exception handling belongs inside the signed decision. Every open item states affected population, consequence, compensating control, exposure limit, owner, deadline, expiry and automatic revocation condition. The release authority accepts or rejects that bounded exception, and an independent challenger records disagreement. Moving the exception to a ticket after approval breaks the decision context.
Proportionate evidenceScale depth by consequence, change and exposure
Consequence and recoverability set the initial evidence depth. Change novelty determines which sections must be challenged again. Exposure controls how much uncertainty can be accepted while runtime labels accumulate. Independence rises when a failure can create material, difficult-to-reverse harm or when the producer has strong delivery incentives.
A canary is a decision state, not a softer word for production. It has an eligible population, maximum volume, restricted authority, duration, stop condition, rollback method, label plan and named watcher. Strong pre-production evaluation can justify limited exposure while delayed outcome labels arrive. It cannot justify unlimited authority on the assumption that monitoring will eventually discover harm.
Automated structural validation should run before human review. It checks required identifiers, referential integrity, signatures, hashes, allowed evidence states, expiry, ownership, exception deadlines and deployment resolution. Humans examine claim relevance, evaluation representativeness, unresolved findings, inference limits, provider scope, operational readiness and the acceptability of residual risk. Automating syntax should increase time for substantive challenge.
The April 2026 US interagency model-risk guidance supersedes the 2011 banking guidance. Its stated scope excludes generative and agentic AI, although banking organisations still need governance and controls for tools outside that guidance. It can inform evidence practices for covered traditional and non-generative models, but should not be cited as direct supervisory coverage of a generative release. The NIST AI RMF 1.0 is also under revision; a pack should record the framework version actually used rather than anticipating future content.
Runtime invalidationThe pack remains alive after deployment
Release evidence expires by condition as well as calendar. A provider model change, retrieval mutation, new tool, material policy update, population shift, queue breach, critical incident, detection failure, subgroup regression or external obligation can invalidate one or more manifest sections immediately. Annual review dates are backstops, not permission to retain contradicted evidence.
OpenTelemetry 1.60.0 offers vendor-neutral tracing, metrics and logs, while W3C PROV-O provides a vocabulary for entities, activities, agents and derivation. Those specifications help capture runtime and lineage evidence. The manifest still defines which release conditions the telemetry observes, the schema version, coverage, access, retention and limitations. Raw telemetry volume is not release assurance.
An invalidation event follows a controlled loop: identify the affected claim and manifest sections; restrict exposure or contain the capability; preserve decision and deployment evidence; reconcile in-flight and completed effects; identify affected people or cases; correct and provide redress where needed; collect targeted evidence; challenge the revised claim; and issue a new release decision or withdraw. Reusing the original approval after a controlling assumption has failed is not revalidation.
Rollback has two layers. Technical rollback restores a prior model, code, prompt, policy, corpus or route. Business reconciliation determines which external actions completed, which state diverged, which users were affected and what correction is authorised. A procurement agent that placed orders cannot be recovered by redeploying yesterday's container alone.
Ten worked casesDecide with delta, descriptor and identity evidence
Each case names the affected manifest sections, controlling descriptor, delta, invalidation trigger, decision and exposure. The cases demonstrate why evidence count cannot decide release quality.
| Case and affected sections | Descriptor and release delta | Decision, exposure and trigger |
|---|---|---|
| 1. Low-risk prompt wording. Identity, behaviour, evaluation and lineage change inside a reversible drafting tool. | Prompt hash, owner, regression method and limitations are refreshed. System map, rights and recovery evidence are reused because scope and validity match. | Approve within the existing envelope. Trigger revalidation on new authority, population, failure or provider change. This is valid reuse. |
| 2. Silent managed-model alias. Identity, provider map, behaviour, evaluation, security, operations and lineage are affected without a source-code diff. | Provider notice resolves the new underlying model. Prior behavioural evidence is invalidated; provenance and dependency evidence are refreshed. | Hold for targeted evaluation and threat evidence. No exposure until endpoint identity and critical cases are resolved. |
| 3. Retrieval corpus update. Data, context, evaluation, security, operations and lineage move. | Corpus snapshot records rights, provenance, entitlements, freshness, deletion and population coverage. Prompt-injection and retrieval-quality tests are added. | Approve a bounded canary for authorised users. Stop on entitlement leakage, missing-source rate or critical coverage failure. |
| 4. New external-action tool. Use envelope, system map, security, human readiness, recovery and decision authority change. | Tool contract, identity, cumulative authority, transaction boundary, intervention and reconciliation rehearsal are new evidence. Recommendation-only evidence cannot carry forward. | Repair control and recovery readiness before any external commitment. Sandbox or shadow operation only. |
| 5. Better average, worse subgroup. Evaluation, decision record, open items and use envelope are controlling. | Evaluation descriptor shows higher aggregate performance and a critical subgroup regression with adequate sample and a credible baseline. The finding remains open. | Reject the broad release. Consider a narrowed envelope only if routing and monitoring prevent exposure of the affected group. |
| 6. Canary with delayed labels. Evaluation, operations, open items and lineage matter. | Pre-production evidence is strong, but runtime outcome coverage is limited and delayed. The descriptor states the inference limit and label arrival plan. | Approve limited exposure with a volume ceiling, restricted authority, watcher, duration and stop condition. No automatic scale-up. |
| 7. Supplier report mismatch. Provider, provenance, security and decision sections are affected. | The report is signed and current but covers another region and endpoint. Its descriptor marks scope mismatch and invalid reuse. | Hold for use-relevant supplier or customer-verifiable evidence. Generic assurance cannot satisfy the condition. |
| 8. Human readiness expires. Human readiness, operations, evaluation and open items move after case volume doubles. | Old trials assumed a shorter queue. Runtime evidence shows intervention-window breaches and inaccessible escalation under surge. | Contain exposure and repair workload, routing or authority. Training attendance does not restore readiness. |
| 9. Broken rollback rehearsal. Recovery, operational readiness and lineage are controlling. | Code restores successfully, but completed payments cannot be reconciled and affected customers cannot be identified. Recovery evidence records the failed test. | Repair reconciliation, action receipts and redress before release. A rollback command is not a recovery plan. |
| 10. Structurally complete mismatch. Identity, integrity, decision and lineage all fail. | Every section exists, yet hashes identify build A, approval names endpoint B and deployment receipt shows policy C. The Pack Integrity Rule fails. | Reject or withdraw immediately, preserve evidence and reconcile effects. Structural completeness cannot cure substitution. |
Interactive instrumentAI Release Evidence Gate
Load a named case or select the closest release conditions. The result is a decision route with a controlling item, not a completeness percentage, document count or automatic approval.
Approve the release within its proven envelope
Resolved release identity: artifact, manifest, approval, deployment and runtime receipt identify the same bounded release.
Affected manifest sections: identity, behaviour, evaluation and lineage.
Controlling item: regression evidence supports the stated low-consequence change.
Evidence reuse: unchanged scoped evidence remains valid; prompt and regression artefacts are refreshed.
Release Delta Graph: predecessor edges record reused, refreshed and retired evidence with reasons.
Required action: retain automated regression, deployment receipt and challengeable limitations.
Exposure and authority: remain inside the approved reversible envelope.
Rollback and invalidation: restore the predecessor and revalidate on provider, authority, population or critical-failure change.
Exception and challenge: no controlling exception; release authority and independent challenger sign the manifest.
Executive cadenceMake release authority inspect identity, delta and expiry
At design, the capability owner defines the use envelope and top assurance claim. Engineering, data, product, security, risk, human-factors, operations and legal teams identify manifest sections, descriptors and evidence producers. The release authority names the consequence-recovery tier and challenger before evidence is generated.
Before challenge, automation validates identities, hashes, signatures, lineage, required descriptor fields, expiry and exception ownership. The producer submits the Release Delta Graph and explains every reused, refreshed, added, retired and invalidated item. The challenger tests scope, method, representativeness, inference and contradictions.
At decision, the authority chooses approval, limited exposure, targeted hold, repair or rejection. Conditions state population, authority, volume, duration, monitoring, rollback, invalidation and exception expiry. Dissent and residual risk stay inside the signed manifest. Deployment is permitted only for the resolved release identity.
During operation, route, action and decision receipts connect outcomes to the release. Alerts target conditions that can invalidate claims. A material contradiction triggers exposure reduction, containment, evidence preservation, reconciliation, correction, redress and a new release decision.
On retirement, the owner preserves the decision record and required evidence, closes or transfers open cases, reconciles state, confirms deletion duties, withdraws authority and records the final lineage edge. The pack explains both why the release entered production and why it ceased to be valid.
Primary evidenceReferences
Links and live status were checked on 2 September 2026. Standards and guidance are applied within their stated scope and pinned version; legal obligations remain subject to applicability and effective time.
- NIST, Artificial Intelligence Risk Management Framework.
- NIST, Generative Artificial Intelligence Profile.
- NIST, Open Security Controls Assessment Language.
- NIST SP 800-218, Secure Software Development Framework Version 1.1.
- NIST SP 800-53A Rev. 5, Assessing Security and Privacy Controls, including Release 5.2.0.
- ISO, ISO/IEC 42001:2023 Artificial intelligence management system.
- European Union, Regulation (EU) 2024/1689, Artificial Intelligence Act.
- Federal Reserve, OCC and FDIC, SR 26-2 Revised Guidance on Model Risk Management.
- SLSA, approved Specification Version 1.2.
- in-toto, Attestation Framework.
- SPDX, Specification Version 3.0.
- CycloneDX, Specification Version 1.7.
- Mitchell et al., Model Cards for Model Reporting.
- Gebru et al., Datasheets for Datasets.
- OpenTelemetry Specification 1.60.0.
- W3C, PROV-O: The PROV Ontology.
- US FDA, Marketing Submission Recommendations for a Predetermined Change Control Plan for Artificial Intelligence-Enabled Device Software Functions, final August 2025.
About this paperA release decision, not a document score
The AI Release Evidence Manifest, Evidence Descriptor, Release Delta Graph, Pack Integrity Rule and AI Release Evidence Gate are this article's synthesis. They package one bounded release case without treating structural validity, document count, signatures, benchmarks, supplier reports or completeness as proof of production fitness.