Part one

The model is no longer the release unit

Published evidence MLOps arose because ordinary application delivery could not account for data-dependent behaviour, training pipelines, model validation and post-release drift. Sculley and colleagues described production ML as an entangled system in which a change to data, features, learning settings or thresholds can alter the behaviour of the whole. Later work made data validation a first-class production concern and proposed concrete tests for models, data and infrastructure. Continuous training platforms then connected those checks to repeatable pipelines. These foundations remain necessary.

They are no longer sufficient as the centre of gravity. A contemporary application may call a hosted foundation model, retrieve evidence from an index, transform that evidence through a prompt, invoke tools through schemas, consult an authorisation policy, write working state, branch through a workflow and present an answer through an interface that changes how a person relies on it. The Berkeley formulation of a compound AI system is useful because it moves attention from a single learned component to several interacting components. Operations must make the same move.

Here B is observable system behaviour. M is the model route and its decoding configuration; P the assembled prompt; D the data and evidence snapshot; R retrieval and reranking; K deterministic calculations and guardrails; T tool contracts and implementations; A identity and authority policy; W workflow control; S persisted state or memory schema; and X the execution environment. The equation is a dependency statement, not a claim that these terms combine linearly. Changing one term can alter the output distribution, the permitted action set, or both.

Prompt effects deserve careful language. Studies have found material performance variation under changes in example order or formatting, while newer work shows that some reported sensitivity came from rigid evaluation methods and that modern models can be more robust than earlier results suggested. The operational conclusion does not require the strongest sensitivity claim. A prompt is behaviourally relevant whenever changing it could alter an approved claim, output contract or action boundary. That possibility is enough to require identity and testing.

The same discipline applies to retrieval. RAG is not one switch. Query rewriting, chunking, embedding, candidate selection, reranking, context ordering and evidence freshness can each change which facts become available to generation. Empirical work comparing RAG choices supports treating these stages as design variables. A release record that says only “RAG enabled” cannot reconstruct the evidence path.

This negative control matters because “version everything” is not a design rule. Logs, dashboards, source documents, feature flags and policy files differ in causal role. The right unit is discovered by tracing what can change served behaviour and what evidence is needed to defend that behaviour. Repository boundaries, team structures and vendor consoles are weak proxies for that causal boundary.

Part two

flowchart LR
  A["The release that nobody could<br/>reproduce"] --> B["On this page"]
  B --> C["The model is no longer the<br/>release unit"]
  C --> D["Release closure, what must<br/>move together"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#007D68,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Release closure: what must move together

Practitioner synthesis A release closure is the smallest set of resolved artefacts and contracts sufficient to reproduce the system behaviour being approved. “Resolved” matters. A branch name, a floating model alias, a mutable index pointer or “latest policy” is a request to discover an artefact later. It is not the identity of the artefact that was evaluated.

For release r, begin at the runtime workflow root. Follow every declared dependency that can alter an approved claim, a control decision or an external outcome. The resulting set C(r) is the behaviour-critical closure. The definition includes transitive dependencies: if a retriever depends on an evidence snapshot and an embedding configuration, those are inside the closure even when the workflow calls only the retriever API.

What belongs inside

The model entry should name the served route, region where relevant, decoding parameters and any routing logic. For self-hosted weights, an artefact digest can identify the weights and runtime image. For a hosted service whose internals are unavailable, the digest can identify the institution’s route configuration and contract, while the manifest records the provider’s model selector and observed service metadata. It must not pretend to possess a weight hash it cannot verify.

The prompt entry should cover the assembled prompt, not merely a template file. System instructions, selected few-shot examples, output schema, prompt compiler and feature flags may jointly determine what the model receives. The retrieval entry should resolve query transformations, embedding and reranking configurations, index build recipe, evidence snapshot or temporal cut, and access-filter policy. The tool entry needs both interface contract and implementation revision because a schema can remain constant while side effects change.

Authority belongs in the closure when it changes what the system may propose or execute. A policy bundle can be independently versioned and signed. Open Policy Agent, for example, supports bundle revisions, signatures and decision logs carrying bundle metadata. This is an implementation example, not a universal requirement. The general requirement is to identify the policy that authorised a decision and to verify that the runtime actually enforced it.

Workflow logic, retry policy, timeouts, compensation rules and idempotency keys are behaviour. So are memory and state schemas when they determine what prior information is retrieved, promoted or forgotten. Persistent business world state itself should usually remain an external, time-varying input, referenced by event or snapshot identifiers rather than packaged into a release. The manifest describes how state is interpreted and which evidence version was visible; it does not freeze the live world.

A logical bundle is not a distributed monolith

A common objection is that one release identity forces every component to deploy together. It should not. A prompt can be promoted without rebuilding a retriever container. A policy bundle can update more frequently than a workflow. A hosted model can be routed independently. Release integrity means atomic resolution, not atomic deployment. At any served moment, the control plane must resolve those independently deployed components to one approved manifest. Mixed states may be used deliberately for canaries, but each mixture needs its own candidate identity and evidence.

This distinction preserves both velocity and reconstructability. Teams keep their repositories and ownership. The release registry becomes a joining layer that records compatible revisions. Consumers never ask for “whatever is current” at runtime; they ask for the revisions named by a release. If an asset cannot be pinned, the manifest states the limitation and the operation compensates with contract monitoring, probe cases and an observed runtime receipt.

Three identities prevent false reproducibility

Teams often compress three different questions into one version string. The first is source identity: which code, prompt text, policy source or index recipe was approved? The second is materialised identity: which container, compiled policy bundle, rendered prompt, vector index or schema artefact was produced? The third is served identity: which resolved artefacts and external services handled a particular request? A Git commit can answer the first question while failing to answer the other two.

The distinction is clearest with prompts. Source control may contain a template with placeholders. A prompt compiler selects examples, injects output schemas and assembles policy instructions. The materialised prompt is the exact byte sequence or structured message list sent to the model. The served receipt also needs the selected model route and any provider-side behaviour identifier available under the contract. Storing only the template commit proves that one input to assembly was known; it does not prove what the model received.

Retrieval has the same layers. An index recipe can be versioned while the materialised index changes because the source snapshot, parser or embedding implementation changed. An apparently immutable index can still serve different results if access filters, query rewriting or reranking float. The release closure should therefore bind recipe, snapshot, materialised index and serving configuration at the granularity needed to reproduce the evidence set. Hashes are useful only when the hashed object is the one that mattered.

Hosted services make perfect reconstruction impossible in some cases. The platform should state that limitation precisely. It can pin the local route, parameters, region, contract version and probe suite; record any provider-supplied revision metadata; and preserve sampled outputs for approved synthetic canaries. This creates bounded reproducibility: the organisation can reconstruct everything it controlled and demonstrate what it observed at service time. It should not claim identical reruns when an external provider does not offer that guarantee.

Part three

Bind change to evidence

A manifest is useful only when it changes promotion decisions. A catalogue of component versions can aid inventory, yet still allow a release whose parts disagree or whose tests apply to another combination. The stronger object binds five things: identity, compatibility, claims, evidence and recovery.

1. Identity: resolve every mutable name

Each behaviour-critical artefact needs a stable identifier, type, immutable reference or digest, and declared dependencies. Software supply-chain systems provide useful precedents. SLSA provenance records build definitions, parameters and resolved dependencies; in-toto’s release attestation links a release name and version to artefact hashes. A compound-system manifest extends this pattern to runtime artefacts that are not traditional build outputs, such as prompts, policy bundles and evidence snapshots.

The analogy has a boundary. SLSA explicitly focuses on build provenance, and runtime dependencies are outside parts of its threat model. A compound release therefore needs additional semantics: which prompt consumes which evidence schema, which workflow expects which tool contract, and which evaluation claims apply to the assembled system. Borrow the integrity machinery without claiming that software provenance alone proves behavioural quality.

The manifest itself needs a trust model

A release manifest assembled on a developer laptop is useful documentation, but weak evidence. The trusted path should resolve artefacts directly from approved registries, compute the behaviour projection and bundle digest, verify signatures or provenance where available, and emit the candidate manifest from a controlled release service. The person requesting promotion may select inputs; they should not be able to rewrite the resulting digests, test outcomes or approval identities after the fact.

That service also needs a rule for external parameters. A tenant-controlled feature flag, environment variable or prompt override can change behaviour without changing a named artefact. Either promote it into the manifest as an explicit parameter, constrain it to a declared non-behavioural field, or reject the candidate. “Configuration lives elsewhere” is an undeclared dependency. SLSA’s separation between external parameters, resolved dependencies and trusted build metadata provides a useful pattern even though the object here is a runtime system release.

Secrets should not be copied into the manifest. Record the secret reference, expected scope and rotation policy, then let runtime identity retrieve the value from an authorised store. The release claim concerns which capability and endpoint the secret enables, not the secret bytes. Similarly, personal data and complete prompts should not be embedded merely to improve reproducibility. Preserve privacy-controlled evidence identifiers and replay fixtures built from synthetic or properly governed cases.

Finally, a manifest must be verifiable at consumption time. A deployment controller can reject a candidate whose signature, digest or compatibility checks fail. A runtime admission check can ensure that the resolved prompt, policy and workflow revisions match the approved release. An incident tool can fetch the same manifest by release ID without relying on a mutable wiki page. Provenance becomes control only when a consumer checks it.

2. Compatibility: reject combinations before evaluation

Compatibility checks are cheaper than end-to-end evaluation and catch a different class of defect. A prompt may require evidence objects with source, valid-time and supersession fields. A retriever may emit only source and score. A workflow may expect tool contract v2 while the endpoint serves v3. A memory reader may interpret a status value removed by the new schema. These are deterministic mismatches and should fail assembly before a model is called.

3. Claims and evidence: test the bundle you promote

A release claim might state that answers cite currently effective evidence, that no write action occurs without approval, or that a fallback activates within a defined timeout. Each claim needs an owner, test pack, threshold, result and tested bundle digest. The digest closes a common loophole: a team runs an evaluation, edits a prompt, and promotes the edited system under the old result. Once a behaviour-critical field changes, the computed digest changes and prior evidence no longer matches.

This does not mean rerunning every test after every change. A change-impact cone maps the changed artefact to the claims it can affect, then to their test packs, approval routes and operating probes. A retrieval snapshot change may select evidence recall, supersession, grounding and citation tests. A tool-contract change selects schema, permission, idempotency and unknown-outcome tests. A broad model-route change can widen the cone to task quality, calibration, safety, format and tool-selection suites.

4. Runtime receipt: join the trace to the release

Promotion establishes what may run. A runtime receipt establishes what did run. Every request trace should carry a low-cardinality release identifier and link to the manifest, policy decision, evidence identifiers and tool effect receipts. OpenTelemetry traces provide the distributed path, and its generative-AI attribute registry defines a data-source identifier for agents and retrieval applications. A project-specific release attribute can join those spans to the registry. Keep the manifest identifier in telemetry, not the prompt or sensitive case content.

Context propagation needs restraint. OpenTelemetry warns that baggage can cross service boundaries, lacks built-in integrity checks and may expose sensitive values. A release ID is usually safe as an allow-listed internal attribute, but it should be set or verified by trusted runtime infrastructure rather than accepted from an untrusted caller. Evidence and effect details belong in governed stores linked by identifiers.

Part four

Promote, observe and recover the released system

A bundle digest solves identity, not reliability. Promotion still needs staged exposure and evidence suited to the failure model. The candidate is first assembled and checked for closure, immutable references and contract compatibility. It then runs offline evaluation against the selected claim packs. Shadow execution can expose production-shaped inputs without serving candidate outputs. A canary limits the population exposed to a candidate while comparing it with the control. Progressive ramp widens exposure only while stop conditions remain clear.

Google’s SRE guidance explains why smaller releases, attributable metrics and automatic pause or rollback reduce risk. Compound AI adds behavioural measures that ordinary service health cannot see. A canary may have normal latency and error rate while citing weaker evidence or selecting the wrong tool. Candidate and control therefore need per-release task, grounding, policy, trajectory and effect measures, with enough human adjudication for claims that automated metrics cannot establish.

Configuration guidance from SRE makes the same point through hermeticity: rollback is difficult when configuration references external resources that can change outside the controlled environment. A compound system contains many such references. Snapshot what can be snapshotted. Resolve names to digests where possible. Where a provider or live source remains mutable, record that fact and distinguish a reconstructed configuration from a claim of identical behaviour.

Rollback also has a hard boundary. Restoring an earlier configuration does not undo a payment, delete a duplicated case note safely or restore the world to an earlier state. External actions need idempotency, readback, compensation and accountable recovery. Data migrations may be irreversible. A policy may have changed because an old one is no longer legally or operationally valid. Release rollback restores a compatible control configuration; outcome recovery repairs the effects already produced.

Canaries have their own boundary. Rare harms, delayed outcomes and low-volume cohorts may not appear in a small exposure window. Shadow traffic cannot test user reliance or real side effects. Automated judges may share the candidate’s blind spots. Staged exposure must therefore sit beside synthetic failure injection, targeted human review and postcondition verification.

Operate claims, not just components

A compound release crosses several teams, so ownership must follow the evidence chain. The asset owner is accountable for the quality and compatibility contract of a prompt, retriever, tool, policy or workflow. The claim owner defines what the system is allowed to assert about itself and which evidence would defeat that claim. The release owner accepts only a candidate whose closure, evidence and recovery route are complete. The service owner watches operating signals and can narrow or withdraw authority. One person may hold several roles in a small team, but the decisions remain distinct.

This separation prevents two familiar failures. In the first, each component owner reports green while nobody owns the behaviour that emerges between components. In the second, a central release board becomes responsible for details it cannot judge and slows every change. The manifest creates a shared object: asset owners attest to their pieces, automated checks establish compatibility, claim owners interpret the selected evidence, and the release owner decides whether the residual uncertainty is acceptable.

Operating reviews should be organised around claims and release cohorts. “Retriever latency increased” is useful diagnosis. “Current-policy support fell below the release threshold for cases using snapshot 42” is a decision signal. The latter can identify the candidate, affected population, failed claim and authorised response. This is why per-release telemetry matters even after promotion: without cohort identity, a mixed rollout can average a regression away.

Incident response uses the same structure in reverse. Preserve the runtime receipt, resolve the release manifest, identify the first violated claim, then inspect the assets and effects in its dependency cone. Containment may disable one tool or force review rather than shut down the whole assistant. The repaired candidate receives a new identity and a regression case linked to the incident. Reusing the old release number would erase the causal history needed for learning and audit.

stateDiagram-v2
  [*] --> Claim
  state "Thought experiment 1: one<br/>model, four released systems" as Claim
  state "Negative control: when<br/>ordinary MLOps is enough" as Test
  state "What belongs inside" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Executable artefact: a release-manifest verifier

The following standard-library Python programme operates on synthetic data. It checks four properties that prose often leaves ambiguous: required asset kinds are present; every behaviour-critical asset is reachable from the workflow root; declared producer and consumer contracts agree; and evaluation claims target the exact bundle digest being promoted. The positive case passes. The negative case introduces a floating model alias, a prompt-retriever schema mismatch and an unsealed prompt edit. Promotion fails for each independent reason.

The behaviour digest deliberately excludes the release label and evaluation pack. Relabelling the same assembly should not create new behaviour, while the evaluation evidence must point to the digest rather than become part of the object it attests. The artefact is intentionally small. It is not a complete supply-chain security system, signature verifier or policy engine. In a real platform, manifest generation should occur in trusted automation, artefacts should be signed or verified against approved registries, and claim evidence should be immutable records with controlled access.

#!/usr/bin/env python3
"""Validate a synthetic compound-AI system release manifest.

The example is intentionally vendor-neutral and uses only Python's standard
library. It demonstrates three controls:

1. Release closure: every behaviour-critical asset is reachable from the
   workflow root.
2. Compatibility: declared producer and consumer contracts agree.
3. Evidence binding: release claims were tested against the exact bundle
   digest being promoted.
"""

from __future__ import annotations

import copy
import hashlib
import json
import re
from dataclasses import dataclass
from typing import Any, Iterable

HEX_64 = re.compile(r"^[0-9a-f]{64}$")
MUTABLE_TOKENS = (":latest", "@latest", "/main", "@main", "/master", "@master")
REQUIRED_KINDS = {
    "model_route",
    "prompt",
    "retriever",
    "evidence_snapshot",
    "tool_contract",
    "policy_bundle",
    "workflow",
    "memory_schema",
}


@dataclass(frozen=True)
class ValidationResult:
    ok: bool
    errors: tuple[str, ...]
    bundle_digest: str


def canonical_json(value: Any) -> bytes:
    """Return stable JSON bytes suitable for hashing."""
    return json.dumps(value, sort_keys=True, separators=(",", ":")).encode("utf-8")


def behaviour_projection(manifest: dict[str, Any]) -> dict[str, Any]:
    """Select and normalise fields that define the proposed runtime behaviour."""
    critical = [
        copy.deepcopy(artifact)
        for artifact in manifest["artifacts"]
        if artifact.get("behaviour_critical", False)
    ]
    critical_ids = {artifact["id"] for artifact in critical}
    for artifact in critical:
        artifact["depends_on"] = sorted(artifact.get("depends_on", []))

    compatibility = [
        copy.deepcopy(rule)
        for rule in manifest.get("compatibility", [])
        if rule["consumer"] in critical_ids and rule["producer"] in critical_ids
    ]
    return {
        "schema": manifest["schema"],
        "system": manifest["system"],
        "root": manifest["release"]["root"],
        "artifacts": sorted(critical, key=lambda item: item["id"]),
        "compatibility": sorted(
            compatibility,
            key=lambda item: (item["consumer"], item["producer"], item["field"]),
        ),
    }


def compute_bundle_digest(manifest: dict[str, Any]) -> str:
    payload = canonical_json(behaviour_projection(manifest))
    return "sha256:" + hashlib.sha256(payload).hexdigest()


def get_path(value: dict[str, Any], dotted_path: str) -> Any:
    current: Any = value
    for part in dotted_path.split("."):
        if not isinstance(current, dict) or part not in current:
            raise KeyError(dotted_path)
        current = current[part]
    return current


def walk_dependencies(root: str, by_id: dict[str, dict[str, Any]]) -> set[str]:
    seen: set[str] = set()
    stack = [root]
    while stack:
        current = stack.pop()
        if current in seen:
            continue
        seen.add(current)
        artifact = by_id.get(current)
        if artifact:
            stack.extend(artifact.get("depends_on", []))
    return seen


def validate_manifest(manifest: dict[str, Any]) -> ValidationResult:
    errors: list[str] = []
    artifacts = manifest.get("artifacts", [])
    by_id = {artifact.get("id"): artifact for artifact in artifacts if artifact.get("id")}

    if len(by_id) != len(artifacts):
        errors.append("artifact identifiers must be present and unique")

    kinds = {artifact.get("kind") for artifact in artifacts}
    missing_kinds = sorted(REQUIRED_KINDS - kinds)
    if missing_kinds:
        errors.append("release is missing required kinds: " + ", ".join(missing_kinds))

    for artifact in artifacts:
        artifact_id = artifact.get("id", "<unknown>")
        ref = str(artifact.get("ref", ""))
        digest = str(artifact.get("digest", ""))
        if any(token in ref.lower() for token in MUTABLE_TOKENS):
            errors.append(f"artifact {artifact_id!r} uses mutable reference {ref!r}")
        if not digest.startswith("sha256:") or not HEX_64.fullmatch(digest[7:]):
            errors.append(f"artifact {artifact_id!r} lacks a valid sha256 digest")
        for dependency in artifact.get("depends_on", []):
            if dependency not in by_id:
                errors.append(f"artifact {artifact_id!r} depends on unknown {dependency!r}")

    root = manifest.get("release", {}).get("root", "")
    if root not in by_id:
        errors.append(f"release root {root!r} is not an artifact")
        reachable: set[str] = set()
    else:
        reachable = walk_dependencies(root, by_id)

    for artifact in artifacts:
        if artifact.get("behaviour_critical", False) and artifact["id"] not in reachable:
            errors.append(
                f"behaviour-critical artifact {artifact['id']!r} is outside release closure"
            )

    for rule in manifest.get("compatibility", []):
        consumer_id = rule["consumer"]
        producer_id = rule["producer"]
        field = rule["field"]
        consumer = by_id.get(consumer_id)
        producer = by_id.get(producer_id)
        if not consumer or not producer:
            continue
        try:
            required = get_path(consumer, f"requires.{field}")
            provided = get_path(producer, f"provides.{field}")
        except KeyError:
            errors.append(
                f"compatibility rule {consumer_id!r} <- {producer_id!r} lacks field {field!r}"
            )
            continue
        if required != provided:
            errors.append(
                f"compatibility mismatch for {field!r}: {consumer_id!r} requires "
                f"{required!r}, but {producer_id!r} provides {provided!r}"
            )

    computed = compute_bundle_digest(manifest)
    recorded = manifest.get("release", {}).get("bundle_digest")
    if recorded != computed:
        errors.append(
            f"recorded bundle digest {recorded!r} does not match computed {computed!r}"
        )

    for claim in manifest.get("claims", []):
        if claim.get("status") != "pass":
            errors.append(f"claim {claim.get('id')!r} has not passed")
        if claim.get("tested_bundle_digest") != computed:
            errors.append(
                f"claim {claim.get('id')!r} was not tested against the promoted bundle"
            )

    rollback = manifest.get("rollback", {})
    rollback_digest = str(rollback.get("bundle_digest", ""))
    if not rollback.get("release_id") or not rollback.get("verified"):
        errors.append("rollback target is absent or unverified")
    if not rollback_digest.startswith("sha256:") or not HEX_64.fullmatch(rollback_digest[7:]):
        errors.append("rollback target lacks a valid bundle digest")

    return ValidationResult(not errors, tuple(errors), computed)


def fake_digest(label: str) -> str:
    return "sha256:" + hashlib.sha256(label.encode("utf-8")).hexdigest()


def make_good_manifest() -> dict[str, Any]:
    artifacts = [
        {
            "id": "model-route",
            "kind": "model_route",
            "ref": "model://reasoning-route@rev-3",
            "digest": fake_digest("model-route-rev-3"),
            "behaviour_critical": True,
            "depends_on": [],
            "provides": {"response_schema": "draft-v2"},
        },
        {
            "id": "prompt",
            "kind": "prompt",
            "ref": "git://assistant/prompts@8f14e45",
            "digest": fake_digest("prompt-8f14e45"),
            "behaviour_critical": True,
            "depends_on": [],
            "requires": {"evidence_schema": "evidence-v3"},
        },
        {
            "id": "retriever",
            "kind": "retriever",
            "ref": "oci://retriever@sha256-pinned",
            "digest": fake_digest("retriever-build-11"),
            "behaviour_critical": True,
            "depends_on": ["evidence-snapshot"],
            "provides": {"evidence_schema": "evidence-v3"},
        },
        {
            "id": "evidence-snapshot",
            "kind": "evidence_snapshot",
            "ref": "evidence://complaints-policy@snapshot-42",
            "digest": fake_digest("evidence-snapshot-42"),
            "behaviour_critical": True,
            "depends_on": [],
        },
        {
            "id": "tool-contract",
            "kind": "tool_contract",
            "ref": "openapi://case-read-api@2.1.0",
            "digest": fake_digest("case-read-api-2.1.0"),
            "behaviour_critical": True,
            "depends_on": [],
            "provides": {"tool_schema": "case-read-v2"},
        },
        {
            "id": "policy",
            "kind": "policy_bundle",
            "ref": "policy://complaints-assistant@revision-19",
            "digest": fake_digest("policy-revision-19"),
            "behaviour_critical": True,
            "depends_on": [],
            "provides": {"authority_schema": "authority-v4"},
        },
        {
            "id": "memory-schema",
            "kind": "memory_schema",
            "ref": "schema://case-working-state@5",
            "digest": fake_digest("case-working-state-5"),
            "behaviour_critical": True,
            "depends_on": [],
            "provides": {"memory_schema": "case-state-v5"},
        },
        {
            "id": "workflow",
            "kind": "workflow",
            "ref": "workflow://complaint-draft@17",
            "digest": fake_digest("complaint-draft-17"),
            "behaviour_critical": True,
            "depends_on": [
                "model-route",
                "prompt",
                "retriever",
                "tool-contract",
                "policy",
                "memory-schema",
            ],
            "requires": {
                "response_schema": "draft-v2",
                "tool_schema": "case-read-v2",
                "authority_schema": "authority-v4",
                "memory_schema": "case-state-v5",
            },
        },
        {
            "id": "evaluation-pack",
            "kind": "evaluation_pack",
            "ref": "eval://complaint-draft@pack-31",
            "digest": fake_digest("evaluation-pack-31"),
            "behaviour_critical": False,
            "depends_on": ["workflow"],
        },
    ]
    manifest: dict[str, Any] = {
        "schema": "compound-release-manifest/v1",
        "system": "complaint-drafting-assistant",
        "release": {
            "id": "complaint-drafting-assistant@17",
            "root": "workflow",
            "bundle_digest": "",
        },
        "artifacts": artifacts,
        "compatibility": [
            {"consumer": "prompt", "producer": "retriever", "field": "evidence_schema"},
            {"consumer": "workflow", "producer": "model-route", "field": "response_schema"},
            {"consumer": "workflow", "producer": "tool-contract", "field": "tool_schema"},
            {"consumer": "workflow", "producer": "policy", "field": "authority_schema"},
            {"consumer": "workflow", "producer": "memory-schema", "field": "memory_schema"},
        ],
        "claims": [
            {
                "id": "grounded-draft-quality",
                "status": "pass",
                "evaluation_run": "eval-run://grounding/104",
                "tested_bundle_digest": "",
            },
            {
                "id": "read-only-authority",
                "status": "pass",
                "evaluation_run": "eval-run://authority/52",
                "tested_bundle_digest": "",
            },
        ],
        "rollback": {
            "release_id": "complaint-drafting-assistant@16",
            "bundle_digest": fake_digest("complaint-drafting-assistant-16"),
            "verified": True,
        },
    }
    digest = compute_bundle_digest(manifest)
    manifest["release"]["bundle_digest"] = digest
    for claim in manifest["claims"]:
        claim["tested_bundle_digest"] = digest
    return manifest


def print_result(label: str, result: ValidationResult) -> None:
    print(f"{label}: {'PASS' if result.ok else 'FAIL'}")
    print(f"bundle_digest={result.bundle_digest}")
    for error in result.errors:
        print(f"- {error}")


def main() -> int:
    good = make_good_manifest()
    good_result = validate_manifest(good)
    print_result("GOOD manifest", good_result)

    bad = copy.deepcopy(good)
    by_id = {artifact["id"]: artifact for artifact in bad["artifacts"]}
    by_id["model-route"]["ref"] = "model://reasoning-route:latest"
    by_id["prompt"]["requires"]["evidence_schema"] = "evidence-v4"
    by_id["prompt"]["digest"] = fake_digest("unreviewed-prompt-edit")
    # Deliberately do not reseal the release or rerun its evidence.
    bad_result = validate_manifest(bad)
    print()
    print_result("BAD manifest", bad_result)

    return 0 if good_result.ok and not bad_result.ok else 1


if __name__ == "__main__":
    raise SystemExit(main())

GOOD manifest: PASS
bundle_digest=sha256:cdf4a1f7614f54417fa6febb8bddee90af8d52b2a1aa769221cc41da3158dc26

BAD manifest: FAIL
bundle_digest=sha256:bd8f44d319d2bf215e27cc99ac4208b3fecfedfa708e6b4e445809d6ef902094
- artifact 'model-route' uses mutable reference 'model://reasoning-route:latest'
- compatibility mismatch for 'evidence_schema'
- recorded bundle digest does not match the computed digest
- both release claims were tested against another bundle

The important property is fail-closed promotion. The bad manifest remains inspectable, but it cannot masquerade as the evaluated release. A platform can extend the same pattern with signatures, transparency logs, policy decisions, approval attestations and runtime verification.

Source notes and evidence boundary

The release-closure model, change-impact cone and executable manifest are practitioner synthesis developed for this article. The sources below establish the prior mechanisms and implementation precedents. They do not, individually, prove the complete architecture proposed here.

1. Primary research. D. Sculley et al., “Hidden Technical Debt in Machine Learning Systems”. Establishes entanglement, configuration debt and the CACE principle in production ML.
2. Primary research. Eric Breck et al., “The ML Test Score”, and “Data Validation for Machine Learning”. Provide concrete production tests and treat data as a first-class production asset.
3. Primary research. Denis Baylor et al., “Continuous Training for Production ML in the TensorFlow Extended Platform”. Describes operation of continuous ML pipelines and the cost of disruption and staleness.
4. Research synthesis. Dominik Kreuzberger, Niklas Kühl and Sebastian Hirschl, “Machine Learning Operations: Overview, Definition, and Architecture”. Supplies the MLOps baseline against which the operational extension is framed.
5. Practitioner research. Matei Zaharia et al., “The Shift from Models to Compound AI Systems”. Defines compound systems as interacting AI components and motivates system-level optimisation.
6. Primary research. Xiaohua Wang et al., “Searching for Best Practices in Retrieval-Augmented Generation”. Demonstrates that RAG behaviour depends on multiple pipeline choices.
7. Competing evidence. Andong Hua et al., “Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMs”. Qualifies broad prompt-sensitivity claims by identifying evaluation artefacts and greater robustness in modern models.
8. Official specification. SLSA build provenance and in-toto release attestation. Provide patterns for resolved dependencies, digests and authoritative release membership.
9. Official engineering guidance. Google SRE, “Configuration Design and Best Practices” and “Canarying Releases”. Support hermetic rollback, gradual change and attributable canary decisions.
10. Official implementation documentation. Open Policy Agent bundles and decision logs. Show revisioned policy/data bundles, signatures and runtime decision metadata.
11. Official observability documentation. OpenTelemetry traces, the generative-AI attribute registry, baggage and sensitive-data guidance. Ground the runtime receipt and its privacy boundary.
12. Official risk guidance. NIST SP 800-218A and the Generative AI Profile. Distinguish AI-model producers, AI-system producers and lifecycle risk management.

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: The release that nobody could reproduce
  M->>E: Thought experiment 1: one model, four released systems
  E-->>M: compare rival explanations
  E->>D: Release the system you can defend
  D-->>Q: revise scope or act

Release the system you can defend

MLOps taught engineering teams to treat models, data and pipelines as operational products rather than research outputs. Compound AI requires the next enlargement. The model remains important, but it sits inside a system whose prompts, evidence, tools, policies, workflows and state interpretations can move independently and interact.

The architecture decision that changes is concrete. Replace “deploy the model and application” with “promote an evidence-bound system release.” Resolve every behaviour-critical asset. Reject incompatible combinations. Compute one bundle digest. Tie claims and tests to that digest. Carry the release identity through runtime traces. Canary the candidate as a system. Recover through a verified predecessor or a deliberately narrower operating mode, while treating external effects as a separate recovery problem.

This discipline does not guarantee good behaviour. It creates the conditions under which good or bad behaviour can be attributed, evaluated and corrected. When an incident arrives, the first question should no longer be “which model version was running?” It should be “which released system made this decision, under which evidence and authority, and can we reconstruct it exactly?” A team that can answer has an operating system for compound AI. A team that cannot still has a collection of components.