The release that passed and still moved

Consider a synthetic credit-memo assistant. Its stable version retrieves approved evidence, calculates ratios through a deterministic service and drafts a narrative for a human underwriter. A candidate release changes the language model, system instruction and reranker. Offline, it passes the reference set. In shadow mode, reviewers prefer its prose. Median response time falls. The release owner sends five per cent of live drafting traffic to the candidate.

Nothing dramatic appears on the main dashboard. Average reviewer score remains inside its ordinary band. Refusal rate and latency look healthy. Yet the candidate adds unsupported policy language in three complex cases. Two are corrected by reviewers. One enters the case note before correction because the note-writing tool is available after superficial approval. The blended quality metric barely moves because routine cases dominate the population.

The team routes new requests back to the old model endpoint and calls that a rollback. It has restored neither the old behaviour nor the old state. The candidate prompt remains active in one worker pool, the new reranker still selects evidence, and case notes created during the canary remain authoritative unless somebody identifies and quarantines them. The release was small in traffic share, yet large in consequence.

The central answer is that a stochastic AI release is a controlled population experiment, not a binary software switch. Its unit of change is the complete configuration that produces behaviour. Its gate combines intolerable-event rules with uncertainty-aware comparisons. Its rollback has three jobs: stop candidate routing, contain candidate-written state and reconcile effects already produced.

This focus is narrower than the broader release-control argument in Shipping LLM Systems in Regulated Environments. It depends on the evaluation portfolio described in The Evaluation Stack for Agentic Systems and hands post-release detection to Behavioural Sentinels. The question here is the seam between those controls: how evidence should govern promotion while two versions coexist.

Part oneBehaviour is a distribution

A conventional service release has many sources of variation, but engineers can usually state an exact functional expectation for a request: this input should produce that response, database write or error. A generative model supplies a conditional distribution over possible continuations. Decoding chooses from that distribution. Retrieval, tools, policy filters and human review then transform the model proposal into an operational result.

Even when sampling is constrained, exact replay may fail across framework releases, devices or nondeterministic operations. PyTorch's reproducibility guidance warns that complete reproducibility is not guaranteed across releases, commits or platforms, and that deterministic operations can cost performance.[1] That implementation fact removes a tempting escape: fixing a random seed does not turn a changing compound system into an ordinary pure function.

The same prompt is one draw

Suppose a candidate answers a policy question correctly nine times and introduces one unsupported condition on the tenth. The stable version answers correctly eight times and omits one caveat twice. Which is better? One example cannot decide. Ten examples barely describe the tail. The relevant object is a vector of behaviour distributions conditioned on task, cohort, evidence state, tool availability and consequence.

The release question is whether the candidate preserved every material behavioural claim within acceptable uncertainty. Those claims can include task success, evidence support, refusal calibration, tool choice, escalation, latency, cost and authority compliance. Some admit statistical tolerance. Others are invariants.

Represent the release claim before sampling

A behavioural metric becomes useful only after its population and decision meaning are fixed. Let p_r,c,k denote the probability that metric k passes under release r for cohort c. Here, r identifies stable or candidate, c might identify routine, complex or sensitive work, and k might identify evidence support. A non-inferiority claim requires p_{candidate,c,k} minus p_control,c,k to be at least minus δ. The margin δ is the largest degradation the consequence owner is willing to rule out before promotion.

When δ becomes smaller, the claim is stricter and usually needs more evidence. When a cohort becomes rarer, its interval shrinks more slowly unless sampling is deliberately enriched. When the oracle is systematically noisy, more observations do not remove the disagreement. These relationships determine canary duration and the authority that may remain exposed while the system learns.

The oracle must also be named. Deterministic checks can verify schema, arithmetic, source identifiers and permissions. Expert reviewers can judge completeness and policy interpretation. Model-based judges may assist, but a judge sharing the candidate's failure mode is correlated evidence, not an independent witness. The release record should preserve oracle version, reviewer rubric and disagreement rather than collapsing all judgements into one unexplained number.

A recent preprint on behavioural fingerprints proposes repeatedly sampling a fixed prompt set, comparing output distributions and detecting endpoint changes that uptime or latency cannot reveal.[2] The work is early and does not establish a complete release method. Its useful contribution here is the object it chooses to monitor: behaviour across samples.

Thought experiment: the locked seed

Imagine two teams testing the same candidate. Team A fixes the seed, runs every golden case once and obtains byte-for-byte repeatability. Team B runs each case across several seeds, realistic evidence states and two tool-failure conditions. Team A can identify a code change precisely. Team B can estimate whether the candidate has acquired a new behavioural tail.

Now vary one feature: make production decoding deterministic. Team A's advantage increases, but Team B still observes changing retrieval results, context ordering, tool responses and provider infrastructure. Remove those sources too, and the system approaches an ordinary deterministic program. At that point repeated behavioural sampling loses value. This is the mechanism's negative control.

flowchart LR
  A["The release that passed and<br/>still moved"] --> B["On this page"]
  B --> C["Part one Behaviour is a<br/>distribution"]
  C --> D["Part two Canarying is an<br/>experiment"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#7B4CD8,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Part twoCanarying is an experiment

The word “canary” is often used for any small traffic split. Google’s SRE workbook gives it a stricter shape: a limited production segment is compared with a control so a bad release can be detected before broad exposure.[3] Online experimentation work likewise treats concurrent assignment, instrumentation and analysis as one system.[4]

For stochastic AI, the comparison must preserve the causal question: did this release bundle change behaviour for equivalent work? A candidate measured against last week's average confounds version with time. A candidate routed only to easy cases confounds version with task mix. A request-level split in a stateful assistant can expose one user to both versions and contaminate memory. Small traffic is not automatically controlled traffic.

Keep a concurrent control

The stable control should receive traffic during the same interval, through the same surrounding dependencies, with release identity attached to every trace. Concurrent control absorbs shared demand seasonality and many infrastructure changes. It does not protect against a common-mode fault that hits both versions, such as a corpus permission error or provider outage. Those require independent invariants and absolute service sentinels.

Where privacy and cost permit, a paired shadow phase is strong for diagnosis: send the same authorised case snapshot to stable and candidate versions, suppress candidate effects, then compare outputs and trajectories. Pairing holds task difficulty constant. It creates a clean explanation when a route, retrieval choice or tool argument changes.

Choose the randomisation unit before the percentage

Randomise at the smallest unit that prevents interference. For a stateless classifier, it may be the request. For a conversational assistant, it may be the conversation. For a case worker, it may be the case or account. For a system that writes long-lived memory, it may be the user or tenant. Assignment should remain sticky for the unit's relevant lifetime, and the trace should record every component version actually used.

A five per cent request split can be less safe than a one per cent account split because the first design creates cross-version state. A user asks the stable version to remember a preference, the candidate reads it under a new schema, then the stable version sees the candidate's rewrite. No metric can cleanly attribute the resulting failure to one arm. The experiment has violated its own boundary.

Prove the experiment works before testing the candidate

Run an A/A exercise in which both arms use the stable release but travel through the proposed assignment, telemetry and analysis path. The arms should have the expected traffic ratio, comparable cohort composition and matching metric definitions. A sample-ratio mismatch can reveal broken routing, dropped events or eligibility rules before those defects are mistaken for model behaviour.

The A/A run should test trace completeness. Every case needs assignment identity, release manifest, evidence snapshot, tool versions, oracle result and outcome status. Missingness must be reported by arm and cohort. If candidate failures are more likely to lose traces, analysing only complete records creates a favourable bias exactly when the release is least reliable.

Human review should be blinded to version where practical. Review order should be randomised, and repeated cases should not let reviewers infer the candidate from style alone. When blinding is impossible, preserve the limitation and sample independent adjudication. A release gate measures the combination of system and oracle; automated arithmetic does not make a subjective rubric objective.

Finally, test one deliberately bad candidate. Add a known omission to a synthetic cohort and verify that the pipeline detects it, stops at the declared boundary and produces an interpretable reason. A gate that has only ever passed healthy releases has not demonstrated detection.

Do not peek with ordinary intervals

A release team naturally watches the canary continuously and stops when a chart looks bad. Repeatedly applying a fixed-sample significance test at every glance raises the chance of a false alarm. Sequential testing and confidence-sequence research exists because the stopping rule is part of the inference. Lindon, Sanden and Shirikian demonstrate continuously monitored regression tests for software canaries while controlling false detection.[5] Howard and colleagues develop time-uniform confidence sequences whose coverage holds across time.[6] Lindon and Malek extend anytime-valid inference to count data used in canary experiments.[7]

The engineering choice need not begin with the most sophisticated method. Teams can use a small number of predeclared looks and spend the error budget across them, as the executable artefact later does. The requirement is more modest and more important: the analysis plan must match how the team will actually watch and stop the rollout.

Thought experiment: the perfect shadow

A candidate drafting assistant shadows one thousand real cases. It never writes or sends anything. Evidence support and reviewer score exceed the control. The team concludes that live release is safe. Now vary one feature: expose the draft through a more confident interface. Reviewers inspect fewer citations, approval time falls and unsupported claims survive more often. The model did not worsen. The human-machine system changed.

Shadowing is excellent for task and trajectory comparison. It cannot measure reliance, queue behaviour, action effects or cross-user interference. A live canary is therefore a distinct evidence stage. It should begin with bounded authority and an observable human route.

Part threeExposure must follow consequence

Traditional rollout plans express exposure as one, five, twenty-five and one hundred per cent of traffic. That works when requests have roughly comparable consequences. AI systems often mix harmless summaries, material recommendations and irreversible actions. One per cent of all requests can include the single payment, account closure or regulatory notice that the system should never execute under experimental authority.

Canary size should be measured in consequence-weighted exposure, not traffic share alone. The first live stage should include frequent, reversible and reviewable work. High-consequence cohorts should remain shadowed, simulation-only or subject to mandatory approval until evidence and recovery match their risk.

This creates an exposure ladder. At the bottom, the candidate observes mirrored traffic and produces no effects. Next, it drafts into an isolated workspace. Then it serves selected low-consequence users with human review. Only later does it receive broader scope. Some actions may never enter a probabilistic release lane; a deterministic authority kernel can continue to enforce limits regardless of model version.

Let the slowest trustworthy label set dwell time

Sample count is only one clock. Route choices, tool errors and unsupported-claim checks may arrive within seconds. Reviewer corrections may arrive after a shift. Complaints, reversals or realised losses may arrive weeks later. A rollout that reaches full exposure as soon as fast metrics clear can outrun the outcome that would falsify its claim.

For each release claim, record the feedback-latency distribution and the operational cycle that must complete. Dwell time is constrained by statistical evidence, outcome lag and the time needed to expose the relevant cohort. This does not require every delayed outcome before any progress. It does require authority to remain bounded while slow evidence is unresolved.

A practical design uses two clocks. The fast clock can stop a candidate on boundary events or leading behavioural signals. The slow clock controls expansion into higher-consequence work. If fast quality clears while delayed corrections remain pending, the candidate may continue in its current low-consequence stage. Time in canary is evidence only when the relevant outcomes had a chance to arrive.

Separate hard stops from statistical gates

A rate metric answers whether an ordinary behaviour is acceptably close to control. It should not decide whether an intolerable event is permitted. A confirmed unauthorised disclosure, effect beyond scope, fabricated approval or bypass of mandatory review can trigger rollback on its first occurrence. The boundary event still needs an oracle, evidence and confirmation route, but its response is not diluted by thousands of routine successes.

The reverse separation also matters. Zero observed catastrophic events in a small sample does not establish a low catastrophic rate. Rare harms require adversarial evaluation, failure injection, containment and later monitoring. The canary can enforce a stop when it sees one; it cannot prove absence from silence.

Staged-rollout research formalises the trade between learning speed and exposure. Zhao, Liu and Deb combine sequential monitoring with adaptive ramping,[8] while rapid-regression work studies continuously monitored production canaries.[5] These methods provide machinery rather than a universal risk policy. The acceptable margin and exposure schedule still belong to the service's consequence owner.

Critical cohorts must clear independently before blending. A ninety-five per cent routine population can hide regression in a five per cent vulnerable or complex population. Promotion should read the worst decision-relevant cohort before the aggregate. When a cohort remains underpowered, the honest state is hold or narrow, not pass-by-average.

A single weighted score is especially dangerous at this boundary. Lower latency or cost can numerically cancel weaker evidence support, even though the gains and harms are not substitutable. Define veto metrics first, then evaluate tradeable objectives among candidates that remain inside every boundary. In decision terms, promotion is an intersection of admissible claims before it becomes an optimisation problem.

Part fourRollback is three different operations

Infrastructure platforms make it easy to restore an earlier container revision. Argo Rollouts and Kubernetes can shift traffic or undo a deployment.[9][10] That is necessary and often insufficient. AI behaviour can depend on configuration and state outside the container.

Sculley and colleagues described how machine-learning behaviour is entangled with data dependencies, configuration and surrounding systems.[11] The ML Test Score similarly treats readiness as tests and monitoring obligations across features, data, models and infrastructure.[12] Compound AI enlarges the release surface again.

Define the behavioural release unit

The release manifest should identify the smallest versioned bundle capable of reproducing the candidate's material behaviour. For a retrieval agent, this can include model endpoint and immutable model identifier, system and task prompts, decoding parameters, tool schemas, policy configuration, reranker, retrieval publication watermark, memory schema, runtime code, feature flags and evaluator version.

Rollback must target the behavioural release unit, not the component mentioned in the change ticket. Restoring old weights against a new prompt and retrieval index creates an untested hybrid. It may remove the suspected cause while introducing a second behaviour nobody evaluated.

Model cards established the value of documenting intended use, evaluation conditions and provenance.[13] A release manifest is narrower and executable. It binds those claims to the exact bundle promoted through the gate and to the previous bundle retained for recovery.

Operation one: route rollback

Route rollback stops assigning new eligible units to the candidate and restores the known-good manifest. Sticky assignments need an explicit rule: finish in-flight work on the candidate, restart it on the stable version, or escalate. Replaying an effectful action without idempotency can be worse than leaving it incomplete, so the runbook must distinguish read-only generation from writes.

Rehearse this path before the first live case. A useful rehearsal restores the stable manifest, invalidates version-sensitive caches, drains or quarantines candidate queues and reads the active versions back from every serving pool. The evidence is the observed restored state, not merely a successful deployment command. If the stable bundle cannot be reconstructed under pressure, the candidate has no credible rollback target.

Operation two: state containment

Candidate-written memory, summaries, embeddings, preferences and case notes should carry release identity and provenance. On rollback, the system can quarantine those writes, recompute derived state from authoritative events or require review before promotion into shared knowledge. Without release identity, the team cannot answer which persistent facts were created under the failed candidate.

Dual-writing everything into the same memory store during a canary is dangerous. Safer patterns use candidate namespaces, write-behind approval or event-sourced records that preserve original evidence. Rollback then changes which derived state is trusted rather than pretending the write never happened.

Unknown outcome is a first-class release state

An effectful tool can accept a command while the caller loses the response. The trace then knows that an action was proposed and transmitted, but not whether the external system committed it. If rollback blindly retries the stable workflow, the organisation may create a duplicate payment, message or case update. If it assumes success, it may leave the intended effect absent.

The action contract therefore needs an idempotency key, effect identifier and independent readback path. The release ledger should represent unknown outcome explicitly. New candidate actions stop, while the uncertain effect enters reconciliation rather than ordinary retry. A responder queries authoritative state, matches effect receipts and either confirms, compensates or escalates.

In-flight work also needs a boundary. Read-only generation can often be discarded and restarted. A multi-step workflow that has crossed an external write should resume from verified state. The rollback runbook must say where the recovery cut lies for each action class. Otherwise old code returns while the business process continues from an invented history.

Operation three: effect recovery

External effects may be reversible, compensatable, disputed or irreversible. A draft can be deleted. A queued message can be cancelled. A payment may require a compensating transaction and investigation. A human decision influenced by a model cannot be erased; it can only be reopened with new evidence. Each action contract should state its recovery class before the action enters a canary.

The NIST AI RMF playbook treats monitoring, incident response and change management as continuing lifecycle practices rather than one-time approval.[14] The Generative AI Profile adds risks specific to generative systems without prescribing a particular canary algorithm.[15] The practitioner synthesis is to bind release, monitoring and recovery evidence to the same manifest.

stateDiagram-v2
  [*] --> Claim
  state "The same prompt is one draw" as Claim
  state "Represent the release claim<br/>before sampling" as Test
  state "Thought experiment, the locked<br/>seed" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Part fiveBuild the release gate

A release gate should be a small executable decision system, not a meeting that interprets whichever dashboard happens to be open. It consumes a version manifest, assignment records, behavioural events, outcome labels and hard-boundary events. It emits one of four states: PROMOTE, HOLD, ROLLBACK or NARROW.

Promotion means every hard invariant passed and each critical cohort has enough evidence to clear its threshold. Hold means evidence is insufficient. Rollback means a boundary failed or material regression is established. Narrow means the candidate may continue with less authority, a smaller cohort or mandatory review while evidence grows.

Make the gate auditable as code

The gate configuration should be versioned beside the release manifest. It names eligible cohorts, assignment unit, metric queries, oracle versions, hard stops, margins, planned looks, dwell rules and rollback target. Changing a threshold during the canary creates a new measurement system. The change may be justified, but it should restart or explicitly reanalyse the decision rather than rewrite the past.

Every state needs reason codes and evidence identifiers. “Hold” should say whether the cause is sample size, missing labels, cohort imbalance, oracle disagreement or unresolved dependency. “Rollback” should identify the triggering invariant or statistical claim. This makes the gate useful to responders and reviewers, not merely to the deployment controller.

The gate also needs an expiry rule. A canary that remains inconclusive while dependencies, prompts or traffic mix continue changing is no longer the experiment that was approved. Expiry can trigger hold, fresh baseline collection or a new candidate identity. It should never allow an old partial result to authorise a materially different bundle.

Human exception authority remains necessary. A service owner may accept a bounded limitation, extend a hold or choose a narrower release. The override should record scope, rationale, owner, expiry and compensating control. It must not be a hidden button that converts a failed hard boundary into promotion. Automation executes the pre-agreed rule; accountable people own residual risk and any departure from it.

Minimal worked example

Take a synthetic answer-support task with control success near ninety per cent. The candidate is acceptable if its success rate is no more than five percentage points worse. The team predeclares six looks at 1,000, 2,000, 4,000, 8,000, 16,000 and 32,000 cases per arm. It adjusts the confidence level across looks and cohorts. Any confirmed restricted-data disclosure triggers immediate rollback.

At 1,000 cases, the candidate's observed score may be slightly higher, yet the uncertainty interval can still include unacceptable degradation. The correct decision is hold. At 32,000 cases, a healthy candidate may finally clear the conservative non-inferiority gate. An unsafe candidate that produces five boundary events in its first 1,000 cases rolls back immediately, even when its average quality interval is inconclusive.

This exposes an important cost: critical cohorts need enough samples. If sensitive cases are rare, waiting for natural traffic can make the canary dangerously long. Authorised replay cases can accelerate evidence, but live effects for that cohort should remain bounded until genuine live evidence arrives.

A release decision record

The following instrument forces a team to name the experiment, authority boundary and recovery path before traffic moves. A missing entry is itself a hold reason.

#!/usr/bin/env python3
"""Transparent release gate for a stochastic AI canary.

The script uses synthetic Bernoulli outcomes. It separates hard boundary
failures from quality non-inferiority and evaluates a fixed set of looks with
conservative family-wise error control. It is a teaching artefact, not a
substitute for local power analysis, causal design or risk ownership.
"""

from __future__ import annotations

from dataclasses import dataclass
from math import sqrt
from random import Random
from statistics import NormalDist
from typing import Iterable


@dataclass(frozen=True)
class Cohort:
    name: str
    control_pass_rate: float
    candidate_delta: float
    traffic_weight: float


@dataclass(frozen=True)
class Scenario:
    name: str
    seed: int
    cohorts: tuple[Cohort, ...]
    candidate_boundary_rate: float
    noninferiority_margin: float = 0.05
    family_alpha: float = 0.05
    looks_per_arm: tuple[int, ...] = (
        1_000, 2_000, 4_000, 8_000, 16_000, 32_000
    )


@dataclass(frozen=True)
class ArmCounts:
    successes: int
    total: int

    @property
    def rate(self) -> float:
        return self.successes / self.total if self.total else 0.0


def wilson_interval(successes: int, total: int, alpha: float) -> tuple[float, float]:
    """Return a two-sided Wilson score interval for a Bernoulli proportion."""
    if total <= 0:
        raise ValueError("total must be positive")
    if not 0.0 < alpha < 1.0:
        raise ValueError("alpha must lie between 0 and 1")

    z = NormalDist().inv_cdf(1.0 - alpha / 2.0)
    phat = successes / total
    z2 = z * z
    denominator = 1.0 + z2 / total
    centre = (phat + z2 / (2.0 * total)) / denominator
    radius = (
        z
        * sqrt((phat * (1.0 - phat) / total) + (z2 / (4.0 * total * total)))
        / denominator
    )
    return max(0.0, centre - radius), min(1.0, centre + radius)


def difference_interval(
    candidate: ArmCounts,
    control: ArmCounts,
    alpha: float,
) -> tuple[float, float]:
    """Conservative interval for candidate pass rate minus control pass rate."""
    # A difference interval combines two arm intervals. Split the
    # comparison-level error budget across arms before applying the union bound.
    arm_alpha = alpha / 2.0
    c_lo, c_hi = wilson_interval(candidate.successes, candidate.total, arm_alpha)
    b_lo, b_hi = wilson_interval(control.successes, control.total, arm_alpha)
    return c_lo - b_hi, c_hi - b_lo


def draw_bernoulli(rng: Random, probability: float, count: int) -> int:
    return sum(rng.random() < probability for _ in range(count))


def allocate_by_weight(total: int, cohorts: Iterable[Cohort]) -> dict[str, int]:
    cohort_list = list(cohorts)
    raw = [total * cohort.traffic_weight for cohort in cohort_list]
    allocated = [int(value) for value in raw]
    remainder = total - sum(allocated)
    order = sorted(
        range(len(cohort_list)),
        key=lambda i: raw[i] - allocated[i],
        reverse=True,
    )
    for index in order[:remainder]:
        allocated[index] += 1
    return {cohort.name: n for cohort, n in zip(cohort_list, allocated)}


def run_scenario(scenario: Scenario) -> str:
    rng = Random(scenario.seed)
    previous_n = 0
    control_counts = {c.name: ArmCounts(0, 0) for c in scenario.cohorts}
    candidate_counts = {c.name: ArmCounts(0, 0) for c in scenario.cohorts}
    boundary_events = 0

    # Bonferroni allocation over all planned looks and all cohort plus overall
    # comparisons. difference_interval then splits each comparison budget by arm.
    comparisons = len(scenario.looks_per_arm) * (len(scenario.cohorts) + 1)
    alpha_per_comparison = scenario.family_alpha / comparisons

    lines = [
        f"Scenario: {scenario.name}",
        (
            "Gate: immediate rollback on any boundary event; otherwise require "
            f"candidate quality no worse than {scenario.noninferiority_margin:.1%} "
            "at a predeclared look."
        ),
    ]

    for look_number, n_per_arm in enumerate(scenario.looks_per_arm, start=1):
        increment = n_per_arm - previous_n
        previous_n = n_per_arm
        allocation = allocate_by_weight(increment, scenario.cohorts)

        for cohort in scenario.cohorts:
            n = allocation[cohort.name]
            control_successes = draw_bernoulli(rng, cohort.control_pass_rate, n)
            candidate_probability = min(
                1.0, max(0.0, cohort.control_pass_rate + cohort.candidate_delta)
            )
            candidate_successes = draw_bernoulli(rng, candidate_probability, n)

            old_control = control_counts[cohort.name]
            old_candidate = candidate_counts[cohort.name]
            control_counts[cohort.name] = ArmCounts(
                old_control.successes + control_successes,
                old_control.total + n,
            )
            candidate_counts[cohort.name] = ArmCounts(
                old_candidate.successes + candidate_successes,
                old_candidate.total + n,
            )

        boundary_events += draw_bernoulli(
            rng, scenario.candidate_boundary_rate, increment
        )

        cohort_intervals = {
            cohort.name: difference_interval(
                candidate_counts[cohort.name],
                control_counts[cohort.name],
                alpha_per_comparison,
            )
            for cohort in scenario.cohorts
        }

        overall_control = ArmCounts(
            sum(value.successes for value in control_counts.values()),
            sum(value.total for value in control_counts.values()),
        )
        overall_candidate = ArmCounts(
            sum(value.successes for value in candidate_counts.values()),
            sum(value.total for value in candidate_counts.values()),
        )
        overall_interval = difference_interval(
            overall_candidate, overall_control, alpha_per_comparison
        )

        if boundary_events > 0:
            decision = "ROLLBACK"
        elif any(
            upper < -scenario.noninferiority_margin
            for _, upper in cohort_intervals.values()
        ):
            decision = "ROLLBACK"
        elif (
            overall_interval[0] > -scenario.noninferiority_margin
            and all(
                lower > -scenario.noninferiority_margin
                for lower, _ in cohort_intervals.values()
            )
        ):
            decision = "PROMOTE"
        else:
            decision = "HOLD"

        worst_cohort, worst_interval = min(
            cohort_intervals.items(), key=lambda item: item[1][0]
        )
        observed_delta = overall_candidate.rate - overall_control.rate
        lines.append(
            f"Look {look_number}: n={n_per_arm:,}/arm, "
            f"observed quality delta={observed_delta:+.2%}, "
            f"overall interval=[{overall_interval[0]:+.2%}, {overall_interval[1]:+.2%}], "
            f"worst cohort={worst_cohort} "
            f"[{worst_interval[0]:+.2%}, {worst_interval[1]:+.2%}], "
            f"boundary events={boundary_events} -> {decision}"
        )

        if decision in {"PROMOTE", "ROLLBACK"}:
            lines.append(f"Final decision: {decision}")
            return "\n".join(lines)

    lines.append("Final decision: HOLD")
    return "\n".join(lines)


def main() -> None:
    healthy = Scenario(
        name="healthy candidate",
        seed=174,
        cohorts=(
            Cohort("routine", 0.94, +0.006, 0.40),
            Cohort("complex", 0.88, +0.004, 0.35),
            Cohort("sensitive", 0.82, +0.005, 0.25),
        ),
        candidate_boundary_rate=0.0,
    )
    unsafe = Scenario(
        name="unsafe candidate",
        seed=4174,
        cohorts=(
            Cohort("routine", 0.94, -0.025, 0.40),
            Cohort("complex", 0.88, -0.040, 0.35),
            Cohort("sensitive", 0.82, -0.080, 0.25),
        ),
        candidate_boundary_rate=0.004,
    )

    print(run_scenario(healthy))
    print("\n" + "=" * 88 + "\n")
    print(run_scenario(unsafe))


if __name__ == "__main__":
    main()

Release the experiment, not the endpoint

Stochasticity does not make disciplined release impossible. It changes what must be disciplined. The system must compare behaviour distributions, keep a concurrent control, assign stable experimental units, preserve critical cohorts and use inference that matches the stopping rule.

The architecture decision changes with that measurement. The deployable object becomes a behavioural release unit. Traffic growth becomes consequence-weighted exposure. Rollback becomes three operations across routing, state and effects. Promotion becomes a claim that every critical boundary and cohort has enough evidence, rather than a feeling that the dashboard looks quiet.

A defensible canary is the last controlled place to discover that a release changed more than its version number. Use it to learn under bounded authority. When evidence is weak, hold. When scope is broad, narrow. When a boundary fails, restore the whole behaviour-producing bundle and reconcile what escaped it.