The alarm that becomes inevitable

A fixed test hides a fixed stopping rule

Imagine a coin known to land heads 2 per cent of the time. You flip it fifty times, count the heads and run a test whose false-positive probability is 5 per cent at that fixed sample size. Used once, as designed, the test has its advertised property. Now inspect after every flip from twenty onward and stop the first time the result looks significant. The calculation at each look may be correct for that look, yet the procedure is different. It has many chances to encounter a rare fluctuation.

The hidden assumption is not independence alone, nor normality, nor a large sample. It is more basic: the rule that chooses when to look is part of the statistical procedure. Classical sequential analysis was built around this fact. Wald's sequential probability ratio test made the stopping rule explicit, while later group-sequential designs allocated error across planned interim analyses.[2][3] A fixed-horizon p-value does not become a sequential test merely because software can recompute it quickly.

Why repeated peeking spends the budget

A false-alarm budget is a promise about a procedure, not a decoration on a threshold. Suppose the acceptable system really does generate harmful outcomes at a conditional rate no greater than 2 per cent. We want the probability of ever stopping it for statistical evidence to be at most 5 per cent. A fixed-horizon test can control error at one declared endpoint. It does not, without adjustment, control the event “the test is significant at least once over all times”. Johari and colleagues describe the same failure in continuously monitored online experiments: endogenous sample size makes conventional inference unreliable.[7]

A false-alarm budget is not an alert-rate target

Operational discussions often use “false-alarm rate” for several different quantities. One team may mean the fraction of alerts later judged unnecessary. Another may mean the probability of at least one mistaken interruption during a mission. A third may mean the expected number of alerts per thousand actions. These are not interchangeable. The e-process boundary used here controls one specific event: the probability that an acceptable evidence account ever crosses its statistical abstention boundary. It does not directly control the positive predictive value of alerts, the number of policy holds, or the burden on reviewers.

This distinction changes how alpha should be chosen. A familiar value such as 0.05 has no privileged operational status. If a false stop merely wastes a few seconds, a larger budget may be reasonable. If a false stop abandons a rescue mission or interrupts a regulated service, the budget may need to be much smaller. Conversely, a tiny alpha can make the monitor so slow that real deterioration passes through many consequential actions. The threshold is therefore an assurance allocation derived from loss and recovery, not a ritual carried over from publication statistics.

It also changes what counts as “spending” the budget. Reading the same anytime-valid account at one hundred additional moments does not open one hundred new null events. Creating a new monitored claim does. Splitting one mission into regional, subgroup and severity monitors does. Reopening after remediation does. The number of clocks is statistically harmless inside an account; the number of claims is not.

There are several legitimate repairs. Pre-plan a finite set of looks and use group-sequential boundaries. Spend alpha over time. Use a sequential probability ratio test for specified models. Or use an anytime-valid process designed so that arbitrary stopping is already covered. The choice depends on the monitoring schedule and model. Anytime-validity is not the only valid method; it is the method that matches an unknown or adaptive horizon most directly.

Alpha-spending and e-processes solve related but different design problems. A spending function allocates a total error budget over information time and can be highly efficient when the analysis plan is controlled. An e-process treats the whole path as one evidence object and remains readable at arbitrary stopping times. Neither excuses post-hoc redefinition of the outcome, selective deletion of observations, or a new test disguised as a software restart.

flowchart LR
  A["The alarm that becomes inevitable"] --> B["Evidence that survives peeking"]
  B --> C["From evidence to an abstention<br/>right"]
  C --> D["Where the guarantee stops"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#7B4CD8,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Evidence that survives peeking

Turn observations into a fair game under the null

Return to the binary harm stream. Let X_t equal 1 when the independently verified outcome at step t is materially harmful, and 0 otherwise. The acceptable-operation claim is deliberately conditional:

Here F_t-1 means the information legitimately available before the next outcome. This null can allow dependence, seasonality and adaptive actions, provided the conditional risk bound remains true. It is stronger and more operational than saying that the long-run average will probably be small.

Pick an alternative harm rate q greater than p₀. For one observation, multiply the evidence by q/p₀ after a harm and by (1-q)/(1-p₀) after a safe outcome. Under the boundary case p_t = p₀, the expected multiplier is exactly one. Under any p_t below p₀, it is at most one. The product therefore behaves like the wealth of a bettor who cannot expect to grow rich when the acceptable-operation claim is true.

A single q detects shifts near that alternative efficiently but can be slow elsewhere. Mix several alternatives with positive weights that sum to one. The weighted sum remains a non-negative supermartingale under the null. In current terminology it is an e-process. Test martingales, e-processes and confidence sequences form the core machinery of safe anytime-valid inference.[4][5][8]

Read the guarantee literally

The phrase “at any stopping time” does not mean that any manipulation is allowed. A stopping time is a decision based on information available so far. The monitor may stop because evidence crossed, because a shift ended, because compute ran out, or because a person inspected the dashboard and chose to pause. What it may not do is use future outcomes to decide which past observations count, then claim the same guarantee. The filtration is the contract that separates allowed adaptation from hindsight.

Consider a single alternative q = 0.20 against p₀ = 0.05. A harm multiplies evidence by four. A safe outcome multiplies it by 0.80/0.95, about 0.842. Two safe outcomes followed by one harm produce about 0.842 × 0.842 × 4 = 2.84. Reverse the order and the final product is the same, although the path differs. If the system can stop early, path order matters operationally because a boundary may be crossed before later safe evidence arrives. Ville's inequality covers that first-crossing event.

The current e-value can decline, which is why the decision rule uses the running maximum or stops immediately at crossing. An anytime-valid p-process can be formed from the reciprocal of that running maximum, capped at one. This makes the interface familiar, but the interpretation should remain precise. An e-value of 20 is not a 95 per cent probability that the null is false. It is evidence whose boundary crossing is calibrated by the null process.

If the operating question is “what harm rate remains plausible?” rather than “has acceptable operation been rejected?”, a confidence sequence is often the better display. It provides a sequence of intervals that jointly covers the parameter over time. The same martingale machinery can support both interfaces, but a confidence sequence does not choose the action either. Estimation, evidence and authority remain separate.

The smallest numerical example

Let the acceptable harm rate be 5 per cent, alpha be 0.05, and the alternatives be 0.10, 0.20 and 0.30 with equal weight. The abstention boundary is therefore 20. Feed the gate the sequence 0, 1, 0, 0, 1, 0, 1, 1, 0, 0, 1, 0. A safe observation usually reduces evidence because it is more compatible with the acceptable rate. A harm can increase it sharply. The mixture crosses 20 at step 7.

One claim gets one evidence account

Anytime-validity removes the cost of repeated looks inside an account. It does not grant free restarts. Suppose a runtime restarts after every hundred actions and its monitor resets evidence to one each time. Each reset opens another chance to cross the 1/alpha boundary. The system has converted one long test into a portfolio of tests. If those tests each receive alpha without allocation, the fleet-level false-alarm promise is gone.

Two multiplicities, two controls

Long-horizon monitoring has a vertical axis and a horizontal axis. Vertical multiplicity is repeated inspection of one evolving claim. The e-process handles that axis. Horizontal multiplicity is the number of claims, accounts or missions that may trigger action. That axis still needs allocation. A simple conservative design gives account i a budget αᵢ and requires the sum of all αᵢ in the governed portfolio to stay below a declared total. More powerful e-value and online-testing methods can be used when the portfolio structure warrants them, but the accounting cannot disappear.

This produces a useful rule for architecture reviews: scale reads freely; scale accounts deliberately. Dashboards, callbacks and human inspections can all read the same persisted peak evidence. A new subgroup monitor, alternative harm definition or post-remediation null receives a distinct account identity and an explicit budget. The ledger should therefore record both the evidence state and its parent allocation.

The distinction also prevents a common misunderstanding about retries. Retrying a reversible tool call may remain inside the same claim if the observation model and event definition already cover retries. Resetting the evidence because the retry “starts a new attempt” is not statistically neutral. It is a new account unless the original evidence construction explicitly treats that reset as part of one valid process.

From evidence to an abstention right

Evidence is not authority

An e-process can justify a statistical statement: under the declared acceptable-operation null, a boundary crossing this extreme would occur with probability at most alpha over the whole monitoring horizon. It cannot decide whether the proper response is to ask a human, retry a reversible tool, enter read-only mode, roll back a transaction, or halt a machine. The evidence gate proposes abstention; an authority kernel chooses the permitted action.

This separation matters because identical evidence can face different consequences. A tool call that drafts an email is reversible before send. A payment release may be difficult to reverse. A physical actuation may be irreversible. Statistical evidence should not silently encode those loss functions. Nor should a low verifier score bypass a deterministic safety invariant. If identity is missing, authority expired, or a typed action violates policy, the system should stop by rule. The e-process is for uncertain cumulative evidence, not for debating a known prohibition.

The Abstention Evidence Contract

The practical contribution of this paper is an Abstention Evidence Contract, or AEC. It binds the theorem to an operating system. The smallest useful contract has seven fields:

1. Governed claim: the exact null, such as “conditional material-failure risk is at most 2 per cent”.
2. Event and filtration: what counts as harm, when it becomes observable, and which prior information may shape the update.
3. Evidence process: the formula, alternatives, weights, software version and calibration data if any.
4. Account identity: a durable identifier that survives runtime restart and prevents duplicate observations.
5. Budget and boundary: alpha, the resulting threshold, and any portfolio allocation across monitors.
6. Authority mapping: the typed action permitted after crossing, including deterministic overrides.
7. Receipt and recovery: the evidence snapshot, decision, readback, investigation and conditions for opening a new account.

This is practitioner synthesis, not a published statistical theorem. Its purpose is to prevent a mathematically valid process from becoming operationally invalid through missing labels, duplicate updates, silent reset or ungoverned action.

A configured-system walkthrough

Consider a synthetic agent that reconciles two ledgers and proposes a typed adjustment. The model can read permitted records and call a deterministic comparison tool. It cannot post an adjustment directly. A separate executor checks identity, amount limits and approval state. After execution, an independent readback labels the outcome harmful only when the posted account, currency or amount differs materially from the authorised adjustment. That event definition is narrower than “the verifier looked worried”, but it is auditable.

The AEC declares p₀ as the maximum conditional rate of such verified mismatches allowed by the assurance policy. Every completed readback receives a monotonically increasing sequence number. Delayed readbacks update the account when they become available, under a construction that has been reviewed for that delay pattern. Duplicate messages are rejected. The evidence snapshot is written outside the agent runtime. A container restart can therefore restore the account rather than reopen the hypothesis.

Suppose the current e-value reaches 7.4 while the threshold is 20. The system may display concern, increase diagnostic sampling or require a second readback, provided those adaptations are valid under the declared filtration. It may not state that acceptable operation has been statistically rejected. When the peak crosses 20, the gate emits a receipt containing account identity, null version, evidence version, sequence, current and peak e-values, alpha, event lineage and threshold. The authority kernel then maps that receipt to “hold further postings, retain read access, preserve pending work and request review”.

Investigation may find a real executor defect, a labelling defect or a rare acceptable fluctuation. All three outcomes matter, but none rewrites the old evidence history. A repair changes the configured system or event path. A reviewed release opens a new account linked to the old receipt and receives a fresh allocation. This continuity is what turns optional stopping mathematics into recoverable execution.

Choosing p₀, alpha and the alternative mixture

The acceptable rate p₀ should come from a risk requirement, established baseline or explicit tolerance. Estimating p₀ from the live stream and then testing against the convenient estimate can invalidate the claim or make it vacuous. When uncertainty in the baseline is material, use a construction that accounts for it, a holdout calibration period, or a composite null designed for that uncertainty.

Alpha expresses the cost of falsely rejecting acceptable operation at the account level. The alternatives q determine efficiency. A q just above p₀ is sensitive to small persistent degradation but may need many observations. A larger q grows rapidly after severe failure but loses wealth during ordinary safe periods. A mixture spreads this risk across several plausible shifts. The supplied grid is illustrative, not universally optimal. Simulation under predeclared safe and harmful scenarios should evaluate detection probability, delay and operational cost before deployment.

Choosing q after seeing which live path looks most dramatic would create another selection problem. Adapting the betting strategy can be valid, but the strategy must be predictable from past information and constructed to preserve the supermartingale condition. “Adaptive” is not a synonym for “post-hoc”.

A decision surface, not one universal threshold

The statistical boundary answers whether evidence against one null has crossed. The action surface also considers reversibility. The matrix below is intentionally conservative. “Watch” is telemetry, not a claim that the null has been rejected. A hard invariant can stop the system in any cell.

What it tests. The artefact tests a stream of verified binary harms against the composite conditional null that risk never exceeds p₀. It encodes a grid of fixed alternatives above p₀, equal mixture weights, one ordered account and a terminal crossing at 1/alpha. It persists its component log-evidence so a process restart does not become a statistical restart.

How to apply it. Define the event before observing the stream. Choose p₀ from a policy or assurance requirement, not from the same data being tested. Choose alpha from the cost of falsely stopping an acceptable system and from the wider portfolio budget. Run the gate after each verified outcome. Persist the snapshot and attach the receipt to the abstention action.

What results permit. A positive result, meaning boundary crossing, permits the bounded claim that the acceptable-operation null has been rejected by this procedure at the configured ever-crossing error level. A negative result means only that this gate has not accumulated enough evidence. It does not prove the system is safe. The gate also cannot establish whether a harm label is correct, whether the monitor is aligned with the relevant failure, or whether the chosen response is proportionate.

def observe(self, harmful: bool, sequence: int) -> DecisionReceipt:
    if self.state is GateState.ABSTAIN:
        raise RuntimeError("terminal gate")
    if sequence != self.sequence + 1:
        raise ValueError("out-of-order or duplicate observation")

    x = 1 if harmful else 0
    for i, q in enumerate(self.spec.alternatives):
        if x:
            self.component_log_e[i] += math.log(q / self.spec.acceptable_rate)
        else:
            self.component_log_e[i] += math.log(
                (1 - q) / (1 - self.spec.acceptable_rate)
            )

    self.log_e = logsumexp([
        math.log(w) + component
        for w, component in zip(self.spec.weights, self.component_log_e)
    ])
    self.peak_log_e = max(self.peak_log_e, self.log_e)

    if self.peak_log_e >= -math.log(self.spec.false_alarm_budget):
        self.state = GateState.ABSTAIN
    return self.receipt(harmful, sequence)

What the supplied artefact demonstrates

With its default p₀ = 0.02, alpha = 0.05 and five-point alternative mixture, the companion's synthetic null check produced an ever-alarm frequency of 0.024 over 1,000 simulated safe streams of 500 observations. In one shifted demonstration, the stream changed from 0.02 to 0.12 after step 120 and the gate crossed at step 222 with peak e = 26.29. These values are seed-specific illustrations. Re-running them can produce different empirical frequencies and delays. The theorem is the supermartingale guarantee, not the output of one Monte Carlo run.

The simulation has two practical uses. It can catch implementation errors, such as applying the wrong safe-outcome multiplier or resetting the running maximum. It can also compare alternative mixtures before deployment. It cannot validate the real harm labels, prove the conditional null, or show that live data satisfy the synthetic Bernoulli model. A monitor that passes simulation but receives selected or corrupted outcomes remains invalid.

stateDiagram-v2
  [*] --> Claim
  state "A fixed test hides a fixed<br/>stopping rule" as Claim
  state "Why repeated peeking spends<br/>the budget" as Test
  state "A false-alarm budget is not an<br/>alert-rate target" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Where the guarantee stops

A valid equation can monitor the wrong world

The e-process guarantee is conditional on the null, event stream and filtration used to construct it. It says nothing about failures the sensor cannot see. A monitor of tool-call syntax may remain perfectly calm while an agent pursues the wrong objective. A conformity score can remain uniform across a distribution change that is cryptic to that score. Recent work gives explicit examples of this “conformal blindness”, where exchangeability breaks but the monitored conformal values do not reveal it.[13] Sequential risk monitors have been proposed for deployed model loss streams, but their guarantees still inherit the definition and availability of the monitored loss.[10] The lesson is general: validity controls false alarms for the chosen signal; it does not guarantee diagnostic completeness.

Five break conditions

First, event ambiguity. “Bad step” is not a measurement specification. The label must identify severity, attribution window, verification path and handling of unresolved outcomes. Delayed labels are not automatically invalid, but the update rule must respect when information becomes available. Outcome-delay research continues to refine which augmentations preserve martingale structure, underscoring that apparently harmless data engineering can change validity.

Second, adaptive censoring. Dropping inconvenient observations, replaying selected failures, or allowing the model to decide which outcomes are scored can violate the conditional expectation bound. Duplicate prevention and ordered sequence numbers are statistical controls, rather than only data-quality conveniences.

Third, estimated evidence. A true likelihood ratio supplies an exact test martingale under its model. A learned density ratio may not. A 2026 preprint on agent verifiers therefore adds a high-probability calibration procedure because the raw estimated process can miss its false-alarm target.[12] This is promising current evidence, not a universal licence to wrap any score in an exponential and call it an e-process.

Fourth, multiplicity across scope. Anytime-validity handles optional stopping within one stream. Multiple harms, subgroups, agents, tenants or repeated post-recovery accounts create horizontal multiplicity. Allocate alpha, use a valid multiple-testing procedure, or state that the guarantee is per account. Vovk and Wang develop ways to combine e-values, while the broader online-testing literature addresses streams of hypotheses.[6]

Fifth, efficiency. A valid gate can be too slow to matter. Alternative choices, mixture weights, batching and signal quality determine growth under real deterioration. Safe testing formalises growth-rate considerations, and confidence-sequence work analyses time-uniform width.[9][5] Validity limits false alarms. It does not promise prompt detection.

Failure injection for the monitor, not only the agent

A release test should attack the AEC as a stateful protocol. Send the same harm event twice and confirm that only one update is accepted. Deliver sequence 42 before 41 and verify that the gate refuses to guess the order. Restart the runtime after a large but sub-threshold e-value and confirm that the restored account produces the same next value as an uninterrupted run. Lose the snapshot write after an external effect and verify that recovery reconciles the evidence receipt before processing another event.

Next attack the observation model. Delay all safe labels while delivering harms immediately. Suppress labels for one subgroup. Change the verifier while leaving the evidence version unchanged. Feed a learned density ratio from a calibration distribution that no longer matches the live score process. Each test should either preserve the stated condition by design, block the update, or invalidate the account visibly. Quiet continuation is a failure.

Finally attack scope. Clone the monitor into ten workers, create two severity definitions, and roll the system through a repair. The budget ledger should reveal how many accounts now exist and what parent allocation each consumes. This is where many seemingly sound implementations fail: the inner process is valid, but orchestration duplicates the opportunity to stop.

A minimal acceptance record therefore contains four results: null simulation below its tolerance with uncertainty reported, harmful-shift detection within an operational delay target, invariant preservation under persistence faults, and explicit rejection of unmodelled observation paths. None proves the deployment safe. Together they show that the statistical claim survived contact with the runtime.

Strong alternatives and their proper place

A Bayesian decision rule can combine prior beliefs, likelihoods and asymmetric losses, including the cost of waiting. Its posterior threshold is not automatically a frequentist ever-false-alarm guarantee, and optional stopping properties depend on the model and interpretation. A CUSUM or sequential likelihood-ratio test can be more efficient when pre-change and post-change distributions are well specified. Group-sequential or alpha-spending designs can be preferable when interim looks are controlled. Conformal test martingales are attractive when exchangeability is a meaningful null and a diagnostic conformity score is available.[11]

The right comparison is therefore not “e-values versus everything else”. It is: which null, horizon, data mechanism, loss function and adaptation pattern does the system actually have? Anytime-validity earns flexibility by demanding an evidence process whose validity survives that flexibility. When a fixed schedule is real, use it. When a parametric change model is trustworthy, exploit it. When the signal is weak or missing, no stopping formalism can manufacture diagnostic information.