A number is the output of a rule

Suppose an unknown quantity is called θ. It might be a defect rate, a sensor bias, a regression coefficient or the probability that a classifier is correct. We observe data X, then apply a rule T to obtain an estimate: θ̂ = T(X).

The notation is intentionally plain. It says that the estimate is not sitting inside the data waiting to be read. A procedure creates it. Change the procedure, the sample, the target or the assumptions connecting them, and the meaning of the same numeral changes.

Before measuring precision, fix the target. A factory might care about the fraction of components currently defective, the probability a newly produced component will be defective, the average defect rate over a week, or the causal effect of changing a machine setting. Those are different estimands. A sample proportion can be a sensible estimator for one and a biased proxy for another.

This distinction matters in machine systems too. “Accuracy” may refer to the mix of cases in a benchmark, while the operational target is error among cases that cross a decision threshold next month. The number may be estimated perfectly for the benchmark and still answer the wrong question. Sampling uncertainty cannot represent target mismatch unless the mismatch is made part of the model.

Bias and variance then describe different ways an estimator can miss. Bias is systematic displacement across repetitions. Variance is instability from sample to sample. Mean squared error combines both, but the combination depends on the scale and loss that matter. A slightly biased estimator may make better decisions if it reduces variance enough. That does not refute unbiasedness results. It changes the comparison class and the decision criterion.

Stopping rules belong to the experiment as well. If inspection stops the moment eight defects are found, then “eight defects in ten observations” arose under a different process from a fixed sample of ten. Likelihood-based arguments can sometimes accommodate optional stopping under carefully specified conditions, but a generic spreadsheet formula cannot infer the design after the fact. Uncertainty begins with an estimand and a sampling contract, not with a standard-error function.

Now change one more variable. Keep the ten unique inspections, but let each component contribute ten photographs. Those images may contain some new information about surface condition, yet they do not create one hundred independent components. A useful analysis becomes hierarchical: variation between components is separated from variation between images of the same component. Treating both levels as interchangeable exaggerates precision; discarding all repeated images wastes signal.

This is why effective sample size is not a mystical discount applied at the end. It summarises a dependence structure that should be modelled at the level where it arises. The same principle covers patients within hospitals, transactions within accounts, tokens within documents and repeated outputs from one stochastic model run. The unit that receives the intervention or defines the target is usually the unit that must anchor the uncertainty calculation.

Likelihood asks what the experiment could have produced

Given observed data x, the likelihood function compares candidate parameter values by asking how compatible each is with the observed sample: L(θ; x) = p(x | θ). It is not the probability that θ is true. It is a function of θ after the data have been fixed.

For eight positives in ten Bernoulli trials, ignoring a constant that does not depend on p, the likelihood is p⁸(1-p)². It peaks at 0.8. Yet the width of the peak matters as much as its location. A broad peak says many alternatives fit nearly as well. A narrow peak says the experiment has separated them more sharply.

Fisher's likelihood programme made this local shape central to estimation.[1] But even a perfect likelihood calculation remains conditional. It assumes that the family p(x | θ) contains a useful description of the experiment, that the sampling units and censoring process are represented, and that the target parameter is identifiable within that family.

An estimate can be computationally exact and epistemically underdetermined. That distinction is the bridge to Fisher information.

flowchart LR
  A["The estimate hides the experiment"] --> B["Curvature makes alternatives<br/>separable"]
  B --> C["Confidence must earn its frequency"]
  C --> D["The confidence warrant"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#007D68,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Move the parameter and watch the distribution move

Imagine turning a hidden dial by a tiny amount. If the observable distribution changes noticeably, data can reveal the turn. If the distribution barely moves, nearby dial settings remain confusable. Fisher information formalises that local sensitivity.

For one observation X, the score is the slope of the log likelihood with respect to the parameter:

Under regularity conditions, the Fisher information is both the expected squared score and the negative expected curvature of the log likelihood:

Large information means that small parameter changes leave a larger statistical trace. For independent observations, information usually adds: Iₙ(θ) = nI₁(θ). This is why precision often improves at the familiar square-root rate. It is also why dependence, duplicated records and hidden common causes matter. If samples do not contribute independent traces, the simple addition rule fails.

There is another route to the same intuition. Compare the distribution at θ with the distribution at a nearby value θ + δ. For a small displacement, their Kullback-Leibler divergence is approximately one half of δ²I(θ) in one dimension. Fisher information is therefore the second-order rate at which neighbouring statistical worlds pull apart. The approximation is local. Large moves can encounter curvature, boundaries or separate modes that a quadratic neighbourhood cannot describe.

The lower bound is a conditional comparison

Rao and Cramér established a celebrated result: under specified regularity conditions, the variance of an unbiased estimator cannot be smaller than the inverse Fisher information.[2][3]

The inequality is powerful because it separates the experiment from the estimator. Information says what precision the experiment makes available locally; efficiency asks how much of it an estimator uses.

It is not a universal speed limit. Bias changes the bound. Boundaries and discrete parameters violate common regularity conditions. Nuisance parameters can consume information. Mixture models can be singular. A finite sample can be strongly non-normal even when asymptotic theory eventually behaves. Efron and Hinkley also showed why observed curvature can sometimes describe the realised sample more appropriately than expected information in one-parameter settings.[4]

With several parameters, the matrix view becomes operational. Suppose a vibration model contains both fault severity and sensor gain. Increasing either can raise the measured amplitude. If the experiment observes only amplitude under one operating condition, the two parameter effects point in nearly the same direction. The information matrix then has one strong eigenvector and one weak one. Severity may appear precise only because gain was silently fixed.

Add a reference signal with known amplitude, or observe the machine under a second load. The effects rotate apart. This is experimental design expressed geometrically: choose observations that enlarge information in the decision-relevant weak direction, rather than only the trace or determinant by habit. An optimal design for estimating every parameter may differ from a design optimised to decide whether severity crosses a maintenance threshold.

The smallest eigenvalue often tells a more uncomfortable story than the dataset size. It identifies the combination of parameters for which the experiment offers the least resistance to confusion. Reporting only marginal standard errors can hide that joint ambiguity, especially when downstream actions depend on a contrast of parameters.

The practical lesson is direct. Before asking which estimator has the smallest reported error, ask whether the experiment creates a non-flat likelihood in the directions the decision cares about. A large dataset can still have a nearly singular information matrix. A small, well-designed intervention can be more valuable than another million passive records.

An interval is a procedure before it is a pair of endpoints

A 95% confidence interval is often read as though the fixed parameter has a 95% chance of lying inside the observed interval. In a frequentist construction, the probability statement belongs to the procedure across repeated samples: under the model, intervals generated by that rule cover the fixed parameter at the advertised rate.

A Bayesian credible interval makes a different conditional statement. Given a prior, likelihood and observed data, it contains a stated amount of posterior probability. Under repeated use, Bayesian intervals may also have good frequentist coverage, or may not, depending on the model, prior and regime. The labels should not be blended. The decision maker needs to know whether the promise is posterior probability, repeated coverage, predictive coverage, or some combination that has actually been checked.

Bootstrap intervals replace an analytic sampling distribution with resampling from the observed data. This can be valuable when formulas are awkward, but resampling cannot recreate population structure absent from the sample. A naive bootstrap of dependent time points breaks their ordering; a row-wise bootstrap of clustered observations breaks clusters. Conformal prediction offers finite-sample marginal coverage under exchangeability, yet that guarantee can still be weak for rare subgroups or under shift. Every method moves assumptions; none abolishes them.

Forecast calibration is agreement between quoted probabilities and frequencies

For binary forecasts, a strong intuitive condition is: among cases assigned probability q, the event should occur with frequency q. In notation, P(Y=1 | Q=q)=q. Dawid's account made this repeated-use idea precise for sequential probability forecasts.[6]

Calibration does not say that an individual event is “30% true”. Nor does it identify the mechanism that produced the forecast. A weather model, a lookup table and a strategically post-processed classifier can all be calibrated on one distribution.

Proper scoring rules reward both honest probability reporting and useful concentration. The Brier score for a binary event is the mean squared difference between forecast probability and outcome. Murphy decomposed it into reliability, resolution and uncertainty, while later work placed proper scoring rules in a broad decision-theoretic framework.[7][8] Calibration is one component of forecast quality, not its synonym.

Calibration is conditional on what you group and where you deploy

Expected calibration error, or ECE, groups predictions into bins and averages the gap between confidence and accuracy. It is useful as a descriptive summary, but its value changes with binning, sample size and the calibration notion being tested. Multiclass systems add choices: confidence calibration, classwise calibration, top-label calibration or calibration of the full predictive distribution are different claims.[11][12]

The audit itself has sampling uncertainty. A reliability point based on twelve cases should not be read like one based on twelve thousand. Show bin counts or uncertainty bands, repeat the result under defensible binning choices, and use a proper score alongside the diagram. A tiny ECE can result from coarse bins that average over a wavy calibration curve. A large ECE can be mostly estimation noise when bins are sparse.

Thresholds change what matters. If action begins at 0.65, excellent calibration between 0.05 and 0.30 cannot compensate for systematic overconfidence around 0.65. Local calibration, ranking quality and the cost of false action should be examined near the decision boundary. This is where a statistical diagnostic becomes a decision instrument rather than a beauty contest between aggregate metrics.

A model can also be calibrated overall and fail within a subgroup. Suppose half the cases come from Site A, where forecasts of 0.7 realise at 0.9, and half from Site B, where they realise at 0.5. Combined, the frequency is 0.7. Aggregate calibration conceals equal and opposite errors.

Distribution shift changes the conditional relationship being audited. Guo and colleagues reported substantial miscalibration in several modern neural classifiers and found temperature scaling effective in their studied settings.[10] Later work by Minderer and colleagues found that newer image architectures could be among the better calibrated models and that earlier trends did not transfer cleanly.[13] The apparent disagreement is productive. Calibration is a configured-system property measured on a stated distribution, not a permanent trait of “neural networks”.

stateDiagram-v2
  [*] --> Claim
  state "A number is the output of a rule" as Claim
  state "Likelihood asks what the<br/>experiment could have produced" as Test
  state "Move the parameter and watch<br/>the distribution move" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

From confidence score to confidence warrant

A dashboard usually displays the final number and hides the chain that makes it trustworthy. The proposed confidence warrant reverses that design. It is a compact, versioned record that travels with an estimate or probabilistic model. It does not certify truth. It states why the uncertainty claim is currently decision-adequate and what would revoke that judgement.

The proposal is a practitioner synthesis of classical estimation, information geometry, metrological uncertainty and calibration research. Its novel contribution is not another scalar confidence score. It is a breakable chain whose failure location determines the remedy.

Each step should carry one of four statuses: supported, conditionally supported, failed or untested. “Untested” must remain distinct from “passed”. A release may tolerate an untested item only when the associated action is reversible, the potential loss is bounded and monitoring can detect harm before escalation. Irreversible or high-cost actions require stronger evidence at the same uncertainty level.

The warrant also has a version boundary. Change the sensor firmware, feature pipeline, model weights, calibration set, action threshold or population mix and identify which claims must be rerun. This avoids a common failure in compound systems: an old calibration plot survives while the configured system that produced it has disappeared.

A configured-system example

Consider a synthetic maintenance system that estimates the chance a machine will fail within seven days. It uses vibration summaries from two plants. A score above 0.65 creates a technician review; below 0.15 defers inspection.

Validation reports an ECE of 0.02 and strong accuracy. The confidence warrant asks what those summaries omit. Plant North samples vibration every minute. Plant South uploads a daily maximum after a local filter discards low-amplitude windows. The model is globally calibrated because North dominates the validation set. At South, distinct degradation patterns collapse into the same maximum. This is partly an information problem, rather than only a probability-mapping problem.

A calibration-only response applies temperature scaling to all scores. Aggregate ECE improves. South's false deferrals remain. The warrant instead triggers a channel intervention: preserve two additional quantiles at South, then reassess the information matrix for degradation parameters. Only after the representation separates the relevant states does local recalibration become meaningful.

Do not calibrate away a measurement failure. Conversely, do not buy new sensors when the data already separate outcomes and the only defect is an overconfident probability map.

The system should be failure-injected before release. Duplicate records to test whether precision incorrectly follows row count. Saturate one feature channel to see whether confidence remains high after distinguishability collapses. Shift the plant mix to test whether aggregate calibration was cancellation. Remove labels near the action threshold to reveal whether the audit depends on convenient missingness. Each injection targets one link in the warrant.

Decision utility must be checked separately. A recalibrated model can move many cases across 0.65, increasing review load even while ECE improves. Report the confusion costs, capacity constraints and abstention volume produced by the new mapping. A probability forecast is an input to a policy; calibration does not choose that policy.

from __future__ import annotations

from dataclasses import dataclass
from math import sqrt
from random import Random
from statistics import fmean, pvariance

Z_95 = 1.959963984540054

@dataclass(frozen=True)
class Result:
    empirical_variance: float
    cramer_rao_bound: float
    wald_coverage: float
    wilson_coverage: float
    fisher_information: float


def binomial(rng: Random, n: int, p: float) -> int:
    return sum(rng.random() < p for _ in range(n))


def wald_interval(x: int, n: int) -> tuple[float, float]:
    phat = x / n
    half = Z_95 * sqrt(phat * (1 - phat) / n)
    return max(0.0, phat - half), min(1.0, phat + half)


def wilson_interval(x: int, n: int) -> tuple[float, float]:
    phat = x / n
    z2 = Z_95 ** 2
    den = 1 + z2 / n
    centre = (phat + z2 / (2 * n)) / den
    half = Z_95 * sqrt(phat * (1 - phat) / n + z2 / (4 * n * n)) / den
    return max(0.0, centre - half), min(1.0, centre + half)


def simulate(n: int = 20, p: float = 0.08,
             repetitions: int = 20_000, seed: int = 2020) -> Result:
    if not (n > 0 and 0 < p < 1 and repetitions > 1):
        raise ValueError("Require n > 0, 0 < p < 1, repetitions > 1")
    rng = Random(seed)
    estimates: list[float] = []
    wald_hits = 0
    wilson_hits = 0
    for _ in range(repetitions):
        x = binomial(rng, n, p)
        estimates.append(x / n)
        lo, hi = wald_interval(x, n)
        wald_hits += lo <= p <= hi
        lo, hi = wilson_interval(x, n)
        wilson_hits += lo <= p <= hi
    information = n / (p * (1 - p))
    return Result(
        empirical_variance=pvariance(estimates),
        cramer_rao_bound=1 / information,
        wald_coverage=wald_hits / repetitions,
        wilson_coverage=wilson_hits / repetitions,
        fisher_information=information,
    )


if __name__ == "__main__":
    print(simulate())

The smallest useful implementation

A team does not need an information-geometry platform to begin. One page is enough: name the target and sampling unit; plot the likelihood or a perturbation response for the critical parameter; compare one naive and one boundary-aware interval in simulation; draw a reliability diagram with counts; slice the action threshold by the two most consequential groups; state one expiry trigger.

A positive result permits bounded use, not general trust. It says that the configured procedure met specified separation, coverage and calibration checks in the tested regime. A negative result is valuable because it routes work. Flat perturbation response calls for measurement redesign. Adequate separation with poor coverage calls for a different uncertainty construction. Good validation calibration that decays later calls for monitoring, recalibration or abstention under shift.

What would weaken the warrant itself? If teams cannot assign failures reliably to its links, or if the links interact so strongly that routing produces worse decisions than end-to-end validation alone, the decomposition should be revised. End-to-end operating evidence remains necessary even when every component check passes.

What the warrant cannot establish

The framework does not decide between frequentist and Bayesian foundations. A posterior can be part of the uncertainty construction, and its predictive consequences can still be checked for calibration. Nor does Fisher information exhaust experimental design. It is local, model-relative and often asymptotic. Decision-focused designs may care about a threshold or loss rather than uniform parameter precision.

Nonlinearity can also make simple uncertainty propagation unreliable. Metrology guidance explicitly distinguishes model construction from uncertainty evaluation, and the 2026 amendment to the Guide to the Expression of Uncertainty in Measurement points to Monte Carlo or higher-order treatment when nonlinearity is significant.[14][15] The warrant should therefore record how uncertainty was propagated, rather than only its final standard deviation.

Finally, calibrated confidence is not consciousness, understanding or felt certainty. A machine can emit well-calibrated probabilities through a statistical procedure without possessing any corresponding experience. This paper concerns the warrant for action under uncertainty, not the metaphysics of the estimator.