Abstention changes the action space

A conventional classifier maps an input x to a prediction. A selective classifier adds a selection function that decides whether the prediction is exposed. That addition looks small, yet it changes what system quality means. Once the model may withhold a prediction, the evaluator must ask which cases remained, which cases left, what happened to them next, and whether the selection rule concentrated harm anywhere.

The vocabulary matters because four operational states are often collapsed into one friendly phrase such as “I’m not sure”. Abstention is the family; gather, defer and decline are distinct members with different authority and recovery paths.

Thought experiment: hold probability constant, change consequence

Consider two systems with the same estimated probability of correctness, 0.82. The first suggests a spelling correction in an internal draft. The second tells a customer that a disputed payment is ineligible for refund. Keep the model, evidence and probability fixed. Change only the consequence of an error.

A universal confidence threshold must treat the cases alike. A decision-aware selector does not. The spelling suggestion is cheap to reverse and easy to inspect, so answering can have lower expected loss. The refund statement may create financial and conduct harm, while a case-record read or qualified review may be available. The system should therefore gather or defer even though epistemic confidence is identical.

The experiment exposes a missing variable. Confidence estimates the chance of correctness under some calibration regime. It does not price an incorrect action. It also does not price delay, review, privacy exposure or customer interruption. Acceptability is a relation among uncertainty, consequence and alternatives, rather than a property of confidence alone.

Risk, coverage and the missing denominator

Selective classification gives a clean starting point. Let f(x) be the model prediction, y the correct label, ℓ a loss function, and g(x) a selector that equals one when the model answers. Coverage is the expected fraction of accepted cases. Selective risk is the loss among those accepted cases.

El-Yaniv and Wiener named this the risk-coverage trade-off and analysed its foundations. Geifman and El-Yaniv later showed how a deep classifier could use a held-out sample to choose a threshold for a desired risk level under stated assumptions. SelectiveNet then learned prediction and selection jointly. These are meaningful mechanisms, not permission to report “accuracy after removing hard cases” without the removed population.

A minimal worked example makes the denominator visible. Suppose a service receives 100 cases. At full coverage it makes 15 mistakes. A selector rejects 20 cases and leaves only five mistakes among the 80 accepted cases. Selective risk improves from 15 per cent to 6.25 per cent. That is useful if the twenty rejected cases receive a competent route. If they disappear into an unstaffed queue, the business has exchanged prediction error for unresolved work.

Three coverage terms should therefore remain separate. Autonomous coverage is the fraction answered by the model. Resolved coverage is the fraction eventually decided by any authorised route. Release coverage is the fraction whose outcomes are verified and eligible for the intended effect. A dashboard that calls all three “coverage” can make a selective model appear healthier than the service around it.

flowchart LR
  A["A confidence score at the<br/>payment gate"] --> B["Abstention changes the action<br/>space"]
  B --> C["Confidence is only one input"]
  C --> D["The capability has a failure<br/>surface"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#007D68,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Confidence is only one input

Teams often respond to the need for abstention by calibrating model confidence and setting a threshold. Calibration is valuable. Guo and colleagues showed that modern neural networks can be poorly calibrated and that temperature scaling was effective on many of their evaluated datasets. Yet calibration answers a different question from selection.

A calibrated score of 0.8 means that, among comparable predictions receiving that score, roughly 80 per cent are correct under the calibration population and definition. A useful selection score must also rank cases so that errors concentrate below the threshold. A model can be calibrated and still provide no ranking power inside a score band. Conversely, a score can rank errors well while its numerical probabilities are miscalibrated.

Thought experiment: a perfectly calibrated coin

Imagine a model that assigns exactly 0.8 confidence to every case and is correct on exactly four of every five cases. It is perfectly calibrated at that score. Now permit it to abstain on 20 per cent of cases, but give the selector no additional feature. Any selected subset has the same expected 20 per cent error. Coverage falls; selective risk does not.

Change only one feature: attach a score that orders the eventual errors below the correct cases while preserving the same overall accuracy. Abstention can now lower retained-case risk. The difference is not “more confidence”. It is information about relative error likelihood. A release gate should test calibration and error ranking separately.

Calibrated Selective Classification goes further by asking whether uncertainty estimates remain calibrated on the accepted subset. This matters because selection changes the population on which probabilities are interpreted. The work is an important warning against assuming that a calibrated base model stays calibrated after a selector filters inputs.

Comparative route loss

Classical reject-option models treat rejection as a cost. Learning-to-defer work adds a more realistic idea: the downstream decision-maker has input-dependent competence. Madras, Pitassi and Zemel modelled a two-stage system that could pass cases to another decision-maker. Mozannar and Sontag developed consistent estimators for choosing between model prediction and an expert. Later work extended the setting to multiple experts, limited expert labels, cost asymmetry and workload constraints.

The conceptual shift is decisive. A difficult case should not be deferred merely because the model is uncertain. It should be deferred when the alternative route has lower expected consequence after its cost and delay are included. A radiologist may be better on one image pattern and worse on another. A fraud analyst may be excellent at identity anomalies and slow on obvious merchant disputes. A deterministic calculation may dominate both on arithmetic. The selector needs a comparative map.

This “escalation dividend” is a practitioner synthesis, not a new theorem. It forces four questions into one inspectable comparison. How likely is each route to be correct on this case? What is the asymmetric harm of each error? What does the route consume? What delay does it introduce now? Hard policy and authority rules still define which actions are feasible. Optimisation only chooses among permitted routes.

When gathering evidence dominates deferral

Gathering deserves its own route because it postpones the final choice while preserving decision authority. It can reduce uncertainty, avoid consuming scarce expert attention and reveal that the original case was outside the model’s assumptions. Yet a generic instruction to “retrieve more context” is not an abstention mechanism. The proposed observation must have a plausible route-changing effect.

A useful decision rule is the expected value of a named observation. Let z be a test, lookup or clarification that may return several results. Let A be the permitted actions, Q(a | x) the current expected total loss of action a, and C_z the full cost of obtaining z. That cost includes money, latency, privacy exposure and the chance that the result arrives too late. Define:

A device check costs 0.12 units. Suppose it has a 60 per cent chance of returning a reassuring result that supports an answer with 0.03 error probability. The remaining 40 per cent produces an adverse or unresolved result and sends the case to the same reviewer. Expected loss after ordering the check is 0.12 + 0.60 × (0.03 × 8) + 0.40 × 1.24 = 0.76. Its value is 1.24 − 0.76 = 0.48 units, so gather is preferred.

Now vary only the observation cost. If the check consumes 0.70 rather than 0.12 units, expected loss becomes 1.34 and direct deferral wins. Vary its informativeness instead: if both possible results lead to the same route with the same expected loss, the observation has no decision value even when it is statistically interesting. Information gain and decision value are different quantities. A surprising datum can leave the action unchanged; a mundane confirmation can unlock an authorised action.

The evidence contract should therefore name the proposition being tested, possible result classes, the route each result may unlock, source authority, freshness, cost, deadline and stopping rule. Repeated retrieval without a decision pivot is unresolved work disguised as research. A practical negative control replaces the observation with a random result of equal latency and cost. If measured performance still improves, the apparent benefit came from delay, selection or another route change rather than from the evidence itself.

After release, compare predicted and realised evidence value. Track how often the observation changed the route, how often it arrived after expiry, whether it merely confirmed the original proposal, and whether availability differs across relevant groups. Those records let the service retire low-value calls, reserve expensive observations for high-consequence cases and distinguish an evidence failure from a model-confidence failure.

A production-shaped worked scenario

Consider a synthetic fraud-alert service handling 10,000 alerts a day. The model can recommend release or hold. A device-evidence call can resolve some low-confidence identity cases in under a second. Sixty analysts can review a bounded queue, but their capacity changes across the day. Some alerts concern classes for which the model has little validation evidence. The service therefore implements four typed outcomes.

High-score, low-consequence alerts are answered. Cases with a named, cheap evidence gap use the device route. Cases whose expected analyst advantage exceeds queue cost are deferred with an evidence pack. Cases outside the validated population, or cases that cannot complete before an irreversible deadline, are declined for automated disposition and placed into a controlled exception process.

At 09:00 the analyst queue is short. The deferral boundary expands because review can improve outcomes cheaply. At 18:00 a campaign-driven spike pushes queue delay beyond the value of review for low-harm cases. The system gathers evidence for more cases, answers some with conservative holds, and declines a small out-of-scope segment. A static confidence threshold would send the same volume at both times and either waste scarce review or leave avoidable error on the table.

Human interaction can itself change performance. In a selective-prediction experiment using conservation data, Bondi and colleagues found that the way deferral was communicated affected human judgement; informing people that the system had deferred while withholding the model’s prediction performed differently from revealing that prediction. The result is domain-bounded, but its implication is broad: the downstream route is part of the treatment, not a fixed oracle.

Queue limits also turn per-case optimisation into allocation. If only sixty reviews are available, the system should not defer the sixty lowest confidence scores automatically. It should rank cases by marginal reduction in expected loss, subject to expertise, deadline and fairness constraints. Research such as DeCCaF models cost-sensitive allocation under synthetic analyst capacity, while OpenL2D and FiFAR provide a public environment with synthetic experts and workload constraints. These are useful evaluation assets, not evidence that a particular operating model will transfer unchanged.

The capability has a failure surface

Abstention often produces an attractive retrospective chart. Sort a test set by confidence, remove the bottom fraction, and accuracy rises. Production can break every term in that calculation. The score distribution shifts. The meaning of a threshold changes. Review queues fill. Experts face cases unlike their historical labels. The accepted subset becomes less representative. Customers learn how to shape inputs. The selector can remain deterministic while its guarantees dissolve.

Ovadia and colleagues showed that predictive uncertainty methods can degrade under dataset shift, with standard post-hoc calibration falling short in their benchmark. Later selective-classification work has explicitly addressed shifted settings, but no method removes the need to detect whether its assumptions still hold. Conformal risk control can provide distribution-free guarantees under exchangeability and extensions can relax particular conditions; those guarantees remain statements about a defined loss and data regime, not a blanket production certificate.

Marginal safety can hide subgroup exposure

A selector may meet a population risk target while concentrating accepted errors in a subgroup. Jones and colleagues found that selective classification can magnify accuracy disparities, especially where models exploit spurious correlations. Shah and colleagues showed analogous concerns for selective regression and proposed subgroup-aware criteria. The mechanism is easy to miss: if confidence is systematically less informative for one group, tightening a global threshold removes many correct cases from that group while leaving some confident errors.

This is not solved by reporting equal deferral rates. Equal rejection can coexist with unequal retained risk, unequal delay, unequal access to expert capacity and unequal costs of unresolved cases. The required audit includes coverage, selective risk and route outcomes by relevant group, task family, source condition and consequence class. Very small groups may lack enough calibration data for strong guarantees; that limitation should produce a visible uncertified state rather than a pooled claim.

A July 2026 preprint on hierarchical group-conditional conformal risk control illustrates both promise and cost. It reports benchmark-specific improvements in group risk violations, alongside substantial participation loss and cases of complete abstention. Because it is recent and not yet broadly replicated, it should inform experiment design rather than be treated as settled operating evidence.

A negative control: remove ranking and downstream advantage

A mechanism claim needs a case where it should fail. In the executable harness below, the positive condition gives the score useful error-ranking power and makes evidence or human review better on difficult cases. The negative control shuffles scores across cases and forces the human and evidence outcomes to equal the model outcome. Only those causal features change.

In the positive run, a threshold reduces autonomous selective risk and error among resolved cases. In the negative run, coverage falls from 100 to 49.5 per cent while selective risk remains essentially unchanged, and operating loss rises because deferral adds cost without competence. The route-aware policy also fails because it trusts route estimates that are false in the negative world.

This is the boundary in executable form: abstention is not intrinsically beneficial. It needs a selector that ranks avoidable error, an alternative route that can improve the outcome, and an operating model that preserves that advantage after cost and delay.

Metric choice can manufacture progress

Risk-coverage curves are necessary but not sufficient. Traub and colleagues showed that common multi-threshold evaluation choices can rank selective systems differently, motivating their generalised risk-coverage metric. Pugnana and colleagues benchmarked eighteen methods across forty-four image and tabular datasets, reinforcing that no single family dominates every setting. These results argue for a measurement stack rather than a favourite threshold metric.

stateDiagram-v2
  [*] --> Claim
  state "Thought experiment: hold<br/>probability constant, change<br/>consequence" as Claim
  state "Risk, coverage and the missing<br/>denominator" as Test
  state "Thought experiment: a<br/>perfectly calibrated coin" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Build abstention as an operating control

A production implementation needs more than a “refuse” token. It needs a control boundary that keeps unlike objects separate. Intent states what the user or process is trying to achieve. Identity names the represented subject and acting workload. World state records current authoritative facts. Context supplies permitted evidence. Reasoning proposes a conclusion. Authority determines who may decide or act. Action is the typed route. Evidence records why that route was selected. Outcome records what happened. Release and recovery decide whether the policy remains in service and how unresolved work returns.

This taxonomy prevents a familiar collapse. Model uncertainty is not missing authority. Missing context is not model uncertainty. A human queue is not a guarantee of competence. A function call is not evidence that its effect occurred. The abstention controller should route a typed deficit, not a vague feeling of doubt.

The abstention release card

The practical decision instrument below turns the mechanism into a release gate. Each row must pass for every consequence class, not merely for the average test set. A “no” does not always block the model itself; it blocks the claim that the configured escalation policy controls risk.

Make the route reconstructable

Every material abstention should produce a decision receipt. The receipt need not expose private chain-of-thought. It should retain the bounded intent, identity, world-state version, evidence available, model proposal, uncertainty or deficit type, candidate routes, policy exclusions, selected route, expected-loss inputs, queue snapshot, expiry, downstream decision, effect receipt and verified outcome.

That record serves three jobs. It lets operators recover unresolved cases. It lets evaluators distinguish model error from route error. It lets governance withdraw a selector or threshold without disabling every underlying component. A generated explanation can accompany the receipt, but the control path must be reconstructable from typed events rather than trusted prose.

Executable escalation harness

The following Python programme creates two synthetic worlds from the same seed. In the positive world, confidence ranks error and downstream routes have comparative advantage. In the negative world, scores are shuffled and the downstream outcomes are forced to equal the model outcome. The artefact compares always-answer, a fixed threshold and a route-aware policy with a human-capacity limit.

from dataclasses import dataclass
import random

@dataclass(frozen=True)
class Case:
    score: float
    model_ok: bool
    human_ok: bool
    evidence_ok: bool
    evidence_available: bool
    harm: float
    group: str

def make_cases(informative: bool, seed: int = 11, n: int = 400):
    rng, cases = random.Random(seed), []
    for i in range(n):
        d = rng.random()
        group = "B" if i % 5 == 0 else "A"
        p_model = max(.42, min(.99, .98 - .50*d - (.05 if group == "B" else 0)))
        score = max(.30, min(.99, p_model + rng.gauss(0, .04)))
        p_human = max(.75, min(.97, .92 - .04*d - (.015 if group == "B" else 0)))
        available = rng.random() < (.62 if d > .35 else .28)
        p_evidence = max(p_model, min(.99, p_model + (.30 if available else 0)))
        harm = 1 + 9*(d**1.6)
        cases.append(Case(score, rng.random() < p_model, rng.random() < p_human,
                          rng.random() < p_evidence, available, harm, group))

    if not informative:                 # negative control
        scores = [c.score for c in cases]
        rng.shuffle(scores)             # destroy error ranking
        cases = [Case(s, c.model_ok, c.model_ok, c.model_ok,
                      c.evidence_available, c.harm, c.group)
                 for c, s in zip(cases, scores)]
    return cases

def threshold_policy(cases, threshold=.75):
    accepted = [c for c in cases if c.score >= threshold]
    deferred = [c for c in cases if c.score < threshold]
    auto_errors = sum(not c.model_ok for c in accepted)
    human_errors = sum(not c.human_ok for c in deferred)
    loss = sum(c.harm for c in accepted if not c.model_ok)
    loss += sum(c.harm for c in deferred if not c.human_ok) + .25*len(deferred)
    return len(accepted)/len(cases), auto_errors/max(1, len(accepted)), \
           (auto_errors + human_errors)/len(cases), loss

def route_policy(cases, human_capacity=60):
    rows, candidates = [], []
    for i, c in enumerate(cases):
        q = {
            "answer": (1-c.score)*c.harm,
            "gather": .05*c.harm + .16 if c.evidence_available else float("inf"),
            "defer": .09*c.harm + .30,
            "decline": .18*c.harm + .40,
        }
        fallback = min(("answer", "gather", "decline"), key=q.get)
        dividend = q[fallback] - q["defer"]
        rows.append((i, fallback))
        if dividend > 0:
            candidates.append((dividend, i))

    human = {i for _, i in sorted(candidates, reverse=True)[:human_capacity]}
    outcomes = []
    for i, fallback in rows:
        c, action = cases[i], ("defer" if i in human else fallback)
        if action == "answer": ok, cost = c.model_ok, 0
        elif action == "gather": ok, cost = c.evidence_ok, .16
        elif action == "defer": ok, cost = c.human_ok, .30
        else: ok, cost = None, .18*c.harm + .40
        realised = cost if ok is not False else cost + c.harm
        outcomes.append((c, action, ok, realised))
    return outcomes

def report(label, cases):
    coverage, risk, resolved_error, loss = threshold_policy(cases)
    outcomes = route_policy(cases)
    decided = [row for row in outcomes if row[2] is not None]
    actions = {a: sum(row[1] == a for row in outcomes)
               for a in ("answer", "gather", "defer", "decline")}
    route_error = sum(row[2] is False for row in decided) / len(decided)
    route_loss = sum(row[3] for row in outcomes)
    print(label)
    print(f"  threshold: coverage={coverage:.1%}, risk={risk:.1%}, "
          f"resolved_error={resolved_error:.1%}, loss={loss:.1f}")
    print(f"  routed:    actions={actions}, resolved_error={route_error:.1%}, "
          f"loss={route_loss:.1f}")

def always_answer(cases):
    errors = sum(not c.model_ok for c in cases) / len(cases)
    loss = sum(c.harm for c in cases if not c.model_ok)
    return errors, loss

def route_metrics(cases):
    outcomes = route_policy(cases)
    decided = [row for row in outcomes if row[2] is not None]
    error = sum(row[2] is False for row in decided) / len(decided)
    subgroup = {}
    for group in ("A", "B"):
        rows = [row for row in decided if row[0].group == group]
        subgroup[group] = sum(row[2] is False for row in rows) / len(rows)
    return error, subgroup

if __name__ == "__main__":
    positive = make_cases(True)
    negative = make_cases(False)
    report("positive", positive)
    report("negative", negative)

    pos_threshold = threshold_policy(positive)
    neg_threshold = threshold_policy(negative)
    pos_route_error, pos_groups = route_metrics(positive)
    neg_route_error, neg_groups = route_metrics(negative)
    pos_always_error, _ = always_answer(positive)

    assert pos_threshold[1] < pos_always_error
    assert neg_threshold[1] > pos_threshold[1]
    assert pos_route_error < neg_route_error
    assert set(pos_groups) == {"A", "B"} == set(neg_groups)
    assert all(0 <= value <= 1 for value in (*pos_groups.values(), *neg_groups.values()))

The embedded runnable programme prints the positive case and negative control below. It also calculates an always-answer baseline and subgroup route errors, then asserts that informative ranking improves selective risk, the negative control destroys the route advantage and every subgroup metric remains bounded.

positive
  threshold: coverage=49.5%, risk=17.2%, resolved_error=14.5%, loss=289.4
  routed:    actions={'answer': 153, 'gather': 179, 'defer': 60, 'decline': 8},
             resolved_error=13.8%, loss=313.1
negative
  threshold: coverage=49.5%, risk=28.8%, resolved_error=29.0%, loss=679.4
  routed:    actions={'answer': 155, 'gather': 176, 'defer': 60, 'decline': 9},
             resolved_error=29.2%, loss=677.7

The threshold wins on the synthetic loss in the positive run because its assumed human route is strong and unconstrained by the sixty-case capacity used in the route-aware policy. That is a useful result rather than an embarrassment. It shows why every policy comparison must share the same feasible downstream service. The route-aware policy resolves 98 per cent within capacity and exposes a group gap that an aggregate loss would hide.

The negative control is more important. Both selective policies spend capacity without reducing error among resolved cases. Any method that still reports success in that world is rewarding lower autonomous coverage, not better decisions.

A staged release sequence

Begin at full coverage in observation mode. Compute risk-coverage curves retrospectively, but also replay where rejected cases would have gone. Next, run shadow routing with real queue snapshots and no changed decisions. Compare predicted route advantage with realised downstream outcomes. Then enable evidence gathering for low-consequence cases, because it usually preserves decision rights while testing the selector’s ability to identify useful information gaps.

Human deferral should follow only after the organisation measures reviewer competence, delay, anchoring and abandonment on the routed population. Decline paths need user remedies before they carry material volume. Consequential autonomous actions come last, with pre-effect freshness, idempotency, effect receipts and independent postcondition verification.

Monitor threshold drift, route mix, queue age, group coverage, high-confidence errors, unnecessary abstention, repeated evidence calls and unresolved outcomes. Recalibration alone is not enough when the model has lost ranking power or the downstream service has changed. Authority withdrawal should be targeted: a failing analyst route can be disabled while evidence gathering remains available; a shifted score can force broader deferral while the base model stays in shadow.

This paper is narrower than two adjacent designs. Before the Agent Acts governs how an agent asks, retrieves, observes and stops. Designing Human-Agent Handoffs focuses on the transfer of evidence, decision rights and accountability. Here the concern is the selector that decides whether those routes are justified at all.

Compact glossary