The strong rivalHuman intervention can make performance worse

People add delay, inconsistency, bias and noise. Unbounded veto can erase automation's value or let local preference defeat a valid policy. Constant confirmation can create fatigue and reduce attention to the rare consequential case. High human control and high automation can coexist when each is assigned to what it does well.

The rival fails when efficiency is used to retain nominal responsibility while removing effective control. A person cannot be accountable for an action they cannot detect, understand, influence or repair. Override should be allocated by consequence, uncertainty, reversibility, detection time and information available only to humans. Low-consequence bounded actions can run with sampling; irreversible or contested decisions may need pre-action or dual control.

The integrity chain also fails when discretion has no boundary. Reviewers need reasons, outcome feedback and institutional policy. If staff repeatedly reject an accurate rule for inconsistent reasons, unlimited override is not the remedy. Examine whether policy, training, incentives or the rule itself should change, then test the redesigned control.

Override Integrity ChainEight links must survive the real workload

The links interact. Independent evidence is useless when the queue allows no time to inspect it. Authority is nominal when a reviewer needs a manager who cannot respond before commitment. A stop control is misleading when it freezes the screen while queued calls continue. Training decays when operators never practise the degraded mode for which they remain responsible.

An override receipt records trigger, state snapshot, evidence available, time budget, human decision, action actually executed, downstream effects, recovery, affected-party remedy and improvement owner. It is this article's synthesis for proving what control existed in one case and which design condition failed.

The chain should be evaluated with drills, not policy review alone. Inject contradictory evidence, queue pressure, expired permissions, partial writes, unavailable managers and lost connectivity. Observe whether a person notices, acts in time and reaches a safe state. Record the gap as a product and operating defect rather than attributing it automatically to the nearest operator.

Four control momentsPlace intervention before its required protection expires

These moments are not equivalent. A reviewer who learns about a denial after the customer receives it may support appeal and remedy but did not prevent the decision. A pause issued while tool calls are in flight must identify which calls can still complete. An institutional committee can remove a use case even when each individual interaction appears valid.

Choose the earliest feasible moment that delivers the claimed protection. Consequence, irreversibility and time to detect determine whether sampling is enough, a pre-action decision is required or the use case should not be automated. Do not add confirmation to every low-consequence action merely to create a record of human presence.

Legal duties depend on role and scope. Article 14 of the EU AI Act contains effective-oversight requirements for applicable high-risk systems, including competence, authority, automation-bias awareness, ability to disregard or reverse outputs and safe interruption. Those provisions do not make every product a high-risk system or settle responsibility outside their application and timeline.

Independent reviewGive the person a route beyond the system's framing

Automation bias can produce commission errors, where a person follows incorrect advice despite contradictory information, and omission errors, where the system's failure to prompt reduces independent action. Expertise alone does not remove the risk. Interface order, time pressure, confidence cues and organisational expectation shape attention.

Independent evidence can be a primary source, raw record, alternative model, direct customer contact, specialist opinion or separate measurement. It should be decision-relevant and accessible within the intervention window. A long provenance drawer that reviewers never have time to open does not satisfy the independence link.

Cognitive forcing may help in selected tasks by eliciting an initial judgement before revealing AI advice or requiring a counter-signal. It can also add cost and reduce preference. Test whether it improves decisions under real workload. A universal pre-judgement step can become another ceremony when the human lacks information or authority.

The same-information trap requires product redesign. If both the model and reviewer lack contract jurisdiction, adding a person cannot recover it. Ask for the fact, route to someone who can obtain it or abstain. Do not treat human presence as a substitute for evidence coverage.

Workload, skill and incentivesMake intervention feasible on the worst credible day

Capacity is not an average staffing ratio. Model the arrival distribution, priority classes, interruption, orientation time, evidence access, decision duration and deadline. A queue can meet its monthly target while failing the rare event whose intervention window is minutes. Reserve capacity or stop the action when the protected lane cannot be served.

Skill requires recent practice. Reliable automation changes situation awareness and can leave people responsible for abnormal conditions they seldom encounter. Bainbridge's ironies of automation remain relevant: the harder residual tasks arrive when context is degraded and automation has shaped competence. Run degraded-mode drills, rotate practice and give operators access to system behaviour during normal work.

Incentives determine authority. If reviewers are rewarded for throughput, penalised for disagreement or require social courage to stop a senior-sponsored product, the formal right may be unusable. Measure queue ageing, time spent on evidence, disagreement, successful interruption and escaped effects without rewarding either agreement or override frequency by itself.

Fatigue is one failure mode, not the whole chain. Paper 058 allocates approval attention across actions. Here workload is tested with detection, independence, authority and safe effect. Removing confirmation volume helps only if the remaining intervention can actually change the system.

Safe effect and remedyKnow what completed after the person pressed stop

An agent sends three of five messages before interruption. The interface cannot assume the operation stopped. It needs stable action identifiers, call status, idempotency, cancellation and reconciliation. Revoke remaining authority, identify messages already delivered, inspect downstream record changes and prevent retries from recreating the effect.

Rollback restores system state only where external reliance is reversible. A sent message may require correction and direct contact. A denied customer may need record repair, reconsideration and compensation. The override receipt distinguishes technical rollback from affected-party remedy and keeps unresolved effects open.

A safe state is defined before launch. It states which calls may finish, which data remain authoritative, who owns pending work and how the user knows interruption succeeded. Tests include duplicated events, network loss, delayed suppliers, partial writes and unavailable notification channels. A successful pause animation proves nothing about these states.

Accountability boundaryDo not place institutional failure on the nearest human

Elish's moral crumple zone describes how a human can absorb responsibility for a complex autonomous system despite limited control. A trace that records only “approved by operator” reproduces the problem. The override receipt preserves system recommendation, evidence, queue, time, authority, executed action and design conditions.

Formal decision rights remain important, but Paper 010's allocation must be exercisable. Product teams control defaults and state, operations controls staffing and escalation, procurement chooses dependencies, leaders approve the use case and the institution funds remedy. Accountability follows these decisions as well as the operator's judgement.

Affected people need contest and remedy even when no individual acted negligently. Correct the record and consequence, explain the route available and change the design that made control impossible. Review should ask why the chain failed before treating agreement as consent or disagreement as misconduct.

Ten boundary casesTest the weakest link under real conditions

Interactive instrumentOverride Reality Check

The Reality Check evaluates consequence, timing and all eight integrity links. It returns one control route with failed links and changes. It never emits a maturity or control score and never treats a button as evidence.

Five control routesChoose the mechanism that can alter the real path

Allow bounded automated action with sampled oversight for low-consequence rules with enforced limits, discoverable effects and proven recovery. Sample outcomes and boundaries rather than demand universal approval. Escalate when novelty or consequence changes.

Require a pre-action human decision when contextual judgement or private human information matters and one competent reviewer can decide before commitment. Hold the action, expose independent evidence and make disregard, modification and stop authority usable.

Require dual control or qualified specialist escalation when consequence, role separation or expertise exceeds one reviewer. Both decisions need independent value; two hurried clicks on the same record are not dual control.

Redesign the workflow so intervention is feasible and reversible when detection, orientation, capacity, authority or safe effect fails but useful automation can remain below a narrower boundary. Move review earlier, reduce queues, expose state, separate evidence, constrain tools and engineer recovery.

Do not automate this consequential action when severe harm is hard to detect, intervention cannot precede irreversible commitment, no competent authority is feasible or safe state and remedy cannot be established. An assistive research route may remain, but it needs its own contract.

Override is an engineered and rehearsed capability. The organisation proves it on realistic queue, degraded-mode and partial-effect drills, then records what the person could see, decide and change. If the institution creates conditions that make control impossible, responsibility cannot be transferred to the operator who happened to touch the screen.

Operational proofDrill the chain from anomaly to repaired outcome

Specify the intervention claim

Begin with the consequential action and affected party. “A human reviews the output” is not a testable claim. “Before an account is suspended, an authorised fraud specialist sees the underlying events and contradictory evidence, can prevent suspension and can restore service if a partial action occurred” identifies timing, information, competence, authority and remedy.

Map the commitment point. A proposal may become binding when a record is written, a message leaves, a payment is released or another service consumes an event. Interfaces often treat the final click as commitment even though asynchronous queues or earlier writes have already created effect. Instrument the actual state transition and expose it to the reviewer.

Define detection signals independently of the system's preferred answer. Include data-quality failures, scope changes, contradictory evidence, unusual tool paths, cumulative limits, policy exceptions and affected-party contest. Sample silent cases to estimate missed detection. An alert rate alone cannot show whether important anomalies are visible.

Orientation should answer five immediate questions: what is happening, what evidence supports it, who or what is affected, what has already occurred and how long remains to intervene. Details can be progressively disclosed, but the material action status cannot hide in a log. Test comprehension under realistic time and interruption.

Prove competence without creating idle nominal experts

Competence combines domain knowledge, system literacy and control rehearsal. A specialist who understands policy but not tool retries may misread a pause. A technical operator who understands queues may lack authority to judge a customer exception. Role design must cover the decision, system and recovery responsibilities or create a timely team route.

Training evidence should include initial qualification, recent practice and degraded-mode drills. Completion of a slide course is weak proof. Use scenarios with conflicting sources, partial writes, missing managers and unavailable suppliers. Observe diagnosis, decision and safe-state verification, then provide feedback without training people merely to agree with the system.

Automation can change the skill it leaves behind. When a system handles ordinary cases, human work becomes rarer and harder. Preserve access to representative cases, rotate controlled practice and monitor whether orientation time or error rises. If competence cannot be sustained at the required response time, reduce automated consequence.

Over-override deserves diagnosis. Compare reasons and outcomes by policy boundary, cohort and reviewer. Frequent rejection may reveal valid local information omitted from the rule. It may also reveal inconsistent discretion or pressure from affected business units. Change policy, evidence or training according to the mechanism; do not maximise either agreement or veto.

Engineer capacity as a constraint

Use arrival and service distributions rather than daily averages. Separate routine review from urgent interruption, allow for shift changes and simultaneous incidents, and include time to contact an affected person or specialist. Set a maximum queue for protected actions. When capacity is unavailable, hold or degrade the action rather than silently shorten review.

Review duration includes orientation, evidence acquisition, independent judgement, consultation, action and confirmation of effect. A six-second timer is not rescued by a fast click. Measure how much of the window remains after queues and system latency. If the remaining time is less than the credible decision demand, capacity fails before any individual acts.

Interruptions impose switching cost. A frontline reviewer cannot simultaneously handle a customer, monitor an agent and reconstruct a complex incident without losing context. Design protected attention, visible priority and resumable work. Count abandoned and deferred cases, not only completed approvals. Staffing and workflow are control implementation, not external assumptions.

Incentives should make challenge safe. Reviewers need assurance that justified delay, disagreement and escalation will not damage performance ratings. Managers need metrics that distinguish unnecessary blockage from prevention. Audit whether agreement correlates with team, shift, sponsor or throughput pressure and investigate causal conditions before drawing conclusions about reviewer quality.

Test authority against real dependencies

List each control verb and its actual reach: disregard a recommendation, modify a proposal, pause a run, cancel queued calls, revoke a token, reverse a record, notify recipients and escalate the use case. A role may possess one and lack another. The interface should not present “reject” if it merely opens a request that arrives after commitment.

Authority must survive absence and urgency. Define alternates, on-call qualification and expiry. A manager approval path that works only during office hours cannot control a global deadline. Dual control needs two independent decisions with the necessary competence and time; duplicating the same confirmation screen adds signatures without adding information.

Tool permission is not business authority. A reviewer may have technical access to stop a service but no mandate to reverse a payment, or formal authority to stop but no credential that reaches the supplier. Test identity, permission and business mandate together. A valid policy sentence cannot cancel a tool call.

Institutional refusal is authority at the highest level. Leaders should decide that no feasible staffing, evidence or recovery model can control some actions. Record the rejected use, affected population and condition for reconsideration. Product pressure should not convert the refusal into a hidden pilot with the same causal effect.

Rehearse safe state and remedy

A safe-state drill begins during partial execution. Introduce duplicate events, delayed responses, network partitions, expired credentials and one irreversible side effect. Press stop, then trace every action identifier. Verify which calls ended, which may retry, which records are authoritative and which person owns unresolved state. The drill ends on evidence, not on a green icon.

Idempotency reduces duplicate system effects but cannot retract external reliance. A corrective message, customer reconsideration, payment reversal or compensation may still be required. Define these routes before launch and involve legal or specialist review where applicable. Affected people should not bear the burden of finding every downstream copy.

The override receipt should be versioned and readable. Record trigger, snapshot, sources, independent evidence, time budget, reviewer role, decision, executed action, effects, recovery, notification and owner. Avoid a raw event dump that cannot reconstruct the human situation. Retain data according to legitimate purpose and access controls.

Outcome review asks whether the intervention prevented or reduced harm, whether the original automation was right, whether the reviewer had private information and whether the control created new delay or inconsistency. This evidence updates thresholds, training, capacity and product design. It does not become a leaderboard for individual agreement.

Audit responsibility and change

After a failure, reconstruct decisions at product, procurement, staffing, policy and frontline levels. Who chose the automation, evidence view, queue target, authority, supplier and rollback design? Which actor could change the relevant condition? Keep the operator's judgement accountable where real discretion existed while preventing institutional design from disappearing behind the approval record.

Material changes trigger new drills. A model may alter confidence or abstention, a prompt may hide a limitation, a tool may add irreversible effect, an interface may move evidence, a policy may narrow reviewer authority and a staffing change may break capacity. Map each change to affected links and retest before relying on the old override claim.

NIST's AI RMF Appendix C calls for differentiated human roles and recognises variation in human-AI interaction. The current published AI RMF 1.0 is being revised. ICO human-review guidance currently states that it is under review following the Data (Use and Access) Act. Use these sources as scoped governance input, not as proof that this product's chain works.

The acceptance question is concrete: can a competent, independent and protected person detect the need, orient to current state, decide in time, change the actual system, reach a known safe state and support remedy? If evidence for any necessary link is missing, the organisation has an interface claim, not an override capability.

Integrity evidenceMeasure links without creating a control score

Use link-specific indicators. Detection can track known anomalies found before effect and sampled misses. Orientation can test state reconstruction and consequence comprehension. Independence can record access to primary or contradictory evidence. Capacity can track protected-lane queue and expired intervention windows. Authority and safe effect can use successful stop, cancellation, reconciliation and remedy drills. Preserve results by consequence and action class.

Do not combine those indicators into one override score. A perfect training result cannot compensate for absent authority, and a fast queue cannot compensate for opaque downstream effects. Treat necessary links as gates, then use ranges and qualitative evidence within them. Report failed links, uncertainty and owner. A release team should see why control works or fails rather than a green aggregate.

Agreement rate needs a paired outcome. Near-universal acceptance may reflect reliable recommendations or ceremonial review. High disagreement may reflect essential human information or arbitrary preference. Sample cases, compare available evidence and review later outcomes. Ask reviewers what they believed their action covered and whether they expected the system to stop. Interface misunderstanding is a product result.

Complaint and appeal volume is incomplete. People may not detect the automated effect, connect it to the product or believe challenge is worthwhile. Proactively sample consequential outcomes and provide an accessible contest route. When an affected person corrects a record, propagate the correction through the action graph and include it in product learning without treating the person as a test resource.

Institutional measures include override claims with a complete receipt, drills that reach a safe state, expired competence, unserved protected queues, informal waivers and unresolved remedies. These direct investment to staffing, tooling and authority. They should not rank teams for suppressing incidents or reward overrides that create unnecessary inconsistency.

The decision remains one of five routes, not a maturity level. Sample bounded automation where every-action review adds no protection. Require pre-action or dual control where competent judgement can change the outcome. Redesign when the chain is repairable. Refuse automation when irreversible consequence outruns feasible control. Revisit the route when evidence, workload, tools or institutional conditions change.

Test boundary escalation as well as ordinary flow. A low-consequence rule can become material when volume accumulates, a new population enters or reversal expires. The system should detect that transition, suspend sampled automation and require the stronger route before consequence crosses the boundary. Reviewers need the cumulative envelope, not only the current item.

Finally, verify institutional learning. A justified override should update a case, rule, prompt, tool boundary or training decision when the mechanism is general. A mistaken override should improve policy or reviewer feedback without discouraging future challenge. The owner records the change and its disconfirming test so one dramatic case does not become an unexamined permanent exception.