01 · Two ways to failThe useful system can be rejected; the persuasive one can be obeyed
One planning team sees a forecasting model make a memorable miss and abandons it, even though it outperforms the managers for the population it was built to support. Across the building, an underwriter reads a fluent explanation, accepts the recommendation and overlooks contradictory source evidence. The first team rejects warranted assistance. The second complies without verification. Neither has calibrated reliance.
Organisations often treat these problems as opposite attitudes requiring different messages. The sceptics need to be made confident; the enthusiasts need a warning. That framing is too shallow. A decision needs an operating rule that changes with evidence strength, context match, error asymmetry, detectability, reversibility and comparative human capability. The useful target is not a warmer feeling about AI. It is behaviour that accepts, checks, challenges, overrides, escalates or rejects an AI contribution in proportion to the conditions now present.
This paper distinguishes that decision-level mechanism from adjacent work. Paper 073 addresses the behaviours through which a product earns trust. Paper 074 designs interfaces that disclose uncertainty. Paper 076 asks whether human intervention can alter a consequential path. Paper 141 explains adoption as a property of a wider work system. Here the narrower question arrives at the instant of a decision: given the evidence, limits and possible harm, what weight may this person or automated process reasonably place on this contribution?
Human-factors research supports caution in both directions. Studies of algorithm aversion found disproportionate withdrawal after visible errors in bounded forecasting experiments; algorithm-appreciation experiments found that people sometimes weight algorithmic advice more heavily than comparable human advice. Automation-bias research identifies omission and commission errors under studied conditions. These are not universal human traits. They are rival patterns whose direction depends on task, display, experience, pressure and comparison.
02 · The vocabularyFive terms, five kinds of evidence
Trust is a person's attitude or expectation that an AI will help achieve a goal under uncertainty. Trustworthiness comprises context-dependent properties of the complete system, including validity, reliability, safety, security, accountability, transparency, privacy and fairness. Reliance is observable weighting or use of AI advice or action in one decision. Compliance means acting on a recommendation or alarm. Verification acquires evidence that can change what may reasonably be accepted.
The distinctions prevent easy category errors. A favourable survey is evidence about attitude, not safe conduct. High accuracy for a model does not establish system trustworthiness when data access, workflow, security or remedy fails. Compliance could be rational acceptance or a response to a default, timer or manager. Looking at an output twice is not verification. A second reviewer who consults the same incomplete record may add confidence without adding information.
| Term | Required evidence | Observable behaviour | Common failure |
|---|---|---|---|
| Trust | Expectation, experience and perceived vulnerability in context | Stated willingness, hesitation or confidence | Sentiment is treated as proof of safety |
| Trustworthiness | System properties against a defined purpose, population and lifecycle | Controls and outcomes remain dependable under stated conditions | Model accuracy stands in for the whole system |
| Reliance | Advice weight, authority, alternatives and consequence | Accept, edit, override, abstain, escalate or reject | More reliance is labelled as better performance |
| Compliance | Action following a recommendation or alarm | The proposed action is taken | Agreement is assumed to be informed or correct |
| Verification | An independent source, test or observation capable of disconfirmation | Evidence changes, confirms or blocks the action | A ceremonial glance becomes a control claim |
NIST frames trustworthiness as socio-technical and context-sensitive, with characteristics that can trade off. The EU AI Act's human-oversight provisions apply within its legal scope to specified high-risk systems and explicitly address competence, authority and awareness of automation bias. The UK Government AI Playbook calls for meaningful human control, clear responsibility and recourse. OECD principles connect agency and oversight to human rights and contextual safeguards. None of these sources says that every output needs a manual signature, or that a signature makes an action safe. Applicable sector law and professional duty still govern the actual decision.
03 · The learning mechanismClose the Reliance Control Loop
The Reliance Control Loop is this article's first synthesis: frame the decision, set the operating envelope, compare human and AI evidence, choose a reliance mode, act or abstain, observe outcomes and near misses, update the envelope, then communicate the change. Repeated use without feedback is habituation, not calibration.
Every loop records the decision and affected population, purpose and alternatives, human and AI baselines, relevant system versions, known exclusions, selected mode and human authority. It preserves what evidence was available before action, how the output was accepted or challenged, and the error type, consequence, detectability and recovery. Workload, time pressure and incentives belong in the record because they alter reliance behaviour. An outcome, affected-party response, learning owner and expiry complete the circuit.
Feedback must reach the decision rule, not only a model backlog. If overrides reveal that a policy exception is systematically absent, the operating envelope changes. If a near miss shows that recovery takes longer than the consequence arrives, the reliance mode narrows. If representative evidence shows stable performance and effective detection for a reversible task, action can become less interruptive. The loop permits movement in either direction.
Thought experiment: a hundred flawless decisions
An assistant succeeds one hundred times on routine supplier invoices. On the next item, the supplier and amount look familiar but the payment route changed. Should the operator rely more because of the streak? Only if prior success is evidence for this condition. If the new route creates an irreversible transfer and removes recovery, the relevant envelope changed before the model produced anything. Experience belongs to a versioned population, not to the product name.
04 · Six axesPlot a calibration surface without averaging it
The Trust Calibration Surface is this article's second synthesis. It plots evidence strength, context match, error asymmetry, detectability and intervention, reversibility and recourse, and human comparative capability. The weakest consequential axis constrains the reliance mode; an average cannot compensate for a missing safeguard.
Evidence strength covers representative field performance, uncertainty, comparison, distribution and independent evaluation. Context match asks how closely the tested population, task, data, environment, workflow and consequence resemble the present decision. Aggregate performance can be strong while subgroup evidence is weak or the current case sits beyond the evaluated boundary.
Error asymmetry compares false positives, false negatives, omissions, commissions, delay and common-mode failure. Detectability and intervention ask whether a problem can be recognised before harm, and whether an authorised person has enough time to change the path. Reversibility and recourse concern containment, explanation, appeal, remedy and recovery. Human comparative capability includes domain skill, situational awareness, workload and access to an independent channel. “Human in the loop” is not a value for this axis.
| Axis | Evidence | Boundary to state | Owner | Trigger to narrow reliance |
|---|---|---|---|---|
| Evidence strength | Comparative results, uncertainty, distribution, external challenge | Population, outcome and evidence age | Evaluation owner | Unsupported group or stale comparison |
| Context match | Task, data, environment, workflow and consequence similarity | Evaluated conditions and exclusions | Product and domain owners | Novel case or changed operational setting |
| Error asymmetry | Type, frequency, severity, correlation and bearer of harm | Permitted error exposure | Business and risk owners | New severe, cumulative or subgroup error |
| Detectability and intervention | Time to detect, independent signal, authority and response window | Latest safe intervention point | Operations owner | Consequence arrives before credible control |
| Reversibility and recourse | Rollback drills, appeal, correction, containment and remedy | Known safe state and affected-party route | Service owner | Irreversible action or failed remedy |
| Human comparative capability | Human-alone baseline, expertise, attention and independent information | Role and workload conditions | Functional leader | Skill expiry, overload or shared error source |
Thought experiment: the better model, the worse team
A new model improves total accuracy, but the cases it now gets wrong were previously the ones experts recognised easily. Old errors disappear and unfamiliar errors emerge behind a more fluent explanation. Model-alone performance rises while team detection falls. If the organisation tracks only aggregate accuracy, it widens reliance precisely when the human-AI route has become less compatible.
05 · Function allocationReliance modes are choices, not ranks
Six Reliance Modes allocate information, analysis, selection and action differently. They do not form a staircase towards autonomy. One workflow can use several modes, and movement towards stronger human control can be the sign of learning.
Independent human route withholds AI output when exposure could anchor judgment or the route is unsupported. Consultative input supplies evidence or options while the person frames and decides independently. Verify before use requires a named disconfirming source or test before action. Execute with human veto permits action after notice inside a narrow, visible and reversible envelope. Conditional autonomy permits bounded action with monitoring, enforced stops and sampled or event-triggered review. Suspended reliance quarantines contribution for an affected population while a fallback continues service.
The correct mode can differ within one case. A benefits officer may use AI consultatively to locate records, verify extracted facts against an independent source and make the eligibility judgment without seeing a generated recommendation. A drafting system may act conditionally on formatting but require human approval before external delivery. A detected subgroup failure may suspend only that population rather than provoke universal withdrawal or average away the harm.
| Mode | AI function | Human role | Permitted action and control | Appropriate context |
|---|---|---|---|---|
| Independent human route | Absent or withheld | Frames and decides | No AI influence; preserve clean evidence channel | Anchoring risk, severe consequence or unsupported population |
| Consultative input | Offers facts, options or critique | Independent framing and choice | Use, edit or disregard with provenance | Human context is decisive and AI can broaden search |
| Verify before use | Proposes a decision or artefact | Tests decisive claims | Action blocked until independent evidence passes | Material errors are detectable before commitment |
| Execute with human veto | Acts after visible notice | Monitors and can prevent commitment | Enforced delay, veto, rollback and event trace | Narrow, reversible, observable action with usable response time |
| Conditional autonomy | Acts inside hard boundaries | Reviews samples and events | Limits, monitoring, stop and recovery are enforced | Stable low-consequence population with proven controls |
| Suspended reliance | Quarantined for affected scope | Uses fallback and supports recovery | No AI contribution until evidence-supported restoration | Incident, drift, change or newly exposed error class |
Reliance has a workload history
Calibration can deteriorate even when technical performance is stable. Reliable automation removes ordinary practice and leaves people with rare, ambiguous exceptions, so skill and situation awareness may decay exactly where intervention becomes hardest. Frequent low-value alerts teach operators that attention is wasted; the next important alert then arrives inside alert fatigue. A fast queue or agreement target encourages complacency by making investigation costly. These are properties of the operating arrangement, not evidence of defective character.
Selective distrust also has more than one cause. An expert may know that one customer group, data source or seasonal condition lies outside the tested boundary. A novice may reject sound advice after a visible failure, or accept it because a precise percentage and fluent rationale look authoritative. Confidence cues therefore require their own evaluation: does the display help users discriminate correct from incorrect cases, or merely move compliance? Hiding limitations to prevent under-reliance is as dangerous as exaggerating certainty to increase use. Representative learning should include success, failure, abstention, escalation and recovery, with outcome feedback that corrects both excessive acceptance and excessive rejection.
Verification must remain proportionate. Requiring every person to reconstruct a reliable low-consequence calculation can destroy efficiency without increasing independence. Conversely, a quick confirmation on a life-changing decision is inadequate even if it is cheap. The selected mode should allocate scarce attention towards error classes where human information or judgment can change the outcome, and place enforced technical controls where people cannot reasonably detect the problem.
06 · Before oversightMap asymmetry and shared failure
The Error Asymmetry Map records false positives and negatives, omissions and commissions, correct and wrong agreement, rare catastrophe, frequent nuisance and cumulative harm. It adds detectability, time to consequence, recovery, distribution and the identity of the person who bears harm. Review design begins here, not with a generic instruction to “keep a human in the loop”.
A missed fraud flag and an unnecessary account freeze have different victims, timing and remedies. A drafting error may be cheap to reverse before publication and expensive afterwards. A modest subgroup false-negative rate can concentrate serious harm even when overall accuracy looks attractive. Aggregate exposure matters because individually small errors can scale across thousands of decisions.
Human and AI errors may also correlate. Both can depend on the same incomplete record, policy assumption, target or historical label. Agreement then represents one evidence lineage expressed twice, not two votes. Independence is a property to demonstrate. It can come from a separate source, a different measurement, privileged situational information, a specialist interpretation or a test whose failure would genuinely block action.
| Error | Consequence | Detectability | Recovery | Affected party | Decision implication |
|---|---|---|---|---|---|
| False positive | Unjustified intervention or denial | May appear only when challenged | Correct record and reverse effect | Person incorrectly flagged | Require counter-evidence and accessible appeal |
| False negative | Risk or need remains unseen | Often discovered after consequence | Contain exposure and restore service | Person or system left unprotected | Narrow scope where detection is weak |
| Commission | Wrong action is actively taken | Visible action can still hide causal error | Undo before dependencies propagate | Recipient of the action | Use pre-action control when reversal is poor |
| Omission | Needed action or escalation is absent | Silence is difficult to sample | Find missed population and reprocess | Person who received no service | Add independent sampling and no-decision review |
| Wrong agreement | Human and AI reinforce the same mistake | Consensus suppresses challenge | Trace the shared evidence lineage | Whoever bears the joint error | Do not count agreement as verification |
| Cumulative small error | Distributed loss becomes material at volume | Individual cases appear tolerable | Aggregate correction and policy repair | Concentrated or dispersed groups | Set population limits and cumulative stops |
Failure boundary for human review
Do not assign a person responsibility when the role lacks independent evidence, adequate time, current competence or authority to intervene. Constant verification can erase the benefit, damage situation awareness and still fail to diversify error. If a credible review route cannot precede irreversible harm, reduce automated consequence, move control earlier or reject the AI route.
07 · Comparative proofMeasure the team, then issue a Calibration Receipt
The Reliance Evidence Ledger compares human-alone, AI-alone and human-AI performance for the relevant population and version. Merely combining two actors does not establish complementarity. The joint route must outperform the best viable alternative on the outcomes and harms that matter.
The ledger records abstention, override, escalation and no-decision rates; error and near-miss type by group and context; calibration of confidence or uncertainty signals; verification yield and false reassurance; time, workload, attention and situation awareness; and recovery, recourse and affected-party outcomes. It tests explanation effects on acceptance and correctness separately. An explanation that raises both correct and incorrect acceptance is persuasive but not calibrating.
Comparative results can justify different modes by skill and case. Bounded suggestions may improve outcomes for novice customer-service agents while experienced staff gain little overall and remain crucial for rare exceptions. That evidence does not imply that novices should obey more often. It supports a narrower envelope with accessible escalation and learning. Research on a jagged technological frontier similarly shows why task position matters, but its studied knowledge-work setting and outcomes cannot be exported as a general performance law.
The receipt at a material decision
A Calibration Receipt preserves what the human knew before seeing the AI, what the system contributed, which evidence was independent, the chosen mode and why the contribution was accepted, edited, overridden, escalated or rejected. It notes unresolved conflict, authority, action, consequence, recourse and the learning returned to the loop. The receipt is proportionate control evidence, not a diary of private thought or a worker-surveillance feed.
A useful receipt reconstructs the decision environment. It avoids forcing lengthy justification for every low-consequence edit while ensuring that a material exception can be audited. Product owns the represented evidence and versions; the business owner owns the outcome and envelope; operations owns workable capacity; risk provides independent challenge; the authorised professional owns the judgement that genuinely falls within the supplied role. Liability cannot be pushed backwards onto the last person who touched the screen.
08 · Expectations expireChange control before reassurance
The Trust Change-Control Rule treats learned reliance as version-bound. A model, prompt, retrieval source, data set, policy, interface or workflow change can invalidate familiar success and failure patterns even when an aggregate performance number improves.
Before release, identify which operating envelopes and user expectations the change disturbs. Compare old and new error types by population, not only total accuracy. Test the human-AI route and reliance behaviour. Announce the affected boundary, show representative changed successes and failures, preserve rollback and fallback, and reset, narrow or expire reliance where prior experience no longer applies. Broader reliance returns through observed outcomes.
After material error, the Calibration Recovery Protocol first contains the affected population while preserving service through a fallback. It separates model, data, interface, workflow, authority, workload and incentive contributors, and investigates under-reliance as well as over-reliance. Operators and affected parties receive appropriate disclosure; harm is remedied and appeals are processed. Practice uses representative successes, failures, abstentions and escalations. Only the evidence-supported mode returns, under monitoring with a named expiry.
A confidence campaign fails this protocol. Reassurance attempts to restore attitude before system behaviour, remedy and evidence justify action. Equally, one dramatic error should not eliminate a demonstrably superior bounded route everywhere. Recovery communicates both what changed and what remains supported, allowing reliance to narrow precisely instead of swinging between loyalty and rejection.
09 · Ten boundary testsCalibration under pressure
Each case closes the control loop, plots the six axes, selects a mode, maps asymmetric error, compares viable routes, issues a proportionate receipt and names accountable business, product and risk owners.
1. Visible forecasting miss
Evidence and error: the algorithm beats managers for the supported demand-forecast population, but one salient miss triggers wholesale rejection. Context match and recovery remain strong; the miss is visible and reversible.
Mode and control: retain a bounded operating envelope with manager challenge, disclose the error class and monitor outcome by category. The planning business owner decides scope, product owns drift, and risk tests exclusions. The receipt prevents anecdote from replacing the comparative record.
2. Persuasive underwriting explanation
Evidence and error: a rationale increases compliance with a wrong recommendation despite contradictory primary documents. The explanation shares the model's evidence and yields no independent verification.
Mode and control: require source verification before use and preserve underwriter authority. Product separates rationale from evidence status; the credit owner protects checking time; risk samples wrong agreement. Applicants receive notice, correction and appeal. Fluency is tested against correctness, not preference.
3. Novices and experts
Evidence and error: bounded suggestions help novice service agents on routine cases, while experts show little average gain and detect rare exceptions. Population and case type change comparative capability.
Mode and control: novices use verified suggestions with escalation; experts receive consultative input and retain exception judgment. The service owner tracks resolution and learning, product tracks error by case, and risk watches unequal customer outcomes. Neither group receives a universal reliance target.
4. Nominal clinical oversight
Evidence and error: a clinician must review an alert under severe time pressure with no independent source. Harm can arrive before adequate assessment; human and model draw from the same incomplete record.
Mode and control: reject rubber-stamp review and redesign the workflow. The clinical business owner funds time or moves detection earlier, product exposes primary evidence, and safety risk owners test intervention. Patients retain the existing route until control is real.
5. Accuracy-improving update
Evidence and error: aggregate accuracy rises, but familiar correct cases become errors and user detection falls. Prior experience is stale for the changed population.
Mode and control: narrow reliance, compare old and new error sets, and revalidate team performance before expansion. Product owns the version boundary, business owns fallback, and risk challenges subgroup results. Receipts carry version identifiers so outcomes can repair the right envelope.
6. Reversible drafting
Evidence and error: a private draft has strong context match, visible defects and easy undo. Consequence remains low until external publication.
Mode and control: allow conditional action inside the draft workspace with later review, while external send remains blocked. The content owner defines quality, product enforces the boundary, and risk samples leakage and unsupported claims. This is appropriate reliance without constant interruption.
7. Consequential benefits decision
Evidence and error: denial has high consequence, decisive omissions are hard to detect and recourse can be slow. An AI recommendation risks anchoring the authorised officer.
Mode and control: use an independent human route for eligibility judgment, with AI limited to locating records through a separate evidence channel. The service owner owns the decision, product blocks recommendation display, and risk tests omitted populations. Claimants receive accessible explanation and appeal.
8. Concentrated subgroup false negatives
Evidence and error: a vendor's single accuracy rate conceals missed cases concentrated in one group. Distribution, context match and recourse fail despite acceptable overall results.
Mode and control: suspend reliance for the affected group, provide human fallback and remediate prior cases. Business owns service continuity, product obtains disaggregated evidence, and risk independently challenges recovery. Scope expands only after representative outcome proof.
9. Shared incomplete record
Evidence and error: human and AI agree because both see the same missing field. Agreement raises confidence but does not change the information set.
Mode and control: escalate to an independent source or specialist. The business owner defines the needed evidence, product marks lineage, and risk samples consensus failures. The receipt states that agreement was correlated and prevents it being counted as confirmation.
10. Confidence campaign after incident
Evidence and error: misplaced reliance harmed customers; leaders propose positive messages before cases are repaired. Trust and system evidence now diverge.
Mode and control: suspend affected reliance, remedy cases, change the system and practise failure boundaries. Business owns affected outcomes, product owns correction, and risk approves only bounded restoration. Communications report facts and limits rather than manufacture confidence.
10 · Interactive instrumentReliance Calibration Workbench
The workbench structures one decision population. Set the evidence fields or load a named case. It returns exactly one of five routes and displays the six-axis surface, reliance mode, error asymmetry, independent evidence, comparative performance, receipt, authority, protections, change control, expiry and owners.
Calibrate one decision population
Mark each item with current evidence. “Unknown” narrows what the system can reasonably ask a person to do.
Worked scenarios
Rely within a bounded operating envelope
Load a scenario or set the evidence fields to produce a decision-specific route.
The workbench will not infer correctness from human-AI agreement, recommend reliance from aggregate accuracy, or convert attitude into a safety claim. Its routes are decisions with expiry. They remain answerable to applicable law, professional responsibility and the rights of the people affected.
11 · The operating conclusionMake reliance movable and responsibility real
Trust will rise, fall and vary between people. An organisation cannot command that attitude into calibration. It can make reliance conditional on evidence and equip the person or automated control to change course.
The operating sequence is disciplined. Define the decision population and alternative route. Close the Reliance Control Loop. Plot all six axes and map asymmetric, distributed and correlated errors. Compare the human, AI and joint paths. Select a mode whose controls can act before consequence. Preserve a receipt for material decisions. Treat every significant system change as a possible expiry of learned experience. After an incident, repair the outcome and the system before inviting reliance again.
This is a system obligation before it is a user obligation. Product and business owners supply representative evidence, limits, interfaces, alternatives and outcome feedback. Operations supplies attention and response capacity. Risk independently challenges error distribution and controls. Leaders protect override, abstention and escalation from punitive incentives. The operator remains responsible for judgment only where the organisation has provided the information, competence, time and authority that make judgment possible.
Calibrated reliance occupies the space between reflexive rejection and automatic acceptance, but it is not a compromise between them. It is a more exact practice. Sometimes it welcomes bounded help after a visible miss. Sometimes it demands an independent check, specialist escalation or suspension. Sometimes it rejects the AI route for a population altogether. The common feature is that evidence, consequence and accountable control decide, not confidence theatre.
Evidence spineReferences and limits
Governance sources are applied within their legal and institutional scope. Laboratory, forecasting and field studies establish bounded findings rather than universal psychology or performance. The Reliance Control Loop, Trust Calibration Surface, six Reliance Modes, Error Asymmetry Map, Reliance Evidence Ledger, Calibration Receipt, Trust Change-Control Rule, Calibration Recovery Protocol and workbench are this article's synthesis.
- NIST AI Resource Center, Artificial Intelligence Risk Management Framework.
- NIST AI Resource Center, AI Risk Management and Human-AI Interaction.
- NIST AI Resource Center, AI Risks and Trustworthiness.
- NIST AI Resource Center, AI RMF Core.
- European Union, Regulation (EU) 2024/1689, especially Article 14.
- UK Government, Artificial Intelligence Playbook for the UK Government.
- OECD, Human-Centred Values and Fairness Principle.
- Lee, J. D. and See, K. A., Trust in Automation: Designing for Appropriate Reliance, 2004.
- Parasuraman, R. and Riley, V., Humans and Automation: Use, Misuse, Disuse, Abuse, 1997.
- Parasuraman, R., Sheridan, T. B. and Wickens, C. D., A Model for Types and Levels of Human Interaction with Automation, 2000.
- Hoff, K. A. and Bashir, M., Trust in Automation: Integrating Empirical Evidence on Factors That Influence Trust, 2015.
- Skitka, L. J., Mosier, K. and Burdick, M., Does Automation Bias Decision-Making?, 1999.
- Dietvorst, B. J., Simmons, J. P. and Massey, C., Algorithm Aversion: People Erroneously Avoid Algorithms After Seeing Them Err, 2015.
- Logg, J. M., Minson, J. A. and Moore, D. A., Algorithm Appreciation: People Prefer Algorithmic to Human Judgment, 2019.
- Dietvorst, B. J., Simmons, J. P. and Massey, C., Overcoming Algorithm Aversion: People Will Use Imperfect Algorithms If They Can Modify Them, 2018.
- Bansal, G. et al., Does the Whole Exceed Its Parts? The Effect of AI Explanations on Complementary Team Performance, 2021.
- Bansal, G. et al., A Case for Backward Compatibility for Human-AI Teams, 2019.
- Dell'Acqua, F. et al., Navigating the Jagged Technological Frontier, 2026.