A reliable answer can still be an unsafe message

The natural response to the 09:17 incident is to ask whether 94% was accurate. That matters, but it is not the whole failure. Suppose the assistant is perfectly calibrated across a thousand similar invoices: among cases labelled 94%, exactly 94% are safe to release. The analyst still cannot know whether this case sits inside the familiar 94% region or near a new failure boundary. A population statistic does not, by itself, identify the assumption carrying one decision.

Research on automation has long distinguished appropriate use from misuse and disuse. Overreliance can make a person accept bad advice; under-reliance can make them reject useful advice.^[1] Trust influences these behaviours, but it is not identical to them. Lee and See describe trust as an attitude that helps govern reliance under uncertainty, while later work separates trust from the actual act of depending on a system.^[2][3] The practical target is not maximum trust. It is warranted reliance on this system, for this task, under these conditions.

Four things that interfaces routinely collapse

Trust is a person’s willingness to accept vulnerability in relation to an expected capability or commitment. Reliance is observable behaviour, such as accepting, rejecting, checking or delegating. Trustworthiness concerns whether the configured system merits reliance. Confidence is an estimate, often produced by a model or user, about a particular proposition. These variables can move separately.

A person can distrust a system yet follow it because policy leaves no choice. They can report high trust yet verify every recommendation. A model can be well calibrated statistically while its evidence is stale. A configured system can be trustworthy for retrieval but not for payments. Hoff and Bashir’s review found dispositional, situational and learned sources of trust variation, which helps explain why identical interfaces produce different reliance patterns.^[4] Yin and colleagues also found that both stated accuracy and observed performance shape trust.^[5]

This creates a scale problem. Most trust displays summarise a model, while most reliance decisions concern a case. “The system is 91% accurate” describes performance under a particular test distribution. “This recommendation is safe to follow” requires a bridge from that distribution to the present case. The bridge may depend on data freshness, subgroup, operating regime, missing fields, tool state or a recent policy change. If the interface hides that bridge, the user is asked to convert a global reputation into a local action by intuition.

Appropriate reliance is consequently at least two-dimensional. A collaborator should accept advice when it is correct and reject it when it is wrong. A person who rejects everything avoids overreliance but creates maximum under-reliance. A person who accepts everything captures all correct advice but also all bad advice. Schemmer and colleagues formalise this as a joint problem rather than one agreement rate.^[12] A single trust scalar cannot reveal whether a person is discriminating good cases from bad ones.

Experiments make the point sharper. Case-specific confidence can help calibrate trust, yet calibration alone need not improve team accuracy when people lack complementary knowledge.^[6] Explanations can also increase acceptance regardless of whether the AI is correct.^[7] Cognitive forcing can reduce overreliance, but users may dislike the added effort and its benefits vary with motivation.^[8] A large meta-analysis found that human–AI combinations often fail to outperform the better solo performer, particularly in decision tasks.^[9]

The hidden assumption is that information which explains a recommendation also equips a decision. Often it does not. A feature-importance plot may say why a model leaned left rather than right, while omitting whether the source data are current, whether the human may override, or whether the action can be reversed. An explanation can be epistemically interesting and operationally empty.

flowchart LR
  A["A reliable answer can still be<br/>an unsafe message"] --> B["The three operations a message<br/>must support"]
  B --> C["From explanation to a control<br/>surface"]
  C --> D["Where sufficiency breaks"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#7B4CD8,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

The three operations a message must support

Return to the payment message. What exactly did the longer version enable? It let the analyst revise a belief about safety. It exposed a concrete challenge: verify the changed account. It stated how to proceed and recover. These are three different operations, and each closes a different gap.

I will call this the Update–Challenge–Continue test. A message is decision-sufficient only relative to a decision and a recipient. The same sentence may be enough for a reversible calendar suggestion and dangerously thin for an irreversible fund transfer. Sufficiency therefore depends on the loss if wrong, uncertainty about the case, delegated authority, available time, reversibility and the human’s complementary competence.

A minimum, not a summary

“Minimum” does not mean short at all costs. It means removing every element that does not change a permitted action while retaining every element needed to distinguish unsafe reliance from safe reliance. Formally, let d be the decision context, m the message and B(d) the attention budget available in that context. Let U, C and R score whether the recipient can update, challenge and continue. The design problem is:

Here, the three thresholds rise with stakes, uncertainty, autonomy and irreversibility. The equation assumes we can operationalise the three capabilities using tasks rather than self-report alone. Remove that assumption and the score becomes a design prompt, not a validated measure. The important move is conceptual: the optimisation target is action competence under a bandwidth constraint, not explanation volume.

The three operations correspond to three failure gaps. The belief gap appears when the recipient cannot connect evidence to the recommendation. The counterfactual gap appears when they cannot say what would make the recommendation wrong. The control gap appears when they do not know what the system may do, what they may stop, or how an effect can be reversed. A message can close one gap while leaving the others open. Confidence mainly addresses belief. A contrastive explanation can address belief and counterfactuals. Authority and recovery close the control gap.

Sufficiency is also recipient-relative. A fraud specialist may know why a recent account change matters; a temporary approver may not. The message for the second person must name the risk and the required test. This does not justify manipulating people with personalised persuasion. It means the interface must represent which decision competence it assumes. When that competence is absent, the correct response is escalation or reduced authority, not a denser confidence display.

The six fields that usually carry the work

The Update–Challenge–Continue test can be implemented through six fields. They are not universal interface furniture. They are candidate obligations that should appear when the decision context requires them.

Proposal and scope state what is being recommended, for which object and with what expected effect. Evidence and provenance identify the observations carrying the proposal and where they came from. Uncertainty expresses case-specific uncertainty, not a decorative percentage. Flip conditions name the smallest observation or assumption change that would reverse the proposal. Authority states whether the system is advising, requesting approval or executing within a grant. Recovery states the stop, rollback, compensation or escalation route.

These fields should not be compressed into one confidence badge. They have different temporal roles. Evidence and provenance describe what supported the proposal at the cutoff. A flip condition points forward to the next discriminating observation. Authority governs the present action. Recovery governs the state after action. Mixing these times creates familiar errors, such as presenting a successful validation check as proof that execution succeeded, or treating a previous permission as current authority.

A useful interface therefore has a stable visible core and optional depth. The visible core contains anything whose absence could change the immediate action. Optional layers can expose model documentation, full evidence lineage and alternative calculations. The split must be established through testing, not aesthetics. If users routinely open a collapsed panel before making the correct decision, that information belongs in the core. If a field never changes an action, challenge or recovery route, it may belong in audit rather than the decision message. Progressive disclosure is an empirical claim about what can remain hidden safely.

Recent work suggests why the flip condition deserves special emphasis. Standard explanations often justify the AI’s choice without engaging the user’s likely alternative. Contrastive explanations that address a predicted human misconception improved independent decision skill in one large experiment, although persistence beyond the short term remains open.^[11] The useful unit is often not “why this?” but “why this rather than the alternative you are likely to choose, and what evidence would make that alternative correct?”

From explanation to a control surface

A decision message should behave less like a persuasive paragraph and more like a compact control surface. It should preserve the distinction between model proposal, world state, authority, typed action, evidence receipt and verified outcome. Once these are collapsed into prose, the human cannot tell whether they are reading a prediction, a permission or a claim that an effect has already occurred.

A message that preserves the boundary

The message does not expose chain-of-thought. It does not provide a story about the model’s internal deliberation. It exposes case evidence, assumptions and control points. This matters because a rationale may be unfaithful, post hoc or simply too expensive to verify. The user needs the smallest set of externally checkable propositions that changes the action.

Nor should the interface pretend every field has equal urgency. The proposal must be readable first. A failure trigger should interrupt the normal visual rhythm. Authority and recovery should sit beside the control they govern, not in a distant policy page. Evidence details may open progressively, but the fact that evidence is stale or dependent cannot be hidden behind interaction. Progressive disclosure is safe only when collapsed content cannot reverse the visible action.

The executable sufficiency test

The lab below converts the argument into a small design instrument. Set the decision context, tick the fields present in the message and inspect the missing obligations. It deliberately produces no “trust score”. A positive result means the message meets this rule set for bounded reliance. It does not prove the evidence is true, the model is calibrated or the organisation has legitimate authority.

The smallest useful implementation is modest: add a typed message object to the system boundary, make missing mandatory fields block higher authority and log both the decision receipt and effect receipt. A TypeScript-like schema makes the separation explicit:

type DecisionMessage = {
  decisionId: string;
  proposal: {
    action: string;
    object: string;
    expectedEffect: string;
  };
  evidence: Array<{
    receiptId: string;
    source: string;
    observedAt: string;
    entitlement: string;
  }>;
  uncertainty: {
    estimate?: number;
    calibrationScope?: string;
    assumptions: string[];
  };
  flipConditions: string[];
  alternatives: Array<{ label: string; discriminatingTest: string }>;
  authority: {
    mode: "advice" | "approval-required" | "bounded-execution";
    expiresAt?: string;
    limits: string[];
  };
  recovery: {
    stopBefore?: string;
    rollback?: string;
    escalateTo: string;
  };
};

A positive result permits the next bounded step, not silent expansion of authority. A negative result means one of two things: enrich the message, or reduce the action’s authority and irreversibility until the current message is adequate. This is a better failure response than adding a longer rationale by default.

To test the message, record the human’s initial decision before advice, the AI proposal, the final decision, eventual ground truth and the actions taken to verify or recover. Then separate beneficial changes from harmful changes. Measure how often a user accepts correct advice, accepts wrong advice, rejects wrong advice and misses useful advice. Add decision time, challenge quality and escalation compliance. A self-reported trust questionnaire can supplement these traces, but it cannot replace them.

A positive experiment should show more discrimination, not simply lower agreement. If both acceptance of correct advice and acceptance of wrong advice fall, the interface may have created blanket distrust. If decision accuracy improves but recovery becomes slower, the design may have moved risk rather than reduced it. The protocol must therefore specify the loss function before the interface is tuned.

Negative results should be diagnosed by type. A coverage failure means a required field was absent. Add the field or reduce authority. A truth failure means a present field was wrong or stale. Repair provenance and source controls. A competence failure means the recipient could not use the message. Change the role, training or escalation route. A control failure means the person understood the risk but could not stop or recover. Redesign the transaction boundary. These failures look similar in an overall accuracy score, but they demand different interventions.

This classification also prevents interface teams from blaming users too quickly. If a reviewer repeatedly accepts bad advice, the cause may be overtrust, but it may also be missing evidence, a misleading default, an authority policy that punishes delay, or an override path that takes too long. Decision sufficiency is a property of the whole work system. Evaluation should therefore include the cost of verification, the organisational consequences of disagreement and whether the human’s challenge actually reaches an authoritative control.

Reliance should be measured behaviourally. Useful outcomes include accepting correct advice, rejecting incorrect advice, changing a wrong initial decision to a right one, avoiding a harmful change, escalating when required and recovering after a detected error. Recent measurement work makes these help–harm distinctions explicit and warns against treating overall agreement as calibration.^[12][13]

stateDiagram-v2
  [*] --> Claim
  state "Four things that interfaces<br/>routinely collapse" as Claim
  state "A minimum, not a summary" as Test
  state "The six fields that usually<br/>carry the work" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Where sufficiency breaks

The framework has a tempting failure mode: once a message contains six neat fields, designers may treat it as a certificate. That would repeat the original mistake. A message can be complete in form and false in substance. Provenance can point to a compromised source. A calibration claim can be out of date. A flip condition can omit an unknown failure mode. An authority label can contradict actual permissions.

Miscalibrated confidence is another boundary. Recent experimental work found that people struggled to detect miscalibration; overconfident systems promoted overreliance, underconfident systems promoted under-reliance, and simply disclosing calibration problems did not necessarily improve decision efficacy.^[14] The lesson is not to hide uncertainty. It is to connect uncertainty to an action rule, such as “below this boundary, seek an independent source” or “this estimate is not valid after a bank-detail change”.

Control can also alter trust without improving correctness. Allowing people to modify an imperfect algorithm slightly has been shown to reduce algorithm aversion.^[15] That is valuable when control enables correction, but dangerous when the control is cosmetic. A slider that lets the user nudge a forecast may create ownership without providing a valid challenge path. Contestability must change the causal route to the outcome, rather than only the feeling of participation.

Recovery has the same regional character. After the compromised payment incident, the organisation should not try to restore a global trust score. It should identify the failed condition, change the control and demonstrate the corrected region. For example: account changes within 48 hours now force callback verification; the callback source is independent; execution remains blocked until a receipt exists. Reliance may then return for payments inside that repaired envelope while staying low elsewhere. Figure 9 depicts this difference between slow-moving reputation and fast case-specific control.

Sufficiency is role-relative

Decision sufficiency also changes with the receiver’s competence. A payment analyst may recognise why a recent bank-account change is diagnostic; a temporary approver may not. The same sentence can therefore support safe challenge for one role and only passive acceptance for another. This is not an excuse to profile individuals or bury them in training. It means the message contract must declare the expected operations and the recipient role that can perform them.

A practical test is substitution. Give the same case to three authorised roles: the system operator, a domain reviewer and a manager who holds approval authority but lacks domain depth. Ask each to identify the case boundary, name a discriminating check and state the safe next action. If only one can do so, the message is not generally sufficient. Either route the decision to that role, add the missing scaffold, or lower authority. Extra prose is not automatically the scaffold. A trusted lookup, a comparison baseline or an explicit hold control may do more work than another explanation paragraph.

Approval authority without challenge capability is not meaningful human control. A role-relative test prevents the governance fiction of assigning accountability to someone who cannot causally affect the decision.

The strongest alternative explanation

Some decisions do not reveal ground truth quickly enough to calculate appropriate reliance case by case. A strategic forecast may take years to resolve. A welfare decision may involve values rather than one correct label. In such settings, the test should shift from “was the AI right?” to process evidence: Were rival options represented? Did the message expose the decisive assumptions? Could affected parties contest the evidence? Was authority exercised as declared? Did later readback match the authorised action? These measures do not prove good judgement, but they keep the decision auditable while outcome evidence matures.

Delayed ground truth also makes learning vulnerable to selective feedback. Teams often observe outcomes for approved cases but not for rejected ones. They may see successful payments but not the fraud avoided by a hold, or see escalations as delay without counting prevented harm. A sufficiency study must record counterfactual missingness and near-misses, otherwise the system may appear helpful merely because its failures are easier to observe than its prevented losses.

Perhaps the entire effect comes from ordinary interface quality. Clear writing, better training and lower cognitive load may explain why structured messages work. That alternative is serious. The proposed framework earns independent value only if Update–Challenge–Continue predicts reliance and recovery beyond readability, message length and delay. Experiments should therefore match reading time, visual salience and recommendation quality.

There are also domains where compression is inappropriate. Value conflicts, contested public decisions and ambiguous diagnoses may require plural perspectives rather than a minimal action message. People may lack the expertise needed to challenge a model. Some consequences cannot be made reversible. Institutional trust can depend on fairness, legitimacy and accountability that no case-level interface can provide. NIST’s AI Risk Management Framework correctly treats trustworthiness as a contextual combination of validity, safety, security, accountability, transparency, explainability, privacy and fairness, not as a single display feature.^[16]

The framework also cannot establish consciousness, agency or moral status. It concerns communication in joint cognitive work. A system may issue a decision-sufficient message without understanding it, and a human may still anthropomorphise the source. The relevant evidence is whether the configured interaction supports better case-specific judgement and safer control.

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: A reliable answer can still be an unsafe message
  M->>E: Four things that interfaces routinely collapse
  E-->>M: compare rival explanations
  E->>D: Grant only the authority the message can support
  D-->>Q: revise scope or act

Grant only the authority the message can support

When an AI interface seems untrustworthy, teams usually reach for one of three remedies: show a confidence score, add an explanation, or require a human approval click. None is sufficient by itself. Confidence can be miscalibrated. Explanation can persuade without discriminating. Approval can become ceremonial when the human lacks evidence, challenge or recovery.

The changed decision is this: before increasing explanation or autonomy, specify the decision message that a competent collaborator needs to update, challenge and safely continue. Then test that message under the actual time, authority and reversibility conditions. If the message fails, do not ask the user to “trust less” or read more. Add the missing evidence or control, or reduce the system’s authority.

The 09:17 assistant did not fail because it communicated too little in the abstract. It failed because it omitted the one fact that separated a routine match from a compromised payment, and it concealed the action boundary behind a fluent assurance. A safer system does not try to become maximally believable. It makes the next decision inspectable, contestable and recoverable.

Trust calibration is therefore an outcome of well-shaped communication and bounded control, not a sentiment to optimise directly.

Glossary

Appropriate reliance

Accepting useful advice and rejecting harmful advice in the cases where each response is warranted.

Decision-sufficient communication

The least attention-costly message that lets a recipient update, challenge and continue or recover for a declared decision context.

Flip condition

An observation or assumption change that would reverse, defer or materially alter the recommendation.

Decision receipt

A record of the proposal, evidence, policy, authority and human or automated decision at the time of action.

Effect receipt

A readback of what the authoritative system actually did, including identifiers needed for rollback or compensation.

Reliance slope

The difference between following AI advice when it is correct and following it when it is wrong, measured across cases or time.

References

1. Parasuraman, R. and Riley, V. (1997). “Humans and automation: use, misuse, disuse, abuse.” Human Factors, 39(2), 230–253. https://doi.org/10.1518/001872097778543886
2. Lee, J. D. and See, K. A. (2004). “Trust in automation: designing for appropriate reliance.” Human Factors, 46(1), 50–80. https://doi.org/10.1518/hfes.46.1.50_30392
3. Jacovi, A., Marasović, A., Miller, T. and Goldberg, Y. (2021). “Formalizing trust in artificial intelligence: prerequisites, causes and goals of human trust in AI.” FAccT ’21, 624–635. https://doi.org/10.1145/3442188.3445923
4. Hoff, K. A. and Bashir, M. (2015). “Trust in automation: integrating empirical evidence on factors that influence trust.” Human Factors, 57(3), 407–434. https://doi.org/10.1177/0018720814547570
5. Yin, M., Vaughan, J. W. and Wallach, H. (2019). “Understanding the effect of accuracy on trust in machine learning models.” CHI ’19. https://doi.org/10.1145/3290605.3300509
6. Zhang, Y., Liao, Q. V. and Bellamy, R. K. E. (2020). “Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making.” FAT* ’20, 295–305. https://doi.org/10.1145/3351095.3372852
7. Bansal, G. et al. (2021). “Does the whole exceed its parts? The effect of AI explanations on complementary team performance.” CHI ’21. https://doi.org/10.1145/3411764.3445717
8. Buçinca, Z., Malaya, M. B. and Gajos, K. Z. (2021). “To trust or to think: cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making.” Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1). https://doi.org/10.1145/3449287
9. Vaccaro, M., Almaatouq, A. and Malone, T. (2024). “When combinations of humans and AI are useful: a systematic review and meta-analysis.” Nature Human Behaviour, 8, 2293–2303. https://doi.org/10.1038/s41562-024-02024-1
10. Swaroop, S., Buçinca, Z., Gajos, K. Z. and Doshi-Velez, F. (2024). “Accuracy-time tradeoffs in AI-assisted decision making under time pressure.” IUI ’24, 138–154. https://doi.org/10.1145/3640543.3645206
11. Buçinca, Z., Swaroop, S., Paluch, A. E., Doshi-Velez, F. and Gajos, K. Z. (2025). “Contrastive explanations that anticipate human misconceptions can improve human decision-making skills.” CHI ’25. https://doi.org/10.1145/3706598.3713229
12. Schemmer, M., Kühl, N., Benz, C., Bartos, A. and Satzger, G. (2023). “Appropriate reliance on AI advice: conceptualization and the effect of explanations.” IUI ’23, 410–422. https://doi.org/10.1145/3581641.3584066
13. Lee, M. H. (2026). “From accuracy to readiness: metrics and benchmarks for human–AI decision-making: an initial exploration.” CHI EA ’26, 1–10. https://doi.org/10.1145/3772363.3798377
14. Li, J., Yang, Y., Zhang, R., Liao, Q. V., Song, T., Xu, Z. and Lee, Y.-C. (2024). “Understanding the effects of miscalibrated AI confidence on user trust, reliance, and decision efficacy.” arXiv:2402.07632. https://arxiv.org/abs/2402.07632
15. Dietvorst, B. J., Simmons, J. P. and Massey, C. (2018). “Overcoming algorithm aversion: people will use imperfect algorithms if they can (even slightly) modify them.” Management Science, 64(3), 1155–1170. https://doi.org/10.1287/mnsc.2016.2643
16. National Institute of Standards and Technology (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1. https://doi.org/10.6028/NIST.AI.100-1