Most agentic systems I audit have a model problem on paper and an interface problem in practice. The model is fine. What's missing is the connective tissue between human and agent: the surfaces where trust is earned, work is handed off, and failure is communicated. That tissue doesn't emerge from prompt engineering. It has to be designed.
I've spent the last three years putting agents into enterprise workflows (procurement, claims, engineering ops) and the pattern is consistent. The teams that treat the human-agent boundary as a first-class design object ship systems people actually use. The teams that bolt a chat window onto an orchestrator ship demos. Here is the design vocabulary I've settled on.
Trust calibration is the product
The goal of a human-agent interface is not to maximize trust. It is to calibrate trust to actual capability: per task, per context, continuously. An operator who over-trusts an agent stops checking its work and inherits its errors silently. An operator who under-trusts it re-does everything and you've built an expensive suggestion engine.
Calibration is a design problem with concrete levers:
- Show confidence where it's meaningful, hide it where it's noise. A raw probability score is worse than nothing; users anchor on it without understanding what it measures. What works is behavioral confidence: "I've done 340 tasks like this with a 2% correction rate" beats "confidence: 0.87" every time.
- Make competence boundaries visible. The interface should communicate what the agent is not good at, unprompted. Agents that volunteer "this contract type is outside my training distribution, review clause 4 carefully" earn more durable trust than agents that never admit limits, because the first correction the user makes doesn't feel like betrayal.
- Let trust be earned incrementally. Design permission tiers the agent graduates through as its track record accumulates. Autonomy is not a launch setting; it's a state the system moves toward, visibly, with the human watching the scoreboard.
The failure mode I see most often: teams optimize for the appearance of reliability (smooth language, no hedging) and produce systems that are trusted far beyond their competence. That's not a UX win. That's a latent incident.
Progressive disclosure of decision evidence
Hidden chain-of-thought is not an audit artefact. It may be unavailable, unstable across runs, misleading when treated as a faithful account of causation, and inappropriate to expose to an operator. Do not make private model deliberation the control. What users and auditors need is a progressively disclosed decision record built from observable evidence, checks, actions and outcomes.
I structure that record in three layers:
Layer 1: the claim
One sentence. What the agent did or wants to do, and the single most load-bearing reason. "Rejected invoice #4471: vendor bank details changed since last payment." This is what 90% of interactions should require.
Layer 2: the decision skeleton
The recorded path, not private deliberation. Show which checks ran, what evidence was retrieved, which policy or precedent applied, which explicit options were evaluated and why the selected option met the stated criteria. Five to ten scannable items are usually enough. This is what a user opens when the claim surprises them.
Layer 3: the operational trace
Tool calls and arguments, retrieval identifiers and excerpts, policy responses, approval events, component versions and outputs: timestamped and exportable. Exclude hidden chain-of-thought. Nobody reads this event record in real time; it exists for validation, incident review and systematic-error analysis. Design it for search and replay, not narrative reading.
The critical rule is that each layer must be supported by the observable layer below it. A clean one-line claim without matching evidence, policy results and actions is a claim with no proof. If a decisive source was missing or a check was unavailable, the first layer must say so.
Interruption design
Agents work asynchronously; humans don't. Every agent interruption is a withdrawal from a finite attention budget, and most agentic systems spend that budget like they printed it.
The design question is never "should the agent ask?" It's "what does this interruption cost, and what does it buy?" I force teams to classify every interruption into one of four types:
- Blocking asks: the agent cannot proceed without input. Minimize these ruthlessly; each one converts an autonomous system back into a form.
- Preemptive flags: the agent can proceed but a human would want to know first. These deserve a queue, not a modal. Batch them.
- FYI notifications: completed work, no action needed. These belong in a digest. If your agent sends more than a handful of real-time FYIs a day, users will mute the channel and miss the one that mattered.
- Escalations: something is wrong. These are the only interruptions allowed to be loud, which means everything else must be quiet or the escalation drowns.
One structural pattern that pays for itself: interruptions carry their own context. When the agent asks a question, the surface includes what it was doing, what it's tried, what happens if the human doesn't answer, and a default it will take on timeout. A question without a timeout default is a process held hostage.
Approval surfaces
The approval step is where most agent deployments quietly die. Design it wrong and you get one of two failure modes: rubber-stamping (the human approves everything without reading, adding latency but no safety) or bottlenecking (approvals pile up and the agent's throughput collapses to the human's).
Principles that have held up in production:
- Approve diffs, not states. Never ask a human to validate an entire output. Show what changed, what's unusual about this instance versus the last hundred, and what the agent is least sure about. The reviewable unit is the delta.
- Risk-tier the surface itself. Low-stakes actions get post-hoc review with an undo window. Medium-stakes get batched approval. High-stakes get a full synchronous review with the layer-2 decision skeleton pre-expanded. One approval UX for all actions guarantees it's wrong for most of them.
- Make rubber-stamping measurable. Track time-to-approve and correction rate per reviewer. When someone approves 200 items in four minutes, your approval surface is theater, and it's better to know that than to carry the illusion of oversight into an audit.
- Reversibility is a design material. The cheapest way to reduce approval friction is to make actions undoable. Every action you make reversible is an approval you can remove.
Communicating failure
Agents fail differently than software. Software fails loudly and consistently; agents fail plausibly and intermittently. The interface has to compensate for the fact that agent failures don't look like failures.
What I build in:
- Failure taxonomies over error messages. "Something went wrong" is unacceptable from a system that took actions in the world. The agent should distinguish: I couldn't do it, I did it but I'm unsure, I did the wrong thing and know it, I may have done the wrong thing and can't tell. Each maps to a different human response.
- Blast-radius statements. Every failure communication answers: what was affected, what was not affected, and what the agent already did about it. Users forgive failure; they don't forgive discovering the scope themselves.
- No apology theater. Agents that over-apologize train users to skim past contrition to find the facts. State what happened, state the fix, move on. Warmth is fine; groveling is noise.
- Post-failure recalibration. After an error, the interface should visibly tighten: more disclosure by default, lower autonomy tier, explicit acknowledgment when the pattern that caused the failure recurs. Trust that resets to full after every failure was never trust.
The team is the unit of design
The through-line in all five patterns: stop designing the agent and start designing the team. A human-agent team has a division of labor, a communication protocol, an escalation path, and a shared memory: the same things any team has. The interface is where all of them live.
The models will keep improving on their own. The interfaces won't. That asymmetry is exactly why this layer is where the durable work is.
Design the division of labour before the screen
The interface is the visible edge of an operating model. It cannot repair an unclear division of labour. Before sketching a panel or notification, assign responsibility for sensing, proposing, checking, deciding, executing and learning.
The agent may retrieve evidence and draft a recommendation. A human may own the final judgement. A deterministic service may execute an approved action. An operations team may own recovery. A policy owner may resolve ambiguity. Those roles should be explicit for each workflow state.
Decision rights should follow competence and consequence. A reviewer may understand a customer case but lack authority to change a payment. An engineer may understand the trace but lack authority to interpret policy. A person in the loop is not automatically the right person in the loop. The interface should route to a role that can understand the evidence and act within a defined ceiling.
| Team function | Agent contribution | Human responsibility | Interface obligation |
|---|---|---|---|
| Sense | Retrieve, classify and detect anomalies | Confirm missing or contested facts | Show source, freshness and gaps |
| Propose | Generate options and a recommendation | Apply context and judgement | Separate options from the recommendation |
| Check | Run policy, permission and consistency checks | Resolve ambiguous rules | Show passed, failed and unavailable checks |
| Decide | Rank or prepare the decision | Accept, amend, reject or escalate | State the decision request and consequence |
| Execute | Prepare a bounded tool call | Authorise material action where required | Preview effect, ceiling and reversal |
| Learn | Aggregate edits, overrides and outcomes | Interpret patterns and change policy | Capture structured reasons without burden |
This matrix exposes false handoffs. If the human is nominally responsible for checking but the interface hides sources, responsibility is fictional. If the human owns a decision but cannot amend the proposal, the screen is an approval gate rather than a judgement surface. Authority, information and available action must meet in the same role.
Build the interface around a decision packet
The primary object should be a decision packet, not a chat transcript. A packet contains the case, the decision request, decisive evidence, material uncertainty, proposed action, consequence and available alternatives. It can link to a trace for investigation without making the trace the normal reading path.
The packet needs a stable hierarchy. Put the current decision and deadline first. Show facts next. Show the recommendation after the reviewer has engaged with evidence when anchoring would matter. Expose uncertainty beside the claim it affects. Keep policy and source context one interaction away.
Progressive disclosure should follow risk and reader need. The first layer contains the decision and decisive facts. The second layer contains source excerpts, policy checks and uncertainty. The third contains the complete operational event trace, configuration and technical diagnostics, not hidden chain-of-thought. A case handler should not need the third layer for routine judgement. A validator or incident investigator must be able to reach it.
Do not confuse hidden detail with simplicity. A clean screen can conceal a weak evidence path. A dense screen can overwhelm until users rely on the recommendation. Test whether reviewers locate the right fact, not whether they say the interface looks clear. The successful design makes scrutiny easier than acquiescence.
Calibrate trust through visible system states
Trust is not a single setting. Users form beliefs from speed, tone, past success, error handling and visual authority. The interface should help those beliefs track actual capability.
Avoid human-like confidence theatre. A polished explanation may increase trust without adding evidence. Numeric confidence may appear precise while remaining uncalibrated. Use concrete states: source verified, source missing, policy conflict, tool unavailable, outside scope, review required. These states tell the user what to do.
Trust also needs memory. After a detected failure, tighten the relevant path. Expand evidence by default. Reduce the autonomy tier. Surface the known failure pattern when it recurs. Return to the previous state only after evidence supports the change. A system that looks identical before and after a serious error teaches users that errors have no operational meaning.
The Google PAIR Guidebook offers practical patterns for human-centred AI products. The ICO explanation guidance emphasises context, audience and meaningful information. These sources support a central design choice: show the kind of evidence and explanation the current decision needs, not every internal detail.
Treat interruption as a scarce resource
An agent can request attention cheaply. Human attention is limited and fragmented. Interruption policy therefore belongs to the operating model.
Classify interruptions by urgency, consequence and expiry. A potential unauthorised payment may require immediate synchronous review. A low-risk drafting uncertainty may wait in a batch. A source refresh failure may alert the service owner rather than every end user. The recipient should be the person who can change the outcome.
Every interruption should contain a reason and a requested action. “Agent needs help” forces the person to diagnose the interruption. “Confirm whether policy exception 4 applies because the customer moved jurisdiction” creates a decision. Include the consequence of delay and the safe state while waiting.
Notification volume needs a budget. Track interruptions per case, per reviewer and per failure class. Repeated low-value alerts indicate a routing or policy problem. Do not solve them only with muted notifications. Remove the cause, batch the decision or change the autonomy boundary. Alert fatigue is evidence that the division of labour is poorly tuned.
Design failure as a recoverable workflow
The interface should distinguish four states. The agent could not act. The agent acted and knows the action failed. The agent acted but cannot verify the result. The agent may have acted incorrectly. Each needs a different response.
For a verified failure, show affected records, the current system state and the recovery action. For an uncertain effect, prevent duplicate execution until reconciliation completes. For possible wrong action, expose blast radius and preserve the trace. For inability to act, provide a safe manual route.
Reversal should be designed before approval. Show whether an action can be undone, by whom and within which window. A reviewer may accept a reversible low-value action with sampled oversight. An irreversible customer notice deserves a stronger pre-action check. Reversibility is not a footnote; it changes the appropriate human-machine allocation.
Accessibility remains part of safety. A reviewer who cannot perceive an evidence marker or operate an escalation control cannot provide effective oversight. Apply WCAG 2.2 to the actual decision workflow. Test keyboard use, screen readers, zoom, colour independence and time limits with realistic packets.
Measure the team, not just the model
Team performance includes outcome quality, time, cognitive load, escalation, correction and learning. Model acceptance rate alone can reward automation bias. Human override rate alone can punish useful correction. Pair measures so that one cannot be optimised at the expense of the decision.
Measure evidence engagement with later outcomes. Measure review time with error detection. Measure escalation rate with escalation precision. Measure autonomy with reversibility and harm. Measure productivity with unresolved work and rework. Segment by consequence and case type.
Qualitative observation remains valuable. Watch reviewers use the interface. Ask them to explain the decision in their own words. Note where they leave the product to find context. Compare stated policy with actual workarounds. A short observed study often exposes a failure that event logs label as success.
Use disagreement as product data. A structured reviewer amendment can reveal a missing source, poor policy representation, wrong route or weak interface. Route each reason to an owner. Close the loop by showing reviewers when recurring feedback changed the system. Feedback without visible consequence becomes another form field people learn to ignore.
The operating review should include product, operations, risk, accessibility, security and policy owners. Review a small sample of complete cases, not only aggregate charts. Examine easy approvals, amendments, escalations and failures. Decide which part of the team needs change: model, evidence, policy, interface, staffing or authority.
Move from interface prototype to operating routine
Start with one consequential decision and reconstruct how it is made today. Observe the real work. Record the evidence people seek, the systems they open, the questions they ask and the points where authority changes hands. Do not assume the written procedure describes the actual path.
Map the proposed team allocation onto that path. Mark agent tasks, deterministic controls, human judgements and policy ownership. For each transition, state the packet transferred and the receiving role. Remove steps that merely ask a person to confirm what the system already decided.
Prototype the decision packet with static content before connecting a model. Use difficult cases and missing evidence. Ask reviewers to think aloud. Check whether they find the decisive fact, understand the consequence and know which action is available. Revise information order before adding visual polish.
Add agent output only after the evidence surface works. Compare a review with the recommendation hidden, delayed and immediately visible. The purpose is not to prove one universal order. It is to find where anchoring or unnecessary effort appears in this workflow.
Instrument interactions sparingly. Record evidence opening, amendment, disagreement, escalation, reversal and later correction. Avoid surveillance measures that encourage speed over judgement. Explain to reviewers how the data improves the system and how it will not be used.
Run a shadow period. The agent can prepare packets while the established process remains authoritative. Compare outcomes and missed information. Use the period to estimate queue demand and to train reviewers on states, limitations and escalation.
Introduce autonomy by consequence tier. Begin with suggestions or reversible internal actions. Increase only where evidence shows that the team, not merely the model, performs well. Keep an explicit route back to the earlier tier after incidents or drift.
Review the routine after launch. Observe real cases again. Check workarounds, alert load and specialist queues. Ask whether reviewers still understand their authority. A human-agent interface is finished only temporarily; the operating routine must adapt when behaviour, demand or policy changes.
The handoff is a designed state change
A handoff should transfer a case, its evidence and a clear decision request. It should not transfer the agent’s entire transcript. The reviewer needs enough context to act, plus a direct path to inspect more.
| Surface element | Question it answers | Failure if absent |
|---|---|---|
| Decision request | What must I decide now? | Reviewer reads without acting |
| Evidence panel | Which facts support the proposal? | Fluency substitutes for verification |
| Uncertainty disclosure | Where is the system least reliable? | Attention goes to easy details |
| Consequence preview | What changes if I approve? | Blast radius stays abstract |
| Reversal path | How can this action be undone? | Review becomes needlessly conservative |
Match interface weight to consequence
| Consequence | Default interaction | Evidence posture | Feedback captured |
|---|---|---|---|
| Low and internal | Inline suggestion | Source on demand | Edit distance |
| Moderate and reversible | Batched review | Key source visible | Accept, amend, undo |
| High and customer-facing | Synchronous review | Evidence first | Reason and counterfactual |
| Irreversible or rights-affecting | Specialist approval | Full event trace and policy record | Signed decision record |
Human oversight is a capability, not a staffing label. The reviewer needs authority, competence, time and an interface that exposes the right evidence. Article 14 of the EU AI Act makes effective oversight and appropriate human-machine interfaces explicit for high-risk systems. The Google PAIR Guidebook offers practical human-centred design patterns. The ICO explanation guidance distinguishes explanation needs by audience and context. WCAG 2.2 supplies the accessibility baseline. The NIST AI RMF links these design choices to governed measurement and risk treatment.
The operational test is simple. Can a trained reviewer locate the decisive evidence, understand the consequence and disagree without friction? If disagreement is hard to express, the interface is asking for compliance rather than judgement.