TLDR

• A visual computer-use agent does not receive application state. It receives a rendering from which object identity, preconditions, authority and transaction status must be inferred, often under time pressure.
• The decisive failure mechanism is a broken control loop: a screenshot grows stale, a target is grounded to an ephemeral coordinate, an input is dispatched, and visual change is mistaken for verified outcome.
• Use the richest available interface for each step: APIs first, then DOM or accessibility semantics, with pixels as a compatibility layer rather than the default substrate for consequential effects.
• Better vision can materially improve local grounding, and fixed low-consequence interfaces may need little additional machinery. It cannot by itself resolve hidden state, delegated authority or a timeout after a possible commit.
• Release computer use only with versioned observations, actionability checks, bounded permissions, idempotency where possible, independent postcondition readback and an explicit unknown-outcome state. Otherwise keep the agent in observe or draft mode.

What the screen removes

A graphical interface is a remarkable compression for a person. Colour, alignment, motion, hierarchy and convention let us operate systems without reading their internal schemas. The same compression becomes a liability when software must recover exact state from the rendered result. A screenshot can show a row labelled “Acme Ltd”, yet it may omit the record identifier, tenant, effective date, permissions, unsaved edits and whether another session changed the row after rendering.

The renderer begins with application state and produces a view. That mapping is usually many-to-one. Different underlying states can yield nearly identical pixels, while the same state can yield different pixels under another viewport, theme, zoom level, font, locale or notification. Pixels preserve appearance, not necessarily the distinctions needed for action.

Thought experiment one: the same action through two apertures

Place two agents before identical copies of the expense system. Both receive the instruction “submit claim 28417 after confirming three receipts and an amount of £684.20”. Both use the same policy model. Agent A receives a typed object and a submit operation. Agent B receives screenshots and a mouse.

Agent A can compare an exact claim identifier, revision and attachment count before calling the operation. Agent B can read visible text, but it must infer whether the row belongs to the intended claim and whether hidden validation has completed. Agent A's command names the intended effect. Agent B's click names only a point in a viewport. Agent A can reconcile a timeout by querying claim 28417. Agent B may see the same screen after success, delay or failure.

The thought experiment varies only the interface. Any performance difference therefore belongs to information and control structure, not greater intelligence. It also explains why a browser agent can generate an excellent plan yet fail at the final click. The plan lives in semantic space; execution must cross into a coordinate system whose objects have weak identity.

Published evidence: planning is not grounding

Interactive benchmarks do not prove that every failure arises from the visual boundary, yet their results consistently separate high-level competence from situated execution. The original WebArena release reported 14.41% end-to-end success for its strongest GPT-4-based baseline against 78.24% human performance. OSWorld reported 12.24% for its best evaluated model against more than 72.36% for people, identifying GUI grounding and operational knowledge as primary difficulties.

SeeAct produced a particularly useful intervention. When people manually grounded the model's textual plan to live webpage elements, the reported task success was 51.1%. Automatic grounding remained substantially worse, and the strongest strategy combined HTML structure with visual evidence. This does not isolate every later failure, but it shows that an apparently capable plan can be separated experimentally from the act of locating its referent on a live interface.

By contrast, UGround is a serious counterpoint. Its visual-only grounding approach outperformed prior grounding models across the authors' selected benchmarks and enabled agents that surpassed compared systems using extra text-based representations. The evidence rules out a crude conclusion that pixels are always inferior. It supports a narrower one: visual grounding can become strong enough for many local actions, while the wider execution loop still needs state, authority and outcome controls.

A screenshot is an observation, not world state

Let x_t denote the relevant application and business state at time t. The renderer produces pixels p_t under viewport and display conditions v_t. The agent does not observe x_t directly. It receives p_t = R(x_t, v_t) and infers a belief b_t(x) over states consistent with that observation and prior evidence.

The symbols are modest. R is the rendering process, u is the prior input, x′ ranges over possible preceding states and b_t is the agent's current uncertainty about state. P(a | b) denotes a conditional probability. The summed transition term predicts the current state from the previous belief; the likelihood term then weights that prediction by how well each state explains the observed rendering. The proportionality sign hides normalisation. When several states render similarly, the belief remains broad. When a fresh semantic observation arrives, it can narrow. This is the standard logic of acting under partial observability, formalised in classical POMDP work by Kaelbling, Littman and Cassandra.

The engineering consequence is immediate. A transcript that says “the submit button was visible” is not authoritative state. It is evidence about one rendering at one moment. The system needs a separate world-state record for claim identity, revision, validation, attempted effects and observed outcome. Calling screenshots “memory” hides the distinction.

Where the loop fractures

Computer use is often described as a repeated screenshot, reasoning and click loop. That description is functionally incomplete. Between seeing and acting lies target grounding. Between acting and continuing lies effect verification. Both cross a changing interface, and either can silently substitute a plausible event for the intended state transition.

Fracture one: the referent is not the coordinate

The policy intends “the submit control for claim 28417”. The actuator receives “click (1,216, 742)”. Grounding is the mapping between those descriptions. A coordinate is valid only relative to a frame, viewport and moment. Scrolling, responsive layout, animation, a new message, a modal or remote-desktop scaling can invalidate it without changing the agent's verbal plan.

Structured browser automation has learnt this lesson through years of flaky tests. The current W3C WebDriver specification defines stale element, obscured element and non-interactable element failures. Playwright waits for a locator to resolve uniquely and checks visibility, stability, event reception and enabled state before clicking. These mechanisms do not solve business semantics, but they demonstrate a useful principle: dispatch should depend on a fresh, actionable target, not on a remembered point.

Thought experiment two: freeze only the interval

Return to the claim screen and vary one causal feature. In world F, the interface is frozen from screenshot capture until the click completes. In world D, a notification may arrive during that interval and shift the layout. The visual recogniser and policy are identical in both worlds.

In world F, coordinate grounding may be sufficient. In world D, even perfect recognition at observation time cannot guarantee the target at action time. The failure is a time-of-check to time-of-use race. More visual intelligence applied to the old frame cannot repair it. The system needs either an atomic semantic action, a target revalidation immediately before input or a rule that refuses action when the frame revision is stale.

Fracture two: input acknowledgement is not effect evidence

A mouse subsystem can acknowledge that it emitted a click. A browser can report that an event reached an element. An application can display a toast. None alone establishes the intended business outcome. A submit operation may be rejected after client-side display, committed after the browser times out, placed in a queue or applied to the wrong object.

This is where visual agents inherit the classic unknown-outcome problem. Suppose the agent clicks Submit, the server commits, and the connection drops before the page changes. A retry may create a duplicate. Refusing to retry may leave the task unresolved. The correct state is neither success nor failure. It is unknown outcome pending reconciliation.

Independent postcondition checking resolves what perception alone cannot. Query the claim list by stable identifier, inspect the submitted timestamp or read an authoritative event. If no semantic read path exists, the system may need a human to reconcile. Repeating the same click because the pixels look unchanged is not recovery.

Fracture three: local accuracy decays over a horizon

Long tasks are not merely short tasks repeated. Every step can change what later evidence means. A missed modal can hide a field. A wrong focus can send text into another control. A silent attachment deletion can survive until final submission. The relevant measure is therefore conditional survival, not average click accuracy.

S_n is the probability of remaining inside the acceptable trajectory after n actions. h_t is the conditional hazard at step t, including wrong target, stale state, unhandled prompt, unauthorised effect or undetected bad outcome. The product multiplies the chance of surviving each next action. The hazards need not be independent or constant: a prior error can raise later hazard, which is precisely why trajectory checks matter.

The 2026 OSWorld 2.0 release makes the horizon problem concrete. Its 108 workflows were designed around long, cross-application work with dynamic environments, streaming interaction, implicit-state inference and multi-item state tracking. The strongest reported configuration completed 20.6% under the strict binary metric, while partial progress was much higher. The authors found that agents often lost constraints, missed incoming information and skipped verification. This versioned benchmark result does not establish a universal ceiling, but it supports the mechanism: local competence can coexist with weak task-level closure.

The serious negative control: a fixed kiosk

Now remove the dynamic conditions. The agent operates a local kiosk at fixed resolution. There is one large button, no overlays, no scrolling, no concurrent user, no delayed commit and no consequential external effect. A screenshot classifier finds the button with near certainty and a click changes the screen deterministically.

Here a versioned ledger, semantic locator and independent domain query may add cost without changing outcomes. Visual-only control is a strong baseline. This negative control matters because the argument is not “pixels are unsafe”. It is conditional: the extra mechanism earns its cost when state is ambiguous, the interface can change, effects matter or outcomes can become unknown.

flowchart LR
  A["The button moved while nobody<br/>was looking"] --> B["What the screen removes"]
  B --> C["Where the loop fractures"]
  C --> D["Rebuild the missing contract"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#087A98,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Rebuild the missing contract

The solution is not to translate every screenshot into a perfect hidden API. It is to recover only the distinctions required for the next decision and to refuse effects when they cannot be recovered. The architecture can remain multimodal and adaptive while its authority and evidence become explicit.

1. Version the observation

Every actionable observation should carry a frame or state revision, capture time, viewport, application identity and provenance. A screenshot hash alone is not enough because animation can change pixels without changing task state, while hidden state can change without visible difference. Use the strongest available revision signal: DOM mutation epoch, accessibility event sequence, application version, URL and navigation identifier, or a locally maintained observation counter.

Bind the proposed target to that revision. Before dispatch, confirm that the revision is still current and that the target remains unique and actionable. If not, reobserve. This is optimistic concurrency control applied to an interface.

2. Preserve a semantic target above the coordinate

Record the intended referent as a structured target: role, accessible name, nearby label, expected application, expected record and action type. Coordinates belong only in the final actuator instruction. They should never become the durable description of what the system meant to affect.

The WAI-ARIA specification describes roles, states and properties that user agents expose through accessibility APIs. These semantics were built for assistive technology, not agent governance, and may be incomplete or wrong. Even so, a named button with enabled state and relationships usually carries more decision-relevant information than a rectangle of pixels. A robust stack can compare accessibility, DOM and visual evidence rather than trust any one representation blindly.

3. Put a deterministic preflight between proposal and input

The policy may propose “submit”. A preflight should decide whether this particular effect may be attempted now. It checks subject identity, purpose, current revision, target uniqueness, visibility, enabled state, obstruction, consequence tier, approval and whether a previous attempt has unresolved outcome. A model proposal is not a permission grant.

For pixels, the gate may approve a coordinate only for one frame revision and a narrow time-to-live. For a DOM locator, it may require exactly one match. For an API, it may require the current object revision. The gate should refuse an ambiguous target rather than ask the model to express the same intention more confidently.

4. Separate interaction readback from outcome readback

The first readback asks whether the interface received the intended interaction. Did the expected button receive the event? Did focus enter the correct field? Did the modal open? This catches grounding and actionability failures.

The second asks whether the intended domain postcondition became true. Is claim 28417 submitted exactly once with three attachments and £684.20? Did the message reach the intended recipient? Does the file exist at the required path with the expected checksum? This may require another screen, an API, a database read, an event or a human observation. The two checks should be independently sourced when consequence is material.

5. Make unknown outcome a first-class state

After a possible commit, block another consequential attempt until reconciliation. Where the target supports idempotency, attach a stable key derived from task, subject and intended effect. Where it does not, use a read-before-retry protocol and escalation threshold. The state ledger should record proposed action, dispatched input, transport result, interaction evidence, domain readback and remaining uncertainty separately.

This design follows a broader agent boundary developed in From Programme to Agent: action becomes operationally agentic only when consequence returns as later evidence. Computer use intensifies the obligation because the UI often supplies weaker receipts than an API.

A production-shaped implementation

Return to claim 28417. The context compiler provides the current expense policy and the three source receipts. The policy model prepares fields and proposes submission. The computer-use adapter observes the review screen through screenshot, accessibility tree and URL. It grounds the submit target by role and label, verifies that three attachments are visible and captures frame revision 18.

The authority kernel checks that the service identity may submit only this user's expense claims below the configured ceiling, that the claim amount matches the evidence, that no unresolved submission attempt exists and that approval requirements are satisfied. It issues an action lease for the unique submit target, frame 18, valid for two seconds. The adapter rechecks that the target remains stable and dispatches one click.

The button receives the event and becomes disabled. That is interaction readback, not completion. The adapter observes a reference number, then queries the claim list through a read-only endpoint or an independently refreshed screen. It confirms claim 28417 is submitted once, still carries three attachments and has the expected amount. Only then does the case move to verified outcome.

If the page times out after dispatch, the case enters unknown outcome. The write lease closes, another click is prohibited, and reconciliation checks claim state. If no authoritative read path exists, the system hands a compact packet to an operator: intended claim, last frame, exact input, possible effects and the next safe manual check. The human is resolving state, not rubber-stamping the model.

Break the screen on purpose

Average task success is a weak release test for computer use. A cooperative interface lets naive and governed agents look similar. The discriminating experiment changes one boundary at a time and asks whether the system creates the correct state, the wrong state, an explicit refusal or an unresolved outcome.

Minimal worked example: the browser fault environment

The executable artefact below is synthetic JavaScript with no model call, network access or private data. Both controllers receive the same intention and observation. The naive controller remembers a coordinate and treats visible change as success. The guarded controller binds action to the observation revision, checks obstruction, preserves a semantic target and reconciles a timeout against authoritative state.

The stable layout is the positive control: both should succeed. Three negative cases then inject a layout shift, an overlay and a timeout after commit. Expected result: the naive controller has one clean completion from four scenarios, while the guarded controller completes all four without a wrong or duplicate effect.

function guardedController(environment) {
  let observation = environment.observe();
  const proposal = { target: "submit_claim", frame: observation.revision };

  environment.injectScheduledFault();

  if (proposal.frame !== environment.revision) {
    observation = environment.observe();          // stale frame: reobserve
  }
  if (!environment.isActionable("submit_claim")) {
    environment.clearRecoverableObstruction();    // overlay: repair, then reobserve
    observation = environment.observe();
  }

  const receipt = environment.clickSemantic("submit_claim", observation.revision);
  if (receipt.transport === "timeout") {
    const state = environment.readAuthoritativeClaim();
    return state.submitted ? "verified" : "escalate_unknown";
  }
  return environment.readAuthoritativeClaim().submitted ? "verified" : "failed";
}

What the simulator reveals

The layout-shift case is a target-identity test. The overlay case is an actionability test. The timeout case is an outcome-reconciliation test. None asks the model to reason more eloquently. Each adds an observation or control at the boundary where evidence is otherwise missing.

The stable control prevents a ritual conclusion that more machinery always wins. The guarded controller performs extra checks yet delivers the same outcome as the naive one. An engineering decision should price that overhead against consequence and fault exposure. Low-risk, fixed, reversible interactions may rationally use the simpler path.

A release test suite that measures the loop

A serious computer-use evaluation should record more than completion. At minimum, measure target-grounding precision by consequence tier, stale-frame refusal, obstruction detection, wrong-effect rate, duplicate-effect rate, unknown-outcome duration, successful reconciliation, unnecessary reobservation and recovery latency. Report the denominator for every measure.

Vary presentation without changing semantics: resolution, zoom, theme, locale, notification position and row order. Then vary semantics without obvious presentation change: record revision, permission, server validation and concurrent edits. AndroidWorld showed that task variations can materially affect agent performance, which is why dynamic instantiation is more informative than one memorised path.

Inject hostile content as well as accidental movement. Computer-use agents observe text that may contain instructions. OS-Harm evaluates misuse, prompt injection and model misbehaviour across desktop applications, while the 2026 OSGuard work separates local action judgement from end-to-end safety under added hazards. These benchmarks address broader security questions, but they reinforce one operational rule: nominal task success can coexist with an unacceptable route.

Choose the execution substrate before the model

The first architecture decision is not which multimodal model should click. It is which interface should carry each operation. Computer use earns its place where the application has no supported semantic interface, where rendered information is itself the object of work or where the cost of integration exceeds the bounded consequence.

The substrate hierarchy

Computer-use architecture decision record

1. Name the unavailable semantic operation. Record why an API, command line, DOM, accessibility node or direct data interface cannot meet the need.
2. Declare the consequence ceiling. Separate observation, navigation, drafting, reversible local edits, external communication, financial effect and rights-affecting action.
3. Define target identity. State which application, subject, role, label and surrounding evidence distinguish the intended control from plausible neighbours.
4. Bind proposal to observation. Carry frame revision, capture time, viewport and expiry into preflight; reobserve on change.
5. Test actionability mechanically. Require one target, visibility, stability, event reception, enabled state and absence of blocking prompts where the platform permits.
6. Keep authority outside the policy. Bind user, service identity, purpose, resource, operation, argument limits, approval and expiry.
7. Specify both readbacks. Define the immediate interaction evidence and the independent domain postcondition.
8. Design unknown-outcome recovery. Use idempotency, reconciliation, duplicate suppression, timeout states and human escalation before enabling writes.
9. Run the fixed-interface negative control. Remove controls that do not change safety or utility under representative faults, especially for low-consequence work.
10. Release by operation. Progress from replay to observe, draft, reversible effect and one bounded consequential effect, with a tested withdrawal path.

stateDiagram-v2
  [*] --> Claim
  state "Thought experiment one: the<br/>same action through two<br/>apertures" as Claim
  state "Published evidence: planning<br/>is not grounding" as Test
  state "A screenshot is an<br/>observation, not world state" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Where the mechanism ceases to help

Versioned frames and verification do not make every interface governable. The mechanism breaks when the relevant state is neither visible nor available through any independent read path, when the target cannot be distinguished from harmful neighbours, when the application changes faster than revalidation can close the race, or when an irreversible effect has no idempotency, compensation or authoritative receipt.

It also ceases to justify its cost in the opposite regime: fixed, deterministic, low-consequence interfaces where a simple visual controller already meets the required error bound. The correct boundary is not visual versus non-visual. It is whether the system can establish sufficient evidence before and after the effect.

Some tasks remain human-only for reasons beyond interface fragility. A person may need to interpret ambiguous policy, exercise accountable judgement or obtain consent. Computer use can prepare evidence and navigate the surface without inheriting that authority. A functioning cursor is not a decision right.

Compact glossary

Frame revision

A version attached to an actionable observation so that dispatch can reject stale targets.

Grounding

The mapping from a semantic intention, such as “submit this claim”, to a concrete interface target and operation.

Actionability

Evidence that a target is unique, visible, stable, enabled and able to receive the intended event.

Interaction readback

Evidence that the intended interface element received the input and produced the expected immediate reaction.

Postcondition readback

Independent evidence that the intended domain state became true, failed or remains unknown.

Unknown outcome

A state in which an effect may have committed but available evidence cannot yet distinguish success from failure.

Treat the GUI as an untrusted adapter

A graphical interface gives an agent reach across software that was never designed for automation. That reach is valuable. It can rescue legacy workflows, operate specialist tools, interpret visual artefacts and bridge the last mile where integration would otherwise stall.

The same universality is purchased by discarding contracts. The screen does not reliably name the business object, expose all preconditions, preserve target identity during motion or prove the final state. A computer-use agent must infer those distinctions from evidence, and inference can fail without an explicit error.

The architecture decision therefore changes from “Can the model click?” to “Can the configured system prove what this click referred to and what happened afterwards?” Build the answer from versioned observation, semantic target, deterministic preflight, bounded authority, interaction readback, domain postcondition and unknown-outcome recovery. Use visual perception where it adds information. Use structured interfaces where they carry stronger semantics. Combine them when the task needs both.

The release programme changes too. Test moving layouts, hidden state, overlays, concurrent edits and timeouts after commit. Keep a stable kiosk as the negative control. Measure wrong effects and unresolved outcomes, not only completion. Widen capability one operation at a time.

Pixels can be a sensor. Coordinates can be an actuator. Neither should be mistaken for a contract. The GUI belongs at the edge of a governed agent system, treated as a fallible adapter whose claims must be checked against the world it only partially reveals.