The refund that succeeded twice

A customer asks a retail assistant to credit £25 against order O-417. The model chooses the correct function and produces a syntactically perfect request. The order exists, its status permits a credit, the amount is within the refundable balance and the request reaches the service. The service commits the credit. Before the response arrives, the connection closes.

The assistant sees a timeout. It reasons that the operation probably failed, creates a fresh request identifier and calls the same function again. The second call also passes the schema. The remaining balance still permits £25, so the service commits a second credit. The customer receives £50. Every emitted object was well formed. Every function name was correct. The failure appeared only when two valid calls were interpreted as one intended action.

A well-typed call is not yet a correct action. Shape validation can prove that required fields exist and values have allowed types. It cannot prove that the call applies to the current world, that the caller may perform it, that the server produced the intended effect or that retrying after an ambiguous response is safe. Those are different claims, supported by different evidence.

The central answer follows. Tool use is learned API interaction when the model learns a policy over one contract containing schema, preconditions, effects and errors. Given an intent, an observed world state, a tool description and an interaction history, it must choose among more than API names. It must decide whether to call, ask, refresh, repair, verify, retry, abandon or escalate. A deterministic gateway then checks what language-model judgement must never be allowed to grant: identity, authority, current entitlement and hard effect limits.

This question is narrower than trajectory assurance across an entire agent decision. Trajectory assurance asks how to evidence the whole path. Here the prior question is whether each tool exposes enough semantics for a path to be meaningfully evaluated at all.

Part oneThe call is a state transition

A function signature invites a familiar picture: choose a name, fill its parameters, receive a value. That picture is adequate for a pure calculator. It becomes misleading as soon as the operation reads changing state, writes durable state, depends on policy or can finish without returning a response.

Represent a tool contract as C = (Σ, P, Δ, E). The schema Σ defines the language of structurally valid actions. The precondition predicate P(s, a) says whether action a applies in world state s. The transition relation Δ(s, a) describes possible new states and observations. The error semantics E(o) maps an observation to what it means for the next decision. The user intent supplies a postcondition Q, such as “one £25 credit exists for this request”.

Here L(Σ) is the set of requests accepted by the schema; s′ is the resulting world state. Each term removes a different failure. Tightening Σ reduces malformed requests. Improving P reduces calls made in the wrong situation. Modelling Δ reveals side effects and uncertain outcomes. Improving E changes recovery after a conflict, timeout or refusal. No single term substitutes for the others.

Thought experiment one: hold the JSON fixed

Place the same valid credit request in two worlds. In World A, the order is delivered, has £50 remaining and has no prior credit under the request key. In World B, the order is still pending. Change nothing else. The tool name and arguments remain identical. A schema-only evaluator awards the same score to both. A stateful contract executes A and rejects B.

Applicability is a relation between an action and a world, not a property of the action string. Training examples that pair a user utterance directly with one gold function call conceal this causal variable. The model can learn textual association while never learning which observation makes the action legal or useful.

Thought experiment two: hold the timeout fixed

Now keep the observation fixed: DEADLINE_EXCEEDED. Change only the operation. The first operation reads a weather forecast. Repeating it may spend time, but it does not intentionally change the forecast. The second issues credit. The first request may have committed before the response disappeared. An automatic retry could duplicate value.

The error token alone is therefore incomplete. Correct recovery depends on the operation's intended effect, idempotency mechanism and available readback. RFC 9110 defines an idempotent method by whether repeated identical requests have the same intended server effect, and cautions against automatically retrying non-idempotent requests without additional knowledge.^[12] A stable request identifier can supply that knowledge by letting the service return the prior result for a duplicate request.^[15]

A timeout after a state-changing call is an unknown outcome, not evidence of failure. The next action should often be readback, not repetition. That decision cannot be learned from request syntax alone.

Negative control: the pure converter

Consider celsius_to_fahrenheit(value). It has no external state, durable effect, permission decision or ambiguous commit. Given a numeric input, the output follows deterministically. Here a schema, a short description and a few boundary tests can be enough. A heavy workflow contract adds ceremony without changing the answer.

This control matters because it blocks an inflated conclusion. The claim is not that every function requires orchestration. It is that contract depth should track state dependence and consequence. Generate-and-validate is a credible baseline for pure, reversible and independently checkable tools. The full mechanism earns its cost when hidden state, policy, side effects or uncertain outcomes can change the next action.

Minimal worked example: when the contract collapses cleanly

Give a model one tool: convert_temperature(value, from_unit, to_unit). Its schema permits finite numbers and the symbols C, F and K. Its result is a number plus the target unit. The operation reads no external state and requests no durable change. For a fixed implementation, the same valid arguments produce the same intended result regardless of customer history, time or concurrent activity.

The evaluation can therefore be compact. A positive case checks that 0°C becomes 32°F. A boundary case checks that −40 has the same numeric value in Celsius and Fahrenheit. A negative case supplies K below absolute zero and expects a typed invalid-argument response. A metamorphic property converts C to F and back, allowing a stated floating-point tolerance. These tests exercise meaning beyond JSON shape, yet none requires an order version, effect receipt or recovery workflow.

Now change one feature. Suppose the “converter” also stores a billable laboratory calibration record. The numeric mapping has not changed, but the operation is no longer pure. The contract must state who may create the record, whether repetition creates a second charge, how a timeout is resolved and how the new record is read back. The extra machinery follows the added effect, not the presence of an LLM. Contract depth should grow when causal exposure grows.

flowchart LR
  A["The refund that succeeded twice"] --> B["Part one The call is a state<br/>transition"]
  B --> C["Part two When state entered<br/>the benchmark"]
  C --> D["Part three Compile a contract<br/>the model and executor…"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#007D68,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Part twoWhen state entered the benchmark

Tool-use research did not begin with a single agreed construct. Toolformer trained a model to decide which APIs to call, when to call them, which arguments to supply and how to incorporate returned results.^[1] API-Bank supplied 73 runnable tools and evaluated planning, retrieval and calls across 314 annotated dialogues containing 753 calls.^[2] Gorilla focused on writing calls against large, changing API collections and used retrieval-aware training to adapt to documentation changes.^[3] ToolLLM expanded coverage to 16,464 collected REST APIs and generated multi-tool solution paths.^[4]

These projects established that tool choice and argument formation are learnable behaviours. They also made documentation a training object rather than an implementation afterthought. One study across six tasks reported that documentation-only prompts could match few-shot tool demonstrations in its evaluated settings.^[5] That result supports exposing precise descriptions. It does not show that prose documentation alone supplies current world state, authority or effect verification.

The evaluation boundary then widened. BFCL covers serial and parallel calls, abstention and stateful multi-step settings; its 2025 paper reports that single-turn calls are much stronger than memory, dynamic decisions and long-horizon behaviour.^[6] ToolSandbox adds stateful execution, implicit dependencies, on-policy conversation and milestone checks.^[8] τ-bench evaluates the final database state after conversations governed by domain policies and introduces pass^k to ask whether repeated runs all succeed, rather than whether one lucky sample succeeds.^[7]

The construct changed from “can the model write a call?” to “can the agent complete a stateful interaction consistently?” That is more than benchmark inflation. It reveals the mechanism that single-turn scoring omits. Once actions modify a shared world, later observations depend on earlier actions, and a locally plausible call can destroy the only successful trajectory.

The strongest objection: let the executor handle everything

A serious competing design says the model should only produce candidate arguments. A deterministic client library can validate them, perform retries, manage versions and return a clean success or failure. This is the right baseline. Transport retries, authentication refresh, request de-duplication and schema enforcement should be removed from language-model discretion whenever their rules are mechanical.

The baseline stops short when recovery changes the meaning of the task. A version conflict may reveal that a customer already cancelled the order. A failed precondition may require a new user decision. A partial result may satisfy one branch of a compound request while invalidating another. A tool may return two alternatives whose choice depends on preference stated earlier in the conversation. A wrapper can make these facts typed and safe; it cannot choose their semantic consequence without an intent-sensitive policy.

Part threeCompile a contract the model and executor can share

OpenAPI defines a language-agnostic description of HTTP interfaces and gives operations identifiers, parameters, request bodies and possible responses.^[9] JSON Schema can assert what a valid JSON document must look like.^[11] Arazzo adds multi-step workflow semantics including success criteria, failure actions, outputs and timeouts.^[10] These standards provide strong raw material. A model-facing contract still needs a compiled view that connects them to the decision the model must make.

A useful compiler emits two related artefacts. The model contract contains concise purpose, applicability, effect and recovery language with examples and counterexamples. The execution contract contains enforceable schemas, identity and permission requirements, version checks, idempotency rules, timeouts, observability fields and postcondition queries. Both derive from one governed source, but they serve different consumers.

Do not ask prose to enforce what code can prove, and do not ask code to infer what the user meant. The model should see that a timeout may mean “effect unknown” and that a version conflict requires fresh state. The gateway should ensure the same request key is reused, block calls without authority and refuse an obsolete version even if the model insists.

Describe effects as transitions, not marketing verbs

Names such as update, submit and process are weak descriptions because they hide what persists. An effect declaration should identify the resource changed, the fields or invariant affected, the version rule, any secondary effects and the observation that proves completion. “Issues a credit” is weaker than “creates one credit receipt keyed by request_id, increments the order's credited total and resource version, and never changes the refundable total”.

Preconditions need the same treatment. Write them as predicates whose evidence can be obtained: order.status == DELIVERED, amount <= refundable_remaining and expected_version == current_version. Record the source and freshness of each value. A condition inferred from an old conversation is not equivalent to a condition read from the order service. For HTTP resources, validators such as entity tags and conditional requests can prevent a stale write from silently overwriting a newer representation.^[12]

The compiled contract should also name uncertainty. A response can establish accepted, rejected, committed, not committed or unknown. These states should not be compressed into a Boolean success flag. A 202 response may mean work has merely started. A dropped connection may hide a completed write. A business rejection can arrive in a technically successful transport response. The recovery policy needs the semantic state, while the execution boundary needs the raw evidence that justifies it.

Finally, version the contract independently from the model. An operation can retain its name while changing required fields, side effects or retry guarantees. Store the contract version with each proposal and execution receipt. Retrieval-aware systems such as Gorilla show why documentation change matters for call formation.^[3] The same principle is more important for effects: a model that learned yesterday's retry rule can be structurally correct and operationally dangerous today.

The contract compiler is a release control

In practice, the source material is scattered. Request and response shapes live in OpenAPI or protobuf. Preconditions sit in service code, product rules or runbooks. Retry behaviour may exist only in a client library. Permissions belong to an identity and policy system. Outcome evidence may require a separate query. A contract compiler does not invent these facts. It joins them into a versioned intermediate representation and fails the build when a consequential gap remains.

For each operation, compilation should answer five tests. Can every precondition be resolved from an authoritative source? Does every write name its intended and possible secondary effects? Does every typed error map to a bounded next-action class? Can an ambiguous commit be resolved by request key or readback? Can the executor check authority without trusting text produced by the model? A missing answer is not a documentation polish item. It is an unmodelled branch in the action space.

The compiler can then generate different views. The model receives a compact description, relevant examples, counterexamples and typed observations. The gateway receives validators, policy hooks, concurrency tokens, retry rules and effect limits. The evaluator receives initial-state builders, fault injectors and postcondition queries. Documentation receives human-readable purpose and failure notes. Because the views share one operation identifier and contract version, a changed precondition can trigger prompt tests, gateway tests and benchmark regeneration together.

This is a practitioner synthesis, not a claim that current interface standards provide such a compiler end to end. OpenAPI and Arazzo supply important syntax; service owners must still expose business state, authority and outcome observers. The release rule is simple: do not publish a consequential tool to an agent catalogue when its interaction contract cannot be compiled without unresolved effect or recovery semantics.

Errors are observations with action semantics

Free-text exceptions force the model to guess. RFC 9457 standardises machine-readable problem details for HTTP APIs.^[13] Google's API guidance similarly recommends canonical codes and actionable details, then distinguishes errors that may be retried automatically from those where repetition would cause unintended state changes.^[14][16] The precise taxonomy will differ by service, but the next-action meaning should not.

The last row marks the system boundary. A model may understand what permission means, but a model proposal never becomes authority by describing itself confidently. Identity, actor, subject, purpose and permission must remain mechanically bound at execution time.

Worked scenario: a refund as a governed transition

Use a synthetic retail platform with four tools: get_order, quote_credit, issue_credit and get_credit_receipt. The customer asks for £25. The order record is current world state. The tool documentation is context. The model's temporary deliberation is reasoning state. The sequence of calls and responses is an evidence ledger. Calling all four “memory” would erase the distinctions needed for recovery.

The interaction starts by binding intent to order O-417. A read returns status DELIVERED, refundable balance £50 and version 7. The quote tool calculates the permissible amount without changing state. After any required confirmation, the model proposes issue_credit with amount £25, expected version 7 and a stable idempotency key derived from the approved intent.

The gateway checks schema, principal, purpose, amount limit, current version and request-key format. The service executes. Success is not the HTTP status alone. The system reads the receipt by idempotency key and verifies that exactly one £25 effect exists. It records both the accepted command and the observed state. Only then does it tell the user that the credit was issued.

Place the contract inside the full control boundary

The tool contract touches ten unlike objects. Intent states the desired change. Identity binds the requester and represented subject. World state describes current reality. Context supplies relevant documentation and policy. Reasoning proposes the next move. Authority decides whether that move may occur. Action is the typed command. Evidence records what was accepted and observed. Outcome is the verified business state. Release and recovery determine what can continue after failure.

Collapsing these objects creates predictable errors. An API connection is not authority. A retrieved policy paragraph is not current world state. A model explanation is not an effect receipt. A successful transport response is not a verified outcome. The interface becomes governable only when each claim points to the object that can support it.

Part fourTrain the recovery policy

If the desired capability is interaction, the training and evaluation unit must be an episode. Each episode needs an initial world state, user intent, contract version, permitted observations, fault schedule and target postcondition. A gold call is useful inside the episode, but it cannot stand in for the episode.

A balanced suite varies one causal feature at a time. Hold the user request fixed and change order status. Hold state fixed and remove one required argument. Hold both fixed and inject a version conflict. Commit the effect and drop the response. Return a transient error before commit. Change the documentation version. Withhold a required fact so the correct action is to ask. Include irrelevant but similarly named tools. Revoke confirmation after the quote. Each case tests a distinct part of the policy.

The scorecard should preserve those distinctions. Report structural validity, tool-selection precision, precondition violations, clarification quality, state-transition success, duplicate-effect rate, recovery correctness, abstention, effect verification and repeated-run consistency. A single end score can still be useful for release, but it should be decomposable into failure mechanisms.

Each L measures one error family; each weight λ reflects consequence. Raising λeffect penalises wrong or duplicated state changes more heavily. Raising λstop rewards abstention when information is missing. The equation is a design template, not a published universal objective. Its value is that it prevents high schema accuracy from numerically hiding effect failure.

What the model should actually learn

The model does not need to memorise TCP behaviour or reproduce a client library. It needs decision-sufficient abstractions. Examples include: “this operation changes durable state”; “this precondition must be observed, not assumed”; “this conflict invalidates the old plan”; “this timeout leaves the outcome unknown”; “this readback can prove the effect”; and “this refusal cannot be repaired by choosing a different tool”.

Training traces should pair those abstractions with action choices, not reward verbose private narration. A concise typed state can carry effect_class=WRITE, retry_class=READBACK_FIRST, version=7 and confirmation=BOUND. The model's job is to map the situation to the next semantic move. The gateway's job is to enforce the move and return a typed observation.

Error recovery is part of tool competence because errors change the action distribution. Removing recovery examples trains a policy for a world in which services never conflict, users never omit information and networks never fail after commit. That is not a simplified production environment. It is a different causal environment.

A training episode is a small causal experiment

An episode manifest should name the intent, initial public state, hidden evaluator state, contract version, available tools, observation budget, injected fault and target postcondition. It should also state which actions are forbidden even if they accidentally reach the target state. That last field prevents an evaluator from rewarding a credit issued without confirmation merely because the final database matches.

Construct matched pairs. In one pair, the only difference is status=DELIVERED versus status=PENDING. In another, the service fails before commit versus after commit. In a third, a required fact is present versus absent. Randomising names, amounts and irrelevant descriptions reduces memorisation, while preserving the causal relation being tested. Hold everything else fixed so a score difference can be attributed to the changed feature rather than to prompt difficulty.

The evaluator should inspect observable decisions, not demand a private chain of thought. It can record selected tool, arguments, state reads, clarification questions, gateway decisions, typed errors, retries, readbacks and final state. Milestone checks catch a trajectory that reaches the right endpoint through a prohibited intermediate action. Final-state checks catch a fluent trajectory whose effect never occurred. Repeated trials estimate consistency; the pass^k framing from τ-bench is useful because consequential systems need all repeated runs to be correct, not one successful sample among many.^[7]

Split evaluation by novelty. One partition varies values inside familiar schemas. Another introduces unseen but well-documented tools. A third changes a familiar tool's contract version. A fourth composes tools whose individual calls were seen but whose state dependency was not. Improvement limited to the first partition suggests memorised call patterns. Transfer to the others is stronger evidence that the model learned contract structure.

Executable recovery benchmark

The included Python 3.10+ artefact creates a synthetic credit API with deterministic fault injection. It tests a governed recovery policy against a happy path, a concurrent version change, unavailability before commit, timeout after commit, two failed preconditions and a blind-retry negative control. No model vendor, network or private data is required.

Its critical property is matched intent. In the timeout case, the governed client reads back the stable idempotency key and records one effect. The naïve client treats the same timeout as failure, changes the key and creates a second £25 credit. The negative case therefore reveals logic that a prose example could conceal.

Decision instrument: admit the interaction, not the endpoint

A tool registry should reject operations whose contract cannot support the decisions their consequences require. The instrument below is intentionally stricter for writes than reads. It is a starting gate for design and evaluation, not evidence that an admitted tool is universally safe.

Keep operating evidence without copying the whole conversation

A recoverable interaction needs a compact trace. Record the contract version, proposal digest, bound actor and purpose, state snapshot identifiers, gateway decision, request key, attempt number, typed observation, readback query and effect receipt. These fields support replay and incident analysis without requiring every private prompt or retrieved document to be duplicated into logs.

Separate a decision receipt from an effect receipt. The first says why a command was admitted: which intent, evidence, policy and version supported it. The second says what the executing system observed after the command. Their identifiers should join across traces, while sensitive payloads remain in their governed stores. OpenTelemetry's stable HTTP span conventions provide a common transport vocabulary, including retry and error attributes, but application fields are still required to express business preconditions and verified outcomes.^[17]

Evidence retention also needs limits. High-cardinality customer data should not become metric labels. Error messages may contain private values and should be redacted or referenced. A trace that exposes everything can make recovery easier while creating a new privacy failure. Operating evidence should preserve the causal joins needed for verification, not become an unbounded copy of context.

Where the mechanism stops helping

A complete declared contract cannot repair a false world model. If the authoritative state is inaccessible, stale or internally inconsistent, the precondition check can certify the wrong world. If the service commits effects it does not document, returns misleading errors, ignores request identifiers or offers no independent readback, the recovery policy lacks the observations needed to distinguish success, failure and partial effect.

The mechanism also does not solve authority by itself. An impeccably trained model can still propose a prohibited action. Nor does it guarantee optimal planning across many tools; it only makes each interaction's semantics available to that planner. Some environments remain too irreversible or opaque for model-directed execution. There, the correct architecture may be simulation, human approval, dual control or no automated write path.

These limits provide falsifiers. The contract thesis weakens in a domain where schema-only agents match stateful agents on held-out changes, ambiguous commits and missing-information cases, with the same verified outcomes and duplicate-effect rate. Conversely, a gain that appears only in format accuracy would not support the thesis.

A decisive evaluation should therefore hold the schema, model and user intent constant while changing only the hidden state or response condition. Score the final authorised state, the number of unintended effects, the recovery decision and the evidence retained for review. Report rates over all eligible episodes, not just calls the agent attempted. This design prevents abstention from masquerading as safety and prevents successful formatting from masking a duplicated or missing effect. It also reveals whether the contract changed behaviour under the very interventions its semantics were meant to govern.

stateDiagram-v2
  [*] --> Claim
  state "The interface is a four-layer<br/>contract" as Claim
  state "Thought experiment one: hold<br/>the JSON fixed" as Test
  state "One valid call, two different<br/>worlds" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Compact glossary

Applicability

Whether a structurally valid action is suitable for the current world state and intent.

Effect receipt

Evidence from the executing system that a particular command produced a recorded effect.

Idempotency key

A stable request identifier that lets a service recognise and de-duplicate the same intended request.

Readback

An independent query of authoritative state or receipt after execution, especially after an ambiguous response.

Typed action contract

A versioned description of request shape, applicability, transition, errors and verification semantics.

Unknown outcome

A state in which the caller lacks evidence about whether an attempted effect committed.

Admit tools as state transitions

The practical decision changes at the registry boundary. Do not approve a tool because the model can name it, fill its JSON and pass a happy-path demonstration. Approve an interaction only when its current preconditions are observable, its effects are stated, its failures have next-action semantics and its consequential outcomes can be read back.

Then train on episodes that make those distinctions causal. Place the model where intent determines whether to call, ask, refresh, verify or stop. Place deterministic control where identity, permission, version, request de-duplication and hard limits can be enforced. Score the resulting system on verified state transitions and repeated consistency, while retaining format accuracy as one component metric.

The durable test of learned tool use is not whether the agent can produce a call. It is whether the system can explain, constrain and recover the state change that the call attempts. That standard turns a catalogue of functions into an operationally meaningful action space.