The call that passed every type check
A customer types: “Tell me what it would cost to send £600 to supplier 7 on Friday. Do not schedule anything.” A model receives two tools. quote_transfer returns an estimate. schedule_transfer creates a future payment. Both accept the same account, payee, amount and date. The second also requires a minimum remaining balance and an idempotency key.
The model returns a pristine call to schedule_transfer. Every required field is present. The account and payee identifiers are strings. The amount has two decimal places. The date is ISO formatted. The key is unique. A strict schema validator reports success. Yet the call is wrong in the most important way: the user asked for information, while the model proposed an effect.
One response is to make the prompt louder: “Never schedule when the user asks for a quote.” That may reduce the error. It does not turn a probabilistic instruction into an execution boundary. Another response is to add a required operation enum. That helps downstream parsing, though the model can still choose the wrong member. The schema can rule out "scheduel"; it cannot infer that "schedule" violates the request.
The decisive distinction is between constraining a representation and validating a proposed transition in the world. Structured output addresses the first problem. Function calling packages a proposed transition. The surrounding system must decide whether that proposal means what the user meant, whether this tool is the least powerful fit, whether the actor may invoke it now, and whether execution produced the intended state exactly once.
This distinction changes system architecture. Parsing should become deterministic as early as possible, but trust should not move with it. A model that reliably emits JSON has become easier to integrate and test. It has not acquired authority.
The schema funnel narrows form before the system earns an effect
A model may participate in the first four stages. Deterministic controls own permission, execution and outcome evidence.
The funnel has more than one gate
“Valid output” sounds singular because a programme often exposes one Boolean result. In a model-mediated system, it hides several propositions. A string may be valid JSON. The parsed value may satisfy a JSON Schema. The fields may accurately preserve the request. The chosen function may be appropriate. The current world may satisfy its preconditions. The actor may possess authority. The effect may occur once and be observable. These claims are related, but none is a synonym for the next.
The JSON specification defines a data interchange syntax. JSON Schema’s validation vocabulary can impose structural constraints such as required properties, types, ranges, patterns and conditional branches. The specification itself draws a useful boundary around format: recognising the lexical form of an email address or URI does not establish that the address exists, belongs to the intended person or is safe to use. Structural validation is powerful precisely because it makes a bounded claim.
Treat each stronger proposition as a separate contract with its own evidence. This prevents a parser’s success from being reported as task success. It also makes failures diagnosable. A missing field belongs to the shape contract. A changed amount belongs to semantic fidelity. A quote request routed to a scheduling tool belongs to selection. A £600 call made under a £500 delegation belongs to authority. A timeout after an accepted command belongs to outcome reconciliation.
Every inner region adds a claim that syntax alone cannot establish
Minimal worked example: an extraction that lies politely
Suppose a support message says, “The parcel arrived after eight days, but the mug was intact.” The application asks for:
{
"type": "object",
"properties": {
"sentiment": {"enum": ["positive", "negative", "neutral"]},
"delay_days": {"type": "integer", "minimum": 0}
},
"required": ["sentiment", "delay_days"],
"additionalProperties": false
}The output {"sentiment":"positive","delay_days":8} is schema-valid. Whether “positive” is faithful depends on the application’s construct. Overall satisfaction, product condition and delivery experience are different labels. The schema does not choose among them. A better contract might separate delivery_sentiment from product_sentiment, define the labelling policy and permit mixed. That is domain modelling, not merely formatting.
This example also reveals why structured output often appears to improve reasoning. A better schema forces the designer to name distinctions that a free-text prompt left vague. The gain may come from a clearer task definition, a narrower decoding space, or both. An evaluation should vary these causes separately: keep the prompt fixed while changing decoder constraint, then keep decoding fixed while changing the schema’s ontology.
The schema is also a theory of the task
Every field encodes a claim about what downstream software needs to know. A required number says the quantity always exists. An enum says the listed categories are jointly sufficient for this decision. A nested object says certain facts belong together. additionalProperties: false says unanticipated distinctions should be rejected rather than preserved. These are modelling choices before they are validation rules.
That matters because a structurally elegant schema can erase the very condition that should block action. Imagine a travel tool with destination, date and cabin, while the user says, “Find options, but do not book until my partner agrees.” If the contract has no field for requested phase or confirmation state, a model can preserve every declared field and still drop the governing condition. Adding a free-text notes property preserves words but does not give the executor a machine-enforceable precondition.
A stronger design separates facts, interpretations and control. Facts include destination and dates. Interpretation includes whether the request is exploratory or effect-seeking. Control includes the confirmation that must exist before booking. The model may infer interpretation with uncertainty; deterministic code should bind control to an authenticated event. A field belongs in the schema only when its meaning, source and downstream authority are explicit.
For each property, write three tests: an admissible value, a lexically valid but semantically wrong value, and a case where the truthful answer is missing or uncertain. This simple discipline exposes whether the contract merely describes a convenient happy path. It also prevents “strict mode” from becoming a substitute for domain analysis.
Thought experiment 1: the compulsory enum
Vary one causal feature: add one truthful escape value while keeping the evidence, model and other fields constant.
A document contains a faint signature that might be “A. Sen” or “A. Senn”. Schema A requires signer ∈ {"A. Sen","A. Senn"}. Schema B adds "uncertain". A constrained decoder using Schema A must finish inside one named branch. It can be perfectly compliant while fabricating certainty. Schema B does not make the model calibrated, but it makes a truthful output representable.
The changed causal feature is not model intelligence. It is the contract’s answer space. When uncertainty is excluded from the schema, downstream systems cannot distinguish confident recognition from forced completion. A schema that cannot represent abstention converts epistemic uncertainty into syntactic confidence.
One extra enum member changes whether uncertainty can remain visible
Closed answer space
enum: ["A. Sen", "A. Senn"]
Evidence: ambiguous
Required result: one named person
Failure branch: false specificity
Truthful escape present
enum: ["A. Sen", "A. Senn", "uncertain"]
Evidence: ambiguous
Permitted result: uncertain
Next action: request a clearer source
Why the distinctions matter operationally
Collapsing the contracts produces the wrong remediation. Teams see invalid JSON and add retries. Later they see a wrong tool and add more retries. Eventually the system sends the same effect twice because a network timeout was mistaken for a generation failure. A single “success rate” hides whether the model could not express the shape, misunderstood the task, lacked current data, exceeded its delegation or created an effect that the caller failed to observe.
A useful evaluation therefore records a stage-specific denominator. Parse conformance is measured over model responses. Schema conformance is measured over parsed responses. Selection precision and abstention are measured over requests with a known admissible tool set. Argument fidelity is measured field by field against authoritative evidence. Execution success is measured over authorised calls. Verified outcome is measured over accepted commands, including timeouts. The funnel is an evidence chain, not a decorative architecture.
flowchart LR A["The call that passed every<br/>type check"] --> B["The funnel has more than one gate"] B --> C["Constraint changes the<br/>decoding path"] C --> D["Function calling proposes an<br/>effect"] D -. "re-test" .-> A classDef first fill:#0a1424,stroke:#007D68,stroke-width:2px,color:#e6f3ff classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff class A first class B,C,D step
Constraint changes the decoding path
Free-form generation repeatedly chooses a next token from a model distribution. Prompt instructions influence that distribution, but they do not remove invalid continuations. A post-generation parser waits until the end and rejects malformed output. A repair loop asks for another attempt after rejection. Constrained decoding acts earlier: after each prefix, a grammar or schema engine computes which next tokens can still lead to a valid completion and masks the rest.
Let h be the generated prefix, zt the model logit for token t, and AC(h) the tokens admitted by constraint C after that prefix. The constrained next-token distribution is:
When a token is outside the allowed set, its constrained probability becomes zero. The remaining probabilities are renormalised. Tightening the constraint can make an invalid comma, field name or enum value impossible. It does not, by itself, change which allowed value best matches the evidence. If "schedule" and "quote" are both legal, a semantic error survives the mask. If only one is legal, the mask can force that value even when the request does not support it.
The mask removes illegal continuations, then renormalises what remains
The schema eliminates four candidates. The model distribution still ranks the two legal operations.
Published evidence: constraints can materially improve formal validity
PICARD incrementally rejected inadmissible tokens while generating SQL and improved results on text-to-SQL tasks. Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning extended the idea across structured NLP tasks using formal and input-dependent grammars. LMQL combined prompt text, control flow and output constraints in a language-model programming abstraction. These results support the mechanism: move some validity work into the decoding procedure rather than hope prose instructions will be obeyed.
Current provider interfaces expose related capabilities under different names. OpenAI distinguishes a structured response from a function call and documents strict schema modes. Anthropic documents strict tool schemas while leaving custom tool execution to the application. Google similarly distinguishes structured response data from function calling and explicitly advises application-side semantic validation. The APIs differ, and supported schema subsets change. Their common architectural signal is more durable: the model emits data or a call description; application code remains responsible for what happens next.
Conformance itself has multiple dimensions. JSONSchemaBench evaluates efficiency, actual coverage of schema features and output quality across constrained-decoding frameworks. Its existence is a warning against treating “supports JSON Schema” as a Boolean product feature. A provider may support a useful subset, compile schemas with restrictions, or reject patterns that a general-purpose validator accepts. Release tests should use the exact schemas, models and API modes that the application will run.
Three mechanisms, three failure signatures
Prompt-only formatting, post-hoc repair and constrained decoding are often grouped together because each can raise parse success. They intervene at different times. Prompting changes the model distribution but leaves every token technically available. Post-hoc validation observes a complete candidate and can reject it. Constrained decoding changes the available next-token set before a candidate exists.
| Mechanism | What it changes | Characteristic failure | Best use |
|---|---|---|---|
| Prompt instruction | Relative probability of formats and values | Occasional malformed or extraneous prose | Low-consequence, human-reviewed output |
| Validate and repair | Acceptance after a full candidate | Retry cost, drift in unrelated fields, repeated failure | Providers without decoder constraints; rare correctable errors |
| Constrained decoding | Admissible token paths during generation | Unsupported schema, forced legal value, compile overhead | Machine-consumed structures with a stable supported contract |
| Application gate | Whether a valid proposal may influence state | Stale evidence, policy denial, unknown effect | Every consequential call |
These mechanisms compose rather than compete. A clear prompt explains the task. A schema defines the representable answer. Constrained decoding enforces the supported structure. A validator checks the final instance. Application gates decide whether the instance is meaningful and actionable. The design error is assigning a later claim to an earlier mechanism.
Choose the earliest mechanism that can guarantee the property, then test the properties it cannot guarantee. Do not spend model tokens repairing braces when the decoder can prevent them. Do not expect the decoder to verify a live account balance. This allocation improves both reliability and diagnostic clarity.
The negative control: when syntax really is the whole problem
Negative control
A deterministic pricing service has already computed subtotal, tax and total as typed decimal values. A language model is asked only to place those values into a fixed presentation object, with no choice of tool, entity or calculation. The object is read-only and a human sees it before use. Here, schema conformance may be almost the entire model-specific requirement. An independent semantic recomputation adds little because authoritative typed values already exist upstream.
This case weakens any universal claim that every structured output needs a large validation estate. The extra gates arise when the model contributes uncertain meaning, selection, state-dependent values or an effect proposal. When those contributions are absent, keep the architecture small. Indeed, a stronger conclusion follows: if a deterministic mapper can copy the authoritative fields, remove the model from that step.
Explicit failure boundary
Constrained decoding ceases to help when the desired truth cannot be represented by the constraint or when the decoder cannot faithfully implement the constraint. The first failure is conceptual: the schema omits abstention, provenance, units, conditional intent or a relevant union type. The second is mechanical: unsupported schema keywords, excessive grammar complexity, tokeniser interactions, ambiguous lexical constraints or a provider fallback weaken the promised guarantee.
Even a complete implementation cannot validate facts outside the instance. A date can have valid syntax and fall on a bank holiday. A customer identifier can match a pattern and refer to the wrong customer. A URI can be well formed and lead to an untrusted host. A tool argument can be numerically in range and exceed the actor’s live limit after another transaction. Those are world-state and policy questions. They require authoritative lookups and rules at the time of decision.
Function calling proposes an effect
Function calling gives the model a catalogue of named operations, descriptions and parameter schemas. The model may choose a function and populate arguments. That interface is valuable because it makes an intended operation inspectable before execution. It is dangerous when “the model called the tool” is allowed to blur two events: a model emitted a call description, then application code chose to execute it.
Official documentation from OpenAI, Anthropic and Google describes an application loop in which the model requests or proposes a tool use, the application executes custom code, and the result returns to the model. This separation is more than SDK choreography. It is the point where deterministic authority can remain outside probabilistic generation.
Treat the emitted call as a typed action proposal, never as ambient permission. A tool description tells the model what an operation means. A parameter schema tells it what arguments look like. Neither proves that the requesting user owns the account, that the model is delegated to act, that a required confirmation is fresh, or that the operation is permissible under current policy.
Tool selection and argument validity are independent axes
Thought experiment 2: the same arguments, one changed verb
Vary one causal feature: change “quote” to “schedule” in the user request while keeping entities, amount, date, tool schemas and model proposal constant.
Request A says, “Quote £600 to supplier 7 for Friday.” Request B says, “Schedule £600 to supplier 7 for Friday.” The model proposes schedule_transfer with exactly the same valid arguments in both cases. Under B, the operation may be semantically appropriate. Under A, it is not. No parameter validator can distinguish the cases because the difference lives in the relation between the request and the selected capability.
A tool-selection evaluator must therefore include negative examples in which a more powerful tool can technically satisfy the nouns in the request. It should measure abstention and least-powerful-fit, not only whether the gold function name appears. The Berkeley Function Calling Leaderboard reflects this wider evaluation problem by covering abstention, stateful settings, and serial and parallel calls rather than reducing tool use to argument syntax.
The catalogue is a capability surface
Tool descriptions are usually written as model guidance, yet the catalogue also defines which capabilities the model can propose. Giving one request access to search_orders, refund_order and delete_customer is not equivalent to giving it only search_orders. A policy check can still deny dangerous calls, but unnecessary exposure increases selection ambiguity, test burden and the chance that untrusted content steers the proposal towards a powerful operation.
Compile the catalogue per request from identity, purpose, workflow phase and current state. Descriptions should distinguish observation from effect, list preconditions and state what the tool does not do. Similar operations need contrastive wording: “returns a quote and creates no payment” beside “creates a scheduled payment after approval”. Names should carry verbs that expose consequence. Hidden aliases and overloaded “manage” functions make evaluation harder.
Catalogue narrowing is not authorisation. A tool may be visible because the model needs to discuss or prepare it while execution remains prohibited. Conversely, hiding a tool does not secure an endpoint available elsewhere. The executor must authenticate and authorise every call independently. Tool availability limits proposal space; capability enforcement limits effect space. Keeping both makes the boundary robust when prompts, descriptions or model behaviour change.
Production-shaped worked example: a conditional future payment
Consider a synthetic servicing request: “Schedule £600 to supplier 7 for Friday, but only if the household account will retain at least £1,000. I confirm this payment.” A useful proposal might contain the account, payee, amount, payment date, minimum-balance condition and idempotency key. The proposal is still only one object inside a larger decision.
Intent. The system compiles the utterance into a requested operation and explicit conditions. It records whether “confirm” refers to this exact amount, payee and date. Identity. It binds the authenticated actor and subject account without asking the model to copy identity from prose. World state. It reads current balance, pending holds, payee status and calendar rules from authoritative systems. Context. It presents only the evidence and tools relevant to this case.
Reasoning. The model may propose the tool and arguments. Authority. Deterministic policy checks the actor’s rights, transaction limit, confirmation freshness and whether this channel may schedule payments. Action. An executor sends a typed request with a stable idempotency key. Evidence. A decision receipt records the request, validated proposal, policy result and evidence versions. An effect receipt records the accepted command and provider identifier.
Outcome. The system reads the schedule back from the authoritative ledger and checks payee, amount, date and status. Release and recovery. If transport times out after the provider accepted the payment, the local result is an unknown outcome, not a failed payment. The system queries by idempotency key before any retry. If the effect exists, it returns the existing receipt. If the state remains unknowable, it holds the case for reconciliation.
A production call crosses six evidence boundaries
Evidence: request record
Evidence: model proposal
Evidence: decision receipt
Evidence: acceptance receipt
Evidence: effect receipt
Evidence: resolved outcome
Practitioner synthesis: the deterministic authority kernel
The model should be free to propose within a typed catalogue and unable to enlarge the catalogue, identity, delegation, limits or release rule. This is the same design boundary developed in The Certainty Gradient: probabilistic components handle interpretation where useful, while deterministic controls own consequential transitions. The Three Principals adds an identity discipline: preserve who requested, who proposed and who executed rather than letting “the agent” collapse them.
Benchmarks should follow the effect, not stop at the call
Abstract-syntax matching is suitable for a component test: did the model choose the expected function and emit comparable arguments? It does not show that a long-running interaction reached the right state. τ-bench evaluates tool-agent-user interaction using domain tasks and final database state, illustrating the value of moving beyond a single call. Production evaluation should go further when needed: inject stale state, revoked authority, tool errors, duplicate messages, contradictory user updates and timeouts around the commit boundary.
The output of such testing is not one score. It is a failure profile: false tool selection, unsafe non-abstention, field-level semantic drift, policy denials correctly honoured, duplicate-effect rate, unknown-outcome duration, readback coverage and verified-outcome rate. A system can have excellent schema conformance and unacceptable selection precision. Reporting both preserves the causal story.
Repair belongs to the failed layer
A generic retry treats every failure as another sampling opportunity. That is reasonable for a transient malformed brace. It is reckless for a denied capability or an ambiguous outcome. The repair policy should be a function of the failed contract because each stage has different invariants.
At the parse or schema stage, the system can return compact machine-readable errors such as missing:payment_date or wrong_type:amount_gbp. It should preserve the selected operation and authoritative fields rather than invite a wholesale rewrite. At the semantic stage, it can point to a mismatch against evidence: “proposal amount £660; request amount £600”. The source value, not the model’s previous prose, becomes the repair anchor.
At tool selection, the system should reclassify or ask a clarifying question. It must not simply transplant arguments into a stronger function. At authority, “repair” is the wrong verb. The system denies, requests a fresh confirmation or escalates to a separately authorised principal. At execution, a network error does not reveal whether the effect happened. Blindly regenerating and resending a call can create duplicates. The repair action is readback and reconciliation.
A safe repair reduces uncertainty without silently changing intent, authority or effect count. That rule is stricter than “eventually return valid JSON”. It gives every retry a bounded purpose and a stop condition.
Give repair a budget and a receipt
A repair loop needs a maximum attempt count, a stable case identifier and a record of what changed. Without them, a system can oscillate between valid alternatives or gradually rewrite a request. A useful repair receipt includes the failed stage, validator code, authoritative source used, fields permitted to change, before and after values, model and schema versions, and the stop reason.
The budget should depend on failure type. One deterministic serialisation repair may be enough for malformed JSON. A field mismatch can permit one evidence-bound correction. Repeated tool-selection disagreement suggests ambiguity or a catalogue problem and should trigger clarification. Authority denials receive zero model retries. Unknown outcomes receive zero execution retries until readback establishes whether the prior command committed.
Repairs also need regression tests. Every discovered malformed prefix becomes a decoder or schema case. Every wrong-tool incident becomes a contrastive selection case. Every semantic drift becomes a field mutation. Every duplicate or unresolved timeout becomes a failure-injection scenario around the commit boundary. An incident is not closed when a second call succeeds; it is closed when the failed contract has a reproducible test.
The correct recovery action depends on the failed contract
Executable artefact: constrained-call harness
The accompanying provider-neutral Python harness makes the distinction executable. It uses synthetic payment data and only the standard library. A raw model response is parsed into a proposal, checked against a small tool schema, compared with the requested operation, reconciled with authoritative request values and world state, checked against delegated authority, executed through an idempotency store and verified by readback.
The positive case asks for a quote and receives a quote with no effect. The first negative case is more important than malformed JSON: it supplies a perfectly shaped scheduling call for a quote request and is rejected at tool selection. Another case changes £600 to £660 while remaining schema-valid and is repaired at semantic validation. A limit violation is denied at authority. Finally, the store commits a payment and then simulates a transport timeout. Readback finds the committed transfer, and a repeated call with the same key returns the existing effect rather than creating another. Reusing that key for a different amount is rejected as an idempotency collision.
The harness’s central property test is not “all outputs parse”. It is “a timeout plus retry produces one committed transfer”. That property connects model integration to a business invariant.
positive_quote ACCEPT stage=outcome reasons=none
wrong_tool_valid_shape REJECT stage=tool_selection reasons=requested:quote, proposed:schedule_transfer
semantic_amount_drift REPAIR stage=semantics reasons=amount_gbp:proposal=660.00:request=600.00
timeout_after_commit ACCEPT stage=readback reasons=transport_timed_out_after_commit | transfer=tr_0001
duplicate_same_key ACCEPT stage=postcondition reasons=none | transfer=tr_0001
same_key_different_effect REJECT stage=idempotency reasons=idempotency key reused for a different effect
valid_but_unauthorised DENY stage=authority reasons=amount_exceeds_limit:500.00
Assertions passed: one committed transfer, zero duplicate effects.Expected deterministic output. Run with python constrained_call_harness.py. Python 3.11 or later is recommended; no third-party packages or credentials are required.
Open the complete runnable Python harness
The code is duplicated in the downloadable companion file so it can be tested independently. The synthetic date, identifiers and balances have no connection to a real account.
#!/usr/bin/env python3
"""Provider-neutral constrained-call harness for Paper 74.
The harness deliberately separates six questions:
1. Can the proposal be parsed?
2. Does it match the selected tool's shape contract?
3. Is the selected tool appropriate for the user's requested operation?
4. Do the arguments preserve the request's meaning and current world state?
5. Is the actor authorised to cause the effect?
6. Did execution produce the intended state exactly once?
It uses only Python's standard library and synthetic data.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from datetime import date
from decimal import Decimal, InvalidOperation
from enum import Enum
import json
from typing import Any, Callable
class RequestedOperation(str, Enum):
QUOTE = "quote"
SCHEDULE = "schedule"
@dataclass(frozen=True)
class UserRequest:
operation: RequestedOperation
account_id: str
payee_id: str
amount_gbp: Decimal
payment_date: str
minimum_remaining_balance_gbp: Decimal
explicit_confirmation: bool
@dataclass(frozen=True)
class AuthorityContext:
actor_id: str
allowed_tools: frozenset[str]
max_single_payment_gbp: Decimal
current_balance_gbp: Decimal
@dataclass(frozen=True)
class ToolSpec:
name: str
required: frozenset[str]
field_types: dict[str, type]
effectful: bool
@dataclass(frozen=True)
class Proposal:
call_id: str
tool: str
arguments: dict[str, Any]
@dataclass(frozen=True)
class Decision:
status: str
stage: str
reasons: tuple[str, ...]
next_step: str
receipt: dict[str, Any] | None = None
@dataclass
class SyntheticPaymentStore:
"""Tiny effect store with idempotency and authoritative readback."""
scheduled: dict[str, dict[str, Any]] = field(default_factory=dict)
by_idempotency_key: dict[str, str] = field(default_factory=dict)
sequence: int = 0
def quote(self, args: dict[str, Any]) -> dict[str, Any]:
# Synthetic tariff: £0.50 plus 0.2% of the amount.
amount = money(args["amount_gbp"])
fee = (Decimal("0.50") + amount * Decimal("0.002")).quantize(Decimal("0.01"))
return {
"kind": "decision_receipt",
"tool": "quote_transfer",
"amount_gbp": str(amount),
"fee_gbp": str(fee),
"effect_created": False,
}
def schedule(
self,
args: dict[str, Any],
*,
fail_after_commit: bool = False,
) -> dict[str, Any]:
key = args["idempotency_key"]
state = {
"account_id": args["account_id"],
"payee_id": args["payee_id"],
"amount_gbp": args["amount_gbp"],
"payment_date": args["payment_date"],
"status": "scheduled",
}
if key in self.by_idempotency_key:
transfer_id = self.by_idempotency_key[key]
existing = self.scheduled[transfer_id]
if existing != state:
raise ValueError("idempotency key reused for a different effect")
return {
"kind": "effect_receipt",
"status": "replayed",
"transfer_id": transfer_id,
"idempotency_key": key,
"state": existing.copy(),
}
self.sequence += 1
transfer_id = f"tr_{self.sequence:04d}"
# Commit occurs before the simulated network failure.
self.scheduled[transfer_id] = state
self.by_idempotency_key[key] = transfer_id
if fail_after_commit:
raise TimeoutError("simulated timeout after the commit boundary")
return {
"kind": "effect_receipt",
"status": "created",
"transfer_id": transfer_id,
"idempotency_key": key,
"state": state.copy(),
}
def readback(self, idempotency_key: str) -> dict[str, Any] | None:
transfer_id = self.by_idempotency_key.get(idempotency_key)
if transfer_id is None:
return None
return {
"kind": "effect_receipt",
"status": "verified_by_readback",
"transfer_id": transfer_id,
"idempotency_key": idempotency_key,
"state": self.scheduled[transfer_id].copy(),
}
TOOLS = {
"quote_transfer": ToolSpec(
name="quote_transfer",
required=frozenset({
"account_id", "payee_id", "amount_gbp", "payment_date"
}),
field_types={
"account_id": str,
"payee_id": str,
"amount_gbp": str,
"payment_date": str,
},
effectful=False,
),
"schedule_transfer": ToolSpec(
name="schedule_transfer",
required=frozenset({
"account_id", "payee_id", "amount_gbp", "payment_date",
"minimum_remaining_balance_gbp", "idempotency_key"
}),
field_types={
"account_id": str,
"payee_id": str,
"amount_gbp": str,
"payment_date": str,
"minimum_remaining_balance_gbp": str,
"idempotency_key": str,
},
effectful=True,
),
}
EXPECTED_TOOL = {
RequestedOperation.QUOTE: "quote_transfer",
RequestedOperation.SCHEDULE: "schedule_transfer",
}
def money(value: Any) -> Decimal:
try:
amount = Decimal(str(value))
except (InvalidOperation, ValueError) as exc:
raise ValueError(f"not a decimal amount: {value!r}") from exc
if amount.as_tuple().exponent < -2:
raise ValueError("money must have at most two decimal places")
return amount.quantize(Decimal("0.01"))
def parse_proposal(raw: str) -> Proposal | Decision:
try:
payload = json.loads(raw)
except json.JSONDecodeError as exc:
return Decision(
"repair", "parse", (f"invalid_json_at_{exc.pos}",),
"repair the serialisation; do not change the intended operation",
)
if not isinstance(payload, dict):
return Decision(
"repair", "parse", ("top_level_must_be_object",),
"emit one call object",
)
required = {"call_id", "tool", "arguments"}
if (
set(payload) != required
or not isinstance(payload.get("call_id"), str)
or not payload.get("call_id")
or not isinstance(payload.get("tool"), str)
or not payload.get("tool")
or not isinstance(payload.get("arguments"), dict)
):
return Decision(
"repair", "envelope", ("invalid_call_envelope",),
"regenerate only the call envelope under its schema",
)
return Proposal(payload["call_id"], payload["tool"], payload["arguments"])
def validate_shape(proposal: Proposal) -> Decision | None:
spec = TOOLS.get(proposal.tool)
if spec is None:
return Decision(
"repair", "tool_catalogue", ("unknown_tool",),
"reselect from the supplied tool catalogue or abstain",
)
keys = set(proposal.arguments)
missing = sorted(spec.required - keys)
extras = sorted(keys - spec.required)
wrong_types = sorted(
name
for name, expected_type in spec.field_types.items()
if name in proposal.arguments
and not isinstance(proposal.arguments[name], expected_type)
)
reasons = tuple(
[*(f"missing:{x}" for x in missing),
*(f"extra:{x}" for x in extras),
*(f"wrong_type:{x}" for x in wrong_types)]
)
if reasons:
return Decision(
"repair", "schema", reasons,
"repair only the fields named by the schema validator",
)
# A schema can express these lexical constraints. They remain shape checks.
try:
amount = money(proposal.arguments["amount_gbp"])
if amount <= 0:
raise ValueError("amount must be positive")
date.fromisoformat(proposal.arguments["payment_date"])
if spec.effectful:
floor = money(proposal.arguments["minimum_remaining_balance_gbp"])
if floor < 0:
raise ValueError("minimum balance cannot be negative")
if len(proposal.arguments["idempotency_key"]) < 8:
raise ValueError("idempotency key too short")
except ValueError as exc:
return Decision(
"repair", "schema", (str(exc),),
"repair the lexical value while preserving the user's meaning",
)
return None
def validate_tool_selection(request: UserRequest, proposal: Proposal) -> Decision | None:
expected = EXPECTED_TOOL[request.operation]
if proposal.tool != expected:
return Decision(
"reject", "tool_selection",
(f"requested:{request.operation.value}", f"proposed:{proposal.tool}"),
"reselect the operation; do not reuse arguments under a more powerful tool",
)
return None
def validate_semantics(
request: UserRequest,
proposal: Proposal,
authority: AuthorityContext,
) -> Decision | None:
args = proposal.arguments
comparisons = {
"account_id": (args["account_id"], request.account_id),
"payee_id": (args["payee_id"], request.payee_id),
"amount_gbp": (money(args["amount_gbp"]), request.amount_gbp),
"payment_date": (args["payment_date"], request.payment_date),
}
mismatches = tuple(
f"{field}:proposal={actual!s}:request={expected!s}"
for field, (actual, expected) in comparisons.items()
if actual != expected
)
if mismatches:
return Decision(
"repair", "semantics", mismatches,
"repair only the mismatched arguments from authoritative request data",
)
if proposal.tool == "schedule_transfer":
proposed_floor = money(args["minimum_remaining_balance_gbp"])
if proposed_floor != request.minimum_remaining_balance_gbp:
return Decision(
"repair", "semantics",
("minimum_balance_condition_was_changed_or_dropped",),
"restore the user's conditional constraint exactly",
)
remaining = authority.current_balance_gbp - request.amount_gbp
if remaining < proposed_floor:
return Decision(
"reject", "world_state",
(f"remaining_balance:{remaining}", f"required_floor:{proposed_floor}"),
"do not schedule; report the failed precondition",
)
return None
def validate_authority(
request: UserRequest,
proposal: Proposal,
authority: AuthorityContext,
) -> Decision | None:
if proposal.tool not in authority.allowed_tools:
return Decision(
"deny", "authority", ("tool_not_delegated",),
"escalate to an authorised actor; prompting cannot create authority",
)
spec = TOOLS[proposal.tool]
if not spec.effectful:
return None
if request.amount_gbp > authority.max_single_payment_gbp:
return Decision(
"deny", "authority",
(f"amount_exceeds_limit:{authority.max_single_payment_gbp}",),
"seek a separately authorised approval",
)
if not request.explicit_confirmation:
return Decision(
"deny", "authority", ("explicit_confirmation_missing",),
"ask the user for confirmation; never infer it from fluency",
)
return None
def execute_and_verify(
proposal: Proposal,
store: SyntheticPaymentStore,
*,
fail_after_commit: bool = False,
) -> Decision:
if proposal.tool == "quote_transfer":
receipt = store.quote(proposal.arguments)
return Decision(
"accept", "outcome", (),
"return the decision receipt; no effect was requested",
receipt,
)
key = proposal.arguments["idempotency_key"]
try:
receipt = store.schedule(
proposal.arguments,
fail_after_commit=fail_after_commit,
)
except ValueError as exc:
return Decision(
"reject", "idempotency", (str(exc),),
"use a new key for a new effect; never overwrite the prior meaning",
)
except TimeoutError:
# The command's transport outcome is unknown. Read authoritative state
# before deciding whether any retry is safe.
observed = store.readback(key)
if observed is None:
return Decision(
"unknown", "outcome", ("timeout_and_no_readback",),
"hold the case for reconciliation; do not issue a blind retry",
)
return Decision(
"accept", "readback", ("transport_timed_out_after_commit",),
"return the effect receipt recovered from authoritative state",
observed,
)
observed = store.readback(key)
if observed is None or observed["state"] != receipt["state"]:
return Decision(
"unknown", "postcondition", ("effect_not_verified",),
"stop and reconcile against authoritative state",
)
return Decision(
"accept", "postcondition", (),
"return the verified effect receipt",
receipt,
)
def process(
request: UserRequest,
raw_proposal: str,
authority: AuthorityContext,
store: SyntheticPaymentStore,
*,
fail_after_commit: bool = False,
) -> Decision:
proposal_or_decision = parse_proposal(raw_proposal)
if isinstance(proposal_or_decision, Decision):
return proposal_or_decision
proposal = proposal_or_decision
gates: tuple[Callable[[], Decision | None], ...] = (
lambda: validate_shape(proposal),
lambda: validate_tool_selection(request, proposal),
lambda: validate_semantics(request, proposal, authority),
lambda: validate_authority(request, proposal, authority),
)
for gate in gates:
decision = gate()
if decision is not None:
return decision
return execute_and_verify(
proposal, store, fail_after_commit=fail_after_commit
)
def proposal_json(tool: str, call_id: str, **arguments: str) -> str:
return json.dumps({
"call_id": call_id,
"tool": tool,
"arguments": arguments,
})
def print_result(name: str, decision: Decision) -> None:
reason = ", ".join(decision.reasons) if decision.reasons else "none"
effect = ""
if decision.receipt and "transfer_id" in decision.receipt:
effect = f" | transfer={decision.receipt['transfer_id']}"
print(
f"{name:26} {decision.status.upper():7} "
f"stage={decision.stage:14} reasons={reason}{effect}"
)
def demo() -> None:
store = SyntheticPaymentStore()
authority = AuthorityContext(
actor_id="user_demo",
allowed_tools=frozenset({"quote_transfer", "schedule_transfer"}),
max_single_payment_gbp=Decimal("1000.00"),
current_balance_gbp=Decimal("3000.00"),
)
quote_request = UserRequest(
RequestedOperation.QUOTE, "acct_demo", "supplier_7",
Decimal("600.00"), "2026-09-04", Decimal("1000.00"), False,
)
schedule_request = UserRequest(
RequestedOperation.SCHEDULE, "acct_demo", "supplier_7",
Decimal("600.00"), "2026-09-04", Decimal("1000.00"), True,
)
collision_request = UserRequest(
RequestedOperation.SCHEDULE, "acct_demo", "supplier_7",
Decimal("700.00"), "2026-09-04", Decimal("1000.00"), True,
)
cases: list[tuple[str, UserRequest, str, dict[str, Any]]] = [
(
"positive_quote",
quote_request,
proposal_json(
"quote_transfer", "c_01",
account_id="acct_demo", payee_id="supplier_7",
amount_gbp="600.00", payment_date="2026-09-04",
),
{},
),
(
"wrong_tool_valid_shape",
quote_request,
proposal_json(
"schedule_transfer", "c_02",
account_id="acct_demo", payee_id="supplier_7",
amount_gbp="600.00", payment_date="2026-09-04",
minimum_remaining_balance_gbp="1000.00",
idempotency_key="quote-became-effect",
),
{},
),
(
"semantic_amount_drift",
schedule_request,
proposal_json(
"schedule_transfer", "c_03",
account_id="acct_demo", payee_id="supplier_7",
amount_gbp="660.00", payment_date="2026-09-04",
minimum_remaining_balance_gbp="1000.00",
idempotency_key="semantic-drift-01",
),
{},
),
(
"timeout_after_commit",
schedule_request,
proposal_json(
"schedule_transfer", "c_04",
account_id="acct_demo", payee_id="supplier_7",
amount_gbp="600.00", payment_date="2026-09-04",
minimum_remaining_balance_gbp="1000.00",
idempotency_key="exactly-once-demo",
),
{"fail_after_commit": True},
),
(
"duplicate_same_key",
schedule_request,
proposal_json(
"schedule_transfer", "c_05",
account_id="acct_demo", payee_id="supplier_7",
amount_gbp="600.00", payment_date="2026-09-04",
minimum_remaining_balance_gbp="1000.00",
idempotency_key="exactly-once-demo",
),
{},
),
(
"same_key_different_effect",
collision_request,
proposal_json(
"schedule_transfer", "c_05b",
account_id="acct_demo", payee_id="supplier_7",
amount_gbp="700.00", payment_date="2026-09-04",
minimum_remaining_balance_gbp="1000.00",
idempotency_key="exactly-once-demo",
),
{},
),
]
results: dict[str, Decision] = {}
for name, request, raw, kwargs in cases:
decision = process(request, raw, authority, store, **kwargs)
results[name] = decision
print_result(name, decision)
over_limit_authority = AuthorityContext(
actor_id="user_demo",
allowed_tools=authority.allowed_tools,
max_single_payment_gbp=Decimal("500.00"),
current_balance_gbp=authority.current_balance_gbp,
)
over_limit = process(
schedule_request,
proposal_json(
"schedule_transfer", "c_06",
account_id="acct_demo", payee_id="supplier_7",
amount_gbp="600.00", payment_date="2026-09-04",
minimum_remaining_balance_gbp="1000.00",
idempotency_key="over-limit-demo",
),
over_limit_authority,
store,
)
print_result("valid_but_unauthorised", over_limit)
# Executable claims made by the article.
assert results["positive_quote"].status == "accept"
assert results["wrong_tool_valid_shape"].stage == "tool_selection"
assert results["semantic_amount_drift"].stage == "semantics"
assert results["timeout_after_commit"].stage == "readback"
assert results["duplicate_same_key"].receipt is not None
assert results["duplicate_same_key"].receipt["status"] == "replayed"
assert results["same_key_different_effect"].stage == "idempotency"
assert over_limit.status == "deny" and over_limit.stage == "authority"
assert len(store.scheduled) == 1, "timeout plus retry created a duplicate effect"
print("\nAssertions passed: one committed transfer, zero duplicate effects.")
if __name__ == "__main__":
demo()
How to test the causal claim
A useful experiment compares four systems on the same request set. System P uses prompt-only JSON instructions and a post-hoc parser. System R adds parse-and-retry. System C uses constrained decoding for the same schema. System G uses constrained decoding plus the independent gates in the harness. The request set should include malformed-output opportunities, ambiguous intents, closely related tools, stale state, missing confirmation, unsupported operations and timeouts before and after commit.
The causal prediction is specific. C should outperform P and R on structural conformance, particularly where grammar violations are common. It need not outperform them on semantic fidelity when the same wrong enum value remains allowed. G should leave structural conformance largely unchanged while improving unsafe-selection rejection, authority compliance and verified outcomes. If C alone improves those later metrics, inspect whether the schema also changed the task ontology or added an abstention branch. That is a useful result, but it is a different mechanism.
Record latency and cost by stage. Constraint compilation may introduce first-use overhead; shorter generations or fewer retries may recover it. Semantic and authority gates add deterministic calls, though many can reuse state already needed for execution. The right comparison is expected cost per verified outcome, not token cost per response. A cheap malformed or unauthorised call is not useful work.
Release decision instrument
Before a structured model output is connected to downstream code, classify both the model’s contribution and the consequence. The grid below is a compact architecture decision record. It assumes the parser and schema validator already pass. “Automatic” means the next deterministic step may proceed without a new human decision; it never means the model bypasses application controls.
Required gate strength rises with model contribution and consequence
| Question | Evidence before release | Blocking failure |
|---|---|---|
| Can the contract represent truth? | Abstention, uncertainty, units, provenance and relevant conditional branches appear in the schema. | The model must choose a false level of certainty or loses a user condition. |
| Does the runtime implement this schema? | Exact schema passes provider compilation and an adversarial conformance suite. | Unsupported keywords, silent fallback or schema-dependent parse failures. |
| Is tool selection evaluated separately? | Positive, negative, no-tool and least-powerful-fit cases, including similar tools. | Only argument accuracy is measured. |
| Are semantics rebound to evidence? | Entity, amount, date, units and conditions compare with authoritative request and state. | Schema-valid values can drift without detection. |
| Is authority outside the model? | Authenticated actor, subject, purpose, limits, confirmation and policy decision are bound mechanically. | Tool visibility or model choice is treated as permission. |
| Can an effect occur exactly once? | Stable idempotency key, timeout semantics, effect receipt and authoritative readback. | A missing response triggers a blind reissue. |
| Does repair preserve invariants? | Stage-specific errors, bounded retries, stop conditions and escalation paths. | A generic retry may alter intent, authority or effect count. |
Teams can convert each row into executable release tests. Generate extra and missing properties. Mutate decimals, units and entity identifiers while preserving type. Present a quote request beside a scheduling tool. Revoke permission after proposal but before effect. Return a timeout immediately before commit and immediately after. Assert that the first may retry safely, the second must read back. These tests reveal logic that a happy-path demo cannot.
stateDiagram-v2 [*] --> Claim state "Minimal worked example: an<br/>extraction that lies politely" as Claim state "The schema is also a theory of<br/>the task" as Test state "Thought experiment 1, the<br/>compulsory enum" as Revision Claim --> Test: expose mechanism Test --> Decision: evidence survives Test --> Revision: rival survives Revision --> Claim: narrow or rebuild Decision --> [*]
Compact glossary
- Structured output
- Model output constrained or validated against a declared data shape, usually so software can consume it without brittle prose parsing.
- Constrained decoding
- An inference procedure that masks next tokens which cannot lead to a completion admitted by a grammar or schema.
- Function call
- A model-emitted proposal naming a tool and its arguments. Custom application code still decides whether and how to execute it.
- Semantic validation
- Checking that values preserve the request, units, entities, conditions and authoritative evidence, beyond satisfying structural types.
- Typed action contract
- A bounded interface that specifies an operation’s inputs, preconditions, effects, errors and evidence without granting ambient authority.
- Decision receipt
- A record of intent, identity, evidence, proposal, policy result and reasons at the point an effect is approved or denied.
- Effect receipt
- Evidence from the executing system that identifies the accepted command and observed resulting state.
- Unknown outcome
- A state in which the caller cannot infer from transport success or failure whether a consequential effect committed.
Move the trust boundary, not the prompt
Structured output solves a real engineering problem. It replaces fragile text scraping with an explicit contract and, under constrained decoding, can make supported syntactic violations impossible. Function calling solves another real problem. It turns a model’s intended operation into inspectable data that application code can route, test and observe.
The mistake is to let integration convenience inflate the claim. Valid JSON is evidence about serialisation. Schema conformance is evidence about structure. Neither proves that the model selected the right operation, preserved the user’s conditions, consulted current state, held delegated authority or caused the intended outcome. Those propositions require different mechanisms and different receipts.
The architecture decision is to end model authority at the proposal boundary. Compile prompts and evidence into a bounded context. Let the model propose a typed call. Validate its shape, then test selection and semantics against authoritative inputs. Bind identity, purpose, permission and confirmation in a deterministic kernel. Execute with idempotency. Read back the effect. Preserve unknown outcome as an explicit state.
The experimental decision changes too. Do not celebrate a near-perfect parse rate without asking which allowed values are wrong. Measure the funnel stage by stage, inject failures around the commit boundary and optimise expected cost per verified outcome. Add an abstention branch where truth requires one. Remove the model where a deterministic mapper is enough.
A well-designed schema makes a stochastic system easier to handle. A well-designed authority and evidence path makes its proposals governable. The first is a language constraint. The second is the system.