Executive briefA correct answer can still produce a wrong external action

A refund agent reads a valid customer record, correctly infers that the complaint is eligible and calls the approved payment tool. The action is nevertheless wrong. The account state was refreshed after the model assembled its context, the customer has already received a manual credit and the service identity still holds authority to commit another payment. Model quality, tool availability and policy wording can each appear satisfactory while the enterprise sends money twice.

Traditional model risk remains necessary. Leaders still need conceptual soundness, implementation accuracy, intended-use boundaries, validation, monitoring, effective challenge and governance. When a model can select or sequence actions, however, the system also interprets changing context, forms a plan, crosses authority boundaries, creates external effects, observes partial results and decides whether to continue. The relevant failure may exist between otherwise acceptable components or emerge only across time.

The risk object is therefore the governed action system. It includes the model, prompts, policies, context, memory, orchestration, identity, delegated authority, tool contracts, humans, runtime gates, observations, interruption, recovery and business outcome. A flawed intermediate inference can be safely contained. A sound output can cause harm because authority, state or order is wrong. Several permitted actions can combine into an impermissible commitment.

This article proposes an Action-Capable Model Risk Record, an Authority-Effect Ladder and a Trajectory Assurance Envelope. Together they preserve familiar validation disciplines while extending evidence where model-mediated planning, uncertainty and feedback materially change action risk. A deterministic workflow that uses a model only for bounded classification may remain within existing treatment.

The strongest rival: join existing controls

Existing model risk, software assurance, access control, operational risk and change management can cover much of this problem when responsibilities and evidence are connected. That is the strongest alternative to a separate agent-risk function. The proposed record should extend, not displace, those systems. Model risk challenges uncertainty and intended use; security governs identity and access; operational risk tests process failure; product owns the design; the business principal owns the objective and commitment; tool owners own side effects and compensation.

The boundary is model-mediated discretion over action. If a deterministic orchestrator receives a bounded classification and performs fixed steps under conventional transaction controls, ordinary validation and software assurance may be sufficient. If a model chooses tools, order, retries, stopping, counterparties or scope, trajectory-specific evidence becomes material.

Action-Capable Model Risk RecordTwelve linked domains make the operating path inspectable

The domains are linked records rather than a checklist. A decision contract points to the authorised identity; the identity points to tool contracts; each tool effect creates a receipt in the trajectory; the trajectory is checked against the assurance envelope; observation triggers intervention; recovery links affected parties to reconciliation and redress. W3C PROV-O offers a useful vocabulary for connecting entities, activities and agents, but the enterprise must still define the business meaning of an authorised effect.

Evidence must be independent of the action system where material. The same process cannot be the only source asserting which policy ran, which credential was used and whether a side effect occurred. Independent policy-decision logs, transaction-system records, identity events, human approval records and customer outcomes provide corroboration.

Authority-Effect LadderThe same model needs different assurance at a higher rung

Authority is not a binary switch. An observing system reads permitted context without changing state. A recommending system offers advice that cannot itself commit the enterprise. A preparing system drafts a proposed transaction for explicit approval. A bounded committer executes a narrow, reversible action under transaction-specific authority. A coordinator sequences several bounded actions across systems. A self-directing system chooses goals or materially expands scope.

Control intensity rises with the rung because the model's uncertainty can reach further into external state. Read access still deserves data minimisation and entitlement controls, but a mistaken observation does not itself transfer funds. Preparation requires complete approval context and protection against substitution between proposal and approval. Bounded commitment requires independent transaction limits, idempotency and compensation. Coordination requires cumulative limits and composition tests. Self-direction should normally be inadmissible in enterprise operation because an authority boundary cannot be established if the system may choose or expand its own goal.

Assign the rung from actual capability, not interface language. A copilot that silently submits a draft is not merely recommending. A workflow presented as bounded commitment may be coordinating if it can create a supplier, issue an order and schedule payment. The highest reachable rung governs assurance even if common usage stays lower. Remove unused permissions and unreachable tool paths rather than arguing that the model is unlikely to use them.

Trajectory Assurance EnvelopeBound the whole path, not only each call

The Trajectory Assurance Envelope defines the authorised state space around an action sequence. It fixes maximum steps, elapsed time, cost, resources, counterparties and cumulative effect. It also names required checkpoints, transaction-specific authorisations, forbidden transitions and safe termination. A prompt asking the model to be cautious does not enforce any of these boundaries.

Limits are multidimensional. Five refunds below an individual cap may still exceed the customer or incident limit. A procurement agent may be allowed to search suppliers, create a vendor draft and prepare a purchase order, yet the combination can establish a commitment that no single tool reports. A retry can duplicate a non-idempotent effect. Parallel branches can each pass a rate gate while their combined exposure breaches the authorised envelope.

Enforcement should sit at trusted boundaries. An identity broker issues short-lived, purpose-bound authority. A policy service evaluates the exact resource and transaction. Tool gateways enforce typed inputs and preconditions. A trajectory monitor aggregates cost and effect across calls. Independent authorisation protects material commitments. The orchestrator receives denials and stop signals but cannot redefine the limit that denied it.

Every external effect produces an action receipt containing the principal, action-system identity, policy and model versions, context reference, tool, request, response, external transaction reference, cumulative exposure and time. A terminal receipt states completed, contained, failed or abandoned, plus unresolved effects. This distinction prevents a stopped workflow from being misreported as a reversed transaction.

Test the envelope with stateful sequences, not isolated prompts. Vary order, delayed responses, stale context, partial tool failure, duplicated events, concurrent branches, injected instructions, revoked authority and unavailable humans. AgentBench, tau-bench, ToolEmu and AgentDojo contribute useful test environments and adversarial patterns. Their results guide test design; they do not certify a production trajectory whose tools, identities, data and recovery paths differ.

Composition and independent evidencePermitted calls can form an impermissible commitment

Independent tool testing is necessary but incomplete. The semantic effect of a sequence can exceed the sum of its calls. A procurement system can discover a supplier, create its master record, negotiate a quantity, draft an order and schedule delivery. Each operation may be allowed for an employee in isolation. Together they can communicate acceptance, create a legal or commercial expectation and bypass the purchasing authority that was supposed to approve the whole commitment.

Composition tests start from prohibited business outcomes and work backwards through reachable paths. Enumerate tools, side effects, shared state and transitions. Identify actions whose order changes meaning, such as approving before refreshing sanctions status or cancelling after a shipment is irreversible. Test alternative plans, retries, partial failures and model re-entry. The controlling assertion is a business invariant, such as no supplier commitment without a transaction-specific approval, not merely every API returned success.

OWASP's excessive-agency guidance connects risk to excessive functionality, permissions and autonomy. NIST zero-trust architecture supports explicit, resource-specific access decisions rather than ambient trust. SP 800-53 provides control families for access, separation of duties, audit, integrity and contingency. SP 800-218A adds secure development practices for generative AI and dual-use foundation models. MITRE ATLAS and AgentDojo inform attack paths where untrusted content influences tool use. None removes the need to validate the organisation's actual contracts and recovery.

A receipt chain should allow an independent reviewer to reconstruct model contribution, human intervention, policy decision, tool effect and resulting state. Record references rather than unrestricted content where possible. Protect logs against modification by the action system. Reconcile receipts with systems of record and with counterparty or customer evidence. Missing receipts are a control failure, not proof that no action occurred.

Human intervention windowAn approval click is not effective challenge

A person can retain formal authority while lacking a meaningful chance to exercise it. The supervisor may receive only the model's summary, face a queue that allows seconds per case and encounter the decision after a payment or message has become difficult to reverse. A bright approval button does not repair missing context, time, skill, independence or stopping power.

Map the intervention window as four times: when material information becomes available, when it reaches the person, when a competent decision can be formed and when the external action becomes irreversible or costly to undo. The usable window is the remainder. Delayed telemetry and rushed queues can reduce it to zero. Test with realistic workload, interruptions, ambiguous evidence and disagreement, not a demonstration case prepared for reviewers.

The human needs authority over the relevant commitment. A reviewer who can reject one call but cannot pause the trajectory, revoke credentials or alter the plan may be bypassed by another branch. Approval must bind the exact proposal, state, amount, counterparty and expiry. If any material field changes, authorisation expires. Record the evidence displayed, time available, decision, reason, model recommendation and later outcome so automation bias and rubber-stamping can be examined.

Human involvement can support recommendation and preparation when commitment remains with an informed professional or customer-service decision maker. It is weaker when the person merely monitors a fast coordinator. Where machine-speed action is necessary, rely on independent preventive limits and tested containment rather than pretending a person can catch every harmful transition.

Contain, reconcile and redressStopping the process does not undo the world

Containment prevents further effects. Reconciliation establishes what actually happened across internal systems, counterparties and affected people. Redress corrects or compensates the consequence. These are separate operations. A kill switch may revoke an identity and stop new calls, yet payments, messages, bookings or supplier expectations already created still exist.

Design recovery from the side effect backwards. For every tool action, ask whether it is reversible, compensatable or irreversible; which owner can act; what deadline applies; which external reference confirms state; and how affected parties are identified. Idempotency keys protect repeated requests only when the external system honours them across the relevant boundary. A workflow-level identifier cannot prevent duplicate settlement if it is not passed into the payment contract.

A harmful refund loop illustrates the sequence. An independent cumulative gate detects exposure and denies the next transaction. The identity broker revokes delegated authority. Operations compare action receipts against the payment ledger and customer account state. Duplicate credits are classified by customer and transaction. Product and service leaders decide whether to reverse, leave funds in place or compensate additional harm. Customers receive accurate notice, a contact route and appeal. Validation uses the incident to add retry, stale-state and concurrent-branch tests before authority can be restored.

Recovery proof belongs in release evidence. Run drills with partial logging, unavailable tool owners, delayed counterparties and disputed customer state. Measure whether the team can identify affected parties, reach a safe state and account for unresolved effects. A successful rollback in a test database is not sufficient where external commitments cannot be rewound.

Ten worked casesThe twelve domains change the disposition, not just the documentation

Each case below is recorded across all twelve domains. The table compresses the decisive fields, while the accompanying analysis names the ladder rung, envelope, independent proof, validation tests and joint owners. The cases include a cumulative trajectory where every call is allowed and a proposed self-directing system that should receive no production action rights.

1. Inbound classification with deterministic action

The decision contract assigns a case category, not a business outcome. The model contribution is bounded classification; retrieved context is provenance-checked; a fixed workflow owns routing and action. Identity cannot invoke external tools. The rung is recommend, with an envelope of one classification, one case and no side effect. Test ambiguity, abstention, distribution shift and workflow misuse. Independent routing and outcome records support validation. Model risk, operations and the case-workflow owner share accountability. Keep the model as bounded input to a deterministic workflow.

2. Payment-investigation copilot

The copilot recommends evidence to examine and cannot freeze, contact or settle. The decision contract prohibits commitment. Context records customer state and investigative sources; the human sees provenance, uncertainty and alternatives. The rung is recommend. The envelope ends when advice is delivered. Test misleading explanations, missing facts, automation bias and prohibited tool reach. The investigator's signed decision and downstream account outcome are independent evidence. Model risk, fraud operations, product and the business principal own the path. Allow recommendation or preparation with human commitment.

3. Refund prepared for approval

The system can draft a transaction but cannot submit it. Context and customer state are refreshed at approval. The exact customer, amount, reason and expiry are bound into an immutable proposal. The rung is prepare. The envelope permits one proposal, no retry and no external effect before transaction-specific human approval. Test substitution, stale state, duplicate prior credit and approval timeout. Payment records and approval signatures are independent proof. Service operations, payments, security, model risk and product share ownership. Allow recommendation or preparation with human commitment.

4. Low-value reversible refunds

The decision contract limits a verified service failure. A purpose-bound identity can commit one reversible refund for the named customer. Tool contracts enforce idempotency and compensation. The rung is commit bounded. The envelope caps amount, customer exposure, hourly rate, total incident exposure, time and retries. Test concurrent branches, non-idempotent responses, stale customer state and revoked authority. Policy logs, payment ledger entries and customer receipts corroborate the trace. Payments, customer operations, security, operational risk, model risk and product jointly own release. Permit bounded action with independent authorisation controls.

5. Procurement composition

The coordinator can call allowed supplier, vendor, order and scheduling tools, but their sequence creates an unauthorised commitment. The rung is coordinate. The decision contract and cumulative envelope are missing the invariant that supplier commitment requires separate purchasing authority. Test alternate ordering, partial failure, implied acceptance, retry and cross-system state. Supplier communication and enterprise procurement records provide independent evidence. Procurement owns the commitment; tool owners, security, legal, operational risk, model risk and product own controls. Keep in sandbox or shadow until trajectory recovery is proven.

6. Stale entitlement and customer state

A retrieval and tool-use system reads an entitlement snapshot that changed before action. Its context record lacks freshness and reauthorisation. The rung may be commit bounded, but the envelope is invalid because authorised state is not bound to action time. Test entitlement revocation, account closure, manual intervention and delayed tool responses. Identity events and customer systems of record must contradict stale internal memory when appropriate. Security, the data owner, customer operations, model risk and product share correction. Keep in sandbox or shadow until trajectory recovery is proven.

7. Non-idempotent retry loop

The original payment succeeds while the tool response times out. The orchestrator retries and multiplies the external effect. The rung is commit bounded, yet the action trajectory exceeds its cumulative envelope. Enforce a business idempotency key at the payment boundary, cap retries and reconcile uncertain outcomes before continuation. Test response loss, duplicated events, concurrent execution and compensation failure. The external ledger, gateway receipts and customer account provide proof. Payments, platform, operational risk, security, model risk and product own the fix. Keep in sandbox or shadow until trajectory recovery is proven.

8. Supervisor without a real window

A supervisor formally approves a coordinated action but sees an incomplete summary seconds before commitment. The human domain fails on information and time; the intervention point cannot pause every branch. The system therefore operates at coordinate even though the interface says human in the loop. Test realistic queue pressure, contradictory evidence, authority revocation and disagreement. Approval timing, displayed content and later outcome are independent evidence. Operations leadership, workforce design, product, model risk, security and the business principal share responsibility. Allow recommendation or preparation with human commitment after moving commitment back to the person.

9. Contained harm with reconciliation and redress

An independent gate contains a refund trajectory after one harmful effect. Authority is revoked, receipts are reconciled to the payment ledger, affected customers are identified and remedy is delivered. The rung is commit bounded. The revised envelope requires checkpoints before cumulative exposure grows and a terminal receipt listing unresolved effects. Test partial logs, unavailable owners, disputed state and compensation failure. Customer contact, payment records and redress receipts corroborate recovery. Operations, payments, product, security, model risk and assurance jointly decide restoration. Permit bounded action with independent authorisation controls only after recovery proof passes.

10. Proposed self-directing enterprise agent

The design permits the model to choose goals, add counterparties and expand its tool scope in pursuit of an open objective. No decision contract can name completion, prohibited transitions or accountable delegation. The rung is self-direct. A credible assurance envelope cannot be drawn because the authorised state space changes under the system's own planning. Benchmark performance and a broad monitoring console do not repair this. The business principal, board delegate, security, operational risk, model risk and product must require bounded redesign. Remove action authority from this design.

Joint ownershipModel risk cannot own the enterprise commitment alone

Model risk challenges conceptual soundness, limitations, use, monitoring and outcomes. It should examine how model uncertainty changes planning and tool selection, but it does not own the commercial goal, credentials, tool implementation or customer remedy. Assigning the entire system to model risk would obscure the people who can actually change those elements.

The business principal owns the objective, prohibited goals, authorised effect and residual risk. Product owns the decision contract and design. Engineering and platform own orchestration, version integrity and receipts. Security owns identity, delegation, policy independence and revocation. Tool owners own preconditions, side effects, idempotency and compensation. Operational risk owns process failure and resilience. Frontline operations own intervention capacity and reconciliation. Legal and compliance advise on obligations. Independent assurance tests evidence and control operation.

Release requires joined decisions. No function can approve beyond its authority. A model validator cannot accept unreconciled payment risk; a tool owner cannot accept model uncertainty; a product leader cannot waive identity controls; a business principal cannot declare an ineffective remedy adequate. Record dissent, conditions and expiry. Escalation goes to an authority able to change scope, budget, capability or release.

Interactive instrumentActing Model Risk Disposition Board

Select a named scenario or alter the twelve operating domains. The Board returns exactly one of five dispositions. It reports the ladder rung, controlling failure, envelope, missing validation, constraints, human role, adversarial tests, receipts, containment and recovery proof, residual risk, release condition and joint owners. It does not produce an agent score or treat a human checkbox as assurance.

Executive operating cadenceReview authority changes, trajectory evidence and unresolved effects

The Action-Capable Model Risk Record, Authority-Effect Ladder, Trajectory Assurance Envelope and Acting Model Risk Disposition Board are this article's synthesis. The April 2026 interagency US banking guidance replaced SR 11-7 and OCC Bulletin 2011-12. It emphasises risk-based model development and use, validation and monitoring, governance and vendor products, but expressly states that generative and agentic AI are outside its scope. This article uses its continuing disciplines as an analogy and foundation, not as a claim that the guidance directly governs action-capable AI.

NIST AI RMF 1.0 remains the current published framework while revision work proceeds, and the July 2024 Generative AI Profile adds generative-specific risk actions. ISO/IEC 23894:2023 and ISO/IEC 42001:2023 support organisational risk management and management-system responsibilities. The EU AI Act provides legal duties in its applicable contexts. These sources guide governance; the enterprise still needs system-specific evidence for authority, trajectories and recovery.

At release review, show the decision contract, actual ladder rung, reachable tools, action and cumulative limits, human window, independent policy path, adversarial test coverage, containment drill, reconciliation result, residual effects and named joint owners. Do not substitute a model benchmark, red-team total or average task-completion measure. Release evidence must cover the production identities, tools, state transitions and counterparties that can create the harm.

During operation, review denied actions, near-boundary trajectories, retries, stale-state detections, human overrides, pauses, revocations, containment events, failed reconciliation and appeals. Look for clusters across models and products. A rising number of successful actions can increase exposure even when per-action quality is stable. Expansion in volume, counterparty class or permitted sequence is a model-risk and operational-risk change.

Revalidate when the model, prompt, context source, memory, plan logic, tool contract, credential, policy, limit, human workflow, outcome pattern, recovery owner or business objective changes. Authority changes deserve immediate review because they can move the same model to a higher ladder rung. Incidents reopen the relevant domains and the envelope. Preserve earlier versions and decisions so effective challenge can reconstruct why production authority existed.

The board-level question is simple but demanding: can the enterprise state exactly what the action system may achieve, show each authorised effect, stop it before cumulative harm grows, account for what happened outside the workflow and remedy affected people? If any answer depends only on the model's instruction or self-report, action authority is not yet governed.

Source recordWorks cited

1. Federal Reserve, OCC and FDIC. Revised Guidance on Model Risk Management, SR 26-2.
2. Office of the Comptroller of the Currency. OCC Bulletin 2026-13: Model Risk Management, Revised Guidance.
3. NIST. Artificial Intelligence Risk Management Framework.
4. NIST. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile.
5. ISO. ISO/IEC 23894:2023, Artificial intelligence, guidance on risk management.
6. ISO. ISO/IEC 42001:2023, Artificial intelligence management system.
7. European Union. Regulation (EU) 2024/1689.
8. NIST. SP 800-207, Zero Trust Architecture.
9. NIST. SP 800-53 Rev. 5, Security and Privacy Controls.
10. NIST. SP 800-218A, Secure Software Development Practices for Generative AI and Dual-Use Foundation Models.
11. OWASP. LLM06:2025 Excessive Agency.
12. MITRE. ATLAS Adversarial Threat Landscape for AI Systems.
13. Ruan, Y. et al. Identifying the Risks of LM Agents with an LM-Emulated Sandbox.
14. Liu, X. et al. AgentBench: Evaluating LLMs as Agents.
15. Yao, S. et al. tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains.
16. Debenedetti, E. et al. AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defences for LLM Agents.
17. W3C. PROV-O: The PROV Ontology.