Anatomy of a Production Multi-Agent System

Every multi-agent architecture diagram I see on LinkedIn has the same shape: a router at the top, three or four agents in boxes, some arrows to "tools" and a cylinder labelled "vector DB". That diagram describes the demo. It does not describe anything that has survived contact with a global financial institution's risk function, an operational resilience review, or six months of real traffic.

I have spent the last several years building multi-agent platforms in exactly that environment. What follows is the architecture that actually ships: the layers that exist not because a framework suggested them, but because something broke, an auditor asked a question we couldn't answer, or a regulator's expectations made them non-negotiable.

Map, not take: this is a description of what a production system looks like when it has to survive scrutiny. Your topology will differ. The layers won't.

Layer 1: orchestration: the part everyone over-designs and under-specifies

Orchestration is where people spend 80% of their design energy and it deserves maybe 30%. The important decisions are not "which framework" but three structural ones:

Deterministic skeleton, probabilistic muscles. In a regulated environment, the top-level flow should be a state machine you can draw on a whiteboard and explain to a model risk reviewer. Agents make decisions within states; they do not invent the state graph at runtime. Fully dynamic agent-decides-everything orchestration is a research pattern, not a production pattern, not because it can't work, but because you cannot bound its behaviour, and bounding behaviour is the entire game in banking.

Explicit handoffs with typed contracts. When the triage agent hands work to the resolution agent, that handoff is a schema-validated message, not a blob of prose. Prose handoffs are where multi-agent systems silently degrade: agent A summarises badly, agent B works from the bad summary, and nobody notices until the output is wrong in a way no single agent caused.

A supervisor that can say no. Every orchestration layer needs a component with the authority to halt a run: budget exceeded, loop detected, confidence below threshold, tool error rate spiking. A healthy supervisor will terminate ordinary production runs, not only pathological test cases. That is the point.

Layer 2: the LLM gateway: the highest-leverage shared component

If you take one thing from this article: put a gateway between every agent and every model, on day one.

The gateway is where you get, in one place:

• Model routing and abstraction. Agents request capabilities ("reasoning-heavy", "fast-classification"), not model names. When a vendor deprecates a model or your procurement team renegotiates, you change routing config, not forty agent definitions.
• Cost and rate governance. Per-agent, per-use-case and per-tenant token budgets, enforced centrally. Cost attribution should expose background jobs and repeated context that disappear inside a shared API key; measure the distribution before setting budgets.
• Centralised guardrails. PII redaction, prompt-injection screening and output filtering live in the gateway, not in each agent. Per-agent implementations will otherwise diverge as teams and release cadences differ.
• Governed evidence capture. Record the minimum metadata needed to attribute and reconstruct a material path: model and prompt-template versions, retrieved-source references, policy decisions, tool outcomes and human actions. Store prompt or completion content only when an approved purpose, access model and retention rule require it; protected content should not become general platform telemetry.
• Failover. Primary model degraded? Route to secondary with an explicit quality flag on the response so downstream consumers know the answer came from the fallback path.

The gateway is boring infrastructure. It is also what makes a model upgrade a controlled routing change rather than a rewrite across every agent.

Layer 3: tool registry: capability is a governed asset

An agent's tools define its blast radius. In a regulated environment, "what can this agent actually do" is a question you will be asked formally, in writing, by people with the power to shut you down. You need to answer it from a system of record, not from reading prompt files.

A production tool registry treats every tool as a governed asset with:

• A schema and a semantic contract: not just parameter types, but preconditions, side effects, and idempotency guarantees. Agents retry. A non-idempotent tool that an agent retries is an incident waiting to happen.
• A risk tier. Read-only lookup tools are tier 1. Tools that mutate customer state are tier 3 and require human approval (Layer 5) or hard eligibility rules before invocation.
• Per-agent entitlements. The fraud triage agent cannot see, let alone call, the payments-release tool. Least privilege applies to agents exactly as it applies to humans, and for the same reasons.
• Versioning. Tools change. Pin agents to tool versions and manage migrations so an upstream API change cannot silently alter behaviour.

The registry also gives you something subtler: an inventory. When an auditor asks “enumerate every action AI systems can take against customer accounts,” the answer should be a query, not a trawl through prompt files.

Layer 4: memory, knowledge, and context: three different problems

Teams conflate these constantly. They have different consistency requirements, different lifecycles, and different failure modes.

Working memory is the structured operational state of a single run: workflow status, validated intermediate results, tool outcomes, evidence references and pending decisions. It is not the model's hidden chain-of-thought or an instruction to persist an unrestricted scratchpad. Externalise only the state needed to pause, resume, inspect and safely replay the workflow, keyed by run ID and governed by its data class. Reconstruction should rely on recorded inputs, decisions and outcomes rather than treating private model reasoning as evidence.

Knowledge is the curated corpus agents retrieve from: policy documents, product terms, procedures. The retrieval pipeline matters less than the corpus governance around it. In a bank, retrieving a superseded policy document isn't a quality bug, it's a compliance event. Every knowledge item carries an owner, an effective date, and a review cycle, and the retrieval layer filters on validity as of now. This is unglamorous data management, and it determines your answer quality more than any embedding model choice.

Long-term memory (persisting things across sessions about customers or cases) deserves the most suspicion. It creates data-protection obligations (what was remembered, on what basis, can the customer see it and request correction or deletion?) and it compounds errors: a wrong “fact” memorised in March poisons every interaction after it. Ship long-term memory narrow and late, with explicit provenance on every stored item.

Layer 5: human-in-the-loop: designed as a workflow, not an apology

Most HITL implementations are a Slack message and a prayer. In production, human oversight is a first-class workflow system with queues, SLAs, and skills-based routing, because the humans are part of the system, and an unmodelled component is an unreliable one.

Three design rules matter:

Route by risk, not by confidence alone. Model confidence is a poor proxy for consequence. A 95%-confident action that moves money needs review; a 70%-confident answer to an internal FAQ does not. Approval routing keys on the risk tier of the action, combined with confidence as a secondary signal.

Make the review real. If reviewers approve nearly everything in seconds, the process is a rubber stamp that appears in the audit trail as evidence against effective oversight. Monitor agreement, challenge and review duration, and seed controlled known-bad cases to verify that the review function remains active.

Feed decisions back. Every human override is a labelled datapoint. If your HITL layer isn't feeding your eval sets and your improvement backlog, you're paying for oversight and throwing away the training signal.

Layer 6: audit and observability: the layer you build for the reader you'll never meet

Material steps emit minimised events into an append-only evidence spine, correlated by run ID. A typical record identifies the event type, component and policy versions, source references, tool or handoff result, human decision and guardrail outcome. Prompt or response content is retained only when the institution has approved the purpose, protection and applicable schedule; otherwise the record uses governed references and non-sensitive metadata. Each evidence class follows its own legal, regulatory and records obligation rather than a casual application-log default.

The design test is narrower and more defensible: for the required review period, can retained evidence reconstruct the material basis and execution path of a specific action: the model and prompt-template versions, decisive source records, policy and tool outcomes, and any human approval? The answer should not depend on hidden chain-of-thought or on retaining every conversational payload.

Observability sits on top of the same spine: cost per resolved case, handoff failure rates, guardrail trigger rates, drift in output distributions. The audit trail and the ops dashboard are two views of one event stream. Build it once.

The honest summary

Notice what's absent from this article: agent frameworks, prompt techniques, model choices. Those are the parts that change every quarter and the parts that matter least. The six layers above are where the actual engineering lives, and roughly five of them are classical distributed-systems and governance work wearing an AI badge.

The demo can take a weekend. The gateway, registry, audit spine and human workflow require sustained platform work. That work is the product.

The system as two interlocking planes

The six layers become easier to reason about when separated into a decision plane and an execution plane. The decision plane interprets intent, retrieves evidence and proposes the next step. The execution plane checks authority, validates parameters and records the outcome. A supervisor may choose a tool. It does not get to waive the tool's contract.

Reasoning is replaceable; the enforcement plane is institutional infrastructure. This boundary also makes model migration less disruptive. Models can change without changing the authorization or evidence contract.

A portfolio matrix for agent boundaries

The useful portfolio question is not “single agent or multi-agent?” It is “how much independent judgement and consequential authority does this task combine?” The matrix below prevents needless agent proliferation while identifying tasks that need stronger separation.

The rule is deliberately conservative. Specialisation earns an agent boundary only when it improves control, evaluation or ownership. A new persona or prompt is not an architectural reason.

Define the execution envelope before the agent graph

A production design starts with an execution envelope: the limits that apply to one business trajectory regardless of which specialist receives the next step. The envelope names the authenticated principal, case, allowed data domains, action ceiling, time limit, cost limit and required terminal states. Every agent inherits it. No handoff can widen it.

This changes the role of orchestration. The supervisor is no longer free to improvise an unlimited graph. It selects from transitions permitted by the envelope and current workflow state. The tool gateway rejects calls outside the declared case or action ceiling. A human approval can widen one named dimension for one named proposal, but it does not grant the whole graph broader authority.

The terminal states deserve attention. “Failed” is not precise enough. A bank workflow normally needs separate states for business refusal, technical failure, human referral, policy conflict, expired request and completed action. Those states drive customer messaging, operational queues and regulatory records. Leaving them as prose in the final answer creates downstream ambiguity.

The graph may be adaptive inside the envelope; the envelope itself is not a model output. That single rule prevents a specialist from turning a diagnostic task into an execution task merely because a retrieved document suggested it.

Make ownership match failure boundaries

The six layers should not become six shared responsibilities. Shared responsibility often means no one owns the failure that crosses two layers. A compact operating model assigns a decision owner, a runtime owner and an evidence owner for each boundary.

This is not a generic RACI exercise. The boundaries correspond to distinct forms of harm. A model-provider outage calls for traffic management. A superseded policy in retrieval calls for corpus withdrawal. A duplicate mutation calls for idempotency repair and case reconciliation. One “AI operations” queue cannot make all three decisions well.

The ownership model should appear in runbooks. On-call staff need explicit authority to disable a tool, pin a model version, withdraw a knowledge source, divert cases to a human queue and stop an agent route. Containment authority must be designed before the incident that needs it.

Release the system as a set of contracts

An agent release is rarely only a prompt release. A change may alter the model route, tool schema, retrieval corpus, state graph, approval rule or evaluation threshold. Treating the application as one undifferentiated version hides interactions. Treating every component independently misses the tested combination.

The release manifest should therefore name both. It records the version of each contract and assigns one tested system version to the combination. A trace can then answer which model, system instruction, tool adapter, knowledge snapshot, policy and evaluator were active together.

Canary evidence should be outcome-based. A lower error rate is useful but incomplete. The release should also preserve referral quality, tool-call validity, groundedness, authority denials, queue impact and cost per resolved case. A change that improves answer style while increasing incorrect tool proposals is a regression.

Rollback must respect state. Reverting code does not undo a case transition or delete an incorrect memory. The release plan needs a state-reconciliation procedure for any component that can create durable effects. For material tools, the procedure belongs in the contract before access is granted.

Reconstruct incidents across agent boundaries

Multi-agent failures are often compositional. The first specialist retrieves a plausible but stale rule. The second compresses away its effective date. The verifier checks consistency with the summary rather than the source. The executor receives valid parameters for the wrong conclusion. No component records an obvious exception.

The incident method must follow causality, not agent names. Start from the external effect and walk backward through the resource mutation, policy decision, proposed action, handoff input, source evidence and original request. Each material event needs a parent identifier, contract version and the minimum event-specific evidence required for reconstruction. Where integrity must be proved and policy permits it, retain a protected content reference or digest; do not make universal input and output capture the price of traceability.

The same chain supports forward replay. Investigators can replace the stale source, preserve the original request and run the trajectory in a non-executing environment. The comparison shows whether the failure came from retrieval, handoff compression, verifier criteria or action policy. Replay is more informative than asking the final model to explain itself.

A multi-agent incident has one causal graph, not one root cause per agent. The corrective action should strengthen the boundary that allowed error to propagate. That may mean an effective-date field becomes mandatory, a verifier reads the primary source, or the executor rejects a proposal without a source identifier.

Capacity is a workflow property

Agent capacity is not simply model requests per second. One customer case may create several model calls, retrievals, tool calls and a human review. The shape varies by route. A small increase in referral rate can overload the human queue while model infrastructure remains healthy.

Capacity planning should therefore start with route classes. For each class, estimate steps per case, concurrency, tool fan-out, token range, expected referral probability and recovery path. Load tests should include slow tools, throttled models, unavailable knowledge sources and a constrained approval queue. The purpose is to expose where back-pressure appears.

Back-pressure should change behaviour safely. Low-priority cases may wait. Draft-only functions may use a smaller model. Consequential actions may stop rather than skip review. Retrieval may serve a last-known-good snapshot only if its freshness policy permits it. Graceful degradation means reducing capability without silently reducing control.

This is why the production system is more than the agent graph. The graph describes possible reasoning. The platform defines which reasoning is permitted, how it is supplied, what it can change, and what happens when any dependency weakens.

Decide whether a capability is an agent, tool or service

Many multi-agent platforms accumulate specialists because an agent is the easiest unit to add. That creates extra prompts, handoffs and evaluation surfaces without adding a meaningful boundary. Before creating a new agent, decide what kind of capability is actually needed.

Use an agent when the step requires interpretation among several plausible paths, benefits from a distinct context contract, and can be evaluated against an owned outcome. A policy-comparison specialist qualifies because it reconciles evidence under explicit criteria. A JSON formatter does not.

Use a tool when the operation has a stable typed contract and should behave deterministically for the same inputs. Calculations, eligibility checks, record lookups and mutations belong here. The model can choose or populate the tool. It should not recreate the tool's business rule in prose.

Use a service when several journeys need the same durable capability with its own scale, security or reliability boundary. Identity resolution, retrieval, policy decisions, model routing and evidence capture usually fit this class. Calling each one an agent understates its operational obligations.

A handoff supplies another useful test. If the receiving component needs only a few typed fields, it may be a tool. If it needs a distinct evidence package, decision rubric and evaluation set, an agent boundary may be justified. If every workflow needs it and it owns persistent state, it is probably a platform or domain service.

The distinction affects incident handling. An agent error is investigated through context, model and evaluation evidence. A tool error is investigated through contract, input and state-transition records. A service failure is investigated through availability, security and dependency telemetry. Blurring the categories blurs the corrective action.

Agent count is not a maturity measure. Boundary quality is. A smaller topology with typed tools and strong services is usually easier to govern than a large cast of agents exchanging summaries.

Revisit the decision after production evidence arrives. A specialist whose outputs are nearly deterministic may become a service or policy rule. A tool that increasingly handles ambiguous exceptions may need a governed human or agent decision before execution. Architecture should follow the observed work, not preserve the first diagram.

Primary references

• NIST, AI Risk Management Framework 1.0, for lifecycle governance and evidence-oriented risk management.
• NIST, Generative AI Profile, for generative-system risks and suggested actions.
• AWS, Use multi-agent collaboration with Amazon Bedrock Agents, for the managed supervisor-and-collaborator pattern.
• Google, Site Reliability Engineering Workbook: Monitoring, for service-level telemetry and actionable monitoring.
• IETF, RFC 8693: OAuth 2.0 Token Exchange, for delegated and exchanged security tokens.

A production multi-agent system is a governed transaction system with probabilistic workers, not a collection of conversational characters.