Executive briefA gateway should stabilise policy, not erase provider reality

A privacy restriction copied into three provider dashboards looks like control until one version drifts and an outage triggers a fourth route. The application sends the same payload shape, but the substitute model handles retention, tool calls, structured output and safety settings differently. The gateway reports success because the request completed. The enterprise discovers later that provider configuration had become the policy source and that fallback had bypassed the reason the restriction existed.

A useful model gateway separates five things that vendors and proxy products often collapse: policy intent, policy decision, enforcement, provider adaptation and evidence. Policy owners publish what the enterprise permits and requires. A compiler produces machine-testable rules. A decision point evaluates the request against current rights and provider evidence. The gateway enforces the authorised route. Typed adapters preserve native semantics or declare loss. Evaluators and telemetry update evidence without silently rewriting the underlying rule.

This separation creates provider neutrality without pretending providers are interchangeable. Applications describe the job, principal, purpose, data envelope, required capability, quality, safety, service and economic constraints. The gateway selects only among eligible routes. Teams responsible for evaluation, incidents, risk and cost can still see the provider, model version, transformation and fallback. Neutrality means that enterprise intent is not trapped inside one supplier’s configuration, not that every implementation difference disappears.

The strongest rival: direct provider integration

Native SDKs expose capabilities early, avoid translation delay and keep provider behaviour visible to developers. For low-consequence experiments and specialised requirements, direct access may be more honest than forcing an immature feature through a generic schema. Permit it under a bounded principal, data set, destination, spend, user group, duration, evidence requirement and prohibition on production action. The exception becomes a learning route, not a permanent bypass.

Promote a native capability into the shared gateway when its semantics, evaluation and operating demand are understood. Some capabilities should remain typed extensions because standardisation would destroy their value. Others reveal a repeated job that deserves a stable contract. Direct integration fails as an enterprise default when every application rebuilds identity, data controls, retries, redaction, evaluation, metering, evidence and exit while supplier terms and model aliases continue to change.

Provider-Neutral Model Invocation ContractDescribe the job before selecting the route

The contract is an application-to-control-plane statement of need. It is provider neutral because its primary fields express enterprise semantics. It is not provider blind. The returned receipt names the selected supplier and version, and extension fields carry justified native features. Contract versions are stable enough for product integration yet explicit enough to reject false equivalence.

The contract distinguishes requirements from preferences. Region, purpose, entitlement, disqualifying quality failure and prohibited data use are typically gates. Preferred latency, marginal quality, carbon information, capacity balance and price may be optimisation inputs after eligibility. An enterprise can choose a different boundary for a different task, but the distinction must be declared and tested rather than inferred by a router.

Gateway policy chainSeparate intent, decision, enforcement, adaptation and learning

Policy source. Authorised owners publish versioned intent, scope, precedence and non-compensable conditions outside provider consoles. Data, security, privacy, domain and risk owners remain identifiable. The gateway operator can implement the rule but cannot expand business risk acceptance merely by changing configuration. Policy source includes effective dates and rollback so an emergency edit does not become an undocumented permanent standard.

Compilation and tests. A compiler converts human-authorised intent into machine-testable decision and enforcement artefacts. Compilation is not policy creation. Reference cases verify eligible and ineligible principals, purposes, data envelopes, regions, provider terms and versions. A change produces a diff, test evidence, affected routes and deployment receipt. Provider dashboards may receive generated settings, but they are outputs rather than the canonical source.

Decision point. The policy decision point evaluates principal, delegation, tenant, purpose, task, data and current provider evidence. It returns an eligible route set or a denial with reasons safe to disclose. Dynamic conditions such as incident state, capacity or expired evidence can remove a candidate. The decision point is separated logically from request forwarding so policy can be tested and observed independently.

Enforcement and adapters. The gateway verifies the decision, applies destination, quota, redaction, transformation and fallback rules, then invokes a typed provider adapter. The adapter maps only demonstrated equivalents. It preserves extension fields, reports unsupported settings and records transformations. It never silently drops a safety setting or converts tool semantics because a target schema happens to accept similar JSON.

Evaluation and evidence. Task evaluations, regression suites, capacity, latency, incidents, cost and provider-term changes update the evidence available to the decision point. They do not silently change enterprise intent. New evidence may make a model ineligible under an existing threshold, trigger a defined degraded mode or open an approval decision. The gateway receipt links the request, policy version, decision, enforcement, adapter, response, telemetry and actual cost.

Routing orderPass non-compensable gates before optimisation

The eligible set begins with authority. The principal and delegation must permit the task, source and action. Purpose must match the declared use. The data envelope must allow the destination, retention, reuse and logging behaviour of the specific provider route. Applicable legal, contractual and supplier conditions must be current. A route that cannot demonstrate these conditions stops before any cost or latency comparison.

Next apply capability and evaluation gates. The model and adapter combination must support the required modality, tool semantics, structured output, grounding and context behaviour. Task, language and cohort evaluations must meet thresholds, and disqualifying failures remain disqualifying. A self-reported model confidence value is not evidence of route fitness. High-consequence use can require independent validation of the model-use combination plus human authority and operating controls.

Only then may the router optimise among eligible candidates. Learned routing, cascades and workload selection can trade quality, latency, capacity and cost within the declared boundary. Their own predictions need calibration, drift monitoring and evaluation against the enterprise workload. A cascade is not automatically safe because its final stage is strong; earlier responses, escalation thresholds and accumulated latency are part of the evaluated route.

Budget pressure can change scope, queue work, select another eligible model, invoke a validated cascade or abstain. It cannot send restricted data to an ineligible region or choose a model below a disqualifying safety threshold. Likewise, a high-quality provider remains unavailable when its terms, incident state, evidence freshness or data handling fail. The router reports that the eligible set is empty rather than converting policy into a soft weight.

Semantic honestyExpose native value through typed extensions

Lowest-common-denominator abstraction can be as dangerous as direct integration. It conceals that one provider supports a useful grounding mode, constrained decoder, tool protocol, batch service or safety control that another does not. An application may then rely on behaviour the gateway cannot name, test or reproduce. The answer is a typed extension associated with a capability contract, not an unstructured bag of vendor parameters.

An extension declares provider and model scope, semantic meaning, allowed values, interaction with base fields, evaluation suite, data implications, telemetry, version, owner and portable fallback. The adapter rejects it on unsupported routes. The receipt records its use. The product’s dependency register marks whether removal causes equivalent substitution, visible degradation, manual operation or inability to serve. This makes native value available without describing it as universal.

Translation requires evidence of equivalence at the level that matters to the task. Matching field names do not prove that tool selection, streaming events, token limits, refusal, citations, schema adherence or retention behave alike. Regression tests compare observable semantics and declared losses. When behaviour differs but remains acceptable, the contract names the variation. When it changes the business or safety meaning, the route needs a different evaluation or is ineligible.

Portability debt is not inherently bad. A provider-specific capability can create enough value to justify dependence. The governance question is whether the debt is visible, funded, bounded and reversible where required. The gateway should not pretend neutrality by stripping the feature, nor should the product call itself portable because a request can be serialised to another endpoint.

Failure semanticsTreat fallback as a new execution decision

Fallback changes more than an endpoint. A substitute may have different tool-call ordering, structured-output guarantees, context window, image treatment, safety behaviour, residency, retention, contractual terms, price and evidence freshness. The gateway therefore re-evaluates the invocation contract against the candidate at failure time. Cached eligibility has an expiry and is invalidated by incidents, provider updates and policy changes.

An equivalent substitute is allowed only when the required semantics and evaluation evidence remain valid. The receipt names the failure, substitute and evaluation basis. If an optional capability is missing, the route can degrade visibly, such as returning a cited draft without tool execution. The user and downstream workflow receive a machine-readable degraded status. The application must not interpret partial output as the original contract’s success.

When no substitute meets the deadline but delayed execution is acceptable, queue the work under a declared recovery objective and notify the caller. If delay would create harm or the task cannot be safely degraded, abstain. A human or alternate process may take over through a defined hand-off. Sensitive traffic never flows to the cheapest remaining provider simply because an availability target is under pressure.

Fallback testing covers provider outage, region loss, capacity exhaustion, rate limiting, model withdrawal, adapter failure and evidence expiry. It verifies not just response delivery but route selection, disclosure, tool suppression, billing, retry limits, idempotency and receipt completeness. Retry and cache behaviour are economic and semantic parts of the route. Unbounded retries can multiply cost or repeat actions; stale caches can make policy changes ineffective.

Negative controlsRun ten routes through the contract

Operational evidenceMake every decision and transformation observable

The gateway receipt binds request identifier, principal and delegation, tenant, purpose, data classification, contract and policy versions, decision result, eligible set, selection rule, provider and model version, adapter and transformations, extension fields, evaluation basis, fallback status, latency, usage, retries, cache, cost allocation, redaction and response identifier. It points to protected detail rather than copying sensitive prompts into every log.

Receipts support audit, incident investigation, billing, replay and exit, but those uses have different access and retention. A finance view may need consumption and allocation without prompt content. Incident responders may need route and transformation evidence. Evaluators may need approved samples under purpose controls. The gateway should not log prompts or responses by default merely because observability is desirable. Evidence policy applies to the evidence itself.

A policy change starts in the authorised source. The compiler identifies affected provider configurations, routes and reference cases. Tests prove that disallowed combinations now block and allowed combinations still pass. Staged deployment measures decision and enforcement agreement. The receipt records effective time and rollback. If one provider cannot express the rule, that route becomes ineligible or an external enforcement step is added, rather than leaving an inconsistent dashboard setting.

Model aliases require resolution. The gateway records the actual provider model identifier and relevant version or snapshot evidence. Changes under a stable alias trigger defined regression and promotion rules. Where the supplier offers no immutable version, the enterprise can increase canary testing, narrow eligible uses, preserve behavioural fingerprints or demand stronger contract terms. A friendly name is never treated as proof of stable behaviour.

Design before crisisExercise provider exit while the route still works

Exit begins with a dependency register linking products to providers, models, adapter versions, extensions, fine-tunes, prompts, evaluations, stored state, batch jobs, data locations, commercial commitments and evidence. Each dependency states an alternate, degraded mode or accepted inability to operate. Products with provider-specific value can remain, but their owners and executives see the concentration and time needed to move.

A controlled exit revokes new traffic according to consequence, preserves receipts, exports approved artefacts, drains queued work, rotates credentials, disables provider reuse where required and tests substitute routes. Regression covers task quality, safety, tool semantics, structured output, context behaviour, latency, cost and user disclosure. Equivalent API calls do not establish equivalent outcomes. Some products may need redesign rather than substitution.

Exit also tests the gateway. If policy rules live only in the departing provider console, if evidence cannot be exported, or if applications bypassed the contract, the control plane has failed its neutrality claim. Provider adapters should be replaceable, but shared policy, contract and receipt semantics must survive. Historical reconstruction retains the route actually used, not a later generic label.

During an incident, exit can be partial. Stop one model version, data class, jurisdiction, capability or use while preserving eligible traffic elsewhere. The decision owner, risk accepter and affected product owners coordinate the boundary. The gateway operator executes it and verifies propagation. Finance and sourcing handle commitments, while legal and privacy owners apply applicable notification, retention and termination terms. No supplier relationship alone decides operational permissibility.

Service disciplineOperate the gateway without turning it into governance theatre

The gateway is a control-plane service, not the whole governance programme. It cannot decide the legitimacy of a product, own domain outcomes, create evaluation truth or replace supplier contracts. It can consistently enforce declared decisions, expose missing evidence and create a reliable route record. Its service contract names supported capabilities, policy availability, latency overhead, failure behaviour, incident paths, evidence retention, extension process and retirement.

Measure decision latency, enforcement failures, eligible-set emptiness, adapter loss, fallback frequency, receipt completeness, retry inflation, cache effects, route concentration and policy drift. Segment by task and consequence. Aggregate success can hide one sensitive use repeatedly entering degraded mode. A latency objective should include policy and adapter work, but pressure on that objective must not encourage bypass of essential checks.

Capacity controls operate after eligibility and before invocation. Quotas can protect tenants, reserve critical workloads and prevent one cascade from consuming every provider allocation. When capacity is unavailable, the contract chooses queue, degraded route or abstention. Load shedding is visible to applications and users. The gateway does not fabricate success or silently change model class to protect its own SLO.

Ownership follows Paper 092’s decision rights. Policy owners author intent; the platform team operates compilation, decision and enforcement services; domains own intended use and outcomes; evaluators and specialist functions supply independent evidence where required; finance sees economic records; sourcing manages commercial terms; incident authorities can kill bounded routes; assurance accesses evidence without designing daily routing. Separation is preserved in permissions and receipts, not only in documentation.

Architecture review should ask whether an application is relying on an undeclared provider behaviour, whether adapter transforms are tested, whether fallback preserves action semantics, whether logs respect data policy, whether model versions are resolvable, and whether exit has been exercised. Direct experiments should surface demand and evidence. The gateway roadmap follows repeated controlled jobs, not a desire to place every AI request behind one proxy.

Test the seamsAssure the gateway as a changing decision system

Adapter assurance begins with a capability manifest. For every supported provider and model version, it identifies request fields, response events, tool protocol, schema behaviour, context limits, streaming order, retention setting, error mapping and extension support. Contract tests use positive and negative examples, including malformed tool calls, interrupted streams, content filtering, oversized context and rate-limit responses. The tests compare observable meaning, not only whether a payload can be parsed. A mapping that loses a refusal reason, truncates a citation or changes an action identifier fails even when the final JSON remains syntactically valid.

Policy deployment uses paired decision and enforcement tests. Reference requests run against the decision point to establish expected allow, deny, route and degraded results. The same requests pass through a non-production enforcement path to verify that destinations, headers, redaction, quotas and adapters implement those results. Differences block promotion or open a bounded exception with an owner. Canary deployment watches both service measures and policy-result distributions. A drop in denials can indicate business change, but it can also reveal a compiler, identity or context defect and therefore needs investigation.

Evidence minimisation needs its own design. The receipt can store hashes, identifiers, categories, versions and protected references while sensitive request or response content remains in an authorised evidence store. Redaction happens before broad telemetry export, and redaction configuration is versioned as part of enforcement. Sampling never removes mandatory incident or transaction evidence. Access is purpose bound, retention follows the applicable object, and deletion or correction propagates to derived indexes where required. This prevents the observability layer from becoming a second, weakly governed copy of prompts and retrieved data.

Evaluation operations connect model evidence to the contract without granting the gateway authority to invent thresholds. Domain owners supply representative tasks, language and cohort cases. Safety, security and privacy specialists contribute disqualifying scenarios. Independent validators review combinations where consequence requires separation. The platform runs suites consistently and preserves comparable results. When workload, provider, adapter, policy or prompt changes materially, the release record identifies which evidence remains valid and which must be refreshed. The optimiser consumes only approved, dated results and exposes uncertainty instead of treating a sparse benchmark as a universal route truth.

Periodic recovery exercises complete the assurance cycle. Teams disable a candidate route, expire its evidence, introduce an incompatible response event and simulate loss of a region. They verify that policy evaluation removes the route, downstream applications recognise degraded status, retries remain bounded and incident owners can reconstruct the decision. The exercise records manual work and stranded capabilities as exit debt. Findings change contracts, adapters or product design through normal ownership rather than an emergency override hidden in gateway configuration.

Interactive instrumentModel Gateway Policy Router

Set eligibility, capability, evidence, fallback and exit conditions. The Router returns exactly one of five routes with the selected contract, policy evidence, fallback condition and portability debt. It never returns a provider or gateway score.

Exactly five outcomesChoose a governed route or a deliberate stop

Evidence boundaryArchitecture and research guide the chain, not a universal ranking

Routing and cascade research demonstrates methods for trading quality and cost across workloads. It does not supply a permanent provider ranking or permission to optimise across failed enterprise constraints. Zero-trust, policy-language, gateway and external-authorisation sources inform separation of decision and enforcement. Open Policy Agent and Cedar are implementation options, not required products.

OpenAPI and OpenTelemetry support machine-readable interfaces and interoperable operational evidence. API compatibility is not semantic equivalence, and telemetry is not complete governance evidence. Production-ML research supports attention to glue code, dependencies and change. ISO and NIST inform lifecycle, context, third-party and monitoring disciplines without turning the gateway into the risk owner.

OWASP informs the sensitive-information disclosure boundary across prompts, context, applications and outputs. The EU AI Act record is applied only where the relevant system, actor, provision, jurisdiction and timing engage. The Provider-Neutral Model Invocation Contract, gateway policy chain, typed extension and Model Gateway Policy Router are this article’s synthesis.

The gateway succeeds when applications can state stable enterprise intent, providers can retain meaningful differences, and every decision remains inspectable. Provider choice may vary request by request. Policy authority, evidence custody and accountability must remain within the enterprise’s governed control plane.