Executive briefBuy roles, not model logos

A procurement workshop can fill a wall with model logos, benchmark columns and supplier discounts without identifying one production role that the enterprise needs. Product teams nominate favourites. Architecture asks for portability. Finance wants leverage across vendors. Risk asks for diversification. The result looks strategic because it contains options, yet each option creates prompts, adapters, tests, monitoring, capacity, incident paths, skills and contractual dependencies that someone must operate.

A model portfolio is not the set of endpoints that employees can call. It is the smallest governed set of model roles required to serve defined classes of work. Each role has users, tasks, exclusions, consequence, data conditions, evaluation, service expectations, economics, owner and exit. The portfolio admits models into those roles. Paper 093’s gateway can then choose among eligible routes for an individual invocation. Runtime routing cannot repair a portfolio that admitted the wrong candidates.

Start with one well-governed default. Add another model only when a named capability, control, economy, resilience or learning option delivers incremental value above its lifecycle complexity. Remove a model when demand, evidence, cost, supplier terms or operating capacity no longer supports the role. Diversity is a means, not a target. Provider contracts, model lineages and infrastructure dependencies determine whether apparent choice creates actual strategic options.

The strongest rival: a single frontier model

A single capable model reduces integration, evaluation, monitoring, skills, support and purchasing complexity. Its rapid improvement may erase specialist advantages before an enterprise recovers the cost of adding them. If it meets task and cohort thresholds, complies with data and policy constraints, operates within cost and service boundaries, and concentration remains within appetite, standardisation can be the stronger strategy.

That choice still needs tested exit and change detection. A provider alias may move, a region may fail, terms may change or a critical capability may be withdrawn. The role charter records the alternative and degraded mode even when no second production model is active. Add diversity when evidence shows a material gap. Refuse symbolic diversity that multiplies interfaces without changing underlying dependencies or outcomes.

Model Role PortfolioUse seven roles without forcing seven models

Roles may share one model, and one model may hold several charters. The mapping remains explicit because evidence, data, service and exit can differ by role. The enterprise need not populate every category. The purpose of the taxonomy is to make a reason for admission testable, not to create seven purchasing targets.

Model Role CharterBind task evidence to operation and exit

The charter starts with tasks, users, affected decisions and exclusions. It names consequence, autonomy and human or deterministic controls around the model. Capability fields cover modalities, languages, context, structured output, tools and deployment. A model is not admitted because it is generally capable; it is admitted because its observable behaviour fits this role and its failure can be managed.

Data fields specify classification, privacy, security, jurisdiction, retention, licence and supplier conditions. Evaluation names production scenarios, cohorts, adversarial cases, calibration or uncertainty treatment and disqualifying failures. The charter distinguishes thresholds that determine eligibility from measures used later for allocation. A benchmark winner that fails the organisation’s critical scenario never reaches routing optimisation.

Operations include latency, throughput, availability, capacity, recovery, fallback, change detection, monitoring, incident and support. Economics cover inference, infrastructure, retries, review, integration, evaluation, skills and idle capacity at workflow level. Dependency fields name provider, version, base-model lineage, region, data, licence, accelerator and key skills. The owner accepts the burden of keeping those facts current.

Portability and exit state replacement, degraded operation, export, dependency migration and retirement. The role also has a demand hypothesis and renewal date. A candidate may create option value during uncertainty, but that option has a cost and expiry. Every admission or renewal produces a portfolio receipt that records the alternative baseline, incremental benefit, incremental burden, concentration effect and decision owner.

Portfolio lifecycleMake admission reversible and renewal demanding

Demand hypothesis. Begin with a recurring unmet task or strategic learning question. Identify the current default and non-model alternative. Estimate volume, consequence, affected cohorts and why the existing route is inadequate. A vendor demonstration or public release is not demand. The hypothesis must be falsifiable through use and outcome evidence.

Role definition and evidence plan. Draft the charter before choosing a candidate. Define tests, data, service, economic and concentration evidence needed for admission. Candidate screening removes options that fail licences, regions, deployment, capability or supplier constraints. Controlled evaluation compares the surviving candidates with the actual baseline on production-like scenarios, including failures and human review.

Admission and allocation. An accountable portfolio owner admits a candidate into a bounded role, with model and provider version, exclusions, owner, capacity, fallback and renewal date. The gateway receives the eligible role mapping. Allocation policy can route individual requests but cannot expand the charter. Experimental evidence does not silently become production permission.

Monitoring and renewal. Observe demand, task performance, cohort results, calibration, failures, review burden, latency, capacity, incidents, cost and dependency change. Provider alias changes or new base-model evidence can pause renewal. Compare the role again with the current default, because improvement elsewhere may remove its advantage. Renewal restates incremental value and burden rather than rolling forward by inertia.

Replacement and retirement. When the role no longer justifies its estate cost, stop new dependencies, select the destination, migrate prompts and adapters, run regression, preserve decision evidence and remove credentials, capacity and support. Retirement is a portfolio outcome, not a failed experiment. It keeps complexity aligned with current need.

Evaluation boundaryUse scenario evidence rather than a horse race

Public evaluations reveal heterogeneous capabilities and can help screen candidates or design tests. They do not reproduce the enterprise task, prompt, context, language, cohort, tool, safety controls or review process. Even a broad benchmark is one bounded measurement. A portfolio decision requires scenario-specific evidence against the actual alternative, with uncertainty and failure distributions visible.

Define disqualifying failures before looking at candidate averages. These may include prohibited disclosure, invalid structured action, severe cohort harm, unsupported language, uncalibrated abstention or inability to meet a deterministic constraint. A high mean cannot compensate for them. Where a verifier or human step mitigates the failure, evaluate the combined workflow and count its cost, latency and residual error.

Specialist admission is local. A language model may outperform the default for a named dialect, document class or domain vocabulary while performing worse elsewhere. The charter limits eligibility to the proved cohort and describes evidence freshness. Expansion needs new scenarios. This keeps a genuine specialist advantage from becoming organisation-wide model shopping.

Evaluation also covers change. Resolve actual model versions, prompts, adapters, retrieval, tool definitions and policies. Regression suites include historical production failures and adversarial cases. If the supplier changes an alias, the candidate does not inherit admission automatically. Canary evidence and rollback accompany promotion. Monitoring tests whether the scenario distribution or user behaviour has moved beyond the evaluated envelope.

Lifecycle economicsCount the workflow, not the token tariff

An economy role earns admission when full workflow cost falls while required outcomes remain within thresholds. Include inference, retries, cache misses, human review, correction, latency, integration, evaluation, monitoring, support and incident burden. For a local model include accelerators, orchestration, security, energy, utilisation, patching, capacity reservation and specialist skills. Open weights can change licence and deployment options; they do not remove operating cost.

Compare incremental burden with the current default. A small model may have a lower unit price but require a custom prompt stack, larger review sample and more failure handling. A specialist may save expensive expert time despite higher inference cost. A private deployment may justify idle capacity because the alternative cannot receive the data. The receipt states assumptions and measured evidence instead of claiming a universal cheaper model.

Procurement discounts can distort portfolio design. A committed-spend rate may make one provider appear economical while increasing concentration and reducing exit flexibility. Allocate commercial commitments transparently and test marginal as well as fully loaded economics. Finance, platform and product owners should see who benefits, which fixed cost remains shared and what happens when volume misses the forecast.

Option value belongs in the decision but has an expiry. An experimental model can teach the organisation about a new modality or interface before demand is certain. Budget the learning question, restricted environment and evidence. If the question is answered negatively or remains unowned, retire the option. Endless sandbox access is not low-cost experimentation when evaluation and security duties persist.

Dependency truthCount shared foundations, not endpoints

Two provider endpoints may expose the same base model, run in the same cloud region, depend on the same accelerator supply or inherit the same training lineage. They can fail together. Conversely, two versions from one provider may behave differently enough for a bounded task but still share contractual and infrastructure concentration. The portfolio map traces provider, base model, region, data, licence, hardware and critical skills separately.

Concentration is contextual. A shared dependency matters more when it supports high-consequence, high-volume or hard-to-replace roles. Record the affected workload, recovery time, substitute evidence and degraded operation. Candidate count is not resilience. A second model improves resilience only when it can take the relevant workload under current data, capacity, evaluation and operational conditions.

Open source is a licence and distribution characteristic, not an automatic private, safe, economical or portable role. The organisation may control deployment while depending on one checkpoint lineage, hardware stack, serving framework and scarce skill group. Licence terms may constrain use or redistribution. Security updates and model provenance remain obligations. The dependency map makes those facts comparable with hosted-provider dependencies without forcing them into one score.

Review correlations in verifiers and cascades. Different endpoints can share data, architecture, prompt biases and failure modes. Empirical joint tests matter more than branding. If a critic agrees with the primary on precisely the disqualifying failure, it adds cost and false confidence. If it detects a defined class and abstains when uncertain, it can add evidence even when it is smaller or less capable overall.

Combined evidenceUse cascades and critics for a defined job

A cascade sends work to a lower-cost or faster model and escalates when a decision rule indicates difficulty. A verifier examines an output or intermediate property. Both can outperform a single route for a workload, but their evidence belongs to the combined system. Evaluate the first-stage errors, escalation calibration, verifier detection, abstention, latency, accumulated cost and final workflow outcome.

The verifier charter names the failure class it is expected to catch. It may check structured completeness, source support, policy classification or consistency with a deterministic rule. It should not be described as an independent judge merely because it is another model. Test correlated errors across production scenarios and shifts. Where the verifier cannot establish the property, abstention must lead to human review, another control or failure.

Cascades require capacity and fallback planning for every stage. Under load, escalation can concentrate traffic on the expensive model precisely when it is unavailable. Retry and review inflation may erase savings. The gateway enforces only candidates admitted to each stage and records the route. Portfolio owners renew the combination when model versions, thresholds or task mix change.

Diversity can add strategic evidence even without active-active substitution. An independent model may challenge a narrow output sample or support periodic evaluation rather than serve production requests. Its value should be measured against the failure it detects and the action taken. Maintaining it without sufficient use, skill or decision consequence is another form of shelfware.

Negative controlsTest ten portfolio decisions

Portfolio receiptRenew roles only against today’s alternative

The receipt names the role, model, provider and resolved version; tasks and exclusions; evaluation suite and date; production demand; workflow economics; service and incident evidence; dependency concentration; prompt and adapter burden; owner; alternative baseline; incremental value; renewal date; replacement and exit. It explains dissent and missing evidence. A receipt is not a scorecard; it is a traceable admission claim.

Renewal compares with the current default and non-model process, not the alternative available when the role first entered. A generalist may improve, a specialist may stagnate, volume may decline or human review may grow. Supplier terms and infrastructure capacity can change the full cost. If advantage remains but evidence is stale, renew conditionally only where consequence and existing controls permit, with a bounded test plan.

Experimental roles expire automatically unless a decision renews the learning question or opens production admission. Low utilisation is not always failure because strategic options can be valuable, but the value, owner and horizon must remain explicit. An orphaned sandbox still consumes access, data, evaluation and incident attention. Remove it when nobody can state the next decision its evidence will inform.

Retirement stops new dependencies first. The product map identifies prompts, adapters, stored state, evaluations, provider data, contracts and affected workflows. Replacement tests the actual role rather than generic API compatibility. Evidence remains accessible for past decisions. Capacity, credentials, licences and support routes close. A model that remains callable after declared retirement has not left the portfolio.

Worked decision systemRun a portfolio review around roles and receipts

Begin with the demand register

The portfolio review starts with jobs that the current estate cannot serve adequately, not a list of newly available models. Each proposal names the requesting products, forecast volume, affected decision, consequence, data conditions and present workaround. The owner states whether the gap concerns capability, control, economics, resilience or learning. Product duplication is consolidated into one demand hypothesis where the job and constraints match. Requests with no owner, no recurring decision or no plausible evidence plan remain observations rather than procurement actions. This prevents supplier announcements from setting the agenda while still giving emerging capability a route into controlled discovery.

Re-establish the default baseline

Every candidate comparison uses the current configured default, including its prompt, retrieval, tools, policy and human review. A stale result from an earlier model release is not a valid baseline. The team reruns representative scenarios and reports distributions by language, cohort and failure class. It measures service behaviour, correction, review and full workflow cost. This baseline can reveal that the supposed gap has already closed or that a process change beats both models. It can also reveal deterioration that requires default renewal, independent of the new proposal. The resulting evidence makes rejection of an additional role a positive portfolio decision rather than passive resistance.

Admit an economy role without exporting hidden work

Consider a stable invoice-field extraction journey with high volume and a deterministic validation layer. A smaller candidate passes the defined document classes and languages, stays within disqualifying error thresholds and reduces total processing cost. The evaluation counts retries, exceptions sent to staff, latency and corrections. The role charter excludes unseen layouts and consequential payment release. The gateway sends only eligible documents, records version and route, and falls back to the default or manual queue on uncertainty. Monitoring watches document-mix drift and reviewer load. Savings fund the added evaluation and support rather than assuming a lower per-request price is captured automatically.

Justify a private role through an infeasible external route

A sensitive low-latency workflow may require local operation because approved external destinations cannot meet the data or timing contract. The review compares the whole local service with the feasible alternative, not with a public endpoint price that the data cannot use. Evidence covers hardware availability, serving software, model licence, security patching, isolation, monitoring, specialist cover, energy, peak capacity, recovery and update cadence. The charter limits the local model to the named workflow. If policy or supplier conditions later permit an adequate managed route, renewal compares again. Local control is an operational responsibility with an exit decision, not a permanent badge of sovereignty.

Separate specialist advantage from general preference

A model for a particular language or regulated document type may show a material advantage on the target cohort. The admission receipt states the number and nature of scenarios qualitatively where no stable numerical claim is needed, the human adjudication process, known blind spots and the minimum advantage required to justify another stack. It also lists tasks where the default remains preferred. Product interfaces select the role by declared task and data rather than allowing users to browse models. If demand expands, the new cohorts enter evaluation before eligibility expands. This preserves the specialist value without turning a bounded result into a global status contest.

Test verifier independence empirically

For a consequential structured response, a critic may check source support and prohibited action before execution. The review samples joint outcomes: primary correct and critic correct, primary wrong and critic detects, primary wrong and critic agrees, and critic abstains. It inspects whether shared training lineage, retrieval context or prompts cause correlated error. A deterministic validator handles schema properties that do not require a model. The critic enters only if its detected failure leads to a useful intervention and its missed failures remain within the declared boundary. The receipt includes added latency, cost, review and operational coupling. A second provider name alone contributes no evidence.

Connect admission to gateway allocation

After admission, the portfolio publishes a role identifier, eligible versions, contract requirements, extensions, evaluation date and expiry to the gateway. Applications request the role or task contract, not a favourite endpoint. The gateway applies principal, purpose, data and current policy before choosing among admitted candidates. Runtime telemetry returns demand, route, failure, cost and fallback evidence to the portfolio. It cannot change the role’s exclusions or invent a new production use. If allocation repeatedly finds no eligible candidate, the signal opens a portfolio review of demand, capacity or contract design rather than silently widening the candidate set.

Negotiate capacity without converting discounts into strategy

Finance and sourcing bring committed spend, price changes, quota, region, termination and service terms into the review. Platform teams add measured utilisation, peak concurrency, queueing and support. Product owners state which workloads can move or degrade. The portfolio owner can reserve critical capacity, allocate flexible work and decide whether redundancy is worth its fixed burden. A discount affects economics only after eligibility and likely consumption. Excess commitment is not a reason to send inappropriate work to a provider. Conversely, a strategically valuable alternate may deserve paid standby capacity when recovery evidence shows it can carry the specified workload.

Report decisions and removals, not model count

Executive reporting shows roles, covered demand, excluded uses, current evidence, concentration, unresolved gaps, incident history, incremental burden, upcoming renewals and retirement progress. It distinguishes active production roles from experimental options and registered alternates. A rising model count is neither progress nor resilience. A smaller portfolio can signal that defaults improved and obsolete specialists were removed. A larger one can be justified when distinct demand and operating capacity grew. The review records who decided, what baseline was used and when evidence expires. This lets management challenge a portfolio claim without reducing incomparable models and components to a single composite indicator.

Interactive instrumentModel Portfolio Allocator

Set task evidence, role value, operating burden, concentration, experimentation and lifecycle conditions. The Allocator returns exactly one of five portfolio routes with a Role Charter, admission evidence, concentration effect and exit condition. It does not calculate a universal portfolio score.

Exactly five outcomesChoose a role decision, not a portfolio score

Evidence boundaryResearch informs portfolio evidence, not a universal winner

Holistic and broad evaluation research demonstrates heterogeneous performance and the limits of a single measure. MMLU is one bounded task benchmark, not an enterprise admission rule. Model Cards inform intended-use, evaluation, limitation and version documentation. Routing and cascade research applies only after the portfolio has declared eligible roles.

Green AI supports reporting computation and efficiency alongside quality. Production-ML research supports tests, dependencies, change and maintenance burden. The AI Index can provide current industry context where a specific figure is checked; no report statistic is used here to rank candidates. ISO, NIST and OECD inform lifecycle, monitoring, third-party and accountability boundaries.

GDPR and the EU AI Act records apply according to processing, role, system, provision, jurisdiction and timing. Open weights or local deployment do not remove those questions. The seven-role Model Role Portfolio, Model Role Charter, concentration dependency map, portfolio receipt and Model Portfolio Allocator are this article’s synthesis.

A multi-model strategy is credible when every candidate performs a proved role that the existing portfolio cannot serve as well after burden is counted. Models enter for durable task evidence, bounded control, resilient option or economical operation. They leave when that reason no longer survives renewal.