TLDR

1. Do not add ordinal ratings as if the distance between categories were measured, equal and comparable across criteria.
2. Apply hard gates for population, lawful authority, ownership, value path and safety before any portfolio comparison.
3. Show evidence grades and unknowns beside each dimension; absence of evidence is not a neutral midpoint.
4. Remove dominated candidates, test the assumptions that can switch a decision and preserve incomparable survivors for explicit judgement.
5. Simple scores remain useful for repeated, calibrated environments with reliable inputs, stable definitions, overrides and outcome back-testing.

Candidate A receives 5 for value, 5 for feasibility and 1 for lawful authority. Candidate B receives 3, 3 and 3. Equal weights produce a tie. What exactly has tied?

Nothing decision-ready. If lawful authority is a condition for action, A fails regardless of its upside. If the figures are ordinal labels, adding them assumes equal spacing within each scale and commensurability across unlike ideas. A move from 1 to 2 in authority cannot be treated as the same gain as a move from 4 to 5 in value. The arithmetic is valid; its interpretation is not.

The weighted score is attractive because it converts a crowded opportunity list into one column. Teams can sort, colour and present a top ten. The decimal creates an impression that trade-offs were resolved objectively. Yet most early AI assessments combine estimates, judgements and unknowns from different owners, evidence dates and causal chains. A crisp total often reflects the scoring architecture more than the opportunity.

The alternative is a staged evidence profile. First apply non-compensable gates. Then grade evidence for each relevant dimension with source, date and expiry. Remove candidates that another candidate matches or exceeds on every material dimension. Preserve unknowns and ranges. Run sensitivity on the few assumptions that can change the next action. Record judgement where rights, values or incomparable consequences remain.

This is not an argument against numbers. Use numbers when the quantity has a unit, population, observation period and causal meaning: queue days, review minutes, loss distribution, comparison effect, model error and lifecycle cost. Refuse to convert unlike ordinal labels into a false common currency. Precision should follow measurement rather than substitute for it.

Figure 1. The precision-theatre machine prints a decimal from qualitative cards

Core mechanism

01 · Manufactured orderThe total conceals five different assumptions

A conventional weighted AI opportunity score manufactures a total order by treating unlike ordinal categories, uneven evidence and negotiable weights as if they formed a stable common scale. Five assumptions make the ranking look inevitable. Each should be defended before addition is allowed.

First, the category labels are assumed to have equal intervals. On a feasibility scale, “prototype demonstrated” may be much closer to “bounded production test” than to “repeatable operations”, or the reverse. Without calibration against outcomes, a 4 is a rank category, not four units of feasibility. Arithmetic on labels can be convenient, but its numerical distance has not been observed.

Second, dimensions are assumed to compensate. A high value estimate can wash out weak safety, missing authority or unlawful purpose. Some criteria are constraints, not preferences. The European Union AI Act defines prohibited practices and obligations for high-risk systems through legal classifications, conditions and duties, not through portfolio points.9 Other jurisdictions and internal policies create their own non-compensable boundaries.

Third, evidence quality is assumed to be equal. One candidate may have a measured baseline and controlled comparison; another may have a sponsor estimate and vendor benchmark. Entering both as 4 for value erases the distinction between claim and observation. Confidence should be shown beside the assessment rather than folded into a lower score whose meaning is ambiguous.

Fourth, unknowns are treated as midpoints. Teams often enter 3 when data quality, integration burden or outcome effect is unknown because a blank cell breaks the spreadsheet. This makes ignorance look moderate and can reward the least investigated candidate. Unknown should remain visible, carry an expiry and trigger a discriminating evidence action.

Fifth, weights are treated as fixed preferences. In practice they are chosen by a group, inherited from a template or adjusted after preferred candidates appear. Small changes can reverse close rankings. A switch is not a defect if it reveals a genuine value trade-off. It is a defect when the reported decimal hides how little change is required.

HM Treasury's 2026 Green Book distinguishes expert-facilitated multi-criteria decision analysis from simple multi-criteria weighting and scoring. It recommends against the latter when it lacks an objective basis and reduces transparency.1 The supplementary MCDA guidance is more demanding than a spreadsheet: objectives, criteria, performance measurement, swing weighting, consistency and sensitivity must be established.2

Figure 2. Upside cannot wash through a veto gate

Compensation trap

Judgement biases can enter before arithmetic. Tversky and Kahneman describe how representativeness, availability and anchoring systematically affect judgement under uncertainty.3 A memorable demo can anchor feasibility; a recent incident can inflate perceived risk; a fashionable benchmark can stand in for local fit. The spreadsheet standardises the output format but does not remove bias from the inputs.

Optimism compounds the problem. The Green Book asks appraisers to adjust benefits, costs and duration using historical forecast error and warns that some quantitative methods can introduce spurious accuracy.1 An organisation with project-outcome history can calibrate evidence grades and ranges. One without it should widen uncertainty, not invent confidence.

Scales also drift between assessors. One person gives 4 when a vendor has demonstrated a prototype; another reserves 4 for monitored production. Written anchors help, but examples and calibration sessions are stronger. Ask assessors to rate the same historical candidates before the live portfolio. Compare their reasons, then relate the categories to outcomes where possible. Persistent variation should appear as a distribution, not disappear inside an average labelled “committee score”.

Figure 3. Unknown evidence is a different state from middling performance

Epistemic hygiene

02 · Staged comparisonGate, profile, filter, stress, judge

Stage one contains hard gates. Define the decision population, legal and policy permissibility, accountable operating owner, plausible value path and safety boundary. A gate outcome needs a reason, source, owner and review date. Failure can mean reject, redesign or bounded non-operational research. It does not mean “add a low governance score”.

Stage two creates the evidence profile. For each dimension, record the current assessment and an evidence grade. A grade can distinguish no evidence, assertion, observed baseline, bounded test and replicated operating evidence. The categories should be local, versioned and supported by examples. Attach sources and expiry because capability, regulation, process and cost change.

Evidence acquisition has a cost, so not every blank deserves research. Rank unknowns by decision relevance: could a plausible result change the gate, dominance relation or next investment? If no, document the unknown and move on. If yes, design the smallest observation that discriminates between actions. A short workflow sample, rights review, data audit or residual-cost measurement can be more valuable than another model benchmark because it addresses the actual switching condition.

Aligned bars or a matrix show the profile without deceptive radial area. Keep each criterion on its own row and place candidates in columns. Mark unknowns in a distinct symbol. Include units where available and a short reason where judgement is qualitative. The visual should make weak evidence impossible to overlook.

Stage three applies dominance. Candidate A strictly dominates B when A is at least as good on every material dimension and better on at least one, under the same evidence basis and gates. Practical dominance can allow immaterial differences, but the tolerance must be declared before comparison. Dominated candidates leave the shortlist unless they provide distinctive information or portfolio diversity.

Check evidence asymmetry before declaring dominance. A candidate with three mature measurements should not automatically dominate one with three unknowns; the relation is unresolved. Nor should a broad category hide a material subpopulation in which the direction reverses. Document which dimensions are treated as material, whether higher is always better and which differences count as practically equivalent. This makes the filter reproducible without pretending it is value-free.

Dominance will often leave several incomparable candidates. One may have stronger value evidence and heavier change burden; another may be safer and less informative. That is useful. The method narrows the set without pretending that all trade-offs have been resolved. The surviving profile becomes the agenda for accountable judgement.

Stage four tests sensitivity and switching conditions. Identify the few uncertain assumptions that can alter the next action: addressable population, effect range, review burden, owner capacity or permissible boundary. Ask how much each must change before preference switches. The Green Book recommends sensitivity and switching-value analysis because decision makers need to know which assumptions matter.1

Do not vary every field mechanically. Focus on uncertain and consequential assumptions, preserve correlations where they matter and state the plausible range. A tornado of decorative sensitivity bars can reproduce precision theatre. The useful output is a sentence such as: “The complaint tool advances unless median review exceeds six minutes per case, at which point integration-first redesign becomes preferable.” That condition tells the team what to measure and when to reconsider.

Stage five records judgement. Name who decided, the trade-off accepted, dissent, expiry, evidence plan and condition for reconsideration. Where consequences or rights are incomparable, a score cannot discharge responsibility. A transparent decision can acknowledge that one group bears a burden another values differently.

Figure 4. An aligned evidence profile preserves shape and unknowns

Profile, not average

Figure 5. Dominance removes weak candidates without forcing a total order

Frontier

03 · Worked comparisonsService and commercial credit expose different failures

Service-copilot portfolio

Consider a popular general assistant, a complaint-evidence tool and a narrow vulnerability-triage service. A conventional sheet scores value, feasibility, adoption, data and risk. The assistant receives high adoption and feasibility because users already know the interface. The complaint tool has a clear outcome and owner but needs integration. Vulnerability triage has material service value and demanding rights and safety requirements.

If adoption and feasibility weights rise slightly, the general assistant leads. If end-to-end outcome and evidence weights rise, the complaint tool leads. If consequence protection becomes a hard condition, vulnerability triage cannot be ranked until its boundary, review and appeal are defined. The reversal shows that the spreadsheet has not found an objective winner; it has encoded a preference about which uncertainty to tolerate.

Apply the staged method. All three need a defined population and owner. The general assistant lacks a causal value path if nominal drafting time does not change completed service. It may receive an evidence-collection action rather than a delivery rank. The complaint tool has stronger value and decision evidence but unknown integration burden. Vulnerability triage passes only with a lawful purpose, affected-population evidence and specialist capacity.

Dominance may remove the general assistant if the complaint tool matches its technical headroom and exceeds it on outcome clarity, ownership and evidence while costs are comparable. If not, both survive for different reasons. A small general-assistant probe can produce reusable capability information while the complaint tool enters value-chain design. One total rank would obscure these different contracts.

A disciplined service comparison should also preserve portfolio interactions. The general assistant and complaint tool may rely on the same retrieval, evaluation or change capability, so costs are not independent. Conversely, launching both may compete for the same review and integration capacity. Record complements and conflicts after candidate-level profiling. Do not inflate each candidate with the full shared benefit or ignore the congestion created when several “top-ranked” projects enter delivery together.

Commercial-credit candidates

Compare covenant extraction, missing-evidence detection and automated approval. Extraction can reduce preparation burden if candidate values, source citations, period comparability and exception review are controlled. Missing-evidence detection may prevent late restarts with a clear early action and short feedback. Automated approval claims larger financial upside but crosses consequential authority, fairness, policy and long-lag outcome boundaries.

A weighted total can let automated approval compensate for low governability with high value. The gate rejects that logic. If lawful authority, safety or accountable decision ownership is absent, the candidate cannot enter a scale comparison. It can be redesigned as evidence assembly or bounded research without operational reliance. Upside does not purchase permission.

Gate evidence should specify the exact proposed action. “Automated approval” may fail while “assemble cited evidence for officer review” passes. This is not a scoring adjustment; it is a different candidate with a different population, authority and downside. Preserve the rejected definition so later teams cannot quietly restore it under a softer label. Re-run the gate whenever the action boundary, affected population or evidence source changes.

Between extraction and missing-evidence detection, neither may dominate. Extraction may have greater measurable time headroom; completeness may have a clearer workflow action and lower reversibility burden. Sensitivity should focus on the assumptions that change the next contract: prevalence of late missing evidence, residual review minutes, integration cost and analyst capacity. A switching statement is more informative than a two-decimal rank.

Figure 6. A small weight change flips a score-led winner

Sensitivity seesaw

The second thought experiment asks ten experts for scores. Five see high value and no feasibility; five see low value and high feasibility. The average is identical across dimensions. Has consensus been achieved? No. The mean has erased a bimodal disagreement about mechanism or evidence. Ask each assessor for a reason and source, then show the distribution.

Disagreement can diagnose different assumed populations, time horizons, technical boundaries or value owners. It can also reveal legitimate value conflict. A facilitated review should resolve factual differences where evidence can discriminate and preserve normative differences where it cannot. Averaging both into 3 silences the most useful information in the room.

Figure 7. Identical averages can conceal opposite expert distributions

Disagreement anatomy

04 · Strongest rivalSimple rules can beat unaided judgement

The strongest rival deserves full acceptance. Dawes showed the robustness of simple, even improperly weighted linear models in repeated decision settings.4 Meehl's earlier work compared clinical and statistical prediction and documented the value of consistent actuarial rules.5 A transparent score can reduce inconsistency, expose criteria and make a large intake manageable.

The conditions determine whether the rival applies. Inputs need stable definitions, reliable measurement and repeated outcomes. Criteria should be commensurable enough for aggregation or connected through a defensible utility model. The environment must permit calibration and back-testing. Overrides must remain explicit. Weights should be fixed before outcomes or candidate identities invite manipulation.

A mature repeated intake decision can meet these conditions. Historical cases can calibrate categories against outcomes, compare a linear rule with human triage, monitor drift and test whether overrides improve or worsen results. The score becomes a model with a documented population and error rate, not a one-off meeting device.

Even there, evaluate the rule as a deployed decision system. Measure who is routed, who is delayed, how overrides distribute, whether labels remain stable and what happens after triage. Back-test on time-separated data and keep a prospective monitoring period. Consistency is valuable only when the criterion it applies consistently remains connected to the intended outcome and does not encode a prohibited or harmful shortcut.

Early AI portfolio choices rarely qualify. Candidate definitions move, technical capabilities change, value paths differ, governance constraints interact and outcomes are sparse. OECD evidence identifies heterogeneous constraints across return, data, skills, regulation and process change.13 A universal scoring template suppresses that local variation.

Keeney and Raiffa show that multi-objective aggregation requires careful value modelling and independence conditions.6 When those conditions can be defended, use the resulting method. When they cannot, a staged profile is more honest. The choice is not mathematics versus intuition; it is calibrated modelling versus arithmetic theatre.

05 · Executable instrumentCompare two candidates without averaging away the evidence

The Opportunity Evidence-Profile Comparator applies gates, profiles, dominance and sensitivity to two candidates. Each input combines assessment with evidence maturity. “Unknown” is a separate value. The sensitivity controls affect only the diagnostic comparison after gates and dominance; they do not create the primary verdict.

Opportunity Evidence-Profile Comparator

Compare two candidates at the same portfolio gate. Use the profile to choose a next action, not to manufacture an ROI rank.

Candidate A

Hard gatesFails authority, population or safety gatePasses defined gates

Value-pool evidenceUnknownWeak assertionObserved baselineBounded causal evidenceReplicated operating evidence

Decision and population clarityUnknownVague activityDecision namedPopulation and action definedVersioned population and owner

Technical headroomUnknownNo demonstrated headroomPrototype evidenceTarget-case evaluationOperating evidence

Workflow and change readinessUnknownNo change routeOwner interestBounded operating routeCapacity and change secured

Data and rights readinessUnknownMaterial gapPartial routeGoverned route with gapsTested representative route

Operating ownerUnknownSponsor onlyNamed without authorityCan change operationsOwns action and capture

Downside and reversibilityUnknownUnboundedNamedRecovery definedSevere cases tested

Full cost coverageUnknownBuild onlyMajor cost classesResidual workflow estimatedLifecycle range tested

Uncertainty and expiryUnknownNo expiryReview plannedUnknowns and dates explicitSwitching evidence monitored

Information yieldUnknownNo learning contractLocal learningLater decision namedReusable boundary evidence

Candidate B

Hard gatesFails authority, population or safety gatePasses defined gates

Value-pool evidenceUnknownWeak assertionObserved baselineBounded causal evidenceReplicated operating evidence

Decision and population clarityUnknownVague activityDecision namedPopulation and action definedVersioned population and owner

Technical headroomUnknownNo demonstrated headroomPrototype evidenceTarget-case evaluationOperating evidence

Workflow and change readinessUnknownNo change routeOwner interestBounded operating routeCapacity and change secured

Data and rights readinessUnknownMaterial gapPartial routeGoverned route with gapsTested representative route

Operating ownerUnknownSponsor onlyNamed without authorityCan change operationsOwns action and capture

Downside and reversibilityUnknownUnboundedNamedRecovery definedSevere cases tested

Full cost coverageUnknownBuild onlyMajor cost classesResidual workflow estimatedLifecycle range tested

Uncertainty and expiryUnknownNo expiryReview plannedUnknowns and dates explicitSwitching evidence monitored

Information yieldUnknownNo learning contractLocal learningLater decision namedReusable boundary evidence

Sensitivity only: value emphasisStandardHigherVery high

Sensitivity only: readiness emphasisStandardHigherVery high

Compare evidenceHard-gate failureDominated caseSensitivity reversalEvidence tie

Assumptions, gates and limits

• Comparable gate: candidates address portfolio choices close enough in objective and stage for profile comparison.
• Hard gate: failed population, authority or safety conditions produce rejection or redesign before comparison.
• Dominance: one candidate must equal or exceed the other on every known profile dimension and be better on at least one; unknowns prevent strict dominance.
• Evidence action: important unknowns or an evidence tie route to a discriminating observation with an owner and expiry.
• Sensitivity: emphasis controls reveal fragility; they do not convert ordinal categories into measured units or produce the primary verdict.
• Switching condition: the result names the assumption or evidence dimension most able to change the next action.
• Limits: the Comparator does not calculate ROI, define legal status, set risk tolerance, prove causal effect or replace accountable judgement.

Figure 8. The comparison board makes gates, unknowns and switches visible

Decision architecture

The named outcomes preserve action. “Reject at gate” stops compensation and can invite a redesigned boundary. “Collect discriminating evidence” identifies the observation most likely to change the next choice. “Advance the dominant candidate” applies only when the comparison is unambiguous on known dimensions. “Decision requires explicit trade-off” keeps accountable judgement visible.

NIST's AI RMF is compatible with this discipline. It treats risk tolerance as contextual, notes that actions are not a simple checklist and asks organisations to document risks that cannot be measured and residual risk against tolerance.78 A portfolio score should not pretend the framework supplies universal weights.

06 · Negative controls and boundarySome trade-offs must remain decisions

The first negative control is a prohibited or ownerless use case with high value and feasibility marks. A low governance rating is averaged away. The correct result is gate failure. The candidate can be rejected, narrowed or kept in non-operational research if that boundary is lawful and useful.

The second negative control enters unknown data quality as 3 of 5. The candidate appears balanced and outranks a better-investigated option whose real constraints are visible. The correction is to display the unknown, assign an evidence owner and ask whether resolving it can change the decision before expiry.

The counter-case is a mature repeated intake decision. Criteria and outcomes have been stable, historical cases are available, a simple linear triage rule is back-tested and overrides are explicit. Consistency can beat unaided judgement here. Monitor errors, drift and distributional effects, and compare the rule with the actual alternative rather than an ideal human.

The failure boundary appears where rights, consequences or stakeholder values are not commensurable. No scoring method can make the choice neutral. Preserve who benefits, who bears harm, which legal and ethical constraints apply and where stakeholders disagree. Decision-makers can still use evidence profiles and sensitivity, but they must own the trade-off.

Uncertainty and irreversibility also change commitment timing. Pindyck's work on investment under uncertainty explains why delay and flexibility can have option value.11 A point estimate can rank immediate commitment above learning by ignoring the value of waiting. Exploration deserves a separate information contract, as March's exploration-exploitation framework suggests.10

Maintain the comparison architecture. Version criteria, gate definitions and evidence grades. Archive forecast and outcome records so the organisation can calibrate optimism and learn where profiles were misleading. Review expired assumptions before each funding round and show material definition changes to decision-makers and owners. A ranking process earns more quantification over time when it can demonstrate stable inputs and predictive validity.

07 · ConclusionUse precision where evidence earns it

Opportunity scoring fails when it gives numerical form to unresolved definitions. Ordinal labels are added, unknowns become midpoints, hard constraints compensate with upside and chosen weights disappear behind decimals. The result is easy to sort and difficult to defend.

A staged evidence profile keeps discipline without abandoning comparison. Gate first. Show assessment and evidence separately. Filter dominance. Stress the assumptions that can switch the action. Preserve incomparable trade-offs for a named decision-maker. Record the result, dissent, owner and expiry.

Simple scores still have a home in repeated, calibrated environments. Their value comes from consistency and back-tested inputs, not aesthetic neatness. Early AI portfolios generally need the more cautious architecture because candidates are heterogeneous, effects are uncertain and constraints interact.

The executive question is not “Which use case has the highest score?” It is “Which candidate survives the gates, which evidence distinguishes it, what assumption would change our next action, and which trade-off must we own?” That answer may be less compact than 4.37. It is far more useful.

SourcesReferences

1. HM Treasury (2026). The Green Book 2026. Used for appraisal transparency, sensitivity, switching values, optimism bias and spurious accuracy.
2. Department for Communities and Local Government (2009). Multi-Criteria Decision Analysis. Green Book supplementary guidance.
3. Tversky, A. and Kahneman, D. (1974). Judgment under Uncertainty: Heuristics and Biases. Science, 185(4157), 1124–1131.
4. Dawes, R. M. (1979). The Robust Beauty of Improper Linear Models in Decision Making. American Psychologist, 34(7), 571–582.
5. Meehl, P. E. (1954). Clinical Versus Statistical Prediction. University of Minnesota Press.
6. Keeney, R. L. and Raiffa, H. (1993 edition). Decisions with Multiple Objectives. Cambridge University Press.
7. NIST (2023). Artificial Intelligence Risk Management Framework 1.0.
8. NIST (2023). AI RMF Core.
9. European Union (2024). Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence.
10. March, J. G. (1991). Exploration and Exploitation in Organizational Learning. Organization Science, 2(1), 71–87.
11. Pindyck, R. S. (1990). Irreversibility, Uncertainty, and Investment. NBER Working Paper 3307.
12. NIST (2023). AI RMF Manage Playbook.
13. OECD (2025). The Adoption of Artificial Intelligence in Firms. OECD Artificial Intelligence Papers.
14. Stanford Institute for Human-Centered Artificial Intelligence (2026). AI Index Report 2026. Used for environment context, not opportunity points.