TLDR

1. The use-case factory is a selection system. Throughput targets favour ideas that are easy to demo, politically safe, locally sponsorable and difficult to falsify.
2. The unit of value is a decision population with a baseline, accountable owner, changed mechanism, measurable outcome, comparison and capture route, not a named use case.
3. Adoption, satisfaction, benchmark accuracy and time saved are activity or intermediate receipts. They become value evidence only through a traced operating and financial consequence.
4. Separate a search portfolio, where every probe buys named information, from a delivery portfolio, where every candidate can fail a value and control gate.
5. The Use-Case Factory Escape Test routes candidates to discovery, value-chain redesign, controlled delivery or stop, with hard overrides for missing population, owner and control boundary.

Two portfolios each complete 20 AI pilots. Portfolio A reports 80 per cent positive participant feedback, full budget use and no serious incident. No pilot closes; none changes an operating baseline. Portfolio B stops 15 pilots within six weeks, transfers two negative lessons into shared controls and scales three narrow services with verified outcomes. Which factory was more productive?

Most executive dashboards would reward A until late in the year. It produced visible movement, engaged more sponsors and avoided the awkward ceremony of killing work. B appears wasteful because its most frequent output is a stopped experiment. Yet B converted uncertainty into decisions and operating change. A converted funding into stories whose success criteria made closure impossible.

The comparison changes when the output unit changes. Count decisions closed, reusable boundaries established and operating outcomes verified rather than pilots completed. Portfolio B then produced 20 decisions: 15 stops, two transfers and three controlled deliveries. Portfolio A produced 20 unfinished questions. Stopping work is not inherently good, but a timely evidence-based stop is productive when it releases attention and prevents weak assumptions from entering operations.

The difference is not project-management maturity. It is production logic. A factory needs countable units, standard stages and a throughput target. “Use cases” appear to supply all three. Once leaders count ideas submitted, proofs built, licences activated, active users or pilots completed, teams learn which proposals move fastest. Demo-friendly tasks enter the belt. Changes requiring workflow redesign, baseline reconstruction, exception analysis, authority shifts and capacity capture wait outside.

The factory then mistakes its own receipts for value. A demonstration proves that an interface can produce an attractive output on selected examples. A pilot proves that selected people used something under temporary conditions. Satisfaction proves a reported preference. None alone proves that a consequential decision changed, that the effect survived exceptions, that the organisation captured it or that the change caused the observed outcome.

Figure 1. The conveyor moves while the value ledger stays disconnected

Core mechanism

01 · Selection mechanismThe factory gets exactly what it rewards

An AI use-case factory produces activity instead of value because its throughput measures select proposals for demoability and non-falsifiability, then defer the mechanisms required to attribute and capture an outcome. The diagnosis is uncomfortable because the participants are behaving rationally. A team rewarded for launching pilots should not volunteer a hard baseline, costly comparison or kill rule that can reduce its launch count.

The mechanism does not require cynical behaviour. A discovery lead wants to keep sponsors engaged, a process owner cannot spare staff to reconstruct a baseline, a risk team prefers a reversible sandbox and a finance partner waits for stable evidence. Each choice can be reasonable. The selection failure arises because the pipeline admits work before these dependencies have carriers, then treats their absence as a later delivery problem rather than an entry condition.

Kerr's classic warning about rewarding one outcome while hoping for another applies directly.1 Executives hope for profitable operating change but reward idea volume, speed to demonstration and visible adoption. Sponsors prefer politically safe tools that add assistance without moving authority. Vendors prefer licence activation. Innovation teams prefer a repeatable stage gate. Users prefer help with tedious work. Each local preference is understandable; their combination severs the route to captured value.

The first selection filter is demoability. Drafting, summarising and search produce immediate outputs that observers can judge. A consequential workflow candidate may require months of outcome history, integration with an official system, a redesigned exception path and agreement about who owns released capacity. The latter can be more valuable and still lose a showcase competition.

The second filter is sponsor safety. An assistant layered beside existing work lets every function retain its territory. A proposal to remove a queue, consolidate review or change a service promise creates winners, losers and hard ownership questions. The factory calls the assistant “low hanging fruit” and the operating change “not ready”, even when the assistant merely adds another screen.

The third filter is immunity from disconfirmation. If success means finishing the pilot, collecting enthusiasm or showing technical feasibility, the pilot cannot fail on value. Baseline quality is reconstructed after the fact, negative cases become enhancement requests and extensions are framed as learning. A system without a negative result has no stop signal.

Current adoption data should not be mistaken for depth. A 2026 US Census working paper found that 18 per cent of firms reported AI use during its reference period, but 57 per cent of adopting firms used it in three or fewer business functions and 65 per cent limited use to three or fewer worker tasks. Among users, 66 per cent reported augmentation only.2 These nationally representative figures describe diffusion, not realised return. The positive correlation with commercial performance in the same paper is observational and does not establish that AI caused it.

Measurement boundaries also move. The Census Bureau revised its core question from use in producing goods or services to use in any business function, and reported overall use between 17 and 20 per cent from December 2025 to May 2026.3 A dashboard that treats each new usage number as comparable without the wording and denominator manufactures progress before any project does.

Figure 2. Selection pressure removes the hardest value work first

Selection funnel

The use case is too loose a unit

“Enterprise assistant” can contain thousands of decisions, users and outcomes. It is impossible to assign one baseline or causal mechanism to the label. Replace it with a decision population: a repeated decision, defined population, starting performance, accountable owner, change mechanism, measurable outcome, comparison and capture path. “Draft responses for tier-one password resets in the English-language queue” is still incomplete, but it can be bounded and tested.

The decision population forces exceptions into view. Which contacts are eligible? What happens when identity evidence is weak? Does the assistant create a review queue? Are repeat contacts counted? Does faster drafting change resolution or only keystrokes? Who can remove staffing capacity, improve service levels or absorb growth if time falls? The answers determine value more than the elegance of the generated text.

02 · Evidence disciplineExchange activity receipts for outcome receipts

An activity receipt proves that something happened inside the initiative: licences assigned, prompts sent, users active, training completed, demonstrations passed, satisfaction reported. These are useful operating measures. They can reveal adoption friction or a broken service. Their error is categorical, not statistical: they answer a different question from whether the organisation created and captured value.

An outcome receipt connects the intervention to a changed operating result. It names the baseline, population, comparison, observation window, exception costs, quality or risk boundary and accountable owner. A value receipt goes further by showing how the outcome was captured through capacity, avoided spend, lower loss, revenue, service improvement or a deliberate public-value objective. A time-saving estimate is not captured benefit until a mechanism changes what the organisation can do.

Define a promotion rule between receipt types. Activity evidence may justify usability work or a larger search probe. It cannot promote a candidate to delivery unless the missing outcome fields are filled prospectively. Outcome evidence may justify scale only when the capture owner accepts the operating action and finance or the relevant public-value authority agrees the measurement logic. Promotion is a decision with an evidence threshold, not the natural reward for reaching the end of a stage.

Figure 3. Activity and outcome receipts answer different questions

Evidence stacks

Worked example: the enterprise service assistant

A company gives 4,000 employees a workplace assistant. Weekly active use reaches 62 per cent, satisfaction is strong and a survey estimates ten minutes saved per user per day. The factory multiplies the self-report by salaries and declares an annual benefit. No queue, staffing plan, service level, revenue target or workload changes. Where did the benefit go?

Some may be real. Employees might finish earlier, improve quality, reduce stress or perform work that was previously deferred. Those are outcomes worth studying. They are not the same as released cost. The value owner must choose a capture mechanism: absorb rising workload without hiring, shorten a customer response target, redirect time to a measured revenue activity, reduce contractor spend or intentionally purchase employee wellbeing. Without that choice, the calculation prices a capacity that no one has authority to use.

A better scorecard starts with decision populations. For service contacts, record eligible contact type, baseline handle time, resolution quality, repeat contact, complaints, escalation, review effort and staffing capacity. Compare a controlled population over an adequate window. Separate time saved on successful cases from time added by exceptions. Then show whether workforce planning, service commitments or demand absorption changed.

The UK Government Digital Service ran a three-month experiment with Microsoft 365 Copilot involving 20,000 employees across 12 organisations and explicitly examined satisfaction, effort, task quality and routine-task time.4 The scale makes it informative, not universally causal. A workplace-tool experiment measures a particular product, population and period; it does not by itself prove firm-level financial capture.

Worked example: financial-crime evidence triage

A generic summariser performs impressively on curated investigation files. Reviewers praise clarity and the model beats a text benchmark. The sample excludes multilingual attachments, corrupted scans, linked entities and cases that trigger enhanced review. In production, those exceptions create most analyst time and severe risk. The polished proof is activity because its selection removed the work that determines value.

Redesign the unit as a residual-case triage service. A human investigator retains disposition. The system assembles evidence for a defined low-complexity population, records coverage and uncertainty, and routes unsupported cases away. Measure analyst changes to the assembly, false-negative severity, escalation quality, cycle time, repeat investigation, review effort and accountable disposition. Freeze a negative set containing the cases the demo excluded.

This service may have low adoption because it touches only a narrow residual population. It can still create more value than a popular assistant if it reduces severe leakage or releases scarce investigator capacity. Reach is a multiplier only after the per-decision effect and capture route are credible.

Figure 4. The pilot treadmill prevents a negative result

Negative control

03 · Portfolio designProtect search by separating it from delivery

The strongest rival explanation deserves to win often. Under fast technical change, broad experimentation can be rational. Requiring a complete business case before testing may suppress discovery, especially when teams do not yet know model boundaries, user behaviour or integration cost. March's exploration-exploitation framework explains why organisations need both search for new possibilities and refinement of known ones.5

The correction is not to eliminate sandboxes. Split the portfolio. A search portfolio buys information. Every probe names the uncertainty, observation, bounded exposure, expiry date and later decision that the result changes. A delivery portfolio buys operating outcomes. Every candidate names the decision population, baseline, causal change, owner, comparison, exception cost, capture mechanism, control boundary and kill rule.

A probe can succeed by returning a negative answer. Perhaps the model fails on the organisation's abbreviation distribution, users cannot challenge quickly enough or retrieval cost erases the opportunity. The information closes or redirects a later decision. A delivery candidate succeeds only when the changed system improves the defined outcome within its quality and risk boundary and the organisation captures the benefit.

Sparse evidence still requires disciplined interpretation. March, Sproull and Tamuz describe how organisations learn from rare or even unobserved events by enriching experience through careful interpretation and simulation.6 That does not permit a single pilot to prove a general business case. It permits a search portfolio to extract hypotheses, boundary conditions and next tests without pretending they are realised outcomes.

NIST's AI RMF makes room for both modes. Its core asks organisations to define business value or context and connects measurement, feedback and go/no-go decisions.7 Its effectiveness section explicitly mentions documented outcomes and go/no-go commissioning and deployment processes.8 Governance is therefore compatible with killing a weak delivery candidate and continuing a bounded discovery programme.

Figure 5. Search and delivery use different contracts

Portfolio split

Thought experiment: the ten-minute saving

Suppose a tool genuinely saves ten minutes per employee per day. Quality is unchanged and employees like it. No workload, staffing, price, service level or revenue mechanism changes. The technical effect is real. The economic outcome is undetermined. The missing link is an owner with authority to convert distributed slack into something the organisation values.

Now suppose the organisation intentionally uses the time to reduce unpaid overtime and burnout. That can be a valid benefit even without headcount reduction. It still needs an outcome measure, observation window and owner. Value capture is not synonymous with cost cutting; it is the deliberate conversion of an effect into a stated organisational objective.

04 · Causal value chainTrace every leak between decision and captured benefit

Start with the changed decision. What does the system cause a person or service to choose differently? Then trace changed work, changed operating outcome and captured benefit. At each link, record a leak. Human review may consume the saved time. Exceptions may move work downstream. Faster completion may create no extra throughput because another constraint governs the process. Released capacity may fragment into minutes no manager can redeploy. Revenue may shift across channels rather than increase.

Field studies show why the population must stay attached. Across 5,172 support agents, Brynjolfsson, Li and Raymond measured meaningfully higher hourly issue resolution on average, with uneven effects across workers.9 Dell'Acqua and colleagues found gains on tasks inside an AI capability boundary and worse performance on a task outside it.10 These are important results in named settings, not universal multipliers for a business case.

Three randomised field experiments involving 4,867 developers provide another reason to preserve experiment identity and outcome definitions: pooled task gains can coexist with noisy or heterogeneous experiment-level estimates.11 A portfolio should not average away the very context needed to decide where deployment works.

The OECD's evidence synthesis distinguishes task-level productivity effects from firm-level productivity and broader value capture.12 Its adoption work also notes that firms can underestimate the process and cultural change required, while uncertainty about return, skills, data maturity and regulation constrains use.13 A factory that measures tool activity but not complementary operating change ignores the costliest part of adoption.

Figure 6. Value leaks at four joins

Causal chain

Carry ownership across the joins

A sponsor can fund a pilot without owning its operating outcome. A product owner can ship a feature without controlling staffing. Finance can validate a calculation without changing work. Operations can absorb a service without controlling the model. Risk can approve a boundary without owning value. The chain needs a baton: a named carrier at each join and one outcome owner with authority to resolve gaps.

Handover should occur before the pilot, not after enthusiasm. The process owner agrees the baseline and changed work. Finance agrees the capture logic and prevents double counting. Risk defines the control boundary. Operations confirms who runs exceptions and support. The product team owns technical delivery. If any party refuses the proposed mechanism, the candidate enters redesign before further build.

Figure 7. The ownership baton often falls between functions

Accountability map

05 · Decision instrumentThe Use-Case Factory Escape Test

Apply the test to one decision population. Score current evidence, not the sponsor's confidence. The result chooses the next portfolio contract. It does not estimate ROI, certify legality or prove causality. Attach a source, owner and expiry date to every non-zero answer.

Run the gate on a fixed cadence with people who can close the result. The search owner decides whether the observation changed the named later decision. The process owner accepts or rejects the operating mechanism. Risk confirms the boundary and recovery route. Finance or the public-value owner tests capture. Record discovery, redesign, delivery or stop as a portfolio transaction, including resources released and reusable evidence. A gate without closure authority becomes another presentation.

Use-Case Factory Escape Test

The test separates information-producing search from controlled delivery and refuses to convert activity into value by arithmetic.

Baseline qualityNo stable baselineAnecdote or reconstructed estimateMeasured outcome with known gapsVersioned baseline with slices

Decision population specificityGeneric use case or enterprise-wideFunction named, cases mixedEligible population and exclusionsEnumerated units with exception route

Accountable outcome ownerSponsor only, no operating ownerNamed owner without change authorityProcess owner can change workOwner controls outcome and handovers

Causal change mechanismAccess or usage is the mechanismTask help, downstream change unknownNamed decision and workflow changeMechanism tested through the value chain

Counterfactual or comparisonNo credible comparisonBefore-after with major confoundersMatched or phased comparisonControlled design with validity checks

Exception and review costExcluded from evaluationObserved but not costedMeasured for material pathsFull residual workflow and severity

Value-capture authorityNo owner can capture the effectTime saving without operating changeNamed capacity, service or revenue routeCapture action approved and tracked

Kill criterion and expiryCompletion defines successSoft review, no binding thresholdPre-set stop or redesign ruleRule enforced with closure owner

Reusable information yieldNo discriminating observationLocal learning, later decision unclearNamed uncertainty and later decisionTransferable boundary or capability evidence

Lawful control boundaryMissing authority or material controlBounded probe with no relianceDelivery controls and recovery definedControls tested with negative cases

Convert activityDiscovery probeValue-chain gapDelivery caseEmpty activity

Assumptions, hard overrides and limits

• Assumptions: one decision population and observation window are in scope; the outcome measure is meaningful; costs include the residual workflow; and all empirical claims remain attached to their population.
• Discovery test: a named uncertainty, discriminating observation, expiry and later decision can justify a bounded probe even when baseline and capture readiness are immature.
• Redesign test: a plausible problem with missing owner, baseline, mechanism or capture authority returns to value-chain design before further build.
• Delivery test: credible baseline, population, causal chain, comparison, residual cost, owner, capture route, control boundary and kill rule support controlled delivery.
• Stop test: a candidate with neither discriminating information nor a plausible value chain stops, regardless of enthusiasm or sunk cost.
• Hard overrides: undefined population, absent outcome owner or missing lawful control prevents delivery. Material exposure cannot be converted into a discovery label to avoid control.
• Limits: ordinal scores can be gamed. The test does not calculate financial value, establish causality, validate safety, classify regulation or replace portfolio judgement.

Figure 8. Escape gates route work by evidence contract

Decision board

06 · Failure boundaryDo not turn discovery into a delivery ledger

Basic research, regulatory learning and early technical exploration can be valuable without near-term outcome capture. A laboratory may study model behaviour before a stable workflow exists. A regulator may test interpretive questions. A technical team may learn whether a new modality is usable. Forcing each into an ROI table creates fabricated precision and discourages important search.

The boundary is an explicit information contract. Name what is unknown, what observation would be informative, who will interpret it, when the work expires and which later decision it informs. Bound exposure and prevent operational reliance. Publish negative findings in a reusable form. If no later decision can be named, the work may still be research, but it should not compete in a delivery portfolio or claim operating value.

The factory critique wins when “learning” protects repeated pilots from closure. If the same uncertainty survives three extensions, the observation is not discriminating, the evidence is not being used or no one owns the later decision. Stop, redesign the experiment or reclassify the programme honestly. Search deserves protection from premature financial claims; funding deserves protection from perpetual search theatre.

No universal pilot-failure percentage is needed. Popular claims about how many AI pilots fail often lack a stable denominator, consistent stage or comparable definition of failure. The mechanism is observable inside one organisation: inspect what enters, which criteria accelerate it, what the gates measure, how often negative results close work and whether scaled services carry outcome receipts.

07 · ConclusionMake the factory produce decisions

The AI use-case factory is not too active. It is active around the wrong unit. Ideas, demos, licences and pilots are easy to count, so the system selects work that can produce them. Baselines, exception cost, comparison, operating ownership and value capture arrive late because they slow the belt and create the possibility of failure.

Replace the generic use case with a decision population. Replace one pipeline with search and delivery portfolios. Replace completion gates with evidence contracts. A search probe should end in information that changes a later decision. A delivery candidate should carry a baseline, causal mechanism, outcome owner, comparison, full residual cost, capture route, control boundary and kill rule. Both should be allowed to stop.

This changes the meaning of productivity. A portfolio that kills 15 weak ideas may outperform one that completes 20 pilots. A narrow service with low adoption may matter more than a popular assistant. A genuine task saving may produce wellbeing rather than cost reduction, if that is the chosen objective. Value is not whatever can be multiplied after a survey; it is the outcome the organisation deliberately changes and captures.

The final executive question is not “How many use cases are in the funnel?” It is “What decision did each experiment buy, what operating outcome did each delivery change, and where is the receipt?” When the factory can answer those questions, activity becomes a means rather than its product.

SourcesReferences

1. Kerr, S. (1975). On the Folly of Rewarding A, While Hoping for B. Academy of Management Journal, 18(4), 769–783.
2. Bonney, K. et al. (2026). The Microstructure of AI Diffusion: Evidence from Firms, Business Functions, and Worker Tasks. US Census Bureau, CES Working Paper 26-25.
3. US Census Bureau (2026). Large Firms With at Least 20 Employees Biggest AI Users. Used with its revised survey wording and measurement period.
4. Government Digital Service (2025). Microsoft 365 Copilot Experiment: Cross-Government Findings Report.
5. March, J. G. (1991). Exploration and Exploitation in Organizational Learning. Organization Science, 2(1), 71–87.
6. March, J. G., Sproull, L. S. and Tamuz, M. (1991). Learning from Samples of One or Fewer. Organization Science, 2(1), 1–13.
7. NIST (2023). Artificial Intelligence Risk Management Framework Core. Used for context, measurement, monitoring, feedback and go/no-go decision practices.
8. NIST (2023). Effectiveness of the AI RMF. Used for documented outcomes and commissioning or deployment decisions.
9. Brynjolfsson, E., Li, D. and Raymond, L. R. (2025). Generative AI at Work. The Quarterly Journal of Economics, 140(2), 889–942.
10. Dell'Acqua, F. et al. (2026). Navigating the Jagged Technological Frontier. Organization Science.
11. Cui, Z. et al. (2026). The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers. Management Science.
12. OECD (2025). The Effects of Generative AI on Productivity, Innovation and Entrepreneurship. OECD Artificial Intelligence Papers.
13. OECD (2025). The Adoption of Artificial Intelligence in Firms. OECD Artificial Intelligence Papers.
14. Aldasoro, I. et al. (2024). Generative AI and labour productivity: a field experiment on coding. BIS Working Paper 1208.
15. Stanford Institute for Human-Centered Artificial Intelligence (2026). AI Index Report 2026. Used for adoption and investment context, not as evidence of realised organisational value.