A credit-review team celebrated its model because approved recommendations defaulted less often. The chart looked decisive: cases where advisers followed the model performed better than cases where they did not. Yet advisers accepted the suggestion mainly on well-documented, low-ambiguity applications. They overrode it when identity evidence was thin, income was irregular or policy exceptions required judgment. The success chart had measured case selection, not model contribution.

Reversing the labels did not rescue the analysis. Raw outcomes after an override made reviewers appear harmful because overrides were concentrated on difficult cases. Counting every good accepted case as machine value and every bad changed case as human error would convert workflow triage into a fictional contest. The useful question was whether offering and operating the combined review policy improved eligible decisions compared with a credible alternative.

That question changes the unit of analysis. The unit is not a prediction, click or person. It is an eligible decision episode under a specified operating policy: who received access, what version produced which output, what the reviewer saw, how they responded, whether an authorised action occurred, what outcome became observable and which counterfactual supports the comparison.

Unit of attributionDefine the combined operating policy

A combined operating policy specifies more than tool access. It states eligibility, output form, uncertainty display, abstention behaviour, review duty, permitted modification, override authority, escalation, action timing and outcome follow-up. Two teams using the same model may therefore operate different treatments. One may require review before any action; another may expose an optional suggestion in an existing interface; a third may automate low-risk cases and route the remainder.

The counterfactual must be equally concrete. “Before AI” is rarely adequate because staffing, demand, prices and decision rules may have changed. Human-alone may mean the same trained team working from identical records without model output. AI-alone may mean an automatic threshold with a defined abstention and escalation rule. The combined policy includes both components plus the handoff. Comparisons are commensurate only when eligibility, outcome horizon and decision authority align.

The first estimand is often the intention-to-treat effect of offering the configuration to eligible episodes. It preserves assignment even when people do not use the recommendation, so non-use remains part of the policy’s real effect. A policy or threshold effect may be more relevant when the governance question concerns which cases are automated, reviewed or abstained. Effects by role, task and baseline skill matter when the average hides gains for novices and losses for experts, or vice versa. Rubin’s potential-outcomes formulation and Hernán and Robins’ target-trial logic provide the counterfactual discipline.[1][2]

This is deliberately not a human share plus a model share. Component contributions may remain unidentified even when a clean experiment identifies the combined policy effect. Leaders can still decide whether to fund, redesign or stop the policy. A forced rupee allocation between participants supplies accounting precision without causal information.

The policy definition must also distinguish availability from obligation. Optional advice tests an offer that a reviewer may ignore. A mandatory examination rule tests a different configuration even if the underlying prediction is unchanged. Default acceptance, required reasons for disagreement and automatic escalation can each alter behaviour. If these settings move during the study, the treatment has changed and the version record must show when.

Eligibility is part of the intervention. A model may serve only low-risk episodes, cases with complete records or customers who consent to a particular data use. Expanding conclusions to excluded episodes silently changes the target population. The study record therefore reports how many episodes entered, why others did not and whether the business case concerns the measured population or a proposed future one.

Cost and value follow the same boundary. Review time, second-line escalation, data remediation and delayed action belong to the combined policy. So do avoided rework, faster resolution and harms outside the primary success measure. A faster decision with more appeals may transfer effort rather than create value. Attribution identifies the outcome difference; the economic case then values the full receipt within the authority and consequence framework established elsewhere in the portfolio.

Observable chainEight linked records around one episode

The joint-production attribution protocol below is this article’s synthesis. Its purpose is to keep operational telemetry aligned with the causal question. A stable episode identifier links records, but access is restricted by role and purpose. The record is about the decision pathway, not a covert productivity score for a named worker.

Separating exposure from access catches a common instrumentation error. A user may be assigned the tool but never receive an output because latency, entitlement or abstention intervenes. Separating the decision from action catches another: an adviser may accept a recommendation, but a later control, customer choice or system failure prevents execution. Outcome attribution begins with the real causal chain, not the interface clickstream.

Pre-treatment difficulty should be captured with variables available before assignment and action. Post-treatment handling time, number of screens opened or escalation are often consequences of the policy and cannot be used casually to make treated and comparison cases look alike. Matching on them may remove part of the effect or open a new bias path. The design note labels each field as baseline, treatment, mediator or outcome before analysts build adjustments.

Timing is equally important. The recommendation timestamp does not prove examination; examination does not prove an action; and an action may affect outcomes only after a lag. The protocol records an admissible sequence and flags reversals, late actions and duplicate episodes. Outcome windows are chosen for the mechanism. A service-resolution effect may appear within days, while repayment, recurrence or employee learning requires longer observation.

Capture must include denominator changes. If a policy diverts difficult cases away from a measured queue, the remaining cases can look faster and more successful even when total system performance is unchanged. Eligible episodes, not completed actions alone, form the denominator. A companion workload ledger follows transfers, abandonment, appeals, rework and cases still unresolved at the end of the window.

Reasons for modification and override can help test mechanisms, but they must be bounded. Free-text surveillance, employee ranking and undisclosed behavioural scoring are not necessary for policy evaluation. A small governance-approved taxonomy, aggregated reporting and role-based access can preserve useful evidence while respecting employment and privacy boundaries. Human oversight under the EU AI Act concerns effective understanding and intervention, not theatre around an automated output.[13]

Comparison disciplineThree baselines and the strict test for synergy

Human-alone, AI-alone and combined policies answer different questions. Human versus combined estimates whether adding the configured system to human work changes outcomes. AI-alone versus combined estimates whether the examination, correction and escalation layer adds value to an automated policy. Human versus AI-alone compares two complete decision policies. None identifies a universal property of “the model” or “the human”.

Synergy has a demanding meaning. The combined policy must outperform both stand-alone policies on commensurate eligible tasks, outcomes and horizons, after accounting for operating cost and adverse effects. If AI-alone is never permitted for high-consequence cases, that missing baseline may be ethically appropriate. The study should then avoid a synergy claim and estimate the authorised comparison. A 2024 meta-analysis found that human-AI combinations do not reliably surpass the better of human or AI performance, reinforcing the need for three-way comparisons rather than celebratory labels.[6]

Field evidence also warns against treating one average as a permanent capability coefficient. Generative assistance has produced heterogeneous effects by worker experience and task boundary, while performance can reverse beyond a technology’s uneven frontier.[7][8] The attribution receipt therefore includes role, task, baseline skill and version. Learning may persist when the tool is absent, which is an organisational effect rather than current model use. Workload may shift to untreated teams, creating displacement that a local throughput metric misses.

A factorial design can sometimes isolate features such as an uncertainty display or required review, but only when the variants are safe, feasible and sufficiently powered. It still estimates effects of policy components under the tested configuration, not metaphysical shares of authorship. Interactions may be estimated as contrasts between assigned configurations. They should not be converted into a permanent percentage of value credited to a person or model.

Baseline competence can change during rollout. Reviewers learn from repeated suggestions, while a model may be retrained from newly captured decisions. A washout period cannot necessarily remove that learning. The protocol therefore asks whether the decision concerns immediate augmentation, sustained capability after training, or a sequence in which people and systems co-adapt. Those are different estimands with different counterfactuals.

Negative controlsSelection, labels, overrides and spillovers

Selective acceptance

Advisers commonly use recommendations on cases that already look tractable. Comparing used with unused recommendations then mixes treatment response with their private information about difficulty. A treatment-on-the-treated estimate from voluntary users needs a defensible instrument, selection model and exclusion assumptions. Without them, intention-to-treat is the safer estimand even if dilution by non-use makes it look modest.

Selective labels

Credit outcomes are observed mainly for approved applicants. A rejected applicant’s repayment under approval is missing by construction. Training or evaluating on observed loans can therefore favour the historical decision policy and distort comparisons between judges and predictions. Kleinberg and colleagues use judge leniency as an instrument to address this selective-label problem in a bounded setting.[5] The lesson is not that every firm possesses such an instrument; it is that outcome availability follows prior action and must be modelled.

Override mythology

Overrides cluster where reviewers see exceptional complexity, uncertainty or policy conflict. Raw post-override error therefore combines case mix with correction quality. A useful analysis first compares like cases or uses assignment, then examines whether override rules improve the combined policy. Automation research has long distinguished use, misuse, disuse and abuse; adherence and override are behaviours to understand, not self-validating evidence of either trust or skill.[9]

Spillover contamination

Trained users may teach control-group colleagues, share templates or change upstream data capture. Access effects then shrink while organisational learning grows. A queue can also displace hard work into the control team, creating an apparently favourable treated outcome at somebody else’s expense. Cluster assignment, exposure mapping and workload measures make these pathways visible. If interference is unavoidable, the estimand may need to become a team-level rollout policy rather than an individual access effect.

Negative controls can reveal residual bias without proving a design valid. An outcome that the policy cannot plausibly affect should not move at assignment. An exposure recorded before assignment should not predict assignment once the stated rule is applied. A placebo intervention date should not reproduce the headline effect in a stable pre-period. Unexpected movement prompts investigation of data lineage, calendar shocks or unmeasured selection.

Heterogeneity analysis should be pre-specified around operational mechanisms. Role, task class, baseline skill, case difficulty and permitted authority are usually more informative than a large search across convenient attributes. Small subgroups yield unstable stories, particularly when only selected outcomes are observed. The primary decision can rely on a pooled policy effect while later windows gather enough evidence for safe routing by subgroup.

Strong rivalAttribution when systems change quickly

The strongest objection is practical: controlled attribution can take longer than the model or workflow remains stable. By the time a long experiment finishes, the result may describe an obsolete configuration. This objection is valid when treatment stability is weak. The answer is not to treat contemporaneous correlation as causation. It is to shorten the decision window, preserve version boundaries and choose the most credible feasible design.

Short randomised windows can assign the combined policy at an episode, adviser, queue or team level. Encouragement designs can vary prompts, training or default access when direct assignment is constrained, provided the encouragement affects the outcome only through policy use and other assumptions are defensible. A stepped rollout can create comparisons across timing, but calendar shocks and heterogeneous effects require explicit treatment. Callaway and Sant’Anna show why multiple-period difference-in-differences needs cohort-aware estimands rather than a single naive before-after contrast.[4]

Matched comparisons and synthetic controls can support bounded cases where randomisation is unavailable and a stable pre-intervention history exists. Synthetic control is most credible when a treated aggregate has a donor pool capable of reproducing its pre-treatment path, not when analysts merely average convenient peers.[3] Regression discontinuities may use an enforced threshold if units cannot manipulate it and effects near the cut-off answer the decision. Shadow recommendations instrument exposure and disagreement before they influence actions, which is useful when event completeness or outcome linkage is not ready.

Each alternative has a failure boundary. Assignment may be confounded. Treated and comparison workflows may differ in unrecorded ways. Outcomes may exist only after selected actions. Spillovers may contaminate controls. Versions may change mid-window. Sample sizes may be inadequate for the intended heterogeneity. A design note lists these limitations and states what decision the evidence can and cannot support.

Randomisation also needs an implementation audit. Assignment probabilities, exclusions and switches are reconciled against the eligible population. Analysts check whether queue managers anticipated assignment, whether users could seek access through another channel and whether the randomisation unit matches the interference mechanism. Protocol deviations remain visible in the intention-to-treat result rather than being deleted to make adherence look clean.

Encouragement designs deserve particular caution. Training invitations, reminders or defaults can be useful instruments only when they change policy use, do not directly change the outcome through another pathway and do not create hidden versions of treatment. A coaching session that improves general decision skill is not merely encouragement to use a tool. Its direct learning effect violates the simple exclusion story and may require a broader policy estimand.

A stepped design does not automatically randomise time. Capacity, readiness or leadership preference may determine which team starts first, and those factors may also predict outcomes. Randomising the entry order strengthens the design where feasible. When entry order is fixed, the analysis reports the selection process, checks pre-trends and narrows the claim. Time-varying effects are expected when teams learn or versions change, so a single pooled coefficient may conceal the mechanism.

Shadow mode has its own limits. Reviewers should not see recommendations intended to be non-influential, and hidden outputs must not leak into escalation or labelling. Shadow predictions can measure coverage, latency, disagreement and outcome linkage, but they do not reveal how humans would respond after exposure. The next live comparison remains necessary before attributing an operating-policy benefit.

The counter-case is a bounded stepped rollout in which teams enter on a published calendar. The organisation records eligibility, assignment, actual exposure, model version, human response, action and outcome. It keeps one holdout long enough to answer the funding decision, monitors contamination and compares the combined policy with the current human-alone workflow. AI-alone is evaluated only on a permitted commensurate subset. The result can identify a useful combined-policy effect without claiming every component contribution.

Interactive instrumentHuman-AI Attribution Designer

The designer chooses a study route, not a credit percentage. Inputs describe the decision episode, available baselines, assignment, observable event chain, outcome linkage, version stability, interference and governance. Its output names the estimand and missing evidence that should govern the next measurement window.

Decision governanceReceipts, confidence and honest limits

An attribution result should travel with an evidence receipt. It names the eligible population, assignment unit, treatment and comparison versions, primary estimand, outcome horizon, adherence pattern, missingness, contamination, side effects and confidence interval. It also records which decision changed because of the result. A policy effect without this context will be reused as though it were a timeless model return.

The receipt separates confirmatory and exploratory work. The primary outcome, horizon, analysis population and stopping logic are fixed before the result is known. Mechanism, subgroup and override analyses may remain exploratory until replicated. This separation prevents a busy event stream from producing whichever attribution story happens to look strongest after many comparisons.

Outcome ownership also needs a named steward. Finance may validate a monetary capture, operations may own service quality, risk may own adverse-event definitions and product may own the policy version. Their records need not contain the same sensitive fields, but they must reconcile to the eligible episode and study window. Access controls allow verification without constructing an unrestricted worker-level dossier.

Confidence limits belong beside point estimates, but sampling uncertainty is only one layer. Identification assumptions, version drift, measurement error and interference can dominate. A tight interval around a confounded contrast is not strong evidence. The receipt separates statistical uncertainty from design limitations and states which assumptions were tested, argued or simply unavailable.

NIST’s AI Risk Management Framework places human roles, responsibilities, oversight and impact assessment inside governance rather than treating a human click as a universal safeguard.[10][11][12] The OECD’s firm evidence likewise shows that adoption depends on complementary skills, data, organisation and support, making the combined configuration the practical object of evaluation.[14]

A treatment-on-the-treated estimate may be valuable when a valid instrument or defensible selection assumptions exist. It should not be improvised by filtering to actual users. Likewise, a mechanism analysis can test whether uncertainty displays, abstention or review rules explain the policy effect. It cannot assign residual value to the human merely because measured model features do not explain it.

Attribution sometimes ends with “unidentifiable”. That is not analytical failure when it prevents a false claim. The result can still reveal which event is missing, which baseline is prohibited, where spillover crosses the unit or why the version cannot be treated as stable. The next investment may be a short shadow period, an outcome-linkage repair or a revised rollout rather than another dashboard.

Leaders should fund the combined policy when credible evidence shows that it improves consequential outcomes within authority, cost and risk boundaries. They should redesign when the handoff, abstention or workflow creates the loss. They should stop when the effect fails across a defensible comparison. In every case, the object is a better operating decision, not a trophy for either participant.

ConclusionAttribute the policy before celebrating the parts

Begin with an eligible episode and a pre-treatment baseline. Preserve assignment. Record actual exposure, recommendation, uncertainty, abstention, human examination, authorised action, timing and outcome. Define the alternative policy with the same care as the treatment. Test selection, missing labels, difficult-case overrides, version change and spillover before using causal language.

Compare human-alone, AI-alone and combined policies only where tasks and authority make the baselines commensurate. Reserve synergy for a combined policy that exceeds both stand-alone alternatives. When only the joint effect is identified, make the funding decision on that evidence and refuse the component split.

The practical discipline is modest but consequential: name the estimand, protect the counterfactual, retain version and contamination records, report confidence and limitations, and connect the finding to a decision. Humans and models can produce value together without the evidence permitting anyone to divide it cleanly.

Evidence spineReferences

1. Rubin, D. B. (1974), ‘Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies’, Journal of Educational Psychology, 66(5), 688–701. doi:10.1037/h0037350.
2. Hernán, M. A. and Robins, J. M. (2020), Causal Inference: What If. Boca Raton: Chapman & Hall/CRC. Open book site.
3. Abadie, A., Diamond, A. and Hainmueller, J. (2010), ‘Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California’s Tobacco Control Program’, Journal of the American Statistical Association, 105(490), 493–505. doi:10.1198/jasa.2009.ap08746.
4. Callaway, B. and Sant’Anna, P. H. C. (2021), ‘Difference-in-Differences with Multiple Time Periods’, Journal of Econometrics, 225(2), 200–230. doi:10.1016/j.jeconom.2020.12.001.
5. Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J. and Mullainathan, S. (2018), ‘Human Decisions and Machine Predictions’, The Quarterly Journal of Economics, 133(1), 237–293. doi:10.1093/qje/qjx032; earlier version: NBER Working Paper 23180.
6. Vaccaro, M., Almaatouq, A. and Malone, T. W. (2024), ‘When Combinations of Humans and AI Are Useful: A Systematic Review and Meta-analysis’, Nature Human Behaviour, 8, 2293–2303. doi:10.1038/s41562-024-02024-1.
7. Brynjolfsson, E., Li, D. and Raymond, L. R. (2023), ‘Generative AI at Work’, NBER Working Paper 31161. doi:10.3386/w31161.
8. Dell’Acqua, F. et al. (2023), ‘Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality’, Harvard Business School Working Paper 24-013. HBS working paper.
9. Parasuraman, R. and Riley, V. (1997), ‘Humans and Automation: Use, Misuse, Disuse, Abuse’, Human Factors, 39(2), 230–253. doi:10.1518/001872097778543886.
10. National Institute of Standards and Technology (2023), ‘AI Risk Management and Human-AI Interaction’, Appendix C to the AI RMF. NIST AIRC.
11. National Institute of Standards and Technology (2023), ‘AI Risk Management Framework: Core’. NIST AIRC Core.
12. Tabassi, E. (2023), Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1. doi:10.6028/NIST.AI.100-1.
13. European Union (2024), Regulation (EU) 2024/1689 Laying Down Harmonised Rules on Artificial Intelligence, Article 14. Official Journal text.
14. OECD, Boston Consulting Group and INSEAD (2025), The Adoption of Artificial Intelligence in Firms: New Evidence for Policymaking. doi:10.1787/f9ef33c3-en.