01 · When the target becomes the workThe successful scorecard that explains the failure
The business unit has exceeded its AI licence-activation target. Pilot count is ahead of plan. Teams report impressive hours saved and no incidents. Behind the dashboard, staff paste sensitive fragments into unsanctioned tools, reviewers correct weak output after hours, service queues have not shortened, and near misses remain private because escalation lowers the team's performance rating. Every visible target is green. The operating system is learning to hide.
This is not primarily a training failure. People have inferred the real contract from promotion decisions, meeting attention, delivery deadlines, vendor commissions and the questions executives ask. If leaders celebrate activity, employees will create activity. If a stopped pilot damages a career, a weak pilot will survive. If incident counts enter a tournament, incidents will be renamed, delayed or kept local. The behaviour is locally rational even when the enterprise result is destructive.
Safe learning means producing information that changes a consequential decision without crossing a prohibited boundary. It includes finding that a model works for a defined population, showing that it fails on a material slice, revealing that verification labour erases expected savings, reporting an attack before harm, transferring a test so another team can use it, or closing a pilot whose remaining uncertainty is not worth buying. The result can be progress, redesign or a stop.
Negative control: pay per failure
A team receives a reward for every failed experiment or near-miss report. It quickly produces low-value tests, fragments one event into several reports and avoids the difficult work of preventing recurrence. This does not reward learning. It prices the occurrence and makes volume the objective. Recognition belongs to the evidence, intervention, system repair and knowledge transfer, never to manufacturing incidents.
02 · Write the real dealThe Safe Learning Incentive Contract
Incentives include far more than variable pay. Protected time makes verification feasible. Authority lets a worker stop a release without begging for cover. Recognition changes whose judgement carries status. Staffing turns review from a nominal control into a usable service. Promotion criteria determine whether coaching and knowledge transfer compete with personal throughput. Manager attention reveals which evidence matters. Financial reward is one instrument among these and often the bluntest.
The Safe Learning Incentive Contract is this article's first synthesis. It names the intended business outcome and affected population, the current stage of work, the decision and consequence boundary, and the behaviours worth repeating. It then specifies outcome, evidence, control and learning claims; measures excluded from reward; effects on workers, customers, controls and downstream teams; and contribution across individuals, teams, platforms, vendors and executives.
The contract joins accountability to the resources needed to discharge it. A manager cannot be accountable for prompt incident review without review capacity. A product owner cannot own benefit capture when finance withholds the baseline. A worker cannot be expected to escalate when the operational target penalises every pause. Authority, time, expertise, access and staffing sit beside each accountabilities field.
Finally, it records protections for good-faith reporting, routes for investigation and remedy, possible gaming and shifted cost, reward form, timing, deferral, reversibility, independent validation, appeal, expiry and refresh. The contract is stage-specific. Exploration can recognise useful information without promising scale. Operation can reward persistent service outcomes. Retirement can recognise an early stop that prevents further cost. One annual target cannot express those different burdens.
| Block | Exploration question | Delivery or operation question | Retirement question |
|---|---|---|---|
| Outcome and population | Which uncertainty could change the decision? | Which service effect is verified for whom? | Which cost or harm ends? |
| Rewardable behaviour | Reversible test and discriminating evidence. | Reliable outcome, control and transfer. | Timely stop, continuity and recovered assets. |
| Excluded proxy | Prompt, pilot or experiment volume. | Deployment count or declared savings. | Keeping a weak capability alive to protect a target. |
| Attached resource | Test capacity, expert access and protected time. | Review, operations and remedy capacity. | Exit authority, transition support and recognition. |
| Validation and timing | Immediate evidence review. | Independent persistence and benefit check. | Closure evidence after affected services stabilise. |
03 · Four irreducible viewsThe Four-Ledger Reward System
The Four-Ledger Reward System is this article's second synthesis. The outcome ledger records verified effects on the decision or service: quality, time, cost, resilience, customer or worker experience, and captured benefit. It prevents activity from impersonating value. A faster draft is not an outcome if the queue remains unchanged or review labour simply moves elsewhere.
The evidence ledger records information that reduced a material uncertainty, distinguished between alternatives, exposed a boundary or made a decision reconstructable. It gives exploration a legitimate result without pretending that learning equals value. Evidence can justify another bounded test, a redesign or a stop. It cannot justify indefinite experimentation when no strategic or economic path remains.
The control ledger covers adherence, challenge, exception handling, abstention, recovery, recourse, reliability and prevented or contained consequence. It resists the idea that speed can cancel control debt. It also rejects a spotless incident count as proof of safety when reporting is costly. The learning ledger records reusable tests, runbooks, patterns, failure analyses, capability transfer, workflow changes, retired assumptions and decisions to stop.
No ledger can settle the reward alone. Strong outcomes with concealed risk are not safe success. Extensive documentation without a changed decision is not useful evidence. Control work without a viable outcome may be necessary for a bounded period, but it cannot rescue a product with no reason to exist. Reusable learning can justify the cost of a disciplined experiment, but not a permanent programme whose original bet has failed.
| Ledger | Credible evidence | What does not substitute |
|---|---|---|
| Outcome | Reconciled effect on the named decision, service and population. | Licence use, output volume or self-reported hours saved. |
| Evidence | A result that reduces a named uncertainty or rejects an alternative. | A slide deck, demo or test chosen after seeing the answer. |
| Control | Observed prevention, detection, intervention, recovery and remedy. | Policy completion or zero reported incidents. |
| Learning | Reused tests, transferred skill, workflow repair or retired assumption. | Private heroics and a lessons-learned document nobody can apply. |
Eight behaviours worth making locally rational
- Frame a consequential uncertainty before building.
- Design a reversible test with explicit stop criteria.
- Verify evidence and state where it does not travel.
- Abstain or escalate when authority or evidence is insufficient.
- Report an error, near miss or adverse outcome promptly.
- Share a reusable test, runbook, artefact or pattern.
- Transfer capability until hero dependency disappears.
- Stop, narrow or retire weak work before cost and harm compound.
04 · Inspect the distortion firstThe Incentive Hazard Map
The Incentive Hazard Map is this article's third synthesis. For every target, it begins with the visible measure and the underlying outcome the organisation actually wants. It identifies who can influence, validate and benefit from the number. It then traces the easiest gaming route, important unmeasured tasks likely to lose attention, and costs or risks that can be shifted to reviewers, customers, operations or a later reporting period.
The map asks which reporting behaviour the target suppresses, whether its time horizon matches the consequence, which populations disappear inside an average and whether executive, team and vendor incentives combine to amplify the same distortion. Leading signals matter because outcome damage may arrive late. A rising manual correction queue can contradict apparent speed gains. A flat incident line alongside falling escalation can signal silence rather than control. A portfolio of easy cases can inflate average quality while excluding the population that decides safety.
Targets become especially dangerous in multitask work. Strong reward on a measurable task pulls effort away from important work that is harder to count. A product team paid for release dates may defer architecture, security and transfer. A vendor paid for adoption may make licence use easy while leaving evaluation, incidents and exit work to the buyer. A service team judged on handling time may stop checking sources just when AI makes confident error cheap.
| Visible target | Likely gaming or neglect | Shifted cost or silenced fact | Leading signal |
|---|---|---|---|
| Active licences | Shallow or unsanctioned use. | Security and review burden. | Growth in unreviewed workflows and exception requests. |
| Pilot or release count | Easy projects and immortal weak pilots. | Architecture, documentation and exit debt. | Ageing pilots with no stage decision. |
| Declared hours saved | Optimistic self-reporting. | Corrections, queues and unchanged capacity. | No reconciled change in service or redeployment. |
| Zero incidents | Reclassification and silence. | Unresolved hazards and worker fear. | Falling escalations while exceptions or rework rise. |
| Average quality | Easy-case selection. | Excluded populations and rare severe loss. | Population mix narrows as the score improves. |
Thought experiment: the metric with perfect integrity
Assume every team reports licence use accurately. The number is still a poor reward target because truthful activity does not establish value, safety or learning. Data integrity cannot repair construct validity. The organisation must first decide what consequence it wants, then collect evidence that discriminates that claim.
05 · Bound what learning may consumeThe Safe-to-Learn Budget
The Safe-to-Learn Budget is this article's fourth synthesis. It specifies a bounded amount of reversible cost, operational error, human review, latency, exception volume or experiment capacity that an accountable owner is willing to consume to acquire information. It creates a stop before exploratory enthusiasm becomes an unlimited claim on other teams.
The budget is not a licence for harm. Severe events, prohibited actions, rights violations, unlawful processing, deliberate discrimination and irreversible affected-party loss do not become acceptable because the experiment remains below a numerical allowance. Hard limits sit outside the budget. When a hard limit is crossed, the test stops and the response path begins. When the ordinary budget is exhausted, expansion stops and the team returns to diagnosis, control repair, a narrower test or retirement.
Review capacity belongs inside the budget because human oversight is a scarce operational resource. A test expected to generate more uncertain cases than qualified reviewers can examine is not safe simply because each case is reversible. Latency and exception volume can damage the service before a dramatic incident appears. The accountable owner must know who bears that load and how it will be restored.
06 · Protection is not immunityThe Candour and Accountability Boundary
The Candour and Accountability Boundary is this article's fifth synthesis. It protects prompt, accurate, good-faith reporting of uncertainty, mistakes, near misses, weak controls and adverse outcomes. Where practical, developmental review is separated from disciplinary investigation. Reporters receive protection from retaliation, and affected parties retain a route to correction or remedy.
The boundary also distinguishes events. Human error may call for system redesign, training or recovery. A risky system condition may make error predictable even when individuals try to comply. An at-risk shortcut may have become normal because incentives and workload reward it. Reckless conduct involves conscious disregard of a substantial boundary. Deliberate concealment, fraud or prohibited action remains subject to fair investigation and accountability. Psychological safety means that speaking up is possible; it does not promise that every act will be consequence-free.
Recognition attaches to the useful act: early disclosure, containment, truthful reconstruction, transferable learning and repair. It never attaches to producing an incident. NASA's aviation reporting system and Google SRE postmortem practice are adjacent operating examples, not legal templates for enterprise AI. Their institutional, confidentiality and sector conditions cannot be imported by slogan. Every organisation must define jurisdiction, due process, privilege, data use, worker representation and remedy locally.
Thought experiment: the early report that worsens the quarter
An engineer reports a prompt-injection route before harm. Containment pauses a launch and makes the quarterly score worse. If the engineer and manager absorb the career cost, the next team learns to wait. Immediate recognition, protected review and executive support make early disclosure rational, while independent investigation determines the system cause and required repair.
07 · Do not finalise value before it existsThe Reward Timing Ladder
The Reward Timing Ladder is this article's sixth synthesis. Immediate recognition can acknowledge useful evidence, candour and a safe intervention while facts are fresh. Short-cycle team acknowledgement can recognise reusable artefacts, coaching and controlled learning. Neither should imply that the business outcome has been achieved.
Stage-gate authority or funding follows independent evidence. Deferred outcome reward follows only after adoption, control performance and captured benefit persist for the relevant period. A material reward can be reversed when a claim was misrepresented, an important population was excluded or cost was shifted and concealed. Good-faith reports are not clawed back merely because they reveal a worse result.
Team reward is usually preferable when outcomes depend on product, platform, domain, control and operations contributions. Individual recognition can still identify a valuable disclosure or artefact, but should not appropriate a system result. Vendor incentives need the same alignment: verified service, evidence access, incident duties, reliability and executable exit matter more than seats activated.
| Timing | Suitable recognition | Evidence gate | Failure protection |
|---|---|---|---|
| Immediate | Prompt disclosure, safe intervention, clear uncertainty. | Basic factual review. | Does not reward the incident itself. |
| Short cycle | Reusable test, coaching, transferred artefact. | Another team can use it. | Recognises shared contribution. |
| Stage gate | Authority, resource or next funding. | Independent evidence across all four ledgers. | Stops when learning budget or hard limit is reached. |
| Deferred | Outcome or team reward. | Adoption, control and benefit persist. | Excludes hidden cost and population harm. |
| Reversal | Clawback of a misrepresented claim. | Material omission, concealment or shifted cost proven. | Never punishes a protected good-faith report. |
The strongest rival: formal incentives are too blunt
The strongest rival argues that complex knowledge work cannot be reduced to targets. Paying for safety behaviours can crowd out professional judgement, invite performative reporting and create bureaucracy. Outcomes are joint, delayed and difficult to attribute. A competent manager, healthy professional culture and clear standards may matter far more than a bonus formula.
The rival is substantially right. The response is to use financial incentives sparingly, keep measures few and stage-specific, prefer team and shared outcomes, inspect distortion, and remove a target when it becomes the work. Protected time, authority, staffing, recognition, career credit and managerial attention often create stronger conditions for candour than cash. The contract organises those conditions; it does not manufacture a universal safe-learning score.
08 · Ten worked casesWhat each system teaches people to reveal or hide
Each case specifies the contract, all four ledgers, the main hazard, budget or hard limit, candour classification, reward timing, worker and affected-party consequences, validation, appeal and accountable owner. The decision is sometimes to redesign the measure and sometimes to remove it entirely.
1. Licence activation creates risky shadow work
The desired outcome is useful, controlled work, but the visible target is active licences. Outcome evidence shows no service change; the evidence ledger contains shallow demonstrations; control evidence reveals unsanctioned data movement; learning remains private. The hazard map finds gaming through trivial use and risk shifted to security. There is no sensible learning budget for prohibited data exposure. The usage target is removed, not softened. Teams receive protected discovery time and recognition for bounded evidence. Security and employee channels accept confidential reports; the CIO owns replacement of the metric, with appeal for workers previously judged by activation.
2. Deployment bonuses keep weak pilots alive
The product team gains credit for every release, so projects avoid a stop decision. The contract changes by stage. Exploration recognises discriminating tests; controlled delivery requires reliability and affected-party evidence; deferred reward requires captured outcome; retirement recognises timely closure. The hazard is easy deployment and hidden maintenance. A bounded experiment budget ends at the next stage gate. Good-faith disclosure of a weak result is protected. Portfolio finance validates value, assurance checks controls and the product executive owns the gate. No individual loses career credit for retiring work that failed its predeclared test.
3. Finance claims hours saved without capacity change
Analysts report faster drafting, yet month-end queues, headcount and overtime are unchanged. The outcome ledger therefore records an unconverted time estimate, not benefit. Evidence confirms a task effect; control and learning ledgers show reconciliation and a reusable method. The hazard is optimistic self-reporting and correction labour shifted to reviewers. Team recognition for the evidence can be immediate, but outcome reward is deferred until finance reconciles capacity that was removed, redeployed or used for a named service improvement. The finance director owns validation, and staff may challenge opaque productivity claims drawn from usage analytics.
4. Prompt-injection near miss is reported early
An engineer identifies an attack path and reports it before external harm. The outcome ledger records containment, not a fictional success; evidence reconstructs the route; control records the stop and repair; learning includes a reusable test. The budget pauses while the hard security limit is examined. The report is classified as prompt, good-faith candour. Immediate recognition and protected review reward disclosure and remediation, never the near miss itself. Security independently validates the fix, the product owner handles any affected-party remedy, and the engineering executive owns the next release gate.
5. Zero escalations reveal a culture of silence
A service team appears perfect, but staff say that each escalation damages their rating. The outcome ledger shows rising rework; the evidence ledger shows missing uncertainty; the control ledger exposes unused escalation; learning is blocked. The hazard map identifies under-reporting and reviewer overload. The zero-escalation target is removed. Managers receive a leading signal for appropriate escalation quality and response time, not volume minimisation. Protected worker interviews and representation support fact finding. Service assurance validates the redesigned route, staff can appeal prior ratings and the operations director owns workload repair.
6. A population failure stops release
Average quality passes, but a targeted population slice fails a material acceptance condition. The team stops release. Outcome remains unproven; evidence has discriminated the boundary; control has worked; learning creates a new population test. Because affected-party protection is a hard limit, no remaining experiment allowance permits release. The safe stop receives immediate recognition, while any outcome reward remains absent. Independent validation confirms the slice and worker assessment cannot penalise the stop. The accountable business owner decides whether to redesign, narrow the population or retire the intervention.
7. Vendor adoption reward shifts work to the buyer
The vendor salesperson earns commission on activated seats while the buyer supplies evaluation, exception review and incident response. The buyer's outcome is therefore incomplete and cost has moved rather than disappeared. The hazard map traces correlated adoption targets across both organisations. The contract replaces activation reward with verified service, evidence access, reliability, incident cooperation and executable exit duties. Deferred payment follows buyer validation. Procurement owns the commercial gate, operations can document shifted load, affected services retain remedy, and the vendor can challenge measurements through a defined contract process.
8. An expert transfers evaluation capability
A specialist shares evaluation assets and coaches peers, reducing personal throughput for a quarter. The outcome ledger records stable service; evidence shows other teams can run the tests; controls retain quality; learning removes hero dependency. The main hazard is a throughput target that makes transfer irrational. The team grants protected coaching time, career recognition and a short-cycle shared reward after recipients demonstrate use. Worker data excludes style surveillance. A domain lead and independent quality reviewer validate transfer; the expert can appeal if contribution is ignored by individual ranking.
9. Repeated prohibited data bypass is concealed
An employee repeatedly bypasses a known data control, hides the action and alters records. This is not protected intelligent failure. Outcome and control ledgers record exposure and response; evidence must preserve facts; learning may still improve the system. The hard limit has been crossed. The candour boundary classifies deliberate concealment for fair investigation, without retaliation against colleagues who report it. No reward applies. Privacy, security and HR retain separate duties, affected parties receive lawful remedy, the employee has due process and appeal, and the accountable executive owns containment.
10. A high score is built from easy cases
A high-performing team selects reversible, familiar cases and excludes difficult populations from a composite score. The outcome claim is invalidated; the evidence ledger records selection bias; control evidence shows an assurance gap; learning identifies a sampling repair. The hazard map exposes average-only reward and a tournament incentive. Independent sampling replaces team selection before any reward attaches. The score is redesigned rather than merely reweighted. Workers can inspect and challenge the case mix, affected populations are represented, and the business owner must accept a lower reported score or narrow the authorised population.
09 · Executable instrumentSafe Learning Incentive Designer
The designer is this article's seventh synthesis. It tests the stage, four-ledger evidence, likely distortion, learning allowance, hard limits, candour classification, contribution, reward timing and independent validation. It returns exactly one of five decisions. It does not calculate a universal score, recommend payment per incident or turn a protected report into immunity from deliberate misconduct.
Outcome boundary
Current stage
Reward environment
Four ledgers
Safe-learning behaviour
Hazard map
Population and time
Learning allowance
Candour
Contribution
Reward and timing
Validation and rights
Reward evidence-producing exploration
Incentive Contract: Acquire one named piece of decision-changing evidence inside a reversible boundary.
Outcome ledger: No scale claim; the business decision and affected population remain explicit.
Evidence ledger: A predeclared test discriminates the material uncertainty.
Control ledger: Stop, review, recovery and hard limits are active.
Learning ledger: The test and boundary conditions become reusable.
Hazard Map: Experiment count, activity and easy-case selection are excluded from reward.
Budget or hard limit: A bounded reversible allowance is owned; prohibited consequence remains outside it.
Candour classification: Good-faith uncertainty and negative evidence receive protection.
Reward form and timing: Protected time and immediate evidence recognition; no premature outcome payment.
Deferred validation: Independent review decides whether another gate is justified.
Protection and appeal: Worker data is purpose-limited and affected parties retain remedy and appeal.
Owner and next gate: The product owner chooses a bounded next test, redesign or stop.
10 · The rule for leadersReward the revelation that improves the decision
A safe-learning incentive system does not make every failure admirable. It makes honest evidence, controlled stopping, prompt reporting, reusable knowledge and verified value easier to choose than theatre. It keeps misconduct accountable, protected reporting safe, affected-party remedy visible and worker assessment challengeable.
The failure boundary is explicit. Remove the incentive when it predictably encourages prohibited behaviour, concealment, population exclusion, shifted harm or control bypass. Redesign the metric when construct validity, gaming, attribution or time horizon is weak. Do not attach consequential reward when independent validation, appeal, accountable ownership or resources are absent. Do not let a bounded learning allowance trade away a hard right or severe consequence.
The board-level question is therefore not “How do we motivate AI adoption?” It is: which evidence, control and learning behaviour must become locally rational at this stage, and which proxy must lose power so people can tell us the truth? That decision changes the architecture of enterprise AI before another model, course or dashboard is purchased.
Evidence and limitsSources checked 3 September 2026
NIST AI RMF 1.0 remains the current published framework and is under revision. NIST, UK, OECD and ILO sources support risk culture, accountable practice, workforce context and participation; they do not prove this article's incentive mechanisms. Incentive, goal and motivation research supplies serious warnings about measurable targets, extrinsic reward and generalisation. NASA and Google SRE are adjacent operating examples, not legal or sector-equivalent templates. Jurisdiction, labour law, collective arrangements, reward design and due process require local review.
The Safe Learning Incentive Contract, Four-Ledger Reward System, eight rewardable behaviours, Incentive Hazard Map, Safe-to-Learn Budget, Candour and Accountability Boundary, Reward Timing Ladder and interactive designer are this article's synthesis. The Kerr DOI in the source pack was corrected to the verified primary record.
- NIST, Artificial Intelligence Risk Management Framework.
- NIST AIRC, AI RMF Core.
- NIST AIRC, AI RMF Playbook.
- UK Government, Artificial Intelligence Playbook.
- UK Government, AI Skills Tools Package.
- OECD, The Impact of AI on the Workplace.
- ILO, Artificial Intelligence Adoption and Its Impact on Jobs.
- ILO, Social Dialogue Critical to Ensure Positive Outcomes of AI in the Workplace.
- Kerr, On the Folly of Rewarding A, While Hoping for B.
- Holmström and Milgrom, Multitask Principal-Agent Analyses.
- Ordóñez et al., Goals Gone Wild, working paper.
- Ryan and Deci, Intrinsic and Extrinsic Motivations.
- Deci, Koestner and Ryan, Extrinsic Rewards and Intrinsic Motivation in Education.
- Edmondson, Psychological Safety and Learning Behavior in Work Teams.
- NASA, Aviation Safety Reporting System Overview.
- Google SRE, Error Budget Policy.
- Google SRE, Postmortem Culture.
- Argote and Ingram, Knowledge Transfer.
About this paperIncentives make evidence cheap or expensive to reveal
The contract, four ledgers, rewardable behaviours, hazard map, learning budget, candour boundary, timing ladder and designer make reward design inspectable at the same resolution as the AI work it shapes.