Policy cannot exert force

A policy can name a desirable state. It can prohibit discrimination, demand human review, limit data use or require explanations. The operating system still needs a way to distinguish an allowed transition from a forbidden one. It needs to know whose identity carries authority, which evidence is current, whether a precondition holds, what happened after a command and what to do when the answer is unknown.

This is familiar in other engineering domains. Nancy Leveson’s system-theoretic account of safety treats accidents as failures to enforce constraints across interacting components, rather than as a simple tally of broken parts. Safety is an emergent property of the system in context, and control depends on feedback about the process being controlled. The analogy is useful for AI governance because an individually accurate model can still participate in an unsafe or unjust workflow. The analogy stops where political legitimacy begins: safety analysis cannot decide which social trade-offs are acceptable. Leveson’s control-theoretic account supplies a mechanism, not a moral constitution.

Thought experiment: the policy twins

Hold the policy constant and vary only enforcement. Organisation A publishes a detailed responsible-AI standard. Teams attest annually that they have read it. Models may call tools with the application service account, and reviewers inspect a random sample after execution. Organisation B adopts the same text, but turns each consequential commitment into a release condition, runtime precondition, evidence requirement and recovery obligation.

Suppose both systems produce the same erroneous recommendation. In A, whether harm follows depends largely on operator vigilance and luck. In B, a model that lacks current evidence cannot reach the execution interface; a decision outside delegated limits requires a named approver; an accepted command without a verified effect becomes an unknown outcome rather than a presumed success. The causal difference is not ethical vocabulary. It is the topology of control.

Frameworks already point towards this lifecycle view. The NIST AI Risk Management Framework Core organises work into Govern, Map, Measure and Manage, makes governance cross-cutting, and explicitly says its actions are not a checklist. ISO/IEC 42001 frames an AI management system as interrelated elements that establish policy, objectives and processes, maintained through continual improvement. For high-risk systems, the EU AI Act requires lifecycle risk management and links it with testing, documentation, oversight, robustness, record-keeping and corrective action.

These sources do not prove that every governance problem is soluble by engineering. They support a narrower point: credible governance must survive contact with design, deployment, operation and change. A principle that never becomes a decision right, a test, a constraint, an observation or a remedy remains an aspiration whose effectiveness cannot be assessed.

From commitment to constraint

The translation starts with harm, not with a catalogue of fashionable controls. “Add a human”, “log everything” and “use explainable AI” are interventions without a causal diagnosis. They may help, do nothing or create new failure paths. The engineer first asks which loss matters, which system state makes that loss possible and which interaction must be constrained.

A useful governance control claim can be written as:

Here C is a consequence to a person, institution or environment. H is a hazardous state from which that consequence can plausibly follow. K is the constraint intended to prevent or limit that state. E is the enforcement point. O is the observation and evidence that tell us whether the constraint operated. R is the response and recovery path. A is the accountable owner with authority to intervene. X is the expiry or retest condition after which old evidence is no longer sufficient.

The tuple is practitioner synthesis, not a standard. Its purpose is diagnostic. A control claim with no hazardous state tends to become generic process. A constraint with no enforcement point becomes advice. Enforcement with no observation creates blind confidence. Evidence with no response merely records harm. A recovery plan with no owner is theatre. An assurance claim with no expiry quietly outlives the system it described.

The translation in eight moves

First state the consequence in human or institutional terms. “Hallucination” is a model behaviour; denial of a benefit, disclosure of private data, an unsafe dose or an unchallengeable allegation is a consequence. This distinction prevents teams from treating aggregate model quality as a proxy for every kind of harm.

Second identify the hazardous system state. A false sentence is not always hazardous. A false sentence inserted into an authoritative customer record without provenance is. A biased score is not the whole state. The state may include an unrepresentative data pipeline, an automated threshold, absent appeal and a reviewer who sees only the score.

Third choose a constraint that changes reachability. Examples include “the model cannot write to the source record”, “no adverse action without disclosed evidence and authorised review”, “special-category data cannot enter this tool context”, or “payment release requires two independent approvals”. Constraints should be specific enough to test and broad enough to survive minor implementation changes.

Fourth place enforcement outside the component being constrained. A prompt saying “do not exceed £10,000” is weak if the same model controls the tool call. The limit belongs in a deterministic authority service or tool contract. Human oversight is also a control only when the reviewer has time, information, competence, independence and actual power to stop the action.

Fifth define evidence before execution. Evidence may include source lineage, policy version, identity propagation, test result, approval, typed request, decision receipt and effect receipt. Model cards and dataset documentation make capabilities, intended uses and provenance more inspectable, but they do not prove a particular runtime decision was authorised. Model Cards for Model Reporting and Datasheets for Datasets are valuable documentary primitives whose scope must not be inflated.

Sixth specify the postcondition. “The API returned 200” is not equivalent to “the customer limit changed exactly once to the approved value”. The observer must read authoritative state or consume a trusted event. Ambiguous commit states deserve the explicit label unknown outcome.

Seventh design response and recovery. Detection without a safe stop, correction path, customer remedy or authority withdrawal is incomplete. Recovery may be compensation, rollback, a new decision, evidence reconstruction, a kill switch, model rollback or manual case reconstruction.

Eighth attach ownership and expiry. Control evidence has a half-life. A passing evaluation may cease to apply after a model, prompt, retriever, policy, data entitlement or tool changes. The owner needs a trigger that reopens the claim.

The harm-blocking asymmetry

A common programme begins with the model: foundation model or classical model, open or closed, general-purpose or specialised. Governance should begin with the action and its consequence. The same model output can be harmless in a private draft, consequential when shown to a caseworker, and irreversible when sent directly to a payment, healthcare or access-control system.

The control burden is asymmetric because false negatives and false positives rarely cost the same, because some effects are easier to reverse than others, and because certain harms remain hidden from the operator. A system may have 98 per cent task accuracy and still be unacceptable if the remaining errors cluster around a protected group, if each error closes an account, or if the organisation cannot identify affected people.

Thought experiment: swap only the consequence

Give one model the same instruction and evidence in two settings. In the first, it drafts a paragraph for an analyst’s private notebook. The analyst can delete it, no authoritative record changes and no customer sees it. In the second, the paragraph is sent automatically as the reason for closing a vulnerable customer’s account. Hold model quality constant. Hold the wording constant. Change only the authority and reversibility of the downstream action.

The appropriate control design changes sharply. The first setting may need source links, confidentiality controls and a visible “draft” state. The second needs legally and ethically justified decision rights, verified identity, a complete evidence basis, independent approval, notice, contestability, effect verification, incident detection and remedy. Risk classification cannot stop at the model boundary because consequence is created by the coupled workflow.

This is why impact assessment matters. ISO/IEC 42005 focuses on foreseeable effects on individuals, groups and society across the lifecycle. The OECD AI Incidents and Hazards Monitor methodology similarly distinguishes actual harm from circumstances that could plausibly lead to it. Both framings move analysis from abstract capability towards affected subjects and real outcomes.

Negative control: when the method should disappear

Consider a deterministic centimetre-to-inch converter embedded in an internal prototype. It accepts public synthetic numbers, performs a fixed formula, cannot write to another system and displays its result beside the input. A conventional unit test covers normal values, boundaries and invalid input. Security review covers the small web surface.

Running a full AI impact assessment, convening an ethics panel, building an assurance case and requiring human approval for each conversion would add process without changing a meaningful hazard. The right governance decision is to classify the component correctly and rely on ordinary software controls. Even if a language model generated the code, the released mechanism is deterministic and independently testable.

This negative control matters because the thesis would be vacuous if it justified maximum ceremony everywhere. Engineering governance is proportional when its controls alter a plausible path to consequential harm. If the analysis cannot name that path, the control should be removed, simplified or treated as an ordinary software, privacy or security requirement.

flowchart TD
  A["Two lending assistants, one policy"] --> B["Policy cannot exert force"]
  B --> C["From commitment to constraint"]
  C --> D["The harm-blocking asymmetry"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#B96D00,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

A minimal worked control

Start with a customer-service summariser. An employee selects three synthetic conversation snippets and asks the model to draft a case note. The note remains a proposal until the employee accepts it. The main consequence is not “the model hallucinates”. It is that an unsupported or sensitive claim enters the authoritative customer record and later changes service, eligibility or investigation.

The hazardous state is therefore: a case note with unsupported material or excess personal data is promoted to the system of record without an accountable reviewer seeing its provenance. That state suggests several constraints. The generator may use only entitled source snippets. Every factual sentence must point to at least one source span. Sensitive categories not required for the service purpose must be suppressed before generation. The model has no record-write credential. Promotion requires an authenticated employee to review the draft and its support. The write service accepts a typed record containing the draft hash, source references, reviewer identity and retention class.

The positive case contains a supported claim: “The customer asked to move the payment date to the 28th,” linked to the exact sentence in the transcript. The negative case contains: “The customer is facing insolvency,” inferred from a late-payment discussion but never stated or evidenced. The negative claim fails support validation and remains outside the record. A reviewer can still add a carefully worded observation, but the system records it as the reviewer’s judgement rather than as a source fact.

Notice what the controls do not claim. Source links do not prove that the customer was truthful. Human approval does not eliminate bias. Data minimisation can omit context needed for a fair decision. The system therefore needs complaint and correction paths, sampled review of promoted notes, and monitoring for groups whose records are edited or challenged unusually often. Governance is an argument about bounded risk, not a certificate of moral purity.

A control record

A production-shaped example

Return to the synthetic small-business overdraft renewal. The model’s job is to organise evidence and propose a decision memorandum, not to possess the bank’s lending authority. The design must keep ten unlike objects separate: intent, identity, world state, context, reasoning, authority, action, evidence, outcome, and release and recovery.

Intent is the underwriter’s request to assess a named facility for renewal. Identity is the authenticated employee and the service identities acting on that person’s behalf. World state is the current authoritative account, exposure, collateral and customer status. Context is the permitted evidence assembled for this decision, including policy and its effective version. Reasoning is the model proposal plus deterministic calculations. None of these is authority.

Authority is the bounded right to approve or veto within delegated limits. Action is a typed request to change a facility or record a decision. Evidence is the chain that allows another reviewer to reconstruct why the proposal and approval occurred. Outcome is the verified business state after execution. Release and recovery govern which component versions may run, how behaviour is canaried, how permissions are withdrawn and how an interrupted case is reconstructed.

The decisive engineering choice is the boundary around the model. The model may identify contradictory figures and suggest questions. It may not silently resolve identity conflicts, invent missing financials, choose a policy version or act under ambient service credentials. Deterministic components should calculate ratios and enforce approval limits because those operations have explicit rules and exact failure tests. The model can explain the calculation, but its prose is not the calculation of record.

The underwriter’s review is designed as a decision, not a ceremonial click. The interface shows evidence quality, missing sources, policy exceptions and the proposed action. It does not hide uncertainty behind a single confidence score. Approval carries the person’s delegated authority only within current limits. A larger or unusual facility routes to another decision right rather than asking the model to “be cautious”.

The action contract contains preconditions: customer identity resolved, case version current, approval unexpired, amount within mandate, no unresolved conflict, and no duplicate idempotency key. Execution returns an acceptance receipt, then a separate observer verifies the authoritative limit. If acceptance arrives but readback is unavailable, the system records an unknown outcome and blocks retry until reconciliation. This avoids turning uncertainty into duplicate effects.

This design connects with narrower mechanisms elsewhere in the article library: an autonomy ledger makes delegated permissions legible; trajectory assurance inspects the path rather than only the final answer; and continuous control evidence keeps runtime observations connected to reviewable claims. Governance engineering is the larger causal frame within which those mechanisms earn their place.

Evidence, tests and recovery

A control is not established because its design sounds plausible. It needs evidence that the control exists, operates under expected conditions, fails safely under adverse conditions and remains applicable to the released configuration. This resembles test, evaluation, verification and validation, but the unit of analysis is broader than the model. It includes data, prompts, retrieval, identities, tools, human decisions, interfaces, operational procedures and the resulting effect.

End-to-end internal algorithmic auditing work has argued for accountability across the full lifecycle rather than a late fairness check. Raji and colleagues’ audit framework is useful here because it makes documentation and organisational process part of the audit object. Research on fairness practice also warns that abstract principles and generic checklists can be misused unless they fit real workflows and organisational incentives. A co-design study with 48 practitioners found that checklists could formalise ad hoc work and empower advocates, while still depending on organisational culture and practitioner needs. The fairness-checklist study supports disciplined use, not checkbox compliance.

Test the transfer function, not the slogan

Each test should trace a disturbance through the control. Corrupt a source date and see whether supersession is detected. Remove the approver’s entitlement and see whether execution stops. Return a tool timeout after acceptance and see whether the case becomes an unknown outcome rather than an automatic retry. Poison long-term memory and see whether unverified content can become authoritative evidence. Change the policy version during a paused case and see whether the decision is reconstructed.

These are failure-injection tests. They ask whether a control blocks or contains a causal path, not whether the happy path still works. Ordinary benchmark scores remain useful for model quality, but they do not answer whether permissions attenuate across delegation, whether evidence is reconstructable or whether recovery creates a second harm.

Recovery deserves equal design weight because controls fail. Google’s site-reliability guidance treats incident response as preparation, detection, response and learning, and uses blameless postmortems to improve the system rather than conceal errors. The same operating logic applies to AI-enabled decisions: alerts should name a symptom with decision consequence, responders need authority to contain it, and the post-incident action should change a test, control or release condition. Google SRE’s postmortem guidance is an operational source, not evidence that every organisation implements it successfully.

Recovery can also include remedy to an affected person. A rollback may restore a database value but fail to undo denial, stigma, delay or distress. The response record should therefore ask: who was exposed, can they be identified, what notice or contest is available, what downstream decisions consumed the result, and who has authority to compensate or reconsider?

Build a living claim, not an evidence warehouse

Evidence is useful only in relation to a claim. “We log prompts” is not a governance claim. “Every adverse-action proposal can be reconstructed from entitled sources, policy, calculation, model version, reviewer identity and effect receipt” is. The claim should name its assumptions and defeaters: a missing data lineage field, an unlogged manual override, a model change outside the release manifest or a complaint pattern that suggests hidden group harm.

Goal Structuring Notation and related assurance-case practice offer a useful discipline: make the top claim, supporting strategy, evidence and unresolved context visible. The GSN Standard Working Group maintains the notation. An assurance diagram does not manufacture truth. It makes the argument inspectable, including where evidence is absent or stale.

The release manifest binds evidence to the actual system: model and adapter, prompt, retrieval index, tool schemas, policy package, identity configuration, action contracts, monitoring rules and rollback target. A change to any dependency does not automatically require full recertification. It requires an impact decision: which claims could the change invalidate, and which focused tests restore confidence?

The governance design canvas

The canvas below is a small executable decision instrument. It does not calculate regulatory compliance or ethical acceptability. It tests whether a proposed use has obvious gaps between its consequence, authority and operating controls. The rules are intentionally conservative for high-consequence execution and intentionally light for low-consequence advice.

Load the positive case to assess a source-linked summary proposal that cannot write without review. Load the negative case to assess automatic account closure with weak observability and no recovery. Then alter one causal feature at a time. The output reveals which design change removes a blocker.

Read the output as an argument

A “bounded release” result does not mean safe, fair or compliant. It means the canvas found no listed structural blocker and the design is ready for domain-specific evidence. “Independent review required” means the scenario may proceed only after unresolved controls and value choices receive accountable treatment. “Do not release” means a plausible high-consequence path remains both reachable and insufficiently observable or recoverable.

The canvas deliberately refuses to multiply the six inputs into a pseudo-scientific risk number. Severity, rights, group effects and reversibility are not commensurable quantities by default. An ordinal decision can be more honest: a single missing control may dominate many minor strengths. This is the harm-blocking asymmetry in operational form.

stateDiagram-v2
  [*] --> Claim
  state "Figure 1. A policy is a<br/>reference signal, not a<br/>control loop" as Claim
  state "Thought experiment, the policy<br/>twins" as Test
  state "Figure 2. Same commitment,<br/>different reachable harm" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

What engineering cannot settle

The claim in this article has a dangerous overextension: if governance is engineering, engineers may appear entitled to decide what society should value. They are not. A control architecture can enforce an eligibility rule perfectly while the rule remains unjust. A metric can make disparate outcomes visible without deciding which equality principle should govern. An appeal workflow can be technically robust while remaining inaccessible to people without time, language, representation or trust.

Research on abstraction in fairness makes this boundary concrete. Selbst and colleagues describe traps that arise when technical work abstracts away the social system in which a model acts. The lesson is not to abandon engineering. It is to keep the system boundary honest and include institutions, people, incentives and affected communities where they are causally relevant.

Governance therefore has two linked layers. The legitimacy layer decides purposes, rights, standing, unacceptable harms, acceptable trade-offs and who may decide. Its methods include law, ethics, political accountability, domain expertise, labour and community participation. The operational layer translates authorised choices into decision rights, constraints, evidence, monitoring, release gates, incident response and remedy. Neither layer substitutes for the other.

The separation also prevents a reverse mistake. Declaring a question “ethical” must not exempt the organisation from implementation. Once a legitimate decision says that applicants must receive reasons and a route to contest, engineering must ensure the right person receives the right evidence, the explanation corresponds to the actual decision process, the appeal can suspend or revise an effect, and the organisation can prove that these things happened.

Unknown harms present a further limit. Incident databases, complaints and red teams reveal only some failures. The OECD monitor explicitly notes that publicly reported incidents are a subset. Controls should leave room for anomaly reporting, whistleblowing, external scrutiny and claims that were not anticipated by the original taxonomy. A governance system that recognises only pre-enumerated risks can become very efficient at ignoring new ones.

Metrics also change behaviour. Teams optimise what release gates reward, reviewers learn which boxes unblock delivery, and suppliers tailor evidence to procurement templates. Independent review, sampled re-performance and attention to complaints are safeguards against this Goodhart-like pressure. The most important signal may be a contradiction between a clean dashboard and the lived experience of affected users.

Strongest competing explanation

A credible objection says governance is fundamentally an organisational and legal discipline. Engineering language may narrow accountability to what can be measured, encourage solutionism and let executives delegate moral responsibility to technical teams. There is strong evidence for the organisational part: lifecycle audits, fairness checklists and ethics-tool reviews repeatedly find that workflows, culture, incentives and authority shape whether tools work. Morley and colleagues’ review of ethics tools describes the persistent gap between principles and practice.

The objection defeats a totalising thesis but not the bounded one advanced here. Governance is not only engineering. It becomes ineffective without engineering wherever software mediates evidence, decisions or effects. Legal accountability determines who owes a duty. Organisational governance assigns decision rights. Ethics tests the purpose and treatment of people. Engineering makes those commitments causally present in the runtime and produces evidence when they are not.

What would weaken this thesis?

The thesis would be weakened if organisations with stronger constraint, evidence and recovery mechanisms did not reduce exposure to severe hazardous states, improve detection or shorten effective remedy compared with policy-led controls under similar conditions. It would also weaken if the mechanism mainly shifted harm into unmeasured channels, burdened affected people with more procedural friction, or created such complexity that operators routinely bypassed it.

Those possibilities should be measured. Useful outcomes include hazardous-state exposure per affected decision, time from deviation to containment, proportion of effects with reconstructable evidence, duplicate or unknown outcomes, appeal success and time to remedy, distribution of errors and reversals across groups, control-bypass frequency, and the number of assurance claims invalidated by change. These measures need context and denominators. None is a universal governance score.