Explanation for Users, Reviewers and Regulators
A defensible explanation is an audience-specific view compiled from one evidential decision record, with causal status, uncertainty and routes to challenge preserved across every view.
TLDR
- There is no universal AI explanation. An affected person, an operational reviewer, a validator and a regulator ask different questions of the same decision.
- The reliable mechanism is an explanation compiler: record evidence, rules, model contributions, human judgements, uncertainty and challenge routes once, then project role-specific views.
- Every view may change language and depth, yet it must preserve outcome, decisive facts, claim status, material limitations and the route to correct or contest the decision.
- Feature importance and counterfactual model outputs do not establish causation or feasible recourse. A fluent narrative can be useful while remaining materially false.
- The architecture decision changes from choosing an explanation technique to designing a reviewable decision receipt and testing each audience view against its intended action.
One decision, four requests
A lender's system refers an application for manual affordability review. The customer sees only a message: “Your application requires further checks.” A case officer sees a score, three highlighted features and a button marked approve or decline. A validator receives a model card and a SHAP chart. A regulator later asks which facts were used, which policy applied on that day, what the human reviewer changed, and how the applicant could challenge an incorrect commitment recorded in the credit file.
Suppose the organisation responds by sending the same colourful feature-attribution chart to all four people. The chart may be technically well formed. It may even be locally faithful to the model. Yet it does not answer the customer's practical question, the officer's control question, the validator's reliability question or the regulator's reconstruction question. More detail would not cure the mismatch. The recipients need different operations, not different page lengths.
The customer needs to know the outcome, the decisive verified facts, what can be corrected and who will reconsider the case. The officer needs enough evidence to detect a bad input, respect the policy boundary and record any override. The validator needs population performance, stability, limitations and proof that the local explanation changes when the model changes. The regulator may need the complete chain from data provenance through policy and human action to the final effect. Calling all of these products an “explanation” hides the engineering problem.
The central answer is to separate the stable evidential record from its audience-specific presentation. The system should first create a decision receipt that identifies what was observed, what was inferred, what rule or model contributed, what a human decided, what remains uncertain and how the decision can be challenged. A compiler then produces a view for each legitimate purpose. Translation is allowed. Evidential mutation is not.
This distinction follows a broader pattern in explainability research and guidance. Human explanations are selective and contrastive rather than exhaustive. NIST treats role and knowledge as material to the description supplied. The UK Information Commissioner's Office distinguishes process explanations from outcome explanations and recommends layered delivery with access to a competent human. These sources do not prescribe the compiler described here. They establish why one static artefact is unlikely to be meaningful across roles.
The audience changes the question
An explanation is successful only relative to a task. “Help me understand” can mean help me predict the system, correct a fact, operate a control, validate a mechanism, allocate responsibility or challenge an outcome. Those tasks demand different evidence. The affected person is usually closest to the consequence and furthest from the implementation. The validator is closest to the model evidence and may know nothing about the individual's circumstances. A regulator can require both, plus the institutional record that joins them.
This is why readability alone is an inadequate objective. A plain-language account may be comprehensible yet omit the very fact that can be corrected. A global model description may be accurate yet fail to explain one outcome. A local attribution may describe the model's sensitivity yet say nothing about the policy that turned a score into an adverse action. A complete technical dump may disclose every parameter while making review practically impossible.
Audience differences also create legitimate conflicts. The applicant may need the identity of the data source so that an error can be corrected. A fraud investigator may need that source protected while an inquiry remains active. A validator may need detailed feature distributions without any person's identifiable record. The solution is not to choose one audience as the true recipient. It is to define entitlements, redactions and escalation paths while keeping the underlying factual account consistent.
An audience is therefore part of the explanation specification, not a formatting preference. Before generating a view, the team should write the recipient's task as an observable verb: correct, decide, validate, reconstruct or appeal. It should then name the smallest evidence set that permits that action, the details that must remain hidden, and the route for requesting the next layer. This turns “make it understandable” into a testable interface requirement.
Figure 1. The four-audience explanation lens
Four objects hidden inside “why”
The word “why” often collapses four different objects. The first is evidence: the records accepted as facts for this decision, together with source, time, status and provenance. “Verified monthly income was £3,200” is evidence if the source and observation can be inspected. It is not yet a reason.
The second is rationale: the rule, model contribution or human judgement that connected evidence to an outcome. “The verified commitments-to-income ratio exceeded the automatic-decision boundary” is a policy rationale. “The model assigned high weight to recent arrears” is a model rationale. Each describes a decision mechanism, but neither proves that the underlying feature caused the real-world condition the system is trying to assess.
The third is a causal explanation: a claim about what would change in the world under an intervention. It requires assumptions or evidence beyond predictive association. A postcode can be important to a model because it acts as a proxy. Changing the postcode field may change the score while moving house does not change the relevant economic circumstances. Causal language therefore needs a declared causal model, intervention evidence or a tightly bounded physical mechanism.
The fourth is contestability: the practical ability to question the input, reasoning, rule, authority or procedure and obtain a competent response. Contestability is not a sentence appended to an explanation. It is an institutional capability with a channel, owner, permitted grounds, response time and power to correct the record or change the outcome.
Figure 2. Four different journeys from a decision
These objects can support one another without becoming interchangeable. Evidence can support a rationale. A rationale can identify a useful decision counterfactual. A causal study can justify an intervention. A contest process can reveal that evidence was wrong. The claim type should travel with the statement because its verification test changes.
Research on explanation in the social sciences finds that people commonly seek contrastive, selective accounts. Rigorous XAI evaluation work separates functionally grounded tests, human-grounded studies and application-grounded evaluation. NIST similarly distinguishes transparency, explainability and interpretability, and states that descriptions should be tailored to role, knowledge and skill. Together, these results support audience specification before interface design.
Thought experiment: the transparent rule
Consider a benefit calculation with one published rule: pay £100 when verified income is below £1,000, otherwise pay £0. The input record, threshold and calculation are all visible. A post hoc explainer produces a bar chart saying income contributed 92 per cent to the result. The chart is not false, but it adds no useful mechanism. The exact rule is shorter, more faithful and easier to contest.
Now add a gradient-boosted model that estimates fraud risk and silently blocks payment before the benefit rule runs. The same rule explanation becomes incomplete. The decision receipt must reveal the model's role, the evidence it used, the blocking policy and the review route. This negative control matters because it prevents explanation engineering from becoming a default demand for decorative XAI. When the operative mechanism is already intelligible, expose the mechanism itself.
flowchart LR A["One decision, four requests"] --> B["The audience changes the question"] B --> C["Plausibility is not faithfulness"] C --> D["Compile views from one receipt"] D -. "re-test" .-> A classDef first fill:#0a1424,stroke:#087A98,stroke-width:2px,color:#e6f3ff classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff class A first class B,C,D step
Plausibility is not faithfulness
Humans are skilled at generating coherent reasons after an outcome. Language models make this cheaper. A system can turn a score, a feature list and a policy paragraph into a persuasive narrative within milliseconds. The resulting prose may be clear, empathetic and actionable. None of those qualities proves that it reflects the actual decision path.
Faithfulness asks whether the explanation changes when the mechanism or evidence changes. Usefulness asks whether the recipient can perform the intended task. These are independent axes. A parameter dump can be faithful and unusable. A polished story can be useful-looking and unfaithful. A defensible product needs both, then adds a third test: whether the recipient can act on an error or disagreement.
Figure 3. The seductive quadrant
Thought experiment: the plausible lie
Take an image classifier and an explanation method that paints a heat map around the object. A reviewer finds the map intuitive. Now randomise the model's weights while preserving the input image. If the heat map remains substantially unchanged, the explanation is responding to image structure rather than the learned decision mechanism. Adebayo and colleagues used this logic to show that visual plausibility alone can mislead. The relevant lesson generalises: an explanation must fail when the mechanism it claims to explain has been destroyed.
Run the same intervention on a narrative explainer. Replace the original model with one that reaches the same outcome for a different reason. If the prose remains “The application was declined because recent arrears indicate repayment difficulty,” the story has not earned its causal wording. It may be a generic justification assembled from plausible domain concepts. A competent test compares the explanation against a decision trace, perturbation, model swap or counterfactual execution that should force a change.
This also exposes a weakness in feature-attribution interfaces. LIME and SHAP answer valuable mathematical questions about a fitted model under specific assumptions. They do not automatically answer why an institution adopted the model, why a policy threshold was chosen, whether a feature is correct, whether the relationship is causal or whether changing the feature is feasible. Research has also shown that post hoc explanations can be manipulated to conceal a biased classifier, and that practitioners may over-trust familiar visualisations. An explanation pack must therefore preserve the method's scope rather than laundering its output into a broader claim.
Here R is the evidential decision receipt. Va is the view compiled for audience a. Ta is the task that audience must perform. Faithfulness means the view does not contradict or outrun the receipt. Sufficiency means it contains the material facts needed for the task. Challengeable means errors and disagreements have an operative route. Proportionate means the detail, privacy exposure and assurance burden match the consequence.
These conditions create two classes of transformation. A permitted transformation can summarise, translate, reorder, layer or redact while preserving meaning and claim status. “The ratio was above the automatic limit” may replace a formula in the first layer, provided the exact values and calculation are available in an entitled layer. A forbidden transformation changes epistemic force: “was associated with” becomes “caused”; “would change this model output” becomes “will improve your outcome”; “the model proposed” becomes “the organisation decided”.
This distinction is useful for generative explanation systems. The language model should never be the source of the reason. It can realise a typed reason into clearer language under constrained vocabulary and citation rules. A second process should compare the generated claims with the receipt, detect unsupported verbs, verify numbers and ensure that uncertainty has not disappeared. When the check fails, the system should return the structured facts or escalate, rather than attempt another increasingly persuasive paraphrase.
The receipt is larger than the explanation
The internal receipt should be deliberately richer than any one public view. It should include identifiers and versions for the model, policy, prompt, retrieval corpus and tool contract that materially shaped the decision. Every evidence item needs source, observation time, effective time where relevant, verification status and any dispute. Every reason needs a claim type and basis. Human action needs identity, authority, timestamp and recorded change. The effect needs a receipt or an explicit unknown-outcome state.
This is not a demand to expose private data, source code or trade secrets to every audience. It is a demand to preserve the accountable record before selective disclosure. Data minimisation can then operate on a complete object. Without that object, “minimal” often means whatever the presenting system happened to retain.
Figure 4. Anatomical cutaway of a decision receipt
The counterfactual ladder
Counterfactual language is especially prone to overreach. “Had your ratio been 0.35, this automatic rule would not have referred the case” is a decision counterfactual. It can be verified by rerunning the actual rule while holding the declared facts fixed. It says how the configured decision function behaves.
“Reducing monthly commitments to £1,120 would cause approval” is much stronger. Approval may depend on other checks, the reduction may alter future cash flow, and the action may be infeasible or harmful. Causal recourse research makes this distinction explicit: reaching a desired model output is not the same as identifying an intervention that produces the desired real-world state. A useful explanation marks which rung it occupies.
Figure 5. The counterfactual ladder
An explanation cannot repair an invalid decision system. It cannot make discriminatory policy acceptable, prove causation from prediction, create an appeal body that does not exist or recover evidence that was never retained. In high-stakes settings, a simpler interpretable model may be preferable to a black box plus a fragile post hoc story.
Compile views from one receipt
The compiler metaphor is precise. A compiler accepts a structured source, applies declared transformations and rejects invalid programmes. Here the source is the decision receipt. The transformation selects fields, changes vocabulary, layers detail and redacts information according to purpose and entitlement. The validator rejects an explanation that cites missing evidence, converts attribution into causation, offers a counterfactual that was never run or omits a required challenge route.
Let the complete receipt be R. For each audience a, an authorised projection function πa produces the view Va = πa(R). The projection is lossy by design. The affected person should not receive unrelated personal data or proprietary model internals. Yet a set of invariants must survive: the outcome, decisive evidence, status of disputed facts, basis of each reason, material uncertainty and accountable path to correction. Tests should compare every generated sentence against those invariants.
Figure 6. Explanation compilation as constrained projection
Worked example: an affordability referral
Use a deliberately small case. Verified monthly income is £3,200. Verified monthly commitments are £1,280. The calculated commitments-to-income ratio is 0.40. Policy version 12 allows automatic progression only at or below 0.35. The configured outcome is therefore “refer for manual affordability review”, not “decline”. Two credit-file entries are disputed and excluded pending review.
The affected-person view says which verified facts were decisive, explains that referral is not a final rejection, and gives the secure route to correct commitments or request human review. It may state a bounded decision counterfactual: with the same verified income and other facts held fixed, commitments of £1,120 produce a ratio of 0.35. It must immediately say that this is neither financial advice nor a guarantee of approval.
The operational view adds source records, dispute status, policy clause, model version, missing-data checks and the reviewer's authority. The validator view adds rule-replay tests, model calibration, segment performance, drift, explanation stability and known exclusions. The regulator view can combine those layers with decision rights, timestamps, human interventions and the effect receipt. All four views derive from the same facts. Their differences are justified by task and entitlement.
Figure 7. A threshold explains the referral, not the whole case
The executable evidence-pack prototype
The companion Python artefact implements the smallest useful compiler. It defines typed evidence items, reasons, a contest route and one decision receipt. Each reason carries a claim kind and evidential basis. The validator rejects causal language unless the basis is an intervention study. It rejects a decision counterfactual unless the actual rule was executed. It also refuses to build an affected-person view without a contest route.
The positive case compiles the synthetic affordability referral. The negative case tries to transform a SHAP attribution into “Living in this postcode caused the decline.” The programme rejects it. This is intentionally a type-safety demonstration rather than a complete production implementation. A real system would add entitlement checks, temporal policy resolution, privacy redaction, signed provenance, multilingual quality controls and immutable effect receipts.
Open the complete runnable Python prototype
Assumptions: Python 3.10 or later, synthetic data only, no third-party packages. Save as explanation_pack_compiler.py and run python3 explanation_pack_compiler.py.
from __future__ import annotations
from dataclasses import asdict, dataclass
from typing import Literal
Audience = Literal["affected_person", "operational_reviewer", "validator", "regulator"]
ClaimKind = Literal["evidence", "policy_rationale", "decision_counterfactual", "causal"]
Basis = Literal["source_record", "rule_execution", "model_attribution", "intervention_study"]
class ExplanationError(ValueError):
"""Raised when an explanation would overstate or omit material evidence."""
@dataclass(frozen=True)
class EvidenceItem:
evidence_id: str
fact: str
value: str
source: str
observed_at: str
status: Literal["verified", "disputed", "missing"] = "verified"
@dataclass(frozen=True)
class Reason:
text: str
kind: ClaimKind
basis: Basis
evidence_ids: tuple[str, ...]
@dataclass(frozen=True)
class ContestRoute:
channel: str
owner: str
allowed_requests: tuple[str, ...]
@dataclass(frozen=True)
class DecisionReceipt:
case_id: str
outcome: str
model_version: str
policy_version: str
evidence: tuple[EvidenceItem, ...]
reasons: tuple[Reason, ...]
uncertainty: str
human_role: str
known_limits: tuple[str, ...]
contest_route: ContestRoute | None
AUDIENCE_FIELDS: dict[Audience, tuple[str, ...]] = {
"affected_person": (
"outcome",
"decisive_evidence",
"plain_reasons",
"decision_counterfactuals",
"uncertainty",
"contest_route",
),
"operational_reviewer": (
"outcome",
"all_evidence",
"reasons_with_basis",
"uncertainty",
"known_limits",
"human_role",
"contest_route",
),
"validator": (
"model_version",
"policy_version",
"all_evidence",
"reasons_with_basis",
"uncertainty",
"known_limits",
),
"regulator": (
"outcome",
"model_version",
"policy_version",
"all_evidence",
"reasons_with_basis",
"uncertainty",
"known_limits",
"human_role",
"contest_route",
),
}
def _index_evidence(receipt: DecisionReceipt) -> dict[str, EvidenceItem]:
index = {item.evidence_id: item for item in receipt.evidence}
if len(index) != len(receipt.evidence):
raise ExplanationError("Evidence identifiers must be unique.")
for item in receipt.evidence:
if not item.source:
raise ExplanationError(f"Evidence {item.evidence_id} has no source.")
return index
def validate_receipt(receipt: DecisionReceipt) -> None:
evidence = _index_evidence(receipt)
if not receipt.reasons:
raise ExplanationError("At least one reason is required.")
for reason in receipt.reasons:
missing = set(reason.evidence_ids) - set(evidence)
if missing:
raise ExplanationError(f"Reason cites unknown evidence: {sorted(missing)}")
if reason.kind == "causal" and reason.basis != "intervention_study":
raise ExplanationError(
"A causal statement needs intervention evidence; attribution is not causation."
)
if reason.kind == "decision_counterfactual" and reason.basis != "rule_execution":
raise ExplanationError(
"A decision counterfactual must be recomputed against the actual decision rule."
)
def compile_explanation(receipt: DecisionReceipt, audience: Audience) -> dict[str, object]:
"""Compile an audience view from one validated evidential decision record."""
validate_receipt(receipt)
if audience == "affected_person" and receipt.contest_route is None:
raise ExplanationError("An affected-person pack requires a contest route.")
evidence_index = _index_evidence(receipt)
decisive_ids = {
evidence_id
for reason in receipt.reasons
if reason.kind in {"policy_rationale", "decision_counterfactual"}
for evidence_id in reason.evidence_ids
}
available: dict[str, object] = {
"outcome": receipt.outcome,
"model_version": receipt.model_version,
"policy_version": receipt.policy_version,
"decisive_evidence": [
asdict(evidence_index[evidence_id]) for evidence_id in sorted(decisive_ids)
],
"all_evidence": [asdict(item) for item in receipt.evidence],
"plain_reasons": [
reason.text for reason in receipt.reasons if reason.kind == "policy_rationale"
],
"decision_counterfactuals": [
reason.text
for reason in receipt.reasons
if reason.kind == "decision_counterfactual"
],
"reasons_with_basis": [asdict(reason) for reason in receipt.reasons],
"uncertainty": receipt.uncertainty,
"known_limits": list(receipt.known_limits),
"human_role": receipt.human_role,
"contest_route": asdict(receipt.contest_route) if receipt.contest_route else None,
}
return {field: available[field] for field in AUDIENCE_FIELDS[audience]}
def positive_case() -> DecisionReceipt:
return DecisionReceipt(
case_id="SYN-1042",
outcome="Application referred for manual affordability review",
model_version="affordability-model-3.4",
policy_version="affordability-policy-12",
evidence=(
EvidenceItem(
"E1",
"Verified monthly income",
"£3,200",
"payroll feed / record 91",
"2026-08-28T09:10:00Z",
),
EvidenceItem(
"E2",
"Verified monthly commitments",
"£1,280",
"credit file / snapshot 55",
"2026-08-28T09:11:00Z",
),
EvidenceItem(
"E3",
"Commitments-to-income ratio",
"0.40",
"calculation E2 / E1",
"2026-08-28T09:11:02Z",
),
),
reasons=(
Reason(
"The verified commitments-to-income ratio was 0.40, above the policy's 0.35 automatic-decision limit, so the case was referred rather than declined.",
"policy_rationale",
"rule_execution",
("E1", "E2", "E3"),
),
Reason(
"Under the current rule, commitments of £1,120 or less would place the ratio at or below 0.35, holding the other verified facts fixed. This is a decision counterfactual, not advice that changing debt will improve overall welfare or guarantee approval.",
"decision_counterfactual",
"rule_execution",
("E1", "E2", "E3"),
),
),
uncertainty="The payroll source is verified; two credit-file entries are under dispute and are excluded pending review.",
human_role="A qualified reviewer decides whether disputed commitments should be corrected and whether the application can proceed.",
known_limits=(
"The model does not establish affordability on its own.",
"The counterfactual holds all other facts fixed and is not a causal claim.",
),
contest_route=ContestRoute(
channel="secure case portal",
owner="affordability review team",
allowed_requests=(
"correct an input fact",
"supply missing evidence",
"request a human review",
"challenge policy application",
),
),
)
def negative_case() -> DecisionReceipt:
return DecisionReceipt(
case_id="SYN-BAD",
outcome="Declined",
model_version="opaque-model-1",
policy_version="unknown",
evidence=(
EvidenceItem("E1", "Postcode attribution", "+0.18", "SHAP output", "2026-08-28T09:00:00Z"),
),
reasons=(
Reason(
"Living in this postcode caused the decline.",
"causal",
"model_attribution",
("E1",),
),
),
uncertainty="Not assessed",
human_role="None recorded",
known_limits=(),
contest_route=None,
)
if __name__ == "__main__":
person_pack = compile_explanation(positive_case(), "affected_person")
print("POSITIVE CASE")
print(person_pack)
print("\nNEGATIVE CASE")
try:
compile_explanation(negative_case(), "affected_person")
except ExplanationError as exc:
print(f"REJECTED: {exc}")
A production-shaped case: disagreement enters the record
Now expand the case without pretending it is a deployment. A lending workflow receives payroll data, open-banking transactions, credit-file commitments and declared household costs. A deterministic service calculates ratios. A predictive model estimates the probability of repayment stress. Policy converts these signals into one of three routes: automatic progress, manual review or decline. A qualified reviewer may correct a disputed input, request documents or apply a documented exception.
The explanation problem changes at each event. Before the decision, the applicant should know the categories of data and the role of automated support. At referral, the person needs the decisive case facts and the meaning of “review”. During review, the officer needs provenance, conflicts and permitted actions. After a correction, the system must preserve the original receipt, append the new evidence, rerun affected calculations and issue a new receipt linked to the first. A later regulator should be able to reconstruct what was knowable and authorised at each point, rather than viewing today's corrected state as though it existed yesterday.
Suppose the customer disputes a £240 monthly commitment. The operational reviewer verifies that the account was closed before the decision date. The evidence status changes from verified to superseded, the ratio falls, and the case progresses. The affected-person explanation should say that the record was corrected and the case reconsidered. The validator should see the frequency and outcome of such disputes as a data-quality signal. The regulator may ask whether the organisation's original verification control was adequate. One correction supports four different judgements, all grounded in the same event history.
Figure 8. Explanation is a temporal product
Tests that belong in the release gate
An explanation view should be tested like a consequential interface, not approved from screenshots. First, run mechanism sensitivity: swap or perturb the model, rule or evidence and confirm that the relevant reason changes. Second, run claim-scope tests: reject causal verbs when only attribution exists, and reject guaranteed recourse when only a model counterfactual exists.
Third, run task tests with the intended audience. Can an affected person identify an incorrect fact and reach a human? Can an officer locate the authoritative source and distinguish referral from decline? Can a validator reproduce the explanation and find known failure segments? Can a regulator trace who had authority to act? Comprehension questions alone are too weak; the recipient should perform the real review or correction task.
Fourth, run consistency and privacy tests. The four views must agree on invariant facts while exposing only entitled information. Fifth, inject failures: stale policy, missing evidence, duplicated identity, inaccessible challenge channel, conflicting human notes and unknown effect. The compiler should stop or mark uncertainty rather than manufacture a smooth account.
Measure the recipient's action, not the story's polish
Explanation evaluation often drifts towards proxy measures such as word count, readability, stated trust or preference. These can diagnose presentation defects, but they do not establish that the explanation improved the consequential task. A clearer adverse-decision notice is valuable only if people can identify the decisive information, recognise what is disputed and reach an effective correction route. A reviewer dashboard is valuable only if it improves detection of wrong evidence or inappropriate policy use without creating automation bias.
A release experiment should therefore assign realistic cases and score outcomes. For affected people, measure correct identification of decisive facts, successful correction attempts, time to competent review and disparity in access across language or disability needs. For operational staff, measure error detection, unsupported overrides, escalation quality and decision time. For validators, measure whether known defects are discovered and whether explanation failures predict model or workflow failures. For regulators, rehearse reconstruction using retained artefacts rather than the system team's verbal walkthrough.
Trust deserves special care. An explanation can increase trust in a bad system, which is a failure rather than a benefit. The target is appropriate reliance: acceptance when evidence is sound, challenge when it is weak, and abstention when the explanation cannot support the decision. Calibration experiments should include both correct and defective cases so that an interface is rewarded for helping recipients discriminate, not merely for making every outcome feel reasonable.
stateDiagram-v2 [*] --> Claim state "Figure 1. The four-audience<br/>explanation lens" as Claim state "Four objects hidden inside “why”" as Test state "Figure 2. Four different<br/>journeys from a decision" as Revision Claim --> Test: expose mechanism Test --> Decision: evidence survives Test --> Revision: rival survives Revision --> Claim: narrow or rebuild Decision --> [*]
Set the explanation burden by consequence
Explanation effort should rise with the harm of error, difficulty of reversal, uncertainty of the mechanism and distance between decision-maker and affected person. A spelling suggestion needs little more than visible control and undo. A credit decline, medical triage recommendation or fraud-account closure demands a reconstructable record, competent review and explicit limits. The same model can therefore need different explanation controls in different uses.
Regulation supplies duties, not a universal visual form. The EU AI Act, for example, contains transparency and information obligations for specified systems and a right to explanation for certain individual decisions under Article 86. Scope and timing remain material. The Commission's current implementation page records that several high-risk dates were extended in 2026. An engineering team should translate the applicable text into evidence fields, owners, effective dates and tests rather than copy a generic “AI Act compliant explanation” badge.
UK guidance offers a useful design vocabulary by separating process and outcome explanations and by identifying rationale, responsibility, data, fairness, safety and performance, and impact. It also recommends layered delivery and access to a competent human. The guidance itself is under review following the Data (Use and Access) Act, which illustrates the temporal requirement: a legal interpretation must be versioned like policy.
Model-risk supervision adds another audience. The Federal Reserve's April 2026 revision emphasises a risk-based approach tailored to model usage and organisational complexity. The Prudential Regulation Authority's current SS1/23 version is effective from April 2026. Neither makes a customer-facing feature chart sufficient. Validation needs evidence of conceptual soundness, performance, limitations, use and governance across the configured decision system.
Disclosure also has a boundary. An explanation pack may contain personal data about other people, security-sensitive controls or commercially protected implementation detail. Faithfulness does not require indiscriminate publication. It requires that redaction be rule-bound, reviewable and unable to hide a material reason. Where direct disclosure is constrained, an independent reviewer may need access to the fuller record, while the affected person receives the decision-relevant fact and an explicit statement that further protected evidence exists and can be considered through the review route.
Figure 9. Explanation burden is shaped by consequence and reversibility
The explanation design card
The practical decision instrument below should be completed before choosing SHAP, counterfactuals, natural-language generation or a dashboard. It forces the team to name the recipient's decision and the claim that must be supported. Empty cells are design failures, not documentation to be filled after launch.
| Design question | Affected person | Operational reviewer | Validator | Regulator or external reviewer |
|---|---|---|---|---|
| Task | Understand, correct, contest | Decide or escalate safely | Test reliability and limits | Reconstruct duty, authority and effect |
| Minimum outcome evidence | Decision, status and consequence | Case state and postconditions | Outcome distribution and failure segments | Case receipt plus aggregate controls |
| Reason object | Plain decisive reasons | Evidence-linked rule, model and human rationale | Mechanism tests and explanation validity | Policy lineage and accountable decision chain |
| Uncertainty | What remains unresolved | Conflicts, missingness, confidence and limits | Calibration, shift and validation gaps | Known limitations and residual risk ownership |
| Action | Correction, human review, appeal | Approve, decline, request evidence, escalate | Restrict, remediate, revalidate | Investigate, require evidence or corrective action |
| Prohibited substitution | Feature chart instead of a challenge route | Score instead of case evidence | Plausible visual instead of mechanism tests | Model card instead of decision reconstruction |
Audience-specific views, shared claims discipline
For an affected person, relevance and agency dominate. State the outcome in ordinary language, distinguish final from provisional decisions, name the material facts and disclose meaningful uncertainty. Give a human route with enough authority to correct evidence and reconsider the decision. Avoid presenting model sensitivity as personal advice. Vulnerability, language, disability and emotional context should shape delivery without diluting the underlying facts.
For an operational reviewer, explanation is part of the control surface. Show authoritative evidence beside derived values. Separate the model proposal from policy and human decision. Make conflicting or stale data visually hard to miss. State what the reviewer may change, what requires escalation and what evidence must be recorded. Measure whether the interface improves error detection, not whether officers report liking it.
For a validator, local explanations are test instruments. Demand model and data sensitivity, reproducibility, stability under irrelevant perturbation, robustness under correlated features and comparison with simpler interpretable baselines. Test the complete decision function, including thresholds, retrieval, policy and human use. A method that explains a model while ignoring the rule that dominates outcomes is locally accurate and operationally misleading.
For a regulator or external reviewer, the evidence pack must support reconstruction and accountability. Preserve the effective policy, model and data state at the decision time. Identify the legal or policy duty being evidenced, the actor who approved the use, the person who exercised judgement and the route by which an affected person could obtain redress. Aggregate monitoring and individual case review answer different questions; both may be necessary.
1. Name the task
What decision or correction must this audience perform after seeing the explanation?
2. Type every claim
Evidence, policy rationale, model attribution, decision counterfactual, causal claim or open uncertainty.
3. Bind claims to sources
Every material reason points to evidence, version and observation time.
4. Test the mechanism
The explanation changes when the relevant model, rule or evidence changes.
5. Exercise the challenge route
A real user can correct a fact and obtain competent reconsideration.
6. Replay the decision
An independent reviewer can reconstruct the result from the retained state.
What would weaken this proposal?
The compiler pattern adds schema, retention, testing and governance cost. It is unnecessary where decisions are trivial, reversible and already governed by a short transparent rule. It would also be weakened if controlled studies showed that a single explanation artefact reliably supports all four audience tasks without producing systematic misunderstanding or privacy loss. The present evidence points in the opposite direction, but the architecture should still earn its cost by reducing review time, correction failure, unsupported causal language and unexplained divergence across views.
Another objection is that explanation selection itself can become paternalistic. An organisation might decide that users need only a simplified story while reserving decisive detail for insiders. The remedy is not identical disclosure. It is layering with invariants and escalation: provide the most relevant explanation first, make additional entitled detail available, preserve the same factual core and give the recipient a route to a competent human. Audience adaptation should reduce cognitive burden, never reduce contestable rights.
Build the receipt before the story
The common question “Which XAI method should we use?” starts too late. It assumes the model output is the decision, the feature attribution is the reason and the recipient merely needs a clearer picture. Consequential systems are larger. Evidence is collected and transformed. Models propose. Policies route. Humans review. Tools act. Outcomes may remain unknown. An explanation that begins from one algorithmic output cannot recover this chain after the fact.
The architecture decision is to make explanation a compiled view of a reviewable decision receipt. The receipt records the facts, versions, claim types, uncertainty, authority, action and outcome. The compiler selects and translates for a legitimate audience while preserving invariants. Validation blocks unsupported causal language, unexecuted counterfactuals, missing provenance and absent challenge routes. Human testing asks whether the recipient can perform the real task.
This design changes what teams measure. For users, success is correction and informed challenge, not satisfaction with a chart. For reviewers, it is error detection and disciplined authority, not faster acceptance. For validators, it is mechanism sensitivity and failure discovery, not visual plausibility. For regulators, it is reconstruction and accountable remedy, not document volume.
The strongest explanation may sometimes be a short transparent rule. In other cases it will be a layered pack joining data, model, policy, human judgement and temporal evidence. The form varies. The discipline does not: say what the evidence supports, preserve what the audience needs to act, and never let a fluent story acquire authority the decision record cannot bear.
sequenceDiagram participant Q as Question participant M as Mechanism participant E as Evidence participant D as Decision Q->>M: One decision, four requests M->>E: Figure 1. The four-audience explanation lens E-->>M: compare rival explanations E->>D: Source ledger D-->>Q: revise scope or act
Source ledger
Official and regulatory sources
- NIST, Four Principles of Explainable Artificial Intelligence. Principles of explanation, meaningfulness, accuracy and knowledge limits.
- NIST AI Risk Management Framework, trustworthiness characteristics. Role-specific transparency, explainability and interpretability.
- ICO and The Alan Turing Institute, What goes into an explanation?. Process and outcome explanations, explanation types and audiences.
- ICO and The Alan Turing Institute, Task 6: build and present the explanation. Layering, timing and dialogue with a competent human.
- Regulation (EU) 2024/1689, Artificial Intelligence Act. Current legal text; applicability depends on system, role, scope and effective date.
- European Commission, AI Act implementation framework. Current application timeline and enforcement context.
- Federal Reserve, SR 26-2 revised guidance on model risk management. Risk-based supervisory approach, issued April 2026.
- Prudential Regulation Authority, SS1/23 model risk management principles. Current supervisory statement effective April 2026.
Primary research
- Miller, Explanation in artificial intelligence: insights from the social sciences. Contrastive and selective properties of human explanation.
- Doshi-Velez and Kim, Towards a rigorous science of interpretable machine learning. Evaluation levels and task-grounded interpretability.
- Adebayo et al., Sanity Checks for Saliency Maps. Model and data randomisation tests.
- Slack et al., Fooling LIME and SHAP. Post hoc explanations can conceal biased classifiers.
- Rudin, Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Interpretable-model baseline and post hoc limits.
- Kaur et al., Interpreting interpretability. How data scientists use and misuse interpretability tools.
- Poursabzi-Sangdeh et al., Manipulating and measuring model interpretability. Human decision experiments and the limits of apparent transparency.
- Ustun, Spangher and Liu, Actionable recourse in linear classification. Formal recourse for actionable changes.
- Karimi, Schölkopf and Valera, Algorithmic recourse: from counterfactual explanations to interventions. Separation of counterfactual endpoints from causal interventions.
- Cobbe, Lee and Singh, Reviewable automated decision-making. Reviewability beyond model-centric explanation.
- Maxwell et al., Meaningful XAI based on user-centric design methodology. Explanation purposes and recipient classes.
- Lundberg and Lee, A unified approach to interpreting model predictions. SHAP's feature-attribution framework.
- Ribeiro, Singh and Guestrin, Why should I trust you?. LIME and local surrogate explanations.
Practitioner synthesis and worked material
- The four-audience lens, typed decision receipt, explanation compiler and release card are original practitioner synthesis in this article. They are proposed patterns, not reported deployments.
- The affordability examples, numbers and timelines are synthetic. They illustrate claim discipline and are not customer, lender or supervisory evidence.
- Evidence-First UX: Showing Claims with Their Provenance. Adjacent treatment of claim provenance.
- Model-Risk Evidence for Agentic Systems. Adjacent treatment of review and trajectory evidence.