AI Research Library

Explanation for Users, Reviewers and Regulators

Rajesh Ranjan Mahapatra

A defensible explanation is an audience-specific view compiled from one evidential decision record, with causal status, uncertainty and routes to challenge preserved across every view.

© 2026 Rajesh Ranjan Mahapatra. 28 min read

Explanation for Users, Reviewers and RegulatorsAll AI Research Library

Explanation for Users, Reviewers and Regulators

A defensible explanation is an audience-specific view compiled from one evidential decision record, with causal status, uncertainty and routes to challenge preserved across every view.

TLDR

  1. There is no universal AI explanation. An affected person, an operational reviewer, a validator and a regulator ask different questions of the same decision.
  2. The reliable mechanism is an explanation compiler: record evidence, rules, model contributions, human judgements, uncertainty and challenge routes once, then project role-specific views.
  3. Every view may change language and depth, yet it must preserve outcome, decisive facts, claim status, material limitations and the route to correct or contest the decision.
  4. Feature importance and counterfactual model outputs do not establish causation or feasible recourse. A fluent narrative can be useful while remaining materially false.
  5. The architecture decision changes from choosing an explanation technique to designing a reviewable decision receipt and testing each audience view against its intended action.
Opening case

One decision, four requests

A lender's system refers an application for manual affordability review. The customer sees only a message: “Your application requires further checks.” A case officer sees a score, three highlighted features and a button marked approve or decline. A validator receives a model card and a SHAP chart. A regulator later asks which facts were used, which policy applied on that day, what the human reviewer changed, and how the applicant could challenge an incorrect commitment recorded in the credit file.

Suppose the organisation responds by sending the same colourful feature-attribution chart to all four people. The chart may be technically well formed. It may even be locally faithful to the model. Yet it does not answer the customer's practical question, the officer's control question, the validator's reliability question or the regulator's reconstruction question. More detail would not cure the mismatch. The recipients need different operations, not different page lengths.

The customer needs to know the outcome, the decisive verified facts, what can be corrected and who will reconsider the case. The officer needs enough evidence to detect a bad input, respect the policy boundary and record any override. The validator needs population performance, stability, limitations and proof that the local explanation changes when the model changes. The regulator may need the complete chain from data provenance through policy and human action to the final effect. Calling all of these products an “explanation” hides the engineering problem.

The central answer is to separate the stable evidential record from its audience-specific presentation. The system should first create a decision receipt that identifies what was observed, what was inferred, what rule or model contributed, what a human decided, what remains uncertain and how the decision can be challenged. A compiler then produces a view for each legitimate purpose. Translation is allowed. Evidential mutation is not.

This distinction follows a broader pattern in explainability research and guidance. Human explanations are selective and contrastive rather than exhaustive. NIST treats role and knowledge as material to the description supplied. The UK Information Commissioner's Office distinguishes process explanations from outcome explanations and recommends layered delivery with access to a competent human. These sources do not prescribe the compiler described here. They establish why one static artefact is unlikely to be meaningful across roles.

Part I

The audience changes the question

An explanation is successful only relative to a task. “Help me understand” can mean help me predict the system, correct a fact, operate a control, validate a mechanism, allocate responsibility or challenge an outcome. Those tasks demand different evidence. The affected person is usually closest to the consequence and furthest from the implementation. The validator is closest to the model evidence and may know nothing about the individual's circumstances. A regulator can require both, plus the institutional record that joins them.

This is why readability alone is an inadequate objective. A plain-language account may be comprehensible yet omit the very fact that can be corrected. A global model description may be accurate yet fail to explain one outcome. A local attribution may describe the model's sensitivity yet say nothing about the policy that turned a score into an adverse action. A complete technical dump may disclose every parameter while making review practically impossible.

Audience differences also create legitimate conflicts. The applicant may need the identity of the data source so that an error can be corrected. A fraud investigator may need that source protected while an inquiry remains active. A validator may need detailed feature distributions without any person's identifiable record. The solution is not to choose one audience as the true recipient. It is to define entitlements, redactions and escalation paths while keeping the underlying factual account consistent.

An audience is therefore part of the explanation specification, not a formatting preference. Before generating a view, the team should write the recipient's task as an observable verb: correct, decide, validate, reconstruct or appeal. It should then name the smallest evidence set that permits that action, the details that must remain hidden, and the route for requesting the next layer. This turns “make it understandable” into a testable interface requirement.

Figure 1. The four-audience explanation lens

Four audience lenses surrounding one decision receipt An affected person, operational reviewer, validator and regulator receive different projections from one central evidential decision receipt. Decision receipt facts · policy · model · human uncertainty · challenge · outcome Affected person What happened? What can I challenge? Reviewer Is the case sound? What must I do? Validator Does it work? Where does it fail? Regulator Can it be reconstructed? Who was accountable?
Conceptual model. The central record remains invariant while selection, vocabulary and depth change by legitimate purpose. A different view is acceptable; a different account of the decisive facts is not.

Four objects hidden inside “why”

The word “why” often collapses four different objects. The first is evidence: the records accepted as facts for this decision, together with source, time, status and provenance. “Verified monthly income was £3,200” is evidence if the source and observation can be inspected. It is not yet a reason.

The second is rationale: the rule, model contribution or human judgement that connected evidence to an outcome. “The verified commitments-to-income ratio exceeded the automatic-decision boundary” is a policy rationale. “The model assigned high weight to recent arrears” is a model rationale. Each describes a decision mechanism, but neither proves that the underlying feature caused the real-world condition the system is trying to assess.

The third is a causal explanation: a claim about what would change in the world under an intervention. It requires assumptions or evidence beyond predictive association. A postcode can be important to a model because it acts as a proxy. Changing the postcode field may change the score while moving house does not change the relevant economic circumstances. Causal language therefore needs a declared causal model, intervention evidence or a tightly bounded physical mechanism.

The fourth is contestability: the practical ability to question the input, reasoning, rule, authority or procedure and obtain a competent response. Contestability is not a sentence appended to an explanation. It is an institutional capability with a channel, owner, permitted grounds, response time and power to correct the record or change the outcome.

Figure 2. Four different journeys from a decision

Evidence, rationale, causal explanation and contestability shown as different paths Evidence traces back to sources, rationale traces through decision logic, causal explanation branches across interventions, and contestability loops to correction or review. Decision outcome + effect Evidence source and status Rationale decision mechanism Causal explanation intervention Contestability review and repair
Conceptual distinction. Evidence asks where a fact came from. Rationale asks how the configured system used it. Causal explanation asks what an intervention would produce. Contestability asks how a person can correct or challenge the decision process.

These objects can support one another without becoming interchangeable. Evidence can support a rationale. A rationale can identify a useful decision counterfactual. A causal study can justify an intervention. A contest process can reveal that evidence was wrong. The claim type should travel with the statement because its verification test changes.

Published evidence

Research on explanation in the social sciences finds that people commonly seek contrastive, selective accounts. Rigorous XAI evaluation work separates functionally grounded tests, human-grounded studies and application-grounded evaluation. NIST similarly distinguishes transparency, explainability and interpretability, and states that descriptions should be tailored to role, knowledge and skill. Together, these results support audience specification before interface design.

Thought experiment: the transparent rule

Consider a benefit calculation with one published rule: pay £100 when verified income is below £1,000, otherwise pay £0. The input record, threshold and calculation are all visible. A post hoc explainer produces a bar chart saying income contributed 92 per cent to the result. The chart is not false, but it adds no useful mechanism. The exact rule is shorter, more faithful and easier to contest.

Now add a gradient-boosted model that estimates fraud risk and silently blocks payment before the benefit rule runs. The same rule explanation becomes incomplete. The decision receipt must reveal the model's role, the evidence it used, the blocking policy and the review route. This negative control matters because it prevents explanation engineering from becoming a default demand for decorative XAI. When the operative mechanism is already intelligible, expose the mechanism itself.

Part II
The argument in one passChart · flow
flowchart LR
  A["One decision, four requests"] --> B["The audience changes the question"]
  B --> C["Plausibility is not faithfulness"]
  C --> D["Compile views from one receipt"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#087A98,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step
A structural route through Explanation for Users, Reviewers and Regulators, from One decision, four requests to Source ledger. Reading use: follow the relation, then test it against the surrounding evidence and limits.

Plausibility is not faithfulness

Humans are skilled at generating coherent reasons after an outcome. Language models make this cheaper. A system can turn a score, a feature list and a policy paragraph into a persuasive narrative within milliseconds. The resulting prose may be clear, empathetic and actionable. None of those qualities proves that it reflects the actual decision path.

Faithfulness asks whether the explanation changes when the mechanism or evidence changes. Usefulness asks whether the recipient can perform the intended task. These are independent axes. A parameter dump can be faithful and unusable. A polished story can be useful-looking and unfaithful. A defensible product needs both, then adds a third test: whether the recipient can act on an error or disagreement.

Figure 3. The seductive quadrant

A phase plane of explanation faithfulness and task usefulness The best explanations are high in both faithfulness and usefulness. Fluent after-stories can appear useful while having low faithfulness. Defensible view mechanism-sensitive and decision-useful Faithful dump accurate detail without task support Seductive story clear and plausible but mechanism-insensitive Decorative output neither faithful nor useful Task usefulness → Mechanism faithfulness →
Illustrative relationship, not measured data. The dangerous quadrant is high apparent usefulness with low faithfulness. Fluency can increase confidence in precisely the explanation that should be rejected.

Thought experiment: the plausible lie

Take an image classifier and an explanation method that paints a heat map around the object. A reviewer finds the map intuitive. Now randomise the model's weights while preserving the input image. If the heat map remains substantially unchanged, the explanation is responding to image structure rather than the learned decision mechanism. Adebayo and colleagues used this logic to show that visual plausibility alone can mislead. The relevant lesson generalises: an explanation must fail when the mechanism it claims to explain has been destroyed.

Run the same intervention on a narrative explainer. Replace the original model with one that reaches the same outcome for a different reason. If the prose remains “The application was declined because recent arrears indicate repayment difficulty,” the story has not earned its causal wording. It may be a generic justification assembled from plausible domain concepts. A competent test compares the explanation against a decision trace, perturbation, model swap or counterfactual execution that should force a change.

This also exposes a weakness in feature-attribution interfaces. LIME and SHAP answer valuable mathematical questions about a fitted model under specific assumptions. They do not automatically answer why an institution adopted the model, why a policy threshold was chosen, whether a feature is correct, whether the relationship is causal or whether changing the feature is feasible. Research has also shown that post hoc explanations can be manipulated to conceal a biased classifier, and that practitioners may over-trust familiar visualisations. An explanation pack must therefore preserve the method's scope rather than laundering its output into a broader claim.

Defensible(Va) = faithful to R ∧ sufficient for Ta ∧ challengeable ∧ proportionate

Here R is the evidential decision receipt. Va is the view compiled for audience a. Ta is the task that audience must perform. Faithfulness means the view does not contradict or outrun the receipt. Sufficiency means it contains the material facts needed for the task. Challengeable means errors and disagreements have an operative route. Proportionate means the detail, privacy exposure and assurance burden match the consequence.

These conditions create two classes of transformation. A permitted transformation can summarise, translate, reorder, layer or redact while preserving meaning and claim status. “The ratio was above the automatic limit” may replace a formula in the first layer, provided the exact values and calculation are available in an entitled layer. A forbidden transformation changes epistemic force: “was associated with” becomes “caused”; “would change this model output” becomes “will improve your outcome”; “the model proposed” becomes “the organisation decided”.

This distinction is useful for generative explanation systems. The language model should never be the source of the reason. It can realise a typed reason into clearer language under constrained vocabulary and citation rules. A second process should compare the generated claims with the receipt, detect unsupported verbs, verify numbers and ensure that uncertainty has not disappeared. When the check fails, the system should return the structured facts or escalate, rather than attempt another increasingly persuasive paraphrase.

The receipt is larger than the explanation

The internal receipt should be deliberately richer than any one public view. It should include identifiers and versions for the model, policy, prompt, retrieval corpus and tool contract that materially shaped the decision. Every evidence item needs source, observation time, effective time where relevant, verification status and any dispute. Every reason needs a claim type and basis. Human action needs identity, authority, timestamp and recorded change. The effect needs a receipt or an explicit unknown-outcome state.

This is not a demand to expose private data, source code or trade secrets to every audience. It is a demand to preserve the accountable record before selective disclosure. Data minimisation can then operate on a complete object. Without that object, “minimal” often means whatever the presenting system happened to retain.

Figure 4. Anatomical cutaway of a decision receipt

Layered anatomy of a decision receipt A decision receipt contains source evidence, derived state, model and policy operations, human authority, uncertainty and outcome verification. Source evidence provenance · time · verification · dispute Derived state calculations · transformations · missingness Decision mechanism model · rule · policy · version · threshold Human authority review · override · reason · accountable actor Outcome and recovery effect receipt · uncertainty · challenge · correction The audience view is a projection of this record, never its substitute.
Practitioner synthesis. The cutaway is an architecture pattern, not a regulatory data schema. Its purpose is to stop a model attribution from standing in for data provenance, policy application, human judgement or verified outcome.

The counterfactual ladder

Counterfactual language is especially prone to overreach. “Had your ratio been 0.35, this automatic rule would not have referred the case” is a decision counterfactual. It can be verified by rerunning the actual rule while holding the declared facts fixed. It says how the configured decision function behaves.

“Reducing monthly commitments to £1,120 would cause approval” is much stronger. Approval may depend on other checks, the reduction may alter future cash flow, and the action may be infeasible or harmful. Causal recourse research makes this distinction explicit: reaching a desired model output is not the same as identifying an intervention that produces the desired real-world state. A useful explanation marks which rung it occupies.

Figure 5. The counterfactual ladder

A ladder from association to feasible recourse Each higher rung needs stronger evidence: attribution, decision counterfactual, causal intervention and feasible recourse. 1. Association or attribution 2. Decision counterfactual 3. Causal intervention 4. Feasible recourse model sensitivity rule re-execution causal assumptions action, agency, cost evidence burden rises
Conceptual ladder. Each rung answers a stronger question and requires additional evidence. A system should never present a lower-rung result with higher-rung language.
Failure boundary

An explanation cannot repair an invalid decision system. It cannot make discriminatory policy acceptable, prove causation from prediction, create an appeal body that does not exist or recover evidence that was never retained. In high-stakes settings, a simpler interpretable model may be preferable to a black box plus a fragile post hoc story.

Part III

Compile views from one receipt

The compiler metaphor is precise. A compiler accepts a structured source, applies declared transformations and rejects invalid programmes. Here the source is the decision receipt. The transformation selects fields, changes vocabulary, layers detail and redacts information according to purpose and entitlement. The validator rejects an explanation that cites missing evidence, converts attribution into causation, offers a counterfactual that was never run or omits a required challenge route.

Let the complete receipt be R. For each audience a, an authorised projection function πa produces the view Va = πa(R). The projection is lossy by design. The affected person should not receive unrelated personal data or proprietary model internals. Yet a set of invariants must survive: the outcome, decisive evidence, status of disputed facts, basis of each reason, material uncertainty and accountable path to correction. Tests should compare every generated sentence against those invariants.

Figure 6. Explanation compilation as constrained projection

A structured receipt passes through validation and audience projection One structured record is validated for claim status and evidence, then projected into four audience-specific views while invariants pass through all views. Decision receipt R typed claims and provenance Validate basis · scope · completeness Person view reason + correction Reviewer view case + control Validator view tests + limits Regulator view lineage + authority Invariants: outcome · decisive facts · claim status · uncertainty · challenge
Architecture pattern. Selection and redaction are explicit compiler operations. The dashed invariant channel represents properties checked across every authorised view, not a second narrative store.

Worked example: an affordability referral

Use a deliberately small case. Verified monthly income is £3,200. Verified monthly commitments are £1,280. The calculated commitments-to-income ratio is 0.40. Policy version 12 allows automatic progression only at or below 0.35. The configured outcome is therefore “refer for manual affordability review”, not “decline”. Two credit-file entries are disputed and excluded pending review.

The affected-person view says which verified facts were decisive, explains that referral is not a final rejection, and gives the secure route to correct commitments or request human review. It may state a bounded decision counterfactual: with the same verified income and other facts held fixed, commitments of £1,120 produce a ratio of 0.35. It must immediately say that this is neither financial advice nor a guarantee of approval.

The operational view adds source records, dispute status, policy clause, model version, missing-data checks and the reviewer's authority. The validator view adds rule-replay tests, model calibration, segment performance, drift, explanation stability and known exclusions. The regulator view can combine those layers with decision rights, timestamps, human interventions and the effect receipt. All four views derive from the same facts. Their differences are justified by task and entitlement.

Figure 7. A threshold explains the referral, not the whole case

Commitments-to-income ratio on a policy threshold line A synthetic ratio of 0.40 lies above a 0.35 automatic-decision limit and triggers manual review, while values at or below 0.35 can progress to other checks. Policy limit 0.35 Observed 0.40 May progress to other checks Manual review 0.00 0.60 The threshold accounts for this routing step only.
Synthetic worked example. Numbers are illustrative. The exact rule supports a faithful local rationale. It does not establish overall affordability, causal welfare effects or final approval.

The executable evidence-pack prototype

The companion Python artefact implements the smallest useful compiler. It defines typed evidence items, reasons, a contest route and one decision receipt. Each reason carries a claim kind and evidential basis. The validator rejects causal language unless the basis is an intervention study. It rejects a decision counterfactual unless the actual rule was executed. It also refuses to build an affected-person view without a contest route.

The positive case compiles the synthetic affordability referral. The negative case tries to transform a SHAP attribution into “Living in this postcode caused the decline.” The programme rejects it. This is intentionally a type-safety demonstration rather than a complete production implementation. A real system would add entitlement checks, temporal policy resolution, privacy redaction, signed provenance, multilingual quality controls and immutable effect receipts.

Open the complete runnable Python prototype

Assumptions: Python 3.10 or later, synthetic data only, no third-party packages. Save as explanation_pack_compiler.py and run python3 explanation_pack_compiler.py.

from __future__ import annotations

from dataclasses import asdict, dataclass
from typing import Literal

Audience = Literal["affected_person", "operational_reviewer", "validator", "regulator"]
ClaimKind = Literal["evidence", "policy_rationale", "decision_counterfactual", "causal"]
Basis = Literal["source_record", "rule_execution", "model_attribution", "intervention_study"]


class ExplanationError(ValueError):
    """Raised when an explanation would overstate or omit material evidence."""


@dataclass(frozen=True)
class EvidenceItem:
    evidence_id: str
    fact: str
    value: str
    source: str
    observed_at: str
    status: Literal["verified", "disputed", "missing"] = "verified"


@dataclass(frozen=True)
class Reason:
    text: str
    kind: ClaimKind
    basis: Basis
    evidence_ids: tuple[str, ...]


@dataclass(frozen=True)
class ContestRoute:
    channel: str
    owner: str
    allowed_requests: tuple[str, ...]


@dataclass(frozen=True)
class DecisionReceipt:
    case_id: str
    outcome: str
    model_version: str
    policy_version: str
    evidence: tuple[EvidenceItem, ...]
    reasons: tuple[Reason, ...]
    uncertainty: str
    human_role: str
    known_limits: tuple[str, ...]
    contest_route: ContestRoute | None


AUDIENCE_FIELDS: dict[Audience, tuple[str, ...]] = {
    "affected_person": (
        "outcome",
        "decisive_evidence",
        "plain_reasons",
        "decision_counterfactuals",
        "uncertainty",
        "contest_route",
    ),
    "operational_reviewer": (
        "outcome",
        "all_evidence",
        "reasons_with_basis",
        "uncertainty",
        "known_limits",
        "human_role",
        "contest_route",
    ),
    "validator": (
        "model_version",
        "policy_version",
        "all_evidence",
        "reasons_with_basis",
        "uncertainty",
        "known_limits",
    ),
    "regulator": (
        "outcome",
        "model_version",
        "policy_version",
        "all_evidence",
        "reasons_with_basis",
        "uncertainty",
        "known_limits",
        "human_role",
        "contest_route",
    ),
}


def _index_evidence(receipt: DecisionReceipt) -> dict[str, EvidenceItem]:
    index = {item.evidence_id: item for item in receipt.evidence}
    if len(index) != len(receipt.evidence):
        raise ExplanationError("Evidence identifiers must be unique.")
    for item in receipt.evidence:
        if not item.source:
            raise ExplanationError(f"Evidence {item.evidence_id} has no source.")
    return index


def validate_receipt(receipt: DecisionReceipt) -> None:
    evidence = _index_evidence(receipt)
    if not receipt.reasons:
        raise ExplanationError("At least one reason is required.")

    for reason in receipt.reasons:
        missing = set(reason.evidence_ids) - set(evidence)
        if missing:
            raise ExplanationError(f"Reason cites unknown evidence: {sorted(missing)}")
        if reason.kind == "causal" and reason.basis != "intervention_study":
            raise ExplanationError(
                "A causal statement needs intervention evidence; attribution is not causation."
            )
        if reason.kind == "decision_counterfactual" and reason.basis != "rule_execution":
            raise ExplanationError(
                "A decision counterfactual must be recomputed against the actual decision rule."
            )


def compile_explanation(receipt: DecisionReceipt, audience: Audience) -> dict[str, object]:
    """Compile an audience view from one validated evidential decision record."""
    validate_receipt(receipt)
    if audience == "affected_person" and receipt.contest_route is None:
        raise ExplanationError("An affected-person pack requires a contest route.")

    evidence_index = _index_evidence(receipt)
    decisive_ids = {
        evidence_id
        for reason in receipt.reasons
        if reason.kind in {"policy_rationale", "decision_counterfactual"}
        for evidence_id in reason.evidence_ids
    }

    available: dict[str, object] = {
        "outcome": receipt.outcome,
        "model_version": receipt.model_version,
        "policy_version": receipt.policy_version,
        "decisive_evidence": [
            asdict(evidence_index[evidence_id]) for evidence_id in sorted(decisive_ids)
        ],
        "all_evidence": [asdict(item) for item in receipt.evidence],
        "plain_reasons": [
            reason.text for reason in receipt.reasons if reason.kind == "policy_rationale"
        ],
        "decision_counterfactuals": [
            reason.text
            for reason in receipt.reasons
            if reason.kind == "decision_counterfactual"
        ],
        "reasons_with_basis": [asdict(reason) for reason in receipt.reasons],
        "uncertainty": receipt.uncertainty,
        "known_limits": list(receipt.known_limits),
        "human_role": receipt.human_role,
        "contest_route": asdict(receipt.contest_route) if receipt.contest_route else None,
    }
    return {field: available[field] for field in AUDIENCE_FIELDS[audience]}


def positive_case() -> DecisionReceipt:
    return DecisionReceipt(
        case_id="SYN-1042",
        outcome="Application referred for manual affordability review",
        model_version="affordability-model-3.4",
        policy_version="affordability-policy-12",
        evidence=(
            EvidenceItem(
                "E1",
                "Verified monthly income",
                "£3,200",
                "payroll feed / record 91",
                "2026-08-28T09:10:00Z",
            ),
            EvidenceItem(
                "E2",
                "Verified monthly commitments",
                "£1,280",
                "credit file / snapshot 55",
                "2026-08-28T09:11:00Z",
            ),
            EvidenceItem(
                "E3",
                "Commitments-to-income ratio",
                "0.40",
                "calculation E2 / E1",
                "2026-08-28T09:11:02Z",
            ),
        ),
        reasons=(
            Reason(
                "The verified commitments-to-income ratio was 0.40, above the policy's 0.35 automatic-decision limit, so the case was referred rather than declined.",
                "policy_rationale",
                "rule_execution",
                ("E1", "E2", "E3"),
            ),
            Reason(
                "Under the current rule, commitments of £1,120 or less would place the ratio at or below 0.35, holding the other verified facts fixed. This is a decision counterfactual, not advice that changing debt will improve overall welfare or guarantee approval.",
                "decision_counterfactual",
                "rule_execution",
                ("E1", "E2", "E3"),
            ),
        ),
        uncertainty="The payroll source is verified; two credit-file entries are under dispute and are excluded pending review.",
        human_role="A qualified reviewer decides whether disputed commitments should be corrected and whether the application can proceed.",
        known_limits=(
            "The model does not establish affordability on its own.",
            "The counterfactual holds all other facts fixed and is not a causal claim.",
        ),
        contest_route=ContestRoute(
            channel="secure case portal",
            owner="affordability review team",
            allowed_requests=(
                "correct an input fact",
                "supply missing evidence",
                "request a human review",
                "challenge policy application",
            ),
        ),
    )


def negative_case() -> DecisionReceipt:
    return DecisionReceipt(
        case_id="SYN-BAD",
        outcome="Declined",
        model_version="opaque-model-1",
        policy_version="unknown",
        evidence=(
            EvidenceItem("E1", "Postcode attribution", "+0.18", "SHAP output", "2026-08-28T09:00:00Z"),
        ),
        reasons=(
            Reason(
                "Living in this postcode caused the decline.",
                "causal",
                "model_attribution",
                ("E1",),
            ),
        ),
        uncertainty="Not assessed",
        human_role="None recorded",
        known_limits=(),
        contest_route=None,
    )


if __name__ == "__main__":
    person_pack = compile_explanation(positive_case(), "affected_person")
    print("POSITIVE CASE")
    print(person_pack)

    print("\nNEGATIVE CASE")
    try:
        compile_explanation(negative_case(), "affected_person")
    except ExplanationError as exc:
        print(f"REJECTED: {exc}")
POSITIVE CASE Compiled affected-person view with outcome, decisive evidence, policy rationale, bounded decision counterfactual, uncertainty and contest route.
NEGATIVE CASE REJECTED: A causal statement needs intervention evidence; attribution is not causation.

A production-shaped case: disagreement enters the record

Now expand the case without pretending it is a deployment. A lending workflow receives payroll data, open-banking transactions, credit-file commitments and declared household costs. A deterministic service calculates ratios. A predictive model estimates the probability of repayment stress. Policy converts these signals into one of three routes: automatic progress, manual review or decline. A qualified reviewer may correct a disputed input, request documents or apply a documented exception.

The explanation problem changes at each event. Before the decision, the applicant should know the categories of data and the role of automated support. At referral, the person needs the decisive case facts and the meaning of “review”. During review, the officer needs provenance, conflicts and permitted actions. After a correction, the system must preserve the original receipt, append the new evidence, rerun affected calculations and issue a new receipt linked to the first. A later regulator should be able to reconstruct what was knowable and authorised at each point, rather than viewing today's corrected state as though it existed yesterday.

Suppose the customer disputes a £240 monthly commitment. The operational reviewer verifies that the account was closed before the decision date. The evidence status changes from verified to superseded, the ratio falls, and the case progresses. The affected-person explanation should say that the record was corrected and the case reconsidered. The validator should see the frequency and outcome of such disputes as a data-quality signal. The regulator may ask whether the organisation's original verification control was adequate. One correction supports four different judgements, all grounded in the same event history.

Figure 8. Explanation is a temporal product

Timeline of a decision, dispute, correction and reconsideration The first decision receipt is preserved. A dispute creates new evidence, a correction, a policy rerun and a linked second receipt. Receipt 1 manual review Dispute £240 commitment New evidence account closed earlier Re-execution calculation + policy Receipt 2 progressed Receipt 2 links to, but never overwrites, Receipt 1
Synthetic temporal sequence. The explanation changes because the evidential state changed. Preserving both receipts allows the organisation to explain the original decision, the correction and the revised outcome without rewriting history.

Tests that belong in the release gate

An explanation view should be tested like a consequential interface, not approved from screenshots. First, run mechanism sensitivity: swap or perturb the model, rule or evidence and confirm that the relevant reason changes. Second, run claim-scope tests: reject causal verbs when only attribution exists, and reject guaranteed recourse when only a model counterfactual exists.

Third, run task tests with the intended audience. Can an affected person identify an incorrect fact and reach a human? Can an officer locate the authoritative source and distinguish referral from decline? Can a validator reproduce the explanation and find known failure segments? Can a regulator trace who had authority to act? Comprehension questions alone are too weak; the recipient should perform the real review or correction task.

Fourth, run consistency and privacy tests. The four views must agree on invariant facts while exposing only entitled information. Fifth, inject failures: stale policy, missing evidence, duplicated identity, inaccessible challenge channel, conflicting human notes and unknown effect. The compiler should stop or mark uncertainty rather than manufacture a smooth account.

Measure the recipient's action, not the story's polish

Explanation evaluation often drifts towards proxy measures such as word count, readability, stated trust or preference. These can diagnose presentation defects, but they do not establish that the explanation improved the consequential task. A clearer adverse-decision notice is valuable only if people can identify the decisive information, recognise what is disputed and reach an effective correction route. A reviewer dashboard is valuable only if it improves detection of wrong evidence or inappropriate policy use without creating automation bias.

A release experiment should therefore assign realistic cases and score outcomes. For affected people, measure correct identification of decisive facts, successful correction attempts, time to competent review and disparity in access across language or disability needs. For operational staff, measure error detection, unsupported overrides, escalation quality and decision time. For validators, measure whether known defects are discovered and whether explanation failures predict model or workflow failures. For regulators, rehearse reconstruction using retained artefacts rather than the system team's verbal walkthrough.

Trust deserves special care. An explanation can increase trust in a bad system, which is a failure rather than a benefit. The target is appropriate reliance: acceptance when evidence is sound, challenge when it is weak, and abstention when the explanation cannot support the decision. Calibration experiments should include both correct and defective cases so that an interface is rewarded for helping recipients discriminate, not merely for making every outcome feel reasonable.

Part IV
How the claim changes under pressureChart · state
stateDiagram-v2
  [*] --> Claim
  state "Figure 1. The four-audience<br/>explanation lens" as Claim
  state "Four objects hidden inside “why”" as Test
  state "Figure 2. Four different<br/>journeys from a decision" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]
The revision loop begins with Figure 1. The four-audience explanation… and keeps counterexamples inside the reasoning path. Reading use: follow the relation, then test it against the surrounding evidence and limits.

Set the explanation burden by consequence

Explanation effort should rise with the harm of error, difficulty of reversal, uncertainty of the mechanism and distance between decision-maker and affected person. A spelling suggestion needs little more than visible control and undo. A credit decline, medical triage recommendation or fraud-account closure demands a reconstructable record, competent review and explicit limits. The same model can therefore need different explanation controls in different uses.

Regulation supplies duties, not a universal visual form. The EU AI Act, for example, contains transparency and information obligations for specified systems and a right to explanation for certain individual decisions under Article 86. Scope and timing remain material. The Commission's current implementation page records that several high-risk dates were extended in 2026. An engineering team should translate the applicable text into evidence fields, owners, effective dates and tests rather than copy a generic “AI Act compliant explanation” badge.

UK guidance offers a useful design vocabulary by separating process and outcome explanations and by identifying rationale, responsibility, data, fairness, safety and performance, and impact. It also recommends layered delivery and access to a competent human. The guidance itself is under review following the Data (Use and Access) Act, which illustrates the temporal requirement: a legal interpretation must be versioned like policy.

Model-risk supervision adds another audience. The Federal Reserve's April 2026 revision emphasises a risk-based approach tailored to model usage and organisational complexity. The Prudential Regulation Authority's current SS1/23 version is effective from April 2026. Neither makes a customer-facing feature chart sufficient. Validation needs evidence of conceptual soundness, performance, limitations, use and governance across the configured decision system.

Disclosure also has a boundary. An explanation pack may contain personal data about other people, security-sensitive controls or commercially protected implementation detail. Faithfulness does not require indiscriminate publication. It requires that redaction be rule-bound, reviewable and unable to hide a material reason. Where direct disclosure is constrained, an independent reviewer may need access to the fuller record, while the affected person receives the decision-relevant fact and an explicit statement that further protected evidence exists and can be considered through the review route.

Figure 9. Explanation burden is shaped by consequence and reversibility

A burden surface for explanation and review Explanation and assurance burden rises as impact increases and reversibility decreases, with uncertainty adding contour bands. Visible reason control and undo Decision receipt review and correction Full assurance case independent validation and replay Impact of an erroneous decision → Difficulty of reversal → higher uncertainty lower uncertainty
Illustrative decision surface. The axes express design factors rather than statutory categories. Uncertainty, vulnerable recipients, scale and weak review capacity can move a use case into a higher assurance region.

The explanation design card

The practical decision instrument below should be completed before choosing SHAP, counterfactuals, natural-language generation or a dashboard. It forces the team to name the recipient's decision and the claim that must be supported. Empty cells are design failures, not documentation to be filled after launch.

Design questionAffected personOperational reviewerValidatorRegulator or external reviewer
TaskUnderstand, correct, contestDecide or escalate safelyTest reliability and limitsReconstruct duty, authority and effect
Minimum outcome evidenceDecision, status and consequenceCase state and postconditionsOutcome distribution and failure segmentsCase receipt plus aggregate controls
Reason objectPlain decisive reasonsEvidence-linked rule, model and human rationaleMechanism tests and explanation validityPolicy lineage and accountable decision chain
UncertaintyWhat remains unresolvedConflicts, missingness, confidence and limitsCalibration, shift and validation gapsKnown limitations and residual risk ownership
ActionCorrection, human review, appealApprove, decline, request evidence, escalateRestrict, remediate, revalidateInvestigate, require evidence or corrective action
Prohibited substitutionFeature chart instead of a challenge routeScore instead of case evidencePlausible visual instead of mechanism testsModel card instead of decision reconstruction

Audience-specific views, shared claims discipline

For an affected person, relevance and agency dominate. State the outcome in ordinary language, distinguish final from provisional decisions, name the material facts and disclose meaningful uncertainty. Give a human route with enough authority to correct evidence and reconsider the decision. Avoid presenting model sensitivity as personal advice. Vulnerability, language, disability and emotional context should shape delivery without diluting the underlying facts.

For an operational reviewer, explanation is part of the control surface. Show authoritative evidence beside derived values. Separate the model proposal from policy and human decision. Make conflicting or stale data visually hard to miss. State what the reviewer may change, what requires escalation and what evidence must be recorded. Measure whether the interface improves error detection, not whether officers report liking it.

For a validator, local explanations are test instruments. Demand model and data sensitivity, reproducibility, stability under irrelevant perturbation, robustness under correlated features and comparison with simpler interpretable baselines. Test the complete decision function, including thresholds, retrieval, policy and human use. A method that explains a model while ignoring the rule that dominates outcomes is locally accurate and operationally misleading.

For a regulator or external reviewer, the evidence pack must support reconstruction and accountability. Preserve the effective policy, model and data state at the decision time. Identify the legal or policy duty being evidenced, the actor who approved the use, the person who exercised judgement and the route by which an affected person could obtain redress. Aggregate monitoring and individual case review answer different questions; both may be necessary.

1. Name the task

What decision or correction must this audience perform after seeing the explanation?

2. Type every claim

Evidence, policy rationale, model attribution, decision counterfactual, causal claim or open uncertainty.

3. Bind claims to sources

Every material reason points to evidence, version and observation time.

4. Test the mechanism

The explanation changes when the relevant model, rule or evidence changes.

5. Exercise the challenge route

A real user can correct a fact and obtain competent reconsideration.

6. Replay the decision

An independent reviewer can reconstruct the result from the retained state.

What would weaken this proposal?

The compiler pattern adds schema, retention, testing and governance cost. It is unnecessary where decisions are trivial, reversible and already governed by a short transparent rule. It would also be weakened if controlled studies showed that a single explanation artefact reliably supports all four audience tasks without producing systematic misunderstanding or privacy loss. The present evidence points in the opposite direction, but the architecture should still earn its cost by reducing review time, correction failure, unsupported causal language and unexplained divergence across views.

Another objection is that explanation selection itself can become paternalistic. An organisation might decide that users need only a simplified story while reserving decisive detail for insiders. The remedy is not identical disclosure. It is layering with invariants and escalation: provide the most relevant explanation first, make additional entitled detail available, preserve the same factual core and give the recipient a route to a competent human. Audience adaptation should reduce cognitive burden, never reduce contestable rights.

Conclusion

Build the receipt before the story

The common question “Which XAI method should we use?” starts too late. It assumes the model output is the decision, the feature attribution is the reason and the recipient merely needs a clearer picture. Consequential systems are larger. Evidence is collected and transformed. Models propose. Policies route. Humans review. Tools act. Outcomes may remain unknown. An explanation that begins from one algorithmic output cannot recover this chain after the fact.

The architecture decision is to make explanation a compiled view of a reviewable decision receipt. The receipt records the facts, versions, claim types, uncertainty, authority, action and outcome. The compiler selects and translates for a legitimate audience while preserving invariants. Validation blocks unsupported causal language, unexecuted counterfactuals, missing provenance and absent challenge routes. Human testing asks whether the recipient can perform the real task.

This design changes what teams measure. For users, success is correction and informed challenge, not satisfaction with a chart. For reviewers, it is error detection and disciplined authority, not faster acceptance. For validators, it is mechanism sensitivity and failure discovery, not visual plausibility. For regulators, it is reconstruction and accountable remedy, not document volume.

The strongest explanation may sometimes be a short transparent rule. In other cases it will be a layered pack joining data, model, policy, human judgement and temporal evidence. The form varies. The discipline does not: say what the evidence supports, preserve what the audience needs to act, and never let a fluent story acquire authority the decision record cannot bear.

The evidence conversationChart · sequence
sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: One decision, four requests
  M->>E: Figure 1. The four-audience explanation lens
  E-->>M: compare rival explanations
  E->>D: Source ledger
  D-->>Q: revise scope or act
Question, mechanism, evidence and decision exchange the burdens developed in Explanation for Users, Reviewers and Regulators. Reading use: follow the relation, then test it against the surrounding evidence and limits.

Source ledger

Official and regulatory sources

  1. NIST, Four Principles of Explainable Artificial Intelligence. Principles of explanation, meaningfulness, accuracy and knowledge limits.
  2. NIST AI Risk Management Framework, trustworthiness characteristics. Role-specific transparency, explainability and interpretability.
  3. ICO and The Alan Turing Institute, What goes into an explanation?. Process and outcome explanations, explanation types and audiences.
  4. ICO and The Alan Turing Institute, Task 6: build and present the explanation. Layering, timing and dialogue with a competent human.
  5. Regulation (EU) 2024/1689, Artificial Intelligence Act. Current legal text; applicability depends on system, role, scope and effective date.
  6. European Commission, AI Act implementation framework. Current application timeline and enforcement context.
  7. Federal Reserve, SR 26-2 revised guidance on model risk management. Risk-based supervisory approach, issued April 2026.
  8. Prudential Regulation Authority, SS1/23 model risk management principles. Current supervisory statement effective April 2026.

Primary research

  1. Miller, Explanation in artificial intelligence: insights from the social sciences. Contrastive and selective properties of human explanation.
  2. Doshi-Velez and Kim, Towards a rigorous science of interpretable machine learning. Evaluation levels and task-grounded interpretability.
  3. Adebayo et al., Sanity Checks for Saliency Maps. Model and data randomisation tests.
  4. Slack et al., Fooling LIME and SHAP. Post hoc explanations can conceal biased classifiers.
  5. Rudin, Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Interpretable-model baseline and post hoc limits.
  6. Kaur et al., Interpreting interpretability. How data scientists use and misuse interpretability tools.
  7. Poursabzi-Sangdeh et al., Manipulating and measuring model interpretability. Human decision experiments and the limits of apparent transparency.
  8. Ustun, Spangher and Liu, Actionable recourse in linear classification. Formal recourse for actionable changes.
  9. Karimi, Schölkopf and Valera, Algorithmic recourse: from counterfactual explanations to interventions. Separation of counterfactual endpoints from causal interventions.
  10. Cobbe, Lee and Singh, Reviewable automated decision-making. Reviewability beyond model-centric explanation.
  11. Maxwell et al., Meaningful XAI based on user-centric design methodology. Explanation purposes and recipient classes.
  12. Lundberg and Lee, A unified approach to interpreting model predictions. SHAP's feature-attribution framework.
  13. Ribeiro, Singh and Guestrin, Why should I trust you?. LIME and local surrogate explanations.

Practitioner synthesis and worked material

  1. The four-audience lens, typed decision receipt, explanation compiler and release card are original practitioner synthesis in this article. They are proposed patterns, not reported deployments.
  2. The affordability examples, numbers and timelines are synthetic. They illustrate claim discipline and are not customer, lender or supervisory evidence.
  3. Evidence-First UX: Showing Claims with Their Provenance. Adjacent treatment of claim provenance.
  4. Model-Risk Evidence for Agentic Systems. Adjacent treatment of review and trajectory evidence.