The forecast that breaks when touched

A predictor answers a question inside the distribution that produced its data. “Among missions that received helicopters, what survival rate followed?” is a legitimate predictive question. It can be estimated with enough representative records. “What survival rate would follow if we dispatched a helicopter to this class of missions?” is not the same question with more confidence. It refers to a different data-generating process.

The notation makes the difference compact. P(Y | X = x) conditions on a value that arose through the existing dispatch policy. P(Y | do(X = x)) refers to a modified system in which the action is set. Standard probability can describe the first distribution. It cannot, by itself, tell us which modified distribution the second expression denotes. That requires causal assumptions, experimental design or both. Pearl’s graphical programme made this distinction explicit by separating a variable observed at a value from a variable externally set to that value (Pearl, 1995).

This is why predictive accuracy and causal adequacy can move independently. A model may predict dispatch with near-perfect accuracy because it has reconstructed the triage policy. It may also predict survival well because severity is visible in the record. Yet the joint predictor can remain silent about what survival would be under another dispatch rule. Better calibration, more parameters and a larger passive sample refine the same observational distribution. They do not select one of the causal rooms in the next thought experiment.

The three rooms also expose a limit on causal discovery. Conditional independences can sometimes rule out graphs, temporal order can orient some arrows, and multiple environments can provide additional constraints. Those methods are valuable. The limit is that their conclusions remain conditional on assumptions such as no hidden common causes, faithful statistical patterns, stable mechanisms or a restricted model class. Causal structure is learned through a partnership between data and declared constraints, not by statistics escaping assumptions.

That distinction matters for frontier models. A representation that predicts an object’s motion across many videos may encode useful regularities. To claim that it has learned a manipulable world model, one must ask whether the representation preserves the consequences of changing mass, force or contact while other mechanisms remain stable. Out-of-distribution prediction is suggestive. A controlled intervention that separates rival generators is stronger.

flowchart LR
  A["The helicopter flag"] --> B["The forecast that breaks when<br/>touched"]
  B --> C["The rule that survives the cut"]
  C --> D["Alternative histories need the<br/>same hidden world"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#7B4CD8,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

The rule that survives the cut

Start with an ordinary support service rather than a graph. Cases arrive with an underlying severity. A triage policy decides whether to make a proactive expert contact. The case later resolves or remains open. Severity affects both contact and resolution. Contact may also affect resolution. These are more than four columns. They are four roles in a generating process.

A structural causal model compresses that process into a set of assignments. Let U contain background factors that the model treats as given from outside its boundary, such as latent severity and random disturbances. Let V contain variables generated inside the model, such as contact X and unresolved outcome Y. Each variable has a function that maps its direct causes and an exogenous disturbance to its value. A distribution over U tells us how cases vary. The tuple is often written M = (U, V, F, P(U)) (Peters, Janzing and Schölkopf, 2017).

The equation for contact is not a fitted identity that must remain true in every imaginable world. It describes how contact is generated in the observational regime. A hard intervention do(X = 1) removes that assignment and fixes contact to one. Graphically, incoming arrows to X are cut. The equations for severity and outcome remain unchanged. This surgical semantics underlies the familiar do-operator (Pearl, 1995).

Identification: recovering the altered world

Writing P(Y | do(X=x)) defines the target, but it does not guarantee that historical data identify it. Identification asks whether the interventional quantity can be expressed using distributions we can observe under the stated graph and assumptions. Estimation comes later. This order prevents an impressively precise number from answering an unlicensed question.

In the contact example, suppose severity S is measured before contact and blocks every non-causal path between contact X and outcome Y. Then the intervention risk can be recovered by standardising both contact groups to the same severity mix:

The first factor asks what happened among people with the same recorded severity and contact status. The second restores the target population’s severity distribution instead of accepting the distribution created by triage. With half the population severe and half mild, the formula produces 30 per cent unresolved under contact and 55 per cent under no contact, exactly as the mental calculation did. The equation compresses a reweighting argument; it does not create the causal assumptions that justify it.

Four conditions carry the result. Severity must be measured well enough to remove the relevant common-cause path. Each severity group must contain both contacted and uncontacted cases, otherwise the missing action is unsupported by data. “Contact” must denote the same intervention across records and the proposed policy. Finally, one case’s contact must not change another case’s outcome within the chosen unit boundary. These are versions of exchangeability, positivity, consistency and no interference used throughout modern causal inference (Hernán and Robins, 2020).

Randomisation addresses the first problem by making assignment independent of pre-treatment background in expectation. It does not make every later contrast causal by magic. Non-adherence, attrition, measurement after treatment, spillovers and a poorly specified action can reopen the gap. A useful discipline is to describe the hypothetical trial the observational analysis is trying to emulate: eligibility, assignment, treatment strategy, follow-up, outcome and estimand. The closer the data-generating process is to that target, the smaller the bridge that assumptions must carry.

The potential-outcome bridge

The same decision can be expressed without structural equations. For each case, let Y(1) be the outcome under contact and Y(0) the outcome under no contact. The average causal effect is the population mean of Y(1) − Y(0). Random assignment makes contacted and uncontacted groups comparable in expectation, which is why experimental contrasts can identify average effects. This treatment-based language grew through the work of Neyman and Rubin, and its limits were famously framed as the fact that only one potential outcome is observed for each unit (Rubin, 1972) (Holland, 1986).

Structural models and potential outcomes are not rival slogans. They are often compatible descriptions of the same intervention, but they emphasise different tasks. Potential outcomes force the analyst to define the treatment and estimand. Graphs expose adjustment, mediation and hidden paths. Structural equations additionally specify how unit-level outcomes are generated, which becomes decisive for counterfactuals.

The mechanism-preservation contract

The formal intervention is clean because it says exactly what changes. Real actions are not always surgical. A new contact policy may alter staffing, waiting times, customer expectations and later triage. A drug may affect several pathways. Clamping an activation in a neural network may move it far outside the states on which downstream computation was trained. Each causal query therefore spends an invariance budget: it assumes that some mechanisms remain usable after another is changed.

This paper’s practical proposal is a mechanism-preservation contract. Before estimating an effect, record the intervention, the rule it replaces, the mechanisms presumed invariant, the population and time window, the unit boundary, and the observations that would revoke the claim. It is not a new causal calculus. It is a discipline for exposing the assumptions that a polished effect estimate can hide.

The intervention must be written at the level actually available. Setting contact to one for every case is a hard intervention. Lowering the triage threshold is a policy intervention because contact still depends on severity and capacity, but through a changed rule. Training staff to interpret risk differently is softer still: it alters the conditional distribution of contact rather than fixing its value. These interventions can have different effects even when each raises the overall contact rate to the same percentage.

This yields a practical break condition. An effect of do(X=1) cannot authorise a threshold policy merely because both involve “more contact”. The policy may target a different mix of cases and change waiting time. The causal licence attaches to an intervention family, not to a variable name. The contract should therefore include an implementation description precise enough that another team could tell whether its proposed action is inside or outside the studied family.

Alternative histories need the same hidden world

An intervention asks about a population under a changed rule. A counterfactual asks about a case whose actual history is partly known: “This severe case was contacted and resolved. Would it have remained unresolved without contact?” The word this changes the mathematical burden. We are no longer free to draw a fresh case from the population.

The standard SCM procedure has three steps. Abduction uses observed evidence to update beliefs about the hidden background. Action replaces the equation for the counterfactual intervention. Prediction propagates the inferred background through the modified equations. The first step is why unit-level counterfactuals are more demanding than average effects: the model must say how observations constrain unobserved disturbances.

Preserving the same background does not mean freezing every recorded field. It means reusing the factors treated as upstream of the intervention, then allowing descendants to change through their equations. If contact changes waiting time, later messages and customer behaviour, those variables should move in the alternative history. Holding them fixed would splice together pieces of incompatible worlds. Conversely, changing a pre-existing diagnosis merely because contact changed would violate the model in the other direction.

This also separates two kinds of uncertainty. We may be unsure about the hidden background for this case even with a fixed structural model. We may also be unsure which structural model and cross-world coupling are correct. More factual evidence can reduce the first. It may leave the second untouched. Responsible individual explanations should report both rather than turning model-dependent precision into personal certainty.

Why feature edits are not automatically counterfactuals

Machine-learning explanations often change an input until the prediction flips: reduce declared debt, increase tenure, remove a symptom, alter an image patch. Such a perturbation can reveal decision sensitivity. It becomes a causal counterfactual only if the edited variable names a coherent intervention and the remaining features are generated consistently with it. Changing age while holding every age-dependent variable fixed does not describe a possible person. Editing a latent activation may not correspond to any stable change in the world.

A feature perturbation is an intervention only when its generating and downstream consequences are specified. Otherwise the result belongs in the function ledger: it tells us how the model behaves under an input change, not what would have happened in the represented world.

stateDiagram-v2
  [*] --> Claim
  state "Identification: recovering the<br/>altered world" as Claim
  state "The potential-outcome bridge" as Test
  state "The mechanism-preservation<br/>contract" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Build the model, then try to break its licence

Causal language is useful only when it changes how a system is built and tested. Consider a synthetic service that predicts whether a case will remain open and proposes proactive expert contact. Historical contact decisions came from human triage. The deployed system will change contact frequency. The configured system therefore needs more than a classifier.

Its causal dossier should name the target population, the outcome window, the action actually available, the triage variables that jointly cause action and outcome, the capacity effects created by more contacts, and the policy whose effect is sought. The model proposal may estimate risk, but authority to change contact policy should depend on an identified effect or an experiment, not on risk association alone.

Four gates should remain separate. Definition asks whether the action and outcome are operationally precise. Identification asks whether the desired effect follows from the available data plus assumptions. Transport asks whether the mechanisms and population used for estimation match the intended deployment. Decision combines the effect with capacity, cost, uncertainty and asymmetric harm. A system can pass the first two gates and still fail the last two.

function generateCase(rng, triageBias, benefit) {
  const U = rng() < 0.5 ? 1 : 0;                  // latent severity
  const pContact = 0.5 + (U ? 1 : -1) * 0.45 * triageBias;
  const EX = rng();
  const EY = rng();
  const X = EX < pContact ? 1 : 0;               // observed policy

  const risk = (x) => {
    const untreated = U ? 0.90 : 0.20;
    const reduction = U ? benefit : benefit * 0.25;
    return Math.max(0.01, untreated - reduction * x);
  };

  const outcome = (x) => EY < risk(x) ? 1 : 0;  // same EY in both worlds
  return { U, X, Y: outcome(X), Y1: outcome(1), Y0: outcome(0) };
}

// Observation groups by the policy-generated X.
// Intervention ignores that assignment and evaluates Y1 and Y0 for every case.

Test the licence before trusting the estimate

The smallest useful assurance sequence has four stages. First, reproduce the observational association and the adjusted or experimental estimand from versioned synthetic or historical data. Second, use negative controls and placebo interventions to expose paths the proposed graph says should be absent. Third, run a bounded pilot at the intended policy scale, including realistic capacity and timing. Fourth, read back outcomes and mechanism indicators, rather than only whether the action executed.

For the contact service, useful mechanism indicators include assignment mix by severity, wait time, expert utilisation, contact completion, downstream channel switching and outcomes for untreated cases. If the estimated benefit remains but untreated cases deteriorate as the queue grows, the intervention has created interference. If severity coding changes after staff see the score, a previously pre-treatment covariate has become part of the policy loop. If the same action label hides shorter or lower-quality contacts, consistency has failed.

This protocol does not remove the need for domain judgement. Negative controls can share hidden biases, pilots can be underpowered, and short horizons can miss delayed effects. Its value is narrower: it turns causal assumptions into prospective failure searches. A surprising result can then lower the claim instead of being explained away after rollout.

Four ways the preservation contract breaks

A causal licence must name its boundary before the system crosses it. The most important failures are not mysterious. Each violates a declared or hidden preservation assumption.

Interference breaks unit separation. Causal designs can model spillovers or partial interference, but they need a larger treatment vector and a declared network or cluster boundary (Hudgens and Halloran, 2008). Feedback breaks a one-pass acyclic story. If outcomes change future triage and resources, cycles may lack a unique equilibrium or even a solution unless explicit solvability conditions hold (Bongers et al., 2021).

Transport shift breaks population invariance. An effect learned in one region, interface or operating regime does not automatically carry to another. Transportability analysis makes differences between source and target populations part of the model rather than a footnote (Bareinboim and Pearl, 2016). Representation shift breaks variable meaning. A stable predictor across environments can be evidence for a causal mechanism under assumptions, but invariance alone does not identify hidden causal variables in general (Peters, Bühlmann and Meinshausen, 2016) (Bing et al., 2023).

What causal intervention can establish about a mind

The same discipline matters when studying biological or artificial minds. Ablating a circuit, patching an activation or disrupting a brain region can support a mechanism claim when the intervention is specific, the candidate system boundary is declared and downstream changes are measured. It can show that a component is necessary or sufficient under the tested conditions.

It cannot by itself show that the component constitutes intelligence, consciousness or experience. A broad perturbation may alter many mechanisms; an artificial activation may be off-manifold; a behavioural change may travel through a compensating route. Intervention upgrades correlation to a model-relative causal claim, not to an ontological verdict. Questions about what a system is, or whether anything is experienced, require additional constitutive and evidential bridges.