Problems, Tasks and Environments

Part IFive objects behind one result

Intelligence is always exhibited relative to some structure outside the intelligent system. Herbert Simon’s account of bounded rationality made the structure of the environment part of the explanation of behaviour. Later formal definitions of machine intelligence also coupled an agent’s achievement to a distribution of environments rather than treating performance as an isolated inner property.[1][2] The broad lesson is useful even when one rejects any particular formal measure: a capability claim silently contains a world in which the capability is exercised.

Engineering discussions often compress that world into one word, “task”. That compression is convenient and dangerous. A business outcome, an executable instruction, an API surface, a stochastic process and a scoring rule are not the same object. They can agree, but agreement must be designed.

The following five-object vocabulary keeps those obligations separate.

The problem states what should be achieved and why. It includes the outcome that matters, protected constraints, the population over which the outcome matters and the consequences of error. “Deliver usable medicine” is a problem. “Maximise destination arrivals” is not an equivalent restatement.

The task is the executable obligation presented to the system. It determines available inputs, observations, actions, goals, prohibitions, horizon and stopping conditions. A task is a sample from, or operationalisation of, a problem. It can omit or distort the problem even when its prose sounds faithful. Measurement theory calls attention to precisely this gap between a construct and its operationalisation.[3]

The environment is the process that turns actions into subsequent observations and consequences. It contains the current state, what the system can observe, transition and failure semantics, time, other actors, data sources, tools and exogenous change. In partially observable settings, the observation process is itself part of the environment. Hidden state is not an implementation detail if it changes which action is defensible.

The configured system is the policy under test. For a language-model application this might include the model, prompt, context policy, planner, memory procedure, parser, tool adapters, retry rule and stopping rule. The exact contents depend on the comparison being made.

The evaluator maps episode evidence to a criterion vector or verdict. It may inspect final state, path compliance, cost, human judgement or a hidden performance function. An evaluator is a measurement instrument. It is never made infallible by being executable.

These objects constrain one another but do not collapse into one another. A problem can support many tasks: intact medicine delivery might be operationalised as navigation, route selection, carrier design or human hand-off. One task can appear in many environments: the same navigation obligation can be tested under clear vision, partial observability, moving obstacles or unreliable actuators. One environment can host several tasks, and one evaluator can score several criterion views over the same episode.

This separation reveals three checks that should precede capability testing. Representativeness asks whether the task instances cover the problem population rather than a convenient corner. Solvability asks whether a feasible policy exists under the stated observations, actions and horizon. Identifiability asks whether the evaluator can distinguish acceptable from unacceptable outcomes with the evidence it receives. A system score is difficult to interpret if any of these checks fails.

Solvability is especially easy to overlook. Suppose the courier can observe only its current cell, cannot probe the floor and must finish in six steps. If the only safe route requires eight steps, intact delivery is impossible under the task contract. A failure says nothing discriminating about which of several policies is more intelligent. The experimental design has created an obligation without a feasible action path. Conversely, if the goal can be achieved by exploiting a simulator artefact that production lacks, the task is solvable for the wrong reason.

This simple notation exposes the attribution problem. One failed verdict, y, has several upstream causes. It does not contain enough information to identify which cause mattered. A verdict is an effect, not a diagnosis. Diagnosis requires additional observations, usually produced by intervention.

Where the boundary belongs

There is no universally correct place to draw the system boundary. Suppose a model proposes malformed tool arguments and a wrapper repairs them. If the experiment selects a base model for an already fixed platform, the wrapper belongs to the environment surrounding the model. If the experiment compares two complete products, each wrapper belongs to its product. Both choices are legitimate. Silent switching is not.

A practical rule is: put inside the system every component whose design is allowed to vary in the decision being tested; hold the rest outside and specify it as experimental context. The rule prevents a model comparison from quietly becoming a product comparison, or a product comparison from erasing the infrastructure that made one system succeed.

Attribution is also not blame. A task omission can originate in product discovery, policy interpretation, data labelling or benchmark construction. An environment mismatch can be a broken connector, an unrealistic simulator or an unmodelled human response. The labels identify the contract that must change first. They do not identify a person to punish.

Nor are the labels mutually exclusive. An under-specified task can train a brittle policy, and an evaluator can then hide the brittleness. A misdeclared tool can trigger a poor recovery policy, after which a weak oracle records success. The protocol therefore records a first broken contract and any contributing causes. The first contract is the earliest point where the observed run ceased to be a valid test of the intended claim. Repairing it may expose a second failure that was previously masked.

This ordering matters economically. If the evaluator cannot observe parcel integrity, collecting more model samples only produces more confidently mis-scored episodes. If the environment cannot distinguish a timeout before commitment from a timeout after commitment, prompting the model to “retry carefully” cannot make duplicate effects impossible. The cheapest effective intervention is often outside the model because it repairs the information or authority structure every model would inherit.

Part IIAttribution through intervention

A post-mortem usually starts from correlation. The model changed and the score fell. The website changed and completion fell. A new evaluator rejected more runs. Correlation can nominate a suspect, but it cannot settle attribution when several changes moved together.

The stronger pattern is a controlled contrast. Keep the problem fixed. Hold four of the five objects stable. Change one variable that represents a plausible cause. Then inspect whether the outcome, failure type or relative ranking changes. This does not make every real system a clean laboratory. It does force the reasoning into a falsifiable form.

Four interventions are especially useful:

Task intervention adds, removes or clarifies a requirement while leaving state and system unchanged. If the system’s action is unchanged but the verdict changes because the task now represents the problem, the earlier success was a task artefact.

Environment intervention varies observations, dynamics, tool reliability, identities or exogenous conditions while preserving task intent. If a ranking reverses across plausible instances, a single-instance claim was too broad. Jayawardana et al. demonstrated this concern directly in deep reinforcement learning: evaluation on a few MDP instances can produce method rankings that differ from evaluation over the task’s wider MDP family.[4] Procgen’s procedural levels respond to the same broad problem by making generalisation across environment instances an explicit benchmark target.[5]

System intervention replaces the candidate with a matched baseline under the same task, environment and evaluator. This is the decisive contrast when claiming system failure. The alternative should be simpler where possible, because a complex baseline can conceal a second source of variation.

Evaluator intervention re-scores the same retained episode with independent criterion implementations. A final-state oracle, path-compliance oracle and human rubric can disagree for legitimate reasons. The disagreement must be explained rather than averaged into false certainty.

Single-factor changes are the starting point, not a dogma. Some failures are interactions. A system may tolerate either longer latency or partial observability but fail when both occur together. A task may be adequate under a reliable API yet become ambiguous under partial commits. After the first-pass interventions, a small factorial design can vary two suspected causes together. The important discipline is to predeclare the factors and preserve a common episode schema, so an interaction is observed rather than invented after the score arrives.

Interventions should also target mechanisms, not just difficulty. Randomly making a benchmark harder can lower every score without revealing why. A useful perturbation has a predicted relation: a meaning-preserving paraphrase should leave the task plan unchanged; revealing a previously hidden obstacle should improve a policy that uses observations; changing the authenticated identity should alter accessible resources according to entitlement; adding irrelevant evidence should not expand authority. These are causal probes because the expected response is specified before execution.

Intervention also exposes underspecification inside the system itself. D’Amour et al. showed that a learning pipeline can produce multiple predictors with similarly strong test performance but materially different behaviour under deployment-relevant stress tests.[6] Equal scores do not imply equal mechanisms or equal future failures. The environment family and the evaluator determine which differences become visible.

Small samples add another layer. Deep reinforcement learning results based on a few runs can change interpretation once uncertainty is represented; Agarwal et al. found discrepancies between point-estimate conclusions and fuller statistical analysis across common benchmarks.[7] Patterson et al. similarly emphasise implementation detail, hyperparameters, baselines and statistical evidence in empirical design.[8] A supposed system failure may therefore be sampling noise, while a supposed environment effect may be an unbalanced experimental design.

Two controlled thought experiments

Thought experiment 1: the transparent wall

Place the same navigation policy in two rooms with identical geometry. In both, a glass wall blocks the direct route. In Room A, the sensor reports the wall. In Room B, glare removes it from the observation. The policy walks into the wall only in Room B.

Has the system failed? The answer depends on the observation contract. If the environment promised that obstacles of this class would be observable, Room B violated the contract. If partial observability was declared and the task demanded cautious probing, then the policy may have failed to manage uncertainty. Hidden state alone does not assign responsibility; hidden state relative to the declared information model does.

Now add a third room in which glare is declared but the task gives no safe probing action. Failure may belong to task design because success is not achievable with the permitted interface. Before judging competence, one must establish that the task is solvable under the environment’s observation and action rules.

Thought experiment 2: swap the judge

Freeze the courier’s complete trajectory: six moves, arrival before the limit, cracked vial. Apply three evaluators. The location-only evaluator passes it. The integrity-aware evaluator fails it. A path-compliance evaluator also fails it because the direct corridor was prohibited. Nothing about the physical episode changed.

AI Safety Gridworlds made this separation concrete by equipping environments with an observed reward and a hidden performance function representing intended safe behaviour.[9] Inverse reward design goes further: it treats a designed reward as evidence about the designer’s objective, interpreted in the context where it was written, rather than as the objective itself.[10]

These results do not prove that every task has a recoverable “true” objective. They establish a narrower discipline: the signal optimised by the system and the criterion used to judge the system must not be assumed to exhaust the intended problem.

Part IIIA minimal failure laboratory

The executable artefact turns the opening case into a seven-by-five grid. The courier begins at coordinate (0, 2) and must reach (6, 2). Three direct-route cells vibrate. A shortest-path policy uses six moves and ignores declared risk. A contract-aware policy assigns cost to declared hazard cells and takes an eight-move detour.

The problem always requires an intact delivery. The task, environment and evaluator are varied independently across cases. Because every object is typed, the test can distinguish a task that omitted integrity from an environment whose realised hazards differ from its declared hazards.

The six cases expose different logical relations. In the healthy control, the contract-aware policy avoids a declared hazard and every layer agrees. In the simple-policy negative control, hazards are absent and the shortest policy passes, showing that extra caution is not a free capability gain. In the task-definition case, the shortest policy passes the literal location-only task while failing the real intact-delivery problem. In the environment-contract case, the cautious policy follows the declared detour but encounters an undeclared hazard on that route. In the evaluator case, an eight-step safe delivery satisfies the task yet a mismatched seven-step oracle rejects it.

Notice that the artefact does not infer task or environment defects from behaviour alone. It compares typed declarations. Operational tests rarely possess direct access to “actual” dynamics, so they need instrumented tool gateways, system-of-record readback and fault injection. The laboratory makes that missing evidence visible rather than pretending a trace can answer every question.

The complete standard-library program is embedded below. It runs six diagnostic cases and writes a typed environment specification plus measured synthetic results. The short excerpt is a pseudocode map of the decision rule:

# Pseudocode summary. Open the complete runnable program below.
def diagnose(case, candidate_episode, baseline_episode):
    diagnosis = Diagnosis()

    if case.problem.requires_intact_delivery != \
       case.task.requires_intact_delivery:
        diagnosis.add("task-definition failure", ...)

    if not case.environment.contract_matches_reality:
        diagnosis.add("environment-contract failure", ...)

    if not evaluator_matches_task(case.task, case.evaluator):
        diagnosis.add("evaluator failure", ...)

    aligned = problem_matches_task(case) \
              and case.environment.contract_matches_reality \
              and evaluator_matches_task(case.task, case.evaluator)

    if aligned \
       and not candidate_episode.evaluator_pass \
       and baseline_episode.evaluator_pass:
        diagnosis.add("system failure", ...)

    return diagnosis

The code is not a universal causal discovery method. Its “actual environment” fields are visible because this is a laboratory. In an operational system, mismatches must be inferred through traces, readbacks, fault injection and independent observation. The value of the artefact is that it makes the evidence rule inspectable: system failure is assigned only after problem, task, environment and evaluator are aligned and a matched alternative succeeds.

from __future__ import annotations

import csv
import json
import random
from dataclasses import dataclass
from pathlib import Path

START = (0, 2)
GOAL = (6, 2)
DIRECT_HAZARDS = frozenset({(2, 2), (3, 2), (4, 2)})
RUNS = 2_000
SEED = 20_260_829


@dataclass(frozen=True)
class Case:
    name: str
    problem_requires_intact: bool
    task_requires_intact: bool
    task_step_limit: int
    declared_hazards: frozenset[tuple[int, int]]
    actual_hazards: frozenset[tuple[int, int]]
    damage_probability: float
    evaluator_requires_intact: bool
    evaluator_step_limit: int
    candidate_policy: str
    baseline_policy: str
    expected: str


def path_for(policy: str, declared_hazards: frozenset[tuple[int, int]]):
    direct = [(x, 2) for x in range(7)]
    if policy == "shortest" or not declared_hazards.intersection(direct):
        return direct
    return [START, (0, 1)] + [(x, 1) for x in range(1, 7)] + [GOAL]


def run_policy(case: Case, policy: str, seed: int = SEED):
    path = path_for(policy, case.declared_hazards)
    rng = random.Random(seed)
    damage_trials = [
        rng.random() < case.damage_probability
        for cell in path[1:]
        if cell in case.actual_hazards
    ]
    intact = not any(damage_trials)
    steps = len(path) - 1
    reached = path[-1] == GOAL
    task_pass = (
        reached
, and steps <= case.task_step_limit
, and (intact or not case.task_requires_intact)
    )
    evaluator_pass = (
        reached
, and steps <= case.evaluator_step_limit
, and (intact or not case.evaluator_requires_intact)
    )
    return {
        "policy": policy,
        "steps": steps,
        "intact": intact,
        "task_pass": task_pass,
        "evaluator_pass": evaluator_pass,
    }


def diagnose(case: Case, candidate: dict, baseline: dict):
    labels = []
    if case.problem_requires_intact != case.task_requires_intact:
        labels.append("task-definition failure")
    if case.declared_hazards != case.actual_hazards:
        labels.append("environment-contract failure")
    evaluator_matches = (
        case.evaluator_requires_intact == case.task_requires_intact
, and case.evaluator_step_limit == case.task_step_limit
    )
    if not evaluator_matches:
        labels.append("evaluator failure")

    aligned = not labels
    if aligned and not candidate["evaluator_pass"] and baseline["evaluator_pass"]:
        labels.append("system failure")
    return labels or ["no failure"]


def cases():
    clear = frozenset()
    detour_fault = frozenset({(3, 1)})
    return [
        Case("healthy control", True, True, 8, DIRECT_HAZARDS,
             DIRECT_HAZARDS, 1.0, True, 8,
             "contract-aware", "shortest", "no failure"),
        Case("simple-policy negative control", True, True, 6, clear,
             clear, 0.0, True, 6,
             "shortest", "contract-aware", "no failure"),
        Case("system failure", True, True, 8, DIRECT_HAZARDS,
             DIRECT_HAZARDS, 1.0, True, 8,
             "shortest", "contract-aware", "system failure"),
        Case("task-definition failure", True, False, 6, DIRECT_HAZARDS,
             DIRECT_HAZARDS, 1.0, False, 6,
             "shortest", "contract-aware", "task-definition failure"),
        Case("environment-contract failure", True, True, 8,
             DIRECT_HAZARDS, detour_fault, 1.0, True, 8,
             "contract-aware", "shortest", "environment-contract failure"),
        Case("evaluator failure", True, True, 8, DIRECT_HAZARDS,
             DIRECT_HAZARDS, 1.0, True, 7,
             "contract-aware", "contract-aware", "evaluator failure"),
    ]


def run_cases():
    print("Diagnostic cases")
    for case in cases():
        candidate = run_policy(case, case.candidate_policy)
        baseline = run_policy(case, case.baseline_policy)
        labels = diagnose(case, candidate, baseline)
        assert labels == [case.expected], (case.name, labels)
        print(f"  {case.name}: {labels[0]}")


def hidden_risk_experiment():
    probabilities = [0.0, 0.10, 0.25, 0.50, 0.75, 1.0]
    rows = []
    print("Hidden-risk experiment")
    for probability in probabilities:
        rng = random.Random(SEED)
        intact = 0
        for _ in range(RUNS):
            trials = [rng.random() < probability for _ in range(3)]
            intact += not any(trials)
        integrity_rate = intact / RUNS
        row = {
            "damage_probability": probability,
            "location_pass_rate": 1.0,
            "integrity_pass_rate": integrity_rate,
        }
        rows.append(row)
        print(
            f"  p={probability:.2f}: location=1.000, "
            f"integrity={integrity_rate:.3f}"
        )
    return rows


def write_outputs(rows):
    specification = {
        "population": "seven-by-five courier grids",
        "state": ["courier position", "vial integrity"],
        "observations": ["position", "declared hazard cells"],
        "actions": ["up", "down", "left", "right"],
        "transition": "one orthogonal cell per action",
        "termination": "goal reached or task step limit",
        "fault_model": "independent damage on each actual hazard entry",
        "evaluator": ["destination", "step limit", "integrity when required"],
        "seed": SEED,
        "runs_per_probability": RUNS,
    }
    Path("environment_specification.json").write_text(
        json.dumps(specification, indent=2) + "\n", encoding="utf-8"
    )
    with Path("courier_hidden_risk_results.csv").open(
        "w", newline="", encoding="utf-8"
    ) as handle:
        writer = csv.DictWriter(handle, fieldnames=rows[0].keys())
        writer.writeheader()
        writer.writerows(rows)


if __name__ == "__main__":
    run_cases()
    results = hidden_risk_experiment()
    write_outputs(results)
    print("Wrote environment_specification.json and courier_hidden_risk_results.csv")

Diagnostic cases
  healthy control: no failure
  simple-policy negative control: no failure
  system failure: system failure
  task-definition failure: task-definition failure
  environment-contract failure: environment-contract failure
  evaluator failure: evaluator failure
Hidden-risk experiment
  p=0.00: location=1.000, integrity=1.000
  p=0.10: location=1.000, integrity=0.721
  p=0.25: location=1.000, integrity=0.419
  p=0.50: location=1.000, integrity=0.131
  p=0.75: location=1.000, integrity=0.021
  p=1.00: location=1.000, integrity=0.000
Wrote environment_specification.json and courier_hidden_risk_results.csv

The hidden-risk experiment keeps the direct policy fixed and varies only the probability of damage on each of three hazard entries. A location-only evaluator remains at 100 per cent pass rate because the destination is always reached. The integrity-aware evaluator tracks actual problem success and falls as hidden risk rises.

The negative control: sometimes the system really failed

A useful framework can become an excuse generator. Every bad result can be blamed on task wording, distribution shift or measurement. The antidote is a serious negative control.

First, remove all hazards. In that ordinary corridor, both policies use the six-step route and deliver an intact vial. The more elaborate planner earns no behavioural advantage. This control shows that risk-aware complexity is not intrinsically superior and should not be rewarded where the environment does not require it.

Second, restore the declared vibration strip, require intact delivery and use an integrity-aware evaluator. Under the same observations, actions, damage dynamics and budget, the shortest policy crosses the strip and fails while the contract-aware policy detours and succeeds. The other contracts have survived inspection, and the matched alternative establishes feasibility. Here, “system failure” is the parsimonious attribution.

The conclusion remains bounded. It applies to this policy under this environment family and criterion set. A later instance may reveal an environmental mismatch too. Causes can be plural, but plural causation is not permission to avoid a decisive comparison.

Part IVWrite the environment before testing the intelligence

Consider a synthetic banking service assistant that prepares a replacement-card action for staff approval. It does not execute the action. The intended problem is: identify the authenticated customer’s affected physical card, establish an allowed replacement reason, prepare exactly one correct operation, preserve restricted data and leave an evidence package for a qualified approver.

A compressed task might say, “Replace the customer’s lost card.” That sentence hides almost everything needed to attribute failure. Which customer identity is authoritative? Which product is physical rather than virtual? Is the old card already blocked? Can the assistant read joint-account records? What happens if the card API times out after committing? What evidence permits the operation? What is a valid terminal state?

The environment includes more than a chat window. It contains identity and entitlement services, current world state across customer and card systems, context returned from governed sources, tool schemas, API transition semantics, concurrent updates, time, faults and readback. The configured system turns observations into a model proposal. Authority remains outside it: a deterministic kernel decides whether the proposed action may be prepared. The evaluator needs evidence of identity, policy, selected card, proposed action, approval state and postcondition. This preserves the distinction among intent, identity, world state, context, reasoning, authority, action, evidence, outcome, and release or recovery.

Now consider four failures.

If authoritative state clearly identifies card A and the model proposal selects card B, while a simpler deterministic selector succeeds, the system has failed. If the task omitted the requirement to distinguish physical and virtual cards, the task has failed even if the system followed its literal instruction. If a timeout occurs after the prepare operation commits but the tool contract reports only “timeout”, the environment has created an unknown outcome; retrying may duplicate the effect. If the evaluator checks that the case is closed but never checks duplicate prepared actions, the evaluator can certify a harmful path.

The remediation sequence differs. Change the selector or reasoning policy for the first. Rewrite the task contract for the second. Add idempotency, effect receipts and readback for the third. Repair the postcondition oracle for the fourth. Model improvement is only one branch of the architecture decision.

The timeout case is worth slowing down. A transport error is an observation about communication, not proof that the external effect did not occur. The environment contract therefore needs three possible outcomes: confirmed not executed, confirmed executed and unknown. The system may retry only the first safely unless the operation is idempotent. The evaluator must treat “unknown” as unresolved rather than converting silence into failure or success. This distinction is invisible in a text-only final answer but decisive in a system that can create effects.

Human approval does not erase the environment problem. A reviewer cannot reliably catch a duplicate if the evidence pack omits the first commit, or select the correct card if the displayed state is stale. Human judgement is another actor with an observation interface. Its placement changes decision rights, but the same requirements for state, evidence and evaluator validity remain.

The environment specification

An environment specification should be written before the test is trusted. It is not a catalogue of infrastructure. It is a contract for how experimental reality is generated and observed.

The distinction between termination and truncation is a small example with large consequences. Gymnasium’s current environment API separates an episode ending because the task reached a terminal condition from an episode being cut off by a time limit or other external boundary.[12] Treating both as one “done” signal can corrupt learning targets and later diagnosis. The task ended and the harness stopped are different claims.

The specification should also declare what cannot be known. A simulation may approximate user behaviour without representing strategic adaptation. A historical replay may know later outcomes that were unavailable at decision time. A tool sandbox may omit production latency and access controls. A simulator earns trust by making its departures from reality explicit, not by being called realistic.

Part VThe failure-attribution protocol

The protocol below is intended for a benchmark review, incident investigation or architecture decision record. It produces a bounded diagnosis and a next experiment rather than a universal label.

1. Restate the problem without implementation language. Name the outcome, protected constraints, target population and consequence of a false success or false failure.
2. Write the executable task contract. Record inputs, observations, allowed actions, prohibitions, horizon, terminal conditions and acceptance criteria. Mark every problem requirement that the task does not represent.
3. Freeze the system boundary. State whether prompts, wrappers, retrieval, tools, retries and human review are inside or outside the compared system. Do not move the boundary after seeing results.
4. Specify the environment family. Describe state, observation, transitions, parameter distributions, other actors, faults, time, reset and authority. Establish whether the failed instance belongs to the promised family.
5. Audit the evaluator against the task. Re-score retained episodes where possible. Test required and prohibited states, path constraints, abstention, unknown outcome and denominator handling.
6. Run matched interventions. Change task wording, environment variables, system policy and evaluator independently. Include a simple baseline and preserve budgets, tools and initial state.
7. Assign the first broken contract and contributing causes. “First” means earliest in the causal and corrective chain, not morally primary. Permit multiple labels and unresolved attribution.
8. Change the corresponding decision. Rewrite the task, repair the environment contract, replace or constrain the system, fix the evaluator, or narrow the problem claim. Then rerun the same intervention set.

The supporting literature establishes several pieces, not the whole protocol. Environment families matter to generalisation and can alter rankings. Proxy rewards and hidden performance criteria can diverge. Equivalent benchmark performance can conceal different model behaviour. Statistical design can change empirical conclusions. Measurement requires an explicit link between the intended construct and the operational instrument.[4][9][6][7][3]

The five-object split, first-broken-contract rule and environment specification are design inference. They have not been validated as a psychometric scale or incident taxonomy. Their value should be tested by whether independent reviewers reach more stable diagnoses, whether proposed fixes address the observed failure and whether unnecessary model changes decline.

Teams that require a problem, task, environment, system and evaluator record before model tuning will make fewer non-causal model changes and will produce failure classes that transfer more reliably from offline tests to shadow operation. A prospective study could compare incident resolution under this protocol with ordinary score-led debugging, using blinded adjudication and time-to-correct-remediation as outcomes.

Where the method stops helping

The five objects are analytical cuts, not natural substances. In adaptive systems, the system changes the environment, users adapt to the system, and evaluators alter development incentives. A fixed partition may become false over long horizons. Multi-agent settings can make one agent part of another agent’s environment, while shared learning couples them.

Attribution can also remain unidentified. Logs may be incomplete. The failed state may be unreplayable. Several contracts may change simultaneously. A human evaluator may disagree for reasons the rubric cannot resolve. The honest output is then unresolved attribution with a discriminating next experiment, not a forced label.

The protocol does not settle whether a task is ethically worth pursuing, whether a target population was selected fairly or whether a successful optimisation should be deployed. Those are problem-level decisions with governance and human consequences beyond episode mechanics. Nor does it prove that a simulator represents production. It only makes the transfer claim inspectable.

Finally, a perfectly specified task can still be the wrong problem. High task success should never silence evidence that the intended outcome was framed too narrowly. The most dangerous evaluator failure is sometimes upstream: choosing to measure the wrong thing with great precision.

Change the experiment before you change the model

The cracked vial does not carry its cause on its surface. Neither does a failed benchmark item, a broken tool call or an incorrect business proposal. Each outcome was produced by an intended problem, an executable task, an environment, a configured system and an evaluator. Compressing them into “the AI failed” discards the information needed to repair the system.

The changed architecture decision is simple: no model change should be approved from a failed score alone. First recover the five contracts. Check that the task represents the problem, the environment kept its declared observation and transition semantics, the evaluator measures the task, and the comparison boundary remained fixed. Then intervene one factor at a time and require a matched baseline before assigning system failure.

This discipline makes room for a strong conclusion when the evidence supports it. In the aligned courier case, the shortest policy really is inferior. It also prevents model work from becoming a ritual response to task omissions, simulator gaps and broken oracles.

The practical output is not a softer verdict. It is a more useful one: which contract broke first, what evidence supports that attribution, what observation would weaken it, and which experiment or design decision changes next.