A route chosen before its explanation

Generated reasoning is often evaluated by reading it. That is natural. Language is the visible part, so a coherent sequence of premises feels like evidence that the model followed those premises. Yet readability answers only whether a human can understand the text. It does not answer whether the text changed the system’s action.

Several research programmes have shown that intermediate reasoning can improve task performance. Chain-of-thought prompting can give a model more room to decompose a problem. ReAct interleaves textual reasoning with actions and observations, allowing external evidence to alter a trajectory. Program-aided methods ask a model to formulate executable steps while an interpreter performs arithmetic or symbolic work. Tree search can explore, score and backtrack across alternatives rather than accept the first continuation.[1][2][3]

These are important results, but they establish a property of a configured method on a task distribution. They do not establish that every visible sentence faithfully reports the internal computation that produced the final action. Turpin and colleagues showed that models can omit a biasing cue while producing an explanation that rationalises the cue’s influence. Lanham and colleagues intervened on chains of thought and found large variation in how much final answers depend on them. Causal mediation work similarly found that models do not reliably use their intermediate steps across tasks and models.[4][5][6]

The practical mistake is to collapse three questions:

1. Outcome lift: did this method improve task success, utility or calibration?
2. Causal mediation: did the stated intermediate commitments help produce the action?
3. Operational auditability: can an operator reconstruct what was known, proposed, permitted, observed and changed?

A system may score well while its explanations are unfaithful. It may expose a causally relevant scratchpad while failing to record the authoritative tool response. It may produce a complete trace that is operationally useful even though it reveals little about computation inside the model weights. These properties can correlate, but none entails the others.

The hidden assumption is temporal

Why do we so readily treat an explanation as a cause? In ordinary conversation, reasons usually arrive after someone has already acted, yet we still use them to infer what guided the action. That inference works tolerably well because people share norms against fabrication, can be questioned, remember earlier commitments and may face consequences for inconsistency. A generated trace has none of those guarantees by default. It is another sequence of tokens sampled under the current context. The final action may influence those tokens just as easily as those tokens influence the action.

Timing alone is not sufficient either. A model can emit a sentence before an action while both are driven by an earlier latent commitment. The sentence is temporally prior but causally redundant. Conversely, a compact plan generated by one model may become genuinely causal when a separate executor is constrained to follow it. The relevant question is therefore not “Was the text written first?” It is “Would a controlled change to this record alter the downstream action while other causes are held fixed?”

This produces four operational roles that should not be mixed. A narrative helps a human understand or contest a choice. A scratchpad gives the model extra computational workspace. A commitment constrains a later decision through a frozen rule. A receipt records an external observation or effect. One artefact may serve several roles, but each role needs a different test. Readability tests narrative quality. Ablation tests scratchpad utility. Intervention tests commitment. Provenance and readback test receipts.

The classification also prevents an unsafe promotion of evidence. A useful scratchpad is not automatically an audit trail because it may omit tool provenance. A well-written narrative is not automatically a commitment because it may be generated after the action. A receipt is not automatically a reason because raw evidence needs a decision rule. Before evaluating a trace, name the job it is expected to perform.

flowchart TD
  A["The guide who explains too late"] --> B["A route chosen before its<br/>explanation"]
  B --> C["The commitments that make<br/>reasoning causal"]
  C --> D["A tool call is an experiment"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#B96D00,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

The commitments that make reasoning causal

A plan becomes more than prose when another part of the system can depend on it, test it and reject it. The key unit is therefore not a sentence called a “thought”. It is a causal commitment: a typed statement made before the relevant evidence or action, with a declared consequence if the world differs from expectation.

Consider “I should inspect the bridge”. That may be sensible, but it does not yet constrain behaviour. Compare it with: “Current probability of an intact bridge is 0.55. Take a drone image because a high-confidence obstruction report would switch the route from north to south. Do not enter the bridge unless the image is timestamped, covers both supports and yields posterior safety above 0.82.” The second form exposes a prior, an information-seeking action, a revision condition, an evidence requirement and a release threshold.

The decisive mechanism is conditional revision. Reasoning guides action when it specifies how particular observations should alter a choice and the configured system actually honours that mapping. A trace that always finds support for the current action, regardless of observation, is not planning under uncertainty. It is explanation under commitment.

This is not a demand to expose unrestricted private chain of thought. A system can produce compact, decision-sufficient commitments without publishing every token generated during inference. The record needs enough structure to support intervention and replay. It does not need to claim that natural-language sentences are a transparent window into all internal computation.

Causal credit is a pattern of controlled dependence

Let X denote the task and permitted context, C the frozen commitment, Z the tool observation and A the released action. A mediated system should show two controlled dependencies. Holding X and Z fixed, an admissible change in C should change A where the two commitments prescribe different actions. Holding X and C fixed, a change in Z should change A where the revision rule crosses its threshold.

Neither dependency should be unlimited. An invalid commitment must not override policy, and an observation outside its validity conditions must not trigger release. This is why causal mediation and authority cannot be collapsed. A plan can have causal force while remaining subordinate to identity, permission and transaction controls. Indeed, a system that follows every substituted plan has demonstrated obedience, not sound reasoning.

The most informative evidence is a dose-response pattern. Small observation changes far from the threshold should leave the action stable. Changes near the threshold should alter confidence or request more evidence. A threshold-crossing observation should revise the action. Wild sensitivity to irrelevant wording suggests prompt fragility. Complete insensitivity suggests a decorative trace. Useful mediation is selective, not maximal.

Four interventions that prose cannot pass by looking good

The trace should be treated like an experimental object. Four interventions reveal different failures.

Commitment swap. Keep the task and tool results fixed, but replace the pre-action plan with another valid plan whose thresholds imply a different response. If the downstream executor ignores the change, the plan was not mediating action. If it follows the change blindly even when the plan violates policy, the system has misplaced authority.

Observation swap. Keep the plan fixed and substitute a counterfactual tool receipt. A bridge-clear receipt and a bridge-blocked receipt should produce different actions when the frozen rule says they should. This tests responsiveness to evidence rather than compliance with words.

Shortcut injection. Add an irrelevant feature correlated with one action, such as always placing the preferred option first. If the action shifts but the trace does not disclose the cue, the explanation is incomplete. Research on biased prompts and reasoning hints shows why this test is necessary.[4][7]

Trace removal. Give the final-answer stage the original task and tool results but remove the intermediate commitments. A large performance drop suggests the trace carried useful information. No drop suggests the trace was redundant, though it may still have audit value. Neither result alone establishes whether hidden computation resembled the visible text.

Recent preprints add a further warning. Some studies report that tool choices can be decoded from activations before the visible reasoning begins, and that steering those representations can flip the choice while the generated text then rationalises the altered route.[8] Such results are preliminary and model-specific, but they sharpen the design principle: recording text is not enough; the system must preserve intervention points before commitment hardens.

A tool call is an experiment

An agent should not call a tool merely because tools are available. A call consumes time, money, privacy budget and attention. It may also introduce stale data, adversarial content or false precision. Under uncertainty, the right question is whether the result could change the decision enough to justify those costs.

This is the logic of value of information. Let the available actions be routes, the unknown world state be bridge condition, and utility encode success, delay and harm. Without new information, choose the action with highest expected utility. A tool is worth calling when its expected improvement in the best available action exceeds the cost and risk of obtaining the observation.[9]

Here, a is an action, s is the unknown state of the world, z is a possible tool observation, and U is the utility of taking that action in that state. The equation assumes that probabilities and utilities are meaningful enough to compare, and that the observation model is known. Remove those assumptions and the number becomes a scenario aid rather than a measured truth.

Planning is the allocation of optionality

A plan under uncertainty is not a long list of steps. It is a policy for preserving and spending options. Search is useful when early choices constrain later possibilities. Tree-of-thought methods make this explicit by generating alternatives, evaluating partial states and backtracking.[3] Tool use adds a second kind of branch: instead of imagining another path, the system can query the world.

The distinction matters because model-generated alternatives are correlated samples from one model, while an external measurement can supply new evidence. Ten restatements of the bridge problem may not be as valuable as one reliable drone image. Conversely, a tool result is not automatically independent or authoritative. A search engine can repeat the model’s own source. A second model can share the same training bias. A sensor can be stale or miscalibrated. The trace must record provenance and expected dependence, not just the word “verified”.

Information gathering needs a stopping rule

Real tool use is sequential. The first observation may reduce uncertainty without resolving it. The agent must then choose among acting, abstaining, asking a person, repeating the measurement or selecting a different source. Without a stopping rule, “reason more” and “search more” become open-ended policies that can consume resources while accumulating correlated evidence.

A practical stopping rule compares the expected value of the next observation with its full marginal cost. That cost includes latency, money, exposure of sensitive context, the chance of tool failure and the risk of delaying a time-critical action. It also includes dependence. If three search results copy the same original report, the second and third do not provide three independent confirmations. A trace should therefore identify the source family or expected correlation when this matters.

Stopping can also be triggered by dominance. If one action remains preferable across every plausible state consistent with the current evidence, more information cannot change the choice. Continue only if the observation may affect execution details or reveal a prohibited condition. At the other extreme, if all available actions violate a safety constraint, more fluent planning should not manufacture a permissible route. The correct result is abstention or escalation.

A good planner buys information only while information can still change a legitimate decision. This principle separates purposeful tool use from ritual retrieval and endless reflection.

Uncertainty must reach the action gate

Many systems generate confidence language without calibrated probabilities. Calibration studies have repeatedly shown that predictive confidence can diverge from empirical correctness, so a number in a trace should not be treated as self-validating.[10] Calibration must be measured on the configured task, model, prompt, tool set and abstention policy.

The operational response is to let uncertainty alter permissions. Low uncertainty may permit an inexpensive reversible action. Moderate uncertainty may trigger a read-only tool call. High uncertainty or conflicting evidence may force abstention, escalation or a safer branch. Uncertainty that cannot change tool choice, authority or release is decorative uncertainty.

stateDiagram-v2
  [*] --> Claim
  state "The hidden assumption is temporal" as Claim
  state "The same words do different<br/>work in two architectures" as Test
  state "The guide who already knows" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

The receipt-gated trace lab

The causal action trace is most useful when implemented as a boundary between a stochastic proposal and an authoritative execution system. The model may propose a belief, plan, tool and action. It does not get to declare its own identity, expand its context, grant itself permission or invent the result of an external effect.

Measure the route, not only the destination

Stateful tool benchmarks have moved evaluation beyond isolated function syntax. BFCL includes serial, parallel, abstention and multi-step settings. τ-bench evaluates conversations, policy adherence and final database state, including repeated-run consistency.[11][12] These developments are valuable because a correct function name can still lead to a wrong outcome, and one successful run can hide trajectory instability.

Yet final state alone can also hide the route. An agent might reach the correct database state after violating a policy and then compensating. It might use a forbidden source but produce an authorised-looking answer. Conversely, two different valid routes may lead to the same outcome. A trace lab therefore needs both outcome tests and causal interventions.

A five-pass evaluation protocol

Pass one establishes the task boundary. Specify intent, candidate actions, authoritative state, permitted context, tool contracts, costs, irreversible effects and success criteria. Keep model proposal separate from authority. Without this manifest, a trace can be praised for solving a task it was never authorised to perform.

Pass two builds matched baselines. Compare direct answering, extra-token reasoning, tool access without an explicit trace, narrated reasoning and receipt-gated planning. Equalise model version, sampling budget and information access where the scientific question requires it. Report both average utility and tail failures. This separates the benefit of more computation from the benefit of a particular trace structure.

Pass three freezes intervention points. Store the commitment before the tool result, sign or hash the record, and version the prompt, policy and tool schema. Prepare semantically valid commitment swaps, observation swaps, irrelevant wording changes and known shortcut injections. The test harness should know which action each frozen rule predicts under each receipt.

Pass four executes and reads back. Record the tool request, raw response digest, parsed observation, validity checks, policy result, action request and authoritative effect receipt. A tool call that times out is not evidence. A successful API response is not an effect unless the relevant state is read back. Preserve abstentions and failed runs rather than filtering them from the dataset.

Pass five grades claims, not personalities. A positive result may justify “this typed commitment mediated this configured action under these interventions”. It does not justify “the model told us how it thought”. A negative result may lead to removing the visible rationale, moving planning into an explicit search procedure, improving tool quality or narrowing authority. The goal is not to make every trace faithful. It is to place assurance on mechanisms that survive testing.

The four metrics

Outcome lift is the change in task utility against a matched baseline. Match model, data, tool access and compute where possible, otherwise a longer trace may simply buy more sampling.

Mediation rate is the fraction of decision-relevant interventions that cause the action change predicted by the frozen rule. It should be reported separately for commitment swaps, observation swaps and trace removal.

Rationalisation rate is the fraction of cases in which a shortcut or contradictory observation changes the action while the trace either omits the influence or rewrites the rule to fit. This requires an experimentally known intervention, not a reviewer’s intuition about wording.

Receipt coverage is the fraction of consequential decisions linked to typed pre-action commitments, authoritative evidence receipts, policy decisions and verified outcome readbacks. Distributed-tracing standards already supply useful concepts such as trace identifiers, parent-child spans, events, attributes and causal links.[13][14] The contribution here is to assign them decision semantics.

from __future__ import annotations
from dataclasses import dataclass
from random import Random
from typing import Literal

Route = Literal["north", "south"]

@dataclass(frozen=True)
class Config:
    prior_safe: float = 0.55
    tool_accuracy: float = 0.90
    tool_cost: float = 5.0
    north_safe_utility: float = 100.0
    north_failed_utility: float = -120.0
    south_utility: float = 35.0
    posthoc_tendency: float = 0.70
    gate_adherence: float = 0.95


def north_value(p_safe: float, c: Config) -> float:
    return p_safe * c.north_safe_utility + (1 - p_safe) * c.north_failed_utility


def choose(p_safe: float, c: Config) -> Route:
    return "north" if north_value(p_safe, c) > c.south_utility else "south"


def posterior(prior: float, report_clear: bool, accuracy: float) -> float:
    p_report_if_safe = accuracy if report_clear else 1 - accuracy
    p_report_if_failed = 1 - accuracy if report_clear else accuracy
    numerator = prior * p_report_if_safe
    denominator = numerator + (1 - prior) * p_report_if_failed
    return numerator / denominator


def expected_value_with_tool(c: Config) -> float:
    p_clear = c.prior_safe * c.tool_accuracy + (1 - c.prior_safe) * (1 - c.tool_accuracy)
    p_safe_clear = posterior(c.prior_safe, True, c.tool_accuracy)
    p_safe_blocked = posterior(c.prior_safe, False, c.tool_accuracy)
    value_clear = max(north_value(p_safe_clear, c), c.south_utility)
    value_blocked = max(north_value(p_safe_blocked, c), c.south_utility)
    return p_clear * value_clear + (1 - p_clear) * value_blocked - c.tool_cost


def run(n: int = 5_000, seed: int = 7, c: Config = Config()) -> dict[str, float]:
    rng = Random(seed)
    totals = {"direct": 0.0, "narrated": 0.0, "gated": 0.0}
    mediated = rationalised = relevant = 0
    use_tool = expected_value_with_tool(c) > max(north_value(c.prior_safe, c), c.south_utility)

    for _ in range(n):
        safe = rng.random() < c.prior_safe
        report_clear = safe if rng.random() < c.tool_accuracy else not safe
        direct = choose(c.prior_safe, c)
        observed = choose(posterior(c.prior_safe, report_clear, c.tool_accuracy), c)
        relevant += int(direct != observed)

        narrated = direct if rng.random() < c.posthoc_tendency else observed
        rationalised += int(direct != observed and narrated == direct)
        gated = observed if rng.random() < c.gate_adherence else direct
        mediated += int(direct != observed and gated == observed)

        for name, route in (("direct", direct), ("narrated", narrated), ("gated", gated)):
            utility = c.south_utility if route == "south" else (
                c.north_safe_utility if safe else c.north_failed_utility
            )
            if name != "direct" and use_tool:
                utility -= c.tool_cost
            totals[name] += utility

    return {
        **{k: v / n for k, v in totals.items()},
        "mediation_rate": mediated / max(relevant, 1),
        "rationalisation_rate": rationalised / max(relevant, 1),
        "tool_positive_evsi": float(use_tool),
    }


if __name__ == "__main__":
    print(run())

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://rajeshrmahapatra.com/schema/causal-action-trace.json",
  "type": "object",
  "required": [
    "trace_id", "decision", "uncertainties", "commitment",
    "tool_request", "evidence_receipt", "policy_decision", "release"
  ],
  "properties": {
    "trace_id": {"type": "string", "minLength": 16},
    "decision": {
      "type": "object",
      "required": ["candidate_actions", "utility_basis"],
      "properties": {
        "candidate_actions": {"type": "array", "minItems": 2, "items": {"type": "string"}},
        "utility_basis": {"type": "string"}
      }
    },
    "uncertainties": {
      "type": "array", "minItems": 1,
      "items": {"type": "object", "required": ["claim", "prior", "decision_relevance"]}
    },
    "commitment": {
      "type": "object",
      "required": ["recorded_at", "prediction", "revision_rule", "stop_rule"],
      "properties": {
        "recorded_at": {"type": "string", "format": "date-time"},
        "prediction": {"type": "string"},
        "revision_rule": {"type": "string"},
        "stop_rule": {"type": "string"}
      }
    },
    "tool_request": {
      "type": "object",
      "required": ["tool_id", "arguments", "authority_scope", "expected_value"]
    },
    "evidence_receipt": {
      "type": "object",
      "required": ["observed_at", "source", "raw_digest", "parsed_result", "validity"]
    },
    "policy_decision": {
      "type": "object",
      "required": ["policy_version", "permitted", "reason_code"]
    },
    "release": {
      "type": "object",
      "required": ["action", "revision_status", "effect_receipt_or_abstention"]
    }
  },
  "additionalProperties": false
}

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: The guide who explains too late
  M->>E: The hidden assumption is temporal
  E-->>M: compare rival explanations
  E->>D: Let evidence change action before commitment
  D-->>Q: revise scope or act

Let evidence change action before commitment

Do not decide whether an agent “can reason” by admiring a transcript. Decide what causal role its intermediate records deserve in the system you are building.

For low-consequence, reversible tasks, outcome lift may be enough. A coding assistant that proposes a correct patch under tests can be useful even if its prose is not a faithful account of model-internal computation. The tests, diff and rollback path carry the assurance burden.

For consequential tool use, require more. The plan should identify decision-relevant uncertainty, state what evidence could change the action, record that rule before the observation, call only permitted tools, preserve raw and parsed receipts, route actions through authority, and verify the effect. Then run counterfactual swaps. A trace becomes release evidence only when it survives interventions that a post-hoc story would fail.

There is also a stop decision. Longer reasoning is not automatically better. It can increase latency, cost and the surface for self-consistent error. If additional tokens no longer change calibrated belief, candidate ranking, information choice or abstention, the system is not preserving optionality. It is spending compute after commitment.

Re-test when the causal route can change

Causal credit expires when the configured system changes. A new model may encode decisions earlier. A prompt revision may move the action before the commitment. A tool update may alter field meanings or reliability. A policy change may turn yesterday’s valid branch into an unauthorised one. Added memory may leak the counterfactual answer into the planner. Even a harmless orchestration optimisation can bypass the executor that once made the plan causal.

Release governance should therefore define change triggers: model or decoding changes, tool-schema changes, new data sources, authority expansion, revised thresholds, altered memory, routing changes and material shifts in task distribution. Re-run a compact intervention suite after each trigger. This is cheaper than assuming that a once-faithful trace remains faithful by identity of wording.

The same rule applies to human review. If reviewers start approving every action under load, the nominal escalation path no longer mediates release. Receipts may show that the control exists while intervention reveals that it has become ceremonial. Governance is a causal property of operation, not a diagram of intended roles.

The strongest alternative explanation is that visible commitments are merely correlated with a hidden decision state. That possibility cannot be removed by better formatting. It requires internal interventions, independent executors, causal mediation designs and repeated testing across task variations. Even then, conclusions remain scoped. A record that mediates behaviour in one architecture may become ceremonial after a model update or orchestration change.

This boundary does not make traces useless. It changes their purpose. The safest trace is not a theatrical window into a machine mind. It is a versioned interface between proposal, evidence, authority and consequence. Its value lies in the counterfactuals it makes executable: What would the agent have done under another observation? Which rule changed? Was the tool worth calling? Did policy permit the action? Did the world actually change?

The operational decision is therefore specific. Stop asking agents for longer explanations by default. Ask them for compact, typed and revisable commitments. Instrument the forks. Price the information. Preserve the receipts. Test the rival causal graph. Reasoning earns authority not by sounding thoughtful, but by exposing where evidence can still change what happens next.