The improvement that vanished with the prompt

Imagine a language model is shown six examples of a new internal code. In each example, a short case description is paired with one of three invented labels: tovan, merek or sulin. The model then labels a seventh case correctly. A dashboard calls this “learning from examples”. The phrase sounds reasonable until the context is cleared. The model receives the seventh case again, without the demonstrations, and the improvement disappears.

Now ask the same model a policy question that needs three dependent deductions. It writes intermediate statements, combines them and reaches the right answer. The dashboard calls this “search”. Yet the system retained only one evolving line of work. It never held a second candidate, backtracked or compared alternative paths.

Finally, wrap the model in a controller that samples sixteen candidate solutions, rejects malformed ones, scores the survivors and selects one. The dashboard calls the result “better reasoning”. The model weights are unchanged, and each candidate may contain only shallow pattern completion. The improvement came from exploring and selecting alternatives.

All three labels describe visible success. None identifies the causal mechanism. The first case adapted inside temporary context. The second transformed a problem state through intermediate commitments. The third allocated compute across a candidate frontier. If the team confuses them, it will reach for the wrong remedy: fine-tuning when a verifier is weak, more tokens when the model lacks a durable skill, or tree search when a direct rule already settles the question.

The central answer is to classify a run by what performance-relevant state changed, how long that change survived and whether more than one alternative remained live. Learning, reasoning and search often cooperate, but their intervention fingerprints differ. That difference is operationally more useful than arguing from fluent traces or product labels.

Part i: ask what state changed

A useful distinction begins before terminology. Picture a system operating on three clocks. The slow clock spans many tasks. It contains weights, fitted adapters or another durable policy state that affects later cases. The middle clock spans one problem. It contains temporary representations, derived constraints and unresolved commitments. The fast branching clock records which alternatives have been opened, scored, deferred or abandoned.

These clocks are related, yet they answer different questions. Learning asks how later behaviour changes because experience altered reusable state. Reasoning asks how a conclusion depends on intermediate transformations within the current problem. Search asks how the system allocates effort among competing possibilities before committing.

A minimal formal vocabulary

Let θ denote durable task-relevant state. It may be a neural network’s parameters, a fitted adapter or a reusable policy table. Let h denote temporary state for the current problem. It may contain hidden activations, a symbolic proof state, a scratchpad or a structured set of derived facts. Let F denote a frontier containing candidate states or trajectories that have not all been collapsed into one commitment.

Learning: θk+1 = U(θk, Dk)

Dk is experience available at episode k, and U is an update rule. The important test is whether changing θ alters behaviour on later tasks after the original context has gone.

Reasoning: ht+1 = Rθ(ht, et), with θ fixed for the problem

et is evidence or an observation introduced at step t. A reasoning claim becomes stronger when a controlled change to intermediate state changes the conclusion in a structurally predicted way.

Search: Fj+1 = Select(Expand(Fj), q), where |F| can exceed 1

q is a scoring or priority rule. Search requires alternatives to remain available for comparison, revision or backtracking. A long single trajectory can consume much compute without being search.

This vocabulary is deliberately functional. It does not assert that every parameter update is valuable learning, every hidden activation is reasoning, or every sampled token is a meaningful branch. Causal relevance matters. A weight change that does not alter future behaviour is an ineffective update. A verbose chain that can be deleted without changing the answer is not evidence that its stated steps carried the result. Twelve candidates that all repeat the same error may be search in form while adding no useful exploration.

Figure 2. same answer, different intervention fingerprint

Durable learner

A classifier is fitted on labelled examples and later handles unseen cases better.

Reset test: restoring the earlier checkpoint removes the gain. Workspace and branch limits need not matter.

Fixed reasoner

A rule system derives a conclusion from premises while its rules remain unchanged.

Workspace test: removing an intermediate derived fact breaks the conclusion. A later unrelated task is unchanged.

Explicit searcher

A route planner keeps several partial paths, scores them and expands the most promising.

Frontier test: a branch cap or disabled backtracking changes success or cost. No training update is required.
Synthetic mechanism comparison. Each column can produce the same outward answer, but a different controlled intervention should remove its advantage.

Why older definitions still matter

The machine-learning tradition gives the slow clock a precise role. Valiant’s theory of learnability asked whether concepts could be acquired from examples under bounded resources. Backpropagation made the update concrete by repeatedly changing connection weights to reduce error, allowing internal representations to capture regularities. These traditions disagree on many details, but both locate learning in an update that improves performance beyond the single example currently being processed.

Classical search gives the frontier a different identity. A* does not improve because the graph changes its rules after each route. It improves by maintaining candidate paths and ordering them with accumulated cost plus a heuristic estimate. Newell and Simon placed heuristic search near the centre of symbolic intelligence, a historically influential view. That history explains why “reasoning” and “search” are often spoken together. It does not make them extensionally identical.

A supplied proof path can be checked through deterministic inference with no competing branch. Conversely, a brute-force password enumerator explores alternatives without deriving rich relational conclusions. Reasoning concerns the transformation of commitments; search concerns the allocation of computation across possibilities. Search may invoke reasoning at every node, but the branching controller remains a separable mechanism.

Published evidence. The cited literature establishes weight-changing learning procedures, frontier-based search and systems that compose learned evaluation with tree search. The exact three-state taxonomy and its intervention protocol are design inference offered by this article.

Part ii: intervention before interpretation

Names are cheap because the output is compatible with many hidden causes. A model may answer a syllogism from memorised textual patterns, a learned direct procedure, a latent multi-step computation, a visible derivation, or explicit search. The safer method is to intervene on candidate mechanisms while preserving the task as closely as possible.

The question is not “does the trace look like learning, reasoning or search?” It is “which state must remain intact for the advantage to survive?” Four tests turn that question into an evaluation design.

1. The persistence test

Give the system experience that improves performance, then remove the immediate context while preserving or resetting durable adaptation state. Test on a later, matched task. If the gain survives only when the updated state survives, the run contains evidence for persistent learning. This test covers ordinary training, fine-tuning, online learning and test-time training that writes back parameter changes.

The test must distinguish useful update from mere mutation. Log the checkpoint or adapter difference, but also measure the later-task effect. A thousand gradient steps that leave task behaviour unchanged do not earn a substantive learning claim. External memory needs the same discipline. Writing a case to a store is persistence; it becomes learning only if the write alters a reusable mapping or decision policy under a stated boundary.

2. The workspace test

Hold durable parameters, input evidence and final decoding policy constant. Intervene on an intermediate state: remove one derived proposition, truncate a scratchpad before a dependency is resolved, replace a latent slot with a control value, or provide the correct intermediate result directly. A structurally selective effect supports the claim that temporary state carried useful inference.

Visible text is only one possible workspace. Chain-of-thought prompting showed that generated intermediate steps can improve performance on multi-step tasks without fine-tuning. The original authors also stated that the result does not settle whether the network is actually “reasoning”. Later intervention studies found that stated chains can omit causal biases, rationalise an already influenced answer, or matter very differently across tasks. A readable explanation is therefore weaker evidence than a controlled dependency.

3. The frontier test

Keep the generator and evaluator fixed, then change how many alternatives can remain live. Compare greedy commitment, several independent samples, beam width, tree depth, backtracking and a matched compute budget. If success changes because viable alternatives are preserved and selected, search contributed.

This test separates search from mere token length. Self-consistency samples multiple reasoning paths and aggregates answers. Tree of Thoughts explicitly maintains and evaluates branches. More recent test-time compute work shows that the value of extra search or adaptation varies with problem difficulty and verifier quality. The relevant quantity is not simply “reasoning tokens”. It is how compute changes the distribution of explored candidates and the chance of selecting a better one.

4. The necessity control

Every mechanism claim needs a simpler competitor. A lookup baseline may solve a fixed benchmark. A direct policy may match a planner on familiar states. A deterministic rule may settle a case before a language model is called. Run the mechanism and then neutralise it. If output quality stays constant, the observed trace may be decorative, redundant or dominated by another path.

This control is serious because modern systems can produce plausible theatre. A model can print eight numbered steps after committing from a shortcut. A controller can branch twelve times but use an evaluator that always chooses the first candidate. A training job can update parameters while a frozen shortcut feature still determines the answer. Mechanism-shaped activity is not the same as mechanism-caused performance.

Figure 3. state cutaway: durable rule, working derivation, candidate frontier

Slow store: θ

  • weights or adapter
  • reusable policy parameters
  • survives the current request
Intervention: restore the prior durable state.

Working state: h

  • derived facts and constraints
  • temporary hidden or symbolic state
  • collapses when the task ends
Intervention: ablate a predicted dependency.

Frontier: f

  • multiple candidate states
  • priority, score or value
  • supports choice and backtracking
Intervention: cap breadth, depth or revisitation.
Author synthesis. The stores are logical roles rather than mandatory physical components. One tensor or process may implement more than one role.

Thought experiment: three maze solvers

Consider a family of grid mazes. Solver A has been trained on millions of examples and emits one move at a time from a learned policy. Solver B has no learned component; it runs A* with Manhattan distance. Solver C uses a learned value network to guide a tree search. All three reach the exit on the first test maze.

Now vary one causal feature at a time. Reset Solver A to its pre-training checkpoint. Its direct policy deteriorates, although no branch budget has changed. Reduce Solver B to one live candidate and forbid reopening. Its guarantee disappears, although no durable state has changed. Replace Solver C’s learned value with a constant. The search can still explore, but it becomes less efficiently guided. Then keep the value network and cap the tree at one branch. The learned component survives while the search advantage disappears.

Figure 4. the maze ablation

A. direct learned policy

One trajectory, no explicit alternatives.

Checkpoint reset harms performance. Branch cap has little meaning.

B. hand-coded a*

Fixed rules, explicit frontier, heuristic ordering.

Branch and reopening controls matter. Training reset does not.

C. learned guided search

Durable value estimates rank an explicit frontier.

Both checkpoint and branch interventions matter for different reasons.
Synthetic thought experiment. The purpose is causal separation, not a claim about relative maze-solving accuracy.

This experiment exposes an important asymmetry. Solver C is not evidence that the three terms are interchangeable. It is evidence that a single system can contain separately necessary learning and search. Its learned value function compresses prior experience. Its search reconstructs task-specific alternatives at runtime. Reasoning may occur in state evaluation or constraint propagation, but it should be tested rather than inferred from the presence of a tree.

A negative control that should fail

Build a dataset in which every input maps to the same answer. Ask a language model to produce a detailed derivation, sample several candidates and vote. It will show intermediate text and multiple paths. A constant classifier will match its accuracy at negligible cost. Truncating the derivation or reducing the candidate count may change wording while leaving the answer untouched.

This is the negative control for mechanism inflation. It demonstrates that a benchmark can invite apparent reasoning and search even when neither is needed. The lesson extends beyond trivial constant labels. Leakage, repeated templates, answer-position cues and memorised benchmark items can all let a direct shortcut dominate a more elaborate trace. Before celebrating a mechanism, construct cases where the shortcut and the intended process disagree.

Failure boundary. No black-box intervention can uniquely recover every internal computation. Redundant pathways, distributed representations and compensating mechanisms can hide causal dependence. The protocol supports bounded engineering claims about a configured system, not a complete theory of machine cognition.

Part iii: hybrids, compilation and boundary cases

The clean examples are useful because the real frontier is hybrid. Learning can manufacture a heuristic for search. Search can generate experience for learning. A reasoning procedure can be distilled into a direct policy. An episode-local learner can update an implicit predictor inside activations while the outer model remains frozen. The distinction survives these combinations if the system boundary and timescale are explicit.

Learned guidance is not the tree

AlphaGo offers a canonical composition. Policy and value networks were trained from expert play and self-play. At runtime, Monte Carlo tree search used those learned estimates to select and evaluate moves. The network without lookahead and the network-guided search were separately measurable configurations. MuZero later learned a model useful for planning, then used tree-based search over its latent predictions.

These systems are often described compactly as “learning to reason” or “learning to search”. The shorthand hides two different investments. Training improves reusable estimates across games. Search spends additional compute on the current position. A deployment can change one without changing the other: update weights, expand more nodes, alter the exploration constant, or replace the evaluator. Each change has its own regression surface.

Figure 5. learned guidance inside explicit search

Across games

Experience updates policy, value or model parameters. The result is reusable guidance.

Evidence: later positions improve when the updated checkpoint is retained.

On this position

The runtime expands alternatives, scores them with learned guidance and backs up values.

Evidence: search budget and branching policy alter this decision.
Mechanism reconstruction from AlphaGo and MuZero. No performance numbers are reproduced. The figure separates training-time reuse from position-specific exploration.

In-context learning sits on a boundary

Few-shot language-model behaviour made the vocabulary harder. GPT-3 was evaluated with demonstrations in the prompt and no gradient updates. The authors called the behaviour in-context learning while remaining agnostic about whether the model learned a new task from scratch or recognised a pattern acquired during pretraining. That caution is still valuable.

Later work on controlled linear-regression tasks showed that fixed transformers can implement familiar learning algorithms inside their activations. An implicit predictor changes as examples arrive, although the outer transformer parameters do not. In this case, “no parameter update” does not mean “no learning-like computation”. It means the update lives on the episode clock and is discarded with the context.

The cleanest language is two-level. The outer system has learned, during training, how to adapt. The inner run performs episode-local adaptation by constructing or updating an implicit predictor in temporary state. Calling both processes learning is defensible if the level is named. Calling the inner process persistent model learning is incorrect because the adapted state does not automatically survive the episode.

Figure 6. in-context adaptation lives between the usual boxes

Outer-loop learning and inner-loop adaptation A large teal outer loop updates model parameters across training episodes. Inside it, an indigo loop shows examples updating an implicit predictor in activations during one context. Clearing context removes the inner predictor while leaving the outer parameters. Outer loop: durable training Parameters θ change across many episodes Inner loop: one context Examples update an implicit predictor in h Clear context: inner adaptation disappears, θ remains
Author synthesis grounded in the in-context learning literature. The inner predictor is established most clearly in controlled tasks; the same mechanism should not be assumed for every language-model behaviour.

Inference time can also contain real parameter learning

Test-time training removes another superficial boundary. In that setting, an incoming test example supplies a self-supervised objective and model parameters are updated before prediction. The clock says “inference”, but the mechanism includes a durable or at least explicit parameter update. A phase label cannot substitute for state inspection.

This matters for operations. A stateless inference service can be replayed from a fixed model, prompt and input. A test-time adapting service may depend on sample order, adaptation scope and rollback policy. It needs checkpoint lineage, contamination controls and a defined answer to whether an update is discarded after one sample, retained for a stream or promoted into the next release.

Search can be compiled away

Suppose an expensive planner solves millions of problems and its chosen actions become training labels for a policy network. At first, the system explores a tree for each problem. After distillation, the policy emits a strong action in one forward pass. Prior search has shaped the weights, but the deployment run may contain no explicit frontier.

This shift is often called amortisation: pay a high cost across training examples, then reuse a compressed mapping at inference. The reverse can also happen. A small model may be weak in one pass but competitive when wrapped in sampling, verification and search. These transformations show why mechanism labels belong to a particular run and boundary. Search in the lineage does not prove search in the current execution.

The strongest objection: everything is state transformation

At a sufficiently low level, the distinction collapses. Training, deduction and tree expansion are all computations that transform physical or digital state. At a sufficiently high level, even evolution can be described as search and a neural forward pass as retrieval from a learned function. Why insist on separate words?

Because engineering explanations are judged by intervention and decision. Parameter learning raises questions about data, generalisation, forgetting and release lineage. Temporary reasoning raises questions about decomposition, faithfulness, workspace and intermediate error. Search raises questions about branching factor, value estimates, stopping, compute and selection bias. Treating them as one undifferentiated “intelligence” mechanism destroys diagnosis.

The objection does establish a limit. The classification is relative to the chosen grain. An in-context predictor may be learning at the inner-loop grain and reasoning at the host-system grain. A theorem prover may view proof-rule application as reasoning while its controller views candidate proof states as search. The article’s claim is therefore modest: the terms are not synonyms at any boundary where their state, persistence and interventions differ.

Design inference. State locus, persistence and branching form an operational coordinate system. It does not impose one metaphysical definition of reasoning, and it permits multi-label classifications when a run genuinely composes mechanisms.

Part iv: what the distinction changes

Mechanism clarity is useful only if it changes a build or release decision. Consider a synthetic document-review service for a regulated workflow. It receives application forms, identity evidence and policy text. Its job is to identify missing or conflicting evidence and draft a refer-back request. It cannot approve the application.

A weak architecture sends all material to one model and calls the response “reasoning”. A stronger architecture separates three uncertainties. A learned extractor maps noisy documents into candidate fields. A reasoning layer derives which policy predicates are satisfied, unsatisfied or unresolved. Search is reserved for ambiguous reconciliation, such as several plausible identity matches or competing document-to-entity assignments. A deterministic authority boundary controls the permitted action.

Figure 7. a production-shaped case with separated mechanisms

Learned extraction

Reusable model maps scans and text to field candidates with confidence and provenance.

Case reasoning

Temporary state applies policy predicates and carries unresolved contradictions.

Bounded search

Only ambiguous cases open alternative matches or evidence hypotheses for comparison.

Human decision

Reviewer receives the residual issue, evidence and requested action. Approval authority remains external.

Worked scenario, not a deployment claim. The separation makes failure ownership and evaluation evidence specific to each mechanism.

Different mechanisms need different evidence

For learning, evaluate generalisation across held-out cases, robustness to distribution shift, calibration, subgroup behaviour, forgetting and checkpoint lineage. A larger reasoning budget cannot repair a classifier that never learned the relevant distinction. More search may amplify its systematic bias by exploring many candidates generated from the same defective representation.

For reasoning, test controlled composition. Vary irrelevant wording, reverse premise order, inject contradictions, remove one needed fact and supply intermediate results. Compare final answers with independent checks. Do not award credit merely because the explanation is long or fluent. The evidence should show that temporary state tracks the task’s dependency structure.

For search, report quality as a function of candidate budget, breadth, depth, stopping rule and evaluator error. Compare against independent sampling and a direct baseline under matched compute. A method that wins with ten times the calls may still be worthwhile, but the gain belongs on a cost-quality curve rather than under a generic “reasoning improvement” label.

Mechanism claim Primary intervention Evidence that supports it Common false positive Release concern
Persistent learning Restore prior durable state Later-task gain disappears under reset and survives context removal Parameter change with no behavioural effect Data lineage, drift, forgetting, rollback
Episode-local adaptation Clear demonstrations or implicit predictor state Current-task gain depends on examples but no durable update is retained Task recognition from memorised format Prompt sensitivity, context isolation, replay
Reasoning Ablate predicted intermediate dependencies Structured changes to workspace cause structured answer changes Plausible post-hoc rationale Faithfulness, contradiction handling, verification
Search Cap branches, depth or backtracking Quality and cost respond to preserved alternatives and selection Repeated samples with no diversity or useful evaluator Compute, latency, verifier bias, stopping

Mechanism-specific telemetry and rollback

Operational traces should preserve the distinction rather than repeat component names. A log event called reasoning_started proves only that a developer emitted a label. Useful telemetry records the state boundary, the intervention-relevant event and the observable consequence. This may be compact: checkpoint identity and adaptation receipt for durable learning; dependency and verification records for temporary inference; candidate lineage, scores and stopping decisions for search.

For a learning mechanism, retain which reusable state changed, the authorised data or feedback that caused the update, the evaluation run that justified release and the prior state needed for rollback. The critical recovery operation is restoration of a validated checkpoint or adapter. Clearing a conversation cannot undo a bad weight update. Conversely, restoring old weights will not remove a poisoned retrieval result that lives only in current context. Rollback must target the state that actually carried the change.

For reasoning, record the claims, premises, calculations, tool results and unresolved contradictions needed to reconstruct the decision. A verbatim natural-language chain may be useful for debugging, but fluency does not make it a faithful causal transcript. The recovery operation is usually replay with corrected evidence, a different decomposition or an independent verifier. Persisting every generated sentence as institutional memory can turn a temporary mistake into durable contamination.

For search, record candidate identity, parentage, evaluator scores, pruning reasons, budget and stopping condition. Without candidate lineage, a final answer selected from twenty attempts is indistinguishable from one direct sample after the fact. Without evaluator telemetry, the system may appear to benefit from breadth while repeatedly selecting the same bias. Recovery may require changing the branch policy, verifier or stopping rule while leaving the underlying model untouched.

Consider a synthetic incident. A policy-review service begins omitting a required exception after a release. The mechanism map yields three fast tests. Restore the previous adapter to test durable learning. Replay the same case with the new policy clause made explicit to test reasoning state. Hold both fixed while widening candidate retrieval and selection to test search. The answer may look like one model failure, yet each intervention points to a different owner, corrective action and regression suite. Mechanism classification becomes operationally valuable when it shortens the path from symptom to reversible change.

The architecture decision record

Before adding a mechanism, write an architecture decision record with six fields. First, name the failure to be reduced. Second, identify the state the mechanism will change. Third, state its persistence horizon. Fourth, define the intervention that would prove causal value. Fifth, identify the simpler baseline. Sixth, record the new failure and operating cost introduced.

This discipline prevents three recurring mistakes. Teams fine-tune a model to solve a retrieval or search-selection problem. They add tree search where a deterministic calculation would be cheaper and more reliable. They accept a chain-of-thought trace as assurance evidence even though the consequential claims and actions were never checked independently.

Add durable learning when reusable competence is missing, add reasoning workspace when a single case needs structured intermediate state, and add search when early commitment is the limiting error. When the failure belongs elsewhere, improve context, tools, verification or authority instead.

Two worked decisions

Minimal worked example. A model repeatedly fails to solve a novel four-step arithmetic schema unless two demonstrations are in the prompt. Fine-tuning is one option, but the persistence test first shows that the gain vanishes with context reset. A compact adapter trained on representative examples then preserves the gain on later tasks. The original improvement was episode-local adaptation; the new intervention creates persistent learning. Calling both “prompt reasoning” would have hidden the actual change.

Production-shaped worked scenario. The document-review service extracts names accurately but often selects the wrong person when two records share similar identifiers. Error review shows that the correct candidate is usually generated but not selected. More extractor training gives little gain. A bounded search over candidate entity assignments, scored by deterministic identifier checks and evidence consistency, improves the matched challenge set. The decision is to invest in candidate diversity and verifier quality, not another general model upgrade.

How this paper relates to the reasoning stack

The live article The Reasoning Stack maps chain-of-thought, tools, search, verifiable rewards and test-time compute as layers used in modern systems. The present paper is narrower and more foundational. It asks which causal state changed before those layers are assembled. The two articles should remain adjacent rather than merged: one classifies mechanisms, the other composes and governs a runtime stack.

Decision instrument. Require a reset test, a workspace intervention, a branch-budget ablation and a matched simpler baseline before using “learning”, “reasoning” or “search” as the explanation for a measured gain.

Executable artefact: a causal trace classifier

The classifier below operates on synthetic trace evidence. It does not inspect prose labels such as thought, reflection or learn. It asks whether a relevant event occurred and whether a targeted ablation materially changed a task score.

The assumptions are explicit. A score change of at least five percentage points is treated as material for the synthetic examples. A real evaluation must replace that threshold with uncertainty intervals and consequence-aware criteria. The classifier permits multiple labels because a learned tree-search system can genuinely contain persistent learning, reasoning and search.

Figure 8. the classifier asks for causal effects, not event names

Observed event Required causal intervention Label earned Without the effect
Durable state updated Reset removes a later-task gain Persistent learning Update observed, value unproven
Implicit predictor updated in context Context reset removes current-task adaptation Episode-local adaptation Pattern recognition remains plausible
Intermediate states emitted or logged Workspace ablation changes the answer selectively Reasoning Ornamental or redundant trace
Several candidates evaluated Branch cap changes quality or cost Search Decorative branching
Synthetic decision logic implemented by the included Python artefact. The code reports cautions when an event is present but its causal contribution is not established.
#!/usr/bin/env python3
"""Classify a synthetic execution trace by causal mechanism.

The classifier distinguishes four forms of evidence:
- persistent learning: a durable update changes later-task behaviour;
- episode-local adaptation: an implicit predictor changes inside one context;
- reasoning: derived intermediate state is causally used within the task;
- search: more than one candidate is evaluated and the branch budget matters.

It deliberately refuses to infer a mechanism from decorative log events alone.
All examples are synthetic and the deltas are illustrative task-score changes.
"""

from __future__ import annotations

from dataclasses import dataclass
from typing import Iterable


EFFECT_THRESHOLD = 0.05


@dataclass(frozen=True)
class TraceEvidence:
    name: str
    durable_updates: int = 0
    later_task_delta_after_durable_reset: float = 0.0
    episode_local_predictor_updates: int = 0
    current_task_delta_after_context_reset: float = 0.0
    derived_state_steps: int = 0
    current_task_delta_after_workspace_ablation: float = 0.0
    candidates_evaluated: int = 1
    current_task_delta_after_branch_cap: float = 0.0


@dataclass(frozen=True)
class Classification:
    labels: tuple[str, ...]
    cautions: tuple[str, ...]


def material(delta: float) -> bool:
    """Return True when an intervention changes the synthetic task score materially."""
    return abs(delta) >= EFFECT_THRESHOLD


def classify(trace: TraceEvidence) -> Classification:
    labels: list[str] = []
    cautions: list[str] = []

    if trace.durable_updates > 0:
        if material(trace.later_task_delta_after_durable_reset):
            labels.append("persistent learning")
        else:
            cautions.append("durable updates observed, but their causal value is unproven")

    if trace.episode_local_predictor_updates > 0:
        if material(trace.current_task_delta_after_context_reset):
            labels.append("episode-local adaptation")
        else:
            cautions.append("episode-local updates observed, but resetting context had no material effect")

    if trace.derived_state_steps > 0:
        if material(trace.current_task_delta_after_workspace_ablation):
            labels.append("reasoning")
        else:
            cautions.append("intermediate steps observed, but they may be ornamental or redundant")

    if trace.candidates_evaluated > 1:
        if material(trace.current_task_delta_after_branch_cap):
            labels.append("search")
        else:
            cautions.append("multiple candidates observed, but branch exploration was not causally necessary")

    if not labels:
        labels.append("mechanism not established")

    return Classification(tuple(labels), tuple(cautions))


def synthetic_cases() -> Iterable[TraceEvidence]:
    return (
        TraceEvidence(
            name="fine_tuned_router",
            durable_updates=240,
            later_task_delta_after_durable_reset=-0.18,
        ),
        TraceEvidence(
            name="fixed_rule_proof",
            derived_state_steps=5,
            current_task_delta_after_workspace_ablation=-0.42,
        ),
        TraceEvidence(
            name="a_star_route",
            candidates_evaluated=37,
            current_task_delta_after_branch_cap=-0.55,
        ),
        TraceEvidence(
            name="in_context_regression",
            episode_local_predictor_updates=6,
            current_task_delta_after_context_reset=-0.31,
        ),
        TraceEvidence(
            name="learned_tree_search",
            durable_updates=5000,
            later_task_delta_after_durable_reset=-0.27,
            derived_state_steps=4,
            current_task_delta_after_workspace_ablation=-0.16,
            candidates_evaluated=64,
            current_task_delta_after_branch_cap=-0.33,
        ),
        TraceEvidence(
            name="decorative_chain",
            derived_state_steps=8,
            current_task_delta_after_workspace_ablation=-0.01,
        ),
        TraceEvidence(
            name="decorative_branching",
            candidates_evaluated=12,
            current_task_delta_after_branch_cap=0.00,
        ),
    )


def main() -> None:
    for trace in synthetic_cases():
        result = classify(trace)
        label_text = ", ".join(result.labels)
        print(f"{trace.name:24} -> {label_text}")
        for caution in result.cautions:
            print(f"{'':27} caution: {caution}")


if __name__ == "__main__":
    main()

Expected output includes four clean positive cases and two negative controls. The decorative_chain contains intermediate steps, yet workspace ablation changes the score by only one point, so the mechanism is not established. The decorative_branching evaluates twelve candidates, yet a branch cap has no effect, so the search events may be redundant.

Expected console output
fine_tuned_router        -> persistent learning
fixed_rule_proof         -> reasoning
a_star_route             -> search
in_context_regression    -> episode-local adaptation
learned_tree_search      -> persistent learning, reasoning, search
decorative_chain         -> mechanism not established
                            caution: intermediate steps observed, but they may be ornamental or redundant
decorative_branching     -> mechanism not established
                            caution: multiple candidates observed, but branch exploration was not causally necessary

The artefact’s most important behaviour is refusal. It will not promote logging activity into a mechanism claim without an ablation effect. That refusal is a small model of the paper’s larger argument: capability names should follow causal evidence, not decorate a successful output.

Compact glossary

Terms used in the article
  • Durable state: task-relevant state that survives the current request and can change later behaviour.
  • Episode-local adaptation: an update to an implicit predictor or policy inside one context, discarded when that context is cleared.
  • Reasoning workspace: temporary state that carries derived facts, constraints or commitments within a problem.
  • Search frontier: more than one live candidate state or trajectory, ordered or selected by a policy, score or value estimate.
  • Amortised search: search effort performed across training examples and compressed into a reusable direct policy.
  • Intervention fingerprint: the pattern of performance changes produced by resetting, ablating or constraining a candidate mechanism.

Source ledger

The sources below support bounded mechanism claims. None establishes the article’s complete taxonomy on its own.

Place the mechanism before naming the capability

A system improves after examples, writes a long derivation and samples several answers. It is tempting to describe the whole performance as learning to reason through search. The phrase may be rhetorically smooth, but it erases the decisions that matter.

Durable learning changes what the system can reuse later. Temporary reasoning changes what the current problem state contains. Search changes how much of the alternative space is visited before commitment. Their boundaries can move. Training can compile search into weights. A context can host an inner learning algorithm. A learned value function can guide a tree. None of those hybrids removes the need to know which state is carrying the gain now.

The practical rule is simple: reset, interrupt and constrain before you interpret. Reset durable state and test whether later competence survives. Interrupt the predicted workspace dependency and see whether the current conclusion changes. Constrain the frontier and measure whether preserving alternatives mattered. Compare every elaborate mechanism with a simpler baseline that could exploit the same benchmark.

This changes research design. Benchmarks should include cases where memorisation, direct policy, inference and search predict different outcomes. It changes architecture. Teams can place training, temporary workspace and search control in separate components with separate budgets and rollback paths. It changes governance. A visible trace is no longer treated as proof of reasoning, and multiple samples are no longer treated as independent evidence unless their generation and selection are tested.

The result is not a final definition of intelligence. It is a sharper causal language for building and evaluating intelligent systems. Before asking whether a machine learned, reasoned or searched, specify the boundary, identify the state, and show the intervention that makes the word earn its place.