The Reasoning Stack, 2026

"Reasoning" in machine learning has gone from a research aspiration to a product tier in about three years, and the vocabulary has outrun the evidence. When someone tells me their system "reasons," I now ask which layer of the stack they mean, because there are at least five, they were developed in sequence, they compose imperfectly. The empirical support for each is very different. This is my working map: what each layer actually is, what the evidence says, and where I think the field is fooling itself.

Layer 1: chain-of-thought: reasoning as decoding strategy

Chain-of-thought started as an observation: prompt a large model to produce intermediate steps before an answer and accuracy on multi-step problems can improve. The original 2022 result has been reproduced across several multi-step settings. Follow-up work since then has complicated the interpretation.

Three findings I treat as load-bearing:

• CoT helps most where problems decompose serially (math, symbolic manipulation, multi-hop lookup) and helps little or negatively on tasks that don't (some commonsense and perception-adjacent tasks show flat or degraded performance with forced verbalization).
• The stated chain is not a faithful trace of the computation. Post-hoc analyses and perturbation studies show models producing correct answers after corrupted chains and wrong answers after apparently valid ones. Generated intermediate text can provide additional inference workspace; it is not a readout of hidden computation. Oversight should therefore rely on observable evidence use, tool and policy events, checked outcomes and a concise structured rationale for the proposed decision, not on an unbounded prose chain.
• CoT is a capability amplifier, not a capability source. It surfaces what pretraining put in; it does not conjure what isn't there. This is why CoT gains correlate strongly with model scale and why small models often generate fluent chains that go nowhere.

Operationally, CoT is not free. Additional generated or internal reasoning tokens can increase inference cost and latency even when the visible answer is short. Use it only where a measured task-level gain justifies that budget; it is the floor of this map, not the whole story.

Layer 2: tool use: externalizing what shouldn't be neural

The second layer admits something important: some of what we were asking models to do internally (arithmetic, fresh facts, code execution) should never have been a neural computation at all. Tool use lets the model delegate to calculators, retrievers, interpreters, APIs.

The evidence here is the cleanest in the whole stack, because the wins are almost definitional. A model that writes and runs code outperforms the same model doing mental arithmetic. Retrieval beats parametric memory on anything time-sensitive. Nothing controversial there.

The genuinely interesting empirical questions are about the decision boundary: when does the model choose to call a tool, and how do errors compound across calls? Two results shape my thinking:

• Tool selection is itself a learned skill with poor out-of-distribution behavior. Models over-call tools on easy instances and under-call on hard ones, and benchmark scores on tool-use suites are inflated by test tasks resembling training tasks. Agentic benchmarks like SWE-bench-style suites keep getting partially saturated and then revealed to leak or reward shortcuts.
• Error compounding is the tax on autonomy. Under an illustrative independence assumption, forty steps that each succeed with probability 0.98 yield an end-to-end success probability of about 45 percent. Real errors are often correlated, so the calculation is not a forecast. It still explains why long-horizon task success, not single-shot accuracy, is the number I watch.

Layer 3: search and planning: spending compute on alternatives

Layer three stops sampling one trajectory and starts exploring several: self-consistency (sample many chains, vote), tree-of-thought variants, MCTS-style exploration over reasoning steps, and planner-executor decompositions.

The honest evidence summary: broad, simple search works; deep, clever search mostly hasn't earned its complexity. Self-consistency's gains are reproducible and unglamorous. Majority voting over samples remains a strong baseline that more elaborate methods often fail to beat under matched compute. Tree search shows real wins on tasks with crisp intermediate-state evaluation, including games, formal mathematics and puzzles. Results are less stable elsewhere because the binding constraint is the value function. Search is only as useful as the score assigned to partial progress, and open-ended work lacks dependable learned verifiers. Formal proof provides the clearest counterexample because verification is unusually strong.

The planning literature has a parallel finding I take seriously: LLM plans that look coherent fail at execution-time preconditions. Neuro-symbolic hybrids (LLM proposes, classical planner/verifier checks) reliably beat LLM-only planning on benchmarks with formal semantics. The lesson generalizes: the model is a strong generator and a weak judge of its own trajectories.

Layer 4: rl with verifiable rewards: the 2024–2025 regime change

The layer that actually changed the field: take domains where correctness is checkable (math with numeric answers, code with unit tests, formal proofs) and run reinforcement learning against that verifiable signal, letting the model discover longer, self-correcting reasoning trajectories. This is the recipe behind the o-series lineage, DeepSeek-R1, and everything that followed; R1's open publication in early 2025 mattered because it showed the recipe was reproducible without exotic ingredients. That behaviors like backtracking and self-verification emerge from RL rather than being hand-engineered.

What the evidence supports:

• Within verifiable domains, the measured gains are large. Competition mathematics and programming benchmarks improved sharply through verifiable-reward training and larger inference budgets. These results demonstrate stronger performance on the evaluated tasks; they do not, by themselves, establish general reasoning competence outside those distributions.
• Transfer out of verifiable domains is real but heavily attenuated. RLVR-trained models write more structured analyses everywhere, but the accuracy gains on open-ended tasks are a fraction of the in-domain gains. We trained models to be excellent where we can check answers, and we're still arguing about how much of that is "reasoning" versus extremely good in-distribution procedure learning.
• The open question I weight most: does RLVR create capabilities or sharpen sampling? A serious line of work argues much of the gain is elicitation (concentrating probability mass on solution paths the base model could already sample at large k) with pass@k studies showing base models catching RL models as k grows. If that view is even half right, the ceiling of this layer is set by pretraining after all, and layer 4 is a very sophisticated version of layer 1's lesson.

Layer 5: test-time compute: thinking longer as a product knob

The top of the stack reframes everything below it economically: reasoning is now a quantity you buy at inference. Longer chains, more samples, more search, more verifier passes: accuracy scales with tokens spent, log-linearly at first, and labs now publish test-time scaling curves the way they once published pretraining ones.

What holds up: the curves are real, and for hard verifiable problems, small models with large thinking budgets genuinely overtake much larger models answering cold: compute-optimal inference is a legitimate result. What the curves conceal:

• They flatten fast, and per-domain. Test-time scaling buys the most on exactly the problems where layer 4 trained the model to use it. On tasks without verifiable structure, extra thinking often plateaus within a few thousand tokens, and overthinking is a documented failure mode: accuracy declining as chains grow, models talking themselves out of right answers.
• Selection is doing quiet work. Many reported test-time gains route through best-of-n with a verifier or reward model. That's a statement about verifiers as much as about thinking. Same lesson as layer 3, one layer up.

Reading the stack whole

Stand back and the five layers tell one story with two morals. First, every layer's ceiling is set by verification. A visible CoT is not assurance evidence, tool chains compound errors without checks, search is blind without value functions, RLVR works best where rewards are checkable. Test-time compute pays off in proportion to the ability to select among candidates. The reasoning stack is really a generation stack waiting for stronger verification outside mathematics, code and other mechanically checkable tasks.

Second, the frontier question in 2026 is not "can models reason?": within checkable domains, yes, demonstrably, superhumanly in spots. It's whether we can manufacture verifiable signal for the domains we actually care about: science, engineering judgment, strategy, the messy open-ended work. Every credible lab roadmap I've seen is, under the branding, an answer to that one question. Watch the verifiers, not the chains.

The enterprise stack is not the research stack

The five research layers describe ways to generate and select candidate reasoning. A production system needs four additional planes around them: context, authority, evidence and operations. Context supplies the permitted facts. Authority limits what a tool call may do. Evidence records what was retrieved, proposed, checked and executed. Operations keeps the service inside latency, cost and recovery bounds.

This distinction prevents a common architecture error. A model with stronger reasoning is installed at the centre and expected to compensate for stale case state, permissive tools and weak release tests. It cannot. More inference may improve a plan while increasing the confidence with which the system acts on the wrong customer record. Reasoning quality cannot repair an invalid system boundary.

Each plane should be independently testable. Context tests cover entitlement, freshness, provenance and conflict. Reasoning tests cover task success and failure modes. Verification tests cover false acceptance as well as false rejection. Authority tests enumerate principal, tool and argument combinations. Operations tests cover timeouts, retries, partial execution and rollback. Putting every concern into one end-to-end score leaves no diagnosis when the score moves.

A verification pyramid for open-ended work

Banking and consulting tasks are rarely verified by one unit test. That does not make them unverifiable. It means verification must be decomposed. At the bottom are exact checks: schema validity, arithmetic, identifiers, dates and policy thresholds. Above them are evidence checks: whether each material claim points to an entitled and current source. Next are process checks: whether required steps, approvals and separation of duties occurred. The top contains human judgement over residual interpretation.

This pyramid changes the economics of reasoning. Do not spend model tokens rediscovering what an exact rule can establish. Use the model for the residual that remains after lower layers have compressed uncertainty. The same ordering improves oversight: a reviewer receives a short list of unresolved judgements rather than an undifferentiated narrative.

Verifiability is designed, not found. A broad instruction such as “assess the credit risk” offers little mechanical purchase. Split it into evidence collection, financial reconciliation, covenant tests, policy exceptions and a bounded risk interpretation. The first four produce checkable artefacts. Human judgement then concentrates on the part that is genuinely interpretive.

Mapping layers to a live banking journey

Consider a credit refer-back workflow. A relationship manager submits an application. The system must identify missing or inconsistent evidence, apply policy and return specific requests without making the lending decision. The temptation is a single long prompt. The stronger design assigns a different reasoning layer to each uncertainty.

Retrieval and exact checks run first. Tool use fetches the application, financial statements, KYC state and product policy through typed interfaces. A planner proposes a short set of checks, but code enforces the permitted sequence. Search or multiple candidates are reserved for ambiguous policy mapping. Test-time compute increases only when material evidence conflicts. The final tool can create a refer-back task but cannot approve credit.

The architecture exposes where an error came from. A missed document may be retrieval. A wrong covenant calculation is deterministic logic or data. An incorrect interpretation is reasoning. An unauthorized action is policy enforcement. A long response time is routing or operations. A named failure owner is more useful than a global “reasoning accuracy” score.

The failure budget belongs to the trajectory

Per-call accuracy is a misleading unit for a multi-step agent. The trajectory succeeds only when context, planning, tool selection, arguments, execution and verification all remain acceptable. Independent-step multiplication is a crude approximation because errors are correlated, but it makes one point unavoidable: adding steps creates reliability debt.

The response is not to demand perfect models. It is to reduce probabilistic surface area. Collapse stable decisions into code. Combine calls that do not need independent reasoning. Stop as soon as required evidence is missing. Use idempotent tools and resumable business state. Place a human at transitions where the consequence is high and the verifier is weak.

The budget should cap more than errors. It should include maximum model calls, tool calls, elapsed time, spend and unverified claims. A trajectory that has not failed yet can still be outside its operating envelope. Enforce the cap in the runtime rather than asking the model to remember it.

Release the stack one layer at a time

Start with the lower verification layers and a human-held action. Establish the evidence bundle, typed tools and replayable traces before adding search or extended compute. Next, automate low-consequence reversible actions whose outcomes can be checked. Only then test whether another reasoning layer produces enough incremental value to justify its new failure modes.

A layer earns release with a matched-compute comparison against the simpler stack. It must improve a named production measure on representative and challenge sets, stay within tail-latency and cost limits, and produce evidence that the control owner can interpret. Roll it back independently when possible. A monolithic prompt revision makes the entire stack one deployment unit and weakens diagnosis.

The stack is therefore less a ladder of intelligence than a sequence of engineering choices. Chain of thought spends inference on additional workspace. Tools buy exact external capability. Search buys alternatives. Verifiable-reward training buys procedures in domains with strong feedback. Test-time compute buys more sampling or refinement. None of them buys governance, context integrity or permission.

An evaluation schema that matches the stack

One end-to-end task score is necessary and insufficient. It tells the release board whether the whole system improved. It does not tell engineering what to change or risk which control weakened. Pair it with component measures tied to the architecture.

For chain-of-thought and direct reasoning, score final checked answers, calibration and sensitivity to irrelevant prompt variation. Do not score the plausibility of the visible chain as proof of internal correctness. For tool use, score whether a tool was needed, whether the correct tool was selected, whether arguments were valid and whether the observed result was incorporated. For search, report marginal gain against sample count and verifier error. For extended compute, report gain and regression by budget, difficulty and consequence.

The evaluation record should preserve both an oracle result and the production result. The oracle run receives the correct evidence and, where useful, the correct tool. The production run must retrieve and select for itself. The difference isolates orchestration and context error from core task error. Without the oracle comparison, teams often respond to retrieval failure by changing the model.

Review false acceptance separately from false rejection. A cautious verifier that rejects every candidate may look accurate on an imbalanced set while destroying straight-through processing. An optimistic verifier may preserve throughput while allowing material errors. Set thresholds from consequence and reviewer capacity rather than a generic F1 target.

The minimum release packet

A reasoning-stack release packet should contain the task contract, allowed tools, authority boundary, context schema, evaluation-set versions, component and end-to-end results, matched-compute comparator, latency and cost distributions, known weak slices, recovery test and rollback identifier. Include trace examples for success, safe abstention, verifier rejection and a recovered partial execution.

The packet should also state what did not change. A model upgrade may leave policy, retrieval and tools constant. An orchestration change may leave the model constant. This narrows the causal claim and the rollback. If every layer changes in one release, a passing average score provides weak evidence about which new component created a tail regression.

Architecture review should challenge three hidden assumptions. First, is a probabilistic decision being used where code could decide? Second, is a generated statement being treated as independent evidence? Third, does the agent possess authority broader than the business outcome requires? Most dangerous designs reveal themselves through one of those questions before a benchmark is run.

Where human judgement actually belongs

“Human in the loop” is too imprecise to design. A person can supply missing evidence, approve an exception, choose among genuinely subjective options, verify a high-consequence interpretation or execute an action. Each role needs a different interface and service expectation.

Do not ask a reviewer to reread the model's entire trajectory. Present the unresolved proposition, supporting and conflicting sources, checks already completed, authority requested and effect of acceptance. The system should show why it could not resolve the case and what changed since similar cases. That turns review into a decision rather than a quality-control scavenger hunt.

Measure human outcomes as part of the stack. Record time to decision, override, reason for override, requests for additional evidence and later correction. High acceptance is not automatically good; it may indicate appropriate automation or automation bias. Periodically seed adjudicated cases and compare decisions with and without the generated recommendation.

The strongest human boundary is one the system cannot route around. Enforce it in the action service or policy engine. A prompt instruction to ask for approval is behaviour, not authorization. The runtime may forget it, a prompt injection may oppose it, or a future version may interpret it differently.

The practical ranking

For most enterprise work, I rank the investments in this order: governed context, typed tools, deterministic verification, trajectory evidence, bounded reasoning, then search or extra compute. This is not a ranking of research importance. It is a ranking of production leverage. The early items remove broad classes of error and strengthen every model. The later items buy narrower gains whose value depends on the foundation.

There are exceptions. A theorem prover or code-repair system with a strong verifier may rationally invest heavily in search and verifiable-reward training. A low-latency customer intent router may need almost none of the upper stack. The architecture follows the task's verification structure, consequence and interaction deadline.

The durable insight is that reasoning becomes useful when it is placed. The question is not whether a model can produce a long chain. It is which part of a workflow deserves probabilistic computation, what constrains it, what checks it and what evidence remains after it acts.

A map for choosing the layer

The research lineage includes chain-of-thought prompting, Toolformer, Tree of Thoughts, DeepSeek-R1 and work on test-time compute scaling. Each paper supports a bounded empirical claim, not the whole stack.