The Bitter Lesson and the Neuro-Symbolic Reply

What the bitter lesson actually teaches

Richard Sutton’s 2019 essay compresses a recurring historical pattern. Researchers encode their understanding of a domain, gain an early advantage, then watch a more general method overtake it as computation becomes cheaper. Sutton identifies search and learning as the methods that can keep absorbing additional computation. The warning is directed at attempts to build the contents of human thought into an intelligent system, particularly when that knowledge becomes a ceiling on improvement.¹

AlphaZero is a compelling exhibit. Earlier chess engines combined deep search with large stores of evaluation knowledge and domain-specific engineering. AlphaZero began from random play and, apart from the rules, used one reinforcement-learning and search procedure across chess, shogi and Go. It learned its evaluation and policy through self-play rather than inheriting a catalogue of human strategies.² The result strongly supports abandoning hand-authored strategic judgement when a scalable learning loop can generate better judgement.

Yet the phrase “apart from the rules” matters. AlphaZero did not need to learn that a bishop moves diagonally by crashing into millions of illegal transitions. The rules defined the simulator, legal actions and terminal conditions within which search and learning operated. That structure did not tell the system which legal move was good. It made the search problem well formed.

MuZero pushes much further towards the bitter lesson. It learns a latent dynamics model sufficient for predicting reward, policy and value, then plans inside that learned model. On Go, chess and shogi it matched AlphaZero without being supplied the game dynamics; on Atari it combined learned dynamics with search across visually complex environments.³ This is strong evidence that even apparently foundational domain structure can sometimes be learned.

MuZero also exposes the qualifier. More search helped substantially in Go, but its gains plateaued in Atari, which the authors associated with model inaccuracy. Search cannot repair a future model that becomes wrong as it rolls forward. Additional computation amplifies whatever transition and value structure the system has learned. Compute is a multiplier, not a substitute for the correctness of the world on which the computation operates.

The bitter lesson is best read as an investment prior: before encoding domain judgement, test whether data, search and compute can discover a better version. The more open-ended the target, the stronger that prior.

It is weaker as a claim about small, extensional boundaries. A legal-move predicate, a type checker and a cryptographic signature test do not attempt to reproduce the contents of intelligence. They reject a formally characterised subset of outputs. Their value does not come from outperforming learning at strategy. It comes from making one proposition decidable before an effect.

The lesson has a jurisdiction: it attacks hand-crafted solutions to open-ended intelligence, not every compact constraint that defines a valid interaction. Extending it beyond that jurisdiction turns a historical observation into an ideology.

Three kinds of structure that should not be collapsed

Arguments about neuro-symbolic systems often treat all human contribution as “knowledge”. That makes the debate impossible to resolve. At least three different objects are being mixed.

First, there is inductive structure. A convolution assumes locality; attention supplies content-addressed interaction; a tree search assumes that imagined continuations are worth comparing. These structures shape what learning can discover efficiently. They are not normally lists of domain facts, and they remain subject to the bitter lesson: a better general architecture may displace them.

Second, there is strategic domain knowledge. Examples include a hand-written chess evaluation, a catalogue of visual features or a large expert-rule system that tries to decide what to do. This is the bitter lesson’s clearest target. Such knowledge can bring data efficiency, but it often freezes the designer’s decomposition and fails to absorb increased compute.

Third, there is admissibility and verification structure. Examples include a grammar, a type relation, a conservation law, a policy threshold, a proof kernel or a precondition on a tool call. This structure does not need to rank every possible action. It says that a candidate cannot be released unless a proposition holds.

The third category is not automatically good. A giant rule base can become another brittle expert system. A local checker can still encode the wrong rule. The relevant question is whether the proposition is smaller, more stable and more directly testable than the behaviour being generated. When it is, explicit representation can reduce the burden on learning without prescribing the solution.

Why rare constraints survive scale

The strongest causal case for explicit constraint is not that neural networks are incapable of learning rules. It is that some important rules are observed mainly through their violation, while the violations are deliberately rare. A learner cannot infer an exception it never encounters unless it transfers the rule from other evidence or receives it through the training objective.

Let an independently sampled action encounter a particular exception with probability ε. After n observations, the probability of seeing at least one such exception is:

To reach confidence 1 − δ of seeing even one example, the required count is:

At an exception rate of 0.1 per cent, roughly 2,995 observations give a 95 per cent chance of seeing one case. At one per million, the count is about three million. Conjunctive rules need the relevant combinations, and time-varying rules make old samples less probative.

Rare-event sample complexity creates a region where an explicit rule can be cheaper than empirical rediscovery, even though learning would eventually approximate it. The word “cheaper” includes failed actions, delayed release, human labelling and the evidence required to demonstrate a residual risk.

Thought experiment: change only rarity, then only consequence

Consider two plants with identical sensors, controllers and actions. In Plant A, a wrong route occurs one time in twenty, pauses a conveyor for five minutes and resets. The error is visible, reversible and frequent, so a reinforcement learner can explore and improve.

Hold the task fixed but reduce the exception to one in a million. The learner now receives far less evidence. Hold rarity fixed again and make the effect an irreversible contaminant release. The learning problem is unchanged, but the exploration budget collapses.

Rarity controls whether feedback arrives; consequence controls whether discovery by violation is tolerable. High values on both axes require information from simulation, synthetic counterexamples, a formal constraint, human review or some combination.

A symbolic guard is not the only response. Oversampling, simulation, transfer and verifier-labelled trajectories can move boundary information into learning. The information need not be written as logic. Scale simply cannot learn from evidence the system never receives.

Behavioural reliability is not logical guarantee

A capable model may behave as though it knows a constraint, which can be strong operational evidence. Reliability is measured over cases and a distribution; a proof quantifies over a formal state space and inherits that formalisation’s assumptions.

A learned rule can be more accurate than a written rule, while a written checker can still provide a stronger claim over the small region it represents correctly. The architecture should preserve both facts instead of turning one into a verdict on the other.

The counterexample learning must receive

A weak neuro-symbolic argument says neural networks cannot be systematic and therefore need an external symbolic engine. The evidence no longer supports that categorical claim. Early results on SCAN showed recurrent sequence models succeeding when test commands resembled training combinations but failing badly when a new primitive had to compose systematically with known operators.⁴ That result made a genuine failure visible, but it did not establish an architectural impossibility.

Meta-learning for compositionality later trained an ordinary neural architecture across a distribution of compositional tasks. On the studied human experiments and benchmarks, it combined flexibility with systematic generalisation better than both rigid symbolic models and conventional neural baselines.⁵ The systematic bias was induced by the training distribution rather than installed as a symbolic module.

This is a serious negative control for the paper’s preferred design. Symbolic machinery is not a necessary condition for learned systematicity. A capability that looks absent under one objective may appear when the data and meta-objective reward it. The bitter lesson is alive inside the compositionality debate.

The same study also marks its boundary. The trained system did not automatically generalise to structures outside its meta-learning distribution and failed particular productivity splits. The result therefore supports a more careful claim: learning can acquire rule-like behaviour, but evidence of acquisition is tied to the training and evaluation regime. It does not turn all future combinations into certified cases.

The practical consequence is demanding. Before adding symbols, construct a learned-only baseline with the right data, objective, search and compute. Before removing symbols, construct a shift set in which rare or novel combinations matter. Neuro-symbolic value exists only in the difference between those matched systems.

The neuro-symbolic reply without nostalgia

The useful reply to the bitter lesson is not to rebuild a large expert system beside a neural model. It is to identify the smallest interface at which explicit semantics changes the experiment. Structure can enter at three different depths, and each supports a different claim.

Semantic loss compiles propositional constraints into a differentiable objective.⁶ DeepProbLog places neural predicates inside probabilistic logic programs.⁷ Neural Logic Machines use differentiable logical operators and report generalisation from smaller to larger synthetic tasks.⁸

These approaches can improve sample efficiency or compositional transfer. They do not all provide the same runtime guarantee. A logic-shaped loss may reduce violations without eliminating them. A probabilistic logic program can preserve interpretable dependencies while still depending on uncertain neural predicates. The words “symbolic” and “neuro-symbolic” therefore say too little about the operating claim.

Geometry shows the complementarity more clearly

AlphaGeometry used a language model to propose auxiliary constructions and a symbolic engine to close their consequences. The model ranked branches; the engine handled exact derivation. On its thirty-problem set, the system solved twenty-five.⁹

The result is impressive, but the ablations and later comparisons are more informative than the label. A re-evaluation found that a stronger fully symbolic combination solved twenty-one of the thirty problems on modest hardware, and that Wu’s method solved two problems missed by AlphaGeometry. Combining the methods reached twenty-seven.¹⁰ The conclusion is not that the hybrid was unnecessary. It is that each component needs a credible independent baseline.

Representation remained a bottleneck. AlphaGeometry2 expanded language coverage from 66 to 88 per cent on the reported 2000–2024 geometry set; combined improvements raised the overall solve rate to 84 per cent.¹¹ Engineering the structure remained part of the research burden.

AlphaProof makes the relationship even starker. Reinforcement learning and large-scale search operate inside Lean, while Lean’s kernel checks the generated proof term. The system learned from millions of formalised problems and used multi-day test-time reinforcement learning on difficult competition questions. The kernel did not invent the proof. It made successful completion mechanically verifiable within the formal system.¹²

The checker is not the solver, and the solver is not the checker. Learning supplies a distribution over promising proof actions. Search converts that distribution into exploration. The proof assistant supplies exact state transitions and a terminal validity test. Removing the learner makes the search combinatorial. Removing the kernel makes correctness an empirical judgement about generated text.

TongGeometry continues the pattern: fine-tuned models guide symbolic exploration, with strong results on IMO-AG-30 and explicit representational limits.¹³ The frontier uses scale to search inside verifiable environments.

The proposal-admissibility split

The general architecture can be stated without committing to a particular neural or symbolic technology. Let a learned model and search process assign an expected utility U_θ(a | x) to candidate action a, given context x. Let a set of explicit predicates C_i inspect authoritative state s, policy version p and the typed action. The selected action is:

This formulation protects the bitter lesson. The model remains free to discover representations, plans and preferences. No human supplies the ranking. The constraint layer says only which proposals may cross the boundary. It can return reasons, request missing evidence or abstain when no candidate is admissible.

The split is strongest when the constraint is local. “A transfer above this amount requires two independent approvals” can be evaluated over a typed request and current approval state. “Always act in the customer’s best interest” cannot be reduced honestly to one Boolean predicate. The latter needs evidence, modelling, human judgement and contestability, not a decorative rule engine.

Explicit structure should be placed at the narrowest boundary where its inputs, semantics, owner and test oracle are all visible. Moving it deeper into the model makes enforcement harder to inspect. Expanding it into a complete world model recreates the brittleness the bitter lesson warns against.

The guarantee ends sooner than most diagrams admit

A checker evaluates a formula over represented inputs. Its guarantee is conditional on five things: the rule is correct; the representation is complete enough; the state is current; the candidate action has the semantics assumed by the checker, and no execution path bypasses enforcement. Violate any one and the system can be formally clean yet operationally wrong.

A proof assistant can verify a theorem that was mistranslated. A policy engine can approve an action against a stale entitlement snapshot. A schema can validate a payload whose identifier refers to the wrong customer. A tool can return “accepted” while the external effect remains unknown. Symbolic certainty over the wrong state is false assurance with excellent formatting.

Safe reinforcement learning through shielding makes the intended separation explicit: a learner optimises reward while a synthesised shield enforces temporal-logic properties before or after the proposed action.¹⁴ The approach is attractive because the safety analysis can be partly agnostic to the learner. Its own limitation is equally important: the shield requires a suitable abstraction of environment state and an accurate specification.

Build the split and test it

Minimal worked example: six release actions

A model proposes six ways to complete a synthetic work order. Its score estimates benefit after latency and uncertainty. Three exact conditions apply: a hazardous job needs a permit; an irreversible action needs dual control, and the evidence snapshot must be no more than thirty days old.

The model-only system selects A. A rule-only system can avoid the invalid actions but may choose poorly among B, E and F if its preference function is crude. The hybrid removes A, C and D, then preserves the model’s learned ranking among the valid candidates and selects B. The constraint contributes no strategic intelligence; the model contributes no authority to waive the constraint.

Production-shaped worked scenario: an evidence-bound refund

An assistant receives a request to investigate and, where permitted, prepare a refund for an apparent duplicate payment. A language model can assemble evidence, distinguish plausible causes and propose the next action. The action kernel should not infer authority from the fluency of that proposal. The control system needs ten separate objects:

The symbolic contribution is deliberately small. It does not encode how to recognise every duplicate-payment pattern. It enforces preconditions such as a verified destination, amount bounds, current entitlement and duplicate prevention. A deterministic calculation establishes the amount. The model remains useful because the evidence is messy, the hypotheses vary and the explanation must fit the case.

If policy is ambiguous, the kernel should not manufacture precision. It can route to a qualified reviewer with the assembled evidence. Bounded autonomy includes a typed refusal to act when no machine-checkable authority exists.

A hybrid benchmark that lets learning win when it can

The accompanying Python benchmark isolates the paper’s mechanism. Each synthetic case contains six candidate release actions. Every model-led system receives the same noisy proposal score. Three observable hazards make an action invalid: a required permit is missing, an irreversible action lacks dual control, or evidence is older than thirty days.

The empirical guard is deliberately favoured: each correct hazard flag is already present, and one labelled violation teaches the whole flag. Training exceptions occur at 0.1 per cent; test exceptions shift to 15 per cent. Sixty seeds each evaluate 1,500 cases, giving 90,000 decisions per system.

Four baselines make the result discriminating: unguarded proposal; correct rules with coarse ranking; the exact hybrid, and a negative control whose freshness limit is twenty days rather than thirty.

Intervals are the tenth to ninetieth percentiles across seeds. Invalid selections receive a synthetic utility of −12; abstentions receive zero. These numbers do not estimate a real operational rate. They demonstrate the causal trade: what changes when only the admissibility mechanism changes.

At 250 training actions, the median run sees no exception and behaves like the unguarded model. At 2,500 it sees three exceptions and learns two hazards, with wide seed variation. At 25,000 it learns all three and matches the exact hybrid.

The benchmark does not say learning fails. It shows learning succeeding once the relevant evidence arrives, while making the cost of waiting for that evidence visible. This is the bitter lesson in its strongest form. Enough data eliminates the performance gap in this favourable finite setting.

The other baselines prevent a symbolic victory lap. Rule-only control removes violations but loses utility. The exact hybrid combines the stronger ranking and correct boundary. The wrong rule also records zero violations, yet lowers utility to 4.505, changes 33.6 per cent of choices and sometimes abstains. A bad rule can look safest on its own metric.

The executable artefact

The program uses only Python’s standard library. Run it with python paper16_hybrid_benchmark.py. The seed family, exception rates, candidate count, invalid-action penalty and evaluation population are declared at the top. Expected output is the table above plus the analytic encounter count for a 0.1 per cent exception.

#!/usr/bin/env python3
"""A synthetic benchmark for the proposal-admissibility split.

The benchmark gives learning an unusually favourable task: every hard failure is
already represented by one of three observable hazard flags. The empirical guard
only has to observe a hazard once to learn that it must be blocked. The hybrid
system receives the three exact constraints directly. All systems use synthetic
data and a fixed family of random seeds.
"""

from __future__ import annotations

from dataclasses import dataclass
import math
import random
from statistics import mean, median
from typing import Callable, Iterable

BASE_SEED = 16016
REPLICATES = 60
TRAIN_EXCEPTION_RATE = 0.001  # 0.1% of training actions
TEST_EXCEPTION_RATE = 0.15    # shifted test: 15% of candidate actions
TEST_CASES = 1_500
CANDIDATES_PER_CASE = 6
INVALID_PENALTY = -12.0


@dataclass(frozen=True)
class Action:
    gain: float
    latency: float
    uncertainty: float
    requires_permit: bool
    permit_present: bool
    irreversible: bool
    dual_control: bool
    evidence_age_days: float
    proposal_score: float


def hazards(action: Action, freshness_limit: float = 30.0) -> frozenset[str]:
    found: set[str] = set()
    if action.requires_permit and not action.permit_present:
        found.add("missing_permit")
    if action.irreversible and not action.dual_control:
        found.add("missing_dual_control")
    if action.evidence_age_days > freshness_limit:
        found.add("stale_evidence")
    return frozenset(found)


def is_valid(action: Action) -> bool:
    return not hazards(action)


def true_utility(action: Action) -> float:
    """Synthetic utility before admissibility is considered."""
    return (
        8.0 * action.gain
        - 2.5 * action.latency
        - 2.0 * action.uncertainty
        + 1.5 * action.gain * (1.0 - action.uncertainty)
    )


def generate_action(rng: random.Random, exception_rate: float) -> Action:
    gain = rng.random()
    latency = rng.random()
    uncertainty = rng.random()
    requires_permit = rng.random() < 0.35
    irreversible = rng.random() < 0.20

    # Safe by construction unless one exceptional condition is injected.
    permit_present = True
    dual_control = True
    evidence_age_days = rng.uniform(0.0, 30.0)

    if rng.random() < exception_rate:
        kind = rng.choice(("permit", "dual", "stale"))
        if kind == "permit":
            requires_permit = True
            permit_present = False
        elif kind == "dual":
            irreversible = True
            dual_control = False
        else:
            evidence_age_days = rng.uniform(30.01, 60.0)

    provisional = Action(
        gain=gain,
        latency=latency,
        uncertainty=uncertainty,
        requires_permit=requires_permit,
        permit_present=permit_present,
        irreversible=irreversible,
        dual_control=dual_control,
        evidence_age_days=evidence_age_days,
        proposal_score=0.0,
    )
    return Action(
        **{
            **provisional.__dict__,
            # Same noisy learned proposal signal for every model-led system.
            "proposal_score": true_utility(provisional) + rng.gauss(0.0, 0.45),
        }
    )


def build_cases(
    rng: random.Random,
    n_cases: int = TEST_CASES,
    candidates: int = CANDIDATES_PER_CASE,
    exception_rate: float = TEST_EXCEPTION_RATE,
) -> list[list[Action]]:
    cases: list[list[Action]] = []
    for _ in range(n_cases):
        case = [generate_action(rng, exception_rate) for _ in range(candidates)]
        if not any(is_valid(action) for action in case):
            case[0] = generate_action(rng, 0.0)
        cases.append(case)
    return cases


def learn_hazards(training: Iterable[Action]) -> frozenset[str]:
    """Optimistic empirical learner: one labelled failure teaches a whole flag."""
    learned: set[str] = set()
    for action in training:
        if not is_valid(action):
            learned.update(hazards(action))
    return frozenset(learned)


def best(case: list[Action], eligible: Callable[[Action], bool], score) -> Action | None:
    allowed = [action for action in case if eligible(action)]
    return max(allowed, key=score, default=None)


def choose_unguarded(case: list[Action]) -> Action | None:
    return best(case, lambda _: True, lambda action: action.proposal_score)


def choose_empirical(case: list[Action], learned: frozenset[str]) -> Action | None:
    return best(
        case,
        lambda action: hazards(action).isdisjoint(learned),
        lambda action: action.proposal_score,
    )


def choose_rule_only(case: list[Action]) -> Action | None:
    # Exact admissibility, but a deliberately coarse hand-written preference.
    return best(
        case,
        is_valid,
        lambda action: 5.0 * action.gain - 4.0 * action.latency,
    )


def choose_hybrid(case: list[Action]) -> Action | None:
    return best(case, is_valid, lambda action: action.proposal_score)


def choose_wrong_rule(case: list[Action]) -> Action | None:
    # Negative control: the explicit rule says 20 days, but truth says 30 days.
    return best(
        case,
        lambda action: not hazards(action, freshness_limit=20.0),
        lambda action: action.proposal_score,
    )


def evaluate(cases: list[list[Action]], chooser: Callable[[list[Action]], Action | None]) -> dict[str, float]:
    realised: list[float] = []
    violations = 0
    abstentions = 0
    changed_from_exact = 0

    for case in cases:
        selected = chooser(case)
        exact = choose_hybrid(case)
        if selected is None:
            abstentions += 1
            realised.append(0.0)
        elif not is_valid(selected):
            violations += 1
            realised.append(INVALID_PENALTY)
        else:
            realised.append(true_utility(selected))

        if selected != exact:
            changed_from_exact += 1

    n = len(cases)
    return {
        "coverage_pct": 100.0 * (n - abstentions) / n,
        "violation_pct": 100.0 * violations / n,
        "mean_realised_utility": mean(realised),
        "changed_from_exact_pct": 100.0 * changed_from_exact / n,
    }


def percentile(values: list[float], q: float) -> float:
    ordered = sorted(values)
    position = (len(ordered) - 1) * q
    lower = math.floor(position)
    upper = math.ceil(position)
    if lower == upper:
        return ordered[lower]
    fraction = position - lower
    return ordered[lower] * (1.0 - fraction) + ordered[upper] * fraction


def summarise(rows: list[dict[str, float]], key: str) -> tuple[float, float, float]:
    values = [row[key] for row in rows]
    return median(values), percentile(values, 0.10), percentile(values, 0.90)


def run() -> None:
    training_sizes = (250, 2_500, 25_000)
    results: dict[str, list[dict[str, float]]] = {
        f"empirical_{size}": [] for size in training_sizes
    }
    for name in ("unguarded", "rule_only", "hybrid_exact", "hybrid_wrong_rule"):
        results[name] = []

    learned_counts: dict[int, list[int]] = {size: [] for size in training_sizes}
    observed_exceptions: dict[int, list[int]] = {size: [] for size in training_sizes}

    for replicate in range(REPLICATES):
        rng = random.Random(BASE_SEED + replicate)
        cases = build_cases(rng)

        for size in training_sizes:
            training = [generate_action(rng, TRAIN_EXCEPTION_RATE) for _ in range(size)]
            learned = learn_hazards(training)
            learned_counts[size].append(len(learned))
            observed_exceptions[size].append(sum(not is_valid(action) for action in training))
            results[f"empirical_{size}"].append(
                evaluate(cases, lambda case, learned=learned: choose_empirical(case, learned))
            )

        results["unguarded"].append(evaluate(cases, choose_unguarded))
        results["rule_only"].append(evaluate(cases, choose_rule_only))
        results["hybrid_exact"].append(evaluate(cases, choose_hybrid))
        results["hybrid_wrong_rule"].append(evaluate(cases, choose_wrong_rule))

    print("Synthetic proposal-admissibility benchmark")
    print(
        f"replicates={REPLICATES}; test_cases/replicate={TEST_CASES}; "
        f"candidates/case={CANDIDATES_PER_CASE}"
    )
    print(
        f"training exceptions={100 * TRAIN_EXCEPTION_RATE:.1f}%; "
        f"shifted test exceptions={100 * TEST_EXCEPTION_RATE:.0f}%"
    )
    print("Intervals are 10th to 90th percentiles across random seeds.")
    print()
    print(
        "system,train_actions,observed_exceptions_median,learned_hazards_median,"
        "violation_pct_median,violation_pct_p10,violation_pct_p90,"
        "utility_median,utility_p10,utility_p90,coverage_pct_median,"
        "changed_from_exact_pct_median"
    )

    order = [f"empirical_{size}" for size in training_sizes] + [
        "unguarded",
        "rule_only",
        "hybrid_exact",
        "hybrid_wrong_rule",
    ]
    for name in order:
        rows = results[name]
        violation = summarise(rows, "violation_pct")
        utility = summarise(rows, "mean_realised_utility")
        coverage = summarise(rows, "coverage_pct")
        changed = summarise(rows, "changed_from_exact_pct")
        if name.startswith("empirical_"):
            size = int(name.split("_")[1])
            observed = median(observed_exceptions[size])
            learned = median(learned_counts[size])
        else:
            size = 0
            observed = 0
            learned = 0
        print(
            f"{name},{size},{observed:.1f},{learned:.1f},"
            f"{violation[0]:.3f},{violation[1]:.3f},{violation[2]:.3f},"
            f"{utility[0]:.3f},{utility[1]:.3f},{utility[2]:.3f},"
            f"{coverage[0]:.3f},{changed[0]:.3f}"
        )

    # A transparent analytic check on the rare-event mechanism.
    for confidence in (0.95, 0.99):
        n = math.log(1.0 - confidence) / math.log(1.0 - TRAIN_EXCEPTION_RATE)
        print(
            f"actions for {100 * confidence:.0f}% chance of seeing at least one "
            f"0.1% exception: {math.ceil(n):,}"
        )


if __name__ == "__main__":
    run()

Synthetic proposal-admissibility benchmark
replicates=60; test_cases/replicate=1500; candidates/case=6
training exceptions=0.1%; shifted test exceptions=15%
Intervals are 10th to 90th percentiles across random seeds.

system,train_actions,observed_exceptions_median,learned_hazards_median,violation_pct_median,violation_pct_p10,violation_pct_p90,utility_median,utility_p10,utility_p90,coverage_pct_median,changed_from_exact_pct_median
empirical_250,250,0.0,0.0,14.933,10.667,15.880,2.974,2.777,3.637,100.000,14.933
empirical_2500,2500,3.0,2.0,5.733,0.000,11.813,4.412,3.452,5.335,100.000,5.733
empirical_25000,25000,25.0,3.0,0.000,0.000,0.000,5.311,5.264,5.364,100.000,0.000
unguarded,0,0.0,0.0,15.167,14.067,16.133,2.925,2.741,3.115,100.000,15.167
rule_only,0,0.0,0.0,0.000,0.000,0.000,4.947,4.894,4.996,100.000,33.500
hybrid_exact,0,0.0,0.0,0.000,0.000,0.000,5.311,5.264,5.364,100.000,0.000
hybrid_wrong_rule,0,0.0,0.0,0.000,0.000,0.000,4.505,4.429,4.566,99.333,33.600
actions for 95% chance of seeing at least one 0.1% exception: 2,995
actions for 99% chance of seeing at least one 0.1% exception: 4,603

A placement decision instrument

A team should not begin with “we need neuro-symbolic AI”. It should begin with a candidate proposition and try to earn the right to enforce it. The following test turns the philosophical dispute into an architecture decision.

Constraint locality test

1. Precision: Can two qualified people state the condition so that independent implementations agree on ordinary and edge cases?
2. Observability: Are the required inputs available from authoritative, fresh and identity-bound state before the action?
3. Consequence: Is violation materially harmful, difficult to reverse or unacceptable during exploration?
4. Rarity: Would representative data contain enough violations and combinations to learn and evaluate the boundary?
5. Ownership: Does the rule have an accountable owner, effective time, version and withdrawal path?
6. Local enforceability: Can the condition be checked at one typed interface without modelling the whole world?
7. Independent outcome evidence: Can the system verify the postcondition and distinguish rejection, success and unknown outcome?

High scores on precision, consequence, ownership and local enforceability favour an explicit runtime guard. Low precision favours learned judgement or human review. Weak observability blocks a hard guarantee even when the policy is crisp. Low consequence and abundant feedback favour learning, because adaptation may matter more than zero empirical violations.

Choose the weakest sufficient form of structure

The experiment should then compare at least three systems under matched proposal compute: learned-only, structure-only and hybrid. Test ordinary data, rare exceptions and distribution shift. Report useful outcome, violation rate, coverage, abstention, latency and total cost. Mutate or age the rule state so that an incorrect constraint cannot win merely by defining its own success metric.

This final step prevents architectural branding. A hybrid earns its complexity only if it reduces a consequential failure or evaluation burden without an unacceptable loss of coverage, utility or adaptability. A learned-only system earns removal of the guard only if fresh evidence shows that the residual risk and recovery path are acceptable. The decision is empirical, but the evidence required depends on the kind of claim being made.

What would overturn the placement

The placement rule is falsifiable. Freeze the proposal model, candidate set, compute budget and outcome evaluator. Then compare learned-only, structure-only and hybrid systems across ordinary cases, deliberately rare combinations, shifted frequencies, stale state and mutated rules. Record useful outcome, material violations, abstention, coverage, latency, review burden and recovery cost. Run the explicit check in shadow mode before giving it authority, so false restrictions become visible without blocking valid work. The guard earns release only when its reduction in consequential error survives this matched comparison and remains larger than the cost of maintaining its rule and state dependencies.

Evidence should count against the guard when a learned system reaches the same bounded violation rate and recovery performance under the relevant shifts, while adapting faster or preserving materially more utility. It should also count against the guard when specification disagreements, stale inputs or bypass paths cause more harm than the boundary prevents. In that case, retire the rule, reduce it to an advisory signal or route the ambiguous proposition to human judgement. Conversely, zero observed violations is never sufficient on its own. Report the evaluation denominator, uncertainty, injected failure coverage and the cases the checker could not represent. The decision is not whether symbols feel reassuring. It is whether the constrained interface improves outcomes under tests capable of making it lose.

Two adjacent articles delimit the neighbouring questions: What Scaling Laws Don’t Tell You covers capability and economics, while The Certainty Gradient covers mechanism placement. This paper supplies the causal boundary.