The fourth decimal place

An arrears team can review 220 of every 1,000 incoming cases. A model decides which cases enter the queue. The review itself costs one unit; an adverse case left outside the queue costs eight. A slide presents a gradient-boosting model as the winner because its area under the receiver operating characteristic curve is 0.009 higher than logistic regression on a random validation split.

The number sounds precise. The decision is not. Were the two models given the same features at the same prediction time? Was feature engineering allowed for one and forbidden for the other? Did hyperparameter search reuse the reported fold? Were probabilities calibrated on untouched data? Does the queue act on a fixed threshold or a fixed capacity? What maintenance, latency and explanation work arrives with the challenger? A model comparison that leaves these questions open is measuring a mixture of algorithm, representation, selection noise and operating design.

Now change only the experimental discipline. Freeze the decision time and place earlier months in development, with later months held for test. Give both models the same raw facts and allow three predeclared interactions in the linear feature map. Tune each within the same validation month, calibrate both on a separate month, and measure the cost of the top 22 per cent of cases. In the executable scenario later in this article, the engineered linear model reaches an AUC of 0.777 and costs 2,041 synthetic units per 1,000 cases. The tuned boosting model reaches 0.768 and costs 2,098. A paired bootstrap places the booster's extra cost between roughly 19 and 107 units per 1,000 cases.

The lesson is wider than this one synthetic result. A baseline is an intervention on the space of explanations. It asks what the available variables can achieve with one global additive boundary, then exposes the structure a challenger must add. If the gain vanishes under a fair split, one learned a property of the evaluation. If one interaction closes the gap, one learned a property of the representation. If a flexible model still wins on later data and business loss, one has evidence that complexity is doing useful work.

This is why “start simple” is too weak. It sounds like a preference for austerity. The stronger rule is: begin with the simplest model whose restrictions make the next claim testable. The baseline is not the model one hopes to ship. It is the model that makes the cost of leaving its hypothesis class visible.

A boundary you can inspect

Consider a binary decision with input vector x. Logistic regression first computes a score and then converts it into a probability. The score is additive in the declared features. Every term is visible, and the threshold for action can be written explicitly.

In two raw features, the equality defines a line. In three it defines a plane; in more dimensions, a hyperplane. Points on one side receive a higher score than points on the other. Regularisation controls how far the coefficients can move in response to the sample. Ridge regularisation shrinks all coefficients smoothly. Lasso regularisation can set some exactly to zero, the defining property introduced by Tibshirani's original formulation.

The value of this restriction is diagnostic. If the model misses because all positive cases sit above a curved trend, a monotone transform or spline is a plausible next intervention. If errors collect only when two conditions co-occur, an interaction is implicated. If mistakes form many alternating local pockets, a tree, kernel or learned representation has a stronger case. The boundary gives the residuals a shape that a modeller can inspect.

The same restriction turns the baseline into a controlled ablation laboratory. Add one feature family at a time and measure the paired change: transaction history, behavioural summaries, external attributes, then interactions. Remove a family and observe which cases move, not merely how the average metric changes. If a new source improves only the development period, it may encode a temporary process. If it improves ranking but destabilises calibration, it may alter scale rather than information. If a derived feature replaces several unstable raw variables without losing decision value, the experiment has discovered a more governable representation. These tests are easier to interpret when the learner itself remains fixed. The linear model becomes a ruler for data and feature design, while the flexible challenger remains available to test the residual structure that the ruler cannot absorb.

Linear in which space?

The notation φ(x) carries most of the intellectual burden. Raw age may become age and age squared. A date may become tenure and season. Two variables may produce a product term. A category may become several indicator columns. Text may become a sparse bag-of-words vector. The resulting classifier is linear in those transformed coordinates even when its boundary in the original world bends or branches.

Hold the learner, sample, optimisation and threshold fixed. Generate four noisy quadrants where the label is positive when x₁ and x₂ share a sign. With only x₁ and x₂, logistic regression has no line that separates opposite quadrants; the executable artefact obtains AUC 0.482. Add exactly one feature, x₁x₂, and the same learner reaches AUC 0.917. Gradient boosting reaches 0.921. The intervention varied representation, not algorithmic complexity.

This example prevents a common category error. The raw linear model's failure does not establish that the problem “needs machine learning”. It establishes that the supplied coordinates do not make the relevant relation additive. Conversely, adding every possible interaction until the feature matrix explodes merely moves complexity into feature construction. The baseline must declare where its complexity lives.

What transparency buys, and what it does not

A coefficient exposes how the fitted score changes when one encoded feature changes while the others are held fixed. That is useful, but it is not a causal effect. Correlated variables can exchange weight. Rescaling changes coefficient magnitude. Regularisation couples estimates. Missingness encodings can proxy for workflow. A coefficient can be stable in aggregate and unstable for a subgroup. The model is inspectable because the computation is short, not because each number arrives with a correct story.

This boundary matters because “interpretable” is often treated as a binary property. A recent user study, Are Logistic Models Really Interpretable?, found that skilled participants could not reliably reproduce even small logistic models from their published parameters. That result does not erase the value of a short computation. It shows that inspectability must be designed for the actual reviewer, with units, feature definitions, score decomposition, uncertainty and counterfactual tests.

flowchart TD
  A["The fourth decimal place"] --> B["A boundary you can inspect"]
  B --> C["Make complexity pay twice"]
  C --> D["A baseline kit for real decisions"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#D95545,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Make complexity pay twice

A flexible model should first pay in prediction and then pay the costs it creates. The first payment is a stable reduction in the loss attached to real actions. The second is enough residual value to cover added data dependencies, compute, latency, monitoring, explanation work, recovery effort and uncertainty about the gain. A tiny AUC improvement can be scientifically real and operationally worthless. A modest AUC change can be valuable when it moves the right cases across a constrained decision boundary.

The equation is an accounting discipline rather than a universal currency. Some costs can be monetised per prediction; others are release conditions. A 30 millisecond latency increase may be negligible in overnight underwriting and disqualifying in a payment hold. An explanation requirement may be a hard constraint. A new real-time feature may add more operational risk than the model family. The important move is to keep the benefit and burden in the same decision record.

This view follows an older empirical warning. David Hand argued that simple classifiers often capture most achievable predictive improvement, while the remaining difference can be smaller than uncertainties omitted from standard comparisons. In his surveyed examples, the simple linear classifier frequently captured more than 90 per cent of the improvement over a trivial floor. That is not a law that linear models always win. It is a reason to measure the marginal claim carefully.

Evidence remains domain dependent. A systematic review of 71 clinical prediction studies found no performance advantage for machine-learning methods over logistic regression among lower-risk-of-bias comparisons, while calibration was omitted in 79 per cent of the included studies. A recent workshop study across 44 binary tabular datasets with no more than 500 observations found L2-regularised logistic regression within three percentage points of the best AutoML or deep model on 55 per cent of datasets. Both results support a baseline discipline within their scopes; neither establishes a universal ranking.

The counterevidence is equally important. Across a controlled benchmark of 45 medium-sized tabular datasets, tree-based models remained stronger than deep neural networks and offered favourable speed. Interpretable additive models with selected pairwise interactions have also matched or exceeded more opaque methods on specific healthcare tasks. These findings imply a ladder rather than a loyalty test. The baseline's purpose is to reveal the shape and price of the remaining error, not to sanctify linearity.

The fair comparison contract

Algorithm names are the least important part of a valid comparison. The contract must freeze six things before results are inspected.

1. Prediction moment: each feature must exist, be permitted and have the same meaning at the instant the model would score.
2. Population and split: entities, groups and time must be separated in the way deployment will encounter them.
3. Representation budget: allowable transformations and feature sources must be stated for every model.
4. Selection budget: tuning trials, early stopping, ensembling and manual retries must be counted, not hidden behind the final fit.
5. Calibration and action: probabilities must be calibrated without touching the test set, then converted into the same threshold, capacity or utility rule.
6. Operating boundary: latency, feature freshness, failure behaviour, explanation and rollback requirements must be evaluated with the score.

A fair comparison also matches the unit of uncertainty to the deployment claim. If the system will score new customers, resample customers rather than individual rows from the same customer. If the concern is performance next quarter, preserve time and compare later windows rather than mixing history at random. If several sites, products or countries matter, report their paired differences before averaging them into one number. Repeated seeds reveal optimisation variance, but they do not replace new cases or future periods. Likewise, a large pooled test set can produce a narrow interval around an average while hiding a reversal in a small, consequential segment. The baseline earns its diagnostic role only when the uncertainty calculation preserves the dependencies that the operating environment will preserve. This is one reason a reusable split manifest is as important as a reusable model configuration: it records exactly which future claim the experiment was allowed to make.

Materiality must also be fixed before the contest. A conventional significance test asks whether the observed difference is compatible with zero under a statistical model. An architecture decision asks whether the conservative benefit is large enough to justify change. Those are different questions. Define a minimum worthwhile reduction in decision loss, perhaps 40 units per 1,000 cases, before opening the test set. Then estimate a paired interval for the challenger minus the baseline. If the entire interval is better than the margin, complexity has earned promotion. If it is worse, retain the baseline. If the interval crosses the margin, the correct result is unresolved, even when one point estimate is numerically higher. This framing also supports non-inferiority decisions: a simpler model may be preferred when its worst plausible loss is within an accepted tolerance and it materially reduces latency, dependency or review burden. Uncertainty should govern the change decision, not decorate the metric table.

The selection budget is easy to miss. Cawley and Talbot showed that overfitting a model-selection criterion can be of the same order as the reported differences between learning algorithms. Varma and Simon demonstrated the mechanism on null data: selecting hyperparameters using cross-validation and then reporting the same cross-validation estimate produced optimistic error; repeating every selection step inside an outer loop substantially reduced the bias. The relevant object of evaluation is the whole procedure that produced the model.

Shuffle the labels before the entire modelling procedure. No real association remains. In the artefact, raw logistic regression returns AUC 0.510 and gradient boosting 0.515 on untouched test data. If a search still reports a striking winner, the pipeline has found leakage, reused evaluation data, exploited grouping, or selected noise. A negative control tests the evaluator before the evaluator certifies a model.

Rank, probability and action are different measurements

AUROC asks how often a randomly selected positive case receives a higher score than a randomly selected negative case. It does not say whether a predicted 0.7 occurs about 70 per cent of the time. It does not encode the cost of a missed case. It ignores a queue that can handle only 22 per cent of arrivals. It can remain unchanged under any strictly increasing transformation of the score, even though probabilities and thresholds become unusable.

Take one ranking of 1,000 cases and give it to two systems. System A reports calibrated probabilities from 0.05 to 0.75. System B cubes every probability. The order is identical, so AUROC is identical. At a policy threshold of 0.20, however, the systems send different populations to review. If capacity selects exactly the top 220, ranking governs the queue but calibration still governs expected benefit and abstention. Vary only the action rule and the preferred model can change.

Calibration must therefore be evaluated alongside discrimination. Guo and colleagues showed that modern neural networks in their image and document experiments were often poorly calibrated and that temperature scaling was an effective post-processing method on most datasets studied. The transferable principle is narrower: calibrate each contender on held-out data, test calibration under the deployment population, and do not infer probability quality from ranking quality.

Complexity outside the model object

A model family can be compact while its system is expensive. Sculley and colleagues described machine-learning technical debt arising from entanglement, hidden feedback loops, undeclared consumers, data dependencies and boundary erosion. They warned against “epsilon features”, small accuracy gains whose complexity overhead may be high. The relevant baseline comparison therefore includes the feature pipeline, serving path, observability, fallback and release process.

Imagine a challenger that lowers loss by 0.4 per cent but requires one low-latency vendor feature, a new online join and a specialised explainer. Its statistical gain may be reproducible while its availability gap worsens the real outcome. Conversely, a tree model that uses the same offline table and reduces missed harm materially may add little operational burden. Count dependencies and failure modes, not parameters alone.

A baseline kit for real decisions

A good kit produces evidence in an order that prevents the challenger from defining the contest after seeing the result. It starts with the decision, not a model catalogue. The artefacts are small: a prediction contract, a feature manifest, a frozen split manifest, a baseline ladder, a paired evaluation, a cost ledger and a release decision.

1. Write the prediction contract

Name the unit, outcome, horizon and action. “Predict arrears” is incomplete. “At the close of each monthly cycle, rank active accounts by the probability of an adverse event within 60 days so a team can review at most 22 per cent” is testable. Record exclusions, label latency, intervention effects, review capacity and the owner of residual harm. This boundary determines which data exist and which metric matters.

2. Establish two floors

The first floor ignores features: base rate, majority class, random ranking or the current operational policy. It answers whether the feature set adds useful information. The second is the regularised linear baseline. It answers whether global additive structure captures that information. A feature-rich linear model that barely beats the constant floor suggests weak labels, low signal, shift or a misspecified horizon before it suggests a need for a deeper network.

3. Build features inside the split

Imputation, scaling, vocabulary construction, selection and target-independent statistics must be fitted only on the development portion and replayed on later data. Target-derived encodings require stricter nesting. Kaufman and colleagues describe leakage as information about the target that would not legitimately be available for prediction, and advocate explicit separation between learning and prediction. A pipeline object helps enforce this boundary; the official scikit-learn pipeline documentation shows how transformations and estimators can be chained so the training boundary is repeatable.

4. Make residuals propose the next model

Do not jump from an unsatisfactory score to the fashionable challenger. Slice residuals by feature, time, subgroup and operational state. Plot binned residuals against each continuous variable. Test a small set of predeclared interactions. Inspect whether later periods or unavailable values dominate errors. The next model should answer the residual pattern: splines for smooth curvature, trees for local partitions, kernels for a similarity geometry, sequence models for order, and representation learning for raw high-dimensional signals.

5. Test absence, stability and fallback

Models rarely encounter the clean feature matrix used in a notebook. Repeat the comparison when a costly source is absent, delayed or stale. Replace a live value with the last authorised value and record whether the model should score, abstain or fall back. Measure coefficient and prediction stability across bootstrap samples, later months and plausible encoding choices. A coefficient that flips sign under small resamples is not a dependable explanation, even when aggregate performance is stable. A complex challenger that wins only while one fragile feature is present has a narrower operating boundary than its headline score suggests.

Fallback deserves its own paired test. If the primary model cannot score, does the system use the raw linear model, a constant risk, the current manual rule or no automated recommendation? Compare the complete policy, including fallback frequency and loss, rather than assuming perfect availability. This can reverse the decision. A challenger with slightly better performance during healthy operation may produce worse expected value when its feature service fails more often. The baseline kit should therefore emit an operating matrix: normal state, missing source, stale source, out-of-range input, subgroup shift and calibration drift. For each state, record the permitted model, confidence treatment, action limit and evidence required to restore the primary path. The result is a bounded model service rather than a score with an implicit promise of continuous validity.

Production-shaped worked example

The companion kit generates 8,400 monthly cases across twelve periods. Eight facts are available at scoring time: affordability, utilisation, missed payments, tenure, a thin-file indicator, a recent shock indicator, channel and income volatility. The outcome process also contains three declared mechanisms: high utilisation combined with low affordability, missed payments following a shock, and a short-tenure curve. Later months have higher event prevalence, which makes random splitting inappropriate.

Months zero to six train the models. Month seven selects regularisation or tree hyperparameters. Month eight fits a one-dimensional calibration model. Months nine to eleven remain untouched until the end. Every contender receives the same raw facts. The engineered linear model alone receives the three declared terms because its purpose is to test whether a small amount of explicit structure explains the challenger. The comparison includes a constant-risk floor, raw logistic regression, engineered logistic regression and histogram gradient boosting.

The action policy reviews at most 220 cases per 1,000. Review costs one synthetic unit. Missing an adverse case costs eight. The threshold first requires expected benefit, then the capacity cap selects the highest probabilities. These units are illustrative rather than estimated banking economics. Their purpose is to make the decision rule executable.

Synthetic test set: 2,100 cases from months nine to eleven; event prevalence 40.3%. Lower Brier score and lower cost are better. The fixed capacity makes all ranked models review 220 cases per 1,000.

The result does not prove that logistic regression is generally superior. The data generator contains mechanisms that the three declared terms approximate well. That is the point of the experiment: when explicit structure captures the useful nonlinearity, a flexible model has little remaining dividend. In another environment, the residuals may remain local and irregular. The same gate can then approve the challenger.

The paired bootstrap matters because both models score the same cases. Resampling cases in pairs estimates the uncertainty of their cost difference rather than comparing two independent summary intervals. Here the mean extra cost of boosting is 61.6 units per 1,000, and the 95 per cent interval remains above zero. The release decision is therefore stronger than saying one point estimate is smaller.

The baseline adoption protocol

Executable baseline kit

The companion Python programme implements all four experiments using synthetic data. It prints the geometry checks, the time-split comparison, a paired bootstrap interval and the shuffled-label negative control. Assertions fail if the interaction does not repair XOR, the flexible model does not solve the checkerboard, the engineered linear model loses the worked scenario, or the null models leave the chance band.

Run the embedded programme in this page. It requires Python, NumPy, pandas and scikit-learn. The exact programme is also embedded below so the release remains inspectable when the companion file is unavailable.

#!/usr/bin/env python3
"""Linear baseline kit for Paper 33.

Purpose
-------
Test whether a flexible classifier earns its extra complexity after holding the
feature availability, temporal split, tuning budget, calibration data, action
capacity, and decision costs constant.

The programme uses synthetic data only. It runs four checks:
1. XOR: a raw linear model fails, while one declared interaction repairs it.
2. Checkerboard: a genuinely local nonlinear surface defeats that interaction.
3. A time-split arrears-review scenario: models are compared by calibrated
   ranking and cost under a fixed review capacity.
4. Shuffled labels: both models should return to chance.

Dependencies
------------
Python 3.11+, numpy, pandas, scikit-learn.

Run
---
    python linear_baseline_kit.py

Expected pattern, allowing small library-version differences
--------------------------------------------------------------
- XOR raw linear AUC is near 0.50; interaction and boosting exceed 0.90.
- Checkerboard boosting AUC exceeds 0.85; the two linear versions stay near 0.50.
- In the arrears scenario, the engineered linear model has lower synthetic
  decision cost than the tuned gradient-boosting challenger.
- Shuffled-label AUCs remain between 0.43 and 0.57.
"""

from __future__ import annotations

from dataclasses import dataclass
from typing import Callable, Iterable

import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    average_precision_score,
    brier_score_loss,
    roc_auc_score,
)
from sklearn.model_selection import ParameterGrid
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

SEED = 33


def sigmoid(z: np.ndarray) -> np.ndarray:
    """Numerically simple logistic link for the bounded synthetic generator."""
    return 1.0 / (1.0 + np.exp(-z))


def split_random(
    X: np.ndarray, y: np.ndarray, train_size: int
) -> tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
    rng = np.random.default_rng(SEED)
    order = rng.permutation(len(y))
    train_idx, test_idx = order[:train_size], order[train_size:]
    return X[train_idx], X[test_idx], y[train_idx], y[test_idx]


def make_xor(n: int = 4_000, noise: float = 0.08, seed: int = SEED):
    """Four quadrants; the sign of x1*x2 determines the label."""
    rng = np.random.default_rng(seed)
    X = rng.normal(size=(n, 2))
    y = (X[:, 0] * X[:, 1] > 0).astype(int)
    flip = rng.random(n) < noise
    return X, np.where(flip, 1 - y, y)


def make_checkerboard(
    n: int = 5_000, noise: float = 0.08, seed: int = SEED + 3
):
    """Alternating local regions that one global interaction cannot represent."""
    rng = np.random.default_rng(seed)
    X = rng.uniform(-1.0, 1.0, size=(n, 2))
    y = (
        np.sin(3.0 * np.pi * X[:, 0]) * np.sin(3.0 * np.pi * X[:, 1]) > 0
    ).astype(int)
    flip = rng.random(n) < noise
    return X, np.where(flip, 1 - y, y)


def linear_model(C: float = 1.0) -> Pipeline:
    return Pipeline(
        [
            ("scale", StandardScaler()),
            ("model", LogisticRegression(C=C, max_iter=2_000)),
        ]
    )


def add_single_interaction(X: np.ndarray) -> np.ndarray:
    return np.column_stack([X, X[:, 0] * X[:, 1]])


def geometry_experiments() -> pd.DataFrame:
    rows: list[dict[str, float | str]] = []
    for dataset, factory in (("XOR", make_xor), ("checkerboard", make_checkerboard)):
        X, y = factory()
        X_train, X_test, y_train, y_test = split_random(X, y, int(0.7 * len(y)))

        raw = linear_model().fit(X_train, y_train)
        interaction = linear_model().fit(add_single_interaction(X_train), y_train)
        boosting = HistGradientBoostingClassifier(
            max_iter=300,
            max_leaf_nodes=31,
            learning_rate=0.08,
            l2_regularization=1.0,
            random_state=SEED,
        ).fit(X_train, y_train)

        candidates = (
            ("raw linear", raw, X_test),
            ("linear + x1*x2", interaction, add_single_interaction(X_test)),
            ("gradient boosting", boosting, X_test),
        )
        for name, model, features in candidates:
            probability = model.predict_proba(features)[:, 1]
            rows.append(
                {
                    "dataset": dataset,
                    "model": name,
                    "accuracy": accuracy_score(y_test, probability >= 0.5),
                    "roc_auc": roc_auc_score(y_test, probability),
                }
            )
    result = pd.DataFrame(rows)

    xor = result[result.dataset == "XOR"].set_index("model")
    checker = result[result.dataset == "checkerboard"].set_index("model")
    assert xor.loc["linear + x1*x2", "accuracy"] > xor.loc["raw linear", "accuracy"] + 0.35
    assert checker.loc["gradient boosting", "roc_auc"] > 0.85
    assert checker.loc["linear + x1*x2", "roc_auc"] < 0.60
    return result


BASE_FEATURES = [
    "affordability",
    "utilisation",
    "missed",
    "tenure",
    "thin_file",
    "recent_shock",
    "digital_channel",
    "income_volatility",
]


def make_arrears_cases(
    months: int = 12, cases_per_month: int = 700, seed: int = SEED
) -> pd.DataFrame:
    """Synthetic cases with drift and three declared nonlinear mechanisms."""
    rng = np.random.default_rng(seed)
    month = np.repeat(np.arange(months), cases_per_month)
    n = len(month)

    affordability = rng.normal(0.45 - 0.035 * month, 0.9, n)
    utilisation = np.clip(rng.beta(2.2, 2.7, n) + 0.018 * month, 0.0, 1.0)
    missed = np.clip(rng.poisson(0.45 + 0.045 * month, n), 0, 4)
    tenure = np.clip(rng.gamma(2.5, 2.0, n), 0.0, 15.0)
    thin_file = rng.binomial(1, 0.18, n)
    recent_shock = rng.binomial(
        1, 0.07 + 0.012 * (month >= 8) + 0.015 * (month >= 10), n
    )
    digital_channel = rng.binomial(1, 0.62, n)
    income_volatility = rng.normal(0.0, 1.0, n)

    high_util_low_margin = ((utilisation > 0.76) & (affordability < 0)).astype(float)
    missed_after_shock = ((missed >= 2) & (recent_shock == 1)).astype(float)
    short_tenure = np.maximum(0.0, 2.0 - tenure) / 2.0

    log_odds = (
        -2.45
        - 0.85 * affordability
        + 1.65 * utilisation
        + 0.50 * missed
        + 0.38 * thin_file
        + 0.72 * recent_shock
        + 0.18 * income_volatility
        + 0.12 * (1 - digital_channel)
        + 1.75 * high_util_low_margin
        + 1.25 * missed_after_shock
        + 0.45 * short_tenure
        + 0.15 * (month >= 9)
    )
    outcome = rng.binomial(1, sigmoid(log_odds))

    return pd.DataFrame(
        {
            "month": month,
            "affordability": affordability,
            "utilisation": utilisation,
            "missed": missed,
            "tenure": tenure,
            "thin_file": thin_file,
            "recent_shock": recent_shock,
            "digital_channel": digital_channel,
            "income_volatility": income_volatility,
            "outcome": outcome,
        }
    )


def engineered_features(frame: pd.DataFrame) -> pd.DataFrame:
    X = frame[BASE_FEATURES].copy()
    X["high_util_low_margin"] = (
        (frame.utilisation > 0.76) & (frame.affordability < 0)
    ).astype(float)
    X["missed_x_shock"] = frame.missed * frame.recent_shock
    X["short_tenure"] = np.maximum(0.0, 2.0 - frame.tenure) / 2.0
    return X


@dataclass(frozen=True)
class CalibratedModel:
    predictor: object
    calibrator: LogisticRegression

    def predict_proba(self, X: pd.DataFrame) -> np.ndarray:
        raw = np.clip(self.predictor.predict_proba(X)[:, 1], 1e-6, 1 - 1e-6)
        score = np.log(raw / (1 - raw))
        return self.calibrator.predict_proba(score[:, None])[:, 1]


def tune(
    factory: Callable[..., object],
    grid: dict[str, Iterable[object]],
    X_train: pd.DataFrame,
    y_train: np.ndarray,
    X_validation: pd.DataFrame,
    y_validation: np.ndarray,
) -> dict[str, object]:
    best_auc, best_parameters = -np.inf, None
    for parameters in ParameterGrid(grid):
        model = factory(**parameters).fit(X_train, y_train)
        auc = roc_auc_score(y_validation, model.predict_proba(X_validation)[:, 1])
        if auc > best_auc:
            best_auc, best_parameters = auc, parameters
    if best_parameters is None:
        raise RuntimeError("No parameter combination was evaluated")
    return best_parameters


def fit_with_calibration(
    model: object,
    X_train: pd.DataFrame,
    y_train: np.ndarray,
    X_calibration: pd.DataFrame,
    y_calibration: np.ndarray,
) -> CalibratedModel:
    predictor = model.fit(X_train, y_train)
    probability = np.clip(
        predictor.predict_proba(X_calibration)[:, 1], 1e-6, 1 - 1e-6
    )
    score = np.log(probability / (1 - probability))
    calibrator = LogisticRegression(C=1e6, max_iter=1_000).fit(
        score[:, None], y_calibration
    )
    return CalibratedModel(predictor, calibrator)


def decision_cost(
    y: np.ndarray,
    probability: np.ndarray,
    review_cost: float = 1.0,
    missed_case_cost: float = 8.0,
    review_capacity: float = 0.22,
) -> tuple[float, float]:
    """Cost per 1,000 cases under a fixed-capacity review queue."""
    n = len(y)
    economic_threshold = review_cost / missed_case_cost
    eligible = np.flatnonzero(probability >= economic_threshold)
    capacity = int(review_capacity * n)
    reviewed = (
        eligible[np.argsort(probability[eligible])[::-1][:capacity]]
        if len(eligible) > capacity
        else eligible
    )
    action = np.zeros(n, dtype=bool)
    action[reviewed] = True
    total = review_cost * action.sum() + missed_case_cost * ((y == 1) & ~action).sum()
    return 1_000 * total / n, 1_000 * action.mean()


def production_experiment() -> tuple[pd.DataFrame, tuple[float, float, float]]:
    data = make_arrears_cases()
    train = data.month <= 6
    validation = data.month == 7
    calibration = data.month == 8
    test = data.month >= 9
    train_plus_validation = data.month <= 7

    feature_sets = {
        "raw linear": data[BASE_FEATURES],
        "linear + declared interactions": engineered_features(data),
        "gradient boosting": data[BASE_FEATURES],
    }

    parameters: dict[str, dict[str, object]] = {}
    for name, X in feature_sets.items():
        if name == "gradient boosting":
            factory = lambda **p: HistGradientBoostingClassifier(random_state=SEED, **p)
            grid = {
                "max_iter": [160, 260],
                "max_leaf_nodes": [7, 15, 31],
                "learning_rate": [0.035, 0.07],
                "min_samples_leaf": [25, 55],
                "l2_regularization": [1.0, 3.0],
            }
        else:
            factory = lambda **p: linear_model(**p)
            grid = {"C": [0.03, 0.1, 0.3, 1.0, 3.0, 10.0]}

        parameters[name] = tune(
            factory,
            grid,
            X.loc[train],
            data.loc[train, "outcome"].to_numpy(),
            X.loc[validation],
            data.loc[validation, "outcome"].to_numpy(),
        )

    y_test = data.loc[test, "outcome"].to_numpy()
    probabilities: dict[str, np.ndarray] = {}
    rows: list[dict[str, float | str]] = []

    base_rate = data.loc[calibration, "outcome"].mean()
    constant_probability = np.full(len(y_test), base_rate)
    constant_cost, constant_reviews = decision_cost(y_test, constant_probability)
    rows.append(
        {
            "model": "constant-risk floor",
            "roc_auc": 0.5,
            "average_precision": y_test.mean(),
            "brier": brier_score_loss(y_test, constant_probability),
            "cost_per_1000": constant_cost,
            "reviews_per_1000": constant_reviews,
        }
    )

    for name, X in feature_sets.items():
        if name == "gradient boosting":
            model = HistGradientBoostingClassifier(
                random_state=SEED, **parameters[name]
            )
        else:
            model = linear_model(C=float(parameters[name]["C"]))

        calibrated = fit_with_calibration(
            model,
            X.loc[train_plus_validation],
            data.loc[train_plus_validation, "outcome"].to_numpy(),
            X.loc[calibration],
            data.loc[calibration, "outcome"].to_numpy(),
        )
        probability = calibrated.predict_proba(X.loc[test])
        probabilities[name] = probability
        cost, reviews = decision_cost(y_test, probability)
        rows.append(
            {
                "model": name,
                "roc_auc": roc_auc_score(y_test, probability),
                "average_precision": average_precision_score(y_test, probability),
                "brier": brier_score_loss(y_test, probability),
                "cost_per_1000": cost,
                "reviews_per_1000": reviews,
            }
        )

    rng = np.random.default_rng(SEED + 8)
    paired_deltas = []
    for _ in range(2_000):
        sample = rng.integers(0, len(y_test), len(y_test))
        boost_cost = decision_cost(
            y_test[sample], probabilities["gradient boosting"][sample]
        )[0]
        linear_cost = decision_cost(
            y_test[sample], probabilities["linear + declared interactions"][sample]
        )[0]
        paired_deltas.append(boost_cost - linear_cost)

    delta = (
        float(np.mean(paired_deltas)),
        float(np.quantile(paired_deltas, 0.025)),
        float(np.quantile(paired_deltas, 0.975)),
    )
    result = pd.DataFrame(rows)

    engineered_cost = result.loc[
        result.model == "linear + declared interactions", "cost_per_1000"
    ].item()
    boosting_cost = result.loc[
        result.model == "gradient boosting", "cost_per_1000"
    ].item()
    assert engineered_cost < boosting_cost
    assert delta[1] > 0.0
    return result, delta


def shuffled_label_control() -> pd.DataFrame:
    X, y = make_xor(seed=132)
    rng = np.random.default_rng(133)
    y = rng.permutation(y)
    X_train, X_test, y_train, y_test = split_random(X, y, 2_800)

    linear = linear_model().fit(X_train, y_train)
    boosting = HistGradientBoostingClassifier(
        max_iter=160, max_leaf_nodes=15, random_state=SEED
    ).fit(X_train, y_train)

    rows = []
    for name, model in (("raw linear", linear), ("gradient boosting", boosting)):
        auc = roc_auc_score(y_test, model.predict_proba(X_test)[:, 1])
        rows.append({"model": name, "roc_auc": auc})
        assert 0.43 <= auc <= 0.57
    return pd.DataFrame(rows)


def print_table(title: str, table: pd.DataFrame) -> None:
    print(f"\n{title}")
    print(table.to_string(index=False, float_format=lambda value: f"{value:.4f}"))


def main() -> None:
    print_table("1. Geometry checks", geometry_experiments())

    production, delta = production_experiment()
    print_table("2. Time-split arrears-review comparison", production)
    mean, lower, upper = delta
    print(
        "\nPaired bootstrap, boosting cost minus engineered-linear cost "
        f"per 1,000 cases: mean={mean:.2f}, 95% interval=[{lower:.2f}, {upper:.2f}]"
    )

    print_table("3. Shuffled-label negative control", shuffled_label_control())
    print("\nAll assertions passed.")


if __name__ == "__main__":
    main()

XOR: raw linear AUC 0.482; linear + x1*x2 0.917; boosting 0.921
Checkerboard: raw linear 0.494; interaction 0.524; boosting 0.921
Arrears cost / 1,000: engineered linear 2,041.0; boosting 2,098.1
Paired delta, boosting minus engineered linear: 61.63 [19.05, 106.67]
Shuffled labels: raw linear 0.510; boosting 0.515
All assertions passed.

Where the discipline stops helping

The baseline is an instrument, and instruments have operating ranges. A global linear score is a weak sole reference when the useful structure is inherently local, compositional, sequential, spatial or relational. Raw pixels, waveforms, source code and graphs usually require a representation before an additive boundary becomes meaningful. In those settings, compare the challenger with a representation-matched simple model, such as a linear probe over frozen public features, a bag-of-words classifier for text, or a shallow temporal model over declared summaries.

The artefact's checkerboard alternates the label across many small regions. Raw logistic regression obtains AUC 0.494. Adding the single XOR interaction reaches only 0.524. Gradient boosting reaches 0.921. The flexible model is not exploiting a cosmetic benchmark advantage; it is representing local partitions that the small feature map cannot express. When the residual surface remains structured after credible feature interventions, complexity has earned a mechanistic reason to exist.

Even then, the claim is bounded. A tree's win on one held-out sample can disappear under population shift. A neural representation can exploit acquisition artefacts. A high-dimensional linear baseline can memorise sparse identifiers. Regularisation reduces variance but does not make unavailable features lawful. A transparent model can reproduce discriminatory labels. Neither simplicity nor accuracy substitutes for label validity, causal analysis, subgroup evaluation or human decision rights.

Interpretability also has a ceiling. Correlated predictors create many near-equivalent linear models with different coefficients. A model can be globally short and locally surprising. Generalised additive models provide an important intermediate option by replacing each linear term with a visible shape, and selected pairwise interactions can retain much of that legibility. The healthcare work of Caruana and colleagues illustrates both the promise and caution: an additive model exposed a counterintuitive asthma pattern that reflected treatment effects in the observed data, making the problematic term visible for investigation and repair.

The final failure mode is organisational. Teams sometimes use the baseline as a veto that protects a familiar model from disconfirming evidence. That reverses its purpose. If a challenger wins under frozen availability, nested selection, later-period testing, calibration, action loss, subgroup review and operational stress, rejecting it because it is less familiar is not discipline. It is inertia.

stateDiagram-v2
  [*] --> Claim
  state "1. Raw space" as Claim
  state "2. Add the declared<br/>interaction x₁x₂" as Test
  state "3. Local checkerboard surface" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Compact glossary

Baseline

A declared lower-complexity procedure used to attribute the marginal value of a challenger, not merely the first model trained.

Feature map

The transformation φ(x) from world measurements into model inputs, including encodings, interactions and lawful basis functions.

Calibration

Agreement between predicted probabilities and observed frequencies within the population and period where decisions are made.

Decision loss

The weighted consequence of actions and errors under a specified threshold, capacity, cost and recovery policy.

Paired comparison

An estimate of model difference that preserves the fact that both contenders score the same cases or time windows.

Complexity dividend

The conservative decision benefit of a challenger after incremental operating and change costs are subtracted.

Promote only the earned complexity

A linear model does not deserve priority because straight lines are pure, coefficients are causal or simple systems never fail. It deserves a place because its restrictions are legible enough to support a discriminating experiment. It tells the team what the raw features achieve, which residual structures remain, how much feature design closes the gap, and whether a flexible model's advantage survives contact with a real action.

The changed architecture decision is concrete. Every predictive component should ship with a baseline ladder and a complexity-dividend record. The record freezes the prediction moment, representation budget, selection procedure, calibration data, action policy, paired uncertainty and operating burden. The accepted model may be linear, additive, tree-based, kernel-based or neural. The burden of proof remains the same.

Choose the least complicated system that clears the decision threshold, and choose a more complicated one without apology when the evidence clears it by a material, repeatable margin. That is the discipline of a baseline: neither reverence for simplicity nor theatre around sophistication, but a fair test of what each additional degree of freedom actually buys.