When perfect fit tells you almost nothing

Suppose a training set contains ten points and a network predicts every label correctly. That statement tells us how the network behaves at ten locations. It does not tell us what curve it draws between them, how it behaves after a coordinate shift, or which input feature carries its decision. A smooth curve, a jagged curve and a rule that memorises point identifiers can all agree on the observations.

The missing object is the selection rule

Supervised training is often written as a search for parameters that minimise average error. A regularised version adds a second term:

The symbols describe a physical training situation. The dataset contains n input and target pairs. The network with parameters θ produces the function f_θ. The loss ℓ measures disagreement on each example. The penalty Ω assigns cost to some parameter configurations, while λ sets how much that cost matters. The equation does not say that small Ω means truth. It says that among solutions with similar observed error, the procedure will prefer some over others.

That preference can enter even when λ is zero. The optimiser, parameterisation, initialisation, stopping time and minibatch noise can favour particular solutions. In a restricted but revealing case, gradient descent on separable logistic regression converges in direction towards a maximum-margin classifier even without an explicit penalty.[7] This is called implicit bias. It means that “unregularised” training still has a selection rule.

Three uncertainties hide inside one accuracy number

The first uncertainty is representational. Distinct parameter settings can implement the same function, and distinct functions can agree on every sampled point. A wide network may contain many redundant routes. A scale increase in one layer can be cancelled by a scale decrease in the next. Counting parameter vectors therefore exaggerates some differences and misses others.

The second uncertainty is optimisation. A function may be representable but practically unreachable under the chosen initialisation, learning rate and update budget. Another may be reached quickly because its gradients align across examples. Training time is therefore part of the bias. Early stopping does not simply stop overfitting after it starts. It prefers functions that the optimiser expresses early.

The third uncertainty is distributional. Generalisation is never a property of a function without a family of future cases. A rule may transfer across random samples from the same process and fail under a location shift, sensor replacement or changed correlation. Saying that a model “generalises” without naming the variation it survives hides the most important design claim.

It is useful to think of the complete learning procedure as assigning an informal prior mass over functions. Some functions occupy large, easily reached regions of parameter space. Some have short descriptions in the architecture. Some appear early along the optimisation path. Some survive noise or parameter shrinkage. The data update this preference, but finite data rarely erase it. The learned function is evidence plus selection rule, not evidence alone.

Modern deep networks make the underdetermination larger, not smaller. Networks with enough capacity can fit random labels, which shows that successful optimisation and low training error do not by themselves explain generalisation.[8] The classical instruction to reduce capacity also has important exceptions. Test error can improve again after a model passes the interpolation threshold, a pattern described as double descent.[9] These results do not imply that capacity is irrelevant. They imply that raw parameter count is too coarse to stand in for the actual selection rule.

A parameter count also ignores how parameters are tied. A convolution with one shared kernel can apply a detector at hundreds of positions, while a dense layer may spend hundreds of unrelated weights learning location-specific copies. The convolution may have fewer free numbers yet a stronger ability to express the relevant family of translated rules. Conversely, a large model can contain a small effective subnetwork selected by optimisation. This is why “smaller”, “simpler” and “more constrained” must not be treated as synonyms.

The practical consequence is that model comparison needs a revealing axis. If two candidates are judged only on the distribution that trained them, the bias remains latent. Move the object, break a feature correlation, alter scale or scramble order. A well-chosen intervention does more than report another score. It identifies which relation the model carried beyond the examples.

flowchart TD
  A["When perfect fit tells you<br/>almost nothing"] --> B["Three ways to bend the search"]
  B --> C["The controlled garden"]
  C --> D["Design constraints that fail<br/>honestly"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#B96D00,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Three ways to bend the search

Regularisation changes what solutions cost

Explicit regularisation modifies the objective or training process so that some solutions become expensive. L2 regularisation penalises squared parameter magnitude. L1 regularisation can favour sparse coefficients. Dropout trains randomly thinned subnetworks and was introduced as a way to reduce brittle co-adaptation.[5] Mixup trains on convex combinations of examples and labels, changing the neighbourhood in which the model is expected to behave smoothly.[15]

The common phrase “reduce overfitting” compresses several mechanisms. A penalty can shrink sensitivity, reduce effective degrees of freedom, favour margin, average subnetworks, or encode a smoothness assumption. Those are different causal claims. The correct question is not whether a method is called a regulariser, but which function property becomes cheaper and under which optimiser that cost is actually implemented.

Even the apparently simple phrase “weight decay” needs care. For ordinary stochastic gradient descent, L2 regularisation and multiplicative decay can be equivalent after appropriate scaling. With adaptive optimisers such as Adam, they are generally not equivalent. AdamW decouples decay from the gradient update so that the shrinkage is not distorted by the adaptive preconditioner.[6]

Regularisers also differ in whether they are hard or soft. A hard sparsity mask removes connections. A norm penalty leaves them available but costly. Dropout changes the stochastic training graph. Data augmentation expands the empirical support with transformed examples. Early stopping limits how long slow-to-learn structure can emerge. These methods can produce similar validation curves through different mechanisms, so replacing one with another is not causally neutral.

The units of the penalty matter. If one feature is measured in metres and another in millimetres, equal coefficient penalties do not express equal physical restraint. The same issue appears inside deep networks when a reparameterisation moves scale across layers. A norm is a property of parameters in chosen coordinates, not a direct measure of functional complexity. Norm-based regularisation can still be useful, but its interpretation belongs to the architecture and parameterisation that give the norm meaning.

There is also a difference between discouraging a solution and supplying evidence against it. Weight decay may favour a smoother or lower-gain rule, but it cannot reveal that a background watermark is non-causal. Only data or an intervention that varies the watermark independently of the label can do that. Regularisation resolves underdetermination by preference; it does not manufacture missing counterfactual evidence.

Normalisation changes coordinates and step dynamics

Consider a hidden activation vector [2, 4, 6]. Its mean is 4. Its population standard deviation is about 1.63. Subtracting the mean and dividing by the standard deviation produces approximately [-1.22, 0, 1.22]. Multiply all three original activations by ten and, ignoring the small numerical stabiliser, the normalised vector is unchanged.

Here h is the pre-normalisation activation, μ and σ are statistics computed over a chosen set of elements, ε prevents numerical division by zero, and γ and β are learned scale and offset parameters. Batch normalisation computes statistics across a minibatch for each feature. Layer normalisation computes them across features within one example and uses the same computation at training and test time.[1][4] Group normalisation divides channels into groups, which avoids dependence on batch size.[3]

Normalisation can make scale directions less troublesome, keep activations in workable ranges and permit larger learning rates. The original batch-normalisation paper framed its effect through “internal covariate shift”.[1] Later work found that reducing such distributional change was not necessary to obtain the benefit and instead emphasised smoother, more predictable gradients or the ability to take larger steps.[2][10] The mechanism remains richer than a single slogan.

The choice of normalisation axes is itself an inductive choice. Batch normalisation couples each example to the other examples used to estimate its feature statistics during training. Layer normalisation instead couples features within one example. Group normalisation chooses intermediate channel groups. These operations can agree on a large image batch and behave differently on a batch of one, a recurrent state, a set of variable size or a signal whose mean is informative.

The learned gain γ and offset β also matter. Normalisation does not permanently force every feature to zero mean and unit variance after training. The affine parameters can restore useful scale and offset, while the normalised parameterisation still changes how gradients move. This is one reason a simple statement such as “normalisation removes magnitude” must be scoped to the statistic, layer and downstream computation.

Train and inference semantics must be checked separately. Batch statistics used during training are usually replaced by accumulated estimates at inference. A small or drifting deployment batch can therefore produce a failure that does not occur in the training graph. Layer normalisation avoids that particular train-test switch, but may impose unwanted invariance across features. The question is not which method is universally safest. It is which statistic is stable, meaningful and available at the point of use.

Normalisation is also not logically indispensable. Carefully designed initialisation and residual scaling can train deep networks without it, as Fixup demonstrated for residual models.[14] That boundary matters. If a normalised model wins, the evidence supports the configured combination of parameterisation, optimiser and training budget. It does not prove that normalisation is the only route to the function.

Architecture changes who may talk to whom

A dense layer gives every output unit its own weight to every input. A one-dimensional convolution uses the same local detector at every position. That weight sharing does more than reduce parameters. It encodes a transformation rule: shift the input and the feature map shifts in a corresponding way. This is translation equivariance. If the network then pools across position, the final output can become approximately translation invariant.

A group-equivariant network extends this idea to specified rotations, reflections or other transformations, using structured weight sharing to reduce sample requirements when the symmetry is valid.[11] A Deep Sets architecture builds permutation invariance into a function over an unordered set.[12] Graph networks encode entities, relations and local message passing, a relational bias intended to support compositional reasoning.[13]

Architectural bias can therefore exclude a vast number of functions while expanding useful reach inside the remaining family. A convolutional detector needs one motif template rather than a separate template for every location. The same restriction is fatal when absolute location determines the label. An architectural symmetry is a claim about the world, written into connectivity.

Depth changes the claim gradually. A small convolution begins with local interactions, but stacked layers enlarge the receptive field. Pooling can turn equivariance into invariance. Skip connections create short optimisation paths that let a deep system behave like an ensemble of shallower transformations. Architecture therefore affects both the function family and the ease with which different members of that family are reached.

Universal-approximation results do not remove this distinction. Two architectures may both approximate a target given enough width, data and optimisation, while differing by orders of magnitude in sample requirement or training time. “Can represent” is an existence statement. Engineering asks whether the representation is reachable with the available evidence and whether the preferred approximation preserves the right relation under shift.

A strong bias is valuable when the symmetry is trustworthy and data are scarce. It is dangerous when exceptions are common or costly. A medical image model may reasonably share a local edge detector across location, yet the absolute side of the body can matter. A graph model may correctly treat node order as arbitrary, yet fail if identifiers encode roles that are not interchangeable. The architecture should preserve the transformation that leaves the target unchanged, not every transformation that makes implementation convenient.

The controlled garden

A useful experiment must make the competing biases disagree. Standard in-distribution accuracy often hides the difference because all models can exploit the same support. The controlled garden below uses a tiny sequence task with one deliberate intervention: the relevant motif appears only near the left during training and only near the right during shifted testing.

The task

Each input is a sequence of 32 values containing exactly sixteen +1 tokens and sixteen −1 tokens. Positive examples contain the ordered motif +1, −1, +1, −1. Negative examples do not. Because every sequence has the same counts, a permutation-invariant set model receives no label information at all. During training, positive motifs start at positions 2 through 7. During the shifted test, they start at positions 22 through 27.

Three models see the same 512 training examples and use the same loss, optimiser, dropout rate, layer-normalisation switch and mild AdamW decay. The dense model has 4,353 parameters and position-specific connections. The convolutional model has 2,241 parameters and shares one four-position detector across the sequence. The set model has 2,369 parameters and averages pointwise features, so it is invariant to every permutation.

The construction removes two common shortcuts. First, token counts cannot identify the class because every example has the same multiset. Second, the shifted test changes only motif location. It does not change motif values, sequence length or class balance. This gives the three architectures a precise point of disagreement: whether a local ordered relation should transfer across position.

The comparison is intentionally not a parameter-matched contest. The convolution is smaller than the dense model, yet carries a more useful rule through sharing. If parameter count alone controlled generalisation, the result would be ambiguous. The intervention instead asks whether the detector itself moves. A stricter follow-up could widen the convolution or shrink the dense network, but neither change would alter the architectural fact that one ties positions and the other does not.

The update budget is fixed at thirty full-batch steps. That choice measures what each configured procedure reaches quickly, not its best possible result after unlimited tuning. This is legitimate because optimisation speed is part of practical inductive bias, but it must be named. A separate convergence study would extend training, tune learning rates and compare equal compute. The present lab chooses causal transparency over exhaustive performance.

The result is not “convolutions are better”. It is narrower and more useful. When the label is a local relation whose position may change, shared local connectivity dramatically reduces the evidence needed to learn the transferable rule. The dense network can represent a detector at every position, but the training support gives it no reason to tie those detectors together. The set model has an even stronger restriction, but it is the wrong one.

Across the three seeds, the dense model averaged about 90.4 per cent training accuracy and 69.7 per cent on a fresh in-support test, yet only 49.7 per cent after the motif moved. The convolution reached 100 per cent on all three splits. The set model remained at 50 per cent everywhere. These numbers separate three failures. The dense model had enough information to learn some left-side regularity but not the shared translated rule. The set model could not represent the ordered relation at all. The convolution both represented and quickly reached the intended detector.

Notice what the result does not establish. It does not show that the dense model could never learn translation from broader data. It does not show that the set model is weak on set-valued tasks. It does not isolate every effect of parameter count. It shows that under a support gap specifically chosen to preserve local order and alter position, the architectural sharing rule supplies the missing bridge.

What normalisation changed in the same model

Within the convolutional architecture, a one-seed ablation held the task, optimiser, learning rate and update count fixed. Without normalisation, shifted accuracy reached 89.8 per cent after 30 full-batch updates. With channel-wise layer normalisation, it reached 100 per cent. The weight-decay switch from 0 to 0.001 made no visible accuracy difference at that budget, and only a tiny change in mean weight norm.

This is a useful negative result. Weight decay did not create the missing translation prior and did not explain the main transfer gap. Normalisation accelerated and stabilised the configured optimisation path, but the unnormalised convolution was already moving towards the same transferable solution. The architecture supplied the relevant rule; normalisation changed how quickly the optimiser exploited it.

The ablation also shows why components should be tested within the same architecture before explanations are compared across architectures. If we compared only the normalised convolution with the unnormalised dense model, the 50-point gap could be credited to either connectivity or normalisation. Holding connectivity fixed reveals a roughly 10-point speed difference at this update budget. Holding normalisation fixed across architectures reveals the much larger symmetry effect.

Three further controls make the inference stronger. Train the dense model with motifs at every location; its shifted gap should shrink if missing positional support caused the failure. Replace the convolution’s global max pool with a head tied to left positions; transfer should deteriorate if pooling supplied the final invariance. Give the set model variable token counts correlated with the label; its score should rise through a count shortcut and fall when equal counts are restored. Each control changes one causal property and predicts a specific movement.

from __future__ import annotations
import argparse, json, random
from dataclasses import dataclass, asdict
from typing import Literal
import numpy as np
import torch
from torch import nn

Architecture = Literal["dense", "conv", "set"]
MOTIF = np.array([1., -1., 1., -1.], dtype=np.float32)
L = 32

def seed_all(seed: int) -> None:
    random.seed(seed); np.random.seed(seed); torch.manual_seed(seed)

def hits(x: np.ndarray) -> list[int]:
    return [i for i in range(L - 3) if np.array_equal(x[i:i+4], MOTIF)]

def balanced(rng: np.random.Generator) -> np.ndarray:
    x = np.array([1.] * 16 + [-1.] * 16, dtype=np.float32)
    rng.shuffle(x)
    return x

def negative(rng: np.random.Generator) -> np.ndarray:
    while True:
        x = balanced(rng)
        if not hits(x):
            return x

def positive(rng: np.random.Generator, allowed: list[int]) -> np.ndarray:
    while True:
        p = int(rng.choice(allowed))
        remainder = np.array([1.] * 14 + [-1.] * 14, dtype=np.float32)
        rng.shuffle(remainder)
        x = np.empty(L, dtype=np.float32)
        x[p:p+4] = MOTIF
        mask = np.ones(L, dtype=bool); mask[p:p+4] = False
        x[mask] = remainder
        positions = hits(x)
        if positions and all(h in allowed for h in positions):
            return x

def dataset(n: int, allowed: list[int], seed: int):
    rng = np.random.default_rng(seed); xs, ys = [], []
    for _ in range(n // 2):
        xs += [positive(rng, allowed), negative(rng)]
        ys += [1., 0.]
    order = rng.permutation(n)
    return (torch.tensor(np.stack(xs)[order]),
            torch.tensor(np.array(ys, dtype=np.float32)[order]))

class ChannelLN(nn.Module):
    def __init__(self, channels: int):
        super().__init__(); self.ln = nn.LayerNorm(channels)
    def forward(self, x):
        return self.ln(x.transpose(1, 2)).transpose(1, 2)

class Dense(nn.Module):
    def __init__(self, norm: bool):
        super().__init__(); layers = [nn.Linear(L, 64)]
        if norm: layers += [nn.LayerNorm(64)]
        layers += [nn.ReLU(), nn.Dropout(.1), nn.Linear(64, 32),
                   nn.ReLU(), nn.Linear(32, 1)]
        self.net = nn.Sequential(*layers)
    def forward(self, x): return self.net(x).squeeze(-1)

class Conv(nn.Module):
    def __init__(self, norm: bool):
        super().__init__(); layers = [nn.Conv1d(1, 96, 4)]
        if norm: layers += [ChannelLN(96)]
        layers += [nn.ReLU(), nn.Dropout(.1),
                   nn.Conv1d(96, 16, 1), nn.ReLU()]
        self.features = nn.Sequential(*layers)
        self.out = nn.Linear(16, 1)
    def forward(self, x):
        h = self.features(x[:, None, :]).amax(-1)
        return self.out(h).squeeze(-1)

class SetModel(nn.Module):
    def __init__(self, norm: bool):
        super().__init__(); layers = [nn.Linear(1, 64)]
        if norm: layers += [nn.LayerNorm(64)]
        layers += [nn.ReLU(), nn.Dropout(.1),
                   nn.Linear(64, 32), nn.ReLU()]
        self.phi = nn.Sequential(*layers)
        self.out = nn.Linear(32, 1)
    def forward(self, x):
        return self.out(self.phi(x[..., None]).mean(1)).squeeze(-1)

def build(name: Architecture, norm: bool) -> nn.Module:
    return {"dense": Dense, "conv": Conv, "set": SetModel}[name](norm)

def accuracy(model, x, y) -> float:
    model.eval()
    with torch.no_grad():
        pred = torch.sigmoid(model(x)) >= .5
        return float((pred == y.bool()).float().mean())

@dataclass
class Result:
    architecture: str; normalise: bool; weight_decay: float; seed: int
    parameters: int; train_accuracy: float
    in_distribution_accuracy: float; shifted_accuracy: float
    mean_weight_norm: float

def run(name, norm, wd, seed, epochs, cache) -> Result:
    seed_all(seed)
    xt, yt = cache[(seed, "train")]
    xi, yi = cache[(seed, "id")]
    xs, ys = cache[(seed, "shift")]
    model = build(name, norm)
    opt = torch.optim.AdamW(model.parameters(), lr=3e-3, weight_decay=wd)
    loss_fn = nn.BCEWithLogitsLoss()
    for _ in range(epochs):
        model.train(); opt.zero_grad(set_to_none=True)
        loss = loss_fn(model(xt), yt); loss.backward(); opt.step()
    norms = [p.detach().norm().item() for p in model.parameters() if p.ndim > 1]
    return Result(name, norm, wd, seed,
        sum(p.numel() for p in model.parameters()),
        accuracy(model, xt, yt), accuracy(model, xi, yi),
        accuracy(model, xs, ys), float(np.mean(norms)))

def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--epochs", type=int, default=30)
    parser.add_argument("--seeds", type=int, default=3)
    parser.add_argument("--output", default="bias_garden_results.json")
    args = parser.parse_args()
    torch.set_num_threads(1); cache = {}
    for seed in range(args.seeds):
        cache[(seed, "train")] = dataset(512, list(range(2, 8)), 1000 + seed)
        cache[(seed, "id")] = dataset(1000, list(range(2, 8)), 2000 + seed)
        cache[(seed, "shift")] = dataset(1000, list(range(22, 28)), 3000 + seed)
    results = []
    for name in ("dense", "conv", "set"):
        for seed in range(args.seeds):
            results.append(run(name, True, 1e-3, seed, args.epochs, cache))
    for norm in (False, True):
        for wd in (0., 1e-3):
            results.append(run("conv", norm, wd, 0, args.epochs, cache))
    with open(args.output, "w", encoding="utf-8") as handle:
        json.dump([asdict(r) for r in results], handle, indent=2)
    for result in results:
        print(asdict(result))

if __name__ == "__main__":
    main()

stateDiagram-v2
  [*] --> Claim
  state "The missing object is the<br/>selection rule" as Claim
  state "Three uncertainties hide<br/>inside one accuracy number" as Test
  state "Regularisation changes what<br/>solutions cost" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Design constraints that fail honestly

The practical mistake is to choose a regularisation recipe before stating the target invariances and deployment shifts. A better order begins with a constraint passport. For every proposed constraint, record four things: what it forbids, what it identifies as equivalent, what training path it favours, and which intervention should make it fail.

The passport is useful because the three constraint families overlap in observed outcomes. A normalised model may have lower validation loss and smaller effective sensitivity. A convolution may use fewer parameters and behave like a regulariser. Dropout changes both capacity during training and the optimisation noise. Rather than classify a method by name, the passport records the causal operation that matters for the decision.

Begin with the target transformation, not the layer. Ask what can change while the correct label stays fixed. Then ask what must not be treated as equivalent. This second question prevents a common asymmetry: teams document desired invariances but rarely document protected distinctions. A handwriting model may ignore small translations but must preserve the distinction between a six and a nine under rotation. A transaction model may ignore record order within a set but must preserve temporal order across authorisation and settlement.

A realistic configured-system example

Consider a vibration classifier fed by eight accelerometers along a conveyor. The engineering team wants to detect a local bearing fault wherever it occurs. A convolution along sensor position expresses useful locality and translation sharing. Per-sensor standardisation may control calibration differences. Mild weight decay may discourage fragile high-gain filters.

Now inspect the passport. The convolution forbids arbitrary long-range interactions in its first layer, so stacked depth or a global branch may be needed for faults defined by coordination between distant motors. Per-sensor standardisation identifies amplitude differences that may be nuisance calibration, but absolute vibration energy may also indicate severity. Weight decay favours smaller filters, but a rare high-frequency signature may need a sharp response. The break tests therefore include moving an injected local pattern across sensors, varying calibration while holding physical amplitude fixed, scaling true event magnitude, and creating a distant two-sensor interaction.

This example shows why a single validation split is insufficient. A model can score well because the test preserves the same sensor locations and calibration regime as training. The passport asks the evaluator to construct the shifts the constraint claims to handle and the shifts it risks erasing.

The team can now choose the strength of the commitment. If local translation is a physical law of the sensor layout, shared convolution is a reasonable hard bias. If sensor roles vary by installation, a softer equivariance penalty or augmentation may be safer. If amplitude sometimes signals severity, normalise calibration factors but retain a separate absolute-energy channel. If rare long-range interactions matter, add a global branch rather than expecting a local stack to discover them cheaply.

This separation supports recovery. When a hard constraint fails, the missing information may not exist anywhere in the representation. When a soft penalty fails, retraining can sometimes relax the preference. When the uncertainty is high, keeping an unrestricted diagnostic branch or logging the pre-normalised signal can preserve evidence for later analysis. Recoverability is therefore part of architectural bias, not an operational afterthought.

The strongest boundary

No finite passport proves that the world will preserve a chosen symmetry. Deployment environments can introduce transformations the team did not anticipate, or make a previously irrelevant variable causal. A convolution that is excellent under translation may fail under rotation. Data augmentation can help cover expected transformations, but it also asserts that labels should remain stable across those generated examples. The validity of the transformation remains a domain claim.

Nor should every preference be made hard. A hard invariant architecture can improve sample efficiency when the symmetry is trustworthy, yet make recovery impossible when exceptions matter. A softer strategy can augment data, add an equivariance penalty, or combine a biased branch with a less restricted branch. The cost is that the model may need more evidence and may relearn shortcuts. Hard constraints buy efficiency with brittleness; soft constraints buy recoverability with evidence cost.

Evaluation-only handling is a third option. When the team cannot justify an invariance, it can keep the model flexible and create dedicated slices that measure the suspected transformation. This does not grant sample efficiency, but it avoids silently erasing information. Evidence can later support promotion from an evaluation hypothesis to a soft penalty or hard structure.

The same discipline applies to normalisation. Log pre-normalisation statistics, test batch-size and distribution changes, and compare training-time with inference-time behaviour. For regularisation, report the optimiser, learning-rate schedule and exact implementation of decay. For architecture, report receptive fields, sharing rules and pooling. These details are not incidental hyperparameters. They define the selection rule whose transfer is being claimed.

Spectral bias supplies another warning. Neural networks can learn lower-frequency functions faster than higher-frequency functions, even when both are representable.[16] That preference often aligns with smooth signal, but not always. A diagnostic fault or cryptographic relation can live in a sharp, high-frequency boundary. “Simple first” is useful only after simple has been defined relative to the task.