flowchart TD
  A["The camera that learned the<br/>timestamp"] --> B["The bottleneck decides what<br/>survives"]
  B --> C["Four prediction games, four<br/>learned worlds"]
  C --> D["The interventional compression<br/>audit"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#5B5CF6,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

The bottleneck decides what survives

Worked example 1: the one-bit predictor. Start with the smallest useful prediction problem. A target token can be either A or B. With no context, both are equally likely. The best honest predictor assigns probability 0.5 to each. When A occurs, its code length under that predictor is −log₂ 0.5 = 1 bit. Now reveal a context in which A occurs three quarters of the time. A calibrated predictor assigns A probability 0.75, and a correct A costs about −log₂ 0.75 = 0.415 bits. The context saves roughly 0.585 bits on that outcome.

This is not a metaphor. A probabilistic model can be turned into a code, and negative log probability is the idealised length of the code assigned to an event. Shannon established the link between probability, entropy and coding; minimum-description-length work later made model plus data description a principle for statistical selection.[1][2] When an autoregressive model minimises next-token log-loss, or a masked model minimises conditional log-loss, it learns probabilities that can compress the target relative to the visible context.

Here C is the visible context, T is the hidden or future target, and q_θ is the model’s predicted distribution. The equation is valid when the score is logarithmic and the training examples are sampled from the distribution being averaged. If deployment changes that distribution, the old codelength can become misleading.

At the optimum, expected log-loss decomposes into the irreducible conditional entropy of the data plus the model’s mismatch from the true conditional distribution. In plain language, some uncertainty remains even for a perfect predictor, and the rest is modelling error. Better prediction means a shorter average description of the hidden target from the visible context.

The left side is the extra target uncertainty caused by replacing the full context with the representation. The right side is the target-relevant information still present in the context but absent from the representation. Zero means predictive sufficiency. The equation does not say that the representation is small, interpretable or causal. Without an explicit capacity limit, noise process, regulariser or architectural constraint, the encoder may preserve much more than the target requires.

Calling two systems “masked models” can therefore conceal different causal contracts. One masks isolated pixels while leaving texture, position and acquisition marks; another removes coherent regions and suppresses metadata. One gives the encoder a narrow code and a weak decoder; another gives the decoder enough capacity to reconstruct from its learned prior. One samples every source equally; another lets a dominant source make device identity predictive. Each pair may share the same named loss, yet each changes the cheapest statistic that reaches the target. The object to evaluate is the full prediction contract: sampling distribution, visible variables, corruption process, target, horizon, encoder capacity, decoder capacity and scoring rule. This contract is the intervention surface. Change one field while holding the others fixed, then ask which representations, transfer results and failure modes move. Only then can an improvement be attributed to the proposed mechanism rather than to an easier route through the task.

The forecast horizon changes the object of learning

Prediction into the future adds another lever: horizon. Consider two sequence generators that behave identically for four steps. One follows an alternating rule after that point; the other enters a persistent state. A one-step predictor can perform well by modelling local transition frequencies. A six-step predictor must distinguish the latent regimes. The longer horizon creates pressure for a state variable that the short horizon never needed.

Four prediction games, four learned worlds

The phrase “self-supervised learning” groups together objectives with different causal demands. They share a method for manufacturing targets from data, but they do not ask the representation to preserve the same relations. A useful comparison therefore starts from the prediction game, not the model brand.

1. Reconstruct the missing observation

Denoising autoencoders corrupt an input and learn to recover the clean version. The corruption prevents a trivial identity map and encourages features stable enough to repair damage. Vincent and colleagues showed that this principle could produce useful intermediate representations and improve classification benchmarks.[5] BERT applies a related idea to text by hiding selected tokens and predicting them from bidirectional context.[4] Masked autoencoders apply it to image patches, using an asymmetric encoder-decoder and a high masking ratio to make vision pretraining efficient and scalable.[10]

The strength of raw reconstruction is also its risk. A pixel loss charges for lighting, sensor noise and exact texture. A token loss charges for lexical identity even when several words express the same proposition. These details may matter for generation, restoration or forensic analysis. They may be wasteful for object recognition or control. Generative fidelity and representational usefulness overlap, but neither contains the other.

2. Predict or identify the future

Autoregressive generative models factor a sequence into conditional predictions of each next element. Contrastive Predictive Coding instead asks a context representation to identify the true future latent among alternatives. Its original experiments reported useful representations across speech, images, text and reinforcement-learning observations.[6] Both approaches reward predictive regularity, but they allocate detail differently. Exact next-observation likelihood must model every predictable distinction in the chosen tokenisation. Contrastive identification needs enough information to separate the positive future from sampled negatives.

Negative sampling quietly defines relevance. If negatives differ mainly by speaker identity, the representation can emphasise identity. If they differ by phonetic content while preserving speaker, it must represent speech units. A future horizon that is too short may reward surface continuity; one that is too long may become dominated by irreducible uncertainty. There is no horizon-free notion of predictive structure.

3. Make two views agree

SimCLR creates two augmented views of an image, encourages their representations to align, and separates other images in the batch. Its ablations showed that data augmentation composition, a nonlinear projection head, normalisation and batch size materially affected performance.[7] Work on view selection argued that useful views should discard some shared nuisance information while retaining information needed by downstream tasks.[8]

This makes augmentation a form of ontology engineering. A colour jitter declares colour non-essential. A crop declares that two partial views should share an identity. A time warp declares a range of timing changes irrelevant. These declarations can be right for one decision and destructive for another. A crop that helps classify objects may erase the spatial relation needed to detect a manufacturing defect. A pitch shift that helps recognise words may destroy a speaker attribute needed for clinical voice analysis.

4. Predict a latent target

Latent-prediction methods replace raw detail with a target encoder’s representation. Data2vec predicts contextualised latent representations from masked inputs across speech, vision and language.[11] I-JEPA predicts representations of image blocks from a context block and reports that target scale and context distribution are crucial design choices.[13] V-JEPA extends masked latent prediction to video.[14] These methods aim to focus prediction on stable structure without spending equal effort on every pixel or waveform fluctuation.

But latent targets do not remove the relevance problem. They relocate it. The target encoder, its update rule, its collapse-prevention mechanism and its training views decide what the student must match. If every input maps to the same latent, prediction becomes trivial. Methods such as BYOL, Barlow Twins and VICReg use architectural asymmetry, variance or redundancy controls to avoid that collapse.[12][15][16]

stateDiagram-v2
  [*] --> Claim
  state "The forecast horizon changes<br/>the object of learning" as Claim
  state "1. Reconstruct the missing<br/>observation" as Test
  state "2. Predict or identify the future" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

The interventional compression audit

A low pretext loss answers one narrow question: can this configured system predict its constructed target on this distribution? It does not tell us which cue carried the prediction, where that cue is represented, whether a cheap decoder supplied the answer, or whether the same code survives a change that preserves the intended structure.

The audit proposed here treats self-supervised pretraining as a causal system. It records five objects: context C, target T, representation Z, suspected nuisance or shortcut S, and downstream decision Y. It then intervenes on S while keeping the intended data-generating structure as stable as possible. The central metric is shortcut debt: the extra predictive codelength paid after a nuisance intervention.

Step 1: write the prediction contract

State exactly what is visible, what is hidden, how examples are paired and where loss is applied. “Masked modelling” is too vague. A useful contract says, for example: contiguous 400-millisecond sensor spans are hidden; the encoder sees the remaining channels plus machine state; the target is a quantised latent from a slowly updated teacher; loss is applied only inside hidden spans; the deployed representation is the encoder output before the predictor.

Then list the properties the downstream decision must retain. A rail-monitoring system may need bearing periodicity, load-dependent harmonics and temporal persistence. It may need invariance to depot identifier, firmware version and sensor gain within a calibrated range. These are engineering hypotheses, not facts revealed by the word “self-supervised”.

Step 2: identify a rival compressor

Build the cheapest alternative mechanism that could achieve the observed loss without the intended structure. For the hospital images, it is scanner metadata. For speech, it might be speaker or recording channel. For source code, file paths and boilerplate may predict missing tokens. For industrial time series, maintenance schedule, sensor identity or stable operating regime may dominate.

The rival should be executable where possible. Remove local order, randomise metadata, swap backgrounds, equalise marginal statistics or train a model with the intended mechanism disabled. The shuffled-word-order experiment is powerful because it constructs a concrete rival: higher-order co-occurrence without normal syntax can explain more downstream performance than a syntax-first story predicts.[17]

Step 3: move the nuisance, not the target

An intervention is useful only if it changes the suspected cue while preserving the property under test. Randomly corrupting everything creates an out-of-distribution mess and does not isolate a mechanism. Better interventions include scanner-style transfer with anatomy held fixed, colour changes with object identity fixed, speaker swaps with phonetic content fixed, or metadata reversal in a synthetic generator whose latent rule is unchanged.

Measure both prediction and transfer. A rise in pretext loss shows dependence of the whole configured predictor. A collapse in a frozen-encoder probe shows that the dependence reached the representation. If only the decoder fails, the encoder may still be useful. If pretext loss remains stable but transfer fails, the objective may preserve multiple interchangeable cues, and the probe selected the unstable one.

Step 4: report four outcomes, not one score

The first outcome is in-distribution log-loss or accuracy. The second is counterfactual log-loss under the nuisance intervention. Their difference in bits is shortcut debt. The third is transfer performance with a declared label budget. The fourth is representation stability across seeds, decoder capacities and mask policies. This extends the spirit of risk-decomposition work, which separates approximation, representation usability, probe generalisation and encoder generalisation rather than treating a single linear-probe score as diagnosis.[18]

#!/usr/bin/env python3
"""Counterfactual Compression Lab for Paper 37.

A tiny self-supervised masked autoencoder is pretrained on synthetic binary
sequences. The latent regime changes the probability of each core bit, while a
single shortcut bit is highly correlated with that regime during training.

Two masking policies are compared:
  1. shortcut_visible: the shortcut remains visible during pretraining.
  2. counterfactual_mask: the shortcut is always replaced by an unknown value.

No regime labels are used during pretraining. A small labelled probe is fitted
only after the encoder has been frozen. The shifted test set flips the shortcut
while preserving the core data-generating process.

Dependency: NumPy only.
"""

from __future__ import annotations

import argparse
import csv
from dataclasses import dataclass, asdict
from pathlib import Path
from typing import Literal

import numpy as np

ShortcutMode = Literal["aligned", "flipped", "random"]


def sigmoid(x: np.ndarray) -> np.ndarray:
    """Numerically stable logistic function."""
    return 1.0 / (1.0 + np.exp(-np.clip(x, -30.0, 30.0)))


def make_data(
    n: int,
    *,
    sequence_length: int = 20,
    shortcut_mode: ShortcutMode = "aligned",
    seed: int = 0,
) -> tuple[np.ndarray, np.ndarray]:
    """Create samples with a hidden regime and a manipulable shortcut.

    Regime 0 produces core bits with probability 0.2; regime 1 uses 0.8.
    Across both regimes the marginal bit frequency is approximately balanced,
    but the conditional structure differs. The first input coordinate is a
    shortcut correlated with the regime in the training distribution.
    """
    rng = np.random.default_rng(seed)
    regime = rng.integers(0, 2, size=n)
    probability = np.where(regime == 0, 0.2, 0.8)
    core = (rng.random((n, sequence_length)) < probability[:, None]).astype(float)

    if shortcut_mode == "aligned":
        shortcut = np.where(rng.random(n) < 0.99, regime, 1 - regime)
    elif shortcut_mode == "flipped":
        shortcut = np.where(rng.random(n) < 0.99, 1 - regime, regime)
    elif shortcut_mode == "random":
        shortcut = rng.integers(0, 2, size=n)
    else:
        raise ValueError(f"Unsupported shortcut_mode: {shortcut_mode}")

    inputs = np.concatenate([shortcut[:, None], core], axis=1)
    return inputs.astype(float), regime.astype(float)


class MaskedAutoencoder:
    """One-hidden-layer masked autoencoder with a deliberately small bottleneck."""

    def __init__(self, input_dim: int, hidden_dim: int, *, seed: int) -> None:
        rng = np.random.default_rng(seed)
        self.w_encoder = rng.normal(0.0, 0.25, size=(input_dim, hidden_dim))
        self.b_encoder = np.zeros(hidden_dim)
        self.w_decoder = rng.normal(0.0, 0.25, size=(hidden_dim, input_dim - 1))
        self.b_decoder = np.zeros(input_dim - 1)

    def encode(self, inputs: np.ndarray) -> np.ndarray:
        return np.tanh(inputs @ self.w_encoder + self.b_encoder)

    def train(
        self,
        inputs: np.ndarray,
        *,
        counterfactual_mask: bool,
        epochs: int,
        batch_size: int,
        learning_rate: float,
        mask_rate: float,
        seed: int,
    ) -> None:
        """Minimise binary cross-entropy on masked core positions only."""
        rng = np.random.default_rng(seed)
        sample_count, input_dim = inputs.shape

        for _ in range(epochs):
            order = rng.permutation(sample_count)
            for start in range(0, sample_count, batch_size):
                batch = inputs[order[start : start + batch_size]].copy()
                masked = rng.random((len(batch), input_dim - 1)) < mask_rate
                corrupted = batch.copy()
                corrupted[:, 1:][masked] = 0.5
                if counterfactual_mask:
                    corrupted[:, 0] = 0.5

                hidden = np.tanh(corrupted @ self.w_encoder + self.b_encoder)
                output = sigmoid(hidden @ self.w_decoder + self.b_decoder)

                normaliser = max(int(masked.sum()), 1)
                output_gradient = (output - batch[:, 1:]) * masked / normaliser
                grad_w_decoder = hidden.T @ output_gradient
                grad_b_decoder = output_gradient.sum(axis=0)

                hidden_gradient = (
                    output_gradient @ self.w_decoder.T * (1.0 - hidden * hidden)
                )
                grad_w_encoder = corrupted.T @ hidden_gradient
                grad_b_encoder = hidden_gradient.sum(axis=0)

                self.w_encoder -= learning_rate * grad_w_encoder
                self.b_encoder -= learning_rate * grad_b_encoder
                self.w_decoder -= learning_rate * grad_w_decoder
                self.b_decoder -= learning_rate * grad_b_decoder

    def reconstruction_metrics(
        self,
        inputs: np.ndarray,
        *,
        counterfactual_mask: bool,
        mask_rate: float,
        seed: int,
    ) -> tuple[float, float]:
        """Return masked-token accuracy and binary log-loss in bits."""
        rng = np.random.default_rng(seed)
        input_dim = inputs.shape[1]
        masked = rng.random((len(inputs), input_dim - 1)) < mask_rate
        corrupted = inputs.copy()
        corrupted[:, 1:][masked] = 0.5
        if counterfactual_mask:
            corrupted[:, 0] = 0.5

        prediction = sigmoid(self.encode(corrupted) @ self.w_decoder + self.b_decoder)
        target = inputs[:, 1:]
        hard_prediction = prediction >= 0.5
        accuracy = float((hard_prediction[masked] == target[masked]).mean())

        epsilon = 1e-12
        bit_loss = -(
            target[masked] * np.log2(prediction[masked] + epsilon)
            + (1.0 - target[masked]) * np.log2(1.0 - prediction[masked] + epsilon)
        ).mean()
        return accuracy, float(bit_loss)


def fit_linear_probe(
    representations: np.ndarray,
    labels: np.ndarray,
    *,
    epochs: int = 500,
    learning_rate: float = 0.1,
    l2: float = 0.01,
) -> tuple[np.ndarray, float]:
    """Fit a logistic probe without external ML libraries."""
    weights = np.zeros(representations.shape[1])
    bias = 0.0
    sample_count = len(labels)

    for _ in range(epochs):
        probabilities = sigmoid(representations @ weights + bias)
        residual = probabilities - labels
        weights -= learning_rate * (
            representations.T @ residual / sample_count + l2 * weights
        )
        bias -= learning_rate * float(residual.mean())
    return weights, bias


def probe_accuracy(
    model: MaskedAutoencoder,
    labelled_inputs: np.ndarray,
    labelled_targets: np.ndarray,
    evaluation_inputs: np.ndarray,
    evaluation_targets: np.ndarray,
) -> float:
    weights, bias = fit_linear_probe(model.encode(labelled_inputs), labelled_targets)
    predictions = sigmoid(model.encode(evaluation_inputs) @ weights + bias) >= 0.5
    return float((predictions == evaluation_targets).mean())


@dataclass(frozen=True)
class Result:
    policy: str
    iid_reconstruction_accuracy: float
    shifted_reconstruction_accuracy: float
    iid_bit_loss: float
    shifted_bit_loss: float
    shortcut_debt_bits: float
    iid_probe_accuracy: float
    shifted_probe_accuracy: float


def run_policy(
    policy: str,
    *,
    train_inputs: np.ndarray,
    train_regimes: np.ndarray,
    iid_inputs: np.ndarray,
    iid_regimes: np.ndarray,
    shifted_inputs: np.ndarray,
    shifted_regimes: np.ndarray,
    hidden_dim: int,
    epochs: int,
    batch_size: int,
    learning_rate: float,
    mask_rate: float,
    seed: int,
    probe_labels: int,
) -> Result:
    counterfactual_mask = policy == "counterfactual_mask"
    model = MaskedAutoencoder(train_inputs.shape[1], hidden_dim, seed=seed + 3)
    model.train(
        train_inputs,
        counterfactual_mask=counterfactual_mask,
        epochs=epochs,
        batch_size=batch_size,
        learning_rate=learning_rate,
        mask_rate=mask_rate,
        seed=seed + 11,
    )

    iid_accuracy, iid_loss = model.reconstruction_metrics(
        iid_inputs,
        counterfactual_mask=counterfactual_mask,
        mask_rate=mask_rate,
        seed=seed + 19,
    )
    shifted_accuracy, shifted_loss = model.reconstruction_metrics(
        shifted_inputs,
        counterfactual_mask=counterfactual_mask,
        mask_rate=mask_rate,
        seed=seed + 19,
    )

    labelled_inputs = train_inputs[:probe_labels]
    labelled_regimes = train_regimes[:probe_labels]
    iid_probe = probe_accuracy(
        model, labelled_inputs, labelled_regimes, iid_inputs, iid_regimes
    )
    shifted_probe = probe_accuracy(
        model,
        labelled_inputs,
        labelled_regimes,
        shifted_inputs,
        shifted_regimes,
    )

    return Result(
        policy=policy,
        iid_reconstruction_accuracy=iid_accuracy,
        shifted_reconstruction_accuracy=shifted_accuracy,
        iid_bit_loss=iid_loss,
        shifted_bit_loss=shifted_loss,
        shortcut_debt_bits=shifted_loss - iid_loss,
        iid_probe_accuracy=iid_probe,
        shifted_probe_accuracy=shifted_probe,
    )


def print_results(results: list[Result]) -> None:
    header = (
        "policy                 iid_rec  shift_rec  iid_bits  shift_bits  "
        "debt_bits  iid_probe  shift_probe"
    )
    print(header)
    print("-" * len(header))
    for result in results:
        print(
            f"{result.policy:<22} "
            f"{result.iid_reconstruction_accuracy:>7.3f}  "
            f"{result.shifted_reconstruction_accuracy:>9.3f}  "
            f"{result.iid_bit_loss:>8.3f}  "
            f"{result.shifted_bit_loss:>10.3f}  "
            f"{result.shortcut_debt_bits:>9.3f}  "
            f"{result.iid_probe_accuracy:>9.3f}  "
            f"{result.shifted_probe_accuracy:>11.3f}"
        )


def write_csv(results: list[Result], path: Path) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    with path.open("w", newline="", encoding="utf-8") as handle:
        writer = csv.DictWriter(handle, fieldnames=list(asdict(results[0]).keys()))
        writer.writeheader()
        for result in results:
            writer.writerow(asdict(result))


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--train-size", type=int, default=10_000)
    parser.add_argument("--test-size", type=int, default=3_000)
    parser.add_argument("--sequence-length", type=int, default=20)
    parser.add_argument("--hidden-dim", type=int, default=3)
    parser.add_argument("--epochs", type=int, default=80)
    parser.add_argument("--batch-size", type=int, default=128)
    parser.add_argument("--learning-rate", type=float, default=0.2)
    parser.add_argument("--mask-rate", type=float, default=0.85)
    parser.add_argument("--probe-labels", type=int, default=1_000)
    parser.add_argument("--seed", type=int, default=1)
    parser.add_argument("--csv", type=Path, default=None)
    return parser.parse_args()


def main() -> None:
    args = parse_args()
    if not 0.0 < args.mask_rate < 1.0:
        raise SystemExit("--mask-rate must be between 0 and 1")
    if args.probe_labels > args.train_size:
        raise SystemExit("--probe-labels cannot exceed --train-size")

    train_inputs, train_regimes = make_data(
        args.train_size,
        sequence_length=args.sequence_length,
        shortcut_mode="aligned",
        seed=args.seed,
    )
    iid_inputs, iid_regimes = make_data(
        args.test_size,
        sequence_length=args.sequence_length,
        shortcut_mode="aligned",
        seed=args.seed + 1,
    )
    shifted_inputs, shifted_regimes = make_data(
        args.test_size,
        sequence_length=args.sequence_length,
        shortcut_mode="flipped",
        seed=args.seed + 2,
    )

    common = dict(
        train_inputs=train_inputs,
        train_regimes=train_regimes,
        iid_inputs=iid_inputs,
        iid_regimes=iid_regimes,
        shifted_inputs=shifted_inputs,
        shifted_regimes=shifted_regimes,
        hidden_dim=args.hidden_dim,
        epochs=args.epochs,
        batch_size=args.batch_size,
        learning_rate=args.learning_rate,
        mask_rate=args.mask_rate,
        seed=args.seed,
        probe_labels=args.probe_labels,
    )
    results = [
        run_policy("shortcut_visible", **common),
        run_policy("counterfactual_mask", **common),
    ]

    print_results(results)
    if args.csv is not None:
        write_csv(results, args.csv)
        print(f"\nWrote {args.csv}")


if __name__ == "__main__":
    main()

The frozen probe sharpens the interpretation. Under the default run, the shortcut-visible encoder scores 0.989 on the aligned test and 0.013 after the shortcut flips. The counterfactually masked encoder scores 0.969 and 1.000. The result is intentionally stark because the synthetic generator isolates one mechanism. It shows why an apparently better pretext score can be worse evidence for reuse.

The lab cannot establish that counterfactual masking is always beneficial. In real data, a supposed nuisance may carry legitimate information; an intervention may damage the target property; the representation may be nonlinear and poorly served by a linear probe; and an operational shift may combine many changes. The experiment is a microscope, not a benchmark.

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: The camera that learned the timestamp
  M->>E: The forecast horizon changes the object of learning
  E-->>M: compare rival explanations
  E->>D: Choose a prediction game whose shortcuts fail
  D-->>Q: revise scope or act

Choose a prediction game whose shortcuts fail

Self-supervision is often justified by data abundance: labels are scarce, so predict one part of the data from another. That is a sound starting point, not a complete design. The hidden target, visible context and prediction boundary decide which regularities pay rent. Scale can make the optimisation more effective without making the objective better aligned.

The strongest alternative explanation for successful transfer is not that the model recovered the world’s true latent causes. It is that natural data contains many overlapping regularities, and a representation useful for one constructed target often happens to support other tasks. This broad statistical reuse is real and valuable. It remains weaker than causal identification. Two environments can share predictive texture, co-occurrence and metadata while differing in the mechanism a consequential decision needs.

There are also cases where exact generative modelling is the right decision. Restoration needs surface detail. Compression needs calibrated probabilities over the source. Simulation may need multimodal futures rather than invariant embeddings. In those settings, removing “nuisance” detail can destroy the product. Conversely, a control or diagnosis system may benefit from latent targets that ignore unpredictable detail and preserve state. No objective is generative, semantic or robust in the abstract. Its value is conditional on the decision and the permitted losses.

The practical decision is therefore simple to state and demanding to follow. Do not choose a self-supervised objective because it is fashionable, label-free or strong on a generic linear probe. Specify the downstream relation that must survive. Design masks, views and horizons that remove cheap alternatives. Declare the encoder boundary. Build at least one rival compressor. Intervene on suspected nuisances. Measure added bits, transfer and stability separately.

Select the prediction game whose cheapest solution remains useful after the world changes in the ways you already know how to test. When that condition is absent, a low pretraining loss is not evidence that the model learned the structure you care about. It is evidence that the model learned to win the game you gave it.