A static recipe hides time

A typical model card says that a Transformer was trained on a corpus with causal language modelling, AdamW and a cosine learning-rate schedule. These facts matter, but they describe categories rather than the realised process. They do not tell us which documents arrived before the learning rate decayed, which samples were duplicated, when gradients were clipped, whether the optimiser state was reset, or which checkpoints first acquired a capability.

The Transformer architecture made sequence computation highly parallel, helping scale next-token training across large datasets.^[1] Yet architecture supplies a family of possible functions, not a unique endpoint. The endpoint is created by repeated state transitions. At step t, a batch produces a stochastic gradient; the optimiser combines it with stored state; the schedule chooses the step size; then the changed model encounters the next batch.

We can compress the process into one typed transition:

Here θ is the model’s parameters, s is optimiser state, g is the gradient from the current batch, η is the learning rate and λ represents regularisation controls. The equation is useful because it names what must persist. It is valid as a schematic, not as a complete physical description. Distributed reduction order, numerical precision, dropped batches and random-number state can also change the transition.

Why order is a causal variable

Take two batches, Alpha and Beta. Alpha rewards a feature that separates birds from tools. Beta rewards a feature that separates red objects from blue ones. If the network has ample independent capacity and both features can be learned without interference, order may make little practical difference. Now impose a narrow bottleneck or strong regularisation. Learning the bird feature first changes the hidden basis available when Beta arrives. Learning colour first changes it in another way. The gradient of Beta is evaluated against a different model in each route.

This is the simplest reason update effects need not commute. Applying Alpha and then Beta is not generally equivalent to applying Beta and then Alpha. The second gradient is not a stored instruction waiting to be added; it is recomputed after the first intervention has changed activations, predictions and optimiser state. Even ordinary stochastic gradient descent therefore carries history in the parameters. Adaptive methods add explicit history in their moments.

A useful diagnostic is to run an order commutator. Start several replicas from one checkpoint. Apply small, controlled batches A then B in one branch and B then A in another, holding random choices fixed. Compare final logits, representations and counterfactual behaviour. A negligible difference under realistic step sizes supports local order-insensitivity. A stable difference identifies an interaction worth observing in the full curriculum. The test does not predict the preferred global order, but it exposes where the commutative intuition fails.

This also clarifies what reproducibility should mean. Bit-for-bit replay is valuable when available, but scientific reproducibility can tolerate tiny numerical differences if the same claims survive across replicas. The required invariant depends on the question. To resume a job after failure, preserve exact process state. To support a mechanism claim, reproduce the behavioural transition, its checkpoint range and its intervention effect across seeds and compatible hardware.

Four controls, one causal chain

The route can be decomposed without pretending the components are independent. Gradient exposure asks which examples and token positions generated errors. Objective pricing asks how those errors were weighted. Update filtering asks how the optimiser, clipping, precision and schedule transformed the gradient into movement. Regime reachability asks which representations and behaviours were accessible within the model, token and compute budget.

flowchart TD
  A["The two runs that ate the same<br/>corpus"] --> B["The model is the trajectory"]
  B --> C["What each batch is allowed to<br/>teach"]
  C --> D["The optimiser remembers the route"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#B96D00,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

One token, two very different lessons

Suppose the correct next token is nest. Model A assigns it probability 0.8. Model B assigns it probability 0.2. Causal language modelling usually applies negative log probability to the correct token. The two losses are approximately 0.223 and 1.609. The second example therefore contributes about 7.2 times as much loss before any explicit weighting.

This is why “the model saw the sentence” is an incomplete statement. What mattered was the probability it assigned at that moment, the surrounding context, the position mask, any token weight, and the other examples sharing the batch. A familiar pattern may generate almost no update. A surprising pattern may dominate. Exposure is filtered by current competence.

For a time-varying data distribution, a useful schematic is:

The distribution q_t decides which sequences appear at step t. The token weight w_i decides how much position i counts. The model probability supplies the surprise. This equation assumes a causal next-token objective. Replace it with masked prediction, span corruption, preference loss or a multimodal contrastive loss and the information pressure changes, even when the raw files are identical.

The dataset is made again at runtime

Raw documents are not the units that reach the optimiser. A runtime pipeline normalises text, tokenises it, chooses sequence boundaries, packs fragments, inserts separators, masks positions and constructs batches. Each operation changes the prediction problem. Two teams can point to the same files yet train on different conditional distributions.

Consider packing. If the end of a medical article and the start of a software licence share one context window without a hard attention boundary, the model is asked to predict the licence while attending to unrelated medical text. Usually the separator makes this relation learnably ignorable, but it still consumes context and contributes gradients. If short documents are grouped by source, local domain runs become longer. If they are mixed aggressively, domain transitions become common. The corpus has not changed as a set of files, while the stream has changed as a learning environment.

Deduplication also has more than one causal effect. It reduces repeated content, changes the frequency of templates and changes when the remaining copies arrive. Filtering can remove noise while also deleting dialect, minority-domain or hard examples. Sequence length changes how many independent examples fit in a batch and which dependencies are available to attention. A tokenizer can make one language consume more positions, thereby changing both its effective context and its share of the token budget.

The operational data object is therefore the ordered token stream plus its construction rules. A defensible manifest records source snapshots, licences and exclusions, but it must also record tokenizer version, normalisation, deduplication keys, packer, sequence boundaries, shuffle seed, sampler state and the data cursor. Without those fields, “same data” can conceal a different training intervention.

Mixture is part of the model specification

Now consider two domains: general prose and source code. A 90:10 token split does not imply a 90:10 gradient influence. Code may be less predictable and produce larger token losses. Long files may be packed differently. Deduplication can remove repeated prose but retain templated code. Sampling with replacement can revisit one domain after another has exhausted its unique examples.

Controlled studies show that these choices matter. DoReMi optimises domain mixture weights with a proxy model and reported improved perplexity across tested Pile domains and faster attainment of its baseline accuracy.^[6] DataComp-LM provides a controlled testbed for filtering, deduplication and mixing, and its baseline experiments identify model-based filtering as important in that setting.^[7] Dolma documents a large open corpus and its curation pipeline so that data effects can be studied rather than inferred from a model endpoint.^[8]

A checkpoint is more than weights

Adam maintains exponentially decaying estimates of the first and second moments of gradients.^[2] AdamW decouples weight decay from the loss-gradient update rather than treating it as ordinary L2 regularisation inside an adaptive optimiser.^[3] These details matter because two runs with identical parameters and the same next batch can move differently if their moment estimates differ.

The first moment m smooths recent gradient direction. The second moment v scales coordinates by recent squared gradients. Bias-corrected versions are marked with hats. The approximate update assumes a standard AdamW-style implementation and omits implementation details. Resetting m, v or the step counter changes the algorithm even when θ is copied exactly.

Gradient conflict changes with the learner

When two domains prefer different updates, their gradients can align, oppose or occupy nearly independent directions. That relationship is not fixed in the dataset. It changes as the representation changes. Early in training, code and prose may both reward basic local syntax. Later, one domain may push towards exact symbolic copying while another rewards semantic paraphrase. A mixture that was cooperative can become competitive.

This matters for adaptive sampling. Raising the weight of the currently worst domain can be sensible when high loss reflects under-training. It can be harmful when high loss reflects label noise, irreducible ambiguity or a conflict with a more important invariant. A loss-only controller cannot distinguish these cases. Add held-out quality checks and measure whether the domain’s gradient improves or damages the decision-relevant probes.

The smallest practical conflict test takes a checkpoint and computes gradients for several labelled slices without stepping the optimiser. Compare their norms and pairwise cosine similarities, then apply small candidate updates in forks. Similarity is only a local clue. The fork tells whether the apparent conflict actually changes the behaviour that matters.

Learning rate is a time-dependent authority

A learning-rate schedule does more than make training numerically stable. It decides when evidence can substantially revise the model. Warmup limits early movement while activations and moments stabilise. A high-rate middle period explores broad regions. Decay reduces movement and can consolidate a representation. The same domain placed in different periods therefore receives different authority over the endpoint.

Batch size also changes the stochastic process. Larger batches reduce sampling noise but use fewer parameter updates for a fixed token budget. Gradient clipping changes unusually large updates. Reduced precision and distributed aggregation can alter small numerical effects. Weight decay pressures parameters continuously, even when a particular feature receives little direct gradient. None of these controls has a single context-free effect; they interact.

Parallel training needs a semantic replay target

Large training jobs distribute examples and parameters across devices. Gradient accumulation decides which microbatches become one update. Data-parallel workers reduce gradients in an order that can vary. A failed worker can cause a batch to be replayed or skipped. Mixed-precision loss scaling can accept or reject an update. These are not reasons to demand impossible hardware-independent bit identity; they are reasons to define what must remain invariant.

For operational recovery, the target is a consistent cut: model weights, optimiser and scheduler state, scaler state, random generators, completed update count and the exact data position must agree. For scientific comparison, the target may be a distribution over trajectories: several seeds should show the same qualitative regime and similar decision-relevant outcomes. For audit, the target is lineage: a reviewer can reconstruct which data and code version had authority over each checkpoint.

Reproducibility is strongest when its tolerance is typed. “Exact resume” permits almost no drift. “Claim replication” permits numerical drift but not a reversal of the causal result. “Product equivalence” may tolerate internal differences while requiring calibrated behaviour and bounded failure on specified slices. Mixing these standards creates false confidence or needless rigidity.

Scale changes what the budget can reach

For a dense Transformer, a common rough approximation for training compute is C ≈ 6ND, where N is the number of non-embedding parameters and D is the number of training tokens. The factor depends on accounting assumptions, but the trade-off is the point: at fixed compute, doubling parameters roughly halves available tokens.

Scaling-law studies found smooth empirical relationships between loss, model size, data and compute over studied ranges.^[4] Hoffmann and colleagues trained more than 400 models and argued that, under their setup, compute-optimal model size and training tokens should grow together rather than scaling parameters while holding data nearly fixed.^[5] These are empirical regularities, not physical laws that determine every architecture, data quality or downstream objective.

Hyperparameters also need a scaling rule. Under Maximal Update Parametrisation, Yang and colleagues found that many tuned hyperparameters could remain stable across width and be transferred from smaller models in their tested settings.^[14] That result is valuable precisely because naive parameter scaling does not guarantee equivalent update dynamics.

stateDiagram-v2
  [*] --> Claim
  state "A static recipe hides time" as Claim
  state "Why order is a causal variable" as Test
  state "Four controls, one causal chain" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Three kinds of transition

Training curves invite stories. A capability appears at 300 million parameters, so we say scale caused emergence. Validation accuracy jumps after 40,000 steps, so we say the model discovered an algorithm. Sometimes that interpretation is useful. Sometimes the observed discontinuity belongs to the measurement.

A measurement transition occurs when a continuous latent quantity crosses a threshold. Exact-match accuracy is zero until every required token wins, even if their probabilities improve smoothly. An optimisation transition occurs when the training process enters a different dynamical regime, perhaps after warmup, weight decay, curriculum change or delayed generalisation. A mechanism transition is stronger: a new internal causal strategy begins to control the answer.

Grokking experiments showed cases where small algorithmic tasks moved from memorisation and poor test performance to strong generalisation long after training error was low.^[11] Work describing emergent abilities focused attention on capabilities that appear absent in smaller models and present in larger ones.^[12] Schaeffer and colleagues demonstrated that nonlinear or discontinuous metrics can make smooth underlying changes appear abrupt in several analysed settings.^[13] These findings are not mutually exclusive. They imply that “phase transition” must be typed before it is explained.

What a phase claim must carry

A serious phase claim needs more than a before-and-after score. It needs a trajectory window narrow enough to locate the change, a continuous behavioural measure, a description of the training-state change, and a rival generator that could produce the same observation. For a reasoning task, the continuous measure might be answer-token margin rather than exact match. The state record might include learning rate, update norm, domain mix and representation probes. The rival might be a memorising lookup strategy.

Next, vary the ruler. Sweep thresholds, prompt formats and partial-credit metrics. If the onset moves freely, the phase is largely measurement-defined. Then vary the route. Repeat seeds, reorder a matched curriculum or continue from checkpoints on either side. If the jump is tightly locked to one scheduler boundary, an optimisation explanation gains weight. Finally, intervene on the candidate mechanism. Patch, ablate or constrain the proposed representation and ask whether the predicted capability changes selectively.

No one test settles the issue. Threshold robustness can coexist with a shortcut. An intervention can create off-distribution states. A representation change can be an effect rather than a cause. The evidence becomes stronger when independent traces converge and their failure modes differ. Phase is a typed causal claim, not a synonym for a steep graph.

An executable path-dependence lab

The following miniature experiment makes the central mechanism visible without claiming to model frontier pretraining. A one-block causal Transformer predicts sequences of the form <bos> colour A/B X/Y -> yes/no <eos>. The correct label is an XNOR relation between the two letter attributes. Colour is causally irrelevant.

One synthetic domain makes colour perfectly predict the label. The other balances colours and breaks that shortcut. Both curricula contain the same pre-generated batches and begin from the same initial weights. Only phase order changes. Phase one uses learning rate 0.002; phase two uses 0.00001. The 200-fold drop intentionally magnifies the interaction between curriculum and plasticity.

The next experiment should vary one causal property at a time. Keep order fixed and flatten the schedule to test whether declining plasticity carried the effect. Keep the schedule and reset Adam moments at the phase boundary to test explicit optimiser memory. Increase model width to test whether spare capacity lets the causal rule coexist with the shortcut. Add label noise to test whether balanced-first success depended on perfectly clean evidence. These interventions convert a demonstration into a mechanism map.

What would strengthen the proposed account? The curriculum effect should grow as the learning-rate contrast grows, shrink when phase two regains plasticity, and change when the bottleneck is relaxed. What would weaken it? Stable differences under a constant schedule and after optimiser-state matching would point towards another route, such as representation competition created by data order alone. A rival explanation is seed-specific luck, which is why the artefact reports medians and retains individual runs.

#!/usr/bin/env python3
"""Synthetic path-dependence lab for a miniature causal language model.

The two curricula contain the same pre-generated batches. Only their order changes.
A high learning rate is used in phase one and a much smaller rate in phase two,
so the experiment makes the interaction between data order and optimiser schedule visible.

Requires: Python 3.10+ and PyTorch 2.0+
Run: python training_trajectory_lab.py --seeds 5 --steps-per-phase 100
"""
from __future__ import annotations

import argparse
import json
import statistics
from dataclasses import asdict, dataclass
from typing import Iterable, Literal

import torch
import torch.nn as nn
import torch.nn.functional as F

Domain = Literal["shortcut", "balanced"]
Curriculum = tuple[Domain, Domain]

TOKENS = [
    "<bos>", "red", "green", "A", "B", "X", "Y",
    "->", "yes", "no", "<eos>",
]
STOI = {token: index for index, token in enumerate(TOKENS)}
COLOURS = ("red", "green")
FIRST_BITS = ("A", "B")
SECOND_BITS = ("X", "Y")


@dataclass(frozen=True)
class Config:
    seeds: int = 5
    steps_per_phase: int = 100
    batch_size: int = 64
    d_model: int = 24
    heads: int = 4
    layers: int = 1
    learning_rate_high: float = 2e-3
    learning_rate_low: float = 1e-5
    weight_decay: float = 5e-2
    beta1: float = 0.9
    beta2: float = 0.95
    clip_norm: float = 1.0


def true_label(first: str, second: str) -> str:
    """XNOR: yes when the two binary attributes agree, otherwise no."""
    return "yes" if ((first == "A") == (second == "X")) else "no"


def sequence(colour: str, first: str, second: str) -> torch.Tensor:
    label = true_label(first, second)
    items = ("<bos>", colour, first, second, "->", label, "<eos>")
    return torch.tensor([STOI[item] for item in items], dtype=torch.long)


def make_domains() -> dict[Domain, list[torch.Tensor]]:
    balanced = [
        sequence(colour, first, second)
        for colour in COLOURS
        for first in FIRST_BITS
        for second in SECOND_BITS
    ]
    shortcut = []
    for first in FIRST_BITS:
        for second in SECOND_BITS:
            label = true_label(first, second)
            colour = "red" if label == "yes" else "green"
            shortcut.append(sequence(colour, first, second))
    return {"shortcut": shortcut, "balanced": balanced}


class CausalBlock(nn.Module):
    def __init__(self, d_model: int, heads: int) -> None:
        super().__init__()
        self.norm_attention = nn.LayerNorm(d_model)
        self.attention = nn.MultiheadAttention(
            d_model, heads, dropout=0.0, batch_first=True
        )
        self.norm_mlp = nn.LayerNorm(d_model)
        self.mlp = nn.Sequential(
            nn.Linear(d_model, 4 * d_model),
            nn.GELU(),
            nn.Linear(4 * d_model, d_model),
        )

    def forward(self, states: torch.Tensor) -> torch.Tensor:
        length = states.size(1)
        causal_mask = torch.triu(
            torch.ones(length, length, dtype=torch.bool, device=states.device),
            diagonal=1,
        )
        normalised = self.norm_attention(states)
        attended, _ = self.attention(
            normalised,
            normalised,
            normalised,
            attn_mask=causal_mask,
            need_weights=False,
        )
        states = states + attended
        return states + self.mlp(self.norm_mlp(states))


class MiniLanguageModel(nn.Module):
    def __init__(self, config: Config) -> None:
        super().__init__()
        self.token_embedding = nn.Embedding(len(TOKENS), config.d_model)
        self.position_embedding = nn.Embedding(8, config.d_model)
        self.blocks = nn.ModuleList(
            CausalBlock(config.d_model, config.heads)
            for _ in range(config.layers)
        )
        self.final_norm = nn.LayerNorm(config.d_model)
        self.output = nn.Linear(config.d_model, len(TOKENS), bias=False)
        self.output.weight = self.token_embedding.weight

    def forward(self, token_ids: torch.Tensor) -> torch.Tensor:
        _, length = token_ids.shape
        positions = torch.arange(length, device=token_ids.device)
        states = self.token_embedding(token_ids)
        states = states + self.position_embedding(positions)[None, :, :]
        for block in self.blocks:
            states = block(states)
        return self.output(self.final_norm(states))


def causal_loss(logits: torch.Tensor, targets: torch.Tensor) -> torch.Tensor:
    """Weighted next-token cross-entropy; the semantic label receives most weight."""
    batch, length, vocabulary = logits.shape
    per_token = F.cross_entropy(
        logits.reshape(-1, vocabulary),
        targets.reshape(-1),
        reduction="none",
    ).reshape(batch, length)
    weights = torch.full((length,), 0.1, device=logits.device)
    weights[4] = 1.0  # after '->', predict yes/no
    return (per_token * weights).sum() / (batch * weights.sum())


def sample_batch(
    pool: list[torch.Tensor], batch_size: int, generator: torch.Generator
) -> torch.Tensor:
    indices = torch.randint(0, len(pool), (batch_size,), generator=generator)
    return torch.stack([pool[index] for index in indices.tolist()])


def precomputed_batches(
    domains: dict[Domain, list[torch.Tensor]], config: Config, seed: int
) -> dict[Domain, list[torch.Tensor]]:
    generators = {
        "shortcut": torch.Generator().manual_seed(10_000 + seed),
        "balanced": torch.Generator().manual_seed(20_000 + seed),
    }
    return {
        domain: [
            sample_batch(domains[domain], config.batch_size, generators[domain])
            for _ in range(config.steps_per_phase)
        ]
        for domain in ("shortcut", "balanced")
    }


@torch.no_grad()
def evaluate(
    model: MiniLanguageModel, balanced_examples: Iterable[torch.Tensor]
) -> dict[str, float]:
    examples = torch.stack(list(balanced_examples))
    inputs, targets = examples[:, :-1], examples[:, 1:]
    logits = model(inputs)
    loss = float(causal_loss(logits, targets))
    predicted = logits[:, 4].argmax(dim=-1)
    expected = targets[:, 4]
    accuracy = float((predicted == expected).float().mean())

    # For every attribute pair, changing colour should not change the answer.
    invariant_and_correct = []
    for first_index, _ in enumerate(FIRST_BITS):
        for second_index, _ in enumerate(SECOND_BITS):
            indices = [
                colour_index * 4 + first_index * 2 + second_index
                for colour_index in range(len(COLOURS))
            ]
            pair_predictions = predicted[indices]
            pair_targets = expected[indices]
            invariant = bool((pair_predictions == pair_predictions[0]).all())
            correct = bool((pair_predictions == pair_targets).all())
            invariant_and_correct.append(invariant and correct)

    return {
        "validation_loss": loss,
        "label_accuracy": accuracy,
        "counterfactual_score": sum(invariant_and_correct)
        / len(invariant_and_correct),
    }


def train_one(
    curriculum: Curriculum, config: Config, seed: int
) -> tuple[dict[str, float], list[dict[str, float | str | int]]]:
    torch.manual_seed(seed)
    domains = make_domains()
    batches = precomputed_batches(domains, config, seed)
    model = MiniLanguageModel(config)
    optimiser = torch.optim.AdamW(
        model.parameters(),
        lr=config.learning_rate_high,
        betas=(config.beta1, config.beta2),
        weight_decay=config.weight_decay,
    )

    trajectory: list[dict[str, float | str | int]] = []
    step = 0
    for phase_index, domain in enumerate(curriculum):
        learning_rate = (
            config.learning_rate_high
            if phase_index == 0
            else config.learning_rate_low
        )
        for group in optimiser.param_groups:
            group["lr"] = learning_rate

        for batch in batches[domain]:
            step += 1
            inputs, targets = batch[:, :-1], batch[:, 1:]
            optimiser.zero_grad(set_to_none=True)
            loss = causal_loss(model(inputs), targets)
            loss.backward()
            gradient_norm = float(
                torch.nn.utils.clip_grad_norm_(model.parameters(), config.clip_norm)
            )
            optimiser.step()

            if step % 20 == 0 or step in {
                config.steps_per_phase,
                2 * config.steps_per_phase,
            }:
                observation = evaluate(model, domains["balanced"])
                trajectory.append(
                    {
                        "step": step,
                        "phase": domain,
                        "learning_rate": learning_rate,
                        "training_loss": float(loss.detach()),
                        "gradient_norm": gradient_norm,
                        **observation,
                    }
                )

    return evaluate(model, domains["balanced"]), trajectory


def median_summary(runs: list[dict[str, float]]) -> dict[str, float]:
    return {
        key: statistics.median(run[key] for run in runs)
        for key in ("validation_loss", "label_accuracy", "counterfactual_score")
    }


def run_experiment(config: Config) -> dict[str, object]:
    curricula: dict[str, Curriculum] = {
        "shortcut_then_balanced": ("shortcut", "balanced"),
        "balanced_then_shortcut": ("balanced", "shortcut"),
    }
    all_runs: dict[str, list[dict[str, float]]] = {name: [] for name in curricula}
    example_trajectories: dict[str, list[dict[str, float | str | int]]] = {}

    for seed in range(config.seeds):
        for name, curriculum in curricula.items():
            final, trajectory = train_one(curriculum, config, seed)
            all_runs[name].append(final)
            if seed == 0:
                example_trajectories[name] = trajectory

    return {
        "experiment": "same pre-generated batches, reversed phase order",
        "config": asdict(config),
        "median_over_seeds": {
            name: median_summary(runs) for name, runs in all_runs.items()
        },
        "runs": all_runs,
        "seed_0_trajectories": example_trajectories,
        "interpretation_boundary": (
            "A difference establishes path dependence in this configured synthetic run. "
            "It does not establish a universal direction or effect size for frontier models."
        ),
    }


def parse_args() -> Config:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--seeds", type=int, default=5)
    parser.add_argument("--steps-per-phase", type=int, default=100)
    parser.add_argument("--batch-size", type=int, default=64)
    parser.add_argument("--learning-rate-high", type=float, default=2e-3)
    parser.add_argument("--learning-rate-low", type=float, default=1e-5)
    args = parser.parse_args()
    if args.seeds < 1 or args.steps_per_phase < 1 or args.batch_size < 1:
        parser.error("seeds, steps-per-phase and batch-size must be positive")
    return Config(
        seeds=args.seeds,
        steps_per_phase=args.steps_per_phase,
        batch_size=args.batch_size,
        learning_rate_high=args.learning_rate_high,
        learning_rate_low=args.learning_rate_low,
    )


def main() -> None:
    torch.set_num_threads(1)
    config = parse_args()
    result = run_experiment(config)
    print(json.dumps(result, indent=2))


if __name__ == "__main__":
    main()

{
  "experiment": "same pre-generated batches, reversed phase order",
  "config": {
    "seeds": 5,
    "steps_per_phase": 100,
    "batch_size": 64,
    "d_model": 24,
    "heads": 4,
    "layers": 1,
    "learning_rate_high": 0.002,
    "learning_rate_low": 1e-05,
    "weight_decay": 0.05,
    "beta1": 0.9,
    "beta2": 0.95,
    "clip_norm": 1.0
  },
  "median_over_seeds": {
    "shortcut_then_balanced": {
      "validation_loss": 3.56215500831604,
      "label_accuracy": 0.5,
      "counterfactual_score": 0.0
    },
    "balanced_then_shortcut": {
      "validation_loss": 0.14094842970371246,
      "label_accuracy": 1.0,
      "counterfactual_score": 1.0
    }
  },
  "interpretation_boundary": "A difference establishes path dependence in this configured synthetic run. It does not establish a universal direction or effect size for frontier models."
}

From lab to training assurance

The miniature model exposes logic that prose hides, but a serious run needs a broader record. At minimum, preserve the data snapshot and ordered shard manifest, tokenizer, objective masks and weights, model configuration, initialisation, optimiser and scheduler state, precision and distributed settings, random generators, data cursor, checkpoints, evaluation code and environment versions.

Then attach instruments to the route. Aggregate validation loss is one. Add per-domain loss, held-out counterfactual probes, calibration, gradient and update norms, clipping rate, token throughput, parameter norm, optimiser statistics and checkpoint-to-checkpoint behavioural deltas. No single instrument certifies a mechanism. Together they can locate when a claim became plausible and which alternative explanations remain.

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: The two runs that ate the same corpus
  M->>E: A static recipe hides time
  E-->>M: compare rival explanations
  E->>D: Govern the trajectory, not just the final checkpoint
  D-->>Q: revise scope or act

The practical decision is more than which architecture to train. It is which sequence of learning opportunities the organisation is willing to authorise. A model can meet a headline benchmark after learning the wrong dependency. It can be impossible to resume faithfully because optimiser state or data position was omitted. It can appear to acquire a new ability because a threshold moved. It can become larger while receiving too little training to use its capacity.

A defensible run therefore begins with a trajectory manifest. Version the ordered data plan, objective components, parameterisation, optimiser, schedule, token and compute budget, evaluation rulers and stop conditions. Persist a complete resumable state. Instrument per-slice losses and updates. Place counterfactual probes at checkpoints, not only at the end. Treat any phase claim as provisional until it survives threshold sweeps, several seeds and a mechanism-relevant intervention.

Set branch points before the expensive run begins. A branch point is a checkpoint at which a disputed choice, such as mixture, objective weight or decay timing, can be changed while upstream history remains shared. It turns post-hoc debate into a controlled counterfactual. Budgeting a few deliberate branches often produces more knowledge than spending every token on one opaque trajectory.

Finally, define the null result in advance. If two routes finish within the agreed behavioural and mechanistic tolerance, record that the tested choice was not decision-relevant under this budget. Do not search indefinitely for a metric that separates them. A training programme becomes scientific when both divergence and equivalence can change the next decision.

This changes scaling decisions too. Ask which parameter-token allocation reaches the decision-relevant regime within the budget. Do not buy parameter count in isolation. When proxy tuning is used, document the invariants that make transfer plausible and the checks that would reject it. When data curation improves aggregate loss, test whether the improvement survives across target slices and rival explanations.

The central thesis now has an operational consequence. A language model is more than found at the minimum of a loss surface; it is made by a history of priced errors under finite authority to change. Preserve that history, observe its regime changes and test the rule the model actually uses. Otherwise, the final checkpoint may be repeatable as a file while remaining irreproducible as an explanation.