The reservoir that trained two different models

Imagine two teams receive the same reservoir of text: web pages, source code, scientific papers, regulatory documents and multilingual conversations. They use the same tokenizer, transformer, optimiser, learning-rate schedule, number of parameters and total token budget. They even publish the same source catalogue. Only one line differs. Team North samples in proportion to the raw number of tokens. Team South caps the largest source, raises the probability of smaller technical sources and rejects near duplicates before sampling.

Both teams can truthfully say that they trained on the same named datasets. They did not train on the same sequence of learning pressure. North repeatedly asks its model to reduce the errors common in the largest web source. South spends more updates on code, specialised language and rare formats. If a troubleshooting page occurs ten thousand times, North may treat it as ten thousand votes unless deduplication or sampling logic says otherwise. If benchmark answers appear in the reservoir, either model may look more capable than it is.

The catalogue says what could be sampled; the mixture says what the optimiser was made to care about. This is the core answer. Architecture defines a space of possible procedures. The mixture steers which procedures become useful under the training objective. A data mixture is therefore not housekeeping around a model. It is one of the model's causal inputs.

The argument is narrower than the broad claim that “better data makes better models”. Quality filters, source choice and scale matter, but they do not tell us how the selected material is converted into updates. It is also narrower than the adjacent question explored in What Scaling Laws Don't Tell You. That article asks what smooth loss curves omit. Here the object of study is the sampler itself, including the hidden weights introduced by duplication and evaluation overlap.

The sampler writes the objective

Start with a simpler training run than a language model. A learner repeatedly sees an example, predicts an answer and receives a loss. The optimiser changes parameters so that future losses tend to fall. The examples are not presented by an impartial universe. A sampler chooses them. If sixty per cent of updates come from source A, then source A contributes roughly sixty per cent of the expected pressure, subject to example length, loss scaling, batching and other implementation details.

For sources D₁ to D_k, let w_s be the probability of choosing source s, and let ℓ(θ; x) be the loss of parameters θ on example x. A static mixture optimises the expected objective:

The same statement appears in recent data-mixture work, although methods differ in how they estimate useful weights. DoReMi uses a small proxy model and distributionally robust optimisation; RegMix fits regressions over many small mixture experiments; Data Mixing Laws and later scaling-law work attempt to predict target losses as a function of mixture and scale.[1][2][3] Their shared premise is modest but consequential: source proportions are variables, not clerical metadata.

Different weights change the expected gradient:

A one-parameter worked example

Consider a model with one parameter, θ, that predicts ŷ = θx. Source A contains the relationship y = x. Source B contains y = −x. Everything else is equal. Under squared loss, an A-heavy 80:20 mixture has the optimum θ = 0.6. Reversing only the sampling weights produces θ = −0.6. The parameter changes sign even though the model class, examples, optimiser and total update count are unchanged.

The mixture becomes behaviour through repeated error correction. This toy system is deliberately small. It does not imitate the internal dynamics of a transformer. Its value is causal isolation: one variable changes, the optimum moves, and the reason can be written on one line. The executable artefact later in the article reproduces the result.

flowchart TD
  A["The reservoir that trained two<br/>different models"] --> B["The sampler writes the objective"]
  B --> C["Four ways the mixture moves"]
  C --> D["From toy slope to a training run"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#B96D00,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Four ways the mixture moves

“Mixture” often means a row of source percentages. That row is necessary and insufficient. The effective probability seen during training can change through explicit source weights, sample multiplicity, time-varying curricula, filters, token-length effects and retries. Evaluation overlap then changes what observed performance can support. Provenance determines whether any of these choices can be reconstructed.

1. Sampling: declared probability is an intervention

The simplest intervention changes the probability of drawing a source. The Pile made this visible by combining 22 constituent datasets and deliberately upweighting selected components rather than sampling only in proportion to raw size.[4] Later work treats mixture search as an optimisation problem, but the practical question arrives first: which target behaviours deserve scarce updates?

A static vector can still hide a schedule. A source may be emphasised early, tapered later or activated only after another competence emerges. Let w_s(t) be source s's probability at training step t. Two runs can have the same average proportion yet present examples in a different order. Non-convex optimisation means path can matter. A complete mixture specification records the schedule, not only its final average.

Mixture search is also target-dependent. A blend that improves code generation may reduce performance in another language or domain. DCLM, FineWeb and Dolma are valuable partly because they expose controlled curation choices, source composition and evaluation regimes rather than presenting a single opaque web corpus.[5][6][7] Their evidence does not imply that one filtering policy transfers to every model size, objective or downstream use.

Thought experiment: the language that is present but scarcely sampled

Build a bilingual reservoir with 900 million tokens in Language A and 100 million in Language B. Both teams keep every token. Both train for 100 million sampled tokens. Team Proportional samples according to raw volume, so the expected exposure is 90 million A tokens and 10 million B tokens. Team Balanced chooses each language with probability one half, producing 50 million sampled tokens from each. Nothing about source membership, architecture or training budget changes. Only the probability rule changes.

The balanced run now asks the optimiser to repair Language B errors five times as often. It also reuses Language B more heavily because the unique pool is smaller. A plausible result is better B performance, some loss in A performance, and greater memorisation risk within B. Another plausible result is little change because shared structure transfers across languages or because tokenisation and model capacity, rather than exposure, are the binding constraints. The experiment is useful precisely because either result discriminates among mechanisms.

Now add a held-out set that is equally balanced across languages. Team Balanced may win the average benchmark even if the intended traffic is 99% Language A. Reverse the test distribution and Team Proportional may look better. A mixture is inseparable from the target distribution used to judge it. “Balanced” is not an intrinsic virtue. It is a decision about whose errors, tasks and risks receive weight.

A serious test therefore reports at least three distributions: the available reservoir, the realised training draws and the intended evaluation or use population. Confusing them creates false arguments. Raw availability does not dictate sampling; sampling does not dictate deployment; and a balanced benchmark does not automatically represent either.

The unit of sampling is another hidden choice

Percentages are ambiguous until the sampling unit is named. Suppose one source contains ten long manuals and another contains ten thousand short messages. Uniform sampling over documents gives each manual the same chance as one message, but a selected manual may contribute far more tokens. Uniform sampling over fixed-length chunks gives a long manual many more tickets in the draw. Uniform sampling over tokens creates yet another distribution. Packing, truncation and sequence-length caps can then change which parts of a selected document produce loss.

These implementations can all display “50% source A, 50% source B” in a configuration file while yielding different document exposure, token exposure and gradient contribution. Loss masking complicates the picture further: prompt tokens may be present in a batch but excluded from the supervised loss. The auditable quantity is the realised contribution to optimisation, not the label on the sampler.

For this reason, a mixture audit reconciles at several levels. It counts selected documents, retained sequences, unmasked tokens and loss-bearing tokens. It reports deviations from configured weights and explains whether the deviation arose from exhausted shards, sequence packing, filtering, worker retries or deliberate curriculum logic. This is less elegant than a pie chart and much closer to the causal process.

2. Deduplication: multiplicity is a hidden weight

Now hold the unique documents constant. In one copy of the reservoir, a troubleshooting page occurs once. In another, a crawler has preserved ten thousand mirrored copies. No mixture table lists a new source. Yet the repeated page has gained probability mass.

This is the first thought experiment with one causal change. Architecture, unique semantic content and total update count remain fixed. Only the number of copies changes. If the repeated page carries a rare format, phrase or answer, its influence can become disproportionate. Lee and colleagues found extensive near duplication in language-model corpora and reported that deduplication reduced memorisation and train-test overlap in their experiments. Kandpal and colleagues connected repeated sequences to extractability and showed that deduplication can mitigate privacy risk.[8][9]

Deduplication is not a universal quality switch. Muennighoff and colleagues found that repeated data can remain useful for several epochs under data constraints, with diminishing returns rather than an immediate cliff. In one of their reported downstream settings, deduplication did not improve benchmark performance, although it may still reduce memorisation.[10] The operational lesson is not “always remove every repetition”. It is to measure multiplicity, choose a policy and expose the resulting effective exposure.

3. Contamination: the score may answer the wrong question

Training contamination is often described as another data-quality defect. Causally, it is a special reweighting: evaluation content or a close transformation receives training probability. Epistemically, it is more serious. The benchmark no longer cleanly tests generalisation beyond the observed items.

Suppose a fraction c of evaluation items has been exposed, and let S_exposed and S_clean denote average scores for the two partitions. The observed mean is:

An n-gram detector can find literal overlap and still miss semantic duplicates. EleutherAI's evaluation harness includes a decontamination procedure based on matching 13-grams, following the form of analysis reported for GPT-3.[11] This is useful evidence, not an oracle. Absence of exact overlap is not evidence of absence of exposure. A release claim should state the detector, normalisation, source snapshot, unit of comparison and residual blind spots.

4. Provenance: not a gradient until it changes a decision

Provenance is often grouped with data-mixture mechanisms, but a distinction matters. A source identifier, licence record or collection note does not directly change gradients. The same bytes with richer documentation train the same mathematical objective. Provenance becomes causal when it controls inclusion, exclusion, weighting, remediation or release. It remains epistemically essential even when it does not alter training, because it determines which claims can be checked.

The Data Provenance Initiative audited more than 1,800 widely used text datasets and found widespread missing or inaccurate licence information.[12] Datasheets for Datasets and Data Statements for NLP proposed structured documentation of motivation, composition, collection and population assumptions before foundation-model training made the scale problem acute.[13][14] ROOTS applied participatory governance and extensive source documentation to the multilingual corpus used for BLOOM.[15]

Provenance is the control surface between a mixture and its accountability. Without a stable source and transformation lineage, a team cannot reliably remove a withdrawn source, recompute overlap, explain why a domain was upweighted or reproduce the blend. A percentage labelled “web 55%” is not a source specification. It is an umbrella over snapshots, filters, countries, languages, hosts and collection times.

From toy slope to a training run

The scalar model proves possibility, not importance in a large transformer. To make a production decision, the mixture needs three additional views: probability mass, unique exposure and target evidence. A source can have a modest weight and still be repeated many times if its unique token pool is small.

Let T be the total number of sampled training tokens, w_s the source probability and U_s the estimated unique tokens retained from source s. The expected source exposure is:

A production-shaped worked scenario

Consider a synthetic 100-billion-token run for a document-intelligence model. Its intended tasks include broad language understanding, code-assisted extraction, regulatory document interpretation and multilingual service correspondence. The team has five retained source groups. The figures below are constructed to expose the audit, not to recommend a real blend.

The weight column alone makes regulated documents look smaller than curated web. Effective exposure reveals the opposite repetition pressure: the specialist source receives an expected 2.5 passes over its estimated unique tokens. The small synthetic source receives three. This does not prove harm. It identifies where memorisation, template lock-in and diminishing returns deserve measurement.

Separate a sampler experiment from a corpus experiment

Many reported “mixture” changes also alter the corpus. A quality filter removes documents, a deduplicator changes multiplicity, or a language classifier reassigns examples before new source weights are applied. The resulting model may improve, but the causal variable is a bundle. That can be operationally useful and scientifically ambiguous.

To isolate sampling, freeze source snapshots and transformations, then vary only the draw schedule. To isolate deduplication, preserve the sampling rule and compare a raw multiset with a documented unique set. To evaluate a complete curation pipeline, change the bundle deliberately and avoid attributing the result to one component. In all three cases, keep the token budget and optimisation settings matched, and inspect whether early stopping causes one run to receive fewer effective updates.

This distinction matters for remediation. If the gain came from removing repeated boilerplate, changing weights later may not reproduce it. If it came from upweighting a rare source, rebuilding the filter stack may be unnecessary. An experiment should identify the intervention that a future team can actually repeat.

A sensible next experiment does not immediately ask for a “better” global mixture. It chooses target slices and matched controls. For example, compare extraction from current regulatory documents, superseded documents, ordinary web tables, low-resource languages and generated forms. Hold architecture, token budget and optimiser fixed. Vary one source weight or deduplication rule. Report target gains, regressions, confidence intervals and overlap. Repeat at more than one scale if the release depends on proxy transfer.

Where mixture engineering stops helping

The strongest counterexample uses the same toy model but removes source disagreement. Source A contains y = x; source B also contains y = x, perhaps at different values of x. Both have the same optimum, θ = 1. Changing 80:20 to 20:80 leaves the optimum unchanged. The executable audit verifies this negative control.

This boundary generalises. If sources are redundant for the target, their expected gradients align and the model has enough capacity and time to reach a shared solution, reweighting may affect convergence or variance without materially changing final behaviour. If a quality filter merely removes examples whose signal is already represented elsewhere, gains may be negligible. If the optimisation bottleneck lies in architecture, context length or objective, mixture work can become elaborate motion around the wrong constraint.

Mixture sensitivity rises when sources make different demands and falls when they are interchangeable for the target. That statement is a useful organising hypothesis, not a proven universal law. High-dimensional gradients change throughout training; local disagreement may not predict final capability; and representational transfer can make distant sources unexpectedly complementary.

There are also scale-transfer limits. Data Mixing Laws reports that source rankings can change with model size or token budget.[3] A separate study of LLM-estimated data utility found that simple token-count heuristics outperformed several manual and learned mixture baselines in its evaluated settings.[16] Recent preprints jointly model source mixtures, repetition and data constraints, but their equations remain empirical claims tied to observed regimes, not guarantees outside them.[17]

Make the mixture a release artefact

Model releases commonly foreground architecture, parameter count, context length and evaluation scores. A defensible release also needs to preserve the probability programme that produced the weights. The appropriate artefact is not a glossy source pie chart. It is a versioned mixture receipt that connects source identity, transformations, effective exposure, contamination evidence and target ablations.

A mixture release receipt should be immutable for a given checkpoint. A later correction creates a superseding receipt; it does not silently rewrite the history of the released weights. The receipt belongs in training lineage beside code, tokenizer, optimiser configuration, random seeds, evaluation manifests and the resulting checkpoint.

The minimum audit sequence

1. Reconstruct the actual draw distribution. Compare configured weights with observed tokens and documents by source, language, host, time period and transformation path.
2. Measure unique exposure. Report raw rows, exact and near-unique units, unique-token estimates, effective epochs and the sensitivity of those estimates to the deduplication threshold.
3. Separate capability targets. Replace one average score with target slices, regressions, calibration, memorisation probes and freshness tests linked to the intended use.
4. Challenge transfer. Include proportional and token-count baselines. Repeat decisive mixture choices at more than one scale or mark transfer as unresolved.
5. Gate the lineage. Missing source identity, permission, overlap evidence or removal capability blocks the affected release claim and may block the checkpoint.

Configured weights are not operating evidence

A distributed training system can violate its own mixture configuration without raising a clean error. One shard may exhaust early. A worker retry may replay batches. A source-specific parser may reject more records than expected. A queue may favour faster-producing sources, while a curriculum transition happens at different steps after resumption. The intended mixture remains in configuration; the realised mixture moves.

The receipt therefore needs observed counters from the training path. Reconcile source selections, accepted sequences, unmasked tokens and loss-bearing tokens over time. Set tolerances before training, not after seeing the result. A deviation can be harmless, material or revealing: a 0.3 percentage-point difference in a large web source may be immaterial, while the same absolute difference can substantially change exposure for a tiny specialist source.

Checkpointing also needs mixture state. Resuming from model and optimiser weights while resetting sampler position or curriculum phase can duplicate one region and omit another. The run may be numerically valid but no longer match its declared schedule. Reproducibility requires replaying the data-selection state, not merely the parameter state.

Run the mixture audit

The companion Python programme turns the article's mechanism into five tests. It fits the one-parameter positive case, runs the aligned-source negative control, shows how exact duplicates change a fitted slope, detects exact benchmark overlap and flags missing provenance. It also computes effective epochs for a synthetic 100-billion-token mixture.

from dataclasses import dataclass

@dataclass(frozen=True)
class Example:
    x: float
    y: float
    text: str

def fit_scalar(sources):
    numerator = denominator = 0.0
    for weight, examples in sources:
        per_example = weight / len(examples)
        for e in examples:
            numerator += per_example * e.x * e.y
            denominator += per_example * e.x * e.x
    return numerator / denominator

positive = [Example(1.0, 1.0, "positive relation")]
negative = [Example(1.0, -1.0, "negative relation")]

assert fit_scalar([(0.8, positive), (0.2, negative)]) == 0.6
assert fit_scalar([(0.2, positive), (0.8, negative)]) == -0.6

# Negative control: both sources imply theta = 1.
aligned_a = [Example(1.0, 1.0, "aligned A")]
aligned_b = [Example(2.0, 2.0, "aligned B")]
assert fit_scalar([(0.8, aligned_a), (0.2, aligned_b)]) == 1.0
assert fit_scalar([(0.2, aligned_a), (0.8, aligned_b)]) == 1.0

#!/usr/bin/env python3
"""Synthetic audit showing that data-mixture weights change a learned model.

Python 3.10+, standard library only. The examples are deliberately tiny so the
causal mechanism is inspectable rather than hidden inside a large training run.
"""

from __future__ import annotations

from dataclasses import asdict, dataclass
import hashlib
import json
from typing import Iterable, Sequence


@dataclass(frozen=True)
class Example:
    x: float
    y: float
    text: str


@dataclass(frozen=True)
class Source:
    name: str
    weight: float
    unique_tokens_b: float
    licence: str | None
    snapshot: str | None
    permitted_use: str | None


def fingerprint(text: str) -> str:
    """Normalise a string and return a stable exact-match fingerprint."""
    normalised = " ".join(text.casefold().split())
    return hashlib.sha256(normalised.encode("utf-8")).hexdigest()


def deduplicate(examples: Iterable[Example]) -> list[Example]:
    """Keep the first example for each normalised text fingerprint."""
    seen: set[str] = set()
    kept: list[Example] = []
    for example in examples:
        key = fingerprint(example.text)
        if key not in seen:
            seen.add(key)
            kept.append(example)
    return kept


def fit_scalar(sources: Sequence[tuple[float, Sequence[Example]]]) -> float:
    """Fit y_hat = theta*x under weighted mean-squared error.

    Each source weight is probability mass. Within a source, examples are
    sampled uniformly. The closed-form optimum is used so the result is exact.
    """
    numerator = 0.0
    denominator = 0.0
    for weight, examples in sources:
        if weight < 0 or not examples:
            raise ValueError("Weights must be non-negative and sources non-empty")
        per_example = weight / len(examples)
        for example in examples:
            numerator += per_example * example.x * example.y
            denominator += per_example * example.x * example.x
    if denominator == 0:
        raise ValueError("Cannot fit a slope when every x is zero")
    return numerator / denominator


def contamination_rate(train: Sequence[Example], benchmark: Sequence[str]) -> float:
    """Return exact normalised-string overlap as a fraction of benchmark items."""
    if not benchmark:
        raise ValueError("Benchmark must not be empty")
    train_keys = {fingerprint(example.text) for example in train}
    exposed = sum(fingerprint(item) in train_keys for item in benchmark)
    return exposed / len(benchmark)


def effective_epochs(total_training_tokens_b: float, source: Source) -> float:
    """Expected source-token exposures divided by unique source tokens."""
    if source.unique_tokens_b <= 0:
        raise ValueError("unique_tokens_b must be positive")
    return total_training_tokens_b * source.weight / source.unique_tokens_b


def provenance_gaps(source: Source) -> list[str]:
    required = {
        "licence": source.licence,
        "snapshot": source.snapshot,
        "permitted_use": source.permitted_use,
    }
    return [field for field, value in required.items() if not value]


def run_audit() -> dict[str, object]:
    # Positive case: the sources require opposite slopes. Only mixture changes.
    positive = [Example(1.0, 1.0, "positive relation")]
    negative = [Example(1.0, -1.0, "negative relation")]
    theta_80_20 = fit_scalar([(0.8, positive), (0.2, negative)])
    theta_20_80 = fit_scalar([(0.2, positive), (0.8, negative)])

    # Negative control: both sources imply the same optimum, so weights do not matter.
    aligned_a = [Example(1.0, 1.0, "aligned source a")]
    aligned_b = [Example(2.0, 2.0, "aligned source b")]
    theta_aligned_80_20 = fit_scalar([(0.8, aligned_a), (0.2, aligned_b)])
    theta_aligned_20_80 = fit_scalar([(0.2, aligned_a), (0.8, aligned_b)])

    # Duplicate case: repeated text silently becomes extra probability mass.
    repeated = [Example(1.0, 1.0, "same troubleshooting page") for _ in range(8)]
    repeated.append(Example(1.0, -1.0, "contradictory troubleshooting page"))
    theta_raw = fit_scalar([(1.0, repeated)])
    theta_dedup = fit_scalar([(1.0, deduplicate(repeated))])

    # Exact contamination test. Semantic paraphrases remain outside this detector.
    train = repeated + [Example(1.0, 1.0, "capital of Freedonia is Belltown")]
    benchmark = [
        "Capital of Freedonia is Belltown",
        "same troubleshooting page",
        "Which port serves Freedonia?",
        "Unseen arithmetic item",
    ]
    exact_overlap = contamination_rate(train, benchmark)

    sources = [
        Source("curated_web", 0.55, 90.0, "documented", "2026-Q2", "pretraining"),
        Source("code", 0.15, 12.0, "documented", "2026-Q2", "pretraining"),
        Source("regulated_docs", 0.20, 8.0, "documented", "2026-Q1", "pretraining"),
        Source("unknown_archive", 0.10, 5.0, None, "2024-Q4", None),
    ]
    total_training_tokens_b = 100.0
    source_rows = []
    for source in sources:
        source_rows.append(
            {
                **asdict(source),
                "effective_epochs": round(effective_epochs(total_training_tokens_b, source), 3),
                "provenance_gaps": provenance_gaps(source),
            }
        )

    assertions = {
        "mixture_flips_learned_slope": theta_80_20 > 0 and theta_20_80 < 0,
        "aligned_control_is_invariant": abs(theta_aligned_80_20 - theta_aligned_20_80) < 1e-12,
        "dedup_changes_hidden_weight": abs(theta_raw - theta_dedup) > 0.5,
        "exact_overlap_detected": abs(exact_overlap - 0.5) < 1e-12,
        "provenance_gate_flags_archive": provenance_gaps(sources[-1]) == ["licence", "permitted_use"],
    }
    if not all(assertions.values()):
        raise AssertionError(assertions)

    return {
        "assumptions": [
            "Sources are sampled by declared probability mass.",
            "Examples are uniform within each source.",
            "The toy model is y_hat = theta*x with squared loss.",
            "Contamination detection covers exact normalised text only.",
        ],
        "positive_case": {
            "theta_at_80_20": round(theta_80_20, 3),
            "theta_at_20_80": round(theta_20_80, 3),
        },
        "negative_control": {
            "theta_at_80_20": round(theta_aligned_80_20, 3),
            "theta_at_20_80": round(theta_aligned_20_80, 3),
        },
        "duplicate_case": {
            "raw_theta": round(theta_raw, 3),
            "deduplicated_theta": round(theta_dedup, 3),
        },
        "contamination": {
            "exact_overlap_fraction": round(exact_overlap, 3),
            "detector_boundary": "Exact normalised-string overlap; semantic overlap is not measured.",
        },
        "sources": source_rows,
        "assertions": assertions,
    }


if __name__ == "__main__":
    print(json.dumps(run_audit(), indent=2, sort_keys=True))

{
  "assertions": {
    "aligned_control_is_invariant": true,
    "dedup_changes_hidden_weight": true,
    "exact_overlap_detected": true,
    "mixture_flips_learned_slope": true,
    "provenance_gate_flags_archive": true
  },
  "assumptions": [
    "Sources are sampled by declared probability mass.",
    "Examples are uniform within each source.",
    "The toy model is y_hat = theta*x with squared loss.",
    "Contamination detection covers exact normalised text only."
  ],
  "contamination": {
    "detector_boundary": "Exact normalised-string overlap; semantic overlap is not measured.",
    "exact_overlap_fraction": 0.5
  },
  "duplicate_case": {
    "deduplicated_theta": 0.0,
    "raw_theta": 0.778
  },
  "negative_control": {
    "theta_at_20_80": 1.0,
    "theta_at_80_20": 1.0
  },
  "positive_case": {
    "theta_at_20_80": -0.6,
    "theta_at_80_20": 0.6
  },
  "sources": [
    {
      "effective_epochs": 0.611,
      "licence": "documented",
      "name": "curated_web",
      "permitted_use": "pretraining",
      "provenance_gaps": [],
      "snapshot": "2026-Q2",
      "unique_tokens_b": 90.0,
      "weight": 0.55
    },
    {
      "effective_epochs": 1.25,
      "licence": "documented",
      "name": "code",
      "permitted_use": "pretraining",
      "provenance_gaps": [],
      "snapshot": "2026-Q2",
      "unique_tokens_b": 12.0,
      "weight": 0.15
    },
    {
      "effective_epochs": 2.5,
      "licence": "documented",
      "name": "regulated_docs",
      "permitted_use": "pretraining",
      "provenance_gaps": [],
      "snapshot": "2026-Q1",
      "unique_tokens_b": 8.0,
      "weight": 0.2
    },
    {
      "effective_epochs": 2.0,
      "licence": null,
      "name": "unknown_archive",
      "permitted_use": null,
      "provenance_gaps": [
        "licence",
        "permitted_use"
      ],
      "snapshot": "2024-Q4",
      "unique_tokens_b": 5.0,
      "weight": 0.1
    }
  ]
}

The programme is an instrument, not a training recipe. Its exact-overlap detector deliberately underclaims semantic contamination. Its provenance gate treats missing documentation as a release blocker, but does not decide legal rights. Its scalar model proves that probability can be causal, not that a particular source weight will transfer to a transformer.

What to report after a mixture experiment

A mixture experiment earns a decision only when it reports the intervention and its boundaries. State which source probabilities changed, whether the schedule changed over time, how many unique tokens were available, which filters ran, how overlap was measured and which target slices improved or regressed. Include uncertainty over repeated seeds or matched runs. Record compute, because a mixture that converges faster can be valuable even when the final optimum is similar.

Do not promote a proxy result directly into a universal rule. DoReMi and RegMix show credible ways to use small experiments; Data Mixing Laws, BiMix and later scaling approaches seek predictive structure.[18] Their practical contribution is a testable proposal: spend smaller runs to reduce the search space. The release still needs evidence that the selected blend survives the actual scale, objective and evaluation boundary.

stateDiagram-v2
  [*] --> Claim
  state "A one-parameter worked example" as Claim
  state "1. Sampling: declared<br/>probability is an intervention" as Test
  state "Thought experiment: the<br/>language that is present but<br/>scarcely sampled" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Compact glossary

Source mixture

The probability distribution, often time-varying, used to select among source distributions during training.

Multiplicity

The retained number of occurrences of an example or near-equivalent group. Under row-uniform sampling, multiplicity is hidden probability mass.

Effective epochs

Expected sampled source tokens divided by estimated unique source tokens. It exposes repetition pressure but not uniform document coverage.

Contamination

Training exposure to evaluation items, answers or sufficiently close derivatives that weakens the intended generalisation claim.

Provenance

Traceable identity and transformation history for a source. It becomes causally active when it controls inclusion, weight, removal or release.

Mixture receipt

A versioned record binding source snapshots, sampling schedule, deduplication, exposure, overlap, permission and target evidence to a checkpoint.

Sources and evidence boundary

The causal derivation and executable examples are self-contained. Empirical claims below are bounded to the cited studies and their reported model sizes, corpora, objectives and evaluations. Dataset documentation sources support provenance practice, not a claim that documentation alone improves model behaviour.

1. Xie et al., “DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining”. Primary research on proxy-based mixture optimisation.
2. Liu et al., “RegMix: Data Mixture as Regression for Language Model Pre-training”. Primary research using many small mixture runs and regression.
3. Ye et al., “Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance”. Primary research on mixture-dependent loss prediction and scale limits.
4. Gao et al., “The Pile: An 800GB Dataset of Diverse Text for Language Modeling”. Primary dataset paper documenting 22 components and deliberate weighting.
5. Li et al., “DataComp-LM: In Search of the Next Generation of Language Model Pretraining Datasets”. Primary controlled testbed for curation and evaluation.
6. Penedo et al., “The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale”. Primary corpus and ablation report.
7. Soldaini et al., “Dolma: An Open Corpus of Three Trillion Tokens for Language Model Pretraining Research”. Primary open-corpus and curation-tooling report.
8. Lee et al., “Deduplicating Training Data Makes Language Models Better”. Primary research on duplication, memorisation and overlap.
9. Kandpal et al., “Deduplicating Training Data Mitigates Privacy Risks in Language Models”. Primary research linking repetition and extractability.
10. Muennighoff et al., “Scaling Data-Constrained Language Models”. Primary research on repeated data, diminishing returns and data constraints.
11. EleutherAI, “Decontamination” in the Language Model Evaluation Harness. Official implementation documentation for literal-overlap screening.
12. Longpre et al., “The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing and Attribution in AI”. Primary audit of dataset lineage and licensing documentation.
13. Gebru et al., “Datasheets for Datasets”. Authoritative dataset-documentation framework.
14. Bender and Friedman, “Data Statements for Natural Language Processing”. Primary proposal for documenting populations and generalisation assumptions.
15. Laurençon et al., “The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset”. Primary corpus and governance report.
16. “Optimizing Pretraining Data Mixtures with LLM-Estimated Data Utility”. Primary negative and comparative evidence for learned mixture selection in its tested settings.
17. “Scaling Laws for Mixture Pretraining Under Data Constraints”. Recent primary preprint jointly considering mixture and repeated exposure; conclusions remain provisional.
18. “BiMix: Scaling Laws for Data Mixtures”. Primary research relating domain proportions, data volume and loss.

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: The reservoir that trained two different models
  M->>E: A one-parameter worked example
  E-->>M: compare rival explanations
  E->>D: Version the river, not only the reservoir
  D-->>Q: revise scope or act

Version the river, not only the reservoir

A source catalogue can be complete and still fail to explain a model. It describes membership. Training is governed by exposure: which source is selected, which example is repeated, when it appears and how often its error contributes to an update. Deduplication changes that exposure by removing hidden votes. Contamination changes both learning pressure and the meaning of an evaluation. Provenance makes the intervention reconstructable and gives controls something stable to act upon.

The argument does not make mixture the sole explanation for capability. Architecture, objective, optimisation, scale, tokenisation and evaluation all remain causal. Nor does it license an elaborate optimiser by default. When sources are aligned, simple proportional or token-count rules may be sufficient. When proxy rankings reverse or target evidence is weak, the correct output is uncertainty, not a confidently tuned percentage.

The changed architecture decision is to treat the sampler and mixture receipt as versioned model components. A checkpoint should not pass release merely because its weights are reproducible. The team must be able to reconstruct source snapshots, transformations, probability schedules, unique exposure, overlap tests, permissions and target ablations. Missing lineage is not an editorial omission. It is a limit on what the model's behaviour can be explained, repaired or legitimately claimed to demonstrate.

The next training run therefore begins before the first token is sampled. It begins with a falsifiable target, a proportional baseline, a declared intervention and a receipt designed to survive the run. The model is the learned parameters plus the causal history that made those parameters likely. Version that history with the same seriousness as the weights.