Five apprentices read the same record

Place one record on a laboratory bench. It contains a photograph of a red crane standing beside a flooded river and the sentence, “The red crane waited beside the river bank after the storm.” Give five identical apprentices the same archive of records, the same parameter budget and the same number of updates. Change only the question for which each apprentice is penalised.

The first sees the sentence from left to right and must predict the next token. The second receives a sentence with bank hidden and may inspect words on both sides. The third receives a whole missing phrase and must reconstruct “beside the river bank”. The fourth sees a batch of photographs and captions and must select the correct pairing. The fifth sees the photograph and must generate the caption token by token.

All five consume the same factual material. Yet the first is repeatedly paid for continuation, the second for bidirectional disambiguation, the third for coherent reconstruction, the fourth for global cross-modal separation, and the fifth for image-conditioned description. The archive did not change. The error made expensive by training did.

This is the core answer. A pretraining objective causally biases learning by choosing the information visible at prediction time, the event that counts as a target, the alternatives compared and the weight assigned to each error. It does not uniquely specify the representation that must emerge. It changes the gradient field within which some regularities become cheaper to learn than others.

Part I

The loss draws a causal boundary

“Objective” is sometimes used as if it meant the name of a model family. It does not. A Transformer can be trained causally, bidirectionally, through denoising, through pair discrimination or through several losses at once. Architecture determines which computations are available. Data determines which regularities exist in the examples. The objective determines which prediction errors are scored during training.

The optimiser follows an estimate of the gradient of this expectation. Parameters that help reduce frequently weighted errors receive repeated reinforcement. Parameters that would support an unscored capability may still emerge because they also help a scored task, but the objective does not pay for them directly. This distinction prevents a common overclaim: next-token prediction does not explicitly reward truth, planning or harmlessness. It rewards probability assigned to observed next tokens. Those broader properties arise only insofar as they help that prediction under the supplied data and model.

A capability receives direct pretraining pressure only when it reduces a scored error under the evidence the model may inspect. Five dials make that pressure legible.

1. Visibility: past tokens, both sides of a gap, an image, another modality, or a corrupted prefix.
2. Target: one token, every next token, a span, a patch, a pair identity, or a whole output sequence.
3. Factorisation: which variables must be predicted first and which may condition later predictions.
4. Comparison set: the vocabulary, sampled negatives, generated corruptions, or alternative sequences.
5. Weighting: which records, tokens, modalities and losses dominate the gradient budget.

Corruption policy and target density sit inside these dials. Mask ten per cent rather than forty per cent and the problem changes. Select independent tokens rather than contiguous spans and the dependency length changes. In contrastive learning, increase the batch and each positive is compared with more negatives. In a mixed objective, change one coefficient and a formerly auxiliary task can become the main training incentive.

A matched experiment becomes discriminating only when the objectives are allowed to disagree. Construct two corpora with the same examples, token counts and marginal target frequencies. In the first, place the decisive cue to the right of the target. In the second, move that cue to the left without changing its reliability. A bidirectional objective can exploit it in both corpora; a causal objective can exploit it directly only after the move. The causal prediction is an interaction between objective and cue position, rather than a generic accuracy advantage.

If both models improve equally before the cue moves, some shared proxy is probably carrying the answer. If neither improves afterwards, capacity, optimisation or corruption may be blocking the intended route. This intervention tests the mechanism more sharply than comparing two final leaderboard scores.

Thought experiment: the future clue

Consider “The crane reached the bank after the storm.” At the moment a causal learner predicts bank, its visible prefix supports at least two readings: a financial institution and a river edge. Add “where floodwater covered the reeds” after the target. A masked learner filling the hidden word can inspect that future clue; the causal learner predicting the original position cannot.

Now vary one feature. In the first condition, the future words resolve the ambiguity. In the second, the future merely repeats information already present before the target. Bidirectional visibility should reduce uncertainty in the first condition and do nothing in the second. That is a discriminating causal prediction. “Bidirectionality is better” is not. The benefit depends on extra conditional information being present.

Part II

Four objective families, four incentive patterns

Next-token prediction: dense, ordered and deployment-aligned

Causal language modelling supplies a target at nearly every position and matches the interface used for autoregressive generation. During training, all next-token losses in a sequence can be computed in parallel. During inference, generated tokens are fed back one at a time. This train-use symmetry is a major engineering advantage, and large causal models have demonstrated broad few-shot behaviour after scale and diverse data. Brown et al.

The objective also imposes an order on stored evidence. A relation appearing as “A composed B” makes predicting B from A directly useful. Recovering A from B may never be scored at the relevant position. The Reversal Curse experiments showed this direction-sensitive failure in autoregressive models trained on fictitious relations, while also finding that models could reverse a relation supplied in the immediate context. Berglund et al.

That result is a failure signature, not a theorem that causal models can never learn symmetric knowledge. Reverse-form data, suitable augmentation or a changed training recipe can add the missing factorisation. A 2026 study, for example, reported partial mitigation through an “identity bridge” data recipe. Ma et al. The durable lesson is narrower: an autoregressive objective pays for the conditionals present in the observed order; logical symmetry is not free.

Masked-token prediction: bidirectional reconstruction with sparse targets

BERT established the practical power of deep bidirectional pretraining for downstream language understanding. Devlin et al. A masked learner can represent a whole input before recovering the missing item, which aligns naturally with classification, retrieval, tagging and extractive tasks that consume a complete sequence.

The price is target sparsity and corruption mismatch. If only a fraction of tokens are hidden, only that fraction receives direct vocabulary prediction loss on a given presentation. The special mask symbol may also be absent from downstream inputs. ELECTRA changed the scored event: it replaced tokens with plausible generator samples and trained a discriminator to identify replacements at every position. Under the paper's matched settings, replaced-token detection was more compute-efficient than masked reconstruction for encoder transfer. Clark et al.

Masking does not inherently produce a good left-to-right generator. Nor does bidirectional visibility guarantee causal understanding. It rewards reconstruction from correlations available around the gap. The model can succeed by exploiting local agreement, templates or near-duplicate phrasing unless evaluations remove those shortcuts.

Span corruption: make missing structure contiguous

Independent token masks ask many small questions. Span corruption removes contiguous material, marks each gap with a sentinel and asks a sequence model to generate the missing spans. The target is therefore both what content is absent and how its tokens cohere in order.

T5 compared several denoising designs under a common text-to-text framework. In its reported study, corrupting 15 per cent of tokens in spans with an average length of three produced marginal gains over independent corruption on most non-translation tasks and shortened sequences enough to improve training efficiency. The authors also reported that differences among nearby span settings were limited. Raffel et al.

That bounded result matters. Span corruption does not magically teach document planning. It makes multi-token reconstruction directly valuable and creates a natural encoder-decoder interface for transformation, infilling and conditional generation. The effect depends on span length, corruption rate, boundary cues, architecture and downstream adaptation.

Multimodal objectives: pair, describe or reconstruct

With image-text pairs, the phrase “multimodal pretraining” hides distinct scored events. CLIP maps images and texts into a shared space and uses a symmetric contrastive loss so matched pairs outrank mismatched pairs in a batch. This created transferable visual representations and useful zero-shot classification through natural-language labels. Radford et al.

A contrastive model can win by preserving the global information needed to distinguish pairs while discarding detail shared across alternatives. Its effective negatives matter. A caption that correctly describes several images may be treated as a false negative for all but one. Batch composition therefore changes the comparison set and the learned geometry.

Captioning asks a different question: given an image and previous text, predict every caption token. SimVLM showed that a single prefix language-modelling objective could support both generative and discriminative vision-language tasks. Wang et al. Yet captioning can exploit language regularity and produce a plausible generic sentence while neglecting fine visual evidence, particularly when captions themselves omit that evidence.

Hybrid systems make both errors expensive. CoCa combined contrastive alignment with autoregressive captioning in one model. Yu et al. A later matched comparison found captioning alone competitive with contrastive pretraining on classification and stronger on the tested vision-language tasks when data, compute and capacity were carefully aligned. Tschannen et al. The result weakens any claim that contrastive loss is intrinsically the superior route to visual representations.

Thought experiment: pairing versus describing

Take two photographs. Both show a red cup and a blue book. In the first, the cup is left of the book; in the second, it is right. Give both the caption “a cup and a book”. A contrastive learner cannot be rewarded for preserving left-right order because the text target does not distinguish it. A captioner cannot be rewarded either, because generating the generic caption never requires the relation.

Now change only the captions to “red cup left of blue book” and “red cup right of blue book”. The contrastive loss can use relational detail to separate the pair, while the captioning loss must generate the relation token. The objective difference matters, but the data had to expose the distinction first. No objective can recover a distinction that its visible evidence and targets never make consequential.

Part III

flowchart LR
  A["Five apprentices read the same<br/>record"] --> B["The loss draws a causal boundary"]
  B --> C["Four objective families, four<br/>incentive patterns"]
  C --> D["What controlled evidence<br/>establishes"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#7B4CD8,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

What controlled evidence establishes

The clean causal claim is not that one objective wins. It is that objective choice changes the training incentives, and those changes matter when evaluations require the affected conditional dependencies. Controlled studies repeatedly show interaction rather than a universal ranking.

This reverses the ranking after a change in adaptation regime. It also exposes a frequent confound: architecture and objective often travel together in public model comparisons. A causal decoder and a bidirectional encoder do not merely receive different losses; they expose different information paths and use different inference interfaces. The study's value lies in crossing these factors rather than comparing product names.

That result supports a schedule, not a metaphysical division between “generative” and “understanding” models. It also remains bounded to the architectures, corpora, scales, tasks and optimisation choices tested. A larger model, different tokenizer or different adaptation budget could move the frontier.

Within an objective family, differences can be small. T5 found limited separation among several nearby denoising variants, even though the selected span scheme offered a useful efficiency and performance trade. Across modalities, the matched captioning study challenged the assumption that contrastive loss is required for strong visual encoders. These are precisely the observations a causal account should expect: objective effects grow when scoring rules expose genuinely different information and shrink when several losses can exploit the same structure.

The strongest objection: data and architecture do the real work

A serious critic can argue that objectives are a secondary detail. Web-scale data contains syntax, facts, code, images and demonstrations; a sufficiently expressive architecture can compress those regularities under many self-supervised losses. Post-training then reorients behaviour more than pretraining does. Aggregate benchmarks may therefore show little objective-specific signature.

This objection is partly right. The objective is one cause in a coupled system, not an independent essence. Data decides which predictive signals exist. Architecture decides which can be represented and routed. Optimisation decides which basin is reached. Scale changes what can be fitted. Post-training changes the use interface and priorities. Any paper that changes all five and attributes the result to the objective has not isolated the mechanism.

The objection becomes too strong only when it claims that the scored event is irrelevant. A causal mask makes future tokens unavailable at a position. A contrastive loss does not directly score caption word order. A mask rate changes the number and difficulty of reconstruction events. These are interventions on the training signal. Their behavioural effect may be small in a particular regime, but the intervention itself is real and produces testable predictions.

The negative control and failure boundary

Return to the future-clue experiment. Let the target bit be predicted from a past cue that is correct 75 per cent of the time. In the positive condition, add an independent future cue that is correct 90 per cent of the time. The optimal conditional uncertainty falls when both are visible. In the negative control, make the future cue an exact copy of the past. Bidirectional visibility has no information gain.

The objective mechanism ceases to help when its additional visibility, target granularity or comparison set contributes no information relevant to the required decision. It can also fail when the model cannot represent the dependency, when optimisation never finds it, when corruption makes the task unlike use, when data offers an easier shortcut, or when post-training overwrites the distinction. An objective audit should therefore measure both the intended dependency and the shortcut that could make it unnecessary.

Mixtures do not remove this boundary. UL2 combined denoisers intended for different regimes and reported a stronger cross-task frontier than its single-objective baselines. Tay et al. CoCa combined global alignment and captioning. In each case, the total objective was a weighted portfolio. The weights, sampling rates and gradient interactions became new design variables. “Use all losses” is not a decision rule.

Part IV

The matched-objective micro-lab

The executable artefact creates 36 synthetic facts. Each has the form BOS, A, relation, B, relation-clue, EOS. For every relation, a fixed permutation maps each source entity A to one target B. Three two-layer Transformers share the initial parameter draw, width, depth, optimiser family, corpus coverage and 350 updates. One receives causal next-token loss, one masked-token loss, and one prefix-conditioned span-reconstruction loss.

This is the minimal worked example. The causal model must complete B from A and the relation. The masked model must recover a left-hand entity while seeing the relation, target and clue on its right. The span model sees a missing two-token span and must generate the source entity and relation. A factorisation diagnostic asks the causal model for the source entity at its original sequence position. At that prediction, it is conditioned only on BOS; the relation, target and clue remain to the right and therefore cannot contribute.

The analytical control in the same script estimates conditional entropy from 100,000 synthetic samples. With an informative future cue, uncertainty falls from 0.810 to 0.394 bits, a 0.416-bit information gain. When the future copies the past, uncertainty remains 0.810 bits. The extra context exists physically but contributes nothing statistically.

Run the file with Python 3.10 or later and PyTorch 2.1 or later. The output is JSON and the assertions cover the positive cases and negative controls. The source uses no private data, network call or hidden checkpoint.

#!/usr/bin/env python3
"""Matched-objective micro-lab for 'Pretraining Objectives and What They Reward'.

The lab uses one synthetic relation corpus and one tiny Transformer backbone.
Three copies receive the same architecture, initial parameter draw, optimiser,
corpus coverage, and update count, but different self-supervised objectives:

1. causal next-token prediction;
2. bidirectional masked-token prediction; and
3. prefix-conditioned span reconstruction.

It also runs an analytical positive case and negative control for contextual
visibility. Results are teaching evidence from synthetic data, not claims about
large language models.

Requirements:
    Python 3.10+
    PyTorch 2.1+

Run:
    python pretraining_objective_matched_lab.py
    python pretraining_objective_matched_lab.py --steps 350 --seed 7
    python pretraining_objective_matched_lab.py --json-out results.json
"""

from __future__ import annotations

import argparse
import json
import math
import random
import warnings
from collections import defaultdict
from dataclasses import asdict, dataclass
from pathlib import Path

import numpy as np
import torch
from torch import Tensor, nn

warnings.filterwarnings("ignore", message="enable_nested_tensor")


@dataclass(frozen=True)
class LabConfig:
    seed: int = 7
    steps: int = 350
    entities: int = 12
    relations: int = 3
    width: int = 48
    heads: int = 4
    layers: int = 2
    learning_rate: float = 0.01
    entropy_samples: int = 100_000


@dataclass(frozen=True)
class AccuracyResult:
    probe: str
    objective: str
    correct: int
    total: int

    @property
    def accuracy(self) -> float:
        return self.correct / self.total


class Vocabulary:
    def __init__(self, entities: int, relations: int) -> None:
        special = ["<pad>", "<bos>", "<eos>", "<mask>", "<sep>"]
        entity_tokens = [f"E{i}" for i in range(entities)]
        relation_tokens = [f"R{i}" for i in range(relations)]
        clue_tokens = [f"C{i}" for i in range(relations)]
        self.tokens = special + entity_tokens + relation_tokens + clue_tokens
        self.stoi = {token: index for index, token in enumerate(self.tokens)}
        self.itos = {index: token for token, index in self.stoi.items()}
        self.pad, self.bos, self.eos, self.mask, self.sep = (
            self.stoi[token] for token in special
        )

    def __len__(self) -> int:
        return len(self.tokens)


class TinyTransformer(nn.Module):
    """One shared architecture used by all three objective conditions."""

    def __init__(self, vocab_size: int, config: LabConfig, max_length: int = 12) -> None:
        super().__init__()
        self.token_embedding = nn.Embedding(vocab_size, config.width)
        self.position_embedding = nn.Embedding(max_length, config.width)
        layer = nn.TransformerEncoderLayer(
            d_model=config.width,
            nhead=config.heads,
            dim_feedforward=4 * config.width,
            dropout=0.0,
            batch_first=True,
            norm_first=True,
        )
        self.transformer = nn.TransformerEncoder(layer, num_layers=config.layers)
        self.normalise = nn.LayerNorm(config.width)
        self.output = nn.Linear(config.width, vocab_size, bias=False)
        self.output.weight = self.token_embedding.weight

    def forward(self, tokens: Tensor, attention_mask: Tensor | None = None) -> Tensor:
        positions = torch.arange(tokens.shape[1], device=tokens.device)
        hidden = self.token_embedding(tokens) + self.position_embedding(positions)[None, :, :]
        hidden = self.transformer(hidden, mask=attention_mask)
        return self.output(self.normalise(hidden))


def causal_mask(length: int) -> Tensor:
    """True entries are blocked, so each position sees only itself and the past."""
    return torch.triu(torch.ones(length, length, dtype=torch.bool), diagonal=1)


def prefix_lm_mask(length: int, prefix_length: int) -> Tensor:
    """Bidirectional corrupted prefix plus a causal reconstruction suffix."""
    mask = torch.ones(length, length, dtype=torch.bool)
    for row in range(length):
        if row < prefix_length:
            mask[row, :prefix_length] = False
        else:
            mask[row, :prefix_length] = False
            mask[row, prefix_length : row + 1] = False
    return mask


def make_corpus(config: LabConfig, vocab: Vocabulary) -> Tensor:
    """Create 36 forward facts of form BOS, A, relation, B, relation-clue, EOS."""
    sequences: list[list[int]] = []
    for relation in range(config.relations):
        permutation = np.random.default_rng(config.seed + relation).permutation(config.entities)
        for source in range(config.entities):
            target = int(permutation[source])
            sequences.append(
                [
                    vocab.bos,
                    vocab.stoi[f"E{source}"],
                    vocab.stoi[f"R{relation}"],
                    vocab.stoi[f"E{target}"],
                    vocab.stoi[f"C{relation}"],
                    vocab.eos,
                ]
            )
    return torch.tensor(sequences, dtype=torch.long)


def new_model(config: LabConfig, vocab: Vocabulary) -> TinyTransformer:
    """Reset the seed so every condition starts from identical parameters."""
    torch.manual_seed(config.seed)
    return TinyTransformer(len(vocab), config)


def train_causal(
    corpus: Tensor, config: LabConfig, vocab: Vocabulary
) -> tuple[TinyTransformer, float]:
    model = new_model(config, vocab)
    optimiser = torch.optim.AdamW(
        model.parameters(), lr=config.learning_rate, weight_decay=0.0
    )
    inputs, targets = corpus[:, :-1], corpus[:, 1:]
    mask = causal_mask(inputs.shape[1])
    for _ in range(config.steps):
        optimiser.zero_grad(set_to_none=True)
        logits = model(inputs, mask)
        loss = nn.functional.cross_entropy(logits.flatten(0, 1), targets.flatten())
        loss.backward()
        optimiser.step()
    return model, float(loss.detach())


def train_masked(
    corpus: Tensor, config: LabConfig, vocab: Vocabulary
) -> tuple[TinyTransformer, float]:
    """Cover each of the four content positions once per update."""
    model = new_model(config, vocab)
    optimiser = torch.optim.AdamW(
        model.parameters(), lr=config.learning_rate, weight_decay=0.0
    )
    examples: list[list[int]] = []
    targets: list[int] = []
    positions: list[int] = []
    for sequence in corpus.tolist():
        for position in (1, 2, 3, 4):
            corrupted = sequence.copy()
            targets.append(corrupted[position])
            corrupted[position] = vocab.mask
            examples.append(corrupted)
            positions.append(position)
    inputs = torch.tensor(examples, dtype=torch.long)
    target_tensor = torch.tensor(targets, dtype=torch.long)
    position_tensor = torch.tensor(positions, dtype=torch.long)
    row_index = torch.arange(len(inputs))
    for _ in range(config.steps):
        optimiser.zero_grad(set_to_none=True)
        logits = model(inputs)
        selected = logits[row_index, position_tensor]
        loss = nn.functional.cross_entropy(selected, target_tensor)
        loss.backward()
        optimiser.step()
    return model, float(loss.detach())


def train_span(
    corpus: Tensor, config: LabConfig, vocab: Vocabulary
) -> tuple[TinyTransformer, float]:
    """Remove source entity plus relation as one span, then reconstruct it."""
    model = new_model(config, vocab)
    optimiser = torch.optim.AdamW(
        model.parameters(), lr=config.learning_rate, weight_decay=0.0
    )
    prefixes: list[list[int]] = []
    targets: list[list[int]] = []
    for sequence in corpus.tolist():
        prefixes.append(
            [vocab.bos, vocab.mask, sequence[3], sequence[4], vocab.eos, vocab.sep]
        )
        targets.append([sequence[1], sequence[2], vocab.eos])
    prefix_tensor = torch.tensor(prefixes, dtype=torch.long)
    target_tensor = torch.tensor(targets, dtype=torch.long)
    inputs = torch.cat([prefix_tensor, target_tensor[:, :-1]], dim=1)
    prefix_length = prefix_tensor.shape[1]
    prediction_positions = torch.tensor(
        [prefix_length - 1, prefix_length, prefix_length + 1], dtype=torch.long
    )
    mask = prefix_lm_mask(inputs.shape[1], prefix_length)
    for _ in range(config.steps):
        optimiser.zero_grad(set_to_none=True)
        logits = model(inputs, mask)
        selected = logits[:, prediction_positions, :]
        loss = nn.functional.cross_entropy(
            selected.flatten(0, 1), target_tensor.flatten()
        )
        loss.backward()
        optimiser.step()
    return model, float(loss.detach())


@torch.no_grad()
def evaluate(
    causal: TinyTransformer,
    masked: TinyTransformer,
    span: TinyTransformer,
    corpus: Tensor,
    vocab: Vocabulary,
) -> list[AccuracyResult]:
    total = len(corpus)
    results: list[AccuracyResult] = []

    # Native causal probe: predict target B after BOS, A, relation.
    prefix = corpus[:, :3]
    prediction = causal(prefix, causal_mask(prefix.shape[1]))[:, -1].argmax(-1)
    results.append(
        AccuracyResult(
            "forward completion B from A and relation",
            "causal next-token",
            int((prediction == corpus[:, 3]).sum()),
            total,
        )
    )

    # Native masked probes: recover a left entity and the relation with both sides visible.
    reverse_cloze = corpus.clone()
    reverse_cloze[:, 1] = vocab.mask
    prediction = masked(reverse_cloze)[:, 1].argmax(-1)
    results.append(
        AccuracyResult(
            "reverse cloze A from relation, B and clue",
            "masked token",
            int((prediction == corpus[:, 1]).sum()),
            total,
        )
    )

    relation_cloze = corpus.clone()
    relation_cloze[:, 2] = vocab.mask
    prediction = masked(relation_cloze)[:, 2].argmax(-1)
    results.append(
        AccuracyResult(
            "relation cloze from A, B and clue",
            "masked token",
            int((prediction == corpus[:, 2]).sum()),
            total,
        )
    )

    # Native span probe: autoregressively reconstruct source entity and relation.
    prefix = torch.stack(
        [
            torch.tensor(
                [vocab.bos, vocab.mask, int(row[3]), int(row[4]), vocab.eos, vocab.sep]
            )
            for row in corpus
        ]
    )
    generated: list[Tensor] = []
    current = prefix.clone()
    prefix_length = prefix.shape[1]
    for _ in range(3):
        mask = prefix_lm_mask(current.shape[1], prefix_length)
        next_token = span(current, mask)[:, -1].argmax(-1)
        generated.append(next_token)
        current = torch.cat([current, next_token[:, None]], dim=1)
    generated_tensor = torch.stack(generated, dim=1)
    expected = torch.stack([corpus[:, 1], corpus[:, 2], corpus[:, 5]], dim=1)
    exact = (generated_tensor == expected).all(dim=1)
    results.append(
        AccuracyResult(
            "two-token span plus EOS exact reconstruction",
            "span reconstruction",
            int(exact.sum()),
            total,
        )
    )

    # Native factorisation diagnostic: source entity A is scored after BOS only.
    # Relation, target B and clue lie to the right and cannot affect this prediction.
    original_prefix = corpus[:, :1]
    prediction = causal(
        original_prefix, causal_mask(original_prefix.shape[1])
    )[:, -1].argmax(-1)
    results.append(
        AccuracyResult(
            "source entity at original causal position",
            "causal next-token diagnostic",
            int((prediction == corpus[:, 1]).sum()),
            total,
        )
    )
    return results


def conditional_entropy_bits(target: np.ndarray, *features: np.ndarray) -> float:
    groups: dict[tuple[int, ...], list[int]] = defaultdict(list)
    for index, value in enumerate(target):
        key = tuple(int(feature[index]) for feature in features)
        groups[key].append(int(value))
    entropy = 0.0
    for values in groups.values():
        probabilities = np.bincount(values, minlength=2) / len(values)
        local = -sum(float(p) * math.log2(float(p)) for p in probabilities if p > 0)
        entropy += len(values) / len(target) * local
    return entropy


def visibility_controls(config: LabConfig) -> dict[str, dict[str, float]]:
    """Positive case: future adds information. Negative control: future repeats past."""
    rng = np.random.default_rng(config.seed)
    target = rng.integers(0, 2, config.entropy_samples)
    past = np.where(rng.random(config.entropy_samples) < 0.75, target, 1 - target)

    informative_future = np.where(
        rng.random(config.entropy_samples) < 0.90, target, 1 - target
    )
    redundant_future = past.copy()

    past_only = conditional_entropy_bits(target, past)
    informative_both = conditional_entropy_bits(target, past, informative_future)
    redundant_both = conditional_entropy_bits(target, past, redundant_future)
    return {
        "positive_case": {
            "H_target_given_past_bits": past_only,
            "H_target_given_past_and_future_bits": informative_both,
            "right_context_information_gain_bits": past_only - informative_both,
        },
        "negative_control": {
            "H_target_given_past_bits": past_only,
            "H_target_given_past_and_redundant_future_bits": redundant_both,
            "right_context_information_gain_bits": past_only - redundant_both,
        },
    }


def run(config: LabConfig) -> dict[str, object]:
    random.seed(config.seed)
    np.random.seed(config.seed)
    torch.manual_seed(config.seed)
    torch.set_num_threads(1)

    vocab = Vocabulary(config.entities, config.relations)
    corpus = make_corpus(config, vocab)
    causal, causal_loss = train_causal(corpus, config, vocab)
    masked, masked_loss = train_masked(corpus, config, vocab)
    span, span_loss = train_span(corpus, config, vocab)
    accuracy_results = evaluate(causal, masked, span, corpus, vocab)
    visibility = visibility_controls(config)

    # Positive cases and negative controls must remain discriminating.
    native = {item.probe: item.accuracy for item in accuracy_results}
    assert native["forward completion B from A and relation"] >= 0.95
    assert native["reverse cloze A from relation, B and clue"] >= 0.95
    assert native["two-token span plus EOS exact reconstruction"] >= 0.95
    assert native["source entity at original causal position"] <= 0.20
    assert visibility["positive_case"]["right_context_information_gain_bits"] > 0.30
    assert abs(
        visibility["negative_control"]["right_context_information_gain_bits"]
    ) < 0.01

    return {
        "scope": {
            "corpus": "synthetic relation facts",
            "facts": len(corpus),
            "backbone": "2-layer, width-48 Transformer",
            "updates_per_condition": config.steps,
            "matching_boundary": (
                "architecture, initial parameters, corpus, optimiser family and update count are matched; "
                "sequence length and scored-token density intentionally differ by objective"
            ),
        },
        "config": asdict(config),
        "final_training_loss": {
            "causal_next_token": causal_loss,
            "masked_token": masked_loss,
            "span_reconstruction": span_loss,
        },
        "accuracy_probes": [
            {
                **asdict(item),
                "accuracy": item.accuracy,
            }
            for item in accuracy_results
        ],
        "visibility_controls": visibility,
        "interpretation": (
            "The objective changes which conditional dependency can reduce loss. "
            "The entropy negative control shows that extra visibility has no value "
            "when it contributes no new information."
        ),
    }


def parse_args() -> tuple[LabConfig, str | None]:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--steps", type=int, default=350)
    parser.add_argument("--seed", type=int, default=7)
    parser.add_argument(
        "--json-out",
        type=str,
        default=None,
        help="optional path for the same JSON payload printed to standard output",
    )
    args = parser.parse_args()
    if args.steps < 50:
        parser.error("--steps must be at least 50 for the assertions to be meaningful")
    return LabConfig(seed=args.seed, steps=args.steps), args.json_out


if __name__ == "__main__":
    config, json_out = parse_args()
    payload = json.dumps(run(config), indent=2)
    if json_out is not None:
        Path(json_out).write_text(payload + "\n", encoding="utf-8")
    print(payload)

Part V

Choose an objective by the error you can afford

An objective decision should begin with a conditional dependency, not a fashionable model family. Ask what the deployed system must infer, which evidence it will possess at that moment, and which wrong answer must become expensive during pretraining. Then design a matched experiment that lets the proposed objective and its strongest simpler rival disagree.

A single causal objective is attractive for the summary generator because its pretraining interface matches free-form decoding. It is a weak default for the document encoder if the decisive tests require bidirectional sentence and passage representations under a tight adaptation budget. Masked reconstruction or replaced-token detection deserves a matched baseline there. Span corruption becomes relevant when missing clauses, table fragments and structured extraction targets must be reconstructed as coherent sequences.

For form images, contrastive alignment is a natural baseline for text-to-region retrieval, but it should be tested on relation-sensitive negatives rather than only broad document categories. Captioning or masked multimodal reconstruction should be added only if token-level visual detail improves the actual retrieval, extraction or generation task. If captions omit check-box state, handwriting location or spatial relation, a captioning loss cannot reward those features merely because they are visible in the image.

The programme may end with three pretrained components, a shared model with several heads, or a staged schedule. The correct choice depends on transfer and operating economics. A shared model reduces duplicated compute and may transfer useful structure. It can also create gradient conflict, force compromises in tokenisation and batch composition, and make release evidence harder to interpret.

The release comparison should preserve four views of the same choice. A frozen probe tests what pretraining made readily accessible. A matched adaptation curve tests how much labelled data and compute are needed to reach the target. A task-level evaluation tests the actual retrieval, generation or extraction decision. An ablation removes the image, future context or relational cue that supposedly causes the gain. An objective-specific advantage that disappears after a cheap, stable adaptation may not justify a more complex pretraining estate. A smaller gain that survives the ablation, lowers adaptation demand and reduces a costly failure can justify it. This keeps the decision tied to marginal operating value rather than to the pretraining loss with the most flattering scale.

The objective audit card

1. Name the decision and native interface. Is the model generating left to right, encoding a complete record, filling a gap, ranking pairs or reconstructing across modalities?
2. Write the required conditional. State it as “predict target T from visible evidence V”. If the notation is unclear, the objective is not yet specified.
3. Record the incentive vector. Measure target density, corruption distribution, factorisation, negative set, sample weights and loss coefficients.
4. Construct a shortcut control. Remove the distinctive future cue, shuffle relation words, replace informative images, or create generic captions that preserve the superficial answer.
5. Match the rival. Hold corpus, tokenizer, backbone capacity, optimiser, update budget and adaptation data as steady as the objectives permit. Report unavoidable mismatches.
6. Probe both native and transferred behaviour. Include generation, representation, infilling, alignment, calibration, data efficiency and the exact downstream outcome that changes the decision.
7. Adopt a stop rule. Keep the simpler objective unless the alternative improves a preregistered measure enough to justify added compute, complexity and release burden.

When to mix or schedule objectives

For a mixed loss, L = Σ λj Lj, the coefficients λj are part of the objective. Raw losses have different scales and target counts, so equal numeric weights do not imply equal training influence. Track per-loss gradient norms, conflicts and downstream marginal value. A loss that improves its own validation metric while degrading the product decision is not complementary.

A schedule is justified when one objective builds a useful starting representation more efficiently and another aligns the model with the final interface. The causal-then-masked result for encoders is one published example, bounded to its evaluation. The reverse schedule may be appropriate for a model that must end as a generator. The decision must be earned with adaptation curves rather than assumed from labels such as “understanding” and “generation”.

stateDiagram-v2
  [*] --> Claim
  state "Thought experiment, the future<br/>clue" as Claim
  state "Next-token prediction: dense,<br/>ordered and deployment-aligned" as Test
  state "Masked-token prediction:<br/>bidirectional reconstruction<br/>with sparse targets" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Compact glossary

Conditional visibility

The evidence a model may inspect when a particular target is scored.

Target density

The proportion of positions or relations that contribute direct loss in a presentation.

Factorisation

The ordered decomposition of a joint distribution into conditional predictions.

Comparison set

The alternatives against which a target must win, such as vocabulary items or batch negatives.

Corruption policy

The procedure that hides, replaces or rearranges evidence before prediction.

Train-use match

The similarity between the pretraining prediction interface and the deployed inference interface.

Train for the error that must become expensive

Pretraining is often described as learning from raw data without labels. The description is incomplete. Self-supervision manufactures labels by choosing what to hide, what to predict, what to compare and what to count. That choice is a causal intervention on learning.

Next-token prediction makes ordered continuation expensive to get wrong. Masking makes reconstruction from both sides expensive. Span corruption makes coherent missing sequences expensive. Contrastive learning makes incorrect global pairing expensive. Captioning makes image-conditioned word choices expensive. None directly prices truth, causal validity, safety or business outcome unless the training record and target construction connect those properties to loss.

The architecture decision therefore changes. Do not ask which objective is generally best. Ask which conditional dependency the system must use, whether the training target exposes it, which shortcut can lower the same loss, and whether a matched rival reaches the required outcome more cheaply. The defensible objective is the smallest scored problem that reliably creates the capability, reveals its boundary and survives the negative control.