Part IThe pair is the task

A supervised classifier receives an explicit target such as “cat”, “pneumonia” or “late payment”. Contrastive learning often looks less prescriptive because it can work without class labels. Yet it still receives a target. The target is encoded in the pairing rule. Two crops from one photograph, two adjacent segments of speech, an image and its caption, or a document and its translation are declared related. Other items in the comparison set are treated as alternatives.

That declaration is a small programme of invariance. A crop pair says that some translation, occlusion and scale change should preserve what matters. An audio-transcript pair says that voice, microphone response and background noise should not block access to the linguistic content. An image-caption pair says that the visual scene should be recoverable through the concepts selected by the caption. None of these statements is universal. A speaker-identification task needs voice; an acoustic-fault detector may need background vibration; an image-forensics task may need the exact crop boundary.

A graph before a geometry

It helps to picture the training data as a graph before picturing an embedding space. Each observation is a node. A positive edge joins two views that should agree. A batch supplies competing nodes that should not win the same match. Training then constructs a geometry that makes those graph relations easy to recover.

Suppose repeated positive edges connect an object across camera angles, crops and captions. In an idealised limit, those edges generate an equivalence relation: observations connected by the permitted transformations belong to one class. The learned map compresses each class into a neighbourhood while keeping rival classes distinguishable. Mathematicians would call the resulting object a quotient of the original observation space. The useful intuition is that the representation operates on “world observations modulo the changes declared irrelevant”.

The analogy has limits. Real positive relations can be noisy and non-transitive. A model rarely maps every positive view to exactly one point. Finite capacity, optimisation, projection heads and auxiliary losses can leave view-specific detail intact. Still, the quotient picture exposes the causal burden: every edge added to the positive graph is a vote for invariance, and every missing distinction must be recovered from some other signal.

Part IIWhat the loss actually asks

Consider a batch of N paired observations. An image encoder maps item x_i to a normalised vector z_i. A text encoder maps its paired caption y_i to a normalised vector w_i. Their dot product is cosine similarity. A common symmetric contrastive objective is:

The first term asks each image to identify its paired text among the batch. The second asks each text to identify its paired image. The temperature τ controls how sharply similarity differences affect the probabilities. A smaller temperature concentrates pressure on the nearest competitors. A larger one spreads credit and blame across more of the batch. Changing τ changes optimisation geometry, but it does not tell the system whether colour, count or causal order deserves preservation.

The gradient has two visible jobs. It pulls the positive score z_i·w_i upward and pushes competing scores downward in proportion to their current softmax probabilities. Wang and Isola described related asymptotic properties as alignment of positive pairs and uniformity of the representation distribution on the hypersphere.^[3] This language is useful because collapse avoidance and pair agreement are separate pressures. A representation that maps every input to one point has perfect agreement but cannot identify a partner.

Contrastive predictive coding introduced a probabilistic contrastive objective for extracting information useful to predict future observations.^[1] InfoNCE is therefore often discussed as a mutual-information estimator or lower bound. That connection is informative, but it should not become a metaphysics of representation. Variational mutual-information bounds can develop severe bias or variance in high-dimensional settings, and a high estimate does not identify which semantic factor has been preserved.^[5] Mutual information is a lens on dependence, not a list of the distinctions a decision will require.

The operational interpretation is simpler. The loss rewards any feature combination that reliably identifies partners. If object category alone separates most items, category may dominate. If every batch contains many instances of the same category but different attributes, the system must use finer distinctions. If captions repeatedly omit a factor, the cross-modal loss offers little reason for the text interface to expose it. The loss optimises discriminability under the sampled comparisons, not semantic completeness.

flowchart TD
  A["The traffic light that became<br/>colourless"] --> B["Part I The pair is the task"]
  B --> C["Part II What the loss actually<br/>asks"]
  C --> D["Part III The shared signal is<br/>a bargain"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#5B5CF6,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Part IIIThe shared signal is a bargain

Multimodal contrastive learning adds a productive asymmetry. An image can contain colour, texture, count, spatial layout, camera artefacts and objects. A caption may name only the objects and one relation. The caption contains syntax, emphasis and cultural knowledge that pixels do not directly carry. The pair overlaps on a latent set of shared causes, but each view also contains private information.

Let S denote information available in both views, U_x information private to the image, and U_y information private to the text. Cross-modal matching can be solved most reliably through S. This makes language a remarkably scalable source of supervision: words gather visually diverse instances around named concepts. CLIP demonstrated broad zero-shot transfer after training a dual encoder on hundreds of millions of image-text pairs, while ALIGN showed similar scaling with over a billion noisier alt-text pairs.^[6][7]

Yet S is not “everything true about the scene”. It is the information that is both present and useful under the pairing distribution. A caption “three red cups beside a plate” can supervise count, colour and relation. A caption “kitchenware” cannot. Even a richly worded caption may be ambiguous about which phrase refers to which region. Language supervision is selective labelling at scale, not lossless translation from the world.

Four meanings of “the feature is gone”

Teams often say that a representation has “lost” a feature when four different observations are being mixed together. First, the source may never have measured the factor. A caption cannot recover an exact temperature that neither pixels nor text contain. Second, the backbone may encode the factor while the projection removes it from the shared interface. Third, the projected vector may retain a weakly decodable trace that the downstream head does not use. Fourth, the factor may remain available within one modality but lack a cross-modal direction, so a text query cannot address it.

These cases imply different remedies. Missing measurement requires another sensor or source. Projection-level suppression may require a wider or multi-headed interface. A readout failure requires task supervision, calibration or a different decision head. Missing cross-modal addressability requires paired examples, grounded attributes or a tool that retrieves the original evidence. Re-training the whole model is wasteful when the required fact already exists upstream and dangerous when the fact never existed anywhere.

A useful diagnostic records four states for each factor. Is it present in the input? Is it decodable at the backbone? Is it exposed by the shared embedding? Does an intervention on the factor change the deployed decision? The states need not move together. Colour can be highly decodable in an image backbone, weakly exposed after projection and irrelevant to an object-retrieval head. Relation direction can appear in token-level features yet vanish after pooling. Source identity may be deliberately absent from semantic similarity but remain mandatory in a separate evidence receipt.

This is why a single linear probe is insufficient. A successful probe shows that some decoder can recover a statistical signal from a chosen layer; it does not show that the operational head uses the signal. A failed probe may reflect probe capacity, data scarcity or a nonlinear encoding rather than true absence. Pair probes with controlled interventions and inspect the exact interface consumed by the application. “Preserved” should mean available to the required decision under a specified access path, not merely visible somewhere inside the network.

For systems that need both broad alignment and specialised detail, one vector need not carry both burdens. A practical representation can contain a shared component for cross-modal discovery, a modality-private component for within-modal work, and typed metadata for provenance, time and measured values. Auxiliary reconstruction, within-modal contrast or supervised heads can support private factors, but each extra objective changes the contract and needs its own controls. The design goal is not maximal retention. It is deliberate retention with a known retrieval path.

A minimal worked example

Create synthetic images with four shapes and four colours. Pair each image with text that names the shape but never the colour. The cross-modal task can be solved by aligning “triangle” with every triangular image. Colour varies freely within those positive groups and contributes nothing to text matching. A zero-shot colour prompt such as “blue” has no reliable cross-modal bridge, even though colour is plainly present in the image vector.

Now mention colour in one caption out of ten. That small intervention changes the learning signal. The system receives some examples in which “blue” must align with blue images across shapes. Colour need not dominate every match, but it acquires a cross-modal direction. Mention it in every caption and the direction strengthens further. Shuffle the image-text pairs and both shape and colour alignment should approach chance. This is the executable experiment later in the article.

Shared space does not require one geometric cloud

Image and text embeddings can remain separable by modality even when paired retrieval works. Research on the modality gap found that independently encoded modalities may occupy different regions of the nominally shared space, influenced by initialisation and contrastive optimisation.^[17] That finding should not be converted into a rule that zero gap is always desirable. A geometry may preserve modality-specific uncertainty or support the trained matching function without pointwise identity. The relevant test is whether the geometry supports the required comparisons without systematic bias, not whether image and text distributions look visually merged.

There is also a strong negative control against simplistic erasure claims. Locked-image text tuning freezes a pretrained image encoder and learns a text interface to read from its representation.^[8] The cross-modal objective cannot overwrite the frozen visual features. Likewise, a trainable projection head may absorb much of the invariance while an earlier backbone layer retains private detail. Pairing creates selection pressure; architecture and capacity determine where that pressure is expressed.

Part IVNegatives carve the boundary

Positive pairs say what may remain close. Negatives say what must be distinguishable in the current comparison. This second half of the objective is easy to underestimate because “all other items in the batch” sounds neutral. It is not neutral. A batch is a sampled boundary around every example.

If an image of a dog chasing a person is contrasted mostly with captions about buses, kitchens and oceans, the words “dog”, “person” and “chasing” are enough. A bag-of-words representation can win. Replace one easy negative with “a person chasing a dog”. The nouns no longer identify the positive. Word order and relation direction become decision-relevant. Nothing about the architecture changed. The sampled alternative changed what the model had to know.

Benchmarks have exposed this gap. ARO was designed around attributes, relations and word order, showing that strong vision-language systems could perform poorly when success required composition rather than unordered concept overlap.^[15] Winoground used two images and two captions containing the same words in different orders, so a correct match required relational understanding.^[16] Such tests do not prove that one benchmark fully isolates composition, but they demonstrate why ordinary retrieval accuracy can reward a weaker solution.

The reverse problem is a false negative. Two different photographs may both depict the same disease pattern, product type or intent. Instance-level contrastive training treats them as competitors merely because they are not the designated pair. Debiased contrastive learning analysed this mismatch and proposed correcting for negatives that may share the latent class.^[10] In specialised domains, false negatives are especially common because many cases legitimately share semantics while only one report is paired to one record.

Hard-negative mining therefore needs a truth test. Select examples that are close under the current model but genuinely different under the required decision. A colour-swapped caption is useful when colour must be retained. It is harmful when the image is monochrome or the colour word is metaphorical. A relation-swapped sentence is useful when the scene visually establishes the relation. It is noise when the image is ambiguous. Hardness is an optimisation property; validity is an evidence property.

Batch size and temperature affect how many competitors matter, but they do not solve semantic sampling. SigLIP replaced the globally normalised softmax with pairwise sigmoid losses and showed that strong image-text learning need not depend on the same giant-batch regime.^[9] This is an important systems alternative. It still requires a defensible account of which pairs are positive and which are negative.

Part VThe invariance contract

Contrastive projects often begin with model family, batch size and compute budget. The causal decision comes earlier: what transformations are allowed to preserve identity, and what distinctions must survive? Write that answer as an invariance contract before training. It is a compact agreement between the pair generator, the representation interface and the downstream decision.

The table is not a universal taxonomy. It forces five questions to meet in one place: the latent factor, the transformation applied to positives, the later decision, the evidence that preserves a distinction, and the test that can reject the design. A useful contract contains explicit contradictions. Lighting should be invariant for object identity, but signal colour should not. Crop should be invariant for texture classification, but not when it changes count.

1. Define the consumed decision. Name the action, retrieval or classification that will use the representation. “General embedding” is not a decision. Record the harm of merging two states and the cost of keeping them separate.
2. Inventory latent factors. Enumerate attributes, relations, quantities, source conditions and temporal variables that could vary. Mark each as must-merge, must-separate, conditionally relevant or currently unknown.
3. Interrogate every positive generator. For each augmentation, pairing rule or temporal window, ask which factors can change across the pair. Run transformation-specific probes rather than one average score.
4. Audit the negative population. Estimate false-negative rates by group, inspect nearest mined negatives, and add valid counterfactual negatives for distinctions that random batches rarely test.
5. Localise retention. Probe the backbone, projection, shared embedding and downstream head. Use controlled interventions as well as decoders. Compare frozen, fine-tuned and auxiliary-objective controls.
6. Release against a factor ledger. Report retrieval or zero-shot quality beside must-keep factor scores, subgroup uncertainty, out-of-distribution transformations and known unresolved distinctions. Withdraw the representation from a decision when its contract is violated.

Dataset curation belongs inside this contract. DataComp held the training procedure and compute budget comparatively fixed while varying selection from a large candidate pool, demonstrating how data design changes downstream outcomes.^[18] The lesson is broader than ranking filtering methods. The pair population is part of the model specification because it defines the empirical equivalence relation the optimiser sees.

stateDiagram-v2
  [*] --> Claim
  state "A graph before a geometry" as Claim
  state "Four meanings of “the feature<br/>is gone”" as Test
  state "A minimal worked example" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Run the contrastive invariance lab

The companion Python lab makes the paper's claim falsifiable on synthetic data. Each “image” vector contains a one-hot shape, a one-hot colour and noise. Each “text” vector always contains shape but contains colour with probability 0, 0.1 or 1. Two linear encoders produce six-dimensional normalised embeddings and train with the symmetric loss above.

After training, text prototypes test zero-shot shape and colour recognition from held-out image vectors. The positive case predicts that shape remains strong when every caption names it, while colour approaches chance when captions omit it and rises when even a minority of captions mention it. The negative case shuffles pairings. No meaningful cross-modal direction should survive.

python -m pip install torch
python contrastive_invariance_lab.py

The first condition is the minimal positive case: shape is consistently shared and reaches 99.9%, while colour remains 27.7%, close to the 25% chance level. Mentioning colour in only 10% of captions raises colour accuracy to 84.1% without hurting shape. Always mentioning colour supports both factors, although a six-dimensional embedding and small synthetic sample produce greater variance. Shuffling pairs collapses both scores towards chance.

This result does not establish that real multimodal models erase every unmentioned factor. The image encoder here is linear, trained from scratch and evaluated through text prototypes. Those assumptions deliberately make the cross-modal bottleneck visible. Change the capacity, pretraining, layer or auxiliary objective and the result can change. The lab's value is causal transparency: pair coverage is the only intended difference across the first three conditions.

"""Synthetic contrastive-invariance lab for paper 55.

Python 3.10+ with PyTorch 2.x. The data are synthetic teaching fixtures.
The positive case preserves shape across image and text. Colour is either
omitted, mentioned in some captions, or always mentioned. The negative case
shuffles image-text pairings. Results are averaged over three seeds.
"""

from __future__ import annotations

from dataclasses import dataclass
from statistics import mean, pstdev

import torch
import torch.nn.functional as F

N = 256
N_SHAPES = 4
N_COLOURS = 4
NOISE_DIMS = 4
EMBED_DIM = 6
STEPS = 300
TAU = 0.07
SEEDS = (0, 1, 2)


def make_data(n: int, colour_mention_probability: float, seed: int):
    """Create paired image/text vectors with shared shape and optional colour."""
    generator = torch.Generator().manual_seed(seed)
    shape = torch.randint(N_SHAPES, (n,), generator=generator)
    colour = torch.randint(N_COLOURS, (n,), generator=generator)

    image = torch.cat(
        [
            F.one_hot(shape, N_SHAPES).float(),
            F.one_hot(colour, N_COLOURS).float(),
            0.25 * torch.randn(n, NOISE_DIMS, generator=generator),
        ],
        dim=1,
    )

    mentions_colour = (
        torch.rand(n, generator=generator) < colour_mention_probability
    ).float().unsqueeze(1)
    text = torch.cat(
        [
            F.one_hot(shape, N_SHAPES).float(),
            mentions_colour * F.one_hot(colour, N_COLOURS).float(),
            0.25 * torch.randn(n, NOISE_DIMS, generator=generator),
        ],
        dim=1,
    )
    return image, text, shape, colour


class DualEncoder(torch.nn.Module):
    def __init__(self, input_dim: int, embedding_dim: int):
        super().__init__()
        self.image = torch.nn.Linear(input_dim, embedding_dim, bias=False)
        self.text = torch.nn.Linear(input_dim, embedding_dim, bias=False)

    def encode_image(self, x: torch.Tensor) -> torch.Tensor:
        return F.normalize(self.image(x), dim=1)

    def encode_text(self, y: torch.Tensor) -> torch.Tensor:
        return F.normalize(self.text(y), dim=1)


def train(
    colour_mention_probability: float,
    seed: int,
    shuffle_pairs: bool = False,
) -> DualEncoder:
    torch.manual_seed(seed)
    image, text, _, _ = make_data(N, colour_mention_probability, seed)
    if shuffle_pairs:
        permutation = torch.randperm(
            N, generator=torch.Generator().manual_seed(seed + 111)
        )
        text = text[permutation]

    model = DualEncoder(image.shape[1], EMBED_DIM)
    optimiser = torch.optim.AdamW(model.parameters(), lr=0.05, weight_decay=1e-3)
    labels = torch.arange(N)

    for _ in range(STEPS):
        z_image = model.encode_image(image)
        z_text = model.encode_text(text)
        logits = z_image @ z_text.T / TAU
        loss = 0.5 * (
            F.cross_entropy(logits, labels)
            + F.cross_entropy(logits.T, labels)
        )
        optimiser.zero_grad()
        loss.backward()
        optimiser.step()

    return model


def prototypes() -> tuple[torch.Tensor, torch.Tensor]:
    """Text prompts for zero-shot shape and colour classification."""
    shape_prompts = []
    for shape in range(N_SHAPES):
        shape_prompts.append(
            torch.cat(
                [
                    F.one_hot(torch.tensor(shape), N_SHAPES).float(),
                    torch.zeros(N_COLOURS + NOISE_DIMS),
                ]
            )
        )

    colour_prompts = []
    for colour in range(N_COLOURS):
        colour_prompts.append(
            torch.cat(
                [
                    torch.zeros(N_SHAPES),
                    F.one_hot(torch.tensor(colour), N_COLOURS).float(),
                    torch.zeros(NOISE_DIMS),
                ]
            )
        )
    return torch.stack(shape_prompts), torch.stack(colour_prompts)


def evaluate(
    model: DualEncoder,
    colour_mention_probability: float,
    seed: int,
) -> tuple[float, float]:
    image, _, shape, colour = make_data(
        N, colour_mention_probability, seed + 1000
    )
    shape_prompts, colour_prompts = prototypes()

    with torch.no_grad():
        z_image = model.encode_image(image)
        z_shape = model.encode_text(shape_prompts)
        z_colour = model.encode_text(colour_prompts)
        shape_prediction = (z_image @ z_shape.T).argmax(dim=1)
        colour_prediction = (z_image @ z_colour.T).argmax(dim=1)

    shape_accuracy = (shape_prediction == shape).float().mean().item()
    colour_accuracy = (colour_prediction == colour).float().mean().item()
    return shape_accuracy, colour_accuracy


@dataclass(frozen=True)
class Result:
    condition: str
    shape_mean: float
    shape_sd: float
    colour_mean: float
    colour_sd: float


def run_condition(
    name: str,
    colour_mention_probability: float,
    shuffle_pairs: bool = False,
) -> Result:
    scores = []
    for seed in SEEDS:
        model = train(colour_mention_probability, seed, shuffle_pairs)
        scores.append(evaluate(model, colour_mention_probability, seed))

    shape_scores = [score[0] for score in scores]
    colour_scores = [score[1] for score in scores]
    return Result(
        condition=name,
        shape_mean=mean(shape_scores),
        shape_sd=pstdev(shape_scores),
        colour_mean=mean(colour_scores),
        colour_sd=pstdev(colour_scores),
    )


def main() -> None:
    torch.set_num_threads(2)
    results = [
        run_condition("colour omitted", 0.0),
        run_condition("colour in 10% of captions", 0.10),
        run_condition("colour always mentioned", 1.0),
        run_condition("shuffled image-text pairs", 0.0, shuffle_pairs=True),
    ]

    print("condition                         shape accuracy      colour accuracy")
    for result in results:
        print(
            f"{result.condition:34} "
            f"{100 * result.shape_mean:5.1f}% ± {100 * result.shape_sd:4.1f}   "
            f"{100 * result.colour_mean:5.1f}% ± {100 * result.colour_sd:4.1f}"
        )

    # Positive and negative cases make the causal claim executable.
    assert results[0].shape_mean > 0.95
    assert results[0].colour_mean < 0.40
    assert results[1].colour_mean > 0.70
    assert results[3].shape_mean < 0.45


if __name__ == "__main__":
    main()

Design the equivalence before training it

The traffic-light example fails only when two decisions are confused. For category retrieval, colour invariance can be desirable. For signal control, it destroys the boundary that matters. The inspection example has the same structure. Aligning images, traces and notes around “bearing fault” improves discovery, but it does not authorise a maintenance action without location, severity, trend and source quality.

Contrastive learning is therefore best understood as controlled forgetting in service of discrimination. Positive pairs define the changes that should cease to matter. Negatives define the alternatives that must remain apart. Multimodal pairs privilege what their views share. Capacity, architecture and optimisation decide how completely those pressures reshape each layer.

The changed engineering decision is concrete. Before scaling a model, write the invariance contract. State what may merge, what must separate, where private evidence remains available, which negatives make the boundary real, and which intervention can reject the design. Then choose augmentations, pair sources, loss, temperature, batch and model size. A representation is defensible when its learned equivalences match the decision's permissible equivalences, not when one retrieval score is high.