The transformation contract

A camera does not receive objects. It receives measurements from a sensor array, a distinction central to computational accounts that separate image measurements from useful scene descriptions.[1] Those measurements depend on the scene, the camera, lighting, optics, exposure, noise and the accident of viewpoint. Let a scene state be s, nuisance conditions be n, and the rendering or sensing process be R. The image is:

The phrase “learn a good representation” hides three different requirements. First, if only a nuisance changes, the task-relevant part of z should stay stable. Second, if the scene undergoes a spatial transformation, geometric outputs should transform in the corresponding way. Third, if a consequence-bearing scene variable changes, the representation must not remain effectively unchanged.

The word nuisance is always relative to the query. Rotating a photograph may be irrelevant when identifying a circular washer, yet decisive when reading an arrow, estimating a grasp angle or distinguishing a six from a nine. Cropping can preserve an image-level class while deleting the one obstacle that makes a route unsafe. An augmentation policy therefore encodes a claim about which changes the task may ignore. When that claim is wrong, training rewards the model for erasing useful evidence.

A practical scene state can be written as four typed views: c for category or identity, g for geometry, r for relations and u for uncertainty. Under a harmless lighting change, c, g and r should remain stable while u may widen. Under an image translation, c should remain stable, image-frame geometry g should translate, and world-frame geometry may remain stable if camera pose is known. If an occluder enters the scene, the visible mask and relation “clear path” should change, and uncertainty about hidden shape should increase.

The common mathematical distinction is between invariance and equivariance. Let g name a declared geometric change and Tg its action on an image or spatial output. A category output c may satisfy c(Tgx) = c(x). A mask m should instead satisfy m(Tgx) ≈ Tgm(x). The same input movement that should disappear from one output must be preserved predictably in another. Group-equivariant networks make this principle architectural for declared transformation groups, rather than relying only on examples to teach it.[9]

This gives a stronger criterion than visual similarity. Let Q be the set of questions or decisions the visual state must support. A representation is approximately sufficient when the best decision rule using it incurs almost no more risk than a rule allowed to inspect the task-relevant scene state:

Scene structure is therefore relative to a query family, not a claim that the model has reconstructed the world in full. A category label can be sufficient for search indexing. A box can be sufficient for coarse counting. A mask may be necessary for area, boundary or contact. Depth and pose may be necessary for motion. The correct representation is the smallest one that preserves the distinctions the action can punish.

This criterion also exposes an asymmetry. Rich internal features may be compressed into a label, but a label cannot reconstruct the discarded geometry. Once a system publishes only valve = 0.94, no downstream component can recover which valve, where it lies, which pixels were visible or whether the handle was blocked. Output compression is an architectural commitment because lost scene variables cannot be repaired later by a more capable decision model.

What each vision task preserves

Computer-vision tasks are often presented as a menu of algorithms. A more durable view treats them as contracts about the shape of the answer. The output type determines which scene distinctions remain available, which metrics are meaningful and which downstream errors become impossible to repair.

Consider one minimal image containing two identical metal washers. The left washer has a hairline crack; the right washer is intact. An image-level classifier can answer “a cracked washer is present”. A detector can separate and localise the affected instance with a box. An instance mask can estimate the visible crack-bearing surface. A pose or depth estimate can place that surface in a tool frame. None of these answers is universally superior. Each is sufficient for a different query, and each makes different errors observable.

The sequence is therefore a ladder of retained distinctions, not a ladder of intelligence. Adding a richer output can reveal an error that a coarser task cannot express. A classifier cannot be penalised for swapping two instances because it does not emit identities. A box detector can be penalised for a missed instance but may still receive a strong score when its boundary cuts through the contact region required by a gripper. The output contract decides both what the system can know and what the evaluator can notice.

Classification: what is present, with location legally discarded

Image classification maps an image to one or more category scores. Its great strength is compression. The ImageNet challenge made large-scale category classification and detection measurable, and AlexNet showed how a deep convolutional network trained at scale could sharply improve that classification benchmark.[3][2] Yet a correct label does not imply that the model found the object, distinguished two instances or represented their relation.

Global pooling is useful precisely because it can remove where information. That is a feature when the query is “does this image contain corrosion?” It is a defect when the query is “which bolt is corroded, how much of its surface is affected, and can a tool reach it?” Classification is not primitive vision. It is a deliberately narrow scene interface.

Detection: preserve a set of instances and coarse extent

Object detection adds a set of labelled regions, usually boxes. Faster R-CNN made region proposals part of the learned network, sharing image features between proposal and detection stages.[4] DETR later formulated detection as direct set prediction, using bipartite matching to connect predicted object queries with target objects.[8] The mechanism changed, but the output contract remained recognisable: identify instances, assign categories and preserve approximate location.

A box supports counting, coarse overlap, tracking seeds and region-level review. It does not identify exact boundaries. Two objects can share a box but differ in usable area, contact points or occlusion. A detector can also produce duplicate or missing instances, so evaluation must test set errors, not only per-box scores.

Segmentation: preserve boundaries, but specify whose boundary

Fully convolutional networks replaced a single image-level answer with dense, per-pixel prediction.[5] Semantic segmentation labels pixels by class, so every road pixel may be “road” without distinguishing one road instance from another. Instance segmentation separates individual objects. Mask R-CNN added a mask branch alongside box and class prediction, making that distinction explicit.[6] Panoptic segmentation combines instance-aware “things” with amorphous “stuff”, aiming to assign every pixel a semantic label and, where appropriate, an instance identity.[7]

These are not merely different visualisations of the same answer. A semantic road mask supports drivable-area estimation but cannot identify two adjacent people. Instance masks support separation but can leave background classes underspecified. Panoptic output gives fuller 2D coverage, yet still says little about hidden surfaces, physical support or metric depth.

flowchart TD
  A["The valve that must stay the<br/>same and move"] --> B["The transformation contract"]
  B --> C["What each vision task preserves"]
  C --> D["Structure beyond masks"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#B96D00,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Structure beyond masks

A mask answers which visible pixels belong together. Many actions need more. A manipulator needs pose and reachable surfaces. Navigation needs free space, depth and motion. Inspection may need part hierarchy. Interaction may depend on support, contact, containment or occlusion. A temporal system must preserve identity across frames rather than rediscovering unrelated masks each time.

Geometry is meaningful only with a frame and a time. A pixel coordinate belongs to an image. A depth value may belong to the camera frame. A grasp pose must usually be transformed into a robot or world frame through calibrated camera intrinsics and extrinsics. If any transform is missing, stale or applied in the wrong direction, numerically precise outputs can describe the wrong place. The same applies through time: a stable track identifier is useful only while the object-state estimate remains current enough for the action.

Occlusion introduces another distinction. A segmentation mask normally describes the visible support of an object, not its complete physical extent. Completing the hidden shape is an inference based on learned regularity, multiple views or a geometric prior. The scene state should keep visible, inferred and measured geometry separate. Otherwise a plausible completion can be consumed as though the camera had observed it.

Scene graphs made one such extension explicit by representing objects and their relationships as a grounded graph.[12] The attraction is clear: “person”, “bicycle” and “road” become more useful when connected by “riding” and “on”. The risk is equally clear. A fluent relation label can be wrong, and a graph can harden an uncertain visual guess into apparently crisp symbolic state. Relations need confidence, provenance and an unresolved value where evidence is insufficient.

Current foundation models blur the old task boundaries without abolishing the contract. DINOv3 reports dense, transferable visual features across many tasks.[13] SAM 3 combines concept-prompted detection, segmentation and tracking, returning identities and masks for matching instances.[14] VGGT reports joint prediction of camera parameters, depth maps, point maps and tracks from one or more views.[15] These are important integrations. They do not make every output interchangeable.

These systems change where the boundaries are implemented. A reusable backbone can provide dense features to several heads; a concept prompt can select the instances whose masks matter; a geometry model can jointly estimate variables previously assembled by separate pipelines. The consumer still needs to know whether a value is image-relative or metric, whether a mask refers to every matching instance or one prompted instance, whether identity survives across frames, and under which shifts the estimate was tested. Integration reduces hand-offs, but it can also concentrate correlated error behind one convenient interface.

A universal backbone can reduce the cost of extracting several scene views, but the consuming system still needs a typed visual contract. A dense feature tensor is not yet an authorised grasp pose. A promptable mask is not metric depth. A 3D point map is not proof of material, load-bearing capacity or current occupancy. The final state should name the objects, coordinate frames, units, uncertainty, valid time and evidence required by the decision.

When pixels underdetermine the world

Vision is an inverse problem. Many scenes can produce similar images, and some produce exactly the same image from one viewpoint. This limits what any model can infer, regardless of scale.

Thought experiment: the painted doorway

Place a camera in a corridor. In World A it faces an open doorway. In World B it faces a high-resolution painting of that doorway positioned so the camera receives the same pixels. Hold the image fixed. The worlds disagree about traversability, depth and what will happen if the robot moves forward. No image-only function can be correct in both worlds because the observation contains no discriminator.

A second view, motion parallax, depth sensing or active probing can break the tie. A prior may make one world more probable, but probability is not observation. The system should preserve the distinction between estimated from appearance, triangulated from views, measured by a sensor and verified through action. Otherwise a model's confidence can silently become a world-state fact.

Formally, the observation identifies a set of compatible scenes rather than one guaranteed scene. Let S(x) contain every world state that could have produced image x within the sensor model. Learning can rank members of S(x) using prior experience, but it cannot remove alternatives merely by assigning one of them a high score. Additional evidence shrinks the compatible set. Camera motion changes perspective, stereo adds disparity, time reveals motion, touch reveals contact and a controlled action can test an affordance.

This gives uncertainty an operational meaning. The useful question is not only “how confident is the network?” It is “which decision-relevant alternatives remain observationally unresolved?” A low-entropy class distribution can coexist with ambiguous depth or blocked visibility. An unknown field should be preserved at the level where evidence is missing, rather than represented by one generic confidence number attached to the entire scene.

Shortcut success is not scene understanding

Even when the required fact is visible, a model may solve the benchmark through another correlated cue. Research on shortcut learning documents how predictors can perform well under ordinary splits while relying on brittle decision rules.[16] ImageNet-trained convolutional networks have also shown strong texture bias under controlled cue conflicts.[17] Clever Hans analyses demonstrate that apparently successful systems can use irrelevant artefacts that happen to predict the label.[18]

The response is not to demand a human-like explanation from every feature. It is to build paired tests. Hold object identity fixed and change texture. Hold geometry fixed and change background. Hold pixels near-constant and move the decision-relevant boundary. A representation that supports the intended scene relation should preserve or change in the declared direction.

The strongest objection: a large backbone can contain everything

A serious competing view says the output taxonomy matters less once a sufficiently large pretrained encoder supplies universal features. A small head can later recover category, mask, depth or correspondence. Recent foundation models provide real evidence for broader transfer, so this objection prevents a false claim that every relation needs a separately trained encoder.

Yet “decodable somewhere” is weaker than “available reliably to the deployed consumer”. A frozen feature can contain position while a pooled output discards it. A probe can recover a relation that the action head never uses. Fine-tuning can alter earlier invariances. Resolution, patch size, prompt, coordinate frame and temporal state still affect what arrives at the decision. The richer backbone changes the economics of obtaining scene views; it does not remove the need to specify and test the final view.

A benchmark for structure, not beauty

The executable artefact turns the argument into a controlled laboratory. It generates 64 by 64 scenes containing a red square and a blue square. The ordinary data makes background brightness perfectly predict whether red is left of blue. A counterfactual split reverses that background correlation while preserving the spatial rule. Paired scenes then translate both objects by three pixels right and three pixels up.

Three representations see the same pixels. global_summary retains colour areas and background counts but no coordinates. A one-dimensional decision stump learns the easy background shortcut. coarse_grid retains colour occupancy in an 8 by 8 grid. components retains exact colour masks and connected components. The point is not that hand-built components solve real vision. It is that each representation exposes a different preservation contract.

The benchmark asks four questions. Does the representation solve the ordinary left-of task? Does it survive the counterfactual background swap? When the image translates, does the red centroid move by the same vector? Does its mask shift rather than merely remain stable? These tests separate benchmark accuracy, nuisance robustness and spatial equivariance.

The relation test is deliberately binary, but the equivariance tests inspect geometry directly. For a known translation vector Δ, centroid error is the distance between the predicted shifted centroid and the original predicted centroid plus Δ. Perfect equivariance gives zero. Shifted-mask Intersection over Union first translates the original predicted mask by Δ, then compares it with the prediction on the translated image. Perfect overlap gives one. These metrics ask whether the output follows the declared transformation, not merely whether both images receive an acceptable task score.

The counterfactual split is equally important. Ordinary accuracy tests performance under the observed correlation. The reversed-background split keeps the named spatial rule unchanged while breaking the shortcut. Because only one causal feature is varied, the failure is interpretable: the global summary never had the coordinates required for “left of”. A random natural-image shift could show degradation, but it would not isolate whether the cause was background, scale, cropping, interpolation or object identity.

The global summary achieves perfect ordinary accuracy and zero counterfactual accuracy. It has learned a real statistical rule in the training distribution, but that rule is not the spatial relation named by the task. Because the summary has no coordinates, centroid and mask tests are not applicable. The coarse grid survives the correlation flip and approximately follows translation, but quantisation makes its reconstructed mask jump at cell boundaries. Exact colour components pass the positive translation controls.

The negative case matters. The script renders two same-colour squares whose edges touch. A four-neighbour connected-component algorithm returns one region, although the scene specification contains two instances. Pixel connectivity is not instance identity. Real instance segmentation needs learned or engineered evidence about boundaries, shape, depth, motion or occlusion. A richer output type does not guarantee the correct grouping mechanism.

A real benchmark can apply the same design without synthetic colours. Define a transformation family and the expected response of every output field. Build matched pairs for illumination, viewpoint, occlusion and instance exchange. Add counterfactual backgrounds and textures. Report the ordinary task metric beside field-level consistency, calibration and abstention. Finally, include a negative control where the coarse representation should win. This prevents a benchmark from rewarding complexity merely because the proposed system is complex.

#!/usr/bin/env python3
"""Synthetic benchmark for representation contracts in computer vision.

The benchmark uses only the Python standard library. It compares three ways of
representing the same 64 x 64 synthetic scenes:

1. global_summary: colour-area and background statistics, with no coordinates;
2. coarse_grid: an 8 x 8 spatial occupancy map for red and blue pixels;
3. components: exact colour masks and connected components.

The ordinary split makes background brightness perfectly predict whether the
red object is left of the blue object. The counterfactual split reverses that
correlation while preserving the spatial rule. Translation pairs test whether
locations and masks move predictably rather than merely staying invariant.

This is a mechanism laboratory, not a benchmark of learned vision models.
"""
from __future__ import annotations

from collections import deque
from dataclasses import dataclass
import math
import random
from typing import Callable, Iterable, Sequence

W = H = 64
RED, BLUE = 1, 2
DARK_BG, LIGHT_BG = 0, 3
SEED = 137

PixelGrid = tuple[tuple[int, ...], ...]
Mask = frozenset[tuple[int, int]]


@dataclass(frozen=True)
class Scene:
    pixels: PixelGrid
    red_mask: Mask
    blue_mask: Mask
    red_centre: tuple[float, float]
    blue_centre: tuple[float, float]
    red_left: bool
    background: int


def square_mask(cx: int, cy: int, half: int) -> set[tuple[int, int]]:
    return {
        (x, y)
        for y in range(cy - half, cy + half + 1)
        for x in range(cx - half, cx + half + 1)
        if 0 <= x < W and 0 <= y < H
    }


def render_scene(
    rng: random.Random,
    *,
    relation: bool | None = None,
    counterfactual_background: bool = False,
    forced_shift: tuple[int, int] = (0, 0),
) -> Scene:
    """Render one non-overlapping red-square / blue-square scene.

    relation=True means red is left of blue. Background is spuriously correlated
    with relation in the ordinary regime and anti-correlated in the
    counterfactual regime.
    """
    red_left = rng.choice((False, True)) if relation is None else relation
    y1, y2 = rng.randint(18, 46), rng.randint(18, 46)
    half_r, half_b = rng.randint(4, 7), rng.randint(4, 7)

    if red_left:
        rx, bx = rng.randint(12, 24), rng.randint(40, 52)
    else:
        bx, rx = rng.randint(12, 24), rng.randint(40, 52)

    dx, dy = forced_shift
    rx, bx, y1, y2 = rx + dx, bx + dx, y1 + dy, y2 + dy

    # Keep all masks in frame for paired translation tests.
    if not (8 <= rx <= 55 and 8 <= bx <= 55 and 8 <= y1 <= 55 and 8 <= y2 <= 55):
        raise ValueError("forced shift moved an object out of frame")

    red = square_mask(rx, y1, half_r)
    blue = square_mask(bx, y2, half_b)
    if red & blue:
        # The x ranges are deliberately separated, but retain a defensive check.
        raise AssertionError("objects unexpectedly overlap")

    ordinary_bg = DARK_BG if red_left else LIGHT_BG
    if counterfactual_background:
        background = LIGHT_BG if ordinary_bg == DARK_BG else DARK_BG
    else:
        background = ordinary_bg

    grid = [[background for _ in range(W)] for _ in range(H)]
    for x, y in red:
        grid[y][x] = RED
    for x, y in blue:
        grid[y][x] = BLUE

    return Scene(
        pixels=tuple(tuple(row) for row in grid),
        red_mask=frozenset(red),
        blue_mask=frozenset(blue),
        red_centre=(float(rx), float(y1)),
        blue_centre=(float(bx), float(y2)),
        red_left=red_left,
        background=background,
    )


def global_summary(scene: Scene) -> tuple[float, ...]:
    counts = [0, 0, 0, 0]
    for row in scene.pixels:
        for value in row:
            counts[value] += 1
    total = W * H
    return tuple(count / total for count in counts)


def fit_decision_stump(
    xs: Sequence[tuple[float, ...]], ys: Sequence[bool]
) -> Callable[[tuple[float, ...]], bool]:
    """Fit a one-dimensional threshold and polarity by exhaustive search."""
    best: tuple[int, float, bool, int] | None = None
    for feature in range(len(xs[0])):
        values = sorted({x[feature] for x in xs})
        thresholds = [values[0] - 1e-9]
        thresholds += [(a + b) / 2 for a, b in zip(values, values[1:])]
        thresholds += [values[-1] + 1e-9]
        for threshold in thresholds:
            for positive_if_greater in (False, True):
                errors = 0
                for x, y in zip(xs, ys):
                    pred = (x[feature] > threshold) == positive_if_greater
                    errors += pred != y
                candidate = (errors, feature, threshold, positive_if_greater)
                if best is None or candidate < (best[3], best[0], best[1], best[2]):
                    # Store as feature, threshold, polarity, errors.
                    best = (feature, threshold, positive_if_greater, errors)
    assert best is not None
    feature, threshold, polarity, _ = best
    return lambda x: (x[feature] > threshold) == polarity


def coarse_grid(scene: Scene, cells: int = 8) -> tuple[tuple[int, ...], tuple[int, ...]]:
    """Return red and blue pixel counts in an cells x cells grid."""
    cell_w, cell_h = W // cells, H // cells
    red, blue = [0] * (cells * cells), [0] * (cells * cells)
    for y, row in enumerate(scene.pixels):
        gy = min(cells - 1, y // cell_h)
        for x, value in enumerate(row):
            gx = min(cells - 1, x // cell_w)
            idx = gy * cells + gx
            if value == RED:
                red[idx] += 1
            elif value == BLUE:
                blue[idx] += 1
    return tuple(red), tuple(blue)


def grid_centroid(counts: Sequence[int], cells: int = 8) -> tuple[float, float] | None:
    total = sum(counts)
    if total == 0:
        return None
    cell_w, cell_h = W / cells, H / cells
    sx = sy = 0.0
    for idx, count in enumerate(counts):
        gy, gx = divmod(idx, cells)
        sx += count * ((gx + 0.5) * cell_w)
        sy += count * ((gy + 0.5) * cell_h)
    return sx / total, sy / total


def grid_mask(counts: Sequence[int], cells: int = 8) -> Mask:
    cell_w, cell_h = W // cells, H // cells
    mask: set[tuple[int, int]] = set()
    for idx, count in enumerate(counts):
        if count == 0:
            continue
        gy, gx = divmod(idx, cells)
        for y in range(gy * cell_h, (gy + 1) * cell_h):
            for x in range(gx * cell_w, (gx + 1) * cell_w):
                mask.add((x, y))
    return frozenset(mask)


def colour_mask(scene: Scene, colour: int) -> Mask:
    return frozenset(
        (x, y)
        for y, row in enumerate(scene.pixels)
        for x, value in enumerate(row)
        if value == colour
    )


def centroid(mask: Iterable[tuple[int, int]]) -> tuple[float, float] | None:
    pts = list(mask)
    if not pts:
        return None
    return (
        sum(x for x, _ in pts) / len(pts),
        sum(y for _, y in pts) / len(pts),
    )


def connected_components(mask: Mask) -> list[Mask]:
    remaining = set(mask)
    components: list[Mask] = []
    while remaining:
        start = remaining.pop()
        queue = deque([start])
        comp = {start}
        while queue:
            x, y = queue.popleft()
            for neighbour in ((x + 1, y), (x - 1, y), (x, y + 1), (x, y - 1)):
                if neighbour in remaining:
                    remaining.remove(neighbour)
                    comp.add(neighbour)
                    queue.append(neighbour)
        components.append(frozenset(comp))
    return components


def iou(a: Mask, b: Mask) -> float:
    union = a | b
    return 1.0 if not union else len(a & b) / len(union)


def shift_mask(mask: Mask, dx: int, dy: int) -> Mask:
    return frozenset((x + dx, y + dy) for x, y in mask)


def relation_from_grid(scene: Scene) -> bool:
    red, blue = coarse_grid(scene)
    rc, bc = grid_centroid(red), grid_centroid(blue)
    assert rc is not None and bc is not None
    return rc[0] < bc[0]


def relation_from_components(scene: Scene) -> bool:
    rc, bc = centroid(colour_mask(scene, RED)), centroid(colour_mask(scene, BLUE))
    assert rc is not None and bc is not None
    return rc[0] < bc[0]


def accuracy(predictions: Iterable[bool], targets: Iterable[bool]) -> float:
    pairs = list(zip(predictions, targets))
    return sum(p == y for p, y in pairs) / len(pairs)


def paired_translation_metrics(
    base: Sequence[Scene], shifted: Sequence[Scene], dx: int, dy: int
) -> dict[str, tuple[float, float]]:
    """Return mean centroid equivariance error and shifted-mask IoU."""
    grid_errors: list[float] = []
    grid_ious: list[float] = []
    comp_errors: list[float] = []
    comp_ious: list[float] = []

    for first, second in zip(base, shifted):
        g1, _ = coarse_grid(first)
        g2, _ = coarse_grid(second)
        c1, c2 = grid_centroid(g1), grid_centroid(g2)
        assert c1 is not None and c2 is not None
        grid_errors.append(math.dist((c1[0] + dx, c1[1] + dy), c2))
        grid_ious.append(iou(shift_mask(grid_mask(g1), dx, dy), grid_mask(g2)))

        m1, m2 = colour_mask(first, RED), colour_mask(second, RED)
        p1, p2 = centroid(m1), centroid(m2)
        assert p1 is not None and p2 is not None
        comp_errors.append(math.dist((p1[0] + dx, p1[1] + dy), p2))
        comp_ious.append(iou(shift_mask(m1, dx, dy), m2))

    return {
        "coarse_grid": (
            sum(grid_errors) / len(grid_errors),
            sum(grid_ious) / len(grid_ious),
        ),
        "components": (
            sum(comp_errors) / len(comp_errors),
            sum(comp_ious) / len(comp_ious),
        ),
    }


def touching_instance_negative_case() -> int:
    """Two same-colour squares touch, so 4-neighbour components merge them."""
    first = square_mask(27, 32, 5)
    second = square_mask(38, 32, 5)  # Edges touch at x=32/33 through adjacency.
    mask = frozenset(first | second)
    return len(connected_components(mask))


def main() -> None:
    rng = random.Random(SEED)

    train = [render_scene(rng) for _ in range(400)]
    iid = [render_scene(rng) for _ in range(200)]
    counterfactual = [
        render_scene(rng, counterfactual_background=True) for _ in range(200)
    ]

    stump = fit_decision_stump(
        [global_summary(scene) for scene in train],
        [scene.red_left for scene in train],
    )

    relation_results = {
        "global_summary": (
            accuracy((stump(global_summary(s)) for s in iid), (s.red_left for s in iid)),
            accuracy(
                (stump(global_summary(s)) for s in counterfactual),
                (s.red_left for s in counterfactual),
            ),
        ),
        "coarse_grid": (
            accuracy((relation_from_grid(s) for s in iid), (s.red_left for s in iid)),
            accuracy(
                (relation_from_grid(s) for s in counterfactual),
                (s.red_left for s in counterfactual),
            ),
        ),
        "components": (
            accuracy(
                (relation_from_components(s) for s in iid), (s.red_left for s in iid)
            ),
            accuracy(
                (relation_from_components(s) for s in counterfactual),
                (s.red_left for s in counterfactual),
            ),
        ),
    }

    # Paired scenes use the same latent scene specification and a controlled shift.
    # Recreate each pair from an identical per-pair seed.
    dx, dy = 3, -3
    bases: list[Scene] = []
    shifts: list[Scene] = []
    for pair_seed in range(1000, 1100):
        r1 = random.Random(pair_seed)
        relation = r1.choice((False, True))
        # Re-initialise with the same seed so all other random draws match.
        bases.append(render_scene(random.Random(pair_seed), relation=relation))
        shifts.append(
            render_scene(
                random.Random(pair_seed), relation=relation, forced_shift=(dx, dy)
            )
        )

    translation = paired_translation_metrics(bases, shifts, dx, dy)
    touching_count = touching_instance_negative_case()

    print("Synthetic scene-structure benchmark (seed=137)")
    print("Numbers are measured on generated 64 x 64 scenes, not real-world model results.\n")
    print(
        f"{'Representation':<20} {'IID relation':>13} {'CF relation':>13} "
        f"{'centroid error':>16} {'shifted mask IoU':>17}"
    )
    print("-" * 84)
    for name in ("global_summary", "coarse_grid", "components"):
        iid_acc, cf_acc = relation_results[name]
        if name == "global_summary":
            centroid_text = "n/a"
            mask_text = "n/a"
        else:
            error, mask_score = translation[name]
            centroid_text = f"{error:.2f}px"
            mask_text = f"{mask_score:.3f}"
        print(
            f"{name:<20} {iid_acc:>12.1%} {cf_acc:>12.1%} "
            f"{centroid_text:>16} {mask_text:>17}"
        )

    print("\nNegative case")
    print(
        "Two touching red instances are returned as "
        f"{touching_count} connected component (expected: 2 instances)."
    )

    # Positive controls and failure controls.
    assert relation_results["global_summary"][0] == 1.0
    assert relation_results["global_summary"][1] == 0.0
    assert relation_results["coarse_grid"][1] == 1.0
    assert relation_results["components"][1] == 1.0
    assert translation["components"][0] == 0.0
    assert translation["components"][1] == 1.0
    assert touching_count == 1


if __name__ == "__main__":
    main()

stateDiagram-v2
  [*] --> Claim
  state "Ignore a nuisance" as Claim
  state "Move with geometry" as Test
  state "React to the world" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

A production-shaped example: the bin-picking cell

Consider a synthetic warehouse cell that must pick one metal coupling from a bin. The camera frame contains several similar parts, shadows, the bin wall and a worker's glove near the edge. The permitted action is not “describe the image”. It is “propose one grasp that can be verified before motion, or decline”.

A class score answers whether a coupling is probably present. Detection separates candidate instances. Instance masks estimate visible extents. Depth supplies a surface estimate. Pose and keypoints support grasp generation. Tracking links the chosen object across the final approach. A typed scene state records the selected instance, coordinate frame, timestamp, visibility, confidence, grasp candidates and the evidence from which each field was derived.

The architecture should not copy every internal tensor into world state. It should compile a bounded state for the next decision. For example:

This state preserves enough structure to decline safely. A system that stores only “coupling, 0.96” cannot explain why motion was withheld. A system that stores a mask without a coordinate frame cannot command a robot. A system that stores a pose without valid time may act on a stale object. The usable scene is a typed claim packet, not the largest available feature map.

The visual pipeline should be evaluated at three boundaries. At the component boundary, test masks, depth, pose and tracks against matched transformations. At the state-compilation boundary, verify that instance identifiers, coordinate transforms, timestamps and uncertainty remain aligned. At the decision boundary, replay scenes through the grasp rule and measure proposed, declined and unsafe actions. A component can improve while the compiled state worsens if post-processing pairs the right mask with the wrong depth.

Failure injection makes those seams visible. Delay the depth frame by 300 milliseconds while leaving the colour frame current. Swap the camera calibration file. Occlude the chosen coupling after the grasp is proposed. Duplicate one detector box. The expected response is not always a correct grasp. Often it is invalidation, reacquisition or abstention. The production requirement is therefore lawful degradation under broken scene evidence, not only high average accuracy on clean frames.

The negative control: when classification is enough

Suppose a fixed camera inspects a bottle cap and the only decision is whether a bright tamper seal is present inside a fixed region. The conveyor position is mechanically constrained, rejected items are diverted harmlessly, and periodic sampling checks drift. A calibrated binary classifier or even a deterministic intensity rule may outperform a full scene stack on cost, latency and auditability.

This negative control prevents a hierarchy in which more visual structure is always better. Detection, segmentation and geometry impose annotation, compute and failure costs. They are justified only when the downstream query needs the distinctions. The smallest sufficient representation is a discipline of restraint.

The scene-structure adequacy test

Before choosing a detector, segmenter or foundation backbone, write the visual contract. The following four tests form a compact architecture decision instrument.

Record six items in the decision note: the downstream queries, the permitted actions, the nuisance set, the transformations whose effects must be preserved, the typed output fields and the unknown policy. Then state a threshold for every field that can block an action. A mask threshold without a decision threshold is incomplete; so is a pose tolerance without a coordinate frame or a track-age limit without a clock.

For each test, create positive cases, negative cases and abstention cases. Positive cases prove that the intended relation is usable. Negative cases expose shortcuts, merged instances, stale tracks or impossible geometry. Abstention cases check that the system can preserve unknowns instead of forcing a label, box or pose.

Then evaluate components and the compiled state separately. A high-quality mask can still be paired with the wrong instance identifier. Accurate depth can use the wrong camera calibration. Stable tracking can preserve a mistaken initial match. The evidence chain should retain source frame, model or algorithm version, coordinate frame, timestamp and post-processing that produced the claim.

Finally, test the decision, not only the image metric. Intersection over Union can improve while grasp success falls if errors concentrate near contact edges. Average precision can rise while a rare obstacle class remains missed. A representation is operationally adequate when its residual errors, uncertainties and unknowns fit the action envelope and recovery design.

The release gate should contain both a minimum capability and a maximum authority. The minimum says what the visual state must support, such as separating adjacent couplings at a declared distance. The maximum says what it may authorise, such as proposing a grasp while a deterministic controller verifies workspace limits. This keeps improved perception from silently expanding the system's permission to act.