Part one

The moment meaning becomes motion

A vision-language model maps images and text into representations from which it can generate language. A vision-language-action model is trained or adapted to generate robot actions as well. RT-1 first showed how a transformer policy could tokenise images, language and robot commands at substantial real-robot scale.[1] RT-2 then represented discretised robot actions with tokens in an existing language-model vocabulary and co-fine-tuned vision-language and robot data.[3] OpenVLA used a related next-token objective over discretised action dimensions, while π0 attached a separate flow-matching action expert that generates continuous action chunks.[7][8]

Those choices alter how a model emits commands. They do not remove the control problem. A number can be syntactically valid and physically impossible. A trajectory can be smooth in the camera frame and collide in the robot base frame. A correct action at time t can be wrong by the time it reaches an actuator. A pose that works for one wrist, gripper and joint configuration can be unreachable for another.

The useful distinction is between semantic sufficiency and action sufficiency. “Pick the blue mug” is semantically sufficient when it identifies the intended object and relation. An action is sufficient only when it also specifies a feasible change of state for this body, from this state, under the time and uncertainty available.

What a VLA policy conditions on

A useful abstraction writes the policy as a distribution over an action chunk rather than a single command:

I≤t is the timestamped visual history, ℓ the instruction, q_t the current proprioceptive state, e the embodiment contract and H the proposed horizon. Increasing H can make motion more coherent, but it also increases the amount of future that can become stale before feedback corrects it.

This form reveals three design questions that the label “end to end” often hides. Which variables are actually observed? In which frame are they expressed? Which component enforces that the proposed action remains valid? A model can learn some of these relations implicitly. The engineering burden is unchanged: tests must show that they are present where consequences depend on them.

A chain of claims, not one opaque mapping

The notation can make the policy look like one indivisible function. Operationally, it is a chain of claims. The image supports a claim that an object exists. A localisation step supports a claim about where it is. Language supports a claim that this object is the intended referent. An affordance supports a claim that a particular contact can advance the task. Kinematics and dynamics support a claim that the body can realise the contact. The effect observer finally supports a claim that the intended state change occurred.

One neural network may contribute to several claims, and an end-to-end loss may couple them. They still fail differently. Misclassification selects the wrong mug. Misregistration selects the right mug in the wrong place. A poor grasp pose reaches the right place with the wrong orientation. Stale evidence reaches yesterday's pose. Slip produces the commanded motion without the intended state change. Treating all of these as “policy failure” prevents targeted repair.

For consequential action, the system should therefore preserve a small evidence record at each boundary: the source observation and timestamp, the bound referent, the frame and transform version, the proposed contact, the embodiment and controller schema, the accepted command, and the observed postcondition. This is not a demand that a model expose a faithful verbal rationale. It is a demand that the system retain the externally testable objects on which execution depended.

A useful VLA trace is causal only where an intervention can change one claim while holding the others stable. If rotating the camera changes the world-frame grasp, the spatial claim failed. If swapping “red” and “blue” changes nothing, the semantic claim failed. If the object moves and the policy renews neither its target nor its horizon, the temporal claim failed. This decomposition turns a vague argument about grounding into an executable diagnosis.

Part two

flowchart TD
  A["One sentence, two bodies"] --> B["The moment meaning becomes motion"]
  B --> C["The four coordinates of an<br/>executable action"]
  C --> D["What current VLA evidence<br/>establishes"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#5B5CF6,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

The four coordinates of an executable action

Consider an action proposal as a claim: “this body can create this state transition, beginning from this observed state, before the evidence expires.” Four coordinates make that claim testable.

1. Semantics: which interaction is intended?

Language has to select more than a noun. “Put the mug on the tray” identifies a source, destination and relation. “Move the object blocking the drawer” also requires a scene-dependent role. The policy must bind those words to particular entities and preserve the binding while the scene changes.

A single-object demonstration is a weak test. A controller that always grasps the nearest item will appear perfectly language-conditioned when only one item is present. The decisive intervention adds a second equally reachable item, swaps the instruction and checks whether the selected trajectory changes. Language grounding is evidenced by controlled behavioural dependence on language, not by success in a scene where language is redundant.

2. Space: where can the intended interaction succeed?

An image location is not yet a robot pose. A pixel must be related to depth, camera calibration, object geometry, the robot base frame and the desired contact. For a mug, the centre of its bounding box may be a poor grasp point. A handle affords a lateral pinch; an open rim may afford suction; the body may make only one of these reachable.

The useful intermediate object is therefore not merely a detected object, but an affordance: a possible interaction under a body and task. SayCan made an earlier, higher-level version of this separation by combining language-model relevance with learned affordance values for available robot skills.[2] RT-Affordance later represented end-effector poses at key task stages, predicted them from language and imagery and used them to condition a low-level policy.[9] The article's scene-to-affordance field generalises that intuition into a test instrument.

A scene-to-affordance field

Let z denote a candidate contact pose and approach. Assign it a task-specific score:

S scores semantic compatibility with the instruction; R reachability from the current robot state; C path and contact clearance; G grasp or interaction quality for this end effector; and H predicted harm or violation. The weights encode task consequences rather than universal constants. The field can be explicit, learned or hybrid. Its value is diagnostic: each term can be intervened on.

From pixels to contact without losing the frame

The spatial translation can be written as a sequence. A pixel and depth estimate define a point in the camera frame. Camera extrinsics map it into the robot base frame. Object geometry proposes contact poses. Inverse kinematics maps a contact pose into joint configurations. A motion planner or learned controller maps the chosen configuration into a trajectory. The inner controller converts that trajectory into torques, velocities or positions.

Every step changes the type of the object being manipulated. A pixel has no unit of metres. A Cartesian displacement has no meaning until its reference frame and convention are named. A joint command is tied to a morphology. Silent conversion is especially dangerous in mixed robot datasets, where one source may encode absolute end-effector pose, another a delta pose, and another joint position. Normalisation makes numbers comparable; it does not make their semantics identical.

Uncertainty also changes as it moves through the chain. A two-pixel localisation error may be harmless for a large box and decisive for a thin handle. A small rotational calibration error creates a position error that grows with distance from the camera. A point estimate hides that geometry. A practical affordance field should therefore penalise candidates whose success is fragile under plausible calibration, depth and object-pose perturbations, not merely candidates that collide at the estimated mean.

A strong spatial test samples those perturbations deliberately. Reproject the same scene from a second calibrated camera, move the camera while preserving the table, or inject a bounded extrinsic error. The desired world-frame contact should remain stable within a declared tolerance. If performance depends on a familiar viewpoint rather than the reconstructed relation, the policy has learned a view-conditioned shortcut rather than a portable action.

3. Time: when does the proposal expire?

Suppose the tray moves 4 cm while a remote policy call is in flight. The intended relation is unchanged, but the old target pose is no longer current. A policy that outputs a long action sequence can be smooth and wrong. A policy that emits one tiny action at a time can react quickly but jitter, hesitate or lose long-range coherence.

This is why temporal representation matters. Action chunking predicts a short sequence and executes only part of it before observing again. Diffusion Policy combined multimodal action distributions with receding-horizon control.[5] Octo used a diffusion head for action chunks, and π0 modelled 50-step continuous chunks with flow matching while conditioning on images, language and proprioception.[6][8] These mechanisms can improve coherence, but only if the system declares how much of a chunk is committed before feedback can revoke it.

Every action proposal has a validity interval. It should carry the observation timestamp, controller period, planned horizon and a rule for replanning after surprise. Without these, delay is silently converted into spatial error.

Chunk length is a control parameter

The right horizon is not a model-family constant. It depends on how quickly the world can invalidate a plan. Let L be sensing, inference and communication latency; K the duration of the action prefix committed before another observation can change it; and Td the characteristic time over which a relevant disturbance changes the task. The ratio ρ = (L + K) / Td is a useful stress variable.

When ρ is small, a longer chunk may improve smoothness with little staleness debt. When it approaches one, the system is committing across the same timescale on which the scene changes. It should shorten the prefix, move inference closer to the robot, predict the disturbance, install a fast local stabiliser or refuse the task. This ratio is illustrative rather than a universal stability theorem, but it forces latency, commitment and environment dynamics into the same experiment.

Evaluation should sweep delay rather than report one hardware configuration. Plot task error, collision rate and stop frequency against injected latency and object speed. Repeat with identical initial frames so the only change is temporal. A policy that succeeds at 5 Hz on static tabletop picking has not thereby shown that it can support 50 Hz bimanual contact. Conversely, a high-frequency action head does not establish long-horizon recovery if its observations or completion tests remain weak.

4. Embodiment: which command can this body realise?

Robot datasets mix arms, grippers, camera layouts, action conventions and control rates. Open X-Embodiment made cross-embodiment learning possible by pooling many datasets, but the mixture does not make all action spaces interchangeable.[4] Octo explicitly designed adapters for new observations and action heads. π0 routed proprioception and actions through a robotics-specific expert while supporting different dimensional action spaces.[6][8]

An embodiment contract should state at least the base and camera frames, action semantics, units, control frequency, joint and velocity limits, end-effector capabilities, reach envelope and available feedback. It is not a model card for the neural network. It is the interface between a proposal and a physical mechanism.

Part three

What current VLA evidence establishes

The strongest reading of the literature is neither “VLAs are general robots” nor “web pretraining does not help control”. The evidence supports a narrower causal picture.

Semantic priors can expand where learned skills are deployed

RT-2 co-fine-tuned large vision-language backbones with robot trajectories and evaluated thousands of real-robot rollouts. Its largest gains appeared under novel objects, backgrounds, environments and instructions. Yet the paper's limitation is decisive: additional web experience did not create new motions. The physical repertoire remained bounded by robot experience.[3]

This is exactly what the preservation view predicts. Web data can improve which learned skill is selected and how a scene is interpreted. It cannot by itself specify the dynamics, contact response and control convention of a body it never operated.

Diverse robot data improves transfer, but the action head matters

OpenVLA trained a 7 billion parameter model on 970,000 demonstrations and combined semantic SigLIP features with spatial DINOv2 features. In the authors' evaluation it improved over RT-2-X across a set of tasks and adapted to new robot setups with small in-domain datasets.[7] Its own discussion also noted a boundary: specialised Diffusion Policy baselines produced smoother, more precise behaviour on some narrow dexterous tasks, and action chunking or temporal smoothing could help.

Octo's ablations likewise found that a diffusion action head and a wider data mixture materially changed performance. Its modular design supported fine-tuning to new observations, action spaces and embodiments, but language-labelled and wrist-camera data were unevenly represented, and novel precise behaviours degraded.[6]

π0 combined a pretrained VLM with a flow-matching action expert and high-frequency action chunks. Its architecture is evidence that semantic scale and motor expressivity need not share one output parameterisation. The model used VLM representations for images and language, while robotics-specific weights processed state and continuous action chunks.[8]

The frontier varies the division of labour

Later systems change where semantic and motor computation meet. π0.5 co-trained heterogeneous high-level semantic outputs and low-level actions for open-world household tasks.[13] Gemini Robotics placed a VLA inside a wider system that also included a separate embodied-reasoning model.[14] GR00T N1 used a dual-system design with a vision-language module and a diffusion-based action module.[15] These architectures are not controlled proof that one split is necessary. They are evidence that leading designs treat semantic reasoning, continuous control and embodiment adaptation as distinct pressures even when training remains joint.

Why a larger mixture is not automatically a better intervention

Cross-embodiment data can teach reusable regularities, but a pooled dataset also contains shortcuts. Robot identity, background, camera height and task label may travel together. A model can infer “which dataset am I in?” and select the associated behaviour without learning the intended relation. The 2025 shortcut-learning study of generalist robot policies identified limited within-dataset diversity and fragmentation among Open X-Embodiment sources as conditions that can encourage such correlations.[12]

Randomly mixing more episodes does not break a correlation that every source preserves. The informative data are counterfactual combinations: the same task on different bodies, different tasks in the same visual setting, the same object under new camera poses, and contradictory instructions over an unchanged scene. These combinations make the causal variable identifiable. They are expensive, but they can be more valuable for evaluation than another large block of near-duplicate demonstrations.

This also changes how cross-embodiment success should be read. Out-of-the-box control on a robot represented in pretraining tests mixture reuse. Fine-tuning to a new action space tests adaptation. Zero-shot transfer to a genuinely new morphology tests something stronger and should not be inferred from the first two. A source pack, benchmark report and model card should keep those regimes separate.

Intermediate spatial guidance can close a specific gap

RT-Affordance held the underlying model family and robot data broadly comparable while changing the policy interface. Its affordance plan encoded end-effector poses at key contact stages. On the authors' novel-object grasp suite, language-conditioned RT-2 achieved 28 per cent average success, while predicted affordances reached 68 per cent; on a small set of placing and articulated tasks, the reported comparison was 3 per cent versus 70 per cent.[9] These are small, study-specific evaluations, not a general VLA ranking. They do show that preserving contact geometry can change outcomes even when semantic object recognition is already plausible.

TraceVLA offers a related intervention. It encoded state-action trajectories as visual traces and reported gains over OpenVLA in simulation and on a WidowX evaluation.[10] The result does not prove that one trace representation is universally best. It supports the narrower proposition that explicit spatial-temporal state can recover information that a single image and instruction did not reliably preserve.

Correct intention and successful execution separate under intervention

The INT-ACT probing suite measured whether a gripper moved towards the correct source object, whether it grasped that object and whether the full task succeeded. In its BridgeV2-based simulation, the fine-tuned π0 condition averaged 84.5 per cent intention correctness, 54.4 per cent grasp success and 30.4 per cent task success across the suite.[11] The gap is not a universal estimate of VLA reliability. It is direct evidence that semantic targeting and motor completion are separable measurements.

Controls that expose shortcuts

A policy should face interventions that make its preferred explanation unnecessary. The strongest competing explanation is simple: perhaps current VLA performance comes mainly from data scale and a powerful imitation model, while explicit preservation variables add little. That hypothesis deserves a fair test.

Start with the negative control that flatters it. Put one object on an empty table. A language-blind, geometry-aware controller can succeed. Add a second object and swap only the instruction. If the same controller now chooses the wrong object, the first result no longer supports language grounding. Next rotate the camera while preserving world geometry. Add delay while preserving the current image. Swap the end effector while preserving scene and words. Each intervention changes one causal feature.

The preservation thesis would weaken if a language-blind, frame-randomised, open-loop and embodiment-agnostic policy matched a fully conditioned policy across these controlled changes. That observation would suggest that task regularities, not preserved interfaces, explain the performance. Existing evidence instead shows recurrent sensitivity to instruction variation, camera and scene changes, precise manipulation, delay and action representation.[11][12][16]

Measure the episode as a sequence of necessary events

Final success is indispensable, but it is too compressed for diagnosis. Record whether the correct referent was selected, whether the proposed contact was feasible, whether the robot reached the pre-contact region, whether contact occurred, whether the object moved as intended, whether the terminal relation held and whether the system recognised failure. These events form a funnel. A drop between two adjacent stages localises the missing capability.

Report denominators and paired conditions. “Eighty per cent grasp success” can mean eight successes in ten demonstrations of one familiar pose, or hundreds of successes across held-out objects, cameras and delays. The latter carries more information. Confidence intervals do not repair a benchmark whose intervention never made the tested capability necessary.

Abstention belongs in the outcome space. On a missing referent, unreachable object or incompatible gripper, refusing to act is the correct behaviour. Accuracy that scores every refusal as failure will reward unsafe guessing. A useful report separates task completion, correct abstention, unnecessary abstention, wrong-object action, collision and unknown outcome. It should also state which failures were detected before effect and which were discovered only after the world changed.

Part four

stateDiagram-v2
  [*] --> Claim
  state "What a VLA policy conditions on" as Claim
  state "A chain of claims, not one<br/>opaque mapping" as Test
  state "Published evidence, bounded" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

A preservation-first engineering pattern

The practical response is not to split every VLA into a rigid classical stack. Learned end-to-end representations can be valuable precisely because semantics, perception and control interact. The response is to make the action claim explicit at the boundary where a stochastic proposal can affect the world.

The action preservation contract

Represent each proposal with a compact, typed contract:

referent and relation preserve intent. frame locates every pose. valid_at and horizon expose staleness. embodiment names the action schema and body. preconditions and bounds make feasibility mechanically checkable. verification says what observation would count as the intended effect.

This contract can be populated directly by a model, compiled from an affordance representation or attached by a wrapper. It does not force a particular neural architecture. Its purpose is to stop hidden coordinate changes from crossing the actuation boundary.

Failure injection before robot scale

Nominal demonstrations teach the happy path. A release test should introduce failures at the same typed boundaries. Expire the camera calibration while leaving images plausible. Skew image and joint timestamps. Drop one camera. Delay the policy response. Insert an obstacle after the plan. Change the gripper state without changing the visual scene. Accept a controller command but suppress its physical effect. Let the object slip after a successful grasp.

Each injection has an expected containment response. Stale calibration should block world-frame execution. Timestamp skew should shorten the horizon or require resynchronisation. A new obstacle should invalidate the path before contact. Suppressed effect should produce an unknown outcome, not a success receipt. Slip should trigger re-observation and recovery only if the object remains identifiable and the new action is authorised.

The objective is not that the VLA itself diagnose every fault. The objective is that no single plausible model output bypasses the checks needed to keep the system's claim true. Faults that can be detected more reliably from encoders, controller state, force thresholds or readback should be detected there. Use the model where interpretation is valuable; use deterministic mechanisms where the condition is already formal.

Worked example one: the smallest scene that distinguishes the mechanisms

Place a red cup alone at coordinates (0.5, 0.0). Ask three policies to pick it. Policy A reads language but treats camera coordinates as base coordinates and commits to one open-loop target. Policy B uses calibrated geometry and feedback but ignores language, always choosing the nearest item. Policy C uses language, calibration, embodiment checks and receding-horizon replanning.

All three can pass the nominal case. Now add a blue cup at (0.8, 0.3) and instruct “pick blue cup”. Policy B still executes a beautiful grasp, on the red cup. Rotate the camera and add an obstacle. Policy A collides because its semantically correct target is expressed in the wrong frame. Move the blue cup during execution. Policy A reaches the old location. Replace the pinch gripper with suction and request a flat lid. Policy C changes the contact contract; the pinch version abstains.

This sequence is more informative than a large success-rate average because it identifies the role of each variable. The runnable artefact below implements exactly these interventions.

Worked example two: mixed-item tote picking

Consider a fulfilment cell asked to “place the damaged blue carton in the inspection bin”. The VLA sees a cluttered tote and selects a carton whose crushed corner matches “damaged”. It proposes an approach pose and a short action chunk. A production-shaped system should still keep five responsibilities outside the unverified proposal.

First, an observation service binds every camera frame, depth map and joint state to a common timestamp. Second, calibration maps the selected pixels into the robot base frame and propagates uncertainty. Third, an embodiment adapter converts the generic pose into the cell's wrist and gripper convention. Fourth, a deterministic kernel checks reach, collision, force, speed, bin identity and human exclusion zones. Fifth, an effect observer confirms that the selected carton left the tote and entered the inspection bin. If the gripper closes but the carton remains, the outcome is unknown or failed, not successful because the command was accepted.

The model may propose the semantic target, contact region and recovery. It does not acquire ambient authority to actuate. If object identity is uncertain, two cartons overlap, calibration residual exceeds its threshold or the scene changes after the proposal, the system re-observes or escalates. Bounded autonomy comes from the whole loop, not from confidence in the central model.

Walk through one failure. The first proposal selects the correct carton but routes behind a rigid divider. The collision checker rejects the path while preserving the semantic target. A second proposal chooses a top grasp, but the suction-quality estimate is below threshold because the damaged surface is porous. The system does not ask the model to “be more confident”. It changes the mechanism, requesting a side pinch available on the same embodiment. The controller executes a short approach prefix, then a wrist camera reveals that another carton has shifted into the corridor. The prefix expires and the robot stops before contact.

After a successful pinch, the gripper sensor reports closure but the tote image still shows the carton in place. The effect observer records an ambiguous grasp and prevents the downstream “place in inspection bin” step. This distinction matters operationally: command acceptance, motor completion and verified business outcome are three different events. A fluent VLA plan can connect them, but only observation can establish which event occurred.

A release instrument

Run the preservation test

The companion Python programme uses only the standard library. A tiny A* planner acts as the deterministic geometry layer. Three policies face seven trials: a semantic but uncalibrated open-loop policy; a language-blind but geometry-aware closed-loop policy; and an affordance-preserving policy that conditions on target, frame, time and embodiment.

#!/usr/bin/env python3
"""A synthetic Vision-Language-Action preservation test.

The simulator is deliberately small. It isolates four interfaces that a VLA
policy must preserve: semantic target selection, camera-to-world geometry,
closed-loop time, and embodiment constraints.

Run:
    python3 vla_affordance_sim.py

No third-party dependencies are required.
"""
from __future__ import annotations

from dataclasses import dataclass, replace
from heapq import heappop, heappush
from math import cos, hypot, pi, sin
from typing import Dict, Iterable, List, Optional, Sequence, Tuple

Vec = Tuple[float, float]
Cell = Tuple[int, int]


def add(a: Vec, b: Vec) -> Vec:
    return (a[0] + b[0], a[1] + b[1])


def sub(a: Vec, b: Vec) -> Vec:
    return (a[0] - b[0], a[1] - b[1])


def scale(a: Vec, k: float) -> Vec:
    return (a[0] * k, a[1] * k)


def norm(a: Vec) -> float:
    return hypot(a[0], a[1])


def distance(a: Vec, b: Vec) -> float:
    return norm(sub(a, b))


def rotate(a: Vec, angle: float) -> Vec:
    c, s = cos(angle), sin(angle)
    return (c * a[0] - s * a[1], s * a[0] + c * a[1])


@dataclass(frozen=True)
class Camera:
    """Rigid transform from world coordinates to camera coordinates."""

    angle: float = 0.0
    translation: Vec = (0.0, 0.0)

    def world_to_camera(self, p: Vec) -> Vec:
        return add(rotate(p, self.angle), self.translation)

    def camera_to_world(self, p: Vec) -> Vec:
        return rotate(sub(p, self.translation), -self.angle)


@dataclass(frozen=True)
class Item:
    name: str
    position: Vec
    velocity: Vec = (0.0, 0.0)
    required_grasp: str = "pinch"  # pinch, suction or either

    def at_tick(self, tick: int) -> "Item":
        return replace(self, position=add(self.position, scale(self.velocity, tick)))


@dataclass(frozen=True)
class Obstacle:
    centre: Vec
    radius: float


@dataclass(frozen=True)
class Embodiment:
    name: str
    base: Vec
    reach: float
    grasp_mode: str
    max_step: float = 0.14


@dataclass(frozen=True)
class Trial:
    name: str
    instruction: str
    items: Tuple[Item, ...]
    obstacles: Tuple[Obstacle, ...]
    camera: Camera
    embodiment: Embodiment
    max_ticks: int = 24
    observation_lag: int = 0


@dataclass(frozen=True)
class Outcome:
    trial: str
    policy: str
    status: str
    selected: str
    ticks: int
    detail: str


class GridPlanner:
    """A small A* planner used as the deterministic geometry layer."""

    def __init__(self, bound: float = 1.4, step: float = 0.1) -> None:
        self.bound = bound
        self.step = step
        self.n = int(round((2 * bound) / step)) + 1

    def to_cell(self, p: Vec) -> Cell:
        x = int(round((p[0] + self.bound) / self.step))
        y = int(round((p[1] + self.bound) / self.step))
        return (max(0, min(self.n - 1, x)), max(0, min(self.n - 1, y)))

    def to_world(self, c: Cell) -> Vec:
        return (c[0] * self.step - self.bound, c[1] * self.step - self.bound)

    def blocked(self, p: Vec, obstacles: Sequence[Obstacle], embodiment: Embodiment) -> bool:
        if distance(p, embodiment.base) > embodiment.reach:
            return True
        return any(distance(p, o.centre) <= o.radius + 0.04 for o in obstacles)

    def plan(
        self,
        start: Vec,
        goal: Vec,
        obstacles: Sequence[Obstacle],
        embodiment: Embodiment,
    ) -> Optional[List[Vec]]:
        start_c, goal_c = self.to_cell(start), self.to_cell(goal)
        if self.blocked(self.to_world(goal_c), obstacles, embodiment):
            return None

        neighbours = [
            (-1, 0), (1, 0), (0, -1), (0, 1),
            (-1, -1), (-1, 1), (1, -1), (1, 1),
        ]
        frontier: List[Tuple[float, Cell]] = [(0.0, start_c)]
        came_from: Dict[Cell, Optional[Cell]] = {start_c: None}
        cost: Dict[Cell, float] = {start_c: 0.0}

        while frontier:
            _, current = heappop(frontier)
            if current == goal_c:
                break
            for dx, dy in neighbours:
                nxt = (current[0] + dx, current[1] + dy)
                if not (0 <= nxt[0] < self.n and 0 <= nxt[1] < self.n):
                    continue
                world = self.to_world(nxt)
                if self.blocked(world, obstacles, embodiment):
                    continue
                move_cost = 1.414 if dx and dy else 1.0
                new_cost = cost[current] + move_cost
                if nxt not in cost or new_cost < cost[nxt]:
                    cost[nxt] = new_cost
                    priority = new_cost + distance(world, self.to_world(goal_c)) / self.step
                    heappush(frontier, (priority, nxt))
                    came_from[nxt] = current

        if goal_c not in came_from:
            return None

        cells: List[Cell] = []
        cursor: Optional[Cell] = goal_c
        while cursor is not None:
            cells.append(cursor)
            cursor = came_from[cursor]
        cells.reverse()
        return [self.to_world(c) for c in cells]


def parse_target(instruction: str, items: Iterable[Item]) -> Optional[str]:
    text = instruction.lower()
    matches = [item.name for item in items if item.name.lower() in text]
    return max(matches, key=len) if matches else None


def compatible(item: Item, embodiment: Embodiment) -> bool:
    return item.required_grasp in {"either", embodiment.grasp_mode}


def collision_on_segment(a: Vec, b: Vec, obstacles: Sequence[Obstacle]) -> bool:
    # Sample densely enough for this illustrative environment.
    steps = max(1, int(distance(a, b) / 0.025))
    for i in range(steps + 1):
        p = add(a, scale(sub(b, a), i / steps))
        if any(distance(p, o.centre) <= o.radius + 0.03 for o in obstacles):
            return True
    return False


class Policy:
    name = "policy"

    def run(self, trial: Trial) -> Outcome:
        raise NotImplementedError


class SemanticOpenLoopPolicy(Policy):
    """Uses the instruction, but drops calibration, geometry and feedback."""

    name = "semantic-only open loop"

    def run(self, trial: Trial) -> Outcome:
        observed = tuple(
            replace(item, position=trial.camera.world_to_camera(item.position))
            for item in trial.items
        )
        target_name = parse_target(trial.instruction, observed)
        if target_name is None:
            return Outcome(trial.name, self.name, "abstain", "none", 0, "instruction names no observed item")
        observed_target = next(i for i in observed if i.name == target_name)
        world_target = next(i for i in trial.items if i.name == target_name)

        # The failure is intentional: camera coordinates are treated as base coordinates.
        proposed = observed_target.position
        if distance(proposed, trial.embodiment.base) > trial.embodiment.reach:
            return Outcome(trial.name, self.name, "unsafe proposal", target_name, 0, "target appears outside reach")
        if collision_on_segment(trial.embodiment.base, proposed, trial.obstacles):
            return Outcome(trial.name, self.name, "collision", target_name, 0, "direct path intersects obstacle")
        if not compatible(world_target, trial.embodiment):
            return Outcome(trial.name, self.name, "incompatible grasp", target_name, 0, "action ignores end-effector capability")

        final_target = world_target.at_tick(trial.max_ticks).position
        if distance(proposed, final_target) <= 0.12:
            return Outcome(trial.name, self.name, "success", target_name, trial.max_ticks, "nominal coordinates happened to remain valid")
        return Outcome(trial.name, self.name, "miss", target_name, trial.max_ticks, "open-loop target was stale or misregistered")


class LanguageBlindClosedLoopPolicy(Policy):
    """Preserves geometry and feedback but deliberately ignores the instruction."""

    name = "language-blind closed loop"

    def __init__(self) -> None:
        self.planner = GridPlanner()

    def run(self, trial: Trial) -> Outcome:
        start = trial.embodiment.base
        # Negative control: always choose the nearest visible item.
        selected = min(trial.items, key=lambda i: distance(i.position, start))
        desired = parse_target(trial.instruction, trial.items)
        if not compatible(selected, trial.embodiment):
            return Outcome(trial.name, self.name, "incompatible grasp", selected.name, 0, "nearest item cannot be grasped by this embodiment")

        position = start
        for tick in range(trial.max_ticks):
            live = selected.at_tick(max(0, tick - trial.observation_lag))
            observed = trial.camera.world_to_camera(live.position)
            target = trial.camera.camera_to_world(observed)
            path = self.planner.plan(position, target, trial.obstacles, trial.embodiment)
            if not path:
                return Outcome(trial.name, self.name, "abstain", selected.name, tick, "no collision-free path")
            waypoint = path[min(2, len(path) - 1)]
            delta = sub(waypoint, position)
            step = min(trial.embodiment.max_step, norm(delta))
            next_position = add(position, scale(delta, step / max(norm(delta), 1e-9)))
            if collision_on_segment(position, next_position, trial.obstacles):
                return Outcome(trial.name, self.name, "collision", selected.name, tick, "geometry layer failed")
            position = next_position
            actual = selected.at_tick(tick).position
            if distance(position, actual) <= 0.12:
                if selected.name == desired:
                    return Outcome(trial.name, self.name, "success", selected.name, tick + 1, "geometry succeeded without testing language")
                return Outcome(trial.name, self.name, "wrong object", selected.name, tick + 1, "closed-loop motion served the wrong semantic goal")
        return Outcome(trial.name, self.name, "timeout", selected.name, trial.max_ticks, "did not reach selected item")


class AffordanceClosedLoopPolicy(Policy):
    """Preserves semantic, spatial, temporal and embodiment constraints."""

    name = "affordance-preserving closed loop"

    def __init__(self) -> None:
        self.planner = GridPlanner()

    def run(self, trial: Trial) -> Outcome:
        target_name = parse_target(trial.instruction, trial.items)
        if target_name is None:
            return Outcome(trial.name, self.name, "abstain", "none", 0, "instruction has no grounded referent")
        target_item = next(i for i in trial.items if i.name == target_name)
        if not compatible(target_item, trial.embodiment):
            return Outcome(trial.name, self.name, "abstain", target_name, 0, "embodiment contract rejects grasp")

        position = trial.embodiment.base
        for tick in range(trial.max_ticks):
            sensed_tick = max(0, tick - trial.observation_lag)
            sensed_item = target_item.at_tick(sensed_tick)
            camera_point = trial.camera.world_to_camera(sensed_item.position)
            target = trial.camera.camera_to_world(camera_point)

            if distance(target, trial.embodiment.base) > trial.embodiment.reach:
                return Outcome(trial.name, self.name, "abstain", target_name, tick, "target lies outside reachable set")

            path = self.planner.plan(position, target, trial.obstacles, trial.embodiment)
            if not path:
                return Outcome(trial.name, self.name, "abstain", target_name, tick, "no collision-free affordance remains")

            # Receding-horizon execution: move only a short prefix, then observe again.
            waypoint = path[min(2, len(path) - 1)]
            delta = sub(waypoint, position)
            step = min(trial.embodiment.max_step, norm(delta))
            next_position = add(position, scale(delta, step / max(norm(delta), 1e-9)))
            if collision_on_segment(position, next_position, trial.obstacles):
                return Outcome(trial.name, self.name, "collision", target_name, tick, "unexpected geometry violation")
            position = next_position

            actual = target_item.at_tick(tick).position
            if distance(position, actual) <= 0.12:
                return Outcome(trial.name, self.name, "success", target_name, tick + 1, "referent, frame, timing and grasp agree")
        return Outcome(trial.name, self.name, "timeout", target_name, trial.max_ticks, "feedback could not close the remaining error")


def make_trials() -> Tuple[Trial, ...]:
    pinch = Embodiment("pinch arm", base=(0.0, 0.0), reach=1.25, grasp_mode="pinch")
    suction = Embodiment("suction arm", base=(0.0, 0.0), reach=1.25, grasp_mode="suction")
    return (
        Trial(
            "nominal single object",
            "pick red cup",
            (Item("red cup", (0.72, 0.0)),),
            (), Camera(), pinch,
        ),
        Trial(
            "instruction contrast",
            "pick blue cup",
            (Item("red cup", (0.42, 0.0)), Item("blue cup", (0.82, 0.28))),
            (), Camera(), pinch,
        ),
        Trial(
            "shifted camera plus obstacle",
            "pick blue cup",
            (Item("blue cup", (0.92, 0.22)),),
            (Obstacle((0.48, 0.10), 0.22),),
            Camera(angle=pi / 6, translation=(0.18, -0.12)), pinch,
        ),
        Trial(
            "moving target with delay",
            "pick blue cup",
            (Item("blue cup", (0.68, -0.30), velocity=(0.0, 0.018)),),
            (), Camera(), pinch, max_ticks=28, observation_lag=1,
        ),
        Trial(
            "embodiment swap: pinch",
            "pick flat lid",
            (Item("flat lid", (0.72, 0.05), required_grasp="suction"),),
            (), Camera(), pinch,
        ),
        Trial(
            "embodiment swap: suction",
            "pick flat lid",
            (Item("flat lid", (0.72, 0.05), required_grasp="suction"),),
            (), Camera(), suction,
        ),
        Trial(
            "missing referent",
            "pick green cup",
            (Item("red cup", (0.50, 0.0)),),
            (), Camera(), pinch,
        ),
    )


def main() -> None:
    policies: Tuple[Policy, ...] = (
        SemanticOpenLoopPolicy(),
        LanguageBlindClosedLoopPolicy(),
        AffordanceClosedLoopPolicy(),
    )
    outcomes = [policy.run(trial) for trial in make_trials() for policy in policies]

    print("VLA preservation test (synthetic)\n")
    print(f"{'trial':31} {'policy':34} {'status':18} {'selected':12} detail")
    print("-" * 126)
    for o in outcomes:
        print(f"{o.trial:31} {o.policy:34} {o.status:18} {o.selected:12} {o.detail}")

    lookup = {(o.trial, o.policy): o.status for o in outcomes}
    assert lookup[("nominal single object", "language-blind closed loop")] == "success"
    assert lookup[("instruction contrast", "language-blind closed loop")] == "wrong object"
    assert lookup[("shifted camera plus obstacle", "affordance-preserving closed loop")] == "success"
    assert lookup[("moving target with delay", "semantic-only open loop")] == "miss"
    assert lookup[("moving target with delay", "affordance-preserving closed loop")] == "success"
    assert lookup[("embodiment swap: pinch", "affordance-preserving closed loop")] == "abstain"
    assert lookup[("embodiment swap: suction", "affordance-preserving closed loop")] == "success"
    assert lookup[("missing referent", "affordance-preserving closed loop")] == "abstain"
    print("\nAll preservation assertions passed.")


if __name__ == "__main__":
    main()

{
  "unit_of_analysis": "closed-loop episode",
  "required_strata": ["instruction", "camera", "delay", "embodiment"],
  "primary_metrics": [
    "correct_referent",
    "collision_free_completion",
    "verified_task_effect",
    "safe_abstention"
  ],
  "negative_controls": [
    "single-object language-redundant scene",
    "language-blind geometry-aware policy",
    "open-loop calibrated-at-start policy"
  ],
  "release_rule": "no critical interface may rely on aggregate success alone"
}

Where this pattern stops helping

The contract does not solve perception under severe occlusion, tactile state estimation, deformable-object dynamics or whole-body control by itself. It cannot make an unseen gripper controllable without calibration and data. It cannot guarantee that an affordance predictor is correct. It does not turn simulation success into real-world evidence.

For contact-rich tasks, force, torque and tactile feedback may dominate vision. For fast locomotion, a large remote VLA may belong above a much faster local policy rather than in the inner loop. For irreversible or safety-critical actions, independent planning, certified limits, human authority or formal safety systems may be required. A model's uncertainty estimate should change the action, but it is not a substitute for enforced bounds.

The pattern is most useful where a VLA genuinely contributes open-vocabulary selection or transferable policy structure, while the physical system can expose calibrated state, bounded action interfaces and observable outcomes. Where a narrow diffusion policy, classical grasp planner or scripted controller already solves the task with stronger evidence and lower cost, adding a VLA can create an unnecessary failure surface.

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: One sentence, two bodies
  M->>E: What a VLA policy conditions on
  E-->>M: compare rival explanations
  E->>D: General action begins at the interface
  D-->>Q: revise scope or act

General action begins at the interface

Vision-language-action models matter because they offer a route from broad visual and linguistic priors into learned control. The important word is not “general”. It is “route”. Every route can preserve a relation, distort it or drop it.

The architecture decision changes from “which VLA is strongest?” to “which interfaces does this system demonstrably preserve under intervention?” Select a model and action representation only after defining the referent test, camera-frame test, delay test, embodiment swap and effect verification. Keep infeasible or unauthorised proposals outside the controller. Execute bounded prefixes, observe the consequence and renew the action claim from current evidence.

A model that recognises the right mug has supplied valuable semantic information. A system that reaches the right contact with the right body, at the right time, without crossing a forbidden state, has begun to turn that information into action. The difference is the engineering content of embodiment.