The world can enter without answering back

A model trained on text alone meets the world through descriptions. A multimodal model meets it through descriptions, images, waveforms and traces. That is a major extension, but it does not by itself settle what the symbols mean for the system.

The classical symbol-grounding problem asks how a formal symbol system could acquire meaning that is more than supplied by an outside interpreter. Harnad framed the difficulty as escaping a dictionary whose definitions only point to more symbols. Modern language models complicate the picture because their dictionaries are vast and their use of language is generative, contextual and interactive. Still, a corpus is made from traces produced by people whose words already participate in perception, action, institutions and shared practice.

Adding images changes the training signal. Contrastive systems such as CLIP learn to place matching images and captions near one another in a shared representation space. The original study showed broad zero-shot transfer after training on hundreds of millions of image-text pairs (Radford et al.). This is not a cosmetic gain. “Jaguar” can now be pulled towards spotted animals in one region of the space and towards cars in another. Texture, shape, scene and language jointly constrain the representation.

Yet the caption “a mug on a table” is still a human-produced link between a visual arrangement and a linguistic practice. The model did not need to turn the mug, discover its hidden handle, feel its mass or learn that this mug is glued down. Multimodality reduces indeterminacy among descriptions, but it can leave the model’s contact with the referent inherited rather than enacted.

Different modalities constrain different ambiguities. An image preserves projective structure and co-location. Audio preserves timing, rhythm and source mixtures. Touch begins at contact and reveals compliance, texture or slippage. Proprioception locates a body relative to its own joints. Their synchrony can be especially informative. A visible impact followed by a sound links two streams through time; a gripper closing while force rises links vision to contact.

Still, a recording of touch is not touch for the receiving system. It is a trace of someone else’s contact. A video of a camera moving around an object contains viewpoint change, but the model did not choose the viewpoint to resolve its uncertainty. The data may preserve the answer to a question that another agent happened to ask. It may even preserve enough alternatives for the model to infer the hidden structure. What remains absent is the ability to decide which missing distinction matters now and to make evidence depend on that decision.

This is why “multimodal” describes an interface more reliably than it describes a cognitive mechanism. Two systems may accept the same image and text tokens while integrating them very differently. One may align global features for retrieval. Another may track objects through time. A third may use vision only to fill a prompt that a language model then processes. The number of modalities does not identify the relation learned among them. A grounding claim must therefore name the source of the signal, the target it concerns and the intervention that could make the relation fail.

Why more pairings do not automatically create a loop

Suppose we scale the museum catalogue. One system receives one thousand photographs of cups with captions. A second receives one billion, including broken cups, cups in paintings, cups in hands and cups described in many languages. The larger model should classify and discuss cups far better. Now ask whether this particular cup will slide when pushed from the left. The corpus may contain enough regularities to make a strong guess. But no amount of passive scale makes the answer contingent on an action taken here and now.

This underdetermination has practical signatures. Winoground was designed so that the same words appear in two captions, but their composition differs. Models tested in that work struggled to match relations reliably (Thrush et al.). SugarCrepe later showed that some vision-language evaluations were vulnerable to linguistic shortcuts, to the extent that blind baselines could perform surprisingly well on earlier constructions (Hsieh et al.). These results do not prove that multimodal models lack meaning. They show that success on image-caption association can be generated by routes weaker than the intended relational competence.

Bender and Koller’s distinction between form and communicative meaning remains relevant here, although their argument is contested and should not be treated as an experimental verdict (Bender and Koller). Bisk and colleagues similarly argue that language research must reconnect words to physical and social experience (Bisk et al.). The useful synthesis is narrower: media can carry grounded structure without giving the receiving model independent control over the tests that originally made that structure reliable.

flowchart TD
  A["The two cameras"] --> B["The world can enter without<br/>answering back"]
  B --> C["The loop is the causal upgrade"]
  C --> D["An experiment that separates<br/>contact from theatre"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#B96D00,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

The loop is the causal upgrade

Action contributes more than another output channel when it changes the evidence available to the system. Feedback then becomes the trace of a selected counterfactual made actual.

A passive multimodal learner can estimate a joint regularity such as “which caption accompanies which image?” An action-conditioned learner faces a different problem: “given this observation and this action, what changes next?” The smallest formal distinction is enough to expose the mechanism.

The notation does not make a system grounded. It clarifies what must be tested. In the second expression, the predicted future depends on an action. In a real closed loop, the agent chooses that action, the environment transitions, and the resulting observation returns to the system. The key comparison is not active versus no data. It is active intervention versus a replay condition that supplies comparable sensory material without target-contingent control.

Why can control reveal something that more observation misses? Observational data can often be generated by several hidden mechanisms. A cup may remain still because it is bolted, because nobody pushed it, because the surface is sticky or because the video begins after motion stopped. If all demonstrations follow the same policy, the policy becomes a confound. The learner sees what skilled demonstrators chose to reveal, not every physically possible transition.

Work on representation identifiability gives a related warning. Locatello and colleagues proved that unsupervised observations alone do not select a unique disentangled representation without additional assumptions or supervision (Locatello et al.). Causal representation learning asks whether changes across interventions, environments or mechanisms can supply stronger constraints (Schölkopf et al.). Those results do not imply that every action discovers a causal variable. They explain why an action chosen to discriminate rivals can add information of a different kind.

Active perception has a long history. Bajcsy defined perception as involving modelling and control strategies, not just passive sensing (Bajcsy). Brooks argued for building intelligence through systems that operate in the world rather than through detached representation alone (Brooks). Sensorimotor accounts of perception emphasise regularities linking movement to sensory change (O’Regan and Noë). These programmes disagree about representation, consciousness and architecture. They converge on one engineering lesson: a sensor becomes more informative when the system can move it or move relative to what it senses.

The smallest useful example: the bolted cup

Consider two visually identical cups. One is free. One is bolted to the bench. A passive model observes one hundred demonstrations, but people always lift the free cup and never touch the bolted one. Appearance and action history jointly predict “liftable”. The hidden mechanism remains confounded with the demonstrator’s choice.

Give the system one safe action: a two-millimetre lateral tap. If the cup moves, the system observes displacement. If it does not, the result supports several possibilities: the cup is fixed, the actuator failed, friction is high, or the sensor missed the motion. One intervention is not magic. It simply converts an untested regularity into a hypothesis with rival explanations.

Embodiment is a boundary claim

Where does this loop live? It may include a model, camera, state estimator, planner, actuator, safety controller and environment. Calling the neural network “embodied” can obscure that most of the causal work occurs outside its weights. The model may propose a tap while a deterministic controller executes it. A state service may record displacement. A separate rule may decide whether the evidence is sufficient.

This boundary discipline prevents two opposite errors. One error attributes the whole agent’s world-contact to a foundation model that merely emits proposals. The other denies any digital embodiment because the body is not biological. A software agent in a stateful game, laboratory or enterprise workflow can possess action-contingent coupling if its actions alter an environment that returns discriminating feedback. A robot can lack useful coupling if its actions are scripted and no internal state is revised by the result.

Embodiment is therefore not a material badge. It is a specification of recurrent causal dependencies across a chosen system boundary. Physical bodies add friction, latency, damage, geometry and sensor noise. Digital environments can add persistence, scarcity, adversaries and irreversible actions. Each substrate creates different tests. Neither makes the model conscious by default.

An experiment that separates contact from theatre

A useful grounding test must hold the sensory spectacle constant while changing whether the system’s choice causes the evidence it receives.

Many evaluations compare a text-only model with a multimodal or robotic system. That establishes the value of extra inputs, extra training and extra machinery together. It cannot isolate intervention. The proposed Intervention Grounding Test uses three matched conditions and a hidden mechanism shift.

The environment contains objects whose visible features are correlated with a hidden property during training. Red objects are usually movable; blue objects are usually fixed. At test time, the correlation weakens or reverses. Recognition remains easy. The question is whether a system can determine the property of the current target.

A defensible protocol fixes five choices before results are seen:

1. Declare the hidden mechanism. State exactly what changes between training and test, such as fastening, friction or action latency.
2. Match the evidence budget. Pairing, replay and intervention receive the same number of target views and comparable extra frames.
3. Constrain the action family. Probes must be typed, reversible and strong enough to distinguish the planned rivals.
4. Record the update path. Save the pre-action belief, chosen action, effect receipt, post-action belief and final answer.
5. Inject broken links. Swap targets, replay receipts and freeze state updates to learn which dependency produced the gain.

The primary outcome should be target-property accuracy under shift. Secondary outcomes reveal whether the system acquired that accuracy intelligently: probes per correct decision, information gained per unit cost, time to revise after surprise and recovery after a failed actuator or corrupted observation. A system that achieves high accuracy by applying every possible action has not solved the same problem as one that selects a single discriminating probe.

The pairing condition predicts from appearance. The replay condition sees an equal number of probe-and-outcome frames, but they concern randomly selected objects. The intervention condition chooses a probe on the target and receives noisy feedback. All three answer the same query. The decisive metric is not raw task success alone. It is the performance advantage under hidden mechanism shift after accounting for probe cost and observation budget.

How to interpret the result

A positive result permits a deliberately limited conclusion: under the tested shift, the configured action-feedback loop provides task-relevant information about the current target that passive and replay-matched conditions do not. It strengthens the claim that the system uses an intervention-grounded representation of movability.

A negative result has several meanings. The passive representation may already contain enough information. The probe may be poorly chosen. Sensor noise or action cost may erase the advantage. The system may receive feedback but fail to update the state used for the decision. The test should therefore record intermediate quantities: selected action, pre-action belief, observed effect, post-action belief and final decision.

The test cannot establish that the system experiences the object, cares about the outcome or possesses intrinsic intentionality. It also cannot show broad causal understanding from one affordance. A grounding result is indexed to the intervention family, environment, system boundary and decision it improves.

The strongest evidence comes from a causal fingerprint rather than one aggregate score. If target control matters, swapping the selected target while keeping the probe should reduce performance. If feedback matters, replaying a stale effect receipt should create a predictable error. If state revision matters, freezing the post-action belief should erase the advantage on the next decision. Each failure injection removes one link while preserving the others.

This matters because closed-loop systems can hide dependence inside orchestration. A planner may appear to learn from feedback when a rule-based controller silently retries. A vision module may appear to identify resistance when a force threshold makes the decision. Neither arrangement is illegitimate. The claim simply belongs to the configured system or component that actually carries the dependency. Instrumentation turns that attribution from a story into a test.

from __future__ import annotations

from dataclasses import dataclass
import random


@dataclass(frozen=True)
class Result:
    pairing_accuracy: float
    replay_accuracy: float
    intervention_accuracy: float
    intervention_utility: float


def bernoulli(p: float, rng: random.Random) -> bool:
    return rng.random() < p


def run_test(
    trials: int = 2_000,
    shift: float = 0.75,
    probe_noise: float = 0.08,
    probe_cost: float = 0.05,
    seed: int = 7,
) -> Result:
    """Compare pairing, matched replay and target intervention.

    Training shortcut:
        P(movable | red) = 0.90
        P(movable | blue) = 0.10

    Test shift linearly reverses that relation.
    """
    if not 0 <= shift <= 1:
        raise ValueError("shift must be in [0, 1]")
    if not 0 <= probe_noise < 0.5:
        raise ValueError("probe_noise must be in [0, 0.5)")

    rng = random.Random(seed)
    p_red = 0.90 - 0.80 * shift
    p_blue = 0.10 + 0.80 * shift

    correct_pair = correct_replay = correct_active = 0

    for _ in range(trials):
        target_red = bernoulli(0.5, rng)
        p_target = p_red if target_red else p_blue
        target_movable = bernoulli(p_target, rng)

        # Pairing uses the training-time colour shortcut.
        pairing_prediction = target_red

        # Replay receives an extra probe outcome, but from another object.
        donor_red = bernoulli(0.5, rng)
        donor_p = p_red if donor_red else p_blue
        donor_movable = bernoulli(donor_p, rng)
        donor_observation = donor_movable ^ bernoulli(probe_noise, rng)
        replay_prediction = pairing_prediction
        if donor_red == target_red:
            replay_prediction = donor_observation

        # Intervention probes this target. The sensor can be noisy.
        target_observation = target_movable ^ bernoulli(probe_noise, rng)
        intervention_prediction = target_observation

        correct_pair += pairing_prediction == target_movable
        correct_replay += replay_prediction == target_movable
        correct_active += intervention_prediction == target_movable

    pairing_accuracy = correct_pair / trials
    replay_accuracy = correct_replay / trials
    intervention_accuracy = correct_active / trials

    return Result(
        pairing_accuracy=pairing_accuracy,
        replay_accuracy=replay_accuracy,
        intervention_accuracy=intervention_accuracy,
        intervention_utility=intervention_accuracy - probe_cost,
    )


if __name__ == "__main__":
    print(run_test())

From laboratory loop to configured system

In a realistic warehouse picker, the multimodal model should not directly convert pixels into unrestricted motor commands. The configured system separates what the model proposes from what the authority layer permits. This separation also makes the grounding claim inspectable.

The intent might be “move the requested carton to bay four”. Identity determines which operator and robot are involved. World state records carton location, gripper status and occupied space. Permitted context limits which camera feeds and inventory records may inform the decision. The model produces a proposal, such as “inspect the left edge, then attempt a low-force lift”. A deterministic authority checks force, zone and collision limits before issuing a typed action.

After execution, the system records an effect receipt from joint encoders and vision, performs readback against the world state, and marks a verified outcome. If the carton does not move as predicted, the plan returns to a recoverable state. In this architecture, grounding evidence consists of linked predictions, authorised interventions and observed effects. Fluency is optional; receipts are not.

The same scenario exposes a negative case. If the camera motion occurs but the second image is appended to a log that the planner never reads, the system is physically embodied yet computationally open-loop for that decision. If the planner reads it but a stale world-state cache still marks the carton as free, the grounding failure sits in state integration rather than perception. A configured test must follow the information all the way to the decision.

This architecture aligns with what recent vision-language-action research is trying to make possible. PaLM-E combined language with continuous sensor inputs for embodied tasks (Driess et al.). RT-2 represented robot actions in a token-compatible training scheme and evaluated transfer from web-scale vision-language data to control (Brohan et al.). OpenVLA and the Open X-Embodiment effort expanded access to cross-robot data and policies (Kim et al.) (Open X-Embodiment Collaboration). Later systems continued to combine broad visual-language priors with specialised action generation, including continuous action experts (Black et al.) and the Gemini Robotics family (Gemini Robotics Team).

These are important engineering advances, not a completed theory of grounding. EmbodiedBench reported a wide gap between high-level multimodal competence and low-level manipulation across the systems it tested (Yang et al.). Such findings reinforce the need to decompose performance. A system may recognise the correct object, plan the right subgoal and still fail because control latency, geometry or recovery is wrong. Conversely, successful manipulation may come from imitation without an explicit causal model.

stateDiagram-v2
  [*] --> Claim
  state "Figure 1. The grounding delta" as Claim
  state "Why more pairings do not<br/>automatically create a loop" as Test
  state "Figure 2. Live intervention<br/>versus matched replay" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Where embodiment stops

Action-feedback coupling strengthens some claims and leaves others untouched. A physical body can supply causal discipline without supplying intrinsic meaning, autonomy, valence or experience.

The strongest functionalist response says that the relevant property is causal organisation, not biological material. If a digital agent maintains state, selects actions, receives consequences and revises policy in a sufficiently rich environment, its loop can instantiate the same functional relation even without motors or skin. On this view, a laboratory simulator may be a legitimate embodiment for the claim being tested.

An enactive or biological-naturalist response demands more. A system may need autonomous self-maintenance, metabolism, affective regulation or a history of development in which its own continued organisation is at stake. From that perspective, externally assigned rewards and resettable episodes may produce control without the normativity that makes outcomes matter to an organism. The Intervention Grounding Test does not decide this disagreement. It isolates a weaker, operational property shared by some candidates.

A social-semantic response points out that humans also learn much of language through testimony, pictures and participation in institutions. Direct manipulation is not required for every concept. No individual has personally visited every country they can discuss. Proxy grounding can therefore be genuine participation in a distributed practice, not a defective substitute. The remaining question is which links are preserved and which are merely copied.

These alternatives are strongest when kept distinct. A system can have excellent proxy grounding for historical concepts, intervention grounding for a robot’s local affordances and no evidence of phenomenal consciousness. It can also have poor physical grounding yet strong social grounding through dialogue. Grounding is plural and claim-relative; the mistake is to compress every relation into one binary label.

Three separations are especially useful. A system can have modality without agency, as when it receives rich video but cannot choose the next view. It can have agency without learning, as when a fixed controller acts successfully but never revises its internal state. It can have learning without stakes, as when a resettable simulator updates a policy but no outcome bears on the system’s own continued organisation. Each separation answers a different question.

Physical embodiment sharpens some of these distinctions because the world resists. Motors saturate, objects deform, sensors drift and actions take time. Yet physical resistance does not guarantee epistemic use. A robot may collide repeatedly without learning. Conversely, a digital agent may encounter genuine resistance through permissions, scarce resources, adversarial users or state transitions that cannot be undone. What matters for the present thesis is whether the resistance enters the same loop that selected the action.

Nor should direct action be treated as the only source of meaning. Concepts such as justice, yesterday or prime number depend on social practices, temporal organisation and proof, rather than only on bodily manipulation. Even concrete language is distributed across people. A child can learn that a stove burns through testimony before touching it. Proxy grounding is therefore often the rational route. The role of intervention is to test action-sensitive claims, not to replace testimony, inference or formal demonstration.

Four break conditions

The intervention claim should be withdrawn when any of four links fail. First, the action was not selected from the system’s uncertainty or goal. Second, the action did not causally affect the target. Third, the returned evidence was not specific enough to distinguish rival mechanisms. Fourth, the evidence did not alter the state that governed the next decision.

These break conditions suggest concrete failure injections. Override the selected probe while keeping the observation. Replay the same effect receipt after a different action. Swap sensor streams between targets. Freeze the world-state update. A genuine intervention advantage should degrade in characteristic ways. If nothing changes, the purported loop may be decorative.

Interventions also have costs. A robot that pokes every uncertain object may be well grounded and unsafe. Active tests can damage equipment, invade privacy or consume scarce opportunities. The smallest useful implementation therefore begins with reversible probes, an operational twin, force and scope limits, and a policy for abstaining when the expected information gain does not justify risk.