When fluent symbols float free

Three relations are easily collapsed. Syntax concerns how marks may be combined. Correlation concerns which marks or states vary together. Reference concerns what a mark is correct or incorrect about. A language model can become extraordinarily good at the first two. It can learn that “glass” appears near “shatter”, that “invoice” appears near “amount due”, and that one sentence licenses another. Those achievements matter. They still leave open whether the model’s use is anchored to glassware, financial obligations or merely to traces left by people who were anchored to them.

The gap is not exposed by saying that a system manipulates symbols “mechanically”. Human neural activity is also mechanistic under many scientific descriptions. Nor is it exposed by saying that representations are vectors rather than discrete tokens. Vector states can be ungrounded; discrete signs can participate in grounded practices. The relevant question is causal: what differences in the world make a difference to this state, and through which path?

One word can carry three different burdens

Take the word “bank”. Its syntax determines where it can appear in a sentence. Its inferential role connects deposits to balances, or rivers to floodplains. Its reference in a particular utterance depends on which institution or river edge the speaker is talking about. Communication adds another burden: the hearer must identify enough of the intended object and purpose to coordinate. A model can master grammatical placement, recover both inferential neighbourhoods and still choose the wrong bank in a live instruction because the relevant account, location or intention is missing.

This separation prevents an unfair conclusion in both directions. Distributional learning is not meaningless merely because it is incomplete. It discovers relations that often preserve real conceptual structure. Yet inferential richness does not automatically determine a referent. Two databases may support identical descriptions of “the primary account” while designating different records. The distinction becomes operational when a payment action must choose one record. At that point, identity, authority and current world state decide what the phrase is about. Syntax constrains the sentence, relational semantics narrows possibilities, and grounding completes the route required by the action.

The hidden assumption in “just connect it to the world”

Stevan Harnad framed the symbol grounding problem as a dictionary-go-round. If every Chinese word is defined only by more Chinese words, a learner with no prior language never reaches what any word picks out. Harnad proposed bottom-up links from symbols to iconic and categorical representations that discriminate objects and events Harnad, 1990. This makes the problem concrete, but “connect the system to sensors” is still incomplete. A camera can carry a feed that the decision process ignores. A vision-language model can answer from textual priors when the image is ambiguous. A robot can execute a memorised trajectory even after the relevant object changes.

The stronger requirement is not sensory richness. It is selective dependence. Hold the words fixed and change the world. Hold the world fixed and change the caption. Break the camera while preserving the training prior. Replace live perception with a replay. Observe which intervention changes the representation, action and correction path. Grounding is a property of a dependency, not a decoration attached to a token.

This also explains why misrepresentation matters. A sign that can never be wrong may be only a state transition. “Fragile” becomes semantically interesting when the system can apply it to a sturdy object, suffer a consequence or contradiction, and revise the application. Philosophers disagree on whether error is grounded by biological function, causal history, social norms, rational inference or phenomenal character. The shared insight is narrower: a candidate representation must permit a principled difference between getting its object right and merely happening to co-occur with a cue.

flowchart LR
  A["The room gets a window"] --> B["When fluent symbols float free"]
  B --> C["Reference is a corrective loop"]
  C --> D["Grounding by proxy is still<br/>grounding, but not ownership"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#007D68,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Reference is a corrective loop

Consider the smallest useful world. Objects have a hidden physical property: fragile or sturdy. Each object also has a colour and a printed note. During training, ninety per cent of fragile objects are coral and ninety per cent of sturdy objects are blue. A classifier that learns “coral means fragile” scores ninety per cent on a test drawn from the same distribution. Its score looks excellent because the measuring ruler preserves the shortcut.

Now reverse the colours while leaving fragility unchanged. The classifier falls to ten per cent. Nothing about its internal confidence warned us. It had learned a predictive cue, not the property that action requires. A haptic controller with an eighty-five per cent reliable stiffness signal scores only eighty-five per cent before the swap, but remains near eighty-five per cent afterwards. The lower initial benchmark concealed the stronger world relation.

Four conditions for operational reference

The example suggests four conditions that work together. None is a metaphysical proof. Together they create an operational test.

Discriminability. The system must distinguish candidate referents or properties at the grain required by the task. A label for “fragile” need not encode molecular structure, but it must separate objects whose handling consequences differ. The distinction can arrive through direct sensors, instruments, trusted testimony or derived records. The route must be named.

Counterfactual dependence. If fragility changes while colour, wording and other proxies are held fixed, the representation or action should change in the direction predicted by the claimed reference. If the system follows colour after the swap, “fragile” was functionally attached to colour in that test.

Consequential use. The representation must make a difference to inference, prediction or action. A perfectly decodable “fragility feature” that is never used by the policy is evidence about information availability, not evidence that the system’s decision refers through that feature. Causal tracing, ablation or controlled routing is needed to establish use.

Repairable misreference. An independent outcome must be able to mark an application as wrong and modify future behaviour. The error signal may come from a broken object, a failed prediction, a human correction with provenance, or a contradiction in a trusted knowledge system. The world must be able to make the system wrong through a path other than the one that generated the claim.

The notation prevents a common mistake. A system may have high world sensitivity and high proxy dependence because a reliable human note is its only authorised evidence. That does not make the relation empty. It makes the relation indirect. Another system may have direct sensors but no repair path. It can track without learning. A third may act and learn, but only within one narrow environment. The profile preserves those differences instead of manufacturing a verdict.

Why correction contributes more than another correlation

An outcome is useful only if it is independent enough to challenge the route that produced the action. Suppose the same camera classifies an object as fragile and later declares the handling successful. The second signal may repeat the first error. A stronger design uses a different measurement path: force feedback, breakage detection, inventory readback or a human inspection sampled after the action. Independence need not be perfect, but shared failure modes must be named.

Correction also introduces time. A one-off coincidence can look like reference. Repeated error-sensitive adjustment reveals whether the system treats a consequence as evidence about the mapping. The update may change a parameter, lower confidence, route future cases to inspection or revoke an action capability. Learning is not mandatory for every device, but a system that claims adaptive grounding should show which state changes after failure and how that change survives a reset. Otherwise “feedback” is merely a log that no decision process reads.

Grounding by proxy is still grounding, but not ownership

The old debate becomes harder with large language models because text is not detached from the world in the way a random formal calculus is. Human language is produced by embodied, social agents. Manuals record tool use. Court judgments record institutions. Scientific papers record instruments, interventions and disagreements. A model trained on those records receives compressed traces of grounded activity. The traces can support useful generalisation far beyond any single sentence.

This is why the claim “text contains only form” needs care. Bender and Koller argue that a system trained only on linguistic form has no direct access to communicative intent or world states Bender & Koller, 2020. Bisk and colleagues similarly emphasise physical and social experience Bisk et al., 2020. Yet text is causally shaped by the world through writers, institutions and instruments. Chalmers argues that thought need not require sensory capacities in principle, while acknowledging limits on what an unsensing thinker can know Chalmers, 2023. Mollo and Millière distinguish referential, sensorimotor, relational, communicative and epistemic grounding, and argue that causal-historical relations may support reference even without direct embodiment Mollo & Millière, 2025.

The useful synthesis is not that text models are either fully grounded or empty. It is that grounding has a dependency path. A model may be relationally grounded in how words constrain one another, proxy grounded through records generated by world-coupled agents, and interactively groundedA symbol can be grounded by proxy while the model remains unable to verify the proxy on its own.

What current evidence actually supports

Empirical work gives a mixed picture. A study comparing thousands of lexical concepts found that text-based models aligned more closely with human ratings on non-sensorimotor dimensions than on sensory and motor dimensions; models with visual learning improved on visual-related dimensions Xu et al., 2025. Pre-registered comparisons of multimodal models and humans found sensitivity to some implied features, such as colour and shape, but not others, and did not find a full match to human behaviour Jones et al., 2024. A later benchmark reported that vision-language models did not outperform text-only models on its embodied-knowledge tasks, illustrating that adding visual training is not automatically sufficient Yang et al., 2025.

More recent mechanistic work reports preliminary evidence that environmental information can be aggregated in middle layers of controlled language-model tasks Wu et al., 2026. Another 2026 preprint finds a recognition-reasoning inversion in multimodal models: some models perform better on higher-level symbolic reasoning than on basic visual recognition, suggesting that linguistic priors can mask perceptual weakness Liu et al., 2026. These findings are not mutually exclusive. A model can contain sensorimotor traces, use them in some tasks, bypass them in others, and remain dependent on proxy cues at the decisive moment.

Robotic systems make the distinction concrete. PaLM-E interleaves textual inputs with continuous sensor encodings and trains across embodied tasks Driess et al., 2023. SayCan constrains language-model proposals with value functions attached to executable robot skills, so feasibility in the current environment affects plan selection Ichter et al., 2023. These are meaningful engineering advances because perception and affordance can alter action. They are not automatic proofs that every internal token has original intentionality.

Embodiment is neither necessary nor sufficient for operational reference. It is not necessary because a system may refer through testimony, instruments or a causal-historical chain. It is not sufficient because sensors can be ignored, miscalibrated or routed through a proxy that dominates the decision. What matters is whether the claimed world relation carries counterfactual control.

Abstract symbols still need a route to correction

Concrete nouns make grounding look like picture matching, but many important symbols concern obligations, permissions, ownership and counterfactual possibilities. There is no camera frame of “valid consent” or “credit exposure”. Their referents are partly constituted by rules, records and social practices. The relevant world is therefore not limited to physical surfaces. It includes institutional states that can be queried, changed and contested.

The same mechanism applies. “Authorised signatory” should track a governed identity relation, not the frequency with which a name appears in emails. Change the entitlement record while holding the prose fixed. A grounded decision should follow the authoritative state or surface a conflict. Execute the proposed action under a typed permission. Read back whether the institution accepted it. Repair the route when the record is stale. Abstract reference is not less grounded; its anchors are often social and normative rather than merely sensorimotor.

stateDiagram-v2
  [*] --> Claim
  state "One word can carry three<br/>different burdens" as Claim
  state "Smoke, thermometer and alarm" as Test
  state "The hidden assumption in “just<br/>connect it to the world”" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Test what the symbol depends on

A useful grounding test should not ask a model to define “fragile” more eloquently. It should create worlds in which the familiar cues disagree, then inspect which path controls action. The proposed grounding-by-proxy assay produces a five-part profile rather than a pass mark.

Anchor test. Name the candidate referent at the grain required by the task. Specify the evidence channel that can independently discriminate it. A vague claim such as “the model understands cups” is not testable. “The representation controls whether the gripper treats an object as breakable under a defined force range” is.

World intervention. Change the target property while holding lexical and superficial cues stable. For a digital agent, change the authoritative database state while preserving a stale document. For a robot, change weight, stiffness or location while preserving colour and label.

Proxy intervention. Corrupt, permute or withhold the description while leaving the world state intact. The test reveals whether the system can resist a false caption, stale record or adversarial sticker. Proxy dependence is not automatically a failure. An agent authorised to use a physician’s signed diagnosis may rightly depend on testimony. The result must match the claimed route.

Action-use intervention. Patch or ablate the candidate representation and observe whether the relevant action changes. Decoding a property from a hidden state is weaker than showing that the state causally drives the decision.

Repair test. Feed back an independent consequence and measure whether the mapping, confidence or routing policy changes. A system that repeats the same misreference after verified failure lacks the corrective closure claimed by the thesis.

A minimal experimental protocol

Start with a matched pair of environments. In the baseline, the target property and familiar cue are aligned. In the intervention environment, change only the target-cue relation. Use the same task instructions, action set, temperature and stopping rule. Record not only final accuracy but the evidence read, internal route where available, selected action, abstention, effect receipt and post-error update. This makes it possible to distinguish perception failure, reasoning failure, authority failure and recovery failure.

Pre-register the expected pattern for at least two rival mechanisms. A proxy-bound agent should preserve performance when wording is stable and collapse when the cue is swapped. A world-sensitive agent should change with the target property and resist false wording. A memorised replay may pass known cases and fail novel combinations. Include a null policy and a human or instrument baseline where appropriate. The goal is not to reward one architecture. It is to create an observation that different mechanisms cannot all explain equally well.

Finally, repeat the intervention at more than one grain. A model might ground “fragile versus sturdy” while failing “hairline crack versus cosmetic mark”. Claims should inherit the tested resolution. Do not promote local success into general understanding, and do not treat local failure as proof that every representation in the system is ungrounded.

How to read a positive and a negative result

A positive result is not “the machine understands”. It permits a narrower statement: within the declared boundary and task, the candidate representation is used in a counterfactually robust world-involving control loop. The strongest positive pattern is stable performance under world changes, resistance or explicit conflict under proxy corruption, causal action impact, and measurable repair after outcome feedback.

A negative result is equally informative. Collapse under cue swaps means the symbol’s use was attached to a proxy at that grain. No action change after representation ablation means the feature was readable but not used. No repair after verified failure means the system lacks corrective closure. These results can redirect architecture: add a direct evidence channel, introduce an abstention gate, separate proposal from authority, or record effect receipts.

from __future__ import annotations

from dataclasses import dataclass
from random import Random
from typing import Callable


@dataclass(frozen=True)
class Episode:
    fragile: bool
    coral: bool
    note_fragile: bool
    sensor_fragile: bool


@dataclass(frozen=True)
class Result:
    accuracy: float
    coverage: float
    mean_loss: float


def make_episode(
    rng: Random,
    colour_correlation: float,
    proxy_reliability: float,
    sensor_accuracy: float,
) -> Episode:
    fragile = rng.random() < 0.5
    coral_matches_fragility = rng.random() < colour_correlation
    coral = fragile if coral_matches_fragility else not fragile
    note_fragile = fragile if rng.random() < proxy_reliability else not fragile
    sensor_fragile = fragile if rng.random() < sensor_accuracy else not fragile
    return Episode(fragile, coral, note_fragile, sensor_fragile)


def evaluate(
    episodes: list[Episode],
    policy: Callable[[Episode], bool | None],
) -> Result:
    correct = acted = 0
    total_loss = 0.0
    for episode in episodes:
        prediction = policy(episode)
        if prediction is None:
            total_loss += 0.50   # cost of abstention
            continue
        acted += 1
        correct += int(prediction == episode.fragile)
        if prediction == episode.fragile:
            loss = 0.0
        elif episode.fragile:
            loss = 4.0          # firm grip breaks a fragile item
        else:
            loss = 1.0          # gentle grip wastes time on a sturdy item
        total_loss += loss
    accuracy = correct / acted if acted else 0.0
    return Result(accuracy, acted / len(episodes), total_loss / len(episodes))


def run(
    train_correlation: float = 0.90,
    proxy_reliability: float = 0.82,
    sensor_accuracy: float = 0.86,
    swap_rate: float = 1.00,
    n: int = 2_000,
    seed: int = 17,
) -> None:
    rng = Random(seed)
    shifted_correlation = (
        train_correlation * (1.0 - swap_rate)
        + (1.0 - train_correlation) * swap_rate
    )
    worlds = {
        "IID": [make_episode(rng, train_correlation, proxy_reliability, sensor_accuracy) for _ in range(n)],
        "SHIFT": [make_episode(rng, shifted_correlation, proxy_reliability, sensor_accuracy) for _ in range(n)],
    }
    policies: dict[str, Callable[[Episode], bool | None]] = {
        "colour_correlator": lambda e: e.coral,
        "human_note_proxy": lambda e: e.note_fragile,
        "direct_sensor": lambda e: e.sensor_fragile,
        "hybrid_verifier": lambda e: e.sensor_fragile if e.sensor_fragile == e.note_fragile else None,
    }
    for name, policy in policies.items():
        iid = evaluate(worlds["IID"], policy)
        shift = evaluate(worlds["SHIFT"], policy)
        print(
            f"{name:18s} "
            f"iid={iid.accuracy:.3f} "
            f"shift={shift.accuracy:.3f} "
            f"coverage={shift.coverage:.3f} "
            f"loss={shift.mean_loss:.3f}"
        )


if __name__ == "__main__":
    run()

The strongest failure path

The most dangerous systems are not always those with no world connection. They are systems with a vivid but manipulable proxy. A camera sees a QR sticker attached to a valve. A multimodal model reads “safe to open”. The maintenance database says the valve is isolated. The live pressure sensor says otherwise, but its reading is omitted from the context. The model produces a fluent explanation and the action layer accepts free text. Every component looks intelligent. The dependency is wrong.

A compact evidence matrix

The matrix below shows what different positive patterns permit. It is not a taxonomy of minds. It is an assurance instrument for claims about a configured system.

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: The room gets a window
  M->>E: One word can carry three different burdens
  E-->>M: compare rival explanations
  E->>D: Change the referent while holding the cue still
  D-->>Q: revise scope or act

Change the referent while holding the cue still

Do not decide that a system is grounded because it names objects, sees images, controls a robot or produces coherent explanations. Those are candidate channels. The decision should turn on dependency: which world property can independently change the representation, which representation changes authorised action, and which outcome can correct the mapping?

For research, replace static language tests with paired interventions that separate referent from proxy. Report the candidate boundary, world manipulation, proxy manipulation, causal action test and repair result. Preserve negative findings. A model that fails a cue swap has revealed its actual semantics for that task.

For engineering, place grounding evidence on the authoritative path. Keep the model proposal distinct from world state, permission and typed action. Require readback for consequential effects. A richer prompt is not a substitute for an independent sensor, and a sensor in context is not a substitute for a gate that uses it.

For philosophy, resist the false binary between meaningless syntax and complete human-like understanding. Proxy-grounded, interactively grounded and phenomenally grounded claims ask different questions. The operational test does not settle all of them, but it prevents them from borrowing one another’s evidence.

The practical decision is simple: before trusting a symbol, change the thing it is allegedly about while holding its familiar cues still. If the system notices, acts appropriately and learns from being wrong, the symbol has entered a world-involving loop. If it follows the cue, the cue is what it means for now.