The world does not arrive labelled

A sensor records effects. Photoreceptors respond to patterns of light. Hair cells respond to pressure waves. Skin receptors respond to deformation, temperature and chemical change. None of these signals carries a label saying “puddle”, “voice” or “warm cup”. The organism must infer the hidden causes that could have produced them.

This is the inverse problem of perception. The forward direction is easy to state: a world state, combined with illumination, viewpoint and sensor properties, produces data. The inverse direction is harder because many world states can produce similar data. A dark patch can arise from lower reflectance, lower illumination, a depression, an occluder or a failing sensor. Perception begins where one-to-one decoding fails.

Expectation is a constraint, not an invention

A common reaction is that prior expectation contaminates otherwise pure sensation. That intuition hides an assumption: the data were sufficient before the prior arrived. In an underdetermined problem, some constraint is unavoidable. The useful question is not whether perception uses prior structure, but which structure, at what strength and with what opportunity for correction.

Bayes’ rule gives the smallest useful compression of this idea:

Suppose the rider initially assigns an 80 per cent prior probability to a puddle and 20 per cent to a pothole. A weak edge cue is 70 per cent likely under a pothole and 20 per cent likely under a puddle. The unnormalised support is therefore 0.14 for pothole and 0.16 for puddle. The posterior is roughly 47 per cent pothole, not because the cue was ignored, but because it was not yet decisive. A second, action-created cue that is nine times more likely under a pothole can reverse the balance.

This toy calculation shows why “top-down” and “bottom-up” are incomplete labels. Evidence does not simply flow upward, then lose a battle against belief. Prior structure and current data jointly determine the best-supported cause. Bayesian models have explained families of perceptual phenomena, including motion illusions under uncertainty and the integration of visual and haptic cues.^{2, 4, 5} They are computational accounts of what a well-adapted estimator might do. They do not, by themselves, prove that neurons literally store explicit probability tables.

Reliability decides how much a cue should count

Consider two estimates of an object’s height. Vision says 10 centimetres with a standard deviation of 3 centimetres. Touch says 14 centimetres with a standard deviation of 1.5 centimetres. Under independent, unbiased Gaussian noise, the minimum-variance estimate weights each cue by inverse variance. Vision receives weight 0.2; touch receives 0.8; the fused estimate is 13.2 centimetres.

The result is not “touch wins”. It is “the more reliable cue counts more in this context”. If the room brightens and visual uncertainty falls, the weights should shift. Human observers can combine visual and haptic information in ways close to this reliability-weighted prediction under controlled conditions, and sensorimotor learning can reflect both task statistics and sensory uncertainty.^{5, 6}

The hidden cause is not the only hidden variable

Even when the object category is known, the signal remains ambiguous. A white mug in shade can return fewer photons than a black mug in sunlight. A nearby quiet voice can create a similar sound level to a distant loud voice. The sensory effect depends on the object, but also on illumination, distance, viewpoint, occlusion and the transfer properties of the sensor. These are sometimes called nuisance variables because the immediate task may not care about them. Causally, however, they are not nuisances at all. They are part of the process that generated the evidence.

Imagine a system that has learned a direct map from pixel intensity to surface colour. It performs well under one lamp, then fails when the lamp moves. A generative account instead asks which combination of reflectance and illumination would produce the observed image.³ The difference becomes visible under intervention. Hold the mug fixed and change the light. A representation tied only to intensity should move with the sensor reading. A representation that has separated likely reflectance from lighting should remain more stable, while its estimate of illumination changes.

This separation need not be perfect or consciously available. It may be approximate, task-specific and distributed across a dynamical circuit. The decisive point is that perceptual constancy requires modelling enough of the data-generating process to discount changes that do not belong to the object property being estimated. Action helps because it can vary one hidden factor while leaving another approximately fixed. A head movement changes viewpoint while the road surface remains in place. Turning a cup changes the visible contour while its material persists. Each controlled change gives the system a chance to discover which parts of the signal travel together.

flowchart LR
  A["The world does not arrive labelled"] --> B["A hypothesis must pay rent in<br/>error"]
  B --> C["The observer changes the next<br/>observation"]
  C --> D["Correction can become concealment"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#087A98,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

A hypothesis must pay rent in error

Bayesian inference describes how evidence should alter support among hypotheses. Predictive coding proposes one family of processes that might implement such updating. The distinction matters. A behavioural fit to Bayes’ rule does not identify a neural circuit. Conversely, a neuron that responds strongly to an unexpected event is not automatically a prediction-error unit.

In the canonical picture, higher levels generate predictions of lower-level activity. Lower levels compare those predictions with incoming activity and transmit residual mismatch. The current latent estimate is adjusted until the generative model accounts for the input sufficiently well.^{1, 10, 11} A prediction becomes explanatory only when mismatch can force revision.

Precision is the gain on correction

Raw mismatch is not enough. A smudged lens, a flickering light or a noisy receptor can produce large differences that should not dominate belief. Predictive accounts therefore weight errors by expected reliability, often called precision. In a simple Gaussian model, precision is inverse variance. A precise error channel commands revision; an imprecise one is discounted.

This creates a mechanism for selective attention without making attention identical to expectation. Expectation concerns what input is likely. Precision concerns which error channels deserve influence. The same expected letter can be processed differently when one location is task-relevant and another is not. Conflating the two makes almost any result compatible with the theory: a smaller response becomes successful prediction, a larger response becomes high precision, and no observation can count against the model.

A defensible predictive-coding claim must therefore specify the predicted sign, timing, layer, cell population and manipulation. The influential Rao-Ballard model showed how hierarchical error minimisation can learn receptive-field-like structure and account for contextual effects.¹ Circuit proposals map representation and error functions onto cortical microcircuits, but the mapping remains an empirical programme rather than a settled anatomical fact.^{10, 11, 12}

One correction step with the numbers exposed

A minimal update makes the roles of mismatch and precision concrete. Let μ be the current estimate of a hidden property, such as normalised surface depth. Let y be the measured cue, and let g(μ) be the cue predicted by the current generative model. A local error-correction step can be written as:

Suppose the current depth estimate is 0.20. It predicts a disparity cue of 0.30, but the sensor measures 0.80. The residual is therefore 0.50. If the combined update gain ηπJ is 0.40, the estimate moves by 0.20 to 0.40. If fog lowers precision so that the gain falls to 0.05, the same residual moves the estimate only to 0.225. The signal did not become less surprising in raw magnitude. It became less trustworthy as a reason to revise depth.

The equation assumes locally smooth predictions, approximately Gaussian error and a model whose latent variable can actually explain the cue. Remove the last assumption and error minimisation can converge neatly on the wrong story. If the dark patch is caused by a sensor fault absent from the hypothesis family, changing “depth” cannot repair the model. The residual may stay high, or precision may be reduced until the system stops listening. A well-designed loop therefore needs an explicit misspecification state. Precision weighting should regulate correction, not provide a convenient way to silence evidence that the model cannot explain.

Same report, different generator

Suppose a cue makes participants more likely to report seeing a face in noise. One mechanism changes early sensory representation. Another leaves the sensory representation untouched but moves the decision criterion for saying “face”. A third reallocates attention. A fourth creates demand characteristics. All can increase the same response.

This rival-generator problem is why reports alone cannot locate the effect. Stronger studies add signal-detection measures, orthogonal tasks, temporally resolved recordings, adaptation controls and causal perturbations. Expectation can sharpen or bias decodable sensory representations in early visual cortex under some paradigms, and unexpected stimuli can enhance feature-selective responses in mouse V1.^{8, 20} Yet systematic reviews find that expectation-suppression results vary with design and can be confounded by repetition, adaptation or attention.^{12, 19}

The observer changes the next observation

Most laboratory displays treat the observer as a camera on a tripod. Natural perception rarely does. Eyes saccade. Heads turn. Fingers trace edges. Nostrils sample. Whiskers sweep. The body approaches, withdraws and changes posture. These movements do more than improve a finished percept. They determine the resolution, timing and causal structure of the sensory stream.

Studies of everyday tasks show that gaze is tightly coupled to the next action. People often fixate an object before grasping it, then move their eyes to the next task-relevant location before the hand finishes.¹³ Active-sensing research treats motor and sensory rhythms as a coordinated sampling process rather than separate modules.¹⁴ Closed-loop accounts go further: the relevant process spans brain, sensor and environment until their interaction converges.¹⁵

Action can seek information, not only reward

A saccade has a cost. It consumes time, creates a transient suppression and moves high-acuity foveal vision away from one location towards another. Its benefit is more than that the new location is interesting. A well-chosen saccade makes unresolved hypotheses predict different observations. The same principle applies when a hand rotates an object, a bat emits a call or a clinician changes an ultrasound angle.

This yields a practical distinction. An action can be pragmatic, changing the world towards a desired state, or epistemic, changing the observation so uncertainty falls. Many actions do both. Moving closer to a suspected pothole improves information but also changes collision risk. A model that optimises information alone may behave recklessly; a model that optimises immediate safety alone may never resolve the ambiguity. The sampling policy therefore needs an explicit trade-off among information gain, action cost and irreversible risk.

The system must predict its own sensory consequences

Movement changes sensation even when the external object remains stable. Eye rotation sweeps the retinal image. Speaking creates sound. Touching produces pressure. Without a model of self-generated change, every action would look like an external event.

Corollary discharge, or efference copy, names signals by which a motor command informs sensory processing about expected consequences. In mouse auditory cortex, motor-related input can suppress predictable sounds associated with movement, providing a circuit-level example of this general logic.¹⁶ The mechanism is not simply “ignore self-generated input”. Unexpected consequences of self-action can be highly informative because they reveal slippage, obstruction or a wrong body model.

Active inference is one formal framework that unifies perceptual updating and action selection through a generative model.¹⁷ It is a stronger commitment than the modest claim needed here. The paper’s thesis requires only that action changes evidence and that the system models enough of this change to interpret the result. One can accept that closed-loop mechanism while remaining agnostic about whether all adaptive behaviour minimises a single free-energy objective.

Recent human work also suggests that active sampling can include bodily timing. In one study, perceptual sensitivity to near-threshold visual targets varied with respiration, and participants adjusted breathing when stimulus timing was predictable.²¹ This is intriguing evidence that sampling policy can reach below deliberate eye movements. It remains one study within a broader and developing literature, not a licence to treat every bodily rhythm as a predictive-control mechanism.

stateDiagram-v2
  [*] --> Claim
  state "Expectation is a constraint,<br/>not an invention" as Claim
  state "Reliability decides how much a<br/>cue should count" as Test
  state "The hidden cause is not the<br/>only hidden variable" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Correction can become concealment

A system can reduce observed prediction error in at least three ways. It can improve its hidden-state estimate. It can act on the world so the prediction becomes true. Or it can choose observations that do not expose the mistake. Only the first is unambiguously perceptual correction. The second may be effective control. The third is epistemic failure disguised as success.

This distinction becomes urgent in active systems because the sampler is no longer neutral. A camera that avoids glare may improve signal quality. A camera that avoids every angle where its preferred object label looks wrong may merely protect the label. Low error on self-selected evidence is not evidence of an accurate percept.

The counterfactual correction test

The proposed instrument is simple: evaluate the model after each active-sampling episode on evidence it could have encountered but did not choose. Call this the held-out probe set. It may contain alternative camera poses, touch locations, sensory timings or simulated interventions. Compare error on that set before and after the chosen actions.

A high score strengthens the claim that active sampling improved the latent estimate. A near-zero score means the actions may have gathered redundant evidence or changed the world without learning it. A negative score indicates overfitting, model drift or a policy that made the internal account less transferable. The measure cannot prove that the model represents the world in a human-like way, still less that it is conscious. It tests a narrower property: correction that survives counterfactual sampling.

A fair probe must be partly outside the policy

Held-out testing becomes harder when the agent changes the world. After a robot pushes an object, the alternative camera view that would have existed before the push may no longer be available. A useful protocol therefore records two kinds of counterfactual. The first is an observation counterfactual: what would another safe sensor pose have shown at the same world state? The second is an action counterfactual: what state would have followed a different intervention? The first can often be approximated with synchronised cameras or a simulator. The second requires an operational twin, repeated trials or a causal transition model.

To prevent the policy from designing its own examination, reserve some probes before the episode begins. They need not all be physically executed. A preregistered pose set, a random forced-probe schedule or a blinded evaluator can provide evidence that the active loop did not select only friendly views. The comparison should match the number of samples, sensing time and action cost. Otherwise an information-seeking policy may appear superior simply because it consumed more opportunities.

Consider an inspection arm that chooses three views of a valve. Its chosen views reduce posterior entropy from 1.4 to 0.2 bits. That looks impressive. On four independently selected safe views, however, the updated model assigns lower likelihood than it did before acting. The loop has become confident but less transportable. The Counterfactual Correction Score is negative even though on-policy uncertainty fell. This is the signature the instrument is designed to expose.

Exogenous probes are not always free or ethical. The solution is not to abandon correction testing, but to state the remaining evidence debt. A dangerous angle may be evaluated in simulation, through a redundant sensor, or during a controlled maintenance window. Where none is possible, the system should carry a narrower claim: “consistent on the sampled path”, not “perceptually resolved”.

An executable lab: information seeking versus confirmation

The following synthetic experiment makes the concealment problem visible. Three hidden objects share one common feature and differ at three diagnostic locations. A passive sampler chooses locations randomly. An information-seeking sampler chooses the location expected to reduce posterior entropy most. A confirmation sampler chooses the location most likely to agree with its current favourite hypothesis. Because the common feature is easy to predict, the confirmation policy can achieve low observed mismatch while learning almost nothing.

A positive result permits a narrow conclusion: within this hypothesis family and noise model, the information-seeking policy gathered evidence that improved identification. A negative result could mean that active selection brought no advantage, that probe cost erased it, or that the chosen information criterion was wrong. Neither result establishes how a biological brain implements perception. The lab exposes a causal possibility that prose can hide: an apparently calm error signal may indicate avoidance rather than understanding.

from __future__ import annotations

import math
import random
from dataclasses import dataclass
from typing import Literal

Policy = Literal["passive", "information", "confirmation"]
PATTERNS = (
    (1, 1, 0, 0),  # H1: common feature + diagnostic feature 1
    (1, 0, 1, 0),  # H2: common feature + diagnostic feature 2
    (1, 0, 0, 1),  # H3: common feature + diagnostic feature 3
)


def entropy(p: list[float]) -> float:
    return -sum(x * math.log2(x) for x in p if x > 0.0)


def posterior(p: list[float], location: int, observation: int,
              noise: float) -> list[float]:
    likelihood = [
        1.0 - noise if observation == PATTERNS[h][location] else noise
        for h in range(3)
    ]
    unnormalised = [p[h] * likelihood[h] for h in range(3)]
    total = sum(unnormalised)
    if total <= 0.0:
        raise ValueError("Posterior has zero mass; check the noise model")
    return [x / total for x in unnormalised]


def expected_entropy(p: list[float], location: int, noise: float) -> float:
    p_one = sum(
        p[h] * (1.0 - noise if PATTERNS[h][location] == 1 else noise)
        for h in range(3)
    )
    return sum(
        p_obs * entropy(posterior(p, location, obs, noise))
        for obs, p_obs in ((1, p_one), (0, 1.0 - p_one))
        if p_obs > 0.0
    )


def choose(policy: Policy, p: list[float], noise: float,
           rng: random.Random) -> int:
    if policy == "passive":
        return rng.randrange(4)
    if policy == "information":
        scores = [expected_entropy(p, loc, noise) for loc in range(4)]
        best = min(scores)
        return rng.choice([i for i, score in enumerate(scores)
                           if abs(score - best) < 1e-12])

    current = max(range(3), key=p.__getitem__)
    agreement = []
    for loc in range(4):
        predicted = PATTERNS[current][loc]
        probability = sum(
            p[h] * (1.0 - noise if PATTERNS[h][loc] == predicted else noise)
            for h in range(3)
        )
        agreement.append(probability)
    best = max(agreement)
    return min(i for i, score in enumerate(agreement)
               if abs(score - best) < 1e-12)


@dataclass(frozen=True)
class Result:
    accuracy: float
    residual_entropy: float
    observed_mismatch: float
    diagnostic_probe_rate: float


def simulate(policy: Policy, *, trials: int = 500, samples: int = 4,
             noise: float = 0.18, prior_bias: float = 0.65,
             seed: int = 42) -> Result:
    rng = random.Random(seed)
    prior = [prior_bias, (1.0 - prior_bias) / 2.0,
             (1.0 - prior_bias) / 2.0]
    correct = total_entropy = mismatches = diagnostic = 0.0

    for _ in range(trials):
        truth = rng.randrange(3)
        belief = prior.copy()
        for _ in range(samples):
            location = choose(policy, belief, noise, rng)
            diagnostic += float(location > 0)
            map_h = max(range(3), key=belief.__getitem__)
            map_prediction = PATTERNS[map_h][location]
            true_value = PATTERNS[truth][location]
            observation = true_value if rng.random() > noise else 1 - true_value
            mismatches += float(observation != map_prediction)
            belief = posterior(belief, location, observation, noise)

        correct += float(max(range(3), key=belief.__getitem__) == truth)
        total_entropy += entropy(belief)

    denominator = trials * samples
    return Result(
        accuracy=correct / trials,
        residual_entropy=total_entropy / trials,
        observed_mismatch=mismatches / denominator,
        diagnostic_probe_rate=diagnostic / denominator,
    )


if __name__ == "__main__":
    for name in ("information", "passive", "confirmation"):
        print(name, simulate(name))

What would strengthen or weaken the proposal

The counterfactual correction hypothesis predicts that systems with higher held-out improvement will transfer better to novel viewpoints, resist adversarial occlusion and calibrate confidence more accurately than systems selected only for low on-policy error. Evidence would strengthen it if this relation held after controlling for model capacity, number of samples and action cost. It would weaken if held-out improvement failed to predict any out-of-distribution performance, or if a simpler measure such as total sample diversity explained the effect fully.

A rival explanation is that information-seeking policies merely collect more varied training data, with no special role for perceptual correction. The discriminating experiment matches sample count and diversity while varying whether probes target disagreement among live hypotheses. Another rival is ecological control: skilled behaviour may stabilise through direct sensorimotor coupling without a rich internal probability distribution. The audit does not require a literal Bayesian code. It asks whether the loop remains exposed to observations that could reveal its mistake.

The proposal should not be applied where probes are destructive, unethical or impossible to simulate. A medical biopsy, a dangerous manoeuvre or an irreversible industrial action cannot be treated as a cheap information query. In those domains, the smallest useful implementation is an operational twin or offline replay that evaluates candidate probes before any physical execution.

When every available hypothesis is wrong

Inference can only redistribute support among the causes it can represent. If the true cause is absent, a posterior of 0.98 does not mean the world has supplied 98 per cent certainty. It means one candidate dominates the current closed set. The distinction is easy to lose when software normalises scores automatically.

A practical loop needs at least three misspecification signals. Residuals should remain visible rather than disappearing after normalisation. Independent probes should be allowed to disagree repeatedly with every candidate. The system should also include an unknown or “none fit” path whose cost is lower than confident action under unresolved conflict. In the valve example, paint damage, an unfamiliar attachment or a shifted camera calibration may lie outside the open, closed and occluded hypotheses. More precision on the same model will not add the missing cause.

This boundary also constrains the proposed score. Held-out improvement can show that the estimate generalises across tested probes, but it cannot show that the hypothesis family is complete. The unknown-object condition is therefore not an optional stress test. It asks whether correction ends in better discrimination or merely in stronger commitment to the least-wrong available label.

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: The world does not arrive labelled
  M->>E: Expectation is a constraint, not an invention
  E-->>M: compare rival explanations
  E->>D: Let action give the world a fair chance to refute
  D-->>Q: revise scope or act

Let action give the world a fair chance to refute

When a perceptual system appears accurate, do not ask only whether its final label matches the answer. Ask how the label was earned. Which prior constrained the estimate? Which evidence channel carried precision? Which action selected the next sample? What observation could have overturned the result? Did improvement survive probes the system did not choose?

For neuroscience, this changes experimental design. A study should perturb prior content, sensory precision and sampling policy separately. It should distinguish early representation from decision criterion, and prediction error from adaptation. It should specify a rival circuit that could produce the same response. The target is not a ceremonial victory for predictive coding, but a causal account whose commitments can fail.

For robotics and applied AI, the change is operational. An active-perception module should log the candidate hypotheses, predicted observations, selected probe, action cost, effect readback, residual and held-out correction. Confidence should fall when the model cannot explain diagnostic views, even if the chosen path looked clean. A system that always returns a label has hidden its most important perceptual state: “none of my current causes fits”.

For human judgement, the lesson is equally practical. Familiarity can make an interpretation arrive quickly, but speed is not the same as warrant. The strongest confidence comes from knowing what you looked for that could have changed your mind. A reliable percept is not the hypothesis with the quietest current error. It is the hypothesis that remains best after the loop has sought a fair chance to refute it.

The cyclist’s puddle becomes a useful model because the head movement did more than add pixels. It created a test. The reflection, parallax and boundary motion were consequences that the competing causes predicted differently. Perception succeeded when action opened the world to correction.