Part I

The action that pays in information

Begin with a two-door problem. One door is safe and one is dangerous. The safe side is hidden, and the prior probability is one half. Committing to the safe door returns 10 units. Choosing the dangerous door loses 14. Before committing, the agent may pay 1 unit to inspect a noisy indicator that identifies the safe side with 90 per cent accuracy.

Committing immediately has expected return −2: half of 10 plus half of −14. After an inspection, the indicator is correct nine times in ten. Following it has expected terminal return 7.6, and the inspection reduces that to 6.6. The inspection is therefore valuable even though its immediate reward is negative.

A myopic controller sees only the unit cost. A reward planner with a sufficiently long horizon sees the future decision benefit. An active-inference agent can reach the same qualitative result by assigning the policy an epistemic value proportional to the expected reduction in uncertainty. In this symmetric case, the indicator contributes about 0.53 bits of expected information about the hidden safe side.

Thought experiment one: scramble the indicator

Hold every feature of the two-door task constant except the indicator’s accuracy. Set it to 50 per cent. The inspection now leaves the posterior exactly where it began. Its expected information gain is zero, and its expected return is the immediate-commit return minus the inspection cost.

This is the decisive negative control. If an alleged epistemic controller continues to prefer the inspection, the behaviour is not explained by useful information seeking. It may be driven by a generic novelty bonus, a numerical artefact, a prior over policies, or an epistemic weight so large that the agent pursues uncertainty reduction without regard to consequence.

Information is valuable only when it can alter a downstream decision. Entropy reduction by itself is not enough. A camera may reveal the colour of a warning label while leaving every available intervention unchanged. That observation is informative in the Shannon sense but decision-irrelevant in the control sense.

Vary the availability, not the information

Change one feature again. Suppose the diagnostic indicator is continuously visible at no cost before the agent reaches the doors. The posterior still changes, but there is no longer an information-gathering action to choose. Both active inference and a reward planner can condition on the observation. The special control burden has disappeared.

Now make the indicator available only after a door has locked. It remains statistically informative, yet arrives too late to affect the irreversible decision. Its retrospective value may support learning or audit, but its expected information gain should not justify the current policy. Timing therefore belongs in the hypothesis: the information must arrive before the decision whose quality it can improve.

These variations separate three claims that are easily merged. Belief updating matters whenever observations are incomplete. Active sensing matters when an action changes which observations arrive. Epistemic control matters only when that active sensing changes a consequential later choice. The distinctive opportunity is action-conditioned observability before commitment.

Name the uncertainty that the policy is buying down

“Reduce uncertainty” is underspecified. The agent may be uncertain about the current fault state, the transition caused by a reset, a sensor’s reliability, a persistent model parameter or the preferences of a human operator. Information about one target need not transfer to another. A thermal image can discriminate iced coil from fan failure while saying little about whether the reset actuator is jammed.

The experiment should name an epistemic target z and connect it to a decision. One useful quantity is the expected mutual information between a future observation and that target, conditional on policy: I(z; o | π). The target may be hidden state for diagnosis, dynamics parameters for system identification, or another agent’s intent in coordination. The target should not be “the world” in general.

This discipline also limits curiosity. A rover surrounded by hundreds of uncertain variables can always find something to inspect. The controller should value information about variables that enter the decision model, within a horizon where their resolution changes expected consequence. Otherwise, a broad epistemic drive can become an expensive search for facts that are true, novel and operationally useless.

Three claims that must remain separate

The language around active inference often moves across three levels without warning. The free-energy principle is proposed as a very general account of self-organising systems. Active inference is a family of formulations for perception and action under a generative model. A concrete agent is an algorithm with particular factorisations, approximations, policy sets, precision parameters and numerical methods.

Evidence for a maze-solving algorithm does not establish a universal principle of life. A mathematical argument for a free-energy bound does not show that one implementation is a better controller than model-predictive control or RL. Conversely, a failed deep implementation does not by itself refute every active-inference formulation. The engineering claim lives at the algorithm-and-task level.

Part II

What active inference computes

The rover does not observe “iced coil” directly. It receives temperatures, vibration, current draw and images. A generative model states how hidden causes are expected to produce observations and how actions are expected to change hidden state. In a finite POMDP formulation, the common ingredients are:

• s_t, the hidden state at time t;
• o_t, the observation;
• a_t, the action;
• A, a likelihood mapping from hidden states to observations;
• B, transition dynamics under actions;
• D, prior beliefs over initial state;
• C, a distribution or log preference over outcomes; and
• E, where used, a prior over policies or habitual tendency.

These letters are conventions, not magic. The design substance lies in the conditional independences and numbers they encode. A model that says vibration is highly diagnostic of fan failure will behave differently from one that treats it as noise. A preference distribution that sharply penalises unsafe pressure will behave differently from a gentle preference for energy efficiency.

A generative model should follow the decision boundary

A generative model need not reproduce every detail of the warehouse. It must preserve the distinctions that change action. If iced coil and fan failure require different interventions, the latent state should keep them separate. If two visual textures never alter diagnosis or control, modelling them may consume capacity without improving policy.

This is an action-oriented modelling choice. Tschantz, Seth and Buckley showed in simulation how active inference can learn compressed models shaped by action rather than reconstructing the sensory stream in full. The engineering translation is cautious: evaluate a representation by interventions and decisions, not only by prediction loss. A model can predict pixels accurately while erasing a rare fault feature, or reconstruct poorly while retaining the one distinction needed for safe control.

The negative control is representation-sensitive. When the indicator is scrambled, the learned model should recognise that it does not predict the hidden target. If an expressive model memorises incidental correlations in the training set, the apparent information gain may survive the scramble on held-out structure. This is why the likelihood model needs calibration under shifts, not merely a good evidence lower bound during training.

Present evidence: variational free energy

For observations already received, the agent needs an approximate posterior over hidden state. One common variational objective is:

q(s) is the tractable belief distribution; p(o,s) is the generative model; p(s|o) is the exact posterior; and D_KL is Kullback-Leibler divergence. Since the evidence term −ln p(o) is fixed for an observed o, minimising F pulls q towards the posterior.

This is state inference. It explains how the thermal image changes belief about the fault. It does not yet choose the image-taking action, because that action concerns observations that have not arrived.

Possible futures: expected free energy

For a candidate policy π, the agent predicts future states and observations. A common decomposition of expected free energy can be written schematically as:

The first term penalises predicted outcomes that are improbable under the preferred-outcome distribution P*(o). The second is expected information gain, following a widely used decomposition discussed by Friston and colleagues: how much the observation is expected to move beliefs about hidden state. Because this term is subtracted, informative policies can receive lower expected free energy.

The pragmatic term asks, “Will this policy lead towards outcomes I prefer?” The epistemic term asks, “Will it reveal state that matters before I must act?” Their numerical balance determines whether the rover inspects, intervenes or escalates.

An alternative decomposition, under particular factorisations, writes expected free energy as a divergence between predicted and preferred outcomes plus expected observation ambiguity:

The first term is often called risk; the second, ambiguity. Here “risk” means distributional mismatch in a mathematical objective. It does not mean that the agent is secure, lawful, fair or within an operational risk appetite. A controller can minimise this technical risk while exploiting a wrong model, obeying harmful preferences or causing an unverified effect.

The relationship among proposed root definitions and decompositions remains an active technical issue. Millidge, Tschantz and Buckley argued that expected free energy is not simply variational free energy projected into the future. Champion and colleagues later formalised a unification problem and showed that different starting definitions recover different subsets of familiar decompositions and can constrain admissible preferences. For engineering, the exact objective must be named and tested rather than invoked by label.

Where the epistemic term enters the causal story

The second term is often described as intrinsic value, but “intrinsic” can obscure its causal role. In the two-door task, the observation changes the posterior; the posterior changes which door is selected; the changed door selection changes expected terminal utility. Remove any link and the information has no instrumental value.

Expected free energy can still value belief change even when a short reward horizon fails to propagate that chain. That is an algorithmic benefit when the coefficient and target are well chosen. It is also a failure route when the belief change is detached from consequence. The lab below exposes both possibilities by letting epistemic weight move independently of expected return.

A useful trace therefore records more than “information gain = 0.31”. It records the prior over the named target, predicted observation distribution, posterior under each possible observation, action selected under each posterior, and consequence difference between those actions. This is an epistemic decision receipt. It lets a reviewer see whether the inspection bought a changed decision or merely a sharper belief.

Preferences are distributions, not a moral faculty

A scalar reward assigns a number to an outcome or transition. Active inference usually expresses goals as prior preferences over outcomes or states. This can expose distributional structure that a mean reward hides.

Consider two policies with the same expected reward of 4. Policy A returns 4 every time. Policy B returns 10 with probability 0.7 and loses 10 with probability 0.3. A risk-neutral expected-reward objective is indifferent. A sharply concentrated preference distribution around acceptable outcomes can distinguish them through divergence. That is useful when the distribution itself matters, but it does not remove the need to justify the preference shape.

This distinction can also be reproduced in reward-based systems through utility curvature, distributional RL, constrained optimisation or coherent risk measures. Once again, active inference offers a particular representation and objective, not an exclusive capacity.

Thought experiment two: the dark room with one variable changed

A common caricature says that a surprise-minimising agent should enter a dark, unchanging room and remain there. Hold the room fixed, but vary the agent’s preferred trajectories. If it strongly expects or prefers changing proprioceptive, social and task-relevant outcomes, an inert room is itself improbable under those preferences. The policy leaves.

Now hold preferences fixed and change only the likelihood model. Suppose the agent incorrectly predicts that the room contains informative signals about a hidden hazard. It may enter and inspect. If no such signals exist, posterior uncertainty will fail to fall, and a sound replanning process should eventually abandon the policy. A brittle implementation may persist because its model is overconfident or because it confuses novelty with useful information.

The thought experiment exposes two independent levers. Preferences shape which outcomes count as desirable. The likelihood model shapes which observations are expected to resolve uncertainty. Neither is licensed by the free-energy vocabulary. Both are design commitments that require evidence, calibration and revision.

Candidate policies are often converted into a distribution such as:

The precision γ controls how sharply policy probability concentrates on lower expected free energy. Low precision spreads probability across alternatives; high precision approaches a hard choice. In practical systems, this parameter can determine whether the agent explores cautiously, dithers, or commits with unjustified certainty. It belongs in the evaluation manifest.

Part III

flowchart LR
  A["The action that pays in<br/>information"] --> B["What active inference computes"]
  B --> C["The comparison that matters"]
  C --> D["From objective to engineered agent"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#7B4CD8,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

The comparison that matters

“Active inference versus RL” is not one experiment. RL ranges from model-free temporal-difference learning to Bayes-adaptive planning, maximum-entropy control, distributional methods, curiosity bonuses and constrained optimisation. Active inference also varies by state space, factorisation, message-passing scheme, policy enumeration, learning rule and deep approximation.

The companion paper Reinforcement Learning and Sequential Choice develops delayed consequence; this article isolates active sensing under hidden state.

A fair comparison starts from a shared task model. Give both agents the same observations, action set, transition and observation models, prior uncertainty, planning horizon and computational budget. Let the reward-based planner maintain a posterior over hidden state. Add an information-gain baseline such as VIME or another intrinsic-reward method when exploration is central. Then change only the decision objective and the machinery strictly required by it.

This design removes a common illusion. A model-free, myopic learner is a weak baseline for a belief-based planner. If active inference wins only because it receives a model of hidden state and the RL agent does not, the experiment has measured model access, not the policy objective.

Research connecting discrete active inference with dynamic programming shows that reward-maximising policies can emerge under stated conditions. Conversely, RL can incorporate information gain explicitly. VIME, for example, rewards actions expected to change beliefs about environment dynamics. Control-as-inference and maximum-entropy RL further narrow the conceptual distance. The frameworks overlap enough that a behavioural difference must be traced to a precise modelling or objective choice.

Behavioural equivalence does not erase engineering differences

Two controllers can choose the same action while exposing different objects to inspection. An expected-reward planner may store a scalar value and a separate uncertainty bonus. An active-inference implementation may retain predicted outcome distributions, posterior beliefs and preference distributions in one factor graph. That representation can make some audits easier, even when behaviour is equivalent.

The reverse can also be true. A conventional POMDP solver may have mature search, approximation and convergence tools for the task, while an active-inference implementation enumerates policies or depends on fragile variational assumptions. Architectural clarity is a legitimate benefit, but it must be measured separately from decision quality. Useful measures include trace completeness, calibration, ease of preference perturbation, computational cost and reviewer agreement about why a policy won.

Da Costa and colleagues have argued that, under broad constructions, an RL agent can be represented by a behaviourally equivalent active-inference model. That result supports using active inference as an interpretive lens, but it also raises the burden for superiority claims. If either formalism can encode the policy, the practical question is which representation makes the relevant assumptions testable, maintainable and cheap enough for this environment.

Four benchmark traps

First, do not give the active-inference agent the true transition model while forcing RL to learn from sparse reward. Second, do not count training samples while ignoring expensive policy inference at deployment. Third, do not tune preferences on test outcomes and call them priors. Fourth, do not use final return alone when the claimed advantage is calibrated uncertainty or safer information gathering.

A benchmark should publish the model access, policy horizon, candidate-policy count, posterior approximation, preference parameters, intrinsic coefficients, random seeds and wall-clock budget. It should report both task outcome and the proposed mechanism: information acquired, belief calibration, action changes attributable to that information, and failure under the scrambled cue. Without this ledger, a performance difference remains difficult to interpret.

The minimum discriminating experiment

Return to the two-door case and sweep indicator accuracy from 0.5 to 1.0. For each accuracy, sweep inspection cost. Compare three controllers:

1. a myopic reward controller that sees the immediate inspection cost;
2. a Bayes-optimal finite-horizon reward planner that values the information through future reward; and
3. an expected-free-energy controller with explicit outcome preferences and an epistemic term.

The first contrast demonstrates why planning under partial observability matters. The second is the discriminating contrast. If the active-inference and Bayes planners choose the same action throughout, the epistemic decomposition may be explanatory or convenient, but the task has not shown a unique control advantage. If they diverge, inspect the cause: distributional preferences, approximation, policy prior, epistemic weight or model mismatch.

A serious negative result

The preferred story says that epistemic value produces intelligent exploration. Champion and colleagues reported a counterexample in their deconstruction of deep active inference. In a dSprites experiment, the expected-free-energy agent could settle on a single action and fail, while a reward-maximising agent explored. Their analysis characterised the agent as a “contrarian information gatherer”: its information objective could favour observations contrary to current beliefs rather than the behaviour an engineer informally expects from curiosity.

The lesson is broader than one environment. Algebraic labels such as information gain do not guarantee useful exploration once approximation, representation learning and policy optimisation interact. An epistemic term must be evaluated by the decisions and outcomes it changes, not by the virtue implied by its name.

The falsification ladder

A useful active-inference result should survive six progressively stronger tests:

1. Scramble the cue. Destroy the relation between observation and hidden state. The epistemic preference for inspecting should disappear.
2. Remove the epistemic term. Behaviour should change exactly where information was claimed to matter.
3. Match information seeking in RL. Add a value-of-information planner or intrinsic bonus using the same belief target.
4. Perturb the likelihood. Test under overconfident, underconfident and shifted observation models.
5. Perturb preferences. Widen, sharpen and conflict the preferred-outcome distribution.
6. Account for compute. Compare latency, memory, sample use and planning calls at equal budgets.

A result that passes only the first two says that information matters. A result that passes the matched RL baseline says something more specific about the active-inference representation or objective. A result that also survives model and preference perturbation begins to support an engineering choice.

Part IV

From objective to engineered agent

The formal loop is not the operational system. A rover may infer that opening a service panel is informative, yet lack permission to do so. A model proposal is not an authorised action. The controller may believe the reset succeeded, yet the compressor state may remain unchanged. Belief update is not effect verification.

Keep stochastic policy selection inside a deterministic authority and recovery envelope. The model may propose capture_thermal_scan with parameters and expected observations. A capability service checks identity, asset, location, maintenance window, energy limit and action scope. After execution, independent telemetry records an effect receipt and readback. An unknown outcome triggers reconciliation rather than optimistic memory.

Worked scenario: a cold-store inspection rover

The generative model predicts which sensor patterns each fault tends to produce and how interventions change state. The preference model places high probability on stable temperature, no unsafe pressure, verified actuator state and low unnecessary downtime. It places lower priority on battery conservation. A candidate thermal-scan policy has immediate energetic cost but high expected information gain when temperature and current draw are ambiguous.

The design does not permit the agent to manufacture any observation it prefers. It cannot lower apparent risk by disabling a sensor or writing a desired state into memory. Observation channels are typed by provenance. Actuator effects are verified against independent telemetry. Preferences apply to world outcomes, not merely to the model’s own reports.

The action-conditioned observability test

Before introducing expected free energy, ask eight questions. A “no” near the top usually ends the case; a “no” near the bottom identifies work required before a trial.

The fourth question is the hinge. If additional evidence cannot change action, an epistemic term may optimise curiosity rather than control. If it can, compare active inference with the simplest belief-state planner that represents the same possibility.

Preference governance is model governance

Preferences are sometimes described as prior beliefs, which can make them sound observational. In an engineered agent, they encode a normative choice about acceptable futures. Someone must own that choice. Preference provenance should record the decision right, source policy, unit, scope, effective time, uncertainty and conflict resolution. A preference that “temperature remains stable” needs a range, duration, exceptions and priority relative to worker safety.

Likelihoods and transitions need their own evidence ledger. Which faults were represented? Which sensor regimes were absent? How often does a reset alter the hidden state? What changed after maintenance? A well-specified preference distribution cannot rescue a false world model. A calibrated model cannot justify a harmful preference.

Monitor at least four quantities separately: posterior calibration over hidden state, realised outcome frequency against predicted outcome distributions, epistemic actions per resolved uncertainty, and regret against a matched baseline. A rising count of inspections with flat information gain is an early warning that the agent is paying for observations that do not improve control.

Learning must not silently rewrite commitments

An agent may learn likelihoods and transitions online as equipment ages. That adaptation should not silently alter protected outcome preferences or authority. Learning that a sensor is less reliable is an empirical update. Deciding that a higher temperature is now acceptable is a policy decision. The two changes require different owners, evidence and release gates.

Version the generative model, preference model, policy set and precision settings separately. Replay consequential cases when any one changes. A new likelihood can make an old inspection worthless; a sharper preference can make the same posterior trigger escalation; a wider policy set can expose an action never covered by the original authority contract. The assurance case should state which claims expire under each change.

For online adaptation, place bounds around what may update automatically. Parameter posteriors can move within validated ranges, while structural changes, new hidden states and preference changes enter a controlled release process. This preserves the central benefit of active inference, adaptive belief and action under uncertainty, without allowing adaptation to become unreviewed self-authorisation.

A decision record for the trial

The architecture decision record should state the named hidden variable, observation channels, active sensing actions, preferred-outcome distribution, expected-free-energy definition, comparator objectives and stop conditions. It should also record the maximum sensing cost, irreversible actions excluded from autonomy, minimum posterior calibration, latency ceiling and human escalation rule.

Predeclare a rejection threshold. For example: reject the challenger if it does not reduce wrong interventions by at least a specified margin at equal sensing cost, if its advantage vanishes under a matched value-of-information baseline, or if model shift causes epistemic actions to rise while verified diagnostic accuracy falls. A design hypothesis without a rejection condition becomes a vocabulary commitment.

What would change the architecture decision?

Adopt an active-inference challenger when the action-conditioned observability test passes and the challenger improves at least one predeclared outcome: lower regret under partial observability, fewer irreversible errors, faster fault identification, better calibration, or lower sensing cost for equivalent decision quality. The improvement must survive equal compute and model access.

Reject or narrow it when a conventional POMDP planner matches the result with less complexity; an information-bonus RL baseline reproduces the advantage; the epistemic term does not change action; performance collapses under modest likelihood shift; or preference tuning becomes an opaque substitute for reward shaping. A theoretically elegant decomposition is not free. It adds assumptions, numerical machinery and validation burden.