The price of changing course
Imagine a commuter arriving at a station with three ways home. The familiar bus is slow but effortless to select. A train is quicker, although using it requires checking a platform change. A bicycle hire promises the earliest arrival, provided the rider notices a temporary road closure and plans a diversion. Maximum expected utility treats the available estimates as if selecting the largest one were free. It chooses the bicycle whenever its expected benefit is even slightly higher. The commuter’s actual problem contains another quantity: how much must the current decision process change to produce that choice reliably?
Pedro Ortega and Daniel Braun’s 2010 preprint gives this question a precise, deliberately abstract answer. It represents behaviour by probability distributions and measures the cost of moving from a reference distribution to a modified one. A reference is the chooser before the new objective is imposed: a habitual policy, a baseline controller, or a distribution encoding current knowledge. A target utility says what the transformed system should favour. The selected distribution must gain utility without making an arbitrarily expensive informational departure from the reference.
The proposal begins at a deeper level than the familiar softmax rule. The authors first ask what mathematical relationship could connect the probability of an event with its utility if utility is real-valued, additive across conditional conjunctions and ordered exactly as probability is ordered. The answer is logarithmic. They then show that the corresponding probability distribution uniquely maximises a free-utility functional. Only after that axiomatic bridge is built do they introduce a reference distribution and interpret relative entropy as the cost of transformation.
This order matters. One could write down an exponential choice rule because it is convenient. Ortega and Braun instead show why an exponential arises from stated structural commitments. Their claim is conditional: accept the proposed relationship between desirability and generative propensity, plus the three utility properties, and the logarithmic conversion follows. The framework does not prove that every organism, institution or program literally prices thought in Shannon bits. It proposes a coherent formalisation whose consequences can be derived and tested.
Thought experiment 1 · Two commuters, one timetable
Setup. Ada and Ben assign the same utilities to bus, train and bicycle. Ada’s reference policy strongly favours the train because she already knows its platform. Ben’s favours the bus because he has never used the ticket machine. With the same temperature, should they receive the same recommendation distribution? The framework says no. Target utility alone does not determine bounded choice; the cost is relative to each chooser’s starting distribution. A policy can therefore be rational relative to one baseline and expensive relative to another.
Three axioms that carry the argument
The paper initially restricts attention to finite probability spaces. An event is a set of outcomes, and a conditional event such as A given B has a probability whenever the conditioning is defined. The authors seek a utility function U assigned to these conditional events. Their first requirement is modest: utility takes real values. It excludes vector-valued or partially ordered desirability at this stage, allowing comparisons and addition to use ordinary arithmetic.
The second requirement is additivity up to a translation constant β. For events A, B and C, the utility of A and B given C equals the utility of A given C plus the utility of B given A and C, less β. Subtracting β from every utility makes this strictly additive. The form mirrors the product rule of conditional probability: joint probability is a product, while joint utility is a sum. Any conversion between the two must therefore turn products into sums.
The third requirement is monotonicity in both directions. One conditional event has higher probability than another if and only if it has higher utility. This is stronger than saying that probability and utility happen to correlate. It makes utility an order-preserving representation of probability. In the authors’ interpretation, a free stochastic system produces an outcome more often exactly when it is more desirable. That teleological reading is an assumption of the construction, not a neutral property of every probability distribution.
Together, additivity and monotonicity leave little freedom. The logarithm is the familiar function that converts multiplication into addition, and the proof adapts an argument used in Shannon’s entropy theorem. The arbitrary positive scale α converts units of information into units of utility. The arbitrary translation β chooses the utility origin. The positivity of α preserves the probability order; a negative value would reverse monotonicity and is excluded.
Additive: U(A∩B|C) = U(A|C) + U(B|A∩C) − β
Monotonic: P(A|B) > P(C|D) ⇔ U(A|B) > U(C|D)
Worked example 1 · Why multiplication asks for a logarithm
Let two conditionally independent stages each succeed with probability one half. Their joint success probability is one quarter. Set β to zero and use base-two logarithms, as the paper does. If α = 3 utility units per bit, each stage has utility 3 log₂(1/2) = −3. The joint event has utility 3 log₂(1/4) = −6, exactly the sum. A linear mapping such as U = 3P would give 1.5 + 1.5 = 3 for the stages but 0.75 for the joint event, violating additivity.
The utility-information equivalence
Theorem 2 states the conversion law U(A|B) = α log₂ P(A|B) + β, where α is strictly positive. Rearranging yields probability as an exponential of utility relative to the utility of the certain event. The pair is called conjugate: given one member, the other is fixed after selecting scale and origin. Since self-information is h(A|B) = −log₂ P(A|B), conjugate utility can also be written as β − αh. Higher surprise corresponds to lower utility under the paper’s interpretation.
At the atomic level, this produces a Gibbs distribution. Assign each outcome an energy equal to negative utility. Probability is then proportional to the exponential of negative energy divided by α. In statistical mechanics α behaves like temperature: low temperature concentrates probability on the highest-utility, lowest-energy outcomes; high temperature spreads mass more evenly. The paper borrows this mathematical structure while giving utility a decision-theoretic interpretation.
One subtlety deserves care. The initial theorem links a single probability measure to its conjugate utility. It does not yet compare behaviour with a non-uniform reference. With no reference term, high α moves the Gibbs distribution towards uniformity. The bounded-rational transformation comes next. Once an initial distribution Pᵢ is present, the final distribution Pᶠ is an exponential tilt of Pᵢ, and high α preserves Pᵢ. Confusing these two settings makes the limit statement appear inconsistent when it is not.
The logarithmic relation also clarifies scale. Multiplying α changes how many utility units correspond to one bit. Adding β changes none of the orderings or probabilities after normalisation. Thus the behaviourally important quantities are utility differences divided by α. A ten-unit advantage can be decisive at α = 0.5 and mild at α = 20. Temperature does not add a second preference ranking; it controls how sharply the existing ranking can move probability away from its baseline.
Thought experiment 2 · The duplicate menu
Setup. A café lists tea once and coffee once. It then prints tea twice under two indistinguishable names. If the chooser treats entries as separate atomic outcomes, the total prior probability of tea may rise even though no new drink exists. The Gibbs conversion operates on the represented outcome space. It cannot decide which descriptions should count as distinct. Representation therefore precedes optimisation: partitioning outcomes can change the policy unless the model preserves the intended event probabilities.
Free utility as a variational principle
For a finite random variable X, take a utility function U and an arbitrary trial distribution Q. The free-utility functional adds expected utility to α times entropy. Because the paper uses base-two logarithms, entropy is measured in bits. Theorem 3 proves that the conjugate distribution P uniquely maximises this functional. Jensen’s inequality supplies the bound, and equality holds precisely when Q equals P.
P = arg maxQ J(Q; U)
The sign can look surprising because the second sum is non-positive. Subtracting it adds entropy. When no reference distribution is specified, free utility rewards both high expected utility and uncertainty. Low α makes utility dominate, concentrating on an optimum. High α makes entropy dominate, approaching a uniform distribution. This is the equilibrium result for a single conjugate pair.
The variational view is more useful than merely inverting U = α log P + β. It turns a static conversion into an optimisation problem. Constraints can be added as a target utility, and the system’s probability measure can be recalculated by maximising the new functional. This creates a common language for deciding, estimating and controlling. The same calculus identifies which distribution should change and which should be held fixed.
Free utility also reveals the role of normalisation. Exponentiating utility values produces non-negative weights but not usually probabilities. Their sum is the partition function Z. Dividing by Z makes the weights sum to one. The logarithm of Z gathers the value of all available alternatives into a single quantity. In sequential control, related normalising constants become soft value functions computed backwards through time.
Worked example 2 · A two-action equilibrium
Let stay have utility 0 and switch utility 2, with α = 2 and no non-uniform reference. The unnormalised weights are 2^(0/2) = 1 and 2^(2/2) = 2. The partition function is 3, so the conjugate probabilities are 1/3 and 2/3. Maximum expected utility alone would put all mass on switch. Free utility retains uncertainty because a two-unit gain is only one information-temperature unit.
From a reference policy to a bounded policy
The decisive step adds a target utility U* to an initial conjugate system (Pᵢ, Uᵢ). The final utility is Uᶠ = Uᵢ + U*. Rewriting the difference between final and initial free utility yields expected target utility minus α times the relative entropy from the final distribution to the initial one. The initial distribution now acts as a reference. Changing a high-probability habit modestly is cheap; assigning substantial probability to an initially rare action is expensive.
P*(x) = P₀(x)·2^(U*(x)/α) / Z
Relative entropy is directional. DKL(Q || P₀) averages log(Q/P₀) under the proposed final distribution Q. A final policy pays heavily for mass placed where the reference assigned little. If P₀(x) is exactly zero, no finite utility can give x positive mass in the exponential tilt. The support of the reference is therefore a hard feasibility boundary. Boundedness here includes both a graded price and an absolute exclusion of initially impossible actions.
Temperature α determines the exchange rate. As α approaches zero, information cost vanishes relative to target utility and the solution approaches a point mass on an action of maximal U*. As α grows, changing the reference becomes expensive and P* approaches P₀. The preprint’s α is the reciprocal of the inverse-temperature parameter often used in later softmax presentations. Keeping that convention prevents a common reversal of the limits.
The solution is an exponential tilt. Each reference probability is multiplied by a utility factor, then all weights are normalised. This factorisation separates two sources of behaviour. P₀ expresses what the agent can or tends to do before the new objective. The exponential factor expresses the pressure of the new objective under the available resource scale. Observing P* alone generally cannot identify these components without additional assumptions.
Worked example 3 · Same utilities, different references
Suppose actions A, B and C have target utilities 0, 1 and 2, with α = 1. Under reference P₀ = (.6, .3, .1), their weights are .6, .6 and .4, giving P* = (.375, .375, .25). Under a uniform reference, the weights are 1/3, 2/3 and 4/3, giving P* = (1/7, 2/7, 4/7). The utility ranking is unchanged. The bounded policies differ because the required transformations differ.
Thought experiment 3 · The action outside support
Setup. A rescue robot’s reference controller assigns zero probability to opening a roof hatch because that actuator was absent during training. A new target utility gives roof escape an enormous reward. The tilt still assigns it zero probability. Is this irrational? Within the specified action space it is a faithful consequence: relative entropy makes new support infinitely costly. For design, the lesson is severe. Exploration or model expansion must enter the reference before optimisation can use it.
From isolated choices to input-output streams
A single menu is only the paper’s entry point. The intended agent is a probability measure over a finite sequence of random variables. Some variables are inputs received from an external system; others are outputs generated by the agent. Conditional probabilities over outputs are propensities: they prescribe what the agent does. Conditional probabilities over inputs are plausibilities: they represent what the agent believes it may observe.
This mixture creates a causal problem. An observed input and a generated output can occupy similar positions in a probability factorisation, yet conditioning on an observation is not the same operation as intervening to choose an action. The authors introduce causal models and an observation operator to distinguish disclosed and undisclosed inputs, and to keep seeing separate from doing. This is necessary before the free-utility transformation can be applied to an interacting agent without treating its own actions as ordinary evidence.
For a multi-step system, the reference P₀ contains current knowledge and baseline action propensities. The target utility may apply across the entire interaction history. The variational objective uses auxiliary measures that select the candidate distribution for controlled variables and the reference distribution for estimated variables, with the roles reversed in the relative-entropy term where appropriate. The notation is dense because control and inference face opposite directions of variation.
The payoff is a unified construction. Controlled probabilities are chosen to improve future utility while paying for deviation from the baseline controller. Estimated probabilities minimise relative-entropy risk against the unknown source. Adaptive control mixes over a family of environment-specific controllers using posterior belief. These are not three casual analogies. They arise by choosing which parts of the I/O system are given and which are varied.
Thought experiment 4 · The umbrella that predicts rain
Setup. An agent often carries an umbrella when rain is likely. Observationally, umbrella and rain are correlated. If it treats its chosen umbrella as ordinary evidence, it may conclude that forcing itself to carry one raises the chance of rain. The paper’s causal distinction blocks that mistake: an output generated by intervention is not disclosed evidence about the environment in the same way as an observed forecast. A bounded controller still needs correct causal bookkeeping.
Control, estimation and adaptive control
In optimal control the environment is known, the utility is given and the action policy is the unknown. The multi-step free-utility objective can be expanded recursively. At each action node, the optimal probability is the reference action probability multiplied by an exponential of immediate utility and expected continuation value, then normalised. The log partition function acts as a soft value. As α approaches zero, this recursion becomes ordinary dynamic programming with a maximum over actions.
At finite α the controller remains stochastic. This is not noise added after an optimal deterministic policy has been found. Randomness is part of the optimum because concentrating probability carries information cost relative to the baseline. A more decisive controller can be worth its price when utility differences are large. When differences are small, preserving the reference can produce greater free utility.
Adaptive estimation reverses what is unknown. An environment parameter determines an observation source, and the agent must form a predictive distribution. With no target utility, minimising relative entropy produces the Bayesian predictive mixture: average each parameter-specific observation model under the posterior probability of that parameter. This section shows that the framework contains a familiar estimator, though the result relies on the chosen probabilistic model and prior.
Adaptive control combines the two tasks. The environment is unknown but belongs to a set of possible environments, and a controller is available for each parameter. The resulting action rule mixes the parameter-specific controllers according to posterior belief. The paper identifies this with the Bayesian control rule developed in related work. Sampling from the predictive mixture can address bandits and other adaptive problems in which acting both earns utility and changes the evidence available later.
How the recursion prices future choices
A one-step tilt can be solved by inspection, but a sequence introduces a dependency: today’s action changes which observations and later choices become possible. The paper expands the trajectory objective from the final time backwards. At the last action, the score contains its immediate target utility plus the expected utility of the last observation. Exponentiating that score, multiplying by the reference action probability and normalising gives the terminal policy. Its partition function summarises the bounded value of that decision point.
The same construction then moves one step earlier. The earlier action receives its immediate utility, expected observation utility and the logarithm of the later partition function. That last term carries forward the value of all bounded future policies, including their information costs. Repeating the operation produces a stochastic counterpart of Bellman recursion. The policy remains history-dependent because every action probability is conditioned on the preceding action-observation string. Resource sensitivity is propagated through the tree rather than charged once at the root.
This explains the maximum-expected-utility limit precisely. As α approaches zero, a log-sum-exponential under the relevant scale approaches a maximum. The soft value becomes an ordinary value function, and the policy collapses onto an action maximising immediate reward plus expected continuation value. At finite α, alternatives below the maximum retain probability according to both their values and their reference propensities. At high α, the continuation calculation remains, but its ability to move the policy away from P₀ fades.
Worked example 6 · A two-stage information bill
At the last step, suppose actions left and right have reference probabilities .5 and .5 and continuation utilities 0 and 2, with α = 2. Their factors are 1 and 2, so the bounded terminal policy is 1/3 and 2/3. Its partition function is 1.5 after retaining the .5 reference weights. An earlier action that opens this branch inherits the corresponding soft continuation value, 2 log₂(1.5), rather than simply inheriting the terminal maximum of 2. The difference records the value lost when concentrating the final choice is informationally costly.
Why estimation uses the other KL direction
For control, a candidate policy Q is varied while P₀ is fixed, so the transformation cost is DKL(Q || P₀). Estimation asks a different question. The data-generating or target distribution is treated as fixed while the approximating distribution is varied. The resulting relative-entropy risk averages log(P₀/Q) under P₀. Swapping the arguments would change which errors are emphasised and would no longer reproduce the same predictive solution.
In adaptive estimation, an unknown parameter θ selects a source. The prior probability of θ and each θ-specific likelihood define a posterior after the observed history. The predictive law for the next observation is the likelihood averaged under that posterior. This is Bayesian prediction written as the minimiser of the estimation functional. Calling it resource-efficient does not remove the need to calculate or approximate the posterior; the result identifies the optimal distribution within the formal objective.
Adaptive control adds a causal safeguard. The posterior over environment parameters must be updated from observations without treating the agent’s own selected actions as if they had been passively observed. The paper marks actions with intervention notation in its histories. Each hypothesised environment comes with a matching controller, and the agent samples from their posterior-weighted predictive mixture. This resembles probability matching, although the preprint presents it through its Bayesian control rule and causal I/O notation.
Worked example 4 · A one-step soft controller
A robot has baseline action probabilities wait .7 and move .3. Moving has expected target utility 3; waiting has utility 0. Let α = 3. The utility factors are 1 and 2, so the unnormalised weights are .7 and .6. The final probabilities are wait 7/13 and move 6/13. Moving becomes more likely, from .3 to about .462, without erasing the baseline. Lowering α to 1 makes the move factor 8 and raises its final probability to 2.4/(2.4+.7), about .774.
Worked example 5 · Resource limits recover two endpoints
Keep the same baseline and utilities. As α tends to infinity, 2^(3/α) tends to 1, so the two weights return to .7 and .3: only transformation cost matters. As α tends to zero, the factor on move overwhelms the factor on wait, so move approaches probability one: target utility dominates. Between these limits lies a continuous family of bounded policies, each corresponding to a different conversion rate between utility and information.
Executable choice laboratory
The laboratory below computes the paper’s one-step control solution for three actions. Enter non-negative reference weights that sum to any positive total; the script normalises them. Utilities can be positive or negative. Temperature α must remain positive. The output reports the tilted policy, expected target utility, KL divergence in bits and free-utility gain. It also tests the support boundary by preserving a zero reference probability.
Information-priced policy bench
The demonstration is deliberately bounded. It does not simulate the paper’s causal I/O formalism, multi-step recursion or adaptive-control mixture. It exposes the transformation at the heart of equation 4. All values are teaching inputs, not empirical measurements of a person or machine. A high computed free utility means only that a policy scores well under the selected reference, utility and α.
What the framework commits to, and what it leaves open
The first boundary is axiomatic. The probability-utility equivalence depends on treating greater generative probability as greater desirability and requiring the ordering in both directions. Descriptive human choice can violate stable probability rankings, and normative utility theory need not identify desirability with how frequently a system produces an outcome. The theorem remains valid as mathematics while its interpretation can be questioned.
The second boundary concerns resources. KL divergence measures the information in changing one distribution into another. It is tractable, compositional and closely connected to coding and statistical physics. Actual computation consumes time, memory, energy, attention and opportunities. Two transformations with equal KL cost can require very different algorithms on a particular machine. The preprint proposes information deviation as a resource cost; it does not prove a universal equivalence with wall-clock complexity or neural effort.
The third boundary is the reference. P₀ carries prior knowledge, habitual behaviour and feasibility, yet choosing it is a substantive modelling decision. If it is fitted from observed actions, the same behaviour can sometimes be explained by a strong prior with weak utility pressure or a weak prior with strong utility pressure. Zero support forbids innovation. A baseline that already encodes expert optimisation may make a difficult policy appear cheap.
The fourth boundary is scale. α converts bits into utility units, but the preprint does not supply a general experimental procedure for measuring it in every domain. Later work by Ortega and Braun elaborates the thermodynamic interpretation and relates the framework to decision experiments. Genewein, Braun and colleagues examine links with ε-optimality. These successors strengthen and test the programme; they should not be cited as results already established in the 2010 manuscript.
A compact rival source pack sharpens the distinctions. Herbert Simon’s bounded rationality emphasises satisficing and procedural limits rather than a single information-priced optimum. Christopher Sims’s rational inattention constrains information flowing from states to actions, commonly through channel capacity or mutual information. Ortega and Braun price deviation from a reference distribution in a broader I/O formalism. These approaches share concern with scarce information but differ in what is constrained, which distribution is prior and whether the result is descriptive, normative or computational.
There is a further interpretive distinction inside the paper. An unconstrained conjugate distribution maximises expected utility plus entropy and approaches uniformity at high temperature. A constrained transformation maximises target utility minus divergence from a reference and approaches that reference at high temperature. Both statements follow from the same calculus, but they answer different questions. Any exposition that silently replaces the reference with a uniform distribution erases the mechanism by which habit and prior structure enter bounded choice.
Thought experiment 5 · Equal bits, unequal seconds
Setup. Two controllers change the same reference probabilities to the same final probabilities, so their KL costs are identical. One computes the tilt with a table lookup; the other searches a huge proof tree before producing it. The information-priced objective calls them equally costly at the distributional level. A runtime budget does not. The example shows what the abstraction suppresses and why information cost should not be casually renamed computational complexity.
A claim boundary worth preserving
The preprint says its formalisation might provide a principled approach to bounded rationality and shows how three classes of probabilistic control or estimation problem fit the variational scheme. It does not report a cognitive experiment, a universal law of brains, or a proof that KL divergence exhausts resource cost. Its strongest result is structural: under the stated axioms, logarithmic utility and the corresponding free-utility transformation follow.
Memory studio
The mechanism is compact enough to memorise, but its surrounding distinctions are easy to lose. The studio below stores four complementary views: an ordered procedure, a bodily cue for temperature, a problem-type comparison and a timed recall ladder. Each aid preserves the reference distribution because removing it turns the bounded transformation into an ordinary Gibbs choice rule.
The PAUT sequence
- Prior: begin at P₀.
- Axioms: real, additive, monotonic.
- Utility tilt: multiply by 2^(U/α).
- Total: divide by Z.
Say it aloud: prior, axioms, utility, total.
The temperature hand
Close thumb and finger: α is small, changing policy is cheap, so probability pinches around the best action. Open the hand: α is large, change is expensive, so probability spreads back toward the reference. This gesture follows the 2010 paper’s α convention.
Control versus estimation card
| Problem | Fixed | Varied | Result |
|---|---|---|---|
| Control | Reference, utility, environment | Action propensity | Exponential policy tilt |
| Estimation | Source family and prior | Predictive plausibility | Bayesian predictive mixture |
| Adaptive control | Controller family | Posterior mixture | Bayesian control rule |
Recall ladder
30 seconds Choose a policy that gains utility but pays KL cost for leaving its baseline.
5 minutes Rebuild P* ∝ P₀2^(U/α), explain Z and give both α limits.
1 hour Derive the log bridge from product-to-sum additivity, then distinguish control, estimation and adaptive control in the I/O model.
Questions that test understanding
Why does the reference policy matter after utilities are known?
Because bounded cost is a divergence from that reference. The same utility gain can require a small change from one baseline and a large change from another.
What does large α mean in this paper?
Each bit of policy change is expensive relative to utility, so the final policy stays near P₀. This is temperature α, not the inverse-temperature notation used in many later texts.
Why can a zero-probability action never appear?
Giving it positive final mass would make DKL(P* || P₀) infinite. The reference support acts as the feasible action set.
Where does causality enter?
The I/O model distinguishes probabilities for observed inputs from probabilities for generated outputs. Actions are interventions, so they must not be treated as ordinary evidence about the environment.
Place in the canon
The first papers in this collection established limits on computation, quantified uncertainty and examined induction and learning. Paper 0012 turns those tools toward agency. An agent is no longer evaluated only by whether it can compute or predict. It must allocate a limited capacity for changing behaviour. The reference distribution records its starting point; target utility defines pressure; relative entropy prices the transformation.
This is an important shift from possibility to allocation. Computability asks whether a rule can in principle be executed. Complexity asks how required resources grow. The free-utility framework asks which stochastic policy is worth adopting when deviation itself is costly under an information measure. The three questions can disagree. An action may be computable and valuable yet receive little probability because the reference makes it informationally remote.
For agency research, the model offers a disciplined middle ground between perfect optimisation and unexplained error. Stochastic choice can be optimal under a resource-sensitive objective. Habit enters through P₀, not as a residual. Exploration is possible when the baseline supports it. None of this supplies goals from nowhere: U*, P₀, α and the causal model must be specified or learned. Agency in this paper is constrained selection over represented alternatives.
The most useful memory is therefore a question: relative to what? A bounded policy is always bounded relative to a reference, an information price and an outcome representation. Asking that question prevents the elegant exponential formula from becoming a generic decoration. It forces every application to expose where its constraints actually live.