Inference, dynamics and causality
Variational inference and the ELBO
The ELBO turns Bayesian inference into optimisation by projecting an inaccessible posterior onto a tractable family, and that projection must be managed as decision-specific approximation debt rather than mistaken for posterior truth.
TLDR
- The identity log evidence = ELBO + KL gap converts posterior approximation into an optimisation target.
- Maximising the ELBO finds the best distribution only inside the chosen family and under the chosen divergence.
- Reverse KL can ignore missing modes, cut correlations and narrow uncertainty even while the objective improves.
- An ELBO value is not a universal accuracy certificate; the relevant test is the posterior quantity that drives the decision.
- Use an approximation-debt ledger and a decision-loss check before letting variational inference carry consequential authority.
The bridge with two plausible hidden states
A monitoring system receives an unusual vibration pattern from a bridge bearing. The engineers’ probabilistic model admits two explanations. The bearing is probably slightly misaligned, centred near a latent severity of −2. A less likely explanation is a crack, centred near +2. The sensor cannot distinguish them cleanly, so the posterior has two separated peaks.
The exact posterior assigns about 85 per cent of its mass to misalignment and 15 per cent to the crack. An inspection costs 0.08 units of expected loss. Continuing without inspection costs one unit only if severity exceeds +1. Under the exact posterior, the crack probability is about 0.15, so inspection is the cheaper action.
Now fit one Gaussian approximation by maximising the evidence lower bound, or ELBO. A wide Gaussian spanning both peaks would place probability in the low-density valley between them. Reverse Kullback–Leibler divergence punishes that invented mass. The optimiser therefore settles on the large misalignment peak and assigns almost no probability to the crack. Its ELBO becomes much better. Its maintenance decision becomes worse.
The puzzle is not that optimisation failed. It succeeded at the objective supplied to it. The hidden assumption was that closeness in the ELBO’s geometry would preserve the uncertainty needed by the decision. Variational inference becomes trustworthy only when that bridge is tested rather than assumed.
The bargain beneath the bound
Bayesian inference begins with a simple instruction: combine a prior belief about an unknown state with the likelihood of the observed data. The difficulty appears in the denominator.
A posterior that cannot be normalised cheaply
Let x be what we observed and z the hidden quantity we want to infer. A model specifies the joint density p(x,z). Bayes’ rule asks for
The evidence p(x) sums or integrates over every hidden explanation. In a small conjugate model, that integral may be analytic. In a large latent-variable model, it can couple millions of possibilities. We can evaluate the joint density at a proposed state, yet still be unable to normalise the posterior or compute its expectations.
Sampling methods attack the integral by constructing draws from the posterior. Variational inference takes a different route. It chooses a tractable family of candidate densities, written qφ(z), and turns inference into a search for parameters φ. This shift from integration to optimisation is the central move described in the modern VI literature.[1]
The identity that makes optimisation legitimate
Choose any density q(z) whose support does not place mass where the joint density is zero. Multiply and divide inside an expectation by q(z). The log evidence can then be decomposed exactly:
The first term is the ELBO:
The second term is a Kullback–Leibler divergence and cannot be negative. Therefore the ELBO is a lower bound on the log evidence. Because log p(x) is fixed while fitting q, maximising the ELBO is equivalent to minimising KL(q || p). The intractable normalising constant disappears from the optimisation even though the posterior remains the target.
What the two forces inside the ELBO actually do
Expand the joint model as p(x,z) = p(x|z)p(z). The same objective can be written as an expected data-fit term minus a divergence from the variational distribution to the prior:
The first term rewards hidden states that explain the observation. The second charges for moving the approximation away from prior expectations. In a variational autoencoder this is often described as reconstruction versus regularisation, but that language can conceal the probabilistic meaning. The first term is an expected log likelihood under a particular observation model. The second is a distributional cost relative to a particular prior. Change either model and the same numeric balance means something different.
There is another equally useful reading. In the form Eq[log p(x,z)] + H(q), the ELBO rewards density in high-joint-probability regions while the entropy H(q) resists collapse. This does not guarantee broad uncertainty. If the target’s geometry makes broad q place mass in a low-density valley, the expected log-joint penalty can dominate the entropy reward. The bridge approximation then narrows around one mode even though entropy is explicitly present.
The equality assumes properly normalised q, a finite expectation, and compatible support. Remove those assumptions and the familiar algebra may be undefined or the Monte Carlo estimator may have pathological variance. The ELBO is exact as an identity under its conditions; its use as a numerical instrument still depends on how those conditions are realised.
One observation, one Gaussian posterior
Suppose z has prior N(0,1), and we observe x from N(z,1). For x = 2, standard Gaussian conjugacy gives the exact posterior N(1, 0.5). Let the variational family contain every one-dimensional Gaussian N(μ,σ²). The exact posterior is inside the family, so the optimum is μ = 1 and σ² = 0.5. The KL gap becomes zero and the ELBO equals the log evidence.
This example matters because it blocks a lazy conclusion. Variational inference is not inherently under-dispersed, mode-seeking or inaccurate. Those behaviours arise from the interaction among the target posterior, the variational family, the divergence, the optimiser and any stochastic estimator. Remove the mismatch, and the debt disappears.
How the expectation becomes computable
Writing the ELBO is not enough. We must estimate its expectation and differentiate it with respect to φ. In conjugate exponential-family models, coordinate updates may be available. Stochastic variational inference uses minibatches and stochastic optimisation to scale such updates to large datasets.[2] Black-box VI instead constructs generic gradient estimators from the log joint and samples from q.[3]
For a reparameterisable Gaussian, draw ε from N(0,1) and set z = μ + σε. Randomness now enters through ε, while μ and σ sit on a differentiable path. The same move powered stochastic backpropagation and variational autoencoders.[4][5] It changes a gradient through a random sample into an ordinary pathwise derivative through a deterministic transformation.
When reparameterisation is unavailable, the score-function identity offers another route: differentiate the log density of q and weight it by the sampled objective. This estimator is broadly applicable, including to many discrete variables, but it can have high variance because every sampled log-joint value multiplies a noisy score. Baselines, Rao–Blackwellisation and control variates reduce that variance; black-box VI made these techniques reusable across models.[3]
The two estimators answer the same derivative question through different causal paths. The score-function estimator asks how changing φ changes the probability of drawing each state. The pathwise estimator asks how the sampled state itself moves when φ changes. When both are available, the pathwise route often supplies a lower-variance signal. That is a computational advantage, not a proof that the resulting family is adequate.
Minibatching adds another approximation layer. If the likelihood factorises across observations, an unbiased minibatch estimate can scale the data-fit term to the full dataset. Yet adaptive optimisation, finite batches and correlated data can still shape the realised trajectory. A reported final ELBO should therefore be accompanied by its estimator definition, sample count, batch scheme and variability, especially when two configurations differ by less than the estimator noise.
Variational inference gains speed by choosing where tractability will live: in a restricted density family, an estimable objective, a differentiable sampling path and an optimiser that can reuse local information. Each choice can also become a source of approximation debt.
flowchart LR A["The bridge with two plausible<br/>hidden states"] --> B["The bargain beneath the bound"] B --> C["The projection writes its own<br/>errors"] C --> D["How approximation debt reaches<br/>a decision"] D -. "re-test" .-> A classDef first fill:#0a1424,stroke:#087A98,stroke-width:2px,color:#e6f3ff classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff class A first class B,C,D step
The projection writes its own errors
The ELBO tells us which member of a chosen family is closest under reverse KL. It does not tell us that the family or the direction of closeness preserves what we care about.
Why reverse KL can select one mode
Imagine a target density with two narrow peaks and almost no density between them. A single Gaussian has two unattractive options. It can spread across both peaks and allocate probability to the impossible valley, or it can sit on one peak and ignore the other. In KL(q || p), every place where q puts mass contributes a penalty proportional to log q/p. Invented mass in a low-p valley is expensive. Missing a region where p has mass but q has almost none contributes little because the expectation is taken under q.
This asymmetry explains the familiar mode-seeking tendency, but only under the relevant geometry. A flexible mixture family may cover both modes. A different divergence can favour mass coverage. A unimodal target may present no such conflict. “Reverse KL is mode-seeking” is a conditional mechanism claim, not a universal personality trait.
Replace the family but preserve the model
Keep the bridge model and its data fixed. Replace the single Gaussian family with a two-component Gaussian mixture. Nothing about the physical bridge, sensor likelihood or prior changes. The new family can place one component on each posterior peak without filling the valley. If the crack probability reappears, the earlier failure belonged to family capacity, not to Bayes’ rule or the data.
Now replace the mixture with a normalising flow. Flows transform a simple base density through invertible maps and can represent richer shapes.[6] If the result still misses the crack, the remaining candidates include optimisation failure, estimator noise, or an objective whose asymmetry is still misaligned with the decision. The replacement experiment localises the cause.
Mean field cuts dependence, even when every marginal looks innocent
A common family factorises the latent variables: q(z) = ∏j qj(zj). This mean-field assumption makes expectations and updates easier, but it removes posterior dependence by construction. The damage can remain hidden when attention stays on means.
A correlated Gaussian whose mean-field optimum is too narrow
Let the exact posterior be a two-dimensional Gaussian with zero mean, unit marginal variances and correlation ρ. Choose a mean-field Gaussian q with diagonal covariance. Minimising reverse KL has a closed-form answer: each variational variance is the reciprocal of the corresponding diagonal entry of the target precision matrix. In this case it is 1 − ρ².
At ρ = 0.9, the exact marginal variance of each coordinate is 1, while the optimal mean-field variance is 0.19. The exact 95 per cent marginal interval is roughly [−1.96, 1.96]. The approximation reports about [−0.85, 0.85]. The means are perfect. The uncertainty is not.
Rotate the coordinates, change the answer
Take the same correlated Gaussian and rotate into its principal axes. In that basis the covariance is diagonal, so a factorised Gaussian can represent the target exactly. Rotate back to the original coordinates and the same family declaration, “independent factors”, becomes restrictive.
No probability law changed. Only the coordinate system in which independence was imposed changed. The expected intuition is that re-labelling should not alter inference. It fails because mean field is not invariant to arbitrary reparameterisation. A tractability assumption can act like an unacknowledged prior over representations. Automatic systems should therefore record transformations and factorisation choices as part of the inference specification, not as invisible compiler details.
Move a latent variable across the approximation boundary
Consider a mixture model with a discrete assignment c and continuous component parameter θ. Version A approximates both c and θ. Version B sums out c exactly and approximates only θ. The generative model and evidence are mathematically unchanged, but the posterior geometry presented to the optimiser differs. Collapsing c can remove a source of combinatorial dependence; in other cases it can make each gradient more expensive.
The experiment changes where exact computation ends and approximation begins. If Version B is markedly more stable, the failure was not “VI versus exact inference” in the abstract. It was a boundary-placement problem. This matters for system design because partial marginalisation, structured factors and local exact updates can buy more accuracy than a globally richer neural approximator.
A thousand weak couplings
Start with two variables whose posterior correlation is only 0.1. Ignoring that link may seem harmless. Now create a decision that depends on the sum of 1,000 similarly coupled variables. The omitted pairwise terms accumulate in the variance of the sum. Nothing discontinuous happened inside the model. The apparent phase change belongs to the aggregate quantity being measured.
This scaling experiment exposes a common error: validating each marginal and assuming the joint decision is safe. A family can be locally plausible and globally misleading. The correct stress test follows the functional used downstream, such as a sum, maximum, threshold crossing or tail probability.
How approximation debt reaches a decision
Approximation error is not one thing. It is a chain of commitments between the model, the projection and the action taken from it.
Five debts, plus one error that belongs elsewhere
This paper uses approximation debt as a practitioner synthesis: the latent obligation created when a faster inference path is accepted without yet demonstrating that it preserves the posterior functionals a decision needs. The debt has five separable sources.
Family debt is the gap between the target posterior and the best member of the chosen family. A diagonal Gaussian cannot express separated modes or dependence. Objective debt appears when the divergence rewards the wrong notion of closeness for the downstream loss. Optimisation debt is the gap between the best available member and the solution actually reached. Estimator debt comes from noisy Monte Carlo, minibatch or discrete-gradient estimates. Amortisation debt appears when one inference network must predict variational parameters for many observations instead of optimising each case separately. Cremer and colleagues separated approximation and amortisation contributions in variational autoencoders, showing why a flexible family does not by itself remove inference suboptimality.[9]
Model misspecification is upstream and must remain distinct. If the likelihood omits a failure mechanism, an exact posterior can still be confidently wrong. Calling that “VI error” would hide the real repair. Conversely, demonstrating that the model is well specified does not erase approximation debt.
p(x,z)
From distributional gap to decision debt
A global divergence can be difficult to interpret. A decision supplies a sharper question. Let a be an available action, z the uncertain state, and ℓ(a,z) the loss if action a meets state z. The action selected under the approximation is
If the exact model posterior were available, it would choose ap. Define the approximation’s decision debt as
Under the same action set and loss, this quantity cannot be negative because ap is optimal under p. It measures the price of acting through q, rather than only the visual difference between densities. In the bridge case, continuing costs about 0.15 under the exact posterior, while inspecting costs 0.08. The approximation selects “continue”, so the decision debt is roughly 0.07.
The exact posterior is unavailable in the problem that motivated VI, so this quantity is usually estimated rather than known. Useful approximations include a high-accuracy sampler on a reduced dataset, exact enumeration on a toy slice, importance sampling from q, simulation-based calibration, or a targeted error bound for the functional itself. Yao and colleagues proposed PSIS and variational simulation-based calibration for diagnosing variational fits.[10] More recent targeted diagnostics ask whether selected means, variances or other posterior functionals are accurate rather than demanding that the entire high-dimensional distribution look good.[12]
| Debt | Typical symptom | Discriminating test | Smallest repair |
|---|---|---|---|
| Family | Modes, skew or dependence absent even after stable optimisation | Fit a structured or mixture family with the same objective | Add one dependency, component or flow layer |
| Objective | ELBO improves while a tail, quantile or action worsens | Hold family fixed and compare divergence or targeted loss | Reweight the functional or use an alternative bound |
| Optimisation | Seeds converge to different fits or gradients stall | Multi-start runs, tighter tolerances, local refinement | Keep the best stable solution and record spread |
| Estimator | High gradient variance or minibatch-sensitive updates | Increase samples, compare estimators, estimate signal-to-noise | Control variates or reparameterised gradients |
| Amortisation | Rare observations fit poorly while common cases fit well | Per-case optimisation from encoder initialisation | Add local refinement for exceptional cases |
| Decision | Action changes under a more accurate reference | Estimate Ddec for the stated loss and action set | Abstain, escalate or change the decision threshold |
An incident-triage model with an amortised posterior
Consider a service-operations model that receives an event trace x and infers latent cause, severity and sensor drift z. The generative model contains global parameters θ learned across incidents. An inference network outputs a diagonal Gaussian and categorical probabilities for each new trace. The system estimates the ELBO with reparameterised continuous samples and minibatches, then proposes either monitor, dispatch or shut down.
The configured system creates all five debts. Rare coupled failures may not fit the factorised family. Reverse KL may undervalue a small severe mode. The optimiser may settle differently after a software upgrade. A low-sample estimator may be noisy. The shared network may be accurate for routine incidents and poor for a novel trace. None of these is exposed by recording only the final ELBO.
A minimal assurance packet records the model version, transformations, family, objective, random seed, estimator sample count, optimisation trace and posterior functionals used by the action. For high-cost actions, it performs local per-case refinement and compares the expected losses. The inference artefact is not “a posterior”; it is a configured approximation with a declared evidence lineage.
Two nearly indistinguishable fits, opposite actions
Suppose the exact posterior probability of a costly fault is 0.12 and an inspection costs 0.08. The exact Bayes action is to inspect. Two variational fits have almost identical means, standard deviations and ELBO estimates. Fit A reports fault probability 0.079; Fit B reports 0.081. A global density plot may make them look interchangeable.
The action threshold makes them operationally different. Fit A chooses continue, which has exact expected loss 0.12 and therefore decision debt 0.04. Fit B chooses inspect, with zero decision debt relative to the exact model posterior. A difference of 0.002 in one posterior functional has become a categorical difference in authority.
This example cuts both ways. A large global KL gap may be harmless if the decision functional is stable far from its threshold. A tiny distributional difference may be material near a sharp boundary. The right diagnostic therefore includes a margin: how far is the posterior functional from the action threshold, and how large is its credible approximation error? When the error band overlaps the threshold, the system should abstain or escalate rather than round the estimate into certainty.
The edge case also explains why comparing only the winning action can miss fragility. Both fits may choose inspect today, yet one may sit 0.001 above the threshold and the other 0.05 above it. Recording the expected-loss difference and threshold margin makes future drift visible before it flips the action.
Why a tighter bound can still be a worse training signal
Importance-weighted objectives use multiple samples to form tighter lower bounds and can improve latent-variable learning.[7] It is tempting to treat tightness as an unqualified improvement. Rainforth and colleagues showed that increasing the number of importance samples can reduce the signal-to-noise ratio of gradients for an inference network, even while the bound tightens.[8] The lesson is not that tighter bounds are bad. It is that bound quality, gradient quality, posterior quality and decision quality are different ledgers.
A laboratory for seeing the debt
The following lab makes two mechanisms visible: correlation debt under mean field, and a decision that worsens while a synthetic ELBO improves.
Executable variational-debt lab
This browser lab uses analytic Gaussian results and a recorded synthetic optimisation trace. It requires no external library. Change the correlation and dimension to see how a small factorisation error compounds in a sum. Replay the bridge fit to see when the approximate action changes.
Experiment A: correlation stress
What the decision sees
The approximation retains 29.5% of the exact standard deviation of the aggregate.
A marginal check can pass while a coupled decision fails. The lab assumes an equicorrelated Gaussian target and the analytic reverse-KL mean-field optimum.
Experiment B: replay the bimodal bridge fit
The exact crack-tail probability is 0.1497 and inspection costs 0.08.
How to apply the lab to a real model
Start with one posterior functional, not the whole model. Name the quantity that changes an action: a tail probability, expected cost, rank, quantile, covariance, or probability of a threshold crossing. Fit the baseline variational approximation and record its optimisation trace. Then construct one more accurate reference on the smallest slice that still contains the hard geometry. This might be exact enumeration for a few latent variables, long-run MCMC on a reduced dataset, a richer structured family, or importance sampling with diagnostics.
Next, perturb one cause at a time. Hold the objective fixed and expand the family. Hold the family fixed and change the divergence or number of importance samples. Repeat with multiple seeds. Refine an amortised fit locally. If the posterior functional and action remain stable, the corresponding debt is lowered. If the action changes, the approximation has crossed a decision boundary and should not receive the same authority.
A positive result permits a bounded claim: this configured approximation is adequate for this functional, data regime and action threshold under the tests performed. A negative result does not prove VI is unusable. It identifies the debt that requires repair or escalation. The experiment cannot establish that the generative model is true, that untested functionals are accurate, or that another deployment distribution will behave similarly.
A seven-step debt assay
A compact protocol keeps the test causal rather than ceremonial. Begin by freezing the generative model, data slice and action loss. Declare the candidate boundary, including every latent variable that is approximated, marginalised or treated as observed. Select the exact posterior functional used by the action and state its threshold or utility curve. Fit the baseline approximation from several seeds and preserve full optimisation traces. Construct the smallest credible reference method. Intervene on one debt source at a time. Finally, compare actions and expected losses, not just density summaries.
The protocol produces three possible outcomes. Bounded acceptance means the functional and action remain stable across the reference and planned perturbations. Repair means a named intervention, such as one structured dependency or local refinement, removes the failure. Escalation means the reference is too costly, unstable or contradictory to lower the debt. Escalation is not a failed experiment. It is evidence that the requested authority exceeds the inference case currently supported.
For repeated use, retain the assay as a versioned test harness. A change in model code, transformation, data regime, variational family, objective, estimator or optimiser invalidates only the evidence it can causally affect, but it should trigger the corresponding replay. This is more efficient than rerunning every diagnostic after every change and more honest than treating a historic ELBO as a permanent certificate.
Run the full Python experiment
The script below reproduces the correlated-Gaussian formula and optimises one Gaussian against the synthetic bimodal target with PyTorch. Run it with Python 3 after installing torch. Different seeds may produce slightly different Monte Carlo traces, but the left-mode optimum has ELBO near log(0.85) ≈ −0.1625.
import math
import torch
# ---------- Analytic mean-field stress test ----------
def mean_field_variance(d: int, rho: float) -> float:
"""Reverse-KL optimum for each marginal of an equicorrelated Gaussian."""
assert d >= 2 and 0 <= rho < 1
return ((1 - rho) * (1 + (d - 1) * rho)) / (1 + (d - 2) * rho)
def aggregate_sd_ratio(d: int, rho: float) -> float:
vq = mean_field_variance(d, rho)
exact_var = d + d * (d - 1) * rho
approx_var = d * vq
return math.sqrt(approx_var / exact_var)
print("d=8, rho=.6, SD ratio =", aggregate_sd_ratio(8, 0.6))
# ---------- Synthetic bimodal ELBO experiment ----------
torch.manual_seed(7)
LOG_2PI = math.log(2 * math.pi)
mu = torch.tensor(0.0, requires_grad=True)
log_sigma = torch.tensor(math.log(1.5), requires_grad=True)
optimiser = torch.optim.Adam([mu, log_sigma], lr=0.03)
def log_normal(z, centre, scale):
centre = torch.as_tensor(centre, dtype=z.dtype)
scale = torch.as_tensor(scale, dtype=z.dtype)
return -0.5 * ((z - centre) / scale) ** 2 - torch.log(scale) - 0.5 * LOG_2PI
def log_target(z):
left = math.log(0.85) + log_normal(z, -2.0, 0.5)
right = math.log(0.15) + log_normal(z, 2.0, 0.35)
return torch.logsumexp(torch.stack([left, right]), dim=0)
def normal_tail(threshold, centre, scale):
a = (threshold - centre) / (scale * math.sqrt(2))
return 0.5 * math.erfc(a)
for step in range(301):
epsilon = torch.randn(16_384)
sigma = log_sigma.exp()
z = mu + sigma * epsilon
log_q = log_normal(z, mu, sigma)
elbo = (log_target(z) - log_q).mean()
if step < 300:
optimiser.zero_grad()
(-elbo).backward()
optimiser.step()
if step % 20 == 0:
tail = normal_tail(1.0, mu.item(), log_sigma.exp().item())
action = "inspect" if tail > 0.08 else "continue"
debt = 0.0 if action == "inspect" else 0.149679 - 0.08
print(step, round(elbo.item(), 4), action, round(debt, 4))Invariant: the target density and decision loss remain fixed while only the variational parameters change. Positive result: a richer family or targeted objective restores the crack-tail probability and inspection action. Negative result: the action remains wrong despite stable optimisation, implying unresolved family or objective debt. This experiment cannot validate the bridge model itself.
stateDiagram-v2 [*] --> Claim state "A posterior that cannot be<br/>normalised cheaply" as Claim state "The identity that makes<br/>optimisation legitimate" as Test state "What the two forces inside the<br/>ELBO actually do" as Revision Claim --> Test: expose mechanism Test --> Decision: evidence survives Test --> Revision: rival survives Revision --> Claim: narrow or rebuild Decision --> [*]
Boundaries and alternative explanations
The approximation-debt view is useful only if it does not claim more than the tests can establish.
A good approximation to a bad posterior is still a bad basis for action
Variational diagnostics compare q with a target implied by the model. They do not establish that the likelihood, prior or observation process captures the world. Under misspecification, even exact Bayes can concentrate on an unhelpful pseudo-truth. Generalised variational frameworks make the loss, divergence and family explicit precisely because the ordinary posterior is not always the only sensible target.[14] That is a model-and-inference design choice, not a free correction.
A rival explanation for the bridge failure is therefore a missing sensor mechanism rather than mode loss. To separate them, compare the variational result with a more accurate inference method under the same model. If both assign little crack risk, revise the model or data. If the reference retains the crack mode and q does not, approximation debt is implicated.
Decision debt is defined relative to the model posterior and stated loss. It cannot certify real-world safety when the model, loss or action set is incomplete. It is an inference assurance measure, not an ontological guarantee.
ELBO optimisation can change the model being learned
When model parameters θ and variational parameters φ are trained jointly, the approximate posterior does more than report uncertainty. It shapes the learned generative model. A restricted encoder may favour model parameters whose posteriors are easy for that encoder to approximate. The resulting model can trade data fit against inference convenience. This is one reason amortisation and approximation gaps matter in variational autoencoders.[9]
Increasing family capacity can help, but it raises optimisation cost and may introduce new estimator problems. Importance weighting can tighten a bound but weaken some gradients.[8] Automatic differentiation can remove hand derivations while leaving family and diagnostic choices intact.[13] There is no single monotonic ladder from “more sophisticated VI” to “more reliable decision”.
What would strengthen or weaken the debt framework
For a fixed class of decisions, targeted estimates of decision debt should predict harmful approximation failures better than raw ELBO level, ELBO improvement or global posterior distance alone.
The hypothesis is strengthened if, across preregistered models and decision losses, the proposed debt estimate ranks approximation-induced action regret consistently and transfers across seeds and data slices. It is weakened if raw ELBO or a simpler calibration metric predicts the same failures just as well, or if the debt estimate is too unstable to guide escalation. Model misspecification is a rival generator because it can make both p and q choose a bad action while appearing mutually consistent.
The framework should not be applied where no meaningful action or loss can be stated, where the reference method is itself unvalidated, or where the consequence of a rare event is omitted from the loss. The smallest useful implementation is one model, one consequential posterior functional, two independently initialised VI fits and one higher-accuracy reference on a reduced case.
A note on “variational free energy” across fields
The negative ELBO is often called variational free energy. Similar algebra appears in machine learning, statistical physics and some theories of biological cognition. Shared equations can justify a structural comparison of optimisation objectives. They do not by themselves show that a brain, organism and software optimiser implement the same variables, gradients or causal mechanism. In a convergence programme, the ELBO is therefore a mathematical instrument first. Any biological or phenomenological bridge requires its own objects, preserved relations and intervention.
Price the projection debt before granting authority
Do not ask, “Did variational inference converge?” Ask, “Which approximation did this configuration converge to, which posterior function drives the action, and what evidence shows that the projection preserves it?”
The practical choice is no longer VI versus exact inference as a slogan. It is a portfolio decision. Use VI where its speed enables useful iteration or scale. Spend accuracy on the dependencies, modes and tails that matter. Add local refinement for exceptional cases. Build a higher-accuracy reference where the action is irreversible or the loss is asymmetric. Abstain when the decision functional has not been tested.
The ELBO earns computational tractability by turning integration into projection. The responsible response is not to reject that bargain, but to price it. Record the family, objective, estimator, optimiser and amortisation boundary. Estimate decision debt on the smallest credible reference. Grant authority only within the tested regime. When the debt cannot be bounded, route the case to a richer approximation, a slower method or human judgement.