Part IThe missing distribution in an ordinary autoencoder

An ordinary autoencoder contains an encoder f and a decoder g. It maps an observation x to a latent code z = f(x), then maps that code back to a reconstruction x̂ = g(z). Training usually minimises a reconstruction loss between x and x̂. A narrow latent layer, regularisation or architectural constraints prevent the pair from copying the input trivially.

Nothing in that objective says where codes should live. If the decoder can reconstruct the training set, a rotated, stretched or translated latent arrangement may be equally acceptable. Codes can form curved ribbons, detached islands or highly unequal clusters. Continuous neural networks often make nearby codes behave somewhat similarly, but continuity is weaker than a probability model. It does not state how frequently a region should be sampled, whether a gap is meaningful or whether the origin is valid.

A reconstruction code is not yet a generative source. To generate, a system needs a distribution from which it can draw a new latent value. Sampling a convenient standard normal distribution after training is an extra assumption. The decoder was never required to make that distribution correspond to the encoded data.

This distinction explains why interpolation demonstrations can mislead. Suppose two training images encode to z₁ and z₂. A straight path (1 - t)z₁ + tz₂ may decode into a visually gradual transition. That result shows how the decoder behaves along one chosen path. It does not show that the path has high probability, that its intermediate states are semantically valid, or that random points elsewhere in latent space will work. A flexible decoder can draw a pleasing bridge through a region that carries almost no encoded mass.

There are several ways to supply the missing distribution. One can fit a Gaussian mixture, normalising flow or diffusion model to deterministic codes after training. One can adversarially match the aggregated code distribution to a prior. A VAE takes a particularly coherent route: it specifies a probabilistic latent-variable model and trains an approximate posterior together with the decoder. The declared prior is part of the objective from the start.

Part IIThe variational bargain

The VAE starts from a generative story. Draw a latent variable z from a prior p(z), then draw an observation x from a decoder likelihood p_θ(x | z). The parameters θ define how latent causes become observable data. A common prior is a standard multivariate normal, N(0, I), because it is easy to sample and its density is tractable.

Training requires the marginal likelihood p_θ(x), which integrates over every possible latent explanation. That integral is usually intractable for a nonlinear neural decoder. The exact posterior p_θ(z | x) is difficult for the same reason. The VAE introduces an encoder distribution q_φ(z | x) to approximate that posterior. One neural network produces its parameters for every input, a move often called amortised inference.

Kingma and Welling's original formulation optimises an evidence lower bound, or ELBO. The identity behind it is simple: the log evidence equals the ELBO plus a non-negative KL divergence between the approximate and exact posterior. Increasing the ELBO therefore improves a lower bound on log likelihood while bringing the approximate posterior closer to the model posterior, although a loose bound can still hide a poor approximation.

The first term rewards expected reconstruction under the decoder likelihood. The second charges the input's posterior distribution for departing from the shared prior.

It is useful to read the two terms as a negotiated contract. The reconstruction term asks the code to preserve distinctions that help predict x. The KL term asks it to remain cheap under the common address system. A narrow posterior centred far from zero is expensive. A broad posterior near the prior is cheap, but may carry little information about the particular input. The VAE does not get sampling for free; it pays with a constrained information channel.

A minimal Gaussian example

Consider one latent dimension with prior p(z) = N(0, 1). For an input photograph, suppose the encoder emits mean μ = 2 and standard deviation σ = 0.5. Its posterior neighbourhood is informative: samples cluster around 2 and vary little. Its KL divergence from the prior is approximately 2.32 nats. If the encoder instead emits μ = 0.4 and σ = 0.9, the charge falls to roughly 0.09 nats, but the code distinguishes that photograph less strongly.

Each squared mean penalises displacement from the prior centre. Each variance term penalises a scale that differs from one. The logarithm prevents a posterior from shrinking to a zero-width private code without cost. In several dimensions the charges add. The decoder can still use distant or precise codes when their reconstruction benefit justifies the rate.

Moving the randomness outside the gradient path

A direct sample from q_φ(z | x) looks like a discontinuous operation to ordinary backpropagation. The reparameterisation trick rewrites a Gaussian sample as a deterministic transformation of input-independent noise. The encoder produces μ and σ; a separate source draws ε ∼ N(0, I); then z = μ + σ ⊙ ε. For a fixed draw of ε, changes in μ and σ change z differentiably.

Thought experiment: add noise, remove the prior charge

Keep the probabilistic encoder and reparameterised sampling, but set the KL weight to zero. The decoder still sees noisy neighbourhoods. It may become robust to small perturbations, and the encoder may learn non-zero variances. Yet no term connects those neighbourhoods to N(0, I). Their centres can drift, their scales can shrink and the aggregate code distribution can take any convenient form.

This is a clean intervention because only one causal feature changes. If random prior samples still fail while reconstructions remain good, stochasticity was insufficient. Noise asks for local tolerance; the KL term asks for global address compatibility. The two effects are often conflated in informal explanations of VAEs.

flowchart TD
  A["Two archives and the unused<br/>postcodes"] --> B["Part I The missing<br/>distribution in an ordinary…"]
  B --> C["Part II The variational bargain"]
  C --> D["Part III What the prior buys<br/>and the rate charges"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#D95545,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Part IIIWhat the prior buys and the rate charges

The negative ELBO is often written as distortion plus rate. Distortion is the expected negative log likelihood of the observation given the code. Rate is the KL divergence from the approximate posterior to the prior. A coefficient β can change their exchange rate:

With β = 0, the model can use as much latent information as it likes and ignore the declared prior. As β rises, private addresses become more expensive. Reconstruction generally worsens because the decoder receives a noisier, less input-specific message. Prior samples may improve because encoded distributions move towards the source used at generation time. At sufficiently high pressure, the cheapest solution is to send no useful message at all.

Alemi and colleagues showed why a single ELBO number can be deceptive. Models with similar objective values can occupy very different points on a rate-distortion plane. One may reconstruct sharply with a high-rate latent code; another may use almost no code and let a powerful decoder model the data marginally. Report rate and distortion separately, because their sum hides the representation you actually trained.

The rate contains two different pressures

Average the per-example KL over the data distribution. It can be decomposed into the mutual information between input and latent code, plus the divergence between the aggregated posterior q(z) and the prior p(z):

The first component is the amount of information the code carries about which input produced it. The second measures whether all encoded samples, considered together, match the declared prior. A single KL term pushes on both. This explains a fundamental tension: improving aggregate prior match by reducing the same rate can also erase input information needed for reconstruction.

The decomposition also corrects a common slogan. Per-example posteriors being close to the prior does not guarantee a perfectly filled latent space, especially with finite capacity and imperfect optimisation. Conversely, a high aggregate match does not guarantee each example has a meaningful, distinct posterior. Measure both operational effects rather than assuming the regulariser did its job.

What “structured latent space” may legitimately mean

In this paper, structure means that the model declares a latent probability law, trains encoded neighbourhoods in relation to that law, and exposes coordinates that can be sampled and compared probabilistically. It may also yield useful local interpolation because the decoder has seen nearby samples around each input. Those are bounded claims.

Structure does not automatically mean that axis one is “rotation”, axis two is “colour”, or that straight lines correspond to causal interventions. Locatello and colleagues showed that unsupervised disentanglement cannot be identified without inductive biases, and that selecting a good model without supervision is itself problematic. A β-VAE can encourage factorised aggregate behaviour, but the discovered axes depend on data, architecture, prior and optimisation.

β is an exchange rate, not a quality dial

The number attached to β is meaningful only inside a fully specified loss. One implementation sums reconstruction error across thousands of pixels; another averages across dimensions; a third learns the decoder variance and therefore changes the effective scale of log likelihood during training. A β of one in those systems does not impose the same information price. Latent dimension, batch reduction and KL warm-up alter the operating point as well.

Choose β by tracing a rate-distortion curve under the exact runtime decoder and sampling procedure. Train several points, then record held-out distortion, average and per-dimension KL, prior-sample validity, mode coverage and the downstream measure that motivated generation. A knee in the curve is useful only after hard constraints pass. If reconstruction of a legally significant field must stay below a fixed error, no improvement in average sample appearance can compensate for crossing that boundary.

Techniques such as free bits or minimum-rate targets make this negotiation more explicit. They allow each latent group to carry a small amount of information before paying the full KL price, or they prevent the optimiser from taking the zero-rate shortcut too early. These methods can stabilise training, but they do not remove the bargain. They replace one global exchange rate with a more deliberate rate-allocation policy. The selected operating point should be recorded as a behavioural contract, because changing it changes both reconstructions and the distribution reached by prior samples.

Reconstruction is defined by the likelihood

The decoder does not optimise an abstract notion of fidelity. It optimises the log likelihood family chosen by the designer. A fixed-variance Gaussian likelihood produces a squared-error term. A Bernoulli likelihood produces binary cross-entropy. These choices specify which residual patterns are cheap and whether uncertainty may be represented in the output distribution.

Pixelwise squared error is especially prone to averaging when many detailed outputs are plausible for one latent message. An averaged edge can look blurry even when it has lower expected squared error than any sharp guess. That visual effect is partly a likelihood and decoder issue, not a mystical consequence of continuous latents. Theis and colleagues cautioned that likelihood, sample appearance and task utility can vary independently. The evaluation must follow the intended use.

Part IVWhere the story breaks

Thought experiment: give the decoder a side channel

Take a VAE whose decoder can reconstruct only by reading z. Now add a side channel that reveals enough of the previous sequence, image context or target structure to predict x without the latent code. Keep the prior, encoder and objective unchanged. The reconstruction benefit of carrying information in z falls, while the KL charge remains. The economical solution is q(z | x) = p(z): zero rate, no input-specific information and a decoder that ignores z.

This is posterior collapse. It is common with powerful autoregressive decoders, but decoder strength is not the only cause. Training dynamics matter. He and colleagues found that an inference network can lag behind the moving model posterior early in training, encouraging the model to learn around the latent path. Other analyses show that even simple linear VAEs can contain local optima associated with collapse. KL warm-up, free bits, altered update schedules and richer posteriors can help, but each intervention changes the optimisation or information budget rather than abolishing the underlying trade-off.

A near-zero KL is not evidence of an elegantly regularised code when the task requires informative latents. Inspect active units, mutual-information proxies, decoder sensitivity to latent interventions and reconstruction under shuffled or zeroed codes.

A convenient prior can have the wrong topology

A unimodal Gaussian is connected. Real latent factors may contain disconnected categories, periodic variables, trees or manifolds with holes. Forcing two genuinely separate modes into one Gaussian can create artificial bridges. Samples along those bridges may decode into hybrid observations that are mathematically reachable but operationally invalid.

Richer priors can reduce the mismatch. VampPrior uses a mixture of variational posteriors conditioned on learned pseudo-inputs. Hierarchical priors and normalising flows increase flexibility. These additions can improve fit, but they also make sampling, inference and validation more complex. The right question is not whether the standard normal is elegant. It is whether its geometry preserves distinctions that the application cannot afford to blur.

Negative control: deterministic code plus a fitted density

Suppose an ordinary autoencoder reconstructs well. Fit a Gaussian mixture to its latent codes and sample from that fitted density instead of pretending the codes follow N(0, I). Ghosh and colleagues developed deterministic regularised autoencoders with ex-post density estimation and reported competitive generative results. Modern latent diffusion systems also separate a learned autoencoding representation from a later generative model over that representation.

This baseline matters because it can make the VAE explanation look unnecessary. It preserves a low-distortion deterministic code, then learns its irregular distribution explicitly. It may outperform a VAE when reconstruction detail dominates and a second-stage density model is acceptable. Its costs are different: two objectives, a possible mismatch between representation training and density fitting, extra release lineage, and no single ELBO tying inference to generation.

The negative control changes the claim from “VAEs make autoencoders generative” to “VAEs jointly train one particular inference-generation contract”. That narrower claim is stronger because it survives a credible alternative.

Three claims to reject at review

First, latent likelihood is not a safety boundary. Deep generative models can assign higher likelihood to some out-of-distribution inputs than to the training distribution. Nalisnick and colleagues demonstrated this on several image-dataset pairs. Use dedicated shift tests, semantic constraints and effect controls rather than treating density as an automatic novelty detector.

Second, low-dimensional visibility is not causal explanation. A two-dimensional plot may reveal clusters or directions, yet visual separability does not show that the decoder causally uses those features or that they correspond to real-world factors. Intervene on coordinates and test outputs against labelled or mechanistic expectations.

Third, the diagonal Gaussian is an approximation choice. It cannot express correlated or multimodal posteriors for one input. Normalising flows, hierarchical latents, importance-weighted objectives and richer priors can tighten or reshape the approximation. They should be justified by a measured failure, not added as ornamental sophistication.

Part VA controlled laboratory and an architecture decision

The companion notebook turns the paper's causal claim into five matched experiments. It creates 2,600 public synthetic cases from two known standard-normal factors. A fixed nonlinear function maps those factors into twelve observed features, with small measurement noise. The first 2,200 cases train identical small networks; 400 are held out.

The comparison keeps the observed data, latent dimension and broad network capacity fixed. It changes only the latent contract. The ordinary autoencoder receives no declared prior. A stochastic autoencoder samples from an encoder distribution but sets β = 0. The moderate VAE uses β = 1. The over-regularised VAE uses β = 20. The serious baseline fits a three-component Gaussian mixture to the ordinary autoencoder's latent codes.

Reconstruction mean squared error measures the input-to-code-to-output path. Sliced-Wasserstein distance compares the two recovered generating factors from model samples with fresh draws from the true factor distribution. It is lower when the sampled distribution matches. A separate manifold residual checks whether the other ten generated features are consistent with the known nonlinear mapping. Active units and mean KL reveal whether the variational models use their two latent dimensions.

The ordinary autoencoder reconstructs best, yet drawing from an assumed standard normal gives the wrong factor distribution. Adding noise without KL pressure performs no better on that generative criterion. The moderate VAE raises reconstruction error by more than an order of magnitude in this deliberately small model. It also reduces factor distance by roughly 89 per cent relative to the ordinary autoencoder sampled from N(0, I). Both latent dimensions remain active.

The fitted-density baseline is the important restraint. It preserves the autoencoder's low reconstruction error and achieves factor distance close to the VAE. The result prevents a triumphalist conclusion. The VAE is valuable when joint prior-compatible training, amortised posterior inference and one probabilistic model matter. A separate density can be better when fidelity dominates and the extra component can be governed.

The β = 20 model exposes collapse. Its mean KL is almost zero, no latent dimension is active, reconstruction is poor and none of its prior samples pass the manifold-consistency threshold. The prior match of its encoded means is superficially excellent because all means sit near zero. That is an empty success: the decoder has learned almost a constant output.

Worked scenario: generating inspection states

Consider a synthetic component-inspection system with high-dimensional sensor traces and images. Engineers want three capabilities: reconstruct a case for review, sample plausible operating states for simulator testing, and vary known factors such as load or temperature without crossing into physically impossible states. No claim is made here about a deployed system; the scenario exposes the architecture decision.

A plain autoencoder may preserve fine defect evidence best. A VAE may support direct sampling and local search from a declared prior, but a standard Gaussian can blend disconnected failure modes. A deterministic autoencoder plus a fitted mixture may keep sharper evidence and model separate operating regimes. A conditional VAE may be preferable when labelled operating conditions should explain some variation, leaving the latent code for residual uncertainty.

The release decision cannot follow a single reconstruction score. The team should test prior samples against physical constraints, rare-mode coverage and allowed state transitions. It should also verify that latent interventions produce the intended change and that strong contextual features do not collapse the posterior. The latent model proposes synthetic observations; deterministic simulation rules and domain review still decide whether those observations are admissible.

Executable artefact: the latent-space bargain lab

The full notebook contains data generation, four trained neural models, the fitted-density baseline, broad assertions and separate plots. Inspect the implementation in this paper or Inspect the implementation in this paper. The core intervention is small enough to inspect:

# Core intervention: keep architecture and data fixed, change only beta.
def train_vae(x, beta, steps=1000):
    model = VariationalAutoencoder()
    optimiser = torch.optim.Adam(model.parameters(), lr=4e-3)
    x = torch.tensor(x)

    for step in range(steps):
        effective_beta = beta * min(1.0, (step + 1) / 250)
        optimiser.zero_grad()
        reconstruction, mean, log_variance, _ = model(x)

        distortion = ((reconstruction - x) ** 2).sum(1).mean()
        rate = (-0.5 * (
            1 + log_variance - mean.pow(2) - log_variance.exp()
        ).sum(1)).mean()

        (distortion + effective_beta * rate).backward()
        optimiser.step()

    return model

# Positive case: moderate prior pressure.
vae = train_vae(train_x, beta=1.0)

# Negative control 1: stochastic code, no prior pressure.
noise_only = train_vae(train_x, beta=0.0)

# Negative control 2: excessive prior pressure, expected collapse.
collapsed = train_vae(train_x, beta=20.0)

The positive assertion checks that the moderate VAE's prior samples match the known factor distribution better than the noise-only encoder. Negative assertions check that excessive pressure produces zero active units and that a separately fitted density repairs the ordinary autoencoder's arbitrary latent source. These are causal comparisons, not a leaderboard.

stateDiagram-v2
  [*] --> Claim
  state "A minimal Gaussian example" as Claim
  state "Moving the randomness outside<br/>the gradient path" as Test
  state "Thought experiment: add noise,<br/>remove the prior charge" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

The structured-latent acceptance card

“Use a VAE” is too coarse an architecture decision. Record the required latent operation, the acceptable information loss and the baseline that would falsify the choice. Approve the VAE only when prior addressability changes the product or experiment and survives the six tests below.

For a release record, add the prior family, latent dimension, likelihood, β schedule, measured rate, active-unit rule, sample-validity test, reconstruction threshold and response to an unknown outcome. If the latent distribution changes, treat it as a model-interface change because sampling behaviour, downstream searches and stored codes may all move.

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: Two archives and the unused postcodes
  M->>E: A minimal Gaussian example
  E-->>M: compare rival explanations
  E->>D: Choose the address system deliberately
  D-->>Q: revise scope or act

Choose the address system deliberately

A variational autoencoder is often introduced as an autoencoder with noise and a regulariser. That description hides the design decision. The model declares a latent source, approximates posterior inference for each observation, and trains the decoder to operate on samples from neighbourhoods whose private information is priced. Its structured latent space is an information contract between reconstruction and prior reachability.

The contract changes three decisions. For research, plot rate against distortion and intervene on the latent path rather than ranking models by ELBO alone. For architecture, compare joint VAE training with a deterministic representation plus an explicit latent density. For assurance, test the exact runtime sampling process, rare-mode coverage, active-unit use and topology-dependent invalid states.

The strongest result is therefore conditional. Use a VAE when direct sampling from a declared prior, amortised posterior inference and a controlled information bottleneck are worth some reconstruction debt. Use another design when the prior is geometrically wrong, the decoder ignores the code, the likelihood punishes the wrong errors or a two-stage density preserves the evidence better.

The question has moved from “does the latent plot look smooth?” to a falsifiable one: does the chosen prior reach valid, useful observations at an acceptable information cost, and does it beat the baseline that learns the latent distribution separately? That is the test a structured latent space must pass.