The motivation begins with representation rather than sample polish. Images, speech and video contain local detail at enormous scale, yet many useful regularities persist over larger spans: an object across pixels, a phoneme across waveform samples, a room across frames. A continuous latent can spend capacity on tiny variations. A discrete bottleneck forces many nearby encoder outputs to share one address and therefore one decoder input.

The system still has familiar parts. An encoder maps an observation x to a vector or field of vectors z_e(x). A dictionary contains K embeddings e₁ through e_K, each D-dimensional. For each encoder vector, the quantiser finds the closest embedding by Euclidean distance. The decoder receives that embedding, written z_q(x), rather than the original encoder output.

This is not simply rounding a scalar. The entries are learnt, so the vocabulary can move towards regions the encoder uses. Nor is it an ordinary classifier with human-labelled classes. Code 37 has no supplied semantic name. Its meaning, if a stable interpretation emerges, comes from which inputs select it and how the decoder uses it.

The paper treats the construction as a variational autoencoder with categorical posterior and prior. Its posterior q(z|x) is deterministic: one code receives probability one and every other code receives zero. Under a uniform prior, the KL term is the constant log K with respect to encoder parameters. This lets training concentrate on reconstruction and codebook alignment before a richer prior is learnt.

The hard choice creates the paper’s central engineering tension. Discreteness gives the bottleneck its character, but nearest-neighbour selection has no useful ordinary derivative at its boundaries. If gradients simply stopped, the encoder would not learn from reconstruction. The authors keep the hard forward choice and supply an approximate backward route. The resulting model is best understood as two computations sharing one training loop: exact selection going forward, surrogate credit assignment coming back.

The distinction matters for every later claim. VQ-VAE does not discover a ready-made symbolic language; it learns a finite index system that helps specified decoders reconstruct or generate data. A compact index may correlate with phonemes or scene structure, but the training objective still determines what distinctions are rewarded and what information can be discarded.

For a codebook e ∈ R^K×D, the assignment is k = argmin_j ||z_e(x) − e_j||₂. The quantised value is z_q(x) = e_k. In two dimensions the rule creates a Voronoi diagram. In the high-dimensional spaces used by the model, the same geometry applies even though it cannot be drawn directly.

Inside one territory, small changes to the encoder output do not alter the decoder input at all: z_q stays equal to the same embedding. At the border, an arbitrarily small move can switch the selected entry. The quantiser is therefore piecewise constant and discontinuous at ties. That is why its exact derivative is zero almost everywhere and undefined at the boundary: not the signal needed to improve reconstruction through an encoder.

Distance and index must be kept separate. The assignment transmits the chosen embedding, not the numeric distance and not the integer label as a magnitude. Code 100 is not “more” than code 20. Indices are addresses into a table. Relabelling every code while permuting the corresponding embeddings leaves the model’s function unchanged.

The vocabulary size K controls how many alternatives exist, while D controls the dimensionality of each alternative. A third quantity: the number of positions in the latent field: controls how many choices an observation gets. An ImageNet example in the paper uses a 32×32 field of choices, each selected from K = 512 vectors. That is very different from compressing the entire image to one of 512 categories.

A larger dictionary can represent more local alternatives, but availability does not guarantee use. Some entries may attract many encoder outputs and others few or none. The governing paper focuses on the training rule and applications; later work studies codebook collapse and utilisation more directly. One should not infer effective capacity from K alone without measuring the empirical code distribution.

Ties also reveal an implementation detail. The mathematical argmin can return any minimiser when two embeddings are equally distant. Software normally has a deterministic tie-breaking convention. Exact ties occupy a measure-zero surface for continuous values, yet finite precision and symmetric initialisation can make them observable. Reproducible analysis therefore records both the metric and tie rule.

The first term scores reconstruction through log p(x|z_q(x)). In a loss-minimisation implementation this is commonly represented as negative log-likelihood or another declared reconstruction cost. The decoder learns normally from it. Through the straight-through estimator, its gradient at the decoder input is copied to z_e, so the encoder also receives a reconstruction direction.

The second term, ||sg[z_e(x)] − e||², moves the selected embedding towards the encoder output. The stop-gradient operator sg is the identity in the forward calculation but has zero derivative. It freezes z_e for this term. Consequently the dictionary is the object that moves; the encoder is not pulled by this copy of the distance.

The third term, β||z_e(x) − sg[e]||², is the commitment cost. Here the stop-gradient freezes the chosen embedding and the encoder is pulled towards it. Without such a term, encoder outputs could change scale or outrun embeddings that learn more slowly. The coefficient β controls this particular pull relative to reconstruction.

The authors report that results did not vary much for β between 0.1 and 2.0 and use β = 0.25 throughout their experiments. This is an empirical statement tied to their reconstruction scales and models, not a dimensionless universal setting. If the data term is rescaled, the same numeric β changes its relative force.

Stop-gradient is not an ordinary mathematical equality. In the forward pass sg[y] equals y; in automatic differentiation it behaves as a constant. Two expressions can therefore have identical values and different gradients. Reading only the scalar loss without its gradient routing loses the mechanism.

The paper also supplies an exponential-moving-average alternative for updating dictionary entries. It accumulates assignment counts and encoder-vector sums, then updates each embedding towards their ratio. The appendix reports γ = 0.99 as working well. Crucially, that alternative is not the update used for the main experiments, so later software defaults should not be silently projected backwards into the evidence.

The paper states the approximation directly: copy gradients from decoder input z_q(x) to encoder output z_e(x). If the decoder says that moving its input north-east would improve reconstruction, the encoder receives that direction as though its continuous output had been the decoder input. On the next forward pass, the shifted encoder output may remain in the same cell or cross into another code’s territory.

This is often called the straight-through estimator. “Estimator” should not be mistaken for an unbiased derivative of the argmin. There is no ordinary gradient through the chosen index that the copied value faithfully estimates. It is a deliberately useful surrogate: a rule for assigning credit where exact calculus would provide none.

The forward and backward stories must remain paired. The decoder never receives the unquantised z_e on the forward pass merely because gradients act as if it did. Conversely, saying that the model uses a hard code without mentioning the surrogate can leave the impression that reconstruction somehow differentiates through argmin. It does not.

The copied gradient can move z_e in a direction that improves the decoder locally around e_k, but selection can then jump. This makes the optimisation path piecewise smooth with discrete events. Momentum, batch composition, dictionary movement and commitment strength can all affect when assignments change. A single scalar loss curve hides that internal reorganisation.

The paper notes that a subgradient through quantisation could be considered, but reports that this simple estimator worked for the initial experiments. It does not prove optimality among gradient estimators. The scientific claim is more modest and more useful: this particular surrogate enabled the declared models to learn discrete codes at the scales tested.

Surrogate gradients also complicate explanations of causality. A successful code is the joint product of forward nearest-neighbour assignments, the copied reconstruction gradient, dictionary attraction, commitment and decoder pressure. One cannot attribute the outcome to discreteness alone without holding the training route and capacity constant.

This separation is easy to blur because both stages appear in samples. Stage one learns a compressor and reconstructor. Its categorical posterior chooses dictionary entries, but the uniform prior contributes a constant KL term and is not trained into a realistic distribution over code maps. Stage two encodes the training data, records the resulting indices and fits an autoregressive prior to those discrete sequences.

For images, the authors use PixelCNN over the spatial code field. Each code is predicted conditionally on earlier codes under a chosen ordering. For audio they use WaveNet over the temporal code sequence. Sampling begins in code space: draw one index at a time from the learnt prior, look up their embeddings and then decode the completed latent structure into pixels or waveform.

This division changes where modelling capacity is spent. The decoder handles local detail conditional on the discrete representation. The prior handles longer-range regularity among the compressed addresses. In the authors’ interpretation, pixel-level autoregression need not spend all its capacity rediscovering local texture before reaching global structure; it can model a reduced spatial grid.

Yet compression does not guarantee semantic factorisation. The decoder and code field may divide responsibility in many ways. A powerful decoder can model detail not explicitly named in a code, while the prior can learn correlations that do not correspond to human concepts. Qualitative samples reveal plausibility, not the unique contents of each latent variable.

The paper explicitly leaves joint training of VQ-VAE and prior to future work. That boundary matters when reconstructing the method historically. Later systems may train components differently, use hierarchical codebooks or replace the prior, but those changes are not evidence from the governing experiment.

Generation and reconstruction also answer different questions. Reconstruction starts with x, selects its codes and asks the decoder to rebuild it. Generation begins from the learnt prior without a source x. A crisp reconstruction does not show that the prior assigns good probability to the code map; an attractive unconditional sample does not measure fidelity to a particular input.

The encoder uses two strided convolutional layers followed by two residual blocks with 256 hidden units; the decoder mirrors that pattern with residual blocks and transposed convolutions. Training uses Adam at learning rate 2×10⁻⁴ for 250,000 steps with batch size 128. For VIMCO, the authors use 50 samples in its multi-sample objective.

The continuous VAE records 4.51 bits per dimension, VQ-VAE 4.67 and VIMCO 5.14. All are reported as lower bounds. For bits per dimension, smaller is better. Thus the discrete model is 0.16 bits per dimension behind the continuous VAE and 0.47 ahead of the VIMCO configuration. “Comparable” is defensible; “state of the art” or “superior to continuous latents” would not follow from this table.

The authors’ novelty claim is narrower: among the discrete-latent approaches considered, VQ-VAE comes close to continuous VAE performance while providing a compressed categorical representation. The comparison is important because earlier gradient estimators for discrete variables had often been demonstrated on smaller data and low-dimensional latent distributions.

Likelihood is not the only objective in the paper. Reconstructions, samples, speaker conversion, phoneme alignment and action-conditioned video reveal different qualities. A method may trade a small likelihood difference for a representation easier to model autoregressively. Those benefits need their own evidence rather than being smuggled into the likelihood number.

The values also depend on the paper’s approximation to the full model likelihood. The decoder is trained using the MAP-selected z_q(x), and the paper reasons that a converged decoder should allocate negligible probability to x under other codes. It evaluates the resulting approximation empirically. This does not convert every VQ-VAE likelihood calculation into an exact marginalisation over all codes.

Protocol details protect the reader from false comparisons. Changing decoder likelihood, preprocessing, number of latent positions, codebook size, optimisation steps or bound estimator can move bits per dimension. The three numbers are meaningful as a declared experiment, not as timeless properties of model names.

For 128×128×3 ImageNet crops, the encoder produces a 32×32×1 field with K = 512. Each address therefore needs nine bits under a fixed-length code. The paper compares 128×128×3×8 input bits with 32×32×9 latent bits and obtains an approximate 42.6-fold reduction. This is a representational count, not a complete compressed-file rate including codebook, prior and entropy coding.

The reconstructions in Figure 2 preserve object identity and broad appearance while becoming slightly blurrier. The authors then fit a PixelCNN prior over the 32×32 address map and decode sampled maps into 128×128 images. The qualitative samples include recognisable object and scene categories, showing that the separated prior and decoder can generate coherent outputs at the displayed scale.

DeepMind Lab frames provide a second image domain. An 84×84×3 frame maps to a 21×21×1 latent field. Reconstructions are described as nearly identical, and PixelCNN samples retain plausible room geometry and texture. These are visual demonstrations rather than blinded human ratings or likelihood leadership claims.

A two-stage model sharpens the distinction between local and global codes. The first VQ-VAE produces a 21×21 field. A second VQ-VAE with a PixelCNN decoder compresses that entire field to only three categorical variables, each with K = 512. Three nine-bit addresses give a 27-bit global representation: less than one 32-bit floating-point value by the paper’s comparison.

At 27 bits the model cannot reproduce the input exactly. Figure 5 shows that room layout, walls and textures can remain broadly similar while pixel details are regenerated. The result demonstrates a severe information bottleneck paired with a powerful conditional generator. It does not show that the three codes enumerate human-interpretable scene descriptions.

Compression ratios require denominators. Raw eight-bit pixels are a convenient baseline but not a competitive image codec. Likewise, a fixed nine-bit address ignores non-uniform code frequencies that entropy coding could exploit. The scientifically durable claim is the declared latent geometry and qualitative reconstruction under it, not a universal 42.6× storage advantage.

The first speech experiment uses VCTK recordings from 109 speakers. Its encoder has six strided convolutions, each with stride two and window size four, yielding a latent sequence 64 times shorter than the original waveform. The representation has one feature map and a 512-entry discrete space. The decoder receives both the chosen codes and a one-hot speaker embedding.

This conditioning establishes an information route: the decoder need not demand speaker identity from the discrete stream because it receives identity separately. When an utterance is encoded from one speaker and decoded with another speaker’s identity, the samples retain linguistic content while changing voice characteristics. The experiment is evidence of useful separation, although “factored out” remains a functional judgement supported by the displayed conversions rather than a proof of statistical independence.

The compressed waveform cannot preserve each sample. Reconstructions have similar content but different exact waveform and altered prosody. This distinction is not a defect hidden by the model: it shows which variations the bottleneck and stochastic decoder can regenerate rather than copy. Listening examples carry information that a pixel-like sample error would miss, but they are also qualitative and subject to selection.

For longer-range unconditional speech, the authors use a 460-speaker dataset and an encoder with 128-fold temporal compression. A 40,960-sample, 2.56-second chunk becomes 320 latent timesteps. A WaveNet prior models those codes. The paper reports clear words and partial sentences, contrasting them with babbling from the original WaveNet configuration, and calls the result a rudimentary phoneme-level language model.

The sharpest quantitative diagnostic maps each of 128 possible latent values, running at 25 Hz after a downsampling factor of 640, to its conditionally most likely label among 41 phonemes. The one-to-one mapping reaches 49.3% accuracy. A baseline that always predicts the prior-most-likely phoneme reaches 7.2%. The codes were learnt without using those labels; the labels enter only the post-hoc evaluation.

Forty-nine per cent is substantial evidence of phoneme-related structure, not equivalence between codes and phonemes. A code may correspond to different phones in different contexts, and a phone may use several codes. The paper’s footnote notes that context-dependent mappings such as bi- or trigrams could score higher. This underscores that the representation’s unit and the linguistic analyst’s unit need not coincide.

DeepMind Lab supplies navigable three-dimensional scenes and action sequences. The experiment provides six initial frames, then samples ten future frames while repeatedly conditioning on an action such as “move forward” or “move right”. The prior generates the latent sequence; after all latent states are produced, a deterministic decoder maps them to pixels.

This order is significant. The model need not generate and re-encode every intermediate image to predict the next latent state. It can represent temporal dependence in the compressed code domain and render frames afterwards. The authors report that local geometry remains coherent and visual quality does not degrade across the shown sequence.

The supplied action is a condition, not a decision. The experiment does not give the model goals, rewards, a policy or authority to act. Repeating “right” and seeing a rightward scene transition indicates action-conditioned prediction. Calling that planning would add a search or choice process not tested here.

Nor is qualitative consistency a complete dynamics evaluation. A model might generate plausible frames that deviate from the exact environment state, forget an occluded object or violate geometry beyond the short horizon. Quantitative prediction error, state probes, counterfactual action comparisons and long-horizon tests would examine those failure modes.

The paper also trains an unconditional version and says it obtains similar results, though those results are omitted for space. That statement supports feasibility but offers less inspectable evidence than the displayed conditioned examples. A faithful companion preserves the asymmetry rather than treating both as equally documented.

The broader insight is architectural: a discrete latent sequence can be an object for a temporal prior. This helps separate representation from control. A system may learn a compressed transition model that a later agent uses, but world modelling, policy selection and subjective experience remain distinct research questions.

The strongest claims are operational. Nearest-neighbour lookup produces categorical addresses. Straight-through copying supplies encoder credit. Codebook and commitment terms route gradients as declared. A second-stage prior models the resulting sequences. These statements can be checked in equations and implementation.

Experimental claims come next. The CIFAR-10 comparison places VQ-VAE close to a continuous VAE on the reported lower bound. Image and video figures show recognisable reconstructions and samples. Audio conversions retain content while changing voice, and code-to-phone mapping far exceeds a frequency baseline. Each result remains tied to its data, architecture and diagnostic.

“Posterior collapse” needs precision. In powerful-decoder VAEs, a latent can be ignored while the decoder models data autoregressively. VQ-VAE’s deterministic quantised route and constant uniform-prior KL avoid the same optimisation pressure in the configurations tested. This does not prove that every discrete bottleneck is fully used or immune to other forms of codebook collapse.

Symbols require more than enumerable tokens. A symbolic system may demand stable reference, combinatorial roles, systematic manipulation or public interpretability. A VQ-VAE index is at least a discrete token internal to a learnt function, but the paper does not test all those stronger criteria. “Code” is the safer term.

Agency is further away. The video model accepts actions; it does not choose them. The image prior samples; it does not pursue a goal. The speech decoder changes speaker identity when instructed; it has no authority or preference. Competent conditional generation should not be redescribed as autonomous action.

Consciousness is not operationalised at all. There are no measures of experience, report, awareness, integration or self-model. A finite latent vocabulary neither establishes nor rules out experience. The paper belongs in a consciousness canon because it sharpens how representations can be structured, not because it offers a consciousness test.

Paper 0055 varied which continuous units were temporarily present. Dropout’s mask is sampled anew during training and usually disappears at inference. Paper 0056 creates a different bottleneck: every encoder vector must choose a persistent entry from a learnt finite dictionary. The first regularises a representation; the second changes its type.

The transition also changes the mathematical obstacle. Dropout has a simple stochastic forward path whose active subnetwork can be differentiated normally. VQ-VAE’s hard argmin blocks ordinary derivatives, so learning depends on explicit surrogate credit and stop-gradient routing. This prepares the canon to distinguish architectures from the optimisation conventions that make them trainable.

Paper 0057 turns back to residual networks: Zhang, Dauphin and Ma’s “Fixup Initialization: Residual Learning Without Normalization” (2019). VQ-VAE asks how to learn through a discrete bottleneck; Fixup asks how to initialise a deep residual path so that normalisation is not required. Both expose invisible training contracts beneath an architectural diagram.