In 2006, a deep net was not merely a long differentiable function awaiting a better optimiser. Hinton, Osindero and Teh consider stochastic binary latent variables arranged in a generative hierarchy. A top-level state produces lower hidden states and eventually visible pixels. Learning the model requires reasoning about which hidden configurations could have generated each observation.

The direct likelihood gradient is expensive because it averages over latent configurations. Even forming a useful recognition distribution is difficult in a densely connected directed model. Two causes that begin independently can become dependent after their shared effect is observed. A factorial approximation then misses precisely the posterior correlations induced by the data.

Markov-chain Monte Carlo could sample from the posterior, but the primary describes it as typically time-consuming. Variational inference replaces the posterior with a tractable distribution and improves a lower bound even when inference is imperfect. Yet a poor approximation leaves a gap, and adjusting every layer jointly can make the gap itself move as the generative model changes.

The proposed escape route is structural. Rather than ignore higher layers while learning a lower one, the algorithm assumes that infinitely many higher layers exist with tied weights. Those tied layers create a prior whose correlations exactly oppose the correlations introduced by the likelihood. Posterior inference through the lower pair becomes factorial.

This constrained model is an undirected restricted Boltzmann machine. It can be trained approximately using contrastive divergence. Once one layer has learnt the current representation, the lower weights are frozen and untied; the inferred hidden activations become a fresh data set for the next RBM. A global problem becomes a staircase of local ones.

The final deep belief net is hybrid. Its top two layers keep undirected connections and form an associative memory. Lower connections point downwards as a generative model. Separate bottom-up recognition connections propose hidden states. After greedy initialisation, a slower “up–down” algorithm fine-tunes the whole system.

Calling this a fast learning algorithm needs a denominator. Greedy training took a few hours per layer on the stated 3 GHz Xeon. The best reported MNIST network also used 300 fine-tuning epochs, 59 additional full-data epochs and about a week of total learning. The fast part is the layer-wise route to a good initial model, not every stage of the final result.

The primary uses a compact logistic example. Earthquake and truck-hits-house are rare, independent causes. Either can make a house jump. Before observing the house, learning that an earthquake occurred does not change the prior chance of a truck collision. After observing the jump, the causes compete: if the earthquake is active, the truck is no longer needed.

The numerical construction gives each cause a bias of −10, the observed effect a bias of −20, and each cause an incoming weight of +20 to the effect. With neither cause, a jump has odds near e⁻²⁰. With exactly one cause, its logit becomes zero and a jump has probability one half. Activating both is wasteful because the joint prior odds are again about e⁻²⁰.

Exact enumeration makes the dependence visible. Conditional on a jump, the states “earthquake only” and “truck only” each receive about 0.49995 posterior probability. “Neither” and “both” each receive about 0.000045. Each cause separately has posterior probability one half, but their joint activation is almost absent.

If the hidden posterior were factorial, two variables with marginal probability one half would be jointly active with probability one quarter. The true value here is roughly 0.000045. The covariance is approximately −0.24995, close to the most negative value possible for two balanced binary variables. A product approximation erases the decisive competition.

In a wide, densely connected belief net, many hidden units share descendants. Every observation can create a web of these dependencies. Exact enumeration grows exponentially, and a single upward pass that computes each hidden probability independently cannot represent the posterior web. This is why adding layers naïvely does not simply add representational power; it adds an inference burden.

The phrase “explains away” can tempt causal overreach. The graph specifies a generative dependence and the posterior changes under observation. It does not by itself establish the real-world causes of a house movement. Within the model, the example is exact Bayesian conditioning; outside it, structure and parameters would need evidence.

Complementary priors attack the dependence rather than approximate around it. The idea is to make hidden causes positively associated in the prior in exactly the way required to cancel their negative association in the likelihood. Multiplying prior and likelihood then yields a factorial posterior that a local upward pass can sample exactly.

Bayes’ rule multiplies likelihood by prior. If the likelihood makes two hidden causes negatively associated after observing data, a positively associated prior can oppose that tendency. A complementary prior is chosen so the two effects cancel exactly, leaving the posterior over the hidden layer factorial.

Factorial does not mean that every hidden unit is uselessly independent in the generative model. It means that, conditional on the current lower-layer vector, the hidden states can be sampled independently. Their marginal activation probabilities still depend on all visible inputs through learned weights. Correlations reappear across data cases and in the model’s unconstrained joint distribution.

The construction uses an infinite sequence of alternating visible-like and hidden-like layers. Successive matrices are transposes and weights are tied: W upward, W^T downward, repeated without end. Starting from data, an upward pass computes a factorial posterior at each layer. Starting sufficiently high, a downward ancestral pass generates the model distribution.

That infinite directed model has the same visible distribution as a finite undirected restricted Boltzmann machine. “Restricted” means there are connections between visible and hidden layers but no within-layer connections. Given visibles, hidden units are conditionally independent; given hiddens, visible units are conditionally independent. Alternating these two factorial conditionals gives block Gibbs sampling.

The equivalence supplies both inference and learning. A data-clamped upward update produces positive-phase correlations between visible and hidden units. A long Gibbs chain produces equilibrium model correlations. Their difference is the maximum-likelihood learning signal for an RBM and, equivalently, for the infinite tied directed network.

The story is exact only at equilibrium and under the tied construction. In practice, contrastive divergence stops the Gibbs chain after a few steps. The paper makes the approximation explicit and notes that the changing short-chain distribution is treated as if it did not depend on the parameters. Efficiency is bought by dropping derivative terms.

Nor must a deployed deep belief net retain infinitely many physical layers. The infinite construction proves a distributional equivalence and provides a prior. The implemented top two layers form one finite undirected associative memory. Their repeated alternating Gibbs transitions play the role of movement through the tied infinite tower.

An RBM assigns low energy to compatible visible and hidden configurations. With no same-layer connections, hidden units can update in parallel given a visible vector, and visibles can update in parallel given a hidden vector. One complete Gibbs step consists of both blocks. Repeating the step defines a Markov chain whose stationary distribution is the model distribution.

The maximum-likelihood gradient for a weight has a memorable form. Increase it according to the visible–hidden correlation when data are clamped, then decrease it according to the correlation under the model at equilibrium. These are conventionally called positive and negative phases. The first asks which features the data activate; the second asks which configurations the current model invents on its own.

Equilibrium can be expensive. Contrastive divergence begins the chain at a data case and runs it for only n complete steps. It replaces the equilibrium negative correlation with a reconstruction-side correlation. In the paper’s information-geometric description, CD-n reduces a difference of two Kullback–Leibler divergences: the data-to-model divergence minus the n-step-to-model divergence.

Gibbs transitions move the chain distribution towards equilibrium, so the displayed difference is nonnegative when sampling noise is ignored. That does not make CD the exact gradient of the difference. The n-step distribution depends on the same model parameters being changed, and CD ignores derivative terms arising from that dependence.

This distinction matters twice. First, a one-step reconstruction that looks like its input is not proof that likelihood improved. Secondly, the variational guarantee used later in the greedy-layer argument assumes sufficiently patient maximum-likelihood Boltzmann-machine learning. The actual experiments use contrastive divergence because it is fast and works well, explicitly giving up the guarantee.

Short chains nevertheless provide a useful local signal. If a feature connection is strong in the data-clamped phase but the model fails to reproduce that correlation after reconstruction, the weight increases. If the model hallucinates a correlation more strongly than the data support, the weight decreases. The update contrasts nearby model behaviour with the observed distribution.

The laboratory below computes these quantities by exact enumeration for a tiny RBM. Its CD value has no Monte Carlo noise: all binary transitions are summed. This makes the approximation boundary visible. As Gibbs steps increase, the negative statistic approaches the exact model statistic, but CD-1 and maximum likelihood can differ substantially at strong weights.

Begin with visible data V₀ and a first hidden layer H₀. Assume all higher weight matrices are tied to W₀. Under the infinite-model equivalence, learning W₀ is learning an RBM. Contrastive divergence supplies a fast approximate solution.

Next, freeze W₀. The algorithm commits to using its transpose, W₀^T, as the bottom-up recognition mapping even if later changes above make that inference no longer exact. Each visible training vector is mapped to the factorial activation distribution over H₀. Those probabilities or sampled activities are the “data” for the next stage.

Now untie W₀ from the higher weights while keeping all remaining higher matrices tied to one another. Learn W₁ as an RBM over the H₀ representations. Freeze it, transform upwards again and repeat. Every new layer models regularities left in the representation produced below.

The relationship to boosting is suggestive but limited. Boosting repeatedly changes case weights so a new weak learner attends to previous errors. Greedy deep learning repeatedly changes the representation of every case. The next learner sees different variables rather than merely different importance weights.

“Invents data” is deliberate shorthand, not a claim that the layer fabricates observations. The original cases remain the evidence. A learnt conditional distribution maps them into hidden activation vectors that function as the training set for the next RBM. Errors or biases in the lower recognition mapping therefore shape everything learned above.

Layer-wise training solves an optimisation coordination problem. A high layer receives a structured signal before global fine-tuning rather than gradients through a randomly initialised tower. The primary’s justification is probabilistic, through complementary priors and a variational bound, not a modern story about vanishing derivatives alone.

Once the stack exists, it is not just a pile of independent autoencoders. The top RBM defines an undirected prior; lower directed weights define how a top state generates pixels. Bottom-up matrices initialise recognition. The pieces share variables and are later adjusted together by the up–down procedure.

For one visible vector v and first hidden state h, the directed model factors into a prior p(h) and likelihood p(v|h). Its log evidence log p(v) requires summing over h. Introduce any recognition distribution Q(h|v). The evidence lower bound averages log p(h)+log p(v|h) under Q and adds Q’s entropy.

The gap between log evidence and the bound is KL(Q(h|v) || p(h|v)). It is nonnegative and vanishes only when Q equals the true posterior. Under the initial tied-weight construction, the complementary prior makes the factorial upward recognition distribution exact. At the moment a lower layer is frozen, the bound touches the evidence.

Freezing W₀ fixes both the likelihood p(v|h) and the recognition distribution Q(h|v). The only trainable term is the higher prior p(h). Improving that prior on hidden states drawn from Q therefore raises the bound. A new RBM can be viewed as a model of the aggregated posterior data.

An increasing lower bound does not normally force the true log probability to increase: a looser bound can rise while evidence falls. Here the initial equality supplies the safeguard. The newly raised bound cannot exceed evidence, and it cannot fall below the evidence value at the tied starting point. Thus the new evidence cannot be below its old value.

The argument can be repeated layer by layer. With exact maximum-likelihood Boltzmann-machine learning for each tied higher stack, adding a layer never decreases the bound. The guarantee is in expectation over the hidden data distribution, as the primary’s footnote notes, and assumes the specified generative and recognition terms remain fixed during the step.

Practice changes the premise. The authors substitute contrastive divergence for exact maximum likelihood because it is far faster. Their wording is direct: this voids the guarantee. The theory still explains why a sufficiently well learnt layer should help, but a CD-trained layer must be checked empirically rather than certified by the proof.

This separation is exemplary. The proof is not discarded because the implementation approximates it, and the implementation is not promoted to theorem because the proof exists nearby. The theory identifies an objective and a safe limiting procedure; engineering chooses a tractable estimator with observed success.

After progressive untying, bottom-up recognition weights and top-down generative weights need not remain transposes. Higher-layer changes destroy the exact complementary priors for lower layers, so a simple factorial upward pass becomes approximate again. The paper addresses this with a contrastive variant of wake–sleep.

The up-pass begins at data. Recognition weights produce stochastic binary states in successive lower hidden layers. At the top, the state initialises the undirected associative memory. Alternating Gibbs sampling then updates its two sides, allowing the high-level state to move towards the model’s conditional distribution.

The down-pass starts from the sampled associative state. Generative weights stochastically reconstruct each layer below until reaching visible pixels. Generative parameters compare their upward data-driven inputs with reconstruction-driven outputs. Recognition parameters learn to recover hidden states from these generated lower-layer activities.

Ordinary wake–sleep has known weaknesses: sleep-phase recognition learning is driven by model fantasies that may be poor, and wake-phase generative learning uses an approximate posterior. The contrastive version begins the top memory from an upward data state and runs it before descending, keeping the negative sample closer to the data-conditioned region.

The MNIST schedule increases the amount of top-memory mixing. During the first 100 of 300 fine-tuning epochs, the upward pass is followed by three full Gibbs iterations; the next 100 use six; the final 100 use ten. Each increase visibly reduced validation error according to the primary.

All parameters are adjusted, but with different roles. The top undirected weights use positive and negative associative-memory statistics. Lower generative weights improve their ability to produce the layer beneath. Recognition weights improve the upward approximation. One learning rate is used throughout the supplied pseudocode for simplicity, although the experiment retunes learning rate, momentum and weight decay conservatively.

Fine-tuning is therefore not ordinary discriminative back-propagation from labels. Labels participate as a ten-way softmax group in the top associative model, and the network learns a joint generative model of images and labels. Its classification decision later compares label-conditioned free energies after an upward recognition pass.

The basic task contains 60,000 training images and 10,000 official test images. It is permutation-invariant: the algorithm receives no knowledge of pixel geometry, performs no special preprocessing and does not augment the set with translated or elastically distorted digits. A fixed random pixel permutation would not change the learning rule.

The displayed network has 784 visible pixels, successive 500-unit layers, a 2,000-unit top layer and ten label units. The top 500-unit representation and the label group connect undirectedly with the 2,000 top units. The two lower transformations become directed generative connections, alongside bottom-up recognition weights.

Development begins with 44,000 balanced cases arranged into 440 mini-batches, each containing ten examples of every class. Each RBM receives 30 epochs of greedy learning. This phase takes a few hours per layer and leaves the network at 2.49% test error: already useful, but not the published endpoint.

A separate 10,000-image validation set selects learning rate, momentum and weight decay during 300 up–down epochs. The top-memory chain length rises from three to six to ten full Gibbs iterations in three 100-epoch blocks. The selected model records 1.39% test error.

The authors then train on all 60,000 official training cases until training error matches the earlier fitted level. Another 59 epochs bring the official test error to 1.25%, or 125 misclassified images, after about a week of total learning. A subsequent six-week low-rate run fluctuates between 1.12% and 1.31%; the paper retains 1.25% as its main result.

Classification does not simply attach a softmax to a deterministic feed-forward net. A deterministic probability up-pass fixes the 500 lower units of the associative memory. Each label is turned on in turn, and the exact free energy of the resulting 510-component vector is computed. Comparing these energies yields the label distribution. Stochastic up-passes or short label Gibbs chains perform worse.

The primary compares 1.25% with a 1.4% support-vector machine and feed-forward results of 1.53% and 1.51% on the basic task. It also lists lower numbers from systems using geometry or extra distortions: 0.95% for LeNet5, 0.56% for a virtual SVM and 0.4% for an elastically augmented convolutional net. Those are different information conditions, not contradictions.

“Better than the best discriminative learning algorithms” in the abstract is therefore scoped by the table and prose. On the permutation-invariant, unaugmented comparison available to the authors, the generative model beats the cited generic discriminative baselines. Hand-crafted connectivity and augmented data obtain lower absolute errors.

To generate a class-conditional digit, the algorithm clamps one label unit and runs alternating Gibbs sampling in the top associative memory until the chain approaches equilibrium. A sampled high-level state then travels down the directed generative connections in one pass, producing pixel probabilities and an image.

Figure 8 presents ten samples for each digit label with 1,000 Gibbs iterations between samples. The separation matters: adjacent images are intended as draws from the class-conditioned model after substantial mixing, not successive frames of a digit slowly being written.

Figure 9 studies a trajectory instead. It begins with a random binary image whose pixels are independently on with probability 0.5, maps that image upward, clamps a label and lets the top memory evolve. Every 20 Gibbs iterations, a downward pass renders what the current high-level state would generate. Rows settle into digit-like regions.

The authors describe digit manifolds as long ravines in the free-energy landscape. A clamped label changes the landscape, and Gibbs sampling explores configurations favoured by that conditional energy. The ravine metaphor conveys many related surface forms associated with one class rather than a single memorised prototype.

Section 7 is titled “Looking into the Mind of a Neural Network”. It then says the word “mind” is not intended metaphorically and offers a definition: a mental state is the state of a hypothetical external world in which a high-level internal representation would constitute veridical perception. The rendered downward sample depicts that hypothetical world.

This is a substantive representational proposal, not casual caption writing. On its own definition, the internal state has intentional content because it can be decoded as a possible input world. The model gives researchers a controlled way to inspect that content by generation. That is stronger than saying a hidden vector is merely uninterpretable machinery.

It is not, however, an experiment on phenomenal consciousness. The figures do not measure awareness, subjectivity, report, metacognition or unified access. A probability model can encode and generate possible worlds without experiencing them. The authors’ stipulated use of “mental state” cannot substitute for an independent consciousness criterion.

Nor does Gibbs movement establish autonomous thought. Transition probabilities follow fixed weights, a clamped label and injected sampling noise. There is no self-chosen objective, environment intervention or persistent goal. The trajectory demonstrates conditional generation and associative completion under the model.

At the operator level, the evidence is concrete. Logistic binary conditionals define local state probabilities. An RBM alternates exact factorial block updates. Contrastive divergence uses positive and short-chain negative correlations. Greedy stacking freezes and transforms representations. Up–down training separates recognition and generative weights.

At the mathematical level, the paper proves an equivalence between an RBM and an infinite tied directed model, derives complementary priors for a family of likelihoods and shows monotonic improvement of a variational bound under exact layer learning. Each result has conditions. None licenses every practical CD-trained network to inherit the theorem.

At the experimental level, one specified MNIST system learns a joint distribution of images and labels, reaches the reported error under a long protocol and generates recognisable class-conditioned digits. The results support representational learning, approximate inference, classification and sampling in that setting.

Agency would require more. A system might be called agentic if it selects objectives, acts on an environment, updates plans under consequences or preserves goals across time. The studied network receives a training objective and, at generation time, a clamped label. Its stochastic transitions explore a supplied conditional distribution rather than choose what to pursue.

“Associative memory” is also technical here. The undirected top layers settle into configurations and can complete or generate patterns. That supports content-addressable dynamics within a run. It does not by itself demonstrate autobiographical memory, temporal self-continuity or deliberate recollection.

The consciousness claim fails at the measurement boundary. There is no contrast between conscious and unconscious conditions, no behavioural report paradigm, no neural correlate criterion and no intervention predicted by a theory of experience. Generated samples are observable outputs from a probabilistic model.

The authors’ literal definition of a mental state can be assessed as a functional-semantic proposal: an internal configuration is about the external state that would make it veridical. A critic may accept that notion of content while denying phenomenal consciousness. The paper supplies no experiment that decides between those positions.

The enduring bridge to contemporary systems is mechanistic. Layer-wise pretraining shows how a difficult objective can be approached by staging tractable local problems; generative decoding shows how internal states can be probed; the bound-versus-CD distinction shows how proofs and approximations should be separated. Those lessons strengthen consciousness research without turning the network into a conscious subject.