Restricted Boltzmann machines were conventionally built from binary stochastic visible and hidden units. A hidden feature detector receives total input x from visible pixels and activates with probability σ(x). That probability can vary continuously, but the sampled state used by the model is still zero or one.

The binary restriction is natural for an elementary event and limiting for a graded cause. One oriented edge may be faint or strong; one face part may match weakly or repeatedly across a receptive field. A single bit must express such intensity indirectly through probability over repeated samples rather than magnitude in one state.

A rectified linear unit outputs zero for negative total input and grows linearly for positive input. Its zero half creates a sparse active set; its positive half carries an unbounded graded value. The unit can therefore say both “this feature is off” and “this feature is present at strength 4.7”.

The paper does not simply import a deterministic ReLU into an RBM. An RBM is a probabilistic energy model, so hidden noise and conditional distributions matter during generative learning. The authors first derive a stochastic integer-valued unit from binary components, then introduce a fast continuous noisy approximation.

This derivation connects three objects that are related but not identical. The infinite stepped-sigmoid unit has an expected count. Softplus closely approximates that expectation. A noisy rectifier approximates samples efficiently. During supervised fine-tuning, the system drops noise and uses max(0,x) with derivative zero at nonpositive inputs and one at positive inputs.

The intervention is tested inside a full pipeline. Gaussian visible units model normalised real-valued pixels. Noisy rectified or binary hidden units undergo contrastive-divergence pretraining. The resulting networks are discriminatively fine-tuned, at which point rectified units are deterministic. Reported differences therefore concern a paired probabilistic-pretraining and deterministic-fine-tuning design.

The title’s “improve” has an empirical scope. Rectified hidden units beat matched binary variants on every displayed Jittered-Cluttered NORB row. On Labeled Faces in the Wild, their mean accuracies are higher, yet the authors explicitly judge the standard deviations too large for firm conclusions. The companion keeps those evidence levels separate.

An RBM has no visible-to-visible or hidden-to-hidden connections. Given an image, hidden units are conditionally independent and can update in parallel. Given hidden states, visible units are conditionally independent and can reconstruct in parallel. Symmetric weights connect the two layers.

For binary units, the paper writes the familiar contrastive-divergence update Δw_ij = ε(⟨v_ih_j⟩_data − ⟨v_ih_j⟩_recon). The positive term measures how often a pixel and feature are active together under training images. The reconstruction term measures the corresponding frequency after the model has taken a short generative step.

The joint energy combines pairwise visible–hidden products and biases. Low-energy configurations receive high probability after normalisation over every visible and hidden state. Contrastive divergence avoids calculating that global partition function directly, using a reconstruction contrast as a practical learning signal.

Object and face images are real-valued. The paper therefore replaces binary visible pixels with Gaussian units. Each input component is normalised to zero mean and unit variance. Visible reconstructions are noise-free top-down values, and the Gaussian variance in the energy is fixed to one.

The hidden layer is the controlled comparison. One network uses ordinary stochastic binary features. The other uses noisy rectified features during generative pretraining. After a label layer or Siamese comparison is attached, the whole classifier is fine-tuned through back-propagation, and NReLUs become deterministic rectifiers.

This design means a performance difference cannot be attributed only to a single forward nonlinearity at test time. The unit type changes how the RBM represents and samples hidden strength during pretraining, how features are initialised and how the deterministic classifier later routes gradients.

It also means that reconstruction should not be mistaken for the final task. The RBM learns an image distribution without labels at each pretraining stage. Labels enter only when the stacked features are used for object classification or when face pairs are fine-tuned for identity agreement.

A conventional binomial unit can be built from N identical binary copies. They share the same weights and bias, so each activates with probability p. The expected active count is Np and its variance is Np(1−p). This count carries more levels than one bit without learning N separate feature templates.

Identical copies have an undesirable response: for small p, their mean grows exponentially with total input, and as p approaches one the variance collapses. The paper changes the copies’ fixed offsets. Copy one receives bias offset −0.5, copy two −1.5, copy three −2.5 and so on.

When total learnt input is x=vW^T+b, expected activity becomes the infinite sum Σ_i=1^∞σ(x−i+0.5). As x rises, successive copies cross their sigmoid transition in turn. The sum looks like rounded steps whose average slope approaches one in the positive region.

The closed-form approximation is softplus, log(1+e^x). It is near zero far below the origin, about log 2 at zero and approaches x for large positive values. The primary’s Figure 1 plots the stepped sum and softplus as virtually indistinguishable at its scale.

The approximation is close rather than algebraically identical. At x=0, summing 200 offset sigmoids gives about 0.68257, while softplus gives 0.69315, a difference near 0.01058. Farther from the transition the gap rapidly shrinks. If only N copies are used, the paper notes a correction involving log(1+e^x−N).

Weight sharing is crucial. Every copy uses the same learnt vector w and base bias b; only its offset is fixed. The construction therefore has no more learnt parameters than an ordinary binary feature, despite representing an unbounded count in the infinite idealisation.

This gives the rectifier a probabilistic ancestry. Its positive activity is not introduced solely as a convenient piecewise-linear function. It approximates how many members of a tied population would be active under offset logistic thresholds. That interpretation makes it natural to use in an RBM.

Let x be the learnt total input. The fast noisy rectified linear unit draws max(0, x + N(0,σ(x))), where the Gaussian has mean zero and variance σ(x). Its standard deviation is therefore √σ(x). The final maximum clips negative perturbed values to zero.

Far into the negative region, σ(x) approaches zero. A detector that is firmly off receives almost no noise and rarely crosses the threshold accidentally. Near x=0, the variance is one half, so stochastic threshold crossing is common. Far into the positive region, the variance approaches one rather than increasing with activity.

This noise profile corrects a drawback of ordinary binomial counts. For N identical copies, variance Np(1−p) can scale with N and then collapses as all copies saturate. The offset-copy construction recruits new units progressively, and its approximation maintains bounded stochasticity across positive strength.

A noisy sample and its expected value should not be conflated. The softplus curve describes the expected stepped population closely. The noisy rectifier is a convenient approximate sampler. A particular perturbation can land above or below both softplus and deterministic ReLU, especially near the threshold.

During discriminative fine-tuning, the paper removes the sampling noise. It sets y=max(0,x), using derivative zero for x≤0 and one for x>0. At x=0 the function is nondifferentiable; the implementation chooses the nonpositive-side derivative and proceeds.

The training path therefore crosses an approximation boundary twice: from infinitely many discrete copies to a continuous noisy unit, then from stochastic NReLU pretraining to deterministic ReLU fine-tuning. Empirical results validate the combined pipeline; they do not isolate which approximation contributes how much.

The same distinction matters when reproducing the method today. Using softplus deterministically throughout, adding constant-variance Gaussian noise, or applying dropout to a ReLU would create related-looking but different conditional models. “Rectified RBM” is not a sufficient implementation specification without the sampling rule.

For α>0, max(0,αx)=α max(0,x). If a unit has zero bias, multiplying every pixel intensity by α multiplies its total input by α. Negative inputs remain negative and inactive; positive activities grow by exactly the same factor.

The property composes through many zero-bias rectified layers. If the first representation scales by α, the next linear map’s input scales by α and its rectifier scales likewise. The whole feature extractor is intensity-equivariant: its output changes predictably with the input scale.

Equivariance is not invariance. The feature vector is different after rescaling; it is α times the original. The face-verification system takes cosine similarity between the two feature vectors. Cosine divides their dot product by both lengths, so independent positive scale factors cancel.

The resulting pair representation is invariant to global positive rescaling of either face image, provided the zero-bias and deterministic conditions hold. Because the same feature extractor is applied to both faces, swapping their order also leaves cosine unchanged. A scalar logistic output then maps the distance to a “same identity” probability.

The paper contrasts this with simply dividing all pixels by their sum. Global input normalisation can still let a bright background alter how local features share total intensity. Equivariance preserves the relative pattern of active feature responses and lets the final cosine remove only overall magnitude.

This is a narrow analytic invariance, not blanket robustness to illumination. Additive brightness, shadows, local highlights, clipping, sensor nonlinearities and geometric changes do not reduce to multiplication by one positive scalar. Nonzero learnt biases would also break exact homogeneity.

NReLU noise breaks the exact sample-by-sample relationship during pretraining. The face classifier obtains the proof-level property only because its discriminative feature extractor is deterministic and omits hidden biases. The probabilistic construction motivates the units; the invariant deployed map uses their noise-free limit.

The data contain five toy classes: humans, animals, cars, planes and trucks: captured as grey-scale stereo pairs, plus a “None” class. The jittered-cluttered version shifts the central object in position, size and intensity, changes viewpoint and lighting, and places distractor objects in the periphery.

Original 108×108×2 stereo images are subsampled to 32×32×2 for speed. Pixels are centred to zero mean and divided by the average standard deviation over all pixels in all training images. This preserves stereo channels while discarding much spatial resolution.

The full training pool contains 291,600 cases, 48,600 per class; the test set contains 58,320, or 9,720 per class. Another 58,320 cases are held out from training for architecture choice and early stopping. The holdout contains all images of one randomly selected object instance from each toy class and an equal random sample of “None”.

One- and two-hidden-layer networks are compared. Candidate first layers have 1,000, 2,000 or 4,000 units; candidate second layers have 1,000 or 2,000. More units improve every reported selection, so the best tested architecture uses 4,000 then 2,000 features.

Each layer is greedily pretrained as an RBM using contrastive divergence. Both layers receive 300 epochs, mini-batches of 100 and a learning rate of 10⁻³ applied to the average per-case CD update, with momentum. A multinomial regression layer is then attached, and all parameters are discriminatively fine-tuned.

Learned first-layer NReLU weights include Gabor-like edge filters, global patterns and point filters that copy pixels. Among the 216 filters selected for highest L2 norm in Figure 4, about 25% are described as Gabor-like. Selection by norm favours well-defined patterns and is not an unbiased census of all 4,000 features.

The activation histogram contains no always-active units, so none is purely linear over the training set. Peaks near activity frequencies 0.2, 0.4–0.5 and 0.6 correspond respectively to Gabor-like, mostly global and point filters under the authors’ inspection. This qualitative mapping complements the classification comparison; it does not prove a universal taxonomy of filters.

With one 4,000-unit hidden layer and random initialisation, the NReLU network records 17.8% test error versus 23.0% for stochastic binary units. Pretraining reduces the respective errors to 16.5% and 18.7%. Both unit types benefit, but the rectified model begins and ends lower.

With two layers of 4,000 and 2,000 units, neither pretrained, NReLU records 17.6% and binary 23.6%. Pretraining the first layer lowers them to 16.5% and 18.8%. Pretraining both produces 15.2% for NReLU and 18.8% for binary.

These rows support two conclusions within the tested architecture. Rectified units outperform binary units regardless of displayed pretraining depth. Pretraining successively helps NReLU, while the binary system shows no additional gain when its second layer is also pretrained in the final two rows.

The one-layer comparison also contains a striking ordering: an unpretrained NReLU network at 17.8% beats a pretrained binary network at 18.7%. This does not prove pretraining is unimportant. It shows that, in this protocol, changing hidden-unit capacity has a larger effect than the binary model’s displayed pretraining gain.

The paper supplies broader baselines: multinomial regression on pixels at 49.9%, a Gaussian-kernel SVM at 43.3%, a convolutional net at 7.2% and a convolutional net with a top SVM at 5.9%. The fully connected NReLU system is much better than the first two and much worse than the convolutional systems.

The authors explain the gap rather than conceal it. Their inputs are heavily subsampled, while convolutional architectures encode image topology and approximate translation invariance. The comparison shows that rectification does not replace spatial inductive bias. A stronger hidden variable and a stronger architecture solve different problems.

Statistical uncertainty is not shown for the NORB tables. The consistency of five matched rows is meaningful, but repeated initialisations or confidence intervals would better quantify robustness. “Outperforms in every reported condition” is supported; a precise population-level advantage is not estimated.

LFW contains 13,233 images of 5,749 people collected from the web. The task presents a pair and asks whether the identities match. Official evaluation defines ten splits, each with 5,400 training pairs and 600 test pairs, balanced between same and different, with identities disjoint across training and test.

The paper merges each pair of official splits and performs five-fold rather than ten-fold cross-validation to speed experiments. This deviation changes the uncertainty calculation and should accompany every quoted mean and standard deviation. It is not the exact official ten-fold protocol.

Each 250×250 colour image is cropped to a central 144×144 window to reduce background cues. Eye coordinates are aligned through rotation and scale, the crop is subsampled to 32×32×3, and pixels receive the same zero-mean, average-standard-deviation normalisation as NORB.

Background control is motivated by a striking observation cited in the paper: humans reach 94.27% accuracy even when the image centre is masked. This indicates that hair, clothing and scene context can identify people. The experiment mostly removes those pixels to focus the learnt face representation, but the human number is not a matched performance baseline.

For each training fold, all 10,800 single faces pretrain one 4,000-unit RBM layer. The routine again uses 300 epochs, batches of 100, learning rate 10⁻³ and momentum. The same learnt feature map is applied independently to both faces in a pair.

Cosine distance combines the two feature vectors symmetrically. A scalar weight and bias feed it to a logistic “same” probability. The complete Siamese system is then fine-tuned on pairs. A PubFig development subset with identities absent from LFW supports architecture choice and early stopping.

Without pretraining, NReLU accuracy is 0.7925±0.0173 and binary accuracy 0.7768±0.0070. With pretraining, they are 0.8073±0.0134 and 0.7777±0.0109. Both NReLU point estimates are higher, and pretraining gives its larger visible improvement.

The primary’s conclusion is deliberately modest: standard deviations are too large for firm conclusions. Contemporary systems at 0.8683 and 0.8529 use respectively commercial automatic alignment and manually collected face attributes, making them informative context rather than controlled unit comparisons.

Imagine data on the surface of a high-dimensional unit hypersphere and a zero-bias rectified unit. Its weight vector defines a plane through the centre. On one side the unit is zero; on the other its activity increases linearly with distance from the plane.

With N such units, every input has an active-set code indicating which side of each plane it occupies. Subject to the sphere having at least N dimensions, the paper states that N units can form 2^N regions. Each region corresponds to one subset of active features.

Within a region, the active subset does not change. Every active rectifier is linear there and every inactive one remains zero, so the network implements a linear model locally. Cross a zero-input plane, and one unit switches status, selecting a neighbouring linear regime.

The implied mixture has exponentially many possible components without listing a parameter set for each one. Latent units grow linearly with N and pairwise weights grow quadratically with layer sizes. Mixing proportions are defined implicitly by the same undirected energy-model parameters used for conditional distributions.

This differs from a directed mixture that separately decides which hidden units enter a component. Such switching can make inference difficult and create jumps in density at component boundaries. Rectified switching occurs exactly where one unit’s input is zero, so adjacent linear pieces meet continuously under the paper’s argument.

The count 2^N is a capacity ceiling under a dimensional assumption. It does not mean training data populate every region, contrastive divergence estimates each mixture weight accurately or every active set corresponds to a semantically meaningful concept. The same parameters couple the regions, making capacity efficient and behaviour constrained.

Nor is “mixture” an explicit softmax over 2^N named experts. The undirected model marginalises latent activities. The mixture interpretation explains piecewise-linear density geometry; it is not an implementation that enumerates exponentially many components.

At the operator level, the paper supplies an explicit construction: tied offset binary copies, a softplus expectation, input-dependent Gaussian noise and deterministic rectification for fine-tuning. These are mathematical and implementable claims.

At the representation level, zero-bias rectifiers are positively homogeneous; cosine can turn that equivariance into scale invariance; active sets define many local linear regimes. These claims describe geometry and transformation behaviour, not what any one feature means to a system.

At the experimental level, matched NORB rows consistently favour NReLU over stochastic binary units. LFW means also favour NReLU, with uncertainty large enough that the authors withhold a firm conclusion. Learnt filter images show edge, part, point and global structure selected by high weight norm.

A graded unit can encode “how much” without representing a conscious intensity. Its numerical activity is a state variable whose causal role is defined by the network. Calling a value of five a stronger perception would require evidence connecting that activity to perception rather than simply to downstream classification.

The same caution applies to sparsity. A unit’s being zero on many images and active on a subset can make its function inspectable, but selectivity is not semantic understanding. A Gabor-like filter responds to oriented contrast; it need not possess a concept of an edge or object.

Agency requires objectives and action loops beyond this study. The networks are trained on fixed datasets under supplied losses. A face pair enters; a probability exits. There is no self-chosen goal, planning horizon, environmental intervention or persistent policy.

Consciousness is still farther outside the measurement. NORB error, LFW accuracy, activation histograms and local linear regions are not contrasts between conscious and unconscious processing. No report, access, integration or phenomenal criterion is operationalised.

The legitimate bridge is architectural. Distributed graded features can form reusable representations, and invariant comparison can support later cognitive systems. Such mechanisms may be ingredients in a model of intelligence. Ingredient status does not establish sufficiency, necessity or the presence of a mind.