Boltzmann Machines, Energy-Based Models, Contrastive Divergence & Deep Belief Networks

Borrowing a page from physics: define an energy over configurations, let low-energy states be probable, and learn by nudging the energy landscape with real data.

• Explain the Boltzmann distribution and how energy-based models turn "low energy ⇒ high probability" into a training principle.
• Write the full Boltzmann Machine energy function and identify why full connectivity makes exact training intractable.
• Derive the Restricted Boltzmann Machine's simplified energy function and its tractable conditional-independence property.
• Derive, step by step, the positive-phase / negative-phase gradient of the RBM log-likelihood.
• Compute one full Contrastive Divergence (CD-1) update by hand on a 2-visible / 1-hidden RBM.
• Describe how Deep Belief Networks stack RBMs via greedy layer-wise pretraining, and why this mattered historically.

1. Energy-Based Models & the Boltzmann Distribution

Every model in this course so far has been trained by directly minimizing a loss between a prediction and a target. Energy-based models take a different route, borrowed from statistical mechanics: physical systems tend to settle into low-energy states — a ball rolls downhill, a magnet aligns its spins to minimize energy. The Boltzmann distribution formalizes this: the probability of a system being found in a particular state is

$$P(\text{state})=\frac{e^{-E(\text{state})/kT}}{Z}$$

where \(E\) is the energy of that state, \(k\) is the Boltzmann constant, \(T\) is temperature, and \(Z=\sum_{\text{all states}}e^{-E(\text{state})/kT}\) is the partition function — the normalizing constant that sums \(e^{-E/kT}\) over every possible state so probabilities sum to 1. The interpretation is direct: lower energy makes \(e^{-E}\) larger, which makes that state more probable.

In deep learning we drop the physical constants — \(k\) and \(T\) are simply absorbed into how the energy function is scaled — and write the relationship in its cleanest form:

$$P(\text{state})\propto e^{-E(\text{state})}$$

Boltzmann Machines take their name, and this core principle, directly from this distribution: they define a learnable energy function over configurations of binary units, and training shapes that energy landscape so that configurations resembling real data end up with low energy (hence high probability).

2. The Boltzmann Machine: Architecture & Energy Function

A Boltzmann Machine has two layers of binary stochastic units:

• Visible units \(v_i\in\{0,1\}\) — represent observed data: pixels, features, ratings.
• Hidden units \(h_j\in\{0,1\}\) — learn latent structure that explains dependencies among the visible units, adding stochasticity so the model can represent complex distributions.

Connections are symmetric (\(w_{ij}=w_{ji}\)) and there are no self-connections. The full energy function includes every possible pairwise interaction: visible biases, hidden biases, visible–visible interactions, hidden–hidden interactions, and visible–hidden interactions:

$$E(v,h)=-\sum_i a_iv_i-\sum_j b_jh_j-\frac12\sum_{i\ne k}v_iw_{ik}v_k-\frac12\sum_{j\ne l}h_jw_{jl}h_l-\sum_{i,j}v_iw_{ij}h_j$$

and the probability of a joint configuration follows the Boltzmann form directly:

$$P(v,h)=\frac{e^{-E(v,h)}}{Z}$$

What do the visible units actually represent? It depends on the data:

3. Why Binary Units? & RBM Variants for Other Data

Binary units were the historical default (Hinton & Sejnowski, 1983) for several concrete reasons: they keep the pairwise energy terms simple; they admit an easy, well-defined stochastic sampling rule via the sigmoid, \(P(v_i=1\mid h)=\sigma\!\big(a_i+\sum_jw_{ij}h_j\big)\); they're cleanly compatible with the Boltzmann distribution's exponential form; and they carry a loose biological analogy to a neuron that either fires or doesn't.

Real data isn't always binary, so the RBM family (introduced formally in Section 5) has variants for other data types:

4. Applications

Boltzmann Machines and RBMs have been used for: feature / representation learning (unsupervised pretraining of features later used by another model), data generation (sampling new visible patterns from the learned distribution), collaborative filtering (RBMs were famously used in the Netflix Prize recommendation system), anomaly detection (high-energy states correspond to unlikely, i.e. anomalous, inputs), and speech, text, and biological sequence modeling.

5. From Boltzmann Machines to RBMs

The full Boltzmann Machine, as elegant as its energy function is, does not scale. Full connectivity means every unit depends on every other unit, which creates exponentially many possible joint states; computing the partition function \(Z\) — a sum over all of them — becomes intractable for anything beyond a toy size. Training requires long Markov-chain (Gibbs sampling) runs just to approximate the expectations the gradient needs, which is slow and doesn't scale past a few dozen units, precisely because visible–visible and hidden–hidden connections make every unit mutually dependent on every other.

$$E(v,h)=-\sum_i a_iv_i-\sum_j b_jh_j-\sum_{i,j}v_iw_{ij}h_j$$

This restriction has one crucial payoff: given \(v\), all the hidden units become conditionally independent of each other (and symmetrically, given \(h\), all visible units become conditionally independent of each other). That's what makes RBM training tractable — the entire hidden layer can be sampled in a single parallel step given the visible layer, and vice versa, with no need to sample one unit at a time while holding the rest fixed.

6. Learning Objective & Gradient Derivation

We want to maximize the log-likelihood of the data under the model, \(\log P(v)\), where \(P(v)=\frac1Z\sum_he^{-E(v,h)}\). It's convenient to write this using the free energy \(F(v)=-\log\sum_he^{-E(v,h)}\), so that \(\log P(v)=-F(v)-\log Z\). Since this splits into two separate terms, we differentiate each one with respect to a weight \(w_{ij}\) in turn: \(\frac{\partial\log P(v)}{\partial w_{ij}}=-\frac{\partial F(v)}{\partial w_{ij}}-\frac{\partial\log Z}{\partial w_{ij}}\). Every algebraic step of both halves is worked out below.

You can replay the same five steps interactively below — useful for testing yourself before moving on:

The positive phase is easy: clamp the visible units to a real data point and compute the exact hidden-unit probabilities — a single, exact computation. The negative phase is the problem: it is an expectation over the model's own distribution, which would require running a long Gibbs-sampling chain to equilibrium to get an unbiased sample — computationally expensive, and the core bottleneck in training any Boltzmann-family model.

7. Contrastive Divergence (CD-k)

Contrastive Divergence is the practical fix that made RBMs trainable at scale: instead of running Gibbs sampling all the way to equilibrium for the negative phase, run just \(k\) steps — most commonly \(k=1\), "CD-1" — starting from the real data point (not from random noise), and treat that short chain's endpoint as a cheap stand-in for the negative-phase expectation.

1. Start at the data point \(v^{(0)}\), sample \(h^{(0)}\sim P(h\mid v^{(0)})\).
2. Run \(k\) Gibbs steps: \(v^{(t+1)}\sim P(v\mid h^{(t)})\), then \(h^{(t+1)}\sim P(h\mid v^{(t+1)})\), for \(t=0,\dots,k-1\).
3. Approximate the negative phase: \(\langle v_ih_j\rangle_{\text{model}}\approx v_i^{(k)}\cdot P(h_j=1\mid v^{(k)})\).

This gives a simple, fully computable update rule:

$$\Delta w_{ij}=\eta\Big[v_i^{(0)}P(h_j{=}1\mid v^{(0)})-v_i^{(k)}P(h_j{=}1\mid v^{(k)})\Big]$$

8. Worked Example: One Full CD-1 Step

A tiny RBM: 2 visible units (\(v_1,v_2\)), 1 hidden unit (\(h\)). Weights \(W=[0.8,\ 0.4]\) (\(w_1\) connects \(v_1\)–\(h\), \(w_2\) connects \(v_2\)–\(h\)), biases \(a_1=a_2=b=0\), learning rate \(\eta=0.1\). Data (clamped): \(v^{(0)}=(1,0)\). Every sigmoid below is computed the same way: evaluate the exponent, exponentiate, add 1, divide into 1 — spelled out in full so nothing is hidden.

You can replay the same four stages interactively below — useful for testing yourself before moving on:

9. Deep Belief Networks (DBN)

A Deep Belief Network is simply a stack of RBMs, trained greedily, one layer at a time: train the first RBM directly on the raw data; then freeze it, run the data through it to get hidden-layer activations, and treat those activations as the "data" for training a second RBM stacked on top; repeat for as many layers as desired. Once every layer has been pretrained this way, the whole stack can be "unrolled" into a single deep network and fine-tuned end-to-end with ordinary backpropagation — typically with a supervised classifier layer added on top for the final task.

Historically (Hinton, Osindero & Teh, 2006), this greedy layer-wise pretraining was one of the first techniques that made training genuinely deep networks practical at all — before that, deep networks were notoriously hard to train from a random initialization. Better weight-initialization schemes, ReLU activations, batch normalization, and residual connections (all covered earlier in this course) have since made layer-wise RBM pretraining largely unnecessary for most modern architectures — but it remains a historically important idea, and the direct ancestor of the encoder-style unsupervised pretraining still used in some settings today.

Summary

10. Code: One CD-1 Step From Scratch

The NumPy code below reproduces every number in the Section 8 worked example exactly — run it and confirm the printed values against the hand derivation.

import numpy as np

def sigmoid(z): return 1 / (1 + np.exp(-z))

# ---- tiny RBM: 2 visible units, 1 hidden unit ----
W = np.array([0.8, 0.4])   # w1 (v1-h), w2 (v2-h)
a = np.array([0.0, 0.0])   # visible biases
b = 0.0                    # hidden bias
eta = 0.1
v0 = np.array([1.0, 0.0])  # clamped data point

# ---- positive phase ----
h0_prob = sigmoid(b + W @ v0)
pos_assoc = v0 * h0_prob
print("h0 =", h0_prob, " positive  =", pos_assoc)

# ---- negative phase: one Gibbs step (CD-1) ----
v1_prob = sigmoid(a + W * h0_prob)          # reconstruct visible units
h1_prob = sigmoid(b + W @ v1_prob)          # resample hidden from reconstruction
neg_assoc = v1_prob * h1_prob
print("v_tilde =", v1_prob, " h1 =", h1_prob, " negative  =", neg_assoc)

# ---- CD-1 parameter updates ----
dW = eta * (pos_assoc - neg_assoc)
da = eta * (v0 - v1_prob)
db = eta * (h0_prob - h1_prob)
print("dW =", dW, " da =", da, " db =", db)

W_new = W + dW
print("Updated W =", W_new)

⬇ Download lecture-22-cd1-rbm.py   More resources for this lecture →