Autoencoders: Architecture, Training, Sparse & Denoising Variants

Train a network to reconstruct its own input, and the bottleneck it must squeeze through becomes a compressed, self-discovered representation — no labels required.

• Explain autoencoders as unsupervised representation learning, and state precisely how a linear autoencoder relates to PCA.
• Describe the encoder → bottleneck → decoder architecture and explain why an unconstrained, bottleneck-free autoencoder can trivially learn the identity function.
• Write the MSE and BCE reconstruction losses and identify autoencoder training as self-supervised learning with target \(y=x\).
• Hand-compute a full encode → decode → loss forward pass through a small linear autoencoder.
• Formulate the L1 and KL-divergence sparsity penalties and connect the KL term back to Lecture 6's definition.
• Explain denoising autoencoders — three standard noise models, and why training against the clean target (not the corrupted input) forces the network to learn robust features.

1. Why Autoencoders? Unsupervised Representation Learning

Every architecture in this course so far — from the perceptron (Lecture 3) through CNNs, RNNs, and Transformers — was trained in a supervised setting: a labeled pair \((x,y)\) told the network exactly what output to produce. An autoencoder breaks that pattern. It is trained to reconstruct its own input: given \(x\), produce \(\hat x \approx x\). No external label is required — the input itself supplies the target. This makes autoencoders a form of unsupervised representation learning: by forcing the network to compress and then reconstruct data, we make it discover, on its own, which features of the data are most informative.

You already know one classical technique for compressing data while preserving as much information as possible: Principal Component Analysis (PCA). PCA finds a linear lower-dimensional subspace — the directions of maximum variance in the data, obtained via eigendecomposition of the covariance matrix (equivalently, SVD) — and projects each point onto that subspace. Every operation in PCA is linear.

An autoencoder generalizes this idea: because its encoder and decoder can include nonlinear activation functions (Lecture 5), it can learn a nonlinear lower-dimensional manifold — a curved, more flexible surface that a purely linear method like PCA cannot represent. This is precisely why autoencoders can capture structure PCA misses.

2. Architecture: Encoder, Bottleneck, Decoder

An autoencoder has three logical parts, chained together:

• Encoder \(f_\theta\): compresses the input into a lower-dimensional code. $$z = f_\theta(x) = \sigma(W_ex + b_e), \qquad x\in\mathbb R^n,\ z\in\mathbb R^m,\ m
• Latent space / bottleneck: the code \(z\) itself — the narrowest point of the network.
• Decoder \(g_\phi\): expands the code back toward the original input dimensionality. $$\hat x = g_\phi(z) = \sigma'(W_dz + b_d), \qquad \hat x \in \mathbb R^n$$

\(W_e,b_e\) and \(W_d,b_d\) are the encoder's and decoder's own weight matrix and bias — the same \(W,b\) notation family from Lecture 4, duplicated for two chained layers (or layer stacks). The activation functions \(\sigma,\sigma'\) are chosen per use case: \(\sigma'\) is often sigmoid when inputs are normalized to \([0,1]\) (e.g. pixel intensities), so \(\hat x\) is guaranteed to land in the same range as \(x\).

3. Working Principle & Reconstruction Loss

The information flow is a straight pipeline: \(x \to \text{Encode} \to z \to \text{Decode} \to \hat x\). Training minimizes a reconstruction loss that measures how far \(\hat x\) is from \(x\). Two standard choices, both already familiar from Lecture 6:

$$L_{MSE} = \frac1n\sum_{i=1}^n (x_i-\hat x_i)^2 \qquad\text{(continuous-valued inputs)}$$ $$L_{BCE} = -\frac1n\sum_{i=1}^n\big[x_i\log\hat x_i + (1-x_i)\log(1-\hat x_i)\big] \qquad\text{(inputs normalized to }[0,1]\text{)}$$

This is exactly the binary cross-entropy formula from Lecture 8's backprop derivation — there it compared a single predicted probability \(Q\) against a single label \(y\); here it is applied per input dimension (e.g. per pixel) and averaged over all \(n\) dimensions of \(x\).

4. Training: Self-Supervision via Backpropagation

Nothing about the training procedure is new. Section 3's loss \(L\) is a differentiable scalar function of every weight in the encoder and decoder, so training an autoencoder is Lecture 7's gradient descent (or a mini-batch/Adam variant from Lecture 9) computing gradients via Lecture 8's backpropagation, applied end-to-end through the encoder and decoder as if they were one deeper network:

$$W \leftarrow W - \eta \nabla_W L \qquad \text{for every } W \in \{W_e,b_e,W_d,b_d\}$$

The one genuinely new idea is where the target comes from: instead of a separately supplied label \(y\), the target is the input itself, \(y = x\). This is called self-supervision — the network supervises itself using structure already present in unlabeled data, which is exactly what makes autoencoders useful whenever labels are scarce or unavailable.

5. Worked Numerical Example: A Linear Autoencoder

To make every equation above completely concrete, consider a linear autoencoder — no activation function in the encoder or decoder, so per Section 1's remark this specific example behaves like a 1-component PCA — compressing a 3-dimensional input down to a 1-dimensional latent code.

You can replay the same four steps interactively below — useful for testing yourself before moving on:

6. Sparse Autoencoders

Even with a bottleneck, an autoencoder is not automatically safe from degenerate solutions — with enough capacity in \(W_e,W_d\), a network can learn a diffuse code that reconstructs adequately without any individual latent unit meaning anything in particular. A sparse autoencoder adds an explicit sparsity constraint: for any given input, most latent units should be inactive (near zero), and only a small subset should fire. This is loosely inspired by biological neurons — for any single stimulus, only a small fraction of the neurons in a cortical region are believed to be highly active — and the resulting representations tend to be more feature-selective (each active unit responds to something specific) and less prone to overfitting.

Two standard ways to enforce sparsity:

This is exactly Lecture 6's KL-divergence definition — there it measured the distance between two general probability distributions; here it is applied to the specific case of two Bernoulli distributions ("fires" vs. "doesn't fire"), one per latent unit.

The full picture, all four latent units, target \(\rho=0.05\):

7. Denoising Autoencoders

Sparsity constrains the latent space directly. Denoising autoencoders take a different approach to blocking the trivial identity shortcut: instead of restricting what the network can do internally, corrupt the input itself. The encoder–decoder is trained to strip the corruption back out:

$$\tilde x = x + \text{noise} \qquad L = \|x - g_\phi(f_\theta(\tilde x))\|^2 \quad\text{(loss measured against the ORIGINAL clean } x\text{)}$$

Three standard ways to corrupt \(x\):

• Gaussian noise: add \(\epsilon\sim\mathcal N(0,\sigma^2)\) independently to every feature/pixel.
• Masking noise: randomly zero out a fraction of the input's features/pixels (a stand-in for missing data).
• Salt-and-pepper noise: randomly flip individual pixels to the two extreme intensity values (pure black/white).

Because the loss is measured against the clean \(x\), not \(\tilde x\), simply copying the corrupted input straight through would not minimize the loss — the corruption would still be present, and the loss would still be large. The only way to genuinely reduce the loss is to learn what the underlying clean structure of the data looks like and reconstruct that, discarding the noise. This is what makes the learned features robust.

The grid and stepper below replay the same steps interactively — useful for testing yourself before moving on:

8. Pitfalls & Practical Notes

9. Summary

10. Code: Autoencoder Forward Pass, Denoising & Sparsity

The NumPy implementation below reproduces the worked examples above exactly — run it and confirm the printed MSE values match the hand-derived numbers (≈0.14442 clean, ≈0.19207 denoised), plus the KL-divergence sparsity penalties from the table.

import numpy as np

# ---- linear autoencoder: 3 -> 1 -> 3 (matches the lecture's worked example) ----
x  = np.array([1.0, 0.5, -0.5])
We = np.array([0.4, 0.3, -0.2])          # encoder weights (3,)
be = 0.1
Wd = np.array([0.5, 0.4, -0.3])          # decoder weights (3,)
bd = np.array([0.05, 0.02, -0.01])

def encode(inp): return inp @ We + be
def decode(z):    return z * Wd + bd
def mse(a, b):     return np.mean((a - b) ** 2)

# ---- clean forward pass ----
z     = encode(x)
x_hat = decode(z)
loss  = mse(x, x_hat)
print("Clean:    z =", z, " x_hat =", x_hat, " MSE =", round(loss, 5))

# ---- denoising: masking noise zeroes feature x2 (index 1) ----
x_tilde     = x.copy(); x_tilde[1] = 0.0
z_tilde     = encode(x_tilde)
x_hat_noisy = decode(z_tilde)
loss_noisy  = mse(x, x_hat_noisy)          # <-- against the CLEAN x, not x_tilde
print("Denoised: z =", z_tilde, " x_hat =", x_hat_noisy, " MSE =", round(loss_noisy, 5))
# expected: Clean MSE ~ 0.14442, Denoised MSE ~ 0.19207

# ---- sparse autoencoder: KL-divergence sparsity penalty ----
def kl_sparsity(rho, rho_hat):
    return rho * np.log(rho / rho_hat) + (1 - rho) * np.log((1 - rho) / (1 - rho_hat))

rho       = 0.05                                    # target average activation
rho_hats  = np.array([0.03, 0.06, 0.30, 0.08])       # 4 latent units' actual average activation
penalties = kl_sparsity(rho, rho_hats)
for j, (rh, p) in enumerate(zip(rho_hats, penalties), start=1):
    print(f"  unit z{j}: rho_hat={rh:.2f}  KL penalty={p:.5f}")
print("Total sparsity penalty (sum over units):", round(penalties.sum(), 5))

⬇ Download lecture-23-autoencoder.py   More resources for this lecture →