CNNs: Motivation & Architecture

Why feeding raw pixels into a dense network is a losing proposition — and the four-layer blueprint, inspired by the visual cortex, that made computer vision tractable for deep nets.

• State, with exact numbers, why a dense/feedforward network scales catastrophically on raw image input.
• List the three concrete disadvantages of MLPs for image data, and how convolution fixes each one.
• Trace CNN history from the Neocognitron and biological vision through LeNet-5.
• Describe the generic 4-layer CNN blueprint (Input → Convolution → Pooling → Fully Connected) and the deep pipeline built from it.
• Explain why convolutional layers are parameter-cheap while fully-connected layers dominate parameter count — and how later architectures exploited that.
• Compare AlexNet, VGG16, GoogLeNet and ResNet on accuracy and parameter budget.

1. Why Not Just Use a Dense Network on Images?

Every network we built through Lecture 11 was a stack of fully-connected (dense) layers: flatten the input into a 1D vector, multiply by a weight matrix, repeat. Nothing stops us from doing the same thing to an image — flatten the pixels into one long vector and feed it to an MLP. The reason nobody does this at scale is a matter of simple arithmetic.

The parameter count is the headline problem, but it isn't the only one. The instructor's framing gives three concrete disadvantages of MLPs on image data:

1. Too many parameters. As above — a fully-connected layer scales as (number of pixels) × (number of hidden units), which explodes for anything but tiny images.
2. No notion of local spatial structure. An MLP treats every pixel as an independent input feature; it has no built-in concept of which pixels are neighbors. It cannot exploit the fact that edges, textures and shapes are formed by small groups of nearby pixels. CNNs instead use small filters (e.g. 3×3, 5×5) that look at a local neighborhood at a time, directly encoding "nearby pixels are related."
3. No translation invariance. If an MLP learns to recognize a cat when it appears in the top-left of the frame, there is no guarantee it recognizes the same cat shifted to the center of the frame — every pixel position has its own independent weights. A CNN's convolution-plus-pooling combination applies the same filter everywhere in the image and then pools over small neighborhoods, so the same pattern is detected regardless of where it appears.

2. A Brief History: From Visual Cortex to LeNet-5

CNNs are not a purely mathematical invention — they are directly inspired by biology, and the architecture evolved over three decades before "deep learning" was a common phrase.

• Hubel & Wiesel (1959) studied the cat visual cortex and found two kinds of cells: simple cells that respond to local, orientation-specific edges in a small receptive field, and complex cells that pool the responses of many simple cells to become invariant to the exact position of the edge. This simple-cell → complex-cell hierarchy is the biological blueprint for convolution → pooling.
• Neocognitron (Fukushima, 1980) was the first computational architecture built directly on that biology — layers of "S-cells" (feature detectors, like convolution) alternating with "C-cells" (spatial pooling for invariance), stacked hierarchically.
• Time-Delay Neural Network (Waibel & Hinton, 1987) applied the same idea to sequences: 16 convolution kernels slid over 15-element windows, with the kernel weights shared across every window position and the whole network trained end-to-end with backpropagation. This introduced the crucial engineering idea of learned, shared-weight convolution kernels trained by gradient descent (rather than hand-designed filters).
• First CNN for images (LeCun, 1989) combined convolution, weight sharing and backprop specifically for 2D image recognition (handwritten digits) for the first time.
• LeNet-5 (LeCun, 1989) is the architecture most people mean by "the first CNN" — small, entirely hand-inspectable, and still the cleanest worked example of the pattern. Its own defining choices: 5×5 filters, stride 1, sigmoid nonlinearity, 2×2 stride-2 pooling, grayscale input, and no zero padding (so the feature maps shrink at every convolution).

3. The 4-Layer CNN Architecture

Strip away the historical detail and a typical CNN reduces to exactly four kinds of layers, always in this order:

Input layer. A 28×28 grayscale image is presented to the network as a 2D matrix — not flattened to a 1D vector the way an MLP would require. This is deliberate: keeping the image 2D preserves the spatial relationships between neighboring pixels, which is exactly what Section 1 said an MLP throws away.

Convolutional layer. In practice a "convolutional layer" is really a composition of three operations applied in sequence:

Fully connected layer. After several convolution+pooling blocks have distilled the image into a compact set of feature maps, those maps are flattened and passed through one or more dense layers (Lecture 4's math, unchanged) ending in a classifier — full details in Lecture 14.

4. The Deep CNN Pipeline

Stacking several convolution+pooling blocks before the classifier head gives the standard deep CNN pipeline:

$$\text{Input} \to \text{CL}_1 \to \text{CL}_2 \to \text{CL}_3 \to \text{CL}_4 \to \text{FCL}_1 \to \text{FCL}_2 \to \text{Output}$$

where CL = convolutional layer (sparse connectivity — each output unit only "sees" a small local patch of its input) and FCL = fully connected layer (dense connectivity — every output unit sees every input unit).

5. Where Do the Parameters Actually Live?

The two halves of the pipeline behave very differently in terms of parameter count. Convolutional layers hold relatively few parameters — each filter is small (e.g. 3×3 or 5×5) and, crucially, the same filter weights are reused (shared) at every spatial position, so the parameter count doesn't grow with image size. Fully connected layers, by contrast, connect every input unit to every output unit, so their parameter count is the product of two potentially large numbers — and they end up dominating the network's total parameter budget.

6. Landmark Architectures: Where This Technology Went

Full convolution arithmetic is Lecture 13's job. Here, as a motivating close, is where the 4-layer blueprint led over the following decade of the ImageNet Large Scale Visual Recognition Challenge (ILSVRC).

AlexNet (2012) took a 227×227×3 RGB input through 5 convolutional layers and 3 fully connected layers, split across two GTX 580 GPUs (a hardware constraint of the time), and trained for 5–6 days. It introduced ReLU activations, dropout regularization on the dense layers, and aggressive data augmentation — and its 15.4% Top-5 error was a dramatic improvement over prior hand-engineered feature pipelines. Interestingly, only about 6% of its 62.3 million parameters sit in the convolutional layers, yet those same convolutional layers consume roughly 95% of the network's compute time — parameters and compute are concentrated in opposite ends of the network.

VGG16 (2014) simplified the design space to a single repeated building block — 3×3 convolution, stride 1, followed by 2×2 max-pooling, stride 2 — stacked 16 convolutional layers deep, and it improved Top-5 error to 7.3%. The cost, as Section 5 showed, was a parameter budget of ~140 million, 85% of it sitting in the three FC layers.

GoogLeNet / Inception (2015) attacked that FC parameter cost directly: it uses average pooling instead of flatten+dense at the end, has no fully-connected layers at all, and needs only 5 million parameters — 12× fewer than AlexNet — while still improving accuracy to 6.7% Top-5 error. Its Inception module runs several kernel sizes in parallel within the same layer, capturing features at multiple scales simultaneously. A key trick that keeps this affordable is the 1×1 convolution, used purely to shrink channel depth before an expensive larger convolution:

ResNet (2015), at 152 layers, is described in the course material as the "beginning of the ultra-deep network era." Its defining trick is the residual (skip) connection: \(F(X)+X\) — the input to a small block of conv+ReLU layers is added back to that block's output before the next activation. During backpropagation, the gradient has a direct, unimpeded path through the shortcut, which is exactly what solves the vanishing-gradient-with-depth problem that had made networks this deep untrainable before. The result: 3.57% Top-5 error, below the commonly cited human-level benchmark on this task.

7. Common Pitfalls & Misconceptions

8. Summary

9. Code: Counting the Parameters

This script reproduces every numeric claim made above — the MLP parameter explosion, the generic conv-layer parameter formula, the 1×1 convolution channel-reduction example, and the landmark-architecture comparison table — as plain, runnable Python (no deep learning framework required).

"""
Lecture 12 -- CNNs: Motivation & Architecture
Reproduces the numeric claims from the lecture: the MLP parameter
explosion on raw images, the conv-layer parameter-count formula, the
1x1 convolution channel-reduction example, and a landmark-architecture
comparison table.
"""

# ---- 1. Why not a dense network on images? ----
H, W, D = 224, 224, 3
flat_inputs = H * W * D
hidden_units = 1000
mlp_weights = flat_inputs * hidden_units

print(f"Flattened 224x224x3 image  -> {flat_inputs:,} inputs")
print(f"Dense layer to {hidden_units} units -> {mlp_weights:,} weights "
      f"(~{mlp_weights/1e6:.1f} million)\n")

# ---- 2. Generic conv-layer parameter count ----
def conv_params(K, D_in, D_k):
    """K x K filter, D_in input channels, D_k output filters (+1 bias each)."""
    return (K * K * D_in * D_k) + D_k

# e.g. 15 filters of 3x3 over a 3-channel input (Lecture 13's example)
p = conv_params(K=3, D_in=3, D_k=15)
print(f"3x3 conv, 3 input channels, 15 filters -> {p:,} parameters "
      f"(compare to the {mlp_weights:,} MLP weights above)\n")

# ---- 3. 1x1 convolution as a channel-depth bottleneck (GoogLeNet trick) ----
in_h, in_w, in_channels = 56, 56, 64
num_1x1_filters = 5
out_shape = (in_h, in_w, num_1x1_filters)
reduction_factor = in_channels / num_1x1_filters
print(f"1x1 conv: {in_h}x{in_w}x{in_channels} -> {out_shape[0]}x{out_shape[1]}x{out_shape[2]}"
      f"  ({reduction_factor:.1f}x channel reduction)\n")

# ---- 4. Landmark architecture comparison ----
architectures = [
    {"name": "AlexNet",   "year": 2012, "params_m": 62.3, "top5_err": 15.4},
    {"name": "VGG16",     "year": 2014, "params_m": 140.0, "top5_err": 7.3},
    {"name": "GoogLeNet", "year": 2015, "params_m": 5.0,  "top5_err": 6.7},
    {"name": "ResNet-152","year": 2015, "params_m": 60.0, "top5_err": 3.57},
]
print(f"{'Model':<12}{'Year':>6}{'Params (M)':>13}{'Top-5 err %':>14}")
for a in architectures:
    print(f"{a['name']:<12}{a['year']:>6}{a['params_m']:>13.1f}{a['top5_err']:>14.2f}")

fewer_params_than_alexnet = architectures[0]["params_m"] / architectures[2]["params_m"]
print(f"\nGoogLeNet uses {fewer_params_than_alexnet:.1f}x fewer parameters than "
      f"AlexNet, yet has lower Top-5 error -- accuracy and parameter count "
      f"are not the same axis.")

⬇ Download lecture-12-cnn-motivation.py   More resources for this lecture →