Backpropagation: The Core Algorithm

How a network turns a single scalar loss into a precise update for every weight — the chain rule, applied mechanically, layer by layer.

• Explain why a naive/brute-force weight search is computationally infeasible, motivating backpropagation.
• Derive gradients of a loss function with respect to every weight using the multivariate chain rule.
• Trace a complete forward pass and backward pass through a small network with real numbers, by hand.
• Write backpropagation in vectorized (matrix) form suitable for mini-batches.
• Recognize common implementation pitfalls: shape mismatches, dead ReLUs, and forgetting the local gradient of the activation.

1. Why Backpropagation?

A neural network's job during training is to find the set of weights \(W\) that minimizes a loss function \(L(W)\). Lecture 7 showed that gradient descent does this by repeatedly moving weights in the direction opposite to the gradient: \(W \leftarrow W - \eta \nabla_W L\). That leaves one question unanswered: for a network with millions of weights spread across many layers, how do we actually compute \(\nabla_W L\) efficiently?

Backpropagation is not a different learning algorithm — it is simply an efficient, systematic application of the chain rule of calculus that computes every one of those gradients in roughly the same time as one forward pass. It was popularized for neural networks by Rumelhart, Hinton & Williams (1986) and remains the computational backbone of every deep learning framework in use today.

2. Setting Up the Network

We'll use the exact notation from the course slides: input \(X\), first weight matrix \(W_1\) with bias \(b_1\), a ReLU-activated hidden layer \(A\), a second weight matrix \(W_2\) with bias \(b_2\), a sigmoid output \(Q\), and a loss \(L\) comparing \(Q\) against the true label \(y\).

3. The Forward Pass

Every backward pass starts by first running forward and caching every intermediate value — we will need \(X\), \(Z_1\), \(A\), \(Z_2\), \(Q\) again during the backward pass.

$$Z_1 = XW_1+b_1 \qquad A=\text{ReLU}(Z_1) \qquad Z_2=AW_2+b_2 \qquad Q=\sigma(Z_2)=\frac{1}{1+e^{-Z_2}}$$

For a binary classification target \(y \in \{0,1\}\), the loss is binary cross-entropy:

$$L = -\big[y\log Q + (1-y)\log(1-Q)\big]$$

4. The Chain Rule, Formally

We want \(\partial L/\partial W_1\) and \(\partial L/\partial W_2\) (and the biases). \(L\) depends on \(W_2\) only through \(Z_2\), and on \(W_1\) through the chain \(Z_1 \to A \to Z_2\). The multivariate chain rule lets us decompose each gradient into a product of local derivatives:

$$\frac{\partial L}{\partial W_2} = \frac{\partial L}{\partial Q}\cdot\frac{\partial Q}{\partial Z_2}\cdot\frac{\partial Z_2}{\partial W_2} \qquad\qquad \frac{\partial L}{\partial W_1} = \underbrace{\frac{\partial L}{\partial Q}\cdot\frac{\partial Q}{\partial Z_2}}_{\delta_2}\cdot\frac{\partial Z_2}{\partial A}\cdot\frac{\partial A}{\partial Z_1}\cdot\frac{\partial Z_1}{\partial W_1}$$

The key algorithmic insight: \(\delta_2 = \frac{\partial L}{\partial Q}\cdot\frac{\partial Q}{\partial Z_2}\), once computed, is reused for every weight upstream of it. This is why backprop walks backward through the graph, layer by layer, instead of recomputing the whole chain for every individual weight.

5. Backward Pass — Layer by Layer

With \(\delta_2=\partial L/\partial Z_2=Q-y\) in hand, the rest of the backward pass is mechanical: walk backward through the same equations used in the forward pass, one layer at a time, applying the chain rule at each step. Every gradient below is written out in full — the interactive recap beneath it replays the same five steps if you want to test yourself afterward.

That's the whole algorithm: \(\delta_2\to(\partial W_2,\partial b_2)\to\partial A\to\delta_1\to(\partial W_1,\partial b_1)\) — five short steps, each one reusing the previous step's output, and each one requiring nothing more than a matrix multiply, a transpose, or an element-wise product. Try it yourself, one step at a time:

6. Worked Numerical Example

Now the exact same five steps, with real numbers, computed entirely by hand from start to finish. Input \(x=[0.5,\ 0.8]\), target \(y=1\), learning rate \(\eta=0.1\).

Prefer to click through it interactively instead of re-reading? Same numbers, same order:

7. Vectorized Form (Mini-Batches)

Real training doesn't process one example at a time — it processes a batch of \(m\) examples stacked as rows of \(X\) (shape \(m\times n\)). Every equation above still holds, just with matrices instead of vectors, and gradients w.r.t. biases summed over the batch dimension:

$$\delta_2 = Q-y \quad(m\times 1) \qquad \frac{\partial L}{\partial W_2}=\frac{1}{m}A^\top\delta_2 \qquad \frac{\partial L}{\partial b_2}=\frac{1}{m}\sum_i \delta_2^{(i)}$$ $$\delta_1 = (\delta_2 W_2^\top)\odot \text{ReLU}'(Z_1) \qquad \frac{\partial L}{\partial W_1}=\frac{1}{m}X^\top\delta_1 \qquad \frac{\partial L}{\partial b_1}=\frac{1}{m}\sum_i \delta_1^{(i)}$$

Here \(\odot\) is element-wise multiplication (Hadamard product) — it appears because each hidden unit's activation function acts independently on its own pre-activation.

8. Common Pitfalls

9. Summary

10. Code: Backpropagation From Scratch

The NumPy implementation below reproduces the worked example above exactly — run it and confirm the printed gradients match the hand-derived numbers.

import numpy as np

def sigmoid(z): return 1 / (1 + np.exp(-z))
def relu(z):    return np.maximum(0, z)
def relu_grad(z): return (z > 0).astype(float)

# ---- data & parameters (matches the worked example in the lecture) ----
X  = np.array([[0.5, 0.8]])                     # (1, 2)
y  = np.array([[1.0]])                          # target
W1 = np.array([[0.3, -0.1], [0.2, 0.4]])         # (2, 2)
b1 = np.array([[0.1, -0.2]])                     # (1, 2)
W2 = np.array([[0.5], [-0.3]])                   # (2, 1)
b2 = np.array([[0.05]])                          # (1, 1)
lr = 0.1

# ---- forward pass ----
Z1 = X @ W1 + b1
A  = relu(Z1)
Z2 = A @ W2 + b2
Q  = sigmoid(Z2)
L  = -(y * np.log(Q) + (1 - y) * np.log(1 - Q))
print("Q =", Q, " Loss =", L)

# ---- backward pass ----
dZ2 = Q - y                       # sigmoid + BCE shortcut
dW2 = A.T @ dZ2
db2 = dZ2.sum(axis=0, keepdims=True)

dA  = dZ2 @ W2.T
dZ1 = dA * relu_grad(Z1)
dW1 = X.T @ dZ1
db1 = dZ1.sum(axis=0, keepdims=True)

print("dW2 =\n", dW2, "\ndW1 =\n", dW1)

# ---- gradient descent update ----
W1 -= lr * dW1; b1 -= lr * db1
W2 -= lr * dW2; b2 -= lr * db2
print("Updated W2 =", W2.ravel(), " Updated b2 =", b2.ravel())

⬇ Download lecture-08-backprop.py   More resources for this lecture →