Feedforward Neural Networks & Forward Propagation

Stacking perceptrons into layers solves XOR — but only if we insert a nonlinearity between the layers. Here is the exact notation we will reuse for the rest of the course.

• Write the forward-propagation equations of a two-layer feedforward network using the course's standard notation.
• Track the matrix shape of every intermediate quantity as data flows through the network.
• Multiply two small matrices by hand, row by column, and explain in words what each output entry means.
• Prove algebraically that stacking linear layers without a nonlinearity collapses to a single linear transform.
• State the Universal Approximation Theorem informally and explain what it does — and does not — guarantee.
• Hand-compute a complete forward pass for a small network given real numbers, tracking every intermediate value.

1. Notation and Setup

Lecture 3 showed that a single perceptron can only separate linearly separable data. A feedforward neural network (also called a multi-layer perceptron) fixes this by chaining several layers of weighted sums and nonlinearities together. We now fix the exact notation this course uses for the rest of the term — the same notation reappears unchanged in Lecture 8's backpropagation derivation.

Let \(X\) be the input — a matrix, not a vector, because we process many examples ("samples") at once. If there are 3 data samples, each with 2 features, then \(X\) has shape \((3,2)\): one row per sample, one column per feature. The first layer's weight matrix \(W_1\) has shape \((2,4)\), meaning 2 input features map onto 4 hidden nodes. The forward equations are:

$$Z_1 = XW_1+b_1 \qquad A=\text{ReLU}(Z_1) \qquad Z_2=AW_2+b_2 \qquad Q=\sigma(Z_2)$$

Here \(A\) is the hidden layer's activation — it has exactly the same shape as \(Z_1\), since ReLU is applied element-wise and changes no shape. \(W_2\) has shape \((4,1)\): 4 nodes in the previous (hidden) layer feeding into 1 output node. \(Q\) is the output probability the network produces, and \(L\) is the loss for each input, comparing \(Q\) against the true label. This is precisely the same \(X, W_1, A, W_2, Q, L\) notation Lecture 8 uses — get comfortable with it now.

2. Why Nonlinearity Is Essential: The Collapse Argument

It's tempting to think that stacking more linear layers automatically makes a model more powerful. It does not — without a nonlinearity between them, any number of stacked linear layers is mathematically identical to a single linear layer. Consider two layers with no activation function in between, applied to an input vector \(x\):

$$h = W_1 x, \qquad \text{output} = W_2 h = W_2(W_1 x) = (W_2 W_1)\,x$$

The product \(W_2 W_1\) is just another matrix — call it \(W_{\text{combined}}\). So \(\text{output}=W_{\text{combined}}\,x\) is exactly what a single linear layer would compute. No matter how many linear layers you stack, the entire network can only ever represent linear functions of its input — it gains zero representational power over Lecture 3's single perceptron.

3. The Universal Approximation Theorem

Once a nonlinearity is present, how powerful does a feedforward network actually become? The Universal Approximation Theorem (Cybenko, 1989; Hornik, 1991 — standard results, not from the course's local slides) gives a remarkable, if informal, answer: a feedforward network with a single hidden layer, given enough hidden units and a suitable nonlinear activation, can approximate any continuous function on a compact (closed, bounded) input domain to arbitrary accuracy.

4. The Network, Diagrammed

Here is the exact architecture implied by the shapes above — 2 input features, 4 hidden nodes (\(W_1\) shape \((2,4)\)), 1 output node (\(W_2\) shape \((4,1)\)):

This picture shows the structure — which nodes connect to which — but no actual numbers flow through it yet. Before we can plug real numbers in, we need to be completely comfortable with what "\(X W_1\)" as a computation actually does, arithmetically. Section 6 builds that up from scratch; Section 7 then runs real numbers all the way through this exact network.

5. Tracking Matrix Shapes

Getting matrix shapes right is the single most common source of bugs when implementing a network. For 3 samples with 2 features each, flowing through the 2 → 4 → 1 network above:

6. How Matrix Multiplication Actually Works

Everything a neural network does — forward pass, backpropagation, all of it — reduces to matrix multiplications like \(XW_1\). If you're already comfortable multiplying matrices by hand, skip straight to the worked example in Section 7. If it's a little rusty or you've never seen the row/column mechanics spelled out, expand the primer below — it builds the whole operation up from a single dot product using the exact numbers Section 7 needs.

New to this? Expand: matrix multiplication from scratch, worked entry-by-entry

6.1 The one operation underneath everything: the dot product

Before matrices, there is one even smaller operation: the dot product of two same-length lists of numbers. Given two lists \(a=[a_1,a_2]\) and \(b=[b_1,b_2]\), their dot product is a single number:

$$a\cdot b = a_1 b_1 + a_2 b_2$$

In words: multiply the numbers in matching positions, then add up all the products. For example, \([0.6,-0.2]\cdot[0.2,-0.3] = (0.6)(0.2)+(-0.2)(-0.3) = 0.12+0.06 = 0.18\). That's it — that is the entire operation. Everything below is just "do this dot product many times, in an organized pattern."

6.2 The matrix-multiplication rule

To multiply a matrix \(A\) (shape \(m\times n\)) by a matrix \(B\) (shape \(n\times p\)), the number of columns of \(A\) must equal the number of rows of \(B\) — both equal to \(n\) here. This shared number \(n\) is exactly the length of the lists we take a dot product of. The result \(AB\) has shape \(m\times p\), and its entry in row \(i\), column \(j\) is defined as:

$$(AB)_{ij} = (\text{row } i \text{ of } A)\ \cdot\ (\text{column } j \text{ of } B)$$

In words: to compute output entry \((i,j)\), take row \(i\) out of the left matrix, take column \(j\) out of the right matrix, and dot-product them. Row \(i\) is only ever paired with row \(i\)'s own output; column \(j\) is only ever paired with column \(j\)'s own output. This is why the "inner" dimensions (columns of \(A\), rows of \(B\)) must match — the two lists being dot-producted have to be the same length — and why the "outer" dimensions (rows of \(A\), columns of \(B\)) survive into the output shape.

🔢 Let's actually do it: X · W1, entry by entry

Use the exact numbers from Section 7's worked example: \(X=[0.6,\,-0.2]\) (shape \(1\times2\)) and \(W_1=\begin{bmatrix}0.2 & 0.4\\-0.3 & 0.1\end{bmatrix}\) (shape \(2\times2\)). Since \(X\) has 2 columns and \(W_1\) has 2 rows, they're compatible, and the result \(XW_1\) will have shape \(1\times2\) (1 row from \(X\), 2 columns from \(W_1\)).

Output entry (1,1) — row 1 of \(X\), dotted with column 1 of \(W_1\):

row 1 of \(X\) = \([0.6,\,-0.2]\)    column 1 of \(W_1\) = \([0.2,\,-0.3]\) (the first number of each row of \(W_1\))

$$(0.6)(0.2) + (-0.2)(-0.3) = 0.12 + 0.06 = \mathbf{0.18}$$

Output entry (1,2) — the same row of \(X\), dotted with column 2 of \(W_1\) instead:

row 1 of \(X\) = \([0.6,\,-0.2]\)    column 2 of \(W_1\) = \([0.4,\,0.1]\) (the second number of each row of \(W_1\))

$$(0.6)(0.4) + (-0.2)(0.1) = 0.24 - 0.02 = \mathbf{0.22}$$

So \(XW_1 = [0.18,\ 0.22]\). Two numbers in, one \(2\times2\) matrix in, two numbers out — one output number per column of \(W_1\), and every single output number used every input number (because each dot product runs over the full row of \(X\)). That's the sense in which every hidden unit in Lecture 3/4's picture "sees" every input feature — it falls straight out of the matrix-multiplication rule, not from anything special about neural networks.

The habit to build

Whenever you see \(Z=XW\) anywhere in this course (or in any deep learning paper/codebase), mentally expand it to: "for every row of \(X\) and every column of \(W\), dot-product them to get one output number." If you can always answer "what are the shapes of \(X\) and \(W\), and therefore what is the shape of the output, and therefore how many dot products am I computing" — you will never again be confused by a matrix-shape error, which the Section 5 table said is the single most common implementation bug.

7. Worked Numerical Example: A Single Forward Pass

To make the equations concrete, we hand-compute one complete forward pass for a single example through a smaller 2 → 2 → 1 network (this is the same architecture family Lecture 8 uses for its full worked example — but with different weights, since this pass only goes forward; how the network learns from its output is the subject of Lectures 7–8). Every arithmetic step is written out below, in order — nothing is left for you to fill in.

You can replay the same four steps interactively below — useful for testing yourself before moving on:

8. Summary

9. Code: A Forward Pass, From Scratch

This NumPy function reproduces the worked example above exactly — run it and confirm the printed values match the hand-derived numbers.

import numpy as np

def relu(z):    return np.maximum(0, z)
def sigmoid(z):  return 1 / (1 + np.exp(-z))

def forward_pass(x, W1, b1, W2, b2):
    Z1 = x @ W1 + b1
    A  = relu(Z1)
    Z2 = A @ W2 + b2
    Q  = sigmoid(Z2)
    return Z1, A, Z2, Q

# ---- matches the worked example in the lecture (2 -> 2 -> 1 network) ----
x  = np.array([[0.6, -0.2]])                     # (1, 2)
W1 = np.array([[0.2, 0.4], [-0.3, 0.1]])          # (2, 2)
b1 = np.array([[0.1, 0.0]])                       # (1, 2)
W2 = np.array([[0.5], [-0.6]])                    # (2, 1)
b2 = np.array([[0.2]])                            # (1, 1)

Z1, A, Z2, Q = forward_pass(x, W1, b1, W2, b2)
print("Z1 =", Z1)   # expect [[0.28, 0.22]]
print("A  =", A)    # ReLU leaves both unchanged (both positive)
print("Z2 =", Z2)   # expect [[0.208]]
print("Q  =", Q)    # expect ~[[0.5518]]

# ---- sanity check: a purely linear network collapses to one matrix ----
W_linear_combined = W1 @ W2
print("\nW2 . W1 collapsed into a single (2,1) matrix (no activation case):")
print(W_linear_combined)

⬇ Download lecture-04-forward-pass.py   More resources for this lecture →