Building Neural Networks in Python (NumPy + Keras)

Everything since Lecture 4 has been building toward this: the exact same network, first written by hand, then written in eight lines of Keras.

• Recognize how Lectures 4–9's math (forward pass, activations, loss, backprop, gradient descent, optimizers) assembles into one trainable program.
• Implement a complete NeuralNetwork class in NumPy with forward, backward, and train methods, generalized to a batch of examples.
• Train that network on a small toy dataset and observe the loss decrease epoch over epoch.
• Build the identical architecture in Keras/TensorFlow and map every tf.keras call back to the specific lecture that derived its math.
• Explain, at a conceptual level, how reverse-mode automatic differentiation replaces hand-derived backpropagation in production frameworks.

1. Recap: From Math to Code

Every lecture since Module A has contributed one ingredient to a single recipe. Lecture 4 gave the forward pass \(Z=XW+b\). Lecture 5 gave activation functions like ReLU and sigmoid. Lecture 6 gave the loss function — binary cross-entropy. Lecture 7 gave the update rule \(w\leftarrow w-\eta\nabla_w L\). Lecture 8 showed exactly how to compute \(\nabla_w L\) via the chain rule. Lecture 9 showed smarter ways to turn that gradient into a step (Adam, RMSProp, etc.). Lecture 10 gave us a way to judge whether any of it worked.

This lecture assembles all of it into working code, twice: once entirely by hand in NumPy — so nothing is hidden — and once in Keras, where a few lines of configuration replace hundreds of lines of hand-written calculus. We reuse the exact architecture from Lecture 8: 2 inputs \(\to\) 2 hidden units (ReLU) \(\to\) 1 output (sigmoid), with binary cross-entropy loss.

2. Building It in NumPy: A NeuralNetwork Class

Lecture 8 hand-computed one forward+backward pass for a single example. Here we generalize that to a small batch of \(m=4\) examples — the two-input AND-gate points first introduced in Lecture 3 (\(X=[[0,0],[0,1],[1,0],[1,1]]\), target \(y=[0,0,0,1]\)) — using precisely the vectorized update equations from Lecture 8's Section 7:

$$\delta_2 = Q-y \quad\ \frac{\partial L}{\partial W_2}=\frac{1}{m}A^\top\delta_2 \quad\ \delta_1 = (\delta_2 W_2^\top)\odot\text{ReLU}'(Z_1) \quad\ \frac{\partial L}{\partial W_1}=\frac{1}{m}X^\top\delta_1$$

The class below implements exactly this: a forward method that caches every intermediate value, a backward method that applies the chain rule using those cached values, and a train method that loops gradient descent (Lecture 7) over many epochs.

class NeuralNetwork:
    def __init__(self, seed=42):
        rng = np.random.default_rng(seed)
        self.W1 = rng.normal(0, 0.5, size=(2, 2))
        self.b1 = np.zeros((1, 2))
        self.W2 = rng.normal(0, 0.5, size=(2, 1))
        self.b2 = np.zeros((1, 1))

    def forward(self, X):
        self.X  = X
        self.Z1 = X @ self.W1 + self.b1
        self.A  = relu(self.Z1)
        self.Z2 = self.A @ self.W2 + self.b2
        self.Q  = sigmoid(self.Z2)
        return self.Q

    def backward(self, y, lr):
        m = y.shape[0]
        dZ2 = self.Q - y                        # sigmoid + BCE shortcut (Lecture 8)
        dW2 = self.A.T @ dZ2 / m
        db2 = np.sum(dZ2, axis=0, keepdims=True) / m
        dA  = dZ2 @ self.W2.T
        dZ1 = dA * relu_grad(self.Z1)
        dW1 = self.X.T @ dZ1 / m
        db1 = np.sum(dZ1, axis=0, keepdims=True) / m
        self.W1 -= lr*dW1; self.b1 -= lr*db1     # gradient descent (Lecture 7)
        self.W2 -= lr*dW2; self.b2 -= lr*db2

    def train(self, X, y, lr=0.5, epochs=1000):
        for epoch in range(epochs):
            Q = self.forward(X)
            self.backward(y, lr)
            if epoch % 100 == 0:
                loss = -np.mean(y*np.log(Q+1e-9) + (1-y)*np.log(1-Q+1e-9))
                print(f"epoch {epoch:4d}  loss={loss:.4f}")

Training this for 1000 epochs at \(\eta=0.5\) on the 4-example AND-gate batch prints a loss that falls steadily and predictions that converge to essentially exact:

▶ Play training

2.1 Watching the Decision Boundary Sharpen, Not Just the Loss Number Fall

A falling loss number is one view of training; a more direct one is watching what the network actually believes about the input space change shape. Evaluate the same trained network's output \(Q\) across a \(5\times5\) grid spanning \(x_1,x_2\in[0,1]\) — chosen so the grid's four corners land exactly on the four real AND-gate training points — at five snapshots during the same training run used above.

3. The Same Network in Keras

Now the identical architecture, trained the identical way, in a modern deep learning framework:

import tensorflow as tf

model = tf.keras.Sequential([
    tf.keras.layers.Dense(2, activation='relu', input_shape=(2,)),  # W1, b1, ReLU
    tf.keras.layers.Dense(1, activation='sigmoid'),                  # W2, b2, sigmoid
])

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
history = model.fit(X, y, epochs=500, verbose=0)
print("final loss:", history.history['loss'][-1])

4. Mapping Keras Calls Back to the Math

Every line of Keras configuration corresponds to a specific mathematical object we derived by hand earlier in the course. The framework has not introduced any new mathematics — it has automated the bookkeeping.

5. Hand-Derived Backprop vs. Framework Autodiff

6. Common Pitfalls

7. Summary

8. Code: NumPy From Scratch & Keras Equivalent

Two complete, runnable scripts: a pure-NumPy implementation (no external ML framework) and its Keras equivalent, both training the same 2→2(ReLU)→1(sigmoid) network on the same AND-gate toy dataset.

import numpy as np

def sigmoid(z):    return 1 / (1 + np.exp(-z))
def relu(z):        return np.maximum(0, z)
def relu_grad(z):   return (z > 0).astype(float)

class NeuralNetwork:
    """2 inputs -> 2 hidden (ReLU) -> 1 output (sigmoid), same architecture as Lecture 8."""
    def __init__(self, seed=42):
        rng = np.random.default_rng(seed)
        self.W1 = rng.normal(0, 0.5, size=(2, 2))
        self.b1 = np.zeros((1, 2))
        self.W2 = rng.normal(0, 0.5, size=(2, 1))
        self.b2 = np.zeros((1, 1))

    def forward(self, X):
        self.X  = X
        self.Z1 = X @ self.W1 + self.b1
        self.A  = relu(self.Z1)
        self.Z2 = self.A @ self.W2 + self.b2
        self.Q  = sigmoid(self.Z2)
        return self.Q

    def backward(self, y, lr):
        m = y.shape[0]
        dZ2 = self.Q - y                          # sigmoid + BCE shortcut (Lecture 8)
        dW2 = self.A.T @ dZ2 / m
        db2 = np.sum(dZ2, axis=0, keepdims=True) / m
        dA  = dZ2 @ self.W2.T
        dZ1 = dA * relu_grad(self.Z1)
        dW1 = self.X.T @ dZ1 / m
        db1 = np.sum(dZ1, axis=0, keepdims=True) / m
        self.W1 -= lr * dW1; self.b1 -= lr * db1  # gradient descent (Lecture 7)
        self.W2 -= lr * dW2; self.b2 -= lr * db2

    def train(self, X, y, lr=0.5, epochs=1000, verbose=True):
        for epoch in range(epochs):
            Q = self.forward(X)
            self.backward(y, lr)
            if verbose and epoch % 100 == 0:
                loss = -np.mean(y * np.log(Q + 1e-9) + (1 - y) * np.log(1 - Q + 1e-9))
                print(f"epoch {epoch:4d}  loss={loss:.4f}")
        return self


if __name__ == "__main__":
    # AND-gate toy dataset (Lecture 3's points)
    X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=float)
    y = np.array([[0], [0], [0], [1]], dtype=float)

    nn = NeuralNetwork(seed=42)
    nn.train(X, y, lr=0.5, epochs=1000)

    print("\nFinal predictions:", nn.forward(X).ravel().round(4))
    print("True labels:      ", y.ravel())

⬇ Download lecture-11-numpy-nn.py

import numpy as np
import tensorflow as tf

# Same AND-gate toy dataset as the NumPy version
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=float)
y = np.array([[0], [0], [0], [1]], dtype=float)

model = tf.keras.Sequential([
    tf.keras.layers.Dense(2, activation='relu', input_shape=(2,)),  # X W1 + b1, then ReLU  (Lectures 4-5)
    tf.keras.layers.Dense(1, activation='sigmoid'),                  # A W2 + b2, then sigmoid (Lectures 4-5)
])

model.compile(
    optimizer='adam',                 # Lecture 9
    loss='binary_crossentropy',       # Lecture 6
    metrics=['accuracy']              # Lecture 10
)

history = model.fit(X, y, epochs=500, verbose=0)  # internal loop = Lectures 7-8, via autodiff

print("Final loss:", round(history.history['loss'][-1], 4))
print("Predictions:", model.predict(X, verbose=0).ravel().round(4))
print("True labels:", y.ravel())

⬇ Download lecture-11-keras-nn.py   More resources for this lecture →