Gradient Descent & Stochastic Gradient Descent

If backpropagation (next lecture) tells us which way every weight should move, gradient descent is the rule that decides how far — and how often — to move it.

• State the gradient descent update rule and interpret it geometrically as a step opposite the steepest-ascent direction.
• Hand-compute successive gradient descent iterates for a simple quadratic loss and recognize geometric convergence.
• Differentiate (Batch) Gradient Descent, Stochastic Gradient Descent, and Mini-Batch Gradient Descent using the instructor's comparison criteria.
• Explain why mini-batch gradient descent (batch sizes 32/64/128/256…) is the practical default for training real networks.
• Diagnose the effect of the learning rate: too small stalls training, too large causes oscillation or divergence.
• Preview how Lecture 8 computes \(\nabla_w L\) efficiently and how Lecture 9's optimizers make smarter use of it.

1. Why Gradient Descent?

Lecture 6 gave us a loss function \(L\) that scores how wrong the network's predictions are — cross-entropy for classification, squared error for regression. Training a network means searching for the parameters (weights and biases, collectively \(w\)) that make \(L\) as small as possible. For any network with more than a handful of weights, the loss surface \(L(w)\) lives in an enormous, high-dimensional space that cannot be searched exhaustively.

Gradient descent is the algorithm that makes this search tractable. Instead of evaluating every possible setting of \(w\), it starts from some initial guess and repeatedly takes small steps that are guaranteed — locally — to reduce the loss. It is, without exaggeration, the single algorithm that makes deep learning possible: every network you will meet in this course, however large, is trained by some variant of it.

2. The Update Rule & Geometric Intuition

The gradient \(\nabla_w L\) points in the direction of steepest ascent of the loss — the direction in which \(L\) increases fastest from the current point. If we want to decrease the loss, the best local direction to move is exactly opposite the gradient: \(-\nabla_w L\). That is the entire idea behind the minus sign in the update rule. Each step nudges \(w\) a little way downhill on the loss surface, and repeating the step traces a path toward a minimum.

3. Worked Example: Watching Convergence

Take the simplest possible loss surface, a 1-D quadratic bowl \(L(w) = (w-3)^2\), whose minimum is obviously at \(w=3\). Its gradient is \(\nabla_w L = 2(w-3)\) — every value used in the six iterations below is just this formula evaluated at the current \(w\). If you're already comfortable differentiating this by hand, skip straight to the iterations below. If you'd like the mechanical derivation spelled out, expand it here.

New to this? Expand: deriving ∇w L = 2(w−3) via the chain rule

Where does ∇w L = 2(w−3) actually come from?

Rather than take this gradient on faith, differentiate it directly. Write \(L\) as a composition \(L=u^2\) where \(u=w-3\), and apply the single-variable chain rule \(\frac{dL}{dw}=\frac{dL}{du}\cdot\frac{du}{dw}\):

$$\frac{dL}{dw}=\underbrace{2u}_{d(u^2)/du}\ \cdot\ \underbrace{1}_{d(w-3)/dw} = 2u = \mathbf{2(w-3)}$$

So \(\nabla_w L = 2(w-3)\) — every gradient value used in the six iterations below is just this formula evaluated at the current \(w\).

Now start from \(w_0=0\) with learning rate \(\eta=0.1\) and apply the update rule \(w\leftarrow w-\eta\nabla_wL\) repeatedly. Every arithmetic step is written out below, in order — nothing is left for you to fill in.

You can replay the same six iterations interactively below — useful for testing yourself before moving on:

4. Batch, Stochastic & Mini-Batch Gradient Descent

The update rule \(w \leftarrow w - \eta\nabla_w L\) says nothing about how much data we use to compute \(\nabla_w L\) on each step — and that choice defines three distinct variants of gradient descent.

• (Batch) Gradient Descent computes the gradient using the entire training set before taking a single step. The gradient estimate is exact and the descent path is smooth, but one step requires a full pass over potentially millions of examples.
• Stochastic Gradient Descent (SGD) computes the gradient from a single randomly chosen training example (or a very small handful) and updates immediately. Each step is cheap and fast, but the gradient estimate is noisy, so the descent path zig-zags rather than gliding smoothly downhill.

5. Mini-Batch Gradient Descent: The Practical Default

Mini-batch gradient descent is the middle ground that virtually all real training uses: compute the gradient over a small, fixed-size batch of examples (commonly 32, 64, 128, or 256), average it, and take one step. This keeps the noise of pure SGD low enough for stable progress while keeping each update far cheaper than a full-dataset pass — and, crucially, batched matrix multiplications map efficiently onto GPU/TPU hardware, which is designed for exactly this kind of parallel arithmetic.

6. Learning Rate: The Critical Hyperparameter

The learning rate \(\eta\) is arguably the single most important hyperparameter in training a neural network:

• Too small — steps are tiny, so the loss decreases correctly but painfully slowly, and training may not converge within any reasonable number of epochs.
• Too large — steps overshoot the minimum. Instead of settling down, the iterate can oscillate around the minimum or, worse, diverge to increasingly extreme values.

We can see divergence directly by re-running the same quadratic \(L(w)=(w-3)^2\) with an aggressive learning rate \(\eta = 1.1\), starting again from \(w_0=0\):

7. Common Pitfalls

8. Summary

9. Code: Gradient Descent From Scratch

The script below reproduces both the convergent (\(\eta=0.1\)) and divergent (\(\eta=1.1\)) numeric examples above, plus a small mini-batch gradient descent demo on a toy linear-regression dataset.

import numpy as np

def L(w):      return (w - 3) ** 2
def grad_L(w): return 2 * (w - 3)

def gradient_descent(w0, lr, steps):
    w = w0
    history = [w]
    for _ in range(steps):
        w = w - lr * grad_L(w)
        history.append(w)
    return history

# ---- convergent run: matches the lecture's worked example ----
conv = gradient_descent(w0=0.0, lr=0.1, steps=6)
print("Convergent (eta=0.1):", [round(w, 4) for w in conv])

# ---- divergent run: learning rate too large ----
div = gradient_descent(w0=0.0, lr=1.1, steps=4)
print("Divergent  (eta=1.1):", [round(w, 4) for w in div])

# ---- mini-batch gradient descent on a toy linear regression ----
rng = np.random.default_rng(0)
X = rng.uniform(-1, 1, size=(200, 1))
true_w, true_b = 2.5, -0.7
y = true_w * X[:, 0] + true_b + rng.normal(0, 0.05, size=200)

w, b, lr, batch_size, epochs = 0.0, 0.0, 0.1, 32, 100
n = len(X)
for epoch in range(epochs):
    idx = rng.permutation(n)
    for start in range(0, n, batch_size):
        batch = idx[start:start + batch_size]
        xb, yb = X[batch, 0], y[batch]
        pred = w * xb + b
        err = pred - yb
        dw = 2 * np.mean(err * xb)
        db = 2 * np.mean(err)
        w -= lr * dw
        b -= lr * db
    if epoch % 20 == 0:
        mse = np.mean((w * X[:, 0] + b - y) ** 2)
        print(f"  epoch {epoch:3d}: w={w:.3f} b={b:.3f} mse={mse:.4f}")

print(f"Learned w={w:.3f}, b={b:.3f}  (true w={true_w}, b={true_b})")

⬇ Download lecture-07-gradient-descent.py   More resources for this lecture →