The Vanishing/Exploding Gradient Problem

A network that can theoretically remember arbitrarily far back — and, in practice, forgets almost everything after a handful of timesteps. Here's the arithmetic behind why.

• Express the gradient of a late-timestep loss w.r.t. an early hidden state as a product of Jacobians.
• Explain why that product typically shrinks toward zero as the gap between timesteps grows.
• Reproduce, by hand, a numeric chain-multiplication example that vanishes toward zero within a handful of steps.
• Contrast this with the exploding-gradient case and identify the shared root cause.
• List the standard mitigations, and name the architectural fix this course covers next (LSTM, Lecture 17).

1. Recap: What BPTT Actually Computes

Lecture 15 established that an RNN, once unrolled across \(T\) timesteps, is trained by Backpropagation Through Time (BPTT) — Lecture 8's chain rule, applied to the unrolled graph, with gradients for the shared weights \(W_h, W_x, W_y\) accumulated across every timestep. To see why that accumulation is dangerous, look at just one piece of it: the gradient of the loss at the final timestep \(T\) with respect to a hidden state far in the past, \(h_1\).

Because \(h_t\) depends on \(h_{t-1}\), which depends on \(h_{t-2}\), and so on, the chain rule forces this gradient to be written as a product of many intermediate Jacobian terms — one factor per timestep in between:

$$\frac{\partial L_T}{\partial h_1} = \frac{\partial L_T}{\partial h_T}\,\prod_{t=2}^{T}\frac{\partial h_t}{\partial h_{t-1}} \qquad\text{where}\qquad \frac{\partial h_t}{\partial h_{t-1}} \approx \text{diag}\big(\tanh'(z_t)\big)\,W_h^\top,\quad z_t = W_h h_{t-1}+W_x x_t$$

Every one of the \(T-1\) factors in that product is itself a combination of two things: the local derivative of the \(\tanh\) activation, and the weight matrix \(W_h\). Both turn out to matter.

2. The Vanishing Gradient, Numerically

Multiply many numbers that are each less than 1, and the product shrinks geometrically — not linearly — toward zero. This is exactly what happens to \(\partial L_T/\partial h_1\) as the gap \(T-1\) grows: it isn't that one factor kills the gradient, it's that a chain of ordinary-looking, individually-plausible sub-1 factors compounds into an almost-zero product astonishingly fast.

Here is the instructor's own worked illustration: each number below stands in for one timestep's combined local-gradient magnitude (the product of a \(\tanh'\) term and a weight factor, exactly like the 0.2796 and 0.3051 just computed above) — a perfectly ordinary value, less than 1. Watch what happens after just five multiplications, worked out in full below:

You can replay the same five steps interactively below — useful for testing yourself before moving on:

3. The Exploding Gradient — Same Root Cause, Opposite Symptom

The vanishing case assumed every factor was comfortably below 1. Flip that assumption — let even one factor in the chain be large — and the same repeated-multiplication mechanism produces the opposite failure: the product grows without bound, or flips sign wildly, instead of decaying to zero. Here is the same style of step-by-step running product, this time exploding:

Same idea, click-through form:

4. Consequences for Learning

Practically, vanishing gradients mean training stalls on long-range structure: the loss keeps improving on short-range, local patterns (which get healthy gradient signal) while dependencies spanning dozens of timesteps simply never get learned, because their gradient contribution is numerically indistinguishable from zero. Exploding gradients are more dramatic and easier to notice: weight updates suddenly become enormous, loss spikes to NaN or diverges, and training visibly breaks — which, in a strange way, makes exploding gradients the "easier" of the two problems, since at least you can tell something has gone wrong.

5. Standard Mitigations

None of these fixes are specific to the local course slides — they are standard, widely-used engineering responses to this well-known problem:

6. Summary

7. Code: Chain Multiplication, Vanishing vs. Exploding

The script below reproduces the vanishing chain from Section 2 numerically, then contrasts it with an exploding chain and a sign-flipping chain.

import numpy as np

def running_products(factors):
    """Print the running product after each multiplication."""
    product = 1.0
    trail = []
    for f in factors:
        product *= f
        trail.append(product)
        print(f"  x {f:>6} -> running product = {product:.6f}")
    return trail

print("VANISHING chain (typical sub-1 local gradients each step):")
vanishing_factors = [0.3, 0.2, 0.5, 0.8, 0.02, 0.1]
vanish_trail = running_products(vanishing_factors)
print(f"-> after {len(vanishing_factors)} steps, gradient ~ {vanish_trail[-1]:.6f}  (essentially 0)\n")

print("EXPLODING chain (same first four factors, then a >1 factor repeats):")
explode_factors = [0.3, 0.2, 0.5, 0.8, 1.8, 1.8, 1.8, 1.8]
explode_trail = running_products(explode_factors)
print(f"-> after {len(explode_factors)} steps, gradient ~ {explode_trail[-1]:.6f}  and still growing\n")

print("SIGN-FLIP / explosive chain (one large negative factor early):")
signflip_factors = [0.3, 0.2, -6.4]
signflip_trail = running_products(signflip_factors)
print(f"-> after just {len(signflip_factors)} steps: {signflip_trail[-1]:.6f}"
      " -- large magnitude AND flipped sign\n")

# Same root cause every time: repeated multiplication by a shared weight
# matrix (here, scalar factors standing in for it) across many timesteps.
# Vanishing: |factor| < 1 typically.  Exploding: some |factor| > 1.

⬇ Download lecture-16-vanishing-gradient.py   More resources for this lecture →