Advanced Optimizers: Momentum, AdaGrad, RMSProp, Adam

Plain SGD knows only one thing — the current gradient. These four optimizers each add memory, and each memory fixes a specific way vanilla gradient descent gets stuck.

• Identify the specific failure modes of plain SGD: narrow-ravine oscillation, saddle points, and vanishing progress along flat directions.
• Derive the Momentum update and explain it physically as a ball accumulating velocity.
• Derive the AdaGrad update and explain how per-parameter accumulated squared gradients equalize progress across axes of different curvature.
• Derive the RMSProp update as a fix for AdaGrad's ever-shrinking learning rate.
• Derive the Adam update, explain why bias correction is needed, and state its default hyperparameters.
• Hand-compute one update step under each optimizer from the same starting point and compare the resulting step sizes.

1. Why Go Beyond Plain SGD?

Lecture 7 gave us the basic rule \(w \leftarrow w - \eta\nabla_w L\). It works, but it treats every parameter identically and remembers nothing about the past. On the simple, symmetric bowl \(L(w)=(w-3)^2\) used throughout Lectures 7–8, that is fine. Real loss surfaces for deep networks are far less friendly, and plain SGD struggles in three specific ways:

• Narrow ravines. If the surface curves steeply in one direction and gently in another, a learning rate large enough to make progress along the gentle direction causes the update to oscillate back and forth across the steep direction.
• Saddle points. Points where the gradient is (near) zero but which are not minima — common in high-dimensional loss surfaces — can stall plain gradient descent almost indefinitely.
• Flat directions. Along directions where the gradient is consistently small, plain SGD takes tiny, painfully slow steps.

Momentum, AdaGrad, RMSProp, and Adam each patch one or more of these problems by giving the optimizer memory — some running statistic of past gradients that shapes the current step.

2. Momentum

We use the velocity form of momentum, which will be our consistent convention throughout this lecture:

$$v_t = \beta v_{t-1} - \eta\,\nabla_w L \qquad\qquad w \leftarrow w + v_t$$

with typical \(\beta=0.9\). \(v_t\) is a running "velocity" — an exponentially-weighted accumulation of past (negative, scaled) gradients. Physically, this is exactly a ball rolling downhill: gravity (the gradient) keeps accelerating it, but it also carries momentum from where it has already been rolling. Momentum's effect is twofold — it accelerates movement in directions where the gradient has consistently pointed the same way, and it damps oscillation in directions where the gradient keeps flip-flopping (the contributions partially cancel in the running average).

3. AdaGrad

$$G_t = G_{t-1} + (\nabla_w L)^2 \qquad\qquad w \leftarrow w - \frac{\eta}{\sqrt{G_t+\epsilon}}\,\nabla_w L$$

\(G_t\) accumulates the sum of squared past gradients for that parameter (\(\epsilon\) is a tiny constant, e.g. \(10^{-8}\), preventing division by zero). Consider the two axes in the instructor's motivating picture:

• Weak-gradient axis — gradients stay small, so \(G_t\) stays small, so dividing by \(\sqrt{G_t+\epsilon}\) (a small number) speeds up the update along that axis.
• Strong-gradient axis — gradients are large, \(G_t\) grows large, and dividing by \(\sqrt{G_t+\epsilon}\) (a large number) slows down the update along that axis.

The net effect is that AdaGrad automatically equalizes the rate of progress across parameters with very different curvature — exactly the narrow-ravine problem from Section 1.

4. RMSProp

RMSProp fixes exactly that downside. Instead of an ever-growing sum, it keeps an exponential moving average of squared gradients — old gradients decay away instead of accumulating forever:

$$G_t = \beta G_{t-1} + (1-\beta)(\nabla_w L)^2 \qquad\qquad w \leftarrow w - \frac{\eta}{\sqrt{G_t+\epsilon}}\,\nabla_w L$$

with typical \(\beta=0.9\). The update rule is identical in form to AdaGrad's — same per-parameter normalization by accumulated squared gradient — but because \(G_t\) is now an EMA rather than a running sum, it can go back down as well as up, tracking the recent gradient magnitude rather than the entire history. This keeps the effective learning rate from decaying to zero and makes RMSProp far more usable for long training runs.

5. Adam

Adam maintains two running statistics per parameter: a first-moment estimate \(m_t\) (momentum-like — an EMA of the raw gradient) and a second-moment estimate \(v_t\) (RMSProp-like — an EMA of the squared gradient):

$$m_t = \beta_1 m_{t-1} + (1-\beta_1)\nabla_w L \qquad\qquad v_t = \beta_2 v_{t-1} + (1-\beta_2)(\nabla_w L)^2$$

Both \(m_0\) and \(v_0\) are initialized to zero. That creates a subtle problem: early on, the EMA is still "warming up" and is biased toward zero (e.g. after one step, \(m_1=(1-\beta_1)\nabla_w L\) is much smaller in magnitude than the true gradient when \(\beta_1\) is close to 1). Adam corrects this with bias correction:

$$\hat m_t = \frac{m_t}{1-\beta_1^{\,t}} \qquad\qquad \hat v_t = \frac{v_t}{1-\beta_2^{\,t}}$$

Dividing by \(1-\beta_1^t\) (which is small for small \(t\) and approaches 1 as \(t\) grows) inflates the early, zero-biased estimates back up to roughly the right scale, while leaving later estimates almost unchanged. The final update is:

$$w \leftarrow w - \eta\,\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon}$$

6. Worked Comparison: One Update Step, Four Ways

Take the same quadratic loss from Lecture 7, \(L(w)=(w-3)^2\), so \(\nabla_w L = 2(w-3)\), starting from \(w=0\) with \(\eta=0.1\). At \(w=0\), \(\nabla_w L = 2(0-3)=-6\) for every optimizer below. To make the comparison meaningful, we assume each optimizer already has some accumulated history from previous steps (the values below are hypothetical, chosen purely to illustrate how each formula behaves — not a claim that one optimizer is always fastest). Every optimizer's arithmetic is worked in full below.

You can replay the same four computations interactively below — useful for testing yourself before moving on:

7. Common Pitfalls

8. Summary

9. Code: Four Optimizers From Scratch

The script below implements plain SGD, Momentum, AdaGrad, RMSProp, and Adam as small, self-contained functions, and reproduces the exact worked comparison numbers from Section 6.

import numpy as np

def grad_L(w):
    return 2 * (w - 3)   # L(w) = (w-3)^2, same loss as Lecture 7

w0, lr, eps = 0.0, 0.1, 1e-8
g = grad_L(w0)   # -6.0 at w=0

# ---- Plain SGD ----
w_sgd = w0 - lr * g

# ---- Momentum (v_t = beta*v_{t-1} - lr*grad ; w <- w + v_t) ----
beta = 0.9
v_prev = 0.5
v = beta * v_prev - lr * g
w_mom = w0 + v

# ---- AdaGrad (G_t = G_{t-1} + grad^2) ----
G_prev = 0.2
G = G_prev + g ** 2
w_ada = w0 - (lr / np.sqrt(G + eps)) * g

# ---- Adam (bias-corrected first & second moments) ----
beta1, beta2 = 0.9, 0.999
m_prev, v2_prev, t = 0.3, 0.1, 2
m = beta1 * m_prev + (1 - beta1) * g
v2 = beta2 * v2_prev + (1 - beta2) * g ** 2
m_hat = m / (1 - beta1 ** t)
v2_hat = v2 / (1 - beta2 ** t)
w_adam = w0 - lr * m_hat / (np.sqrt(v2_hat) + eps)

print(f"gradient at w=0:      {g:.4f}")
print(f"Plain SGD   -> w_new = {w_sgd:.4f}")
print(f"Momentum    -> w_new = {w_mom:.4f}   (v_t={v:.4f})")
print(f"AdaGrad     -> w_new = {w_ada:.4f}   (G_t={G:.4f})")
print(f"Adam        -> w_new = {w_adam:.4f}   (m_hat={m_hat:.4f}, v_hat={v2_hat:.4f})")

⬇ Download lecture-09-optimizers.py   More resources for this lecture →