Activation Functions
Four functions, four very different training dynamics: sigmoid's vanishing gradients, tanh's zero-centered balance, ReLU's dying neurons, and Leaky ReLU's fix.
Section 2 of Lecture 4 proved something important but incomplete: a nonlinearity between layers is non-negotiable, or the whole network collapses to a single linear transform. It never said which nonlinearity, or why that choice is one of the most consequential decisions in a network's design. This lecture is that missing half.
- Explain why activation functions must be nonlinear, referencing the collapse argument from Lecture 4.
- State the formula, range, and pros/cons of sigmoid, tanh, ReLU, and Leaky ReLU.
- Compute the vanishing-gradient behavior of sigmoid and tanh at large |x| numerically.
- Explain, mechanistically, why non-zero-centered outputs (sigmoid) produce inefficient, same-direction weight updates while zero-centered outputs (tanh) do not.
- Compute the probability that a ReLU neuron is "dead" at initialization and after its pre-activation mean shifts, and explain why Leaky ReLU avoids this.
1. Why Activation Functions Must Be Nonlinear
Lecture 4 proved this algebraically: stacking linear layers with no nonlinearity between them collapses into a single linear layer, since \(W_2(W_1x)=(W_2W_1)x\). Every function studied in this lecture exists to break that collapse — each takes a real-valued pre-activation \(z\) and applies a nonlinear transform, which is what allows depth to actually add representational power, and (per the Universal Approximation Theorem) lets a network approximate complex, curved functions rather than only straight lines and hyperplanes.
But not all nonlinearities are equally good to train with. The rest of this lecture compares four of the most widely used activation functions along the axis that matters most in practice: how well gradients flow backward through them during training.
2. Sigmoid
$$\sigma(x)=\frac{1}{1+e^{-x}}, \qquad \text{range } (0,1)$$- Pros: smooth, differentiable everywhere, and its (0,1) output is naturally interpreted as a probability — this is exactly why Lecture 8's output layer uses it.
- Cons: saturates and vanishes for large |x| — the gradient becomes vanishingly small exactly where the function is most confident. Numerically, \(\sigma'(5)\approx0.0067\): a neuron pushed to a confident output of \(\sigma(5)\approx0.9933\) receives almost no gradient signal to learn from. Its output is also not zero-centered — always positive — which causes the inefficient weight-update pattern examined in Section 4.
3. Tanh
$$\tanh(x)=\frac{e^x-e^{-x}}{e^x+e^{-x}}, \qquad \text{range } (-1,1)$$- Pros: zero-centered output — balanced positive and negative values lead to more balanced weight updates than sigmoid (Section 4).
- Cons: still saturates for large |x|, so vanishing gradients are not solved, only improved. Numerically, \(1-\tanh^2(4)\approx1.34\times10^{-3}\) — an almost completely flat gradient once the pre-activation is only moderately large.
4. Worked Comparison: Why Non-Zero-Centered Outputs Hurt
Consider two neurons that share the same downstream (upstream, in the backward pass) gradient \(g=1.0\), one using sigmoid and one using tanh, each with a small local pre-activation formed from a weight and an input: weights \(w=[0.5,0.5]\), inputs \(x=[1,-1]\), learning rate \(\eta=0.1\). Multiplying weight by input elementwise gives local pre-activations \(w\odot x = [0.5,-0.5]\), which is what each activation function sees:
Sigmoid: \(\sigma(0.5)\approx0.62\), \(\sigma(-0.5)\approx0.38\) — both outputs are positive, even though the underlying pre-activations \(w\odot x=[0.5,-0.5]\) had opposite sign. One step of gradient descent subtracts \(\eta\cdot g\cdot(\text{activation})\) from each weight (\(\eta=0.1\), \(g=1.0\)):
$$w_1'=0.5-0.1(1.0)(0.62)\approx\mathbf{0.44} \qquad\qquad w_2'=0.5-0.1(1.0)(0.38)\approx\mathbf{0.46}$$
Both weights move in the same direction — both decrease from 0.5 — even though the underlying inputs \(x_1=1\) and \(x_2=-1\) point opposite ways. That's the inefficiency: the direction of the update is dictated entirely by the sign of the upstream error \(g\), not by each weight's own input.
Tanh: \(\tanh(0.5)\approx0.46\), \(\tanh(-0.5)\approx-0.46\) — tanh preserves the sign of its input. The same update rule now gives:
$$w_1'=0.5-0.1(1.0)(0.46)\approx\mathbf{0.45}\ (\text{decreases}) \qquad\qquad w_2'=0.5-0.1(1.0)(-0.46)\approx\mathbf{0.55}\ (\text{increases})$$
The two weights split into opposite directions — one goes down, one goes up — each following the sign of its own input rather than being dragged along by the other.
The mechanism: a neuron's gradient with respect to an upstream weight is proportional to the neuron's own output activation (that activation is what multiplies the incoming weight in the next layer's pre-activation). Sigmoid's output is always positive, in the range \((0,1)\) — regardless of whether the underlying pre-activation was positive or negative. This means the gradient's sign, for every weight feeding out of that neuron, is forced to match the sign of the single upstream error term \(g\), all at once. Every weight connected to that neuron is therefore forced to increase together or decrease together on a given step — even when the ideal update would move some of them up and others down. This produces an inefficient, zig-zagging path toward the minimum. Tanh avoids this because its output can itself be positive or negative, so the sign of each weight's gradient can differ from its neighbors', letting the optimizer adjust each one somewhat independently. This is precisely why tanh is generally preferred over sigmoid for hidden layers, even though sigmoid remains standard for output layers that must produce a genuine probability.
5. ReLU
$$f(x)=\max(0,x), \qquad \text{range } [0,\infty)$$- Pros: extremely simple and cheap to compute; on the positive side its gradient is exactly 1, so it does not vanish the way sigmoid/tanh do for large positive inputs.
- Cons: "dying ReLU" — the gradient is exactly 0 for any negative input, so a neuron that is consistently pushed negative during training stops receiving any gradient and effectively stops learning, permanently.
At initialization, if a neuron's pre-activation \(x\sim\mathcal N(0,1)\) (standard normal, symmetric around zero), then exactly half of the bell curve's area sits to the left of 0 and half to the right, so \(P(x<0)=0.5\) — on average, roughly 50% of ReLU neurons are inactive for any given input right at initialization. That alone is not fatal (different inputs activate different neurons).
The real danger is if training shifts a neuron's pre-activation mean into negative territory — the whole bell curve slides left, so more of its area now falls on the negative side of zero. To compute how much, convert the cutoff \(x=0\) into a z-score — how many standard deviations away from the (shifted) mean it sits — and read off the standard normal CDF \(\Phi\) at that point: \(Z=\frac{x-\mu}{\sigma}\). If training pushes the mean to \(\mu=-1\) (still unit variance \(\sigma=1\)):
$$P(x<0)=P\!\left(\frac{x-\mu}{\sigma}<\frac{0-(-1)}{1}\right)=P(Z<1)=\Phi(1)\approx\mathbf{0.8413}$$
In words: shifting the mean one standard deviation to the left of zero drags the cutoff \(x=0\) to one full standard deviation above the new mean, so roughly 84% of the curve's area (nearly all of it except the right-hand tail) now lies below zero — meaning about 84% of that neuron's activations land on ReLU's flat, zero-gradient side, and it becomes very hard for that neuron to receive any further training signal.
6. Leaky ReLU
$$f(x)=\begin{cases}x & x\ge0\\ \alpha x & x<0\end{cases}, \qquad \text{range } (-\infty,\infty)$$- Pros: a small, nonzero gradient on the negative side (controlled by \(\alpha\), typically a small constant like 0.01) prevents neurons from dying completely.
- Cons: less sparse than plain ReLU (negative inputs are never fully suppressed to zero), and \(\alpha\) is an extra hyperparameter that needs tuning.
With \(\alpha=0.01\) and \(x=-3\): \(f(-3)=0.01\times(-3)=-0.03\), and the gradient \(f'(-3)=\alpha=0.01\) — small, but non-zero. Gradients keep flowing through this neuron even when its pre-activation is negative, unlike plain ReLU where \(f'(-3)=0\) exactly.
7. Summary Table
| Function | Formula | Range | Main weakness |
|---|---|---|---|
| Sigmoid | \(1/(1+e^{-x})\) | (0, 1) | Vanishing gradient + not zero-centered |
| Tanh | \((e^x-e^{-x})/(e^x+e^{-x})\) | (−1, 1) | Still saturates for large |x| |
| ReLU | \(\max(0,x)\) | [0, ∞) | Dying ReLU (zero gradient when x<0) |
| Leaky ReLU | \(x\) if \(x\ge0\), else \(\alpha x\) | (−∞, ∞) | Less sparse; α needs tuning |
8. Summary
- Every activation function here exists to break the linear collapse from Lecture 4 — but the choice among them shapes how well gradients survive backpropagation.
- Sigmoid saturates and is not zero-centered, which produces inefficient, same-direction weight updates — it survives mainly as an output-layer probability function.
- Tanh is zero-centered (better hidden-layer updates) but still saturates for large |x|.
- ReLU avoids saturation on the positive side but can "die" permanently on the negative side; Leaky ReLU fixes this with a small nonzero negative-side gradient.
- These four functions, and their gradient behavior, are exactly what Lecture 8's backpropagation derivation multiplies through at every layer — a saturated or dead activation there directly kills the gradient signal for every weight upstream of it.
9. Code: Plotting Activations and Their Gradients
The script below plots all four activation functions and their derivatives with matplotlib, and separately reproduces the sigmoid-vs-tanh weight-update comparison from Section 4.
import numpy as np
import matplotlib.pyplot as plt
def sigmoid(x): return 1 / (1 + np.exp(-x))
def sigmoid_grad(x): s = sigmoid(x); return s * (1 - s)
def tanh_grad(x): return 1 - np.tanh(x) ** 2
def relu(x): return np.maximum(0, x)
def relu_grad(x): return (x > 0).astype(float)
def leaky_relu(x, a=0.01): return np.where(x >= 0, x, a * x)
def leaky_relu_grad(x, a=0.01): return np.where(x >= 0, 1.0, a)
x = np.linspace(-6, 6, 400)
fig, axes = plt.subplots(2, 2, figsize=(10, 8))
axes[0,0].plot(x, sigmoid(x), label="sigmoid"); axes[0,0].plot(x, sigmoid_grad(x), '--', label="sigmoid'")
axes[0,1].plot(x, np.tanh(x), label="tanh"); axes[0,1].plot(x, tanh_grad(x), '--', label="tanh'")
axes[1,0].plot(x, relu(x), label="ReLU"); axes[1,0].plot(x, relu_grad(x), '--', label="ReLU'")
axes[1,1].plot(x, leaky_relu(x), label="Leaky ReLU"); axes[1,1].plot(x, leaky_relu_grad(x), '--', label="Leaky ReLU'")
for ax in axes.flat:
ax.legend(); ax.axhline(0, color='gray', lw=0.5); ax.axvline(0, color='gray', lw=0.5)
plt.tight_layout()
plt.savefig("activations.png", dpi=120)
print("Saved activations.png")
# ---- numeric facts referenced in the lecture ----
print("\nsigmoid'(5) =", round(sigmoid_grad(5), 4)) # ~0.0067
print("1 - tanh(4)^2 =", round(tanh_grad(4), 6)) # vanishing at large x
import math
def normal_cdf(x, mu=0, sigma=1):
return 0.5 * (1 + math.erf((x - mu) / (sigma * math.sqrt(2))))
print("P(x<0), mu=0 =", round(normal_cdf(0, mu=0), 4)) # 0.5, standard normal symmetry
print("P(x<0), mu=-1 =", round(normal_cdf(0, mu=-1), 4)) # ~0.8413
# ---- Section 4: sigmoid vs tanh weight-update comparison ----
w = np.array([0.5, 0.5]); x_in = np.array([1, -1]); eta, g = 0.1, 1.0
z = w * x_in # elementwise local pre-activations
a_sig = sigmoid(z)
a_tanh = np.tanh(z)
w_sig_new = w - eta * g * a_sig
w_tanh_new = w - eta * g * a_tanh
print("\nSigmoid activations:", np.round(a_sig, 2), " -> updated w:", np.round(w_sig_new, 2))
print("Tanh activations: ", np.round(a_tanh, 2), " -> updated w:", np.round(w_tanh_new, 2))
print("(Sigmoid: both weights move the SAME direction. Tanh: weights split in OPPOSITE directions.)")
⬇ Download lecture-05-activations.py More resources for this lecture →