LSTM Networks: Gates & Memory

Instead of overwriting the hidden state every timestep, an LSTM cell learns exactly what to forget, what to add, and what to reveal — three learned decisions replacing one blunt overwrite.

• Name and define all four LSTM gates/candidates, and give the intuitive 0/0.5/1 reading of each sigmoid gate.
• Write the cell-state update equation and explain why it is (near-)additive across timesteps.
• Explain precisely why the additive cell-state path fixes the vanishing gradient problem from Lecture 16.
• Compute one full LSTM timestep by hand, gate by gate, with real numbers.
• Cite the original LSTM paper and describe its place in the sequence-modeling toolkit.

1. A Memory Cell With Gates

Lecture 16 showed that a plain RNN's hidden state is rewritten at every timestep by repeated multiplication with the shared weight matrix \(W_h\) — the very thing that causes gradients (and, in practice, long-range memory) to vanish. The Long Short-Term Memory network, introduced by Hochreiter & Schmidhuber, "Long short-term memory," Neural Computation, 9(8):1735–1780, 1997, fixes this with a structural change: alongside the usual hidden state \(h_t\), an LSTM carries a second running signal, the cell state \(C_t\), and uses a small set of learned gates to control, at every timestep, exactly how much of the old memory to keep and how much new information to add.

Picture the sentence "I will swim today" being read one word at a time. At the first timestep ("I"), the cell has almost nothing in memory yet, so it mostly writes new information in. At the second timestep ("will"), it must decide: keep what it has about "I" (the subject), and add information about "will" (future tense) — a running memory is built up incrementally, word by word, gate by gate. That is the intuitive picture; the rest of this lecture makes it precise.

2. The Gates, One at a Time

Every gate below is a small feedforward layer that looks at the same two things — the previous hidden state \(h_{t-1}\) and the current input \(x_t\), concatenated as \([h_{t-1}, x_t]\) — and squashes the result with either a sigmoid (\(\sigma\), output in \((0,1)\)) or a tanh (output in \((-1,1)\)).

Every weight matrix above (\(W_f, W_C, W_i, W_o\)) is, like the plain RNN's \(W_h, W_x\), shared across all timesteps. In a real LSTM, \(h_{t-1}, x_t, C_t\) are vectors, and every equation above applies element-wise / via matrix-vector products — for clarity, the worked example in Section 5 treats everything as scalars, but the mechanics generalize directly.

3. All Six Equations, Together

4. Why This Fixes the Vanishing Gradient Problem

Recall Lecture 16's diagnosis: a plain RNN's gradient backward through time is a product of many Jacobian terms, each involving a repeated multiplication by \(W_h\) — and products of many sub-1 terms vanish geometrically.

Now look at the cell-state update again: \(C_t=f_t\odot C_{t-1}+i_t\odot\tilde C_t\). The path from \(C_{t-1}\) to \(C_t\) is an element-wise multiplication by the forget gate \(f_t\), plus an addition — there is no shared weight matrix repeatedly multiplying the cell state itself at every step. When we backpropagate through the \(C\) path, the gradient at each step is scaled by \(f_t\) (a per-timestep, learned, data-dependent number that can sit close to 1) rather than by the same fixed \(W_h\) every single time.

5. Worked Numerical Example — One Timestep

Take a single scalar LSTM timestep: previous hidden state \(h_{t-1}=0.2\), previous cell state \(C_{t-1}=0.3\), current input \(x_t=0.5\). Weights (already merging \([h_{t-1},x_t]\) into two scalar coefficients per gate, for arithmetic simplicity):

Now every gate, worked by hand, in order — including the sigmoid/tanh arithmetic itself, not just the final gate value. Recall \(\sigma(z)=\dfrac{1}{1+e^{-z}}\) and \(\tanh(z)=\dfrac{e^{z}-e^{-z}}{e^{z}+e^{-z}}\) (Lecture 5).

You can replay the same six gates interactively below — useful for testing yourself before moving on:

All six computed values, side by side:

6. Limitations

7. Summary

8. Code: One LSTM Cell Timestep From Scratch

Reproduces every number in Section 5's worked example exactly.

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

# ---- state and input (matches the worked example in the lecture) ----
h_prev, C_prev, x = 0.2, 0.3, 0.5

# ---- gate weights (scalar, for a single-unit illustrative LSTM cell) ----
Wfh, Wfx, bf = 0.5, 0.6, 0.1     # forget gate
Wih, Wix, bi = 0.4, 0.3, 0.0     # input gate
WCh, WCx, bC = 0.3, 0.5, 0.2     # candidate memory
Woh, Wox, bo = 0.6, 0.4, 0.1     # output gate

# ---- forward pass through one LSTM timestep, gate by gate ----
f_t = sigmoid(Wfh * h_prev + Wfx * x + bf)
i_t = sigmoid(Wih * h_prev + Wix * x + bi)
C_tilde = np.tanh(WCh * h_prev + WCx * x + bC)
C_t = f_t * C_prev + i_t * C_tilde
o_t = sigmoid(Woh * h_prev + Wox * x + bo)
h_t = o_t * np.tanh(C_t)

print(f"f_t (forget) = {f_t:.4f}")
print(f"i_t (input)  = {i_t:.4f}")
print(f"C~_t (candidate) = {C_tilde:.4f}")
print(f"C_t (cell state)  = {C_t:.4f}")
print(f"o_t (output) = {o_t:.4f}")
print(f"h_t (hidden state) = {h_t:.4f}")

expected = dict(f_t=0.6225, i_t=0.5573, C_tilde=0.4699, C_t=0.4487, o_t=0.6035, h_t=0.2540)
computed = dict(f_t=f_t, i_t=i_t, C_tilde=C_tilde, C_t=C_t, o_t=o_t, h_t=h_t)
for k in expected:
    assert abs(computed[k] - expected[k]) < 1e-3, f"{k} mismatch!"
print("\nAll values match the lecture's hand-worked example.")

⬇ Download lecture-17-lstm-cell.py   More resources for this lecture →