LSTM Variants: BiLSTM, Stacked, Encoder-Decoder, GRU

One LSTM cell, read left-to-right, is only half the picture. Reading both directions, stacking layers, and chaining an encoder into a decoder unlock the architectures behind translation and summarization.

• Explain why a standard (unidirectional) LSTM cannot use future context, and why that's sometimes a problem.
• Describe how a Bidirectional LSTM combines two independent LSTMs to see the whole sequence.
• Describe how stacking BiLSTM layers builds progressively more abstract sequence representations.
• Draw and explain the encoder-decoder (seq2seq) architecture, including the role of the <SOS> token and the context vector.
• State the fixed-context bottleneck limitation and connect it to the motivation for attention (Lecture 20).
• Write the GRU equations and describe how it simplifies the LSTM.

1. Past Context Isn't Always Enough

Every LSTM (and plain RNN) we've built so far processes a sequence strictly left to right: at timestep \(t\), the hidden state \(h_t\) can depend on \(x_1,\ldots,x_t\) — the past — but never on \(x_{t+1},\ldots,x_T\) — the future. For generation tasks (predict the next word given only what came before) that's the right constraint. But for many understanding tasks, it's a real handicap.

2. Bidirectional LSTM (BiLSTM)

The fix is architecturally simple: run two independent LSTM cells over the same sequence — one forward (left→right, learning past context, exactly as before) and one backward (right→left, learning future context, processing the sequence in reverse). At every timestep, the two cells' outputs are concatenated to form a representation informed by the entire sequence, not just what came before:

$$h_t^{\text{BiLSTM}} = \big[\,h_t^{\text{forward}}\ ;\ h_t^{\text{backward}}\,\big] \qquad C_t^{\text{BiLSTM}} = \big[\,C_t^{\text{forward}}\ ;\ C_t^{\text{backward}}\,\big]$$

Each direction has its own full set of LSTM weights (its own \(W_f,W_C,W_i,W_o\)) — they do not share parameters with each other, only within their own direction across time.

Walking through the 3-word sequence "I / Like / Cats": the forward LSTM reads left→right, one word ahead of where it started, while the backward LSTM reads the same sentence right→left, independently:

You can replay the same steps interactively below — useful for testing yourself before moving on:

3. Stacked BiLSTM

Just as CNNs stack convolutional layers to build progressively more abstract visual features (Lecture 13), BiLSTM layers can be stacked on top of each other: the concatenated forward/backward output of one BiLSTM layer becomes the input sequence fed into the next BiLSTM layer above it. Using the same "I / Like / Cats" example, tokens flow upward through multiple LSTM layers per direction, with a Concat operation combining the forward and backward states at every layer, not just the last one.

4. The Encoder-Decoder (Seq2Seq) Architecture

Stacked BiLSTMs are the building block for one of the most important sequence-to-sequence architectures: the Encoder-Decoder, used for machine translation, summarization, and any task that maps one sequence to another sequence of possibly different length.

• Encoder — a stacked BiLSTM that reads the entire input sequence and compresses it into a single summary: the final concatenated states \(C_{\text{forward}}, C_{\text{backward}}, H_{\text{forward}}, H_{\text{backward}}\).
• Decoder — a unidirectional LSTM chain. It starts from a special <SOS> (start-of-sequence) token, and its initial hidden/cell state is set using the encoder's concatenated context (not zeros). At each step it produces one output token via a softmax layer, then feeds that token back in as the next input — generating the output sequence one token at a time.

This is the classic seq2seq architecture: the encoder's job is understanding, the decoder's job is generation, and the context vector is the single channel connecting the two.

5. The Bottleneck Problem

In practice, because the encoder's final state is produced by repeatedly overwriting a fixed-size hidden state (Lecture 15/16's recurrence), the words processed last — closest to the end of the sequence — tend to dominate the final context, while information from early words has had many more overwrite-and-forget-gate steps in which to fade. For short sentences this is a minor effect; for long sentences (a paragraph to translate, say) it becomes a serious accuracy problem.

The same reasoning, broken into three steps you can replay interactively:

6. GRU — A Simplified Gate (standard textbook material)

The Gated Recurrent Unit (GRU) is not covered in this course's source slides, but it is standard, widely-used material that belongs in any survey of LSTM variants — a simpler gated cell with comparable performance and fewer parameters. It merges the forget and input gates into a single update gate, and merges the cell state and hidden state into one signal:

$$z_t=\sigma(W_z[h_{t-1},x_t]) \qquad r_t=\sigma(W_r[h_{t-1},x_t])$$ $$\tilde h_t=\tanh\big(W[r_t\odot h_{t-1}, x_t]\big) \qquad h_t=(1-z_t)\odot h_{t-1}+z_t\odot\tilde h_t$$

Here \(z_t\) (update gate) plays a role similar to the LSTM's forget+input gates combined — it directly interpolates between the old hidden state and the new candidate \(\tilde h_t\) — while \(r_t\) (reset gate) controls how much of the past hidden state is used when computing that candidate. With only two gates and one state (instead of three gates, a candidate, and two states), a GRU has fewer weight matrices to learn, which can mean faster training and less overfitting on smaller datasets, often at little to no cost in accuracy.

7. Summary

8. Code: Bidirectional LSTM & a Toy Encoder-Decoder

Illustrative Keras/TensorFlow snippets — architecture wiring, not a full training pipeline.

"""
Lecture 18 -- BiLSTM layer + a toy Encoder-Decoder skeleton (Keras).
Illustrative only: shows how the pieces connect, not a trainable model.
Full trainable seq2seq-with-attention tutorial:
https://www.tensorflow.org/text/tutorials/nmt_with_attention
"""
from tensorflow.keras import layers, Model

VOCAB_SIZE, EMBED_DIM, HIDDEN = 5000, 64, 128

# ---- 1. A Bidirectional LSTM layer (Section 2) ----
inputs = layers.Input(shape=(None,), dtype="int32")
x = layers.Embedding(VOCAB_SIZE, EMBED_DIM, mask_zero=True)(inputs)
bilstm_out = layers.Bidirectional(layers.LSTM(HIDDEN, return_sequences=True))(x)
# bilstm_out shape: (batch, time, 2*HIDDEN) -- forward/backward concatenated
bilstm_model = Model(inputs, bilstm_out, name="bilstm_demo")
bilstm_model.summary()

# ---- 2. A minimal Encoder-Decoder skeleton (Section 4) ----
# Encoder: (Bi)LSTM compresses the source sequence into final states.
enc_inputs = layers.Input(shape=(None,), name="encoder_tokens")
enc_emb = layers.Embedding(VOCAB_SIZE, EMBED_DIM, mask_zero=True)(enc_inputs)
enc_out, fh, fc, bh, bc = layers.Bidirectional(
    layers.LSTM(HIDDEN, return_state=True))(enc_emb)
state_h = layers.Concatenate()([fh, bh])   # H_forward || H_backward
state_c = layers.Concatenate()([fc, bc])   # C_forward || C_backward

# Decoder: unidirectional LSTM, initialised from the encoder's context,
# starts from a  token (handled upstream in the input pipeline),
# predicts one token at a time via softmax.
dec_inputs = layers.Input(shape=(None,), name="decoder_tokens")  # teacher-forced
dec_emb = layers.Embedding(VOCAB_SIZE, EMBED_DIM, mask_zero=True)(dec_inputs)
dec_lstm = layers.LSTM(2 * HIDDEN, return_sequences=True)
dec_out = dec_lstm(dec_emb, initial_state=[state_h, state_c])
dec_softmax = layers.Dense(VOCAB_SIZE, activation="softmax")(dec_out)

seq2seq = Model([enc_inputs, dec_inputs], dec_softmax, name="encoder_decoder_demo")
seq2seq.summary()

⬇ Download lecture-18-bilstm-seq2seq.py   More resources for this lecture →