Attention Mechanism & Self-Attention (Transformers)

How a model learns to look back at every word in a sentence and decide, dynamically, which ones matter right now — the single mechanism underneath every modern large language model.

• Explain why Word2Vec's static embeddings (Lecture 19) and the encoder-decoder bottleneck (Lecture 18) both point to the same missing ingredient: attention.
• State the Bahdanau attention formulation — alignment scores, softmax weights, and the context vector — and name the standard family of alignment-score functions.
• Derive scaled dot-product self-attention, \(\text{Attention}(Q,K,V)=\text{softmax}(QK^\top/\sqrt{d_k})V\), and compute it by hand on a real 4-token example.
• Explain why the softmax is scaled by \(\sqrt{d_k}\) and what breaks if you skip it.
• Describe multi-head attention as several independently-learned attention "perspectives" run in parallel, and causal masking as the mechanism that lets decoders train in parallel while respecting left-to-right generation.
• Connect BERT's MLM and NSP pretraining objectives back to cross-entropy (Lecture 6), backprop (Lecture 8), and tokenization (Lecture 19) as one trainable system.

1. From Static to Contextual Embeddings

Lecture 19 gave every word in the vocabulary one fixed embedding, learned by Word2Vec and then frozen. That embedding does not change no matter what sentence the word appears in. Recall the running example: "I went to bank for opening an account" and "I went to bank of a river for the walk" — two completely different senses of "bank," yet Word2Vec hands both occurrences the exact same vector. Separately, Lecture 18 ended on the encoder-decoder bottleneck: a plain sequence-to-sequence model compresses an entire input sequence, however long, into a single fixed-size context vector — and whatever doesn't fit gets lost, especially for long sequences.

Both problems have the same root cause: forcing a single, fixed-size representation to do too much work. And both are solved by the same idea. Instead of handing the decoder one static summary, let the model look back at all positions in the sequence and weigh their relevance dynamically, separately for every step it takes. That dynamic re-weighting is attention. Applied to a sequence's own tokens rather than between an encoder and a decoder, it becomes self-attention — and self-attention is what finally makes word embeddings context-dependent, closing Lecture 19's "bank" problem for good. We build up to that in two stages: first the original, pre-Transformer attention mechanism (Bahdanau attention), then the Transformer's scaled dot-product self-attention, which is the main event of this lecture.

2. Bahdanau Attention: Learning Where to Look

The attention mechanism introduces a set of additional learned parameters whose job is to compute attention weights — numbers that express how important or relevant each element of the input sequence is, at a given decoding step. Instead of the decoder seeing only the encoder's final hidden state (Lecture 18's bottleneck), it now receives a fresh context vector \(c_i\) at every decoding step \(i\), built from the decoder's own previous hidden state and all of the encoder's hidden states — not just the last one.

The one open design choice is the alignment function \(\text{align}(\cdot,\cdot)\) itself. Several standard forms exist:

Keep that last row in mind — the rest of this lecture is essentially "scaled dot-product alignment," generalized from a single decoder query against encoder states, to every token in a sequence querying every other token, all at once, with the query/key/value roles themselves learned. That generalization is the Transformer.

3. The Transformer Architecture

The Transformer was introduced by Vaswani et al. in "Attention Is All You Need."^† It replaces recurrence entirely — no RNN, no LSTM cell, nothing processes the sequence step by step. Instead, every token attends directly to every other token in a single operation, which is both more expressive and vastly more parallelizable than Lecture 15–18's recurrent architectures.

The architecture has two modules, encoder and decoder, each a stack of 6 identical layers. Each encoder layer has exactly two sub-layers: a self-attention mechanism and a feed-forward network. A residual ("skip") connection wraps each sub-layer, followed by layer normalization:

$$\text{LayerNorm}\big(x+\text{Sublayer}(x)\big)$$

The decoder repeats this pattern but inserts a third sub-layer between the two: multi-head attention over the encoder's output stack (so the decoder can look back at the source sequence, exactly as Bahdanau attention did — just computed the Transformer's way). The decoder's own self-attention sub-layer is additionally masked, a detail we return to in Section 6.

^† The course slides cite this as "Vaswani et al., Attention is all you need" (2023 in the deck's own reference line); the paper was in fact originally published in 2017 (NeurIPS) — the standard, well-known fact — so the 2023 in the source slide most likely reflects when that particular deck was compiled, not the paper's publication date.

4. Scaled Dot-Product Self-Attention: A Full Worked Example

Here is the equation that makes the entire Transformer, and by extension every modern large language model, work:

Three matrices — Query \(Q\), Key \(K\), Value \(V\) — combine through a dot product, a scale, a softmax, and a weighted sum, to produce a new set of embeddings in which every token's representation has absorbed information from every other token, weighted by relevance. We now walk through this exactly, step by step, on the instructor's own worked example, sentence "The dog ran fast" — reproducing every number as given on the course slides.

Step 1 — Tokenize & embed

The sentence is first tokenized with a word-piece tokenizer (Lecture 19) into token IDs — the slide's own illustrative snippet shows [1234, 1876, 3456] — which are then looked up in an embedding matrix. The course slide states the embedding dimension used in practice as 768 (this is the real BERT-base embedding size, used here illustratively). For the hand-worked example, a small initial embedding matrix is shown below, truncated to a few dimensions with "…" standing in for the rest of a 768-dimensional row:

Step 2 — Project into Query, Key, and Value matrices

Three separate learned weight matrices \(W_Q, W_K, W_V\) — learned during training, just like \(W_1, W_2\) in Lecture 8's feedforward network — project the same input embedding matrix \(X\) into three different matrices, each a different "view" or projection of the initial embeddings:

$$Q=XW_Q \qquad K=XW_K \qquad V=XW_V$$

The course slide's own example numbers (again truncated to a few dimensions, rows in order The / Dog / ran / fast):

Step 3 — Compute attention scores: \(QK^\top\), scale, softmax

First, every query is dotted against every key, giving a raw similarity/score matrix (rows = query token, columns = key token, order The/Dog/ran/fast for both):

These raw scores are then scaled by dividing every entry by \(\sqrt{d_k}\) — the slide explicitly shows this as its own "Scale" step ("÷ \(\sqrt{d_k}\)") before softmax. For the full 768-dimensional case used in practice, \(\sqrt{d_k}=\sqrt{768}\approx 27.7\), so every raw score above would be divided by roughly 27.7 before proceeding — concretely, using the actual raw scores shown in the figure above: the raw score 110 for (The, The) becomes \(110/27.7\approx\mathbf{3.97}\), and the raw score 58 for (The, ran) becomes \(58/27.7\approx\mathbf{2.09}\). Scaling divides every entry in a row by the same constant, so it never changes which score is largest — it only compresses the spread between scores before softmax sees them (more on exactly why that compression matters below).

After scaling, a row-wise softmax turns each row of scores into a probability distribution over "which tokens should I attend to." The course slide gives the resulting attention matrix directly:

Read it concretely. Row 1, token "The," attends almost entirely to itself — weight 0.99, with the other three tokens receiving essentially nothing (0.0004, 0.0003, 0.0005). That's a common, sensible pattern for a function word with little to disambiguate — "The" doesn't need context from "dog," "ran," or "fast" to know what it means. Row 3, token "ran," spreads its attention much more broadly — 0.38 to itself, 0.269 to "Dog," 0.094 to itself again... (0.38, 0.269, 0.094, 0.25 across The/Dog/ran/fast) — plausibly because a verb's meaning and grammatical role depends heavily on its subject and object elsewhere in the sentence, not just on itself.

Step 3b — The Raw-Score → Scale → Softmax Mechanism, Fully Worked by Hand

Step 3's numbers above are the instructor's own slide values, but — as the transparency note already flagged — they come from full 768-dimensional \(Q\) and \(K\) vectors of which the slide (and this page) only shows a few truncated columns, so the exact arithmetic that produces 110 or 0.99 genuinely can't be reproduced from what's on the page. That must not be allowed to hide the mechanism itself. So here is the identical pipeline — dot-product raw score, divide by \(\sqrt{d_k}\), exponentiate, sum, divide — carried out completely, start to finish, on a small, fully-specified toy example with \(d_k=4\) instead of 768, which we construct ourselves precisely so every number is visible. (These toy numbers are illustrative only, not the slide's; only the mechanism should carry over — the same caveat as Step 4's arithmetic below.)

Step 4 — Weighted sum of Values → new, contextual output embedding

The last step multiplies the attention matrix by the Value matrix: \(Z=\text{Attention}\times V\). Take token "The" (row 1 of the attention matrix: weights 0.99 / 0.0004 / 0.0003 / 0.0005) applied to the four rows of \(V\) shown in Step 2. Showing the arithmetic explicitly for the first dimension (column \(d_1\) of \(V\), values 3.0, 1.4, 5.4, 0.6 for The/Dog/ran/fast):

$$Z_{The,1}=0.99(3.0)+0.0004(1.4)+0.0003(5.4)+0.0005(0.6)=2.97+0.00056+0.00162+0.0003\approx 2.972$$

That is close to, but not exactly, the course slide's stated \(Z_{The}\approx[2.96,\ 7.54,\ \ldots,\ 5.45]\) — the small discrepancy comes from the same source we flagged in Step 3: the truncated ellipsis dimensions and display rounding in the slide's numbers, not an error in the method. The mechanism and the near-exact match matter far more than the last decimal.

Walking through exactly why: the projection matrices \(W_Q, W_K, W_V\) are the same learned weights regardless of which sentence "bank" appears in — nothing about them changes between the two sentences. What changes is the Key and Value matrices "bank"'s Query gets dotted against, because those come from \(X W_K\) and \(X W_V\) applied to that sentence's own token embeddings, and the two sentences contain different surrounding tokens. In "I went to bank for opening an account," bank's Query is dotted against Keys for "account," "opening," "for," etc. — words like "account" plausibly produce a high raw score (Step 3's mechanism, exactly as worked out above), so after softmax most of the attention weight lands on "account," and \(Z_{bank}\) ends up as a blend dominated by "account"'s Value vector, i.e. the financial-institution sense. In "I went to bank of a river for the walk," the very same \(W_Q\) produces bank's Query, but it is now dotted against Keys for "river," "walk," etc. instead — "river" plausibly scores highest, so after softmax \(Z_{bank}\) instead ends up dominated by "river"'s Value vector, the geographic sense. Same weights, same equation, same mechanism — but a different sentence supplies a different set of Keys and Values to attend over, which alone is enough to produce two different output vectors for the identical input word.

5. Multi-Head Attention

One attention computation gives one "view" of how tokens relate to each other. Multi-head attention runs the entire Q/K/V/softmax/weighted-sum process of Section 4 multiple times in parallel, each head with its own independently-learned \(W_Q, W_K, W_V\) projection matrices — so each head is free to specialize in a different kind of relationship: one head might learn to track syntactic subject–verb agreement, another might specialize in coreference (which pronoun refers to which noun), another in something no human-nameable pattern at all. The resulting per-head outputs are concatenated and passed through one more learned linear projection to produce the final multi-head attention output:

$$\text{MultiHead}(Q,K,V)=\text{Concat}(\text{head}_1,\ldots,\text{head}_h)W_O \qquad \text{head}_i=\text{Attention}(QW_Q^i,\,KW_K^i,\,VW_V^i)$$

Nothing new is happening arithmetically here — it's exactly Section 4's mechanism, run \(h\) times with different learned weights, then merged. The conceptual leap is what matters: attention is no longer a single fixed lens on the sentence, but several independent, simultaneously-learned lenses whose combined output is far richer than any one of them alone.

6. Masked Multi-Head Attention (Decoder Self-Attention)

In the decoder, self-attention as described above has a problem: at training time the decoder is shown the entire target sequence at once (for efficiency), but it must not be allowed to "see" tokens further to the right that it hasn't generated yet — that would be cheating during training, and is literally impossible at inference time, since those tokens don't exist yet. So the decoder's self-attention sub-layer applies causal masking: right-hand context is hidden during the attention-score computation.

Mechanically, this is simple: before the softmax, a very large negative number (or \(-\infty\)) is added to the raw score of every masked (future) position. After softmax, \(\exp(-\infty)=0\), so those positions' contribution vanishes to (numerically) exactly zero — while every unmasked position's score is untouched and softmax still redistributes correctly among the positions that remain visible. The course slide frames this using <SOS> (start-of-sequence), <PAD>, and <Mask> tokens: at each decoding step only the tokens generated (or known) so far are visible, shown progressively filling in — e.g. 0.99 | *** | *** | *** at the first step, with more real values appearing at each later step as more of the sequence becomes visible, and *** denoting a masked, not-yet-computed position.

The illustrative reveal below shows the same idea on a clean 4-token causal mask (the exact numbers here, beyond the first row's 0.99, are illustrative rather than reproduced from the source table, per the lecture's design — the point is the masking pattern, a lower-triangular reveal):

7. BERT: Applying the Encoder Side

BERT (Bidirectional Encoder Representations from Transformers) is built from the encoder portion of the Transformer only — no causal masking is needed, because BERT isn't autoregressively generating text; it's building a representation of a whole sentence it can already see in full. BERT is used to produce context-aware embeddings, directly closing the loop opened in Section 1 / Lecture 19. It is pretrained with two objectives:

Why this matters pedagogically: BERT is the clearest practical proof that everything covered so far in this course — Lecture 6's cross-entropy, Lecture 8's backprop, Lecture 19's tokenization, and this lecture's self-attention — composes into a single trainable system that produces the contextual word embeddings underlying essentially all modern NLP.

8. Summary & Where This Fits in the Course

• Attention replaces a single fixed summary with a dynamic, per-step weighted look-back over the whole sequence — solving both the encoder-decoder bottleneck (Lecture 18) and static word embeddings (Lecture 19).
• Self-attention's core equation, \(\text{Attention}(Q,K,V)=\text{softmax}(QK^\top/\sqrt{d_k})V\), projects the same input into Query/Key/Value views, scores every pair of tokens, scales to keep softmax well-behaved, and produces a weighted blend of Values as the new, contextual output.
• Multi-head attention runs several independently-learned copies of that mechanism in parallel; masked (causal) attention adds \(-\infty\) to future positions' scores so decoders can train in parallel while still respecting left-to-right generation.
• BERT (encoder-only, MLM + NSP) and GPT-style models (decoder-only, causal masking) are the two dominant ways of applying this machinery in practice.

9. Code: Scaled Dot-Product Self-Attention From Scratch

The full Transformer (6-layer encoder/decoder stacks, multi-head attention, positional encodings, layer norm, feed-forward blocks) is too large for a lecture-demo script — see the box below for a complete reference implementation. What follows instead is a small, runnable, self-contained NumPy implementation of the core mechanism this lecture is about: scaled dot-product self-attention, following exactly the four steps of Section 4 (embed → project to Q/K/V → score/scale/softmax → weighted sum of V). It uses made-up, seeded numbers (not the instructor's truncated 768-dim slide numbers, which can't be reproduced from a small demo) — but the printed attention matrix has the same qualitative shape as the lecture's heatmap: rows that sum to 1, with some peaked and some spread out.

"""
Lecture 20 - Scaled Dot-Product Self-Attention from scratch (NumPy).

Mirrors the lecture's Section 4 walkthrough:
  Step 1: token embeddings (X)
  Step 2: project to Query, Key, Value   Q = X.Wq   K = X.Wk   V = X.Wv
  Step 3: scores = Q K^T / sqrt(d_k), then row-wise softmax
  Step 4: output Z = Attention @ V  (a context-blended, "contextual" embedding)

The numbers below are made up (seeded random) -- NOT the instructor's
truncated 768-dim slide numbers, which can't be reproduced exactly from
a small demo. The qualitative shape should match the lecture's heatmap:
rows sum to 1, some rows peaked (near-certain self-attention), some spread
across several tokens.
"""
import numpy as np

np.random.seed(20)


def softmax(x, axis=-1):
    x = x - np.max(x, axis=axis, keepdims=True)   # numerical stability
    e = np.exp(x)
    return e / np.sum(e, axis=axis, keepdims=True)


tokens = ["The", "dog", "ran", "fast"]
n_tokens, d_model, d_k = 4, 8, 8

# ---- Step 1: token embeddings X (n_tokens x d_model) ----
X = np.random.randn(n_tokens, d_model)

# ---- Step 2: learned projection matrices -> Q, K, V ----
Wq = np.random.randn(d_model, d_k) * 0.5
Wk = np.random.randn(d_model, d_k) * 0.5
Wv = np.random.randn(d_model, d_k) * 0.5

Q = X @ Wq
K = X @ Wk
V = X @ Wv

# ---- Step 3: raw scores QK^T, scale by sqrt(d_k), softmax per row ----
scores = Q @ K.T
scaled_scores = scores / np.sqrt(d_k)
attention = softmax(scaled_scores, axis=1)

print("Attention matrix (each row is a probability distribution, sums to 1):")
for tok, row in zip(tokens, attention):
    row_str = "  ".join(f"{v:.3f}" for v in row)
    print(f"  {tok:>5}: {row_str}   sum={row.sum():.3f}")

# ---- Step 4: weighted sum of Values -> contextual output Z ----
Z = attention @ V

print("\nOutput Z (contextual embeddings, one row per token, first 4 dims shown):")
for tok, row in zip(tokens, Z):
    row_str = "  ".join(f"{v:.3f}" for v in row[:4])
    print(f"  {tok:>5}: {row_str} ...")

⬇ Download lecture-20-attention.py   More resources for this lecture →