Word Embeddings & Tokenization

To understand language, understand words. To let a machine understand words, turn them into vectors — the whole art is in choosing HOW.

• Explain why words must be converted to numeric vectors before a neural network can process language.
• Build a one-hot encoding by hand and state its two core disadvantages.
• Describe the CBOW and Skip-gram Word2Vec architectures, including the exact matrices involved and what becomes the embedding table after training.
• Explain precisely why Word2Vec produces a static, context-independent embedding — and what problem that causes.
• Describe the out-of-vocabulary problem in word-level tokenization and how sub-word tokenization (BPE, WordPiece) solves it.
• Trace the BPE merge-training algorithm by hand on a small corpus, pair-score round by pair-score round.

1. From Words to Vectors

Every neural network we've studied — feedforward, CNN, RNN, LSTM — computes with numbers: matrix multiplications, dot products, gradients. None of that machinery has any native notion of a "word." So before any language model can do anything, there is one unavoidable question: how do we turn a word into a vector of numbers?

The instructor's framing is exactly this chain of reasoning: to understand language, a model must understand words; the only way a machine can compute with words is if they live in a vector space. This isn't a minor implementation detail — it's foundational. There are more than 7,100 different languages spoken in the world; no fixed rule-based dictionary or hand-coded feature scheme scales to that. What's needed is a general, learnable, numeric representation of "word meaning" that a model can compute with directly. This lecture builds that representation up in three stages: one-hot (naive), Word2Vec (dense, learned), and sub-word tokenization (fixes the vocabulary problem underneath both).

2. One-Hot Encoding

If you already know what a one-hot vector is, skip straight to the worked example below. If not, expand for the definition.

New to this? Expand: what one-hot encoding is

The simplest possible scheme: build a vocabulary of every unique word in your corpus, then represent word \(i\) as a vector of length \(V\) (the vocabulary size) that is 0 everywhere except a single 1 at position \(i\).

So \(\text{Tokyo} = [1,0,0,0,0,0,0,0,0,0]\) and \(\text{is} = [0,1,0,0,0,0,0,0,0,0]\), and so on for every word.

3. Word2Vec: Continuous Bag of Words (CBOW)

Word2Vec fixes both one-hot disadvantages at once: it represents words as dense, much lower-dimensional vectors, learned so that semantically similar words end up close together in the embedding space. There are two Word2Vec architectures; CBOW predicts a target word from its surrounding context words.

Mechanically: each one-hot context word \(X_i\) (shape \(1\times V\)) is multiplied by a shared weight matrix \(W_{V\times N}\) (\(N\) = embedding dimension, chosen much smaller than \(V\)), and the results for all context words are averaged to produce the hidden layer \(H\) (shape \(1\times N\)):

$$H = \frac{1}{|\text{context}|}\sum_{i} X_i W$$

\(H\) is then multiplied by a second matrix \(W'_{N\times V}\) to produce the output layer, and a softmax turns that into a probability distribution over the entire vocabulary for the predicted target word. Training minimizes cross-entropy loss (Lecture 6) between this distribution and the true target word, with weights updated by ordinary backpropagation (Lecture 8) — nothing new algorithmically, just a specific choice of architecture and training task.

4. Word2Vec: Skip-Gram

Skip-gram runs the same idea in reverse: given the target word, predict each context word individually. For a window size \(w\), this prediction is repeated \(k=2w\) times per target — once per context position.

Input is the one-hot target word \(x\) (shape \(1\times V\)); the hidden layer is simply \(H = xW\) (a row lookup into \(W_{V\times N}\), since \(x\) is one-hot); the output is \(H W'\), a length-\(V\) score vector, again turned into a probability distribution via softmax and trained with cross-entropy loss — exactly the same ingredients as CBOW, just with the roles of input and output swapped. As with CBOW, \(W_{V\times N}\) after training is the embedding table.

5. The Limitation: Static, Context-Independent Embeddings

6. Tokenization and the Out-of-Vocabulary Problem

Before any of the above can happen, raw text has to be split into discrete units — tokens — in the first place. The most obvious approach is word-level tokenization: split on spaces and punctuation, count how often each resulting word occurs across the corpus, assign each word an index by frequency rank, and (to keep the vocabulary a manageable size) discard words below some frequency threshold.

7. Sub-Word Tokenization: BPE and WordPiece

Sub-word tokenization fixes the OOV problem by never requiring a word to be a single, indivisible unit in the first place. Instead of building a vocabulary of whole words, it builds a vocabulary of frequently-occurring pieces of words (down to individual characters, in the worst case) — so any string, seen or unseen, can always be represented as some sequence of known pieces.

Both algorithms share the same overall shape: start from individual characters, repeatedly find the best-scoring adjacent pair of symbols currently in use, merge it into a new single vocabulary entry, and repeat — growing the vocabulary one merge at a time until a target size (or no more useful merges) is reached. They differ only in the scoring rule used to pick the winning pair each round.

8. Worked Example: Merge-Training a Sub-Word Vocabulary

Corpus: {huggingface, hugging, face, hug, hugger, learning, learner, learners, learn}. Follow the merge process round by round below (using the ##-continuation convention: ##x marks a character that is not at the start of a word).

You can replay the same five steps interactively below — useful for testing yourself before moving on:

9. Summary

10. Code: Byte Pair Encoding From Scratch

A toy BPE trainer (word-frequency-based pair counting — the GPT-style algorithm) on the exact instructor corpus, printing each merge as it happens.

"""
Lecture 19 -- Byte Pair Encoding (BPE) merge-training from scratch.
Toy implementation on the instructor's example corpus. Uses classic
BPE (pair FREQUENCY, summed across word counts) -- the algorithm
GPT-style tokenizers use. (WordPiece, used by BERT, scores pairs
differently -- see the lecture notes.)
"""
from collections import defaultdict, Counter

corpus = ["huggingface", "hugging", "face", "hug",
          "hugger", "learning", "learner", "learners", "learn"]
word_freq = Counter(corpus)   # each word appears once here

def split_word(w):
    """Represent a word as a list of symbols; '##' marks a
    non-initial character, following the lecture's convention."""
    return [w[0]] + ["##" + c for c in w[1:]]

splits = {w: split_word(w) for w in word_freq}

def get_vocab(splits):
    v = set()
    for symbols in splits.values():
        v.update(symbols)
    return v

def pair_freqs(splits, word_freq):
    freqs = defaultdict(int)
    for w, symbols in splits.items():
        for a, b in zip(symbols, symbols[1:]):
            freqs[(a, b)] += word_freq[w]
    return freqs

def merge_pair(pair, splits):
    a, b = pair
    merged = a + (b[2:] if b.startswith("##") else b)
    new_splits = {}
    for w, symbols in splits.items():
        out, i = [], 0
        while i < len(symbols):
            if i < len(symbols) - 1 and symbols[i] == a and symbols[i+1] == b:
                out.append(merged); i += 2
            else:
                out.append(symbols[i]); i += 1
        new_splits[w] = out
    return new_splits, merged

print("Initial vocab:", sorted(get_vocab(splits)))

NUM_MERGES = 6
for step in range(1, NUM_MERGES + 1):
    freqs = pair_freqs(splits, word_freq)
    if not freqs:
        break
    best_pair = max(freqs, key=freqs.get)
    splits, merged_token = merge_pair(best_pair, splits)
    print(f"Merge {step}: {best_pair} (freq={freqs[best_pair]}) -> '{merged_token}'")

print("\nFinal splits:")
for w in corpus:
    print(f"  {w:12s} -> {splits[w]}")

⬇ Download lecture-19-bpe.py   More resources for this lecture →