Probability, Loss Functions & Cross-Entropy

Every loss function is secretly a statement about probability distributions — this lecture builds that connection from random variables up to the cross-entropy loss Lecture 8 will differentiate.

• Define a random variable and distinguish discrete from continuous random variables with concrete examples.
• Explain the difference between a probability mass function (PMF) and a probability density function (PDF).
• Describe a Gaussian Mixture Model and what its parameters (mean, covariance, mixing probability) represent.
• State the KL divergence and show why minimizing cross-entropy is equivalent to minimizing KL divergence to the true data distribution.
• Compute \(D_{KL}(P\|Q)\) and \(D_{KL}(Q\|P)\) by hand for a concrete example and confirm they are different numbers.
• Explain, mechanistically, why minimizing forward KL is "mean-seeking" while minimizing reverse KL is "mode-seeking," and contrast their behavior across four illustrative cases.
• Derive the binary cross-entropy loss and connect it to Lecture 8's backpropagation example.

1. Random Variables

A random variable is a variable which represents the outcome of a trial, experiment, or event — a specific number which is different each time the trial is repeated. It is a function from outcomes to real numbers, formally \(X:\Omega\to\mathbb R\), where \(\Omega\) is the set of all possible outcomes.

2. Probability Distributions: PMF vs. PDF

How likely is each value of a random variable? That question is answered differently depending on whether the variable is discrete or continuous.

3. Gaussian Mixture Models

Real data rarely looks like one clean bell curve — it often has several distinct "clumps." A Gaussian Mixture Model (GMM) models a distribution as a weighted sum of several Gaussians, indexed \(k\in\{1,\dots,K\}\), each with its own mean (center), covariance (width/shape), and mixing probability (weight):

$$p(x) = \sum_{k=1}^{K}\pi_k\,\mathcal N(x\,;\,\mu_k,\Sigma_k), \qquad \sum_{k=1}^K \pi_k = 1$$

The mixing coefficients \(\pi_k\) are themselves probabilities — each between 0 and 1, and required to sum to 1 across all \(K\) components, just like a PMF over "which Gaussian generated this point." A \(K=3\) mixture, for instance, models a distribution as three overlapping bell curves, each contributing its own share of the total probability mass. GMMs matter here as a bridge: they are one concrete answer to the question "how do we model the true, unknown distribution the data comes from?" — which is exactly the question the rest of this lecture addresses more generally.

4. What Determines Model Performance

Zooming out: every supervised deep learning system, at training time, is trying to approximate an unknown distribution of real data using a parameterized model θ. The instructor's own framing of the training loop makes the moving parts explicit:

Every stage in that loop can help or hurt final performance — bad embeddings lose information before the model even sees it, an ill-suited architecture cannot represent the true function, and a poorly chosen loss can point gradients in a direction that doesn't actually reflect what we care about. This lecture focuses on that last piece: what loss functions actually measure, and why.

5. Maximum Likelihood Estimation and Cross-Entropy

Training a classifier means choosing the model's parameters to make its predicted distribution \(Q\) match the true (but unknown) data distribution \(P\) as closely as possible. Maximum Likelihood Estimation (MLE) formalizes "as closely as possible" as: choose parameters that make the observed data most probable under the model. Remarkably, this turns out to be mathematically identical to minimizing cross-entropy.

The KL divergence between two distributions is the expected log-likelihood ratio, averaged over the true distribution — it measures how much "extra surprise" you incur by using \(Q\) to describe data that actually came from \(P\):

$$D_{KL}(P\|Q)=\sum_x P(x)\log\frac{P(x)}{Q(x)}$$

Expand this using the log quotient rule:

$$D_{KL}(P\|Q) = \sum_x P(x)\log P(x) - \sum_x P(x)\log Q(x) = -H(P) + H(P,Q)$$

where \(H(P)=-\sum_x P(x)\log P(x)\) is the entropy of the true distribution, and \(H(P,Q)=-\sum_x P(x)\log Q(x)\) is the cross-entropy between \(P\) and \(Q\). Rearranging:

$$H(P,Q) = H(P) + D_{KL}(P\|Q)$$

6. Forward vs. Reverse KL: Mean-Seeking vs. Mode-Seeking

Here is a fact that surprises most students the first time they see it: \(D_{KL}(P\|Q)\) and \(D_{KL}(Q\|P)\) are, in general, two different numbers. KL divergence measures a kind of "distance" between distributions, but it is not a true distance metric — it is not symmetric. Swapping which distribution you call \(P\) and which you call \(Q\) changes the answer. This is not a minor technicality: the two directions penalize mistakes in opposite ways, and that choice quietly shapes what kind of model you end up with. Write both directions out explicitly, side by side, so the difference is visible in the formula itself before we compute anything:

$$\underbrace{D_{KL}(P\|Q)=\sum_x P(x)\log\frac{P(x)}{Q(x)}}_{\text{"Forward" KL — weighted by }P(x)} \qquad\qquad \underbrace{D_{KL}(Q\|P)=\sum_x Q(x)\log\frac{Q(x)}{P(x)}}_{\text{"Reverse" KL — weighted by }Q(x)}$$

Both sums run over the exact same log-ratio-flavored quantity, but each one is weighted by a different distribution — the first by \(P\), the second by \(Q\). Since a term weighted by a near-zero number contributes almost nothing to the sum no matter how large the rest of that term is, which distribution does the weighting turns out to control everything.

6.1 First, prove to yourself the two numbers really differ

Before any conceptual discussion, let's just compute both directions for one concrete pair of distributions and see that they are not equal. Suppose the true label distribution over {bike, not-bike} is \(P=[0.9,\,0.1]\) (genuinely 90% likely to be a bike), and the model's current prediction is the under-confident \(Q=[0.6,\,0.4]\).

6.2 The bigger picture: mean-seeking vs. mode-seeking

The asymmetry above has a striking visual consequence when the true distribution \(P\) has more than one "peak" (exactly the Gaussian Mixture Model from Section 3). The instructor's own slides illustrate this with real data generated from a GMM, fit by logistic regression under each KL direction as the loss. Here is the same experiment, reproduced: let the true distribution \(P\) be a bimodal mixture of two Gaussians (two separate clusters of likely values), and ask what single, unimodal (one-peak) Gaussian \(Q\) each KL direction would prefer.

This is precisely the same asymmetry the "identify Bike" example below demonstrates in a discrete, decision-focused setting — reverse KL tolerates a generator that fails to cover a real pattern (mode-seeking: ignoring a peak) far more than it tolerates confidently inventing something fake. Watch for the same two ideas — "penalized for missing a real mode" vs. "penalized for inventing a fake one" — reappearing in each of the four cases below.

Click through the interactive version below for the reasoning behind each row:

7. Binary Cross-Entropy, Closing the Loop

For a binary classification target \(y\in\{0,1\}\) and a model prediction \(Q=P(\hat y=1)\), the true label distribution puts all its probability mass on the single correct value. Plugging this two-outcome case into the general cross-entropy formula \(H(P,Q)=-\sum_x P(x)\log Q(x)\) (summed over the two outcomes, "class 1" with probability \(Q\) and "class 0" with probability \(1-Q\)) gives:

This is exactly the loss function Lecture 8 differentiates step by step to derive backpropagation, and exactly the loss whose gradient with respect to a sigmoid output collapses to the remarkably clean \(\partial L/\partial Z_2 = Q - y\). Everything in this lecture — random variables, distributions, MLE, KL divergence — was building toward this one formula, which is not an arbitrary design choice but the direct consequence of maximizing the likelihood of the true labels under the model.

8. Summary

9. Code: Binary Cross-Entropy and a Manual KL Divergence

The script below computes binary cross-entropy for a small toy prediction, then computes KL divergence by hand on a small discrete distribution and verifies the \(H(P,Q)=H(P)+D_{KL}(P\|Q)\) identity numerically.

import numpy as np

# ---- binary cross-entropy on a toy prediction ----
y = np.array([1, 0, 1, 1])
Q = np.array([0.9, 0.2, 0.6, 0.4])   # model's predicted P(class=1)

bce = -np.mean(y * np.log(Q) + (1 - y) * np.log(1 - Q))
print("Binary cross-entropy loss:", round(bce, 4))

# ---- manual KL divergence on a small discrete distribution ----
# P = true distribution, Q = model distribution, over 4 outcomes
P = np.array([0.10, 0.40, 0.35, 0.15])
Qd = np.array([0.20, 0.30, 0.25, 0.25])
assert np.isclose(P.sum(), 1) and np.isclose(Qd.sum(), 1)

def entropy(p):
    return -np.sum(p * np.log(p))

def cross_entropy(p, q):
    return -np.sum(p * np.log(q))

def kl_divergence(p, q):
    return np.sum(p * np.log(p / q))

H_P   = entropy(P)
H_PQ  = cross_entropy(P, Qd)
D_KL  = kl_divergence(P, Qd)

print(f"\nH(P)          = {H_P:.4f}")
print(f"H(P,Q)        = {H_PQ:.4f}")
print(f"D_KL(P||Q)    = {D_KL:.4f}")
print(f"H(P) + D_KL   = {H_P + D_KL:.4f}  (should equal H(P,Q) above)")

⬇ Download lecture-06-crossentropy-kl.py   More resources for this lecture →