Evaluation Metrics: Regression & Classification

A model is only as trustworthy as the number we use to judge it — and the wrong number can make a broken model look perfect.

• Compute MSE and MAE by hand and explain why squaring/absolute value is necessary (the "errors cancel out" pitfall).
• Define every entry of a confusion matrix (TP, TN, FP, FN) including their names as Type-1 and Type-2 errors.
• Compute Accuracy, Precision, Recall, Specificity, and F1 from a confusion matrix, and explain the precision/recall tradeoff as the decision threshold changes.
• Plot and interpret an ROC curve and a Precision-Recall curve from a full threshold scan, and state what AUROC and AUPRC each measure.
• Choose the right metric for a given deployment scenario (spam filtering, medical diagnosis, imbalanced classes).
• Describe, at a high level, why sequence-generation tasks need specialized metrics like BLEU.

1. Regression Metrics: MSE & MAE

When the network predicts a continuous number (regression), we need a single scalar summarizing how far predictions are from the truth, averaged over all examples.

$$MSE = \frac{1}{n}\sum_{i=1}^n (y_{pred}^{(i)} - y_{true}^{(i)})^2 \qquad\qquad MAE = \frac{1}{n}\sum_{i=1}^n |y_{pred}^{(i)} - y_{true}^{(i)}|$$

Why not simply average the raw signed error \((y_{pred}-y_{true})\)? Because positive and negative errors cancel out.

Worked numerical example

Take \(y_{true} = [3,\ -0.5,\ 2]\) and \(y_{pred} = [2.5,\ 0.0,\ 2]\). Every arithmetic step is written out below, in order.

You can replay the same three steps interactively below — useful for testing yourself before moving on:

Because MSE squares each error, a single large error contributes disproportionately to the total — MSE is sensitive to outliers, and useful precisely when large errors are especially undesirable. MAE weights every error linearly regardless of size, giving a more robust, "typical error magnitude" reading that is less swayed by a few bad predictions.

2. Classification Metrics: The Confusion Matrix

Before applying any of this to real numbers, we need the standard vocabulary: what each of TP/TN/FP/FN means, and how Accuracy, Precision, Recall, Specificity, and F1 are each defined in terms of them. If you're already comfortable with these definitions, skip to the worked confusion matrix below, which applies them to a real 100-example population. If not, expand below.

New to this? Expand: confusion matrix terminology and metric definitions

For classification, we first sort every prediction into one of four buckets by comparing it against the true label:

Term	Meaning
TP (True Positive)	Predicted positive, actually positive
TN (True Negative)	Predicted negative, actually negative
FP (False Positive)	Predicted positive, actually negative — a Type-1 error
FN (False Negative)	Predicted negative, actually positive — a Type-2 error

From these four counts, every standard classification metric follows directly:

$$Accuracy = \frac{TP+TN}{TP+TN+FP+FN} \qquad Precision = \frac{TP}{TP+FP} \qquad Recall = \frac{TP}{TP+FN} \qquad F1 = \frac{2\cdot Precision\cdot Recall}{Precision+Recall}$$

Specificity (true negative rate) is the negative-class analogue of recall: \(Specificity = \dfrac{TN}{TN+FP}\).

3. Worked Confusion Matrix

Consider a population of 100 predictions with \(TP=40,\ FP=10,\ FN=5,\ TN=45\):

You can replay the same walkthrough interactively below — useful for testing yourself before moving on:

4. The Precision/Recall Tradeoff

A classifier typically outputs a probability, and we choose a decision threshold (default 0.5) above which we call the prediction "positive." Moving that threshold trades precision against recall.

Consider a population of 20 (10 positive, 10 negative) scored at two different thresholds:

Raising the threshold from 0.50 to 0.60 makes the classifier more conservative about calling something "positive." Two fewer true positives are now missed as false negatives (FN goes 1→3), so recall drops (0.9→0.7). At the same time precision drops too here (0.81→0.77) because the classifier's accuracy on the examples it still does call positive didn't improve enough to offset the lost true positives — in general, raising the threshold pushes precision up and recall down, but the exact numbers depend on the score distribution, as this example shows. The broader point: precision and recall move in tension as the threshold changes, and F1 (which combines them) captures that tradeoff in a single number — here F1 falls from 0.857 to 0.733 as the threshold rises.

5. ROC & Precision-Recall Curves

Rather than picking one threshold, we can scan across every possible threshold and plot how the metrics move together. The table below scans thresholds from 1.00 down to 0.00 over the same population of 20 (10 positive, 10 negative):

6. Choosing the Right Metric

• Precision matters when false positives are costly — e.g. spam detection: flagging a legitimate email as spam (FP) is worse than letting one spam email through (FN).
• Recall matters when false negatives are costly — e.g. medical diagnosis: missing an actual disease case (FN) is far worse than a false alarm that gets ruled out by a follow-up test (FP).
• F1 balances both when neither type of error clearly dominates, or when you need one number to compare models.
• Accuracy is only meaningful when classes are roughly balanced — with a 99%/1% class split, a model that always predicts the majority class scores 99% accuracy while being useless.

7. Aside: Metrics for Sequence Generation (BLEU)

8. Common Pitfalls

9. Summary

10. Code: Computing Evaluation Metrics

The script below reproduces every worked number above: the MSE/MAE example, the 100-sample confusion matrix, and the full 21-row threshold scan used to draw the ROC/PR curves.

import numpy as np

# ---- Regression metrics ----
y_true = np.array([3, -0.5, 2])
y_pred = np.array([2.5, 0.0, 2])
errors = y_pred - y_true
mse = np.mean(errors ** 2)
mae = np.mean(np.abs(errors))
print(f"MSE = {mse:.4f}   MAE = {mae:.4f}")   # 0.1667, 0.3333

# ---- Classification metrics from a confusion matrix ----
def cm_metrics(TP, TN, FP, FN):
    acc  = (TP + TN) / (TP + TN + FP + FN)
    prec = TP / (TP + FP)
    rec  = TP / (TP + FN)
    spec = TN / (TN + FP)
    f1   = 2 * prec * rec / (prec + rec)
    return dict(accuracy=acc, precision=prec, recall=rec, specificity=spec, f1=f1)

example2 = cm_metrics(TP=40, TN=45, FP=10, FN=5)
print("Example 2 (TP=40,FP=10,FN=5,TN=45):", {k: round(v, 4) for k, v in example2.items()})

# ---- Full threshold scan (population of 20: 10 positive, 10 negative) ----
scan = [
    # thr,  TP, TN, FP, FN
    (1.00, 0,10,0,10), (0.95, 1,10,0,9), (0.90, 2,10,0,8), (0.85, 2,9,1,8),
    (0.80, 3,9,1,7),   (0.75, 4,9,1,6),  (0.70, 5,9,1,5),  (0.65, 5,8,2,5),
    (0.60, 6,8,2,4),   (0.55, 7,8,2,3),  (0.50, 8,8,2,2),  (0.45, 9,8,2,1),
    (0.40, 9,7,3,1),   (0.35, 9,6,4,1),  (0.30, 9,5,5,1),  (0.25, 9,4,6,1),
    (0.20, 9,3,7,1),   (0.15, 9,2,8,1),  (0.10, 9,1,9,1),  (0.05, 10,1,9,0),
    (0.00, 10,0,10,0),
]

print("\nthr    Acc   Prec  Recall Spec  F1    FPR")
roc_points, prc_points = [], []
for thr, TP, TN, FP, FN in scan:
    m = cm_metrics(TP, TN, FP, FN)
    fpr = 1 - m['specificity']
    roc_points.append((fpr, m['recall']))
    prc_points.append((m['recall'], m['precision']))
    print(f"{thr:.2f}  {m['accuracy']:.2f}  {m['precision']:.3f} {m['recall']:.2f}  "
          f"{m['specificity']:.2f}  {m['f1']:.3f}  {fpr:.2f}")

# crude trapezoidal AUROC estimate from the scanned points (sorted by FPR)
roc_sorted = sorted(roc_points)
auroc = np.trapz([r for _, r in roc_sorted], [f for f, _ in roc_sorted])
print(f"\nApprox AUROC (trapezoidal, from this 21-point scan) = {auroc:.3f}")

⬇ Download lecture-10-metrics.py   More resources for this lecture →