Machine Learning Foundations Recap

Before we build a single neuron, we recall what "learning from data" formally means — and the one tradeoff that governs every model you will ever train.

• Distinguish machine learning from explicit programming, and supervised, unsupervised and reinforcement learning from each other with concrete examples.
• Describe the general ML pipeline: data → features → model → loss → optimization → evaluation.
• Explain why data is split into train/validation/test sets, and what each split is for.
• Recognize underfitting and overfitting visually and diagnose them from training/test error patterns.
• State the bias-variance decomposition of expected test error and compute it for a small numeric example.
• Preview L1/L2 regularization as a tool for controlling model complexity.

1. What Is Machine Learning?

Classical programming means writing explicit rules: a human decides the logic, the computer executes it. Machine learning inverts this. We show the computer many examples of inputs and their correct outputs, and ask it to learn a function \(f\) that maps new, unseen inputs to correct outputs — without a human ever writing down the rule explicitly. Formally, given a dataset of examples, we search over a family of candidate functions (a "hypothesis space") for the member that best explains the data and, crucially, generalizes to data it has not seen.

Machine learning problems are usually grouped into three families:

Almost everything in this course — from Lecture 3's perceptron through Lecture 20's Transformers — is supervised learning. Lectures 21–24 return to unsupervised methods.

2. The General ML Pipeline

Nearly every supervised learning system, no matter how complex, is built from the same six stages:

3. Train / Validation / Test Splits

A model that has simply memorized its training data is useless — we care about performance on new, unseen inputs, a property called generalization. To measure generalization honestly, we never evaluate a model on the same data it was trained on. Instead we partition the available data into three disjoint sets:

A common split for moderate-sized datasets is roughly 70/15/15 or 80/10/10 (train/validation/test), though the right proportions depend on how much data you have overall.

4. Underfitting, Good Fit, and Overfitting

Every model has some amount of capacity — its ability to represent complicated functions. Too little capacity, and the model cannot capture the real pattern in the data (underfitting). Too much capacity relative to the amount of data, and the model starts fitting the noise in the training set instead of the underlying trend (overfitting). Both hurt generalization, for opposite reasons. The figures below show the same 11 noisy data points fit by three models of increasing complexity: a straight line, a smooth quadratic, and a high-degree polynomial that wiggles through nearly every point.

5. The Bias-Variance Tradeoff

Underfitting and overfitting are two symptoms of one underlying tradeoff. If we imagine retraining the same model architecture on many different random samples of training data, its expected error on a new test point decomposes into three additive parts:

$$\mathbb{E}\big[(y-\hat f(x))^2\big] \;=\; \underbrace{\big(\mathbb{E}[\hat f(x)]-f(x)\big)^2}_{\text{Bias}^2} \;+\; \underbrace{\mathbb{E}\big[(\hat f(x)-\mathbb{E}[\hat f(x)])^2\big]}_{\text{Variance}} \;+\; \underbrace{\sigma^2}_{\text{Irreducible noise}}$$

Simple models (a straight line) tend to have high bias and low variance. Complex models (a high-degree polynomial, or later, a very large neural network) tend to have low bias and high variance. The best generalizing model sits at the sweet spot that minimizes the sum of all three terms — not the model with the lowest training error, and not the simplest model either.

6. Regularization: A Quick Preview

If overfitting comes from a model having too much freedom relative to its data, one fix is to explicitly discourage that freedom during training. Regularization adds a penalty term to the loss that grows with the size of the model's weights, trading a little training accuracy for a model that generalizes better:

$$L_{\text{regularized}} = L_{\text{original}} + \lambda \cdot R(w)$$

We are only previewing this idea now. It resurfaces with real teeth in Lecture 23, where the "sparse autoencoder" enforces an L1-style penalty directly on hidden-layer activations rather than on weights.

7. Summary

8. Code: Underfitting vs. Overfitting on Synthetic Data

The script below generates the same kind of noisy, hump-shaped synthetic dataset used in Section 4, fits a degree-1 (underfit), degree-2 (good fit), and degree-12 (overfit) polynomial regression with scikit-learn, and prints the training-set mean squared error for each — watch it fall as the degree increases, even as the fit visually gets worse on new data.

import numpy as np
from numpy.polynomial import polynomial as P
from sklearn.metrics import mean_squared_error

rng = np.random.default_rng(0)

# ---- synthetic data: a mild "hump" shape plus noise ----
X = np.linspace(0, 10, 11)
true_y = -0.15 * (X - 4.5) ** 2 + 4.5
y = true_y + rng.normal(0, 0.3, size=X.shape)

def fit_and_eval(degree):
    coefs = np.polyfit(X, y, degree)          # highest power first
    y_hat = np.polyval(coefs, X)
    mse = mean_squared_error(y, y_hat)
    return coefs, mse

print(f"{'Degree':<8}{'Model':<12}{'Train MSE':>10}")
for degree, name in [(1, "Underfit"), (2, "Good fit"), (12, "Overfit")]:
    _, mse = fit_and_eval(degree)
    print(f"{degree:<8}{name:<12}{mse:>10.4f}")

# A degree-12 polynomial on 11 points interpolates almost exactly (near-zero
# training error) but will swing wildly between the training x-values --
# exactly the "memorized noise" behaviour shown in the lecture figure.
# Try evaluating each fitted polynomial at a dense grid (e.g. np.linspace(0,10,200))
# and plotting it against X, y with matplotlib to see this directly.

⬇ Download lecture-02-under-overfit.py   More resources for this lecture →