Self-Organizing Maps & K-Means Clustering

Neurons that compete instead of cooperate: from a single winner-take-all update rule to Kohonen's topology-preserving map and the ubiquitous k-means algorithm.

• Define competitive learning and derive the winner-take-all update rule from first principles.
• Explain how a Self-Organizing Map extends competitive learning with a neighborhood function to preserve topology.
• Compute Best Matching Units, neighborhood function values, and weight updates for a small SOM by hand.
• Describe how a SOM performs dimensionality reduction, and list real-world applications of SOMs.
• Implement k-means clustering from its assign/update principles and connect the assignment step to competitive learning's nearest-neighbor rule.
• Compare k-means and SOM as, respectively, topologically-unconstrained and topologically-constrained prototype learners.

1. Why Unsupervised Learning?

Every model built so far in this course — from the perceptron in Lecture 3 to the Transformer in Lecture 20 — was trained with labels: an input \(x\) paired with a target \(y\), and a loss function that measured how far the network's prediction strayed from that target. Most data in the world doesn't come with labels attached. Raw sensor logs, unannotated images, customer transaction histories, gene expression matrices — someone would have to hand-label all of it before supervised learning could touch it.

Unsupervised learning asks a different question: given only \(x\), with no \(y\) at all, can a model discover useful structure on its own — groups of similar points, a compact representation, or a map of how the data is organized? This lecture builds up unsupervised learning from a single, elegant mechanism — neurons competing for the right to respond to an input — through two of its most important instantiations: the Self-Organizing Map and k-means clustering.

2. Competitive Learning

Let the input be \(x\in\mathbb{R}^d\) and let each neuron \(i\) hold a weight vector \(w_i\in\mathbb{R}^d\). The winner is the neuron whose weight vector is closest to the input by Euclidean distance:

$$i^{*}=\arg\min_i \lVert x-w_i\rVert$$

Only the winning neuron's weights move — every other neuron is left untouched. With learning rate \(\eta(t)\):

$$w_{i^{*}}(t+1)=w_{i^{*}}(t)+\eta(t)\big(x-w_{i^{*}}(t)\big) \qquad\text{(all other } w_i \text{ unchanged)}$$

The update simply nudges the winner's weight vector a fraction \(\eta(t)\) of the way toward the input. Over many inputs, each neuron drifts toward the center of the region of input space it keeps winning — it becomes a prototype for that region.

Competitive learning is the mechanism underneath: clustering (it behaves like an online, incremental version of k-means — the explicit bridge we'll make precise in Part C), prototype learning, feature specialization, dimensionality reduction & visualization — the foundation the Self-Organizing Map builds on — and general pattern discovery.

3. Worked Example: Winner-Take-All

Input dimension 2, three neurons, learning rate \(\eta=0.5\). Initial weights: \(N_1=(0.2,0.6)\), \(N_2=(0.8,0.4)\), \(N_3=(0.5,0.9)\). Input \(x=(0.1,0.5)\).

You can replay the same three steps interactively below — useful for testing yourself before moving on:

4. Self-Organizing Maps (SOM)

Competitive learning finds prototypes, but the prototypes it finds have no relationship to one another — neuron 7's prototype could be nothing like neuron 8's. The Self-Organizing Map (Kohonen, 1982) fixes this by arranging the output neurons on a low-dimensional grid — usually 2D — and forcing neighboring grid neurons to learn similar prototypes. It projects high-dimensional input onto that grid while preserving topology: inputs that are close in the original space end up mapped to neurons that are close on the grid. This topology preservation is exactly what distinguishes a SOM from plain competitive learning.

The output layer is a grid of neurons — e.g. \(m\times n\) — where each neuron \(i\) has both a weight vector \(w_i\in\mathbb{R}^d\) (as before) and a fixed grid position \(r_i\) that never changes during training. Training combines two ideas: competition (find the winner, exactly as in Part 2) with neighborhood cooperation (also update the winner's neighbors, scaled by how far they sit from the winner on the grid).

What's new is the Gaussian neighborhood function, which measures how "close" neuron \(i\) is to the BMU on the grid (not in weight space):

$$h_{i,i^{*}}(t)=\exp\!\left(-\frac{\lVert r_i-r_{i^{*}}\rVert^2}{2\sigma^2(t)}\right)$$

Every neuron in the map is updated — not just the winner — scaled by this neighborhood value:

$$w_i(t+1)=w_i(t)+\eta(t)\cdot h_{i,i^{*}}(t)\cdot\big(x-w_i(t)\big)$$

Two limits make this rule click. When \(i=i^{*}\), the grid distance is 0, so \(h_{i,i^{*}}=1\) — the winner gets the full update, exactly as in ordinary competitive learning. As the grid distance from the winner grows, \(h_{i,i^{*}}\to 0\) — distant neurons barely move at all. Neurons that sit close to the winner on the grid get a partial nudge toward the same input. Repeat this over many inputs, and neighboring grid neurons are continually pulled toward similar regions of input space — which is precisely what makes the map preserve topology.

5. Worked Example: BMU & Neighborhood Update

A 2×2 grid with fixed positions \(r_{11}=(0,0),\ r_{12}=(0,1),\ r_{21}=(1,0),\ r_{22}=(1,1)\). Input \(x=(0.2,0.8)\), \(\eta=0.5\), \(\sigma=1.0\).

You can replay the same five steps interactively below — useful for testing yourself before moving on:

The neighborhood values from this step form a small, clean picture of how influence fades with grid distance:

6. SOM as Dimensionality Reduction

Because every input is ultimately reduced to "which grid cell won," a SOM doubles as a dimensionality-reduction technique: a high-dimensional input is summarized by a low-dimensional (often 2D) grid coordinate that a human can inspect on a map. Consider a SOM with input dimension 3 and a 2×2 grid:

For input \(x=[0.8,0.2,0.5]\), computing all four distances shows N12 is closest, so the BMU is (0,1) — the SOM's 2D "output" for this 3D input is simply the coordinate (0, 1). Running the same computation over several inputs produces a mapping table:

Applications of SOMs: clustering & prototype discovery (customer segmentation, text topic discovery, bioinformatics), dimensionality reduction (as preprocessing for other ML models), anomaly detection (inputs that land far from every learned prototype), robotics & control (building sensor and motor maps), and industrial/IoT settings (fault detection, equipment health monitoring).

T. Kohonen, "The Self-Organizing Map," Proc. IEEE, 1990.

7. K-Means Clustering

K-means is the simplest widely-used clustering algorithm, and it shares its core mechanism with competitive learning:

1. Choose the number of clusters \(k\) and initialize \(k\) centroids (e.g. randomly, or with the k-means++ heuristic).
2. Assign every data point to its nearest centroid by Euclidean distance — exactly the same winner-takes-all, nearest-neighbor idea as the competitive-learning winner rule in Part 2.
3. Update each centroid to the mean of all points currently assigned to it.
4. Repeat steps 2–3 until centroids stop moving (convergence) or a maximum iteration count is reached.

K-means minimizes the within-cluster sum of squares:

$$J=\sum_{i=1}^{k}\sum_{x\in C_i}\lVert x-\mu_i\rVert^2$$

8. Worked Example: K-Means by Hand

Since the local slides don't include a k-means example, here is a small, self-contained one with \(k=2\) and six 2D points, run for two full iterations.

You can replay the same iterations interactively below — useful for testing yourself before moving on:

9. K-Means vs. SOM

Summary

10. Code: Competitive Learning & K-Means From Scratch

The NumPy code below reproduces the Section 3 winner-take-all update exactly, then implements k-means from scratch on a small synthetic 2D dataset, printing centroids after every iteration.

import numpy as np

# ---------------------------------------------------------------
# Part A: Competitive learning — reproduces the worked example
# ---------------------------------------------------------------
neurons = np.array([[0.2, 0.6], [0.8, 0.4], [0.5, 0.9]])  # N1, N2, N3
x = np.array([0.1, 0.5])
eta = 0.5

dists = np.linalg.norm(neurons - x, axis=1)
winner = np.argmin(dists)
print("Distances:", np.round(dists, 3), " Winner: N", winner + 1)

neurons[winner] = neurons[winner] + eta * (x - neurons[winner])
print("Updated winner weights:", np.round(neurons[winner], 3))
print("Other neurons unchanged:", np.round(neurons, 3))

# ---------------------------------------------------------------
# Part C: K-means from scratch — small synthetic dataset, k=2
# ---------------------------------------------------------------
def kmeans(X, centroids, n_iter=3):
    centroids = centroids.copy()
    for it in range(n_iter):
        dists = np.linalg.norm(X[:, None, :] - centroids[None, :, :], axis=2)
        assign = np.argmin(dists, axis=1)
        new_centroids = np.array([
            X[assign == k].mean(axis=0) if np.any(assign == k) else centroids[k]
            for k in range(len(centroids))
        ])
        print(f"Iter {it+1}: assignments={assign.tolist()}  "
              f"centroids={np.round(new_centroids, 3).tolist()}")
        if np.allclose(new_centroids, centroids):
            print("Converged.")
            centroids = new_centroids
            break
        centroids = new_centroids
    return centroids, assign

X = np.array([[1, 1], [1.5, 2], [3, 1], [5, 7], [3.5, 5], [4.5, 5]])
init_centroids = np.array([[1, 1], [5, 7]])
final_centroids, final_assign = kmeans(X, init_centroids, n_iter=3)
print("Final centroids:", np.round(final_centroids, 3))

J = sum(np.linalg.norm(X[i] - final_centroids[final_assign[i]]) ** 2 for i in range(len(X)))
print("Within-cluster sum of squares J =", round(J, 3))

⬇ Download lecture-21-som-kmeans.py   More resources for this lecture →