The Perceptron & Biological Inspiration

The first trainable artificial neuron — a loose analogy to a biological neuron, a precise learning rule, and a hard limitation that shaped 15 years of AI history.

• Describe the biological-neuron analogy behind artificial neurons, and state clearly where the analogy stops.
• State the McCulloch-Pitts neuron (1943) and Rosenblatt's Perceptron (1958) and the equation each computes.
• Derive and apply the perceptron learning rule to update weights from a misclassified example.
• Interpret a trained perceptron geometrically as a linear decision boundary (hyperplane).
• Hand-train a perceptron on the AND logic gate and verify the result numerically.
• Explain why the perceptron cannot represent XOR, and why this motivated multi-layer networks.

1. A Biological Analogy (Not a Literal Model)

Artificial neurons take their name — and a loose structural inspiration — from biological neurons, but the resemblance is a starting metaphor, not an engineering blueprint. It is still a useful way to build intuition for the first time:

2. The McCulloch-Pitts Neuron (1943)

The first mathematical model of a neuron predates the perceptron by 15 years. McCulloch and Pitts (1943) proposed a binary threshold unit: sum the binary inputs, and fire (output 1) if the sum meets or exceeds a fixed threshold, otherwise stay silent (output 0). It had no learning rule — the threshold and connections were fixed by hand — but it established the core idea that a network of simple binary units could, in principle, compute logical functions.

3. Rosenblatt's Perceptron (1958)

Rosenblatt's key addition was to attach a learning rule to the McCulloch-Pitts idea, and to allow real-valued, adjustable weights instead of fixed connections. The perceptron computes a weighted sum of its inputs, adds a bias, and passes the result through a step function:

$$\hat y = \text{step}(w\cdot x + b), \qquad \text{step}(z) = \begin{cases}1 & z \ge 0 \\ 0 & z < 0\end{cases}$$

Here \(w=[w_1,\dots,w_n]\) are the weights, \(b\) is the bias (equivalent to a threshold), and \(x=[x_1,\dots,x_n]\) is the input vector. Unlike the McCulloch-Pitts neuron, these weights are learned from labeled examples rather than fixed by a designer.

4. The Perceptron Learning Rule

Training a perceptron means repeatedly presenting labeled examples \((x,y)\) and nudging the weights whenever the prediction is wrong:

$$w \leftarrow w + \eta(y-\hat y)\,x \qquad\qquad b \leftarrow b + \eta(y-\hat y)$$

Here \(\eta\) (eta) is the learning rate, a small positive number controlling the size of each update. Notice what happens in each case: if \(\hat y = y\) (correct), the error term \((y-\hat y)\) is zero and nothing changes. If the perceptron predicted 0 but should have predicted 1, the error is \(+1\) and weights move in the direction of \(x\) (making the weighted sum larger next time). If it predicted 1 but should have predicted 0, the error is \(-1\) and weights move away from \(x\).

5. Geometric Interpretation

The equation \(w\cdot x + b = 0\) defines a straight line in 2D (a hyperplane in higher dimensions). The perceptron's decision rule simply asks which side of that line a point falls on: \(w\cdot x + b \ge 0\) classifies as 1, otherwise 0. Training a perceptron is therefore geometrically equivalent to searching for a straight line that separates the "1" points from the "0" points — which is only possible if the data is linearly separable.

6. Worked Example: Learning the AND Gate

We train a 2-input perceptron on the AND logic function, which is linearly separable. Data (in the order they are processed), starting weights \(w=[0,0]\), bias \(b=0\), learning rate \(\eta=0.1\), threshold convention \(\text{step}(z)=1 \text{ if } z\ge 0\):

We now trace one full pass over the four examples, in order, updating the weights whenever the prediction is wrong. Every number is written out below — nothing is left hidden.

You can replay the same pass interactively below — useful for testing yourself before moving on:

After this single pass, \(w=[0.1, 0.1]\), \(b=0\) — closer to a correct solution, but not there yet (check: for \((0,0)\), \(z=0\ge0\) still predicts 1 instead of 0). Continuing this exact update rule for two more full passes converges: re-running the update rule (and the downloadable NumPy code below) shows Pass 2 ends at \(w=[0.2,\,0.1]\), \(b=-0.1\), and Pass 3 ends at \(w=[0.2,\,0.1]\), \(b=-0.2\), with zero updates on the verification pass that follows — a valid separating solution. (An earlier version of this page stated the converged weights as \(w\approx[0.2,0.2]\), \(b\approx-0.3\); independently re-deriving the update trace found that number was wrong for the standard floating-point run — the corrected trace is below, and the box after the animation explains a genuinely interesting subtlety this convergence value exposes.) Watch the separating line itself rotate and shift, pass by pass:

Isolated for reference, the boundary the downloadable code actually reports:

7. The XOR Problem and the First AI Winter

AND, OR, and NOT can all be computed by a single perceptron, because their "1" and "0" outputs are linearly separable. XOR cannot: plot its four points — \((0,0)\to0\), \((0,1)\to1\), \((1,0)\to1\), \((1,1)\to0\) — and no single straight line can separate the two classes; the "1" points sit on opposite corners from each other. Minsky and Papert proved this formally in 1969, and their book's broader critique of the perceptron's limitations contributed to a sharp, decade-long collapse in neural network research funding and interest — the first "AI winter."

8. Summary

9. Code: Perceptron Learning From Scratch

This NumPy implementation reproduces the AND-gate example above exactly, training until convergence and printing the final weights and bias.

import numpy as np

def step(z):
    return np.where(z >= 0, 1, 0)

# ---- AND-gate dataset, in the exact order used in the lecture ----
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
y = np.array([0, 0, 0, 1])

w = np.zeros(2)
b = 0.0
eta = 0.1
max_epochs = 20

for epoch in range(1, max_epochs + 1):
    updates = 0
    for xi, yi in zip(X, y):
        z = np.dot(w, xi) + b
        y_hat = step(z)
        error = yi - y_hat
        if error != 0:
            w += eta * error * xi
            b += eta * error
            updates += 1
    print(f"Epoch {epoch}: w={w}, b={b:.3f}, updates={updates}")
    if updates == 0:
        print("Converged -- no updates this epoch.")
        break

print("\nFinal weights:", w, " Final bias:", round(b, 3))
print("\nPredictions on AND:")
for xi, yi in zip(X, y):
    pred = step(np.dot(w, xi) + b)
    print(f"  x={xi} -> predicted={pred}, target={yi}")

⬇ Download lecture-03-perceptron.py   More resources for this lecture →