A computer does not see a photo. It sees a grid of numbers — one number per pixel per colour channel, nothing else. So any claim that a program “recognises a cat” has to cash out as arithmetic on that grid. The question that built modern computer vision is a small one: what is the simplest useful arithmetic you can do on a grid of numbers?
The answer turned out to be: take a much smaller grid, lay it on top of the image, multiply the overlapping numbers, add them up, write the total down. Then slide it one pixel and do it again. That is a convolution. It sounds too plain to matter, and it is the single operation underneath almost every vision system built in the last decade.
Sliding a small grid over a big one
Call the small grid a kernel. A 3×3 kernel has nine numbers. Put it over the top-left 3×3 patch of the image, multiply each kernel number by the pixel under it, sum the nine products, and that sum becomes one pixel of a new image. Move right one pixel, repeat, and keep going until you have covered everything. One step of that is drawn below.
Small teal numbers are the kernel weights, large black numbers the pixels. Nine multiplications, one sum, one output cell — then the kernel slides one column right and it repeats.
The output is a new grid — a feature map. It is slightly smaller than the input, because the kernel cannot hang off the edge; pad the border with zeros if you want to keep the size. Slide two pixels at a time instead of one — a stride of 2 — and the output is roughly half as wide and half as tall.
What the output means depends entirely on the nine numbers. Nine values of 1/9 average each neighbourhood: that is a blur. A large positive centre with negative neighbours amplifies the gap between a pixel and its surroundings: that is a sharpen. The mechanics never change. Only the nine numbers do.
Edge detection you can do by hand
Here is the case worth understanding properly, because everything else follows from it. Take a kernel whose left column is negative and whose right column is positive:
import numpy as npfrom scipy.signal import convolve2d
img = np.zeros((8, 8))img[:, 4:] = 1.0 # black on the left, white on the right
sobel_x = np.array([[-1, 0, 1], [-2, 0, 2], [-1, 0, 1]]) # left column negative, right column positive
print(convolve2d(img, sobel_x, mode="valid").astype(int))Run it and almost the whole output is zeros. The only nonzero values sit in the columns straddling the boundary between dark and light.
The reason is worth saying out loud. In a flat region the pixels on the left and on the right are the same, so the negatives and positives cancel and the sum is zero. Where brightness changes they do not cancel, and you get a large number. This kernel does not “know” about edges. It computes a difference, and a vertical edge is a horizontal difference in brightness. Transpose it and you find horizontal edges instead.
No edge detector is hiding in there. A kernel is a small question asked at every position in the image at once. “Is it brighter on the right than the left, here?”
The change that mattered: learn the numbers
For thirty years, computer vision was the craft of designing those numbers by hand: edge maps, gradient histograms, corner detectors, hand-tuned descriptors, all feeding a classifier. Progress was slow, because a human had to guess in advance which patterns mattered.
The break came from noticing that the nine numbers in a kernel are just parameters. Nothing distinguishes them from the weights in any other layer of a neural network. So do not design them. Initialise them randomly, wire the whole thing to a loss function, and let gradient descent move them until the network classifies images correctly. The kernels that survive training are the ones that produce useful features — chosen by the data, not by a researcher’s intuition.
This is the same shift from hand-built features to learned ones seen in tabular work, only more dramatic, because images have so many pixels that human intuition runs out fast. It does not make feature thinking obsolete — feature engineering that survives production is still most of the work on structured data — but for raw pixels, learning won decisively.
Why stacking layers builds up meaning
One convolutional layer sees a 3×3 patch. Stack a second on top and each of its outputs depends on a 5×5 region of the original image. Stack ten and a single output depends on a large fraction of the picture. This growing window is called the receptive field. Depth buys reach.
The consequence is a rough hierarchy. Early layers, looking at tiny patches, learn what a tiny patch can express: edges at various angles, colour blobs, spots. Middle layers combine those into textures, corners, and repeated patterns. Later layers, with a wide receptive field, respond to parts — a wheel-ish thing, an eye-ish thing — and the final layers to whole objects. Nobody programmed that ladder. It falls out of stacking a local operation many times.
Two more properties come free. Weight sharing: the same nine numbers apply everywhere, so a feature learned in the top-left corner is detected in the bottom-right too, and the layer needs nine parameters instead of one per pixel position — a fully connected layer on a modest image needs millions of weights for the same job. And locality: nearby pixels are related and distant ones usually are not, which is true of images and is baked into the operation rather than learned.
For the sliding-and-summing itself, and for where the same maths turns up outside images, watch But what is a convolution? by 3Blue1Brown.
Where the intuition breaks
Convolutional networks are not doing what your visual system does, and pretending otherwise leads to bad predictions about failure.
- Translation, yes. Rotation and scale, no. Weight sharing gives you robustness to shifting an object sideways. Turn it 40 degrees or halve its size and the learned kernels may not fire. That robustness comes from data augmentation, not from the architecture.
- Texture beats shape. Trained on natural photographs, these networks lean heavily on local texture. This is why an image can look obviously like one thing to a person and score confidently as another.
- The features are learned, not legible. Visualising kernels is fun and it is not an explanation. The same caution applies here as with feature importance on tabular models: a picture of what a unit responds to is a hypothesis, not a reason.
It is worth marking where the whole approach stops applying. Convolution assumes the input has a grid on it, where being next to something means something — pixel beside pixel, sample beside sample, word beside word. A customer table has no such geometry. Column 4 is not adjacent to column 5 in any sense a kernel can use, and reordering the columns would change what the filters see. Run a convolution over that data and it trains without error, reports a loss, and learns nothing you could not have got from a tree.