What a neural network actually is

A neural network is a function. Numbers go in, numbers come out. That is the whole of it. The word “neuron” is a historical accident that makes people expect something alive in there, and there isn’t. There is arithmetic, arranged in a particular shape, with a lot of adjustable constants.

The useful way to meet the idea is on a concrete problem, so take the one the field cut its teeth on: read a handwritten digit. The image is 28 by 28 pixels in grayscale. Flatten it and you have 784 numbers, each between 0 (black) and 1 (white). The answer you want is one of ten labels. So the function you need takes 784 numbers in and gives 10 numbers out, where the largest output says which digit it thinks it saw.

One neuron is a weighted sum with a threshold

Start with a single unit. It looks at all 784 inputs and produces one number. It does this in the dullest possible way: multiply each input by its own weight, add the results together, then add one more number called the bias.

In full: w1*x1 + w2*x2 + ... + w784*x784 + b.

The weights say what this unit cares about. A large positive weight on a pixel means “if this pixel is bright, that’s evidence for me firing”. A negative weight means the opposite. The bias shifts the whole thing up or down — it sets how much total evidence is needed before the unit gets excited at all. A unit with a bias of -10 is a sceptic; it stays quiet unless the weighted sum is strongly positive.

If you stopped here you would have linear regression with a fancier name. The next part is what makes it different.

The squash is not a detail

The weighted sum can be any number at all: -400, 0.02, 17. Feeding raw sums into the next layer would be a problem, and not only for scale reasons. Stack two linear steps and the result is still linear — the composition of two straight-line functions is a straight line. A hundred layers of pure weighted sums collapse into one weighted sum. You would have spent a lot of compute to build a slightly slower version of a very simple model.

So each unit passes its sum through a nonlinear function before handing it on. For years that was the sigmoid, an S curve that squashes anything into the range 0 to 1. Today it is almost always ReLU: output the sum if it is positive, otherwise output zero. Comically simple, and it works better, largely because it does not flatten out for large inputs the way the sigmoid does.

The nonlinearity is the entire reason a deep network can express anything a shallow one cannot. Treat it as load-bearing, not garnish.

Layers, and what the middle ones are for

Now put units side by side into a layer, and stack layers. A small classic network for digits is 784 inputs, then a layer of 16 units, then another 16, then 10 outputs. Each layer takes the previous layer’s outputs as its inputs. Shrink that to 4 inputs, 3 units and 2 outputs and the whole shape fits on a page.

Thicker lines are larger weights. The teal unit does exactly what the section above described — multiply, add, add the bias, squash — and so does every other unit in the picture. Nothing else happens anywhere in the network.

The hopeful story is that the middle layers learn parts: the first finds edges, the second assembles edges into loops and strokes, and the last says a loop on top of a vertical stroke is a 9. It is a good story for building intuition and it is roughly what happens in large image networks. In a tiny fully connected net on digits, it is mostly not what happens — inspect the learned weights and you find messy patterns that work without being interpretable. Keep the story as scaffolding, not as a claim about the model. That gap between what we imagine a model learned and what it actually keyed on bites people constantly; see feature importance is not explanation.

The whole forward pass is two lines of arithmetic per layer:

import numpy as np

def forward(x, layers):
    """x: input vector. layers: list of (weights, bias) pairs."""
    for W, b in layers[:-1]:
        x = np.maximum(0, W @ x + b)   # ReLU
    W, b = layers[-1]
    scores = W @ x + b
    return np.exp(scores) / np.exp(scores).sum()   # softmax

That function, given the right weights, reads handwriting. Given the wrong weights it outputs confident nonsense. Nothing else changes between the two.

Learning means picking 13,002 numbers

Count the adjustable constants in that 784-16-16-10 network. The first layer has 784 × 16 weights plus 16 biases. The second has 16 × 16 plus 16. The last has 16 × 10 plus 10. Add them up: 13,002 numbers.

“Training a neural network” means searching for values of those 13,002 numbers that make the outputs right on examples where you know the answer. That is the complete job description. No rules are written, no features are hand-designed, no logic is programmed. A search process nudges thirteen thousand dials until the errors get small.

Two questions follow immediately, and they are the two questions the rest of deep learning answers. First: how do you measure “wrong” as a single number, so the search has something to minimise? Second: with 13,002 dials, how do you know which way to turn each one? The answers are the cost function with gradient descent, and backpropagation. Modern networks have billions of these numbers instead of thousands, and the two answers are unchanged.

What this buys you, and what it costs

You get a model that can fit relationships nobody could write down by hand, from raw inputs, with no feature-engineering phase. You pay with a model that has no account of why it answered as it did, needs a lot of labelled examples, and will happily memorise your training set if you let it.

For images, audio and text that trade is usually excellent. For a table of fifty business columns and forty thousand rows, it usually is not — a gradient boosted tree will beat it and train in seconds. The architecture is a tool with a shape, not a general upgrade. Start with the simplest model that could work and let the problem argue you upwards.

3Blue1Brown’s But what is a neural network? draws the layers and weights moving as the network runs.

Anyone who has understood weighted sums, a squashing function, and the fact that thirteen thousand constants are sitting there waiting to be tuned has understood the architecture completely. Everything after that — convolutions, attention, residual connections — is a different arrangement of the same three ideas. What the picture leaves out is how the constants get set: gradient descent, run on the derivative of the loss with respect to every one of them, repeated a few hundred thousand times. That mechanism is longer to explain than the architecture is.