Every model takes numbers. Language is made of symbols. Bridging that gap is the first decision in any text project, and the obvious bridges are all bad in instructive ways.
Give each word an integer and you have told the model that aardvark is 1 and zebra is 40,000, and therefore that zebra is forty thousand times something. It is not. The numbers imply an order and a scale that the vocabulary does not have. So instead you go one-hot: one column per word, a single 1 in the column for the word present, zeros everywhere else. No fake ordering. This is what sits underneath bag-of-words models, and it works better than it has any right to — see TF-IDF still works — but it has a specific, fatal blind spot.
The problem with one-hot: everything is equally far from everything
Take the one-hot vectors for cat, dog, and bureaucracy. Compute the distance between any two of them. It is the same in all three cases. In a one-hot space, every word is exactly as similar to every other word, which is a formal way of saying the representation contains no information about meaning at all.
The practical cost is that nothing generalises. Train a classifier on tickets containing refund and it learns nothing whatsoever about tickets containing reimbursement. Those are separate columns and, to the model, separate universes. Every synonym, inflection, and spelling variant has to be learned from its own examples — a losing arrangement in a language with a long tail where most word forms appear a handful of times.
What you want is a representation where cat and dog sit close together and bureaucracy sits somewhere else. Which requires deciding, without a human writing it down, what “close” means for words.
You shall know a word by the company it keeps
The idea that solves it is older than the neural networks that made it famous. The linguist J. R. Firth put it as: you shall know a word by the company it keeps. This is the distributional hypothesis, and its claim is that words appearing in similar contexts tend to mean similar things.
It is easy to test on yourself. Read this: “she poured the wug into a glass and drank it.” You now know a great deal about wug. It is a liquid, probably potable, probably not motor oil. You learned that from the neighbours alone.
The hypothesis turns a philosophical question — what does a word mean? — into a counting problem. Meaning becomes a fingerprint of co-occurrence, and two words with similar fingerprints get similar representations. No dictionary, no linguist, no labels. Just a large pile of ordinary text.
How training turns company into geometry
The training setup is almost a trick. Give each word a short vector of, say, 300 numbers, initialised at random. Then set the model a chore: slide a window across a text corpus and, from the word in the middle, predict the words around it. Adjust the vectors by gradient descent so the predictions get better. That recipe is skip-gram, one of the two Mikolov and colleagues published as word2vec in Efficient Estimation of Word Representations in Vector Space in 2013.
Nobody wants the predictor. It is a pretext task, and its outputs are thrown away. What you keep is the vectors, because of what the training pressure does to them. To predict the context of coffee well, the model needs a vector that encodes “appears near drink, cup, morning”. The vector for tea faces almost the same pressure. Two words that keep the same company end up pushed toward the same region of the space, not because anyone declared them similar, but because similar vectors are the cheapest way to satisfy the objective.
The result is that geometry now carries meaning. Distance is similarity; nearest neighbours are near-synonyms and topical relatives. And some directions turn out to be consistent: the vector difference between many singular and plural pairs points roughly the same way, as does the difference between many country and capital pairs.
What king − man + woman actually shows
That last property produced the most-repeated result in the field: take the vector for king, subtract man, add woman, and the nearest vector to the result is queen. Flatten the space to two dimensions and the four words sit on the corners of a parallelogram.
Nobody put a “royalty” axis or a “gender” axis into the model. Both directions fell out of counting which words appear near which. The arithmetic works because the four vectors happen to sit on a parallelogram — not because the model knows what a queen is.
It is a real phenomenon and it deserves a careful reading.
What it shows is that a relationship — here, something like gendered counterpart — is encoded as a roughly constant offset applying across many word pairs. That is genuinely surprising. Nothing in the training objective asked for arithmetic to work. It fell out of the co-occurrence structure of language.
What it does not show is reasoning, or that dimensions are meaningful. Three things worth knowing before you quote the example:
- The standard way of running these analogies excludes the three input words from the answer. Without that exclusion, the nearest vector to the result is very often king itself, because you have moved only a short way from where you started.
- The clean examples are the ones that get published. Analogy accuracy across a full test set is respectable, not magical, and it collapses on relations that are rare in the training text.
- The same mechanism that captures king/queen captures whatever else the corpus contains, including occupational and gender stereotypes. Embeddings learn the statistics of the text they were given. That is the deal, and it is not optional.
Vectoring Words (Word Embeddings) by Computerphile builds the same geometry up from the counting.
Where one vector per word runs out
The classic setup assigns each word exactly one vector, and that assumption breaks in ways you will meet immediately in real text.
Polysemy. Bank has one vector, a smear between river banks and savings accounts, useful for neither. Newer contextual models produce a different vector for each occurrence of a word, computed from the sentence it sits in — the main reason they replaced static embeddings for hard tasks.
Opposites are neighbours. Hot and cold keep almost identical company — both precede water, day, coffee. Distributionally they are twins, so they land close together. If your task depends on the difference, similarity in embedding space will quietly mislead you.
No negation, no facts. The space encodes association, not truth. “Not a scam” and “a scam” share nearly all their words. Sentiment and factuality need something on top.
The through-line is worth holding onto: an embedding is not a definition. It is a compressed summary of where a word tends to appear, and every strength and failure follows from that one sentence. For how this plays out when you actually build something — normalisation, chunking, retrieval evaluation — the practical companion to this post is Embeddings, explained for people who ship things.