LLMs

Large language models: what next-token prediction produces, why it fails the way it does, and how to build on it honestly.

6 articles, newest first.

• Generative models and LLMs · Part 28 of 33 · 7 min How a model gets trained, from pretraining to RLHF The three stages that turn raw text into a chat assistant, what each one actually changes about the model, and why the base model and the assistant are different things.
• Generative models and LLMs · Part 27 of 33 · 6 min What a language model is actually doing A first-principles look at large language models: next-token prediction, why that simple goal produces apparent reasoning, and what temperature and sampling really control.
• Generative models and LLMs · Part 26 of 33 · 7 min Attention is the whole trick Queries, keys and values in plain language: how a word picks up meaning from its neighbours, and why one matrix multiplication solved the long-range dependency problem.
• Generative models and LLMs · Part 25 of 33 · 7 min Transformers explained from scratch Tokens, embeddings and a stack of identical blocks — how a transformer turns next-word prediction into a training objective, and why it trains in parallel where RNNs could not.
• Generative models and LLMs · Part 29 of 33 · 5 min Why prompting works at all Prompting is conditioning, not magic words. Understanding what the prompt does to the probability distribution explains which techniques help and which are folklore.
• Generative models and LLMs · Part 30 of 33 · 5 min Why language models make things up Hallucination is not a bug in the usual sense. It follows from what the model is trained to do, and only some of the popular fixes address the actual cause.