Foundations

The core ideas of data science and AI explained from scratch — what each method actually does, before any library gets involved.

33 articles, newest first.

• Before any of the maths · Part 2 of 33 · 4 min Always build the dumb baseline first Majority class, yesterday-equals-today, one rule of thumb: an hour spent on a deliberately dumb baseline is the best ROI in a data project.
• Before any of the maths · Part 3 of 33 · 4 min Start with the simplest model that could possibly work Linear regression is not beneath you. Reaching for deep learning first is usually a mistake — here is the discipline that replaces it, and when to escalate.
• Before any of the maths · Part 1 of 33 · 3 min The job is mostly cleaning data, and no one warns you Courses spend 5% of the time on the 80% of the work. What data cleaning actually involves, why it is the real skill, and how to get good at it deliberately.
• Reasoning under uncertainty · Part 10 of 33 · 7 min Markov chains, from states to stationary distributions What a Markov chain is, why the memoryless assumption is less crazy than it sounds, how a chain settles into a long-run distribution, and where you meet one in real work.
• Generative models and LLMs · Part 28 of 33 · 7 min How a model gets trained, from pretraining to RLHF The three stages that turn raw text into a chat assistant, what each one actually changes about the model, and why the base model and the assistant are different things.
• Generative models and LLMs · Part 27 of 33 · 6 min What a language model is actually doing A first-principles look at large language models: next-token prediction, why that simple goal produces apparent reasoning, and what temperature and sampling really control.
• Generative models and LLMs · Part 26 of 33 · 7 min Attention is the whole trick Queries, keys and values in plain language: how a word picks up meaning from its neighbours, and why one matrix multiplication solved the long-range dependency problem.
• Generative models and LLMs · Part 25 of 33 · 7 min Transformers explained from scratch Tokens, embeddings and a stack of identical blocks — how a transformer turns next-word prediction into a training objective, and why it trains in parallel where RNNs could not.
• Generative models and LLMs · Part 29 of 33 · 5 min Why prompting works at all Prompting is conditioning, not magic words. Understanding what the prompt does to the probability distribution explains which techniques help and which are folklore.
• Reasoning under uncertainty · Part 9 of 33 · 7 min Entropy, or how to measure surprise Information as surprise, why bits fall out of a logarithm, and the straight line from there to the cross-entropy loss you train with and the splits a decision tree picks.
• Generative models and LLMs · Part 30 of 33 · 5 min Why language models make things up Hallucination is not a bug in the usual sense. It follows from what the model is trained to do, and only some of the popular fixes address the actual cause.
• Reasoning under uncertainty · Part 6 of 33 · 7 min The central limit theorem, from first principles Why sums of many small independent effects turn into a bell curve, what the theorem promises about sample means, where it fails, and why "n > 30" is folklore.
• Neural networks · Part 23 of 33 · 7 min Convolution, and how computers learned to see What a convolution actually does to an image, why sliding a tiny grid of numbers finds edges, and why learning those numbers instead of designing them changed computer vision.
• Generative models and LLMs · Part 31 of 33 · 8 min How diffusion models make images The idea behind AI image generators, explained plainly: teach a model to remove noise from a picture, and running it in reverse from pure static gives you a picture.
• Reasoning under uncertainty · Part 8 of 33 · 11 min Monte Carlo methods: answering hard questions by rolling dice When the maths is too hard, sample instead. What Monte Carlo is from first principles, why error falls as one over root n, and where it earns its keep in real work.
• Generative models and LLMs · Part 33 of 33 · 8 min Reinforcement learning, and what AlphaGo taught Learning from consequences instead of answers: agent, reward, and the credit assignment problem — plus an honest account of why reinforcement learning is rare in normal data work.
• Reasoning under uncertainty · Part 5 of 33 · 6 min Bayes theorem from first principles Bayes as areas and counts rather than a formula: the medical-test example worked properly, why base rates dominate, and where the idea earns its keep in practice.
• Neural networks · Part 24 of 33 · 7 min What word embeddings really are Why representing words as IDs fails, how counting the company a word keeps turns language into geometry, and what the famous king-minus-man analogy does and does not prove.
• Models that learn from tables · Part 17 of 33 · 7 min Gradient boosting, built one mistake at a time How boosting works from scratch: fit a model, fit the next one to what the first got wrong, then repeat. Why that is not bagging, and what the learning rate really buys.
• Models that learn from tables · Part 14 of 33 · 7 min Overfitting and the bias-variance tradeoff What overfitting really is, what bias and variance each mean in concrete terms, how the tradeoff shows up in model choice, and where the classic curve stops holding.
• Models that learn from tables · Part 18 of 33 · 9 min K-means clustering, and what it is really doing The assign-and-update loop from scratch, what k-means actually minimises, why it only finds round clusters of similar size, and how to choose k without fooling yourself.
• Generative models and LLMs · Part 32 of 33 · 7 min GANs, and why deepfakes work How training two networks against each other produces convincing fake images, why that setup is famously unstable, and what it sensibly changes about trusting a video.
• Models that learn from tables · Part 19 of 33 · 8 min PCA explained from first principles What a principal component actually is, why scaling comes first, how to read a scree plot, and the two things people wrongly expect PCA to do for them.
• Models that learn from tables · Part 15 of 33 · 7 min Logistic regression from first principles Why a straight line cannot predict probabilities, what the log-odds link fixes, how to read a coefficient as an odds ratio, and why the loss is cross-entropy.
• Models that learn from tables · Part 16 of 33 · 7 min Decision trees and random forests, from the split up How a tree decides which question to ask, why a single deep tree memorises its training data, and why averaging hundreds of decorrelated trees fixes it.
• Neural networks · Part 22 of 33 · 7 min Backpropagation explained plainly How a network works out which weight was to blame for a wrong answer, why the chain rule is really credit assignment, and why backprop is bookkeeping rather than magic.
• Reading data honestly · Part 12 of 33 · 6 min Simpson's paradox from first principles Why an average over groups can point the opposite way to every group inside it, what the Berkeley admissions case actually showed, and how to decide which number to act on.
• Neural networks · Part 21 of 33 · 6 min Gradient descent from first principles The cost surface, why the gradient points uphill, what the learning rate really controls, and why local minima matter far less in practice than everyone expects.
• Neural networks · Part 20 of 33 · 7 min What a neural network actually is Layers, weights, biases and activations built up from scratch on the handwritten-digit problem, and why "learning" means nothing more exotic than adjusting a pile of numbers.
• Reasoning under uncertainty · Part 7 of 33 · 8 min P-values: what they say and what they do not What a p-value actually measures, the four things it gets mistaken for, how forking paths produce small p-values from nothing, and what to report instead.
• Reading data honestly · Part 13 of 33 · 5 min Automation and jobs: picking the right unit of analysis Why "will automation take my job" is the wrong question, what a task-level view shows that a job-level view hides, and how to reason about it without hype or doom.
• Reading data honestly · Part 11 of 33 · 8 min Correlation is not causation, but The three things that can produce any correlation, why the famous slogan stops thought instead of starting it, and what actually establishes a causal claim.
• Reasoning under uncertainty · Part 4 of 33 · 7 min The Monty Hall problem, and why switching wins Three doors, one car, a host who knows where it is. Switching wins two thirds of the time — here is the intuition, the enumeration, and the lesson underneath it.