Glossary

98 terms that turn up again and again across the articles here, each one defined in a sentence or two of plain English. Where a term has an article of its own, the term links to it — or search the site if you already know the word you want.

Read it end to end and you have the working vocabulary of the field. Read one line and you can follow the meeting.

Statistics and probability 25 terms

What a test actually claims, and the extra meaning it gets read as claiming.

p-value: The chance of seeing a result at least as extreme as yours if there were no real effect. It is not the chance that your finding is wrong, and not the size of the effect.
Null hypothesis: The dull explanation a test tries to rule out, usually that the difference is exactly zero. Failing to rule it out is not evidence that it is true.
Confidence interval: A range built so that, over many repeated experiments, 95 per cent of such ranges hold the true value. Any single interval either holds it or does not.
Bootstrap: Resampling your own data with replacement, thousands of times, to see how much an estimate moves. It needs no formula for the sampling distribution.
Central limit theorem: Averages of many independent values follow a bell curve, whatever shape the raw values have. It applies to the average, not to the raw data.
Bayes' theorem: The rule for updating a belief with evidence: what you believed before, multiplied by how strongly the evidence favours one answer over another.
Prior: What you believed before the new data arrived, written as a probability.
Posterior: What you believe after the new data, once the prior has been updated by the evidence.
Base rate: How common something is before you look at any specific case. Ignore it and rare things start to look far more likely than they are.
Simpson's paradox: A pattern that holds inside every group can reverse once the groups are pooled, because the groups differ in size and in mix.
Confounder: Something that moves both the supposed cause and the effect, so the two move together without one causing the other.
Causal inference: Estimating what an action actually caused, rather than what it happened alongside. It needs a design or an assumption about the comparison, never just a bigger dataset.
Randomisation: Assigning units to treatment and control by chance, so the groups differ only by luck. It is what makes the comparison in an experiment fair.
Statistical power: The chance a test finds a real effect of a given size. When power is low, most true effects are missed and the ones that do reach significance are exaggerated.
Minimum detectable effect: The smallest effect an experiment of a given size can reliably find. You choose it before the test, and it sets the sample size.
Interference (spillover): When one unit being treated changes another unit’s outcome — shared friends, shared drivers, shared stock. It breaks the independence the standard test assumes.
Monte Carlo simulation: Answering a probability question by simulating the process many times and counting what happens, instead of solving it on paper.
Markov chain: A process where the next state depends only on the current state, not on the path that led there.
Robust statistic: A summary that a few extreme values cannot drag around — the median rather than the mean, for example.
Entropy: The average surprise in a distribution, measured in bits. High entropy means the outcome is hard to guess.
Cross-entropy: The average surprise you suffer when you describe what really happened using your predicted probabilities. It is smallest when the predictions match reality.
Log loss: Cross-entropy used as a score for a classifier. It punishes confident wrong answers far harder than hesitant ones.
Perplexity: Cross-entropy turned back into a count: roughly how many equally likely options the model is choosing between at each step. Lower is better.
Mutual information: How much knowing one variable cuts your uncertainty about another. Unlike correlation, it catches relationships that are not straight lines.
KL divergence: How many extra bits you pay for describing one distribution using another. It is not symmetric, so it is not a distance.

↑ All sections

Evaluation 20 terms

How a model is scored, and why a good score is so often the wrong one.

Precision: Of the cases the model flagged, the share that really were positive.
Recall: Of the cases that really were positive, the share the model found.
F1: The harmonic mean of precision and recall, in one number. It treats the two as equally costly, which almost no real problem does.
ROC AUC: The chance that a random positive case scores above a random negative one. It measures ranking only — it says nothing about whether the probabilities are right, and it looks generous when positives are rare.
Precision–recall curve: Precision plotted against recall across every threshold. On rare positives it shows the trade-off far more honestly than an ROC curve.
Calibration: Whether predicted probabilities match what happens: of the cases scored 0.7, about 70 per cent should turn out positive. A model can rank cases well and still be badly calibrated.
Brier score: The mean squared difference between predicted probabilities and outcomes. It rewards being right and being honest about how sure you are.
Cross-validation: Splitting the data into folds, training on some and scoring on the rest, then rotating. It estimates how the model does on rows it did not learn from.
Data leakage: Information from the answer, or from the future, reaching the model during training. It is the usual reason a brilliant test score collapses in production.
Overfitting: The model has learned the noise in the training sample, so it scores well there and worse on anything else. A complex model is not automatically overfitting; complexity only matters relative to how much data you have.
Bias: Error from a model too rigid to follow the real pattern. It is wrong in the same direction no matter which sample you train on. Not the same as the bias term in a network, nor as social bias.
Variance: Error from a model so flexible that a different training sample would give a noticeably different model.
Regularisation: Deliberately limiting how closely a model can fit — penalising large weights, pruning trees, stopping training early. It trades a little bias for a lot less variance.
Baseline: The simplest sensible prediction: last week’s value, the majority class, the group average. Every model has to beat it, and plenty do not.
Class imbalance: When one outcome is rare — fraud, churn, disease. Accuracy stops meaning anything, because always saying "no" scores well.
Threshold: The score above which a prediction counts as positive. Where you put it is a decision about the cost of false alarms against missed cases, not a default of 0.5.
MAE and RMSE: Two ways to summarise regression error. MAE is the average size of the miss; RMSE squares first, so a few large misses dominate it.
Feature importance: A ranking of which inputs a model leans on. It describes the model, not the world, and it is not evidence of cause.
SHAP value: The contribution one feature made to one prediction, measured against the average prediction. It explains that single prediction, not the model as a whole.
Partial dependence: The average prediction as one feature is moved across its range and the others are left alone. It assumes that feature can move independently of the rest, which is often false.

↑ All sections

Machine learning 22 terms

The models that do most of the work on tables of rows and columns, and the steps that feed them.

Feature engineering: Turning raw records into the columns a model can use — counts, ratios, time since the last event. It is where most of the gain on tabular problems comes from.
One-hot encoding: Replacing a category column with one 0/1 column per category. Safe and simple, and unmanageable once there are thousands of categories.
Target encoding: Replacing a category with the average outcome for that category. Strong on high-cardinality columns, and the easiest way to leak the answer if the average is computed over rows the model also trains on.
Gradient descent: Fitting by repeatedly nudging the parameters in the direction that reduces the loss fastest.
Learning rate: The size of each nudge. Too large and training bounces around or diverges; too small and it crawls.
Loss function: The single number training tries to make small. It defines what "good" means for the model, so picking it is a modelling decision, not a technicality.
Logistic regression: A linear model whose output is squeezed into a probability. Fast, well calibrated out of the box, readable, and a hard baseline to beat.
Decision tree: A chain of yes/no splits on single columns, ending in a prediction. Easy to read, and unstable on its own — small data changes redraw it.
Random forest: Many trees, each grown on a random sample of rows and columns, then averaged. The randomness makes their mistakes cancel out.
Bagging: Training the same kind of model on many bootstrap samples and averaging the results. It cuts variance; it does not fix bias.
Boosting: Adding models one at a time, each one fitting what the models before it got wrong.
Gradient boosting: Boosting where each new tree is fitted to the gradient of the loss. It is the default first choice for tabular data, and it needs early stopping to stay honest.
k-means: Clustering that splits points into k groups around k centres. You must choose k, and it assumes round, similarly sized clusters.
DBSCAN: Clustering by density: crowded regions become clusters and isolated points are left as noise. It finds odd shapes and does not need a cluster count.
Silhouette score: How much closer a point sits to its own cluster than to the next nearest, averaged over all points. A high score means the clusters are separated, not that they mean anything.
PCA: Rewriting correlated columns as a smaller set of uncorrelated ones that keep as much of the variation as possible.
Principal component: One of those new axes: a weighted mix of the original columns. The first holds the most variation. A component is a direction of variance, not a hidden concept.
Embedding: A list of numbers that places an item — a word, a product, a user — in space, so that similar items sit close together.
Cosine similarity: The angle between two vectors, reported from −1 to 1. It compares direction and ignores length, which is why it suits embeddings.
TF-IDF: Scoring a word by how often it appears in one document and how rare it is across all of them. Still competitive for search and for text features.
Multi-armed bandit: An experiment that keeps shifting traffic towards the options doing well while still sampling the rest.
Reinforcement learning: Learning by acting, seeing a reward, and adjusting. It needs an environment you can act in cheaply and often, which most business problems do not have.

↑ All sections

Deep learning and LLMs 18 terms

The parts a neural network is built from, and the vocabulary that has grown around language models.

Neural network: Layers of weighted sums with a non-linear step between them, fitted by gradient descent. It is not a model of the brain.
Weight: One number inside a network that multiplies an input. Training a network means choosing its weights.
Bias term: A constant added after a weighted sum, letting a unit shift its output up or down. It has nothing to do with statistical or social bias.
Activation function: The non-linear step after each weighted sum, such as ReLU. Without it, stacking layers would collapse into a single linear model.
Backpropagation: Applying the chain rule backwards through the network to work out each weight’s share of the error. It is efficient bookkeeping for gradients, not a learning rule of its own.
Convolution: Sliding a small patch of weights across an image and recording the response at each position, so the same detector works anywhere in the frame.
Kernel (filter): The small grid of weights that gets slid across the image. In machine learning "kernel" also means a similarity function in methods such as SVMs — different idea, same word.
Transformer: An architecture built from stacked attention layers. It reads a whole sequence at once instead of stepping through it, which is what made large models trainable.
Attention: A step where every position pulls information from the other positions, weighted by a relevance score it works out itself.
Token: The chunk of text a language model actually reads. Usually a word piece, sometimes punctuation — rarely a whole word.
Next-token prediction: The training task behind language models: given the text so far, predict the next token. Everything else the model appears to do comes from doing this well.
Pretraining: The first and longest training stage: next-token prediction over an enormous body of text.
Fine-tuning: Carrying on training a pretrained model on a smaller, targeted set of examples to adapt what it does.
RLHF: Reinforcement learning from human feedback: people rank answers, and the model is tuned towards the preferred ones. It shapes tone and helpfulness, not knowledge.
Hallucination: A fluent, confident statement that is false. It is not a lookup failure — it is the ordinary behaviour of a model that predicts plausible text and has nothing to check it against.
Diffusion model: An image generator trained to strip noise away one step at a time. Generation starts from pure noise and runs that process in reverse.
GAN: Two networks trained against each other: one produces fakes, the other tries to spot them, and both improve.
Prompt: The text you give a language model. Prompting works because the wording moves the model into the kind of text where the answer you want is likely to follow.

↑ All sections

Data and SQL 8 terms

Getting the rows into a shape worth modelling, and keeping them that way.

Window function: A SQL function that looks across a set of rows related to the current row — running totals, ranks, previous values — without collapsing them into one row, as GROUP BY would.
CTE: A common table expression: a named subquery declared with WITH, so a long query reads as named steps. It is naming, not caching.
Incremental model: A table rebuilt only for new or changed rows instead of from scratch. Cheap to run, and it quietly drifts from a full rebuild when late data arrive.
Cohort: A group defined by when it started, then followed over time. Cohorts stop a flood of new users from hiding the fact that older ones are leaving.
Schema validation: Checking that arriving data have the columns, types and ranges you expect, and failing loudly when they do not.
Imputation: Filling a missing value with an estimate. Before you fill anything, record that it was missing — why a value is absent is often the useful signal.
Sampling: Taking a subset that stands in for the whole. The rule that picks rows must have nothing to do with what you are measuring, or the sample is biased.
Seasonality: A pattern that repeats on the calendar — hour, weekday, holiday, season. Ignore it and you read the calendar as a trend.

↑ All sections

Working practice 5 terms

The habits that decide whether the work survives contact with production.

Unit of analysis: What one row stands for: a user, a session, an order. Choose it wrongly and counts inflate, p-values shrink, and the test lies.
Drift: Live data, or the relationship the model learned, moving away from the training conditions. Scores decay without a line of code changing.
Feature store: One place where feature definitions live, so training and live serving compute the same number the same way.
Backtesting: Testing a forecast against the past as if it were running live, using only what was known at each point in time.
Reproducibility: Someone else, or you in six months, can rerun the work and get the same numbers — same data, same code, same versions, same random seeds.

↑ All sections