PCA explained from first principles

Picture a cloud of points measured on two variables — say height and weight for a few thousand people. The cloud is not a round blob. It is a stretched ellipse, leaning up and to the right, because tall people tend to be heavier. Most of the spread in that cloud runs along the long axis of the ellipse. Very little of it runs across the short axis. The figure below shows exactly that cloud.

PC1 is the direction of greatest variance in the cloud; PC2 is at right angles to it and picks up what little spread is left.

That observation is the whole of principal component analysis, and it is how Pearson first framed it, in an 1901 paper titled On Lines and Planes of Closest Fit to Systems of Points in Space; Hotelling gave the method its modern form and its name in 1933. The original axes — height, weight — were handed to you by whoever designed the measuring instrument. They are not necessarily the directions in which the data actually varies. PCA looks at the cloud and asks a different question: along which direction is the spread largest? That direction becomes the new first axis. Then it asks the same question again, restricted to directions at right angles to the first, and gets a second axis. And so on, until you have as many new axes as you had original variables.

Nothing is lost in that step. You have simply rotated the coordinate system to line it up with the shape of the data. The gain comes next: because the new axes are ordered by how much spread they capture, the last few usually capture almost none. Drop them, and you have fewer columns with nearly the same information.

What a component actually is

A principal component is a direction, and a direction in a table of numbers is a weighted recipe over the original columns. If your columns are height, weight and resting heart rate, the first component might be 0.62 × height + 0.70 × weight - 0.35 × heart_rate, with the weights scaled so they form a unit vector. Those weights are called the loadings.

Each row of your data then gets a score on that component: run its values through the recipe and you get one number. The scores are the new columns. So PCA gives you two different objects, and confusing them is the most common source of muddle:

The eigenvalue attached to each component is the variance of its scores. That is the number you compare across components, and it is what “explained variance” means. It is a statement about spread, and only about spread.

Scaling is not optional

Variance has units. If height is in centimetres and weight in kilograms, the height column has a variance in cm² and weight in kg². Switch height to millimetres and its variance grows a hundredfold, which means the first component swings round to point almost entirely along height. Nothing about the people changed. Only the ruler did.

So unless every column is genuinely on the same scale and you want the larger-range columns to dominate, standardise first: subtract the mean, divide by the standard deviation. Centring matters too — PCA measures spread around the origin, so an uncentred dataset produces a first component that mostly just points at the mean.

The run below takes five columns from the public dataset: the monthly price of a subscription, and how many times that customer viewed, logged in, exported and purchased in their first thirty days.

import pandas as pd
from sklearn.decomposition import PCA
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

subs = pd.read_csv("https://dataacademy.ai/data/subscriptions.csv",
                   parse_dates=["signed_up_at", "cancelled_at"])
events = pd.read_csv("https://dataacademy.ai/data/events.csv",
                     parse_dates=["occurred_at"])

ev = events.merge(subs[["subscription_id", "customer_id", "signed_up_at"]],
                  on="customer_id")
month1 = ev[(ev.occurred_at - ev.signed_up_at).dt.days.between(0, 29)]
counts = month1.pivot_table(index="subscription_id", columns="event_type",
                            values="event_id", aggfunc="count", fill_value=0)

d = subs.set_index("subscription_id").join(counts)
d[counts.columns] = d[counts.columns].fillna(0)
X = d[["monthly_price", "view", "login", "export", "purchase"]]
X_train, X_test = train_test_split(X, test_size=0.3, random_state=0)

pca = make_pipeline(StandardScaler(), PCA())
scores = pca.fit_transform(X_train)
print(pca[-1].explained_variance_ratio_.round(3))
print(pca[-1].explained_variance_ratio_.cumsum().round(3))

loadings = pd.DataFrame(pca[-1].components_.T, index=X.columns,
                        columns=[f"PC{i}" for i in range(1, 6)])
print(loadings.round(2))
print(X_train.corr().round(2))

[0.616 0.199 0.113 0.056 0.016]
[0.616 0.815 0.928 0.984 1.   ]
                PC1   PC2   PC3   PC4   PC5
monthly_price  0.05  1.00  0.01  0.03 -0.01
view           0.54 -0.02 -0.19 -0.46 -0.68
login          0.54 -0.01 -0.14 -0.40  0.73
export         0.49 -0.05 -0.40  0.77 -0.02
purchase       0.42 -0.04  0.89  0.19 -0.04
               monthly_price  view  login  export  purchase
monthly_price           1.00  0.07   0.08    0.04      0.04
view                    0.07  1.00   0.92    0.75      0.58
login                   0.08  0.92   1.00    0.76      0.61
export                  0.04  0.75   0.76    1.00      0.48
purchase                0.04  0.58   0.61    0.48      1.00

Read the loadings before the ratios. PC1 gives the four activity counts weights between 0.42 and 0.54 and the price almost nothing, so it is one axis of “how much did this customer do”, and it holds 61.6% of the variance because those four columns correlate between 0.48 and 0.92 with each other. PC2 is the price and nothing else, at a loading of 1.00 — the price correlates at most 0.08 with any count, so it gets a whole component to itself and takes 19.9% with it. Five columns went in, and the first two axes carry 81.5% of the spread.

Fit the scaler and the PCA on the training data only, then apply both to the test data. Fitting on everything is a textbook case of leakage — the test rows have quietly influenced the axes your model was trained in.

Reading a scree plot

Plot the explained variance of each component against its rank. The curve starts high and falls. What you are looking for is the elbow: the point where the fall flattens into a long, boring tail. Components before the elbow carry structure. Components in the tail carry mostly noise.

This is a judgement, not a test. Three habits keep it honest:

• Read the cumulative curve, not just the individual bars. “The first four components hold 91% of the variance” is a more useful sentence than “there is an elbow at four”.
• Decide the threshold before you look, if the components feed a downstream model. Otherwise you will pick whichever cut makes the final score look best, which is just a slow way of overfitting.
• Expect no elbow sometimes. A gently sloping curve with no break means the variance is spread evenly across directions. That is a real answer: this data does not compress.

StatQuest: Principal Component Analysis (PCA), Step-by-Step by StatQuest with Josh Starmer walks through the geometry one rotation at a time.

What PCA is for, and what it is not

It is genuinely good at four things. Compressing wide, correlated data before a model that struggles with collinearity or dimension count. Killing multicollinearity in linear models, since the components are uncorrelated by construction. Making a two-dimensional plot of high-dimensional data that is at least honest about how much it left out. Speeding up training when you have hundreds of nearly redundant sensor channels.

Two expectations are misplaced, and both are common.

PCA is not feature selection. It does not tell you which of your columns matter. Every component is built from all of them, so you still have to collect, clean and serve every original variable in production. If your aim is to drop columns, use a method that drops columns.

Components are rarely interpretable. Occasionally the first component has an obvious reading — “overall size”, “general affluence” — and it is fine to say so. But a component is chosen to maximise variance, not to mean anything. A recipe that mixes a price, a click rate and a tenure in months is a direction in space, not a concept. Treating it as one, and then explaining it to a stakeholder, is how a plausible story gets attached to an arbitrary axis.

There is also a limit built into the method: PCA only finds directions that are straight lines. If your data lies on a curved surface — a spiral, a shell — the largest-variance straight direction can cut clean through the structure and tell you nothing about it.

So the test to apply before fitting one: name what the next step gains. Fewer inputs for a model that handles collinearity badly, two axes for a plot, a decorrelated basis for a distance metric — all answers. “To reduce dimensionality” is not an answer, and a run that cannot pass that test is better not started.