A customer table has a country column with 47 distinct values. Six of them
are spellings of the United Kingdom. Two are empty strings that mean different
things, and nobody left is sure which is which. No course has a lesson on that
column. Courses hand over clean data, a defined target, and one job: pick an
algorithm and tune it. Real work spends most of its hours before any model
exists, getting the data into a state worth trusting. That isn’t a detour from
the work — it is the work, and the sooner it’s treated as the core skill,
the faster everything else improves.
What “cleaning” actually means
Finding out what the columns really mean. The documentation is missing,
wrong, or describes the schema from two migrations ago. status = 3 means
“cancelled” except for rows before 2023, when it meant “pending review”.
Nobody knows why amount_v2 exists, but the one report everyone trusts uses
it. This work is archaeology plus diplomacy: reading pipeline code, asking the
longest-tenured engineer, and writing down what you learn so the next person
doesn’t repeat the dig.
Handling missing values honestly. Not just dropna() — understanding
why values are missing, because the mechanism decides the fix. Values
missing at random are a nuisance; values missing because of what they are
(high earners skipping the income field, a sensor dying exactly when it
overheats) bias every downstream conclusion, and no imputation formula
repairs that without you noticing it first.
Catching the silent errors. Duplicates from a retried batch job. Amounts in cents for one source and euros for another. Dates in three formats, one of them ambiguous. A sensor that reports zero when it’s actually broken, which is different from reporting nothing. Test accounts generating a third of the “engagement”. None of these throw exceptions; all of them survive into models and dashboards if no one goes looking.
Reconciling sources that disagree. The billing system and the analytics events differ on revenue by 4%. Which is truth? (Answer: neither, for different reasons, and finding the two reasons takes a week and is worth it.)
Why this is the real skill
Skip this work and your beautiful model learns garbage and reports it confidently — most real-world model failures are data failures wearing a model’s clothes. But the deeper reason is leverage: modelling improvements are capped (the difference between a good and great algorithm on the same features is a few points), while data improvements compound — a corrected label definition, a de-duplicated source, one honest new feature routinely beat any amount of hyperparameter tuning. The practitioners who develop a reputation for being trustworthy are, almost always, the ones who are simply better at this unglamorous layer.
Getting good at it deliberately
It’s a learnable craft, not a tax:
- Profile before using. For every new table: row counts over time, null rates, value distributions, min/max dates, top categories. Fifteen minutes, every time. Most surprises live here.
- Turn discoveries into checks. Every bug you find becomes an assert or a schema rule at the pipeline boundary, so it can never sneak back in quietly.
- Keep raw data immutable and cleaning in code. No hand-edited files. If every transformation is a script, every number is reproducible, and the cleaning itself becomes reviewable work instead of private heroics.
- Write down the semantics. A short data dictionary with the landmines (“status=3 changed meaning in 2023”) is the highest-value document most teams don’t have.
Courses can’t teach this well, because every dataset’s mess is its own. The habits transfer, though, and there’s a way to start them this week: take the table your team argues about most and profile it. Row counts by month, null rate per column, value counts on every categorical, min and max on every date. Fifteen minutes. Then write whatever surprised you into a file next to the pipeline code and send the link round. That file is the data dictionary the team doesn’t have yet.