Data Cleaning
The bulk of the job. Making messy data trustworthy before any model touches it.
7 articles, newest first.
- Outliers: diagnose before you delete An outlier is an error, a whale, or a discovery — three different treatments. Robust statistics, sensible detection, and the customers you should not trim away.
- Schema checks: make your pipeline refuse bad data Upstream will change without telling you. Explicit schema and expectation checks at pipeline boundaries turn silent corruption into loud, early failures.
- Pandas cleaning patterns that scale past the toy example Method chaining, assign, vectorized string ops, coercion audits, and validation asserts — a working style for cleaning code you can still read in six months.
- Missing data is information, not an inconvenience Why dropna() is a modelling decision in disguise, the three mechanisms of missingness, and imputation patterns that do not bury the signal.
- Data leakage is why your model looks too good If your validation score seems miraculous, the most likely explanation is that the answer leaked into the features. The common leaks, two worked through end to end, and how to audit for them.
- How to sample data without lying to yourself head(1000) is not a sample. Random, stratified, and entity-level sampling in SQL and pandas — and the bias traps in each.
- The job is mostly cleaning data, and no one warns you Courses spend 5% of the time on the 80% of the work. What data cleaning actually involves, why it is the real skill, and how to get good at it deliberately.