Reproducibility: the checklist nobody follows until it burns them

“Where did the 0.83 come from?” The slide is four months old, the person asking has it open, and rerunning the notebook now returns 0.79. Not wildly different — different enough that neither number can be defended, because nobody can explain the gap. The audience for reproducibility isn’t peer reviewers. It’s you, six months from now, under pressure.

The gap always comes from one of four moving parts. Pin all four.

1. The code that ran

“The notebook” is not the code that ran — notebooks execute in human order, not top-to-bottom, and half the state comes from cells since deleted. Minimum standard: results worth keeping come from a script or notebook executed top-to-bottom in one shot (jupyter nbconvert --execute, papermill, or plain python train.py), committed to git, with the commit hash stored next to the outputs.

import subprocess
commit = subprocess.check_output(
    ["git", "rev-parse", "--short", "HEAD"]).decode().strip()
results["code_version"] = commit

An uncommitted-changes check alongside it turns “which version produced this?” from archaeology into a lookup.

2. The environment it ran in

pip install pandas today and four months ago yield different pandas — with changed defaults, fixed bugs your results depended on, and different transitive dependencies. Lock the whole tree, not the shortlist you remember typing: requirements.txt with exact versions via pip freeze at minimum; a lockfile (uv lock, poetry.lock, conda-lock) as the grown-up option; a container image when OS-level libraries matter. The test is brutal and simple: can a colleague on a fresh machine produce your number with the instructions in the README? If the README says “you might also need to…”, it fails.

3. The data it saw

The silent killer. Code and environment pinned, number still different — because the query ran against a live table. Yesterday’s late-arriving events, a backfill, a deleted-users purge: the training data no longer exists unless you kept it. Options in ascending robustness: store the query with its exact time bounds (WHERE created_at < '2026-07-01' — never NOW() or an open upper bound); snapshot the extracted dataset to immutable storage and record a checksum; use data versioning (DVC, lakeFS, warehouse time-travel) when datasets are too large to copy around.

import hashlib
digest = hashlib.sha256(
    pd.util.hash_pandas_object(df, index=True).values).hexdigest()[:16]
print(f"training data fingerprint: {digest}")

Eight lines, and “did the data change?” becomes answerable in one comparison instead of a week of diffing.

4. The randomness inside

Train/test splits, weight initialisation, subsampling in tree ensembles — all seeded, or every rerun is a new experiment. Set seeds explicitly and pass them everywhere (random_state=SEED in every sklearn call; numpy, python, and the framework’s own RNG for deep learning — plus the deterministic-ops flag if you need bit-exactness on GPU).

The subtler point: vary the seed on purpose, once. Run the pipeline with five seeds and look at the spread. If your celebrated 0.5-point improvement is smaller than the seed-to-seed variation, you’ve learned it’s noise — better to find out yourself than have a skeptic find out for you.

The one-command test

The four pins converge on a single standard: one command, from a fresh clone, regenerates the headline result — make reproduce, a run.sh, anything. What it must not be is a sequence of notebook cells executed in an order that lives in your memory. Try it on the current project this week: clean checkout, one command, on a machine that isn’t yours. Whatever breaks in the first five minutes is the part of the checklist still outstanding.