Tommy TangData Scientist & Applied Machine Learning Developer

How I think about feature leakage

· 1 min read

A short checklist I run through to catch data leakage before it inflates my validation scores.

  • machine learning
  • evaluation
  • best practices

Feature leakage is one of the first things I investigate when a validation score looks unexpectedly strong. I use this checklist before trusting the result:

  1. Split first, transform second. Fit scalers, encoders, and imputers on the training split only, then apply to validation/test.
  2. Watch time. For anything temporal, split by time, not randomly. No peeking at the future.
  3. Group leakage. If rows share an entity (a patient, a user), keep all of an entity’s rows in the same split.
  4. Target-derived features. Be suspicious of any feature computed using the target (target encoding, aggregates over labels) without proper folds.
  5. Suspiciously high scores. If a metric looks too good, assume leakage until proven otherwise.

My default safeguard is to keep preprocessing inside the training pipeline so each scaler, encoder, and imputer only sees the training portion of a fold.