Knowledge Base / Glossary

Data leakage

An accidental flow of information across an evaluation boundary that makes held-out performance look more trustworthy than it is.

Data leakage occurs when information that should be unavailable during model training influences the training process or the choices used to evaluate it. In a train-test evaluation, the test set is meant to imitate genuinely unseen observations. If test measurements help choose imputation values, feature scales, selected features, or model settings, that evaluation boundary has leaked.

Leakage does not require exposing test labels. Test feature values alone can leak when they influence learned preprocessing.

The leaking workflow fits preprocessing on all rows before splitting, while the protected workflow splits first and fits imputation, scaling, principal component analysis, and the model using training rows only. — Top: fitting learned preprocessing on all rows lets the future test rows affect the values and directions used during training. Bottom: split independent units first, fit every data-adaptive step on training rows, and only transform the untouched test rows with the already fitted pipeline. Fixed parsing and declared unit conversions are not shown because they do not learn from the observed sample.

A small example

Suppose the observed training widths are \(2.9\) and \(3.1\) millimeters, with one training width missing. Their median is \(3.0\). The future test widths are \(3.7\) and \(3.9\).

If the analyst combines both subsets before imputation, the median of all four observed values is \(3.4\). The missing training cell is then filled with \(3.4\), so the test measurements have changed the training table. That is leakage even though no test label was used. The numbers are intentionally small toy values, not an empirical estimate of the size of leakage in a real study.

The protected sequence

Define the independent unit and split raw units into training and test sets.
Fit data-adaptive preparation, such as imputation and standard scaling, on training rows only.
Fit feature selection, principal component analysis, and the prediction model on the training data only.
Apply the already fitted transformations to the test rows.
Evaluate once on those held-out rows using a metric chosen for the task.

Cross-validation repeats this boundary. Each fold needs its own preprocessing fit, which is why software pipelines are useful.

Not every operation before a split leaks. A fixed file parser, documented unit conversion, or validity rule declared without inspecting outcomes may be applied consistently. The risky steps are those that learn values, directions, thresholds, or choices from the observed data.

Descriptive PCA is a different goal

Using every row for a purely descriptive PCA is not automatically leakage. If the stated question is “what patterns appear in these exact observations?”, then the full dataset is the object being described and there is no held-out performance claim.

The problem begins when an analyst reuses globally imputed, globally scaled, or globally fitted PCA features and then claims that a later test split measures performance on unseen data. For that claim, the split must happen before any data-adaptive preprocessing is fitted.

What preventing leakage does not establish

A clean split does not repair biased sampling, mislabeled rows, dependence between related samples, or distribution shift.
Leakage-free accuracy is not automatically biological understanding or a causal result.
Randomly splitting rows is not enough when several rows come from the same organism, patient, plot, plate, or time series. Related rows may need to stay in the same split.
Repeatedly checking the test result and changing the analysis in response can turn the test set into another training signal.