Every data scientist knows the feeling: a model comes back with 99% accuracy, and instead of celebrating, we get suspicious. Nine times out of ten, the cause is leakage: some piece of information in the training data that would not actually be available at prediction time.
We wanted two real numbers. First, how dramatic is that accuracy jump on real data, not a toy example? Second, if we describe the pipeline to an AI model before anyone runs it, with no hint that anything is wrong, does it catch the leak, and does the answer change depending on which model we ask?
So we took four real data projects, planted a different, well-known leakage pattern into each one, trained a real model with and without the leak, and recorded the actual numbers.
Then, separately, we described each pipeline in plain terms and asked four Claude models, from fastest to strongest, what they would flag before trusting the result.
Source: KDnuggets — Read the full article on the original site ↗