When a column is removed from a source database or dataset, the next pipeline run usually fails, making the problem easy to detect. But when a field keeps the same name and data type but its meaning changes (known as semantic drift), the pipeline can continue running normally, allowing the problem to go unnoticed.
This happens because a schema describes the structure of data, not what the values actually represent. It tells us which fields exist, their data types, and whether they can contain null values, but not whether a timestamp uses UTC, a durability score should range from 10 to 100, or an empty string means something different from null.
As a result, most pipeline checks are designed to catch obvious errors, such as missing columns or incorrect data types, but often fail to detect less obvious problems where the data looks correct but its meaning or values have changed.
I wanted to see how far this limitation reaches in practice, so I built a small lab where I introduced nine types of pipeline failures and tested four detectors against each one. This article looks at what each detector caught and what happened when changes went unnoticed and corrupted the data.
Source: freeCodeCamp News — Read the full article on the original site ↗