The Anatomy of BPE: Why Python Wastes 46% of Tokens

How BPE Tokenization Works and What It Means for Language Design Who this is for. If you want to understand how ChatGPT "sees" your code and why the same program costs different amounts in different languages — read on. All terms explained in footnotes and the glossary at the end. In the previous article, we established that inference cost grows quadratically with token count. The natural question: can we reduce token count without losing semantics? To answer that, we need to understand how LLMs see code. Not as text — as a sequence of tokens. And between how a programmer sees def factorial(n): and how GPT-4 sees it, there's a chasm. How BPE Works BPE (Byte Pair Encoding)1 is the algorithm that converts text into sequences of integers (tokens). It underlies all modern LLMs: GPT-4 uses the cl100k_base2 vocabulary, Claude uses a modified BPE, and Llama uses SentencePiece3 BPE. The algorithm is simple: Start with an alphabet of individual bytes (256 characters). Find the most frequent pai