How Tokenization Powers Large Language Models

01 The Bridge Between Text and Numbers

Before a transformer model can process a single word, that word must become a number. This conversion — tokenization — is the foundational interface between human language and machine learning. When you type a prompt into ChatGPT or Claude, the model does not see letters or words. It sees a sequence of integer IDs, each pointing to a row in an embedding matrix. The tokenizer is the system that performs this translation, and its design choices ripple through every aspect of model behavior.

Tokenization is not a neutral preprocessing step. The choice of token vocabulary directly affects how many tokens a given text produces, which in turn determines inference cost, context window utilization, and even the model's ability to spell or count. A tokenizer that splits "unbelievable" into "un" + "believ" + "able" gives the model compositional clues about meaning, but one that keeps it as a single token saves context length. Every tokenizer makes these trade-offs millions of times across its vocabulary.

02 Byte Pair Encoding: The Algorithm That Built GPT

The dominant tokenization method for modern LLMs is Byte Pair Encoding, or BPE. Originally a data compression technique proposed by Philip Gage in 1994, BPE was adapted for neural language models by Sennrich, Haddow, and Birch in 2015. The algorithm starts with a vocabulary of individual bytes or characters, then iteratively merges the most frequent adjacent pair into a new token. After thousands of merge operations, the vocabulary contains common subwords, whole words, and even multi-word phrases.

OpenAI's GPT models use a BPE variant called tiktoken. GPT-2 had a vocabulary of roughly 50,257 tokens, while GPT-4 uses cl100k_base with approximately 100,000 tokens. The larger vocabulary means fewer tokens per text, but a bigger embedding table. Andrej Karpathy's walkthrough of building a GPT tokenizer from scratch reveals just how many edge cases lurk in this seemingly simple process — from handling Unicode characters to dealing with the fact that a single emoji can require four or more tokens.

03 The Token Tax: Why Encoding Matters for Cost

Every token costs money. API pricing for models like GPT-4 is quoted per million tokens, and the tokenizer determines how many tokens your text consumes. English text is relatively efficient — roughly one token per four characters. But other languages fare far worse. A sentence in Japanese or Arabic may require two to three times more tokens than its English translation, because BPE vocabularies are disproportionately built from English training data.

This creates a hidden language tax. Users working in non-Latin scripts pay more for the same semantic content, both in API costs and in reduced effective context windows. OpenAI's o200k_base tokenizer, introduced with GPT-4o, explicitly addressed this by expanding the vocabulary to 200,000 tokens and including more non-English subword merges. The result was a measurable reduction in token counts for many languages, though English remains the most efficiently encoded.

04 Why Models Struggle With Letters and Counting

If you have ever asked an LLM how many letters are in a word and received a wrong answer, tokenization is the culprit. A model that sees "strawberry" as a single token has no direct access to the individual characters s-t-r-a-w-b-e-r-r-y. It would need to have learned, during training, that this particular token contains three r's. For common words this memorization sometimes works, but for rare words or novel combinations, the model genuinely cannot count characters it has never seen decomposed.

This same limitation explains why LLMs are notoriously bad at spelling corrections, character-level manipulations, and certain types of code generation involving individual characters. The token boundary acts as an information bottleneck. Some newer tokenizers attempt to preserve more character-level structure, but the fundamental tension between compression and granularity remains. A tokenizer that represents every character separately would eliminate spelling errors but would make every prompt four to five times longer, blowing through context limits.

05 Special Tokens and the Architecture of Control

Beyond encoding text, tokenizers define a set of special tokens that serve structural purposes. The end-of-text token signals the model to stop generating. Padding tokens fill out batch sequences to equal length. Chat-format tokens like those used in ChatGPT mark the boundaries between system messages, user turns, and assistant responses. These special tokens are invisible to users but essential to how the model structures its output.

The design of these control tokens has real security implications. Prompt injection attacks often exploit the boundary between user content and system instructions — a boundary defined by special tokens. If an attacker can craft input that mimics or disrupts these tokens, they may be able to override system-level constraints. Tokenizer robustness is therefore not just a performance concern but a safety one.

06 Multilingual Tokenization and the Push for Fairness

The disparity in token efficiency across languages has pushed researchers and companies to develop more equitable tokenizers. Meta's Llama models use the SentencePiece library, which was designed with multilingual text in mind. Google's Gemini employs a SentencePiece-based tokenizer that reportedly handles over 100 languages with more balanced efficiency. The trade-off is always the same: a more multilingual vocabulary means more tokens in the embedding table, which increases model size and training cost.

The practical consequence is that tokenization decisions shape who can afford to use AI. A developer building a chatbot in Swahili pays more per conversation than one building the same chatbot in English. As AI infrastructure becomes a public utility, these encoding disparities take on a civic dimension. The tokenizer is not just a technical artifact — it is a policy decision about whose language is cheap to process and whose is expensive.

07 The Future: Beyond Subword Tokenization

Several research directions aim to move beyond traditional BPE. Byte-level models that process raw UTF-8 bytes directly eliminate the tokenization step entirely, though they trade vocabulary efficiency for sequence length. MegaByte, a architecture proposed by Microsoft Research, uses a hierarchical structure to make byte-level processing tractable. Other approaches explore dynamic tokenization, where the tokenizer adapts based on context, or token-free models that operate directly on character sequences with efficient attention mechanisms.

Despite these alternatives, BPE-based tokenization remains dominant in production systems. The infrastructure investment in existing tokenizers — from tiktoken's optimized Rust implementation to the embedding tables baked into trained model weights — creates significant switching costs. For the foreseeable future, the tokenizer will remain the invisible gateway through which all human text must pass before a language model can begin to think.