AI Tokens and Tokenization: How Language Models Read Text
Tokenization converts text into identifiers from a model-specific vocabulary. Tokens may represent words, word fragments, punctuation, spaces, characters or bytes, so token counts do not map consistently to word counts.
Timeline
- 2018-11-01: The SentencePiece paper presented language-independent subword tokenization trained directly from raw sentences.
- 2025-04-01: NAACL research examined how subword boundaries affect character-level reasoning by language models.
- 2026-10-11: Current tokenizer implementations, research and API concepts were reviewed for this guide.
A token is a discrete unit that a language model maps to a numerical identifier. Before generation, a tokenizer transforms input text into a sequence of those identifiers; the model then predicts a distribution for the next token and repeats the process. A token is not necessarily a word. Depending on the vocabulary, it can represent a common whole word, part of a word, punctuation, a leading space, a character, a byte sequence or a special control marker. [1][2][3]
Subword tokenization balances two extremes. A word-only vocabulary can become enormous and still needs an unknown symbol for unseen words. Character- or byte-level systems avoid many unknowns but create longer sequences. Methods such as byte-pair encoding, WordPiece and unigram models keep frequent patterns as larger units while breaking rarer strings into smaller pieces. The exact segmentation is learned or configured for a particular tokenizer, so the same sentence can produce different token sequences in different models. [2][3][4]
Spaces and normalization matter. OpenAI’s documentation shows that a common word with a preceding space can be one token while “ tokenization” can split into multiple pieces. SentencePiece instead treats raw Unicode text and represents whitespace with a visible meta-symbol before reversible detokenization. Other systems may normalize character forms, merge byte sequences or apply a pre-tokenizer. Consequently, changing capitalization, punctuation, accents or whitespace can change the count even when a human considers the wording nearly identical. [1][2][3]
English rules of thumb are estimates, not conversion formulas. OpenAI offers an approximate relationship of four characters or three-quarters of an English word per token for its models, but it tells users to test the actual string. Languages, scripts, code, emoji, numbers and unusual names can behave very differently. A tokenizer trained on one data mixture may represent common English fragments efficiently while splitting another language into more pieces, which can affect usable context and processing cost. [1][2][4]
Token counts matter because a model has a maximum context measured in tokens. The input, conversation history, retrieved documents, tool results and generated answer may all consume that budget. Reaching the limit can require truncation or summarization, and the placement of relevant information can affect performance. Some services also meter input, output or cached usage by token. Users should consult the current model documentation and tokenizer rather than multiplying a word count by a universal constant. [1]
Token boundaries can influence reasoning about text itself. Research comparing model tokenizers shows that a visually simple word may be divided into fragments that do not align with individual letters. A next-token model can generate fluent language without directly receiving a separate symbol for every character, which helps explain why spelling counts or character manipulations can be unexpectedly difficult. This is one contributor, not proof that every such error is caused only by tokenization. [4]
Special tokens carry structural meaning. Implementations may reserve identifiers for the beginning or end of a sequence, padding, unknown material, roles or other controls. These are part of the model’s formatting contract and may not correspond to visible user text. A vocabulary and its mapping must match the model checkpoint; substituting another tokenizer can feed different identifiers and corrupt behavior even if both tools use an algorithm with the same general name. [2][3]
To estimate usage, select the exact target model and run the complete text through its supported tokenizer, including system instructions and expected output allowance when relevant. Recheck after changing models because vocabulary and normalization rules can change. For readers, the durable mental model is simple: words are a human linguistic unit, while tokens are model-specific computational pieces. They often overlap, but they are neither interchangeable nor consistently sized across languages, formats and model families. [1][2][3][4]
Sources
- OpenAI Developers — Key concepts: Tokens
- Google SentencePiece — Official repository and documentation
- ACL Anthology — SentencePiece: A simple and language independent subword tokenizer and detokenizer
- ACL Anthology — Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenization