Tokenization

A support team plans to translate their English documentation into Hindi. The token budget that comfortably fit the English version overflows immediately once translated — for the exact same meaning, in the exact same context window.

Tokens Are Not Words

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")

for text in ["Hello world", "I love machine learning", "supercalifragilisticexpialidocious"]:
    ids = enc.encode(text)
    print(f"{text!r}")
    print(f"  tokens: {ids}")
    print(f"  count:  {len(ids)}")
    print(f"  pieces: {[enc.decode([t]) for t in ids]}")
    print()

$ python tokenize_basics.py
'Hello world'
  tokens: [9906, 1917]
  count:  2
  pieces: ['Hello', ' world']

'I love machine learning'
  tokens: [40, 3021, 5780, 6975]
  count:  4
  pieces: ['I', ' love', ' machine', ' learning']

'supercalifragilisticexpialidocious'
  tokens: [13066, 3035, 278, 333, 4193, 321, 4633, 4683, 532, 307, 78287]
  count:  11
  pieces: ['sup', 'erc', 'al', 'if', 'rag', 'il', 'istic', 'exp', 'ial', 'id', 'ocious']

Four common words became four tokens — a clean 1:1 mapping. A word absent from the training corpus’s frequent-sequence list becomes 11 tokens, each a fragment the vocabulary did have frequent evidence for. This is byte-pair encoding: a fixed vocabulary of whole words, common subwords, and individual bytes, chosen so that frequent text compresses well and rare text still has some encoding, one fragment at a time.

Why Different Languages Cost Different Token Counts

import sys
sys.stdout.reconfigure(encoding="utf-8")
import tiktoken

enc = tiktoken.get_encoding("cl100k_base")

sentences = {
    "English": "Data engineering in India",
    "Hindi":   "भारत में डेटा इंजीनियरिंग",
    "Chinese": "印度的数据工程",
}

for lang, text in sentences.items():
    ids = enc.encode(text)
    chars = len(text)
    print(f"{lang:8} chars={chars:3}  tokens={len(ids):3}  chars/token={chars/len(ids):.2f}")

$ python tokenize_languages.py
English  chars= 25  tokens=  4  chars/token=6.25
Hindi    chars= 25  tokens= 27  chars/token=0.93
Chinese  chars=  7  tokens=  7  chars/token=1.00

All three lines mean roughly the same thing. The English version packs 6.25 characters into every token; the Hindi version needs a new token for every single character — nearly 7x more tokens for equivalent meaning. cl100k_base’s vocabulary was built on a training corpus that is overwhelmingly English, so English got efficient multi-character vocabulary entries and Devanagari script did not. This is a real, measurable cost difference, not a rounding effect: a context window and a per-token price both apply to the token count, not the character count or the “amount of meaning.”

Vocabulary and Special Tokens

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")
print(f"Vocabulary size: {enc.n_vocab:,}")
print(f"Special tokens:  {sorted(enc.special_tokens_set)}")

text = "The cat sat<|endoftext|>The dog ran"
ids_allowed = enc.encode(text, allowed_special={"<|endoftext|>"})
ids_disallowed = enc.encode(text.replace("<|endoftext|>", ""))
print()
print(f"With <|endoftext|> as a real special token: {len(ids_allowed)} tokens -> {ids_allowed}")
print(f"Same text with the tag stripped:            {len(ids_disallowed)} tokens -> {ids_disallowed}")

$ python tokenize_special.py
Vocabulary size: 100,277
Special tokens:  ['<|endofprompt|>', '<|endoftext|>', '<|fim_middle|>', '<|fim_prefix|>', '<|fim_suffix|>']

With <|endoftext|> as a real special token: 7 tokens -> [791, 8415, 7731, 100257, 791, 5679, 10837]
Same text with the tag stripped:            6 tokens -> [791, 8415, 7731, 791, 5679, 10837]

<|endoftext|> becomes exactly one token (100257) — a single vocabulary entry the model was trained to treat as a hard document boundary, not as the 13 literal characters. That single ID carries structural meaning ordinary text tokens don’t: it’s how training data for multiple unrelated documents gets concatenated into one stream without the model learning to bleed context across the boundary.

Practice

import tiktoken
enc = tiktoken.get_encoding("cl100k_base")

for t in ["3.14159", "$1,234.56", "user@example.com"]:
    ids = enc.encode(t)
    print(f"{t!r}: {len(ids)} tokens -> {[enc.decode([i]) for i in ids]}")

'3.14159': 4 tokens -> ['3', '.', '141', '59']
'$1,234.56': 6 tokens -> ['$', '1', ',', '234', '.', '56']
'user@example.com': 3 tokens -> ['user', '@example', '.com']

import tiktoken
enc = tiktoken.get_encoding("cl100k_base")

for t in ["Hello", " Hello", "HELLO", "hello"]:
    print(f"{t!r}: {enc.encode(t)}")

'Hello': [9906]
' Hello': [22691]
'HELLO': [51812, 1623]
'hello': [15339]

import tiktoken
enc = tiktoken.get_encoding("cl100k_base")

for lang, t in [("English", "Machine learning is powerful"), ("Arabic", "التعلم الآلي قوي")]:
    ids = enc.encode(t)
    print(f"{lang}: chars={len(t)} tokens={len(ids)} chars/token={len(t)/len(ids):.2f}")

English: chars=28 tokens=4 chars/token=7.00
Arabic: chars=16 tokens=13 chars/token=1.23

import tiktoken
enc = tiktoken.get_encoding("cl100k_base")

text = "Summarize the quarterly earnings report for our investors." * 50
ids = enc.encode(text)
INPUT_COST_PER_MILLION = 3.00
cost = len(ids) * INPUT_COST_PER_MILLION / 1_000_000
print(f"tokens={len(ids)} estimated_cost=${cost:.6f}")

tokens=501 estimated_cost=$0.001503