Tokenizer benchmark: tokens per word by language
Arabic costs 3.88× English per character on cl100k_base and 1.80× on o200k_base. Measure the tokenizer before you blame the model, the prompt or the provider.
Token cost by language
The estimate uses measured tokens per 1,000 characters from the sample below, not a live tokenizer call. For an exact count on your own text, use the tokenizer.
Result Arabic
Word estimate uses the measured tokens-per-word ratio for space-delimited languages. For Chinese and Japanese it is derived from characters, because words are not whitespace-delimited.
Tokenizer premium: 12 languages, 8 tokenizers
The table is the cleanest cross-tokenizer comparison available: the same 2,000 FLORES-200 sentences translated into each language, tokenised by eight tokenizer families, then divided by the English token count for the same tokenizer. A value of 3.00 means the text costs three times as many tokens as the English original. The last column is the spread between the best and worst tokenizer for that language.
| Language | GPT-2 | Llama | cl100k | Qwen | XLM-R | mT5 | BLOOM | NLLB | Spread |
|---|---|---|---|---|---|---|---|---|---|
| Standard Arabic | 4.40 | 3.42 | 3.04 | 1.63 | 1.18 | 1.35 | 1.14 | 1.40 | 3.9× |
| Hindi | 7.46 | 4.60 | 4.79 | 4.47 | 1.25 | 1.59 | 1.28 | 1.22 | 6.1× |
| Bengali | 9.65 | 5.38 | 5.84 | 5.09 | 1.38 | 1.58 | 1.17 | 1.28 | 8.2× |
| Urdu | 6.30 | 4.37 | 4.39 | 3.19 | 1.23 | 1.52 | 1.36 | 1.30 | 5.1× |
| Korean | 5.07 | 3.18 | 2.38 | 1.64 | 1.16 | 1.27 | 2.79 | 1.03 | 4.9× |
| Hebrew | 4.39 | 3.29 | 3.66 | 1.48 | 1.12 | 1.22 | 2.92 | 1.24 | 3.9× |
| Vietnamese | 4.54 | 2.92 | 2.45 | 1.41 | 1.18 | 1.95 | 1.27 | 1.18 | 3.8× |
| Chinese (Simplified) | 3.21 | 2.00 | 1.91 | 1.07 | 0.97 | 0.92 | 0.95 | 1.11 | 3.5× |
| Japanese | 3.00 | 2.24 | 2.30 | 1.46 | 1.11 | 0.90 | 1.81 | 1.01 | 3.3× |
| Turkish | 2.43 | 2.09 | 1.91 | 1.61 | 1.04 | 1.12 | 1.96 | 1.14 | 2.3× |
| Spanish | 1.99 | 1.45 | 1.55 | 1.52 | 1.20 | 1.31 | 1.21 | 1.24 | 1.7× |
| Indonesian | 1.98 | 1.76 | 1.55 | 1.54 | 0.94 | 1.08 | 0.96 | 0.93 | 2.1× |
Arabic costs 4.40× English on GPT-2 but 1.14× on BLOOM — a 3.9× swing from the tokenizer alone. Bengali is worse: 9.65× versus 1.17×. If you pay per token, the tokenizer is a procurement decision.
Measured tokens per word and per 1,000 characters
The second table is our own measurement, run on 20 November 2026. We took the first 10,000 characters of the Wikipedia article on artificial intelligence in each language, counted whitespace-delimited words, and tokenised with tiktoken 0.12 (cl100k_base and o200k_base). The text is encyclopedic prose in both cases, which is close enough for a cost ratio and not a substitute for your own corpus.
| Language | Tok/1k chars cl100k | Tok/1k chars o200k | Cost × English cl | Cost × English o200k | $/1M chars cl | $/1M chars o200k | Tok/word cl | Tok/word o200k |
|---|---|---|---|---|---|---|---|---|
| English | 214.3 | 213.3 | 1.00× | 1.00× | $0.54 | $0.53 | 1.24 | 1.23 |
| Spanish | 288.7 | 234.1 | 1.35× | 1.10× | $0.72 | $0.59 | 1.66 | 1.34 |
| Turkish | 446.1 | 321.7 | 2.08× | 1.51× | $1.12 | $0.80 | 3.14 | 2.26 |
| Standard Arabic | 832.0 | 383.6 | 3.88× | 1.80× | $2.08 | $0.96 | 4.57 | 2.11 |
| Korean | 1,215.7 | 732.4 | 5.67× | 3.43× | $3.04 | $1.83 | 3.77 | 2.27 |
| Hindi | 1,179.8 | 381.4 | 5.51× | 1.79× | $2.95 | $0.95 | 5.27 | 1.71 |
| Japanese | 1,054.7 | 759.3 | 4.92× | 3.56× | $2.64 | $1.90 | n/a | n/a |
| Chinese (Simplified) | 1,092.3 | 713.7 | 5.10× | 3.35× | $2.73 | $1.78 | n/a | n/a |
Chinese and Japanese do not delimit words with spaces, so a tokens-per-word figure would compare a whitespace segment to a word and inflate the number. We report “n/a” and use characters instead. Arabic, Hindi, Korean and Turkish are space-delimited and comparable.
What the multiplier costs you
The bars show cost per character relative to English. The newer tokenizer is the largest single improvement in the dataset: Arabic falls from 3.88× to 1.80× and Hindi from 5.50× to 1.79×. Chinese and Japanese improve far less, because the bottleneck is the script, not the vocabulary.
cl100k_base · o200k_base — cost per character relative to English, from the measured table above.
At $2.50 per million input tokens, a million characters of English costs about $0.54. Arabic costs $2.08 on cl100k and $0.96 on o200k; Hindi costs $2.95 and $0.95. At corpus scale that is a budget line, not a rounding error.
Why Arabic and Hindi move so much
Arabic is morphologically rich: one surface form can carry stem, person, number, gender and case, and the orthography often omits short vowels. A tokenizer trained mostly on English learns few Arabic subwords and falls back to characters. Hindi has the same problem with Devanagari conjuncts.
A tokenizer trained on more multilingual text learns those subwords and needs fewer tokens. That is why Qwen, XLM-R and BLOOM land near parity for Arabic while GPT-2 does not. Tokenizer coverage belongs on the evaluation sheet next to price and context.
Method and reproducibility
The premium table is a re-analysis of the published FLORES-200 tokenization lengths from Petrov et al. (2023), normalised to English per tokenizer. The measured table is our own run; the corpus is the plain-text extract of the same Wikipedia article in each language, truncated to 10,000 characters, with the API's own word count.
import json, re, tiktoken, urllib.parse, urllib.request
TITLE = {"en": "Artificial intelligence", "ar": "ذكاء اصطناعي", "hi": "कृत्रिम बुद्धिमत्ता"}
enc = tiktoken.get_encoding("cl100k_base")
for code, title in TITLE.items():
q = urllib.parse.urlencode({"action": "query", "prop": "extracts",
"explaintext": "1", "titles": title, "format": "json", "redirects": "1"})
with urllib.request.urlopen(f"https://{code}.wikipedia.org/w/api.php?{q}") as r:
page = list(json.load(r)["query"]["pages"].values())[0]
text = page["extract"][:10000]
words = len([w for w in re.split(r"\s+", text.strip()) if w])
print(code, len(enc.encode(text)), words)
Limits: one article per language, 10,000 characters, one point in time. Wikipedia register is not your product's register, so treat the ratios as a planning estimate and run the harness on your own documents before committing to a per-token budget.
Sources
Links checked 20 November 2026. We cite the primary page for each figure. Where a provider publishes a price behind a login, we say so and do not guess.
- Petrov et al., Language Model Tokenizers Introduce Unfairness Between Languages (NeurIPS 2023) — FLORES-200 tokenization lengths across 17 tokenizers
- tiktoken — cl100k_base and o200k_base encodings used for the measured table
- Wikimedia API — plain-text extracts used as the language sample
FAQ
How many tokens is Arabic compared with English?
It depends on the tokenizer. On the FLORES-200 sample, Standard Arabic costs 4.40× the English token count with GPT-2, 3.04× with cl100k_base, 1.63× with Qwen and 1.14× with BLOOM. The tokenizer choice changes the bill by almost 4×.
What is a tokenizer premium?
The ratio of tokens needed for the same content in another language to the tokens needed in English, using the same tokenizer. Parity is 1.00. A premium of 3.00 means the text costs three times as much to send.
Why is tokens per word “n/a” for Chinese and Japanese?
Because those languages do not separate words with spaces. Counting whitespace segments and calling them words would overstate the ratio. For those languages, tokens per 1,000 characters is the comparable metric.
Does the newer OpenAI tokenizer fix multilingual cost?
It closes most of the gap for Arabic and Hindi: Arabic falls from 3.88× to 1.80× the English cost per character, and Hindi from 5.50× to 1.79×. It helps Chinese and Japanese far less, because the cost is driven by the script rather than the vocabulary.
How was the benchmark measured?
The 8-tokenizer table re-analyses the published FLORES-200 tokenization lengths from Petrov et al. (2023). The measured table runs tiktoken 0.14 on the first 10,000 characters of the same Wikipedia article in each language, collected on 20 November 2026.
Run the same check on your own traffic
Plugsky's API is OpenAI-compatible, with published plan caps and 39 models on one key. Start free with no card.
Start free → See plans