Free tool · measured 20 November 2026

Tokenizer benchmark: tokens per word by language

Arabic costs 3.88× English per character on cl100k_base and 1.80× on o200k_base. Measure the tokenizer before you blame the model, the prompt or the provider.

Token cost by language

Open the tokenizer

The estimate uses measured tokens per 1,000 characters from the sample below, not a live tokenizer call. For an exact count on your own text, use the tokenizer.

Result Arabic

83,200 tokens for 100,000 characters, costing $0.21 at $2.50 per 1M input tokens. That is 3.88× the English cost for the same character count.
Tokens83,200
Input cost$0.21
Cost vs English3.88×
Approx. words18,200

Word estimate uses the measured tokens-per-word ratio for space-delimited languages. For Chinese and Japanese it is derived from characters, because words are not whitespace-delimited.

Tokenizer premium: 12 languages, 8 tokenizers

The table is the cleanest cross-tokenizer comparison available: the same 2,000 FLORES-200 sentences translated into each language, tokenised by eight tokenizer families, then divided by the English token count for the same tokenizer. A value of 3.00 means the text costs three times as many tokens as the English original. The last column is the spread between the best and worst tokenizer for that language.

Tokenizer premium relative to English on FLORES-200. Lower is better. Source: Petrov et al., NeurIPS 2023.
LanguageGPT-2Llamacl100kQwenXLM-RmT5BLOOMNLLBSpread
Standard Arabic4.403.423.041.631.181.351.141.403.9×
Hindi7.464.604.794.471.251.591.281.226.1×
Bengali9.655.385.845.091.381.581.171.288.2×
Urdu6.304.374.393.191.231.521.361.305.1×
Korean5.073.182.381.641.161.272.791.034.9×
Hebrew4.393.293.661.481.121.222.921.243.9×
Vietnamese4.542.922.451.411.181.951.271.183.8×
Chinese (Simplified)3.212.001.911.070.970.920.951.113.5×
Japanese3.002.242.301.461.110.901.811.013.3×
Turkish2.432.091.911.611.041.121.961.142.3×
Spanish1.991.451.551.521.201.311.211.241.7×
Indonesian1.981.761.551.540.941.080.960.932.1×

Arabic costs 4.40× English on GPT-2 but 1.14× on BLOOM — a 3.9× swing from the tokenizer alone. Bengali is worse: 9.65× versus 1.17×. If you pay per token, the tokenizer is a procurement decision.

Measured tokens per word and per 1,000 characters

The second table is our own measurement, run on 20 November 2026. We took the first 10,000 characters of the Wikipedia article on artificial intelligence in each language, counted whitespace-delimited words, and tokenised with tiktoken 0.12 (cl100k_base and o200k_base). The text is encyclopedic prose in both cases, which is close enough for a cost ratio and not a substitute for your own corpus.

Measured on 10,000-character samples, 20 November 2026. Cost per 1M characters at $2.50 per 1M input tokens.
LanguageTok/1k chars cl100kTok/1k chars o200kCost × English clCost × English o200k$/1M chars cl$/1M chars o200kTok/word clTok/word o200k
English214.3213.31.00×1.00×$0.54$0.531.241.23
Spanish288.7234.11.35×1.10×$0.72$0.591.661.34
Turkish446.1321.72.08×1.51×$1.12$0.803.142.26
Standard Arabic832.0383.63.88×1.80×$2.08$0.964.572.11
Korean1,215.7732.45.67×3.43×$3.04$1.833.772.27
Hindi1,179.8381.45.51×1.79×$2.95$0.955.271.71
Japanese1,054.7759.34.92×3.56×$2.64$1.90n/an/a
Chinese (Simplified)1,092.3713.75.10×3.35×$2.73$1.78n/an/a

Chinese and Japanese do not delimit words with spaces, so a tokens-per-word figure would compare a whitespace segment to a word and inflate the number. We report “n/a” and use characters instead. Arabic, Hindi, Korean and Turkish are space-delimited and comparable.

What the multiplier costs you

The bars show cost per character relative to English. The newer tokenizer is the largest single improvement in the dataset: Arabic falls from 3.88× to 1.80× and Hindi from 5.50× to 1.79×. Chinese and Japanese improve far less, because the bottleneck is the script, not the vocabulary.

English
Spanish
Turkish
Standard Arabic
Korean
Hindi
Japanese
Chinese (Simplified)

cl100k_base · o200k_base — cost per character relative to English, from the measured table above.

At $2.50 per million input tokens, a million characters of English costs about $0.54. Arabic costs $2.08 on cl100k and $0.96 on o200k; Hindi costs $2.95 and $0.95. At corpus scale that is a budget line, not a rounding error.

Why Arabic and Hindi move so much

Arabic is morphologically rich: one surface form can carry stem, person, number, gender and case, and the orthography often omits short vowels. A tokenizer trained mostly on English learns few Arabic subwords and falls back to characters. Hindi has the same problem with Devanagari conjuncts.

A tokenizer trained on more multilingual text learns those subwords and needs fewer tokens. That is why Qwen, XLM-R and BLOOM land near parity for Arabic while GPT-2 does not. Tokenizer coverage belongs on the evaluation sheet next to price and context.

Method and reproducibility

The premium table is a re-analysis of the published FLORES-200 tokenization lengths from Petrov et al. (2023), normalised to English per tokenizer. The measured table is our own run; the corpus is the plain-text extract of the same Wikipedia article in each language, truncated to 10,000 characters, with the API's own word count.

import json, re, tiktoken, urllib.parse, urllib.request

TITLE = {"en": "Artificial intelligence", "ar": "ذكاء اصطناعي", "hi": "कृत्रिम बुद्धिमत्ता"}
enc = tiktoken.get_encoding("cl100k_base")

for code, title in TITLE.items():
    q = urllib.parse.urlencode({"action": "query", "prop": "extracts",
        "explaintext": "1", "titles": title, "format": "json", "redirects": "1"})
    with urllib.request.urlopen(f"https://{code}.wikipedia.org/w/api.php?{q}") as r:
        page = list(json.load(r)["query"]["pages"].values())[0]
    text = page["extract"][:10000]
    words = len([w for w in re.split(r"\s+", text.strip()) if w])
    print(code, len(enc.encode(text)), words)

Limits: one article per language, 10,000 characters, one point in time. Wikipedia register is not your product's register, so treat the ratios as a planning estimate and run the harness on your own documents before committing to a per-token budget.

Sources

Links checked 20 November 2026. We cite the primary page for each figure. Where a provider publishes a price behind a login, we say so and do not guess.

FAQ

How many tokens is Arabic compared with English?

It depends on the tokenizer. On the FLORES-200 sample, Standard Arabic costs 4.40× the English token count with GPT-2, 3.04× with cl100k_base, 1.63× with Qwen and 1.14× with BLOOM. The tokenizer choice changes the bill by almost 4×.

What is a tokenizer premium?

The ratio of tokens needed for the same content in another language to the tokens needed in English, using the same tokenizer. Parity is 1.00. A premium of 3.00 means the text costs three times as much to send.

Why is tokens per word “n/a” for Chinese and Japanese?

Because those languages do not separate words with spaces. Counting whitespace segments and calling them words would overstate the ratio. For those languages, tokens per 1,000 characters is the comparable metric.

Does the newer OpenAI tokenizer fix multilingual cost?

It closes most of the gap for Arabic and Hindi: Arabic falls from 3.88× to 1.80× the English cost per character, and Hindi from 5.50× to 1.79×. It helps Chinese and Japanese far less, because the cost is driven by the script rather than the vocabulary.

How was the benchmark measured?

The 8-tokenizer table re-analyses the published FLORES-200 tokenization lengths from Petrov et al. (2023). The measured table runs tiktoken 0.14 on the first 10,000 characters of the same Wikipedia article in each language, collected on 20 November 2026.

Run the same check on your own traffic

Plugsky's API is OpenAI-compatible, with published plan caps and 39 models on one key. Start free with no card.

Start free → See plans