Short answer. Most Arabic benchmarks test Modern Standard Arabic and call it done. This one separates MSA from Gulf and Egyptian dialect across nine task families, scores accuracy, latency and cost per 1,000 items, and publishes the item set and rubric. The baseline run opens on 20 November 2026; scores appear only after two annotators have reviewed every item.
Baseline: 20 November 2026. No model has been scored yet. Publishing an Arabic leaderboard without a run would mean inventing numbers, and Arabic is exactly the language where invented numbers are easiest to spot in production.
What the benchmark measures
Arabic is not one register. A model that writes polished MSA can fail a Gulf customer-support reply, and a model that handles Egyptian sentiment can mangle an MSA legal summary. The benchmark therefore reports three subtotals — MSA, Gulf and Egyptian — and never a single “Arabic” score without them.
Each item produces four measurements: task accuracy, latency (p50 and p95), token cost at the provider’s published input and output rates, and a failure label. The failure label matters more than the score: “wrong dialect”, “wrong entity”, “hallucinated citation” and “refused” are different defects with different fixes.
The task set
| Family | Variety | Items | Scoring |
|---|---|---|---|
| News comprehension | MSA | 80 | Exact match on 4 options |
| Named entity recognition | MSA | 80 | Entity-level F1 |
| Summarisation | MSA | 80 | Rubric 1–5, two annotators |
| Grounded answer with citation | MSA | 80 | Answer accuracy + citation faithfulness |
| Customer-support intent | Gulf | 100 | Macro F1 over 12 intents |
| Reply generation | Gulf | 100 | Rubric 1–5 for dialect and policy |
| Sentiment | Egyptian | 120 | Macro F1, three classes |
| Dialect to MSA rewrite | Egyptian | 80 | BLEU + rubric 1–5 |
| Translation | MSA to English | 80 | COMET-style ranking + spot check |
Total: 800 scored items plus 400 held-out items used only for a final check. The public set is 800 items; the held-out 400 stay private so a model cannot be tuned to the test.
The 12-model cohort
All twelve models are served through one OpenAI-compatible endpoint, which keeps the harness identical. Context windows are the values published in the catalogue on 20 November 2026.
| Model | Context | Tier | Why it is in the cohort |
|---|---|---|---|
plugsky-max | 262,144 | Paid | Top quality reference |
plugsky-ultra | 262,144 | Paid | Top quality, long context |
plugsky-pro | 262,144 | Paid | General production model |
plugsky-plus | 262,144 | Paid | Mid-tier default |
plugsky-thinker | 262,144 | Paid | Reasoning-heavy tasks |
plugsky-kimi | 262,144 | Paid | Long-document Arabic |
plugsky-longctx | 262,144 | Paid | Long-context control |
plugsky-qwen-next | 262,144 | Paid | Efficient multilingual model |
plugsky-deepseek-pro | 262,144 | Paid | Reasoning comparison |
plugsky-gemma-4 | 131,072 | Paid | Small-model baseline |
plugsky-mistral-medium | 131,072 | Paid | European model baseline |
plugsky-parse | 262,144 | Paid | Vision and document parsing |
Scoring and cost
| Dimension | Method | Reported as |
|---|---|---|
| Accuracy | Task-specific metric, two annotators on rubric items | Percentage with 95% confidence interval |
| Dialect fidelity | Annotator judgement: MSA, Gulf, Egyptian, mixed | Share of replies in the requested variety |
| Latency | Sequential requests, streaming, fixed 300-token output | p50 and p95 milliseconds |
| Cost | Published input and output rates × measured tokens | USD per 1,000 items |
| Reliability | Non-2xx, malformed JSON, empty content | Failure rate by class |
Cost is reported per 1,000 items rather than per million tokens, because that is the unit a product team budgets. It includes the prompt, the retrieved context for RAG items, and the completion, at the rates published on 20 November 2026.
Dataset construction and licensing
MSA news items are drawn from public-domain and CC BY-SA sources; dialect items are written for this benchmark by native speakers and reviewed by a second annotator. No customer data is used, and no item is generated by a model that is also in the cohort. Each item carries its provenance, the annotator IDs, and a licence tag.
Dialect coverage is the hard part. Gulf Arabic varies by country, and Egyptian Arabic has its own register split between formal writing and speech. The benchmark labels each item with country and register, and reports Gulf and Egyptian subtotals separately rather than pretending the dialects are uniform.
Methodology and limitations
- Prompting. One system prompt per task family, identical across models, with the output schema in the prompt. No few-shot examples, because example selection is a hidden variable.
- Annotators. Two native speakers per rubric item; disagreements are adjudicated by a third. Inter-annotator agreement is reported with the first results.
- Held-out set. 400 items never published, used once per model version. A model that scores far above its public result is flagged, not celebrated.
- Versioning. Model IDs and snapshot dates are pinned. A provider update starts a new row; results are never overwritten.
Three limitations are worth stating up front. Eight hundred items is enough to rank models coarsely, not to detect a two-point difference. Rubric scoring is subjective even with two annotators. And a benchmark written by one team reflects that team’s idea of a good Arabic answer; the item set is public so that judgement can be challenged.
What we will publish first. The first release will contain the cohort, the task set, the harness and the annotation guide. Scores follow when the run completes and the second annotation pass closes. If a model fails to complete a task family, the cell will show the failure class rather than a zero, because a refusal and a wrong answer are not the same result.
Sources
Links checked 20 November 2026. We cite the primary page for each figure. Where a provider publishes a price behind a login, we say so and do not guess.
- ArabicMMLU — a public MSA knowledge benchmark used as a design reference
- Petrov et al., tokenizer fairness — evidence that Arabic costs 1.14× to 4.40× English tokens depending on tokenizer
- Anthropic model deprecations — example of the snapshot discipline applied to the cohort
FAQ
Why separate MSA, Gulf and Egyptian Arabic?
Because they behave differently in production. A model can write strong Modern Standard Arabic and still fail a Gulf customer-support reply or an Egyptian sentiment item. Reporting one blended Arabic score hides the failure that matters for a given product.
How many items are in the benchmark?
Eight hundred scored items plus 400 held-out items. The public set covers nine task families: comprehension, entity recognition, summarisation, grounded answering, Gulf intent and reply, Egyptian sentiment and rewrite, and MSA-to-English translation.
Are there scores yet?
Not yet. The baseline run opens on 20 November 2026 and scores are published after two annotators review every rubric item. This page publishes the cohort, the task set and the method so the first results can be checked.
How is cost calculated?
Published input and output rates multiplied by measured tokens, reported as USD per 1,000 items. It includes the prompt and any retrieved context for RAG items, so it reflects what a product team would actually spend.
Is customer data used?
No. MSA items come from public-domain and CC BY-SA sources; dialect items were written for the benchmark by native speakers. No customer data and no model-generated items from the cohort are used.
Can I reproduce the benchmark?
Yes, once the item set and harness are released. Prompts, output schemas, model snapshots and annotation guidelines are published together, and the held-out 400 items stay private to catch overfitting.
Use the data, then test it on your own workload
Plugsky publishes its plan caps and model limits. Start free with no card, or compare the flat plans against per-token billing.
Start Free → See plans