Data · Arabic

Arabic LLM benchmark: 12 models compared

Nine task families across Modern Standard, Gulf and Egyptian Arabic, scored on accuracy, latency and cost per 1,000 items — with the method published before the scores.

P
Plugsky Data desk
Baseline 20 November 2026 · reviewed 20 November 2026

Short answer. Most Arabic benchmarks test Modern Standard Arabic and call it done. This one separates MSA from Gulf and Egyptian dialect across nine task families, scores accuracy, latency and cost per 1,000 items, and publishes the item set and rubric. The baseline run opens on 20 November 2026; scores appear only after two annotators have reviewed every item.

Baseline: 20 November 2026. No model has been scored yet. Publishing an Arabic leaderboard without a run would mean inventing numbers, and Arabic is exactly the language where invented numbers are easiest to spot in production.

What the benchmark measures

Arabic is not one register. A model that writes polished MSA can fail a Gulf customer-support reply, and a model that handles Egyptian sentiment can mangle an MSA legal summary. The benchmark therefore reports three subtotals — MSA, Gulf and Egyptian — and never a single “Arabic” score without them.

Each item produces four measurements: task accuracy, latency (p50 and p95), token cost at the provider’s published input and output rates, and a failure label. The failure label matters more than the score: “wrong dialect”, “wrong entity”, “hallucinated citation” and “refused” are different defects with different fixes.

The task set

FamilyVarietyItemsScoring
News comprehensionMSA80Exact match on 4 options
Named entity recognitionMSA80Entity-level F1
SummarisationMSA80Rubric 1–5, two annotators
Grounded answer with citationMSA80Answer accuracy + citation faithfulness
Customer-support intentGulf100Macro F1 over 12 intents
Reply generationGulf100Rubric 1–5 for dialect and policy
SentimentEgyptian120Macro F1, three classes
Dialect to MSA rewriteEgyptian80BLEU + rubric 1–5
TranslationMSA to English80COMET-style ranking + spot check

Total: 800 scored items plus 400 held-out items used only for a final check. The public set is 800 items; the held-out 400 stay private so a model cannot be tuned to the test.

The 12-model cohort

All twelve models are served through one OpenAI-compatible endpoint, which keeps the harness identical. Context windows are the values published in the catalogue on 20 November 2026.

ModelContextTierWhy it is in the cohort
plugsky-max262,144PaidTop quality reference
plugsky-ultra262,144PaidTop quality, long context
plugsky-pro262,144PaidGeneral production model
plugsky-plus262,144PaidMid-tier default
plugsky-thinker262,144PaidReasoning-heavy tasks
plugsky-kimi262,144PaidLong-document Arabic
plugsky-longctx262,144PaidLong-context control
plugsky-qwen-next262,144PaidEfficient multilingual model
plugsky-deepseek-pro262,144PaidReasoning comparison
plugsky-gemma-4131,072PaidSmall-model baseline
plugsky-mistral-medium131,072PaidEuropean model baseline
plugsky-parse262,144PaidVision and document parsing

Scoring and cost

DimensionMethodReported as
AccuracyTask-specific metric, two annotators on rubric itemsPercentage with 95% confidence interval
Dialect fidelityAnnotator judgement: MSA, Gulf, Egyptian, mixedShare of replies in the requested variety
LatencySequential requests, streaming, fixed 300-token outputp50 and p95 milliseconds
CostPublished input and output rates × measured tokensUSD per 1,000 items
ReliabilityNon-2xx, malformed JSON, empty contentFailure rate by class

Cost is reported per 1,000 items rather than per million tokens, because that is the unit a product team budgets. It includes the prompt, the retrieved context for RAG items, and the completion, at the rates published on 20 November 2026.

Dataset construction and licensing

MSA news items are drawn from public-domain and CC BY-SA sources; dialect items are written for this benchmark by native speakers and reviewed by a second annotator. No customer data is used, and no item is generated by a model that is also in the cohort. Each item carries its provenance, the annotator IDs, and a licence tag.

Dialect coverage is the hard part. Gulf Arabic varies by country, and Egyptian Arabic has its own register split between formal writing and speech. The benchmark labels each item with country and register, and reports Gulf and Egyptian subtotals separately rather than pretending the dialects are uniform.

Methodology and limitations

  • Prompting. One system prompt per task family, identical across models, with the output schema in the prompt. No few-shot examples, because example selection is a hidden variable.
  • Annotators. Two native speakers per rubric item; disagreements are adjudicated by a third. Inter-annotator agreement is reported with the first results.
  • Held-out set. 400 items never published, used once per model version. A model that scores far above its public result is flagged, not celebrated.
  • Versioning. Model IDs and snapshot dates are pinned. A provider update starts a new row; results are never overwritten.

Three limitations are worth stating up front. Eight hundred items is enough to rank models coarsely, not to detect a two-point difference. Rubric scoring is subjective even with two annotators. And a benchmark written by one team reflects that team’s idea of a good Arabic answer; the item set is public so that judgement can be challenged.

What we will publish first. The first release will contain the cohort, the task set, the harness and the annotation guide. Scores follow when the run completes and the second annotation pass closes. If a model fails to complete a task family, the cell will show the failure class rather than a zero, because a refusal and a wrong answer are not the same result.

Sources

Links checked 20 November 2026. We cite the primary page for each figure. Where a provider publishes a price behind a login, we say so and do not guess.

FAQ

Why separate MSA, Gulf and Egyptian Arabic?

Because they behave differently in production. A model can write strong Modern Standard Arabic and still fail a Gulf customer-support reply or an Egyptian sentiment item. Reporting one blended Arabic score hides the failure that matters for a given product.

How many items are in the benchmark?

Eight hundred scored items plus 400 held-out items. The public set covers nine task families: comprehension, entity recognition, summarisation, grounded answering, Gulf intent and reply, Egyptian sentiment and rewrite, and MSA-to-English translation.

Are there scores yet?

Not yet. The baseline run opens on 20 November 2026 and scores are published after two annotators review every rubric item. This page publishes the cohort, the task set and the method so the first results can be checked.

How is cost calculated?

Published input and output rates multiplied by measured tokens, reported as USD per 1,000 items. It includes the prompt and any retrieved context for RAG items, so it reflects what a product team would actually spend.

Is customer data used?

No. MSA items come from public-domain and CC BY-SA sources; dialect items were written for the benchmark by native speakers. No customer data and no model-generated items from the cohort are used.

Can I reproduce the benchmark?

Yes, once the item set and harness are released. Prompts, output schemas, model snapshots and annotation guidelines are published together, and the held-out 400 items stay private to catch overfitting.

Use the data, then test it on your own workload

Plugsky publishes its plan caps and model limits. Start free with no card, or compare the flat plans against per-token billing.

Start Free → See plans