Home / Add-ons / Benchmark Report
Benchmark Report — model latency measured over six days
A dated PDF and CSV of p50, p95 and p99 latency for all 39 public model IDs, measured by a five-minute production probe from 16 to 22 November 2026.
On this page · 11 sections
Who it is for
Teams writing a business case, sizing a migration or defending a latency claim who need a dated number instead of a vendor adjective.
The problem it removes
- Latency claims are quoted from a launch post with no date.
- A benchmark mixes models, regions and probe shapes, so nobody can reproduce it.
- The CSV behind a chart is never published.
What you get
- 1,828 probe cycles, 71,292 model checks, six days
- p50, p95 and p99 per model, plus ok / slow / broken / rate-limited counts
- 39 public model IDs in 10 probe streams, stated explicitly
- Plan economics and published limits in the same document
- PDF for the appendix, CSV for your own analysis
How to use it
1. Download
No account required. The ZIP contains the PDF, the CSV, the README and the licence.
2. Read the method
The PDF states the probe shape and window before the latency table, and lists the exclusions in a dedicated limits section.
3. Recompute
The CSV carries every row, so your analyst can reproduce the percentiles.
curl -sS https://plugsky.com/downloads/plugsky-benchmark-2026-11.zip -o plugsky-benchmark-2026-11.zip
What the report measures
Every five minutes, a production cron on Plugsky infrastructure sends one short chat probe to each distinct upstream: the prompt Reply OK. with max_tokens=128. The result is recorded against every Plugsky alias that shares that upstream. The report covers six days, 1,828 cycles and 71,292 model checks.
| Section | Contents | Question it answers |
|---|---|---|
| Method | Probe shape, cadence, window, probe streams | Can I reproduce this? |
| Latency by model | p50, p95, p99 and mean for 39 public model IDs | How fast is this model in practice? |
| Reliability | ok, slow, broken and rate-limited counts | How often does it fail? |
| Plan economics | Prices, daily caps, concurrency, RPM | Which plan fits this latency budget? |
| Limits | Ten explicit exclusions | What can I not conclude? |
Headline numbers
The window is 16–22 November 2026, UTC. Across all checks, 96.87% returned ok. The table below is a sample of the full 39-row ranking, which is in both the PDF and the CSV.
| Model | Samples | ok | p50 | p95 | p99 |
|---|---|---|---|---|---|
| plugsky-embed | 1,828 | 100.00% | 77 ms | 112 ms | 181 ms |
| plugsky-translate | 1,828 | 100.00% | 164 ms | 426 ms | 882 ms |
| plugsky-parse | 1,828 | 96.72% | 439 ms | 1,927 ms | 9,782 ms |
| plugsky-minimax | 1,828 | 93.82% | 599 ms | 3,146 ms | 9,822 ms |
| plugsky-mistral-medium | 1,828 | 89.06% | 3,842 ms | 28,366 ms | 39,871 ms |
The three embedding aliases share one probe stream, so their identical numbers are expected, not a coincidence. The report labels each stream (S1–S10) and says how many aliases share it.
What it does not measure
- No competitor latency. We do not run other providers, so we do not quote them.
- No throughput under load. The probe is one request at a time, not a concurrency test.
- No long-context timing. The probe is short; a 100K-token prompt is a different measurement.
- No streaming. Time to first token is not the same as total request time.
- No customer network time. The probe runs from Plugsky production, so your RTT is additional.
- No quality score. A fast wrong answer is still wrong.
Those exclusions are printed in a dedicated limits section on page 6 of the PDF, after the latency table. A benchmark that hides its shape is a marketing page with a chart.
How to read the CSV
Each row is one alias: model, samples, ok percentage, p50/p95/p99, mean, the window and a notes column that records the stream and the counts behind the percentages. Percentiles use successful probes only; failures and rate limits are counted separately. The CSV carries one row per model; request-level rows are not shipped. The README states the method so your analyst can recompute the percentiles from a comparable log contract.
The notes column also records watchdog upstream overrides where they exist. That matters: an alias whose upstream changed mid-window can mix two populations, and the report says so rather than smoothing it away.
A note on the bad cycles
A six-day average hides the hours that hurt. The report counts 112 of 1,828 cycles in which ten or more model IDs were not ok, and the worst cycle had 34 of 39 model IDs failing or rate-limited at once. Those clusters are why the report publishes slow, broken and rate_limited separately instead of folding them into an error rate.
They also explain the gap between p50 and p95 on the slowest model. A p50 of 3.8 seconds with a p95 of 28 seconds is not a model that is usually slow; it is a model that is occasionally starved. If your workload is latency-sensitive, the p95 is the number to plan against, and the report prints it for every row.
The probe does not attribute the cause. A cluster can come from an upstream worker at capacity, a rate limit on our key, or a network path between regions. The report says which status fired; it does not pretend to know why.
Who should skip it
If you need a latency number for your own prompts, this report is not that measurement. If you need competitor pricing, this is the wrong product. And if a single number from the pricing page is enough for your decision, the free pages already answer it.
Version and licence
Version 2026-11. Four files, about 100 KB. Commercial use inside your organisation is covered; reselling or republishing the report or the CSV is not. The licence is in the ZIP and repeats the point in one page.
The report is dated and will go stale. That is deliberate: a latency figure without a date is a rumour. Re-run the probe after a model or upstream change rather than quoting this edition as current.
Questions
What exactly do I get?
A ZIP with report.pdf (6 pages), benchmark.csv (39 rows), README.md and LICENSE.txt.
Does it benchmark other providers?
No. It measures Plugsky only. Competitor latency is excluded because we do not run those systems.
How was latency measured?
A production cron sends a short synthetic chat probe to each distinct upstream every five minutes. Aliases sharing an upstream share the measurement.
Are the percentiles independent?
No. 39 public model IDs collapse to 10 probe streams; the report says so rather than implying 71,292 independent observations.
Will the numbers match my workload?
Probably not exactly. The probe is short and non-streaming, so it excludes long-context time, streaming and your network.
Sources
- Plugsky pricing and fair-use limits — checked 22 November 2026
- Plugsky model catalogue — checked 22 November 2026
- Plugsky API documentation — checked 22 November 2026
Related add-ons
AI Cost Index Pro
A dated JSON dataset of Plugsky plan prices, intervals and the live model capability matrix, built for procurement comparison and internal budgeting.
Usage Export
Download every request your workspace makes — model, tokens, cost, status and latency — as a CSV with a fixed header. Built for finance reconciliation and warehouse loads.
Free download. No account required. The ZIP includes the PDF, the data and the licence.
Download the ZIPCompare add-ons