● Deploy anywhereGet API key

Home / Add-ons / Benchmark Report

Benchmark Report — model latency measured over six days

A dated PDF and CSV of p50, p95 and p99 latency for all 39 public model IDs, measured by a five-minute production probe from 16 to 22 November 2026.

Free

Free download · no account required · licence inside the ZIP

Download the ZIP

On this page · 11 sections
  1. Who it is for
  2. The problem it removes
  3. What you get
  4. How to use it
  5. What the report measures
  6. Headline numbers
  7. What it does not measure
  8. How to read the CSV
  9. A note on the bad cycles
  10. Who should skip it
  11. Version and licence

Who it is for

Teams writing a business case, sizing a migration or defending a latency claim who need a dated number instead of a vendor adjective.

The problem it removes

What you get

How to use it

1. Download

No account required. The ZIP contains the PDF, the CSV, the README and the licence.

2. Read the method

The PDF states the probe shape and window before the latency table, and lists the exclusions in a dedicated limits section.

3. Recompute

The CSV carries every row, so your analyst can reproduce the percentiles.

curl -sS https://plugsky.com/downloads/plugsky-benchmark-2026-11.zip -o plugsky-benchmark-2026-11.zip

What the report measures

Every five minutes, a production cron on Plugsky infrastructure sends one short chat probe to each distinct upstream: the prompt Reply OK. with max_tokens=128. The result is recorded against every Plugsky alias that shares that upstream. The report covers six days, 1,828 cycles and 71,292 model checks.

What is in the report, and what each part answers.
SectionContentsQuestion it answers
MethodProbe shape, cadence, window, probe streamsCan I reproduce this?
Latency by modelp50, p95, p99 and mean for 39 public model IDsHow fast is this model in practice?
Reliabilityok, slow, broken and rate-limited countsHow often does it fail?
Plan economicsPrices, daily caps, concurrency, RPMWhich plan fits this latency budget?
LimitsTen explicit exclusionsWhat can I not conclude?

Headline numbers

The window is 16–22 November 2026, UTC. Across all checks, 96.87% returned ok. The table below is a sample of the full 39-row ranking, which is in both the PDF and the CSV.

Selected rows. Full data in benchmark.csv.
ModelSamplesokp50p95p99
plugsky-embed1,828100.00%77 ms112 ms181 ms
plugsky-translate1,828100.00%164 ms426 ms882 ms
plugsky-parse1,82896.72%439 ms1,927 ms9,782 ms
plugsky-minimax1,82893.82%599 ms3,146 ms9,822 ms
plugsky-mistral-medium1,82889.06%3,842 ms28,366 ms39,871 ms

The three embedding aliases share one probe stream, so their identical numbers are expected, not a coincidence. The report labels each stream (S1–S10) and says how many aliases share it.

What it does not measure

Those exclusions are printed in a dedicated limits section on page 6 of the PDF, after the latency table. A benchmark that hides its shape is a marketing page with a chart.

How to read the CSV

Each row is one alias: model, samples, ok percentage, p50/p95/p99, mean, the window and a notes column that records the stream and the counts behind the percentages. Percentiles use successful probes only; failures and rate limits are counted separately. The CSV carries one row per model; request-level rows are not shipped. The README states the method so your analyst can recompute the percentiles from a comparable log contract.

The notes column also records watchdog upstream overrides where they exist. That matters: an alias whose upstream changed mid-window can mix two populations, and the report says so rather than smoothing it away.

A note on the bad cycles

A six-day average hides the hours that hurt. The report counts 112 of 1,828 cycles in which ten or more model IDs were not ok, and the worst cycle had 34 of 39 model IDs failing or rate-limited at once. Those clusters are why the report publishes slow, broken and rate_limited separately instead of folding them into an error rate.

They also explain the gap between p50 and p95 on the slowest model. A p50 of 3.8 seconds with a p95 of 28 seconds is not a model that is usually slow; it is a model that is occasionally starved. If your workload is latency-sensitive, the p95 is the number to plan against, and the report prints it for every row.

The probe does not attribute the cause. A cluster can come from an upstream worker at capacity, a rate limit on our key, or a network path between regions. The report says which status fired; it does not pretend to know why.

Who should skip it

If you need a latency number for your own prompts, this report is not that measurement. If you need competitor pricing, this is the wrong product. And if a single number from the pricing page is enough for your decision, the free pages already answer it.

Version and licence

Version 2026-11. Four files, about 100 KB. Commercial use inside your organisation is covered; reselling or republishing the report or the CSV is not. The licence is in the ZIP and repeats the point in one page.

The report is dated and will go stale. That is deliberate: a latency figure without a date is a rumour. Re-run the probe after a model or upstream change rather than quoting this edition as current.

Questions

What exactly do I get?

A ZIP with report.pdf (6 pages), benchmark.csv (39 rows), README.md and LICENSE.txt.

Does it benchmark other providers?

No. It measures Plugsky only. Competitor latency is excluded because we do not run those systems.

How was latency measured?

A production cron sends a short synthetic chat probe to each distinct upstream every five minutes. Aliases sharing an upstream share the measurement.

Are the percentiles independent?

No. 39 public model IDs collapse to 10 probe streams; the report says so rather than implying 71,292 independent observations.

Will the numbers match my workload?

Probably not exactly. The probe is short and non-streaming, so it excludes long-context time, streaming and your network.

Sources

AI Cost Index Pro

A dated JSON dataset of Plugsky plan prices, intervals and the live model capability matrix, built for procurement comparison and internal budgeting.

$49/month · Read more →

Usage Export

Download every request your workspace makes — model, tokens, cost, status and latency — as a CSV with a fixed header. Built for finance reconciliation and warehouse loads.

$19/month · Read more →

Download the pack

Free download. No account required. The ZIP includes the PDF, the data and the licence.

Download the ZIPCompare add-ons

Planning a budget first? The Flat-Rate AI Budget Book 2027 is a free 14-page workbook. Buyers can inspect sample deliverables.