Short answer. Latency is not one number. Time to first token (TTFT), end-to-end p50 and p95, and output tokens per second measure different things, and a provider can win one while losing another. This index fixes the harness, the regions and the model cohort, then publishes monthly snapshots. The first snapshot window opens on 20 November 2026; no latency figures are published before the first completed run.
Baseline: 20 November 2026. This edition publishes the measurement matrix, the harness and the metric definitions. Publishing numbers we did not measure would defeat the purpose of an index.
What the index measures
| Metric | Definition | Why it matters |
|---|---|---|
| TTFT p50 | Milliseconds from request sent to first content token received | Perceived responsiveness in a streaming UI |
| TTFT p95 | 95th percentile of the same measurement | The wait your slowest users remember |
| End-to-end p50 | Request sent to final token, at a fixed output length | Batch and agent-step cost in wall-clock time |
| End-to-end p95 | 95th percentile of the same | Timeout and retry budget |
| Output tokens/sec | Output tokens divided by generation time after the first token | Throughput for long answers |
| Error rate | Non-2xx and malformed responses, by class | A fast 429 is not a fast success |
Every sample uses streaming, a fixed prompt of 1,000 input tokens, max_tokens of 300 and temperature 0. Input length is fixed because TTFT grows with prompt size; output length is fixed because tokens per second is only comparable at a known generation length.
The 10 regions
Probes run from one host per region, in the same cloud region as the label. The provider’s nearest endpoint is used; if a provider has no endpoint in a region, the cell is left empty rather than measured over a transatlantic hop and labelled local.
| Region | Probe location | Nearest endpoint assumption |
|---|---|---|
| us-east | Virginia, US | US East |
| us-west | Oregon, US | US West |
| ca-central | Montréal, Canada | Canada Central |
| eu-west | Ireland | EU West |
| eu-central | Frankfurt, Germany | EU Central |
| uk-south | London, UK | UK South |
| ap-south | Mumbai, India | Asia South |
| ap-southeast | Singapore | Asia Southeast |
| ap-northeast | Tokyo, Japan | Asia Northeast |
| sa-east | São Paulo, Brazil | South America East |
Ten regions covers the three largest API markets plus India, Japan, Singapore and Brazil. It does not cover Africa, the Middle East or Australia; those are the first candidates for expansion.
The 20-model cohort
Twelve Plugsky models and eight external baselines. The cohort is fixed for the quarter so month-over-month changes reflect the provider, not a new model. A retirement replaces a model at the next quarter boundary and the change is logged.
| Model | Provider | Context | Role in the cohort |
|---|---|---|---|
plugsky-plus | Plugsky | 262,144 | General text |
plugsky-pro | Plugsky | 262,144 | General text |
plugsky-max | Plugsky | 262,144 | Top quality |
plugsky-ultra | Plugsky | 262,144 | Top quality |
plugsky-thinker | Plugsky | 262,144 | Reasoning |
plugsky-kimi | Plugsky | 262,144 | Long context |
plugsky-qwen-next | Plugsky | 262,144 | Efficient text |
plugsky-deepseek-pro | Plugsky | 262,144 | Reasoning |
plugsky-coder | Plugsky | 262,144 | Code |
plugsky-parse | Plugsky | 262,144 | Vision and documents |
plugsky-vision-fast | Plugsky | 131,072 | Vision, latency-sensitive |
plugsky-translate | Plugsky | 32,768 | Translation |
gpt-6-luna | OpenAI | 272K short | Low-cost baseline |
gpt-6.2-sol | OpenAI | 272K short | Mid-tier baseline |
claude-haiku-5-5 | Anthropic | See model page | Low-cost baseline |
claude-sonnet-5-5 | Anthropic | See model page | Mid-tier baseline |
claude-opus-5-5 | Anthropic | See model page | Top-tier baseline |
| Gemini 2.5 Pro | 1,000,000 | Long-context baseline | |
| DeepSeek V3 | DeepSeek | 128,000 | Open-weight baseline |
| Llama 4 Maverick | Meta (hosted) | 1,000,000 | Open-weight baseline |
The harness
The harness is deliberately boring: one process per region, a fixed prompt, streaming responses, and a timestamp on every chunk. It records TTFT, total time, token count and the HTTP status. It does not retry on latency, because a retry turns a slow sample into a fast one and hides the tail.
import time, json, urllib.request
def sample(base_url, api_key, model, prompt):
body = json.dumps({
"model": model,
"messages": [{"role": "user", "content": prompt}],
"max_tokens": 300,
"temperature": 0,
"stream": True,
}).encode()
req = urllib.request.Request(
base_url.rstrip("/") + "/chat/completions",
data=body,
headers={"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"},
)
t0 = time.perf_counter()
ttft = None
chunks = 0
with urllib.request.urlopen(req, timeout=60) as r:
for line in r:
if not line.startswith(b"data: "):
continue
payload = line[6:].strip()
if payload == b"[DONE]":
break
delta = json.loads(payload)["choices"][0]["delta"].get("content")
if delta:
chunks += 1
if ttft is None:
ttft = time.perf_counter() - t0
total = time.perf_counter() - t0
return {"ttft_ms": round(ttft * 1000), "total_ms": round(total * 1000), "chunks": chunks}
Each cell is 600 sequential samples: 200 per day over a 30-minute window, at the same hour, on three consecutive days. Sequential rather than parallel, because concurrency changes queueing and therefore latency. The reported p95 is the p95 of the pooled samples, not the average of daily p95s.
Why the first snapshot has no numbers
A latency index is only useful if the method survives contact with the provider. Three things have to be true before a number is publishable: the model must accept the fixed prompt, the response must stream, and the run must complete without a rate limit. On the first pass, several of the 200 cells will fail one of those tests, and a failed cell must be shown as failed rather than dropped.
Publishing the matrix first lets readers challenge the design before the numbers harden. Fixing a wrong region now is cheaper than explaining a revision later.
- p50 is the typical wait; p95 is the planning number. A 300 ms p50 with a 4 s p95 will still generate timeouts in production.
- TTFT and total time diverge with output length. A model with a fast first token and slow generation feels responsive and finishes last.
- Region beats provider. A mid-tier model in the same region usually beats a top-tier model across an ocean for interactive work.
- Queueing is invisible in a single sample. Run at your own concurrency, not at one request at a time, before trusting a published number.
Methodology and limits
Snapshots run monthly, in the first full week, and are published with the raw sample count per cell. The index is not a benchmark of quality: it does not measure accuracy, and a faster model is not a better one. It also does not measure cold starts, provisioned throughput or batch latency. Those are separate products with separate behaviour.
Two biases: one probe host per region measures that path, not every network, and a 30-minute window misses diurnal peaks. The monthly snapshot rotates the hour so the series eventually covers a full day.
Sources
Links checked 20 November 2026. We cite the primary page for each figure. Where a provider publishes a price behind a login, we say so and do not guess.
- OpenAI API pricing — context bands and service tiers used to define the cohort
- Anthropic pricing — model names and regional inference options
- Google Cloud regions — region inventory used to place probes
FAQ
What is TTFT and why measure it separately?
Time to first token is the delay before the first content token arrives. It dominates perceived responsiveness in a streaming interface, and it can be fast while total generation is slow. Measuring only end-to-end latency hides that difference.
Why does the index publish no latency numbers yet?
Because the first measurement window opens on 20 November 2026. Publishing a number before the run would mean inventing it. The page publishes the regions, the 20-model cohort, the harness and the metric definitions so the first snapshot can be reproduced.
How many samples are behind each cell?
Six hundred sequential samples per cell: 200 a day over a 30-minute window, on three consecutive days. Sequential execution avoids the queueing effect that concurrency introduces.
Is a lower p50 always better?
No. A provider can have a low p50 and a heavy tail, which produces timeouts even though the typical request is fast. For production planning, p95 and the error rate matter more than the median.
Does region matter more than model choice?
For interactive workloads, often yes. A mid-tier model in the same region usually returns a first token sooner than a top-tier model across an ocean. Test both before choosing a provider on latency alone.
Use the data, then test it on your own workload
Plugsky publishes its plan caps and model limits. Start free with no card, or compare the flat plans against per-token billing.
Start Free → See plans