Data · Performance

LLM latency index by provider and region

TTFT, p50, p95 and output tokens per second across 10 regions and 20 models — with the harness published before the first number, so the method can be checked.

P
Plugsky Data desk
Baseline 20 November 2026 · reviewed 20 November 2026

Short answer. Latency is not one number. Time to first token (TTFT), end-to-end p50 and p95, and output tokens per second measure different things, and a provider can win one while losing another. This index fixes the harness, the regions and the model cohort, then publishes monthly snapshots. The first snapshot window opens on 20 November 2026; no latency figures are published before the first completed run.

Baseline: 20 November 2026. This edition publishes the measurement matrix, the harness and the metric definitions. Publishing numbers we did not measure would defeat the purpose of an index.

What the index measures

MetricDefinitionWhy it matters
TTFT p50Milliseconds from request sent to first content token receivedPerceived responsiveness in a streaming UI
TTFT p9595th percentile of the same measurementThe wait your slowest users remember
End-to-end p50Request sent to final token, at a fixed output lengthBatch and agent-step cost in wall-clock time
End-to-end p9595th percentile of the sameTimeout and retry budget
Output tokens/secOutput tokens divided by generation time after the first tokenThroughput for long answers
Error rateNon-2xx and malformed responses, by classA fast 429 is not a fast success

Every sample uses streaming, a fixed prompt of 1,000 input tokens, max_tokens of 300 and temperature 0. Input length is fixed because TTFT grows with prompt size; output length is fixed because tokens per second is only comparable at a known generation length.

The 10 regions

Probes run from one host per region, in the same cloud region as the label. The provider’s nearest endpoint is used; if a provider has no endpoint in a region, the cell is left empty rather than measured over a transatlantic hop and labelled local.

RegionProbe locationNearest endpoint assumption
us-eastVirginia, USUS East
us-westOregon, USUS West
ca-centralMontréal, CanadaCanada Central
eu-westIrelandEU West
eu-centralFrankfurt, GermanyEU Central
uk-southLondon, UKUK South
ap-southMumbai, IndiaAsia South
ap-southeastSingaporeAsia Southeast
ap-northeastTokyo, JapanAsia Northeast
sa-eastSão Paulo, BrazilSouth America East

Ten regions covers the three largest API markets plus India, Japan, Singapore and Brazil. It does not cover Africa, the Middle East or Australia; those are the first candidates for expansion.

The 20-model cohort

Twelve Plugsky models and eight external baselines. The cohort is fixed for the quarter so month-over-month changes reflect the provider, not a new model. A retirement replaces a model at the next quarter boundary and the change is logged.

ModelProviderContextRole in the cohort
plugsky-plusPlugsky262,144General text
plugsky-proPlugsky262,144General text
plugsky-maxPlugsky262,144Top quality
plugsky-ultraPlugsky262,144Top quality
plugsky-thinkerPlugsky262,144Reasoning
plugsky-kimiPlugsky262,144Long context
plugsky-qwen-nextPlugsky262,144Efficient text
plugsky-deepseek-proPlugsky262,144Reasoning
plugsky-coderPlugsky262,144Code
plugsky-parsePlugsky262,144Vision and documents
plugsky-vision-fastPlugsky131,072Vision, latency-sensitive
plugsky-translatePlugsky32,768Translation
gpt-6-lunaOpenAI272K shortLow-cost baseline
gpt-6.2-solOpenAI272K shortMid-tier baseline
claude-haiku-5-5AnthropicSee model pageLow-cost baseline
claude-sonnet-5-5AnthropicSee model pageMid-tier baseline
claude-opus-5-5AnthropicSee model pageTop-tier baseline
Gemini 2.5 ProGoogle1,000,000Long-context baseline
DeepSeek V3DeepSeek128,000Open-weight baseline
Llama 4 MaverickMeta (hosted)1,000,000Open-weight baseline

The harness

The harness is deliberately boring: one process per region, a fixed prompt, streaming responses, and a timestamp on every chunk. It records TTFT, total time, token count and the HTTP status. It does not retry on latency, because a retry turns a slow sample into a fast one and hides the tail.

import time, json, urllib.request

def sample(base_url, api_key, model, prompt):
    body = json.dumps({
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "max_tokens": 300,
        "temperature": 0,
        "stream": True,
    }).encode()
    req = urllib.request.Request(
        base_url.rstrip("/") + "/chat/completions",
        data=body,
        headers={"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"},
    )
    t0 = time.perf_counter()
    ttft = None
    chunks = 0
    with urllib.request.urlopen(req, timeout=60) as r:
        for line in r:
            if not line.startswith(b"data: "):
                continue
            payload = line[6:].strip()
            if payload == b"[DONE]":
                break
            delta = json.loads(payload)["choices"][0]["delta"].get("content")
            if delta:
                chunks += 1
                if ttft is None:
                    ttft = time.perf_counter() - t0
    total = time.perf_counter() - t0
    return {"ttft_ms": round(ttft * 1000), "total_ms": round(total * 1000), "chunks": chunks}

Each cell is 600 sequential samples: 200 per day over a 30-minute window, at the same hour, on three consecutive days. Sequential rather than parallel, because concurrency changes queueing and therefore latency. The reported p95 is the p95 of the pooled samples, not the average of daily p95s.

Why the first snapshot has no numbers

A latency index is only useful if the method survives contact with the provider. Three things have to be true before a number is publishable: the model must accept the fixed prompt, the response must stream, and the run must complete without a rate limit. On the first pass, several of the 200 cells will fail one of those tests, and a failed cell must be shown as failed rather than dropped.

Publishing the matrix first lets readers challenge the design before the numbers harden. Fixing a wrong region now is cheaper than explaining a revision later.

  • p50 is the typical wait; p95 is the planning number. A 300 ms p50 with a 4 s p95 will still generate timeouts in production.
  • TTFT and total time diverge with output length. A model with a fast first token and slow generation feels responsive and finishes last.
  • Region beats provider. A mid-tier model in the same region usually beats a top-tier model across an ocean for interactive work.
  • Queueing is invisible in a single sample. Run at your own concurrency, not at one request at a time, before trusting a published number.

Methodology and limits

Snapshots run monthly, in the first full week, and are published with the raw sample count per cell. The index is not a benchmark of quality: it does not measure accuracy, and a faster model is not a better one. It also does not measure cold starts, provisioned throughput or batch latency. Those are separate products with separate behaviour.

Two biases: one probe host per region measures that path, not every network, and a 30-minute window misses diurnal peaks. The monthly snapshot rotates the hour so the series eventually covers a full day.

Sources

Links checked 20 November 2026. We cite the primary page for each figure. Where a provider publishes a price behind a login, we say so and do not guess.

FAQ

What is TTFT and why measure it separately?

Time to first token is the delay before the first content token arrives. It dominates perceived responsiveness in a streaming interface, and it can be fast while total generation is slow. Measuring only end-to-end latency hides that difference.

Why does the index publish no latency numbers yet?

Because the first measurement window opens on 20 November 2026. Publishing a number before the run would mean inventing it. The page publishes the regions, the 20-model cohort, the harness and the metric definitions so the first snapshot can be reproduced.

How many samples are behind each cell?

Six hundred sequential samples per cell: 200 a day over a 30-minute window, on three consecutive days. Sequential execution avoids the queueing effect that concurrency introduces.

Is a lower p50 always better?

No. A provider can have a low p50 and a heavy tail, which produces timeouts even though the typical request is fast. For production planning, p95 and the error rate matter more than the median.

Does region matter more than model choice?

For interactive workloads, often yes. A mid-tier model in the same region usually returns a first token sooner than a top-tier model across an ocean. Test both before choosing a provider on latency alone.

Use the data, then test it on your own workload

Plugsky publishes its plan caps and model limits. Start free with no card, or compare the flat plans against per-token billing.

Start Free → See plans