LLM API latency: 30 models, measured live

Method

On 7 August 2026 we ran real inference requests against every model on the public Plugsky API (api.plugsky.com/v1/chat/completions) with a production API key. Each model received the same prompt ("Reply with exactly: HELLO"), max_tokens=20, non-streaming. Latency was measured end-to-end from request to final response. All 30 chat models were also probed by the automated watchdog (/api/health/models), and 3 embedding models were tested with both English and Arabic input.

Results: 30/30 chat models responded

Every chat model returned a valid completion. Median latencies by tier:

Speed tier	Models	Median latency
Fastest	plugsky-phi, plugsky-gemma3-nano-2b, plugsky-lite, plugsky-gemma3-nano-4b, plugsky-gemma-4, plugsky-vision-fast, plugsky-qwen-vl	0.23–0.30s
Fast	plugsky-micro, plugsky-kimi, plugsky-minimax, plugsky-tiny, plugsky-coder-fast, plugsky-mistral-small, plugsky-coder-ultra, plugsky-thinker	0.46–0.78s
Mid	plugsky-pro, plugsky-frontier, plugsky-reasoning, plugsky-gpt-oss, plugsky-qwen-next, plugsky-coder, plugsky-nano, plugsky-longctx, plugsky-ultra, plugsky-plus	0.67–0.93s
Slower (OK)	plugsky-llama4 (1.4s), plugsky-deepseek-pro (3.4s), plugsky-mistral-medium (12.1s)	1.4–12.1s

Full per-model measurements are published in the audit appendix of this report. Latency varies with load and prompt length; these are single-run medians on 2026-08-07.

Health: 30/30 healthy, 0 broken

The watchdog probe (/api/health/models, every 5 minutes, 80-token prompt) reported 30/30 chat models healthy, 0 failed, 0 slow, 0 rate-limited. Four models currently run on watchdog-applied upstream overrides after their original upstreams were retired by NVIDIA NIM — the watchdog swapped them to healthy same-profile peers automatically, and users saw no interruption.

Fallback: forced-failure test

We forced one model's primary upstream to a nonexistent endpoint. The request still returned 200 with content in 0.93s, routed automatically to a healthy fallback upstream. Session history was preserved through the failover (a 10-message conversation correctly recalled earlier context). Embeddings similarly failed over: plugsky-embed, plugsky-embed-nim, and plugsky-embed-multilingual all returned valid vectors (2,048 / 4,096 / 4,096 dimensions) including Arabic input.

What this means for buyers

• Independent AI clouds can be fast — sub-300ms responses are achievable on hosted open-weight models.
• Failover is real, not marketing — automatic cascade kept a request alive through a total upstream failure.
• Arabic support is measurable — multilingual embeddings handle Arabic input on the same endpoint as English.

Appendix: full measurements (2026-08-07)

Model	Latency	Result
plugsky-micro	0.55s	OK
plugsky-lite	0.24s	OK
plugsky-plus	0.80s	OK
plugsky-pro	0.67s	OK
plugsky-max	0.71s	OK
plugsky-frontier	0.70s	OK
plugsky-reasoning	0.71s	OK
plugsky-kimi	0.60s	OK
plugsky-deepseek-pro	3.40s	OK
plugsky-deepseek-flash	0.73s	OK
plugsky-gpt-oss	0.70s	OK
plugsky-qwen-next	0.82s	OK
plugsky-coder	0.67s	OK
plugsky-minimax	0.53s	OK
plugsky-vision-fast	0.30s	OK
plugsky-llama4	1.39s	OK
plugsky-qwen-vl	0.30s	OK
plugsky-longctx	0.81s	OK
plugsky-mistral-medium	12.10s	OK
plugsky-gemma-4	0.28s	OK
plugsky-nano	0.85s	OK
plugsky-tiny	0.62s	OK
plugsky-coder-fast	0.71s	OK
plugsky-phi	0.23s	OK
plugsky-gemma3-nano-2b	0.23s	OK
plugsky-gemma3-nano-4b	0.27s	OK
plugsky-mistral-small	0.61s	OK
plugsky-ultra	0.93s	OK
plugsky-thinker	0.78s	OK
plugsky-coder-ultra	0.46s	OK