Short answer. Treat concurrency, RPM and TPM as three separate constraints. Size the in-flight pool with Little's Law, queue only when the wait is shorter than the client deadline, and retry only within a budget. A 429 is a signal to slow down, not to retry immediately.
Concurrency versus RPM versus TPM
| Limit | What it counts | What it protects | What you see when it binds |
|---|---|---|---|
| Concurrency | Requests in flight at one instant | Memory, KV cache, worker slots | Queue wait rises; p95 latency climbs before errors |
| RPM | Requests started per minute | Admission rate and control-plane load | 429 with Retry-After |
| TPM | Input plus output tokens per minute | Token throughput and cost | 429 on long prompts even at low request rate |
| Daily cap | Requests or tokens per day | Plan fair use | 429 with an upgrade link until 00:00 UTC |
A fast model can hit RPM at low concurrency; a long prompt can hit TPM on one slot. Track all four counters.
Queue math
Little's Law is the only formula most teams need: in-flight = throughput × latency. If you serve 20 requests per second and each takes 2 seconds, you need 40 requests in flight. If the model slows to 4 seconds, the same traffic needs 80. A pool capped at 40 now has a queue, and the queue adds latency to every request behind it.
| Throughput | Average latency | Required in-flight | If the pool cap is 40 |
|---|---|---|---|
| 10 req/s | 2 s | 20 | Headroom; p95 may still spike |
| 20 req/s | 2 s | 40 | At the cap; no room for retries |
| 40 req/s | 2 s | 80 | Queue grows; reject or shed load |
| 20 req/s | 4 s | 80 | Queue grows even at the earlier rate |
Size for p95, not the average, and add a separate allowance for retries. If 5% of calls retry once, the effective request rate is 1.05× the original. Queue wait is roughly queue_depth / service_rate; if the wait exceeds the remaining client deadline, reject early rather than spending a worker slot on a response nobody will read.
Health checks and weighted routing
Health is a score, not a boolean. A provider that returns 500s on 30% of calls is not healthy; a provider that is 200 ms slower but has capacity may still be the better choice. Track error rate, p95 latency, 429 rate and timeouts over a rolling window, then apply hysteresis so a single bad minute does not flap the pool.
| Signal | Starting threshold | Action |
|---|---|---|
| 5xx rate | >5% over 60 s | Reduce weight by half; probe |
| 429 rate | >2% over 60 s | Lower rate; honour Retry-After |
| p95 latency | >2× baseline for 5 min | Shift weight; keep a canary |
| Timeouts | >1% over 60 s | Open circuit for 30 s |
Examples, not universal thresholds. Weighted routing should combine remaining quota, observed latency and a cost preference. Least-outstanding requests is usually better than round robin because it reacts to slow workers. Consistent hashing helps when prompt caching or session affinity matters.
import asyncio, random
class Pool:
def __init__(self, name, weight, limit):
self.name, self.weight, self.limit = name, weight, asyncio.Semaphore(limit)
self.inflight = 0
self.healthy = True
async def call(self, fn):
async with self.limit:
self.inflight += 1
try:
return await fn()
finally:
self.inflight -= 1
def choose(pools):
live = [p for p in pools if p.healthy]
# power of two choices: sample two, take the least loaded
a, b = random.sample(live, 2)
return min((a, b), key=lambda p: (p.inflight + 1) / p.weight)
Retry budgets and 429 handling
Retries are borrowed capacity. Give them a budget: for example, allow retries equal to 10% of successful requests over a rolling window. When the budget is empty, fail fast. This prevents a provider outage from being amplified by every client at once.
| Response | Retry? | Action |
|---|---|---|
| 429 rate limit | Yes, within budget | Honour Retry-After; add full jitter |
| 429 daily quota | No | Fail fast; alert; wait for reset or upgrade |
| 500 / 502 / 503 | Yes, once | Backoff; try another healthy pool |
| 400 / 401 / 403 / 404 | No | Fix the request, key or model ID |
| Timeout | Yes, if idempotent | Use an idempotency key; cap attempts |
import asyncio, random
async def with_retry(call, budget, max_attempts=3):
for attempt in range(max_attempts):
try:
return await call()
except RateLimited as e:
if not budget.take():
raise
delay = min(8, 0.25 * 2 ** attempt)
await asyncio.sleep(random.uniform(0, delay)) # full jitter
if e.retry_after:
await asyncio.sleep(e.retry_after)
raise RuntimeError("retry budget exhausted")
Use an Idempotency-Key so a retry does not double-charge; Plugsky caches it for 24 hours. See automatic failover and upstream resilience for the server-side picture, and the rate-limit planning guide for the capacity model.
Latency benchmark
Do not publish a single latency number for a model. Latency depends on prompt length, output length, region, time of day and concurrency. Run a sweep and report the shape.
| Test point | What to record |
|---|---|
| Concurrency 1, 4, 8, 16, 32 | Time to first token, total latency, queue wait |
| 200 requests per point | p50, p95, p99 and error rate |
| Same prompt set | Input tokens, output tokens, model ID |
| Warm and cold | First call after deploy versus steady state |
| Region pair | Client region, API region, network RTT |
Report queue wait separately. A p95 that doubles at concurrency 16 may be your semaphore, not the provider. The rate-limit calculator turns your RPM, concurrency and token estimates into a headroom plan.
Configuration example
pools:
- name: primary
base_url: https://api.plugsky.com/v1
weight: 70
concurrency: 16
rpm: 600
- name: secondary
base_url: https://api.plugsky.com/v1
weight: 30
concurrency: 8
rpm: 240
retry:
budget_ratio: 0.10
max_attempts: 3
base_ms: 250
max_ms: 8000
retry_on: [429, 500, 502, 503, timeout]
health:
window_s: 60
error_rate: 0.05
p95_factor: 2.0
probe_interval_s: 30
Keep the config in version control and change one variable at a time; log pool, attempt, queue wait, status and request ID.
FAQ
What is the difference between concurrency and RPM?
Concurrency is the number of requests in flight at one instant. RPM is the number of requests started in a minute. A slow model can exhaust concurrency while staying under RPM, and a fast model can hit RPM while using little concurrency.
How do I size concurrency?
Use Little's Law: in-flight requests equals throughput multiplied by average latency. At 20 requests per second and 2 seconds per request, you need 40 in flight. Add headroom for p95 latency and retries, not just the average.
Should I retry a 429?
Only with a retry budget, exponential backoff with jitter, and respect for Retry-After. A blind retry loop turns a rate limit into a retry storm. If the 429 is a plan quota with no reset before the next day, fail fast instead.
How do I benchmark LLM latency?
Run the same prompt set at concurrency 1, 4, 8, 16 and 32. Record time to first token, total latency, queue wait and error rate at each point, then report p50, p95 and p99. Do not compare a warm single request with a cold concurrent run.
Does Plugsky do automatic failover?
Plugsky documents automatic failover for smart-routed requests and a 502/503 response after upstream failure. Client-side balancing is still needed across regions and providers. Verify the current failover behavior and SLA in the contract.
Sources
AWS Builders' Library on timeouts, retries and backoff with jitter, checked 20 November 2026: aws.amazon.com/builders-library. Google SRE Book, Handling Overload, checked 20 November 2026: sre.google/sre-book/handling-overload. LiteLLM routing documentation, checked 20 November 2026: docs.litellm.ai/docs/routing. OpenAI rate-limit guide for 429 and Retry-After, checked 20 November 2026: developers.openai.com/api/docs/guides/rate-limits.
Reviewed 20 November 2026. Thresholds are examples, not vendor guarantees. Verify plan limits, failover behaviour and SLAs in the current Plugsky documentation and contract before relying on them in production.