Feature · reliability

LLM load balancing for concurrency and failover

Balance traffic across keys, regions and providers without turning a rate limit into a retry storm. The hard part is not the round robin; it is the queue, the health signal and the retry budget.

Short answer. Treat concurrency, RPM and TPM as three separate constraints. Size the in-flight pool with Little's Law, queue only when the wait is shorter than the client deadline, and retry only within a budget. A 429 is a signal to slow down, not to retry immediately.

Concurrency versus RPM versus TPM

LimitWhat it countsWhat it protectsWhat you see when it binds
ConcurrencyRequests in flight at one instantMemory, KV cache, worker slotsQueue wait rises; p95 latency climbs before errors
RPMRequests started per minuteAdmission rate and control-plane load429 with Retry-After
TPMInput plus output tokens per minuteToken throughput and cost429 on long prompts even at low request rate
Daily capRequests or tokens per dayPlan fair use429 with an upgrade link until 00:00 UTC

A fast model can hit RPM at low concurrency; a long prompt can hit TPM on one slot. Track all four counters.

Queue math

Little's Law is the only formula most teams need: in-flight = throughput × latency. If you serve 20 requests per second and each takes 2 seconds, you need 40 requests in flight. If the model slows to 4 seconds, the same traffic needs 80. A pool capped at 40 now has a queue, and the queue adds latency to every request behind it.

ThroughputAverage latencyRequired in-flightIf the pool cap is 40
10 req/s2 s20Headroom; p95 may still spike
20 req/s2 s40At the cap; no room for retries
40 req/s2 s80Queue grows; reject or shed load
20 req/s4 s80Queue grows even at the earlier rate

Size for p95, not the average, and add a separate allowance for retries. If 5% of calls retry once, the effective request rate is 1.05× the original. Queue wait is roughly queue_depth / service_rate; if the wait exceeds the remaining client deadline, reject early rather than spending a worker slot on a response nobody will read.

Health checks and weighted routing

Health is a score, not a boolean. A provider that returns 500s on 30% of calls is not healthy; a provider that is 200 ms slower but has capacity may still be the better choice. Track error rate, p95 latency, 429 rate and timeouts over a rolling window, then apply hysteresis so a single bad minute does not flap the pool.

SignalStarting thresholdAction
5xx rate>5% over 60 sReduce weight by half; probe
429 rate>2% over 60 sLower rate; honour Retry-After
p95 latency>2× baseline for 5 minShift weight; keep a canary
Timeouts>1% over 60 sOpen circuit for 30 s

Examples, not universal thresholds. Weighted routing should combine remaining quota, observed latency and a cost preference. Least-outstanding requests is usually better than round robin because it reacts to slow workers. Consistent hashing helps when prompt caching or session affinity matters.

import asyncio, random

class Pool:
    def __init__(self, name, weight, limit):
        self.name, self.weight, self.limit = name, weight, asyncio.Semaphore(limit)
        self.inflight = 0
        self.healthy = True

    async def call(self, fn):
        async with self.limit:
            self.inflight += 1
            try:
                return await fn()
            finally:
                self.inflight -= 1

def choose(pools):
    live = [p for p in pools if p.healthy]
    # power of two choices: sample two, take the least loaded
    a, b = random.sample(live, 2)
    return min((a, b), key=lambda p: (p.inflight + 1) / p.weight)

Retry budgets and 429 handling

Retries are borrowed capacity. Give them a budget: for example, allow retries equal to 10% of successful requests over a rolling window. When the budget is empty, fail fast. This prevents a provider outage from being amplified by every client at once.

ResponseRetry?Action
429 rate limitYes, within budgetHonour Retry-After; add full jitter
429 daily quotaNoFail fast; alert; wait for reset or upgrade
500 / 502 / 503Yes, onceBackoff; try another healthy pool
400 / 401 / 403 / 404NoFix the request, key or model ID
TimeoutYes, if idempotentUse an idempotency key; cap attempts
import asyncio, random

async def with_retry(call, budget, max_attempts=3):
    for attempt in range(max_attempts):
        try:
            return await call()
        except RateLimited as e:
            if not budget.take():
                raise
            delay = min(8, 0.25 * 2 ** attempt)
            await asyncio.sleep(random.uniform(0, delay))  # full jitter
            if e.retry_after:
                await asyncio.sleep(e.retry_after)
    raise RuntimeError("retry budget exhausted")

Use an Idempotency-Key so a retry does not double-charge; Plugsky caches it for 24 hours. See automatic failover and upstream resilience for the server-side picture, and the rate-limit planning guide for the capacity model.

Latency benchmark

Do not publish a single latency number for a model. Latency depends on prompt length, output length, region, time of day and concurrency. Run a sweep and report the shape.

Test pointWhat to record
Concurrency 1, 4, 8, 16, 32Time to first token, total latency, queue wait
200 requests per pointp50, p95, p99 and error rate
Same prompt setInput tokens, output tokens, model ID
Warm and coldFirst call after deploy versus steady state
Region pairClient region, API region, network RTT

Report queue wait separately. A p95 that doubles at concurrency 16 may be your semaphore, not the provider. The rate-limit calculator turns your RPM, concurrency and token estimates into a headroom plan.

Configuration example

pools:
  - name: primary
    base_url: https://api.plugsky.com/v1
    weight: 70
    concurrency: 16
    rpm: 600
  - name: secondary
    base_url: https://api.plugsky.com/v1
    weight: 30
    concurrency: 8
    rpm: 240
retry:
  budget_ratio: 0.10
  max_attempts: 3
  base_ms: 250
  max_ms: 8000
  retry_on: [429, 500, 502, 503, timeout]
health:
  window_s: 60
  error_rate: 0.05
  p95_factor: 2.0
  probe_interval_s: 30

Keep the config in version control and change one variable at a time; log pool, attempt, queue wait, status and request ID.

FAQ

What is the difference between concurrency and RPM?

Concurrency is the number of requests in flight at one instant. RPM is the number of requests started in a minute. A slow model can exhaust concurrency while staying under RPM, and a fast model can hit RPM while using little concurrency.

How do I size concurrency?

Use Little's Law: in-flight requests equals throughput multiplied by average latency. At 20 requests per second and 2 seconds per request, you need 40 in flight. Add headroom for p95 latency and retries, not just the average.

Should I retry a 429?

Only with a retry budget, exponential backoff with jitter, and respect for Retry-After. A blind retry loop turns a rate limit into a retry storm. If the 429 is a plan quota with no reset before the next day, fail fast instead.

How do I benchmark LLM latency?

Run the same prompt set at concurrency 1, 4, 8, 16 and 32. Record time to first token, total latency, queue wait and error rate at each point, then report p50, p95 and p99. Do not compare a warm single request with a cold concurrent run.

Does Plugsky do automatic failover?

Plugsky documents automatic failover for smart-routed requests and a 502/503 response after upstream failure. Client-side balancing is still needed across regions and providers. Verify the current failover behavior and SLA in the contract.

Sources

AWS Builders' Library on timeouts, retries and backoff with jitter, checked 20 November 2026: aws.amazon.com/builders-library. Google SRE Book, Handling Overload, checked 20 November 2026: sre.google/sre-book/handling-overload. LiteLLM routing documentation, checked 20 November 2026: docs.litellm.ai/docs/routing. OpenAI rate-limit guide for 429 and Retry-After, checked 20 November 2026: developers.openai.com/api/docs/guides/rate-limits.

Reviewed 20 November 2026. Thresholds are examples, not vendor guarantees. Verify plan limits, failover behaviour and SLAs in the current Plugsky documentation and contract before relying on them in production.