Key facts
| Failover layer | Plugsky gateway reroutes to a healthy upstream when a provider fails |
| Upstream down | 503 — smart-routed requests automatically fail over |
| Provider failure | 502 after auto-failover — retry with backoff, then check /status |
| Safe retries | Idempotency-Key on POST endpoints returns the cached result for 24 hours |
| Rate limiting | 429 includes Retry-After; SDKs retry with exponential backoff |
| Embedding resilience | Same-profile peers in the embed family serve as fallbacks |
| Visibility | Per-request logs include model, region, latency, status and request ID |
| Product status | Live |
TL;DR
- Failover happens at the gateway — the request shape never changes.
- 502 and 503 are the two signals to distinguish provider recovery from a retryable failure.
- Send Idempotency-Key on POSTs so a retry cannot double-execute work.
- Honor Retry-After; let SDK backoff handle the common case.
- Track which model answered each request — failover can change it.
How it works, step by step
- Wire client-side retries with exponential backoff and jitter for 429, 500, 502 and 503.
- Add Idempotency-Key to every POST so retries return the cached result instead of repeating work.
- Honor Retry-After exactly on 429 rather than retrying immediately.
- Record the model field from each response so failover-driven substitutions are visible.
- Set a retry budget per request to avoid amplifying an incident.
- Alert on sustained 5xx rates and subscribe to the status page for upstream incidents.
- For residency-critical workloads, pin a region and test the failover posture during a pilot.
Original data
Try it yourself
Open the OpenAI-compatible API tester →
Where failover sits in the request path
Your application talks to one OpenAI-compatible endpoint. Behind it, the Plugsky gateway selects an upstream provider for the requested model and monitors health. If the provider fails mid-flight or is unavailable, the gateway reroutes to a healthy peer when one exists for that model class. Because the contract stays identical, failover requires no code change — but it does mean the model field in a response may name the peer that actually answered, so log it.
Reading 502 and 503 correctly
The two status codes answer different questions:
- 503 — upstream down: smart-routed requests automatically fail over; retry with backoff if you still receive it.
- 502 — provider failure after auto-failover: the gateway tried to recover and could not; retry with backoff and open a ticket if it persists.
- 500 — internal error: retry with backoff; persistent errors deserve a support ticket with the request ID.
- 429 — rate limit: the upstream is healthy but you are over your request rate; honor
Retry-After.
All API errors share the OpenAI error shape, so a single parser handles every provider and the gateway itself.
Retries that are actually safe
A retry is only safe when the server can recognise the duplicate. Plugsky supports an Idempotency-Key header on POST endpoints: resending the same key returns the cached result for 24 hours, so a timeout followed by a retry cannot produce two completions or two batch submissions.
curl -X POST https://api.plugsky.com/v1/chat/completions \
-H "Authorization: Bearer $PLUGSKY_API_KEY" \
-H "Idempotency-Key: $(uuidgen)" \
-H "Content-Type: application/json" \
-d '{"model":"plugsky-pro","messages":[{"role":"user","content":"hello"}]}'Use a fresh key per logical request, cap total attempts, and add jitter so a fleet of workers does not retry in lockstep.
Designing for resilience beyond the gateway
- Model fallbacks: define a secondary model for each critical workload and switch on sustained errors.
- Region strategy: pin the region that satisfies residency and confirm recovery behaviour during onboarding.
- Deployment tiers: VPC, on-prem and air-gapped deployments remove internet dependencies for regulated workloads.
- Observability: export request logs to your APM or SIEM and alert on error-rate and latency SLOs, not only outages.
- Status awareness: subscribe to incident history on the status page so your on-call knows when a 502 is platform-wide.
Failover protects availability, not correctness. Keep user-visible fallbacks — cached answers, queued jobs, a clear degraded-mode message — for the cases where no upstream is healthy.
Honest comparison
| Capability | Plugsky | Single-provider API | Self-hosted gateway |
|---|---|---|---|
| Upstream failover | Automatic rerouting to healthy peers | None — provider outage is your outage | You build and operate it |
| Error contract | OpenAI schema for gateway and providers | Provider-specific semantics | Yours to define |
| Idempotent retries | Idempotency-Key cached for 24 hours | Varies by endpoint | Custom implementation |
| Region choice | Pinned regions plus VPC/on-prem options | Provider regions | Whatever you deploy |
| Ops burden | Managed | Low | High |
| Visibility | Per-request logging and status page | Provider status page | Custom telemetry |
Frequently asked questions
Does failover change my request or response format?
No. Requests and responses keep the OpenAI-compatible shape. The model field may identify the peer that answered, so log it for observability.
What is the difference between 502 and 503?
503 means an upstream is down and smart-routed requests can automatically fail over. 502 means a provider failed after failover was attempted — retry with backoff and check the status page.
Are retries safe on chat completions?
Yes, when you send an Idempotency-Key on the POST. Reusing the same key returns the cached result for 24 hours instead of executing the request twice.
How should I handle 429 responses?
Honor the Retry-After header. Plugsky SDKs already retry with exponential backoff; custom clients should add jitter and a maximum attempt count.
What happens to embeddings during an incident?
Requests fall back to same-profile peers in the embed family automatically. Live component health is published on the status page.
Can I require that traffic never leaves a region?
Yes. Pin the workspace region and, for stricter requirements, move to a VPC, on-prem or air-gapped deployment where the data path is fully inside your control.
How do I know failover happened?
The response model field and per-request logs (model, region, latency, status, request ID) show which upstream answered, and the status page shows platform incidents.
Does failover affect billing?
Self-serve plans are flat monthly with fair-use usage rather than per-token billing, so a retried request does not create a per-token charge. Idempotency keys prevent duplicate work.