What happens if an upstream model fails?

Key facts

Error behaviour	Standard HTTP status codes in OpenAI-compatible error shape
Automatic substitution	Not on self-serve — no silent model swaps
Application fallback	Retry, switch model, or degrade gracefully
Enterprise failover	Gateway-level options in dedicated deployments
Status visibility	Component health published on the status page
Rate limiting	429 responses with retry guidance
Multi-region	Available in enterprise deployments
Product status	Live

TL;DR

• A failure surfaces as a standard API error, not a wrong answer.
• On self-serve, no other model is silently substituted for yours.
• Your application owns retries, fallback and graceful degradation.
• Enterprise gateways can add automatic failover by policy.
• Design for failure with timeouts, caps and model alternates from day one.

How it works, step by step

1. Handle error classes distinctly: retryable (429, 5xx) versus terminal (4xx schema errors).
2. Use exponential backoff with jitter for retries, and cap total attempts.
3. Define an alternate model per critical path and switch on repeated failure.
4. Add timeouts and a degraded response path so UX fails softly, not silently.
5. Subscribe to status and alert on error-rate and latency anomalies.
6. Rehearse failures: inject model errors in staging before you need the logic in production.

Try it yourself

Open the OpenAI error decoder →

The failure modes you will actually see

Model failures are less dramatic than an outage and more frequent than you expect:

• Rate limits: 429 responses under burst load; retryable with backoff.
• Upstream errors: transient 5xx responses from model infrastructure; retryable.
• Timeouts: requests that exceed your client deadline, common with long generations.
• Invalid requests: schema or parameter errors that retries will not fix; fix and surface to the caller.
• Quality failures: a technically successful response that fails validation — the most dangerous kind, because nothing errors.

Treat the last category as a failure mode too: validate outputs and route to a stronger model or a safe fallback when validation fails.

Fallback patterns that work

Keep fallback explicit and simple:

1. Retry the same model with exponential backoff and jitter for transient errors.
2. Switch to a designated alternate after one or two failures — pick a model with similar behaviour so output stays consistent.
3. Degrade: return a cached or simplified answer, or queue the request, rather than failing the user experience outright.
4. Fail loudly when nothing safe is available, with a clear error and an alert.

Enterprise deployments can move failover into the gateway where policy allows. Either way, log which model served each request so incidents are diagnosable after the fact.

What we do and what we do not do

What we do: return standard errors promptly, publish component health on the status page, apply rate limits predictably, and support gateway-level failover in enterprise deployments. What we do not do: silently swap models on self-serve — a different model can change formatting, tone and correctness, which is worse than an honest error. That also means uptime is partly your design responsibility: applications that retry sensibly and hold a fallback path survive upstream hiccups that take down naive ones.

Honest comparison

Behaviour	Plugsky self-serve	Enterprise gateway	Typical multi-provider setup
Error signalling	Standard API error	Standard API error	Reworked per provider
Retry logic	Your application	Gateway plus app	You build across providers
Model substitution	None — explicit only	Policy-based	Manual routing
Degradation path	You design it	You design it	You design it
Status visibility	Status page	Status page plus contract	Multiple dashboards
Complexity	Low	Moderate	High

Frequently asked questions

Will Plugsky automatically use another model if mine fails?

Not on self-serve. Silent substitution changes output behaviour without notice, so failover is explicit — your application chooses whether to retry or switch.

What error format should I expect?

Standard HTTP status codes with an error body in the OpenAI-compatible shape, so existing error handling works unchanged.

Which errors are safe to retry?

429 rate limits and 5xx upstream errors are retryable with exponential backoff. Request validation errors are terminal and should be fixed, not retried.

How many retries should I attempt?

Two or three with backoff and jitter covers most transient failures. Then switch to an alternate model or degrade, rather than looping.

How do I pick a fallback model?

Choose one with similar behaviour for the same task and validate it against your evaluation set, so fallback responses stay consistent.

Is there multi-region failover?

It is available in enterprise deployments scoped with the team. On self-serve, region behaviour depends on the deployment you use.

How do I monitor model health?

Watch the status page, alert on your own error and latency metrics, and log the serving model per request so incidents can be attributed quickly.