How do you build AI routing with an OpenAI-compatible API?

Key facts

Endpoint	POST https://api.plugsky.com/v1/chat/completions, the same shape as OpenAI chat completions
Routing mechanism	The model field selects the alias; payload stays OpenAI-compatible
Migration	Keep the OpenAI SDK; change base_url and model names
Models	30+ models behind one endpoint, from free tiers to frontier reasoning
Fallbacks	Retry with a different alias on transport error or schema validation failure
Pricing model	Flat monthly self-serve plans with unlimited fair-use usage
Deployment	Plugsky cloud, VPC, on-prem and air-gapped options
Roadmap	Batch and fine-tuning endpoints are coming soon

TL;DR

• Routing is a config decision: the same payload goes to different aliases.
• Start with static rules for known call sites, then add a classifier for the long tail.
• Shadow routing first; let the router write logs without controlling traffic.
• Version your routing table and roll back by reverting one config file.
• OpenAI compatibility keeps your SDK, tests and retry logic intact.

How it works, step by step

1. Change the base URL to Plugsky and confirm existing SDK calls work unchanged.
2. Create a versioned routing table mapping workload names to model aliases, timeouts and output caps.
3. Implement static rules for known call sites and tenants.
4. Add a cheap classifier route for unclassified traffic using a JSON schema.
5. Configure a fallback chain: primary alias, secondary alias, then a typed error.
6. Run shadow mode for several days, compare per-route quality, then flip traffic route by route.

Try it yourself

Open the AI model pricing calculator →

Architecture: a gateway in front of the SDK

You do not need a new SDK or service. A routing gateway is a module that wraps the client:

• Config: a versioned table from workload to alias, timeout, max_tokens and schema.
• Decision: rules first, classifier second, memoized per request so one request never runs two routers.
• Execution: call Plugsky through the existing OpenAI-compatible client.
• Fallback: on transport error or invalid JSON, retry once with the next alias and mark the request.
• Telemetry: log workload, alias, latency, validation result and escalation marker.

Because only the model field changes, you can test an alias against production traffic without touching application code.

From static rules to semantic routing

Mature routers grow in stages. Each stage should be independently revertible:

1. Rules: map features and tenants to aliases. Audit-friendly and instant.
2. Classifier: a JSON-mode call to plugsky-micro returning tier and confidence for unclassified traffic.
3. Semantic: embed queries with plugsky-embed and match labeled examples when latency budgets allow.
4. Cascade: attempt the cheap alias, validate, escalate once to plugsky-pro or plugsky-frontier.

Keep the routing table small enough to review. A gateway with forty routes is harder to reason about than forty services with clear contracts.

Evaluation and safe rollout

Route quality is measurable if you log the counterfactual:

• Shadow mode: run both the current model and the proposed route, diff outputs and score them offline.
• Per-route quality: maintain a frozen evaluation set per workload so cheap aliases keep their pass rates.
• Latency budget: report routing overhead separately from model latency.
• Rollback drill: rehearse reverting the config under load; a routing table you cannot roll back is an outage waiting to happen.
• Canary: migrate one tenant or 5% of traffic, watch error and escalation rates, then expand.

Limitations and caveats

Routing does not make models interchangeable. Context windows, tool support and JSON reliability differ across the catalogue, so a route must carry its own prompt and validation rules.

• Region and deployment constraints apply to routing too: do not send regulated data to an alias hosted in the wrong region.
• Client-side routing lives in your process, so deploy the table with your app and version it alongside code.
• Fallback chains can mask persistent failures; alert on fallback rate instead of treating it as normal.
• Batch and fine-tuning endpoints are coming soon; build current routing on chat completions and embeddings.
• If you already run a gateway, keep one router of record. Two routing layers produce unverifiable quality data.

Honest comparison

Capability	Plugsky routing gateway	Static SDK calls	Multi-vendor gateway
Integration	One base URL, alias per route	Hard-coded model per call	Per-provider adapters
Model access	30+ models behind one endpoint	Only what you wired	Vendor-dependent
Fallbacks	Alias chain on errors and schema failures	Manual retries	Usually supported
Rollout	Shadow mode and config rollback	Deploy code to change	Gateway config
Governance	Scoped keys, audit logs, region pinning	Your application only	Varies by gateway

Frequently asked questions

Do I need a separate routing service?

No. A module wrapping your existing OpenAI-compatible client is enough; the routing table maps workload names to Plugsky model aliases.

How does the model field work?

Each request names a Plugsky model alias in the standard model field. Changing the alias changes the route without altering the payload shape.

What is shadow routing?

Running the proposed route alongside the live one and logging both outputs, so you can score the new route offline before it controls production traffic.

How many fallbacks should a chain have?

One or two. A primary alias, a secondary alias and a typed error is easier to reason about than a long chain that hides failures.

Can routing respect data residency?

Yes, but only if you encode residency as a hard constraint in the routing table. Never let a classifier choose a region for regulated data.

Does this replace a model gateway?

It can for most teams. If you already run a gateway, keep that as the single router of record rather than stacking two routing layers.

Can I evaluate routing on the free plan?

Yes. plugsky-micro and plugsky-lite are free with no card, and a 14-day full-access trial lets you compare paid aliases on real traffic.