Key facts
| Definition | Directing requests to different models based on task, cost, latency or capability |
| Why it matters | Avoids overpaying with frontier models for simple work and under-serving hard tasks |
| Routing signals | Task classification, prompt length, required tools, confidence and fallback errors |
| Control point | One OpenAI-compatible endpoint; the model parameter selects the target |
| Strategy order | Static rules first, then measured escalation and learned routing |
| Evaluation | Per-task acceptance, latency and spend instead of a single global score |
| Pricing model | Flat monthly self-serve plans with unlimited fair-use usage |
| Status | Chat, streaming, function calling and embeddings are live |
TL;DR
- Route by task type before you route by cleverness.
- Keep rules explicit and versioned so behavior is debuggable.
- Escalate on failure signals, not on every request.
- Log the routing decision with the outcome to tune it.
- One API, 30+ models — the router is configuration, not a rewrite.
How it works, step by step
- Classify incoming requests by task type, expected output and latency budget.
- Map each class to a model tier: fast, coding, long-context or reasoning.
- Implement the router as configuration that returns a model name and parameters.
- Define escalation triggers — validation failure, low confidence, explicit retry.
- Log model, latency, tokens and outcome for every routed request.
- Re-evaluate monthly and promote cheaper models when they meet the quality bar.
Try it yourself
Open the AI workload router simulator →
How routing works
A router sits between your application and the model API. It inspects the request — task type, prompt size, whether tools are needed — and returns a model name and parameters. Static rules cover most needs. More advanced routers use a small classifier or heuristics based on prompt features, and the most mature systems learn from logged outcomes which model succeeds per request class. The router never changes application code; it changes configuration.
Routing strategies
- Task-based: summarize to a fast alias, refactor to a coding alias, analyze with reasoning.
- Length-based: short prompts to small models, long contexts to extended-window models.
- Cost-capped: start cheap and escalate only on failure.
- Latency-aware: interactive requests get fast models; batch jobs can use stronger ones.
- Confidence-based: route uncertain outputs to a verifier or stronger model.
Combine strategies carefully — every added rule increases the surface you must evaluate.
Common mistakes
- Routing on vibes rather than logged outcomes.
- Escalating every request, which removes the savings routing exists to create.
- No fallback when the chosen model errors or times out.
- Ignoring capability differences — routing a tool-calling task to a model without tools.
- Letting the routing table drift undocumented, so nobody knows why a model is used.
Routing on Plugsky
Because Plugsky exposes 30+ models through one OpenAI-compatible endpoint, a router is a thin function that picks a model name. Your SDK stays the same; there is no second integration to maintain. Use flat monthly pricing to keep cost predictable as routing spreads traffic across tiers, and private deployment options when routing must happen inside your network. Evaluate per task, and keep a written routing table so the team understands each route's purpose.
Honest comparison
| Strategy | Signal | Cost effect | Complexity |
|---|---|---|---|
| Single model | None | Predictable, often wasteful | Lowest |
| Static task rules | Task type | Large savings | Low |
| Length-based | Token count | Moderate savings | Low |
| Cost-capped escalation | Failure signal | Savings with retry cost | Moderate |
| Learned routing | Logged outcomes | Highest potential | High |
Frequently asked questions
What is model routing?
Directing each request to a specific model based on task type, cost, latency or capability, rather than sending everything to one model.
Is routing the same as load balancing?
No. Load balancing spreads identical requests across capacity; routing deliberately chooses a different model per request based on the task.
Should I start with rules or a learned router?
Rules. They are transparent, testable and usually capture most of the savings. Move to learned routing only once you have outcome logs to train and validate against.
How do I know routing is working?
Track cost per task and quality metrics per route. If total spend falls without acceptance or accuracy dropping, routing is working.
Does routing require multiple SDKs?
No. With an OpenAI-compatible API, one SDK works for all 30+ models and the router only changes the model parameter.
Can routing run inside my own network?
Yes. Private Plugsky deployments keep the endpoint — and therefore routing traffic — within your VPC, on-prem environment or air-gapped network.