Key facts
| Core idea | Task-to-model routing instead of one model for everything |
| API surface | One OpenAI-compatible endpoint; switch models by changing the model parameter |
| Model catalogue | 30+ models covering fast, coding, long-context and reasoning tiers |
| Routing controls | Static rules by task type first; learned routing only when you have outcome data |
| Evaluation | Per-task acceptance, latency and token spend rather than one global score |
| Pricing model | Flat monthly self-serve plans with unlimited fair-use usage |
| Free tier | Free plan with 2 free AI models, no card required |
| Status | Chat, streaming, function calling and embeddings are live |
TL;DR
- Map tasks to model tiers before writing any routing code.
- Start with static rules; add measured fallbacks later.
- Keep one API and one SDK — only the model name changes.
- Evaluate per task, not with a single benchmark score.
- Log routing decisions so you can prove the split pays off.
How it works, step by step
- Inventory your AI touchpoints: autocomplete, review, test generation, repo Q&A, debugging, documentation.
- Assign a tier to each: fast and cheap, coding-focused, long-context or reasoning.
- Implement a thin routing layer that maps task type to a model and records the decision.
- Run the same prompts across candidate models and compare accepted output, latency and spend.
- Add fallbacks: retry on errors, escalate on validation failure, downgrade when a fast model suffices.
- Review routing monthly as models change — retire aliases that no longer earn their cost.
Try it yourself
Open the AI workload router simulator →
Why one model is the wrong default
Using a frontier model for commit messages wastes money and latency; using a small model for multi-file refactors produces rejected diffs. Tasks differ in reasoning depth, context size and tolerance for mistakes, so the efficient workflow matches the model to the task. The gain is not only cost: fast models make interactive flows feel instant, while strong models raise acceptance rates on hard changes.
A practical routing table
- Autocomplete and commit messages: micro or lite aliases — speed dominates.
- Single-file edits and tests: a coding-focused alias with strong instruction following.
- Repository Q&A and refactor planning: a long-context alias.
- Debugging and architecture: a reasoning-tier alias, used sparingly.
- Embeddings: a dedicated embedding model for code and docs search.
Encode this as configuration, not scattered constants, so you can change a tier in one place.
Implementing the router
Keep the router boring: a function that takes a task type and returns a model name, with per-model parameters for temperature and max tokens. Your application still calls one OpenAI-compatible endpoint and one SDK. Log the task type, model, latency, token count and whether the result was accepted — that log is the evidence base for changing rules. Add a fallback chain for provider or validation errors, and a budget guard for the expensive tier.
Evaluation and maintenance
Score each route on its own terms: autocomplete on latency and acceptance, edits on tests passed, repo Q&A on citation correctness, reasoning on task success. New models arrive frequently, so re-run the evaluation set on a schedule and promote a cheaper model into a tier when it matches quality. Retire aliases that no longer win anywhere, and keep a written record of why each route exists.
Honest comparison
| Task | Fast alias | Coding alias | Reasoning alias |
|---|---|---|---|
| Commit messages | Best fit — instant and cheap | Overkill | Overkill |
| Single-file edit | Often too weak | Best fit | Works, slower |
| Repo-wide question | Fails on context | Works with retrieval | Works, costlier |
| Hard debugging | Unreliable | Sometimes | Best fit |
| Cost profile | Lowest | Moderate | Highest |
Frequently asked questions
Do I need multiple SDKs for multiple models?
No. With an OpenAI-compatible API and 30+ models, your SDK stays fixed and only the model parameter changes. The router is configuration, not separate integrations.
How should routing decisions be made?
Start with static task-to-model rules. Move to outcome-based routing only after you have logs showing which model succeeds per task type.
What should the router log?
Task type, chosen model, prompt and completion token counts, latency, retries and whether a human accepted the result. Those fields let you evaluate and tune routes.
When should a request escalate to a stronger model?
On validation failure, low-confidence output or an explicit retry signal — not on every request. Escalation should be rare and measured.
Is multi-model routing expensive to maintain?
The router itself is small. The recurring work is re-evaluation when models change, which is why per-task evaluation sets matter more than the routing code.
Can I prototype this on the free plan?
Yes. The free plan includes 2 free AI models with no card, and a 14-day full-access trial lets you compare stronger aliases on your real tasks.