Key facts
| Tool type | Free model shortlist builder across the live catalogue |
| Inputs | Task type, context length, latency target, budget model, structured-output needs |
| Outputs | Ranked model shortlist with trade-offs per candidate |
| Catalogue | 30+ models from free to frontier tiers |
| Specialised selectors | Coding, RAG and agent selector variants |
| Free plan | 2 free AI models (plugsky-micro, plugsky-lite), no card required |
| API | OpenAI-compatible chat completions; change the model name per call |
| Product status | Live |
TL;DR
- Define the task and its failure cost before comparing models.
- Context window, JSON reliability and latency eliminate more models than raw quality.
- Shortlist two or three tiers: a small default, a mid tier, a frontier escalation.
- Run your own eval set on the shortlist; public benchmarks are only a filter.
- Route per task with one OpenAI-compatible endpoint instead of standardising on one model.
How it works, step by step
- Write down the task in one sentence and what a wrong answer costs you.
- Estimate input and output size for a typical call using real examples, not guesses.
- Open the AI model comparison tool and enter task, context, latency and budget constraints.
- Review the shortlist and note which models trade quality for speed or cost.
- Collect 20-50 representative prompts with expected outputs as your eval set.
- Run the eval set against the top candidates and score correctness, JSON validity and latency.
- Set a per-task routing default and revisit the decision when the catalogue changes.
Try it yourself
Open the AI model comparison tool →
Constraints that do the real work
Most selection debates dissolve once constraints are explicit. A 200-page document pipeline rules out small context windows. A latency-sensitive autocomplete rules out reasoning models that think before answering. A strict JSON contract rules out models that wrap output in prose. Budget model matters too: per-token billing punishes long agent loops, while flat monthly self-serve pricing makes iteration cheap. The comparison tool applies these as filters, which is why a shortlist of two or three is more useful than a ranking of thirty.
How to read model tiers
Think in tiers rather than names. Micro and lite tiers handle classification, extraction, routing and formatting at low latency. Mid tiers handle summarisation, customer-facing chat and moderate reasoning. Frontier tiers handle hard reasoning, long documents and code. Most production systems want at least two tiers wired to one API, so a single integration covers both. Plugsky exposes 30+ models through the same OpenAI-compatible endpoint, which means changing tiers is a model-name change, not a new SDK.
Benchmarks filter, evals decide
Public benchmarks tell you which models are worth testing; they cannot tell you which model fits your data. Build a golden set from real traffic, include the awkward cases — empty inputs, tables, mixed languages, adversarial prompts — and score each candidate on correctness, format adherence and latency. Track cost per successful task, not cost per token. When two models are close, pick the one with the simpler failure mode, because predictable failures are cheaper to handle. Re-run the eval when a new model lands; selection is maintenance, not a one-time decision.
Honest comparison
| Selection factor | Why it matters | Fast filter | Plugsky advantage |
|---|---|---|---|
| Task type | Determines the quality bar | Chat, extraction, RAG, agents, code | 30+ models across all tiers |
| Context window | Long inputs truncate or fail | Estimate worst-case input tokens | Long-context models in catalogue |
| Structured output | Parsers depend on valid JSON | Require JSON mode and tool support | JSON mode and function calling live |
| Latency | Sets the user experience | Measure time to first token | Small tiers for high-volume calls |
| Budget model | Agent loops amplify per-token cost | Model cost per successful task | Flat monthly self-serve plans |
| Operations | One integration beats five | Count SDKs and keys | One OpenAI-compatible endpoint |
Frequently asked questions
Should I just use the largest model?
No. Use the smallest model that passes your eval set for each task, and reserve frontier tiers for work that genuinely needs them.
How many models should I test?
Two or three is usually enough: one small, one mid, one frontier. More candidates rarely change the decision.
What is a good eval set size?
Twenty to fifty representative prompts with expected outputs is enough to expose most differences, as long as the set includes your failure cases.
Does the comparison tool use public benchmarks?
It structures your requirements against catalogue attributes. Use benchmarks to filter candidates and your own evals to make the final call.
Can I switch models later?
Yes. With an OpenAI-compatible endpoint you change the model name per request, so routing changes do not require code changes.
How do I handle structured output?
Prefer models and modes with JSON mode and function calling, validate every response against your schema, and retry on parse failures.
What about free models?
The free plan includes 2 free models (plugsky-micro and plugsky-lite) with no card, which is enough to prototype routing before paying.
Does Plugsky charge per token?
Self-serve plans are flat monthly with unlimited fair-use usage. See the live pricing page for current plans.