Key facts
| Tool type | Free benchmark explorer plus an evaluation framework |
| Dimensions | Accuracy, format validity, latency and cost per successful task |
| Eval set size | 20-50 representative prompts including known failure cases |
| Benchmark role | Filters candidates; your golden set makes the decision |
| Cost view | Cost per successful task, not per token |
| Models | 30+ models from free to frontier tiers |
| Free plan | 2 free AI models (plugsky-micro, plugsky-lite), no card required |
| Product status | Live |
TL;DR
- Shortlist with benchmarks, decide with a golden set drawn from real traffic.
- Score cost per successful task, because retries and format failures change the ranking.
- Include the awkward cases: empty input, mixed language, long tables, adversarial prompts.
- Freeze prompts and settings during a run, or you are measuring noise.
- Re-run the suite whenever a model, prompt or retrieval setting changes.
How it works, step by step
- Define the task and what a correct output looks like in checkable terms.
- Collect 20-50 real prompts with expected outputs, including deliberate failure cases.
- Use the benchmark explorer to shortlist two or three candidate models per tier.
- Freeze prompt templates and parameters, then run every candidate on the same set.
- Score correctness, format validity, latency and cost per successful task.
- Review disagreements manually to confirm the rubric is measuring the right thing.
- Record the winner per task and re-run the suite when models or prompts change.
Original data
Try it yourself
Build a golden set that reflects production
A golden set is only useful if it looks like your traffic. Pull real prompts from logs, strip sensitive data, and label the expected behaviour. Include the cases that break systems: empty fields, mixed Arabic and English, tables with merged cells, prompts that should trigger a refusal, and inputs near the context limit. Twenty to fifty examples will expose most differences if the set is honest; a thousand synthetic examples will hide them. Version the set alongside the prompt so a regression is traceable.
Metrics that separate similar models
Accuracy alone rarely decides. Add format validity — does the output parse against your schema on the first try? Add latency percentiles, not averages, because tail latency shapes user perception. Add cost per successful task: a cheaper model that fails one call in ten and retries is often more expensive than the model you rejected. Track these four together and a clear winner usually appears. When two models tie, compare failure modes: a model that fails predictably is easier to wrap in validation than one that fails creatively.
Turning evaluation into a regression suite
An evaluation that runs once is an anecdote. Wrap the golden set in a test runner that executes on demand and in CI, and store scores per run so trends are visible. Alert when accuracy or format validity drops beyond a threshold, not on every small wobble. Because Plugsky exposes 30+ models behind one OpenAI-compatible endpoint, changing a model under test is a configuration change, which makes multi-model regression runs cheap. Pair the suite with model routing so each task automatically runs on the model that last passed its tests.
Honest comparison
| Metric | How to measure it | Pass signal | Common pitfall |
|---|---|---|---|
| Accuracy | Score against expected outputs with a rubric | Meets the agreed threshold on the golden set | Testing only easy examples |
| Format validity | Parse every response against your schema | High first-try validity | Ignoring retry cost |
| Latency | Track p50 and p95 time to completion | Meets the user-facing target | Averaging away tail latency |
| Cost per task | Spend divided by successful completions | Lower than the alternative in production | Comparing sticker token rates only |
| Safety behaviour | Include prompts that should be refused | Refusals are correct, not random | Skipping adversarial cases |
Frequently asked questions
How large should an LLM eval set be?
Twenty to fifty well-chosen examples covering your real traffic and failure cases beat hundreds of easy ones. Expand the set as new failure modes appear.
Are public benchmarks useless?
No. They are an efficient filter for which models are worth testing. They cannot tell you how a model behaves on your data, prompts and schemas.
What is cost per successful task?
Total spend divided by the number of outputs that passed validation. It captures retries and failures that per-token comparisons miss.
How do I score open-ended answers?
Combine rule-based checks for format and required facts with human review on a sample. Keep the rubric written down so scores stay comparable over time.
How often should I re-run evals?
After any model, prompt, retrieval or schema change, and on a regular schedule so slow drift is visible before users notice.
Can I evaluate models from different providers?
Yes. With an OpenAI-compatible endpoint, switching the model under test is a parameter change, so the same harness covers the whole catalogue.
What about latency spikes?
Measure percentiles and test at realistic concurrency. A model that is fast alone can degrade under load.
Is there a free way to start evaluating?
Yes. The free plan includes 2 free AI models (plugsky-micro and plugsky-lite) with no card required.