How do you reduce the cost of model evaluation with model routing?

Key facts

Router model	plugsky-fusion escalates per request; useful for judge tiering (live)
Judge tiers	Cheap tiers with rubrics for most checks, stronger models for ambiguity
Caching	Hash outputs and judge verdicts to avoid re-paying
Models	30+ models; separate the model under test from the judge
JSON mode	Live for rubric scores and structured verdicts
Pricing	Flat monthly self-serve plans with no per-token charges on self-serve
Free tier	plugsky-micro and plugsky-lite on the free plan, no card required
Roadmap	The batch endpoint is coming soon for offline evaluation runs

TL;DR

• Born from a fixed sample, not the whole suite, on every change.
• Judge routine checks with cheap tiers and tight rubrics.
• Cache outputs and verdicts by hash; identical cases should never re-run.
• Reserve strong models for ambiguous or adversarial cases.
• Track evaluation spend per release like a production budget.

How it works, step by step

1. Split your suite into fast regression checks and deep quality reviews.
2. Cap the fast suite at a representative sample that still catches regressions.
3. Write rubrics so cheap judges can score reliably; validate against human labels.
4. Cache model outputs by prompt hash and judge verdicts by output hash.
5. Route ambiguous, adversarial or high-stakes cases to strong judges.
6. Run deep reviews on milestones, not on every commit.
7. Report evaluation spend alongside accuracy deltas per release.

Try it yourself

Open the prompt diff and evaluator →

Evaluation runs can outspend production

A production request serves one user; an evaluation run pays for every case in the suite, often multiple models, plus a judge for each output. Teams that re-run full suites on every commit can easily spend more on measurement than on their product. The fix is the same discipline production gets: sampling, caching and tiering.

Start by separating concerns. Fast regression checks catch obvious breakage and need only a representative sample. Deep quality reviews ask harder questions and deserve strong judges but run rarely. Mixing them into one nightly full run is how budgets disappear.

Judge design and tiering

LLM-as-judge quality depends on rubric specificity, not on model size alone. A tight rubric — explicit criteria, short scales, structured output — lets cheap tiers agree with human labels on routine checks. Reserve strong judges for ambiguity, subtle tone, factual verification against sources, and adversarial cases.

• Validate judge agreement against a human-labelled set before trusting any tier.
• Separate the model under test from its judge; self-judging hides failures.
• Use JSON mode so scores are machine-checkable and comparable across runs.
• Never tune prompts against the same cases you report quality on; hold out a set.

Caching, sampling and reporting

Cache aggressively. Model outputs keyed by prompt hash mean unchanged cases never regenerate, and judge verdicts keyed by output hash mean unchanged outputs are never re-scored. With per-token billing gone on self-serve flat plans, the remaining costs are latency and fair-use headroom, but caching still saves wall-clock time on every run.

Report evaluation spend per release next to accuracy deltas; if a change costs more to evaluate than it saves in production, say so explicitly. Start on the free plan with plugsky-micro and plugsky-lite as judges and models under test, then use the 14-day full-access trial to calibrate stronger judges; the live pricing page lists plans, and the batch endpoint for large offline runs is coming soon.

Honest comparison

Evaluation choice	Routed evaluation on Plugsky	Strong judge for every check	No evaluation
Routine checks	Cheap judge with tight rubric	Frontier price per case	Regressions ship silently
Ambiguous cases	Escalated to strong judges	Native strength	Unmeasured
Re-runs	Hash-based caching	Paid every run	Not applicable
Cadence	Fast suite on change, deep on milestones	Full suite nightly	Ad hoc
Evidence	Judge-human agreement measured	Rarely measured	None

Frequently asked questions

Why is evaluation more expensive than production?

Each run pays for every case, often across several models, plus judge calls. Re-running full suites on every commit multiplies that, which is why sampling and caching matter so much.

Can cheap models be judges?

Yes, for routine rubric-based checks, if the rubric is specific and you have validated agreement against human labels. Reserve strong judges for ambiguity, tone and factual verification.

How do I keep judge quality high?

Write explicit criteria and scales, use structured output, validate judge agreement against human labels, and never let the model under test judge itself.

How does caching work in evaluation?

Hash prompts to reuse model outputs and hash outputs to reuse judge verdicts. Unchanged cases then cost nothing on a re-run.

How large should the fast suite be?

Large enough to catch your known regressions, small enough to run on every change. Build it from real failures and grow it deliberately rather than importing thousands of generic cases.

Should I hold out test cases?

Always. Keep a held-out set that never informs prompt tuning, and report quality on it, otherwise your metrics overstate real performance.

Is batch evaluation available?

Not yet — the batch endpoint is coming soon. Today, run evaluation with bounded concurrency and a durable queue, and cache results between runs.

Can I evaluate on a free plan?

Yes. plugsky-micro and plugsky-lite are on the free plan with no card, and the 14-day full-access trial lets you calibrate stronger judge models.