Key facts
| Harness pattern | Generator agent → candidate calls → judge scoring → report |
| API surface | Chat completions with function calling and JSON mode (live) |
| Models | 30+ models behind one endpoint for candidates and judges |
| Structured scores | JSON mode for rubric scores, reasons and pass/fail flags |
| Routing | Route bulk generation to small models, judging to stronger ones |
| Reproducibility | Pin model versions and temperature per run and store every trace |
| Deployment | Region-locked planes, VPC, on-prem and air-gapped options |
| Free tier | plugsky-micro and plugsky-lite on the free plan, no card required |
TL;DR
- Use agents to build and expand the test set, not to score their own work.
- Keep candidates and judges on different models to reduce correlated bias.
- Score with a rubric and structured output so results aggregate cleanly.
- Pin model versions and settings — an evaluation without reproducibility is an anecdote.
- Turn failures into regression cases and re-run them on every prompt change.
How it works, step by step
- Define the task, the rubric and the pass criteria before writing any harness code.
- Build a golden set of real inputs with human-verified expected outputs, then use an agent to generate edge cases for review.
- Implement the harness: candidate calls through the chat endpoint, with tool-based checks for anything deterministic.
- Add a judge model with a rubric and JSON-mode output containing score, reason and pass flag per case.
- Validate the judge against a sample of human ratings and adjust the rubric until agreement is acceptable.
- Run the suite across candidate models, aggregate scores with confidence intervals, and store traces per case.
- Promote failures into the regression set, and re-run before any prompt or model change ships.
Try it yourself
Open the LLM benchmark explorer →
Evaluation as an agent loop
Evaluation is naturally agentic: generate cases, execute them, inspect results, refine. An agent can expand a golden set with paraphrases and edge cases, run deterministic checks as tools, and package outputs for scoring. The important discipline is separation — the model that produces an answer should not be the model that grades it.
On Plugsky the harness uses the same live primitives as production: /v1/chat/completions for candidate calls, function calling for programmatic checks, and JSON mode for structured judge output. With 30+ models behind one endpoint, switching candidates is a string change rather than a new integration.
Judge models and scoring patterns
An LLM judge is a measurement instrument, and instruments need calibration. Write the rubric with explicit criteria and anchored levels, ask for a score plus a reason, and validate against human ratings on a sample before trusting it at scale. Use a different model family for judging than for generating to reduce correlated errors.
- Rubric first: criteria, weights and pass thresholds defined before the first run.
- JSON output: score, reason and flags per case so aggregation is arithmetic, not parsing.
- Deterministic checks: exact-match, schema validity, tool-call correctness and refusal detection as tools.
- Drift watch: re-validate the judge whenever you change judge model or prompt.
From scores to release decisions
A score is only useful if it changes what you ship. Define thresholds in advance: what improvement is meaningful, what regression is disqualifying, and which categories are must-pass. Store per-case traces so a failure can be reproduced, and keep a regression suite that grows with every incident.
Run evaluation at three moments: before a model switch, before a prompt change, and on a schedule against production samples. Route bulk generation to plugsky-micro or plugsky-lite and reserve stronger models for judging and hard cases, which keeps runs fast and predictable. For sensitive datasets, pin the workspace to a region-locked plane or deploy VPC, on-prem or air-gapped.
Honest comparison
| Aspect | Plugsky-based harness | Single-vendor evaluation | Manual spot checks |
|---|---|---|---|
| Candidate models | 30+ models behind one endpoint | One vendor's catalogue | Whatever fits in a day |
| Judge setup | Separate model family on the same API | Often the same vendor | Human only |
| Reproducibility | Pinned versions, settings and traces | Varies | Rare |
| Regression suite | Failures promoted automatically | Manual | Ad hoc |
| Deployment | Cloud, VPC, on-prem, air-gapped | Vendor cloud | Local scripts |
Frequently asked questions
Can the same model generate and judge?
It can, but avoid it. Using the same model for generation and judging correlates errors and inflates scores. Pick a different model family as the judge and validate it against human ratings.
How many test cases do we need?
Start with a golden set that covers each requirement and failure mode, then expand with generated edge cases after human review. Coverage matters more than raw count.
Is function calling needed for evaluation?
It helps. Deterministic checks — schema validity, tool-call correctness, exact match — are best implemented as tools so the judge only handles semantic quality.
How do we keep runs reproducible?
Pin model versions and settings, store the full trace per case, and version the rubric and prompts alongside the results. Otherwise scores cannot be compared across runs.
What should we measure besides quality?
Latency, refusal correctness, tool-call accuracy and structured-output validity. These are the dimensions that break in production even when prose quality looks fine.
Can we evaluate on private data?
Yes. The harness runs in your environment, and Plugsky supports region-locked planes plus VPC, on-prem and air-gapped deployment for sensitive datasets.
How do we start?
Use the free plan's plugsky-micro and plugsky-lite models with no card to build the harness on a small golden set, then scale to more candidates on a paid plan.