Key facts
| Evaluation approach | One client, one suite, swap the model parameter |
| API compatibility | OpenAI-shaped chat completions with streaming, JSON mode and function calling (live) |
| Models | 30+ models behind one endpoint for side-by-side runs |
| Structured scoring | JSON mode for rubric scores and pass/fail flags |
| Deterministic checks | Tool-based validation of outputs and tool calls |
| Pricing | Flat monthly plans with unlimited fair-use usage on self-serve |
| Deployment | Region-locked planes, VPC, on-prem and air-gapped options |
| Free tier | plugsky-micro and plugsky-lite on the free plan, no card required |
TL;DR
- Make the model a parameter in your harness, not an assumption in your code.
- Replay identical prompts and settings so differences come from the model alone.
- Score quality with a rubric and JSON output; check schema validity in code.
- Measure latency and refusal behaviour alongside accuracy.
- Keep the suite small, fast and growing — every incident becomes a case.
How it works, step by step
- Build the harness around a single client with the base URL in configuration and the model as a per-run parameter.
- Assemble a suite of real prompts with expected answers, plus edge cases and must-refuse inputs.
- Add deterministic checks for schema validity, required fields and tool-call correctness.
- Add a judge model with a rubric and JSON output for semantic quality, calibrated on human ratings.
- Run the suite across candidate models, recording quality, latency and failure categories per model.
- Compare arms head to head and pick a primary model plus a fallback for each workload.
- Schedule the suite to re-run on prompt changes and periodically against production samples.
Try it yourself
Open the best AI model picker →
One client, many models
The main cost of model evaluation used to be integration: every vendor had its own SDK, auth and response shape. With an OpenAI-compatible API that overhead disappears. Your harness keeps one client, changes the model value and replays the same prompts — differences in output come from the model rather than the plumbing.
Plugsky exposes 30+ models through that single endpoint with streaming, JSON mode and function calling live, so candidate models can be tested on exactly the behaviours your product depends on: structured output, tool calls, long inputs and refusals.
Designing a trustworthy suite
A useful suite is small, labelled and honest about failure. Include representative real prompts, boundary cases, structured-output tasks, tool-use tasks and inputs the system must refuse. Keep expected outputs human-verified, and version everything — prompts, temperature, max tokens — so a re-run is comparable to the last one.
- Stratify: score by category, not just overall, so a model strong on writing but weak on extraction is visible.
- Determinism where possible: set temperature to zero for extraction tests; score creative tasks with a rubric.
- Cost and latency: record tokens, wall-clock time and error rates per model alongside quality.
- Adversarial cases: prompt injection attempts and malformed inputs belong in the suite.
From results to a routing decision
Evaluation output should be an operational decision: which model for which workload, and what fallback to use. Many teams land on a tiered arrangement — plugsky-micro or plugsky-lite for cheap, high-volume classification and extraction, a mid-tier model for general chat, and a frontier model for hard reasoning. With everything behind one endpoint, that decision is expressed in configuration rather than architecture.
Keep the suite running. Prompts drift, models get updated and traffic changes shape. Periodic runs against production samples catch regressions early, and each incident becomes a new case. Because the API is OpenAI-compatible, adding or replacing a candidate later is a one-line change to the harness.
Honest comparison
| Aspect | Plugsky | Multi-provider adapters | Single-vendor testing |
|---|---|---|---|
| Candidate coverage | 30+ models, one client | One adapter per vendor | Vendor catalogue only |
| Setup effort | Model parameter change | SDK and auth per provider | None, but narrow |
| Structured checks | JSON mode plus function calling | Varies by vendor | Vendor-specific |
| Cost shape | Flat monthly with unlimited fair use on self-serve | Mixed per-provider bills | Per-token |
| Reproducibility | Pin model and settings per run | Good, with effort | Limited alternatives |
Frequently asked questions
Do I need different code per model?
No. Change the model parameter on the same client. That is the practical benefit of an OpenAI-compatible endpoint for evaluation and for production routing.
How do we judge quality without humans?
Use a rubric-driven judge model and calibrate it against human ratings on a sample. Keep deterministic checks — schema validity, exact matches, tool-call correctness — in code rather than with a model.
Should temperature be zero?
For extraction and classification tests, yes. For tasks where variation is expected, score multiple samples with a rubric instead of pretending a single deterministic answer exists.
What metrics belong in the report?
Quality by category, schema validity rate, tool-call accuracy, refusal correctness, latency and error rate. Quality alone hides operational regressions.
How do we compare cost fairly?
Self-serve plans are flat monthly with unlimited fair-use usage rather than per-token billing. Check the live pricing page for current plan terms before modelling heavy evaluation runs.
Can evaluation run on private prompts?
Yes. The harness runs in your environment, and Plugsky supports region-locked planes plus VPC, on-prem and air-gapped deployment for sensitive data.
How do we begin?
Start on the free plan with plugsky-micro and plugsky-lite and no card, build a 30–50 case suite, then expand candidates and volume on a paid plan or the 14-day full-access trial.