Key facts
| Evaluation layers | Retrieval metrics first, then answer metrics |
| Embeddings | plugsky-embed and plugsky-embed-multilingual via /v1/embeddings (live) |
| Judge models | Structured scoring with JSON mode on /v1/chat/completions (live) |
| Golden set | Questions paired with expected source passages and answers |
| Models | 30+ models behind one endpoint, candidates and judges alike |
| Deterministic checks | Citation IDs, recall at k and refusal detection as tools |
| Deployment | Region-locked planes, VPC, on-prem and air-gapped options |
| Free tier | plugsky-micro and plugsky-lite on the free plan, no card required |
TL;DR
- Separate retrieval evaluation from generation evaluation — they fail differently.
- A golden set needs expected source passages, not just expected answers.
- Measure recall@k before blaming the model for a weak answer.
- Score groundedness and citations with a judge calibrated on human ratings.
- Re-run the suite whenever chunks, embeddings, prompts or models change.
How it works, step by step
- Write 50–150 questions with two labels each: the passage that should be retrieved and the answer a careful human would give.
- Implement retrieval scoring: recall@k, mean reciprocal rank and a zero-result rate over the golden set.
- Implement answer scoring with a judge model and JSON output covering groundedness, citation correctness, completeness and refusal.
- Calibrate the judge against human ratings on a sample and revise the rubric until agreement is acceptable.
- Run the suite across chunking strategies, embedding models and prompts, keeping traces per case.
- Promote every production failure into the golden set so regressions are caught on the next run.
- Gate releases on both layers: retrieval thresholds must hold before answer quality is even assessed.
Original data
Try it yourself
Open the best model for RAG selector →
Two layers, two failure modes
RAG pipelines fail at retrieval or at generation, and the fixes are different. If the right passage never reaches the prompt, no instruction tuning will rescue the answer; if retrieval is correct but the answer drifts, the prompt or the model needs work. Evaluate the layers separately so you always know which one to fix.
Plugsky supports both halves of the harness on live endpoints: /v1/embeddings for indexing the corpus under test, and /v1/chat/completions with JSON mode for judge output. Candidates and judges can be different models from the same 30+ model catalogue.
Building a golden set that measures retrieval
Most evaluation sets only record expected answers, which hides retrieval failures behind fluent responses. Pair every question with the passage or document that contains the answer, then measure whether retrieval surfaces it. That single change turns your suite from a vibe check into a diagnostic.
- Recall@k: share of questions whose expected passage appears in the top k results.
- MRR: how high the right passage ranks when it is retrieved.
- Refusal cases: questions the corpus cannot answer, to test honest failure.
- Tracing: store retrieved IDs per question so ranking changes can be explained.
Scoring answers, then acting on results
Answer scoring needs a rubric: groundedness (is every claim supported by the retrieved passages), citation accuracy (do citations point to the passages that support the claims), completeness and refusal correctness. Ask the judge for structured output with a reason per score, and calibrate on human ratings before trusting it at scale.
Then set thresholds and use them. Recall below target means fixing chunking, metadata filters or the embedding model; groundedness failures mean tightening the prompt or the context assembly; citation failures mean the answer format is too loose. Re-run after every change — chunk size, embedding model, prompt, candidate model — because each one can move both layers. Run it all inside your chosen region, including on-prem when policy requires it.
Honest comparison
| Dimension | Layered RAG evaluation | Answer-only scoring | No evaluation |
|---|---|---|---|
| Retrieval visibility | Recall@k and MRR per question | Invisible | None |
| Groundedness | Judged against retrieved passages | Judged against expected answer | None |
| Citation checks | Deterministic ID matching | Rarely checked | None |
| Regression safety | Failures promoted to golden set | Manual review | None |
| Judge independence | Different model family from candidate | Often the same | N/A |
Frequently asked questions
Why evaluate retrieval separately?
Because retrieval and generation fail for different reasons and require different fixes. Without retrieval metrics you cannot tell whether a weak answer came from missing context or from the model.
How large should the golden set be?
Start with enough questions to cover every document type and failure mode, then grow it. Fifty well-labelled questions with known source passages outperform hundreds of unlabelled ones.
Can one model judge another reliably?
With a clear rubric and calibration against human ratings, yes. Use a different model family from the candidate, ask for reasons alongside scores, and re-validate the judge when you change it.
What is the most common retrieval failure?
Chunking. Passages split away from their headings or tables without captions often rank poorly. Recall@k falls, and no prompt change recovers it.
How do we test refusals?
Include questions your corpus cannot answer and check that the assistant declines while offering the nearest sources. Refusal correctness is a scored dimension, not an afterthought.
How often should the suite run?
Before any prompt, chunking, embedding or model change ships, and on a schedule against production samples to catch drift.
Can evaluation run on private corpora?
Yes. The harness runs in your environment, and Plugsky supports region-locked planes plus VPC, on-prem and air-gapped deployment.