Key facts
| Tool type | Checklist plus test set template, scored with the prompt diff evaluator |
| Test set fields | Question, expected answer, gold passages, must-cite document |
| Retrieval metrics | Recall at k, rank of the gold passage, empty-result rate |
| Answer metrics | Groundedness, citation accuracy, refusal correctness |
| RAG status | RAG, embeddings and agents are live |
| Models | 30+ models for generation and embeddings |
| Free plan | 2 free AI models (plugsky-micro, plugsky-lite), no card required |
| Product status | Live |
TL;DR
- Separate retrieval failures from generation failures; they have different fixes.
- Include unanswerable questions so refusal behaviour is measured, not assumed.
- Record the gold passage for every question, not just the expected answer.
- Score citation accuracy: a correct answer citing the wrong source is still a failure.
- Keep the test set versioned and re-run it after chunking or prompt changes.
How it works, step by step
- Collect 30-50 real questions, including some your knowledge base cannot answer.
- For each question, write the expected answer, the gold passages and the must-cite document.
- Run the test set through your RAG pipeline and log retrieved passages and answers.
- Score retrieval recall and the rank of the gold passage within the top k results.
- Score groundedness, citation accuracy and correct refusals for unanswerable questions.
- Use the prompt diff and evaluator to compare variants on the identical test set.
- Version the test set and re-run after every chunking, embedding or prompt change.
Try it yourself
Open the prompt diff and evaluator →
The checklist dimensions
Split the checklist into retrieval and generation. Retrieval checks whether the right passage appears in the top k results and at what rank; an empty or noisy result set guarantees a bad answer regardless of the model. Generation checks whether the answer is grounded in the retrieved context, whether it cites the correct source and whether it refuses when the context does not contain the answer. Track these separately, because a retrieval fix and a prompt fix are different work items — and a system that scores poorly overall may only be failing on one side.
Building the test set template
A useful test set row has four fields: the question, the expected answer, the gold passages that should be retrieved, and the document that must be cited. Write questions the way users ask them, including typos and mixed languages where relevant. Add at least five unanswerable questions to measure refusal correctness. Keep the set small enough to maintain by hand — thirty to fifty rows is enough to expose most regressions — and store it as data, not prose, so it can run automatically. Version it beside your prompts and embedding settings.
Comparing variants with the evaluator
Change one variable at a time: chunk size, overlap, top k, embedding model or prompt. Run every variant on the identical test set and use the prompt diff and evaluator to see where outputs diverge and which rows changed verdict. Watch for trades: a larger top k may raise recall while lowering precision and pushing irrelevant context into the answer. When scores tie, prefer the configuration that is cheaper and simpler to operate, and record the winning settings in the same repository as the pipeline configuration.
Honest comparison
| Check | What it catches | How to score | Pass signal |
|---|---|---|---|
| Retrieval recall | Missing passages | Gold passage present in top k | High recall on answerable questions |
| Gold passage rank | Weak ranking | Position of the gold passage | Near the top, not the tail |
| Groundedness | Hallucination beyond context | Claims supported by retrieved text | No unsupported claims |
| Citation accuracy | Right answer, wrong source | Cited document equals the must-cite document | Citations match gold sources |
| Refusal correctness | Inventing answers for missing data | Unanswerable questions refused | Refusals, not guesses |
| Latency and cost | Heavy retrieval or generation | End-to-end time and spend per task | Meets targets at the chosen settings |
Frequently asked questions
What should a RAG evaluation checklist include?
Retrieval checks (recall, gold passage rank), generation checks (groundedness, citation accuracy) and refusal correctness, measured on a versioned test set with gold passages.
How big should the RAG test set be?
Thirty to fifty questions with gold passages is enough to catch regressions and small enough to maintain by hand. Include at least five unanswerable questions.
Why separate retrieval from generation scoring?
Because the fixes differ. Poor retrieval calls for chunking, embedding or top-k changes; poor groundedness calls for prompt or model changes.
What is citation accuracy?
The share of answers whose cited source matches the document that actually contains the answer. A correct answer citing the wrong passage is still a failure for trust.
How do I test refusal behaviour?
Add questions your corpus cannot answer and check that the system declines instead of inventing an answer. Score refusals as correct when the data is genuinely absent.
Can I compare prompt versions with this checklist?
Yes. Run each variant on the identical test set and use the prompt diff and evaluator to see which rows changed verdict and why.
How often should I re-run the evaluation?
After any chunking, embedding, prompt or model change, and on a schedule so slow drift is visible before users report it.
Is there a free way to prototype RAG?
Yes. The free plan includes 2 free AI models with no card, and the RAG sandbox lets you test retrieval before building a pipeline.