RAG retrieval benchmark: 10 datasets and a reproducible evaluator
Compute recall@k, MRR@k and nDCG@k from your own qrels and run files, then plan chunking, embedding and reranking cost. No leaderboard, no hidden benchmark.
Evaluate a run
Paste qrels (query, doc, grade) and a run (query, doc, rank or score). Everything is computed in your browser; nothing is uploaded.
| Query | Relevant | Retrieved | Recall@k | MRR@k | nDCG@k |
|---|---|---|---|---|---|
| No evaluation yet. | |||||
Plan the index and reranker
Chunking and embedding
Query cost and latency
Embedding is treated as a batch job priced per token. Reranking is priced per 1,000 passages. Latency is your measured or expected number, not a vendor claim. This planner does not include vector-store, compute or egress costs.
The 10 datasets
The table lists the ten BEIR datasets used in the protocol, with corpus size and query count as published in the BEIR dataset table. They are not a leaderboard. Each has a different domain, document length and relevance pattern, and a retriever that wins on one can lose on another.
| Dataset | Domain | Corpus | Queries | Task | Why use it |
|---|---|---|---|---|---|
| MS MARCO | Web passages | 8.84M | 6,980 | Passage retrieval | The default web-scale sanity check; large corpus, short passages. |
| Natural Questions | Wikipedia | 2.68M | 3,452 | Open-domain QA | Natural question phrasing; good for answer-bearing passage retrieval. |
| HotpotQA | Wikipedia | 5.23M | 7,405 | Multi-hop QA | Questions need two or more passages; tests whether top-k covers both. |
| FiQA-2018 | Finance | 57K | 648 | Domain retrieval | Financial questions and opinions; a small, domain-specific corpus. |
| ArguAna | Arguments | 8.67K | 1,406 | Argument retrieval | Counter-argument retrieval; tests semantic matching beyond keyword overlap. |
| SciFact | Science | 5K | 300 | Claim verification | Scientific claims with evidence abstracts; small and fast to iterate. |
| NFCorpus | Nutrition | 3.6K | 323 | Biomedical retrieval | Nutrition and medical claims; useful for high-stakes phrasing. |
| SCIDOCS | Science | 25K | 1,000 | Citation prediction | Scientific paper retrieval; longer documents and domain vocabulary. |
| TREC-COVID | Biomedical | 171K | 50 | Epidemic retrieval | Small query set, large corpus; a stress test for recall. |
| Climate-FEVER | Climate | 5.42M | 1,535 | Fact verification | Claims about climate science; evidence retrieval at scale. |
Source: BEIR dataset table, checked 20 November 2026. Counts are rounded in the upstream table; use the downloaded dataset for exact figures.
Metrics that matter: recall@k, MRR and nDCG
Retrieval evaluation is not one number. Recall@k asks whether the relevant documents made it into the candidate set. MRR@k asks how high the first relevant document ranked. nDCG@k asks how good the entire ranked list is, weighted by graded relevance and discounted by position. A retriever that finds every relevant document at ranks 9 and 10 has perfect recall@10 but a poor MRR@10.
recall@k = |relevant ∩ top_k| ÷ |relevant|
MRR@k = mean(1 ÷ rank_of_first_relevant)
DCG@k = Σ (2^grade − 1) ÷ log2(rank + 1)
nDCG@k = DCG@k ÷ IDCG@k
Use recall when the downstream reader or LLM can handle a noisy candidate set. Use MRR when the user sees the first result. Use nDCG when relevance is graded and the order of the whole list matters. Report all three, plus the number of queries and the k, or the number is not comparable.
Chunking and reranking change the result
Chunking changes the denominator
Chunk size changes what counts as a document. A 512-token chunk with 64-token overlap produces more passages than a 1,024-token chunk, which raises recall@k because there are more chances to include the answer, but it also raises index size, embedding cost and the chance of retrieving near-duplicates. A 128-token chunk can split an answer across two chunks and lower recall even though the retriever is fine.
The planner uses a simple estimate: chunks per document equals the document length divided by the stride (chunk size minus overlap), rounded up. Real chunkers split on headings, sentences or semantic boundaries, so the actual count differs. Use the planner for order-of-magnitude cost, then count the chunks your pipeline actually produces.
Rerankers: measure the lift, not the demo
A reranker takes the top N candidates from the retriever and reorders them with a cross-encoder or a hosted model. It can raise MRR and nDCG without changing recall@k, because the candidate set is unchanged. It can also lower recall if the reranker drops a relevant document that the retriever found, especially at small k.
Measure the lift at the same k and on the same qrels. Report the retriever-only metrics, the reranked metrics and the extra latency and cost. A reranker that adds 20 points of nDCG@10 but triples p95 latency may still be the right trade for a research tool and the wrong trade for a chat product.
Cost and latency
Indexing cost is driven by tokens, not documents. A 10,000-document corpus with 1,200 tokens per document and 512-token chunks with 64-token overlap produces about 26,800 chunks and 13.7 million chunk tokens. At $0.02 per million tokens, the index costs about $0.27 per rebuild. That is small; the query path usually dominates at scale.
Query cost is driven by rerank depth. Reranking 50 passages per query at $0.20 per 1,000 passages costs $0.01 per query, or $1,000 for 100,000 queries. Retrieval latency and rerank latency add directly to time-to-first-token if the LLM waits for the context. Use the planner to see which term dominates before you optimize the wrong one.
Reproducibility protocol
- Pin the dataset. Record the BEIR dataset name, download URL and checksum. Do not mix versions.
- Pin the embedder. Record the model ID, revision, dimensions, normalization and max input length.
- Pin the chunker. Record chunk size, overlap, splitter and any preprocessing (HTML stripping, deduplication).
- Pin the candidate depth. Retrieve a fixed N before reranking; report N and the final k.
- Pin the reranker. Record the model, revision, batch size and score normalization.
- Export the run. Keep the qrels and the ranked run file with the metrics JSON. A metric without the run is not reproducible.
Run the same protocol on at least three datasets: one in your domain, one general web dataset and one small dataset for fast iteration. If the ranking of your candidates changes across datasets, the choice is not robust.
Methodology and limits
The evaluator parses three-column qrels and run files, computes recall@k, MRR@k and nDCG@k per query, and averages over the queries in the qrels. A query with no run entry scores zero. The planner uses a stride-based chunk estimate and user-supplied prices and latencies. It does not call an embedding model, a vector database or a reranker, and it does not publish a leaderboard.
The dataset table is a snapshot of the BEIR dataset metadata, not a benchmark result. The page does not claim that any chunker, embedder or reranker wins. Related: the RAG evaluation guide, embedding model comparison, the RAG sandbox and the RAG cost calculator.
FAQ
What is recall@k?
Recall@k is the share of relevant documents that appear in the top k results. If a query has three relevant documents and two are in the top 10, recall@10 is 0.67. It measures coverage, not ordering. A retriever can have high recall and poor ranking, which is why MRR and nDCG are reported alongside it.
What is the difference between MRR and nDCG?
MRR looks only at the first relevant result: it is 1 divided by that result's rank, averaged over queries. nDCG uses the full graded relevance list and discounts results by position, so a highly relevant document at rank 1 counts more than a marginally relevant one at rank 5. Use MRR for “how fast does the user see something useful” and nDCG for “how good is the whole ranked list”.
Which of the 10 datasets should I use?
Use the one whose domain and task match your corpus. MS MARCO is web passages, FiQA is finance, NFCorpus is nutrition, TREC-COVID is biomedical, SciFact and SCIDOCS are scientific, HotpotQA is multi-hop, ArguAna is argument retrieval, Climate-FEVER is fact verification and NQ is Wikipedia question answering. Running all 10 is a robustness check, not a substitute for your own data.
Why doesn't the page publish a leaderboard?
Because retrieval quality depends on the chunker, embedder, index, query expansion and reranker, and a single aggregate score hides those choices. The page provides the datasets, the metric implementation and a protocol so you can produce a result that someone else can reproduce. It does not claim that one embedder or chunker wins.
How do I make my run reproducible?
Pin the dataset version and checksum, the embedder model and revision, the chunk size and overlap, the number of retrieved candidates, the reranker model and the top-k. Record the date and the hardware or API. Export the JSON report and keep the qrels and run files with it. A score without those fields is not reproducible.
Sources and verification
Primary sources were opened and checked on 20 November 2026. Product terms and datasets change; the linked source is authoritative.
- BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models — dataset construction, metrics and zero-shot evaluation checked 20 November 2026
- BEIR dataset repository — dataset list, download links and checksums checked 20 November 2026
- pytrec_eval — reference implementation of recall, MRR and nDCG checked 20 November 2026
- Lost in the Middle: How Language Models Use Long Contexts — why retrieval position and context length matter downstream checked 20 November 2026
Need retrieval and generation in one API?
Plugsky provides hosted chat and embedding models through one OpenAI-compatible endpoint. Use the free tier to prototype, then size the plan against your query volume.
Browse models → See plans