How do you reduce the cost of RAG with model routing?

Key facts

Router model	plugsky-fusion escalates per question across tiers (live)
Embeddings	Live embeddings API; re-embed only changed chunks by hash
Retrieval	Tune chunk size and top-k against recall, not index size
Models	30+ models; lookups cheap, synthesis escalated
Long context	plugsky-longctx for wide synthesis when needed
Pricing	Flat monthly self-serve plans with no per-token charges on self-serve
Free tier	plugsky-micro and plugsky-lite on the free plan, no card required
Roadmap	Files and batch endpoints are coming soon; keep ingestion in your pipeline

TL;DR

• Index incrementally; unchanged chunks should never be re-embedded.
• Retrieve fewer, better passages — context is the recurring cost.
• Route lookups cheap and synthesis strong.
• Cache question embeddings and repeat answers.
• Measure cost per answered question with citation accuracy.

How it works, step by step

1. Break spend into indexing, retrieval and generation, and measure each per workload.
2. Hash chunks and re-embed only new or changed content.
3. Tune chunk size and top-k together, measuring recall and context cost.
4. Deduplicate retrieved passages and drop anything below a relevance threshold.
5. Classify questions and route lookups to cheap tiers, synthesis to stronger ones.
6. Cache question embeddings and answers where freshness allows.
7. Re-measure cost per answered question and citation accuracy monthly.

Try it yourself

Open the RAG cost calculator →

The three cost centres of RAG

Indexing is the one-time cost that recurs as content changes; retrieval is the per-query cost of finding passages; generation is the per-query cost of writing an answer with those passages in context. Teams often focus on model choice while a nightly full re-embed or an oversized top-k quietly dominates the budget.

Measure all three separately before optimising. The largest line is usually generation context on high-volume query workloads, followed by index maintenance on fast-changing corpora. Fixing the wrong one wastes effort — and worse, can reduce quality.

Chunk size, top-k and reranking

Chunk size and top-k decide both quality and cost. Small chunks mean more vectors and finer retrieval; large chunks mean better context per result but more tokens in every prompt. Tune them against a golden question set and track recall and context size together.

• Deduplicate near-identical passages before generation.
• Apply a relevance threshold so weak matches never enter the prompt.
• Rerank selectively on query classes where it measurably helps; reranking everything doubles retrieval work.
• Blend keyword search for exact identifiers so they do not need semantic retrieval.

Generation tiering and measurement

Generation is where routing applies. Factual lookups answered from one or two passages run well on cheap tiers; comparisons and cross-document synthesis benefit from stronger models; long-context models earn their place when breadth is genuinely required rather than chosen by habit. Use citations so quality remains measurable at every tier.

Cache question embeddings for repeated queries and cache answers with a time-to-live tied to content change. Report cost per answered question alongside citation accuracy and refusal quality on a held-out set. Start on the free plan with plugsky-micro and plugsky-lite to build and measure the pipeline, then use the 14-day full-access trial for strong and long-context models; plans are on the live pricing page.

Honest comparison

RAG cost lever	Routed RAG on Plugsky	Frontier model on every query	Re-embed everything nightly
Index maintenance	Incremental by hash	Incremental if implemented	Full cost daily
Retrieval	Tuned top-k with thresholds	Often oversized	Often oversized
Lookups	Cheap tier	Frontier price for facts	Depends on generation tier
Synthesis	Escalated or long-context	Native strength	Depends on generation tier
Evidence	Cost per answered question	Cost per query	Cost per refresh

Frequently asked questions

What is the biggest RAG cost?

It depends on workload. High query volume makes generation context the largest line; fast-changing corpora make index maintenance dominant. Measure the three centres separately before optimising any.

How do I reduce indexing cost?

Hash chunks and re-embed only changed content, and retire deleted documents. Nightly full re-embeds are the most common avoidable expense.

How many chunks should I retrieve?

Tune against recall on a golden question set, then watch context size. More chunks can improve answers but add cost and noise on every single query.

Should reranking always be on?

No. Apply it to query classes where it measurably changes outcomes. Running a reranker on every query doubles retrieval work for limited benefit.

When is a long-context model worth it?

When synthesis genuinely spans many passages and selecting a few is unreliable. Use it deliberately for those questions, not as a default.

Can I cache RAG answers?

Yes, with a time-to-live tied to content change frequency. Always cache question embeddings, since repeated questions are common.

How do I keep quality measurable?

Require citations, keep a held-out golden set, and track citation accuracy and refusal quality alongside cost per answered question.

Can I build this for free?

Yes. plugsky-micro and plugsky-lite are on the free plan with no card, and the 14-day full-access trial covers stronger and long-context models.