How do you build AI routing with RAG?

Key facts

Retrieval	POST /v1/embeddings with plugsky-embed returns vectors; top-k similarity scores drive routing
Generation	POST /v1/chat/completions with the retrieved context inserted
Routing signals	Top similarity score, agreement across chunks and number of distinct sources
Model tiers	30+ models; plugsky-micro and plugsky-lite for extractive, plugsky-pro and plugsky-frontier for synthesis
RAG status	RAG is live; embeddings plus chat completions are the documented building blocks
Pricing model	Flat monthly self-serve plans with unlimited fair-use usage
Governance	Scoped keys, audit logs and usage analytics for the retrieval path
Roadmap	Rerankers are not in the documented live stack; batch ingestion is coming soon

TL;DR

• Retrieve before you route: evidence quality is a better signal than query length.
• Three lanes cover most traffic — direct answer, single-source extraction, multi-source synthesis.
• Route misses to clarification or refusal instead of forcing a weak answer.
• Calibrate thresholds on labeled data; raw cosine similarity is not relevance.
• Keep routing and retrieval in one pipeline so one API call pattern serves both.

How it works, step by step

1. Label a sample of real questions with the lane that produced a correct answer and the number of sources required.
2. Build the index: chunk structure-aware, embed with plugsky-embed, store metadata and document hashes.
3. At query time, retrieve a candidate set and compute routing signals: top score, source spread and agreement.
4. Define thresholds per lane and route to the matching model tier.
5. Generate with citations, and validate that cited sources exist in the retrieved set.
6. Review misroutes weekly and adjust thresholds or the index, not just the model.

Original data

Try it yourself

Open the best model for RAG selector →

Why retrieval should drive routing

Query length and keyword lists are weak routing signals. Retrieval results are stronger because they arrive before generation and cost almost nothing to inspect.

• Direct lane: one chunk answers with high similarity. Send to plugsky-micro or plugsky-lite with a strict extractive instruction.
• Extraction lane: two or three chunks from one document agree. Still cheap, but allow a short synthesis of the excerpts.
• Synthesis lane: many chunks, multiple documents or conflicting statements. Route to plugsky-pro or plugsky-frontier.
• Miss lane: top scores below threshold. Ask a clarifying question or return a bounded refusal instead of generating unsupported text.

Implementation details

Compute the lane inside your retrieval function so routing stays testable and deterministic:

1. Embed the query once with plugsky-embed and reuse that vector for both the search and any cache lookup.
2. Retrieve a wider candidate set than you will send to the model, then select the final chunks under a character budget.
3. Derive signals: maximum similarity, mean of the top three, distinct source count and overlap between chunks.
4. Map signals to a lane with thresholds you can version in config.
5. Send the lane, context and instruction together so the prompt matches the model tier.

Store the lane and signals with every logged request. Without them you cannot tell whether a bad answer came from retrieval or routing.

Evaluating retrieval-driven routing

Measure the router against the same labeled question set you use for retrieval:

• Lane accuracy: did the query go to the lane that previously produced a correct answer?
• Miss handling: on unanswerable questions, how often does the system refuse instead of hallucinating?
• Quality per lane: pass rate and groundedness by lane, so cheap lanes are not quietly worse.
• Threshold sensitivity: how much quality moves when you shift each boundary by a small amount.

If the extraction lane fails more than expected, the cause is usually chunking, not the model. Fix retrieval before you upgrade the alias.

Limitations and trade-offs

Similarity scores are language, chunking and embedding-model dependent, so thresholds do not transfer between corpora. Additional honest limits:

• Multi-hop questions can look like misses after the first retrieval pass; an agent loop that performs a second query handles them better than a bigger model on thin context.
• Conflicting documents defeat single-pass routing; the synthesis lane needs instructions for handling disagreement.
• A stale index misroutes confidently, so track document freshness alongside scores.
• Plugsky returns embeddings rather than hosting an index, so ranking quality is your responsibility.
• Rerankers are not in the documented live stack today; evaluate any substitute before depending on it.

Honest comparison

Capability	Retrieval-driven routing	Length-based routing	Single model for all queries
Signal	Evidence score, source spread, agreement	Token count or keywords	None
Cheap lane	Clean single-source lookups stay cheap	Short queries only	Never
Miss handling	Explicit refusal or clarification path	Often unanswered	Confident hallucination risk
Evaluation	Lane accuracy and per-lane quality	Aggregate quality only	Aggregate quality only
Tuning	Thresholds versioned in config	Prompt length heuristics	Not applicable

Frequently asked questions

What is retrieval-driven routing?

It is the practice of scoring retrieved evidence before generation and choosing the model tier, or a refusal path, based on those scores.

Which similarity threshold should I use?

Start at 0.7 for cosine similarity with plugsky-embed on English corpora, then calibrate on labeled questions from your own domain. Thresholds do not transfer between embedding models.

Can a cheap model really answer RAG questions?

Yes for extractive questions where one passage contains the answer. Reserve stronger models for synthesis across sources and conflicting evidence.

How do I handle unanswerable questions?

Add a miss lane with a similarity floor. When nothing clears it, return a bounded refusal or ask a clarifying question instead of generating an answer.

Should I rerank before routing?

Reranking can improve ordering, but it is not part of the documented live Plugsky stack today. Build first on top-k similarity and your own scoring, and evaluate any reranker separately.

Does this work for Arabic or mixed-language corpora?

Yes. Use plugsky-embed-multilingual for the index and include per-language questions in the evaluation set, since similarity distributions differ by language.

Can I test this without paying?

Yes. The free plan includes two free models with no card, and a 14-day full-access trial unlocks the paid catalogue for routing experiments.