Key facts
| Pattern | Rewrite → retrieve → judge → retry → cited answer |
| Runtime | Function calling loop on /v1/chat/completions (live) |
| Embeddings | plugsky-embed and plugsky-embed-multilingual via /v1/embeddings (live) |
| Models | 30+ models behind one endpoint, route per step |
| Guardrails | Turn and tool-call budgets enforced in code |
| Citations | Chunk IDs carried into structured answer output |
| Observability | Per-turn traces with queries, hits and scores |
| Free tier | plugsky-micro and plugsky-lite on the free plan, no card required |
TL;DR
- Make retrieval a tool the agent can call with different queries, not a fixed step.
- Add a self-check: does the retrieved evidence actually answer the question?
- Bound the loop with turn and tool budgets so it terminates predictably.
- Retry with filters, synonyms or another source before escalating to a human.
- Trace every retrieval attempt so quality issues are diagnosable.
How it works, step by step
- Build the retrieval service first: chunking, embeddings and a search tool that returns passages with stable IDs and scores.
- Give the agent tools for search, document fetch and, where relevant, structured filters such as date or entity.
- Write the loop so the agent rewrites the query, retrieves, and evaluates evidence sufficiency before answering.
- Add a retry policy: reformulate, relax or tighten filters, or switch sources, up to a fixed budget.
- Require structured output — answer, cited chunk IDs and a confidence indicator — or an explicit cannot-answer result.
- Route rewriting and judging to small models and final synthesis to a stronger one.
- Evaluate retrieval attempts and answers on a golden set, and log each turn for failure analysis.
Original data
Try it yourself
What makes RAG agentic
Classic RAG embeds the question once, retrieves top-k, and generates. It fails quietly when the first query is poorly phrased, the filter is wrong, or the evidence is split across documents. Agentic RAG adds judgement: the model sees what came back, decides whether it is sufficient, and acts again if not.
On Plugsky that is a function-calling loop over the live chat endpoint. Retrieval is exposed as a tool with typed arguments, so the agent controls the query, filters and result count. Embeddings run on /v1/embeddings, and 30+ models behind the same endpoint let each step run on the cheapest model that performs well.
Query rewriting and evidence judging
Two additions do most of the work. Query rewriting expands abbreviations, adds synonyms and extracts filters the user implied, such as a date range or product name. Evidence judging asks a direct question: do these passages contain enough information to answer, and which passage supports which claim?
- Rewrite small: produce two or three variants, not ten; retrieve in parallel.
- Judge cheaply: a small model can classify sufficiency from passage summaries.
- Escalate honestly: if the budget is exhausted, return the best sources instead of guessing.
- Keep IDs: every passage carries a stable ID so citations survive retries.
Budgets, evaluation and operations
An open-ended loop is a cost and latency risk. Cap turns, tool calls and tokens per request, and record why a retry happened. A useful default is one rewrite plus one relaxed-filter retry before answering, but tune it against your own data rather than assuming.
Evaluate agentic RAG differently from a fixed pipeline: measure first-pass retrieval success, retry success, final groundedness and citation accuracy, and how often the agent correctly declines. Those numbers tell you whether the extra loop earns its cost. Trace every attempt — query, filters, hits, scores — so a bad answer can be attributed to rewriting, retrieval or synthesis. Deploy in a region that matches your data policy, including on-prem where required.
Honest comparison
| Concern | Agentic RAG | Fixed retrieve-then-answer | Search plus chatbot |
|---|---|---|---|
| Query handling | Rewrites and retries as needed | Single query | Manual reformulation |
| Evidence check | Explicit sufficiency judgement | None | User judgement |
| Failure mode | Declines or retries | Answers anyway | Open-ended chat |
| Cost control | Turn and tool budgets | One pass | Unbounded |
| Observability | Per-attempt traces | Single log line | None |
Frequently asked questions
Is agentic RAG always better?
No. For simple FAQ retrieval, a fixed pipeline is cheaper and easier to operate. Add the loop when questions are ambiguous or evidence is fragmented across sources.
How many retries should the agent get?
One or two in most systems, with a hard cap. Beyond that, escalate to a human or return the best sources with a clear signal that the evidence was insufficient.
Which model rewrites the query?
A small, fast model usually suffices — plugsky-micro or plugsky-lite. Reserve stronger models for judging complex evidence and writing the final answer.
How do we prevent invented answers?
Require citations to retrieved chunk IDs and allow an explicit cannot-answer output. The evidence judgement step exists precisely to authorise refusal.
Does this need function calling?
Yes, or an equivalent structured output path. Function calling is live on Plugsky and is the simplest way to let the agent choose retrieval tools.
How do we evaluate it?
Score first-pass retrieval, retry success, groundedness, citation accuracy and correct refusals. Compare against a fixed pipeline to confirm the loop is worth its cost.
Can it run on-prem?
Yes. Plugsky supports region-locked cloud planes plus VPC, on-prem and air-gapped deployment, and retrieval infrastructure remains under your control.