How do you build multi-agent systems with RAG?

Key facts

Pattern	Shared retrieval service, scoped tools per agent
Embeddings	plugsky-embed and plugsky-embed-multilingual via /v1/embeddings (live)
Generation	Cited synthesis with streaming and JSON mode (live)
Models	30+ models behind one endpoint, route per agent
Scoping	Access labels filtered inside the retrieval tool, per agent
Citations	Chunk IDs carried through handoffs to the final answer
Deployment	Region-locked planes, VPC, on-prem and air-gapped options
Free tier	plugsky-micro and plugsky-lite on the free plan, no card required

TL;DR

• Build retrieval once as a service; let every agent call it through a scoped tool.
• Pass chunk IDs and evidence between agents so citations survive synthesis.
• Give each agent a query template and filters suited to its sub-task.
• Deduplicate retrieved passages before synthesis to keep prompts lean.
• Evaluate retrieval and synthesis separately, then evaluate the whole pipeline.

How it works, step by step

1. Index the corpus once with plugsky-embed (or plugsky-embed-multilingual) and store access labels per chunk.
2. Wrap retrieval in one tool that resolves scope server-side, applies filters and returns passages with stable IDs.
3. Assign each agent a retrieval profile: which sources, which filters and how many passages it may read.
4. Have workers return structured findings — claim, evidence IDs, confidence — rather than prose.
5. Pass findings and IDs through typed handoffs, and deduplicate evidence before the synthesiser runs.
6. Generate the final answer with mandatory citations mapped back to source documents.
7. Trace retrieval IDs per agent and score recall, groundedness and citation accuracy end to end.

Try it yourself

Open the RAG architecture builder →

Retrieval as a shared service for agents

In a multi-agent system, retrieval is infrastructure, not a per-agent feature. One index, one embedding pipeline, one retrieval tool with server-side scoping — then each agent calls it with a profile matched to its sub-task. This avoids duplicated indexes, inconsistent chunking and the classic bug where one agent receives passages another agent should never see.

The model side is live on Plugsky: /v1/embeddings for indexing and /v1/chat/completions with function calling for the agent loops. Retrieval services and access rules stay in your code, which is where scoping decisions must live.

Scoping and handoffs for retrieved context

The hard part is what travels between agents. Passing full passages through several handoffs inflates prompts and loses provenance. Instead, have each worker return structured findings that reference evidence by ID, then let the synthesiser fetch or receive the passages it actually needs, deduplicated.

• Scope per agent: a research agent may read broadly while a drafting agent sees only approved sources.
• Stable IDs: chunk and document IDs let citations survive every handoff.
• Freshness: carry updated dates so superseded passages are demoted, not silently used.
• Budgets: cap passages and tokens per agent so retrieval cannot crowd out reasoning.

Cost, citation and evaluation

Retrieval-heavy pipelines amplify model costs, so route aggressively: small models for extraction and filtering, stronger models for synthesis. With 30+ models behind one endpoint, each agent's model is a configuration value. Cache embeddings, reuse retrieval results across agents working the same sub-question, and trim passages before they reach a prompt.

Evaluate in layers. Retrieval first: recall@k against questions with known source passages. Then synthesis: groundedness, citation accuracy and refusal behaviour. Finally the pipeline: does the orchestrator route work correctly, and does evidence survive handoffs intact? Persist traces with retrieval IDs per agent so any failure can be attributed to the right stage. For sensitive corpora, the same design runs in a region-locked plane or on-prem via Plugsky deployments.

Honest comparison

Concern	Multi-agent RAG	Single RAG assistant	Text search plus chat
Retrieval	One scoped service, profiles per agent	One retrieval path	Manual copy-paste
Evidence handling	Structured findings with chunk IDs	Inline passages	None
Model routing	30+ models, per agent	One model	One model
Citations	Preserved through handoffs	Present if prompted	Rarely
Evaluation	Retrieval and synthesis scored separately	End-to-end only	None

Frequently asked questions

Do all agents need access to the same index?

One index is simpler and cheaper, but agents should have different retrieval profiles. Enforce source and access filters server-side per agent so scope is a tool property, not a prompt request.

How do citations survive agent handoffs?

Carry stable chunk and document IDs in structured findings, and map them back to source links at synthesis. Passages that lose their IDs cannot be cited accurately later.

Is agentic RAG worth the complexity?

When a task needs decomposition — comparing sources, separating extraction from analysis — yes. For straightforward question answering, a single RAG loop is cheaper and easier to debug.

Which models should each agent use?

Small models for retrieval filtering and extraction, stronger models for cross-source synthesis. Everything sits behind one endpoint, so routing is a configuration change per agent.

How do we keep prompts from exploding?

Return findings with evidence IDs rather than full text, deduplicate passages, and cap the number of chunks and tokens each agent may consume.

How should we evaluate the pipeline?

Score retrieval recall first, groundedness and citations second, then orchestration correctness with end-to-end tasks. Traces with retrieval IDs make attribution possible.

Can it run on-prem?

Yes. Plugsky supports region-locked cloud planes plus VPC, on-prem and air-gapped deployment, and retrieval infrastructure stays under your control.