How do you build document Q&A with AI agents?

Key facts

Endpoint	POST /v1/chat/completions with tools for retrieval, lookup and verification
Retrieval	POST /v1/embeddings with plugsky-embed for each sub-query
Multi-hop handling	A planner decomposes the question and merges evidence across sub-queries
Verification	Check that each cited passage supports its claim; abstain when it does not
Models	30+ models; cheap aliases for sub-queries and stronger ones for synthesis
Pricing model	Flat monthly self-serve plans with unlimited fair-use usage
RAG status	RAG is live; embeddings plus chat completions are the building blocks
Roadmap	Rerankers are not in the documented live stack; batch ingestion is coming soon

TL;DR

• Decompose first: multi-part questions fail single-pass retrieval.
• Make retrieval a tool the agent calls once per sub-query.
• Verify citations before composing the final answer.
• Abstain when evidence is thin; a wrong answer costs more than a missing one.
• Track latency and step count — the loop is the cost of multi-hop accuracy.

How it works, step by step

1. Index the document set with structure-aware chunks and provenance metadata.
2. Build a planner step that decomposes the question into sub-queries with expected evidence types.
3. Expose retrieval as a tool so the agent fetches evidence for each sub-query.
4. Collect candidate passages, deduplicate them and check for contradictions.
5. Compose an answer with citations, then run a verification pass that confirms each citation supports its claim.
6. Abstain or ask a clarifying question when verification fails, and log the trace for evaluation.

Original data

Try it yourself

Open the chat with PDF tool →

Why document Q&A needs a loop

Single-pass retrieval answers lookup questions well and fails predictably everywhere else:

• Comparisons: 'Which contract has the longer termination notice?' needs two separate retrievals and a comparison step.
• Aggregates: totals and counts require finding every relevant passage, not the top few.
• Follow-ups: 'And what does the amendment change?' depends on the previous answer's context.
• Verification: fluent answers can cite passages that do not actually support them, so a check step is required.

The loop adds latency and work, but it converts unsupported answers into grounded ones or honest abstentions.

Architecture of an agentic Q&A system

Five components in a bounded loop:

• Planner: decomposes the question into sub-queries and states what evidence each needs.
• Retriever tool: embeds each sub-query with plugsky-embed, searches the index and returns passages with ids.
• Evidence store: collects passages across sub-queries, deduplicates and flags contradictions.
• Composer: drafts an answer with mandatory citations to evidence ids.
• Verifier: checks each claim against its cited passage and triggers abstention or another retrieval round.

Cap the loop at two or three retrieval rounds; beyond that, ask the user a clarifying question instead of burning steps.

Implementation and evaluation

Keep the loop tight and measurable:

• Use a cheap alias for planning and sub-query rewriting, and a stronger alias for composing multi-document synthesis.
• Return passages with stable ids so citations resolve and verification is mechanical.
• Deduplicate near-identical passages across sub-queries to avoid repetitive citation.
• Persist the retrieval trace with the answer so a wrong response can be replayed.

Metrics: sub-query recall (was the needed passage retrieved for each part?), citation support rate, answer accuracy on a labeled set, abstention correctness and steps per question. Compare against single-pass retrieval on the same set to confirm the loop is paying for itself.

Limitations

Agentic Q&A is more capable and more expensive:

• Latency and cost scale with the number of retrieval rounds; for simple lookups, skip the agent and answer directly.
• Contradictory documents can stall verification; define a conflict policy — present both with citations or escalate.
• A stale or incomplete index turns into confident abstentions or wrong answers; monitor freshness and coverage.
• Rerankers are not part of the documented live stack, and batch ingestion is coming soon, so initial large indexes need rate-limited jobs you operate.
• You own the index, access filtering and provenance; the model only reasons over evidence you supply.

Honest comparison

Capability	Agentic document Q&A	Single-pass RAG	Keyword search
Multi-hop questions	Plans sub-queries and merges evidence	Usually fails	Manual queries
Citations	Verified against supporting passages	Often asserted without checking	Links only
Abstention	Triggers when verification fails	Rarely	Not applicable
Latency	Higher due to multiple rounds	Low	Lowest
Evaluation	Sub-query recall plus citation support	Answer accuracy only	Click-through

Frequently asked questions

What is agentic document Q&A?

A retrieval loop where an agent decomposes a question, retrieves evidence for each part, composes a cited answer and verifies that citations support the claims before responding.

When is a single retrieval pass enough?

For direct lookup questions with one obvious source. Use the agent loop only when questions involve comparisons, aggregates or follow-ups that need several retrievals.

How do I stop the agent from looping forever?

Cap retrieval rounds at two or three, then ask a clarifying question or abstain. Track steps per question and investigate outliers.

How can citations be verified?

Retrieve passages with stable ids, require the answer to reference those ids, and run a verification step that checks each claim against its cited passage before returning the answer.

Which models should the loop use?

Cheap aliases such as plugsky-micro or plugsky-lite for planning and sub-query rewriting, with a stronger alias for multi-document synthesis and verification.

What should the system do with contradictory evidence?

Apply an explicit policy: present both passages with citations, or escalate to a human. Silent resolution of conflicts hides risk.

Can I build this on the free plan?

Yes for prototyping. Embeddings and chat completions are live, two free models are available with no card, and the 14-day full-access trial supports evaluation with stronger aliases.