How do you build document Q&A with RAG?

Key facts

Pipeline	Parse → chunk → embed → retrieve → generate with citations
Embeddings	plugsky-embed and plugsky-embed-multilingual via /v1/embeddings (live)
Generation	OpenAI-compatible /v1/chat/completions with streaming and JSON mode (live)
Models	30+ models from free to frontier behind one endpoint
Long context	plugsky-longctx for synthesising across many retrieved passages
Deployment	Plugsky cloud, your VPC, on-prem or air-gapped
Free tier	plugsky-micro and plugsky-lite on the free plan, no card required
Roadmap endpoints	Files and batch endpoints are coming soon — run parsing in your own pipeline today

TL;DR

• Treat document Q&A as retrieval first, generation second — chunking decides quality.
• Use plugsky-embed, or plugsky-embed-multilingual when documents and queries mix languages.
• Keep source, page, section and access labels in vector metadata so you can filter before you rank.
• Instruct the model to answer only from retrieved context and to refuse when context is missing.
• Run a golden question set on every change and track citation accuracy, not just answer fluency.

How it works, step by step

1. Collect a small, representative corpus (30–100 documents) and write 20 questions with known answers and expected sources.
2. Parse documents to text, preserving headings and page numbers; run OCR separately for scanned pages before chunking.
3. Split text into overlapping chunks of a few hundred tokens and attach source, page, section and access-label metadata.
4. Embed every chunk with plugsky-embed (or plugsky-embed-multilingual) and upsert the vectors plus metadata into your store.
5. At query time embed the question with the same model, filter by the user's permissions, and retrieve the top candidates.
6. Send the retrieved passages to a chat model with a strict answer-only-from-context instruction and a citation format.
7. Evaluate recall, citation accuracy and refusals on your golden set, then tune chunk size, overlap and top-k before launch.

Original data

Try it yourself

Open the RAG sandbox →

The four-stage RAG pipeline

Document Q&A is a retrieval problem before it is a generation problem. The pipeline has four stages: parse each document into text, split it into overlapping chunks, embed every chunk with an embedding model, and write the vectors plus metadata into a store you control. At query time you embed the question with the same model, retrieve the closest chunks, and pass them to a chat model as context.

Plugsky covers the two model calls — embeddings and generation — through one OpenAI-compatible API. Parsing, chunking and the vector store stay in your code, which is where document permissions, tenancy and residency belong.

Chunking, embeddings and retrieval quality

Chunk size is the highest-leverage setting in the stack. Chunks that are too small lose context; chunks that are too large dilute the embedding and crowd the prompt. Start with a few hundred tokens and one or two sentences of overlap, keep headings and page numbers in metadata, and measure recall against questions you already know the answers to.

• Embeddings: use plugsky-embed for English-heavy corpora and plugsky-embed-multilingual when documents and queries mix languages.
• Metadata: store source, page, section and access labels next to every vector so you can filter before you rank.
• Retrieval: blend keyword search with vector search; pure embeddings miss exact identifiers such as part numbers and clause numbers.
• Generation: require the model to answer only from the provided context and to say when the context is insufficient.

Production concerns: permissions, freshness and evaluation

In production the questions change from "does it work" to "is it safe and current". Filter retrieval by the requesting user's permissions, not just by tenant, because the model can only leak what you place in the prompt. Re-index when source documents change, and store the document version alongside the vector so stale chunks can be expired.

Run every release against a golden set of questions with expected sources. Track citation accuracy — did the answer cite the document that actually contains the answer — and refusal quality on questions the corpus cannot answer. Log retrieval hits and misses so failures can be traced to parsing, chunking, ranking or generation, and keep every cited source one click away for the reader.

Honest comparison

Capability	Plugsky	Vector-only pipeline	Building in-house
Embeddings	plugsky-embed and plugsky-embed-multilingual (live)	Third-party or self-hosted	Train or host your own model
Generation	30+ chat models behind one OpenAI-compatible API	Bring your own model endpoint	Serve every model yourself
Citations	JSON mode plus prompt patterns for source-linked answers	Manual formatting	You design the schema
Deployment	Cloud, VPC, on-prem and air-gapped options	Varies by vendor	Full control at full ops cost
Time to first answer	Managed model layer, your retrieval stack	Managed retrieval, separate model	Months of platform work

Frequently asked questions

What do I need to build document Q&A with RAG?

A parser, a chunking step, an embedding model, a vector store and a chat model. Plugsky provides the embedding and chat calls through one OpenAI-compatible API; parsing, chunking and storage stay in your application.

Which embedding model should I use?

Use plugsky-embed for English-heavy corpora and plugsky-embed-multilingual when documents or queries mix languages. Whichever you choose, embed documents and queries with the same model.

Does Plugsky store my documents?

No. Documents stay in your storage and vector database. Only the text you send as prompt context reaches the model, and prompt retention is configurable per workload.

Can I keep using the OpenAI SDK?

Yes. The chat and embeddings endpoints are OpenAI-compatible, so you change the base URL and model names and keep your existing client code.

How do I stop the model from making things up?

Constrain it to the retrieved context, require a citation for every claim, and let it refuse when the context is insufficient. Then measure citation accuracy on a golden question set.

Is there a free way to start?

Yes — the free plan includes plugsky-micro and plugsky-lite with no card required. A 14-day full-access trial is available when you need frontier models.

Do I need a vector database?

For a demo you can score embeddings in memory. For anything production-grade, use a vector database or pgvector so you get metadata filtering, index updates and permission-aware retrieval.