How do you build voice assistants with RAG?

Key facts

Embeddings	Live embeddings API for spoken-query retrieval over your knowledge base
Generation	OpenAI-compatible chat completions with streaming (live)
Audio endpoints	Speech-to-text and text-to-speech are coming soon; bring your own STT/TTS today
Answer style	Prompt for one or two spoken sentences with no formatting
Models	30+ models; route routine spoken queries to plugsky-micro or plugsky-lite
Pricing	Flat monthly self-serve plans with no per-token charges on self-serve
Free tier	plugsky-micro and plugsky-lite on the free plan, no card required
Roadmap	Files and batch endpoints are coming soon; ingest your corpus in your own pipeline

TL;DR

• Retrieve on the transcript, then compress the answer for speech.
• Target one or two sentences; long answers cannot be skimmed by ear.
• Stream text so TTS starts before the full answer exists.
• When retrieval is weak, say so and offer a human or follow-up.
• Measure retrieval hit rate and turn latency, not just answer accuracy.

How it works, step by step

1. Ingest your knowledge sources, chunk them, and embed with plugsky-embed or plugsky-embed-multilingual.
2. Connect an STT provider and normalize transcripts: casing, numbers and likely misrecognitions.
3. Retrieve top candidates for the transcript, filtering by permissions and locale.
4. Generate a spoken-style answer constrained to the retrieved context, with a one-sentence fallback when evidence is weak.
5. Stream the answer and synthesize sentence by sentence with your TTS provider.
6. Handle follow-up questions with a short conversation memory instead of resending everything.
7. Evaluate with recorded spoken questions: retrieval hits, answer correctness, sentence length and latency.

Try it yourself

Open the RAG chunk size calculator →

Spoken queries need tighter retrieval

Speech-to-text output is messier than typed text: missing punctuation, phonetic errors, half-finished sentences. Before retrieval, normalize the transcript and consider a cheap model pass that rewrites it into a clean query. Then retrieve a small number of high-confidence chunks — precision matters more than recall here, because a wrong passage becomes a confidently spoken wrong answer.

Keep locale and language in metadata so retrieval respects the conversation's language, and filter by user permissions before ranking. For names and numbers, blend keyword search with vector search; embeddings alone miss exact identifiers that speech recognition may already have distorted.

Answer shaping for text-to-speech

An answer that reads well often sounds terrible. Prompt for one or two short sentences, no lists, no markdown, no parentheticals, and no raw identifiers unless asked. When the retrieved context is insufficient, the assistant should say it does not know and offer the next step instead of improvising.

• Use max_tokens sized for speech, not for completeness.
• Stream tokens and synthesize complete sentences as they finish.
• Spell out ambiguous numbers and confirm critical ones back to the caller.
• Keep a short rolling summary of the conversation so follow-ups stay grounded.

Latency budget and fallbacks

Every layer adds wait: recognition, retrieval, generation, synthesis. Measure each separately and set a per-turn budget, then degrade gracefully when a layer is slow — a shorter answer beats a perfect answer that arrives late. If the model stalls, speak a holding phrase while the stream catches up, and cancel everything when the caller interrupts.

Evaluate with recorded spoken questions and track retrieval hit rate, answer correctness, spoken length and end-to-end latency. Audio endpoints are coming soon, so keep the brain decoupled from your STT/TTS choices. The free plan with plugsky-micro and plugsky-lite covers development, and the 14-day full-access trial tests stronger models on hard questions.

Honest comparison

Concern	RAG voice assistant on Plugsky	Chat-only RAG app	Scripted IVR
Knowledge	Retrieved from your corpus	Retrieved from your corpus	Fixed decision tree
Answer style	Short spoken sentences	Paragraphs with citations	Recorded prompts
Latency	Streamed and synthesized early	Time to full answer	Immediate
Coverage	Open-ended questions	Open-ended questions	Predefined paths
Extensibility	Add documents by re-indexing	Same	Rebuild the tree

Frequently asked questions

Do I need Plugsky audio endpoints for a voice assistant?

They are not required today. Audio endpoints are coming soon, so you connect your own speech-to-text and text-to-speech providers and use Plugsky for retrieval and generation, which are live.

How do I make answers sound natural?

Prompt for one or two spoken sentences with no formatting, cap the output length, and stream text into synthesis as sentences complete. Avoid lists and parentheticals.

What if speech recognition gets the question wrong?

Normalize the transcript, optionally rewrite it with a cheap model, and confirm critical details such as names and numbers back to the caller before acting on them.

Should retrieval use the transcript directly?

Clean it first. Punctuation and misrecognitions hurt embedding quality, so a light normalization or rewrite pass improves retrieval accuracy.

How many chunks should I retrieve for a spoken answer?

Fewer than for a text answer. Precision matters more than recall because the response is short; two or three strong passages usually beat a larger, noisier set.

How do I keep turn latency down?

Stream everything: start synthesis on the first complete sentence, keep answers short, and route routine queries to plugsky-micro or plugsky-lite. Measure each layer separately.

Can it handle multiple languages?

Yes, with the right embedding and generation models. Use plugsky-embed-multilingual and multilingual chat tiers, and evaluate recognition and answer quality per language.

Can I build it for free?

Yes. The free plan includes plugsky-micro and plugsky-lite with no card, and a 14-day full-access trial is available for stronger models.