How do you build voice assistants with AI agents?

Key facts

Agent brain	Chat completions, streaming and function calling are live
Agent runtime	Agents with memory, tools and orchestration are live
Audio endpoints	Speech-to-text and text-to-speech endpoints are coming soon; use your own STT/TTS today
Models	30+ models; route short turns to plugsky-micro or plugsky-lite and hard ones upward
Pricing	Flat monthly self-serve plans with no per-token charges on self-serve
Free tier	plugsky-micro and plugsky-lite on the free plan, no card required
Deployment	Cloud, VPC, on-prem or air-gapped for sensitive audio pipelines
Governance	Scoped keys, RBAC and audit logs for every agent action

TL;DR

• Bring your own STT and TTS; Plugsky runs the agent brain today.
• Design for short turns; spoken answers must be briefer than chat replies.
• Stream text so TTS can start before the reply finishes.
• Put lookups and actions in tools, not in the model's memory.
• Budget latency end to end: recognition, model, synthesis.

How it works, step by step

1. Pick STT and TTS providers that fit your languages and latency budget, and pin their versions.
2. Build the agent loop on chat completions with tools for lookups, bookings and account actions.
3. Design prompts for speech: short sentences, no markdown, confirmations for any action.
4. Stream model output and feed sentence fragments to TTS as they complete.
5. Handle barge-in: when the caller speaks, cancel the model stream and stop playback.
6. Set per-turn timeouts and a fallback line for slow or failed turns.
7. Test with real accents and noisy audio, and measure recognition error, tool accuracy and turn latency.

Try it yourself

Open the AI agent prompt generator →

Anatomy: STT, agent brain, TTS

Divide the assistant into three failure domains. Speech-to-text turns audio into a transcript and owns recognition errors. The agent brain decides what to say and does, using the conversation, memory and tools. Text-to-speech turns the reply back into audio. Plugsky supplies the brain today — chat completions, streaming, function calling and agent orchestration are live — while audio endpoints are coming soon, so the STT and TTS layers come from providers you choose and validate.

Keeping the brain provider-agnostic matters because speech vendors change often. Your orchestration layer should accept a transcript and return a reply stream, with tools as the only way to affect the outside world.

Designing agents for spoken turns

Speech punishes long answers. Prompts should demand short sentences, no formatting, and explicit confirmation before any action with a side effect. Replace lists with a question — 'do you want the morning or afternoon slot?' — and keep one decision per turn. Memory needs to be tighter than in text chat: store the caller's intent, confirmed facts and pending action, and summarize the rest.

• Expose tools such as find_booking, change_booking and escalate_to_human with strict schemas.
• Read numbers and identifiers back before acting; recognition confuses similar digits.
• Detect frustration and offer a human rather than looping.
• Keep a text transcript for audit even when the interaction is voice-first.

Latency, barge-in and evaluation

The user experience is the sum of recognition, model and synthesis time. Stream the model's text, synthesize sentence by sentence, and start speaking before the full reply exists. When the caller interrupts, cancel the stream and the playback immediately; a talkative assistant that cannot be stopped feels broken regardless of answer quality.

Evaluate with recorded calls: word error rate from your STT, tool selection accuracy, task completion, unnecessary confirmations and median turn latency. Because audio endpoints are not live yet, build the brain first against text transcripts, then swap in audio when the endpoints ship. The free plan with plugsky-micro and plugsky-lite covers development, and the 14-day full-access trial lets you test stronger models on difficult calls.

Honest comparison

Layer	Voice assistant on Plugsky	All-in-one voice platform	Fully self-hosted
Agent brain	Live chat completion and tool calling	Platform-managed	You host and scale models
Speech	Bring your own STT/TTS; Plugsky audio endpoints coming soon	Built in	Self-host ASR and TTS
Tools	Function calling with your permission checks	Platform actions	Custom loop
Model choice	30+ models behind one endpoint	Vendor models	Models you can run
Data control	Cloud, VPC, on-prem, air-gapped	Vendor cloud	Full control, full ops cost

Frequently asked questions

Does Plugsky provide speech-to-text and text-to-speech?

Not yet. Audio endpoints are coming soon. Today Plugsky provides the agent brain through live chat completions and function calling, and you connect an STT and TTS provider of your choice.

How do I keep voice answers short?

Write speech-specific prompts: one decision per turn, no formatting, explicit confirmations before actions. Summarize memory aggressively and ask a question instead of listing options.

Can the assistant interrupt or be interrupted?

Barge-in is your responsibility in the audio layer: detect caller speech, cancel the model stream and stop TTS playback. Design the pipeline so cancellation propagates to the model call.

How do I stop wrong actions?

Route every side effect through a tool with server-side permission checks, confirm details back before executing, and keep an audit log of tool calls with the transcript.

Which models should handle calls?

Route short routine turns to plugsky-micro or plugsky-lite and escalate complex requests to stronger tiers. All 30+ models sit behind one OpenAI-compatible endpoint.

What latency should I target?

There is no universal number, but perception degrades quickly as turn latency grows. Measure recognition, time to first token and synthesis separately, then stream everything you can.

Can it run on-prem?

Yes. Plugsky deploys in our cloud, your VPC, on-prem or air-gapped, so sensitive call pipelines can stay inside your perimeter.

Can I start without paying?

Yes. The free plan includes plugsky-micro and plugsky-lite with no card, and a 14-day full-access trial is available for stronger models.