Key facts
| Vision | Image content in chat completions messages (live) |
| Embeddings | /v1/embeddings for semantic search and clustering (live) |
| RAG | Built-in vector store and per-project retrieval collections (live) |
| Body limit | Maximum request body of 16 MB |
| Image generation | /v1/images/generations with plugsky-imagine-xl and plugsky-imagine-fast — coming soon |
| Audio | Transcription and speech endpoints with whisper-plugsky — coming soon |
| Compatibility | Same OpenAI SDKs and frameworks as text-only calls |
| Moderation | /v1/moderations is coming soon |
TL;DR
- Vision, embeddings and RAG are live and OpenAI-compatible.
- Send images as message content; keep pages under the 16 MB body limit.
- Combine vision extraction with RAG retrieval in one pipeline.
- Build an abstraction now so image and audio endpoints drop in later.
- Test the document pipeline with the chat-with-PDF tool before shipping.
How it works, step by step
- Decide which live capability the feature needs: vision extraction, semantic search or retrieval-augmented answers.
- Send images or scans as message content in chat completions and force structured output with JSON mode.
- Chunk and embed your document corpus, then create a retrieval collection per project or use case.
- Validate every extraction against a schema and return failures to the model for a retry.
- Split large documents into bounded pages to stay inside the 16 MB request limit.
- Design one internal multimodal interface so the coming-soon image and audio endpoints slot in without refactoring.
Original data
Try it yourself
A document QA app with what is live
The fastest useful build is document question answering. Take a PDF page or screenshot, send it as image content in the messages array, and ask for structured extraction with JSON mode enabled. Ingest the same corpus into embeddings, store vectors in a per-project collection, and retrieve passages to answer questions with citations.
Because the shape is OpenAI-compatible, this composes with the frameworks you already use. LlamaIndex or LangChain handle chunking and retrieval; your model call stays a chat completion with tools if you need function calling alongside vision.
Planning for image and audio
Two endpoint families are coming soon and should not be in your critical path: image generation via /v1/images/generations with the plugsky-imagine models, and audio via /v1/audio/transcriptions and /v1/audio/speech with whisper-plugsky planned for speech-to-text and a voice catalogue for speech output.
- Abstraction: wrap model calls behind one internal interface so a new modality is a provider change, not an app rewrite.
- Feature flags: ship UI behind flags so enabling generation or transcription later is a toggle.
- Honesty: tell users what is live; do not stub capabilities you cannot deliver.
Handling files, limits and quality
Two practical constraints shape the build. First, request bodies cap at 16 MB, so split multi-page scans into page-sized calls and stitch results afterwards. Second, vision and extraction are probabilistic: validate output against a strict schema, retry on failure, and surface uncertainty rather than silently accepting a malformed field.
For retrieval quality, chunk deliberately and keep collections scoped — one per project or use case — so a support corpus cannot pollute a legal corpus. Log token usage and model choice per request for observability, and route heavy vision work to a capable tier while keeping classification cheap on 30+ available models.
Honest comparison
| Capability | Plugsky status | DIY stack | Third-party multimodal API |
|---|---|---|---|
| Vision understanding | Live in chat completions | Self-hosted vision model | Separate endpoint |
| Embeddings | Live endpoint | Open-source embedding server | Vendor lock-in |
| Retrieval | Built-in RAG collections | You build retrieval | Vendor-managed |
| Image generation | Coming soon | Diffusion pipeline | Available |
| Audio | Coming soon | ASR and TTS services | Available |
Frequently asked questions
How do I send an image to the model?
Include image content in the chat completions messages array, exactly as the OpenAI vision shape works, and keep the payload within the 16 MB request limit.
Can I use my existing SDK?
Yes. Vision, embeddings and RAG calls work through the same OpenAI-compatible endpoints, so the SDK and framework code you already have applies.
Can I generate images or transcribe audio?
Not yet. Image generation and audio endpoints are coming soon per the docs; build an abstraction so you can enable them without refactoring.
How should I chunk documents for RAG?
Chunk on natural boundaries — sections or pages — and keep collections scoped per project or use case so retrieval stays relevant.
What happens with very large PDFs?
Split them into per-page or per-section calls to respect the body limit, then combine results in your application layer.
How do I improve extraction reliability?
Use JSON mode or a strict schema, validate every response, retry failures with the error returned to the model, and route low-confidence cases to a human.
Is there a tool to test a document pipeline?
Yes. The chat with PDF tool lets you try retrieval and answering over a document before wiring the pipeline into your app.