Key facts
| Definition | Models that process multiple input or output modalities in one request |
| Why it matters | Documents, screenshots and diagrams no longer need separate OCR pipelines |
| Plugsky support | Vision-capable chat models are reachable through the OpenAI-compatible chat completions API |
| Typical inputs | Text plus images; audio and video support varies by model |
| Use cases | Document Q&A, UI testing, chart reading, accessibility and content moderation review |
| Status | Multimodal chat is live; image generation is a coming-soon endpoint |
| Models | 30+ models behind one API, with capability documented per model |
| Cost driver | Images consume tokens based on resolution and count, so size inputs deliberately |
TL;DR
- Multimodal means more than text in one request — usually text plus images.
- Vision models remove separate OCR steps for many document workflows.
- Image token cost scales with resolution, so downscale where possible.
- Capability differs per model — check the catalogue before committing.
- Plugsky serves multimodal chat live; image generation is coming soon.
How it works, step by step
- Identify where images carry information today: screenshots, scans, charts or photos.
- Check which models in the catalogue accept image input for your task type.
- Encode images per the API format and send them alongside the text prompt.
- Downscale images to the minimum resolution that still answers the question.
- Validate outputs against extracted ground truth on a labeled sample.
- Fall back to OCR plus a text model where vision quality or cost does not fit.
Try it yourself
Open the best AI model selector →
What multimodal means in practice
A text-only model sees characters. A multimodal model receives a prompt built from several modalities — commonly text and images — and reasons across them. In practice this means you can send a screenshot of a broken UI and ask what is wrong, or a scanned invoice and ask for structured fields. The model handles the perceptual step that previously required a separate OCR or computer-vision pipeline.
When to use multimodal models
- Document understanding: scanned forms, invoices and PDFs where layout matters.
- UI and QA: screenshot-based regression triage and accessibility review.
- Data extraction: reading charts, tables and gauges from images.
- Support: customers attach screenshots and expect an answer in one turn.
- Content review: flagging policy issues in uploaded images.
If the input is already clean text, a text model is cheaper and faster — use vision only when the image itself carries the information.
Common mistakes
- Sending full-resolution photos when a downscaled image answers the question, inflating token cost.
- Assuming every model in the catalogue accepts images — capability varies and must be checked per model.
- Trusting extracted numbers without a verification step on financial or medical data.
- Mixing many images into one request and losing track of which one supports the answer.
- Ignoring privacy rules when images contain faces, identity documents or customer data.
How Plugsky implements it
Plugsky serves multimodal chat through the OpenAI-compatible chat completions endpoint, so images are passed in the same message format used by the OpenAI SDK — change the base URL and your existing code path works. Vision-capable models are listed in the catalogue with their capabilities. Embeddings and RAG remain live for text pipelines that complement vision, while image generation is documented as a coming-soon endpoint. For regulated images, private deployment options keep traffic inside your network.
Honest comparison
| Approach | Multimodal model | OCR plus text model | Custom computer vision |
|---|---|---|---|
| Setup effort | Low — one API call | Medium — OCR pipeline | High — training and serving |
| Layout understanding | Strong on documents and UIs | Varies by OCR engine | Task-specific |
| Flexibility | General-purpose prompts | Fixed extraction schema | Narrow |
| Cost | Image tokens per request | OCR plus text tokens | GPU hosting |
| Best for | Mixed visual and text reasoning | High-volume structured forms | Narrow repetitive checks |
Frequently asked questions
What are multimodal models?
AI models that process more than one data type in a single request — most commonly text and images, sometimes audio or video — and reason across the combined input.
Which plugsky models support images?
Vision support varies by model, so check the model catalogue for current capabilities before building a workflow that depends on image input.
Do multimodal models replace OCR?
For many document and screenshot workflows, yes — the model reads the image directly. For very high-volume, fixed-schema extraction, a dedicated OCR pipeline can still be cheaper.
How does image cost work?
Images are converted to tokens based on resolution and count, so they consume part of the context window. Downscale images to the smallest size that still contains the information.
Can Plugsky generate images?
Image generation is documented as a coming-soon endpoint. Multimodal understanding — text plus images in chat — is live today.
Is multimodal chat OpenAI-compatible?
Yes. Images are sent in the same message format as the OpenAI API, so existing SDK code works after changing the base URL.