What can you build with Plugsky vision and multimodal models?

Key facts

Vision input	Image content blocks on POST /v1/chat/completions
Vision models	plugsky-qwen-vl, plugsky-vision-fast among the 30+ model catalogue
Good fits	Document extraction, screenshots, chart reading, inspection, captioning
Structured output	Combine vision with JSON mode for typed extraction
Image generation	POST /v1/images/generations is coming soon and returns 501
Audio endpoints	Transcription and speech are coming soon
Deployment	Region pinning, VPC, on-prem and air-gapped options
Product status	Vision inputs live; image generation, audio and moderation coming soon

TL;DR

• Vision is an input type on chat completions — no separate endpoint to learn.
• Pick a vision-capable model and check its context limits on /models.
• Pair vision with JSON mode to turn images into typed data.
• Image generation and audio are roadmap items, not shippable today.
• Downscale and crop images before upload to control token cost and latency.

How it works, step by step

1. Confirm the task needs vision rather than OCR plus text reasoning.
2. Choose a vision-capable model from the catalogue and read its image limits.
3. Downscale or crop images so the important region is large in the frame.
4. Send the image content with a precise extraction or description instruction.
5. Request JSON with a defined schema when downstream code consumes the result.
6. Validate output against the schema and retry with a correction if it fails.
7. Measure accuracy on a labelled image set before rolling out to users.

Original data

Try it yourself

Open the OpenAI-compatible API tester →

What multimodal means at the API level

Vision is not a separate service. You send a normal chat completion whose user message contains both text and image content, and a vision-capable model reasons over both:

resp = client.chat.completions.create(
    model="plugsky-qwen-vl",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Extract the invoice number and total as JSON."},
            {"type": "image_url", "image_url": {"url": image_url_or_data_uri}},
        ],
    }],
    response_format={"type": "json_object"},
)

Because it is the same endpoint, streaming, function calling and structured output compose with vision rather than requiring new plumbing.

Builds that work today

• Document extraction: invoices, receipts, ID documents and forms into typed JSON, with a review step for low-confidence fields.
• Screenshot and UI understanding: support triage, accessibility checks and test verification from screenshots.
• Chart and table reading: convert visuals in reports into structured series for downstream analysis.
• Inspection and defect spotting: flag anomalies in product photos and route them to a human queue.
• Captioning and alt text: multilingual descriptions for media libraries, paired with a moderation policy for user-generated content.

Image handling and cost awareness

Image resolution and count drive both latency and token usage. Downscale to the smallest size that preserves the detail your task needs, crop to the region of interest rather than sending a full screenshot, and send one image per call when asking for a focused extraction. For multi-page documents, process pages in parallel with a bounded worker pool and merge results with the page number in each record. Always keep the original file in your own storage and treat model output as a draft: extraction errors are rare but consequential, so define confidence thresholds and human review for anything that touches money or identity.

What is not available yet

Be explicit with stakeholders about the roadmap. Image generation through /v1/images/generations is coming soon and returns 501 today. Audio transcription and speech synthesis are also coming soon, as is moderation. Vision-capable chat models are the live capability, so design workflows that only need understanding now, and keep generation features behind a flag for when they ship. Check /models for vision model cards and /docs for endpoint status before committing to a delivery date.

Honest comparison

Capability	Plugsky vision chat	Separate vision API	Self-hosted VLM
Interface	Image content on chat completions	Dedicated endpoint and schema	You run the serving stack
Model choice	Multiple vision models in the catalogue	Vendor's vision model	Whatever you deploy
Structured output	JSON mode composes with vision	Varies	You build it
Generation	Coming soon	Often available	Depends on weights
Residency	Region pinning plus VPC/on-prem	Vendor regions	Fully in your control
Ops burden	Managed	Managed	High

Frequently asked questions

Can Plugsky models read images?

Yes. Vision-capable models such as plugsky-qwen-vl and plugsky-vision-fast accept image content on /v1/chat/completions alongside text, and you can combine that with JSON mode for structured extraction.

Can Plugsky generate images?

Not yet. POST /v1/images/generations is coming soon and currently returns 501 until an image model is enabled on the catalogue.

Which vision model should I choose?

Check the model cards at /models for context limits, image support and status. plugsky-qwen-vl is the general multimodal option; plugsky-vision-fast targets lower-latency visual tasks.

How do I keep image costs predictable?

Downscale and crop before upload, send one focused image per call, and process batches with bounded concurrency. Self-serve plans are flat monthly, and token counts are returned for observability.

Is OCR still needed?

For dense text with strict character accuracy, dedicated OCR can still beat a general vision model. Many teams run OCR for text and vision for layout and context, then merge both.

Can I process documents in one region?

Yes. Pin the workspace region so images and outputs stay put, and use VPC, on-prem or air-gapped deployment where the data path must remain inside your perimeter.

How accurate is extraction?

Accuracy depends on image quality, model choice and schema design. Measure on a labelled sample, define confidence thresholds, and route uncertain fields to human review.

Are audio and moderation available?

Both are coming soon according to the docs. Plan workflows that do not depend on them yet and track status before promising timelines.