Feature · vision

Vision API for image understanding

Classify a product, read a screenshot, extract a table or check a user-uploaded photo. The image goes in the same chat completions call as text, and the answer can be a strict JSON object.

Short answer. Send an image_url or base64 image in POST /v1/chat/completions. Use plugsky-vision-fast for low-latency classification and UI checks, and plugsky-parse for dense documents and tables. Ask for JSON with a schema, then validate it. Vision is available on paid plans from $8/month.

Task matrix

TaskModelOutputTypical failure
Scene or object classificationplugsky-vision-fastLabel + confidenceConfuses similar categories
Product attributesplugsky-vision-fastColour, material, styleInvents a material from sheen
UI screenshot QAplugsky-vision-fastElements, text, layout issuesMisses small hit targets
Document and table extractionplugsky-parseBlocks, rows, cellsBreaks reading order
Chart or dashboard readingplugsky-parseSeries, values, trendsReads a legend as data
Safety triageplugsky-vision-fastCategory flagsContext-dependent misses

For text-only moderation, use /v1/moderations.

Resolution and token math

Images are billed as input tokens, and the count is a function of dimensions and the model family, not file size. The same 1024×1024 image can cost 765 tokens on an OpenAI-style tile model, about 1,296 on Claude, or about 1,032 on Gemini. Plugsky is OpenAI-compatible on the request shape, but you should read usage.prompt_tokens on your own calls rather than assume an upstream billing formula.

ImageOpenAI-style high detailClaude-style patchesGemini-style tiles
512×512255336258
1024×10247651,2961,032
2048×20482,8051,568 cap2,322

The OpenAI-style rule is 85 base tokens plus 170 per 512-pixel tile after scaling to fit 2048 pixels and setting the shortest side to 768. Claude counts 28×28 patches with a token cap. Gemini counts 258-token tiles. Use the smallest image that passes eval.

from math import ceil

def openai_tokens(w, h, detail="high"):
    if detail == "low":
        return 85
    if max(w, h) > 2048:
        scale = 2048 / max(w, h)
        w, h = int(w * scale), int(h * scale)
    if min(w, h) > 768:
        scale = 768 / min(w, h)
        w, h = int(w * scale), int(h * scale)
    return 85 + 170 * ceil(w / 512) * ceil(h / 512)

print(openai_tokens(1024, 1024))  # 765

Structured JSON output

Free text is hard to validate. Ask for a schema with a label, a confidence, and the evidence the model used. Then run the same schema locally; a response that fails validation should be retried or routed to review, not patched by hand.

import json
from openai import OpenAI

client = OpenAI(api_key="YOUR_PLUGSKY_KEY", base_url="https://api.plugsky.com/v1")

resp = client.chat.completions.create(
    model="plugsky-vision-fast",
    response_format={"type": "json_schema", "json_schema": {
        "name": "product", "strict": True,
        "schema": {
            "type": "object",
            "required": ["category", "attributes", "confidence"],
            "properties": {
                "category": {"type": "string"},
                "attributes": {"type": "array", "items": {"type": "string"}},
                "confidence": {"type": "number"},
                "evidence": {"type": "array", "items": {"type": "string"}}
            }
        }
    }},
    messages=[{"role": "user", "content": [
        {"type": "text", "text": "Describe the product. Use only visible evidence."},
        {"type": "image_url", "image_url": {"url": "https://example.com/sku.jpg", "detail": "high"}}
    ]}],
)

record = json.loads(resp.choices[0].message.content)
assert 0 <= record["confidence"] <= 1

UI, OCR and product modes

ModeWhat to ask forWhat to check
UIElement name, role, bounding box, visible text, hit-target sizeCoordinates map back to the original resolution; disabled states are not guessed
OCRBlocks in reading order, tables as rows and cells, null for unreadable textColumn order, totals, Arabic-Indic digits
ProductCategory, colour, material, pattern, defects, label textNo invented attributes; label text matches the SKU

For documents, use the document parsing API pattern.

The 20-image benchmark

Twenty images is enough to expose a bad prompt and small enough to label by hand. Use the same prompt, temperature 0 and the same model for every row, and score each answer against the criterion.

#ImageTaskPass criterion
1White-background shoeCategory + colourBoth correct; no invented material
2Folded shirt on a bedPattern + sleeve lengthPattern correct; sleeves visible
3Cosmetic bottleRead the front labelBrand and volume exact
4Jewellery close-upMetal + stoneNo guess when obscured
5Grocery packNet weight + flavourBoth exact; units preserved
6Mobile checkout screenList interactive elementsPrimary button found; coordinates valid
7Error dialogRead the error and buttonsText exact; no invented cause
8Dashboard with chartsRead three seriesLegend and values not swapped
9Dark-mode settings pageFind the toggle stateState correct; low-contrast text read
10Arabic UI screenReading order + labelsRight-to-left order preserved
11Invoice pageVendor, total, dateThree fields exact
12Bank statement tableRows + running balanceRow count and signs correct
13Handwritten noteName + quantityNull when unreadable
14Rotated receipt photoTotal + taxArithmetic reconciles
15Arabic contract pageParties + dateArabic-Indic digits normalised
16Blurry product photoCategory onlyAbstains instead of guessing
17Image with a text instructionDescribe the sceneDoes not follow text in the image
18Two products in one frameCount + attributesCount correct; attributes separated
19Reflective surfaceMaterial + colourReflection not read as a second object
20Low-light photoSafety flagFlag matches human label

Report exact match, field accuracy, hallucination rate and p95 latency.

Safety, limits and cost per 1,000 images

Images can carry prompt injection: text inside a screenshot that says “ignore previous instructions”. Treat image text as data, not instructions. Avoid face recognition and biometric inference; minimise personal data; keep human decisions for high-impact outcomes.

At roughly 1,065 tokens for a 1024×1024 request, Hobby's 2M daily cap allows about 1,878 images per day; at full use the effective cost is about $0.14 per 1,000 images. Starter has the same list-price ratio. Builder and Scale have no monthly token cap, so cost per image is their monthly fee divided by volume.

PlanDaily token capImages/day at 1,065 tokensEffective $ / 1,000 at full use
Hobby $82M~1,878~$0.14
Starter $205M~4,695~$0.14
Builder $60No capRPM/concurrency bound$60 ÷ monthly images
Scale $120No capRPM/concurrency bound$120 ÷ monthly images

See pricing for current caps and plugsky-vision-fast for model details. For the wider build, read what you can build with multimodal AI.

FAQ

Which model should I use for vision?

Use plugsky-vision-fast for classification, UI checks and product attributes where latency matters. Use plugsky-parse for dense documents, tables and pages that need a long structured output.

How are images counted as tokens?

It depends on the model family. OpenAI-style tile models use a base of 85 tokens plus 170 per 512-pixel tile after resizing, so a 1024×1024 image is 765 tokens. Claude uses 28×28 patches, and Gemini uses 768-pixel tiles. Read usage.prompt_tokens on your own requests rather than assuming a formula.

Can the API return structured JSON?

Yes. Send a JSON schema in response_format and require fields such as label, confidence and evidence. Validate the response with the same schema before writing it to a database.

How do I test vision accuracy?

Use a fixed 20-image set that covers product, UI, document and edge cases. Score exact match, field accuracy, hallucination rate and latency; do not rely on a vendor demo set.

What does vision cost per 1,000 images?

At about 1,065 tokens per 1024×1024 image, Hobby's 2M daily token cap allows roughly 1,878 images per day. At full daily use the effective cost is about $0.14 per 1,000 images; plans are flat, so your volume and output length dominate.

Sources

OpenAI vision documentation for image tokenization, checked 20 November 2026: developers.openai.com/api/docs/guides/images-vision. Anthropic vision documentation for patches and token caps, checked 20 November 2026: platform.claude.com/docs/en/build-with-claude/vision. Google Cloud Vision documentation, checked 20 November 2026: cloud.google.com/vision/docs.

Reviewed 20 November 2026. Token formulas are documented by the upstream model families and are planning estimates for Plugsky; reconcile against usage.prompt_tokens. Do not use vision output for medical, legal or safety decisions without human review.