How do you build computer-use agents with an OpenAI-compatible API?

Key facts

Endpoint	POST https://api.plugsky.com/v1/chat/completions with JSON mode or a fixed tool schema for actions
Compatibility	Same shape as OpenAI; change base_url and model to adopt it
Observation	Screenshots or accessibility trees you generate; verify multimodal support per model
Models	30+ models; choose a vision-capable alias if screen understanding is required
Pricing model	Flat monthly self-serve plans with unlimited fair-use usage
Free tier	Free plan with 2 free AI models, no card required
Isolation	Disposable VM or browser profile; credentials scoped per task
Roadmap	Files, images and responses endpoints are coming soon

TL;DR

• One action per API call keeps the loop verifiable and debuggable.
• Use JSON mode so every action parses into your executor reliably.
• Serialize state explicitly — the model has no memory between calls unless you send it.
• Run in a disposable VM with task-scoped credentials and full session recording.
• Confirm vision support for your chosen model before relying on screenshots.

How it works, step by step

1. Change the base URL to Plugsky and confirm a structured-output request parses as expected.
2. Define the action schema: action type, target, value, a short reason and a done flag.
3. Build the executor that captures observations and applies actions inside a disposable VM.
4. Send one observation per call and request exactly one action in JSON mode.
5. Verify the expected state change after each action; abort on repeated no-progress states.
6. Log observation, action, result and evidence for every step; gate irreversible actions behind approval.

Original data

Try it yourself

Open the API latency tester →

Architecture: API as policy, executor as hands

The trust boundary is explicit:

• API: receives the observation and goal, returns one action in a fixed schema.
• Executor: owns the machine — captures observations, applies actions, enforces allowlists and approvals.
• State serializer: sends the model a compact view: current screen, goal, recent actions and their results.
• Recorder: stores screenshots, actions and reasons for audit and replay.

Because there is no server-side session, you control exactly what the model remembers. That is more work than a managed agent runtime and considerably easier to audit.

Implementation details

Keep each call small and deterministic:

• Use JSON mode with an action enum, and reject any response that does not parse or match the schema.
• Send one observation per call, not a growing history of screenshots; include only the last action and result.
• Normalize coordinates to relative units if you must click by position, so resolution changes do not break the agent.
• Add a done action plus a completion check so the agent cannot loop forever.
• Cap steps and elapsed time per task, and abort on repeated observations of the same state.
• Keep the loop synchronous: apply, verify, then observe again. Parallel actions on one UI cause races.

Evaluation and limitations

Score the loop honestly:

• Task success: completed without human intervention, verified by an end-state check.
• Steps per task: lower is better and indicates less flailing.
• Recovery rate: failed actions followed by a successful alternative.
• Destructive incidents: unapproved or irreversible actions taken — target zero.
• Latency per action: observation, model call and settle time, which compounds over a session.

Limitations to plan around: screen understanding requires a model that accepts visual input, and that support varies by model and must be verified in the docs. There is no built-in browser or desktop tool, so sandboxing, recording and approval logic are your responsibility. Anti-bot measures and site terms of service constrain what automation is acceptable, and UI changes break position-based actions faster than semantic ones.

Honest comparison

Capability	Computer-use loop on Plugsky API	Managed GUI agent service	RPA scripts
Model choice	30+ aliases, one OpenAI-compatible call	Service-selected models	None
Control	You own executor, sandbox and approvals	Vendor-managed runtime	Fully scripted
Observation	Your screenshots or accessibility trees	Vendor capture tools	Structured app APIs
Audit	Your recordings plus API logs	Vendor dashboards	Deterministic logs
Fit	Custom, security-sensitive workflows	Fast prototypes	Stable repetitive flows

Frequently asked questions

Does the API support image input?

Support depends on the model. Some catalogue entries are vision-capable, but you should verify the current multimodal capability in the docs and test with your own screenshots before building around it.

How do I keep the agent from looping?

Send one action per call, compare consecutive observations, and abort when the same state repeats. Cap steps and elapsed time per task.

Should I send full history each turn?

No. Send a compact serializer state: current observation, goal, the last action and its result. Growing screenshot history wastes context and slows the loop.

How do I make actions reliable?

Use JSON mode with a strict schema, reject invalid responses, and verify the expected state change after each action before continuing.

How do I protect the host machine?

Run sessions in a disposable VM or browser profile, inject credentials at the executor layer, allowlist domains and actions, and record everything.

Can I use this for web scraping?

Automation is subject to each site's terms of service and anti-bot measures. Where an official API exists, prefer it over driving the UI.

Can I prototype on the free plan?

The loop and schema work with free models, but check that a vision-capable alias is available for the observation step before committing to the design.