How do you build computer-use agents with AI agents?

Key facts

Loop endpoint	POST /v1/chat/completions with a fixed action schema; function calling is live
Perception	Screenshots or accessibility trees you capture; verify multimodal model support in the docs
Action space	Constrain to click, type, scroll, key, wait and done — nothing broader
Isolation	Run in a disposable browser or VM with credentials scoped to the task
Models	30+ models; vision-capable entries in the catalogue determine what the agent can observe
Pricing model	Flat monthly self-serve plans with unlimited fair-use usage
Approvals	Human confirmation for payments, submissions and irreversible actions
Roadmap	Files, images and responses endpoints are coming soon

TL;DR

• Keep the action space tiny; every extra action type is a new failure mode.
• Prefer structured observations (accessibility tree) over pixels when available.
• Run sessions in disposable VMs with task-scoped credentials only.
• Require confirmation for irreversible steps and log every action.
• Verify current multimodal model support before designing around screenshots.

How it works, step by step

1. Choose one workflow with a measurable outcome and a low cost of failure.
2. Define the action schema and the observation format: screenshot, accessibility tree or both.
3. Build the executor that applies actions in a disposable VM or browser profile.
4. Run the loop: observe, decide one action, execute, verify state, repeat under a step and time cap.
5. Add approval gates for purchases, submissions, deletions and anything hard to undo.
6. Record sessions and evaluate task success, steps taken, recovery rate and destructive-action incidents.

Try it yourself

Open the browser automation generator →

The perception-action loop

Each turn is deliberately small:

1. Observe: capture the current screen or accessibility tree, plus the goal and recent actions.
2. Decide: ask the model for exactly one action in a fixed schema — click, type, scroll, key, wait, done — with a short reason.
3. Execute: apply the action in the sandbox and wait for the UI to settle.
4. Verify: check that the expected state change occurred before the next turn.

Structured observations are cheaper and more reliable than screenshots when the application exposes an accessibility tree or DOM. Use pixels only where structure is unavailable.

Safety architecture

Computer-use agents act with a human's privileges, so the architecture is the safety mechanism:

• Disposable VM or browser profile per session, destroyed afterward.
• Credentials scoped to the task, injected at the executor layer, never visible to the model.
• Domain and action allowlists; anything off-list requires approval.
• Confirmation gates for payments, submissions, deletions and permission changes.
• Full session recording — screenshots, actions and model reasons — retained for audit.
• Hard caps on steps, wall-clock time and total actions per task.

Implementation and evaluation

Implementation notes that reduce flailing:

• Reset to a known state at the start of every task rather than continuing from the previous screen.
• Keep a step budget; agents that exceed ten steps rarely recover without human input.
• Detect no-progress loops by comparing consecutive observations and aborting on repeats.
• Use a cheap model for routine navigation and a stronger one only for ambiguous screens.
• Log every decision so failures can be replayed deterministically.

Evaluate task success, steps per task, recovery after a failed action, and destructive incidents. The last metric must be zero; if it is not, tighten approvals before scaling up.

Limitations

Computer use is the least mature agent pattern, and honesty matters here:

• Screen understanding depends on multimodal model support; verify the current capability in the docs and test on your own interfaces before committing.
• UI drift breaks prompts that encode pixel positions; prefer semantic targets such as element labels.
• Latency compounds: each step includes a capture, a model call and a wait, so plan for slow sessions.
• Captchas, anti-bot measures and site terms of service restrict what automation is acceptable; respect them.
• Plugsky does not ship a browser tool, so the executor, sandbox and recording layer are yours to build.

Honest comparison

Capability	Computer-use agent on Plugsky	Recorded script automation	Human operator
Adaptivity	Re-decides every step from observations	Breaks on UI change	Full judgment
Observability	Per-step reasons and recordings	Deterministic logs	Notes and tickets
Safety	Sandbox, allowlists and approvals	Whatever the script allows	Human restraint
Setup cost	High: executor, sandbox, evaluation	Medium	Training only
Best for	Varying screens and multi-step decisions	Stable repetitive flows	Exceptional cases

Frequently asked questions

What is a computer-use agent?

An agent that observes a graphical interface, decides one action at a time — click, type, scroll, key or wait — and executes it in a controlled environment until the task completes.

Does Plugsky provide a browser tool?

No. Plugsky provides the model API; you build the executor, sandbox and recording layer. Function calling is live for the action loop.

How does the agent see the screen?

You capture a screenshot or accessibility tree and include it in the request. Whether a given model accepts image input depends on that model, so check the docs and test before designing around vision.

How do I stop the agent from doing damage?

Keep the action space minimal, run in a disposable VM, scope credentials to the task, require approval for irreversible actions, and cap steps and time.

Which model should drive the loop?

A vision-capable model for screen understanding, with a cheap alias for routine navigation steps. Verify multimodal support for the specific alias you choose.

How do I evaluate a computer-use agent?

Task success rate, steps per completed task, recovery rate after failed actions and destructive-action incidents. The last one must be zero.

Can I prototype this on the free plan?

You can prototype the loop with free models, but verify that the model you need for screen understanding is available before relying on it.