How do you build computer-use agents with RAG?

Key facts

Retrieval	POST /v1/embeddings with plugsky-embed over runbooks, UI maps and past trajectories
Loop endpoint	POST /v1/chat/completions with a constrained action schema
Index content	Step-by-step procedures, element descriptions, error signatures and recovery actions
Models	30+ models; observation capability depends on the model you choose
Pricing model	Flat monthly self-serve plans with unlimited fair-use usage
RAG status	RAG is live; embeddings plus chat completions are the building blocks
Governance	Audit logs plus recorded trajectories for review
Roadmap	Files, images and batch endpoints are coming soon

TL;DR

• Retrieve the procedure before acting; exploration is the expensive failure mode.
• Index UI maps and element labels, not pixel coordinates.
• Store error signatures with known recoveries so the agent stops repeating mistakes.
• Keep actions small; a retrieval miss should cost one step, not the task.
• Refresh the index whenever the target application changes.

How it works, step by step

1. Document one workflow as numbered steps with the screens, elements and expected results involved.
2. Convert each step into an index entry: task, screen, element labels, action and success signal.
3. Record known error signatures with their recovery actions as separate entries.
4. Embed everything with plugsky-embed and store application version metadata with each entry.
5. At run time, retrieve the current step and its recovery options before deciding the next action.
6. Log the retrieved evidence with each executed action, and re-index when the UI changes.

Original data

Try it yourself

Open the multi-agent workflow generator →

Why retrieval helps where pixels are ambiguous

Computer-use agents fail in predictable places: unfamiliar screens, multi-step procedures with branching rules, and error states. Retrieval addresses each:

• Procedures: the agent follows your documented order instead of inventing one.
• UI maps: element labels and expected outcomes make screens identifiable without memorizing positions.
• Error recovery: known failure signatures map to tested recovery actions rather than improvised retries.
• Trajectory memory: previously successful sessions become examples the current run can imitate.

What to index

Useful entries for a computer-use agent:

• Procedure steps: one chunk per step with the screen name, element labels, action and success signal.
• UI map entries: screen identifiers, navigational landmarks and stable element names.
• Error signatures: the visible message or state, its meaning and the recovery action that resolved it.
• Successful trajectories: condensed action sequences with their outcomes, deduplicated.
• Policy notes: steps that require approval or are prohibited entirely.

Tag every entry with application name and version, and never index credentials or personal data captured during recordings.

Implementation and evaluation

The loop with retrieval is observe, retrieve, act, verify:

1. Identify the current screen from structured elements or a screenshot.
2. Retrieve the matching procedure step and any relevant recovery entries.
3. Ask the model for one action, with retrieved evidence available in the prompt.
4. Execute, verify the expected state change, and log the evidence ids used.

Evaluate task success, steps per task, retrieval hit rate and recovery rate. When a task fails, check whether the right entry existed and was retrieved — those are two different problems with different fixes.

Limitations

Retrieval cannot see the current screen, and it can go stale:

• UI changes invalidate indexed element labels; treat index refresh as part of every application release.
• Recorded trajectories contain sensitive data; scrub and store only what policy allows.
• Retrieval adds latency to every step, which compounds in long sessions.
• Observation still depends on model support for screenshots if structured access is unavailable — verify before designing around it.
• Files, images and batch endpoints are coming soon, so screenshot storage and bulk processing remain your responsibility today.

Honest comparison

Capability	Computer-use agent with RAG	Agent without retrieval	Scripted automation
Procedure adherence	Follows indexed steps	Improvises from the screen	Fixed sequence
Error recovery	Known signatures map to tested fixes	Retries blindly	No recovery
Adaptability	Handles variation within documented flows	Explores and wastes steps	None
Maintenance	Re-index on UI change	Prompt tuning	Script rewrite
Audit	Evidence ids per action	Screen recordings only	Deterministic logs

Frequently asked questions

What should I index for computer-use agents?

Procedure steps, UI map entries, error signatures with recoveries, condensed successful trajectories and policy notes about approval-required steps.

Should I index screenshots?

Prefer structured element labels. If you store screenshots, keep them short-lived, scrubbed of personal data, and separate from the retrieval index.

How do I keep runbooks in sync with the UI?

Treat index refresh as part of the application release process, and store version metadata so stale entries can be detected and purged.

Does retrieval replace vision?

No. Retrieval supplies procedures and recovery knowledge; observing the current screen still requires the model to process visual or structured input.

How do I handle steps that need approval?

Index them explicitly as approval-required and make the executor enforce the gate regardless of what the model proposes.

How do I evaluate retrieval here?

Track retrieval hit rate, steps per completed task and recovery rate separately, so you can tell whether failures came from missing knowledge or bad execution.

Can I prototype on the free plan?

Yes for the retrieval and loop logic. Verify that the model you need for screen observation is available before depending on it.