How do you build code generation with AI agents?

Key facts

Endpoint	POST /v1/chat/completions with tools for test runs, linting and file reads
Coding models	plugsky-coder and plugsky-coder-fast are in the catalogue alongside 30+ other models
Verification	Execute tests and linters as tools and feed failures back into the loop
Loop limits	Cap iterations, wall-clock time and diff size per task
Pricing model	Flat monthly self-serve plans with unlimited fair-use usage
Free tier	Free plan with 2 free AI models, no card required
Governance	Scoped keys, audit logs and sandboxed execution
Roadmap	Fine-tuning and batch endpoints are coming soon

TL;DR

• Generation is the easy half; the loop that runs tests is what makes the output usable.
• Send interfaces, conventions and the failing test, not the whole repository.
• Cap iterations and diff size so a bad task cannot spiral.
• Sandbox all execution and treat generated code as untrusted input.
• Measure compile rate, test pass rate and review acceptance — not lines generated.

How it works, step by step

1. Define the task contract: target files, expected behavior and the tests that must pass.
2. Assemble context from the repository: interfaces, call sites, conventions and the failing test.
3. Generate a candidate patch with a coding-focused model and apply it in a sandbox.
4. Run the compiler, linter and tests through tools; return failures to the model for one correction pass.
5. Stop on green checks or budget exhaustion, and surface the diff with a summary of what changed.
6. Require human review before merge, and record whether the diff was accepted or reverted.

Original data

Try it yourself

Open the best model for coding selector →

Architecture of a code-generation agent

Four stages, each with a clear owner:

• Task intake: normalize the request, collect target files and locate relevant tests.
• Context builder: retrieve interfaces, existing patterns and the failing test — never the whole repository.
• Generate and verify: the model proposes a patch; your runner applies it in a sandbox and executes checks.
• Review gate: a human reviews the diff; acceptance and revert data feeds back into evaluation.

The model never executes code directly. Tools expose read-only inspection and check execution, with the sandbox enforcing filesystem and network limits.

Implementation choices

Decisions that separate a useful agent from a demo:

• Use plugsky-coder for complex changes and plugsky-coder-fast for routine edits; keep the rest of the 30+ catalogue available for explanation and review tasks.
• Ask for a unified diff or a patch object rather than full file rewrites — smaller outputs are easier to review and apply.
• Feed back the exact compiler or test error, not a summary; precise failure text is the highest-value context.
• Allow one correction pass on failure, then stop. Repeated attempts on the same error rarely converge.
• Freeze dependencies and run tests in a hermetic sandbox so results are reproducible.
• Log the diff, checks, iterations and outcome for every task.

Evaluating code generation

Score the pipeline, not the prose:

• Compile rate: share of generated patches that build without manual fixes.
• Test pass rate: on held-out tests, including tests not shown to the model.
• Review acceptance: share of diffs merged with only minor edits.
• Revert rate: patches later reverted for defects — the metric that matters most for trust.
• Cost per accepted change: iterations and tokens spent divided by accepted diffs.

Held-out tests are essential. If the model can see the test, passing it proves less than you think.

Limitations and risks

Generated code is untrusted code:

• Verify everything in a sandbox with no network and no credentials, and never run generated scripts on developer machines.
• Tests are a weak oracle: code can pass them and still be wrong, insecure or unmaintainable.
• Models hallucinate APIs and configuration keys; retrieval of real interfaces reduces but does not eliminate this.
• Large monorepos exceed context windows, so task scoping matters more than model choice.
• Fine-tuning and batch endpoints are coming soon; current pipelines should rely on prompt engineering and retrieval.

Honest comparison

Capability	Code agent on Plugsky	Autocomplete-only assistant	Manual development
Verification	Tests and linters run in the loop	None	Developer runs them
Context	Retrieved interfaces and failing tests	Open file window	Full human knowledge
Model choice	Coding models among 30+ aliases	Vendor model	Not applicable
Review	Diff plus check summary	Line suggestions	Full review process
Cost shape	Flat monthly self-serve, unlimited fair use	Per-seat subscription	Engineering hours

Frequently asked questions

Do I need an agent to generate code?

The agent is valuable for the verification loop, not the generation call itself. If you cannot run tests or checks automatically, a single generation call with human review is a better starting point.

Which Plugsky model is best for code?

plugsky-coder for complex changes and plugsky-coder-fast for routine edits are the obvious starting points. Benchmark both on your own repository tasks before committing.

How do I stop the agent from rewriting too much?

Request diffs instead of full files, cap diff size and iterations, and scope the task to specific files and tests.

Is it safe to let the agent run tests?

Yes, inside an isolated sandbox with no network and no credentials. Treat every generated artifact as untrusted input.

What is the most reliable evaluation metric?

Revert rate: patches later undone for defects. It reflects real maintenance cost better than pass rates measured on tests the model could see.

Can it work on a large monorepo?

Partially. Retrieval and clear task scoping help, but context limits still apply. Start with well-bounded packages rather than repository-wide requests.

Can I evaluate coding models on the free plan?

Yes. The free plan includes two free models with no card, and a 14-day full-access trial lets you compare coding aliases on your own tasks.