How do you build classification with AI agents?

Key facts

Endpoint	POST /v1/chat/completions with JSON mode for the decision and tools for lookups
Deterministic output	Fixed label enums; reject any response outside the label set
Models	30+ models; plugsky-micro for single-pass labels, plugsky-pro for ambiguous multi-signal cases
Escalation	Confidence thresholds and a review queue instead of repeated model retries
Pricing model	Flat monthly self-serve plans with unlimited fair-use usage
Free tier	Free plan with 2 free AI models, no card required
Governance	Scoped keys, audit logs and per-label telemetry
Roadmap	The batch endpoint is coming soon; run large label jobs on chat completions with concurrency limits

TL;DR

• Plain classification is one JSON-mode call; add an agent only when context is missing.
• Let tools fetch account or document signals, then force the decision into a fixed enum.
• Escalate once on low confidence; repeated retries hide ambiguity instead of resolving it.
• Measure per-class precision and recall, not aggregate accuracy.
• Sample reviewed items into your evaluation set to keep it honest.

How it works, step by step

1. Write the taxonomy down with one-line definitions and unambiguous boundary rules.
2. Decide which labels need external context and expose only those signals as tools.
3. Run the agent to gather signals, then make the final decision with JSON mode constrained to the label enum.
4. Attach a confidence score and route low-confidence items to a human review queue.
5. Validate outputs against the schema and reject anything outside the taxonomy.
6. Track per-class precision, recall and review-queue precision, and refresh the evaluation set monthly.

Original data

Try it yourself

Open the prompt optimizer →

When classification needs an agent

Most classification is a single call: text in, label out. An agent earns its cost when the decision depends on information that is not in the input.

• Missing context: a support ticket that cannot be routed without the customer's plan or recent order history.
• Multiple signals: fraud or risk labels that combine the message with account activity and prior cases.
• Recovery: inputs that fail validation or split across two labels and need one clarifying lookup before a final decision.

If none of those apply, skip the agent. Two model calls produce more variance and latency than one, and labels rarely improve enough to pay for it.

Implementation pattern

Separate gathering from deciding:

1. Signal step: the agent calls read-only tools such as get_account_flags to fill context gaps.
2. Decision step: a JSON-mode call receives the input plus gathered signals and returns one label from the enum, a confidence value and a one-line reason.
3. Validation: reject outputs outside the enum or with confidence below the floor.
4. Escalation: send borderline cases to a second pass with a stronger alias or to a human queue — never loop the same prompt.
5. Logging: store input, signals, label, confidence, evidence and reviewer outcome for every item.

The decision step must stay deterministic and cheap; the agent adds context, not authority.

Evaluating classification with review in the loop

Per-class metrics are mandatory because aggregate accuracy hides the labels you care about:

• Precision and recall per label: track the rare, high-cost labels separately.
• Confusion pairs: identify label boundaries the model blurs, and either sharpen definitions or merge labels.
• Confidence calibration: reviewed items above the threshold should be more accurate than those below it.
• Review queue precision: if reviewers agree with almost everything queued, raise the threshold; if they disagree, lower it.
• Coverage: how often the agent could not gather the signals needed for a decision.

Feed reviewed outcomes back into a frozen evaluation set so improvements are measurable over time.

Limitations

Agentic classification adds failure modes that plain calls do not have:

• Tool errors and timeouts become classification errors unless you handle missing signals explicitly.
• Confidence values are model-generated and uncalibrated until you measure them against reviews.
• Taxonomy drift is the most common long-term issue; review label usage quarterly and retire labels nobody uses.
• The batch endpoint is coming soon, so large backfills need concurrency limits, retries and idempotent writes today.
• Latency and cost per item increase with each agent step; keep the gather phase narrow.

Honest comparison

Capability	Agentic classification	Single-call classification	Human-only review
Context gathering	Read-only tools fetch missing signals	Input text only	Manual lookup
Decision output	Fixed enum in JSON mode	Fixed enum in JSON mode	Human judgment
Escalation	Confidence threshold with review queue	Usually none	All items reviewed
Evaluation	Per-class metrics plus queue precision	Per-class metrics	Sampled audits
Throughput	Lower, due to extra steps	High	Lowest

Frequently asked questions

Is an agent always better for classification?

No. Use an agent only when the label depends on context that is not in the input. For straightforward text classification, a single JSON-mode call is faster, cheaper and more consistent.

How do I keep labels consistent?

Constrain output with a strict enum in JSON mode, validate every response, and reject anything outside the taxonomy rather than mapping it to the nearest label.

What confidence threshold should I use?

Start at 0.8 and calibrate against human review. The right threshold depends on label cost: a wrong fraud label and a wrong topic label have very different consequences.

Should the agent retry on low confidence?

No. Escalate once to a stronger alias or a human queue. Repeated retries on the same evidence add variance without adding information.

How do I handle a growing taxonomy?

Version the taxonomy, keep per-label definitions in the prompt, and review label usage quarterly. Large taxonomies often benefit from a retrieval step instead of a longer prompt.

Can I use cheap models for labeling?

Yes. plugsky-micro covers most single-pass labeling, and plugsky-lite handles routine multi-signal cases. Reserve plugsky-pro for genuinely ambiguous items.

Is there a free way to start?

The free plan includes plugsky-micro and plugsky-lite with no card, and a 14-day full-access trial lets you test stronger models on ambiguous items.