2027 planning guide · Agents and RAG

AI agents and RAG in 2027: architecture, retrieval and evaluation

Agents and retrieval solve different problems. Choose the simplest structure that works, then measure the parts separately.

Short answer. An AI agent chooses its own steps and tools; a workflow follows a path you defined. RAG retrieves external evidence before generation. In practice, many production systems use workflows with bounded agent steps and custom retrieval, not autonomous agents. Choose the simplest structure that meets the task, then evaluate retrieval and tool use separately.

Quick answers to the questions buyers ask first.
QuestionOne-sentence answer
What is the difference between a workflow and an agent?A workflow has predefined steps; an agent chooses steps and tools at runtime.
When should I use an agent?When the path cannot be enumerated in advance and errors are recoverable or reviewable.
What does an AI gateway do?It centralises routing, credentials, retries, rate limits and observability across models.
Is a gateway an agent platform?No; it does not own durable state, tool execution or evaluation.
What is managed RAG?A vendor-hosted retrieval pipeline; faster to start, with less control over ranking.
What is custom RAG?It is your own ingestion, chunking, index, retrieval and reranking.
Which metric matters most?Task success with evidence matters most; retrieval recall and groundedness explain why it fails.
Do I need MCP?Use it when you want a standard tool interface; it is not a requirement for agents and not a security boundary.

Agents and retrieval are often sold together, but they solve different problems. RAG improves what the model knows at the moment of generation. An agent changes what the system does next. A workflow can use RAG without being an agent, and an agent can run without retrieval at all.

The architecture question comes first. Get it wrong and the evaluation plan will be wrong too, because you will be measuring the wrong thing.

Workflow or agent?

The distinction is control flow, not the model, the framework or the demo. A workflow is a sequence you can draw. An agent chooses the sequence at runtime.

What is the difference between an AI workflow and an agent?

A workflow is a sequence of steps you define in code: retrieve, classify, generate, validate, respond. An agent uses a model to choose actions, tools and stopping conditions at runtime. A workflow can contain an LLM call at every step and still be a workflow; an agent can use one tool and still be an agent. Choose a workflow when the process is known and auditable. Choose an agent only when the path cannot be enumerated in advance. The label matters less than the operating model: can you replay a failed task, bound the cost, and explain why a tool was called?

When is an agent the wrong choice?

When the steps are known, the cost of a wrong action is high, or you cannot evaluate the trajectory. Agents add non-determinism, latency and token cost, and they fail in ways that are harder to reproduce. A support triage process with fixed categories is a workflow. A refund agent that can move money needs human approval and a hard cap, even if it is technically an agent. The right question is not whether the demo looks autonomous, but whether the organisation can operate and audit the loop.

A control-flow decision, not a maturity ladder.
QuestionWorkflowAgent
Who chooses the next step?CodeThe model
PredictabilityHighVariable
Cost per taskBoundedVariable, depends on steps
Typical failureA step failsLoop, wrong tool or unsafe action
EvaluationPer stepOutcome and trajectory
Best fitKnown processOpen-ended task with recoverable errors

Gateways and agent platforms

Both sit around the model, but they govern different things. A gateway governs model traffic. An agent platform governs task state.

What does an AI gateway actually solve?

An AI gateway centralises model access: credentials, routing, retries, rate limits, logging and cost attribution. It gives platform teams one place to enforce policy across providers and one place to see which model served a request. It does not own task state, tool execution or evaluation. A gateway can make a multi-provider architecture manageable, but it will not make an unreliable agent reliable. Buy or build one when you have more than one model, team or provider to govern. The provider comparison is a useful input to the routing policy.

How is an agent platform different from a gateway?

An agent platform adds durable execution, tool registration, memory, tracing and evaluation. It owns the lifecycle of a task: start, pause, resume, retry and stop. A gateway sits in front of model calls; a platform sits around the work. Some platforms include a gateway, and some gateways add agent features, which is why the categories blur. Ask what happens when a step fails halfway through, where state is stored, and whether you can export traces and run the same task outside the platform. The exit path matters as much as the feature list.

Gateway and agent platform are different control planes.
ConcernAI gatewayAgent platform
Model credentials and routingYesUsually via a gateway
Retries and rate limitsYesApplication-level
Durable task stateNoYes
Tool registry and executionLimitedYes
Tracing and evaluationPartialYes
Lock-in riskLowerHigher

Managed or custom RAG

RAG is a pipeline, not a single product. The build-versus-buy question applies to each stage, not to the whole stack.

What does a RAG pipeline need?

Ingestion, parsing, chunking, embedding, indexing, retrieval, reranking, prompt assembly and citation. Each stage has its own failure mode: a bad parse loses a table, a bad chunk splits an answer, a weak retriever returns plausible but irrelevant text, and a missing citation makes the answer unverifiable. Instrument each stage rather than only the final answer. The embedding model and its dimensions are part of the schema: changing them means re-embedding the corpus and rebuilding the index. Start with the retrieval benchmark before tuning prompts.

When should you use managed RAG versus custom RAG?

Managed RAG is enough when the corpus is small, the document types are standard and ranking control is not a differentiator. It gets you to a working prototype quickly and removes the operational burden of ingestion and indexing. Build custom when retrieval quality is the product, your documents are domain-specific, or you need hybrid search, metadata filters and reranking control. Custom does not mean from scratch: use a vector store, an embedding API and an evaluation harness, and write only the parts that differentiate. Start managed, measure where it fails, and replace the failing stage rather than the whole stack.

How should you chunk documents?

Chunk by semantic structure, not by a fixed character count. Headings, sections, tables and code blocks carry meaning that a blind 500-token window destroys. Keep parent-child relationships so a retrieved chunk can expand to its section, add metadata for access control and source, and test overlap rather than assuming it helps. There is no universal chunk size; measure recall on a labelled query set. For tables, preserve the header row with every chunk or the numbers become unreadable. For contracts, keep clause boundaries intact so a citation points to the right obligation.

Evaluation that catches real failures

Evaluate retrieval, generation and tool use separately. A single end-to-end score hides which component failed.

How large should an evaluation set be?

Large enough to cover each intent and failure mode, not large enough to look impressive. Start with 50 to 100 real tasks per intent, stratified by document type, query length and edge case, then grow the set from production failures. Keep a frozen holdout so a prompt change cannot be tuned to the test. Track inter-rater agreement if humans grade answers. A small, well-labelled set beats a large synthetic one that does not resemble your traffic. Version the set with the prompt and model ID so a result is reproducible.

How do you evaluate retrieval and agent trajectories?

For retrieval, measure recall at k, mean reciprocal rank and nDCG against labelled relevant passages. For generation, measure groundedness and answer correctness against the retrieved evidence. For an agent, evaluate both the outcome and the path: allowed tools, step count, cost and whether a human approval was required. Replay traces to reproduce failures. A correct answer produced by an unsafe path is still a failure. The RAG retrieval benchmark is a starting harness for the retrieval layer.

An evaluation checklist for RAG and agent systems.
DimensionQuestionEvidence
Retrieval recallDid the right passage appear in the top k results?Labelled queries, recall at k
GroundednessIs every claim supported by the retrieved text?Citation audit
Answer correctnessDoes the answer resolve the user's question?Rubric or blind human review
Tool successDid calls validate and complete?Traces, error rates
TrajectoryWas the path allowed and efficient?Step logs, tool allowlist
LatencyWhat is p50 and p95 time per task?Traces
CostWhat is cost per successful task?Usage logs, retries
SafetyDid injection and PII tests pass?Red-team set
RegressionDid a model or prompt change break known cases?Frozen evaluation set

MCP, tools and security

Tool access is where an agent becomes an attack surface. The protocol standardises the interface; it does not supply the permissions model.

What is MCP, and what are the main agent security risks?

The Model Context Protocol is an open interface for exposing tools and context to AI applications. It standardises how a client discovers and calls a server, which reduces bespoke integration work. It is not a security boundary. The main risks are prompt injection, over-broad tool scopes, data exfiltration through tool calls, and confused-deputy access where the agent acts with more authority than the user. Use least privilege, allowlists, sandboxing, human approval for irreversible actions and audit logs. Treat every tool description and every retrieved document as untrusted input, because both can carry instructions. The MCP specification defines the interface, not your policy.

What to verify before you ship

  • Control flow. A diagram that shows where the model chooses a step and where code does.
  • Tool permissions. Least privilege, allowlists and a human approval gate for irreversible actions.
  • Data path. What is retrieved, what is sent to the model, what is logged and where it is retained.
  • Evaluation set. A frozen, versioned set with retrieval, generation and trajectory cases.
  • Fallback. A degraded path when retrieval is empty, a tool fails or the model is unavailable.
  • Cost caps. Step limits, token budgets and a per-task ceiling. See unlimited API limits for plan-level constraints.
  • Human escalation. A route for low-confidence or high-impact cases, with the context a reviewer needs.
  • Observability. Traces that connect a user request to retrieved chunks, model calls and tool results.

An agent is a control-flow decision, not a product category. If you cannot draw the loop, you cannot operate it.

What we did not verify. We did not benchmark a specific agent framework or run a retrieval evaluation for this page. The checklist is an operating model, not a score. Test it against your own corpus, tools and traffic, and treat vendor benchmark claims as inputs to your evaluation rather than substitutes for it.

Frequently asked questions

Is RAG better than fine-tuning?

They solve different problems. RAG gives the model fresh, attributable evidence at query time; fine-tuning changes style, format or task behaviour by training on examples. RAG is usually the better first move for knowledge that changes and answers that need citations. Fine-tuning can help when the model must follow a rigid output contract or a house style. Many production systems use both, with retrieval for facts and a tuned model for form.

Do I need a vector database?

Not always. A small corpus can run on keyword search or a database extension such as pgvector, and that may be enough to prove the workflow. A dedicated vector database earns its place when you need scale, hybrid search, metadata filtering, replication or low-latency approximate nearest-neighbour queries. The index is only one stage; a weak parser or chunker will not be fixed by a faster database. Measure retrieval quality before buying infrastructure.

Which embedding model should I use?

Use the model that performs best on your own retrieval set, not the one at the top of a general benchmark. Plugsky Embed offers 2048-dimensional vectors with an 8,192-token context for RAG and semantic search; the multilingual variant is a separate model ID. Check the dimension, normalisation, language coverage and maximum input length before committing. Changing the model later means re-embedding the corpus, so record the model ID and dimension with every vector. Store both in the index metadata so a later mismatch is visible.

What is hybrid search?

Hybrid search combines lexical matching, such as BM25, with vector similarity. Lexical search is strong for exact terms, product codes, names and rare words; vectors are strong for paraphrase and semantic similarity. Combining them can improve recall on messy real queries, especially when users mix identifiers with natural language. You then need a fusion method and a reranker. Measure each component separately so you know which one helped. Fuse, then evaluate again.

What is reranking?

Reranking is a second-stage model that reorders the top candidates from retrieval. It improves precision by scoring query and passage together, at the cost of extra latency and compute. It is useful when recall is good but the right passage sits below the cutoff. Do not rerank a weak candidate set; fix retrieval first. Evaluate the reranker on the same labelled queries and report the latency budget it consumes. Keep that budget explicit.

How do I stop hallucinations?

You cannot eliminate them, but you can make them easier to catch. Ground answers in retrieved passages, require span-level citations, allow the model to abstain when evidence is weak, and validate claims against the source before showing them. Evaluate groundedness separately from answer correctness. A confident answer with no supporting passage should fail the check even if it happens to be true. Make abstention a valid outcome, and keep human review for high-stakes outputs.

What is an agent loop?

An agent loop is the cycle of model reasoning, tool call, observation and repeat until a stopping condition. The loop needs a hard step cap, a cost cap and a rule for what happens when a tool fails. Log every step with its input and output so a failure can be replayed. Without those controls, one ambiguous request can consume a large budget or repeat an unsafe action. The loop is the unit of evaluation, not just the final answer.

When should a human review an agent action?

Review when the action is irreversible, financially material, legally significant or outside the agent's tested scope. Also review when confidence is low, retrieval is empty or a tool returns an unexpected shape. The reviewer needs the evidence, the proposed action and the reason, not a bare approval button. Design the escalation path before launch, and measure how often it fires; a review rate near zero can mean the threshold is too high.

How do I estimate cost per task?

Instrument tokens, tool calls, retrieval queries and retries for every successful task, then divide total cost by completed tasks. Include embedding and reranking calls, vector storage and human review time. An agent with a variable number of steps has a cost distribution, not a single number, so report p50 and p95. Compare that figure with the value of the task, not with a per-million-token rate. Recompute it when the model or prompt changes.

What is the first metric to instrument?

Task success with evidence. If the system cannot complete the user's task, latency and cost are irrelevant; if it completes the task without supportable evidence, the success is fragile. Pair it with retrieval recall so you can tell whether a failure came from missing evidence or from generation. Add latency and cost once the quality signal is stable. A dashboard with ten metrics and no task-success definition is noise. Define task success before launch.

Sources

Primary sources and official documentation for this 2027 planning guide — evidence checked 20 November 2026.

  1. Model Context Protocol specification — checked 20 November 2026.
  2. Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — checked 20 November 2026.
  3. Yao et al., ReAct: Synergizing Reasoning and Acting in Language Models — checked 20 November 2026.
  4. NIST AI Risk Management Framework — checked 20 November 2026.
  5. OWASP Top 10 for Large Language Model Applications — checked 20 November 2026.

Last checked 20 November 2026.

Related reading

Measure the pipeline, not the demo

Start with a retrieval benchmark, add a frozen evaluation set, and keep a human path for high-impact actions.

Start Free → Read the docs