Architecture
A local AI agent architecture has four layers:
┌─────────────────────────────────────────────────┐ │ Your Application │ │ (CLI, web app, desktop, automation script) │ ├─────────────────────────────────────────────────┤ │ Agent Orchestrator │ │ - Sends messages to local inference engine │ │ - Handles tool call → execute → feed back loop │ │ - Manages conversation context / token budget │ ├─────────────────────────────────────────────────┤ │ Local Inference Engine │ │ Ollama │ vLLM │ LM Studio │ llama.cpp │ │ Runs open-weight models on your GPU / CPU │ ├─────────────────────────────────────────────────┤ │ Tool Execution Layer │ │ Web search (local scraper) │ File system │ │ Local DB │ Code interpreter │ Custom scripts │ └─────────────────────────────────────────────────┘
Every component runs on your machine. No cloud dependency. The local inference engine exposes an OpenAI-compatible API, and your agent code uses the standard OpenAI SDK.
Function calling with local models
Many local models support OpenAI-compatible function calling (tool use). The model receives a tools array alongside the messages and returns structured tool_calls when it decides a tool should be invoked. Your code executes the tool and feeds the result back.
Models with reliable function calling:
- Llama 3.2 / 3.3 — best overall function calling among open-weight models
- Qwen 2.5 — strong tool-use performance, supports parallel calls
- Mistral / Nemo — good function calling in the 7B-12B range
- DeepSeek-V3-Lite — excellent for complex tool schemas
- Command R+ — designed for agentic use cases
Integrating tools
Local agents can integrate any tool that runs on your machine. Unlike cloud agents, there are no rate limits, no IP whitelists, and no data egress concerns. Common local tools include:
| Tool type | Examples | Implementation |
|---|---|---|
| File system | Read, write, search local files | Python os/glob/shutil |
| Web search | Local scraping, self-hosted SearXNG | httpx + BeautifulSoup |
| Database | SQLite, PostgreSQL, DuckDB | DB-API connectors |
| Code execution | Python sandbox, SQL queries | subprocess, Docker containers |
| Document processing | PDF, DOCX, CSV parsing | PyMuPDF, python-docx, pandas |
| Local APIs | Home Assistant, Jellyfin, Plex | httpx to local endpoints |
RAG on local documents
Local RAG keeps your documents entirely on your hardware. The pipeline uses three local components:
1. Local embedding model
# Using Ollama for embeddings
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
def embed(texts):
resp = client.embeddings.create(
model="nomic-embed-text",
input=texts,
)
return [e.embedding for e in resp.data]2. Local vector database
# ChromaDB — runs in-process, no server needed
import chromadb
db = chromadb.PersistentClient(path="./agent-knowledge")
collection = db.get_or_create_collection("docs")
# Add documents
collection.add(
ids=["doc1", "doc2"],
embeddings=[embedding1, embedding2],
metadatas=[{"source": "report.pdf"}, {"source": "email.pdf"}],
)
# Query at runtime
results = collection.query(
query_embeddings=[query_embedding],
n_results=5,
)3. Local LLM (via Ollama/vLLM)
# Feed retrieved context into the agent
context = "\n".join(results["documents"][0])
agent_prompt = f"Context:\n{context}\n\nQuestion: {user_query}"
resp = client.chat.completions.create(
model="llama3.2",
messages=[
{"role": "system", "content": agent_prompt},
{"role": "user", "content": user_query},
],
tools=tools,
)The agent loop
The core loop for a local agent is identical to a cloud agent. The only difference is the base_url pointing to your local engine:
User input → LLM (with tools) → tool_calls?
├─ No → return response
└─ Yes → for each call:
execute tool locally
append result as "tool" message
loop back to LLM
Local agents typically run with a step limit of 5-10 iterations to avoid runaway loops. Use tool_choice: "auto" for the model to decide when to call tools and when to respond directly.
Complete code example
from openai import OpenAI
import json
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
tools = [
{
"type": "function",
"function": {
"name": "search_files",
"description": "Search local files by keyword",
"parameters": {
"type": "object",
"properties": {
"query": {"type": "string"},
"path": {"type": "string"}
},
"required": ["query"]
}
}
},
{
"type": "function",
"function": {
"name": "read_file",
"description": "Read the contents of a file",
"parameters": {
"type": "object",
"properties": {
"path": {"type": "string"}
},
"required": ["path"]
}
}
}
]
def local_agent(user_input, max_steps=5):
messages = [{"role": "user", "content": user_input}]
for step in range(max_steps):
r = client.chat.completions.create(
model="llama3.2",
messages=messages,
tools=tools,
)
msg = r.choices[0].message
if not msg.tool_calls:
return msg.content
messages.append(msg)
for tc in msg.tool_calls:
fn = tc.function.name
args = json.loads(tc.function.arguments)
if fn == "search_files":
result = search_vs_code(args["query"])
elif fn == "read_file":
result = open(args["path"]).read()
messages.append({
"role": "tool",
"tool_call_id": tc.id,
"content": json.dumps({"result": result})
})
return "Max steps reached"
# Example
output = local_agent("Find all Python files related to AI and summarize them")
print(output)Local vs managed agent API
| Factor | Local agent (DIY) | Plugsky (managed agent API) |
|---|---|---|
| Privacy | Complete — data never leaves | Configurable — VPC/on-prem available |
| Model choice | 1-3 models (limited by VRAM) | 30+ models, swap instantly |
| Tool ecosystem | Any local tool, script, or API | Built-in tools + any OpenAI-compatible tool |
| Function calling | Depends on model quality | Reliable across all models |
| Uptime | Machine-dependent | 99.9% SLA |
| Team access | Manual (LAN or VPN) | Built-in API keys + RBAC |
| Scaling | Single GPU or machine | Auto-scaling, multi-node |
| RAG | Manual setup (ChromaDB + embeddings) | Managed — upload and query |
| Cost model | Hardware + electricity | Flat monthly ($7-$249) |
Migration path to Plugsky
When your local agent needs team access, reliability, or access to more capable models, the migration takes one line change:
# Before (local)
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
# After (Plugsky)
client = OpenAI(base_url="https://api.plugsky.com/v1", api_key="sk-live-...")
# Everything else stays the same:
# tools, messages, tool_calls, streaming, agent loop — all identicalPlugsky's agent API adds managed capabilities on top of the same OpenAI-compatible interface: built-in RAG, sub-agent orchestration, audit logging, model routing, and team access controls. Your agent code never changes — only the endpoint and model name.
Build private agents, deploy with confidence
Prototype agents locally with complete privacy. Migrate to Plugsky when you need team access, uptime, and 30+ models — without rewriting a single line of agent code.
Start Free → Back to Local AI →