Best Local AI for Agents — run AI agents on your hardware

Qwen2.5 — best for function calling

Qwen2.5 is the best local model for AI agents. Its function calling capability is the strongest of any sub-10B model — it reliably parses tool definitions and returns valid JSON tool calls.

Why Qwen2.5 leads for agents:

• Tool adherence: Returns valid JSON tool calls >95% of the time in our tests
• Multi-turn tool use: Maintains consistent function calling across sequential turns
• Parallel tool calls: Can invoke multiple tools in a single response
• Strong reasoning: Better than other 7B models at deciding which tool to use

Available in 7B (8 GB VRAM), 32B (16 GB), and 72B (24 GB+). The 7B variant is the sweet spot for local agent development.

Mistral — strong instruction following

Mistral 7B and Mistral 12B (Nemo) are excellent for agent tasks that prioritise instruction following over raw tool adherence. Mistral models understand nuanced instructions well and are less likely to misinterpret tool descriptions.

Mistral strengths for agents:

• Instruction precision: Follows complex multi-part instructions accurately
• Concise output: Generates compact tool calls without extraneous text
• Long context: 32K (7B) and 128K (12B) for maintaining agent state
• Multilingual agents: 12B variant supports 100+ languages

Llama-3.1 — decent but requires care

Llama-3.1 8B (Meta) has function calling support but requires more careful prompting to produce valid tool calls consistently. It tends to describe what it would do rather than returning structured tool call JSON.

Tips for using Llama-3.1 with agents:

• Use a system prompt that explicitly demands JSON-only responses
• Provide clear few-shot examples of tool call formats
• Implement retry logic with error correction for malformed calls
• Consider the 70B variant for significantly better instruction following

Model comparison for agents

Model	Function calling	Instruction follow	Reasoning	Tool reliability	Min VRAM
Qwen2.5 7B	Excellent	Very good	Good	95%+	8 GB
Qwen2.5 32B	Superior	Excellent	Very good	97%+	16 GB
Mistral 7B	Good	Excellent	Good	90%+	8 GB
Mistral 12B	Good	Excellent	Very good	92%+	16 GB
Llama-3.1 8B	Fair	Good	Good	80%+	8 GB
Llama-3.1 70B	Very good	Excellent	Very good	95%+	48 GB

Agent frameworks that work locally

Because Ollama and local inference servers expose the OpenAI-compatible API, most agent frameworks work with zero modification:

Framework	Local support	Notes
LangChain / LangGraph	Full	Built-in Ollama integration, tool decorators, agent executor
CrewAI	Full	Set OpenAI base URL to localhost:11434/v1
AutoGen	Full	Works with any OpenAI-compatible endpoint
Vercel AI SDK	Full	Configure provider with custom base URL
OpenAI Assistants API	No	Server-side only, requires cloud API key

Setup guide

Run an AI agent locally in 10 minutes:

1. Install Ollama and pull a model: ollama pull qwen2.5:7b
2. Install Python packages: pip install openai langchain
3. Write your agent:
   from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
response = client.chat.completions.create(
  model="qwen2.5:7b",
  messages=[{"role": "user", "content": "search the web for latest AI news"}],
  tools=[your_tool_definitions]
)
4. Use LangChain for simplicity: LangChain's Ollama integration handles tool parsing automatically.

Limitations of local agent models

Local models are great for agents, but they have real limits:

• Weak multi-step reasoning: Complex chains of tool calls (5+ steps) degrade in quality
• Tool call errors: Even the best local model returns malformed JSON ~5% of the time
• Context limitations: 32K-128K is good, but agent state across many turns consumes it fast
• No parallel execution: Local models process one tool call at a time (some cloud models batch)
• No built-in retry: You must implement error handling yourself

Bridge: local dev, cloud production

The practical approach to AI agents is a hybrid one:

1. Develop locally with Qwen2.5 or Mistral. Prototype your agent loops, test tool definitions, iterate quickly with zero API costs.
2. Deploy to Plugsky when you need reliable autonomy. Plugsky's frontier models (GPT-4o class) handle complex multi-step reasoning, maintain coherent agent state, and include built-in fallback and retry.
3. Switch with one line: base_url: "http://localhost:11434/v1" → base_url: "https://api.plugsky.com/v1"

Plugsky's agent-optimised models include plugsky-frontier and plugsky-fusion (for tool-intensive workflows). Same API, no code changes.

Frequently asked questions

What is the best local model for AI agents?

Qwen2.5 7B is the best local model for agent tasks. It has the strongest function calling adherence of any sub-10B model and reliably returns valid JSON tool calls. Mistral 7B is a close second with good instruction-following. Llama-3.1 8B is decent but more prone to formatting errors in tool calls.

Can I build a fully autonomous agent with local models?

Yes for simple autonomous agents (web search, data lookup, file operations). For complex multi-step reasoning, tool chaining, and dynamic decision-making, local models struggle compared to GPT-4o or Claude 3.5. The practical approach is local for simple agents, cloud for complex ones.

What frameworks work with local agent models?

Any OpenAI-compatible framework works: LangChain, CrewAI, AutoGen, and Vercel AI SDK all support local models through a base URL change. Point the framework at http://localhost:11434/v1 and it functions identically to cloud APIs.

How do I set up a local AI agent?

Install Ollama, pull qwen2.5:7b, write a Python script using the OpenAI SDK with base_url=http://localhost:11434/v1, and define your tools as JSON schema. LangChain makes this even easier with built-in Ollama integration and tool decorators.

When should I use cloud models for agents instead of local?

Use cloud models (via Plugsky) for agents that need multi-step reasoning, complex tool chaining, dynamic task planning, handling ambiguous instructions, or maintaining coherent state across many turns. Local models are best for single-turn tool execution and simple workflows.