Best Local AI for RAG — private document Q&A on your hardware

The local RAG stack

A RAG system has three components: an LLM for answering, an embedding model for converting text to vectors, and a vector database for storing and searching those vectors. Here is the recommended local stack:

Component	Recommended	Alternatives
LLM	Mistral 7B / 12B	Qwen2.5, Llama-3.1, Phi-3
Embeddings	nomic-embed-text	BGE-M3, jina-embeddings-v3
Vector DB	ChromaDB	Qdrant, LanceDB, FAISS
Framework	AnythingLLM	LangChain, LlamaIndex, Open WebUI

Best models for RAG

Not all models are equally good at RAG. The ideal RAG model must follow instructions precisely, stay grounded in provided context, and produce concise answers without hallucination.

Mistral (7B / 12B) — best for RAG

Mistral models excel at RAG because of their instruction-following ability and long context handling. The 12B variant (Mistral-Nemo) adds multilingual support and a 128K context window.

Qwen2.5 — strong alternative

Qwen2.5 handles long contexts well and is particularly strong for multilingual RAG. The 7B variant works on most GPUs; the 72B offers cloud-comparable quality.

Llama-3.1 — decent, but larger

Llama-3.1 8B is a capable RAG model but requires more VRAM than Mistral 7B for similar quality. The 70B variant needs enterprise-grade hardware.

Local embedding models

Model	Dimensions	Context	Languages	Best for
nomic-embed-text	768	8K	English	All-purpose local RAG
BGE-M3	1024	8K	100+	Multilingual RAG
jina-embeddings-v3	1024	8K	English	Code-heavy RAG
all-MiniLM-L6-v2	384	256	English	Lightweight / CPU

All four models run locally via Ollama or sentence-transformers. For most use cases, nomic-embed-text offers the best quality-to-performance ratio on consumer hardware.

Vector databases

ChromaDB — easiest to start

ChromaDB runs in-process with no separate server. It stores vectors as files on disk and is fully Python-native. Best for individual or small-team RAG with up to 1M documents.

Qdrant — best for performance

Qdrant runs as a client-server system with a REST API. It handles 10M+ vectors efficiently, supports filtering, and has built-in quantization for memory reduction.

LanceDB — embedded analytics

LanceDB stores vectors in Lance columnar format, making it ideal for analytics-heavy RAG workflows where you also run aggregations on metadata.

Setup with Ollama + AnythingLLM

Build a private RAG system in 15 minutes:

1. Install Ollama — ollama pull mistral and ollama pull nomic-embed-text
2. Install AnythingLLM — desktop app for macOS, Linux, Windows
3. Configure AnythingLLM to use Ollama as the LLM provider and embedding provider
4. Upload documents — PDFs, Word files, text files, code repos
5. Start asking questions — AnythingLLM handles chunking, embedding, and retrieval

For custom deployments, use LangChain or LlamaIndex pointed at your local Ollama endpoint + ChromaDB.

Local RAG vs cloud RAG

Dimension	Local RAG	Cloud RAG (Plugsky)
Answer quality	Good (Mistral 12B)	Excellent (GPT-4o, Claude 3.5)
Data privacy	Your machine, your data	GDPR-compliant, sovereign options
Query latency	2-5s on GPU	1-3s
Document capacity	100K-1M docs	Unlimited
Setup effort	30 min initial	API call
Cost at 10K queries/day	Electricity only	$10-50/day

Plugsky also offers managed RAG through the same OpenAI-compatible API, with built-in embedding, chunking, and vector storage — no infrastructure needed.

Performance benchmarks

On consumer hardware, here are typical RAG query times:

Setup	Embedding	Search	Generation	Total
Mistral 7B (Q4) + RTX 3090	80ms	40ms	1.8s	~1.9s
Mistral 12B (Q4) + RTX 4090	80ms	40ms	3.2s	~3.3s
Mistral 7B (Q4) + Apple M2 (CPU)	300ms	100ms	8.5s	~8.9s
Mistral 7B (Q4) + 16 GB VRAM	80ms	40ms	2.1s	~2.2s

When local RAG wins

• Sensitive documents. Financial records, legal contracts, medical data, trade secrets — never leave your network.
• High query volume. At 10,000+ queries per day, cloud API costs make local RAG the clear economic choice.
• Offline environments. Air-gapped factories, field research, military deployments — RAG works without internet.
• Regulatory compliance. GDPR, HIPAA, PDPL, and many other regulations restrict data processing to local infrastructure.

For all other cases, Plugsky's cloud RAG gives you better models, zero maintenance, and the same API. Start local, scale to cloud with a one-line config change.

Frequently asked questions

What is the best local model for RAG?

Mistral (7B and 12B) is the best local model for RAG. It handles long context well, follows instructions precisely, and generates concise grounded answers. Qwen2.5 is a strong alternative, especially for Arabic or multilingual document collections.

What embedding model should I use for local RAG?

nomic-embed-text is the best all-purpose local embedding model (768-dim, 8K context). For multilingual documents, use BGE-M3 (1024-dim, 8K context, 100+ languages). For code-heavy RAG, use jina-embeddings-v3.

Which vector database should I use locally?

ChromaDB is the easiest for local RAG — pure Python, no external dependencies, runs in-process. For larger collections (1M+ vectors), Qdrant offers better performance and a client-server architecture. LanceDB is ideal for embedded analytics-heavy workloads.

How long does a local RAG query take?

On a consumer GPU (RTX 3090), expect 2-5 seconds per query total: ~100ms for embedding, ~50ms for vector search, and ~1-4s for generation depending on model size. On CPU-only, expect 5-15 seconds per query with a 7B model.

When is local RAG better than cloud RAG?

Local RAG is better when your documents contain PII, trade secrets, or regulated data that cannot leave your network. It is also better for high-frequency queries (1000s/day) where cloud API costs add up fast, and for offline or air-gapped environments.