Local RAG — private RAG pipeline setup guide

Local RAG stack

A local RAG pipeline has four components, all running on your machine:

┌──────────────┐     ┌──────────────┐     ┌──────────────┐
│  Documents    │ ──▶ │  Embedding    │ ──▶ │  Vector DB    │
│  (PDF, DOCX,  │     │  Model        │     │  (ChromaDB,    │
│   TXT, HTML)  │     │  (nomic, bge) │     │   Qdrant)      │
└──────────────┘     └──────────────┘     └──────────────┘
                                                    │
                                                    ▼
┌──────────────┐     ┌──────────────┐     ┌──────────────┐
│  Response     │ ◀── │  Local LLM   │ ◀── │  Retrieved   │
│  to user      │     │  (Ollama/     │     │  chunks      │
│               │     │   vLLM)       │     │  + query     │
└──────────────┘     └──────────────┘     └──────────────┘

Every component runs locally. No cloud services, no API calls to external providers, no data leaving your network.

Local embedding models

Embedding models convert text into vector representations. These run efficiently on CPU and are small enough to fit alongside your LLM:

Model	Dimensions	Context	Best for	RAM use
nomic-embed-text	768	8192	General purpose, best all-around	~500 MB
bge-m3	1024	8192	Multilingual, dense retrieval	~2 GB
bge-small-en-v1.5	384	512	Fast, low-resource	~100 MB
mxbai-embed-large-v1	1024	512	High quality, English-focused	~700 MB
all-MiniLM-L6-v2	384	256	Lightest, CPU-friendly	~80 MB

Running embeddings with Ollama

bash

# Pull an embedding model
ollama pull nomic-embed-text

# Test embedding
curl http://localhost:11434/api/embed \
  -d '{"model": "nomic-embed-text", "input": "Your text here"}'

python

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

def embed(texts):
    resp = client.embeddings.create(
        model="nomic-embed-text",
        input=texts if isinstance(texts, list) else [texts],
    )
    return [e.embedding for e in resp.data]

Local vector databases

Vector databases store and retrieve embeddings. These run locally with zero cloud dependency:

Database	Mode	Persistence	Best for
ChromaDB	In-process / client-server	Disk (SQLite)	Quick prototyping, small to medium collections
Qdrant	Client-server (Docker/binary)	Disk	Production-like local testing, high performance
LanceDB	Embedded (Arrow)	Disk (Lance format)	Large collections, ML-native workflows
FAISS (manual)	In-memory	No (export/load)	Maximum performance, static datasets

ChromaDB setup

bash

pip install chromadb

python

import chromadb

db = chromadb.PersistentClient(path="./rag-data")
collection = db.get_or_create_collection(
    name="knowledge-base",
    metadata={"hnsw:space": "cosine"},
)

# Add documents in batches
collection.add(
    ids=["doc-001", "doc-002"],
    embeddings=[emb1, emb2],
    documents=["Content of doc 1", "Content of doc 2"],
    metadatas=[{"source": "report.pdf"}, {"source": "email.txt"}],
)

# Query
results = collection.query(
    query_texts=["What is our policy on data retention?"],
    n_results=5,
)

Local LLM for generation

The generation step combines the user query with retrieved context and sends both to a local LLM. Use any engine running an OpenAI-compatible endpoint:

Engine	Recommended models for RAG	Min RAM
Ollama	llama3.2, qwen2.5, mistral	8 GB
vLLM	Llama 3.2, Qwen 2.5, Mistral	16 GB (GPU)
llama.cpp	Any GGUF model	8 GB
LM Studio	Any GGUF model (GUI)	8 GB

Complete pipeline example

python

import chromadb
from openai import OpenAI

# ── 1. Connect to local embedding + LLM ──
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

# ── 2. Load or create vector store ──
db = chromadb.PersistentClient(path="./rag-data")
collection = db.get_or_create_collection("docs")

def index_documents(file_paths):
    """Chunk, embed, and store documents."""
    for path in file_paths:
        text = open(path).read()
        chunks = [text[i:i+1000] for i in range(0, len(text), 1000)]
        for idx, chunk in enumerate(chunks):
            resp = client.embeddings.create(
                model="nomic-embed-text",
                input=[chunk],
            )
            embedding = resp.data[0].embedding
            collection.add(
                ids=[f"{path}-{idx}"],
                embeddings=[embedding],
                documents=[chunk],
                metadatas=[{"source": path, "chunk": idx}],
            )

def ask(query, top_k=5):
    """Query the RAG pipeline."""
    # Embed the query
    q_resp = client.embeddings.create(
        model="nomic-embed-text",
        input=[query],
    )
    q_emb = q_resp.data[0].embedding

    # Retrieve
    results = collection.query(
        query_embeddings=[q_emb],
        n_results=top_k,
    )

    # Build context
    context = "\n---\n".join(results["documents"][0])

    # Generate answer
    response = client.chat.completions.create(
        model="llama3.2",
        messages=[
            {"role": "system", "content": f"Answer based on this context:\n\n{context}"},
            {"role": "user", "content": query},
        ],
    )
    return response.choices[0].message.content

# Usage
index_documents(["policy.pdf", "handbook.docx"])
answer = ask("What is our vacation policy?")
print(answer)

Local vs cloud RAG

Factor	Local RAG (DIY)	Plugsky (managed RAG)
Privacy	Complete — data never leaves	Configurable (VPC/on-prem on Enterprise)
Setup time	Hours to days	Minutes (upload and query)
Embedding models	1-2 models (limited by RAM)	Multiple models, auto-optimized
Vector DB maintenance	Manual (ChromaDB, Qdrant)	Fully managed
Chunking strategy	Must implement and tune	Auto-chunking with configurable strategy
Re-ranking	Must implement separately	Built-in cross-encoder re-ranking
Hybrid search	Must implement (BM25 + vector)	Built-in hybrid search
Scaling	Single machine	Auto-scaling to millions of documents
Cost	Hardware + electricity	Flat monthly ($7-$249)

When to stay local

Local RAG is the right choice when you need complete data sovereignty, are prototyping and iterating rapidly, have a small document set (under 10,000 pages), or are building a single-user application. It's also ideal for air-gapped environments where no cloud connectivity is permitted.

Local RAG becomes impractical when you need to share the knowledge base across a team of 10+ users, need 99.9% uptime, have document collections exceeding 100,000 pages, or need multi-language hybrid search with re-ranking.

Migration to Plugsky

Plugsky's managed RAG API handles chunking, embedding, indexing, retrieval, and re-ranking automatically. Migration is straightforward:

python

# Before (local)
# Manual pipeline with ChromaDB + Ollama embedding + local LLM

# After (Plugsky)
from openai import OpenAI

client = OpenAI(
    base_url="https://api.plugsky.com/v1",
    api_key="sk-live-...",
)

# Upload documents — Plugsky handles chunking + embedding
client.files.create(file=open("policy.pdf", "rb"), purpose="assistants")

# Query — built-in retrieval + re-ranking
response = client.chat.completions.create(
    model="plugsky-pro",
    messages=[
        {"role": "system", "content": "Answer from your knowledge base."},
        {"role": "user", "content": "What is our vacation policy?"},
    ],
)

# No vector DB to manage. No embedding pipeline. No chunking config.

Build local RAG, deploy with Plugsky

Prototype your RAG pipeline locally with complete privacy. Migrate to Plugsky's managed RAG when you need team access, scaling, and enterprise features — no code changes required.

Start Free → Back to Local AI →

Local RAG stack

Local embedding models

Running embeddings with Ollama

Local vector databases

ChromaDB setup

Local LLM for generation

Complete pipeline example

Local vs cloud RAG

When to stay local

Migration to Plugsky

Build local RAG, deploy with Plugsky

Related

Local AI agents

Best local LLMs

RAG API