Running LLMs locally gives you privacy, offline capability, and zero API costs. But the tooling landscape is fragmented. Ollama, vLLM, and llama.cpp occupy different niches across the ease-of-use to performance spectrum.
Quick overview
Each engine targets a different user profile:
- Ollama — the beginner-friendly option. One-command install, built-in model registry, CLI + API. Wraps llama.cpp under the hood with a model management layer.
- vLLM — the production workhorse. PagedAttention for high-throughput GPU serving, continuous batching, tensor parallelism. Requires GPU expertise to set up.
- llama.cpp — the edge/embedded specialist. Pure C/C++, runs on anything including Raspberry Pi, phones, and Apple Silicon with Metal acceleration.
Feature comparison
| Capability | Ollama | vLLM | llama.cpp |
|---|---|---|---|
| Setup complexity | One command | Python + pip + CUDA | Compile from source or binary |
| Inference speed | Good (single-user) | Excellent (batched) | Excellent (single-user) |
| GPU required | No (CPU fallback) | Yes (NVIDIA CUDA) | No (CPU, Metal, Vulkan) |
| CPU support | ✓ (via llama.cpp) | ✗ (GPU only) | ✓ (best in class) |
| Apple Silicon (MPS) | ✓ (Metal) | ✗ | ✓ (Metal — best) |
| OpenAI-compatible API | ✓ (with adapter) | ✓ (native, full) | Basic HTTP server |
| Continuous batching | ✗ (planned) | ✓ (PagedAttention) | ✓ (parallel decoding) |
| Quantization support | ✓ (GGUF) | ✓ (AWQ, GPTQ, FP8) | ✓ (GGUF — most formats) |
| Model registry | ✓ (ollama.com/library) | ✗ (manual download) | ✗ (manual download) |
| Community & ecosystem | Medium (growing fast) | Large (UC Berkeley) | Large (Gerganov) |
When to choose Ollama
Ollama is the best entry point for local LLMs. Its one-command install (curl -fsSL https://ollama.com/install.sh | sh) and ollama pull llama3 workflow makes it trivial for developers and hobbyists. The built-in model library means no hunting for weights.
Ollama wraps llama.cpp for CPU/Metal inference and adds a model server with basic OpenAI compatibility. It's ideal for personal use, prototyping, and single-user applications where setup time matters more than throughput.
Choose Ollama if: you want the fastest path from zero to running a model, you value a CLI experience, or you're experimenting with local AI for the first time.
When to choose vLLM
vLLM is the gold standard for production local inference on GPUs. Its PagedAttention algorithm achieves near-100% GPU utilization with continuous batching, making it 2-10x more throughput-efficient than naive implementations. It natively serves the OpenAI Chat Completions API format.
vLLM supports tensor parallelism across multiple GPUs, AWQ/GPTQ/FP8 quantization, and structured output (JSON mode, regex, grammar). It's what many commercial API providers run under the hood.
Choose vLLM if: you have NVIDIA GPUs (A100, H100, RTX 4090+), you need high throughput for multiple concurrent users, or you're building a production inference pipeline.
When to choose llama.cpp
llama.cpp is the most portable inference engine. Written in pure C/C++, it compiles on virtually any platform — x86, ARM, RISC-V, WebAssembly. Its GGUF format supports a vast range of quantization levels (Q2 through Q8, IQ2 through IQ4) for running models on minimal hardware.
Apple Silicon users benefit from Metal acceleration that rivals GPU performance on smaller models. The llama-server binary provides a basic HTTP API, though it lacks OpenAI compatibility out of the box.
Choose llama.cpp if: you're on a Mac, you need to run on CPU-only hardware, you're targeting edge devices (Raspberry Pi, phones), or you want the finest control over quantization and memory usage.
When to outgrow local (Plugsky)
All three engines share a ceiling: your hardware. A single GPU can serve at most one or two concurrent users for 70B-class models. You hit limits with:
- Model size: cannot run 70B+ models without multiple GPUs
- Concurrency: more than a few simultaneous users degrades latency
- Uptime: local hardware fails, needs reboots, consumes power
- GPU cost: a single H100 costs $30K+ — hard to justify for variable workloads
Plugsky gives you cloud-hosted inference with the same OpenAI-compatible API. Flat pricing from $99/mo means no GPU capex, no power bills, no ops. You get access to 30+ models including 70B-class and MoE architectures without buying hardware.
Think of Plugsky as "vLLM as a service" — production-grade inference without the GPU cluster.
Frequently asked questions
Which is faster: Ollama, vLLM, or llama.cpp?
vLLM is generally fastest for high-throughput GPU serving with PagedAttention. llama.cpp is fastest on CPU and Apple Silicon. Ollama is slower per-request but adds a user-friendly CLI and model management layer on top of llama.cpp.
Can I run Ollama, vLLM, and llama.cpp on CPU?
llama.cpp has the best CPU support with optimized ARM and x86 kernels. Ollama inherits llama.cpp CPU support. vLLM is GPU-first and not recommended for CPU-only systems.
Which local engine has the best API compatibility?
vLLM ships with a native OpenAI-compatible API server (Chat Completions, Completions, Embeddings). Ollama has its own API that is OpenAI-compatible with an adapter. llama.cpp offers a basic server endpoint that is less feature-complete.
What hardware do I need for each engine?
Ollama: 8GB+ RAM for 7B models, runs on CPU/GPU/MPS. vLLM: NVIDIA GPU with 16GB+ VRAM recommended, requires CUDA. llama.cpp: runs on anything — CPU, GPU, Apple Silicon, AMD, Intel ARC, minimal 4GB RAM for quantized models.
When should I choose Plugsky over local inference?
Choose Plugsky when you outgrow local hardware: need larger models (70B+), need concurrent users, need 99.9% uptime, or want flat predictable pricing without GPU capex. Plugsky runs on cloud GPUs with no setup.
Outgrew your local setup?
Plugsky gives you cloud inference at flat pricing. No GPU, no setup.
Start Free → See pricing