VRAM requirements by model size
The most important specification for local LLM inference is VRAM (GPU memory). Here is how much you need for each model size at common quantization levels:
| Model size | Q4 (4.5 bit) | Q8 (8 bit) | FP16 | Recommended GPU |
|---|---|---|---|---|
| 1-3B | ~2-3 GB | ~3-5 GB | ~4-7 GB | Any GPU with 4+ GB |
| 7-8B | ~5-7 GB | ~8-10 GB | ~14-16 GB | RTX 3060 12 GB, RTX 4060 |
| 13-14B | ~8-10 GB | ~14-16 GB | ~26-28 GB | RTX 3090 24 GB |
| 30-34B | ~18-22 GB | ~32-36 GB | ~60-68 GB | RTX 4090 24 GB, A6000 48 GB |
| 70-72B | ~38-42 GB | ~68-74 GB | ~130-140 GB | 2x RTX 3090, A100 80 GB |
| 120-180B | ~65-100 GB | ~115-175 GB | ~220-340 GB | 4x A6000, H100 |
Formula: VRAM ≈ (parameters × bytes per parameter) × 1.2 (overhead). At Q4: 4.5 bits = ~0.56 bytes per parameter. At Q8: 1 byte per parameter. At FP16: 2 bytes per parameter.
Context window overhead
In addition to model weights, you need VRAM for the KV cache. Each token of context consumes roughly:
- 7B model: ~1 MB per 1K tokens of context
- 13B model: ~2 MB per 1K tokens
- 70B model: ~10 MB per 1K tokens
For an 8K context window with a 7B model, add ~8 MB extra VRAM. For 128K context with a 70B model, add ~1.3 GB.
CPU vs GPU inference
| Factor | CPU inference | GPU inference |
|---|---|---|
| Speed | 2-8 tok/s (7B Q4) | 30-120 tok/s (7B Q4) |
| Cost | Free (existing hardware) | $200-$3,000+ for GPU |
| Model size limit | Limited by RAM (32-128 GB typical) | Limited by VRAM (8-80 GB typical) |
| Energy efficiency | Lower (CPU draws full power) | Higher (GPU efficient for matrix ops) |
| Multi-user | Struggles with concurrent requests | Handles batch inference well |
| Best engine | llama.cpp, Ollama | vLLM, Ollama, llama.cpp (CUDA/Metal) |
Recommendation: Use CPU inference for experimentation, small models (3B and under), or when you don't have a GPU. Use GPU inference for production-like workloads, larger models, and any use case requiring more than 10 tok/s.
Apple Silicon (M1, M2, M3, M4)
Apple Silicon Macs are excellent for local AI because of unified memory — the GPU and CPU share the same pool, so your model can access more than a typical GPU. llama.cpp with Metal acceleration achieves strong performance.
| M-chip | Unified memory | Max model | Performance (7B Q4) |
|---|---|---|---|
| M1 (base) | 8 GB | 3B Q4 | 15-25 tok/s |
| M1 Pro | 16-32 GB | 8B Q4 | 20-35 tok/s |
| M1 Max | 32-64 GB | 13B Q4 | 25-40 tok/s |
| M2 Pro | 16-32 GB | 8B Q4 | 25-40 tok/s |
| M2 Max | 32-96 GB | 30B Q4 | 20-35 tok/s |
| M3 Pro | 18-36 GB | 13B Q4 | 30-50 tok/s |
| M3 Max | 36-128 GB | 70B Q3 | 15-25 tok/s |
| M4 Pro | 24-48 GB | 30B Q4 | 30-50 tok/s |
| M4 Max | 36-128 GB | 70B Q4 | 20-35 tok/s |
Note: Apple Silicon + Metal via llama.cpp or Ollama is the most cost-effective way to run 30B+ models on consumer hardware. A Mac Studio with 128 GB unified memory can run 70B Q4 models that would require a $15,000+ multi-GPU workstation on PC.
RAM for context windows
System RAM matters for CPU inference and for storing context. When running on CPU or CPU+GPU hybrid (some layers on GPU, rest on CPU), the model weights are loaded into system RAM. Large context windows also consume system memory:
| Context length | 7B model | 13B model | 70B model |
|---|---|---|---|
| 4K (default) | ~16 GB RAM | ~32 GB RAM | ~80 GB RAM |
| 8K | ~20 GB RAM | ~36 GB RAM | ~90 GB RAM |
| 32K | ~32 GB RAM | ~48 GB RAM | ~130 GB RAM |
| 128K | ~64 GB RAM | ~96 GB RAM | ~280 GB RAM |
Rule of thumb: Double the model's VRAM requirement for system RAM. If a 7B Q4 needs 6 GB VRAM, ensure at least 16 GB system RAM. For a 70B Q4 needing 40 GB VRAM, ensure at least 64 GB system RAM.
Storage for model files
Model files are large. Plan your storage accordingly:
| Model size | Q4 file size | Q8 file size | FP16 file size |
|---|---|---|---|
| 3B | ~1.8 GB | ~3.2 GB | ~6 GB |
| 7-8B | ~4.5 GB | ~8 GB | ~15 GB |
| 13-14B | ~8 GB | ~14 GB | ~27 GB |
| 30-34B | ~18 GB | ~32 GB | ~62 GB |
| 70-72B | ~40 GB | ~70 GB | ~140 GB |
If you plan to experiment with 5-10 models, budget 100-200 GB of free storage. An NVMe SSD is strongly recommended — model loading time is bound by disk read speed.
Recommended hardware builds
Budget build (~$800) — entry-level local AI
| Component | Recommended | Estimated cost |
|---|---|---|
| GPU | Used RTX 3060 12 GB | $200 |
| CPU | Ryzen 5 5600 / Intel i5-12400 | $120 |
| RAM | 32 GB DDR4 | $60 |
| Storage | 1 TB NVMe SSD | $60 |
| Rest (mobo, PSU, case) | B550 / B660, 650W, ATX | $360 |
Capable of: 7B-8B Q4 models at 30-50 tok/s, 13B Q4 at 15-20 tok/s (partial offload), 3B models comfortably. Good for learning, prompt engineering, and single-user agent prototypes.
Mid-range build (~$2,000) — serious local inference
| Component | Recommended | Estimated cost |
|---|---|---|
| GPU | Used RTX 3090 24 GB | $700 |
| CPU | Ryzen 7 7800X3D / i7-13700K | $350 |
| RAM | 64 GB DDR5 | $180 |
| Storage | 2 TB NVMe SSD | $120 |
| Rest (mobo, PSU, case) | X670 / Z790, 850W, ATX | $650 |
Capable of: 30B Q4 models at 15-25 tok/s, 70B Q3 at 8-12 tok/s, 7B-13B at Q8. Excellent for multi-model experimentation, local agents with RAG, and team of 2-3 users.
Workstation build ($5,000+) — production-like local setup
| Component | Recommended | Estimated cost |
|---|---|---|
| GPU | 2x Used RTX 3090 (48 GB total) or 1x A6000 48 GB | $1,400-$4,000 |
| CPU | Ryzen 9 7950X / i9-14900K | $550 |
| RAM | 128 GB DDR5 | $350 |
| Storage | 4 TB NVMe SSD | $250 |
| Rest | Workstation mobo, 1200W PSU, large case | $800 |
Capable of: 70B Q4 models at 15-25 tok/s, 120B Q3 at 8-12 tok/s, multiple concurrent models. Adequate for simulating production workloads, multi-user teams, and fine-tuning small models.
Note on cost
Before spending $2,000+ on hardware, consider that Plugsky's Enterprise plan starts at $249/month. At that price, it takes 8+ months of subscription to equal the workstation build — and you get 30+ models, 99.9% uptime, auto-scaling, and zero hardware maintenance. Use local hardware for prototyping and learning. Use Plugsky for production.
Not sure what hardware to buy?
Start with Plugsky's free tier — no hardware needed. Use it to test models and workloads before committing to a local hardware purchase.
Start Free → Back to Local AI →