Budget tier: under $300
You do not need a flagship GPU to run local LLMs. A used last-generation card with 12 GB of VRAM handles 7B and small 13B models at usable speeds.
| GPU | VRAM | Used price | Models supported |
|---|---|---|---|
| RTX 3060 | 12 GB | $180-220 | 7B Q4, 8B Q4, 13B Q4 (tight) |
| RTX 2060 12GB | 12 GB | $150-180 | 7B Q4, 8B Q4 |
| RTX 2080 Ti | 11 GB | $250-300 | 7B Q4, 8B Q4, 13B Q4 (tight) |
| RTX 4060 | 8 GB | $280 new | 7B Q4 only |
Best pick: RTX 3060 12 GB. The 12 GB VRAM is the key differentiator. An RTX 4060 has faster cores but only 8 GB — it cannot run 13B models at any quantization. The RTX 3060 can. Buy used on eBay or Facebook Marketplace.
With a 3060, expect ~20-30 tok/s on a 7B Q4 model and ~12-15 tok/s on a 13B Q4. This is fast enough for interactive chat and most single-user workloads.
Mid-range: $550-700
This is the sweet spot for most users. You get enough VRAM for 13B-14B models at good quantizations and solid token speeds.
| GPU | VRAM | Price | Models supported |
|---|---|---|---|
| RTX 4070 | 12 GB | $550 new | 7B Q8, 13B Q4, 14B Q4 |
| RTX 4070 Super | 12 GB | $600 new | 7B Q8, 13B Q4, 14B Q4 |
| RTX 3090 | 24 GB | $700-800 used | 13B Q8, 30B Q4, 32B Q4 |
| RTX 4070 Ti Super | 16 GB | $800 new | 13B Q8, 14B Q8, 30B Q4 |
Best pick: RTX 3090 24 GB (used). At ~$700-800, the 3090 offers 24 GB of VRAM — the highest capacity of any consumer GPU under $1000. It runs 30B Q4 models comfortably and handles 70B Q2 models. A used 3090 is the best value in local LLM hardware, period.
Runner-up: RTX 4070 Ti Super 16 GB. If you want a new card with warranty, the 4070 Ti Super offers 16 GB and excellent token speeds (~40-50 tok/s on 7B). It handles 14B Q8 and smaller 30B Q4 models.
High-end: $1600+
For running 70B models, training, or production serving, you need a flagship GPU or a professional card.
| GPU | VRAM | Price | Models supported |
|---|---|---|---|
| RTX 4090 | 24 GB | $1600-1800 | 30B Q8, 32B Q8, 70B Q2 |
| RTX 5090 | 32 GB | $2000+ | 70B Q4, 72B Q4 |
| RTX 6000 Ada | 48 GB | $5500 | 70B Q8, 72B Q8 |
| A6000 | 48 GB | $4000 used | 70B Q8, 72B Q8 |
Best pick: RTX 4090 24 GB. The 4090 is the fastest consumer GPU for local LLMs. You get 80+ tok/s on a 7B Q4 model. The 24 GB VRAM fits 30B-32B Q8 models and 70B Q2. If you can afford it, this is the best single-GPU setup.
Upgrade pick: RTX 5090 32 GB. The 5090's extra 8 GB over the 4090 is significant — it fits Llama 3.3 70B Q4 (needs ~35 GB + KV cache) when paired with some CPU offloading, or runs 32B Q8 with generous context.
Enthusiast and multi-GPU setups
For running 70B+ models at full precision or with long context, multiple GPUs are the most cost-effective approach.
| Setup | Total VRAM | Price | Models supported |
|---|---|---|---|
| 2x RTX 3090 | 48 GB | $1400-1600 | 70B Q4, 72B Q4 |
| 2x RTX 4090 | 48 GB | $3200+ | 70B Q4, 72B Q4 (fast) |
| 2x RTX 5090 | 64 GB | $4000+ | 70B Q8, 72B Q8, 120B Q4 |
| 4x RTX 3090 | 96 GB | $2800-3200 | 70B Q8, 120B Q8 |
Best pick: 2x RTX 3090. Dual 3090s give you 48 GB total for ~$1400-1600 — less than a single RTX 4090. This is enough for Llama 3.3 70B Q4 with reasonable context. NVLink is not required for inference; llama.cpp and vLLM handle multi-GPU automatically via PCIe.
Ensure your power supply can handle the load. Two RTX 3090s draw ~700W under full load. You will need a 1000W+ PSU and a motherboard with enough physical space and PCIe lanes.
AMD GPU alternatives
AMD GPUs have improved significantly for local LLMs thanks to ROCm support in llama.cpp and Ollama, but NVIDIA remains the safer choice.
| GPU | VRAM | Price | Notes |
|---|---|---|---|
| RX 7900 XTX | 24 GB | $850-950 | Best AMD option. 24 GB VRAM at a great price. ROCm support is good but ~10-20% slower than equivalent NVIDIA. |
| RX 7900 XT | 20 GB | $700-800 | Good 1440p card for LLMs. 20 GB fits most 13B Q8 and small 30B Q4 models. |
| RX 7800 XT | 16 GB | $500-600 | Decent mid-range option. 16 GB VRAM at a competitive price. Slower than RTX 4070 in ML workloads. |
| RX 7600 XT | 16 GB | $320-350 | Budget AMD option. 16 GB VRAM for cheap, but compute performance is significantly lower. |
Verdict: The RX 7900 XTX is the only AMD card worth considering over NVIDIA. Its 24 GB VRAM at ~$900 matches the used RTX 3090 on capacity and beats it on new price. For everything else, go NVIDIA — you get better performance, broader software support, and fewer compatibility headaches.
Apple Silicon (M1-M4) for local LLMs
MacBooks with Apple Silicon are surprisingly capable for local AI because unified memory acts as VRAM. A Mac with 64 GB unified memory can run models that would need a 48 GB GPU on a PC.
| Mac | Unified memory | Models supported | Token speed (7B Q4) |
|---|---|---|---|
| M1 MacBook Air | 8 GB | 1-3B Q4 | 10-20 tok/s |
| M1 Pro/Max | 16-32 GB | 7B Q4, 13B Q4 | 20-40 tok/s |
| M2/M3 Pro/Max | 16-48 GB | 7B Q8, 13B Q8, 30B Q4 | 30-60 tok/s |
| M3/M4 Ultra | 64-128 GB | 70B Q4, 72B Q4 | 40-80 tok/s |
| M4 Max | 36-128 GB | 13B Q8, 30B Q8, 70B Q4 | 50-90 tok/s |
Apple Silicon uses Metal GPU acceleration through llama.cpp and Ollama. Performance is excellent — an M4 Max at 50-90 tok/s on a 7B Q4 is competitive with an RTX 4070. The catch is that you cannot add an eGPU to a Mac, so you are limited to whatever unified memory you bought at purchase time.
Read our full guide: Local LLM on MacBook.
Token speed benchmarks
Here are real-world tokens-per-second benchmarks for popular GPUs running a 7B Q4_K_M model in llama.cpp with default settings:
| GPU | VRAM | Prompt processing | Generation |
|---|---|---|---|
| RTX 3060 12 GB | 12 GB | ~800 tok/s | ~25 tok/s |
| RTX 4060 | 8 GB | ~1200 tok/s | ~30 tok/s |
| RTX 4070 | 12 GB | ~1800 tok/s | ~40 tok/s |
| RTX 3090 | 24 GB | ~3500 tok/s | ~55 tok/s |
| RTX 4070 Ti Super | 16 GB | ~2500 tok/s | ~45 tok/s |
| RTX 4090 | 24 GB | ~5000 tok/s | ~85 tok/s |
| RX 7900 XTX | 24 GB | ~2800 tok/s | ~45 tok/s |
| M4 Max (64 GB) | 64 GB | ~3000 tok/s | ~60 tok/s |
These numbers are for single-user inference. Batch processing and serving can achieve higher throughput but at the cost of per-request latency.
Skip the GPU decision entirely
Plugsky gives you 30+ models including Llama, Qwen, Mistral, and DeepSeek. No hardware, no setup, no rate limits.
Start Free → VRAM calculatorFrequently asked questions
What is the best budget GPU for local LLMs?
The RTX 3060 12GB is the best budget GPU at ~$200 used. It fits 7B Q4 models easily and runs 13B Q4 models. For sub-$200, look for a used RTX 2060 12GB or RTX 3060 12GB on eBay or Facebook Marketplace.
Is NVIDIA or AMD better for local LLMs?
NVIDIA is better for local LLMs. CUDA and cuBLAS are the gold standard — llama.cpp, Ollama, and all major runners ship with CUDA support, fast kernels, and broad testing. AMD ROCm support has improved but still has 10-30% lower performance and more compatibility issues.
Is more VRAM or faster GPU speed better?
VRAM is king. More VRAM lets you run larger models with better quality. A 12GB RTX 3060 ($200) runs bigger models than an 8GB RTX 4070 ($550). Token speed matters only once your model fits. Always prioritise VRAM capacity over clock speed.
Can I use multiple GPUs for local LLMs?
Yes. llama.cpp and vLLM support multi-GPU inference across NVIDIA GPUs (NVLink not required). Two RTX 3090s give 48GB total — enough for Llama 3.3 70B Q4. Make sure your PSU and motherboard support the load.
Is Apple Silicon good for local LLMs?
Yes, Apple Silicon (M1/M2/M3/M4) is excellent for local LLMs because unified memory acts as VRAM. A MacBook Pro with 64GB unified memory can run models up to 50GB. Metal GPU acceleration gives 40-80 tok/s on a 7B Q4 model. The trade-off is no eGPU support.