Example recommendation (8GB GPU, code generation)
Example output — your results will vary based on your answers.
Table of Contents
How to choose a local model
Picking the right local LLM comes down to three factors: your hardware, your use case, and your quality requirements. Here's how to think about each.
Hardware is the gatekeeper. Your GPU VRAM or system RAM determines the maximum model size you can run. A 7B model at Q4 needs ~5GB. A 30B model at Q4 needs ~18GB. A 70B model at Q4 needs ~42GB. If your hardware can't fit the weights plus KV cache, the model simply won't load. Use the VRAM calculator for precise estimates.
Use case determines model family. Not all models are good at everything. Llama 3.1 excels at general chat and creative writing. DeepSeek Coder is purpose-built for code generation. Mistral has strong function calling for agent workflows. Qwen 2.5 is the best choice for Arabic and multilingual tasks. Pick the family that matches your primary workload.
Quality vs speed is a sliding scale. Quantization shrinks models by reducing the precision of weights. Q8 is nearly lossless but saves only 50%. Q4 saves 75% memory with minimal quality loss (1-3% on benchmarks). Q2 saves 87.5% but quality degradation is noticeable, especially for reasoning and math. Start with Q4 and adjust based on your experience.
Context length adds VRAM. The KV cache for a 7B model at 4K context uses ~1GB. At 128K context it uses ~32GB — often more than the model weights themselves. If you need long context, make sure your hardware can handle the KV cache overhead, or choose a model with efficient attention like the MQA/GQA architectures used in Llama 3.1 and Qwen 2.5.
Model size vs hardware requirements
Updated Jul 2026 — estimates include model weights + 4K KV cache + ~15% overhead
| Model | Params | FP16 | Q8 | Q4 | Q2 | Min hardware |
|---|---|---|---|---|---|---|
| Llama 3.2 3B | 3B | 6 GB | 3.5 GB | 2.2 GB | 1.5 GB | CPU / 4GB GPU |
| Qwen 2.5 Coder 1.5B | 1.5B | 3 GB | 2 GB | 1.2 GB | 0.8 GB | CPU / any GPU |
| Gemma 2 2B | 2B | 4 GB | 2.5 GB | 1.6 GB | 1 GB | CPU / 4GB GPU |
| Mistral 7B | 7B | 14 GB | 8 GB | 5 GB | 3 GB | 8GB GPU / M1 |
| Llama 3.1 8B | 8B | 16 GB | 9 GB | 5.5 GB | 3.5 GB | 8GB GPU / M1 |
| DeepSeek Coder 6.7B | 6.7B | 13 GB | 7.5 GB | 4.5 GB | 3 GB | 8GB GPU / M1 |
| Qwen 2.5 7B | 7B | 14 GB | 8 GB | 5 GB | 3 GB | 8GB GPU / M1 |
| Phi-3 Medium 14B | 14B | 28 GB | 15 GB | 8 GB | 5 GB | 16GB GPU / M4 Pro |
| Llama 3.1 13B | 13B | 26 GB | 14 GB | 8 GB | 5 GB | 16GB GPU / M4 Pro |
| Qwen 2.5 32B | 32B | 64 GB | 34 GB | 18 GB | 11 GB | 24GB GPU / multi-GPU |
| DeepSeek Coder 33B | 33B | 66 GB | 35 GB | 19 GB | 12 GB | 24GB GPU / multi-GPU |
| Llama 3.1 70B | 70B | 140 GB | 75 GB | 40 GB | 22 GB | Multi-GPU / A100 |
| Qwen 2.5 72B | 72B | 144 GB | 77 GB | 41 GB | 23 GB | Multi-GPU / A100 |
| Mixtral 8x7B | 47B | 94 GB | 50 GB | 28 GB | 16 GB | 24GB+ GPU |
Quantization guide
Quantization reduces model precision to save memory and increase speed. Here's what you need to know about each level:
| Level | Bits | Memory savings | Quality | Speed | When to use |
|---|---|---|---|---|---|
| FP16 | 16 | — | Reference | 1× | Datacenter GPUs / A100 |
| FP8 | 8 | 50% | Near-lossless | 1.5-2× | H100/H200 native |
| Q8 | 8 | 50% | Excellent (<0.5% loss) | 1.3× | 16GB+ GPUs, Apple Silicon |
| Q6 | 6 | 62.5% | Very good (~1% loss) | 1.4× | Good balance for 16GB GPUs |
| Q4 | 4 | 75% | Good (1-3% loss) | 1.8× | Sweet spot — most users |
| Q3 | 3 | 81% | Fair (3-5% loss) | 2× | Low-memory GPUs (8GB) |
| Q2 | 2 | 87.5% | Degraded (5-10% loss) | 2× | Edge / last resort only |
Pro tip: GGUF Q4_K_M is the most popular quantization variant — it uses 4-bit quantization with "importance" weighting that preserves more quality on critical weights. It's llama.cpp's recommended default and works on CPU, Apple Silicon, and GPU.
Frequently asked questions
How do I choose the right local LLM?
Start with your hardware: GPU VRAM is the most important factor (weights + KV cache must fit). Then consider your use case (chat, code, RAG, agent) to pick the right model family. Quantization lets you fit larger models into limited memory at a small quality cost. Use this interactive recommender to find the best match.
How much does quantization affect model quality?
Q8 is near-lossless (<0.5% perplexity increase). Q4 loses ~1-3% on benchmarks — the sweet spot for most users. Q2 loses ~5-10% quality, most noticeable in reasoning and math. For production use, Q4 is recommended. For maximum quality with enough hardware, FP16 is best.
Can I run a 70B model on a consumer GPU?
A 70B model at Q4 requires ~42GB VRAM — too much for a single consumer GPU (RTX 4090 has 24GB). At Q2 it needs ~22GB, fitting on a 24GB card with quality loss. For 70B+ models, you need multi-GPU setup, a datacenter GPU (A100 80GB), or use cloud inference via Plugsky.
What's the best model for code generation on 8GB VRAM?
For 8GB VRAM, DeepSeek Coder 6.7B at Q4 (~4.5GB with KV cache) is the best option for code. Qwen 2.5 Coder 7B at Q4 is also excellent. Both support 16K+ context and outperform much larger models on coding benchmarks. Use Ollama to run them locally.
What if I don't have enough hardware for local models?
You have three options: (1) use more aggressive quantization to fit models in less memory, (2) offload layers to system RAM via llama.cpp CPU+GPU mode, (3) use Plugsky cloud inference — run any model on cloud GPUs with no local hardware requirements and flat monthly pricing from $20/mo.
Last updated Jul 2026. Model availability and performance verified at time of writing — check Ollama and Hugging Face for latest versions.
Don't have enough hardware?
Run any model on cloud GPUs through Plugsky. No hardware limits, flat pricing.
Start free → API docs