Best models for 8GB VRAM
With 8 GB of VRAM (typical: RTX 2060, RTX 3060, RTX 4060, RTX 3070), your best bet is 7-8B parameter models at Q4 quantization. These fit comfortably with room for 4-8K context.
1. Qwen 2.5 7B — Q4_K_M
VRAM usage: ~4.5 GB (weights) + ~1 GB (4K context) = ~5.5 GB total. Speed: ~25 tok/s on an RTX 3060. Strengths: Top-tier reasoning for its size, strong at coding and maths, 32K native context. The best all-rounder for 8 GB GPUs. Trade-off: Slightly weaker creative writing than Llama 3.1 8B.
2. Llama 3.1 8B — Q4_K_M
VRAM usage: ~5 GB (weights) + ~1 GB (4K context) = ~6 GB total. Speed: ~22 tok/s on an RTX 3060. Strengths: Excellent instruction following, broad knowledge base, great for chat and general Q&A. The most popular local model family. Trade-off: Higher VRAM usage than Qwen 7B — tight on 8 GB with 8K context.
3. Mistral 7B — Q4_K_M
VRAM usage: ~4 GB (weights) + ~1 GB (4K context) = ~5 GB total. Speed: ~28 tok/s on an RTX 3060 (fastest of the three). Strengths: Very fast inference, strong at summarisation and classification, efficient architecture. Trade-off: Older model — lags behind Qwen 2.5 7B and Llama 3.1 8B on reasoning benchmarks.
8GB tier summary
| Rank | Model | Quant | VRAM | Context | tok/s |
|---|---|---|---|---|---|
| 1 | Qwen 2.5 7B | Q4_K_M | ~5.5 GB | 8K | ~25 |
| 2 | Llama 3.1 8B | Q4_K_M | ~6 GB | 4K | ~22 |
| 3 | Mistral 7B | Q4_K_M | ~5 GB | 8K | ~28 |
To run these: ollama run qwen2.5:7b, ollama run llama3.1:8b, or ollama run mistral:7b.
Best models for 16GB VRAM
With 16 GB of VRAM (typical: RTX 4070 Ti Super, RTX 4080, RTX 3080 Ti), you have options. You can run 8B models at Q8 for maximum quality per token, or step up to 14B models at Q4.
1. Llama 3.1 8B — Q8_K_M
VRAM usage: ~8.5 GB (weights) + ~2 GB (8K context) = ~10.5 GB total. Speed: ~35 tok/s on an RTX 4070 Ti Super. Strengths: Near-lossless quality (Q8 retains ~99.5% of FP16 performance). The 8B model at Q8 outperforms many 14B models at Q4 on fine-grained reasoning. Trade-off: You are paying for quality per token rather than raw model size.
2. Qwen 2.5 14B — Q4_K_M
VRAM usage: ~8 GB (weights) + ~2 GB (8K context) = ~10 GB total. Speed: ~18 tok/s on an RTX 4070 Ti Super. Strengths: More parameters = better knowledge and reasoning than any 8B model. Excels at coding, analysis, and multilingual tasks. Trade-off: Slower generation than the 8B Q8 option. Weaker on a per-token quality basis.
3. DeepSeek-Coder-V2-Lite — Q4_K_M
VRAM usage: ~8 GB (weights) + ~2 GB (8K context) = ~10 GB total. Speed: ~16 tok/s. Strengths: Specialised for code generation. Matches GPT-4 on coding benchmarks (HumanEval, MBPP). The best choice for developers. Trade-off: Weaker at general knowledge and creative tasks compared to Qwen 14B.
16GB tier summary
| Rank | Model | Quant | VRAM | Context | tok/s |
|---|---|---|---|---|---|
| 1 | Llama 3.1 8B | Q8_K_M | ~10.5 GB | 8K | ~35 |
| 2 | Qwen 2.5 14B | Q4_K_M | ~10 GB | 8K | ~18 |
| 3 | DeepSeek-Coder-V2-Lite | Q4_K_M | ~10 GB | 8K | ~16 |
The 8B Q8 option is best for general use. The 14B Q4 is best when you need more knowledge. DeepSeek-Coder is best for code.
Best models for 24GB+ VRAM
With 24 GB of VRAM (RTX 3090, RTX 4090, RX 7900 XTX), you can run serious models: 32B at Q4, 70B at Q3, or mixtures of experts like Mixtral. This is the enthusiast tier.
1. Llama 3.3 70B — Q3_K_M (or Q4_K_M with dual GPU)
VRAM usage: ~28 GB (Q3_K_M weights) + ~4 GB (4K context) = ~32 GB total (needs dual GPU or CPU offloading). Speed: ~8 tok/s on an RTX 4090 with Q3. Strengths: GPT-3.5-class performance on most benchmarks. Best-in-class local reasoning and knowledge. Trade-off: Does not fit on a single 24 GB GPU at Q4. Requires Q3 or CPU offloading. Slow generation.
2. Qwen 2.5 32B — Q4_K_M
VRAM usage: ~18 GB (weights) + ~3 GB (8K context) = ~21 GB total. Speed: ~12 tok/s on an RTX 4090. Strengths: The sweet spot for 24 GB GPUs. Fits comfortably with 8K+ context. Excellent coding, reasoning, and multilingual performance. Approaches GPT-4 on several benchmarks. Trade-off: Not as knowledgeable as 70B models, but faster and easier to run.
3. Mixtral 8x7B — Q4_K_M
VRAM usage: ~20 GB (weights) + ~3 GB (8K context) = ~23 GB total. Speed: ~15 tok/s on an RTX 4090. Strengths: Mixture-of-experts architecture gives 47B total parameters but only uses ~13B per token. High capability at lower effective compute. Great speed-to-quality ratio. Trade-off: Fills 24 GB completely — minimal room for context beyond 8K. Less capable than Qwen 32B on coding tasks.
24GB+ tier summary
| Rank | Model | Quant | VRAM | Context | tok/s |
|---|---|---|---|---|---|
| 1 | Qwen 2.5 32B | Q4_K_M | ~21 GB | 8K | ~12 |
| 2 | Mixtral 8x7B | Q4_K_M | ~23 GB | 8K | ~15 |
| 3 | Llama 3.3 70B | Q3_K_M | ~32 GB | 4K | ~8 |
Qwen 32B Q4 is the best single-GPU model for 24 GB cards. For dual-GPU setups, Llama 70B Q4 is the top choice. Use our model recommender for personalised suggestions.
Cross-tier comparison
How do the tiers stack up against each other? Here is a direct comparison of the top recommendation from each VRAM tier:
| Dimension | 8GB: Qwen 7B Q4 | 16GB: Llama 8B Q8 | 24GB: Qwen 32B Q4 |
|---|---|---|---|
| VRAM used | ~5.5 GB | ~10.5 GB | ~21 GB |
| Generation speed | ~25 tok/s | ~35 tok/s | ~12 tok/s |
| MMLU score | ~72% | ~74% | ~83% |
| Coding (HumanEval) | ~75% | ~77% | ~85% |
| Context | 8K | 8K | 8K |
| GPU cost | ~$200 used | ~$800 | ~$1600 |
The jump from 16 GB to 24 GB gives the biggest quality improvement — the 32B model scores ~10 points higher on MMLU and HumanEval. But it also costs 2x more on the GPU side.
Quantization trade-offs: large Q4 vs small Q8
A common dilemma: should you run a larger model at low quantization or a smaller model at high quantization?
- 14B Q4 (16 GB VRAM) vs 8B Q8 (16 GB VRAM): The 14B Q4 has more knowledge and better reasoning. The 8B Q8 has higher quality per token (less quantisation noise). For most tasks, the 14B Q4 wins. But for code generation, the 8B Q8 is close.
- 32B Q4 (24 GB VRAM) vs 14B Q8 (24 GB VRAM): The 32B Q4 beats the 14B Q8 on every benchmark. More parameters at Q4 outperform fewer parameters at Q8. Always choose the larger model at Q4 if it fits.
- 70B Q3 (32 GB VRAM) vs 32B Q8 (32 GB VRAM): Here the trade-off is real. The 70B Q3 has more knowledge, but the 32B Q8 has cleaner outputs. For creative writing, pick Q8. For factual Q&A, pick the 70B.
For a detailed comparison of all quantization levels, see our quantization calculator.
Not sure what fits your setup?
Use our free tools to find the perfect model, or skip the hardware entirely with Plugsky — 30+ models on a flat-rate plan.
Start Free → Model recommenderFrequently asked questions
What is the best local LLM for 8GB VRAM?
The three best models for 8GB VRAM are: Qwen 2.5 7B Q4_K_M (~4.5GB, ~25 tok/s), Llama 3.1 8B Q4_K_M (~5GB, ~22 tok/s), and Mistral 7B Q4_K_M (~4GB, ~28 tok/s). All three fit comfortably with room for 8K context.
What is the best local LLM for 16GB VRAM?
The three best models for 16GB VRAM are: Llama 3.1 8B Q8_K_M (~8.5GB, ~35 tok/s), Qwen 2.5 14B Q4_K_M (~8GB, ~18 tok/s), and DeepSeek-Coder-V2-Lite Q4_K_M (~8GB, ~16 tok/s). The 8B model in Q8 offers the best quality per token.
What is the best local LLM for 24GB VRAM?
The three best models for 24GB VRAM are: Llama 3.3 70B Q4_K_M (needs 48GB — use Q3_K_M at ~28GB), Qwen 2.5 32B Q4_K_M (~18GB, ~12 tok/s), and Mixtral 8x7B Q4_K_M (~20GB, ~15 tok/s). The 32B Qwen offers the best quality-to-speed ratio.
How much context can I use with these models?
On 8GB VRAM with a 7B Q4 model, use 4-8K context (2-3GB KV cache). On 16GB with 14B Q4, use 8-16K (2-4GB KV cache). On 24GB with 32B Q4, use 8K (4GB KV cache). Longer context means less room for model quality — balance based on your task.
Should I use a smaller model at higher quantization or a larger model at lower quantization?
Generally, a larger model at lower quantization (e.g., 14B Q4) outperforms a smaller model at higher quantization (e.g., 8B Q8) on reasoning and knowledge tasks. But for code generation, the 8B Q8 may match the 14B Q4. Use our model recommender to compare.