Best local LLMs — tested models for local inference in 2026

Models by use case

Use case	Top pick	Alternative	Min hardware
General chat	Llama 3.2 8B (Q4)	Qwen 2.5 7B, Mistral 7B	8 GB RAM
Coding	DeepSeek-Coder-V2-Lite (Q4)	CodeLlama 13B, Qwen2.5-Coder 7B	12 GB RAM
Reasoning	Qwen 2.5 32B (Q4)	Llama 3.3 70B (Q3), Mixtral 8x7B	24 GB VRAM
RAG / retrieval	Mistral 7B (Q4_K_M)	Llama 3.2 8B, Nemo 12B	8 GB RAM
Arabic / bilingual	Jais 13B (Q4)	AceGPT 7B, Llama 3.2 8B (fine-tuned)	12 GB RAM
Agents / tool use	Llama 3.2 8B (Q4_K_M)	Qwen 2.5 7B, Command R 7B	8 GB RAM
Fast / low latency	Llama 3.2 3B (Q4)	Qwen 2.5 3B, Gemma 2 2B	4 GB RAM

Models by hardware tier

8 GB VRAM (budget GPU / Apple Silicon base)

Model	Quantization	VRAM used	Tokens/sec	Quality
Llama 3.2 3B	Q4_K_M	~2.5 GB	80-120	Good (light)
Qwen 2.5 3B	Q4_K_M	~2.3 GB	90-130	Good (light)
Llama 3.2 8B	Q4_K_M	~6.5 GB	30-50	Excellent
Mistral 7B	Q4_K_M	~5.8 GB	35-55	Excellent
Qwen 2.5 7B	Q4_K_M	~6.0 GB	30-50	Excellent
Gemma 2 9B	Q4_K_M	~7.0 GB	25-40	Very good

16 GB VRAM (mid-range GPU / M2 Pro)

Model	Quantization	VRAM used	Tokens/sec	Quality
Llama 3.2 8B	Q8_0	~9.0 GB	25-40	Excellent
Qwen 2.5 14B	Q4_K_M	~10 GB	15-25	Excellent
DeepSeek-Coder-V2-Lite	Q4_K_M	~12 GB	12-20	Excellent (coding)
Mixtral 8x7B	Q4_K_M	~14 GB	20-35	Very good
Llama 3.3 70B	Q2_K	~14 GB	8-12	Good

24 GB VRAM (RTX 3090/4090, M3 Max)

Model	Quantization	VRAM used	Tokens/sec	Quality
Qwen 2.5 32B	Q4_K_M	~20 GB	12-20	Excellent
Llama 3.3 70B	Q3_K_M	~20 GB	8-14	Very good
DeepSeek-Coder-V2	Q4_K_M	~22 GB	10-18	Excellent (coding)
Mixtral 8x22B	Q4_K_M	~22 GB	5-10	Excellent

48 GB+ VRAM (workstation, A6000, Mac Studio)

Model	Quantization	VRAM used	Tokens/sec	Quality
Llama 3.3 70B	Q4_K_M	~40 GB	15-25	Excellent
Qwen 2.5 72B	Q4_K_M	~45 GB	12-20	Excellent
DeepSeek-V3	Q4_K_M	~42 GB	10-18	Frontier
Command R+ 104B	Q3_K_M	~40 GB	6-10	Excellent (agents)

Quantization guide

Quantization reduces model precision to fit larger models on limited hardware. Lower precision = smaller file + faster inference + some quality loss. Higher precision = better quality + more VRAM required.

Format	Bits	File size (7B)	File size (70B)	Quality	Best for
Q2_K	2.6	~3 GB	~26 GB	Low	Fitting large models on limited VRAM
Q3_K_M	3.4	~4 GB	~34 GB	Medium	Balance for 70B+ models
Q4_K_M	4.5	~4.5 GB	~42 GB	Good	Best general-purpose sweet spot
Q5_K_M	5.3	~5.5 GB	~50 GB	Very good	Quality-sensitive applications
Q8_0	8.0	~7.5 GB	~70 GB	Near lossless	Maximum quality, plenty of VRAM
FP16	16.0	~14 GB	~140 GB	Lossless	Multi-GPU setups, training

Other quantization formats

• GGUF — llama.cpp format. Best for CPU + GPU hybrid inference. Widest model support. Use with Ollama, LM Studio, and llama.cpp directly.
• GPTQ — GPU-only format. Slightly faster than GGUF on pure GPU. Used by vLLM and AutoGPTQ.
• AWQ — GPU-only, higher quality than GPTQ at same bit width. Supported by vLLM and TGI.
• FP8 — Native to H100/H200 GPUs. Same quality as FP16 at half the memory. Best for NVIDIA Hopper.

Where to download

• Ollama Library — easiest. One command: ollama pull model-name. GGUF pre-configured with recommended quantization.
• Hugging Face — largest selection. Search by model name + GGUF/GPTQ/AWQ. The TheBloke user has thousands of pre-quantized models.
• bartowski — active quantizer on Hugging Face, frequent GGUF releases of new models.
• LM Studio Model Hub — in-app browser, one-click download.

Benchmark methodology

All benchmarks measured on a single RTX 4090 (24 GB VRAM) with Ubuntu 24.04, Ollama 0.5.x, and llama.cpp b3817. Tokens per second measured with a context window of 4096 tokens and temperature 0.7. Quality scores based on MMLU-Pro, HumanEval, and internal instruction-following tests. Results vary by hardware, driver version, and system load.

When local is not enough

Even the best local models can't match frontier models for complex reasoning, coding, or multilingual tasks. If you need:

• DeepSeek-V3, GPT-4 class, or Qwen 3 frontier models
• 99.9% uptime with zero hardware management
• Access to 30+ models without downloading 200 GB of weights
• Team access with API keys, budgets, and audit logs

Plugsky gives you all of that through the same OpenAI-compatible API you already use locally. Switch from localhost:11434/v1 to api.plugsky.com/v1 and get 30+ models — including frontier models that no local hardware can run.