How Much VRAM Do You Need for a Local LLM?

Quick reference: VRAM by model size and quantization

Here is the most important table on this page. It shows how much VRAM each model size needs at different quantization levels. These are approximate — actual usage varies by implementation (llama.cpp, transformers, etc.) and context length.

Values above are for model weights only. Add 0.5-8 GB for the KV cache depending on context length (see below).

Use our interactive VRAM calculator to see exactly which models fit your GPU.

How VRAM works for LLMs

When you run a local LLM, VRAM is consumed by two things:

1. Model weights — the trained parameters stored as numbers. At 16-bit precision, each parameter takes 2 bytes. At 4-bit quantization, each parameter takes 0.5 bytes. A 7B model at 16-bit = 14 GB. At 4-bit = 3.5 GB.
2. KV cache — stores computed key-value pairs during generation so the model doesn't re-process the entire prompt on every token. Scales with context length.

The model must fit entirely in VRAM for GPU inference. If it doesn't, llama.cpp and other runners can offload layers to system RAM, but this slows generation dramatically (10-50x slower per offloaded layer).

Model weights by quantization level

Quantization reduces the precision of model weights, trading a small quality loss for a large memory reduction. Here is the formula:

VRAM for weights = number_of_parameters × bytes_per_parameter

FP16: 2 bytes/param   → 7B × 2 = 14 GB
Q8:   1 byte/param    → 7B × 1 = 7 GB
Q6:   0.75 bytes/param → 7B × 0.75 = 5.25 GB
Q4:   0.5 bytes/param  → 7B × 0.5 = 3.5 GB

In practice, add ~10% overhead for model architecture buffers, so a 7B Q4 model uses ~4 GB, not exactly 3.5 GB.

For a detailed breakdown of quantization levels, see our quantization calculator.

KV cache and context overhead

The KV cache is the second major VRAM consumer. It grows linearly with context length and model size.

Notice that at 32K context, the KV cache for a 70B model requires 64 GB — nearly twice the model weights themselves. For long-context work, the KV cache is often the bottleneck, not the model size.

Some runners support KV cache quantization (Q4 KV cache), which cuts this overhead by 4x with minimal quality impact. llama.cpp supports this via --cache-type-k q4_0.

Total VRAM: putting it together

Here are real-world total VRAM requirements for popular model + context combinations:

Concurrency overhead

If you plan to serve multiple users from one GPU, VRAM usage multiplies:

• Batch size 1: one KV cache per user at the full context length.
• Continuous batching: VRAM overhead per active sequence. For a 7B model at 4K context, each additional concurrent user needs ~1 GB of KV cache.
• vLLM / TGI: adds additional overhead for PagedAttention blocks and scheduler state.

As a rule of thumb, serving N concurrent users on a 7B Q4 model at 8K context needs roughly 5 GB + (2 GB × N). For production serving, double the single-user estimate.

Recommendations by use case

Light use (chat, summarisation)

8 GB VRAM — Qwen 2.5 7B Q4 or Llama 3.1 8B Q4. Up to 8K context. Perfect for single-user interactive chat, basic RAG, and document summarisation. Budget GPUs like the RTX 3060 12 GB or RTX 4060 work well.

Power use (coding, analysis)

16-24 GB VRAM — Qwen 2.5 14B Q4 or Llama 3.1 8B Q8. 16K+ context. Good for complex code generation, data analysis, and multi-turn conversations. An RTX 3090 24 GB or RTX 4070 Ti Super covers this tier comfortably.

Heavy use (large models, long context)

48 GB+ VRAM — Llama 3.3 70B Q4 or Qwen 2.5 32B Q8. Requires dual RTX 3090/4090 or a professional GPU like the A6000 48 GB. Best for enterprise workloads, batch processing, and tasks that need GPT-4-class quality.

Not sure what fits your setup? Use the VRAM calculator to check, or skip the hardware entirely with Plugsky — 30+ models on a flat-rate plan, no GPU required.

Frequently asked questions