What is a Local LLM?

A local LLM is a large language model that runs entirely on your hardware — your laptop, workstation, or on-premises server. Unlike ChatGPT or the OpenAI API, where your prompts are processed on shared cloud infrastructure, a local LLM keeps everything inside your network. The model weights live on your disk, inference consumes your GPU, and your data never reaches a third party.

A large language model (LLM) is a neural network trained on massive text datasets to predict and generate human-like text. When you run one locally, you download the model weights (the trained parameters) to your machine and perform inference — the process of generating text — using your own compute resources.

The key distinction is where inference happens. In the cloud, inference happens on the provider's GPU. Locally, it happens on your GPU. The model itself is the same either way — what changes is control, privacy, and cost structure.

How local LLMs differ from cloud APIs

Dimension	Local LLM	Cloud API
Where inference runs	Your GPU / CPU	Provider's GPU cluster
Data handling	Never leaves your machine	Sent to provider's servers
Pricing model	Upfront hardware + electricity	Per-token or monthly
Internet required	No (after download)	Yes
Rate limits	None (limited by hardware)	Requests per minute
Model choice	Any open-weight model	Provider's model catalogue
Setup time	10-60 minutes	Instant via API key

Model sizes explained

LLMs are measured by parameter count — roughly, the number of weights in the neural network. More parameters generally means more capability but requires more memory and compute.

1B-3B parameters (Phi-3 mini, Llama 3.2 3B, Qwen 2.5 1.5B): Run on CPU, phone, or weak GPU. Good for simple tasks, classification, and chat.
7B-8B parameters (Llama 3 8B, Mistral 7B, Qwen 2.5 7B): The sweet spot for consumer hardware. Needs 8GB+ VRAM at 4-bit. Handles most tasks well.
13B-14B parameters (Qwen 2.5 14B, Llama 2 13B): Requires 16-24GB VRAM. Strong performance approaching GPT-3.5 levels.
30B-34B parameters (Yi 34B, Qwen 2.5 32B): Requires 24-48GB VRAM. Excellent quality, best on workstation GPUs.
70B-72B parameters (Llama 3 70B, Qwen 2.5 72B): Needs 48-80GB VRAM. Top-tier local performance, rivals GPT-3.5-turbo and approaches GPT-4 on some benchmarks.

Hardware requirements by model size

Model size	Minimum VRAM (4-bit)	Recommended GPU	CPU fallback
1B-3B	2-4 GB	Any GPU or Apple Silicon	Yes, runs well
7B-8B	6-8 GB	RTX 3060 12GB / RTX 4060	Slow but works
13B-14B	10-14 GB	RTX 3090 / 4070 Ti Super	Very slow
30B-34B	20-28 GB	RTX 3090 24GB / RTX 4090	Not practical
70B-72B	40-48 GB	2x RTX 3090 / A6000 48GB	Not practical

Check our VRAM calculator to see exactly which models fit your GPU configuration.

Popular local models

Llama 3.2 (Meta) — Available in 1B, 3B, and 8B. The most widely used local family. Excellent instruction following and reasoning for its size.
Mistral 7B / Mixtral 8x7B (Mistral AI) — Strong performance per parameter. Mixtral uses a mixture-of-experts architecture for higher capability at lower compute.
Qwen 2.5 (Alibaba) — Available in 0.5B to 72B. Top-tier performance across the board, especially strong at coding and maths.
DeepSeek Coder V2 (DeepSeek) — Specialised for code generation. Matches GPT-4 on coding benchmarks at a fraction of the size.
Phi-3 (Microsoft) — Miniature model (3.8B) that punches above its weight. Runs on phones and laptops with ease.

What is quantisation?

Quantisation shrinks a model by reducing the precision of its weights. A full-precision model uses 16-bit floating point numbers. A 4-bit quantised version uses 4-bit integers, reducing memory by roughly 75% while retaining 95-99% of the quality.

For example, Llama 3 70B at 16-bit needs 140GB of VRAM — impossible for consumer hardware. At 4-bit it needs ~40GB, fitting on a single A6000 or two RTX 3090s. This trade-off is the reason local LLMs are practical at all.

Common quantisation formats include GGUF (llama.cpp), AWQ, and GPTQ. Ollama handles this automatically — when you type ollama pull llama3.2:3b, it downloads the optimal quantised version for your hardware.

How to run a local LLM

The simplest way:

bash

# Install Ollama (macOS, Linux, Windows)
curl -fsSL https://ollama.com/install.sh | sh

# Download and run a model
ollama run qwen2.5:7b

This downloads the model, quantises it if needed, and starts an interactive chat session. Ollama also exposes an API at http://localhost:11434/v1 that is compatible with the OpenAI SDK — just change the base_url.

Local vs cloud quality comparison

The quality gap between local and cloud LLMs has narrowed significantly in 2025-2026. On standard benchmarks:

Qwen 2.5 72B matches GPT-4 on MMLU (knowledge) and HumanEval (coding).
Llama 3 70B trails GPT-4 by ~5% on most reasoning benchmarks.
Mistral 7B matches GPT-3.5-turbo on many tasks at 1/50th the size.
All local models trail GPT-4o and Claude 3.5 on long context, creative writing, and nuanced instruction following.

For 80% of everyday tasks — summarisation, coding, Q&A, brainstorming — a well-chosen local model is indistinguishable from a cloud API. The edge cases are complex reasoning, very long documents, and tasks requiring the absolute frontier of AI capability.

Use our local model recommender to find the best model for your hardware and use case, or skip the hardware entirely with Plugsky for 30+ models on a flat-rate plan.

No GPU? No problem

Plugsky gives you 30+ models including Llama, Qwen, Mistral, and DeepSeek. No hardware, no setup, no rate limits.