Local Inference Benchmark — July 2026

Methodology

All tests were run between July 8-10, 2026 on dedicated hardware with no other GPU workloads. Each engine-model-hardware combination was tested 50 times with different prompts of varying length (100-2000 input tokens). Results show the median of 50 runs after 5 warm-up iterations.

Benchmark results

All values are median of 50 runs. Tokens/s measures output generation throughput only (excludes TTFT). Peak VRAM includes model weights + KV cache for the specified context length. Latency p50 is end-to-end request time.

* Uses system RAM via unified memory, not dedicated VRAM. M4 runs 70B models via memory swapping with significant performance penalty.

Key takeaways from the benchmark data:

• vLLM is the fastest on GPU — 185 tok/s on RTX 4090 with Llama-3.1-8B, 44% faster than Ollama on the same hardware.
• llama.cpp leads on Apple Silicon — 52 tok/s on M4 vs 42 tok/s for Ollama, a 22% improvement.
• SGLang matches vLLM on throughput but adds structured output capabilities that neither Ollama nor llama.cpp offer natively.
• 70B models require server-grade hardware for usable performance. On a single RTX 4090, expect 5 tok/s — too slow for interactive use.
• Mixtral-8x7B is particularly challenging — its MoE architecture uses all 24GB of RTX 4090 VRAM, leaving no room for batch size or context.

Engine comparison

Ollama

Best for: Quick experimentation, simple setup, good CPU performance.

Ollama is the easiest local inference engine to install and use. A single binary, one command to pull a model, and you have an OpenAI-compatible API running on port 11434. It works on CPU, Apple Silicon, and NVIDIA GPUs. Performance is good but not class-leading — on GPU it trails vLLM by 30-45%. On CPU and Apple Silicon, it trails llama.cpp by 15-22%.

Strengths: Easiest setup, large model library via ollama pull, built-in modelfile system for customisation, works everywhere.

Weaknesses: Lowest throughput of the four, limited batching, no PagedAttention, no structured output support.

vLLM

Best for: Highest throughput, production workloads, requires GPU.

vLLM is the throughput king. Its PagedAttention mechanism manages KV cache as pages rather than contiguous blocks, eliminating memory fragmentation and enabling 2-4× larger batch sizes. On RTX 4090 with Llama-3.1-8B, it delivers 185 tok/s — 44% faster than Ollama. On multi-GPU setups with tensor parallelism, the gap widens further.

Strengths: Highest token throughput, continuous batching for concurrent requests, PagedAttention, tensor/pipeline parallelism, full OpenAI API surface including function calling.

Weaknesses: Requires NVIDIA GPU (CUDA), no CPU inference, more complex setup, higher VRAM overhead for small models.

llama.cpp

Best for: Apple Silicon, CPU optimisation, edge deployment.

llama.cpp is the most versatile engine for non-NVIDIA hardware. Its Metal backend on Apple Silicon delivers 52 tok/s on M4 with Llama-3.1-8B — outperforming Ollama by 22%. It supports all major quantisation formats (GGUF, IQ, Q4_K_M) and can run on anything from a Raspberry Pi to a 96-core server.

Strengths: Best CPU and Apple Silicon performance, broadest quantisation support, tiny binary size, runs on ARM/x86/Metal/CUDA/Vulkan.

Weaknesses: Lower GPU throughput than vLLM, no PagedAttention, server setup requires compilation, OpenAI API surface is limited.

SGLang

Best for: Complex reasoning, structured output, constrained decoding.

SGLang adds a radical idea: compile the generation graph ahead of time. This allows joint optimisation of the prompt processing and token generation phases. For structured output tasks (JSON mode, tool calling, grammar-constrained generation), SGLang is 1.5-3× faster than vLLM. Its RadixAttention caches prefix KV across requests for even higher throughput on repeated system prompts.

Strengths: Fastest for structured/constrained output, RadixAttention for prefix caching, ahead-of-time compilation, competitive raw throughput.

Weaknesses: Requires NVIDIA GPU, smaller community than vLLM, fewer pre-built models, steeper learning curve.

Hardware guide

Choosing the right hardware for local inference depends on your model size, latency requirements, and budget. Here is a practical guide based on our benchmark data:

Laptop tier (Apple Silicon): MacBooks with M4 chips are surprisingly capable. A 24GB M4 runs 7B models at 40-55 tok/s with llama.cpp — fast enough for interactive chat and most development workflows. 70B models are technically possible via memory swapping but run at 2-4 tok/s, which is too slow for practical use.

Desktop tier (RTX 4090): The RTX 4090 hits the sweet spot for local inference. Its 24GB GDDR6X runs all 7B models at over 128 tok/s with any engine. With vLLM, you get 185 tok/s on Llama-3.1-8B. Mixtral-8x7B fits but fills VRAM completely, limiting concurrency. 70B models run via Q4_K_M but at just 5 tok/s — usable for batch processing but not chat.

Server tier (A100/H100): Datacenter GPUs with NVLink enable tensor parallelism across multiple cards. Two A100-80GB cards run Llama-3.3-70B at 219 tok/s with vLLM — competitive with cloud API performance. The key advantage is large model support: with 8× H100 (640GB total VRAM), you can run Llama-4 (405B) locally.

Cost analysis

The economics of local vs cloud inference depend entirely on your token volume. We calculated total cost of ownership including hardware amortisation (3-year lifespan), electricity, and maintenance.

Local cost includes hardware amortised over 3 years ($1,600 M4 Mac mini, $3,000 RTX 4090 desktop, $30,000 2× A100 server) + electricity at $0.12/kWh. Plugsky API shown at Hobby/Starter pricing for Llama-3.1-8B equivalent. OpenAI API shown at GPT-4o-mini pricing.

Key findings:

• At low volume (1M tok/mo): Local inference appears cheaper ($0.44-5.20/MTok) because the hardware is a sunk cost. Cloud APIs cost $20-150/MTok. However, the upfront hardware investment ($1,600-30,000) must be considered.
• At medium volume (10M tok/mo): Local is dramatically cheaper ($0.44-3.50/MTok) for anyone who already owns the hardware. Cloud APIs at $10-150/MTok are hard to justify if you have a capable GPU sitting idle.
• At high volume (100M tok/mo): Local costs $0.30-0.53/MTok — approaching the marginal cost of electricity. Cloud APIs cost $2-150/MTok. At this scale, self-hosting pays for the hardware in 1-3 months.
• Break-even point: For an RTX 4090 system ($3,000 total), the break-even vs Plugsky API ($10/MTok at scale) is at ~2M tokens/month over 2 years. Vs OpenAI ($150/MTok), break-even is immediate.

When local beats cloud

Local inference has clear advantages in several scenarios:

• Privacy: Data never leaves your machine. No third party sees your prompts, completions, or embeddings. Essential for healthcare, legal, finance, and any workload with PII or trade secrets.
• Latency: Local GPU inference is 50-180ms p50 for 7B models vs 100-500ms for cloud APIs. The difference is noticeable in interactive chat — local feels instant.
• Offline operation: No internet required. Run your AI pipeline in air-gapped environments, during travel, or through network outages.
• Cost at high volume: Beyond 10M tokens/month, local inference costs $0.30-3.50/MTok compared to $2-150/MTok for cloud APIs. If you have sustained high throughput, local pays for itself.
• Development iteration: Zero-latency feedback during prompt engineering. No rate limits, no queueing, no API costs during experimentation.

When cloud beats local

Cloud APIs are the better choice in other situations:

• No hardware investment: Start building immediately without buying a GPU. $20/month gets you production inference with no upfront cost.
• Access to large models: Cloud APIs run 70B-405B models on multi-GPU backends. Running Llama-4 (405B) locally requires 8× H100 GPUs ($200K+). Cloud access costs pennies per request.
• Scalability: Cloud APIs auto-scale from 1 request to 10,000 concurrent requests. Local hardware has a fixed throughput ceiling.
• Team access: Cloud APIs provide API keys, usage tracking, team management, and SSO. Self-hosted requires DIY auth infrastructure.
• Multi-model access: Cloud APIs give you 30+ models with zero setup. Local requires downloading and managing each model separately.
• Uptime and reliability: Cloud APIs offer 99.9%+ SLA. Local inference goes down with your internet, power, or hardware failure.

Plugsky Bridge

You do not have to choose between local and cloud. The most productive setup uses both — develop locally with Ollama or vLLM, then deploy to Plugsky for production without changing your code.

Every local engine tested here exposes an OpenAI-compatible API. Code written against any of them works against Plugsky with just a base URL change:

# Local development — Ollama, vLLM, llama.cpp, or SGLang
client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama",
)

# Production — Plugsky, zero code changes beyond config
client = OpenAI(
    base_url="https://api.plugsky.com/v1",
    api_key="sk-live-PLUGSKY...",
)

# The rest of your code stays exactly the same
resp = client.chat.completions.create(
    model="plugsky-pro",
    messages=[{"role": "user", "content": "Hello!"}],
)

This pattern means you can:

• Develop locally with real models, real data, zero latency, and zero API cost.
• Test with your CI pipeline against a local vLLM instance for deterministic results.
• Deploy to Plugsky when you need production scale, team access, or models your hardware cannot run.
• Route by sensitivity — keep PII workloads local, route general traffic to the cloud.

import os
from openai import OpenAI

def get_client():
    """Route based on environment."""
    env = os.getenv("APP_ENV", "development")
    if env == "development":
        return OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
    return OpenAI(
        base_url="https://api.plugsky.com/v1",
        api_key=os.getenv("PLUGSKY_API_KEY"),
    )

def route_by_sensitivity(prompt: str):
    """Route sensitive prompts locally, others to Plugsky."""
    sensitive_keywords = ["ssn", "salary", "health", "password"]
    if any(kw in prompt.lower() for kw in sensitive_keywords):
        return OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
    return OpenAI(
        base_url="https://api.plugsky.com/v1",
        api_key=os.getenv("PLUGSKY_API_KEY"),
    )

Download the test data

The full benchmark dataset is available for download. It includes raw per-run results, system configurations, and the test harness used to generate these numbers.

• Full results (CSV): All 4,200 individual runs across every engine-model-hardware combination with per-iteration timing.
• Test harness (Python): The benchmarking script we used, including prompt generation, warm-up logic, and metric collection.
• Methodology document: Complete description of hardware configuration, software versions, environment variables, and test parameters.

Use these to verify our numbers on your own hardware, or to benchmark your own setup against our baselines.

Dataset last updated: July 10, 2026. License: CC BY 4.0.

Frequently asked questions