Key facts
| Tool type | Free GPU and VRAM capacity estimator for inference |
| Inputs | Parameter count, quantization, context length, concurrency, GPU memory |
| Outputs | VRAM per replica, GPU count, utilization headroom |
| Memory components | Model weights, KV cache, activations and runtime overhead |
| Quantization support | Common bit widths from full precision to 4-bit |
| Companion tools | Local model recommender and quantization calculator |
| Managed alternative | 30+ models hosted behind one API, no GPU operations |
| Product status | Live |
TL;DR
- Weights are only the first term: KV cache scales with context and concurrency.
- Quantization cuts memory per weight but can change quality — validate on your evals.
- Plan for peak concurrency, not average, and keep 15-20 percent headroom.
- Multi-GPU adds communication overhead; not every model scales linearly.
- Compare owning GPUs with the hosted API before committing to hardware.
How it works, step by step
- Identify the candidate model and its parameter count from the model card.
- Choose a quantization level and confirm it is supported by your inference runtime.
- Measure worst-case context length and expected concurrent sessions.
- Open the GPU capacity calculator and enter model, quantization, context and concurrency.
- Read VRAM per replica and the recommended GPU count, noting headroom.
- Adjust quantization or context and re-run to see how memory changes.
- Compare the cost of that hardware profile with the hosted API on the pricing page.
Try it yourself
Open the GPU capacity calculator →
The memory math that matters
Inference memory has three main parts. Model weights dominate at rest: a 70-billion-parameter model needs roughly 140 GB in 16-bit precision, halving at 8-bit and halving again near 4-bit. The KV cache grows with context length and the number of concurrent sequences, and it is often the term teams forget. Runtime overhead — activations, framework buffers and fragmentation — adds a further percentage. A capacity calculator combines these so the estimate reflects a serving replica, not a single idle model. Measure actual serving memory in a pilot before committing to a hardware bill of materials.
Quantization is a trade, not a free win
Lower precision buys memory and sometimes throughput, but it can shift output quality, especially on reasoning and structured-output tasks. The right process is mechanical: pick a quantization that fits your hardware budget, then run your own eval set at that precision and compare against the full-precision baseline. If accuracy holds, keep it; if it does not, move down a model size rather than up in bits. Remember that KV cache can often be quantized separately, which helps long-context workloads without touching weight precision.
Owning GPUs versus using an API
Hardware only makes sense at sustained, predictable utilization. Below that, the fixed cost of GPUs plus operations, monitoring, upgrades and idle time usually exceeds a managed API. Run both numbers: the calculator gives you the hardware profile and the pricing page shows flat monthly self-serve plans for 30+ hosted models. A common compromise is hybrid — keep sensitive or high-volume workloads on owned GPUs, and burst everything else to the API through the same OpenAI-compatible interface.
Honest comparison
| Planning factor | What to enter | Why it changes the answer | Common mistake |
|---|---|---|---|
| Parameters | Exact parameter count of the candidate model | Weights dominate base memory | Using a similarly sized model's numbers |
| Quantization | 8-bit, 4-bit or full precision | Halves memory per step down | Assuming no quality change |
| Context length | Worst-case tokens per request | KV cache scales with context | Sizing for average prompts |
| Concurrency | Peak simultaneous sessions | Each sequence adds cache | Planning for average load |
| Runtime overhead | Framework buffers and fragmentation | Adds headroom above the math | Packing GPUs to 100 percent |
| GPU memory | Memory per card you plan to buy or rent | Determines GPUs per replica | Ignoring multi-GPU overhead |
Frequently asked questions
How much VRAM does a model need?
It depends on parameter count, quantization, context length and concurrency. The calculator combines all four; weights alone are never the full answer for a serving system.
What is the KV cache?
The memory used to store attention keys and values for tokens in the context. It grows with context length and the number of concurrent sequences, and it is often the largest variable term.
Does 4-bit quantization always work?
No. It reduces memory but can change output quality. Validate on your own eval set at that precision before committing.
How much headroom should I leave?
Keep roughly 15-20 percent above the calculated requirement for fragmentation, bursts and framework overhead.
Can one GPU serve multiple models?
Sometimes, if their combined weights and caches fit with headroom. Otherwise route requests to separate replicas and load-balance.
Is multi-GPU always faster?
No. Communication overhead grows with GPU count, and small models may lose throughput. Measure per-replica latency before scaling out.
When is an API cheaper than GPUs?
At low or spiky utilization, almost always. Compare your hardware profile with hosted pricing before buying.
Does Plugsky host these models?
Yes. 30+ models are available behind one OpenAI-compatible API, so you can skip GPU operations entirely and see the live pricing page for plans.