Free tool · no sign-up

GPU memory bandwidth calculator: tokens/sec for 30 GPUs

Estimate LLM decode throughput from memory bandwidth, model size, quantization, batch and KV cache. Compare 30 GPUs and find the batch that actually fits.

Model and GPU

VRAM calculator

KV GB per 1K tokens is for FP16 cache. Halve it for FP8. MoE models use total parameters for weight memory; active parameters affect compute, not the weight read.

Result fits

Enter model and GPU details to estimate decode throughput.
Weights—
KV cache—
Total memory—
Bandwidth—
Decode tok/s—
Per-request tok/s—
Max batch @ context—
GB/s per watt—

Roofline estimate. Real kernels, paged attention and tensor-parallel communication will move the number. Use it to compare hardware, not to promise a benchmark.

Why bandwidth sets the decode ceiling

Decode generates one token per sequence per step. To do that, the GPU reads the model weights and the KV cache for every active sequence, then performs a relatively small amount of arithmetic. The ratio of compute to bytes is low, so the memory bus is the constraint. The formula is:

decode_tok_s = batch × bandwidth_GB_s × efficiency
               ÷ (weights_GB + batch × kv_GB_per_sequence)

kv_GB_per_sequence = kv_GB_per_1k_tokens × context_tokens ÷ 1000

At batch 1, the denominator is weights plus one KV cache. At batch 32, the weights are still read once, but 32 tokens are produced, so throughput rises until the KV term dominates. That is why batch size is the cheapest throughput lever — until memory runs out.

30 GPUs compared

The table applies the formula to each GPU's published memory bandwidth at 70% efficiency, with 4-bit weights and an 8,192-token context. The 70B column shows the maximum batch that fits at 4-bit weights plus KV cache on a single GPU; “No” means the weights alone exceed the 90% memory budget.

8B Q4 tokens/sec at batch 1 and 32, 8K context, 70% bandwidth efficiency. 70B max batch is for Q4 weights plus FP16 KV cache on one GPU.
GPUMem GBBW GB/sTDP WGB/s/W8B Q4 b=18B Q4 b=3270B Q4 max batch70B fits
NVIDIA H100 SXM803,3507004.79463199811Yes
NVIDIA H100 PCIe802,0003505.71277119311Yes
NVIDIA H200 SXM1414,8007006.86664286330Yes
NVIDIA B200 SXM1928,00010008.001106477246Yes
NVIDIA A100 SXM 80GB802,0394005.10282121611Yes
NVIDIA A100 PCIe 80GB801,9353006.45268115411Yes
NVIDIA A100 SXM 40GB401,5554003.892159280Yes
NVIDIA A30249331655.651295570No
NVIDIA A10246001504.00833580No
NVIDIA A10G246001504.00833580No
NVIDIA L40488643002.881195151Yes
NVIDIA L40S488643502.471195151Yes
NVIDIA L424300724.17411790No
NVIDIA T416320704.57441910No
NVIDIA V100 SXM2 32GB329003003.001245370No
NVIDIA V100 PCIe 32GB329002503.601245370No
NVIDIA RTX 4090241,0084502.241396010No
NVIDIA RTX 4080 Super167363202.301024390No
NVIDIA RTX 3090249363502.671295580No
NVIDIA RTX 3080107603202.381054530No
NVIDIA RTX 5090321,7925753.1224810690No
NVIDIA RTX 5080169603602.671335730No
NVIDIA RTX 5070 Ti168963002.991245340No
NVIDIA RTX A6000487683002.561064581Yes
NVIDIA RTX 6000 Ada489603003.201335731Yes
NVIDIA RTX 5000 Ada325762502.30803440No
AMD Instinct MI300X1925,3007507.07733316246Yes
AMD Instinct MI250X1283,2005605.71442190926Yes
AMD Instinct MI210641,6003005.332219546Yes
Intel Gaudi 31283,7006006.17512220726Yes

GPU memory, bandwidth and TDP are vendor datasheet figures. Tokens/sec are model outputs, not measured benchmarks. The 70B max batch assumes 8K context and 10% memory overhead.

Quantization changes capacity, not bandwidth

Lower precision reduces the bytes per parameter, so the weights are smaller and the decode ceiling rises. It does not increase the GPU's memory bandwidth. The table below shows the weight footprint at three precisions and the minimum number of 80 GB GPUs at 4-bit, with 10% headroom for overhead.

Weight memory and 80 GB GPU count. KV cache is additional; max batch is for one 80 GB GPU at 8K context with 10% overhead.
ModelFP16 GB8-bit GB4-bit GB80 GB GPUs at Q4Max batch on one 80 GB
7B14.57.23.6165
13B26.013.06.5149
32B65.632.816.4126
70B141.270.635.3113
120B234.0117.058.5122
405B810.0405.0202.530

Quantization is not free. Aggressive 2-bit or 3-bit formats can degrade quality, and some formats require calibration data. The right question is not “what is the smallest file?” but “what is the smallest file that still passes my evaluation?”. A 4-bit model that fits one GPU is cheaper than a 2-bit model that needs a second eval cycle.

Batch and KV cache

Batch size raises aggregate throughput because the weights are amortized across more sequences. The gain is steep at first and then flattens as the KV cache term grows. At the same time, per-request latency rises: each request shares the decode step with more sequences. The calculator reports both aggregate and per-request tokens per second so you can see the trade-off.

Context length multiplies the KV cache. Doubling context from 8K to 16K doubles the cache per sequence and roughly halves the batch that fits. If your workload has a long tail of 32K-token requests, size for the tail, not the mean. Paged attention and KV quantization can reduce the waste, but they do not repeal the memory budget.

Multi-GPU scaling

Tensor parallelism splits the weights and the KV cache across GPUs, and aggregate bandwidth rises with the GPU count. It does not rise perfectly: every layer needs an all-reduce or all-gather, and interconnect latency eats into the gain. The calculator's multi-GPU efficiency field defaults to 85% and applies to the aggregate bandwidth. For NVLink systems, 85–90% is a reasonable planning range; for PCIe or Ethernet, it can be lower.

A second effect matters for memory: tensor parallelism replicates some activations and buffers, and the KV cache is partitioned by attention heads. The simple count × memory model is optimistic at small batch and long context. Measure the actual memory before you buy the GPUs.

Methodology and limits

The calculator uses vendor memory bandwidth, memory capacity and TDP, plus the model size, quantization, batch, context and KV-cache inputs you provide. It assumes dense decode, one sequence length, ideal memory coalescing and no prefix caching, speculative decoding or scheduler overhead. It does not model prefill, which is compute-bound and covered by the throughput calculator.

For local deployment, pair this with the VRAM calculator and the GPU capacity calculator. For hosted capacity, see GPU rental options and the local AI guide.

FAQ

Why does memory bandwidth limit tokens per second?

Each decoded token requires reading the model weights and the KV cache for every active sequence. The GPU performs relatively little arithmetic per byte read, so the memory bus, not the compute units, sets the ceiling. A GPU with 3.35 TB/s and a 4 GB 4-bit model has a theoretical ceiling near 800 tokens per second at batch 1 before efficiency losses and KV cache traffic.

Does 4-bit quantization make the model four times faster?

It makes the weights roughly four times smaller than FP16, so the bandwidth-bound decode ceiling rises. It does not make the compute units four times faster, and it does not shrink the KV cache unless you also quantize the cache. In practice, 4-bit weights improve tokens per second and reduce the GPU count, but the gain is rarely the full 4×.

How much KV cache do I need?

For a grouped-query model in FP16, the cache is about 2 × layers × KV heads × head dimension × 2 bytes per token. An 8B model with 32 layers, 8 KV heads and 128 head dimension uses about 131 KB per token, or 1.07 GB for an 8,192-token sequence. A 70B model with 80 layers uses about 328 KB per token, or 2.7 GB per sequence.

Why does my GPU run out of memory before the weights are full?

The KV cache grows with batch size and context length. At 8K context and batch 32, a 70B model needs about 86 GB of KV cache on top of 35 GB of 4-bit weights. An 80 GB GPU cannot hold that. The calculator reports the maximum batch at the current context, which is usually the number that matters.

Is the tokens-per-second figure a benchmark?

No. It is a roofline estimate: bandwidth × efficiency divided by bytes read. It assumes ideal memory coalescing, one sequence length and no tensor-parallel communication cost. Real serving stacks add paged attention, continuous batching and kernel overhead. Use the estimate to compare hardware and size memory, then measure on your workload.

Sources and verification

Primary sources were opened and checked on 20 November 2026. Product terms and datasets change; the linked source is authoritative.

Skip the GPU shopping list

Plugsky serves hosted models through one OpenAI-compatible API. You get published plan caps instead of a memory budget, and the free tier starts at 100 requests and 50K tokens per day.

See plans → Read the local AI guide