GPU memory bandwidth calculator: tokens/sec for 30 GPUs
Estimate LLM decode throughput from memory bandwidth, model size, quantization, batch and KV cache. Compare 30 GPUs and find the batch that actually fits.
Model and GPU
KV GB per 1K tokens is for FP16 cache. Halve it for FP8. MoE models use total parameters for weight memory; active parameters affect compute, not the weight read.
Result fits
Roofline estimate. Real kernels, paged attention and tensor-parallel communication will move the number. Use it to compare hardware, not to promise a benchmark.
Why bandwidth sets the decode ceiling
Decode generates one token per sequence per step. To do that, the GPU reads the model weights and the KV cache for every active sequence, then performs a relatively small amount of arithmetic. The ratio of compute to bytes is low, so the memory bus is the constraint. The formula is:
decode_tok_s = batch × bandwidth_GB_s × efficiency
÷ (weights_GB + batch × kv_GB_per_sequence)
kv_GB_per_sequence = kv_GB_per_1k_tokens × context_tokens ÷ 1000
At batch 1, the denominator is weights plus one KV cache. At batch 32, the weights are still read once, but 32 tokens are produced, so throughput rises until the KV term dominates. That is why batch size is the cheapest throughput lever — until memory runs out.
30 GPUs compared
The table applies the formula to each GPU's published memory bandwidth at 70% efficiency, with 4-bit weights and an 8,192-token context. The 70B column shows the maximum batch that fits at 4-bit weights plus KV cache on a single GPU; “No” means the weights alone exceed the 90% memory budget.
| GPU | Mem GB | BW GB/s | TDP W | GB/s/W | 8B Q4 b=1 | 8B Q4 b=32 | 70B Q4 max batch | 70B fits |
|---|---|---|---|---|---|---|---|---|
| NVIDIA H100 SXM | 80 | 3,350 | 700 | 4.79 | 463 | 1998 | 11 | Yes |
| NVIDIA H100 PCIe | 80 | 2,000 | 350 | 5.71 | 277 | 1193 | 11 | Yes |
| NVIDIA H200 SXM | 141 | 4,800 | 700 | 6.86 | 664 | 2863 | 30 | Yes |
| NVIDIA B200 SXM | 192 | 8,000 | 1000 | 8.00 | 1106 | 4772 | 46 | Yes |
| NVIDIA A100 SXM 80GB | 80 | 2,039 | 400 | 5.10 | 282 | 1216 | 11 | Yes |
| NVIDIA A100 PCIe 80GB | 80 | 1,935 | 300 | 6.45 | 268 | 1154 | 11 | Yes |
| NVIDIA A100 SXM 40GB | 40 | 1,555 | 400 | 3.89 | 215 | 928 | 0 | Yes |
| NVIDIA A30 | 24 | 933 | 165 | 5.65 | 129 | 557 | 0 | No |
| NVIDIA A10 | 24 | 600 | 150 | 4.00 | 83 | 358 | 0 | No |
| NVIDIA A10G | 24 | 600 | 150 | 4.00 | 83 | 358 | 0 | No |
| NVIDIA L40 | 48 | 864 | 300 | 2.88 | 119 | 515 | 1 | Yes |
| NVIDIA L40S | 48 | 864 | 350 | 2.47 | 119 | 515 | 1 | Yes |
| NVIDIA L4 | 24 | 300 | 72 | 4.17 | 41 | 179 | 0 | No |
| NVIDIA T4 | 16 | 320 | 70 | 4.57 | 44 | 191 | 0 | No |
| NVIDIA V100 SXM2 32GB | 32 | 900 | 300 | 3.00 | 124 | 537 | 0 | No |
| NVIDIA V100 PCIe 32GB | 32 | 900 | 250 | 3.60 | 124 | 537 | 0 | No |
| NVIDIA RTX 4090 | 24 | 1,008 | 450 | 2.24 | 139 | 601 | 0 | No |
| NVIDIA RTX 4080 Super | 16 | 736 | 320 | 2.30 | 102 | 439 | 0 | No |
| NVIDIA RTX 3090 | 24 | 936 | 350 | 2.67 | 129 | 558 | 0 | No |
| NVIDIA RTX 3080 | 10 | 760 | 320 | 2.38 | 105 | 453 | 0 | No |
| NVIDIA RTX 5090 | 32 | 1,792 | 575 | 3.12 | 248 | 1069 | 0 | No |
| NVIDIA RTX 5080 | 16 | 960 | 360 | 2.67 | 133 | 573 | 0 | No |
| NVIDIA RTX 5070 Ti | 16 | 896 | 300 | 2.99 | 124 | 534 | 0 | No |
| NVIDIA RTX A6000 | 48 | 768 | 300 | 2.56 | 106 | 458 | 1 | Yes |
| NVIDIA RTX 6000 Ada | 48 | 960 | 300 | 3.20 | 133 | 573 | 1 | Yes |
| NVIDIA RTX 5000 Ada | 32 | 576 | 250 | 2.30 | 80 | 344 | 0 | No |
| AMD Instinct MI300X | 192 | 5,300 | 750 | 7.07 | 733 | 3162 | 46 | Yes |
| AMD Instinct MI250X | 128 | 3,200 | 560 | 5.71 | 442 | 1909 | 26 | Yes |
| AMD Instinct MI210 | 64 | 1,600 | 300 | 5.33 | 221 | 954 | 6 | Yes |
| Intel Gaudi 3 | 128 | 3,700 | 600 | 6.17 | 512 | 2207 | 26 | Yes |
GPU memory, bandwidth and TDP are vendor datasheet figures. Tokens/sec are model outputs, not measured benchmarks. The 70B max batch assumes 8K context and 10% memory overhead.
Quantization changes capacity, not bandwidth
Lower precision reduces the bytes per parameter, so the weights are smaller and the decode ceiling rises. It does not increase the GPU's memory bandwidth. The table below shows the weight footprint at three precisions and the minimum number of 80 GB GPUs at 4-bit, with 10% headroom for overhead.
| Model | FP16 GB | 8-bit GB | 4-bit GB | 80 GB GPUs at Q4 | Max batch on one 80 GB |
|---|---|---|---|---|---|
| 7B | 14.5 | 7.2 | 3.6 | 1 | 65 |
| 13B | 26.0 | 13.0 | 6.5 | 1 | 49 |
| 32B | 65.6 | 32.8 | 16.4 | 1 | 26 |
| 70B | 141.2 | 70.6 | 35.3 | 1 | 13 |
| 120B | 234.0 | 117.0 | 58.5 | 1 | 22 |
| 405B | 810.0 | 405.0 | 202.5 | 3 | 0 |
Quantization is not free. Aggressive 2-bit or 3-bit formats can degrade quality, and some formats require calibration data. The right question is not “what is the smallest file?” but “what is the smallest file that still passes my evaluation?”. A 4-bit model that fits one GPU is cheaper than a 2-bit model that needs a second eval cycle.
Batch and KV cache
Batch size raises aggregate throughput because the weights are amortized across more sequences. The gain is steep at first and then flattens as the KV cache term grows. At the same time, per-request latency rises: each request shares the decode step with more sequences. The calculator reports both aggregate and per-request tokens per second so you can see the trade-off.
Context length multiplies the KV cache. Doubling context from 8K to 16K doubles the cache per sequence and roughly halves the batch that fits. If your workload has a long tail of 32K-token requests, size for the tail, not the mean. Paged attention and KV quantization can reduce the waste, but they do not repeal the memory budget.
Multi-GPU scaling
Tensor parallelism splits the weights and the KV cache across GPUs, and aggregate bandwidth rises with the GPU count. It does not rise perfectly: every layer needs an all-reduce or all-gather, and interconnect latency eats into the gain. The calculator's multi-GPU efficiency field defaults to 85% and applies to the aggregate bandwidth. For NVLink systems, 85–90% is a reasonable planning range; for PCIe or Ethernet, it can be lower.
A second effect matters for memory: tensor parallelism replicates some activations and buffers, and the KV cache is partitioned by attention heads. The simple count × memory model is optimistic at small batch and long context. Measure the actual memory before you buy the GPUs.
Methodology and limits
The calculator uses vendor memory bandwidth, memory capacity and TDP, plus the model size, quantization, batch, context and KV-cache inputs you provide. It assumes dense decode, one sequence length, ideal memory coalescing and no prefix caching, speculative decoding or scheduler overhead. It does not model prefill, which is compute-bound and covered by the throughput calculator.
For local deployment, pair this with the VRAM calculator and the GPU capacity calculator. For hosted capacity, see GPU rental options and the local AI guide.
FAQ
Why does memory bandwidth limit tokens per second?
Each decoded token requires reading the model weights and the KV cache for every active sequence. The GPU performs relatively little arithmetic per byte read, so the memory bus, not the compute units, sets the ceiling. A GPU with 3.35 TB/s and a 4 GB 4-bit model has a theoretical ceiling near 800 tokens per second at batch 1 before efficiency losses and KV cache traffic.
Does 4-bit quantization make the model four times faster?
It makes the weights roughly four times smaller than FP16, so the bandwidth-bound decode ceiling rises. It does not make the compute units four times faster, and it does not shrink the KV cache unless you also quantize the cache. In practice, 4-bit weights improve tokens per second and reduce the GPU count, but the gain is rarely the full 4×.
How much KV cache do I need?
For a grouped-query model in FP16, the cache is about 2 × layers × KV heads × head dimension × 2 bytes per token. An 8B model with 32 layers, 8 KV heads and 128 head dimension uses about 131 KB per token, or 1.07 GB for an 8,192-token sequence. A 70B model with 80 layers uses about 328 KB per token, or 2.7 GB per sequence.
Why does my GPU run out of memory before the weights are full?
The KV cache grows with batch size and context length. At 8K context and batch 32, a 70B model needs about 86 GB of KV cache on top of 35 GB of 4-bit weights. An 80 GB GPU cannot hold that. The calculator reports the maximum batch at the current context, which is usually the number that matters.
Is the tokens-per-second figure a benchmark?
No. It is a roofline estimate: bandwidth × efficiency divided by bytes read. It assumes ideal memory coalescing, one sequence length and no tensor-parallel communication cost. Real serving stacks add paged attention, continuous batching and kernel overhead. Use the estimate to compare hardware and size memory, then measure on your workload.
Sources and verification
Primary sources were opened and checked on 20 November 2026. Product terms and datasets change; the linked source is authoritative.
- NVIDIA H100 datasheet — H100 memory bandwidth, capacity and TDP checked 20 November 2026
- NVIDIA H200 product page — H200 141 GB HBM3e and 4.8 TB/s bandwidth checked 20 November 2026
- AMD Instinct MI300X specifications — MI300X 192 GB HBM3 and 5.3 TB/s bandwidth checked 20 November 2026
- Intel Gaudi 3 product brief — Gaudi 3 128 GB HBM2e and 3.7 TB/s bandwidth checked 20 November 2026
Skip the GPU shopping list
Plugsky serves hosted models through one OpenAI-compatible API. You get published plan caps instead of a memory budget, and the free tier starts at 100 requests and 50K tokens per day.
See plans → Read the local AI guide