Free tool · no sign-up

LLM throughput and capacity calculator: batch, KV cache, concurrency and cost

Estimate decode tokens per second, KV cache, maximum batch, queue latency and cost per million tokens from model size, GPU bandwidth and context length.

Model and hardware

Advanced: memory overhead and KV override

Use the override for models with multi-head latent attention (MLA) or a custom cache layout, where the standard keys-and-values formula overestimates memory.

Rate limit calculator

All numbers are estimates. The model assumes one sequence length, no prefix caching and no speculative decoding. Measure before you commit to a GPU count.

Result fits

Enter model and hardware details to estimate throughput.
Decode tokens/sec—
Per-request tok/s—
KV cache / sequence—
Total memory—
Max batch at context—
Request latency—
Capacity (req/s)—
Queue wait—
Cost / 1M output—
Cost / 1M total—

Roofline arithmetic. It does not model kernel efficiency, tensor-parallel interconnect, scheduler overhead or model quality.

Capacity tables

Batch scaling

Decode throughput, KV memory and cost as batch size changes at the current context length and GPU count.
BatchKV cache GBTotal memory GBDecode tok/sPer-request tok/s$ / 1M outputFits

Arrival rate and queue latency

Queue wait at fractions of the estimated request capacity. The approximation is M/M/1-style; it is directional, not a guarantee.
Arrival req/sUtilizationQueue waitLatency incl. queueState

The roofline model behind the calculator

LLM inference has two phases with different bottlenecks. Prefill processes the prompt in parallel. It is a large matrix multiplication and is limited by compute. Decode generates one token per sequence per step. It reads the weights and the KV cache for every active sequence, so it is limited by memory bandwidth. A capacity plan that uses only FLOPS will overestimate decode throughput by a wide margin.

weights_GB = params_B × bytes_per_weight
kv_GB_per_token = 2 × layers × kv_heads × head_dim × kv_bytes / 1e9
decode_tok_s = batch × bandwidth_GB_s × efficiency / (weights_GB + batch × kv_GB_per_seq)
prefill_tok_s = gpu_tflops × 1e12 × mfu / (2 × params_B × 1e9)

The formula is deliberately simple. It assumes the weights are read once per decode step and the KV cache is read once per sequence. Real kernels may fuse operations, use paged attention, cache prefixes or quantize the KV cache. Those changes can improve the result; they rarely make the bandwidth term disappear.

Decode is bandwidth-bound; prefill is compute-bound

At batch 1, decode reads the entire model to produce one token. A 70B model in FP16 is about 141 GB; at 3.35 TB/s and 70% efficiency, the ceiling is roughly 16 tokens per second. That is close to the experience of running a large model on a single GPU. At batch 32, the weights are still read once, but 32 tokens are produced, so aggregate throughput rises sharply until the KV cache traffic dominates.

Prefill is the opposite. A 2,000-token prompt on a 70B model needs about 2.8 × 1014 FLOPs. On one H100 at 35% MFU, that is under a second; on eight GPUs, it is a fraction of a second. This is why time-to-first-token is usually a prefill and queueing problem, while tokens-per-second is a decode and bandwidth problem.

KV cache is the hidden capacity limit

Every active sequence stores keys and values for every layer. For a 70B model with 80 layers, 8 KV heads, head dimension 128 and FP16 cache, one token costs about 327 KB. An 8,192-token sequence costs about 2.7 GB. Thirty-two concurrent sequences cost about 86 GB before weights. At 128 concurrent sequences, the cache alone is 343 GB, which is why long-context serving needs more GPUs than the weight size suggests.

Grouped-query attention reduced the KV cache by sharing key/value heads, and multi-head latent attention compresses it further. The calculator's override field exists for those architectures. If you enter the vendor's bytes-per-token figure, the memory estimate is more useful than the generic formula.

Concurrency, queueing and latency

Throughput and latency trade off. Increasing batch size raises aggregate tokens per second and lowers cost per token, but each request waits longer because it shares the decode step with more sequences. The calculator reports both aggregate throughput and per-request tokens per second, then estimates request latency as prefill time plus output tokens divided by the per-request decode rate.

Queueing adds a second delay. If requests arrive faster than the batch can finish them, the queue grows without bound. The arrival table shows utilization and an M/M/1-style wait; at 95% utilization the wait is long and unstable. A production service should plan for 60–80% utilization, not 95%, because traffic is bursty and retries amplify load.

Cost per million tokens

The cost KPI divides GPU-hours by generated tokens. At 1,000 aggregate decode tokens per second on one $2.50/hour GPU, the output cost is about $0.69 per million tokens before utilization and failure overhead. At 100 tokens per second, it is about $6.94. The difference is throughput, not the hourly price.

Cost per total token includes prefill. For a workload with a long prompt and a short answer, prefill can dominate; for a chat workload with short prompts and long answers, decode dominates. Enter your own prompt and output sizes rather than using a generic 1:1 ratio.

Worked example: 70B on eight H100s

With 70.6B parameters in FP16, 80 layers, 8 KV heads, head dimension 128, an 8,192-token context, batch 32 and 70% memory efficiency, the calculator estimates about 141 GB of weights, 86 GB of KV cache, 2,646 decode tokens per second aggregate, 83 tokens per second per request, and about $2.10 per million output tokens at $2.50/GPU-hour. The memory fits in 8 × 80 GB with room for overhead. Change the context to 32,768 and the KV cache roughly quadruples; the batch that fit at 8K may no longer fit.

These are model outputs, not measured benchmarks. The purpose is to show which term moves the answer. In this example, weights and KV cache are the same order of magnitude, which is why long-context serving is a memory problem as much as a compute problem.

Methodology and limits

The calculator uses published GPU memory bandwidth, memory capacity and dense FP16 throughput, plus the model dimensions you enter. It assumes dense transformer decoding, one sequence length, no prefix caching, no speculative decoding and no tensor-parallel communication cost. It does not model kernel fusion, paged attention, continuous batching or scheduler overhead. It does not predict model quality.

For a production plan, measure tokens per second at your p95 prompt and output length, then calibrate the efficiency field until the model matches. Use the calibrated model to explore context length and batch size, not to claim a benchmark. Related: the rate limit calculator, the GPU capacity calculator, and Plugsky plans.

FAQ

Why is decode called memory-bandwidth bound?

Each generated token requires reading the model weights and the KV cache for every active sequence. The GPU's arithmetic units are mostly idle during that read. Aggregate decode throughput is therefore approximately bandwidth × efficiency divided by bytes read per token step, not peak FLOPS. Prefill is the opposite: it processes many prompt tokens at once and is compute-bound.

What is the difference between batch size and concurrency?

In this calculator, batch size is the number of sequences decoded together in one step. Concurrency is the number of in-flight requests, which is usually the same number for a single model replica. Both consume KV cache. More concurrency raises aggregate throughput until memory or bandwidth saturates, but it also raises per-request latency because each request waits for its turn in the batch.

Why does my real throughput differ from the estimate?

The model is a roofline estimate. It assumes perfect memory coalescing, no kernel launch overhead, no tensor-parallel interconnect cost, no scheduler overhead and a single sequence length. Real serving stacks add paged attention, continuous batching, prefix caching and speculative decoding, each of which changes the result. Use the estimate for sizing, then measure.

How do I estimate KV cache per token?

For a standard multi-head or grouped-query attention model with FP16 cache: 2 (keys and values) × layers × KV heads × head dimension × 2 bytes. A 70B model with 80 layers, 8 KV heads and 128 head dimension uses about 327 KB per token, or 2.7 GB for an 8,192-token sequence. Models with multi-head latent attention compress the cache differently; use the vendor's formula.

Does the cost include prefill and idle time?

The cost KPI is GPU-hours divided by generated tokens, so it includes the time spent on prefill for the requests in the batch and the decode time. It does not include idle time, failed requests, retries, CPU overhead or storage. Enter the hourly price you actually pay, including tax and network, and add a utilization factor if the GPUs are not busy.

Sources and verification

Primary sources were opened and checked on 20 November 2026. Product terms and datasets change; the linked source is authoritative.

Don't want to size the GPUs yourself?

Plugsky serves hosted models through one OpenAI-compatible API with published plan caps. The free tier starts at 100 requests and 50K tokens per day.

See plans → Browse models