Local LLM Speed Calculator: Estimate Tokens per Second

By .

Check whether a local LLM roughly fits your GPU, then estimate generation speed from memory bandwidth, model size, and quantization.

Bandwidth over active weight size, calibrated on our RTX 5080.

Estimated speed by quantization (your model + GPU)

QuantWeights sizeGPU-only est. tokens/secEst. VRAM need

This estimates generation (decode) speed, which is memory-bandwidth-bound: each new token requires reading the active weights from VRAM once, so tokens/sec ≈ (bandwidth ÷ active weights size) × efficiency. On a dense model every parameter is active, so total size is what streams.

On a mixture-of-experts model only a slice does — fill in Active params/token (gpt-oss 20b is 20.9B total / ~3.6B active; Qwen3-30B-A3B is 30.5B / 3.3B) and the speed estimate uses that slice while the weights and VRAM figures stay on the total, which is what has to fit.

MoE decode also realizes less of a card's bandwidth than dense decode, so a 0.60 derate is applied on that path, calibrated against our own measured RTX 5080 throughput.

It's a ballpark — most accurate for larger models; small models and long contexts run slower than the pure-bandwidth math suggests (compute, sampling, and KV-cache reads add overhead), so treat results as ±30%. The tool shows speed only when its rough VRAM allowance fits the selected accelerator.

It does not estimate CPU or partial-GPU offload. For a context-aware fit check, use the LLM VRAM Calculator.

Worked examples

These are this calculator’s own outputs, computed with the same formula the tool runs in your browser — so you can see a real answer without touching a single input.

Mainstream 2026 gaming/AI desktop running a small everyday chat model — an 8B model (Llama 3.1 8B class) at Q4_K_M on an RTX 5070 Ti, efficiency left at the 70% default

  • Inputs: Model size = 8 B params; Quantization = Q4_K_M (0.60 bytes/param); GPU = RTX 5070 Ti (896 GB/s); Efficiency = 70%
  • Result: Est. generation speed 131 tokens/sec · Model weights 4.8 GB (at 0.60 bytes/param) · Rough VRAM need 5.8 GB · Memory bandwidth 896 GB/s × 70% efficiency. Table: Q8 8.5 GB / 74.0 tok/s / 10.2 GB; Q5_K_M 5.7 GB / 110 tok/s / 6.8 GB; Q4_K_M 4.8 GB / 131 tok/s / 5.8 GB; Q3_K_M 3.9 GB / 160 tok/s / 4.7 GB.

Common homelab box on a used 24 GB card — a 32B model (Qwen2.5 32B / QwQ class) at Q4_K_M on an RTX 3090, efficiency left at 70%

  • Inputs: Model size = 32 B params; Quantization = Q4_K_M (0.60 bytes/param); GPU = RTX 3090 (936 GB/s); Efficiency = 70%
  • Result: Est. generation speed 34.1 tokens/sec · Model weights 19.2 GB (at 0.60 bytes/param) · Rough VRAM need 23.0 GB · Memory bandwidth 936 GB/s × 70% efficiency. Table: Q8 33.9 GB / 19.3 tok/s / 40.7 GB; Q5_K_M 22.7 GB / 28.8 tok/s / 27.3 GB; Q4_K_M 19.2 GB / 34.1 tok/s / 23.0 GB; Q3_K_M 15.7 GB / 41.8 tok/s / 18.8 GB.
    Note the 23.0 GB VRAM figure against a 24 GB card — genuinely borderline, which is the useful reader takeaway.

How to use this calculator

Enter the model’s total size in billions of parameters, pick the quantization you’ll run, and choose your GPU (or enter a custom memory bandwidth). If the model is a mixture-of-experts (gpt-oss, Qwen3-A3B, DeepSeek and friends), also fill in active params/token - the slice that actually streams from VRAM on each token.

Leave it blank for a dense model. The calculator first compares the rough memory need with the selected accelerator’s capacity. When it fits, it estimates generation speed in tokens per second — the speed you feel while the model is typing its reply — and compares common quant levels.

Apple unified-memory and custom accelerator choices require you to enter the memory available to the workload.

If you’ve measured your real tokens/sec on a known model, nudge the efficiency slider until the estimate matches — then its predictions for other models and quants on your rig get more accurate.

Why bandwidth, not TFLOPS, decides speed

Local LLM generation is memory-bandwidth-bound.

To produce each token, the GPU streams the model’s weights out of VRAM once and does comparatively little arithmetic per byte.

So the limiting factor is how fast the card can read memory — its GB/s — not its raw compute.

That’s why a memory-bandwidth number predicts token speed far better than a TFLOPS rating, and why the formula is simply:

tokens/sec ≈ (memory bandwidth ÷ weights size) × efficiency

A 70B model at Q4_K_M is 42.0 GB of weights at the 0.60 bytes-per-parameter figure this calculator uses. That bandwidth division is meaningful only when the deployment keeps the model in accelerator memory. Halve the weights with a lighter quant and the fully resident bandwidth estimate roughly doubles.

Speed vs. fit: two different questions

“How fast will it run?” and “will it even fit?” are separate problems, but a GPU-bandwidth speed estimate is useful only after a fit check. This tool uses weights plus a 20% planning allowance as that first gate. If the requirement exceeds the selected memory, it shows the mismatch and withholds tokens/sec instead of presenting an impossible single-device result.

For a detailed fit calculation — weights plus the KV cache at your context length and runtime headroom — use the LLM VRAM Calculator. CPU offload, partial GPU offload, and multi-device scaling are not estimated here, so a failed fit check has no speed number.

Picking a model for your card

If the estimated speed is comfortable (say 15+ tok/s for chat) and the model fits, you’re set. If it’s too slow, your levers are: a lighter quant (faster and smaller, slight quality cost), a smaller model, or a card with more bandwidth.

For which models actually fit each card in the first place, the VRAM-tier guide to local LLMs by GPU maps it tier by tier; for build-level guidance, see the single-RTX-5060 local-LLM walkthrough.

A note on accuracy

These are estimates from the bandwidth-bound model, most reliable for larger models and short-to-medium contexts. Real numbers vary with context length, the inference backend, and runtime overhead. Passing the rough capacity gate does not replace a context-aware fit calculation.

The backend gap is smaller than most people expect: measured on one RTX 5080, Ollama and llama.cpp decode within 2–6% of each other on dense models. Use the figure to compare GPUs and quants and to size expectations — not as a guaranteed benchmark.

Assumptions and sources

  • GPU memory capacities come from NVIDIA’s official RTX 50-series, RTX 40-series, RTX 3090, RTX 3060-family, and A100 specifications. Apple unified-memory and custom choices use the capacity you enter because available memory varies by configuration and workload.
  • GPU memory-bandwidth figures are the vendors’ published specifications, cross-checked against the TechPowerUp GPU database - for example the RTX 4090 at 1,008 GB/s and the RTX 3090 at 936 GB/s.
  • Bytes-per-parameter for each quantization (0.60 for Q4_K_M, i.e. about 4.83 bits per weight) follow the GGUF quantization sizes implemented in llama.cpp, the runtime family this model assumes.
  • The 70% default efficiency factor and the ±30% accuracy framing are TechFuelHQ calibration assumptions, stated so you can tune the slider against a tokens/sec figure you have actually measured on your own rig.
  • On the mixture-of-experts path the tool multiplies that efficiency by a further 0.60. MoE decode reads scattered per-token expert slices rather than one contiguous block, so it realizes less of a card’s bandwidth than dense decode. The derate is calibrated against our own measurement - gpt-oss:20b on an RTX 5080 (20.9B total, about 3.6B active per token) decodes at ~188 tok/s on a 960 GB/s card, and (960 / (3.6 x 0.60)) x 0.70 x 0.60 = 187. It is one calibration point on one card, so tune the efficiency field if you have measured your own.
  • The rough VRAM figure adds 20% to the weights for KV cache and overhead - a TechFuelHQ planning allowance; for a real fit check use the LLM VRAM Calculator, which models the cache properly.

Related guides and tools

Data

RTX 5080 Local LLM Throughput: Measured Tokens/Second Dataset

Measured decode and prefill tokens/second for gpt-oss 20B, Qwen 2.5 14B and 7B, and Llama 3.2 3B on a retail RTX 5080, …

LLM VRAM Calculator

Find which local AI models fit your GPU: pick card, model size, quantization, and context length to estimate VRAM …

Articles

Which Local LLM Fits Your GPU in 2026? VRAM Tiers

Which local LLM fits your GPU in 2026: models for 8, 12, 16, 24, and 48GB+ cards at Q4_K_M, with real GGUF file sizes …

Tutorials

How to Self-Host a Local LLM on a Single RTX 5060 in 2026

Run a private LLM on a $300 RTX 5060. VRAM tradeoffs, software stack picks, Windows + Linux setup, real tokens/sec …

Frequently asked questions

How is local LLM speed (tokens per second) estimated?
Generation speed is dominated by memory bandwidth when the model is resident in accelerator memory: each new token requires reading the active weights. So tokens/sec ≈ (memory bandwidth ÷ active weights size in GB) × an efficiency factor. The default 32B Q4_K_M model is 19.2 GB of weights and has a 23.0 GB rough requirement; on an RTX 4090 (1008 GB/s, 24 GB) at 70% efficiency the estimate is 36.8 tokens/sec. The calculator suppresses this speed estimate when its rough memory requirement exceeds the selected device’s capacity.
Why is LLM inference memory-bandwidth-bound and not compute-bound?
During generation, the model processes one token at a time, and each step reads every active weight from memory but does relatively little math per byte read. That makes the memory subsystem the bottleneck, not the GPU’s compute units — which is why a card’s GB/s memory bandwidth predicts token speed far better than its TFLOPS. Prompt processing (reading your input) is the opposite — compute-bound and much faster per token — so this tool estimates the generation speed you feel while the model is replying.
How many tokens per second will a 4090 or 3090 do on a 70B model?
A 70B model at Q4_K_M has 42.0 GB of weights and a 50.4 GB rough requirement in this calculator, so it cannot fit on one 24 GB RTX 4090 or RTX 3090. The tool therefore does not show a one-card GPU speed for that setup. Multi-GPU layouts and CPU or partial-GPU offload have different memory paths and are outside this calculator’s model. Smaller models that pass the capacity check can use the displayed GPU-only estimate.
Does quantization make a model faster, or just smaller?
Both. A heavier quant means fewer bytes per parameter, so there are fewer bytes to read per token — which directly raises tokens/sec on a bandwidth-bound workload, on top of shrinking the VRAM footprint. Q4_K_M reads roughly a third of the data of FP16 - 0.60 bytes per parameter against 2.0, or 30% - so it’s both far smaller and about three times faster. The trade is quality: lower quants lose some accuracy, with Q4_K_M widely considered the sweet spot for most local use.
Why is my real tokens/sec slower than this estimate?
The pure-bandwidth model is an upper-ish bound and is most accurate for larger models. Real speed is lower because of KV-cache reads that grow with context length, sampling overhead, CPU/offload bottlenecks if the model doesn’t fully fit in VRAM, and backend efficiency (llama.cpp vs vLLM vs others). That’s why the tool exposes an efficiency slider — dialing it to match a number you’ve actually measured makes its other estimates more accurate. Treat results as ±30%.
Does this tell me whether a model fits in my GPU?
It performs a conservative first-pass check: model weights plus a 20% allowance must fit the selected accelerator memory before the tool shows a GPU-only speed estimate. Actual memory use still depends on context length, batch size, runtime, and KV-cache settings. Use the companion LLM VRAM Calculator for that detailed fit check. CPU or partial-GPU offload speed is not estimated here.