Local LLM Speed Calculator: Estimate Tokens per Second

By .

Estimate local LLM generation speed (tokens/sec) from your GPU bandwidth, model size, and quantization — the memory-bandwidth math behind real inference.

Bandwidth over active weight size, calibrated on our RTX 5080.

Estimated speed by quantization (your model + GPU)

QuantWeights sizeEst. tokens/secEst. VRAM need

This estimates generation (decode) speed, which is memory-bandwidth-bound: each new token requires reading the active weights from VRAM once, so tokens/sec ≈ (bandwidth ÷ active weights size) × efficiency. On a dense model every parameter is active, so total size is what streams. On a mixture-of-experts model only a slice does — fill in Active params/token (gpt-oss 20b is 20.9B total / ~3.6B active; Qwen3-30B-A3B is 30.5B / 3.3B) and the speed estimate uses that slice while the weights and VRAM figures stay on the total, which is what has to fit. MoE decode also realizes less of a card's bandwidth than dense decode, so a 0.60 derate is applied on that path, calibrated against our own measured RTX 5080 throughput. It's a ballpark — most accurate for larger models; small models and long contexts run slower than the pure-bandwidth math suggests (compute, sampling, and KV-cache reads add overhead), so treat results as ±30%. For whether a model fits in VRAM, use the LLM VRAM Calculator.

Worked examples

These are this calculator’s own outputs, computed with the same formula the tool runs in your browser — so you can see a real answer without touching a single input.

Mainstream 2026 gaming/AI desktop running a small everyday chat model — an 8B model (Llama 3.1 8B class) at Q4_K_M on an RTX 5070 Ti, efficiency left at the 70% default

  • Inputs: Model size = 8 B params; Quantization = Q4_K_M (0.60 bytes/param); GPU = RTX 5070 Ti (896 GB/s); Efficiency = 70%
  • Result: Est. generation speed 131 tokens/sec · Model weights 4.8 GB (at 0.60 bytes/param) · Rough VRAM need 5.8 GB · Memory bandwidth 896 GB/s × 70% efficiency. Table: Q8 8.5 GB / 74.0 tok/s / 10.2 GB; Q5_K_M 5.7 GB / 110 tok/s / 6.8 GB; Q4_K_M 4.8 GB / 131 tok/s / 5.8 GB; Q3_K_M 3.9 GB / 160 tok/s / 4.7 GB.

Common homelab box on a used 24 GB card — a 32B model (Qwen2.5 32B / QwQ class) at Q4_K_M on an RTX 3090, efficiency left at 70%

  • Inputs: Model size = 32 B params; Quantization = Q4_K_M (0.60 bytes/param); GPU = RTX 3090 (936 GB/s); Efficiency = 70%
  • Result: Est. generation speed 34.1 tokens/sec · Model weights 19.2 GB (at 0.60 bytes/param) · Rough VRAM need 23.0 GB · Memory bandwidth 936 GB/s × 70% efficiency. Table: Q8 33.9 GB / 19.3 tok/s / 40.7 GB; Q5_K_M 22.7 GB / 28.8 tok/s / 27.3 GB; Q4_K_M 19.2 GB / 34.1 tok/s / 23.0 GB; Q3_K_M 15.7 GB / 41.8 tok/s / 18.8 GB.
    Note the 23.0 GB VRAM figure against a 24 GB card — genuinely borderline, which is the useful reader takeaway.

How to use this calculator

Enter the model’s total size in billions of parameters, pick the quantization you’ll run, and choose your GPU (or enter a custom memory bandwidth). If the model is a mixture-of-experts (gpt-oss, Qwen3-A3B, DeepSeek and friends), also fill in active params/token - the slice that actually streams from VRAM on each token. Leave it blank for a dense model. The calculator estimates the generation speed in tokens per second — the speed you feel while the model is typing its reply — plus the weights size and a rough VRAM figure. The table compares common quant levels so you can see the speed-vs-quality trade for your exact hardware.

If you’ve measured your real tokens/sec on a known model, nudge the efficiency slider until the estimate matches — then its predictions for other models and quants on your rig get more accurate.

Why bandwidth, not TFLOPS, decides speed

Local LLM generation is memory-bandwidth-bound. To produce each token, the GPU streams the model’s weights out of VRAM once and does comparatively little arithmetic per byte. So the limiting factor is how fast the card can read memory — its GB/s — not its raw compute. That’s why a memory-bandwidth number predicts token speed far better than a TFLOPS rating, and why the formula is simply:

tokens/sec ≈ (memory bandwidth ÷ weights size) × efficiency

A 70B model at Q4_K_M is 42.0 GB of weights at the 0.60 bytes-per-parameter figure this calculator uses; divide a card’s bandwidth by that and you’re in the right ballpark. Halve the weights (a lighter quant) and you roughly double the speed.

Speed vs. fit: two different questions

“How fast will it run?” and “will it even fit?” are separate problems. This tool answers the first. A model can fit comfortably in VRAM and still be slow (big model, modest bandwidth), or be fast in theory but not fit at all (not enough VRAM, forcing slow CPU offload). For the fit side — VRAM for weights plus the KV cache at your context length — use the LLM VRAM Calculator. Run both before you buy a card or pick a model.

Picking a model for your card

If the estimated speed is comfortable (say 15+ tok/s for chat) and the model fits, you’re set. If it’s too slow, your levers are: a lighter quant (faster and smaller, slight quality cost), a smaller model, or a card with more bandwidth. For which models actually fit each card in the first place, the VRAM-tier guide to local LLMs by GPU maps it tier by tier; for build-level guidance, see the single-RTX-5060 local-LLM walkthrough and the $1,000 local-AI workstation for a tuned setup.

A note on accuracy

These are estimates from the bandwidth-bound model, most reliable for larger models and short-to-medium contexts. Real numbers vary with context length (the KV cache grows and gets re-read), the inference backend, and whether the whole model fits in VRAM. The backend gap is smaller than most people expect: measured on one RTX 5080, Ollama and llama.cpp decode within 2–6% of each other on dense models. Use the figure to compare GPUs and quants and to size expectations — not as a guaranteed benchmark.

Assumptions and sources

  • GPU memory-bandwidth figures are the vendors’ published specifications, cross-checked against the TechPowerUp GPU database - for example the RTX 4090 at 1,008 GB/s and the RTX 3090 at 936 GB/s.
  • Bytes-per-parameter for each quantization (0.60 for Q4_K_M, i.e. about 4.83 bits per weight) follow the GGUF quantization sizes implemented in llama.cpp, the runtime family this model assumes.
  • The 70% default efficiency factor and the ±30% accuracy framing are TechFuelHQ calibration assumptions, stated so you can tune the slider against a tokens/sec figure you have actually measured on your own rig.
  • On the mixture-of-experts path the tool multiplies that efficiency by a further 0.60. MoE decode reads scattered per-token expert slices rather than one contiguous block, so it realizes less of a card’s bandwidth than dense decode. The derate is calibrated against our own measurement - gpt-oss:20b on an RTX 5080 (20.9B total, about 3.6B active per token) decodes at ~188 tok/s on a 960 GB/s card, and (960 / (3.6 x 0.60)) x 0.70 x 0.60 = 187. It is one calibration point on one card, so tune the efficiency field if you have measured your own.
  • The rough VRAM figure adds 20% to the weights for KV cache and overhead - a TechFuelHQ planning allowance; for a real fit check use the LLM VRAM Calculator, which models the cache properly.

Related guides and tools

Data

RTX 5080 Local LLM Throughput: Measured Tokens/Second Dataset

Measured decode and prefill tokens/second for gpt-oss 20B, Qwen 2.5 14B and 7B, and Llama 3.2 3B on a retail RTX 5080, …

LLM VRAM Calculator

Find which local AI models fit your GPU: pick card, model size, quantization, and context length to estimate VRAM …

Articles

Which Local LLM Fits Your GPU in 2026? VRAM Tiers

Which local LLM fits your GPU in 2026: models for 8, 12, 16, 24, and 48GB+ cards at Q4_K_M, with real GGUF file sizes …

Tutorials

How to Self-Host a Local LLM on a Single RTX 5060 in 2026

Run a private LLM on a $300 RTX 5060. VRAM tradeoffs, software stack picks, Windows + Linux setup, real tokens/sec …

Frequently asked questions

How is local LLM speed (tokens per second) estimated?
Generation speed is dominated by memory bandwidth: to produce each new token, the GPU must read the model’s weights out of VRAM once. So tokens/sec ≈ (memory bandwidth ÷ weights size in GB) × an efficiency factor. A 70B model quantized to Q4_K_M is 42.0 GB of weights (this calculator uses 0.60 bytes per parameter for Q4_K_M); on an RTX 4090 (1008 GB/s) at 70% efficiency that’s 16.8 tokens/sec. This calculator runs that math for your GPU, model size, and quant.
Why is LLM inference memory-bandwidth-bound and not compute-bound?
During generation, the model processes one token at a time, and each step reads every active weight from memory but does relatively little math per byte read. That makes the memory subsystem the bottleneck, not the GPU’s compute units — which is why a card’s GB/s memory bandwidth predicts token speed far better than its TFLOPS. Prompt processing (reading your input) is the opposite — compute-bound and much faster per token — so this tool estimates the generation speed you feel while the model is replying.
How many tokens per second will a 4090 or 3090 do on a 70B model?
Ballpark, at Q4_K_M (42.0 GB of weights): an RTX 4090 (1008 GB/s) lands around 15–20 tok/s and an RTX 3090 (936 GB/s) around 14–18 tok/s, assuming the model fits in VRAM (a single 24 GB card is tight for 70B Q4 — many people use two). Smaller models are much faster: an 8B at Q4 on either card runs well over 80–100 tok/s. Use the calculator with your exact card and quant for a per-setup estimate.
Does quantization make a model faster, or just smaller?
Both. A heavier quant means fewer bytes per parameter, so there are fewer bytes to read per token — which directly raises tokens/sec on a bandwidth-bound workload, on top of shrinking the VRAM footprint. Q4_K_M reads roughly a third of the data of FP16 - 0.60 bytes per parameter against 2.0, or 30% - so it’s both far smaller and about three times faster. The trade is quality: lower quants lose some accuracy, with Q4_K_M widely considered the sweet spot for most local use.
Why is my real tokens/sec slower than this estimate?
The pure-bandwidth model is an upper-ish bound and is most accurate for larger models. Real speed is lower because of KV-cache reads that grow with context length, sampling overhead, CPU/offload bottlenecks if the model doesn’t fully fit in VRAM, and backend efficiency (llama.cpp vs vLLM vs others). That’s why the tool exposes an efficiency slider — dialing it to match a number you’ve actually measured makes its other estimates more accurate. Treat results as ±30%.
Does this tell me whether a model fits in my GPU?
Not directly — this tool estimates speed, not fit. It shows the weights size and a rough VRAM figure (weights + ~20% for the KV cache and overhead), but actual VRAM use also depends on context length and batch size. For a proper fit check, use the companion LLM VRAM Calculator, which models the KV cache and context window. The two together answer ‘will it fit, and how fast will it run.’