MXFP4 vs Q4_K_M: Why 21B Beats 14B on an RTX 5080

The strangest row in our RTX 5080 throughput dataset is the one where the biggest model is not the slowest. gpt-oss 20B — 20.9B parameters as ollama reports it — decodes at 187–190 tokens per second. Qwen 2.5 14B, six billion parameters smaller, manages 94–97. The “bigger” model is twice as fast, and it lands within a few percent of the dense 7B. None of that is a fluke, and the explanation is worth having because it changes which models you should even shortlist on a 16GB card.

The numbers, and where they come from

Everything below is from our own bench: ollama 0.32.1 on a retail RTX 5080 (driver 610.88), medians of three fresh-prefill runs per cell, model digests pinned, with a nonce prepended each rep so the prompt cache cannot fake the prefill numbers. The full matrix, method, and raw CSV are on the dataset page.

Decode is a memory race, and only active bytes run it

Generating a token means streaming the model’s weights through the GPU once. Do that a hundred times a second and the limiting factor is not compute — it is how fast VRAM can feed the cores. The RTX 5080’s memory bus moves about 960 GB/s. So the ceiling is simple division: bandwidth over bytes-touched-per-token.

Run it for the dense 14B. Q4_K_M stores roughly 4.5–4.8 bits per weight, so 14.8B parameters is about 8.5–8.9GB that every single token must read. 960 ÷ 8.7 ≈ 110 tokens per second as the theoretical ceiling — and the measured 94–97 is about 90% of that. The arithmetic and the measurement agree, which is how you know the model is bandwidth-bound.

Now gpt-oss 20B. It is a mixture-of-experts model: per OpenAI’s model card, 21B total parameters but only about 3.6B active per token, with the expert weights shipped natively in MXFP4 — 4-bit floating point with shared scale factors, the OCP microscaling format. The 21B has to fit in VRAM (hence the ~14GB resident), but each token only streams the active slice. A few gigabytes per token instead of nine. That is the whole trick, and it is why the 21B model decodes like a 7B — because per token, it approximately is one.

Total parameters decide whether a model fits. Active bytes decide how fast it runs. Those are different questions, and the spec sheets only advertise the first.

What context length actually costs

The other quietly useful result: at prompts of ~500 and ~1,700 tokens, decode speed was flat between num_ctx 4096 and 16384 for every model — within about 1%. The 16k configurations cost VRAM (the KV cache has to live somewhere) but not speed at these lengths. If you have been keeping context small to protect tokens per second, at normal prompt sizes you are paying a real capacity price for an imaginary speed benefit. Push into genuinely long prompts and attention will eventually take its cut — but that boundary is much further out than the folk wisdom says.

Prefill tells the same story from the other side: it is compute-bound and parallel, and gpt-oss chewed through prompts at 9,000–10,500 tokens per second, roughly double the dense 14B there too.

What to shortlist on a 16GB card

If you want the strongest model that still feels instant on this class of hardware, the measured answer is the MoE: gpt-oss 20B gives 14B-and-up class capability at dense-7B interactive speed, for about 14GB of VRAM. The dense 14B is the choice when a specific dense model’s quality on your task justifies half the speed. And the 3B remains the bulk-work option — 300 tokens per second is a different kind of tool.

Where your GPU lands if it is not a 5080: the local LLM by GPU VRAM guide maps models to cards, the LLM speed calculator estimates decode from bandwidth and quant, and the VRAM calculator does the fit math. What running this card 24/7 costs in electricity is measured in the 5080 perf-per-watt piece.

The dataset behind every number here is CC BY 4.0 — take it: /data/rtx-5080-llm-throughput/.