Local 70B: What It Costs in Hardware and Watts

By LK Wood IV · Published 2026-09-07 · ~9 min read · St. Louis County, MO

A dense 70B is the model everyone means when they say they want to run a real LLM at home. Here is what it actually does once it is loaded: 12.6 tokens per second.

That is Hardware Corner’s March 2026 measurement of Llama 3.3 70B at Q4 on an RTX A6000, the cheapest 48GB card that holds the file. A thousand-token answer takes 79 seconds. Push the context to 16K and the same card gives 9.1 tokens per second, which is 110 seconds for the same answer.

At that rate the electricity tops out at $1.21 per million tokens. The card costs $5,490.00 new. Watts are not what makes a local 70B expensive. Capital and patience are.

Some retailer links below are affiliate links; if you buy through them TechFuel HQ may earn a small commission at no extra cost to you. As an Amazon Associate I earn from qualifying purchases. Commissions never influence what gets recommended — see our disclosure.

42.52 gigabytes decides which cards are even in the conversation

The Llama 3.3 70B Q4_K_M GGUF is 42,520,398,816 bytes — 42.52GB of weights before the KV cache and runtime buffers exist. Hugging Face’s repository API reports that byte count directly, and the whole quantization ladder with it:

Exact sizes from the bartowski GGUF repository file listing.

Read the columns rather than the rows. A 24GB card gets you IQ2_S with 1.8GB left over for everything else — a 70B in name only. A 32GB card gets you Q3_K_S with about a gigabyte of headroom, which is why an RTX 5090 does not solve this problem despite costing what it costs. 48GB is the first tier where the file loads at a quantization worth using and leaves room to work. That is the line, and it does not move because you want it to.

If you are choosing between the specific cards that sit either side of it, the RTX A6000 versus RTX 5090 comparison is the card-by-card version of that decision. This page is about what the whole tier costs once you are in it.

What 48GB actually delivers on a 70B

Hardware Corner runs its GPU benchmarks on Ubuntu 24.04 with llama.cpp’s llama-bench and CUDA 12.8 at 4-bit quantization, and publishes the per-card tables. These are its Llama 3.3 70B rows:

Decode and prefill figures come from Hardware Corner’s per-card pages, one page per row: RTX A6000, RTX 6000 Ada and RTX PRO 5000 Blackwell. Methodology is on its GPU ranking page, which states the 4-bit runs use Q4_K_XL rather than the Q4_K_M file sized in the table above; the two are close enough that both need 48GB and far enough apart that these rates are not a measurement of that exact file. Memory and maximum power from NVIDIA’s professional desktop GPU specifications, which rates all three boards at 300W and lists the PRO 5000 Blackwell in 48GB and 72GB versions; Hardware Corner’s page does not say which one it benchmarked.

Two things follow. The 70B row exists only for 48GB cards — Hardware Corner’s RTX 3090, RTX 4090 and RTX 5090 pages carry no Llama 3.3 70B measurement at all, and the 3090 and 4090 pages say why in one line: 24GB cannot hold the file. The only 4090 with a 70B row on that site is the modified 48GB variant, which is a different card. And the spread inside the tier is nearly 2x, entirely on architecture: the same 48GB and the same 300W, five years apart, gets you 12.6 or 24.0 tokens per second.

The A6000 is the cheap way in and it is the slow one. That is the trade you are making.

For contrast, here is 16GB

I measured a retail RTX 5080 across four models with Ollama at temperature 0, three fresh-prefill runs per cell, and published the throughput dataset as open CSV. gpt-oss 20B in MXFP4 decodes at 178 to 190 tokens per second across two dated captures, fully resident in about 14GB. Qwen 2.5 14B at Q4_K_M runs 94 to 97.

That is fifteen times the A6000’s 70B rate on a card costing a quarter as much. The 20B is not the same model and nobody should pretend it is — but in my own eight-task coding evaluation gpt-oss 20B passed all eight, including two-turn tool calling. The gap in delivered usefulness is much narrower than the gap in parameter count, and the gap in speed runs the other way by an order of magnitude.

Hardware Corner’s own A6000 table makes the same point on one board: Qwen3 30B A3B, a mixture-of-experts model, runs at 131.2 tokens per second on the card where the dense 70B manages 12.6. Ten times faster, same silicon.

The electricity is the cheap part

Divide a published power rating into a published token rate and you get the electricity per million tokens. At NVIDIA’s 300W rating for the professional 48GB boards, and the EIA US residential average of 18.34 cents per kilowatt-hour for June 2026:

The 5080 rows are measured — board power sampled at 1Hz alongside the generation that produced the rate, published in the power-efficiency dataset, which is why their rates differ slightly from the throughput capture above. The 70B rows pair Hardware Corner’s token rates with NVIDIA’s rated wattage, so they are ceilings: a card that draws less than 300W lands under them.

Even the worst row on this table is under two dollars per million tokens. Running the 70B costs roughly sixteen times what running gpt-oss 20B costs, and sixteen times almost nothing is still almost nothing. Anyone budgeting a local 70B around its power draw is solving the wrong problem.

The watts that do matter are the idle ones. My 5080 holds 52 to 71W at the Windows desktop rather than the 8 to 15W reviews quote, which is $85 to $115 a year before a single token. A mostly-idle inference node’s bill is set by what it does the other twenty-three hours. I have not measured an A6000’s idle draw, so there is no number for it here. The logic still applies, and it applies twice to a two-card machine.

Two used 3090s are the value answer at 48GB

Two RTX 3090s total 48GB, which clears 42.52GB with the same 5.48GB of headroom a single 48GB board gives you. NVIDIA lists the 3090 at 24GB GDDR6X and 350W.

The price is why people do it. Hardware Corner’s two published figures for a used 3090 disagree: its dual-GPU comparison puts a used RTX 3090 at “around $800”, while the price panel on its RTX 3090 benchmark page states a $1,000 market average. They are the same publisher on two pages with different update dates, and the used market moves fast enough that both can have been right when written. Call it $1,600 to $2,000 for the pair and price the actual listing in front of you.

Against a $5,490.00 new A6000, that is a third of the price for the same capacity. It is the value answer and it has been for years.

What you give up is not subtle. 700W of card rating instead of 300W. Two boards’ worth of idle draw. Two slots, the airflow between them, and a power supply sized for both. Blower-style cards exist to be stacked in a chassis rather than to sit near you, and you will hear it. Neither card carries a warranty you can rely on, and no standalone RTX 3090 appears anywhere in the August 17 Newegg feed: the rows that name the card are water blocks, cables, cooling parts and prebuilt systems containing one, and not a single graphics card. This is a used-market purchase with used-market recourse.

Splitting buys capacity, not speed

The thing people expect from two cards is twice the throughput. They do not get it, and the reason is in llama.cpp’s own argument documentation, which describes the default split mode as splitting “layers and KV across GPUs (pipelined)” against the row and tensor modes it labels “parallelized”. Pipelined means the layers on card two wait for the layers on card one. Only one GPU is computing at a time.

Hardware Corner’s dual-card test shows the shape of it from the other direction. Testing two RTX 5060 Ti 16GB against one RTX 3090 with llama.cpp, the pair reached 131K context on a 30B MoE model and 45K on a dense 32B, well past what the single card sustained — while the single 3090 generated dense-model tokens roughly twice as fast at 4K to 16K context. Capacity to the pair, speed to the one fast card.

That comparison varies bandwidth as well as card count, and Hardware Corner attributes the speed gap to bandwidth explicitly, so read it narrowly: a second card reliably buys you memory and does not reliably buy you tokens per second. No dual-3090 70B rate appears in Hardware Corner’s tables — its 48GB 70B rows are all single cards — so treat the 12.6 to 24.0 range above as the tier’s speed and do not expect two 3090s to beat it because there are two of them.

NVLink does not rescue this either. It moves the needle for training and for concurrent users. One person typing is neither.

Do not buy hardware for a dense 70B

Buy a 16GB-class card and run a mixture-of-experts model. That is the recommendation for nearly everyone who arrives at this question, and the numbers above are why: 178 to 190 tokens per second on my own measurement against 12.6 on the cheapest board that holds the dense 70B, for a quarter of the price and a sixteenth of the electricity per token.

The cheapest RTX 5080 in the August feed was $1,399.99 , and thirteen boards in all sat between that price and $1,999.99, so there is room to shop the exact model.

Prices move. Check before you order, and run your context budget through the VRAM calculator before you commit to any card — the 5.48GB of headroom a 48GB board leaves a 70B is enough for ordinary context and not for long context, and that failure arrives mid-session rather than at load time.

If a dense 70B is genuinely the requirement

Some workloads want the dense model and nothing else will do. In that case: two used RTX 3090s. A third of the price of a new A6000 for the same 48GB, at the cost of 700W of card rating, two slots, the noise and no warranty worth the name.

Buy a single 48GB board instead when you are paying for the things the pair cannot give you — one slot, 300W, ECC memory, the enterprise driver branch, and a card you can put in a case without planning the airflow around it. Hardware Corner states a $4,600 market average for a used A6000 against the $5,490.00 new list; a used listing asking near the new price is not a deal, because the discount is the only reason to accept a 2020 board with somebody else’s hours on it.

How this was checked

The capacity table uses exact byte counts from the Hugging Face repository API for the bartowski Llama-3.3-70B-Instruct-GGUF repo, not rounded parameter math. Llama-3.3-70B-Instruct-Q4_K_M.gguf is 42,520,398,816 bytes.

The 70B speed figures are Hardware Corner’s, published March 2026 and captured with llama.cpp’s llama-bench on Ubuntu 24.04 with CUDA 12.8 at 4-bit. I did not benchmark a 70B model.

The 16GB figures are mine: a retail ASUS ROG Astral RTX 5080 OC under Ollama at temperature 0, three fresh-prefill runs per cell for throughput, and board power sampled at 1Hz over the same generations for the efficiency figures. Both raw CSVs are published CC BY 4.0 at /data/rtx-5080-llm-throughput/ and /data/rtx-5080-llm-power-efficiency/.

Card memory and power ratings come from NVIDIA’s own specification pages. The kilowatt-hours and dollars per million tokens are arithmetic dividing a rated wattage into a measured token rate at the June 2026 EIA residential average of $0.1834/kWh; the 70B rows are ceilings because a rated maximum is not a measured draw.

Prices are the lowest in-stock USD offer per exact model from the August 17 Newegg feed. Every used-card figure on this page is Hardware Corner’s stated market value, named as theirs where it appears.

This page gets re-checked when a new 48GB-class card ships or a major 70B release changes the file size.