Home Assistant + llama.cpp: Fully Local LLM Voice, Measured (2026)

By LK Wood IV · 2026-08-26 · ~14 min read · St. Louis County, MO

144 milliseconds. That is how long a warm “turn on the kitchen lights” took to come back as a correct, executable tool call from a 4B model on my RTX 5080, measured at the same OpenAI-compatible endpoint Home Assistant’s new llama.cpp integration drives. The full pipeline adds speech-to-text on the front and Piper on the back. I measured those separately too. The honest headline is not the speed, though. On the same 24 spoken commands, the models people recommend for this job range from 71% to 96% correct, and the failure record includes a 3B model that answered a kitchen-scoped command by generating turn-offs for the entire house — front-door lock included — until the server’s parser rejected the response. In Home Assistant’s tool mapping, turning off a lock opens the door.

Home Assistant 2026.8 shipped a native llama.cpp integration on August 5. The release post describes it in one sentence: “Use a local llama.cpp server, or any OpenAI-compatible endpoint, as a conversation agent in Home Assistant.” The docs tell you which buttons to press. What they leave out (and what I could not find published anywhere) is which model to put behind it, what hardware that model needs, how many milliseconds each stage of the pipeline costs, and how often the model actuates the right device. This guide is those numbers.

What I measured, and what I did not

Every latency and accuracy figure on this page is first-party, from one bench: my Ryzen 7 7800X3D / RTX 5080 (16GB) machine, running llama.cpp build 10507 (llama-server) on Windows, driver 610.88. The integration is a thin OpenAI-API client, so I benchmarked the exact request path it drives. Each request carried the same system prompt Home Assistant 2026.8.3 assembles (its default instructions, its device-control and area prompts, and a fixed 20-entity, 9-area “bench home” serialized the way async_get_exposed_entities does it) and the same Assist tools (HassTurnOn, HassTurnOff, HassLightSet, HassClimateSetTemperature, HassSetPosition, HassStopMoving, GetLiveContext), mirrored from the 2026.8.3 intent-tools source. Temperature 0, seed 42, five warm repetitions per cell, medians reported. The summary CSV is published, and so are the full receipts: the corpus and scorer plus every raw per-request record.

What I did not do is run Home Assistant itself on this bench. My HA-side steps below follow the official integration docs and the 2026.8.3 source rather than a click-through on this machine, and I say so here instead of pretending otherwise. Two consequences are worth knowing. First, my “command time” is the model-server leg only; Wyoming transport and Home Assistant’s own intent execution add a little on top. Second, my accuracy corpus was typed text at the API, and my Whisper timings used Piper-synthesized speech as input. That audio is clean and close-mic, which makes the STT accuracy numbers a ceiling rather than a promise about your echoey kitchen. Latency transfers. Perfect transcription rates don’t.

The architecture, corrected

Most early coverage of this integration calls it “built-in llama.cpp.” It is not, and the real design is better. The integration is a client. You run llama.cpp’s llama-server (or llama-cpp-python, or vLLM) anywhere on your network, and Home Assistant connects to its OpenAI-compatible endpoint, default http://localhost:8080/v1. The integration page lists it plainly, and the config flow asks for a URL, an optional API key, and then which model the server offers.

That client design is exactly what a homelab wants. Your Home Assistant box stays a low-power N100 or a HAOS VM on Proxmox, and the model runs wherever the compute already lives. A gaming PC. A workstation. The same host, if it has the memory. You can register several agents against different servers and models, each with its own instructions.

Voice needs two more pieces, both older and both solid. Whisper handles speech-to-text and Piper handles text-to-speech, wired in over the Wyoming protocol as add-ons or standalone containers. One subtlety the docs never connect. Home Assistant’s low-power STT default, Speech-to-Phrase, is a closed recognizer that matches against known sentence patterns, which is why it runs “under one second” on a Pi. An LLM agent exists precisely so you can say things no template anticipated, so it needs open-ended Whisper STT. Choosing an LLM brain silently commits you to the heavier STT path. That is where the hardware question lives.

Hardware tiers, honestly

I sell nothing here. A voice brain does not need an RTX 5080. This bench has one, so that is the GPU I can measure, but the tiers below are what the numbers support.

For sizing a specific model against a specific card, the VRAM calculator does the arithmetic. My measured VRAM residency per model is in the results table.

Step 1: Run llama-server where the compute is

Grab a llama.cpp release for your platform (CUDA build for NVIDIA), or use the official containers. Tool calling — the thing that lets the model actuate anything — runs through the Jinja chat-template engine. On the build I tested (10507) the server’s own --help reports --jinja as default-enabled, and I pass it explicitly anyway; older builds shipped it off, so if your agent chats but nothing actuates, --no-jinja or an ancient build is the first thing to rule out.

Windows or Linux binary:

llama-server -hf unsloth/Qwen3-4B-Instruct-2507-GGUF:Q4_K_M --host 0.0.0.0 --port 8080 -ngl 99 -c 8192 --jinja --api-key changeme

Docker with an NVIDIA GPU:

docker run --gpus all -p 8080:8080 -v /path/to/models:/models ghcr.io/ggml-org/llama.cpp:server-cuda -m /models/Qwen3-4B-Instruct-2507-Q4_K_M.gguf --host 0.0.0.0 --port 8080 --n-gpu-layers 99 -c 8192 --jinja --api-key changeme

Notes from running this, not from the README:

• -hf user/repo:QUANT downloads straight from Hugging Face and defaults to Q4_K_M. It saved me the download-and-point dance for every model in this guide.
• -ngl 99 offloads all layers to the GPU, and -ngl 0 forces CPU. -c 8192 gives the ~1,500-2,500-token Home Assistant prompt room to grow with your entity count.
• If you already run Ollama, llama-server can load most of Ollama’s blobs directly from ~/.ollama/models/blobs/. That is how I benched Qwen2.5. The exception that cost me an evening was gpt-oss — Ollama’s blob for it carries its own architecture tag and will not load in llama-server, so you need the upstream GGUF conversion. I hit the same wall building the Ollama vs llama.cpp comparison.
• Set --api-key on anything listening beyond localhost. llama-server itself warns at startup that it ships with CORS open and no key.

Sanity-check with curl http://SERVER_IP:8080/health before touching Home Assistant. It answers {"status":"ok"} once the model is loaded.

Step 2: Connect Home Assistant

On Home Assistant 2026.8 or newer (I verified against 2026.8.3, the latest release as I write this), go to Settings → Devices & services → Add integration → llama.cpp. The flow asks for the base URL — your server’s http://SERVER_IP:8080/v1 — plus the optional API key, then queries the server and prompts you to select the model. From the integration’s entry you add a conversation agent subentry, which holds the two settings that matter.

• Control Home Assistant grants the agent the Assist API. Without it you get a chatbot that can’t touch a switch.
• Instructions is the system prompt, templated. The default plus the Assist API’s own guidance is what I benchmarked. Resist stuffing it until you’ve read the latency section, because every token you add here is prefill the model re-reads on a cold start.

The agent only sees entities you expose, under Settings → Voice assistants → Expose. Expose deliberately. My bench home is 20 entities and its serialized state costs ~1,500-2,500 tokens of prompt depending on the model’s tokenizer. A 300-entity house pushes the prompt up with it, along with the cold-start prefill and the model’s targeting difficulty.

Then build the voice side in Settings → Voice assistants. Create an Assist pipeline, pick your llama.cpp agent as the conversation agent, Whisper for STT, Piper for TTS. Whisper and Piper install as add-ons on HAOS (defaults model: auto, and Piper’s en_US-lessac-medium voice — the exact voice in my TTS table), or as containers on any Docker host, including your GPU box:

docker run -d -p 10300:10300 -v whisper-data:/data rhasspy/wyoming-whisper --model small-int8 --language en

There is a GPU build of faster-whisper (lscr.io/linuxserver/faster-whisper:gpu) if you want STT on the CUDA box too; my STT table below shows what that buys. Point Home Assistant’s Wyoming integration at port 10300 and the pipeline picks it up. Wake word (openWakeWord/microWakeWord on a satellite or the Voice Preview Edition puck) is the one stage I have no hardware to measure, so I leave it to HA’s docs.

Model choice: the accuracy table I went looking for and had to build

Speed was never the interesting question. Every model below answers a warm command in under a second on a GPU. The question is whether the tool call that comes back does the right thing. I ran a fixed corpus of 24 typed commands against each model at temperature 0 — simple on/off, area-scoped commands (“turn off all the lights in the office”), parameters (brightness, color, thermostat), locks and covers, state queries, and five deliberately hard cases covering ambiguity, multi-area targeting, an implicit request, a command for a device that doesn’t exist, and a timer request when no timer tool is offered. Scoring is mechanical. Correct tool, correct target, correct parameters, against a published acceptance set. The corpus and scorer live in the bench receipts along with every raw response, so you can recompute any cell of this table.

The failures are more instructive than the percentages, because they are not random. Each model has a signature way of being wrong.

• Llama-3.2-3B floods, guesses, and crashes. Three of its 24 responses were rejected outright by llama-server’s tool-call parser — the model generated malformed tool output the server refused to hand back, which a Home Assistant user experiences as a failed interaction. The server log shows what it was trying to do on the kitchen-scoped “turn everything off” command: an enumeration of HassTurnOff calls marching far beyond the kitchen through the whole bench home, the Front Door Lock among them, until the parse died. Home Assistant maps turn-off on a lock to opening it; the only thing between that response and an open front door was a parser error. Asked the ambiguous “turn off the lights,” it silently guessed one room instead of asking which. It turned on the nonexistent disco ball. And its answer to “Who wrote The Hobbit?” was a GetLiveContext call with name: "The Hobbit" — it looked the novel up as if it were a device in the house (rejection number three). Fast, cheap, and the one model here I would not connect to real locks.
• Qwen3-4B drops the scope. Its area commands came back domain-correct but house-wide. “Turn off all the lights in the office” produced HassTurnOff {domain: [light]} with no area, which would darken the whole house. Its instinct on ambiguity is investigation (GetLiveContext) rather than actuation, which is at least safe.
• Qwen2.5-7B under-reaches. It targeted the entity literally named “Office Lights” and left the desk lamp on. Technically responsive, functionally half a command. On the ambiguous prompt it enumerated and switched off lights in five rooms instead of asking which one. On the disco ball it checked live context before telling me no such device exists — reasonable behavior that still scored as a miss under the strict no-tool answer key, where the 4B and 14B simply said so without calling anything.
• Qwen2.5-14B just… behaves. Its one miss in 24: asked what’s playing in the living room, it answered (correctly) straight from the entity snapshot in its prompt instead of calling GetLiveContext as its instructions demand. Right answer, wrong route. On a real install, where that snapshot goes stale between commands, the habit can bite. It nearly had a second miss, but that one was mine — told to turn everything off in the kitchen, it also switched off the kitchen speaker, which my original answer key had forgotten existed. The model was more thorough than my gold labels. I corrected the key (corpus v1.0.1) and scored it as the pass it was.
• gpt-oss-20b omits. It turned off the kitchen lights but forgot the dining room on the two-area command, and went off to inspect the thermostat instead of acting on “I’m cold.” Nothing dangerous, since its misses are omissions rather than overreach, but you feel its reasoning pass in the latency table below.

Home Assistant’s own prompt tells the model to ask for an area when a device-type command is ambiguous, and to prefer name-plus-domain targeting. The source is right here. The spread above is how differently five models follow the same instructions.

Two scoped notes. This is one fixed corpus at temperature 0 on one bench home, a reproducible comparison rather than a universal ranking, and the integration’s own default temperature is 0.7, which can only make behavior less deterministic. Also, a fine-tuned home-control model like Home-LLM exists precisely to attack this problem class; it predates the native integration and I did not bench it here.

Latency results

Command completion time is the number that matters for actuation: request sent to complete tool call back. First-token time flatters voice pipelines; a tool call is only actionable when it is finished. “Cold” is the first request after the server loads, which prefills the entire system prompt. “Warm” is the median of five repeats, where llama-server’s prompt cache means only your new sentence gets processed. That cache is the single most important performance fact in this whole stack. The same request that costs a second cold on the 14B costs 371 ms warm.

LLM leg, RTX 5080 (all layers offloaded):

Server VRAM is nvidia-smi total-used delta, before load versus after generation, on a card with an ~865 MiB desktop baseline. Every row above came from a verified-quiet capture window, with the baseline recorded per run in the receipts. This box runs other inference workloads, and two earlier contaminated captures were discarded rather than published.

CPU only: the numbers I am not publishing. I ran the same bench script with -ngl 0 on the 7800X3D and got decode rates near 1 token per second on a 3B. Absurd for this CPU, and I know exactly why. This bench box is temporarily on a single stick of DDR5 (a dead stick is out for RMA), and during the CPU window other inference workloads on the machine were sharing that one memory channel. CPU decode is memory-bandwidth-bound, so those runs measured my bus contention instead of the tier. Numbers measured badly are worse than no numbers, so the CPU rows stay out of the table until I can capture them on a quiet, dual-channel box; the dataset will grow the rows when they’re real. What holds regardless of my bad window: CPU prefill of a ~2,100-token Home Assistant prompt is a cold-start cost measured in seconds rather than milliseconds, warm commands ride the prompt cache exactly as on the GPU, and Home Assistant’s own docs put open-ended Whisper — the other CPU-hungry stage — at ~8 seconds on a Pi 4 versus under a second on a NUC. A resident server plus a 3-4B model is the only CPU-only configuration I would attempt, and I would set expectations in seconds.

STT (faster-whisper, beam 1, on the 24 synthesized commands, ~2-3s of audio each):

Two honest readings of that match column. First, the method-section reminder: the input audio is synthetically clean, so these are ceilings with a good mic, not field guarantees. Second, verbatim match is a harsh metric that cuts both ways. Several “misses” were harmless formatting (“Set the bedroom lights to 30%.” for “30 percent”, “10 minutes” for “ten minutes”), which the downstream LLM reads fine. The misses that matter are the garbled entity names: base-int8 heard “turn off the desk lamp” as “Turn off the disc lip,” and even small-int8 turned the disco ball into a “disco bowl.” A garbled name is a dead command no matter how good the model behind it is. (Base scoring below tiny here is a 24-sample quirk rather than a finding.)

TTS (Piper, en_US-lessac-medium, the add-on’s default voice): a non-problem, and I can now say so with numbers. On the 7800X3D’s CPU, Piper synthesized the short acknowledgment “Turned on the kitchen lights.” in 93 ms, a 2.9-second sentence in 101 ms, and a deliberately long 8.9-second status answer in 315 ms — a real-time factor of 0.03-0.07. The voice model loads once, in 1.5 s. Piper is the one pipeline component where the Raspberry-Pi-class advice in HA’s docs simply extends upward: their voice page calls it “optimized for the Raspberry Pi 4”, and a desktop CPU has an order of magnitude to spare.

Adding it up. A warm GPU pipeline — Whisper small on the card (99 ms), the 4B brain (144 ms), a short Piper acknowledgment (93 ms) — comes to roughly a third of a second of measured compute between end-of-speech and start-of-spoken-reply, before Wyoming transport and Home Assistant’s own execution overhead. That is inside the territory where an assistant feels instant. One honest asterisk. Like every OpenAI-style agent loop, the integration feeds the tool result back through the model to phrase the spoken confirmation (the loop is visible in its conversation-entity source), so the acknowledgment costs a second warm round trip. The actuation itself doesn’t wait for it. The lights are on before the voice says so.

Tuning notes from the bench

• Keep the server resident. Everything above says cold starts are the tax. If the server is on a box that sleeps, or something else evicts your VRAM, every “first command of the evening” pays load plus full prefill. This is also my argument against giant system prompts and exposing 300 entities.
• --jinja, again. It is the difference between tool calls and vibes. If your agent chats but nothing actuates, check this first.
• Watch the token budget. The integration defaults to a 3,000-token response cap; temperature 0.7 and top-p 1.0 live in the same file. My bench capped at 512, and the 3B’s whole-house flood ran that cap out mid-JSON before the parser rejected it. If you see failed responses in the logs, the model rather than the budget is usually the problem.
• Qwen3 pickers, use the Instruct variant. Qwen’s hybrid “thinking” builds spend tokens reasoning before every answer. I did not bench one, and for voice I would not; the 2507 Instruct line I benched answers directly. gpt-oss shows what reasoning costs even when it works: for the same warm command it generated 112 tokens where the 14B needed 30, which is most of why it sits at 585 ms warm even though its 212 tok/s decode is two and a half times the 14B’s.
• Streaming is a chat feature. The integration exposes a streaming toggle, and it helps perceived latency when the model reads you a paragraph. A tool call is atomic. Nothing actuates until the JSON is complete.

Why local, briefly

Because the audio of your kitchen never leaves your LAN, because it works when your ISP doesn’t, and because the per-query price is your electricity. I will not pretend the cloud assistants aren’t slicker at barge-in and follow-ups. They are, today. What you get here is a voice interface whose brain, vocabulary, and failure modes are yours to choose, measure, and fix. The accuracy table is exactly that. Pick the brain whose mistakes you can live with, and wire it to locks only when the answer is “none of them.”

Sources

• Home Assistant 2026.8 release post (llama.cpp integration announcement): https://www.home-assistant.io/blog/2026/08/05/release-20268/
• llama.cpp integration docs: https://www.home-assistant.io/integrations/llama_cpp/
• HA 2026.8.3 source — Assist tools + prompts: https://github.com/home-assistant/core/blob/2026.8.3/homeassistant/components/intent/llm.py · exposed-entity serialization + GetLiveContext: https://github.com/home-assistant/core/blob/2026.8.3/homeassistant/components/homeassistant/llm.py · the integration (defaults in const.py): https://github.com/home-assistant/core/tree/2026.8.3/homeassistant/components/llama_cpp
• Local voice pipeline + hardware guidance: https://www.home-assistant.io/voice_control/voice_remote_local_assistant/
• Whisper / Piper add-on defaults: https://github.com/home-assistant/addons/tree/master/whisper · https://github.com/home-assistant/addons/tree/master/piper
• wyoming-faster-whisper container: https://github.com/rhasspy/wyoming-faster-whisper · GPU build: https://docs.linuxserver.io/images/docker-faster-whisper/
• llama.cpp docker images: https://github.com/ggml-org/llama.cpp/blob/master/docs/docker.md
• llama-server flag reference (-hf, --jinja default): https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md
• Bench dataset (CSV, CC BY 4.0): /data/ha-voice-brain-bench-2026-08.csv — method ha-voice-brain-v1, corpus v1.0.1, captured 2026-08-25/26. Full receipts (corpus, scorer, raw per-request records, rescore output): /data/ha-voice-bench/README.md.