Run Ollama in Docker bound to loopback (-p 127.0.0.1:11434:11434) for GPU and CPU host tests so unauthenticated port 11434 never opens on 0.0.0.0. On Windows Docker Desktop, keep the Windows NVIDIA driver on WSL2; on Linux Engine, configure NVIDIA Container Toolkit. Reserve LAN or remote access for private networks or an authenticated reverse proxy.
Contents
At a glance
- Class
- Tutorial
Provenance
- Aug 15, 2026Published
- Nov 13, 2026Last substantive update
By LK Wood IV · Published 2026-08-15 · Updated 2026-11-13 · ~10 min read · St. Louis County, MO
If you are on WSL2, the most common first move is the wrong one, and it feels like progress while you make it.
The presence of nvcc inside your distro proves you have a compiler. It tells you nothing about whether Docker can pass through the GPU.
NVIDIA’s CUDA on WSL guide separates the Windows display driver from development tools inside Linux.
WSL gets its CUDA driver interface from Windows. Do not install the Linux cuda-drivers or nvidia-open packages inside the distro. There is a version distinction here: starting with CUDA 13.4 the generic cuda meta-package no longer installs a driver. The compiler toolkit is still unnecessary for the prebuilt Ollama image.
You do not need a driver inside WSL. You already have one. It came from Windows.
The two toolkits people conflate
This is the root of it. The naming is genuinely unhelpful:
| Package | What it does | Needed for Ollama in Docker? |
|---|---|---|
CUDA Toolkit (nvcc, headers) | compiles CUDA applications | No |
NVIDIA Container Toolkit (nvidia-container-toolkit) | lets Docker hand a GPU to a container | Yes |
They share a brand. Nothing else. nvcc --version printing happily tells you precisely nothing about whether Docker can reach your GPU — I have watched that exact screenshot get posted as evidence in bug reports where the real failure was that the container toolkit was never installed.
You want the second one.
On Debian or Ubuntu running its own Docker Engine, install the toolkit there. If your WSL terminal is a client of Docker Desktop for Windows, skip these Linux daemon steps. Docker Desktop uses its WSL2 backend; the GPU driver belongs on Windows.
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
| sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
| sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
| sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
For a Linux Docker Engine you manage, follow the NVIDIA runtime configuration. For the standard rootful engine, run sudo nvidia-ctk runtime configure --runtime=docker, then restart that Docker daemon when interrupting containers is safe. The linked guide has different commands for rootless Docker.
Running it
Every host-bound example below binds port 11434 explicitly to 127.0.0.1:11434:11434 rather than copying the bare -p 11434:11434 flag from Ollama’s Docker quickstart. That single prefix keeps an unauthenticated model API confined to your machine while you set up and test.
NVIDIA GPU (docker run, loopback-only):
docker run -d --gpus=all \
-v ollama:/root/.ollama \
-p 127.0.0.1:11434:11434 \
--name ollama ollama/ollama
docker exec -it ollama ollama run llama3.2
CPU-only baseline (docker run, loopback-only):
docker run -d \
-v ollama:/root/.ollama \
-p 127.0.0.1:11434:11434 \
--name ollama ollama/ollama
docker exec -it ollama ollama run llama3.2
Running the CPU container first is a fast sanity check when debugging a fresh host: it verifies Docker storage, loopback port binding, and model pulls before introducing --gpus=all, so any failure after recreating the container with GPU flags isolates the NVIDIA runtime layer.
AMD ROCm (docker run, loopback-only):
docker run -d --device /dev/kfd --device /dev/dri \
-v ollama:/root/.ollama \
-p 127.0.0.1:11434:11434 \
--name ollama ollama/ollama:rocm
Note the volume is named (-v ollama:/root/.ollama).
/root/.ollama holds your downloaded models, and those are large. Recreating the container when switching from CPU to --gpus=all (docker stop ollama && docker rm ollama, then re-running with -v ollama:/root/.ollama) keeps your weights intact, whereas an anonymous volume is how people re-download forty gigabytes after a cleanup — see which prune command eats data for why that happens.
Where 11434 really listens
There is bad advice in circulation. It says Ollama binds 127.0.0.1 by default, so publishing the port is safe.
That is true of a native host install. It is not true of the official container image. Ollama’s image sets OLLAMA_HOST=0.0.0.0:11434 so processes outside the container namespace can reach it, and Docker’s port-publishing documentation explains what happens next: writing -p 11434:11434 without an IP binds 0.0.0.0:11434 across all host interfaces and inserts iptables/nftables forwarding rules ahead of standard ufw or firewalld INPUT chains. Anyone on your Wi-Fi, LAN, or an unfirewalled VPS public IP can pull models, delete weights, or saturate your GPU because Ollama’s API has no built-in authentication.
For local CLI tools, IDE extensions, and scripts on the Docker host, keep 127.0.0.1:11434:11434. On Linux Docker Engine releases older than 28.0, Docker also documents a direct-routing localhost exception on adjacent LANs; run Docker Engine 28+ or omit ports: altogether if only sibling containers talk to Ollama.
Verify the active socket binding right after starting the container:
docker port ollama 11434/tcp
curl --fail http://127.0.0.1:11434/api/tags
The first command must report 127.0.0.1:11434 (never 0.0.0.0:11434 or [::]:11434). The second lists installed models; an empty {"models":[]} JSON response on a fresh volume is expected. If docker port shows 0.0.0.0 or [::], stop and remove the container, then recreate it with -p 127.0.0.1:11434:11434 attached to the same ollama volume.
LAN and remote access: private network or authenticated reverse proxy
When a laptop, workstation, or phone off the Docker host genuinely needs your Ollama instance, expose it deliberately instead of dropping 127.0.0.1 from -p:
- Same-host web UIs and agents (Open WebUI, n8n): omit
ports:on theollamaservice completely and connect over an internal Docker Compose network athttp://ollama:11434. Authentication happens at the web UI login screen rather than the raw model port. - Personal devices over a private network (SSH tunnel or Tailscale/WireGuard): keep Ollama bound to
-p 127.0.0.1:11434:11434on the server and forward the port over authenticated SSH (ssh -N -L 11434:127.0.0.1:11434 user@docker-host), or bind explicitly to the server’s privatetailscale0IPv4 address (-p 100.x.y.z:11434:11434) restricted by tailnet ACLs so LAN and WAN interfaces never see the port. - Shared LAN or HTTPS remote API clients (authenticated reverse proxy): keep Ollama bound to
127.0.0.1:11434:11434and put Caddy, Nginx, or Traefik in front of it terminating TLS and enforcing anAuthorization: Bearertoken, HTTP basic auth, or OIDC policy before forwarding tohttp://127.0.0.1:11434— the exact pattern Ollama’s API authentication guide recommends because client SDKs accept custom headers whileollama serveitself performs no auth check.
Compose, where --gpus does not exist
This is the second thing that costs people an evening.
--gpus=all is a docker run flag. Compose does not accept it as a service key, and the validation error does not point you toward deploy.resources.reservations.devices. Both Compose templates below keep the explicit "127.0.0.1:11434:11434" loopback quote so YAML never exposes 0.0.0.0 (and never parses bare colon numbers as sexagesimal integers).
NVIDIA GPU compose.yaml (loopback-only):
services:
ollama:
image: ollama/ollama
container_name: ollama
volumes:
- ollama:/root/.ollama
ports:
- "127.0.0.1:11434:11434"
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
restart: unless-stopped
volumes:
ollama:
CPU-only compose.yaml (loopback-only):
services:
ollama:
image: ollama/ollama
container_name: ollama
volumes:
- ollama:/root/.ollama
ports:
- "127.0.0.1:11434:11434"
restart: unless-stopped
volumes:
ollama:
Two rules inside the GPU devices block, per Docker Compose’s GPU specification, are easy to trip:
capabilities: [gpu]is mandatory. Docker’s Compose specification states that omittingcapabilitieserrors on deployment even though it looks optional.countanddevice_idsare mutually exclusive. Specifycount: all(or an integer on Linux) ordevice_ids, never both in the same entry.
Recent Compose releases also accept a shorter service-level gpus: all key, though deploy.resources.reservations.devices remains the widest baseline across distro-packaged Compose plugins and Portainer/Dockge stacks.
The WSL2 restriction nobody mentions
On WSL2, NVIDIA documents that only --gpus all is supported.
You cannot filter by index or UUID.
So a compose file pinning device_ids: ['0'] is documented as not workable under WSL2, even though the identical file is fine on native Linux. If you have two GPUs in a Windows box and were planning to give one to Ollama and keep the other for something else, that plan does not survive contact with WSL2.
That single limitation is the strongest argument I know for putting the LLM host on native Linux rather than WSL2, assuming you have the choice, and it is the kind of constraint that only shows up after you have already built the thing around the assumption that a GPU is a GPU.
Adding Open WebUI
The obvious next step is a browser interface. Open WebUI’s official stack runs it alongside Ollama, and their :cuda image tag is documented as Nvidia GPU support, to be paired with --gpus all.
Open WebUI’s two-service Compose stack publishes the UI and leaves the Ollama service without a published host port. The UI container can reach the API by service name on their shared network. That keeps the raw API off the host’s published interfaces. On native Linux, the host itself may still reach the container’s bridge IP; see Docker’s bridge-network behavior.
If only the UI needs inference, keep the backend unpublished and use http://ollama:11434 from the UI container. If a host application needs inference too, use the loopback mapping above. For a different computer you control, forward a local port through authenticated SSH to the Docker host’s loopback address; do not expose the raw API on your router.
Past a chat UI, the next layer most people add is tool access via MCP servers. Before connecting a stack of them, know the resident cost: we measured 11 popular MCP servers at 178-422MB of RAM each on the same box class this guide targets.
Verifying the GPU is doing the work
The failure mode here is quiet. Everything runs. It just runs slowly, on the CPU, while you assume otherwise.
docker exec -it ollama nvidia-smi
docker exec ollama ollama ps
Run nvidia-smi inside the container first to confirm Docker passed the GPU device nodes through, then load a model and check ollama ps while it is resident. Compare what both commands report:
| What you see | Layer involved | Fix |
|---|---|---|
docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]] | Docker Engine runtime registration | On native Linux, install nvidia-container-toolkit, run sudo nvidia-ctk runtime configure --runtime=docker, and restart Docker. On Windows, enable the WSL2 backend in Docker Desktop and install the Windows NVIDIA driver. |
nvidia-smi works inside ollama, ollama ps shows 100% GPU under PROCESSOR | Healthy GPU offload | Full model and KV cache fit in VRAM; verify docker port ollama 11434/tcp still prints 127.0.0.1:11434. |
nvidia-smi works inside ollama, ollama ps shows 100% CPU or 45%/55% CPU/GPU | Model / context VRAM fit (OLLAMA_NUM_CTX) | The quantised weights plus context window exceed available VRAM and spilled to system RAM; pick a smaller quant or check local LLMs by GPU VRAM and the Ollama FAQ. |
What I have not tested
I have not run the ROCm image. The command above comes from Ollama’s documentation and I have no AMD card here to confirm it against, so treat that line as documented rather than verified by me.
I have not tested the newer top-level gpus: Compose attribute, and Compose version support for it is exactly the sort of thing that varies by distro package.
I have not benchmarked WSL2 against native Linux for inference throughput. I believe native is better and I have not measured it.
The wrapper question, though, I have measured: Ollama against bare llama.cpp on the same RTX 5080 put the decode gap on dense models at 2-6%, with the whole serving stack — HTTP, sampling, model management — costing about 13% against the raw engine loop. None of the dramatic slowdowns the forums promise showed up.
What getting it wrong costs
Installing a second driver inside WSL can disrupt the Windows-provided interface. Verify which engine runs your container before changing anything in either operating system.
The -p 11434:11434 mistake costs more, and it costs it quietly. An unauthenticated inference API on a public address does not announce itself. It just becomes somebody else’s free GPU until you notice the electricity bill or the fan noise.
Frequently asked questions
Do I need to install CUDA inside WSL2 for GPU acceleration?
What is the difference between the CUDA Toolkit and the NVIDIA Container Toolkit?
--gpus=all depends on. Checking nvcc --version tells you nothing about whether Docker can reach your GPU. They are unrelated packages that happen to share a brand.What is the docker run command for Ollama with a GPU?
docker run -d --gpus=all -v ollama:/root/.ollama -p 127.0.0.1:11434:11434 --name ollama ollama/ollama. To test a CPU-only baseline first, run docker run -d -v ollama:/root/.ollama -p 127.0.0.1:11434:11434 --name ollama ollama/ollama. Both commands keep port 11434 off 0.0.0.0; if only sibling containers like Open WebUI need Ollama, omit -p completely and connect to http://ollama:11434 on a shared Compose network.Why does --gpus=all not work in my compose file?
Can I pick a specific GPU on WSL2?
--gpus all is supported, so you cannot filter by index or UUID. A compose file pinning device_ids on a multi-GPU WSL2 box is documented as not workable. On native Linux the filtering works normally.Does Ollama in Docker listen only on localhost?
-p 127.0.0.1:11434:11434. Ollama’s official container image sets OLLAMA_HOST=0.0.0.0:11434, and Docker’s port-publishing rules bind bare -p 11434:11434 to 0.0.0.0:11434 across host interfaces while bypassing standard ufw or firewalld INPUT rules. Always bind 127.0.0.1:11434:11434 for local clients, and reserve LAN or remote access for a private overlay network or an authenticated reverse proxy.How do I expose Ollama in Docker to my LAN or remote devices safely?
-p 11434:11434 on 0.0.0.0, because Ollama’s API has no built-in authentication and allows model pulls, deletes, and GPU inference by anyone who can reach the port. Instead, keep Ollama bound to 127.0.0.1:11434:11434 and place an authenticated HTTPS reverse proxy (Caddy, Nginx, or Traefik checking Bearer tokens or SSO) in front of it, or tunnel over an authenticated private network such as SSH port forwarding or Tailscale.Why can I not reach port 11434 in the Open WebUI stack?
"127.0.0.1:11434:11434" under ports: on the ollama service and verify the loopback binding with docker compose ps.How much VRAM do I need?
Corrections
- Last updated
- Methodology
- See our methodology for research and review standards.
- Corrections
- Spotted an error? Email a correction with the affected line and what you found. The draft includes this page’s address. Verified changes are recorded here. Contact details.