Quick answer

Run Ollama in Docker bound to loopback (-p 127.0.0.1:11434:11434) for GPU and CPU host tests so unauthenticated port 11434 never opens on 0.0.0.0. On Windows Docker Desktop, keep the Windows NVIDIA driver on WSL2; on Linux Engine, configure NVIDIA Container Toolkit. Reserve LAN or remote access for private networks or an authenticated reverse proxy.

Contents

At a glance

Class
Tutorial

Provenance

  • Aug 15, 2026Published
  • Nov 13, 2026Last substantive update

By LK Wood IV · Published 2026-08-15 · Updated 2026-11-13 · ~10 min read · St. Louis County, MO

If you are on WSL2, the most common first move is the wrong one, and it feels like progress while you make it.

The presence of nvcc inside your distro proves you have a compiler. It tells you nothing about whether Docker can pass through the GPU.

NVIDIA’s CUDA on WSL guide separates the Windows display driver from development tools inside Linux.

WSL gets its CUDA driver interface from Windows. Do not install the Linux cuda-drivers or nvidia-open packages inside the distro. There is a version distinction here: starting with CUDA 13.4 the generic cuda meta-package no longer installs a driver. The compiler toolkit is still unnecessary for the prebuilt Ollama image.

You do not need a driver inside WSL. You already have one. It came from Windows.

The two toolkits people conflate

This is the root of it. The naming is genuinely unhelpful:

PackageWhat it doesNeeded for Ollama in Docker?
CUDA Toolkit (nvcc, headers)compiles CUDA applicationsNo
NVIDIA Container Toolkit (nvidia-container-toolkit)lets Docker hand a GPU to a containerYes

They share a brand. Nothing else. nvcc --version printing happily tells you precisely nothing about whether Docker can reach your GPU — I have watched that exact screenshot get posted as evidence in bug reports where the real failure was that the container toolkit was never installed.

You want the second one.

On Debian or Ubuntu running its own Docker Engine, install the toolkit there. If your WSL terminal is a client of Docker Desktop for Windows, skip these Linux daemon steps. Docker Desktop uses its WSL2 backend; the GPU driver belongs on Windows.

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

For a Linux Docker Engine you manage, follow the NVIDIA runtime configuration. For the standard rootful engine, run sudo nvidia-ctk runtime configure --runtime=docker, then restart that Docker daemon when interrupting containers is safe. The linked guide has different commands for rootless Docker.

Running it

Every host-bound example below binds port 11434 explicitly to 127.0.0.1:11434:11434 rather than copying the bare -p 11434:11434 flag from Ollama’s Docker quickstart. That single prefix keeps an unauthenticated model API confined to your machine while you set up and test.

NVIDIA GPU (docker run, loopback-only):

docker run -d --gpus=all \
  -v ollama:/root/.ollama \
  -p 127.0.0.1:11434:11434 \
  --name ollama ollama/ollama

docker exec -it ollama ollama run llama3.2

CPU-only baseline (docker run, loopback-only):

docker run -d \
  -v ollama:/root/.ollama \
  -p 127.0.0.1:11434:11434 \
  --name ollama ollama/ollama

docker exec -it ollama ollama run llama3.2

Running the CPU container first is a fast sanity check when debugging a fresh host: it verifies Docker storage, loopback port binding, and model pulls before introducing --gpus=all, so any failure after recreating the container with GPU flags isolates the NVIDIA runtime layer.

AMD ROCm (docker run, loopback-only):

docker run -d --device /dev/kfd --device /dev/dri \
  -v ollama:/root/.ollama \
  -p 127.0.0.1:11434:11434 \
  --name ollama ollama/ollama:rocm

Note the volume is named (-v ollama:/root/.ollama).

/root/.ollama holds your downloaded models, and those are large. Recreating the container when switching from CPU to --gpus=all (docker stop ollama && docker rm ollama, then re-running with -v ollama:/root/.ollama) keeps your weights intact, whereas an anonymous volume is how people re-download forty gigabytes after a cleanup — see which prune command eats data for why that happens.

Where 11434 really listens

There is bad advice in circulation. It says Ollama binds 127.0.0.1 by default, so publishing the port is safe.

That is true of a native host install. It is not true of the official container image. Ollama’s image sets OLLAMA_HOST=0.0.0.0:11434 so processes outside the container namespace can reach it, and Docker’s port-publishing documentation explains what happens next: writing -p 11434:11434 without an IP binds 0.0.0.0:11434 across all host interfaces and inserts iptables/nftables forwarding rules ahead of standard ufw or firewalld INPUT chains. Anyone on your Wi-Fi, LAN, or an unfirewalled VPS public IP can pull models, delete weights, or saturate your GPU because Ollama’s API has no built-in authentication.

For local CLI tools, IDE extensions, and scripts on the Docker host, keep 127.0.0.1:11434:11434. On Linux Docker Engine releases older than 28.0, Docker also documents a direct-routing localhost exception on adjacent LANs; run Docker Engine 28+ or omit ports: altogether if only sibling containers talk to Ollama.

Verify the active socket binding right after starting the container:

docker port ollama 11434/tcp
curl --fail http://127.0.0.1:11434/api/tags

The first command must report 127.0.0.1:11434 (never 0.0.0.0:11434 or [::]:11434). The second lists installed models; an empty {"models":[]} JSON response on a fresh volume is expected. If docker port shows 0.0.0.0 or [::], stop and remove the container, then recreate it with -p 127.0.0.1:11434:11434 attached to the same ollama volume.

LAN and remote access: private network or authenticated reverse proxy

When a laptop, workstation, or phone off the Docker host genuinely needs your Ollama instance, expose it deliberately instead of dropping 127.0.0.1 from -p:

  1. Same-host web UIs and agents (Open WebUI, n8n): omit ports: on the ollama service completely and connect over an internal Docker Compose network at http://ollama:11434. Authentication happens at the web UI login screen rather than the raw model port.
  2. Personal devices over a private network (SSH tunnel or Tailscale/WireGuard): keep Ollama bound to -p 127.0.0.1:11434:11434 on the server and forward the port over authenticated SSH (ssh -N -L 11434:127.0.0.1:11434 user@docker-host), or bind explicitly to the server’s private tailscale0 IPv4 address (-p 100.x.y.z:11434:11434) restricted by tailnet ACLs so LAN and WAN interfaces never see the port.
  3. Shared LAN or HTTPS remote API clients (authenticated reverse proxy): keep Ollama bound to 127.0.0.1:11434:11434 and put Caddy, Nginx, or Traefik in front of it terminating TLS and enforcing an Authorization: Bearer token, HTTP basic auth, or OIDC policy before forwarding to http://127.0.0.1:11434 — the exact pattern Ollama’s API authentication guide recommends because client SDKs accept custom headers while ollama serve itself performs no auth check.

Compose, where --gpus does not exist

This is the second thing that costs people an evening.

--gpus=all is a docker run flag. Compose does not accept it as a service key, and the validation error does not point you toward deploy.resources.reservations.devices. Both Compose templates below keep the explicit "127.0.0.1:11434:11434" loopback quote so YAML never exposes 0.0.0.0 (and never parses bare colon numbers as sexagesimal integers).

NVIDIA GPU compose.yaml (loopback-only):

services:
  ollama:
    image: ollama/ollama
    container_name: ollama
    volumes:
      - ollama:/root/.ollama
    ports:
      - "127.0.0.1:11434:11434"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    restart: unless-stopped

volumes:
  ollama:

CPU-only compose.yaml (loopback-only):

services:
  ollama:
    image: ollama/ollama
    container_name: ollama
    volumes:
      - ollama:/root/.ollama
    ports:
      - "127.0.0.1:11434:11434"
    restart: unless-stopped

volumes:
  ollama:

Two rules inside the GPU devices block, per Docker Compose’s GPU specification, are easy to trip:

  • capabilities: [gpu] is mandatory. Docker’s Compose specification states that omitting capabilities errors on deployment even though it looks optional.
  • count and device_ids are mutually exclusive. Specify count: all (or an integer on Linux) or device_ids, never both in the same entry.

Recent Compose releases also accept a shorter service-level gpus: all key, though deploy.resources.reservations.devices remains the widest baseline across distro-packaged Compose plugins and Portainer/Dockge stacks.

The WSL2 restriction nobody mentions

On WSL2, NVIDIA documents that only --gpus all is supported.

You cannot filter by index or UUID.

So a compose file pinning device_ids: ['0'] is documented as not workable under WSL2, even though the identical file is fine on native Linux. If you have two GPUs in a Windows box and were planning to give one to Ollama and keep the other for something else, that plan does not survive contact with WSL2.

That single limitation is the strongest argument I know for putting the LLM host on native Linux rather than WSL2, assuming you have the choice, and it is the kind of constraint that only shows up after you have already built the thing around the assumption that a GPU is a GPU.

Adding Open WebUI

The obvious next step is a browser interface. Open WebUI’s official stack runs it alongside Ollama, and their :cuda image tag is documented as Nvidia GPU support, to be paired with --gpus all.

Open WebUI’s two-service Compose stack publishes the UI and leaves the Ollama service without a published host port. The UI container can reach the API by service name on their shared network. That keeps the raw API off the host’s published interfaces. On native Linux, the host itself may still reach the container’s bridge IP; see Docker’s bridge-network behavior.

If only the UI needs inference, keep the backend unpublished and use http://ollama:11434 from the UI container. If a host application needs inference too, use the loopback mapping above. For a different computer you control, forward a local port through authenticated SSH to the Docker host’s loopback address; do not expose the raw API on your router.

Past a chat UI, the next layer most people add is tool access via MCP servers. Before connecting a stack of them, know the resident cost: we measured 11 popular MCP servers at 178-422MB of RAM each on the same box class this guide targets.

Verifying the GPU is doing the work

The failure mode here is quiet. Everything runs. It just runs slowly, on the CPU, while you assume otherwise.

docker exec -it ollama nvidia-smi
docker exec ollama ollama ps

Run nvidia-smi inside the container first to confirm Docker passed the GPU device nodes through, then load a model and check ollama ps while it is resident. Compare what both commands report:

What you seeLayer involvedFix
docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]]Docker Engine runtime registrationOn native Linux, install nvidia-container-toolkit, run sudo nvidia-ctk runtime configure --runtime=docker, and restart Docker. On Windows, enable the WSL2 backend in Docker Desktop and install the Windows NVIDIA driver.
nvidia-smi works inside ollama, ollama ps shows 100% GPU under PROCESSORHealthy GPU offloadFull model and KV cache fit in VRAM; verify docker port ollama 11434/tcp still prints 127.0.0.1:11434.
nvidia-smi works inside ollama, ollama ps shows 100% CPU or 45%/55% CPU/GPUModel / context VRAM fit (OLLAMA_NUM_CTX)The quantised weights plus context window exceed available VRAM and spilled to system RAM; pick a smaller quant or check local LLMs by GPU VRAM and the Ollama FAQ.

What I have not tested

I have not run the ROCm image. The command above comes from Ollama’s documentation and I have no AMD card here to confirm it against, so treat that line as documented rather than verified by me.

I have not tested the newer top-level gpus: Compose attribute, and Compose version support for it is exactly the sort of thing that varies by distro package.

I have not benchmarked WSL2 against native Linux for inference throughput. I believe native is better and I have not measured it.

The wrapper question, though, I have measured: Ollama against bare llama.cpp on the same RTX 5080 put the decode gap on dense models at 2-6%, with the whole serving stack — HTTP, sampling, model management — costing about 13% against the raw engine loop. None of the dramatic slowdowns the forums promise showed up.

What getting it wrong costs

Installing a second driver inside WSL can disrupt the Windows-provided interface. Verify which engine runs your container before changing anything in either operating system.

The -p 11434:11434 mistake costs more, and it costs it quietly. An unauthenticated inference API on a public address does not announce itself. It just becomes somebody else’s free GPU until you notice the electricity bill or the fan noise.

Frequently asked questions

Do I need to install CUDA inside WSL2 for GPU acceleration?
Ollama’s prebuilt image does not need a CUDA development toolkit in the WSL distro. Keep the GPU driver on Windows. Do not install driver-bearing CUDA meta-packages inside WSL. Use Docker Desktop’s WSL2 GPU support, or configure the NVIDIA Container Toolkit in the environment that runs your own Docker Engine.
What is the difference between the CUDA Toolkit and the NVIDIA Container Toolkit?
The CUDA Toolkit gives you nvcc and headers for COMPILING CUDA applications. The NVIDIA Container Toolkit is what lets Docker hand a GPU to a container, and it is what --gpus=all depends on. Checking nvcc --version tells you nothing about whether Docker can reach your GPU. They are unrelated packages that happen to share a brand.
What is the docker run command for Ollama with a GPU?
For NVIDIA GPU acceleration bound safely to host loopback, run docker run -d --gpus=all -v ollama:/root/.ollama -p 127.0.0.1:11434:11434 --name ollama ollama/ollama. To test a CPU-only baseline first, run docker run -d -v ollama:/root/.ollama -p 127.0.0.1:11434:11434 --name ollama ollama/ollama. Both commands keep port 11434 off 0.0.0.0; if only sibling containers like Open WebUI need Ollama, omit -p completely and connect to http://ollama:11434 on a shared Compose network.
Why does --gpus=all not work in my compose file?
Because it is a docker run flag, not a Compose service key. In Compose you either use a deploy.resources.reservations.devices block or the newer top-level gpus attribute. In the reservations block, capabilities is not optional — Docker’s docs state that omitting it errors on deployment — and count and device_ids are mutually exclusive.
Can I pick a specific GPU on WSL2?
No. NVIDIA documents that under WSL2 only --gpus all is supported, so you cannot filter by index or UUID. A compose file pinning device_ids on a multi-GPU WSL2 box is documented as not workable. On native Linux the filtering works normally.
Does Ollama in Docker listen only on localhost?
Not unless you explicitly bind -p 127.0.0.1:11434:11434. Ollama’s official container image sets OLLAMA_HOST=0.0.0.0:11434, and Docker’s port-publishing rules bind bare -p 11434:11434 to 0.0.0.0:11434 across host interfaces while bypassing standard ufw or firewalld INPUT rules. Always bind 127.0.0.1:11434:11434 for local clients, and reserve LAN or remote access for a private overlay network or an authenticated reverse proxy.
How do I expose Ollama in Docker to my LAN or remote devices safely?
Never publish bare -p 11434:11434 on 0.0.0.0, because Ollama’s API has no built-in authentication and allows model pulls, deletes, and GPU inference by anyone who can reach the port. Instead, keep Ollama bound to 127.0.0.1:11434:11434 and place an authenticated HTTPS reverse proxy (Caddy, Nginx, or Traefik checking Bearer tokens or SSO) in front of it, or tunnel over an authenticated private network such as SSH port forwarding or Tailscale.
Why can I not reach port 11434 in the Open WebUI stack?
In the standard two-service stack, Open WebUI reaches Ollama at http://ollama:11434 on their shared Compose network without publishing port 11434 on the host. If a host CLI or editor plugin also needs the API, add "127.0.0.1:11434:11434" under ports: on the ollama service and verify the loopback binding with docker compose ps.
How much VRAM do I need?
That is the question that decides everything else, and it depends on the model and quantisation rather than on Ollama. It is worth working out before buying anything, because a model that does not fit in VRAM spills to system RAM and the speed difference is not subtle.

Corrections

Last updated
Methodology
See our methodology for research and review standards.
Corrections
Spotted an error? Email a correction with the affected line and what you found. The draft includes this page’s address. Verified changes are recorded here. Contact details.

About the author

Written by Lowell K. Wood IV, who builds and runs TechFuelHQ from St. Louis, Missouri.