Ollama in Docker with GPU: Stop Installing CUDA Inside WSL2 (2026)

By LK Wood IV · 2026-08-15 · ~9 min read · St. Louis County, MO

If you are on WSL2, the most common first move is the wrong one, and it feels like progress while you make it.

People open the distro, run apt install cuda or cuda-drivers, check nvcc --version, see a version string, and treat the prerequisite as handled.

NVIDIA’s own CUDA-on-WSL documentation warns against exactly this. More than once. One of those warnings is in capitals.

Here is why it is not merely unnecessary but harmful. Under WSL2 the Windows driver is projected into the distro as a libcuda.so stub. That stub is the GPU access path. The cuda, cuda-12-x and cuda-drivers meta-packages pull in a Linux NVIDIA driver, which installs over that stub and breaks the thing that was working.

You do not need a driver inside WSL. You already have one. It came from Windows.

The two toolkits people conflate

This is the root of it. The naming is genuinely unhelpful:

They share a brand. Nothing else. nvcc --version printing happily tells you precisely nothing about whether Docker can reach your GPU — I have watched that exact screenshot get posted as evidence in bug reports where the real failure was that the container toolkit was never installed.

You want the second one.

On Debian or Ubuntu:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

This is the same package on native Linux and inside WSL2. The only difference is where you stop: on WSL2 that command is the end of it, whereas on native Linux you also need a real NVIDIA driver underneath, which is the step WSL2 users are trying to replicate when they break their stub.

Running it

docker run -d --gpus=all \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  --name ollama ollama/ollama

docker exec -it ollama ollama run llama3.2

Drop --gpus=all and you get a working CPU install. That is a reasonable first move: it confirms the rest of your setup before you start fighting the GPU layer, and it means a later failure has exactly one new variable in it.

For AMD, the image and the flags both change:

docker run -d --device /dev/kfd --device /dev/dri \
  -v ollama:/root/.ollama -p 11434:11434 \
  --name ollama ollama/ollama:rocm

Note the volume is named.

/root/.ollama holds your downloaded models, and those are large. An anonymous volume there is how people re-download forty gigabytes after a cleanup — see which prune command eats data for why that happens.

Where 11434 really listens

There is bad advice in circulation. It says Ollama binds 127.0.0.1 by default, so publishing the port is safe.

That is true of a native install. It is not true of this image. The official container sets OLLAMA_HOST=0.0.0.0:11434 in its Dockerfile, so inside the container it listens on everything, and whatever you publish with -p is genuinely reachable.

Ollama’s API has no authentication. -p 11434:11434 on a machine with a public IP publishes an unauthenticated inference endpoint on someone else’s hardware budget. Bind it to localhost explicitly if you want it host-only:

-p 127.0.0.1:11434:11434

Compose, where –gpus does not exist

This is the second thing that costs people an evening.

--gpus=all is a docker run flag. Compose does not accept it as a service key, and the error you get does not point anywhere useful.

services:
  ollama:
    image: ollama/ollama
    volumes:
      - ollama:/root/.ollama
    ports:
      - "127.0.0.1:11434:11434"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

volumes:
  ollama:

Two rules inside that block, both documented and both easy to trip:

• capabilities is not optional. Docker’s docs state that omitting it errors on service deployment. It looks like decoration and it is required.
• count and device_ids are mutually exclusive. Pick one, or it errors.

There is also a newer top-level gpus: attribute if your Compose version has it, which is less verbose for the common case.

The WSL2 restriction nobody mentions

On WSL2, NVIDIA documents that only --gpus all is supported.

You cannot filter by index or UUID.

So a compose file pinning device_ids: ['0'] is documented as not workable under WSL2, even though the identical file is fine on native Linux. If you have two GPUs in a Windows box and were planning to give one to Ollama and keep the other for something else, that plan does not survive contact with WSL2.

That single limitation is the strongest argument I know for putting the LLM host on native Linux rather than WSL2, assuming you have the choice, and it is the kind of constraint that only shows up after you have already built the thing around the assumption that a GPU is a GPU.

Adding Open WebUI

The obvious next step is a browser interface. Open WebUI’s official stack runs it alongside Ollama, and their :cuda image tag is documented as Nvidia GPU support, to be paired with --gpus all.

One deliberate default worth understanding before you “fix” it: in Open WebUI’s official compose stack, the ollama service publishes no host port at all. 11434 is reachable on the Compose network and nowhere else. That is not an oversight. It means the only thing exposed is the UI, which has accounts, rather than the raw API, which does not.

If you genuinely need the API from the host, they ship a separate api overlay for that. Use it rather than editing the base file, so the reason stays visible.

Verifying the GPU is doing the work

The failure mode here is quiet. Everything runs. It just runs slowly, on the CPU, while you assume otherwise.

docker exec -it ollama nvidia-smi

If that errors, the container cannot see the GPU and no amount of Ollama configuration will help — the problem is the container toolkit or the driver.

Then run a prompt and watch nvidia-smi on the host. If VRAM use does not move and your CPU fans do, the model is not on the GPU. The usual cause is that the model does not fit in VRAM, which is a hardware question rather than a Docker one, and which I have written up separately in local LLMs by GPU VRAM.

What I have not tested

I have not run the ROCm image. The command above comes from Ollama’s documentation and I have no AMD card here to confirm it against, so treat that line as documented rather than verified by me.

I have not tested the newer top-level gpus: Compose attribute, and Compose version support for it is exactly the sort of thing that varies by distro package.

I have not benchmarked WSL2 against native Linux for inference throughput. I believe native is better and I have not measured it, so I am not going to give you a number.

What getting it wrong costs

The CUDA-inside-WSL mistake costs you a working GPU and then several hours, because the symptom appears after the install that felt like progress. You were closer before you started than after.

The -p 11434:11434 mistake costs more, and it costs it quietly. An unauthenticated inference API on a public address does not announce itself. It just becomes somebody else’s free GPU until you notice the electricity bill or the fan noise.