How to Use an NVIDIA GPU Inside a Docker Container

Takeo — Mon, 27 Jul 2026 18:00:38 +0000

I do not normally use Docker very often. However, I recently needed to install PyTorch3D, a library that is no longer actively updated, and I could not get it working properly in my current environment.

I could have launched a separate GPU machine running the older Ubuntu 22.04 or installed another CUDA version on the host system. However, I was concerned that adding multiple CUDA versions might cause problems with library paths and symbolic links.

So, somewhat reluctantly, I decided to use Docker.

Running a normal Docker container is relatively straightforward: pull an image and start the container. However, when using a GPU machine, some additional configuration is required to make the host GPU available inside the container.

This article is a short memo describing the setup process.

Host environment

My host environment is Ubuntu 24.04 with CUDA 13.1.

The following command also displays information about the NVIDIA driver and the GPU:

# nvidia-smi

The output looks like this:

Although nvidia-smi may display a different CUDA version, the CUDA Toolkit installed on the host is CUDA 13.1, as shown by the output of nvcc --version.

Target Docker environment

For this test, I wanted to launch a container with the following combination:

Ubuntu 22.04
CUDA 12.4.1

To make the host GPU available inside Docker containers, the NVIDIA Container Toolkit must first be installed.

Installing the NVIDIA Container Toolkit

I installed it using the following commands:

# curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

# apt update
# apt install nvidia-container-toolkit

The installation completed without any problems.

Configuring Docker to use the NVIDIA runtime

Next, configure Docker to use the NVIDIA Container Runtime and restart the Docker service.

# nvidia-ctk runtime configure --runtime=docker
# systemctl restart docker

You can confirm that the NVIDIA runtime has been registered by running:

# docker info | grep -i runtime
 Runtimes: nvidia runc io.containerd.runc.v2
 Default Runtime: runc

You can also inspect Docker's configuration file:

# cat /etc/docker/daemon.json
{
    "runtimes": {
        "nvidia": {
            "args": [],
            "path": "nvidia-container-runtime"
        }
    }
}

The default runtime remains runc, but the NVIDIA runtime is now available when a container is started with GPU support.

Selecting a CUDA Docker image

For this example, I used a CUDA 12.4.1 runtime image based on Ubuntu 22.04.

Other CUDA and Ubuntu combinations can be found on the NVIDIA CUDA image page on Docker Hub:

Selecting a CUDA Docker image

For this example, I used a CUDA 12.4.1 runtime image based on Ubuntu 22.04.

Other CUDA and Ubuntu combinations can be found on the NVIDIA CUDA image page on Docker Hub:
CUDA Image

Testing GPU access inside the container

To confirm that the GPU is available inside the Docker container, run nvidia-smi in the container:

# docker run --rm --gpus all \
  nvidia/cuda:12.4.1-runtime-ubuntu22.04 \
  nvidia-smi

The options used here are:

--rm: Automatically removes the container after it exits.
--gpus all: Makes all available host GPUs accessible inside the container.
nvidia/cuda:12.4.1-runtime-ubuntu22.04: Specifies the CUDA 12.4.1 runtime image based on Ubuntu 22.04.
nvidia-smi: Runs the NVIDIA system management command inside the container.

If the setup is working correctly, information about the host GPU should be displayed:

The downloaded Docker image can also be confirmed with:

# docker image ls

Conclusion

By installing the NVIDIA Container Toolkit and starting the container with the --gpus all option, I was able to use the host GPU from a Docker container running an older combination of Ubuntu and CUDA.

This approach is useful when a library requires an older CUDA or operating-system environment and you do not want to modify the CUDA configuration of the host machine.

In my case, Docker allowed me to prepare an Ubuntu 22.04 and CUDA 12.4.1 environment without adding another CUDA installation directly to my Ubuntu 24.04 host.

DEV Community: Takeo

How to Use an NVIDIA GPU Inside a Docker Container