DEV Community

Cover image for Building a vLLM Container Image
Sanskriti Harmukh for Vultr

Posted on with Aashish Chaurasiya Originally published at docs.vultr.com

Building a vLLM Container Image

vLLM is a fast inference and serving library for Large Language Models (LLMs). It offers integration with popular Hugging Face models, optimized CUDA kernels for NVIDIA GPUs, tensor parallelism support, and fast model execution. This guide walks through cloning the vLLM project repository and building container images for both CPU-only and GPU-powered systems, then pushing the result to a container registry. By the end, you'll have a working vLLM container image tagged and pushed to a registry, ready to deploy.


1. Prerequisites

Before you begin:

  • Deploy a GPU-enabled server with NVIDIA GPU drivers, CUDA, and Docker preinstalled.
  • Access the server using SSH.
  • Start the Docker service.

1. Start the Docker service:

$ sudo systemctl start docker
Enter fullscreen mode Exit fullscreen mode

2. Set Up the Server

1. Create a new directory to store your vLLM project files:

$ mkdir vllm-project
Enter fullscreen mode Exit fullscreen mode

2. Switch to the directory:

$ cd vllm-project
Enter fullscreen mode Exit fullscreen mode

3. Clone the vLLM project repository using Git:

$ git clone https://github.com/vllm-project/vllm/
Enter fullscreen mode Exit fullscreen mode

4. List files and verify that a new vllm directory is available:

$ ls
Enter fullscreen mode Exit fullscreen mode

5. Switch to the vllm project directory:

$ cd vllm
Enter fullscreen mode Exit fullscreen mode

6. List the directory files and verify that the necessary Dockerfile resources are available:

$ ls
Enter fullscreen mode Exit fullscreen mode

Output:

benchmarks      collect_env.py   Dockerfile       docs       LICENSE                 pyproject.toml          requirements-common.txt  requirements-dev.txt     rocm_patch  vllm
cmake           CONTRIBUTING.md  Dockerfile.cpu   examples   MANIFEST.in             README.md               requirements-cpu.txt     requirements-neuron.txt  setup.py
CMakeLists.txt  csrc             Dockerfile.rocm
Enter fullscreen mode Exit fullscreen mode

The vLLM project directory includes the following Dockerfile resources:

  • Dockerfile: Contains the main vLLM library build context with support for NVIDIA GPU systems.
  • Dockerfile.cpu: Contains the vLLM build context for CPU systems.
  • Dockerfile.rocm: Contains the build context for AMD GPU systems.

Use these resources in the next sections to build a CPU or GPU system container image.

3. Build a vLLM Container Image for CPU Systems

Use Dockerfile.cpu, which contains the build context with all necessary packages and dependencies for CPU-based systems.

1. Build a new container image using Dockerfile.cpu with all files in the project working directory. Replace vllm-image with your desired image name:

$ docker build -f Dockerfile.cpu -t vllm-image .
Enter fullscreen mode Exit fullscreen mode

2. View all Docker images on the server and verify that your new vLLM image is available:

$ docker images
Enter fullscreen mode Exit fullscreen mode

Output:

REPOSITORY   TAG       IMAGE ID       CREATED              SIZE
vllm-image   latest    70f07e7c923f   About a minute ago   3.22GB
Enter fullscreen mode Exit fullscreen mode

4. Build a vLLM Container Image for GPU Systems

The vLLM project directory contains two Dockerfile resources for building container images for GPU-powered systems. Use the main Dockerfile resource to build a new container image for GPU systems.

1. Build a new container image vllm-gpu-image using Dockerfile with all files in the project directory:

$ docker build -f Dockerfile -t vllm-gpu-image .
Enter fullscreen mode Exit fullscreen mode

2. View all Docker images on the server and verify that the new vllm-gpu-image is available:

$ docker images
Enter fullscreen mode Exit fullscreen mode

Output:

REPOSITORY        TAG       IMAGE ID       CREATED       SIZE
vllm-gpu-image   latest    bf92416d18b4   8 hours ago   8.88GB
Enter fullscreen mode Exit fullscreen mode

To run the vLLM GPU container image, verify that your target host runs the minimum or higher CUDA version referenced in the Dockerfile, and use the --gpus all option when starting the container.

3. Verify the minimum CUDA version:

$ cat Dockerfile | grep CUDA_VERSION=
Enter fullscreen mode Exit fullscreen mode

Output:

ARG CUDA_VERSION=12.3.1
Enter fullscreen mode Exit fullscreen mode

5. Upload the vLLM Container Image to a Container Registry

1. Open your cloud provider's console and navigate to your container registry to view the registry access credentials. Copy the Registry URL, Username, and API Key.

2. Switch to your server terminal session and log in to your container registry. Replace your-registry.example.com, exampleregistry, exampleuser, registry-password with your actual registry details:

$ docker login https://your-registry.example.com/exampleregistry -u exampleuser -p registry-password
Enter fullscreen mode Exit fullscreen mode

3. Tag the vLLM container image with your desired registry tag. For example, your-registry.example.com/exampleregistry/vllm-gpu-image:

$ docker tag vllm-gpu-image your-registry.example.com/exampleregistry/vllm-gpu-image
Enter fullscreen mode Exit fullscreen mode

4. View all Docker images on the server and verify that the new tagged image is available:

$ docker images
Enter fullscreen mode Exit fullscreen mode

Output:

REPOSITORY                                                TAG       IMAGE ID       CREATED       SIZE
vllm--gpu-image                                           latest    bf92416d18b4   8 hours ago   8.88GB
your-registry.example.com/exampleregistry/vllm-gpu-image  latest    bf92416d18b4   8 hours ago   8.88GB
Enter fullscreen mode Exit fullscreen mode

5. Push the tagged image to your container registry:

$ docker push your-registry.example.com/exampleregistry/vllm-gpu-image
Enter fullscreen mode Exit fullscreen mode

6. Open your container registry's management panel and verify that the new repository is available.

Next Steps

  • Reference the pushed image from your orchestrator (Docker Compose, Kubernetes, etc.) to deploy vLLM at scale.
  • Benchmark inference throughput with vLLM's built-in benchmarking scripts before moving to production.
  • Automate the build with a CI pipeline so new vLLM releases are rebuilt and republished automatically.
  • Explore vLLM's tensor parallelism options for multi-GPU serving.

For the full guide with additional tips, visit the original article on Vultr Docs.

Top comments (0)