Building a Llama.cpp Container Image

Llama.cpp is a high-performance inference platform designed for Large Language Models (LLMs) like Llama, Falcon, and Mistral. It provides a streamlined development environment compatible with both CPU and GPU systems. This guide explains how to build Llama.cpp into a Docker container image for CPU and GPU systems and push the finished image to a container registry. By the end, you'll have CPU and GPU container images of Llama.cpp built, tagged, and pushed to a container registry.

---

Prerequisites

• A server with Docker installed (a GPU-enabled server with NVIDIA drivers and the NVIDIA Container Toolkit for the GPU build).
• Access to the server using SSH.

• Start the Docker service.
  

  $ sudo systemctl start docker
  

• Add the non-root user to the Docker group. For example, linuxuser.
  

  $ sudo usermod -aG docker linuxuser
  

• Switch to the user:
  

  $ su - linuxuser
  

1. Set Up the Project Directory

1. Create a new directory to store your Llama.cpp project files:

$ mkdir llama-project

2. Switch to the directory:

$ cd llama-project

3. Clone the Llama.cpp project repository using Git:

$ git clone https://github.com/ggerganov/llama.cpp/

4. List files and verify that a new llama.cpp directory is available:

$ ls

Output:

llama.cpp

5. Switch to the llama.cpp project directory:

$ cd llama.cpp

6. List all hidden files and verify that a new .devops directory is available:

$ ls -a

Output:

..                 cmake           convert.py                     flake.lock     ggml-common.h        ggml-metal.h      ggml-quantsh     .git                         llama.cpp        
CMakeLists.txt  .devops    

The .devops directory contains the following Dockerfile resources:

• main.Dockerfile: Contains the build context for CPU systems.
• main-cuda.Dockerfile: Contains the build context for GPU systems.

Use these resources in the next sections to build a CPU or GPU system container image.

2. Build a Llama.cpp Container Image for CPU Systems

Follow the steps below to build a new Llama.cpp container image using main.Dockerfile, which contains the required build context and installs all necessary dependency packages for CPU-based systems.

1. Copy the main.Dockerfile from the .devops directory to the main llama.cpp project directory:

$ cp .devops/main.Dockerfile .

2. Build a new container image using main.Dockerfile with all files in the project directory. Replace llama-image with your desired image name:

$ docker build -f main.Dockerfile -t llama-image .

3. View all Docker images on the server and verify that a new llama-image is available:

$ docker images

Output:

REPOSITORY                                  TAG       IMAGE ID       CREATED              SIZE
llama-image                                 latest    f4fcb571a2f7   About a minute ago   80.1MB

3. Build a Llama.cpp Container Image for GPU Systems

The Llama.cpp main-cuda.Dockerfile resource contains the build context for NVIDIA GPU systems that run the latest CUDA driver packages. Follow the steps below to build a Llama container image compatible with GPU systems.

1. Copy main-cuda.Dockerfile to the Llama.cpp project directory:

$ cp .devops/main-cuda.Dockerfile .

2. Build a new container image llama-gpu-image using the main-cuda.Dockerfile with all files in the working project directory:

$ docker build -f main-cuda.Dockerfile -t llama-gpu-image .

3. View all Docker images on the server and verify that the new llama-gpu-image is available:

$ docker images

Output:

REPOSITORY                                  TAG       IMAGE ID       CREATED          SIZE
llama-gpu-image                             latest    40421f1469cf   2 minutes ago    2.08GB

To run the Llama.cpp GPU container image, verify that your target host includes the minimum or higher CUDA version referenced by the ARG CUDA_VERSION= directive within main-cuda.Dockerfile.

4. Verify the target CUDA version:

$ cat main-cuda.Dockerfile | grep CUDA_VERSION=

Output:

ARG CUDA_VERSION=12.3.1

4. Push the Llama.cpp Container Image to a Container Registry

1. Open your container registry's management panel and copy the registry URL, username, and API key (or password) to use when accessing the registry.

2. Switch to your server terminal session and log in to your container registry. Replace your-registry-url, exampleregistry, exampleuser, and registry-password with your actual registry details:

$ docker login https://your-registry-url/exampleregistry -u exampleuser -p registry-password

3. Tag the Llama.cpp container image with your desired registry tag. For example, your-registry-url/exampleregistry/llama-gpu-image:

$ docker tag llama-gpu-image your-registry-url/exampleregistry/llama-gpu-image

4. View all Docker images on the server and verify that the new tagged image is available:

$ docker images

Output:

REPOSITORY                                           TAG       IMAGE ID       CREATED          SIZE
your-registry-url/exampleregistry/llama-gpu-image    latest    40421f1469cf   3 minutes ago    2.08GB
llama-gpu-image                                      latest    40421f1469cf   3 minutes ago    2.08GB

5. Push the tagged image to your container registry repository:

$ docker push your-registry-url/exampleregistry/llama-gpu-image

6. Open your container registry's management panel and check the repositories list to verify that the new repository is available.

Next Steps

• Automate the CPU and GPU builds in a CI/CD pipeline so new images are rebuilt whenever the Llama.cpp repository updates.
• Pull the GPU image onto a multi-GPU host and benchmark inference throughput across different quantization levels.
• Pin the CUDA_VERSION in main-cuda.Dockerfile to match the driver version installed on your target hosts.
• Explore Llama.cpp's server mode to expose an OpenAI-compatible API from the container.

For the full guide with additional tips, visit the original article on Vultr Docs.