Building an AI-powered Docker Solution with Llama and k8sGPT

In this article, I will walk you through building an efficient solution leveraging Docker, Kubernetes, and AI tools like Llama and k8sGPT. This guide assumes you have a solid understanding of Docker, Kubernetes, and basic command-line operations.

Motivation

The goal is to have a k8s analysis tool which can identify the issues within the cluster and provide a human-readable description and possibly a solution thanks to the LLM. All of that ideally packed as a single Docker image, so it can be executed anywhere without further configuration.

Solution Architecture Overview

This solution is structured around a Docker image tailored to execute an AI model via the Llama framework. Diagnostic capabilities are enhanced using k8sGPT, a tool optimized for Kubernetes environments. The primary components include:

• ollama: Serves as the container-based model deployment framework, enabling efficient hosting and serving of large language models (LLMs). It provides API endpoints for model inference and handles model weight management, offering optimized serving capabilities with support for various model formats and quantization levels.
• k8sGPT: Functions as an AI-powered diagnostic tool specifically designed for Kubernetes environments. It analyzes cluster events, logs, and metrics to provide automated troubleshooting insights, detect anomalies, and generate human-readable explanations for identified issues within the Kubernetes infrastructure.
• Docker: Provides the containerization platform that encapsulates the application, its dependencies, and runtime environment into standardized units. This ensures consistent deployment across different environments while offering isolation, resource management, and scalability features necessary for production workloads.

Step 1: Crafting the Dockerfile

FROM ollama/ollama

# Install needed dependencies into the image
RUN apt-get update && apt-get install -y \
    curl \
    git \
    build-essential \
    && rm -rf /var/lib/apt/lists/*

# Download and install the kubectl tool for managing the k8s clusters
RUN curl -LO "https://dl.k8s.io/release/v1.25.3/bin/linux/arm64/kubectl" && \
    chmod +x kubectl && \
    mv kubectl /usr/local/bin/

# Download and install k8sGPT tool for k8s analysis.
RUN curl -LO "https://github.com/k8sgpt-ai/k8sgpt/releases/latest/download/k8sgpt_Linux_arm64.tar.gz" && \
    tar xzf k8sgpt_Linux_arm64.tar.gz && \
    chmod +x k8sgpt && \
    mv k8sgpt /usr/local/bin/ && \
    rm k8sgpt_Linux_arm64.tar.gz

# Start ollama and download llama3.2:3b model right into the image
RUN mkdir -p /root/.ollama && \
    ollama serve & \
    sleep 10 && \
    ollama pull llama3.2:3b && \
    sleep 5 && \
    ollama list && \
    pkill ollama

# Copy kube config configuration into the image + Copy entry point script
WORKDIR /opt
COPY ["./k8s/", "/opt/k8s/"]
COPY ["run.sh", "/opt/run.sh"]

ENV KUBECONFIG=/opt/k8s/kubeconfig.yaml
ENV TZ="Europe/Prague"

# Run the script when the container is started
RUN chmod +x /opt/run.sh
ENTRYPOINT ["/opt/run.sh"]

Step 2: Designing the Entry Point Script

#!/bin/bash

# Start ollama in the background
ollama serve &
while ! curl -s localhost:11434 >/dev/null; do
    echo "Waiting for ollama to start..."
    sleep 1
done

# Configure the k8sGPT tool to use ollama and installed llama3.2:3b model by default
k8sgpt auth add --backend ollama --model llama3.2:3b
k8sgpt auth default -p ollama

# Configure the active k8s configuration
kubectl config use-context "context name"
k8sgpt analyze --explain

Step 3: Build and Run The Docker Image

docker build -t k8sgpt-ollama-llama3-analyzer .
docker run k8sgpt-ollama-llama3-analyzer

Step 4: Analyze the output

The possible output might be something similar to this. The k8sGTP provides an error detection and solution in human readable form thanks to the locally integrated LLM.

Ingress kube-system/hubble-ui()
- Error: Ingress kube-system/hubble-ui does not specify an Ingress class.
Error: Ingress kube-system/hubble-ui does not specify an Ingress class.

Solution:
1. Create a default Ingress Class: `kubectl create ingressclass hubble-ingress --default`
2. Update Ingress resource: `kubectl edit ing ingress kube-system/hubble-ui`

Conclusion and Possible Improvements

The Dockerfile and run.sh script can be parameterized to externalize all necessary configurations, allowing for greater flexibility without modifying the Docker image. Currently, the analysis results are directly printed to the console. Adding options for output delivery, such as email notifications, could improve usability.

The docker image can be used for example in CI/CD environment and executed daily to check your Kubernetes clusters.