Les avancées dans l’Intelligence Artificielle Générative et sa mise en oeuvre simplifiée via certains outils, transforment profondément la gestion des clusters Kubernetes via ce concept d’ AIOps (acronyme anglais de Artificial Intelligence for IT Operations) énoncé par Gartner, un processus dans lequel vous utilisez des techniques d’intelligence artificielle (IA) pour maintenir par exemple une infrastructure. L’un des domaines dans lesquels les ingénieurs DevOps et les opérateurs de cluster débutants sont souvent confrontés, est relatif aux défis de l’identification, de la compréhension et de la résolution de problèmes au sein d’un cluster Kubernetes …
- AIOps - Wikipédia
- Qu'est-ce que l'intelligence artificielle pour les opérations informatiques (AIOps) ? - Explication de l'intelligence artificielle pour les opérations informatiques - AWS
- Intelligence artificielle générative - Wikipédia
Dans cet article, nous allons donc explorer comment configurer et utiliser K8sGPT, un outil open source basé sur l’IA générative, en combinaison avec Ollama et le modèle Falcon3, pour identifier et résoudre les problèmes dans un cluster Kubernetes.
Pour cela, je pars ici d’une instance Ubuntu 24.04 LTS dans DigitalOcean :
En y installant localement le moteur Docker …
(base) root@k8sgpt:~# curl -fsSL https://get.docker.com | sh -
# Executing docker install script, commit: 4c94a56999e10efcf48c5b8e3f6afea464f9108e
+ sh -c apt-get -qq update >/dev/null
+ sh -c DEBIAN_FRONTEND=noninteractive apt-get -y -qq install ca-certificates curl >/dev/null
Scanning processes...
Scanning candidates...
Scanning linux images...
+ sh -c install -m 0755 -d /etc/apt/keyrings
+ sh -c curl -fsSL "https://download.docker.com/linux/ubuntu/gpg" -o /etc/apt/keyrings/docker.asc
+ sh -c chmod a+r /etc/apt/keyrings/docker.asc
+ sh -c echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu noble stable" > /etc/apt/sources.list.d/docker.list
+ sh -c apt-get -qq update >/dev/null
+ sh -c DEBIAN_FRONTEND=noninteractive apt-get -y -qq install docker-ce docker-ce-cli containerd.io docker-compose-plugin docker-ce-rootless-extras docker-buildx-plugin >/dev/null
Scanning processes...
Scanning candidates...
Scanning linux images...
+ sh -c docker version
Client: Docker Engine - Community
Version: 27.4.1
API version: 1.47
Go version: go1.22.10
Git commit: b9d17ea
Built: Tue Dec 17 15:45:46 2024
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 27.4.1
API version: 1.47 (minimum version 1.24)
Go version: go1.22.10
Git commit: c710b88
Built: Tue Dec 17 15:45:46 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.7.24
GitCommit: 88bf19b2105c8b17560993bee28a01ddc2f97182
runc:
Version: 1.2.2
GitCommit: v1.2.2-0-g7cb3632
docker-init:
Version: 0.19.0
GitCommit: de40ad0
================================================================================
To run Docker as a non-privileged user, consider setting up the
Docker daemon in rootless mode for your user:
dockerd-rootless-setuptool.sh install
Visit https://docs.docker.com/go/rootless/ to learn about rootless mode.
To run the Docker daemon as a fully privileged service, but granting non-root
users access, refer to https://docs.docker.com/go/daemon-access/
WARNING: Access to the remote API on a privileged Docker daemon is equivalent
to root access on the host. Refer to the 'Docker daemon attack surface'
documentation for details: https://docs.docker.com/go/attack-surface/
================================================================================
Et je suis capable d’y lancer directmeent Ollama via son image officiellle pour exécuter des modèles de langage de grande taille (LLM) localement en y utilisant les CPU premium d’Intel présents dans l’instance Ubuntu :
Ollama is now available as an official Docker image · Ollama Blog
(base) root@k8sgpt:~# docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama:latest
Unable to find image 'ollama/ollama:latest' locally
latest: Pulling from ollama/ollama
6414378b6477: Pull complete
9423a26b200c: Pull complete
629da9618c4f: Pull complete
00b71e3f044c: Pull complete
Digest: sha256:18bfb1d605604fd53dcad20d0556df4c781e560ebebcd923454d627c994a0e37
Status: Downloaded newer image for ollama/ollama:latest
7b09d9fcdacff4319e553c41f741a15266eb5a5ec745959363e7754c53a203ef
(base) root@k8sgpt:~# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
7b09d9fcdacf ollama/ollama:latest "/bin/ollama serve" About a minute ago Up About a minute 0.0.0.0:11434->11434/tcp, :::11434->11434/tcp ollama
(base) root@k8sgpt:~# netstat -tunlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:11434 0.0.0.0:* LISTEN 46227/docker-proxy
tcp 0 0 127.0.0.54:53 0.0.0.0:* LISTEN 744/systemd-resolve
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 744/systemd-resolve
tcp6 0 0 :::22 :::* LISTEN 1/init
tcp6 0 0 :::11434 :::* LISTEN 46235/docker-proxy
udp 0 0 127.0.0.54:53 0.0.0.0:* 744/systemd-resolve
udp 0 0 127.0.0.53:53 0.0.0.0:* 744/systemd-resolve
Et en y chargeant le grand modèle Falcon 3 développé par le Technology Innovation Institute (TII) d’Abu Dhab et présent dans la bibliothèque de grands modèles d’Ollama :
(base) root@k8sgpt:~# docker exec -it ollama ollama pull falcon3
pulling manifest
pulling 3717a52b7aea... 100% ▕██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ 4.6 GB
pulling 803b5adc3448... 100% ▕██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ 218 B
pulling 58f83c52a4e3... 100% ▕██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ 13 KB
pulling 35e31ed4c388... 100% ▕██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ 101 B
pulling acb75345e14b... 100% ▕██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ 487 B
verifying sha256 digest
writing manifest
success
(base) root@k8sgpt:~# docker exec -it ollama ollama list
NAME ID SIZE MODIFIED
falcon3:latest 472ea1c89f64 4.6 GB 11 seconds ago
Je peux à cette étape y charger K8sGPT, un outil pour scanner vos clusters kubernetes, diagnostiquer et trier les problèmes en anglais simple. Il dispose d’une expérience SRE codifiée dans ses analyseurs et aide à extraire les informations les plus pertinentes pour les enrichir avec de l’IA générative.
K8sGPT fonctionne en trois étapes :
- Extraction : Récupération des détails de configuration de toutes les charges de travail déployées dans le cluster.
- Filtration : Un composant appelé « analyzer » filtre les données nécessaires.
- Génération : Les données filtrées sont traitées pour générer des insights et des rapports en anglais simple.
(base) root@k8sgpt:~# curl -LO https://github.com/k8sgpt-ai/k8sgpt/releases/download/v0.3.48/k8sgpt_amd64.deb
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 34.5M 100 34.5M 0 0 30.5M 0 0:00:01 0:00:01 --:--:-- 30.5M
(base) root@k8sgpt:~# dpkg -i k8sgpt_amd64.deb
(Reading database ... 74447 files and directories currently installed.)
Preparing to unpack k8sgpt_amd64.deb ...
Unpacking k8sgpt (0.3.48) over (0.3.48) ...
Setting up k8sgpt (0.3.48) ...
(base) root@k8sgpt:~# k8sgpt
Kubernetes debugging powered by AI
Usage:
k8sgpt [command]
Available Commands:
analyze This command will find problems within your Kubernetes cluster
auth Authenticate with your chosen backend
cache For working with the cache the results of an analysis
completion Generate the autocompletion script for the specified shell
custom-analyzer Manage a custom analyzer
dump Creates a dumpfile for debugging issues with K8sGPT
filters Manage filters for analyzing Kubernetes resources
generate Generate Key for your chosen backend (opens browser)
help Help about any command
integration Integrate another tool into K8sGPT
serve Runs k8sgpt as a server
version Print the version number of k8sgpt
Flags:
--config string Default config file (/root/.config/k8sgpt/k8sgpt.yaml)
-h, --help help for k8sgpt
--kubeconfig string Path to a kubeconfig. Only required if out-of-cluster.
--kubecontext string Kubernetes context to use. Only required if out-of-cluster.
Use "k8sgpt [command] --help" for more information about a command.
La ligne de commande est alors disponible pour y vérifier les fournisseurs disponibles localement dans cette instance :
(base) root@k8sgpt:~# k8sgpt auth list
Default:
> openai
Active:
Unused:
> openai
> localai
> ollama
> azureopenai
> cohere
> amazonbedrock
> amazonsagemaker
> google
> noopai
> huggingface
> googlevertexai
> oci
> ibmwatsonxai
Ollama sera utilisé comme fournisseur de backend IA pour K8sGPT au travers de LocalAI (qui agit comme l’API REST de remplacement compatible avec les spécifications de l’API OpenAI pour une inférence locale).
Voici la commande pour configurer K8sGPT avec Ollama et le modèle Falcon3 :
(base) root@k8sgpt:~# k8sgpt auth add --backend localai --model falcon3 --baseurl http://localhost:11434/v1
localai added to the AI backend provider list
Je lance un cluster Kubernetes managé sur DigitalOcean via DigitalOcean Kubernetes (DOKS) :
DigitalOcean Managed Kubernetes | Starting at $12/mo.
Récupération du fichier Kubeconfig depuis ce cluster pour l’insérer localement sur l’instance Ubuntu pour l’utiliser avec le client Kubectl :
(base) root@k8sgpt:~# curl -LO https://dl.k8s.io/release/v1.32.0/bin/linux/amd64/kubectl && chmod +x ./kubectl && mv kubectl /usr/local/bin/ && kubectl
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 138 100 138 0 0 1000 0 --:--:-- --:--:-- --:--:-- 1007
100 54.6M 100 54.6M 0 0 120M 0 --:--:-- --:--:-- --:--:-- 120M
kubectl controls the Kubernetes cluster manager.
Find more information at: https://kubernetes.io/docs/reference/kubectl/
Basic Commands (Beginner):
create Create a resource from a file or from stdin
expose Take a replication controller, service, deployment or pod and expose it as a new Kubernetes service
run Run a particular image on the cluster
set Set specific features on objects
Basic Commands (Intermediate):
explain Get documentation for a resource
get Display one or many resources
edit Edit a resource on the server
delete Delete resources by file names, stdin, resources and names, or by resources and label selector
Deploy Commands:
rollout Manage the rollout of a resource
scale Set a new size for a deployment, replica set, or replication controller
autoscale Auto-scale a deployment, replica set, stateful set, or replication controller
Cluster Management Commands:
certificate Modify certificate resources
cluster-info Display cluster information
top Display resource (CPU/memory) usage
cordon Mark node as unschedulable
uncordon Mark node as schedulable
drain Drain node in preparation for maintenance
taint Update the taints on one or more nodes
Troubleshooting and Debugging Commands:
describe Show details of a specific resource or group of resources
logs Print the logs for a container in a pod
attach Attach to a running container
exec Execute a command in a container
port-forward Forward one or more local ports to a pod
proxy Run a proxy to the Kubernetes API server
cp Copy files and directories to and from containers
auth Inspect authorization
debug Create debugging sessions for troubleshooting workloads and nodes
events List events
Advanced Commands:
diff Diff the live version against a would-be applied version
apply Apply a configuration to a resource by file name or stdin
patch Update fields of a resource
replace Replace a resource by file name or stdin
wait Experimental: Wait for a specific condition on one or many resources
kustomize Build a kustomization target from a directory or URL
Settings Commands:
label Update the labels on a resource
annotate Update the annotations on a resource
completion Output shell completion code for the specified shell (bash, zsh, fish, or powershell)
Subcommands provided by plugins:
Other Commands:
api-resources Print the supported API resources on the server
api-versions Print the supported API versions on the server, in the form of "group/version"
config Modify kubeconfig files
plugin Provides utilities for interacting with plugins
version Print the client and server version information
Usage:
kubectl [flags] [options]
Use "kubectl <command> --help" for more information about a given command.
Use "kubectl options" for a list of global command-line options (applies to all commands).
(base) root@k8sgpt:~# kubectl cluster-info
Kubernetes control plane is running at https://738af175-32d4-43e9-9e31-b7ae3058be3e.k8s.ondigitalocean.com
CoreDNS is running at https://738af175-32d4-43e9-9e31-b7ae3058be3e.k8s.ondigitalocean.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
(base) root@k8sgpt:~# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
pool-kuaxj3k47-ejvl3 Ready <none> 6m40s v1.31.1 10.110.0.3 159.65.200.52 Debian GNU/Linux 12 (bookworm) 6.1.0-27-amd64 containerd://1.6.31
pool-kuaxj3k47-ejvl8 Ready <none> 6m38s v1.31.1 10.110.0.2 164.92.147.176 Debian GNU/Linux 12 (bookworm) 6.1.0-27-amd64 containerd://1.6.31
(base) root@k8sgpt:~# kubectl get po,svc -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system pod/cilium-4gxht 1/1 Running 0 6m46s
kube-system pod/cilium-lw8gd 1/1 Running 0 6m48s
kube-system pod/coredns-c5c6457c-bnzfc 0/1 Running 0 36s
kube-system pod/coredns-c5c6457c-nz6gr 0/1 Running 0 36s
kube-system pod/cpc-bridge-proxy-ebpf-7ncbq 1/1 Running 0 55s
kube-system pod/cpc-bridge-proxy-ebpf-qth8w 1/1 Running 0 55s
kube-system pod/hubble-relay-67597fb8-kmlw5 1/1 Running 1 (51s ago) 8m40s
kube-system pod/hubble-ui-79957d9f7b-4n9kj 2/2 Running 0 74s
kube-system pod/konnectivity-agent-7ml7p 1/1 Running 0 61s
kube-system pod/konnectivity-agent-tnf8j 1/1 Running 0 61s
kube-system pod/kube-proxy-ebpf-4gt2z 1/1 Running 0 6m48s
kube-system pod/kube-proxy-ebpf-ztjql 1/1 Running 0 6m46s
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/kubernetes ClusterIP 10.108.32.1 <none> 443/TCP 9m54s
kube-system service/hubble-peer ClusterIP 10.108.54.62 <none> 443/TCP 8m40s
kube-system service/hubble-relay ClusterIP 10.108.41.20 <none> 80/TCP 8m40s
kube-system service/hubble-ui ClusterIP 10.108.52.164 <none> 80/TCP 8m40s
kube-system service/kube-dns ClusterIP 10.108.32.10 <none> 53/UDP,53/TCP,9153/TCP 36s
Installation d’Headlamp, une interface web qui remplace le dashboard traditionnel de Kubernetes, facile à utiliser et extensible.
- Headlamp
- GitHub - headlamp-k8s/headlamp: A Kubernetes web UI that is fully-featured, user-friendly and extensible
Headlamp a été créé pour combiner les fonctionnalités traditionnelles des autres interfaces web et tableaux de bord (c’est-à-dire pour lister et visualiser les ressources) avec des fonctionnalités supplémentaires.
(base) root@k8sgpt:~# kubectl apply -f https://raw.githubusercontent.com/kinvolk/headlamp/main/kubernetes-headlamp.yaml
service/headlamp created
deployment.apps/headlamp created
secret/headlamp-admin created
(base) root@k8sgpt:~# kubectl get po,svc -n kube-system
NAME READY STATUS RESTARTS AGE
pod/cilium-4gxht 1/1 Running 0 11m
pod/cilium-lw8gd 1/1 Running 0 11m
pod/coredns-c5c6457c-bnzfc 1/1 Running 0 5m7s
pod/coredns-c5c6457c-nz6gr 1/1 Running 0 5m7s
pod/cpc-bridge-proxy-ebpf-7ncbq 1/1 Running 0 5m26s
pod/cpc-bridge-proxy-ebpf-qth8w 1/1 Running 0 5m26s
pod/csi-do-node-gxlmb 2/2 Running 0 4m24s
pod/csi-do-node-swqfv 2/2 Running 0 4m24s
pod/do-node-agent-7bgsh 1/1 Running 0 4m11s
pod/do-node-agent-hwt6l 1/1 Running 0 4m11s
pod/headlamp-7dfd97b98b-wmn66 1/1 Running 0 48s
pod/hubble-relay-67597fb8-kmlw5 1/1 Running 1 (5m22s ago) 13m
pod/hubble-ui-79957d9f7b-4n9kj 2/2 Running 0 5m45s
pod/konnectivity-agent-7ml7p 1/1 Running 0 5m32s
pod/konnectivity-agent-tnf8j 1/1 Running 0 5m32s
pod/kube-proxy-ebpf-4gt2z 1/1 Running 0 11m
pod/kube-proxy-ebpf-ztjql 1/1 Running 0 11m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/headlamp ClusterIP 10.108.38.247 <none> 80/TCP 48s
service/hubble-peer ClusterIP 10.108.54.62 <none> 443/TCP 13m
service/hubble-relay ClusterIP 10.108.41.20 <none> 80/TCP 13m
service/hubble-ui ClusterIP 10.108.52.164 <none> 80/TCP 13m
service/kube-dns ClusterIP 10.108.32.10 <none> 53/UDP,53/TCP,9153/TCP 5m7s
en l’exposant localement et récupérant la clé nécessaire à l’accès à son interface web :
(base) root@k8sgpt:~# nohup kubectl port-forward -n kube-system service/headlamp 8080:80 &
(base) root@k8sgpt:~# cat nohup.out
Forwarding from 127.0.0.1:8080 -> 4466
Forwarding from [::1]:8080 -> 4466
(base) root@k8sgpt:~# kubectl -n kube-system create serviceaccount headlamp-admin
serviceaccount/headlamp-admin created
(base) root@k8sgpt:~# kubectl create clusterrolebinding headlamp-admin --serviceaccount=kube-system:headlamp-admin --clusterrole=cluster-admin
clusterrolebinding.rbac.authorization.k8s.io/headlamp-admin created
(base) root@k8sgpt:~# kubectl create token headlamp-admin -n kube-system
eyJhbGciOiJSUzI1NiIsImtpZCI6Ikd0UHltNHV6c1liSkY0VkhNRWFZMXJKYklqY1R6ckZrMDZQS281dEg3dUUifQ.eyJhdWQiOlsic3lzdGVtOmtvbm5lY3Rpdml0eS1zZXJ2ZXIiXSwiZXhwIjoxNzM1MzkyNDEwLCJpYXQiOjE3MzUzODg4MTAsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwianRpIjoiNTVkNjI1ZTItNjA2Yi00MTNhLTk2OTgtODFmYjdjZDU4MWY4Iiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJoZWFkbGFtcC1hZG1pbiIsInVpZCI6IjdmOGQwMTU0LWRiYWQtNGU2MS04NTUzLWU1NWI3ZWU0ZjhlOSJ9fSwibmJmIjoxNzM1Mzg4ODEwLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06aGVhZGxhbXAtYWRtaW4ifQ.jbrtuXS7uMP6HfwR3CbIbnpRTq4CDaacq0okwm_4tvmJNNcExi9-Dti3cGj1J3tteszpxVzurWPhrWgFlL4UkEacY9fD1TRH4GAZDCFldJ_jvyeaclzGeymrjEGAZ9TbBdoyuXtLeIVhApdICF1KNM-s8mfr1oOREDwlR9HzzrhoECozYxVS9uM1WIEZpum4FwMEl6cKPqOyNx1Rn5MtKPcc87JyK0FxuXzg9WC-cPSNOxu_rUFrZYHyrVapCDpl_XLymD3pFUUuB8XPVidVXcVOthH1Djwm8TRE6aAD4XlkHTcyTYchvN_CpOI2JQ6DVY60unSU8nq2pxfqLC6G2Q
Avant de lancer une analyse via K8sGPT, introduction d’un problème dans le cluster Kubernetes pour simuler une situation réelle. Vous pouvez utiliser des exemples de déploiements défectueux disponibles sur des répositories comme celui de Robusta :
GitHub - robusta-dev/kubernetes-demos: YAMLs for creating Kubernetes errors and other scenarios
Exemple avec ce déploiement d’un Pod cassé via :
apiVersion: apps/v1
kind: Deployment
metadata:
name: payment-processing-worker
spec:
replicas: 1
selector:
matchLabels:
app: payment-processing-worker
template:
metadata:
labels:
app: payment-processing-worker
spec:
containers:
- name: payment-processing-container
image: bash
command: ["/bin/sh"]
args: ["-c", "if [[-z \"${DEPLOY_ENV}\"]]; then echo Environment variable DEPLOY_ENV is undefined ; else while true; do echo hello; sleep 10;done; fi"]
base) root@k8sgpt:~# kubectl apply -f https://raw.githubusercontent.com/robusta-dev/kubernetes-demos/main/crashpod/broken.yaml
deployment.apps/payment-processing-worker created
(base) root@k8sgpt:~# kubectl get po
NAME READY STATUS RESTARTS AGE
payment-processing-worker-747ccfb9db-dzjqx 0/1 CrashLoopBackOff 1 (11s ago) 17s
(base) root@k8sgpt:~# kubectl logs po/payment-processing-worker-747ccfb9db-dzjqx
Environment variable DEPLOY_ENV is undefined
Il s’est en effet crashé …
Lancement de l’analyse avec une sortie en JSON :
(base) root@k8sgpt:~# k8sgpt analyze -o json --explain --filter=Pod --backend localai | jq .
{
"provider": "localai",
"errors": null,
"status": "ProblemDetected",
"problems": 1,
"results": [
{
"kind": "Pod",
"name": "default/payment-processing-worker-747ccfb9db-dzjqx",
"error": [
{
"Text": "the last termination reason is Completed container=payment-processing-container pod=payment-processing-worker-747ccfb9db-dzjqx",
"KubernetesDoc": "",
"Sensitive": []
}
],
"details": "Error: The pod \"payment-processing-worker-747ccfb9db-dzjqx\" has completed its execution with a \"Completed\" termination reason, indicating the container \"payment-processing-container\" has finished successfully.\n\nSolution: Verify the logs for the container to ensure data integrity, then check related services for expected outcomes; if successful, mark the pod as ready in the cluster.",
"parentObject": "Deployment/payment-processing-worker"
}
]
}
ou en mode texte :
(base) root@k8sgpt:~# k8sgpt analyze --explain --backend localai --with-doc
100% |████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| (1/1, 7960 it/s)
AI Provider: localai
0: Pod default/payment-processing-worker-747ccfb9db-dzjqx(Deployment/payment-processing-worker)
- Error: the last termination reason is Completed container=payment-processing-container pod=payment-processing-worker-747ccfb9db-dzjqx
Error: The pod "payment-processing-worker-747ccfb9db-dzjqx" has completed its execution with a "Completed" termination reason, indicating the container "payment-processing-container" has finished successfully.
Solution: Verify the logs for the container to ensure data integrity, then check related services for expected outcomes; if successful, mark the pod as ready in the cluster.
Comme le montre cette sortie, K8sGPT a identifié et signalé le pod problématique et a également fourni des conseils sur les mesures potentielles à prendre pour comprendre et résoudre le problème.
Autre exemple avec ce Pod Nginx problématique :
apiVersion: v1
kind: Pod
metadata:
name: inventory-management-api
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
command:
- wge
- "-O"
- "/work-dir/index.html"
- https://home.robusta.dev
(base) root@k8sgpt:~# kubectl apply -f https://raw.githubusercontent.com/robusta-dev/kubernetes-demos/refs/heads/main/crashloop_backoff/create_crashloop_backoff.yaml
pod/inventory-management-api created
(base) root@k8sgpt:~# kubectl get po
NAME READY STATUS RESTARTS AGE
inventory-management-api 0/1 ContainerCreating 0 5s
(base) root@k8sgpt:~# kubectl get po
NAME READY STATUS RESTARTS AGE
inventory-management-api 0/1 RunContainerError 1 (1s ago) 10s
Et une nouvelle analyse met le doigt sur le Pod problématique …
(base) root@k8sgpt:~# k8sgpt analyze --explain --backend localai --with-doc
100% |████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| (1/1, 2 it/min)
AI Provider: localai
0: Pod default/inventory-management-api()
- Error: the last termination reason is StartError container=nginx pod=inventory-management-api
Error: The Kubernetes error indicates that there was a StartError issue with the nginx container for the pod named inventory-management-api.
Solution:
1. Check the nginx configuration file for syntax errors.
2. Ensure all required resources and permissions are correctly set.
3. Verify network accessibility within the pod.
4. Confirm proper image pull secrets if using Docker images.
5. Review any recent changes to the deployment or service configurations.
Dans cet autre exemple il peut ne pas détecter de problématique avec cette simulation de faux positif via busybox :
apiVersion: batch/v1
kind: Job
metadata:
name: java-api-checker
spec:
template:
spec:
containers:
- name: java-beans
image: busybox
command: ["/bin/sh", "-c"]
args: ["echo 'Java Network Exception: \nAll host(s) tried for db query failed (tried: prod-db:3333) - no available connection and the queue has reached its max size 256 \nAll host(s) tried for db query failed (tried: prod-db:3333) - no available connection and the queue has reached its max size 256 \nAll host(s) tried for db query failed (tried: prod-db:3333) - no available connection and the queue has reached its max size 256 \nAll host(s) tried for db query failed (tried: prod-db:3333) - no available connection and the queue has reached its max size 256'; sleep 60; exit 1"]
restartPolicy: Never
backoffLimit: 1
(base) root@k8sgpt:~# kubectl delete -f https://raw.githubusercontent.com/robusta-dev/kubernetes-demos/refs/heads/main/crashloop_backoff/create_crashloop_backoff.yaml
pod "inventory-management-api" deleted
(base) root@k8sgpt:~# kubectl apply -f https://raw.githubusercontent.com/robusta-dev/kubernetes-demos/refs/heads/main/job_failure/job_crash.yaml
job.batch/java-api-checker created
(base) root@k8sgpt:~# kubectl get po
NAME READY STATUS RESTARTS AGE
java-api-checker-5s6dc 1/1 Running 0 7s
(base) root@k8sgpt:~# kubectl logs po/java-api-checker-5s6dc
Java Network Exception:
All host(s) tried for db query failed (tried: prod-db:3333) - no available connection and the queue has reached its max size 256
All host(s) tried for db query failed (tried: prod-db:3333) - no available connection and the queue has reached its max size 256
All host(s) tried for db query failed (tried: prod-db:3333) - no available connection and the queue has reached its max size 256
All host(s) tried for db query failed (tried: prod-db:3333) - no available connection and the queue has reached its max size 256
(base) root@k8sgpt:~# k8sgpt analyze --explain --backend localai --with-doc
AI Provider: localai
No problems detected
Pour une intégration plus complète, vous pouvez installer l’opérateur K8sGPT dans votre cluster Kubernetes. Cet opérateur surveille en continu les problèmes dans le cluster et génère des insights que vous pouvez consulter en interrogeant la ressource personnalisée (CR) de l'opérateur.
GitHub - k8sgpt-ai/k8sgpt-operator: Automatic SRE Superpowers within your Kubernetes cluster
$ helm repo add k8sgpt https://charts.k8sgpt.ai/
$ helm repo update
$ helm install release k8sgpt/k8sgpt-operator -n k8sgpt-operator-system --create-namespace
# Installer l'opérateur k8sGPT
$ kubectl apply -n k8sgpt-operator-system -f - << EOF
apiVersion: core.k8sgpt.ai/v1alpha1
kind: K8sGPT
metadata:
name: k8sgpt-ollama
spec:
ai:
enabled: true
model: falcon3
backend: localai
baseUrl: http://localhost:11434/v1
noCache: false
filters: ["Pod"]
repository: ghcr.io/k8sgpt-ai/k8sgpt
version: v0.3.48
EOF
Debugging your Rancher Kubernetes Cluster the GenAI Way w...
Il est également possible d’analyser plusieurs clusters Kubernetes en spécifiant le chemin vers le fichier Kubeconfig concerné :
$ k8sgpt analyze --explain --backend localai --with-doc --kubeconfig <chemin vers le fichier Kubeconfig>
L’opérateur recherchera les problèmes dans le cluster et générera des résultats d’analyse. En fonction de la puissance de votre machine (pour accélerer les temps de réponse d’Ollama, des ressources GPU sont nécessaires), il faut un certain temps à l’opérateur pour appeler le LLM et générer les informations …
Pour conclure, K8sGPT en combinaison avec Ollama offre une solution puissante pour déboguer et gérer les clusters Kubernetes de manière efficace. Cette intégration utilise l'intelligence artificielle pour fournir des insights clairs et des recommandations pour résoudre les problèmes, simplifiant ainsi la vie des opérateurs de cluster. En suivant ces étapes, vous pouvez mettre en place une solution de diagnostic automatisée et basée sur l'IA pour votre environnement Kubernetes …
À suivre !
Top comments (0)