Why Your Ollama LLMs Are Running Painfully Slow on K3s (And How to Fix GPU Pass-Through)
Imagine this scenario: You finally decided to take control of your data privacy and build an in-house AI infrastructure. You spun up a lightweight Kubernetes cluster using K3s, deployed Ollama, connected it to an Open-WebUI frontend, and asked your local LLM a simple question.
Then... you wait.
The cursor blinks. One word crawls onto the screen every two seconds ($1 \text{ token/s}$). Your CPU fan sounds like a jet engine taking off, system memory spikes to $100\%$, and the user experience is thoroughly ruined.
What went wrong? You have an expensive NVIDIA GPU seated right inside your server hardware! Why is your local AI running like a turtle on a treadmill?
In this deep-dive article, we will unpack the root cause behind Ollama’s sluggish performance on Kubernetes, explore how K3s handles hardware resources, and walk through a step-by-step production guide to enable true NVIDIA GPU Pass-Through using the NVIDIA Container Toolkit and Device Plugin.
The Root Cause: Why Is Ollama Defaulting to CPU?
When you run Ollama locally on bare-metal desktop OS (like macOS or Windows with CUDA drivers installed), Ollama automatically detects available GPUs and offloads neural network layers to VRAM.
However, Kubernetes operates in an isolated, containerized universe.
By default, container runtimes inside Kubernetes (specifically containerd, which comes bundled with K3s) are deliberately sandboxed from the underlying host's physical PCI hardware.
What Happens Behind the Scenes:
Hardware Blindness: The containerized Ollama process cannot "see"
/dev/nvidia*host device nodes.CPU Fallback: Because no CUDA runtime is exposed inside the pod, Ollama gracefully falls back to CPU matrix multiplication (using AVX/AVX-512 instructions).
Resource Exhaustion: CPU inference on a 7B or 13B parameter model requires continuous, high-latency memory transfers between system RAM and the CPU cache. This leads to thread contention, thermal throttling, and sub-par generation speeds compared to 50 tex - 80 text tokens/s on GPU VRAM).
To fix this, we must build a hardware bridge from the Linux kernel driver all the way into our K3s pod namespace.
Architectural Overview: The 3-Layer GPU Bridge
To achieve zero-latency GPU pass-through in K3s, we need three distinct layers working in harmony:
┌─────────────────────────────────────────────────────────┐
│ Ollama Pod (K3s Namespace) │
│ resources: limits: { nvidia.com/gpu: "1" } │
└────────────────────────────┬────────────────────────────┘
│
┌────────────────────────────▼────────────────────────────┐
│ NVIDIA K8s Device Plugin │
│ (Exposes GPU count to K3s Scheduler) │
└────────────────────────────┬────────────────────────────┘
│
┌────────────────────────────▼────────────────────────────┐
│ NVIDIA Container Toolkit & Runtime (Host) │
│ (Injects CUDA libraries into containerd) │
└────────────────────────────┬────────────────────────────┘
│
┌────────────────────────────▼────────────────────────────┐
│ Linux Kernel & NVIDIA Drivers │
│ (/dev/nvidia0, /dev/nvidia-uvm) │
└─────────────────────────────────────────────────────────┘
Let's implement this step-by-step.
Step 1: Prepare the Host Node (NVIDIA Drivers & Container Toolkit)
Before touching Kubernetes or K3s manifests, your Linux host kernel must have functioning NVIDIA drivers and the NVIDIA Container Toolkit installed.
1. Verify NVIDIA Driver Installation
Ensure the official NVIDIA proprietary drivers are loaded on your host machine:
Bashnvidia-smi
You should see a clean driver status table showing your GPU device name (e.g., RTX 3090 / RTX 4090 / A100), driver version (e.g., 535.x or 550.x), and CUDA version capability.
2. Install NVIDIA Container Toolkit
The Container Toolkit configures containerd to pass host GPU devices into OCI container specs.
On Ubuntu / Debian systems:
Bash# Add NVIDIA Container Toolkit repository
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
&& curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed 's#deb [^ ]* #&[signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] #' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
Step 2: Configure K3s Containerd Engine to Support NVIDIA Runtime
By default, K3s uses its own embedded containerd instance located at /var/lib/rancher/k3s/agent/etc/containerd/config.toml.
Crucial Note: Never edit this config.toml directly! K3s overwrites this file every time the service restarts. Instead, we must use the K3s template extension mechanism.
1. Create a Containerd Template File
Create a template configuration file at /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl:
sudo cp /var/lib/rancher/k3s/agent/etc/containerd/config.toml /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl
2. Register the nvidia-container-runtime
Configure the NVIDIA Container Runtime as a default/available runtime handler inside /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl:
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia]
runtime_type = "io.containerd.runc.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia.options]
BinaryName = "/usr/bin/nvidia-container-runtime"
3. Restart K3s Service
Apply the changes by restarting K3s:
Bashsudo systemctl restart k3s
Step 3: Deploy the NVIDIA Kubernetes Device Plugin
Now that containerd knows how to launch CUDA-enabled containers, we need to advertise physical GPU resources to the Kubernetes Control Plane scheduler.
We use Helm to deploy the official NVIDIA Kubernetes Device Plugin:
Bash# Add NVIDIA Helm repository
helm repo add nvdp https://nvidia.github.io/k8s-device-plugin
helm repo update
# Install Device Plugin into kube-system namespace
helm install nvidia-device-plugin nvdp/k8s-device-plugin \
--namespace kube-system \
--set runtimeClassName=nvidia
Verify GPU Resource Capacity
Check if your K3s node now recognizes the physical GPU as a schedulable capacity unit:
Bashkubectl get nodes -o jsonpath='{.items[*].status.allocatable}' | jq .
You should see an allocatable resource key similar to this:
JSON{
"cpu": "16",
"memory": "32Gi",
"nvidia.com/gpu": "1"
}
If [nvidia.com/gpu](https://nvidia.com/gpu) appears in the list, congratulations! Your K3s cluster is officially GPU-aware.
Step 4: Deploying Ollama with GPU Resource Requests
Now comes the moment of truth: deploying Ollama with explicit GPU resource limits so the Kubernetes scheduler mounts the CUDA drivers into the container.
Create a production deployment file named ollama-gpu-deployment.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: ollama
namespace: ai-workloads
labels:
app: ollama
spec:
replicas: 1
selector:
matchLabels:
app: ollama
template:
metadata:
labels:
app: ollama
spec:
runtimeClassName: nvidia # Enforces NVIDIA container runtime
containers:
- name: ollama
image: ollama/ollama:latest
imagePullPolicy: IfNotPresent
ports:
- containerPort: 11434
name: http
resources:
requests:
cpu: "2000m"
memory: "4Gi"
nvidia.com/gpu: "1" # Requests 1 Physical GPU
limits:
memory: "16Gi"
nvidia.com/gpu: "1" # Caps GPU usage to 1
volumeMounts:
- name: ollama-storage
mountPath: /root/.ollama
volumes:
- name: ollama-storage
persistentVolumeClaim:
claimName: ollama-pvc
---
apiVersion: v1
kind: Service
metadata:
name: ollama-service
namespace: ai-workloads
spec:
type: ClusterIP
ports:
- port: 11434
targetPort: 11434
name: http
selector:
app: ollama
Apply the manifest:
Bashkubectl create namespace ai-workloads
kubectl apply -f ollama-gpu-deployment.yaml
Step 5: Verification & Performance Benchmark
Let's verify that Ollama is actively using GPU hardware acceleration instead of CPU rendering.
1. Exec into the Pod & Monitor VRAM Usage
Run nvidia-smi directly from your host or exec into the pod while pulling a model:
# Exec into running Ollama pod
kubectl exec -ti -n ai-workloads deployment/ollama -- ollama run llama3:8b
On your host machine, open a separate terminal window and run:
Bashwatch -n 1 nvidia-smi
2. Check Key Metrics
Look for the following signals:
Process Name: You should see
ollamaorollama_llama_serverregistered under the GPU Processes table.VRAM Allocation: VRAM usage should jump from $0 \text{ MiB}$ to $4.5\text{--}5.5 \text{ GiB}$ (for a quantized 8B model).
Generation Speed: Tokens should stream onto your terminal screen instantly at $60\text{--}90 \text{ tokens/s}$!
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.14 Driver Version: 550.54.14 CUDA Version: 12.4 |
|-----------------------------------+------------------------+--------------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
|===================================+========================+==========================|
| 0 NVIDIA RTX 4090 On | 00000000:01:00.0 Off | N/A |
| 30% 45C P2 120W / 450W | 5620MiB / 24564MiB | 88% Default |
+-----------------------------------+------------------------+--------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 0 N/A N/A 18294 C ...bin/ollama_llama_server 5612MiB |
+---------------------------------------------------------------------------------------+
Troubleshooting Common Edge Cases
Even with a detailed setup, distributed container runtime quirks can occur. Here are the top three issues engineers face when deploying Ollama on K3s:
Issue 1: Insufficient [nvidia.com/gpu](https://nvidia.com/gpu) Error
Symptom: Pod remains stuck in
Pendingstate.kubectl describe podshows0/1 nodes are available: 1 Insufficient [nvidia.com/gpu](https://nvidia.com/gpu).-
Fix: The Device Plugin failed to register the GPU. Verify that
Bashnvidia-smiworks on the host and restart thenvidia-device-plugindaemonset:kubectl rollout restart daemonset nvidia-device-plugin-daemonset -n kube-system
Issue 2: CUDA Out Of Memory (OOM)
Symptom: Ollama pod crashes during model loading with
CUDA error: out of memory.Fix: The model parameter size exceeds your GPU VRAM capacity. Use smaller quantized models (e.g.,
llama3:8b-instruct-q4_K_Minstead of FP16), or enable CPU Offloading parameters in Ollama environment flags (OLLAMA_NUM_PARALLEL=1).
Issue 3: Driver Mismatch After Linux Kernel Updates
Symptom: GPU pass-through worked yesterday, but broke after a system reboot.
-
Fix: Unattended Linux kernel upgrades break proprietary NVIDIA DKMS kernel modules. Disable automatic updates for kernel/NVIDIA packages or pin your driver versions:
Bashsudo apt-mark hold nvidia-driver-550
Conclusion & Next Steps
Building a Private AI Cloud on top of K3s gives you the ultimate combination: the lightweight orchestration of K3s paired with the raw computing power of local GPU hardware.
By setting up the NVIDIA Container Toolkit and enforcing runtimeClassName: nvidia on your deployments, you unlock the full throughput of your hardware—slashing LLM response latencies from 30+ seconds down to milliseconds.
Ready to Build Your Complete Private AI Stack?
GPU pass-through for Ollama is just one piece of the puzzle. To run an enterprise-ready local AI ecosystem, you also need:
Vector Databases (Qdrant / Milvus) for Document Retrieval-Augmented Generation (RAG).
Automated Cluster Diagnostics using K8sGPT with Telegram/Discord alerts.
High-Availability Storage (Longhorn) for persistent vector indices.
If you want to master these architectures step-by-step with production-ready manifests and real-world scripts, explore my complete course:
👉 Masterclass: From Zero to Private AI Cloud on K3s HA
Have questions about GPU passthrough, CNI configurations, or local LLM inference? Drop a comment below and let's troubleshoot together!
Top comments (0)