DEV Community: Vishnu Hari Dadhich

What the heck is an AI Agent?

Vishnu Hari Dadhich — Sat, 12 Sep 2026 18:45:00 +0000

Not a chatbot with more steps

“Agentic AI” gets used for basically anything with a system prompt now. It isn’t a chatbot that’s more polite about calling tools. The actual difference is structural: an agent runs a loop, a chatbot answers a turn.

A chatbot takes input, produces output, done. State lives in the conversation history you paste back in. An agent runs its own loop: observe, decide, act, observe again, using its own outputs to decide what to do next, without a human in the middle of every step.

Example. Ask a chatbot “is my API returning 500s right now” and it tells you it can’t check. Ask an agent the same thing and it calls a monitoring tool, reads the response, decides the error rate looks elevated, calls a log-search tool to pull recent 500s, reads those, and comes back with “yes, 12 in the last 10 minutes, all from the /checkout endpoint, here’s the stack trace.” Same model, same question. The difference is the loop in between.

The model doesn’t loop, the harness does

This is the part most explanations skip. The LLM itself is stateless and single-shot, feed it tokens, get tokens back. It has no concept of “keep going until the task is done.” The loop, the tool execution, the deciding-when-to-stop, none of that lives in the model weights.

All of that lives in the agent harness , the code wrapped around the model that turns single-shot completions into a running agent. The harness is what:

Sends the prompt to the model and gets a response back
Parses that response for a tool call
Actually executes the tool (runs the shell command, hits the API, reads the file)
Feeds the result back into context
Decides whether to call the model again or stop

Swap the model and keep the harness, you get the same agent with different reasoning quality. Swap the harness and keep the model, you get a completely different agent, different tools available, different context management, different stopping conditions. The harness is doing more of the actual engineering work than people give it credit for, the model is just the decision-maker it calls into on each turn.

Strip the harness out entirely and there’s no agent left, just a model that can produce a tool-call-shaped output with nothing to execute it, feed a result back, or call the model again. In RL terms, the model is the policy, the harness is the environment loop that runs the policy against the world and feeds observations back. A policy with no environment isn’t an agent, it’s a function you can call once.

This is also why “build your own agent” and “build your own harness” are the same task described two ways. There’s no shortcut where you write an agent and skip the harness, hand-rolling one just means you’re the one writing the loop: send the prompt, parse the response for a tool call, execute it, append the result, decide whether to call the model again. Do that in 40 lines of Python or reach for pi.dev or Claude Code’s, it’s the same role either way.

Example. Two different harnesses wrapping the same underlying model can behave nothing alike. One harness might cap the loop at 5 iterations and summarize aggressively to save context. Another might allow 50 iterations, keep full history, and let the model spawn sub-agents. Same model, same weights, same API calls to it, wildly different agent because the harness around it is different.

The four things a harness provides

Calling the LLM as decision-maker. The harness sends the current state to the model and treats its output as a decision about what to do next, not just a completion to display.

A tool/function-calling interface. The harness defines what tools exist, executes them when the model calls one, and returns results in a format the model can read.

Memory/state across steps. The harness owns the context window, what goes in, what gets summarized or dropped, and any external memory store.

Loop control. The harness decides when to call the model again and when to stop, whether that’s a fixed iteration cap, a “the model said it’s done” signal, or something more custom.

Example, walking through one iteration. User asks the agent to find the total size of log files older than 30 days. Step 1, harness sends the request to the model, model decides it needs to list files, returns a tool call for find /var/log -mtime +30. Step 2, harness executes that command, gets a list of paths, appends the result to context. Step 3, harness calls the model again, model decides it needs sizes, returns a tool call for du -ch on those paths. Step 4, harness executes it, appends the total to context. Step 5, harness calls the model once more, model has enough info, returns a final answer with no further tool call, harness detects that and stops the loop. Five round trips through the harness, one user question, no human in between.

Why the loop gets expensive

Every loop iteration appends to context: the tool call, the tool result, the model’s next reasoning step. None of that gets discarded between iterations, unless the harness explicitly manages it.

Tie this back to KV cache: cache grows with every token in context, and context in an agent loop grows every single step, not just per conversation turn. A single-shot inference request builds a cache once and discards it after. An agent loop keeps extending the same cache, iteration after iteration, and the cost compounds because every added token gets re-attended-to on every subsequent forward pass.

Worked example. Say a tool call and its result add roughly 300 tokens to context per iteration. A 10-iteration loop adds 3,000 tokens on top of the original prompt, all of it sitting in KV cache. Run the log-file-size agent above against a directory tree with a lot of nested paths and that 300-token estimate climbs fast, since raw find and du output isn’t exactly compact.

Practical implication: a long-running agent has to actively manage context, not just let it grow unbounded. This is a harness responsibility, not a model one, good harnesses summarize old steps, drop stale tool outputs, and truncate before the sequence gets unwieldy. That’s not an optimization, it’s what keeps a loop viable past a handful of iterations regardless of what it’s running on.

Single-shot vs loop, at the token level

Single-shot: prompt in, tokens out, KV cache built once, discarded after.

Agent loop: prompt in, tokens out (including a tool call), harness executes the tool outside the model, tool result gets tokenized and appended by the harness, full sequence goes back through the model, cache either recomputed from scratch or extended depending on the serving setup. Repeat.

The cost isn’t “the model thinks harder.” It’s “the model re-processes a longer sequence every iteration.” That’s the whole reason agent loops cost more to serve than a single chatbot request, and why serving techniques like continuous batching and prefix caching (vLLM does both) matter more for agents than for one-off completions.

Some harnesses and agents you’ve probably heard of

Frameworks like CrewAI and LangGraph are toolkits for building your own harness, not already-built ones. Since the interesting question is which finished harnesses and agents are worth knowing, here’s a rough split, open source and paid, as of when this was written:

Open source / self-hostable harnesses

Pi (pi.dev), a minimal, aggressively extensible terminal coding-agent harness. Deliberately skips features like sub-agents, plan mode, and MCP support out of the box, the pitch is you build those in yourself with extensions rather than accept whatever the harness maker decided. It also makes the model-vs-harness split from earlier concrete: it supports 15+ model providers and lets you switch mid-session, the harness stays constant, the model underneath it doesn’t have to.
Hermes Agent, an open-source agent from Nous Research that can run locally or on a server and build reusable skills from previous work.
OpenHands (formerly OpenDevin), an open-source autonomous software engineer harness, the open equivalent of Devin below.
OpenClaw, a fast-growing self-hosted personal agent harness, notable for running locally with your own model of choice.
AutoGPT, the original viral agent demo, now a maturer platform with a visual builder and self-hosting support. Still the reference point most people mean when they say “autonomous agent.”

Paid / proprietary harnesses and agents

Claude Code, Anthropic’s coding agent harness, runs in a terminal, reads and edits across a whole codebase, runs tests, commits changes.
Devin, a fully autonomous coding agent from Cognition, runs in its own sandboxed cloud environment rather than your terminal.
OpenAI Codex / ChatGPT Agent, OpenAI’s equivalents, spanning terminal, cloud, and chat surfaces.
Perplexity Comet, a browsing agent that navigates and completes tasks inside the browser rather than a terminal.

Coding is where agents and harnesses are most mature right now, since code execution gives the loop a fast, checkable signal of whether the last action actually worked. That’s not a coincidence, it’s the same reason the log-file-size example earlier in this post works cleanly: shell commands succeed or fail in an unambiguous way, which is exactly the kind of feedback a harness’s loop needs to decide what to do next.

Up next

Next post in the local tools series is the practical version of this: standing up a Hermes-style agent, harness, tool calling, loop, the works, on the local setup already covered in this series. This post is the vocabulary you need before that one makes sense.

What Happens When You Ask an LLM a Question

Vishnu Hari Dadhich — Fri, 11 Sep 2026 04:00:00 +0000

I have been running models with Ollama and vLLM. Then I stopped to ask: what is actually happening behind that API call?

A learning chat turned into these notes. Let’s follow one request all the way through:

Why did my Kubernetes pod restart? The container’s last termination reason is OOMKilled.

This is an illustrative example, not a live diagnosis. We will use a hypothetical 2B model throughout. Technically, the container restarted within the pod.

1. An answer grows one token at a time

An LLM , or large language model, is a neural network trained on language. For the text-generating models here, the basic loop is: use the text so far to predict what comes next.

Our pod question + the OOMKilled clue
                  |
                  v
             Tokenizer
          text -> token IDs
                  |
                  v
      Embeddings + position information
                  |
                  v
        Transformer layer 1
        attention + other maths
                  |
                  v
        Transformer layer 2
                  |
                 ...
                  |
                  v
         Scores for next tokens
                  |
                  v
           Choose one token
                  |
                  v
        Append it to the answer
                  |
                  v
           Stop condition?
            / \
          yes no
           | |
           v v
     Final answer Process chosen token
                    through the model
                    using the KV cache
                         |
                         +--> Score and choose again

A token can be a word, part of a word, or punctuation. The tokenizer assigns each piece an ID. An embedding turns that ID into a list of numbers the network can process. The model also needs information about token positions.

For our prompt, a tokenizer might split OOMKilled into smaller pieces. Exact splits depend on the tokenizer. Each piece gets an ID, then a numerical vector.

The answer might grow as The → container → was → killed. These are illustrative token boundaries. The model chooses from next-token scores using its generation settings, then repeats.

2. What does a 2B model actually contain?

About two billion adjustable numbers , called parameters. Most are weights used in the network’s calculations. They collectively encode learned patterns, rather than one fact per number.

TRAINING
Examples about language, code, containers...
                  |
                  v
        Predict the next token <---------+
                  | |
                  v |
       Compare with training target |
                  | |
                  v |
           Calculate the error |
                  | |
                  v |
         Adjust learned numbers ---------+

INFERENCE
Our pod question + learned numbers
                  |
                  v
       Calculations through layers
                  |
                  v
       Generate a restart explanation
       (learned numbers stay fixed)

A neural network is a stack of mathematical operations. A simple neuron combines weighted inputs, adds a bias, and applies an activation function. Many such operations let the network learn complex patterns.

During training, examples about containers and memory can shape these numbers. Our question then uses those learned patterns to connect OOMKilled with an out-of-memory event. There is no single “Kubernetes parameter.”

Training changes the parameters. Ordinary inference uses them. Chatting adds context; it does not normally retrain the model. A bigger parameter count alone does not guarantee better answers.

3. Transformers connect the relevant pieces

A Transformer is the architecture behind many LLMs. Its layers combine attention with other neural-network calculations.

Attention mixes information from tokens in the available context. In a typical text generator, a token can attend to itself and earlier tokens.

Token representations from our prompt
                  |
          +-------+-------+
          | | |
          v v v
          Q K V
       Queries Keys Values
          | | |
          +---+---+ |
              | |
              v |
      Compare Q with K |
              | |
              v |
      Mask future tokens |
      and form weights |
              | |
              +-----+-----+
                    |
                    v
       Weighted mixture of values
                    |
                    v
        Further layer calculations
                    |
                    v
      Later layers -> next-token scores

In our request, OOMKilled is a useful clue for explaining restart. Attention helps combine information from those positions while the layers build the response.

The diagram is an intuition, not a measured attention map. Q, K, and V are learned numerical representations , not literal questions or database entries. Multiple attention heads can capture different relationships. Google’s Transformer introduction explains the architecture further.

4. Why can it reason, and still be wrong?

Learning to predict language can build useful patterns for code, maths, and problem solving. Further training can improve instruction following and reasoning.

A model can use intermediate steps to work through a problem. Some reasoning models spend additional computation before answering. This can help, but a fluent explanation is not proof.

Question: why did the container restart?
                  |
                  v
       Supplied clue: OOMKilled
                  |
                  v
      Learned relationship: memory
                  |
                  v
      Suggest checking limit and usage
                  |
                  v
       Verify against the real cluster

This is an example of a useful explanation, not a trace of the model’s hidden internal computations.

A useful answer: “OOMKilled indicates an out-of-memory kill. Check the container’s memory limit and memory usage.”

It would be a leap to say “Your app definitely has a memory leak.” Our prompt gives no evidence of a leak. The model also has not inspected the cluster; it only has the information we supplied.

5. Context is the input. KV cache saves work.

The context window limits how many tokens a request can accommodate, including input and generated output. Instructions, included chat history, and supplied documents all take space.

Our question + OOMKilled clue
              |
              v
           PREFILL
     Process the prompt
       / \
      v v
 Save prompt K/V Score first token
      | |
      v v
  [KV cache] Choose "The"
      | |
      | v
      +---------> DECODE <----------------+
      | Process "The" |
      | using earlier K/V |
      | | |
      | v |
      | Save new K/V |
      | Score and choose |
      | next token |
      | | |
      | v |
      | Stop condition? |
      | / \ |
      | yes no |
      | | | |
      | v +-----------+
      | Finish answer Process next token
      |
      +-- Cache grows as more tokens are processed

Prefill processes the prompt and produces the scores for the first output token. Decode continues generation, typically one token per sequence per step.

The KV cache stores keys and values from earlier tokens so the model can reuse them. It is temporary attention data, not a permanent memory of you.

For our request, prefill reads the question and OOMKilled clue. Decode builds the answer. When generating the next piece after The container, the model reuses cached keys and values from earlier tokens.

Pasting 500 lines of pod logs would add input tokens and usually increase cache needs. It would not add parameters to the model.

Two useful measurements:

Metric	What I notice
Time to first token (TTFT)	Wait until “The” appears, including queueing and prompt processing
Output tokens per second	How quickly the rest of the restart explanation appears

6. Why does a model need so much memory?

GPUs accelerate the large matrix calculations. But the model also has to fit in memory.

Quantization represents numbers using fewer bits. The chart shows ideal weight storage: parameters multiplied by bits, divided by eight. Actual formats add overhead and may keep some weights at higher precision.

FP16 and BF16 both use 16 bits, with different numerical ranges and precision. INT4 uses 4-bit integers. AWQ and GPTQ are quantization methods.

For our 2B model, 16-bit weights take about 4 GB; ideal 4-bit storage takes about 1 GB. The question stays the same. We store the learned numbers more compactly, leaving more of my 8 GB GPU for the cache and runtime. Whether it fits depends on the model, format, and context. Quality can fall; speed gains depend on hardware and software.

7. What does vLLM add?

Our pod question (request A)
            |
            v
     vLLM request queue
            |
            v
 Scheduler chooses work <-------------------+
            | |
            v |
 Batch of scheduled tokens from A, B, C |
            | |
            v |
 GPU runs model using weights + KV cache |
            | |
            v |
 Return generated tokens to each user |
            | |
            v |
 Finished? -- no: schedule more work -------+
     |
    yes
     |
     v
 Release request resources

vLLM is serving software that runs a model efficiently. Our pod question is request A. Other users send requests B and C. vLLM can process them together while maintaining each request’s own context.

Continuous batching updates the active batch as requests finish and capacity becomes available. PagedAttention manages KV cache in blocks that need not sit together in memory, reducing wasted space. The vLLM team’s explanation connects these ideas to serving more requests.

If B finishes while our restart explanation is still generating, D can join when capacity allows. Our request A continues, with its own KV blocks.

8. More GPUs, and fewer active experts

Split maths, split layers, or serve separate requests

TENSOR PARALLELISM
One layer's input for our question
             |
       +-----+-----+
       v v
     GPU 1 GPU 2
    part A part B
       | |
       +-----+-----+
             |
       Communicate / combine
             |
         Next layer

PIPELINE PARALLELISM
Our question -> GPU 1 -> GPU 2 -> Output
             early later
             layers layers

DATA PARALLELISM FOR SERVING
              Request routing
               / \
              v v
        Our question Other request
              | |
              v v
        Model copy 1 Model copy 2
              | |
              v v
        Our answer Their answer

Each replica can itself use multiple GPUs. For example, four GPUs could run two replicas, with two GPUs per replica.

For the same pod question: TP shares each layer’s calculations across GPUs; PP passes the work through groups of layers; DP sends our whole request to one model replica while another serves someone else. These are possible layouts, not a claim that our small model needs multiple GPUs.

NCCL is NVIDIA’s GPU communication library. It can move and combine data over connections such as PCIe and NVLink. Extra GPUs can also add waiting time, so scaling is not automatically a speedup. NVIDIA’s overview describes that communication layer.

How GPU communication connects the pieces

GPU 1 partial result GPU 2 partial result
          | |
          +------------+------------+
                       |
              Collective operation
                 (for example,
                NCCL AllReduce)
                       |
          +------------+------------+
          v v
GPU 1 combined result GPU 2 combined result

Communication travels over available connections,
such as NVLink or PCIe, depending on the hardware.

MoE routes work to selected experts

Representation of a token from our question
                       |
                       v
                  Learned router
                       |
               +-------+-------+
               v v
            Expert 2 Expert 7
               | |
               +-------+-------+
                       |
                       v
             Combine expert outputs
                       |
                       v
              Continue through layers

Other experts are not selected for this token
in this illustrative MoE layer.

MoE is an alternative architecture, not a serving switch for our dense 2B model. A mixture-of-experts model answering the same question selects a subset of expert networks for each token at an MoE layer. The routing is learned; experts are not necessarily named subject specialists. For a token in our question, a router might select experts 2 and 7. That does not make either a “Kubernetes expert.” Fewer active parameters reduce computation, but all weights still need storage somewhere. Mixtral’s paper provides a concrete example.

Memory capacity answers “Will it fit?” Memory bandwidth answers “How fast can data move?” Compute throughput answers “How fast can the maths run?” Any of these, plus GPU communication, can limit performance.

The whole conversation flow

The app supplies the conversation history it wants the model to use. The next turn is another request, with an updated context.

MODEL SETUP OUR CONVERSATION
Learned parameters Pod question + OOMKilled
       | |
       v v
Choose weight format App assembles context
(optional quantization) instructions + messages
       | |
       v v
Load model on GPU(s) Tokenize and schedule
       | |
       +----------------+-------------------+
                        |
                        v
             Prefill through the model
             Build/reuse available KV cache
                        |
                        v
                  First output token
                        |
                        v
             Decode + extend KV cache
             Repeat until stopping
                        |
                        v
             Restart explanation to user
                        |
                        v
           Follow-up: "What should I check?"
                        |
                        v
            App includes relevant history
            plus this follow-up question
                        |
                        v
                  Next request

Cache reuse between requests depends on the engine and matching context. Even without reuse, the app can send the history again and the engine can recompute it.

The five things I want to remember

Term	My reminder
Parameters	Learned numbers linking patterns such as OOMKilled and memory
Tokens	Pieces of our question and restart explanation
Attention	Combine the restart question with the OOMKilled clue
KV cache	Reuse earlier calculations as the explanation grows
Inference engine	Schedule our request and run the model

Our final answer might be:

The container was killed because of an out-of-memory event. Check its memory limit and usage to investigate why.

One question, learned numbers, context, repeated calculations, and an answer to verify against the cluster.

Run vLLM on Kubernetes with Minikube, WSL2 and NVIDIA GPU

Vishnu Hari Dadhich — Tue, 08 Sep 2026 05:30:00 +0000

This is the vLLM entry in my local AI series. After testing Ollama, llama.cpp, and FreeToken, I wanted to run vLLM as a Kubernetes Deployment on my Windows/WSL2 setup.

The plan was simple: deploy vLLM, request nvidia.com/gpu: 1, expose an OpenAI-compatible API endpoint through a Service, and tie it into the Kubernetes workflows I write about regularly.

Getting a GPU into Kubernetes on WSL2 turned into an investigation. Not because vLLM is hard, but because container runtimes and nested clusters handle GPU passthrough in non-obvious ways. Here is what happened, how the device plugin and node prerequisites work, and how to get a working GPU cluster running with Minikube.

What You’ll Build

Windows
   │
   ▼
WSL2 Ubuntu
   │
   ▼
Docker Engine + NVIDIA Container Toolkit
   │
   ▼
Minikube
   │
   ▼
Kubernetes
   │
   ├── NVIDIA GPU Operator
   │
   ├── NVIDIA Device Plugin
   │
   └── vLLM
         │
         ▼
      NVIDIA GPU

By the end of this guide, you’ll have vLLM running on Kubernetes with GPU acceleration and serving an OpenAI-compatible API.

The Test Rig

My test machine for this run:

Host Machine: Windows 11 Laptop
GPU: NVIDIA GeForce RTX 4070 Laptop GPU (8.0 GiB VRAM)
Host Memory: 32 GB DDR5
WSL2 Environment: Ubuntu 24.04 LTS (noble) with systemd enabled
NVIDIA Drivers: Driver version 616.56 / CUDA 13.4 user-mode driver
Kubernetes: Minikube v1.39.0 provisioning Kubernetes v1.37.0
Model: Qwen3.5-0.8B (Qwen/Qwen3.5-0.8B) served by vLLM v0.28.0

Troubleshooting vLLM GPU Support on Docker Desktop WSL2 Kubernetes

Docker Desktop offers a one-click Kubernetes cluster in its settings. Turning it on provisions a single-node cluster named desktop-control-plane running on containerd://2.3.4.

On the WSL2 host, nvidia-smi works fine, and running standalone GPU containers (docker run --gpus all) works without issues. The next step was applying the standard NVIDIA Kubernetes device plugin DaemonSet to enable GPU scheduling:

kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.16.2/deployments/static/nvidia-device-plugin.yml

The DaemonSet pod started, but immediately stalled in a retry loop:

E... Incompatible strategy detected auto
E... If this is a GPU node, did you configure the NVIDIA Container Toolkit?
I... No devices found. Waiting indefinitely.

Checking the node capacity confirmed the issue:

kubectl describe node desktop-control-plane | grep -A5 "Capacity:\|Allocatable:"

Checking Docker Desktop’s desktop-control-plane: nvidia-smi finds the RTX 4070, but the node exposes zero GPU capacity.

In this tested Docker Desktop Kubernetes setup, GPU resources were not exposed to the Kubernetes node. Despite GPU access working in WSL2 and standalone Docker containers, the Kubernetes node could not expose nvidia.com/gpu to the device plugin.

How Kubernetes Detects NVIDIA GPUs Using the NVIDIA Device Plugin

The NVIDIA Kubernetes Device Plugin is a DaemonSet that exposes GPU hardware to the Kubernetes control plane. It does not run inference, and it does not install drivers or container runtimes.

Under standard Kubernetes GPU scheduling, pods request GPU resources by specifying limits for extended resources. The device plugin integrates with Kubelet through the Kubernetes Device Plugin API over gRPC, using a Unix domain socket at /var/lib/kubelet/device-plugins/kubelet.sock. Its responsibility comes down to three functions:

Discovery: The plugin queries the host system using NVML (NVIDIA Management Library) to check how many physical GPUs are present and verify their health status.
Registration: It registers with Kubelet and advertises the discovered GPUs as an extended allocatable resource named nvidia.com/gpu.
Allocation: When a pod requesting nvidia.com/gpu: 1 gets scheduled to the node, Kubelet calls the plugin’s Allocate gRPC endpoint. The plugin picks a healthy GPU and sends back the device IDs and environment variables (NVIDIA_VISIBLE_DEVICES=<UUID>) to Kubelet. Kubelet then passes those variables to the container runtime so the container gets access to /dev/nvidia*.

Worker Node Prerequisites

Because the device plugin only handles discovery and Kubelet registration, it assumes the worker node already has a functional GPU stack. For worker nodes in any Kubernetes cluster, three layers must be in place before the device plugin can run:

Host Kernel Drivers: The node OS must have the NVIDIA kernel modules loaded (nvidia.ko, nvidia-uvm.ko) and device nodes created in /dev. Running nvidia-smi on the host must return clean output.
NVIDIA Container Toolkit: Packages such as libnvidia-container and nvidia-container-toolkit must be installed on the host. Refer to the NVIDIA Container Toolkit install guide for distribution setup. This toolkit includes the OCI prestart hook that inspects container environment variables and mounts host GPU driver libraries into the target container.
Runtime Configuration: The node container runtime (containerd or Docker) must have the NVIDIA runtime configured in its daemon settings (such as /etc/containerd/config.toml or /etc/docker/daemon.json) and set up to handle CDI (Container Device Interface) or NVIDIA runtime hooks.

If any of those three pieces is missing or unconfigured, the device plugin pod will fail during startup or report zero allocatable GPUs.

Why Docker Desktop Kubernetes Cannot Detect My NVIDIA GPU

Based on the runtime architecture observed in this setup, the issue comes down to how Docker Desktop isolates its cluster node.

desktop-control-plane runs inside Docker Desktop’s private utility VM (docker-desktop), isolated using sysbox-runc rather than the standard OCI runtime runc. Sysbox provides nested container virtualization to spin up systemd, kubelet, and containerd within an unprivileged container environment.

In this setup, that isolation layer prevents GPU passthrough:

Missing OCI Passthrough: The node container was not launched with NVIDIA GPU passthrough flags (--gpus all). Inside the sandbox, containerd cannot access NVIDIA device nodes (/dev/nvidia*) or the WSL2 DirectX driver mapping (/usr/lib/wsl/lib).
Locked-Down Lifecycle: You cannot configure the NVIDIA Container Toolkit inside the node’s containerd because Docker Desktop internally supervises the container.
Daemon Defaults Do Not Propagate: Setting nvidia as the default OCI runtime in docker-desktop WSL does not alter the sysbox-runc runtime used by the Kubernetes node container.

Docker has an open roadmap issue requesting native GPU passthrough for Docker Desktop Kubernetes. It remains an architectural boundary in the current setup.

Because of these issues, I pivoted to running Minikube with WSL2 using native Docker.

Installing Docker CE and Containerd Inside WSL2

To satisfy those node prerequisites and bypass Docker Desktop’s VM isolation, we install the native Docker Community Edition engine directly inside the Ubuntu 24.04 WSL2 environment:

# Add Docker's official GPG key
sudo mkdir -p /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Set up the repository
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu noble stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# Install Docker CE and containerd
sudo apt update
sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Configuring the official Docker CE repository and installing containerd and the Docker daemon inside Ubuntu WSL2.

Verify that both the Docker daemon and containerd services are running cleanly under systemd:

systemctl status docker
systemctl status containerd

Confirming active (running) status for both docker.service and containerd.service under WSL2 systemd.

Before moving to Kubernetes, sanity check that native Docker has full GPU access through the WSL2 NVIDIA container runtime:

docker container run --gpus all --rm nvidia/cuda:13.3.1-base-ubuntu26.04 nvidia-smi -L

Direct GPU passthrough test: Docker successfully identifies GPU 0 as the NVIDIA GeForce RTX 4070 Laptop GPU.

Run Minikube with NVIDIA GPU Support on WSL2

Install the latest Minikube Debian package:

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube_latest_amd64.deb
sudo dpkg -i minikube_latest_amd64.deb

Installing Minikube v1.39.0 via Debian package.

Following the official Minikube GPU documentation, starting Minikube with GPU acceleration requires specific driver and runtime parameters. Now start Minikube. Two flags matter here and both are easy to miss:

minikube start \
  --driver=docker \
  --container-runtime=docker \
  --gpus=all \
  --memory=12g \
  --cpus=10 \
  --extra-config=apiserver.service-account-issuer=https://kubernetes.default.svc \
  --extra-config=apiserver.service-account-signing-key-file=/var/lib/minikube/certs/sa.key

Minikube starting Kubernetes v1.37.0 on Docker, automatically detecting and enabling the nvidia-device-plugin addon.

[!IMPORTANT] Why --container-runtime=docker is required: By default, even when using --driver=docker, Minikube configures containerd as its internal in-node runtime. However, Minikube’s --gpus=all flag currently hooks into Docker’s OCI runtime wrapper. If you omit --container-runtime=docker, Minikube fails during preflight checks with an invalid flag combination error.

Notice the startup output: Minikube automatically detects the GPU and enables the nvidia-device-plugin addon for you.

Verifying the Cluster and GPU Allocation

Install kubectl to talk to the cluster:

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
kubectl version --client
kubectl get nodes
kubectl get pods -A

Node minikube is Ready (v1.37.0) and the nvidia-device-plugin-daemonset is 1/1 Running in kube-system.

Next, inspect the node’s schedulable capacity:

kubectl describe node minikube | grep -A7 "Capacity:\|Allocatable:"

Cluster verification: nvidia.com/gpu: 1 is now officially registered in Capacity and Allocatable.

A quick test pod confirms the GPU is reachable from inside Kubernetes:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: gpu-test
spec:
  restartPolicy: Never
  containers:
    - name: cuda-test
      image: nvidia/cuda:13.3.1-base-ubuntu26.04
      command: ["nvidia-smi", "-L"]
      resources:
        limits:
          nvidia.com/gpu: 1
EOF

Check the test pod logs:

kubectl logs gpu-test

Test pod running nvidia-smi inside the Kubernetes cluster and successfully accessing the RTX 4070.

Real GPU scheduling and nvidia.com/gpu resource accounting, without Docker Desktop getting in the way.

Deploy vLLM on Kubernetes with NVIDIA GPU

Model choice for 8 GB VRAM: Qwen3.5-0.8B (Qwen/Qwen3.5-0.8B). Small enough to leave real headroom for vLLM’s KV cache, which is the main reason to use vLLM over llama.cpp or Ollama in the first place, and it is a capable model at that size. As detailed in the vLLM documentation, the server exposes an OpenAI-compatible API endpoint over HTTP.

The Shared Memory (`/dev/shm`) Gotcha

When running vLLM via the Docker CLI, passing --ipc=host lets workers exchange tensors and state across processes using host shared memory. In Kubernetes, pods do not share the host IPC namespace by default, and Kubernetes provisions /dev/shm as a minimal 64 MB tmpfs mount.

Some PyTorch and vLLM multiprocessing workloads can require significantly more shared memory than Kubernetes’ default /dev/shm (which defaults to a minimal 64 MB tmpfs mount). If insufficient shared memory is available, initialization or worker processes may fail. The Kubernetes solution is mounting an emptyDir volume backed by host RAM (medium: Memory) with a dedicated sizeLimit:

# vllm-qwen-k8s.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-qwen
  labels:
    app: vllm-qwen
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-qwen
  template:
    metadata:
      labels:
        app: vllm-qwen
    spec:
      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.28.0
          args:
            - "--model=Qwen/Qwen3.5-0.8B"
            - "--gpu-memory-utilization=0.7"
            - "--max-model-len=8192"
          ports:
            - containerPort: 8000
          resources:
            limits:
              nvidia.com/gpu: 1
          volumeMounts:
            - name: hf-cache
              mountPath: /root/.cache/huggingface
            - name: dshm
              mountPath: /dev/shm
      volumes:
        - name: hf-cache
          emptyDir: {}
        - name: dshm
          emptyDir:
            medium: Memory
            sizeLimit: 2Gi
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-qwen
spec:
  selector:
    app: vllm-qwen
  ports:
    - port: 8000
      targetPort: 8000
  type: ClusterIP

Two practical notes on this manifest:

Image version pinning: I pin the image version (v0.28.0) here so the deployment remains reproducible. You can update the image tag after validating compatibility with your CUDA and model version.
Hugging Face cache: emptyDir is used here for simplicity. The Hugging Face model cache is lost when the pod is recreated. For repeated testing or larger models, consider using a PersistentVolumeClaim (PVC) or a host-mounted path.

Apply the manifest and watch the deployment roll out:

kubectl apply -f vllm-qwen-k8s.yaml
kubectl get pods,svc

Applying vllm-qwen-k8s.yaml and confirming the vLLM pod reaches 1/1 Running status alongside the ClusterIP service.

Stream the pod logs to inspect the vLLM initialization sequence:

kubectl logs vllm-qwen-5c8bbc786f-gdpmj -f

vLLM v0.28.0 engine initialization: resolving Qwen3_5ForConditionalGeneration, setting max model length to 8192, and warming up CUDA graphs.

Once PyTorch compilation and CUDA graph capture finish, the ASGI application completes startup and begins listening on port 8000:

vLLM application startup complete: exposing OpenAI-compatible endpoints (/v1/chat/completions) with live throughput metrics.

Testing Inference and Hardware Telemetry

Forward port 8000 from the cluster service to the local machine:

kubectl port-forward svc/vllm-qwen 8000:8000

Now issue an OpenAI-standard chat completion request using curl:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3.5-0.8B",
    "messages": [{"role": "user", "content": "Say hello in one sentence."}]
  }'

Inference test: 48 tokens generated cleanly through the OpenAI-compatible vLLM API running in Minikube.

The response returned without issues, with server logs showing generation throughput averaging 23.5 tokens/s.

Finally, checking Windows Task Manager confirms the hardware utilization on the host:

Windows Task Manager telemetry: 4.4 GB of 8.0 GB dedicated VRAM allocated to the WSL2 vLLM pod at 45°C.

Dedicated GPU memory held at 4.4 / 8.0 GB (in line with the --gpu-memory-utilization 0.7 setting plus desktop display overhead), with the GPU temperature at 45°C.

Beyond Local Dev: Why Production Clusters Use the NVIDIA GPU Operator

For a single-node laptop or a Minikube sandbox, managing drivers on the host and running a standalone device plugin DaemonSet gets the job done.

In production Kubernetes clusters, manual node configuration does not scale. When you operate multi-node clusters across cloud providers or bare metal, nodes get provisioned dynamically by autoscalers, Linux kernel patch levels drift, and different GPU architectures (such as A100, H100, or L40S) coexist in the same cluster.

This is why production setups deploy the NVIDIA GPU Operator instead of managing the device plugin directly.

The GPU Operator automates the deployment and lifecycle management of the NVIDIA software stack required for GPU-enabled Kubernetes nodes. Driven by the ClusterPolicy Custom Resource Definition, it manages several components depending on your cluster configuration:

Driver Management: Depending on the operating environment, the Operator can compile and load the NVIDIA driver container matched to the node kernel, or use pre-installed host drivers (common in managed Kubernetes environments like GKE, EKS, or AKS).
Container Toolkit and CDI Configuration: Depending on the Operator version and runtime configuration, GPU injection can be configured through direct NVIDIA runtime integration or CDI-based device injection (Container Device Interface), which modern releases enable by default.
Device Plugin Supervision: It deploys and manages the lifecycle of the NVIDIA Kubernetes Device Plugin DaemonSet, keeping it in sync with driver and runtime availability.
Node Feature Discovery (GFD): It automatically detects physical GPU capabilities and labels each node (such as nvidia.com/gpu.product: NVIDIA-A100-SXM4-80GB or nvidia.com/gpu.family: ampere). Workloads can then target specific GPU models using standard Kubernetes nodeSelector or affinity rules.
Cluster Metrics with DCGM Exporter: It runs the Data Center GPU Manager (DCGM) exporter to collect metrics (GPU compute utilization, VRAM usage, temperature, power draw, and memory bandwidth) and expose them to Prometheus for cluster monitoring and alerting.
MIG Management: On enterprise GPUs that support Multi-Instance GPU (MIG), the Operator can dynamically partition a single physical GPU into isolated hardware instances, allowing smaller workloads to share an A100 or H100 without memory interference.

In short: the device plugin provides basic GPU scheduling for pods. The GPU Operator manages the operational lifecycle, monitoring, and driver stack required to run GPU infrastructure reliably in production.

Key Takeaways

Docker Desktop Kubernetes isolates nodes without GPU passthrough: In tested setups, desktop-control-plane runs in a nested sysbox-runc sandbox without GPU device passthrough flags.
Minikube on native Docker CE in WSL2 works reliably: Launching Minikube with --driver=docker --container-runtime=docker --gpus=all automatically activates the NVIDIA device plugin addon.
Understand device plugin vs node prerequisites: The device plugin only manages discovery and Kubelet allocation. Host drivers, Container Toolkit, and runtime configuration must already exist on the worker node.
Always size /dev/shm in Kubernetes: Multiprocessing in PyTorch and vLLM quickly exceeds the default 64 MB tmpfs mount. Mount an emptyDir memory volume backed by host RAM.
Budget VRAM carefully: Setting --gpu-memory-utilization 0.7 on a compact model like Qwen3.5-0.8B preserves headroom for the KV cache on an 8 GB GPU.
Deploy the NVIDIA GPU Operator in production: Production multi-node clusters rely on the GPU Operator to manage drivers, CDI device injection, node feature discovery, and DCGM monitoring.

Thanks for reading! If you run into issues setting up GPU passthrough on your machine, have questions, or have suggestions for other local AI setups to explore, leave a comment below.

Running a 35B MoE Model on an 8 GB Laptop GPU: Testing FreeToken

Vishnu Hari Dadhich — Mon, 07 Sep 2026 06:30:00 +0000

Running large language models locally usually comes down to a hard hardware boundary: video RAM. If a model does not fit into your GPU’s VRAM, performance usually falls off a cliff as soon as standard runtimes fall back to system memory over the PCIe bus.

Mixture of Experts (MoE) architectures offer an attractive theoretical escape hatch. While the total parameter count can be large (30B to 70B+), only a sparse subset of expert layers activates for any given token. However, standard local runtimes still require loading the entire weight footprint into memory, which puts 30B+ models out of reach for everyday consumer laptops with 8 GB of VRAM.

Enter FreeToken by FlashML, backed by their research paper (arXiv:2608.16157). FreeToken is an edge inference runtime designed specifically to run frontier MoE models on consumer hardware by dynamically managing a hierarchical cache between GPU VRAM and host RAM.

I recently downloaded FreeToken to put their claims to the test on my modest laptop GPU. Here is what the setup looked like, how FreeToken handles memory under the hood, and the real-world generation numbers I observed.

The Test Rig

My test machine is a portable laptop, not a high-end multi-GPU workstation:

CPU: 13th Gen Intel Core i9-13900H (14 cores, 20 threads)
GPU: NVIDIA GeForce RTX 4070 Laptop GPU (8.0 GiB VRAM)
Host RAM: 32 GB DDR5 (31.7 GiB usable)
Operating System: Windows 11
Runtime: FreeToken Desktop (v0.2.0-beta.17)

The FreeToken landing page highlights bringing frontier models to consumer edge hardware.

On paper, an 8 GB VRAM budget makes running a 35-billion parameter model look impossible. In traditional setups, 8 GB VRAM limits you to 7B or 8B parameter models in 4-bit quantizations (such as Q4_K_M). Attempting to load a 35B model typically triggers out-of-memory errors or slows inference to a crawl.

Installing FreeToken and Exploring the Library

The FreeToken desktop app is available to download directly from the FlashML website. It provides a self-contained installer that is quick to set up on Windows. When you launch it, the interface automatically detects your hardware specs, available VRAM, and system RAM, and presents a curated library of models optimized for edge offloading.

FreeToken accurately detects the RTX 4070 Laptop GPU (8 GB) and 32 GB RAM, recommending compatible models.

For this test, I selected Qwen3.6-35B-A3B NVFP4 (nvidia/Qwen3.6-35B-A3B-NVFP4). This is a 35-billion parameter MoE model using NVIDIA’s 4-bit floating point (NVFP4) format, with an initial download size of 21.9 GiB.

Before loading the model, the console dashboard showed clean baseline resource usage:

GPU VRAM: 0.4 / 8.0 GiB
Host RAM: 11.0 / 31.7 GiB
GPU Temp: 42°C at idle

Idle console telemetry: 0.4 GiB VRAM used, ready for model weights.

Weight Conversion: The FTW Format

Once the 21.9 GiB download finished, FreeToken flagged the model as needing conversion before it could be launched.

Raw model download complete (21.9 GiB), prompting for weight conversion.

FreeToken converts raw Hugging Face weights into its proprietary format called FTW (FreeToken Weight format). During this process, the engine repacks and organizes the tensors into memory-mapped structures optimized for rapid streaming between host RAM and GPU memory.

Repacking raw weights into FTW format for fast memory streaming.

The conversion took just over a minute on the i9-13900H and NVMe storage, producing a compact 19.5 GiB weight package.

Conversion complete: 19.5 GiB ready for execution.

Loading the Model and Memory Allocation

Starting the model triggers the weight allocation phase. Instead of attempting to cram the entire 19.5 GiB into the 8 GB VRAM, FreeToken partitions the workload across both memory tiers.

Starting model and populating active weights into VRAM.

Once loaded, the memory profile was eye-opening:

VRAM Usage: 6.8 GiB / 8.0 GiB (85% utilization, leaving a healthy buffer for OS display compositing)
Host RAM Usage: 30.2 GiB / 31.7 GiB (95% utilization)
GPU Status: Running at 45°C, drawing only 7W at idle

Model running: 6.8 GiB in VRAM and 30.2 GiB in system RAM.

The Cache Architecture Under the Hood

The console view provides detailed insight into how FreeToken manages this memory footprint:

MoE Expert Cache: FreeToken allocated 916 active expert slots in VRAM out of a total pool of 10,240 slots, consuming 1.51 GiB of VRAM. As different experts are needed during generation, they are dynamically paged in and out from the host RAM cache.
KV Cache: 8K tokens of context were allocated in VRAM, consuming just 0.16 GiB.
Mamba State Slots: 24 state slots were reserved, using 1.44 GiB of VRAM.

Detailed cache breakdown: MoE expert cache, KV cache, and Mamba state slots in VRAM.

This hierarchical design is the core reason the model can run without crashing. Inactive experts reside in system RAM (filling ~30 GB of system memory), while the active attention layers, KV cache, and currently triggered experts reside in the 8 GB VRAM.

Real-World Performance & Generation Speed

To evaluate real-world performance, I tested a conversational prompt with reasoning enabled:

User: “hi, how are you ?”

The model engaged its full thinking process, taking 1 minute and 9 seconds to plan and structure its response, and then generated 393 tokens at 4.3 tokens per second.

Chat generation benchmark: 393 tokens generated at 4.3 tok/s with full chain-of-thought thinking.

While 4.3 tokens per second is not real-time voice conversational speed, it is remarkably steady for running a 35-billion parameter model on an 8 GB laptop GPU. For tasks like code review, background agent execution, document analysis, or local drafting, 4.3 tok/s is completely practical.

The model detail sheet confirms the architecture and licensing details:

Base Repository: nvidia/Qwen3.6-35B-A3B-NVFP4
Context Length: 262,144 tokens (256K context window)
License: Apache 2.0

Model details: 262K context window, NVFP4 quantization, and Apache 2.0 license.

Built-in Agentic Tooling & Local Endpoints

FreeToken also includes built-in support for developer tools and coding agents.

The Apps tab exposes both OpenAI-compatible and Anthropic-compatible local HTTP endpoints:

OpenAI Endpoint: http://127.0.0.1:1919/v1
Anthropic Messages Endpoint: http://127.0.0.1:1919/v1/messages

Local OpenAI and Anthropic compatible endpoints with one-click configurations for Claude Code, Codex, opencode, and more.

FreeToken provides quick configuration presets for popular coding assistants:

Claude Code: Configurable via local Anthropic endpoint routing
Codex / opencode / openclaw: One-click environment setups
Hermes / DeepSeek Harness: Direct command-line integration

Having drop-in compatibility with both OpenAI and Anthropic API schemas means you can point tools like Claude Code directly to your localhost port without running a reverse proxy.

Where FreeToken Fits

FreeToken is a relatively young product, but its approach solves a real constraint for local inference.

Strengths

Bypasses the VRAM limit: Running a 35B MoE model on an 8 GB laptop GPU without manual layer-by-layer offload tuning works out of the box.
Hierarchical caching: The dynamic MoE expert cache keeps VRAM usage around 6.8 GiB, preventing driver resets and CUDA out-of-memory crashes.
Local developer endpoints: Dual support for OpenAI and Anthropic endpoints makes integration with modern coding agents straightforward.
Simple setup on Windows: No manual compilation, no CUDA toolkit troubleshooting, and a clean desktop GUI.

Limitations

System RAM is the real requirement: While VRAM requirements drop significantly, your host RAM must be large enough to hold the repacked weights. On my machine, the model took 30.2 GB of system RAM. If your laptop only has 16 GB of RAM, you will not be able to run 35B models this way.
Initial conversion step: Each new model requires a one-time repacking step into the FTW format before the first run.
Generation throughput: At roughly 4.3 tokens per second, it is well suited for asynchronous workloads, coding agents, and complex reasoning queries, but not for instant conversational back-and-forth.

Wrap-up

If you have a laptop with 32 GB of system RAM and a modest 8 GB NVIDIA GPU, FreeToken is worth checking out. It makes 35B-class MoE models runnable on consumer hardware that would otherwise choke on them, keeping your data completely local.

You can download the desktop app directly from FlashML, check out the project on GitHub, and read their research paper on arXiv:2608.16157.

Kubernetes is Adding a K to YAML: Why KYAML Makes Sense

Vishnu Hari Dadhich — Sat, 05 Sep 2026 19:00:00 +0000

Kubernetes 1.37 (“Garhwal”) just made KYAML Stable. YAML with a K bolted on, and it’s a genuinely useful feature.

The problem

YAML has two well known landmines.

Type coercion.

country: NO # becomes boolean false
enabled: yes # becomes boolean true
version: 3.10 # becomes float 3.1

Those values were meant to be strings. YAML quietly turned them into something else.

Whitespace as structure. Indentation defines meaning in normal YAML. Get one space wrong and the file still parses. It just means something else now, silently.

spec:
  containers:
  - name: app
    image: nginx
    ports:
    - containerPort: 80
      protocol: TCP # one space off, this attaches to the wrong container

No error. No warning. Just wrong.

Helm makes this worse, since you’re injecting indentation from outside the YAML context and hoping it lines up.

What KYAML actually changes

Nothing about the language. It’s YAML’s existing “flow style,” just enforced. Every KYAML file is valid YAML. Old tooling, old parsers, old kubectl versions all read it fine.

Three rules: objects use { }, arrays use [], strings are always quoted. Numbers and booleans stay bare, since they’re actually meant to be those types.

One example, everything at once:

# before
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 2
  template:
    spec:
      automountServiceAccountToken: false
      hostNetwork: true
      containers:
        - name: app
          image: nginx
          ports:
            - containerPort: 8080
          env:
            - name: REGION
              value: NO


# after (KYAML)
apiVersion: "apps/v1"
kind: "Deployment"
metadata: { name: "api" }
spec: {
  replicas: 2,
  template: {
    spec: {
      automountServiceAccountToken: false,
      hostNetwork: true,
      containers: [
        {
          name: "app",
          image: "nginx",
          ports: [{ containerPort: 8080 }],
          env: [{ name: "REGION", value: "NO" }],
        },
      ],
    },
  },
}

Everything in one shot:

{ } for objects (metadata, spec, each container)
[] for arrays (containers, ports, env)
strings always quoted ("api", "nginx", "REGION")
replicas: 2 and containerPort: 8080 stay bare, they’re real numbers
automountServiceAccountToken: false and hostNetwork: true stay bare, they’re real booleans
value: "NO" is quoted, because it’s a string that would otherwise get read as false

KYAML doesn’t remove types, it just stops guessing which one you meant.

Why it matters

Since structure comes from { } and [], not from spaces, indentation stops being load bearing. Squash a KYAML file onto one line and it means exactly the same thing. That kills the Helm whitespace problem outright.

Trying it

kubectl get deployment api -o kyaml

Convert existing files:

go install github.com/google/yamlfmt/cmd/yamlfmt@latest
yamlfmt --kyaml manifest.yaml

Also ships as a pre-commit hook and a Docker image for CI.

Nothing forces the switch. Plain YAML keeps working. Mixed repos, some converted, some not, are fine.

My plan if I touch this on a real repo: one directory first, verify round-trip parsing, check Helm and Kustomize don’t reformat it differently, then expand once it’s boring.

Not required. Cheap upgrade. Worth exploring.

For a broader breakdown of what else landed in this release, check out my notes on Kubernetes v1.37: An Operator’s Look at Garhwal.

Running llama.cpp on a 32 GB MacBook Air: A Direct Comparison with Ollama

Vishnu Hari Dadhich — Thu, 03 Sep 2026 02:30:00 +0000

In the previous post, I ran Ornith 1.5 9B on my 32 GB MacBook Air using Ollama and recorded baseline token-generation speeds on short prompts. Ollama is great for getting up and running quickly, but under the hood, its inference engine is built on llama.cpp.

In this post, we go one level down: building and running llama.cpp directly, offloading inference to Apple Silicon’s Metal GPU, and comparing performance numbers side by side with Ollama on the exact same model and quantization level.

Step 1: Build llama.cpp from Source

Building llama.cpp from source on macOS is fast and straightforward. On Apple Silicon, CMake automatically enables Metal support (GGML_METAL=ON) and compiles GPU compute kernels tailored for Apple’s unified memory architecture.

Clone the repository and compile the release binaries using all available CPU cores:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j$(sysctl -n hw.logicalcpu)

Once the build finishes, the binaries live in build/bin. Add them to your current session’s $PATH:

export PATH="$(pwd)/build/bin:$PATH"

To make this permanent across terminal sessions, append it to your shell configuration:

echo "export PATH=\"$(pwd)/build/bin:\$PATH\"" >> ~/.zshrc
source ~/.zshrc

Confirm that the build succeeded:

llama-server --version

Step 2: Serve the Model (Out-of-the-Box Baseline)

llama.cpp can download GGUF models directly from Hugging Face Hub using the -hf flag, caching weights under ~/.cache/huggingface/hub:

llama-server -hf ornith-ai/Ornith-1.5-9B-GGUF --port 8080 -ngl 99

Here is what the flags do:

-hf ornith-ai/Ornith-1.5-9B-GGUF: Resolves the model on Hugging Face and downloads the default Q4_K_M quant.
-ngl 99: Offloads all 99 model layers to the GPU (Metal on Apple Silicon unified memory).
--port 8080: Binds the OpenAI-compatible HTTP server to port 8080.

Starting llama-server with automatic Hugging Face download and full Metal GPU offload.

During startup, llama.cpp downloads both the multimodal vision projector (mmproj-Ornith-1.5-9B-BF16.gguf) and the main model weights (Ornith-1.5-9B-Q4_K_M.gguf), offloads the layers into Metal, initializes inference slots, and listens on port 8080.

We test this default setup with a short greeting prompt through the OpenAI-compatible API:

curl http://localhost:8080/v1/chat/completions \
  -d '{
    "model": "ornith-1.5-9b",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

First chat completion request via curl, returning detailed server-side timing breakdowns.

The response returns a complete timing breakdown in the timings object:

prompt eval time = 496.21 ms / 12 tokens (24.18 tokens per second)
eval time = 5527.17 ms / 96 tokens (17.19 tokens per second)

Compare that to Ollama’s numbers on the same model and quant from the earlier post:

Metric	Ollama (Default)	llama.cpp (Default, no `-fa`)
Prompt Eval Speed	37.13 tokens/s	24.18 tokens/s
Generation Speed	16.68 tokens/s	17.19 tokens/s

Generation speed is practically neck-and-neck (~17 tokens/s). However, prompt evaluation (prefill latency) was noticeably slower on vanilla llama.cpp out of the box (24.18 tokens/s vs. Ollama’s 37.13 tokens/s).

Ollama enables FlashAttention and sensible batch sizes by default, while raw llama.cpp keeps conservative baseline settings unless configured explicitly.

Step 3: Verifying the Quants

Before comparing benchmark numbers, it is critical to confirm that both engines are executing the exact same quantization format. Comparing different quants (e.g. Q4_K_M vs Q8_0) would invalidate any performance conclusions.

Check the quant llama.cpp loaded:

curl -s http://localhost:8080/v1/models | jq


"ftype": "Q4_K - Medium"

Check the quant Ollama is using:

curl -s http://localhost:11434/api/show -d '{"model": "ornith-1.5:9b"}' | jq '.details.quantization_level'


"Q4_K_M"

Both runtimes are confirmed to be executing Q4_K_M. Precision and model size are strictly identical.

Step 4: Leveling the Playing Field with Flash Attention & Batch Tuning

To give llama.cpp parity with Ollama’s runtime optimizations, we restart llama-server with FlashAttention enabled and explicit batch sizes configured:

llama-server -hf ornith-ai/Ornith-1.5-9B-GGUF --port 8080 -ngl 99 -fa on -b 2048 -ub 512

Here is what these tuning flags configure:

-fa on (or --flash-attn on): Enables FlashAttention kernels for Apple Silicon Metal. This significantly accelerates prompt evaluation (prefill) and reduces memory bandwidth overhead.
-b 2048 (--batch-size): Sets the logical batch size for prompt evaluation.
-ub 512 (--ubatch-size): Sets the physical micro-batch size dispatched to Metal compute passes, keeping the GPU pipelines fully saturated.

Step 5: Side-by-Side Benchmark on Longer Generation

Both engines were given the same prompt:

"give a 25 line story"

A longer generation produces several hundred tokens, giving a much more reliable measurement of steady-state generation speed than short one-liners.

Here are the side-by-side results on the 32 GB MacBook Air:

Metric	Ollama (`ornith-1.5:9b`)	llama.cpp (`-fa on -b 2048 -ub 512`)
Prompt Eval Count	17 tokens	18 tokens
Prompt Eval Speed	46.01 tokens/s	48.68 tokens/s
Generation Count	496 tokens	436 tokens
Generation Speed	16.42 tokens/s	17.07 tokens/s
Total Duration	30.58 s	25.85 s

With FlashAttention and batch tuning enabled, llama.cpp matches or slightly edges out Ollama across both prompt evaluation ( 48.68 vs. 46.01 tokens/s ) and token generation speed ( 17.07 vs. 16.42 tokens/s ).

Ollama Command & Stats

ollama run ornith-1.5:9b --verbose
>>> give a 25 line story
Thinking...
...done thinking.

[story output omitted]

total duration: 30.582007541s
load duration: 2.821208ms
prompt eval count: 17 token(s)
prompt eval duration: 369.481ms
prompt eval rate: 46.01 tokens/s
eval count: 496 token(s)
eval duration: 30.204706s
eval rate: 16.42 tokens/s

llama.cpp Server Startup & Timing Logs

Server startup and live slot execution log:

➜ ~ llama-server -hf ornith-ai/Ornith-1.5-9B-GGUF --port 8080 -ngl 99 -fa on -b 2048 -ub 512
0.00.729.018 I cmn common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.729.664 W srv llama_server: -----------------
0.00.729.667 W srv llama_server: CORS is set to allow all origins ('*') and no API key is set
0.00.729.668 W srv llama_server: this can be a security risk (cross-origin attacks)
0.00.729.668 W srv llama_server: more info: https://github.com/ggml-org/llama.cpp/pull/25655
0.00.729.669 W srv llama_server: -----------------
0.00.731.435 I srv load_model: loading model 'ornith-ai/Ornith-1.5-9B-GGUF'
0.01.177.317 W model has unused tensor blk.32.attn_norm.weight (size = 16384 bytes) -- ignoring
0.01.177.324 W model has unused tensor blk.32.post_attention_norm.weight (size = 16384 bytes) -- ignoring
0.01.177.329 W model has unused tensor blk.32.attn_q.weight (size = 18874368 bytes) -- ignoring
0.01.177.331 W model has unused tensor blk.32.attn_k.weight (size = 2359296 bytes) -- ignoring
0.01.177.333 W model has unused tensor blk.32.attn_v.weight (size = 3440640 bytes) -- ignoring
0.01.177.338 W model has unused tensor blk.32.attn_output.weight (size = 9437184 bytes) -- ignoring
0.01.177.340 W model has unused tensor blk.32.attn_q_norm.weight (size = 1024 bytes) -- ignoring
0.01.177.341 W model has unused tensor blk.32.attn_k_norm.weight (size = 1024 bytes) -- ignoring
0.01.177.343 W model has unused tensor blk.32.ffn_gate.weight (size = 28311552 bytes) -- ignoring
0.01.177.345 W model has unused tensor blk.32.ffn_down.weight (size = 41287680 bytes) -- ignoring
0.01.177.347 W model has unused tensor blk.32.ffn_up.weight (size = 28311552 bytes) -- ignoring
0.01.177.350 W model has unused tensor blk.32.nextn.eh_proj.weight (size = 18874368 bytes) -- ignoring
0.01.177.353 W model has unused tensor blk.32.nextn.enorm.weight (size = 16384 bytes) -- ignoring
0.01.177.354 W model has unused tensor blk.32.nextn.hnorm.weight (size = 16384 bytes) -- ignoring
0.01.177.359 W model has unused tensor blk.32.nextn.shared_head_norm.weight (size = 16384 bytes) -- ignoring
0.01.799.972 I cmn init: llama threadpool init, n_threads = 4
0.01.959.375 W load_hparams: Qwen-VL models require at minimum 1024 image tokens to function correctly on grounding tasks
0.01.959.377 W load_hparams: if you encounter problems with accuracy, try adding --image-min-tokens 1024
0.01.959.377 W load_hparams: more info: https://github.com/ggml-org/llama.cpp/issues/16842

0.02.172.437 I srv load_model: loaded multimodal model, '/Users/vishnuhd/.cache/huggingface/hub/models--ornith-ai--Ornith-1.5-9B-GGUF/snapshots/abdd624b12ebf020b767fff532ff44fe552b28c3/mmproj-Ornith-1.5-9B-BF16.gguf'
0.02.399.797 I srv load_model: initializing, n_slots = 4, n_ctx_slot = 262144, kv_unified = 'true'
0.02.402.615 I srv init: chat template supports preserving reasoning, consider enabling it via --reasoning-preserve
0.02.402.621 I srv llama_server: model loaded
0.02.402.623 I srv llama_server: listening on http://127.0.0.1:8080
0.02.402.623 W srv llama_server: NOTICE: server default port will be changed to :9931 in a future release
0.02.402.623 W srv llama_server: ref: https://github.com/ggml-org/llama.cpp/pull/26508

0.36.386.802 I slot get_availabl: id 3 | task -1 | selected slot by LRU, t_last = -1
0.36.386.826 I slot launch_slot_: id 3 | task 0 | processing task, is_child = 0
0.42.586.497 I slot print_timing: id 3 | task 0 | n_gen = 100, tg = 16.98 t/s, tg_3s = 17.15 t/s
0.45.633.605 I slot print_timing: id 3 | task 0 | n_gen = 152, tg = 17.01 t/s, tg_3s = 17.07 t/s
0.48.679.586 I slot print_timing: id 3 | task 0 | n_gen = 204, tg = 17.03 t/s, tg_3s = 17.07 t/s
0.51.718.658 I slot print_timing: id 3 | task 0 | n_gen = 256, tg = 17.04 t/s, tg_3s = 17.11 t/s
0.54.746.036 I slot print_timing: id 3 | task 0 | n_gen = 308, tg = 17.07 t/s, tg_3s = 17.18 t/s
0.57.778.672 I slot print_timing: id 3 | task 0 | n_gen = 360, tg = 17.08 t/s, tg_3s = 17.15 t/s
1.00.817.109 I slot print_timing: id 3 | task 0 | n_gen = 412, tg = 17.08 t/s, tg_3s = 17.11 t/s
1.02.233.148 I slot print_timing: id 3 | task 0 | prompt eval time = 369.74 ms / 18 tokens ( 20.54 ms per token, 48.68 tokens per second)
1.02.233.155 I slot print_timing: id 3 | task 0 | eval time = 25476.17 ms / 436 tokens ( 58.57 ms per token, 17.07 tokens per second)
1.02.233.157 I slot print_timing: id 3 | task 0 | total time = 25845.92 ms / 454 tokens
1.02.233.159 I slot print_timing: id 3 | task 0 | graphs reused = 434
1.02.233.187 I slot release: id 3 | task 0 | stop processing: n_tokens = 453, truncated = 0

And the raw curl request (with story output trimmed):

curl http://localhost:8080/v1/chat/completions \
  -d '{
    "model": "ornith-1.5-9b",
    "messages": [{"role": "user", "content": "Give me a 25 line story"}]
  }'


{
  "choices": [{
    "finish_reason": "stop",
    "index": 0,
    "message": {
      "role": "assistant",
      "content": "[story text omitted]",
      "reasoning_content": "[thinking omitted]"
    }
  }],
  "created": 1788423730,
  "model": "ornith-ai/Ornith-1.5-9B-GGUF",
  "system_fingerprint": "b10712-daef7b687",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 436,
    "prompt_tokens": 18,
    "total_tokens": 454,
    "prompt_tokens_details": {
      "cached_tokens": 0
    }
  },
  "id": "chatcmpl-dLoeYkd6rvuFieOvUSr9dBxB798twvZi",
  "timings": {
    "cache_n": 0,
    "prompt_n": 18,
    "prompt_ms": 369.742,
    "prompt_per_token_ms": 20.54,
    "prompt_per_second": 48.68,
    "predicted_n": 436,
    "predicted_ms": 25476.174,
    "predicted_per_token_ms": 58.57,
    "predicted_per_second": 17.07
  }
}

Key Takeaways

Ollama is not faster than llama.cpp : Under the hood, Ollama is llama.cpp. Its initial out-of-the-box advantage in prompt evaluation comes entirely from default runtime tuning (FlashAttention and batch sizes), not a secret runtime or different quantization.
FlashAttention is essential on Apple Silicon : Enabling -fa on doubled prompt evaluation speed from 24.18 tokens/s to 48.68 tokens/s , drastically cutting time-to-first-token.
Consistent throughput : At 9B parameters with Q4_K_M quantization, Apple Silicon unified memory sustains a rock-solid ~17.1 tokens/sec.
Why run llama.cpp directly? : Ollama provides exceptional developer ergonomics for local apps and testing. But direct llama.cpp gives you full control over context allocation, slot limits, KV cache quantization (-ctk, -ctv), and immediate access to upstream features and bugfixes.

In the next post, we will explore KV cache quantization and memory profiling to see how far we can stretch long-context windows on a 32 GB machine.

Running Ollama on a 32 GB MacBook Air: A Practical First Setup

Vishnu Hari Dadhich — Thu, 03 Sep 2026 01:30:00 +0000

I have a 32 GB MacBook Air. It is not a workstation GPU box, but its unified memory makes it a surprisingly capable machine for local models, provided I choose models that fit and keep expectations sensible.

This is the first post in a small, practical series about running models locally. I am starting with Ollama because it gets a model running quickly without building a runtime from source or hand-managing dependencies.

Why Ollama

Ollama manages model downloads, exposes a straightforward CLI, and starts a local HTTP API. On Apple silicon, it supports the Apple GPU; Ollama’s current macOS requirement is Sonoma (14) or newer. The app stores models and configuration under ~/.ollama. The macOS documentation is the useful reference for install, storage, and logs.

Under the hood, Ollama packages model management and an API around inference backends including llama.cpp. That distinction is useful: Ollama is the convenient front door; llama.cpp is a lower-level route I can use later when I want to compare runtimes directly.

Step 1: Install Ollama

Download the macOS app from ollama.com/download, mount the DMG, and drag Ollama to /Applications. Start it once. If the CLI is not already available, the app will offer to add it to your path.

Confirm that both the CLI and the local server are available:

ollama --version
ollama list
ollama ps

On a new installation, ollama list and ollama ps should be empty. The first reports downloaded models; the second reports models currently loaded into memory.

Step 2: Pull a Model

For this machine, I started with Ornith 1.5 9B:

ollama pull ornith-1.5:9b

This feels deliberately familiar if you work with containers: ollama pull downloads the model and its layers, while ollama run starts an interactive session. The current Ollama build of ornith-1.5:9b is 6.6 GB with a 256K context window, which is a comfortable starting point on a 32 GB laptop. Ollama’s model page lists the available tags; the 35B download is 23 GB, so I would not make that the default on an Air.

Pulling Ornith 1.5 9B, then confirming it is available locally.

After the pull completes, confirm it is available:

ollama list

Step 3: Run It Interactively

Start a chat session with the model:

ollama run ornith-1.5:9b

Use a question that resembles the work you actually do. I tested a simple greeting first, then moved on to infrastructure questions. Exit the interactive prompt with /exit or Ctrl-D.

A first interactive conversation with the locally running model.

For an initial sanity check, the model was responsive and produced a natural answer. That is useful confirmation that the model loads and runs locally, but it is not a benchmark. A real comparison needs the same prompt, context length, generation settings, and output length.

Step 4: Inspect Performance with `--verbose`

Ollama’s --verbose flag is a quick way to see timings after every response:

ollama run ornith-1.5:9b --verbose

On this MacBook Air, my short greeting test produced 114 output tokens at 16.68 tokens/sec , with a total duration of 7.22 seconds. The model’s thinking trace was visible before its answer.

I ran the same kind of test with Gemma 4 E4B:

ollama run gemma4:e4b --verbose

That run produced 228 output tokens at 27.62 tokens/sec , with a total duration of 8.49 seconds. It is faster in this small test, but it also generated a different and longer response. These figures are useful as a personal baseline, not as an apples-to-apples model ranking.

Ornith 1.5 9B vs. Gemma 4 E4B

Both models fit well on a 32 GB MacBook Air, but they are aimed at slightly different trade-offs.

Model	What I observed	Practical fit
`ornith-1.5:9b`	6.6 GB download; 16.68 tokens/sec in my short verbose run	A capable 9B-class, text-and-image model with plenty of memory headroom
`gemma4:e4b`	9.6 GB download; 27.62 tokens/sec in my different short verbose run	An efficient edge model for local chat, reasoning, coding, and multimodal work

The E in Gemma 4 E4B means effective parameters. Ollama describes E4B as a 4.5B-effective-parameter edge model (8B including embeddings), with a 128K context window and text, image, and audio support. It is designed to do useful local work without the memory cost of the larger Gemma 4 workstation models. The Gemma 4 library page has the current tags, sizes, and capabilities.

In practice, I would start with ornith-1.5:9b if I want the smaller download and a roomy 256K context window, or gemma4:e4b if I want the efficient Gemma 4 feature set. Neither of these two quick runs says which model is universally better; use the prompts you care about and record the result.

Step 5: Call the Local API

The terminal chat is only the first test. Ollama exposes an API locally at http://localhost:11434/api, so the model can be part of a script or an application.

curl http://localhost:11434/api/chat \
  -d '{
    "model": "ornith-1.5:9b",
    "messages": [{"role": "user", "content": "Hello!"}],
    "stream": false
  }'

Calling the same model through Ollama’s local chat API.

The response includes the answer and useful timing fields. Keep this endpoint local by default. If I later expose it to another device, I will put authentication and a proper reverse proxy in front of it. I will not publish port 11434 directly. Ollama’s API documentation covers the local base URL and client libraries.

Step 6: Try the App UI

The CLI is great for testing and scripts, but the Ollama app also gives me a simple chat interface. Here, Ornith is selected in the model picker and used for a weather question.

The Ollama app with Ornith 1.5 9B selected for a web-assisted prompt.

The app can also give a model access to web tools. In this example, Ornith searched for current Singapore weather before answering. That extends the model with fresh online information when the tool is enabled, but it does not replace the model’s built-in training knowledge. Treat retrieved results as sources to verify, especially for technical or time-sensitive answers.

The UI is useful when I want to compare prompts casually. The local API is the path I will use when I want to integrate models into tooling.

Commands Worth Remembering

# Download without entering an interactive chat
ollama pull ornith-1.5:9b

# Start an interactive chat
ollama run ornith-1.5:9b

# Show downloaded models and their disk usage
ollama list

# Show models currently loaded by the runner
ollama ps

# Remove a model I no longer need
ollama rm ornith-1.5:9b

Models are not small. Treat ollama pull the same way you would a sizeable docker pull: check disk space before collecting a pile of models “just in case.”

What This MacBook Air Is Good At

An 8B to 9B class model is a good fit for private note summarisation, explaining logs, drafting YAML, lightweight coding help, and experimenting with local integrations. This is not where I expect a 70B-class model to be effortless or where I would host production inference.

There are many ways to run models locally, but Ollama is an excellent Apple-silicon starting point. It gets the plumbing out of the way so I can spend time evaluating the models themselves.

Next, I will run llama.cpp directly on the same machine, then see whether a vLLM setup is worth comparing. Stay tuned.

Kubernetes v1.37: An Operator's Look at Garhwal

Vishnu Hari Dadhich — Wed, 02 Sep 2026 12:30:00 +0000

Kubernetes v1.37 landed recently under the release theme Garhwal , named after the Himalayan region of Uttarakhand, India. The release packages 67 enhancements across alpha, beta, and stable.

The upstream changelog is massive, but as someone who spends most of the day managing control planes, dealing with storage, and watching cloud bills, only a handful of changes stand out as immediately relevant to day-to-day work.

Here are the features I am keeping an eye on, along with practical examples for how they work.

1. HPA Can Finally Scale to Zero (`minReplicas: 0`)

For years, one of the most frustrating limitations in vanilla Kubernetes autoscaling was that HorizontalPodAutoscaler could not scale below one replica. If you had a queue consumer waiting on messages in SQS or RabbitMQ, or a GPU worker waiting on batch jobs, you had to keep at least one pod running around the clock, or pull in third-party tools like KEDA just to do basic scale-to-zero.

In v1.37, HPA scale to zero graduated to Beta and is enabled by default. You can now configure minReplicas: 0:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: queue-worker-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: queue-worker
  minReplicas: 0
  maxReplicas: 10
  metrics:
    - type: External
      external:
        metric:
          name: aws_sqs_approximate_number_of_messages_visible
        target:
          type: Value
          averageValue: 30

When the queue is empty, the controller scales the deployment down to 0 pods:

kubectl get hpa queue-worker-hpa
# NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
# queue-worker-hpa Deployment/queue-worker 0/30 (avg) 0 10 0 18m

Under the hood, the HPA controller adds a ScaledToZero condition so you can easily distinguish an automated scale-down from a manual shutdown:

status:
  currentReplicas: 0
  desiredReplicas: 0
  conditions:
    - type: ScaledToZero
      status: "True"
      reason: ScaledToZero
      message: "all pods scaled down based on metrics"

The catch: This only works with object or external metrics. It does not work with CPU or memory metrics, because a deployment with zero running pods produces no CPU or memory data to trigger an upscale. For event-driven workers and GPU workloads, this saves real money on cloud bills.

2. Finding Orphaned PVCs Just Got Much Easier

PersistentVolumeClaims have a habit of outliving the workloads that created them. Someone deletes a Deployment or StatefulSet, but the PVC stays behind in the namespace. Weeks or months later, dozens of unused EBS volumes or cloud disks are quietly burning infrastructure budget.

Until now, Kubernetes gave you no native way to know when a PVC was last used without writing custom scripts to inspect every pod spec in the cluster.

Kubernetes v1.37 addresses this by graduating PVC last-used tracking to Beta (enabled by default). The PVC protection controller now adds an Unused condition to status.conditions on PersistentVolumeClaims:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: abandoned-redis-data
  namespace: analytics
status:
  phase: Bound
  conditions:
    - type: Unused
      status: "True"
      reason: NoPodsUsingPVC
      lastTransitionTime: "2026-08-15T09:30:00Z"

When the last pod referencing a PVC terminates, Status flips to "True" with reason NoPodsUsingPVC.
The lastTransitionTime tells you the exact timestamp since the volume became idle.
As soon as a new pod attaches, the condition flips back to False.

Kubernetes does not delete anything automatically (which is good; you do not want the control plane guessing whether data is safe to purge). But this gives you a clean field to query across your cluster:

# List all PVCs across all namespaces that are currently unused, with their idle timestamp
kubectl get pvc -A -o custom-columns=\
'NAMESPACE:.metadata.namespace,\
NAME:.metadata.name,\
CAPACITY:.status.capacity.storage,\
UNUSED:.status.conditions[?(@.type=="Unused")].status,\
IDLE_SINCE:.status.conditions[?(@.type=="Unused")].lastTransitionTime'

With one command, you can spot storage that has been abandoned for 30+ days and clean it up safely.

3. ClusterTrustBundles Go GA: Retire Your CA Sync Cronjobs

If you manage enterprise clusters with internal PKI, private container registries, or corporate proxies, you have almost certainly solved the root certificate problem with a workaround. Most teams either run a custom controller that clones a secret into every namespace as a ConfigMap, or bake internal certificates into container base images.

In v1.37, ClusterTrustBundles and Pod Certificates officially graduated to Stable (GA) under certificates.k8s.io.

A ClusterTrustBundle is a cluster-scoped object that holds trusted X.509 certificate anchors:

apiVersion: certificates.k8s.io/v1alpha1
kind: ClusterTrustBundle
metadata:
  name: corporate-root-ca
spec:
  signerName: "corp.internal/pki"
  trustBundle: |
    -----BEGIN CERTIFICATE-----
    MIICjTCCAjSgAwIBAgIUTkX...
    -----END CERTIFICATE-----

Workloads in any namespace can project this bundle directly into their filesystem via standard projected volumes:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: internal-api
spec:
  template:
    spec:
      containers:
        - name: app
          image: my-registry.internal/api:v1
          env:
            # Point common TLS libraries to the projected CA bundle
            - name: SSL_CERT_FILE
              value: /etc/ssl/certs/corporate-root-ca.crt
          volumeMounts:
            - name: trusted-certs
              mountPath: /etc/ssl/certs/corporate-root-ca.crt
              subPath: corporate-root-ca.crt
              readOnly: true
      volumes:
        - name: trusted-certs
          projected:
            sources:
              - clusterTrustBundle:
                  name: corporate-root-ca

The kubelet handles mounting and rotation natively. That is one less custom controller or sync cronjob to maintain.

4. Control Plane Breathing Room: etcd RangeStream & Concurrent Decode

When large clusters experience network hiccups or controller restarts, hundreds of controllers can hit the API server with list requests at the exact same moment. Historically, etcd built full responses in memory before sending them, and the API server processed watch events serially on a single goroutine. During recovery storms, control plane memory would spike, occasionally sending nodes into Out-Of-Memory (OOM) loops.

In v1.37, two features land on by default to fix this:

etcd RangeStream (Beta): Instead of buffering massive key-value sets in memory, etcd streams responses back in chunks. This turns what used to be a memory-heavy dump into a smooth stream.
Concurrent Watch Object Decode (Beta): The API server now decodes incoming watch events across a pool of worker goroutines rather than processing them one by one.

If you interact with etcd 3.7+ directly, you can test the new chunked streaming RPC via etcdctl:

# Stream keys in chunks rather than buffering everything into a single gRPC response
etcdctl get /registry/pods/ --prefix --stream

In upstream benchmarks across 150,000 pods, these two improvements combined cut cache initialization time by more than 50%. For operators of large clusters or CRD-heavy environments, this means significantly less control plane jitter during rolling restarts.

5. `kubectl get -o kyaml` Goes Stable

YAML has plenty of parsing quirks (such as unquoted country codes like NO being interpreted as boolean false).

In v1.37, KYAML reaches Stable. KYAML is a safer, unambiguous subset of YAML tailored specifically for Kubernetes. You do not need to change any of your existing manifests or pipelines; KYAML is strictly backwards compatible with standard YAML.

You can now run:

# Output clean, normalized YAML without parser ambiguity
kubectl get deployment queue-worker -o kyaml

Comparing the output against regular YAML:

# Standard YAML output often strips quotes from country codes or booleans:
country: NO # Parsed by some YAML 1.1 loaders as boolean false!

# KYAML output strictly normalizes and quotes scalar types:
country: "NO"

It formats cleanly, avoids parser edge cases, and provides predictable output for gitops diffs and scripts.

6. SELinuxMount Reaches GA

If your nodes run SELinux (common in Red Hat, Rocky, or Fedora environments), volume mounts historically required recursive file relabeling on every pod start, which could cause painfully slow startup times on large volumes.

In v1.37, SELinuxMount is now Stable and enabled by default for CSI drivers that support it. Volumes are mounted with a single mount context rather than recursively walking the directory tree.

The catch: A volume mount can only carry one SELinux context. If you run multiple pods with different SELinux labels that share the same volume on the same node, those pods may now fail to mount.

If your workloads rely on shared volumes with mixed labels, you can preserve the legacy recursive relabeling behavior in your Pod spec:

apiVersion: v1
kind: Pod
metadata:
  name: shared-data-consumer
spec:
  securityContext:
    seLinuxChangePolicy: Recursive
  containers:
    - name: app
      image: alpine
      volumeMounts:
        - name: shared-storage
          mountPath: /data
  volumes:
    - name: shared-storage
      persistentVolumeClaim:
        claimName: shared-data-pvc

Where This Leaves Us

Do not rush to upgrade production this weekend. If you use a managed provider (EKS, GKE, Rancher), wait for .1 or .2 patch releases. If you run physical data centers like me and manage the control plane yourself, be even more conservative: test locally, promote through lower environments, and comfortably stay on n-1 or n-2.

The real reason to track v1.37 today is preventing fresh technical debt. If your team was about to build custom controllers to sync CA certs or write scripts to hunt dead storage, tell them to hold off. Upstream is solving it natively.

Let the early patches bake, keep your clusters healthy, and start planning which legacy workarounds you get to delete.

Multi-region YugabyteDB deployment on AWS EKS with Istio

Vishnu Hari Dadhich — Thu, 02 May 2024 06:34:52 +0000

In today’s distributed cloud landscape, deploying applications across multiple regions and clusters is crucial for scalability, reliability, and performance. This blog post will guide you through setting up a multi-region, multi-cluster YugabyteDB deployment on AWS EKS with Istio service mesh.

WHY YUGABYTEDB?

YugabyteDB is a transactional database that brings together four must-have needs of cloud native apps – namely SQL as a flexible query language, low-latency performance, continuous availability, and globally-distributed scalability. Other databases do not serve all 4 of these needs simultaneously.

Monolithic SQL databases offer SQL and low-latency reads, but neither have the ability to tolerate failures, nor can they scale writes across multiple nodes, zones, regions, and clouds.
Distributed NoSQL databases offer read performance, high availability, and write scalability, but give up on SQL features such as relational data modelling and ACID transactions.

WHY AWS EKS AND ISTIO?

AWS EKS provides a managed Kubernetes service, simplifying cluster management and deployment. Istio, an open-source service mesh, enables traffic management, security, and observability across microservices.

Combining Yugabyte with AWS EKS and Istio creates a robust, scalable, and secure cloud-native architecture that spans across multiple regions.

DEPLOYMENT OVERVIEW

Our deployment consists of:

Three AWS regions (Singapore, Mumbai and Hyderabad).
One EKS cluster in each region.
One YugabyteDB master and one YugabyteDB tserver are deployed in each EKS cluster.
Istio is deployed in each cluster with an east-west gateway to provide a multi-cluster service mesh.

DEPLOYMENT STEPS

Pre-requisites

AWS account with at least three regions enabled
AWS user with access to create VPC and EKS using eksctl eksctl, aws cli, kubectl and git installed on your local system

Clone the following repo to follow along with this blog:

git clone https://github.com/vishnuhd/yugabyte-multiregion-aws-eks-istio.git
cd yugabyte-multiregion-aws-eks-istio

Deploy AWS EKS clusters

Deploy EKS clusters in three different regions (namely Singapore, Mumbai and Hyderabad) using eksctl:

{
  for region in mumbai singapore hyderabad; do
    echo -e "Creating EKS cluster in ${region}...\n"
    eksctl create cluster -f ${region}/cluster-config.yaml
    echo -e "-------------\n"
  done
}

Rename the kube contexts for the simplicity of this demo:

kubectl config rename-context 'yb-mumbai.ap-south-1.eksctl.io' mumbai
kubectl config rename-context 'yb-singapore.ap-southeast-1.eksctl.io' singapore
kubectl config rename-context 'yb-hyderabad.ap-south-2.eksctl.io' hyderabad

NOTE: By default, EKS does not provide EBS permissions, follow this article to enable EKS PVC dynamic provisioning.

Setup Istio

When configuring a production deployment of Istio, key considerations include whether the mesh will be in single or multiple clusters, Istio control plane setup for high availability, and the choice between a single multicluster service mesh or federated multi-mesh deployment. These factors represent independent dimensions of configuration for Istio deployment.

This guide describes the various options and considerations when configuring your Istio deployment. For this demo, we are gonna Install Multi-Primary on different networks.

Download Istio:

curl -L https://istio.io/downloadIstio | sh -
cd istio-1.21.0
export PATH=$PWD/bin:$PATH

Plug in CA Certificates for Istio

In a multi-cluster environment, we would want to set up one root CA and use the root CA to issue intermediate certificates to the Istio CAs that run in each cluster. This would ensure the services behind it can only be accessed by services with a trusted mTLS certificate.

Create a cert directory :

mkdir -p istio-1.21.0/certs

Generate the root CA certificate and key:

cd istio-1.21.0/certs
make -f ../tools/certs/Makefile.selfsigned.mk root-ca

For each cluster, generate an intermediate certificate and key for the Istio CA:

cd istio-1.21.0/certs

{
  for region in mumbai singapore hyderabad; do
    echo -e "Generating certs for cluster - ${region}...\n"
    make -f ../tools/certs/Makefile.selfsigned.mk yb-${region}-cacerts
    echo -e "-------------\n"
  done
}

In each cluster, create a secret called cacerts including all the input files ca-cert.pem, ca-key.pem, root-cert.pem and cert-chain.pem :

{
  for region in mumbai singapore hyderabad; do
    echo -e "Creating namespace and secret for cluster - ${region}...\n"

    kubectl --context ${region} create namespace istio-system
    kubectl --context ${region} create secret generic cacerts -n istio-system \
          --from-file=istio-1.21.0/certs/yb-${region}/ca-cert.pem \
          --from-file=istio-1.21.0/certs/yb-${region}/ca-key.pem \
          --from-file=istio-1.21.0/certs/yb-${region}/root-cert.pem \
          --from-file=istio-1.21.0/certs/yb-${region}/cert-chain.pem

    echo -e "-------------\n"
  done
}

With this step completed, we are now prepared to install Istio on every cluster.

Install Istio

Install Istio using istioctl:

{
  for region in mumbai singapore hyderabad; do
    echo -e "Installing istio for cluster - ${region}...\n"

    istioctl install --context ${region} -f ./${region}/istio.yaml

    echo -e "-------------\n"
  done
}

Install a gateway in each cluster, that is dedicated to east-west traffic. By default, this gateway will be public on the Internet. Production systems may require additional access restrictions (e.g. via firewall rules) to prevent external attacks.

{
  for region in mumbai singapore hyderabad; do
    echo -e "Installing the east-west gateway for cluster - ${region}...\n"

    ./istio-1.21.0/samples/multicluster/gen-eastwest-gateway.sh \
        --mesh mesh1 --cluster yb-${region} --network network-${region} | \
        istioctl --context ${region} install -y -f -

    echo -e "-------------\n"
  done
}

Since the clusters are on separate networks, we need to expose all services (*.local) on the east-west gateway in all three clusters. While this gateway is public on the Internet, services behind it can only be accessed by services with a trusted mTLS certificate and workload ID, just as if they were on the same network.

{
  for region in mumbai singapore hyderabad; do
    echo -e "Exposing the services for cluster - ${region}...\n"

    kubectl --context ${region} apply -n istio-system -f \
        ./istio-1.21.0/samples/multicluster/expose-services.yaml

    echo -e "-------------\n"
  done
}

Install a remote secret in each cluster that provides access to the other cluster’s Kube API server.

{
  for region1 in mumbai singapore hyderabad; do
    for region2 in mumbai singapore hyderabad; do
      if [[ "${region1}" == "${region2}" ]]; then continue; fi
      echo -e "Create remote secret of ${region1} in ${region2}...\n"

      istioctl create-remote-secret \
        --context ${region1} \
        --name=yb-${region1} | \
        kubectl apply -f - --context ${region2}

      echo -e "-------------\n"
    done
  done
}

Install YugabyteDB

To install YugabyteDB using helm charts, add the chart repo:

helm repo add yugabytedb https://charts.yugabyte.com
helm repo update

Create a YugabyteDB namespace (yb-demo) in each cluster and enable Istio injection:

{
  for region in mumbai singapore hyderabad; do
    echo -e "Creating namespace for cluster - ${region}...\n"

    kubectl --context ${region} create namespace yb-demo
    kubectl label --context ${region} namespace yb-demo istio-injection=enabled

    echo -e "-------------\n"
  done
}

Install YugabyteDB:

{
  for region in mumbai singapore hyderabad; do
    echo -e "Installing YugabyteDB in cluster - ${region}...\n"

    helm upgrade --install ${region} yugabytedb/yugabyte \
        --version 2.19.3 \
        --namespace yb-demo \
        -f ${region}/overrides.yaml \
        --kube-context ${region} --wait

    echo -e "-------------\n"
  done
}

This will install YugabyteDB in each EKS cluster with 1 master and 1 tserver connected with each other through the Istio service mesh. At this point, it is important to understand each parameter being set in the overrides.yaml file. Each master and tserver pod needs to know all the master addresses to replicate data and elect leaders.

In addition to the Istio setup, an extra step is required, which involves creating identical Kubernetes services in all clusters to enable DNS service discovery. More information can be found here. Therefore, we need to replicate the yugabyte-master and yugabyte-tserver services present in the Mumbai region to both the Singapore and Hyderabad regions, and vice versa.

{
  for region1 in mumbai singapore hyderabad; do
    for region2 in mumbai singapore hyderabad; do
      if [[ "${region1}" == "${region2}" ]]; then continue; fi
      echo -e "Creating services of ${region2} in ${region1}...\n"

      kubectl --context ${region1} apply -f ${region1}/services-${region2}.yaml -n yb-demo

      echo -e "-------------\n"
    done
  done
}

Check the YugabyteDB pods and services, all of them should be up and running:

{
  for region in mumbai singapore hyderabad; do
    echo -e "Checking YugabyteDB pods and svcs for cluster - ${region}...\n"

    kubectl --context ${region} get pods,svc -A

    echo -e "-------------\n"
  done
}

Finally, we need to configure global data distribution, for Yugabyte to handle the data distribution properly across regions:

kubectl --context mumbai exec -n yb-demo mumbai-yugabyte-yb-master-0 -- bash \
-c "/home/yugabyte/master/bin/yb-admin --master_addresses mumbai-yugabyte-yb-master-0.yb-demo.svc.cluster.local,hyderabad-yugabyte-yb-master-0.yb-demo.svc.cluster.local,singapore-yugabyte-yb-master-0.yb-demo.svc.cluster.local modify_placement_info aws.ap-south-1.ap-south-1a,aws.ap-south-2.ap-south-2a,aws.ap-southeast-1.ap-southeast-1a 3"

Voila, your YugabyteDB multi-regional setup is now complete!

Access the YugabyteDB UI

Find the yb-master-ui service in yb-demo namespace for any cluster and open it in the browser along with the port 7000:

As we can see the masters are spread across regions, with Hyderabad one as the leader.

Explore the tablet servers:

Similarly, we can see the tablet servers being distributed across multiple regions, each of them able to hand synchronous reads and writes.

Run a sample Yugabyte application in any of the clusters:

kubectl run yb-sample-apps \
    -it --rm \
    --image yugabytedb/yb-sample-apps \
    --namespace yb-demo \
    --context singapore \
    --command -- sh

java -jar yb-sample-apps.jar java-client-sql \
    --workload SqlInserts \
    --nodes yb-tserver-common.yb-demo.svc.cluster.local:5433 \
    --num_threads_write 1 \
    --num_threads_read 2

Here, we are targeting yb-tserver-common service for reads and writes, which will choose any of the tserver in any of the regions randomly. This also helps in load-balancing the traffic across regions.

We can also see the tables created by this sample app:

DISASTER RECOVERY

Disasters can happen anytime, YugabyteDB provides us with a feature called Replication Factor (RF). Configurations usually include a Replication Factor (RF) of 3. In this setup, a write to the leader requires an acknowledgement from one follower before being committed, as the leader and one follower together constitute the majority. In the event of a failure, operational replicas in the Raft group can support consistent reads and writes, while those that have become separated from the Raft consensus quorum cannot make progress. We will now see this feature of Yugabyte in action.

Current leaders for master and tservers:

Currently, the master pod in the Hyderabad region is the LEADER, while most of the transactions are handled by the tablet server in the Mumbai region.

Let’s make the Hyderabad region go down:

kubectl scale sts hyderabad-yugabyte-yb-master-0 --replicas 0 -n yb-demo --context hyderabad
kubectl scale sts hyderabad-yugabyte-yb-tserver-0 --replicas 0 -n yb-demo --context hyderabad

As soon as we decrease the pod replicas to 0 for both master and tserver, we can see the errors in the Yugabyte master UI:

Here we can see that the master server from the Mumbai region is elected as a new Leader and all transactions are still operating even if one whole region is down.

Yugabyte also keeps track of the under-replicated tables, so that they can be replicated to the region as soon as it comes back online:

When the region comes online again, the data is replicated back to the Hyderabad region:

Awesome, now you have a multi-regional fault-tolerant YugabyteDB setup on AWS EKS clusters using Istio as a service mesh.

BONUS: SETUP KIALI FOR OBSERVABILITY

The Istio download package comes by default with Kiali and Prometheus, let’s set it up to have a nice view of all the services in the mesh.

Install Kiali and Prometheus:

{
  for region in mumbai singapore hyderabad; do
    echo -e "Checking Kiali and Prometheus for cluster - ${region}...\n"

    kubectl apply -f istio-1.21.0/samples/addons/prometheus.yaml --context ${region}
    kubectl apply -f istio-1.21.0/samples/addons/kiali.yaml --context ${region}

    echo -e "-------------\n"
  done
}

Open Kiali dashboard:

istioctl dashboard kiali --context singapore

The above Kiali graph shows the various clusters and services from the POV of EKS cluster in the Singapore region.

CLEANING UP THE RESOURCES

Uninstall YugabyteDB:

{
  for region in mumbai singapore hyderabad; do
    echo -e "Un-installing YugabyteDB in cluster - ${region}...\n"

    helm uninstall ${region} --namespace yb-demo --kube-context ${region}
    kubectl delete pvc --namespace yb-demo \
      --selector component=yugabytedb,release=${region} \
      --context ${region}

    echo -e "-------------\n"
  done
}

Delete additional YB services:

{
  for region1 in mumbai singapore hyderabad; do
    for region2 in mumbai singapore hyderabad; do
      if [[ "${region1}" == "${region2}" ]]; then continue; fi
      echo -e "Deleting services of ${region2} in ${region1}...\n"

      kubectl --context ${region1} delete -f ${region1}/services-${region2}.yaml -n yb-demo

      echo -e "-------------\n"
    done
  done
}

Un-install Kiali and Prometheus:

{
  for region in mumbai singapore hyderabad; do
    echo -e "Checking Kiali and Prometheus for cluster - ${region}...\n"

    kubectl delete -f istio-1.21.0/samples/addons/prometheus.yaml --context ${region}
    kubectl delete -f istio-1.21.0/samples/addons/kiali.yaml --context ${region}

    echo -e "-------------\n"
  done
}

Uninstall Istio:

{
  for region in mumbai singapore hyderabad; do
    echo -e "Un-installing Istio in cluster - ${region}...\n"

    istioctl uninstall --purge -y --context ${region}

    echo -e "-------------\n"
  done
}

Delete EKS clusters:

eksctl delete cluster yb-mumbai --region ap-south-1
eksctl delete cluster yb-singapore --region ap-southeast-1
eksctl delete cluster yb-hyderabad --region ap-south-2

CONCLUSION

A multi-region, multi-cluster YugabyteDB deployment on AWS EKS with Istio provides a highly available, scalable, and secure architecture for distributed applications. By leveraging YugabyteDB deployed on multiple AWS regions and EKS clusters, this setup ensures redundancy and failover capabilities, minimizing downtime and ensuring business continuity. Istio’s service mesh capabilities provide advanced traffic management, security, and observability features, allowing for fine-grained control and monitoring of the application traffic. This setup is ideal for organizations requiring a robust and resilient infrastructure for their critical applications.

The original tech blog is here, please follow/subscribe to get notifications directly in your inbox when new content goes live. You can also find me on LinkedIn @ in/vishnuhd.

DEV Community: Vishnu Hari Dadhich

What the heck is an AI Agent?

Not a chatbot with more steps

The model doesn’t loop, the harness does

The four things a harness provides

Why the loop gets expensive

Single-shot vs loop, at the token level

Some harnesses and agents you’ve probably heard of

Up next

What Happens When You Ask an LLM a Question

1. An answer grows one token at a time

2. What does a 2B model actually contain?

3. Transformers connect the relevant pieces

4. Why can it reason, and still be wrong?

5. Context is the input. KV cache saves work.

6. Why does a model need so much memory?

7. What does vLLM add?

8. More GPUs, and fewer active experts

Split maths, split layers, or serve separate requests

How GPU communication connects the pieces

MoE routes work to selected experts

The whole conversation flow

The five things I want to remember

Run vLLM on Kubernetes with Minikube, WSL2 and NVIDIA GPU

What You’ll Build

The Test Rig

Troubleshooting vLLM GPU Support on Docker Desktop WSL2 Kubernetes

How Kubernetes Detects NVIDIA GPUs Using the NVIDIA Device Plugin

Worker Node Prerequisites

Why Docker Desktop Kubernetes Cannot Detect My NVIDIA GPU

Installing Docker CE and Containerd Inside WSL2

Run Minikube with NVIDIA GPU Support on WSL2

Verifying the Cluster and GPU Allocation

Deploy vLLM on Kubernetes with NVIDIA GPU

The Shared Memory (/dev/shm) Gotcha

Testing Inference and Hardware Telemetry

Beyond Local Dev: Why Production Clusters Use the NVIDIA GPU Operator

Key Takeaways

Running a 35B MoE Model on an 8 GB Laptop GPU: Testing FreeToken

The Test Rig

Installing FreeToken and Exploring the Library

Weight Conversion: The FTW Format

Loading the Model and Memory Allocation

The Cache Architecture Under the Hood

Real-World Performance & Generation Speed

Built-in Agentic Tooling & Local Endpoints

Where FreeToken Fits

Strengths

Limitations

Wrap-up

Kubernetes is Adding a K to YAML: Why KYAML Makes Sense

The problem

What KYAML actually changes

Why it matters

Trying it

Running llama.cpp on a 32 GB MacBook Air: A Direct Comparison with Ollama

Step 1: Build llama.cpp from Source

Step 2: Serve the Model (Out-of-the-Box Baseline)

Step 3: Verifying the Quants

Step 4: Leveling the Playing Field with Flash Attention & Batch Tuning

Step 5: Side-by-Side Benchmark on Longer Generation

Ollama Command & Stats

llama.cpp Server Startup & Timing Logs

Key Takeaways

Running Ollama on a 32 GB MacBook Air: A Practical First Setup

Why Ollama

Step 1: Install Ollama

Step 2: Pull a Model

Step 3: Run It Interactively

Step 4: Inspect Performance with --verbose

Ornith 1.5 9B vs. Gemma 4 E4B

Step 5: Call the Local API

Step 6: Try the App UI

Commands Worth Remembering

What This MacBook Air Is Good At

Kubernetes v1.37: An Operator's Look at Garhwal

1. HPA Can Finally Scale to Zero (minReplicas: 0)

2. Finding Orphaned PVCs Just Got Much Easier

3. ClusterTrustBundles Go GA: Retire Your CA Sync Cronjobs

4. Control Plane Breathing Room: etcd RangeStream & Concurrent Decode

The Shared Memory (`/dev/shm`) Gotcha

Step 4: Inspect Performance with `--verbose`

1. HPA Can Finally Scale to Zero (`minReplicas: 0`)

5. `kubectl get -o kyaml` Goes Stable