DEV Community: Suresh Kumar Pallapothu

Day 4: Optimizing Model Weights — Quantization Basics (GGUF, AWQ) for Efficient Inference

Suresh Kumar Pallapothu — Tue, 10 Nov 2026 20:35:09 +0000

Production AI runs into the memory wall. LLMs are bound by memory bandwidth, because every generated token moves billions of parameters from memory to the compute cores. A standard 8-billion-parameter model in FP16 needs roughly 16 GB of VRAM for the weights alone, before the KV-cache and context. Quantization compresses the weights so these models fit on commodity enterprise hardware without a collapse in quality.

Bytes per parameter

FP16 stores each weight in 16 bits: 1 sign bit, 5 exponent bits and 10 mantissa bits, which is 2 bytes. INT4 uses 4 bits, which is 0.5 bytes, a 75% saving.

Diagram: FIG 1 · Weights only, before KV-cache and runtime overhead. INT4 cuts weight memory by 75% compared with FP16. See the animated version.

The mathematics

Quantization maps a float weight to a small integer with a scale and an offset.

q = round(w / S) + Z
w ≈ S × (q − Z)        (de-quantize)

w is the original weight, S a scale factor, Z a zero-point offset and q the stored integer. At run time the weights are de-quantized on the fly, or used directly by low-bit kernels, which cuts memory by up to 75%.

Diagram: FIG 2 · Quantization maps many float values onto a small set of integer levels. The rounding error is the price paid for memory savings. See the animated version.

Here is a tiny runnable demonstration of the rounding error you trade for that saving. It uses simple 4-bit quantization of a random weight vector, so it shows the principle rather than what production formats do (they quantize in small groups and keep extra scales to reduce the error).

import numpy as np

rng = np.random.default_rng(0)
w = rng.normal(0, 0.05, size=4096).astype(np.float32)      # pretend these are weights

bits = 4
levels = 2**bits - 1                                       # 15 steps for 4-bit
S = (w.max() - w.min()) / levels                           # scale factor
Z = round(-w.min() / S)                                    # zero point

q = np.clip(np.round(w / S) + Z, 0, levels).astype(np.uint8)   # quantize: stored as small integers
w_hat = S * (q.astype(np.float32) - Z)                         # de-quantize

print("max level:", q.max(), " memory: 16 bits ->", bits, "bits per weight")
print("mean absolute error:", float(np.abs(w - w_hat).mean()))

Choosing a format

Diagram: FIG 3 · GGUF for portable CPU and mixed deployments, AWQ for high-throughput GPU serving, GPTQ for calibrated 4-bit GPU models. See the animated version.

GGUF: the format used by llama.cpp and Ollama. It targets CPUs and Apple Silicon, supports offloading layers between CPU and GPU, and ships as one portable file.
AWQ (Activation-aware Weight Quantization): suits high-throughput GPU serving through engines such as vLLM and TensorRT-LLM. It protects the small set of weights that matter most, and holds up well at 4-bit.
GPTQ: a post-training method for GPUs that calibrates 4-bit weights against a sample dataset.

The format has to match the runtime you plan to use, so decide the engine first.

The quality trade-off

Diagram: FIG 4 · Lower bit widths shrink the file but cost quality. Sizes are approximate for an 8B model. Quality bars are illustrative, since real loss depends on the model and the task. See the animated version.

Q8_0 is very close to FP16 and halves the memory.
Q4_K_M is a common default for local deployment. An 8B model drops to about 4.8 GB and runs on a 16 GB workstation, usually with a small quality loss.
Q2_K compresses hardest and can show clear degradation, such as weaker reasoning and malformed JSON output.

How much quality you lose depends on the model and the task, so run your own prompts and a few edge cases before you commit. For example, with Ollama you can pull a specific quantization by tag: ollama pull llama3.1:8b-instruct-q4_K_M.

Run it yourself

This is the exact example from the video. A real script with a real result.

import numpy as np

rng = np.random.default_rng(0)
w = rng.normal(0, 0.05, 4096).astype(np.float32)

bits = 4
levels = 2**bits - 1
S = (w.max() - w.min()) / levels
Z = round(-w.min() / S)

q = np.clip(np.round(w / S) + Z, 0, levels).astype(np.uint8)
w_hat = S * (q.astype(np.float32) - Z)

print("levels used:", int(q.max()) + 1)
print("mean abs error:", round(float(np.abs(w - w_hat).mean()), 5))
print("size in bytes:", w.size * 2, "->", w.size * bits // 8)

Run it:

python quantize.py

Output I got while recording the video:

levels used: 16
mean abs error: 0.00604
size in bytes: 8192 -> 2048

Coming Up Next

Day 5: Local AI runtimes: deploying Ollama and vLLM for bare-metal execution.

Quantization #MachineLearning #Optimization #LLMs #GGUF #vLLM #DataScience #AIEngineering

Originally published at https://sureshpallapothu.in/blog/ai-int-day-4-quantization, where this post includes animated diagrams.

Day 3: Hardware Acceleration in Docker — GPU Passthrough and Limits for Constrained Hardware

Suresh Kumar Pallapothu — Mon, 09 Nov 2026 23:08:11 +0000

A leading reason AI workloads fail in containers is the boundary between the container's isolated namespace and the host's hardware. Running inference on raw CPUs creates big latency bottlenecks, and exposing a GPU or accelerator takes deliberate host setup.

The acceleration architecture

A container cannot talk to a PCIe device without driver support. The NVIDIA Container Toolkit supplies it by wrapping the standard runtime with a hardware-aware one (nvidia-container-runtime).

Diagram: FIG 1 · The NVIDIA Container Toolkit injects the host's driver libraries and device nodes into the container at launch. Your image does not bundle the kernel driver. See the animated version.

Three ideas make it work.

Diagram: FIG 2 · Decoupling the driver from the image is what makes accelerated containers portable. Compatibility between the host driver and your image's CUDA version still matters. See the animated version.

Driver decoupling. The host runs the kernel driver (nvidia.ko). The image never contains it.
Runtime injection. At launch the toolkit injects the matching user-space driver library (libcuda.so) and the device nodes (/dev/nvidia*) into the container.
Application agility. The image needs only the high-level CUDA runtime libraries, so it does not depend on the host's driver build. The host driver must still be recent enough for the CUDA version inside the image.

Launching an accelerated container

With the toolkit installed, register it with Docker and restart the daemon.

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

That edits /etc/docker/daemon.json. You can also set it by hand, and making it the default runtime is optional, since --gpus selects it per container.

{
  "default-runtime": "nvidia",
  "runtimes": {
    "nvidia": { "path": "nvidia-container-runtime", "runtimeArgs": [] }
  }
}

docker run -d --name local-inference \
  --gpus all \
  --shm-size="2g" \
  -p 8000:8000 \
  enterprise-model-server:latest

Verify it: docker run --rm --gpus all ubuntu nvidia-smi should print the same GPU table as the host.

Why `--shm-size` matters

PyTorch passes tensors between worker processes through shared memory (/dev/shm). Docker's default is only 64 MB, so parallel data loading can crash suddenly with bus errors. Raising it, for example to 2g, avoids that.

Diagram: FIG 4 · PyTorch uses shared memory to pass tensors between worker processes. Docker's 64 MB default is too small for that, so raise it with --shm-size. See the animated version.

Hard limits for a 16 GB machine

An unconstrained container is a danger to the host. A multi-step prompt can grow the KV-cache and demand several extra gigabytes of RAM at once, so enforce limits with Linux cgroups.

docker run -d --name constrained-model \
  --cpus="4.0" \
  --memory="10g" \
  --memory-swap="10g" \
  -p 11434:11434 \
  ollama/ollama:latest

Diagram: FIG 3 · An unconstrained model container can take the whole machine down. Kernel cgroup limits keep the damage inside the container. The 10 GB cap is an example for a 16 GB host. See the animated version.

--memory="10g" caps the container at 10 GB and leaves about 6 GB for the host OS, networking daemons and file cache.
--memory-swap="10g" (equal to --memory) disables swap for the container. Swapping model weights to disk slows inference by orders of magnitude, so it is better to fail fast.
When the container exceeds its cap, the kernel's OOM killer stops that container, not the host. This is the default behaviour (--oom-kill-disable=false), so you do not need to set it. Never use --oom-kill-disable=true without a memory limit, because then a spike can hang the whole machine.
Add --gpus only when a GPU is present. On a CPU-only laptop, the memory and CPU caps are what protect you.

Treat these flags as part of your deployment definition, and in Kubernetes you will express the same limits as resource requests and limits.

Run it yourself

This is the exact example from the video. These are real docker run commands that read the limits from inside the container. There is no GPU on my laptop, so only the limits half of the lesson is shown.

# no GPU on this machine, so limits only
docker run --rm --cpus=2 --memory=256m --memory-swap=256m \
  --shm-size=2g alpine:3.20 sh -c '
  cd /sys/fs/cgroup
  for f in memory.max memory.swap.max cpu.max; do
    echo "$f: $(cat $f)"
  done
  echo shm: $(df -h /dev/shm | tail -1 | tr -s " " | cut -d" " -f2)'

docker run --rm --memory=64m --memory-swap=64m alpine:3.20 \
  sh -c 'x=$(dd if=/dev/zero bs=1M count=200 | tr "\0" x)'
echo "over the cap: exit code $?"

Run it:

bash limits.sh

Output I got while recording the video:

memory.max: 268435456
memory.swap.max: 0
cpu.max: 200000 100000
shm: 2.0G
over the cap: exit code 137

Coming Up Next

Day 4: Optimising model weights: quantization basics with GGUF and AWQ.

HardwareAcceleration #Docker #GPUs #DevOps #Linux #TechInfrastructure #CUDA #SystemAdministration

Originally published at https://sureshpallapothu.in/blog/ai-int-day-3-gpu-passthrough, where this post includes animated diagrams.

Day 2: Advanced Docker for AI — Multi-Stage Builds for Heavy Python Runtimes

Suresh Kumar Pallapothu — Sun, 08 Nov 2026 18:28:19 +0000

Containerising an AI workload is a different problem from containerising a web service. The dependencies are huge, the build tools are heavy, and an unoptimised image can pass 10 GB. Moving that across a registry slows every CI/CD run, and during a Kubernetes autoscale event each new pod must pull it before it can serve a single request.

The naive Dockerfile

This is the Dockerfile most teams start with.

FROM python:3.11
RUN apt-get update && apt-get install -y build-essential gcc
RUN pip install torch transformers fastapi uvicorn
COPY . /app
WORKDIR /app
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

It works, and it has three production problems.

Diagram: FIG 1 · A single-stage build carries compilers, caches and wheels into production. Layer sizes here are illustrative, not measured. See the animated version.

Build artifact accumulation. Compilers, intermediate wheels and pip caches are baked into read-only image layers and shipped everywhere.
Excessive attack surface. Development and compilation tools exist in production, which gives an attacker more to work with after a break-in.
Deployment latency. Pulling a large image can take minutes, which causes request timeouts while new pods start.

The multi-stage fix

Keep the build dependencies in an ephemeral builder stage. Copy only the compiled output into a minimal runtime stage.

Diagram: FIG 2 · Dependencies are built in stage 1. Only the finished virtual environment is copied into the clean runtime stage; everything else is discarded. See the animated version.

# syntax=docker/dockerfile:1

# ---- Stage 1: builder (heavy, discarded) ----
FROM python:3.11-slim AS builder
WORKDIR /build
RUN apt-get update \
 && apt-get install -y --no-install-recommends build-essential curl \
 && rm -rf /var/lib/apt/lists/*
RUN python -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"
COPY requirements.txt .
RUN --mount=type=cache,target=/root/.cache/pip \
    pip install --no-cache-dir -r requirements.txt

# ---- Stage 2: runtime (small, ships to production) ----
FROM python:3.11-slim
WORKDIR /app
RUN apt-get update \
 && apt-get install -y --no-install-recommends libgomp1 \
 && rm -rf /var/lib/apt/lists/*
COPY --from=builder /opt/venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"
RUN useradd -m -u 10001 aiuser
USER aiuser
COPY --chown=aiuser:aiuser ./src /app
EXPOSE 8000
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

Build it with BuildKit enabled, which current Docker versions use by default: docker build -t model-api:1.0 .. A few notes on this file:

The cache mount (--mount=type=cache) keeps pip's download cache outside the image, so repeat builds in CI do not re-download multi-gigabyte wheels. Because the cache is not in the layer, --no-cache-dir here only avoids writing a second copy.
libgomp1 is the OpenMP runtime that many numerical libraries need at run time. If your stack needs other shared libraries, install them in the runtime stage too, otherwise the app fails to start with a missing .so error.
If you need GPU support, use a CUDA-enabled base image for both stages, or install a CUDA build of PyTorch. That is the subject of Day 3.

Image hygiene checklist

Diagram: FIG 3 · Five small habits that keep AI images lean and safe. Do the apt cleanup in the same RUN command, or the deleted files still live in an earlier layer. See the animated version.

Add a .dockerignore listing .git, .venv, __pycache__ and raw test data, so the build context stays small and secrets do not leak in.
Run as a non-root user (USER aiuser).
Clean package caches in the same RUN command that created them. Deleting files in a later layer does not shrink the image, because the earlier layer still holds them.

Diagram: FIG 4 · Smaller images start faster. The sizes and network speed are illustrative inputs to simple arithmetic, so measure your own registry and network. See the animated version.

Image size is not vanity. At 100 MB/s, an 8 GB image takes about 80 seconds to pull and a 2.5 GB one about 25 seconds, and during a scale-up every new node pays that cost. Lean, deterministic images turn research dependencies into a repeatable, auditable artifact that a cluster scheduler can start quickly.

Run it yourself

This is the exact example from the video. Both images are really built and measured with Docker. The 100 MB random file in the builder stage stands in for compilers and wheels, so the size gap is real but exaggerated on purpose.

# syntax=docker/dockerfile:1
# ---- builder: heavy, thrown away ----
FROM alpine:3.20 AS builder
WORKDIR /build
RUN dd if=/dev/urandom of=toolchain.bin bs=1M count=100
RUN echo "model-api v1" > app.txt

# ---- runtime: small, ships ----
FROM alpine:3.20
RUN adduser -D -u 10001 aiuser
USER aiuser
COPY --from=builder /build/app.txt /app/
CMD ["cat", "/app/app.txt"]

Helper file Dockerfile.naive, used by the run:

FROM alpine:3.20
WORKDIR /build
RUN dd if=/dev/urandom of=toolchain.bin bs=1M count=100
RUN echo "model-api v1" > app.txt
CMD ["cat", "/build/app.txt"]

Helper file compare.sh, used by the run:

set -e
docker build -q -f Dockerfile.naive -t d2naive . >/dev/null
docker build -q -t d2lean . >/dev/null
echo "naive image: $(( $(docker save d2naive | wc -c) / 1000000 )) MB"
echo "multi-stage: $(( $(docker save d2lean | wc -c) / 1000000 )) MB"
echo "app says: $(docker run --rm d2lean)"
echo "runs as uid: $(docker run --rm d2lean id -u)"
docker rmi d2naive d2lean >/dev/null

Run it:

bash compare.sh

Output I got while recording the video:

naive image: 108 MB
multi-stage: 3 MB
app says: model-api v1
runs as uid: 10001

Coming Up Next

Day 3: Hardware acceleration in Docker: configuring GPU passthrough and resource limits for constrained hardware.

Docker #Containerization #DevOps #AIEngineering #FastAPI #CloudNative #PlatformEngineering #CyberSecurity

Originally published at https://sureshpallapothu.in/blog/ai-int-day-2-advanced-docker, where this post includes animated diagrams.

Day 1: The AI Infrastructure Shift — Why the Model is Only 10% of the Production System

Suresh Kumar Pallapothu — Sat, 07 Nov 2026 18:14:38 +0000

In enterprise AI, a common belief is that business value comes mainly from the choice of foundation model. Teams spend weeks benchmarking a 7-billion-parameter model against a 70-billion one, debating fine-tuning curves and polishing system prompts. Then, once a model is chosen, they hit a wall trying to move from a prototype on a laptop to a real production environment.

In real-world deployments the model is roughly 10% of the total engineering effort. The other 90% is the platform infrastructure needed to serve, scale, observe and secure that intelligence. This series is about that 90%.

In plain terms: A model is the engine. Nobody ships an engine to customers. They ship a car: fuel system, brakes, dashboard, locks, service schedule. This series builds the car.

Diagram: FIG 1 · The model is one block in the grid. The other eight blocks are the platform that serves, scales, observes and secures it. See the animated version.

Walk through the grid and you see the real job. Data has to flow in (ingestion and pipelines). Traffic has to be routed safely (ingress and a service mesh). State has to survive restarts (persistent storage and vector databases). Every change must ship safely (CI/CD and GitOps), every resource must be scheduled and capped, every request must be traceable (observability), and untrusted model-generated code must be contained (sandboxes and egress control). The model is one block of nine.

Diagram: FIG 2 · Teams spend weeks choosing a model, then discover that most of the work is the platform around it. See the animated version.

The hidden technical debt in machine learning

In 2015, Google researchers published a now-classic paper, Hidden Technical Debt in Machine Learning Systems. It showed that the machine-learning code in a real system is a small box surrounded by a much larger maze of infrastructure plumbing. With today's large language models (LLMs), that maze has grown:

Infrastructure plumbing: API rate limits, load balancing for high-concurrency prompts, context-window exhaustion, and database connections.
Resource allocation: inference is compute-bound and memory-bound. Unlike a stateless Node.js or Go service that sips memory, an LLM needs continuous GPU memory (VRAM), careful KV-cache management and hardware acceleration passed through to the container.
State and synchronization: real-time systems must keep vector databases, caches such as Redis, and model weights in step with low latency.

The memory problem, in numbers

The KV-cache is the model's working memory for the conversation. Every token a request holds in its context adds to it, for every layer of the network. That is why a model that fits comfortably for one user can run out of memory at ten.

Diagram: FIG 4 · Weights are a fixed cost. The KV-cache grows with every long request, so a model that fits one user can run out of memory at ten. Approximate, before runtime overhead. See the animated version.

Here is the arithmetic as a few lines of Python. The architecture numbers match a typical 8-billion-parameter model (32 layers, 8 key-value heads, a head size of 128). It is a rough budget that ignores runtime overhead, and I have run it:

# Rough GPU memory budget for serving one 8-billion-parameter model.
GIB = 2**30

def weights_gib(params: float, bytes_per_param: float) -> float:
    return params * bytes_per_param / GIB

def kv_cache_gib(tokens: int, layers=32, kv_heads=8, head_dim=128, bytes_per_value=2) -> float:
    # Keys and values (x2), for every layer, for every token held in context.
    return 2 * layers * kv_heads * head_dim * bytes_per_value * tokens / GIB

gpu_gib = 24
for name, bytes_per_param in [("FP16", 2), ("INT8", 1), ("INT4", 0.5)]:
    w = weights_gib(8e9, bytes_per_param)
    per_request = kv_cache_gib(8192)
    fits = int((gpu_gib - w) // per_request)
    print(f"{name}: weights {w:5.1f} GiB, one 8k-token request {per_request:.1f} GiB, "
          f"about {fits} concurrent requests fit in {gpu_gib} GiB")

Notice what changes the answer: quantization shrinks the weights, which leaves room for more concurrent requests. We cover that in Day 4.

The fragility of unmanaged inference

The fastest way to get a model online is to load it in a Python script and wrap it in Flask or a basic FastAPI app. It works in a demo. In production it fails in three predictable ways:

Memory leaks and out-of-memory panics: concurrent long prompts exhaust memory, and the Linux kernel's OOM-killer terminates the process with no warning.
No fault isolation: if the model runtime crashes, the whole application gateway goes with it, dropping live connections and losing in-flight work.
Security gaps: unsandboxed execution lets a prompt-injection attack reach the host filesystem, internal network sockets or environment variables, which often hold secrets.

Diagram: FIG 3 · A raw script takes the whole gateway down when it runs out of memory. Isolated pods contain the failure, and the platform replaces the failed one. See the animated version.

The platform engineering mandate

Reliable AI systems come from platform engineering habits. Stop treating AI as an isolated data-science experiment, and treat the inference runtime as a mission-critical infrastructure tier:

Containerization: package the runtime as a deterministic, immutable, multi-stage Docker image (Days 2 and 3).
Resource throttling: enforce memory and CPU quotas with Linux kernel cgroups, so one runaway request cannot take down a node (Day 3).
Network isolation: use a zero-trust service mesh and eBPF-based networking to restrict what a model or agent can reach (Week 3).

Diagram: FIG 5 · Containerize for repeatability, throttle with kernel quotas, and isolate the network, so one misbehaving model cannot hurt anything else. See the animated version.

A quick readiness check

Before you call a model "production-ready", ask:

What happens when the process runs out of memory, and who restarts it?
How many concurrent long requests fit before it fails, and has that been measured?
Can the image be rebuilt identically next month, on any machine?
What can the model, or an agent running its code, reach on the network?
If an answer is wrong at 3 a.m., can you trace which step caused it?
How does a new version reach production, and how do you roll it back?

If you cannot answer most of these, the model is not the problem. The platform is.

What the next 30 days build

Each week wraps another ring around the model, until you have something you could actually run in production, on a laptop with 16 GB of RAM.

Diagram: FIG 6 · Each week adds a ring around the model: containers, Kubernetes, network and security, then agents and CI/CD. See the animated version.

Reliable enterprise AI begins at the operating system and networking layer, not in the prompt window.

Coming Up Next

Day 2: Advanced Docker for AI: multi-stage builds for heavy Python runtimes.

AIInfrastructure #MLOps #PlatformEngineering #CloudNative #DevOps #EnterpriseAI #SystemDesign #TechLeadership

Originally published at https://sureshpallapothu.in/blog/ai-int-day-1-ai-infrastructure-shift, where this post includes animated diagrams.

Day 30: The 10-Year Horizon — Architecting IT Strategy for the AI-Native Enterprise

Suresh Kumar Pallapothu — Fri, 30 Oct 2026 18:47:28 +0000

Over the coming decade, AI is likely to move from an experimental add-on to the operating fabric of enterprise technology. Nobody can predict the details, but the direction is clear enough to plan for. For IT leaders running shared services and infrastructure portfolios, the job shifts from managing a pile of applications to architecting a unified platform that people and autonomous agents both use.

In plain terms: Think of city planning, not building one house. You lay down roads, power and water so that whatever gets built on top, today or in ten years, simply plugs in.

Diagram: An AI-native enterprise rests on a unified, governed data fabric, an orchestration layer that can swap models, identity and access management for autonomous agents, and human-centric leadership in which engineers direct and audit automated systems. See the animated version.

Pillar 1: A unified data fabric

AI is only as effective as the data it can reach. Replace fragmented application databases with a governed data fabric that exposes business logic and data to human teams and to agents through the same controlled interfaces. Day 11 to Day 14 showed how retrieval depends on it, and Day 29 showed why you must monitor it.

Pillar 2: Commoditised compute and a swappable model layer

Models and hardware keep getting cheaper and more alike. The lasting advantage is rarely owning one particular model. It is having an orchestration layer that handles routing, evaluation, cost and policy behind a stable interface, so you can swap the model underneath as price and quality change, and your applications do not notice.

Diagram: Models and hardware keep commoditising, so the lasting advantage is an orchestration layer that handles routing, evaluation, cost and policy behind a stable interface, so any foundation model can be swapped in as price and quality change without changing the applications. See the animated version.

Pillar 3: Zero-trust identity for autonomous entities

When agents execute tasks, manage infrastructure and deploy code, identity and access management has to cover non-human workers: granular permissions, short lifetimes and a full audit trail, exactly the zero-trust ideas from Day 26.

Diagram: As agents run tasks and deploy code, identity and access management must authenticate non-human workers with granular, time-bound and auditable privileges, much like giving a contractor a visitor badge for one job. See the animated version.

Pillar 4: Human-centric leadership

Automation takes over repetitive execution, which lifts engineers toward designing, directing and auditing complex systems. The teams that do well will be those who can orchestrate and question autonomous systems, not just operate them.

Diagram: As automation handles repetitive execution, engineers spend more time directing, auditing and architecting autonomous systems. The shares shown are illustrative and are not a forecast. See the animated version.

Here is the swappable-model idea in a few lines. It assumes OpenAI-compatible endpoints, which most hosted and self-hosted engines (Ollama, vLLM and many providers) offer. The model names and prices are placeholders.

from openai import OpenAI

# One place that knows about models. Apps ask for a capability, not a vendor.
MODELS = {
    "cheap-fast": {"base_url": "http://ollama.internal:11434/v1", "model": "llama3.1:8b",   "key": "x"},
    "strong":     {"base_url": "https://api.provider.example/v1",  "model": "big-model-v2", "key": "env:PROVIDER_KEY"},
}
ROUTES = {"summarise": "cheap-fast", "contract-review": "strong"}   # change a line, not an app

def ask(task: str, prompt: str) -> str:
    cfg = MODELS[ROUTES[task]]
    client = OpenAI(base_url=cfg["base_url"], api_key=resolve_secret(cfg["key"]))
    reply = client.chat.completions.create(model=cfg["model"],
                                           messages=[{"role": "user", "content": prompt}])
    return reply.choices[0].message.content

Where this leaves you

Over 30 days we went from what AI is, through data, models, APIs, retrieval, infrastructure, containers and Kubernetes, to agents, security, observability, human oversight and governance. The through-line is simple: build systems that are deterministic where they must be, secure by default, observable, and led by people. Do that today and you are laying the foundation for the AI-native enterprise of tomorrow.

Coming Up Next

AI Intermediate: Infrastructure and Agentic Systems is the next course, starting on 7 November, after a one-week break. Thank you for following along, and share this series with someone who is starting their AI journey.

ITStrategy #FutureOfTech #CIO #EnterpriseArchitecture #DigitalTransformation

Originally published at https://sureshpallapothu.in/blog/day-30-ten-year-horizon, where this post includes animated diagrams.

Day 29: Managing Technical Debt, Drift and Model Governance in Enterprise IT

Suresh Kumar Pallapothu — Thu, 29 Oct 2026 19:10:25 +0000

Shipping a model is not the finish line. Ordinary software stays exactly as correct as it was on release day until someone changes the code. An AI system can get worse without anyone touching it, because the world feeding it keeps moving. That slow decay is called drift, and managing it is the heart of long-term AI maintenance.

In plain terms: A map is accurate the day it is printed. A year later new roads exist, and the map still looks confident while sending you down a closed street.

Diagram: Data drift happens when the distribution of production inputs moves away from the training data, for example when a firmware update changes the units a device reports. The model was never trained on the new range, so its accuracy degrades quietly. See the animated version.

1. Two kinds of drift

Data drift (covariate shift): the statistics of the production inputs move away from the training data. For example, a firmware update makes switches report a value in new units, and a model that parses that telemetry suddenly sees numbers in a range it never learned.
Concept drift: the relationship between inputs and the correct answer changes. A fraud model trained on old spending patterns starts misclassifying transactions after procurement policy changes. The inputs look normal, but the right answer is different.

Diagram: Data drift means the inputs change, such as new units or formats. Concept drift means the relationship between inputs and the correct answer changes, such as a fraud model facing a new procurement policy. Both cause silent decay without raising an error. See the animated version.

2. Governance controls

Automated drift detection. Continuously compare production data with the baseline using a statistic such as the Population Stability Index (PSI) or Kullback-Leibler divergence, and alert when it crosses a limit.
Model inventory and lineage. Keep one registry of every active model: its training data version, quantization settings, licence obligations (for example Apache 2.0 versus a non-commercial licence) and business owner.
Retraining and retirement. Set schedules for retraining and a clear path for retiring old models, so obsolete weights and unmaintained integrations do not pile up as technical debt.

Diagram: Population Stability Index compares production data with the baseline, feature by feature. A common rule of thumb treats under 0.1 as stable, 0.1 to 0.25 as worth watching and above 0.25 as needing action. Kullback-Leibler divergence is another option, and the thresholds should be tuned to your data. See the animated version.

Diagram: Governance means keeping a central registry of every active model with its training data version, quantization, licence obligations and business owner, and defining a path from registration through monitoring and retraining to retirement. The example entry is illustrative. See the animated version.

Here is a PSI check in Python. It assumes baseline and current are NumPy arrays of one numeric feature. The 0.1 and 0.25 limits are common rules of thumb, not laws, so tune them to your data.

import numpy as np

def psi(baseline, current, bins: int = 10) -> float:
    edges = np.quantile(baseline, np.linspace(0, 1, bins + 1))      # bins from the training data
    edges[0], edges[-1] = -np.inf, np.inf
    b = np.histogram(baseline, edges)[0] / len(baseline)
    c = np.histogram(current, edges)[0] / len(current)
    b, c = np.clip(b, 1e-6, None), np.clip(c, 1e-6, None)           # avoid log(0)
    return float(np.sum((c - b) * np.log(c / b)))

score = psi(train_latency_ms, last_week_latency_ms)
if score > 0.25:
    print(f"PSI {score:.2f}: significant drift, investigate or retrain")
elif score > 0.10:
    print(f"PSI {score:.2f}: moderate drift, watch closely")
else:
    print(f"PSI {score:.2f}: stable")

Run a check like this per feature on a schedule, send the results to your monitoring system, and tie alerts to an owner listed in the inventory. Proactive governance catches silent failures before your users do.

Coming Up Next

Day 30: The 10-year horizon: architecting IT strategy for the AI-native enterprise.

ModelGovernance #TechnicalDebt #MLOps #DataDrift #EnterpriseIT

Originally published at https://sureshpallapothu.in/blog/day-29-drift-and-governance, where this post includes animated diagrams.

Day 28: Human-in-the-Loop Architecture — Machine Second, Human First in Practice

Suresh Kumar Pallapothu — Wed, 28 Oct 2026 19:03:07 +0000

Efficiency is the goal of automation, but in high-consequence settings, full autonomy is the wrong target. Actions that change production data, move money, reconfigure network hardware or contact customers should combine the machine's speed with verified human judgment. The principle is Human First, Machine Second.

In plain terms: A junior colleague can read, research and draft all day without asking. But before they wire money or delete a database, they get a sign-off from someone senior.

1. Define the decision boundary

Divide actions into two classes with automatic gates.

Reversible, low-impact (fully autonomous): reading documentation, summarising logs, querying read-only metrics, drafting a remediation plan.
Irreversible or high-impact (gated): dropping a table, changing a switch VLAN, deploying to production, approving spending.

Decide the class in code and policy, using an allow-list of action types. Do not ask the model whether its own action is risky, because a confused or manipulated model will answer wrongly.

Diagram: Low-impact reversible actions such as reading docs, summarising logs and drafting plans can run autonomously. Irreversible or high-impact actions such as dropping tables, changing network configuration, deploying to production or approving spend need human approval. The classification is set in code and policy, not by the model. See the animated version.

2. Engineer the approval gate

A human-in-the-loop workflow needs an asynchronous pause and resume inside your orchestration.

Interrupt. When an action is high-impact, the agent stops, saves its state to durable storage and enters a WAITING_FOR_APPROVAL state.
Notify with context. An alert goes to Slack, Teams or a portal with the proposed command, the agent's reasoning and the expected outcome.
Authorise. The human approves or rejects, and the decision is authenticated and signed so it cannot be forged.
Resume or revise. On a verified approval the saved state is restored and the action runs. On a rejection the human's note is added to the agent's context so it drafts a safer alternative.

Diagram: When an action meets the high-impact criteria the agent halts and saves its state, a notification with the proposed command and reasoning goes to a human, the human approves or rejects with authenticated authorisation, and the workflow resumes from stored state or revises its plan. See the animated version.

Diagram: A good approval request shows the proposed command, the agent's reasoning and the expected outcome in one place, with approve and reject actions. The example is illustrative. See the animated version.

Diagram: If the human rejects a proposed action, their note is added to the agent's context so it drafts a safer alternative. This keeps people reviewing only the actions that matter while the machine does the legwork. See the animated version.

Here is the core logic in Python. It assumes a database helper db for saving state and a shared secret for signing approvals. In production you would use your identity provider and a proper signing scheme, and a framework such as LangGraph offers built-in interrupt and resume.

import hmac, hashlib, json

HIGH_IMPACT = {"drop_table", "change_vlan", "deploy_prod", "approve_spend"}   # policy, not model opinion
SECRET = b"load-from-a-secrets-manager"

def execute_or_pause(run_id: str, action: dict, state: dict):
    if action["type"] not in HIGH_IMPACT:
        return perform(action)                                   # reversible: just do it

    db.save(run_id, {"status": "WAITING_FOR_APPROVAL", "action": action, "state": state})
    notify_approvers(run_id, action, reasoning=state["reasoning"])   # command + why + expected outcome
    return "paused"

def sign(run_id: str, decision: str, approver: str) -> str:
    msg = f"{run_id}:{decision}:{approver}".encode()
    return hmac.new(SECRET, msg, hashlib.sha256).hexdigest()

def resume(run_id: str, decision: str, approver: str, signature: str, note: str = ""):
    if not hmac.compare_digest(signature, sign(run_id, decision, approver)):
        raise PermissionError("Invalid approval signature")      # never trust an unsigned decision
    saved = db.load(run_id)
    if decision == "approve":
        return perform(saved["action"])
    saved["state"]["human_feedback"] = note                      # rejection: teach the next attempt
    return replan(saved["state"])

Balancing autonomous reasoning with human oversight keeps the organisation safe while still capturing most of the productivity gain, because people only review the small share of actions that carry real risk.

Coming Up Next

Day 29: Managing technical debt, drift and model governance in enterprise IT.

HumanInTheLoop #HITL #EnterpriseGovernance #TechLeadership #ProcessAutomation

Originally published at https://sureshpallapothu.in/blog/day-28-human-in-the-loop, where this post includes animated diagrams.

Day 27: Observability and Tracing for Agentic Systems — Telemetry, Logs and Metrics

Suresh Kumar Pallapothu — Tue, 27 Oct 2026 18:52:51 +0000

Standard monitoring watches status codes, latency and memory. Those numbers can look perfect while an agent is failing. It can return HTTP 200 OK with a completely made-up answer, or burn through forty planning steps going in circles. To debug an agent you need to see its decisions, not only its servers.

In plain terms: A flight tracker that only shows "plane is in the air" tells you nothing about whether it is heading to the right city. You need the flight path.

Diagram: An agent can return HTTP 200 with normal latency while giving a hallucinated answer or looping on planning. Standard server health metrics cannot see that, so agent systems need traces that record their decisions. See the animated version.

The three pillars of agent observability

Traces capture the end-to-end path of a run. Every model call, retrieval and tool call becomes a nested span with its time, input tokens and output tokens, plus enough prompt context to reproduce the step.
Metrics track the cost and health of the system: token use, cache hit ratio, context-window utilisation and cost per transaction, by team or workflow.
Logs record structured state changes in a multi-agent graph, so you can find the step where the reasoning diverged from what you expected.

Diagram: A trace records every step of an agent run as nested spans: planning, retrieval, tool calls and answer generation, each with its latency and its input and output token counts. The span names and numbers are illustrative. See the animated version.

Diagram: Agent observability uses three kinds of data. Traces show the path of one run, metrics track tokens, cache hits, context use and cost, and logs record state changes so you can find where reasoning diverged. See the animated version.

Standards: OpenTelemetry and correlation IDs

Vendor-neutral instrumentation. Using OpenTelemetry (OTel) means your telemetry can flow into Prometheus, Grafana, Jaeger or a commercial backend without locking you to one vendor. OpenTelemetry also has emerging conventions for generative AI attributes, which are still evolving, so check the current names.
Correlation IDs. Give each incoming request a unique trace ID that follows it through the gateway, application, vector database and agent workers. Search that ID and you see one user interaction end to end.

Diagram: A unique trace or correlation ID attached to each incoming request travels through the gateway, application, vector database and agent workers, so operators can follow one user interaction end to end. OpenTelemetry propagates it across services. See the animated version.

Here is a minimal Python example. It assumes you have installed opentelemetry-sdk and opentelemetry-exporter-otlp, and that a collector or a recent Jaeger is listening for OTLP on port 4317. The llm and run_tool functions stand in for your own.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint="localhost:4317", insecure=True)))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("support-agent")

def handle(question: str, request_id: str):
    with tracer.start_as_current_span("agent.run") as run:
        run.set_attribute("app.request_id", request_id)

        with tracer.start_as_current_span("llm.plan") as span:
            plan = llm(question)                                  # your model call
            span.set_attribute("gen_ai.usage.input_tokens", plan.input_tokens)
            span.set_attribute("gen_ai.usage.output_tokens", plan.output_tokens)

        with tracer.start_as_current_span("tool.run_sql") as span:
            span.set_attribute("tool.name", "run_sql")
            rows = run_tool("run_sql", plan.sql)
            span.set_attribute("tool.rows_returned", len(rows))   # 0 rows is a useful signal
        return rows

One caution: prompts can contain customer or internal data, so redact sensitive fields before they go into spans or logs. With tracing in place, agent debugging moves from guesswork to something you can inspect, replay and audit.

Coming Up Next

Day 28: Human-in-the-loop architecture: machine second, human first in practice.

Observability #OpenTelemetry #DistributedTracing #MLOps #SRE

Originally published at https://sureshpallapothu.in/blog/day-27-agent-observability, where this post includes animated diagrams.

Day 26: Zero-Trust Security for AI — Sandboxing Ephemeral Execution and Model Egress

Suresh Kumar Pallapothu — Mon, 26 Oct 2026 19:31:01 +0000

Once an agent can run code (a Python script to analyse a dataset, a command on a server), the old network perimeter stops protecting you. The code was written by a model, and you cannot treat it as trusted. A zero-trust design starts from one assumption: the model will eventually produce something buggy, unsafe or compromised, so the system must stay safe anyway.

In plain terms: Letting an agent run code on your server is like letting a stranger use your office computer. You would give them a locked guest room with no network cable, a time limit, and a visitor badge that opens nothing important.

Diagram: Under zero-trust principles you assume the model will eventually produce buggy, unsafe or compromised code. Generated code must never run on a host or production container. It runs inside a sandbox that limits what it can steal, escalate or change. See the animated version.

1. Runtime sandboxing: shrink the blast radius

Running generated code directly on a host or in a production container risks data theft, privilege escalation and damaging changes. Instead:

Isolation. Run it in an unprivileged, ephemeral environment such as a gVisor sandbox, a Firecracker microVM or an unprivileged throwaway container. MicroVMs and gVisor give a stronger boundary than a plain container, because they do not share the host kernel directly.
Immutable filesystem. Give it read-only access to the base system and a temporary in-memory disk that is destroyed as soon as the task ends.
Resource throttling. Use Linux cgroups limits on CPU, memory, disk I/O and process count, so a runaway or malicious script cannot starve the machine.

Diagram: Agent-generated code runs inside layered isolation: a microVM or gVisor boundary, a read-only filesystem with a temporary in-memory disk, and kernel cgroup limits on CPU, memory and I/O. The whole environment is destroyed as soon as the task ends. See the animated version.

2. Egress and privilege minimisation

No network by default. A sandbox should start with no network access at all. If a task needs one external API, allow that single destination and keep everything else blocked, so nothing can be sent out.
Least privilege. Database credentials handed to an agent should be read-only, limited to non-sensitive schemas and short-lived. Administrative actions, such as schema changes, need an authenticated elevation step.

Diagram: A code-execution sandbox should have no network access by default. If a task needs an external API, allow that single destination and keep everything else, including the local network and the internet, blocked so data cannot leak out. See the animated version.

Diagram: Credentials given to an agent should be read-only, limited to non-sensitive schemas and short-lived. Administrative actions such as schema changes require an authenticated elevation step, and everything is logged. See the animated version.

Here is a practical starting point: a Python helper that runs untrusted code in a locked-down throwaway container. It assumes Docker is installed. For stronger isolation, add --runtime=runsc if you have gVisor set up, or use a microVM platform.

import subprocess, tempfile, pathlib

def run_untrusted(code: str, timeout_s: int = 10) -> str:
    with tempfile.TemporaryDirectory() as d:
        script = pathlib.Path(d, "task.py")
        script.write_text(code)
        cmd = [
            "docker", "run", "--rm",
            "--network", "none",                 # zero network access
            "--read-only",                       # read-only root filesystem
            "--tmpfs", "/tmp:size=64m",          # small in-memory scratch space
            "--memory", "256m", "--cpus", "0.5", # cgroup limits
            "--pids-limit", "64",                # no fork bombs
            "--cap-drop", "ALL",                 # drop Linux capabilities
            "--security-opt", "no-new-privileges",
            "--user", "65534:65534",             # unprivileged user
            "-v", f"{script}:/task.py:ro",
            "python:3.11-slim", "python", "/task.py",
        ]
        try:
            done = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout_s)
        except subprocess.TimeoutExpired:
            return "Stopped: time limit exceeded"
        return (done.stdout + done.stderr)[:4000]     # cap the output size too

Treat the agent's execution environment as an untrusted runtime boundary. That is what lets you give agents real capabilities without handing them the keys to the enterprise.

Coming Up Next

Day 27: Observability and tracing for agentic systems: telemetry, logs and metrics.

ZeroTrust #CyberSecurity #DevSecOps #Sandboxing #AIAgents

Originally published at https://sureshpallapothu.in/blog/day-26-zero-trust-ai, where this post includes animated diagrams.

Day 25: Automated CI/CD — Automating the Software Lifecycle for AI Models

Suresh Kumar Pallapothu — Sun, 25 Oct 2026 17:12:01 +0000

A normal unit test checks that a function returns an exact value. AI is non-deterministic: editing a system prompt, changing the temperature or pointing at a newer model can quietly reduce accuracy, add hallucinations or break the JSON your code depends on, and no exact-match test will notice. MLOps closes the gap by putting evaluation into the Git workflow.

In plain terms: A restaurant does not taste-test every dish by checking its weight. It has a tasting panel with a score sheet, and a new recipe only goes on the menu if the score does not drop.

1. The evaluation pipeline

Any pull request that touches prompt templates, RAG indexing or agent tools should trigger automatic checks before it can merge.

Schema and lint tests. Assert that outputs follow the declared JSON schema and regex rules across a set of test inputs.
Gold-standard suite. Run the pipeline against a version-controlled set of real edge cases and compare the pass rate with the last release.
LLM-as-judge. Use a separate evaluator model to grade faithfulness, relevance and use of context, and flag a drop below the baseline.

Diagram: Any pull request that changes prompt templates, RAG indexing or agent tooling must pass automated checks before merging: schema and regex tests, a gold-standard test suite, and an LLM judge scoring faithfulness and relevance. See the animated version.

Diagram: Running a version-controlled gold-standard test suite on every change gives a pass-rate trend. A prompt change that drops the score below the agreed baseline is flagged before it ships. The numbers are illustrative. See the animated version.

2. GitOps and automated rollout

Release through GitOps, with a tool such as ArgoCD.

Declarative infrastructure. Model settings, image tags and replica counts live as manifests in a Git repository. ArgoCD keeps the cluster in line with that repository, so a rollback is a revert of one commit.
Canary rollouts. A new model version first receives a small share of real traffic, for example 5%, while error rate and latency are watched. If the numbers hold, the share grows step by step to 100%, and if a threshold is crossed the rollout reverts on its own. In the Argo ecosystem the canary itself is run by Argo Rollouts, which works alongside ArgoCD.

Diagram: In GitOps the model version, image tag and replica limits are declared in a Git repository, and ArgoCD keeps the cluster in sync with it, pulling drifted resources back. A rollback is a revert of one commit. See the animated version.

Diagram: A new model version receives a small share of traffic first, for example 5%, while error rate and latency are watched. If metrics stay healthy the share grows step by step to 100%, and if a threshold is exceeded the rollout reverts automatically. ArgoCD syncs the manifests, and Argo Rollouts performs the canary. See the animated version.

Below are the two main pieces. They assume a tests/evals/ folder with your gold cases, a helper called run_pipeline, and a judge model you provide. Treat the thresholds as starting points to tune.

# tests/evals/test_quality.py  (run by CI with: pytest tests/evals)
import json
from pipeline import run_pipeline          # your RAG or agent entry point
from judge import faithfulness_score       # calls an isolated judge model

BASELINE = 0.88                            # pass rate of the currently deployed version
CASES = json.load(open("tests/evals/gold_cases.json"))

def test_outputs_match_schema():
    for case in CASES:
        out = run_pipeline(case["input"])
        assert set(out) >= {"answer", "sources"}           # schema check

def test_quality_not_below_baseline():
    passed = sum(faithfulness_score(c, run_pipeline(c["input"])) >= 0.8 for c in CASES)
    assert passed / len(CASES) >= BASELINE - 0.02          # small tolerance for judge noise

# Argo Rollouts: a canary that shifts traffic in steps
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: model-api
spec:
  replicas: 4
  selector:
    matchLabels: { app: model-api }
  strategy:
    canary:
      steps:
        - setWeight: 5
        - pause: { duration: 10m }      # watch error rate and latency here
        - setWeight: 25
        - pause: { duration: 10m }
        - setWeight: 50
        - pause: { duration: 10m }
        - setWeight: 100
  template:
    metadata:
      labels: { app: model-api }
    spec:
      containers:
        - name: model-api
          image: registry.example.com/model-api:1.5.0

For automatic rollback you attach an analysis that queries your metrics and fails the rollout when they cross a threshold. Automating testing and release turns AI deployments from manual experiments into repeatable, verifiable engineering.

Coming Up Next

Day 26: Zero-trust security for AI: sandboxing ephemeral execution and model egress.

MLOps #CICD #GitOps #ArgoCD #PlatformEngineering

Originally published at https://sureshpallapothu.in/blog/day-25-ai-cicd, where this post includes animated diagrams.

Day 24: Multi-Agent Orchestration — Supervisor, Worker and Evaluator Patterns

Suresh Kumar Pallapothu — Sat, 24 Oct 2026 16:53:08 +0000

As a workflow grows, handing one agent every tool and a long list of responsibilities makes it worse. The prompt gets crowded, the model picks the wrong tool and the context window fills up. The fix is the one human organisations use: divide the work among specialists who collaborate.

In plain terms: A hospital does not give one doctor every job. A coordinator sends each patient to the right specialist, and a second doctor reviews the plan before treatment begins.

Diagram: Giving one agent every tool and a huge prompt dilutes its focus and causes tool-selection mistakes. Splitting work across specialised agents with narrow scopes and short prompts is usually more reliable and easier to test. See the animated version.

Three patterns

Worker (specialisation). A small agent with a narrow scope and its own limited tools. A log-parser worker can only read logs. A database worker has a read-only SQL connection. A tight scope improves accuracy and limits damage.
Supervisor (routing and delegation). The coordinator. It reads a high-level request, splits it into sub-tasks, hands them to the right workers and combines the results into one answer.
Evaluator (validation and guardrails). An independent agent that reviews worker output against policy, accuracy rules and business logic before anything is committed or shown to the user. Keeping it separate matters, because a worker checking its own work tends to approve it.

Diagram: A supervisor splits a request into sub-tasks and routes them to narrow worker agents, each with its own limited tools. An independent evaluator reviews their output against policy and quality rules before anything is committed or returned to the user. See the animated version.

State and orchestration

Coordinating several agents calls for a real state machine. Frameworks such as LangGraph (Python) and Mastra (TypeScript) model the system as a directed graph.

Nodes are agents, tool executions or human-approval checkpoints.
Edges are routing rules based on what earlier nodes returned.
Durable state records progress after each step (a checkpoint), so if a pod crashes mid-run the workflow resumes from the last confirmed node instead of starting over.

Diagram: Frameworks such as LangGraph or Mastra model multi-agent workflows as directed graphs. Nodes are agents, tool calls or human approval checkpoints, and edges are routing decisions based on the outputs of earlier nodes. See the animated version.

Diagram: Saving a checkpoint after each step means that if a pod crashes mid-run the workflow resumes from the last confirmed step instead of restarting from the beginning. Frameworks such as LangGraph provide checkpointing for this. See the animated version.

Here is the pattern without any framework, so the idea is clear. The ask_* functions stand in for model calls with different system prompts and tools, and a framework adds the graph, retries and persistent checkpoints for you.

def run_workflow(request: str, state: dict) -> str:
    # Supervisor splits the request into sub-tasks (cached in state so a restart skips done work)
    state.setdefault("plan", ask_supervisor(request))            # e.g. ["parse_logs", "check_db"]

    for task in state["plan"]:
        if task in state.setdefault("results", {}):
            continue                                              # resume: this step already finished
        worker = WORKERS[task]                                    # each worker has its own narrow tools
        state["results"][task] = worker(request)
        save_checkpoint(state)                                    # durable state after every step

    draft = ask_supervisor_combine(state["results"])
    verdict = ask_evaluator(draft, policy=POLICY)                 # independent quality gate
    if verdict != "approved":
        return "Held for human review"                            # an approval node in a real graph
    return draft

Structuring agents as modular teams mirrors high-performing engineering organisations: clear roles, small scopes and a reviewer, which is what makes complex tasks dependable.

Coming Up Next

Day 25: Automated CI/CD: automating the software lifecycle for AI models.

MultiAgentSystems #LangGraph #SystemDesign #AgenticAI #SoftwareEngineering

Originally published at https://sureshpallapothu.in/blog/day-24-multi-agent-orchestration, where this post includes animated diagrams.

Day 23: Tool Use and Function Calling — Connecting LLMs to APIs, Databases and Shells

Suresh Kumar Pallapothu — Fri, 23 Oct 2026 17:57:27 +0000

By default a language model is an isolated reasoning engine. It cannot read the current time, query your private database or run a command. Function calling, also called tool use, is the bridge that connects that engine to real systems, so the model becomes a router that decides what to call while your software does the calling.

In plain terms: The model is a manager who can write instructions but cannot touch anything. It hands a clear, structured request to an assistant (your application), who does the task and reports back.

Diagram: On its own an LLM only has its trained weights. It cannot read the current time, query a private database, call a live API or run commands. Function calling is the bridge: the model asks for an action and your software performs it. See the animated version.

The four-step protocol

A common misconception is that the LLM runs the code. It does not. Function calling is a deterministic, four-step exchange.

Declare the tools. Your application sends the model a list of tools described with JSON schemas: the name, what it is for and its typed parameters.
Model invocation. If the question needs outside data or an action, the model replies with a structured tool call (the function name and exact arguments) instead of ordinary text.
Client-side execution. Your application intercepts that reply, validates it and runs the real code: a SQL query, a REST call, a script.
Observation injection. Your application sends the result back as a tool message, and the model uses it to finish its answer.

Diagram: In function calling the application declares tools as JSON schemas, the model replies with a structured tool call instead of text, the application validates and executes the real code, and the result is returned to the model as an observation so it can write the final answer. The model never runs code itself. See the animated version.

Diagram: The application describes a tool with a name, a purpose and typed parameters. The model answers with a structured JSON tool call naming the function and its arguments, and the application sends the real result back as an observation. The names and values are illustrative. See the animated version.

Type safety and validation

Free text breaks software pipelines. Use structured arguments and validate them with JSON Schema or a library such as Pydantic, so a malformed value is rejected before it reaches your backend. Then add checks the model cannot override: an allow-list of tools and a permission check for the current user.

Diagram: Before running a tool call, the application checks the arguments against a schema, checks the tool is on an allow-list, checks the user has permission, and only then executes and logs it. Bad or unsafe calls are rejected. Pydantic or JSON Schema keeps arguments type-safe. See the animated version.

Here is the loop with an OpenAI-compatible API. Ollama and vLLM from Day 16 expose the same interface, so the pattern works with a local model too. It assumes a model that supports tool calling, and get_gpu_status is a stand-in for your own function.

import json
from openai import OpenAI
from pydantic import BaseModel

client = OpenAI()                     # or OpenAI(base_url="http://localhost:11434/v1", api_key="x")

class GpuArgs(BaseModel):             # type-safe arguments
    node: str

def get_gpu_status(node: str) -> dict:
    return {"node": node, "status": "degraded"}   # replace with a real lookup

TOOLS = [{"type": "function", "function": {
    "name": "get_gpu_status",
    "description": "Returns GPU health for a node",
    "parameters": GpuArgs.model_json_schema(),
}}]
ALLOWED = {"get_gpu_status": (GpuArgs, get_gpu_status)}

messages = [{"role": "user", "content": "Is gpu-07 healthy?"}]
reply = client.chat.completions.create(model="gpt-4o-mini", messages=messages, tools=TOOLS)
msg = reply.choices[0].message

if msg.tool_calls:
    messages.append(msg)
    for call in msg.tool_calls:
        schema, fn = ALLOWED[call.function.name]                       # allow-list
        args = schema.model_validate_json(call.function.arguments)     # validate
        result = fn(**args.model_dump())                               # execute
        messages.append({"role": "tool", "tool_call_id": call.id,
                         "content": json.dumps(result)})               # observation
    final = client.chat.completions.create(model="gpt-4o-mini", messages=messages)
    print(final.choices[0].message.content)

Restricting the model to proposing structured arguments keeps execution under your control: authenticated, validated and logged inside your own application.

Coming Up Next

Day 24: Multi-agent orchestration: supervisor, worker and evaluator patterns.

FunctionCalling #ToolUse #APIs #Pydantic #SoftwareArchitecture

Originally published at https://sureshpallapothu.in/blog/day-23-function-calling, where this post includes animated diagrams.

DEV Community: Suresh Kumar Pallapothu

Day 4: Optimizing Model Weights — Quantization Basics (GGUF, AWQ) for Efficient Inference

Bytes per parameter

The mathematics

Choosing a format

The quality trade-off

Run it yourself

Coming Up Next

Quantization #MachineLearning #Optimization #LLMs #GGUF #vLLM #DataScience #AIEngineering

Day 3: Hardware Acceleration in Docker — GPU Passthrough and Limits for Constrained Hardware

The acceleration architecture

Launching an accelerated container

Why --shm-size matters

Hard limits for a 16 GB machine

Run it yourself

Coming Up Next

HardwareAcceleration #Docker #GPUs #DevOps #Linux #TechInfrastructure #CUDA #SystemAdministration

Day 2: Advanced Docker for AI — Multi-Stage Builds for Heavy Python Runtimes

The naive Dockerfile

The multi-stage fix

Image hygiene checklist

Run it yourself

Coming Up Next

Docker #Containerization #DevOps #AIEngineering #FastAPI #CloudNative #PlatformEngineering #CyberSecurity

Day 1: The AI Infrastructure Shift — Why the Model is Only 10% of the Production System

The hidden technical debt in machine learning

The memory problem, in numbers

The fragility of unmanaged inference

The platform engineering mandate

A quick readiness check

What the next 30 days build

Coming Up Next

AIInfrastructure #MLOps #PlatformEngineering #CloudNative #DevOps #EnterpriseAI #SystemDesign #TechLeadership

Day 30: The 10-Year Horizon — Architecting IT Strategy for the AI-Native Enterprise

Pillar 1: A unified data fabric

Pillar 2: Commoditised compute and a swappable model layer

Pillar 3: Zero-trust identity for autonomous entities

Pillar 4: Human-centric leadership

Where this leaves you

Coming Up Next

ITStrategy #FutureOfTech #CIO #EnterpriseArchitecture #DigitalTransformation

Day 29: Managing Technical Debt, Drift and Model Governance in Enterprise IT

1. Two kinds of drift

2. Governance controls

Coming Up Next

ModelGovernance #TechnicalDebt #MLOps #DataDrift #EnterpriseIT

Day 28: Human-in-the-Loop Architecture — Machine Second, Human First in Practice

1. Define the decision boundary

2. Engineer the approval gate

Coming Up Next

HumanInTheLoop #HITL #EnterpriseGovernance #TechLeadership #ProcessAutomation

Day 27: Observability and Tracing for Agentic Systems — Telemetry, Logs and Metrics

The three pillars of agent observability

Standards: OpenTelemetry and correlation IDs

Coming Up Next

Observability #OpenTelemetry #DistributedTracing #MLOps #SRE

Day 26: Zero-Trust Security for AI — Sandboxing Ephemeral Execution and Model Egress

1. Runtime sandboxing: shrink the blast radius

2. Egress and privilege minimisation

Coming Up Next

ZeroTrust #CyberSecurity #DevSecOps #Sandboxing #AIAgents

Day 25: Automated CI/CD — Automating the Software Lifecycle for AI Models

1. The evaluation pipeline

2. GitOps and automated rollout

Coming Up Next

MLOps #CICD #GitOps #ArgoCD #PlatformEngineering

Day 24: Multi-Agent Orchestration — Supervisor, Worker and Evaluator Patterns

Three patterns

State and orchestration

Coming Up Next

MultiAgentSystems #LangGraph #SystemDesign #AgenticAI #SoftwareEngineering

Day 23: Tool Use and Function Calling — Connecting LLMs to APIs, Databases and Shells

The four-step protocol

Type safety and validation

Coming Up Next

FunctionCalling #ToolUse #APIs #Pydantic #SoftwareArchitecture

Why `--shm-size` matters