DEV Community: Suresh Kumar Pallapothu

Day 22: Autonomous Agents — From Passive LLMs to Goal-Driven Execution

Suresh Kumar Pallapothu — Thu, 22 Oct 2026 18:08:44 +0000

Early generative AI was passive: you send a prompt, the model returns text, and a human decides what to do next. That is useful for drafting, but it cannot run a business workflow on its own. The next step is the autonomous agent: a system that can plan, check its own progress and carry out a multi-step goal with little human intervention.

In plain terms: A chatbot is a colleague who answers your question and then waits. An agent is a colleague you give a task to, who comes back when it is done, or when they are stuck.

Diagram: A passive LLM returns one text answer and a human decides the next step. An autonomous agent takes a goal and plans, acts, checks the result and repeats with little human intervention. That makes it useful for workflows and also riskier, so guardrails matter. See the animated version.

1. The core agentic loop

An agent runs a repeating cycle, borrowed from classic control systems.

Perceive: take in the goal, the current state and any context.
Plan: break an abstract goal, such as "audit this quarter's server error logs and flag failing optics", into an ordered list of concrete tasks.
Act: pick a capability and use it: call an API, run a database query, read a file.
Reflect: compare the result with the goal. If something went wrong, adjust and try again.

Diagram: An agent loops through four stages: perceive the goal and context, plan the tasks, act by calling a tool, and reflect on the result. It repeats until the goal is met or a limit such as a step budget is reached. See the animated version.

Reflection is what makes an agent more than a script. If a query returns no rows, the agent can notice why, correct its approach and run it again. In practice you always cap the number of steps and the cost, because an agent that keeps retrying can loop forever.

Diagram: After an action the agent observes the result. When a query returns no rows it reflects, notices the wrong column name, corrects the query and tries again. The example is illustrative, and a step limit prevents endless retries. See the animated version.

2. Memory: short-term and long-term

Working memory is the context window: the current plan, intermediate values and recent tool outputs. It is fast but limited, and it disappears when the run ends.
Episodic memory is a record of past runs kept in a vector database. When the agent meets a similar problem, it can retrieve how it solved a related one before.

Diagram: Working memory is the context window holding the current plan, state and recent tool outputs. Episodic memory stores past runs in a vector database so an agent can recall how it fixed a similar problem before. See the animated version.

Here is the loop in a few lines of Python. The llm and tools pieces are placeholders for your model call and your real functions, so treat this as the shape of an agent, not a finished one.

MAX_STEPS = 8                      # a hard stop so the loop cannot run forever

def run_agent(goal: str) -> str:
    history = [f"Goal: {goal}"]
    for step in range(MAX_STEPS):
        # Plan + act: the model chooses the next tool call, or says it is done
        decision = llm("\n".join(history))          # e.g. {"tool": "run_sql", "args": {...}} or {"done": "..."}
        if "done" in decision:
            return decision["done"]

        result = tools[decision["tool"]](**decision["args"])      # act

        # Observe + reflect: put the result back so the model can adjust
        history.append(f"Called {decision['tool']} -> {result}")
    return "Stopped: step limit reached. A human should review."

Turning a language model into an agent shifts AI from an assistant into something that executes. That power is why the next lessons are about tools, teams of agents, testing and safety.

Coming Up Next

Day 23: Tool use and function calling: connecting LLMs to APIs, databases and shells.

AIAgents #AutonomousAI #EnterpriseAutomation #AgenticWorkflows #SoftwareEngineering

Originally published at https://sureshpallapothu.in/blog/day-22-autonomous-agents, where this post includes animated diagrams.

Day 21: High-Performance Networking — Ingress and Egress for AI Clusters

Suresh Kumar Pallapothu — Wed, 21 Oct 2026 18:13:21 +0000

AI traffic does not look like ordinary web traffic. Requests carry multi-megabyte documents, and replies stream token by token over long-lived connections (Server-Sent Events). Network settings built for short page loads break that pattern. This lesson covers three places to tune: traffic coming in, traffic between services, and traffic going out.

In plain terms: Think of the cluster as a building. The front door must stay open long enough for a slow conversation, the corridors between rooms must not jam, and the exits must be locked except for the few doors you approve.

1. Ingress: keep long streams alive

An ingress controller (NGINX, Traefik or Envoy, for example) sits at the front door. Its defaults are tuned for quick requests and break AI streaming in two ways.

Timeouts. Default proxy read and send timeouts are often around 30 seconds, which can cut a connection while a model works through a large context. Raise them to several minutes for your slowest replies.
Buffering. If the proxy buffers the response, it waits for the entire reply before sending anything, which ruins real-time token streaming. Turn buffering off.

Diagram: If an ingress proxy buffers the response it waits for the whole LLM reply before sending anything to the user, which ruins real-time token streaming. Turning buffering off lets each token through as it is generated, over Server-Sent Events. See the animated version.

Diagram: Default proxy timeouts of about 30 seconds cut off long generations over big contexts. Raising the read and send timeouts to several minutes keeps the connection open until the model finishes. See the animated version.

2. East-west traffic: eBPF and Cilium

Standard Kubernetes networking uses kube-proxy, which builds chains of Linux iptables rules. As services multiply, evaluating those chains becomes a bottleneck for chatty traffic such as embeddings moving between gateways, vector stores and inference engines.

An eBPF-based network plugin such as Cilium makes routing decisions with small programs running inside the Linux kernel and direct lookups, and it can replace kube-proxy. In high-throughput datacenters, specialised network cards and kernel-bypass techniques can push latency lower still. How much you gain depends on your cluster, so measure before and after you switch.

Diagram: kube-proxy builds long chains of iptables rules that get slow to evaluate as services multiply. An eBPF-based network plugin such as Cilium makes routing decisions with kernel programs and direct lookups, which helps heavy service-to-service traffic. Gains depend on the cluster, so measure first. See the animated version.

3. Egress: stop data leaving

AI workloads add a new risk: an autonomous agent or a compromised model runtime may try to send internal context to an outside server. Use a NetworkPolicy with default-deny egress, so AI Pods can reach only the internal endpoints you list, and everything else, including the open internet, is blocked. Policies are only enforced if your network plugin supports them, and Cilium and Calico do.

Diagram: A NetworkPolicy with default-deny egress lets an AI Pod reach only approved destinations such as the vector database, plus cluster DNS, and blocks everything else, including unauthorised internet traffic. It needs a network plugin that enforces policies, such as Cilium or Calico. See the animated version.

Here are the two pieces of configuration. They assume the NGINX ingress controller and a namespace called ai, with a vector database Pod labelled app: qdrant.

# Ingress: long timeouts and no buffering, for streaming replies
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: model-ingress
  annotations:
    nginx.ingress.kubernetes.io/proxy-read-timeout: "300"    # seconds
    nginx.ingress.kubernetes.io/proxy-send-timeout: "300"
    nginx.ingress.kubernetes.io/proxy-buffering: "off"
spec:
  ingressClassName: nginx
  rules:
    - host: llm.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service: { name: model-svc, port: { number: 80 } }
---
# Egress: AI Pods may talk only to the vector DB and cluster DNS
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: ai-default-deny-egress
  namespace: ai
spec:
  podSelector: {}                 # every Pod in the namespace
  policyTypes: [Egress]
  egress:
    - to:
        - podSelector: { matchLabels: { app: qdrant } }
    - to:                          # allow DNS, or nothing will resolve
        - namespaceSelector: {}
          podSelector: { matchLabels: { k8s-app: kube-dns } }
      ports:
        - { protocol: UDP, port: 53 }
        - { protocol: TCP, port: 53 }

Get the allow-list right and your AI infrastructure handles big payloads with low latency, while keeping strict boundaries around what can leave. That wraps up Week 3. Next week moves to enterprise workflows, starting with autonomous agents.

Coming Up Next

Day 22: Autonomous agents: from passive LLMs to goal-driven execution.

KubernetesNetworking #eBPF #Cilium #NetworkSecurity #DevSecOps

Originally published at https://sureshpallapothu.in/blog/day-21-ai-cluster-networking, where this post includes animated diagrams.

Day 20: Persistent Storage in Kubernetes — Managing State, Weights and Vector Indices

Suresh Kumar Pallapothu — Tue, 20 Oct 2026 18:05:45 +0000

Containers are ephemeral. When a Pod crashes or Kubernetes moves it, everything on its local disk is gone. For a web server that is fine. For AI it is costly: vector indices take time to rebuild, and model weights are many gigabytes that you do not want to download again every restart.

In plain terms: A container's disk is a whiteboard that gets wiped whenever the room is reassigned. A persistent volume is a filing cabinet in the corridor that every new room can open.

Diagram: A container's local filesystem is discarded when its Pod crashes or is rescheduled. Vector indices and multi-gigabyte model weights must outlive the Pod, so they need persistent volumes. See the animated version.

1. The storage abstraction

Kubernetes separates the storage from the application with two resources.

PersistentVolume (PV): a piece of real storage in the cluster, such as local NVMe disks, an NFS share or a SAN. It records the size, access modes and backing hardware.
PersistentVolumeClaim (PVC): the application engineer's request for storage. The Pod references the claim, and Kubernetes binds it to a matching volume. Many clusters also have a StorageClass that creates a volume on demand when a claim arrives.

Diagram: A PersistentVolume is a piece of real storage in the cluster. A PersistentVolumeClaim is the application's request for storage. The Pod references the claim, which binds to a matching volume, or a StorageClass can create one on demand. See the animated version.

2. Access modes for AI workloads

The access mode decides how Pods may use the volume.

ReadWriteOnce (RWO): mounted read-write by a single node. This is the usual choice for a stateful vector database such as Milvus or Qdrant, where uncoordinated writers could corrupt the index.
ReadOnlyMany (ROX): mounted read-only by many nodes at once. This suits model weights: keep one copy of, say, 15 GB of quantized weights on a shared, fast network mount, and let many stateless inference Pods attach to it instead of each holding its own copy.

Check that your storage backend actually supports the mode you ask for. Shared read-only mounts typically come from NFS or a similar network filesystem, while plain block disks are usually single-node.

Diagram: ReadWriteOnce mounts a volume read-write on a single node, which suits a vector database that must not have uncoordinated writers. ReadOnlyMany mounts read-only on many nodes at once, so one copy of the model weights can serve many inference Pods. The storage backend has to support the mode you request. See the animated version.

3. StatefulSets for databases

For a vector database, a standard Deployment is usually the wrong tool. A StatefulSet gives each Pod a stable name (db-0, db-1), starts them in order, and binds each Pod to its own persistent volume that follows it across restarts. That keeps replication between the database Pods consistent through cluster restarts.

Diagram: A StatefulSet gives each Pod a stable name such as db-0 and db-1, starts them in order, and binds each Pod to its own persistent volume that follows it across restarts. That is why stateful databases use StatefulSets instead of Deployments. See the animated version.

Here is an example. It assumes your cluster has a default StorageClass, and the Qdrant image and sizes are illustrative.

# 1. Shared, read-only model weights (needs a backend that supports ReadOnlyMany)
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: model-weights
spec:
  accessModes: [ReadOnlyMany]
  resources:
    requests: { storage: 20Gi }
---
# 2. A vector database with its own volume per Pod
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: qdrant
spec:
  serviceName: qdrant
  replicas: 1
  selector:
    matchLabels: { app: qdrant }
  template:
    metadata:
      labels: { app: qdrant }
    spec:
      containers:
        - name: qdrant
          image: qdrant/qdrant:v1.9.0        # pin a version you have tested
          volumeMounts:
            - { name: data, mountPath: /qdrant/storage }
  volumeClaimTemplates:                       # one PVC is created per Pod: data-qdrant-0, ...
    - metadata: { name: data }
      spec:
        accessModes: [ReadWriteOnce]
        resources:
          requests: { storage: 50Gi }

Done well, this eliminates repeated weight downloads, preserves your database indices and protects the system's memory across cluster updates.

Coming Up Next

Day 21: High-performance networking: ingress and egress for AI clusters.

KubernetesStorage #DataEngineering #CloudNative #VectorDatabase #StatefulSet

Originally published at https://sureshpallapothu.in/blog/day-20-kubernetes-storage, where this post includes animated diagrams.

Day 19: Kubernetes for AI 101 — Pods, Deployments and Services for Model Serving

Suresh Kumar Pallapothu — Mon, 19 Oct 2026 17:40:09 +0000

Docker Compose coordinates services on one machine. Enterprise AI needs more: traffic surges, nodes fail, and upgrades must happen with zero downtime. Kubernetes (often written K8s) is the industry-standard control plane for that. Learning it starts with three building blocks.

In plain terms: Compose is a manager running one kitchen. Kubernetes is a head office running a chain of kitchens: it opens a new branch when the queue grows, replaces a branch that burns down, and rolls out a new menu one branch at a time.

Diagram: Docker Compose runs services on one host. Kubernetes is a control plane over many nodes: it reschedules workloads after failures, scales on demand and performs rolling updates without downtime. See the animated version.

1. Pods: the atomic unit

A Pod is the smallest thing Kubernetes schedules. It wraps one or more tightly coupled containers that share a network address and can share storage volumes. An inference Pod usually pairs the model server with a sidecar container that handles logging, telemetry or request sanitising.

Diagram: A Pod is the smallest unit Kubernetes schedules. An inference Pod usually pairs the model server with a sidecar container for logging or telemetry. The containers share one network address and can share volumes. See the animated version.

2. Deployments: updates and self-healing

You rarely create Pods by hand. A Deployment declares what you want, for example "three replicas of this model server image", and a controller keeps reality matching that.

Rolling updates. When you change the model version or a prompt template, the Deployment starts new Pods, waits for their readiness checks to pass, then retires the old ones, so in-flight requests are not dropped.
Self-healing. If a heavy context pushes a container past its memory limit and it is killed (OOMKilled), Kubernetes restarts or replaces it to bring the count back to what you declared.

Diagram: A Deployment keeps the declared number of Pod replicas running. For an update it starts new Pods, waits for their readiness checks, then stops old ones so in-flight requests are not dropped. If a Pod is killed for running out of memory, Kubernetes restarts or replaces it to restore the desired count. See the animated version.

3. Services: one stable address

Pods come and go, and each new Pod gets a new IP. A Service sits in front of them with a stable virtual IP and DNS name.

ClusterIP exposes the model only inside the cluster, so internal applications can reach it while it stays hidden from the public internet.
Readiness-aware routing. The Service only sends traffic to Pods that pass their readiness probe. If a Pod is busy and its readiness check starts failing, new requests go to its idle siblings. Tune the probe to your model server, since a probe that is too strict can take every Pod out of rotation at once.

Diagram: A Service gives a changing set of Pods one stable virtual IP and DNS name. With a ClusterIP the model is reachable only inside the cluster. The Service sends traffic only to Pods that pass their readiness probe, so a Pod that reports itself busy is skipped. See the animated version.

Here is a compact example. It assumes an image called model-api that serves a /health endpoint on port 8000. Adjust the names, resources and probe to your own server.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: model-api
spec:
  replicas: 3
  selector:
    matchLabels: { app: model-api }
  strategy:
    rollingUpdate: { maxUnavailable: 0, maxSurge: 1 }   # never drop capacity during an update
  template:
    metadata:
      labels: { app: model-api }
    spec:
      containers:
        - name: model-api
          image: registry.example.com/model-api:1.4.0
          ports: [{ containerPort: 8000 }]
          resources:
            requests: { cpu: "2", memory: 8Gi }
            limits:   { memory: 8Gi }                   # exceeding this gets the container OOMKilled
          readinessProbe:
            httpGet: { path: /health, port: 8000 }
            periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: model-svc
spec:
  type: ClusterIP            # reachable only inside the cluster
  selector: { app: model-api }
  ports: [{ port: 80, targetPort: 8000 }]

Apply it with kubectl apply -f model.yaml, and inside the cluster your application calls http://model-svc. These three abstractions are the baseline for scaling AI microservices with automated resilience.

Coming Up Next

Day 20: Persistent storage in Kubernetes: managing state, weights and vector indices.

Kubernetes #K8s #MLOps #DevOps #CloudNative

Originally published at https://sureshpallapothu.in/blog/day-19-kubernetes-for-ai, where this post includes animated diagrams.

Day 18: Docker Compose for Multi-Container AI Stacks

Suresh Kumar Pallapothu — Sun, 18 Oct 2026 16:47:19 +0000

A serious AI application is a small distributed system. A RAG pipeline cannot be one monolithic process, because inference, data indexing and the user interface have different needs and change at different speeds. Docker Compose lets you describe all of them in one file and start them together on a single host.

In plain terms: Compose is a seating plan for a dinner party. You say who sits where, who can talk to whom, and who must arrive before the others, and Docker sets the table.

The core topology: a 3-tier RAG stack

Diagram: A self-contained RAG deployment has three isolated services: an application layer that orchestrates, a vector database that holds the embeddings, and a model server such as vLLM or Ollama that generates the answer. Docker Compose starts and connects them on a single host. See the animated version.

Model server (vLLM or Ollama). A dedicated compute container that manages GPU or system memory and exposes inference over an internal port.
Vector database (Qdrant, Milvus or pgvector). An independent, stateful service that holds the embeddings and answers nearest-neighbour queries.
Application layer (FastAPI, plus Streamlit or Next.js). The orchestrator: it takes the user's question, fetches matching context from the vector database, builds the prompt and asks the model server for the answer.

Three configuration practices

1. A private bridge network. Put every container on one user-defined network. Docker then resolves service names automatically, so the app can call http://ollama:11434 or http://qdrant:6333. Publish only the app's port to the host, and the database and model server stay unreachable from outside.

Diagram: Containers on a user-defined bridge network find each other by service name through Docker's built-in DNS, for example http://ollama:11434 or http://qdrant:6333. Only the app's port is published to the host, so the database and model server are not reachable from outside. See the animated version.

2. Volumes for state. Mount separate volumes for the vector database storage and the model cache (/root/.ollama, or /root/.cache/huggingface for Hugging Face models). Containers can then be updated, restarted or rebuilt without losing the index or downloading the weights again.

Diagram: Mounting separate volumes for the vector database storage and the model cache lets you rebuild, restart or upgrade containers without losing the index or downloading the model weights again. See the animated version.

3. Healthchecks and start order. Add a healthcheck to each service and use depends_on with condition: service_healthy, so the app never queries a database or model server that is not listening yet.

Diagram: With a healthcheck on each service and depends_on with condition service_healthy, the application container waits until the database and model server are actually listening, so it never queries a service that is not ready. See the animated version.

Here is a compact example. It assumes your app lives in ./app with a Dockerfile like yesterday's, and that the healthcheck commands suit the image versions you use. Check each image's documentation, since minimal images do not always include the tools a check needs. In production, pin a specific image tag instead of latest.

services:
  app:
    build: ./app
    ports:
      - "8000:8000"            # the only port published to the host
    environment:
      OLLAMA_URL: http://ollama:11434
      QDRANT_URL: http://qdrant:6333
    depends_on:
      ollama:
        condition: service_healthy
      qdrant:
        condition: service_healthy
    networks: [ai-network]

  qdrant:
    image: qdrant/qdrant:latest
    volumes:
      - qdrant_data:/qdrant/storage
    healthcheck:
      test: ["CMD-SHELL", "bash -c ':> /dev/tcp/127.0.0.1/6333' || exit 1"]
      interval: 10s
      retries: 10
    networks: [ai-network]

  ollama:
    image: ollama/ollama:latest
    volumes:
      - ollama_models:/root/.ollama
    healthcheck:
      test: ["CMD", "ollama", "list"]
      interval: 10s
      retries: 10
    networks: [ai-network]
    # For an NVIDIA GPU, add a device reservation:
    # deploy:
    #   resources:
    #     reservations:
    #       devices:
    #         - driver: nvidia
    #           count: all
    #           capabilities: [gpu]

volumes:
  qdrant_data:
  ollama_models:

networks:
  ai-network:

Start it with docker compose up -d, and stop it with docker compose down (add -v only if you want the volumes deleted too). Decoupling the stack this way means any one component can be upgraded, scaled or replaced without destabilizing the rest.

Coming Up Next

Day 19: Kubernetes for AI 101: pods, deployments and services for model serving.

DockerCompose #Microservices #RAG #SystemDesign #DevOps

Originally published at https://sureshpallapothu.in/blog/day-18-docker-compose, where this post includes animated diagrams.

Day 17: Containerizing AI Workloads — Production Dockerfiles for Python APIs

Suresh Kumar Pallapothu — Sat, 17 Oct 2026 16:37:15 +0000

Moving a model from a notebook to a service means putting it in a container. But AI images are not like a Node.js or Go service. Deep learning frameworks bring heavy CUDA shared libraries, C++ build tools and multi-gigabyte wheel files, and a careless Dockerfile can balloon past 10 GB.

In plain terms: Imagine shipping a finished chair. A naive Dockerfile mails the chair together with the whole workshop: saws, sawdust and offcuts. A multi-stage build ships only the chair.

Diagram: A naive Dockerfile that installs everything in one image ends up carrying compilers and build caches along with the large CUDA and PyTorch libraries. A multi-stage build ships only the runtime pieces. The bar widths are illustrative. See the animated version.

The anatomy of a bloated image

A naive Dockerfile that runs RUN pip install torch transformers fastapi in a default base image causes three operational problems.

Diagram: A naive AI image duplicates build caches inside its layers, leaves compilers in production which widens the attack surface, and is so large that new pods start slowly when traffic spikes. See the animated version.

Layer duplication. Build caches and downloaded wheels stay trapped in read-only image layers, which slows registry pushes and pulls.
Security exposure. Leaving gcc, g++ and make in the production image widens the attack surface.
Slow deployments. Huge images saturate the network during autoscaling, so new pods start late exactly when traffic spikes.

Three production patterns

Multi-stage builds. Split the Dockerfile in two. The builder stage has the compilers; it downloads and builds dependencies into a virtual environment at /opt/venv. The runtime stage starts from a clean, minimal base such as python:3.11-slim and copies in only that environment.

Diagram: In stage one, the builder stage has compilers and installs every dependency into a virtual environment. In stage two, a clean slim runtime image receives only that virtual environment, and everything else from stage one is discarded. See the animated version.

Wheel caching. A BuildKit cache mount (--mount=type=cache,target=/root/.cache/pip) keeps pip's download cache outside the image, so repeated CI builds do not download the heavy packages again.

Non-root security. Never run the process as root. Create a dedicated user and start the ASGI server as that user.

Diagram: A production-ready AI image uses a multi-stage build, caches pip downloads with a BuildKit cache mount, runs the server as a non-root user and pins its dependency versions, so builds are reproducible and auditable. See the animated version.

Here is a complete example for a FastAPI model server. It assumes a requirements.txt with pinned versions and an app/ folder with a main.py, and it is a template to adapt rather than a tuned production file. A GPU image would use a CUDA-enabled base and PyTorch build instead.

# syntax=docker/dockerfile:1

# ---- Stage 1: builder (has compilers, gets thrown away) ----
FROM python:3.11-slim AS builder
RUN python -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"
COPY requirements.txt .
RUN --mount=type=cache,target=/root/.cache/pip \
    pip install -r requirements.txt

# ---- Stage 2: runtime (clean and minimal) ----
FROM python:3.11-slim
COPY --from=builder /opt/venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"
RUN useradd -m -u 1000 appuser
WORKDIR /app
COPY --chown=appuser:appuser app/ ./app/
USER appuser
EXPOSE 8000
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

Build it with DOCKER_BUILDKIT=1 docker build -t model-api .. Writing lean, deterministic Dockerfiles turns chaotic research dependencies into reproducible, auditable artifacts that a cluster scheduler can start quickly.

Coming Up Next

Day 18: Docker Compose for multi-container AI stacks.

Docker #Containerization #DevOps #AIEngineering #FastAPI

Originally published at https://sureshpallapothu.in/blog/day-17-dockerfiles, where this post includes animated diagrams.

Day 16: Local Model Execution — Running Open-Weight LLMs Securely with Ollama and vLLM

Suresh Kumar Pallapothu — Fri, 16 Oct 2026 14:10:02 +0000

Relying only on commercial cloud APIs carries real exposure: vendor lock-in, recurring cost for every token, rate limits, and the need to send sensitive intellectual property across public networks. Running an open-weight model such as Llama 3, Mistral or Gemma 2 on your own infrastructure addresses each of those. The price is that you now operate the hardware, so choosing the right runtime matters.

In plain terms: Ordering from a restaurant is easy, but you pay per dish, you queue at busy hours, and the restaurant sees what you ate. Cooking at home means buying the stove, and in return the menu, the speed and the privacy are yours.

Diagram: Relying only on commercial cloud APIs brings vendor lock-in, recurring costs, rate limits and the transmission of sensitive data over public networks. Running open-weight models on your own infrastructure addresses each of these, at the price of operating the hardware yourself. See the animated version.

1. Prototyping and edge: Ollama and llama.cpp

On a workstation, a laptop or an edge box with limited memory (a 16 GB machine is a typical target), a plain Python script adds overhead you do not want. Ollama is built on llama.cpp, a lean C and C++ engine, so it avoids most of that Python overhead.

GGUF quantization: Ollama runs models stored in the GGUF format, usually at 4-bit or 8-bit precision. That shrinks multi-gigabyte weights enough to fit in ordinary RAM or a modest GPU, with a small loss in quality for most tasks. Check the quality on your own prompts.
Operational simplicity: it runs a local service and handles downloading models, spreading them across GPU and CPU memory, and formatting prompts. It exposes an OpenAI-compatible REST API at http://localhost:11434.

2. High-throughput serving: vLLM

When hundreds of users hit the same model at once, handling requests one after another creates a queue. vLLM is a serving engine built for that case, with two key ideas.

PagedAttention. Older engines reserve one contiguous block of KV-cache memory per request, sized for the longest possible reply, so much of it sits empty. The vLLM authors measured 60 to 80% waste in earlier systems. PagedAttention borrows the idea of virtual-memory paging from operating systems and splits the cache into small blocks that are handed out as a request grows.

Diagram: Older inference engines reserve one contiguous block of KV-cache memory per request, sized for the maximum length, and much of it sits empty. vLLM's PagedAttention splits the cache into small blocks allocated as a request grows, like paging in an operating system. The vLLM authors reported 60 to 80% waste in earlier systems. See the animated version.

Continuous batching. A static batch must finish completely before new requests are accepted, so slots sit idle while the longest reply completes. vLLM slips new requests into the running batch at the token level. Reported gains are commonly 2x to 4x higher throughput than basic pipelines, but the real figure depends on your model, hardware and traffic.

Diagram: With static batching, a whole batch must finish before new requests are accepted, so slots sit idle while the longest reply finishes. Continuous batching slips new requests into free slots at the token level, which commonly raises throughput by 2x to 4x depending on the model, hardware and traffic. See the animated version.

Which one should you pick?

Diagram: Ollama, built on llama.cpp, suits prototyping and edge machines with limited memory and runs quantized GGUF models with a single command. vLLM suits GPU servers handling many users at once. Both expose an OpenAI-style API. See the animated version.

Here is how little your application code changes. Both engines speak an OpenAI-style API, so you can point the same client at either one. This assumes you have installed Ollama and pulled a model, or started a vLLM server.

# Ollama: pull and chat with a quantized 8B model in one command
ollama run llama3.1:8b

# vLLM: serve a model with an OpenAI-compatible API on port 8000 (needs an NVIDIA GPU)
vllm serve Qwen/Qwen2.5-7B-Instruct

from openai import OpenAI

# Same client, different base_url. Ollama listens on 11434, vLLM on 8000.
client = OpenAI(base_url="http://localhost:11434/v1", api_key="not-needed")

reply = client.chat.completions.create(
    model="llama3.1:8b",
    messages=[{"role": "user", "content": "Summarise RAG in one sentence."}],
)
print(reply.choices[0].message.content)

Self-hosting turns an LLM from an outside cost center into a private utility inside your own network boundary. The next lessons package it properly, starting with containers.

Coming Up Next

Day 17: Containerizing AI workloads: writing production Dockerfiles for Python APIs.

LocalAI #vLLM #Ollama #OpenSourceAI #DataSovereignty

Originally published at https://sureshpallapothu.in/blog/day-16-local-models, where this post includes animated diagrams.

Day 15: Introduction to AI Infrastructure — Hardware, Runtimes and Compute

Suresh Kumar Pallapothu — Thu, 15 Oct 2026 14:12:30 +0000

Until now we called AI over an API. Moving to self-hosted models is how you guarantee your data never leaves your network, and it also means you now own the hardware, the runtime and the operating system. This week starts with the three building blocks.

In plain terms: Hosting your own model is like running your own kitchen instead of ordering delivery. You get control and privacy, but you now need the right stove, the right pans and a kitchen that stays up.

1. The hardware constraint: VRAM

A model has to fit in the memory of the GPU, called VRAM, before it can run. The parameter count sets the requirement: memory is roughly the number of parameters times the bytes used per parameter. Shrinking the numbers (quantization) is how you fit bigger models on smaller cards.

Diagram: The weights of an 8-billion-parameter model take about 16 GB at 16-bit precision, 8 GB at 8-bit and 4 GB at 4-bit. The KV cache and runtime add roughly 1 to 3 GB, so a 4-bit 8B model needs roughly 6 to 8 GB of VRAM in practice. See the animated version.

The figures are for the weights alone. The KV cache, which holds the conversation so far, and the runtime itself take extra space, so plan for a little headroom. A quantized 8-billion-parameter model needs roughly 6 to 8 GB of VRAM in practice.

2. Inference runtimes

You do not run a raw Python script in production. You use a dedicated, optimized runtime. llama.cpp is a lean C++ runtime that runs quantized models well on laptops and small servers. vLLM is built for GPU servers and serves many requests at once with smart batching and memory management. Same model, far more speed from the same hardware.

Diagram: You do not run raw Python scripts in production. Dedicated runtimes such as llama.cpp and vLLM squeeze much more speed from limited hardware using optimized code, batching and efficient memory use. See the animated version.

3. The hosting OS

Linux is the standard host for production AI. A headless Ubuntu Server, which means no desktop environment at all, is a common choice. It leaves every bit of memory and CPU for your workloads, and it passes GPU resources cleanly into Docker containers and Kubernetes pods.

Diagram: Headless Ubuntu Server has no desktop overhead, passes GPU drivers cleanly to Docker and Kubernetes, and offers long-term support releases that are easy to automate, which is why Linux is the standard host for production AI. See the animated version.

Put the pieces together and you get a stack that is built from the bottom up.

Diagram: A self-hosted AI server is built bottom up: GPU hardware, a headless Ubuntu Server with the GPU driver, the NVIDIA container toolkit that exposes the GPU to containers, Docker or Kubernetes pods, and finally the model running in an optimized runtime. See the animated version.

Once the driver and the container toolkit are in place, you can check the GPU from inside a container. The commands below are the standard checks, and they need an NVIDIA GPU on the host.

# On the host: is the GPU visible to the driver?
nvidia-smi

# Inside a container: is the GPU passed through?
docker run --rm --gpus all ubuntu nvidia-smi

If the second command prints the same GPU table as the first, your containers can use the GPU. The next lessons build on that: local models with Ollama and vLLM, then Dockerfiles and Kubernetes.

Coming Up Next

Day 16: Local model execution: running open-weight LLMs securely with Ollama and vLLM.

AIInfrastructure #Hardware #Ubuntu #Docker #SelfHostedAI

Originally published at https://sureshpallapothu.in/blog/day-15-ai-infrastructure, where this post includes animated diagrams.

Day 14: RAG Evaluation and Hallucination Guardrails

Suresh Kumar Pallapothu — Wed, 14 Oct 2026 14:07:03 +0000

A RAG pipeline is powerful, but it adds new places to fail. The model is only as good as what you hand it, and when it is handed the wrong thing it does not say "I'm confused". It answers confidently.

In plain terms: An open-book exam only works if the student opens the right book. Hand them the wrong chapter and they will write a polished, convincing, wrong answer.

Diagram: If the retrieval step pulls the wrong document, the LLM will still write a fluent answer from it. The result is confident and wrong, so a hallucination can start in retrieval and not in the model. See the animated version.

1. Measure the two halves

You evaluate a RAG system by scoring its two halves separately, because they fail in different ways.

Context relevance asks whether the retrieved documents actually contain the answer to the user's question. This checks the retrieval step.
Faithfulness asks whether the generated response is strictly derived from the retrieved context, or whether the model brought in outside information. This checks the generation step.

Diagram: Context relevance checks whether the retrieved documents actually contain the answer. Faithfulness checks whether the generated answer is strictly derived from the retrieved context instead of invented. See the animated version.

2. Spot an unfaithful answer

Faithfulness is checked claim by claim. If the context says the REIT paid a dividend in Q3 and the answer adds that its yield is 7.4%, that second claim is not in the context. It was invented, and that is a hallucination, even though it sounds perfectly plausible.

Diagram: A faithfulness check compares each claim in the answer with the retrieved context. A claim the context supports passes. A claim that is not in the context, such as an invented yield percentage, is flagged as a hallucination. The example text is illustrative. See the animated version.

3. Guardrails: an LLM as judge

You can automate that check. A secondary, smaller LLM acts as a judge in the pipeline. Before the final answer reaches the user, the judge compares it with the retrieved text and decides whether to show it or block it. When a check fails you can retry retrieval, or return "I don't know". Judges can be wrong too, so sample their decisions and review them.

Diagram: A secondary, smaller LLM acts as a judge. Before the user sees the answer, it compares the answer to the retrieved text and either lets it through or blocks it. Judges can be wrong too, so teams sample and review their decisions. See the animated version.

Here is the idea as a short Python sketch. The llm_judge function stands in for a call to a small model that you prompt to compare an answer against a context.

def guarded_answer(question: str) -> str:
    chunks = search(embed(question), top_k=3)
    answer = generate(question, chunks)

    # Judge: is every claim in the answer supported by the retrieved text?
    verdict = llm_judge(
        "Reply SUPPORTED or UNSUPPORTED. Is every claim in the answer "
        "backed by the context?\n\n"
        f"Context: {chunks}\n\nAnswer: {answer}"
    )
    if verdict.strip().upper().startswith("SUPPORTED"):
        return answer
    return "I could not find a reliable answer in the documents."

Keep a small test set of real questions with known answers, and run it every time you change chunking, the embedding model or the prompt. That is how you catch a regression before your users do.

Coming Up Next

Day 15: Introduction to AI infrastructure: hardware, runtimes and compute.

AIQuality #MLOps #Hallucination #TechLeadership #DataGovernance

Originally published at https://sureshpallapothu.in/blog/day-14-rag-evaluation, where this post includes animated diagrams.

Day 13: Retrieval-Augmented Generation (RAG) — Chatting with Proprietary Documents

Suresh Kumar Pallapothu — Tue, 13 Oct 2026 13:56:20 +0000

A language model only knows what it was trained on, and that stopped at a fixed date. It has never read your internal wiki, your tickets or this morning's incident report. Retrieval-Augmented Generation (RAG) is the architecture that lets it answer questions about your own, up-to-date documents, without retraining the model.

In plain terms: An open-book exam. Instead of making the student memorise the whole library, you let them look up the right pages first, then write the answer from those pages.

Diagram: A language model is frozen at its training date and has never seen your private documents. RAG looks them up when a question arrives and gives them to the model, so no retraining is needed and updated documents change the answers. See the animated version.

RAG chains together the two previous lessons: embeddings give every paragraph a position by meaning, and a vector database finds the closest ones fast. Then the LLM does what it is best at, which is writing a clear answer.

Diagram: RAG has three steps: ingest your documents by chunking, embedding and storing them, retrieve the most relevant paragraphs for each question, then generate an answer from that retrieved context. See the animated version.

Step 1: Ingestion

Ahead of time, your private documents are split into chunks (paragraph-sized pieces), each chunk is converted into an embedding, and the vectors are stored in a vector database. Chunk size is a real design decision. Chunks that are too big blur the meaning, and chunks that are too small lose the surrounding context.

Diagram: Ingestion splits your documents into chunks, converts each chunk into an embedding, and stores the vectors in a vector database. Chunk size is a trade-off: large chunks blur meaning and tiny chunks lose context. See the animated version.

Step 2: Retrieval

When a user asks a question, that question is embedded as well. The vector database runs a similarity search and returns the top few most relevant paragraphs. Three is a common starting point, and you tune it for your documents.

Step 3: Augmented generation

The retrieved paragraphs are injected into the LLM's prompt, together with an instruction: "Answer the user's question using ONLY the following retrieved context." The model writes its answer from the supplied facts instead of from memory.

Diagram: The retrieved paragraphs are injected into the prompt with an instruction to answer using only that context. The model then writes its answer from the supplied facts instead of from memory. The example text is illustrative. See the animated version.

Here is the whole loop in Python. The embed, search and llm functions stand in for your embedding model, your vector database and your LLM API, so you can plug in the ones you use.

def answer(question: str) -> str:
    # 2. Retrieval: embed the question and fetch the 3 closest chunks
    query_vector = embed(question)
    chunks = search(query_vector, top_k=3)

    # 3. Augmented generation: put the chunks in the prompt
    context = "\n".join(f"- {c}" for c in chunks)
    prompt = (
        "Answer the question using ONLY the context below. "
        "If the context does not contain the answer, say you don't know.\n\n"
        f"Context:\n{context}\n\nQuestion: {question}"
    )
    return llm(prompt)

# 1. Ingestion (run ahead of time, and again whenever documents change)
for chunk in split_into_chunks(load_documents()):
    store(embed(chunk), chunk)

Notice the "say you don't know" rule. It gives the model a safe way out when retrieval finds nothing useful. Tomorrow's lesson covers what happens when retrieval goes wrong, and how to measure it.

Coming Up Next

Day 14: RAG evaluation and hallucination guardrails.

RAG #GenerativeAI #EnterpriseAI #SoftwareEngineering #AIImplementation

Originally published at https://sureshpallapothu.in/blog/day-13-rag, where this post includes animated diagrams.

Day 12: Vector Databases — Storing and Searching Enterprise Knowledge

Suresh Kumar Pallapothu — Mon, 12 Oct 2026 14:01:56 +0000

Yesterday you turned text into embeddings. Now you have a new problem: where do millions of those vectors live, and how do you find the closest ones to a question fast?

In plain terms: A normal database is a filing cabinet with labelled folders. You ask for the folder called "open" and it hands it over. A vector database is a librarian who understands what you mean and brings the books that are closest to your idea, even if no label matches.

Diagram: A relational database answers exact-match questions. A vector database answers the question closest to this meaning by ranking stored embeddings by distance, which standard databases are not built to index. See the animated version.

1. The vector store

Standard relational databases such as PostgreSQL or MySQL are built to search for exact matches: WHERE status = 'open'. They have no efficient way to answer "which stored vectors are closest to this one?" That job needs geometry. A vector database such as Milvus, Qdrant or Pinecone is engineered to store and index millions of embeddings and to answer that question efficiently.

2. Approximate nearest neighbour (ANN)

You could compare your question against every stored vector, but at a million documents that is far too slow. Vector databases build an index instead. HNSW (Hierarchical Navigable Small World) is a popular one: it links vectors into a layered graph, so a search starts at a coarse layer and hops closer and closer to the answer. The trade is a little accuracy for a lot of speed, which is why it is called approximate.

Diagram: Comparing a query against every vector is too slow at scale. Approximate nearest neighbour indexes such as HNSW hop through a layered graph to reach close matches in milliseconds, trading a little accuracy for a lot of speed. The comparison counts are illustrative. See the animated version.

3. pgvector: no new system required

If your team already runs PostgreSQL and does not want another database to operate, the pgvector extension adds a vector column type and similarity search to it. Your embeddings sit right next to your normal relational data, so one query can filter by customer and rank by meaning.

Diagram: You can store embeddings in a dedicated vector database such as Milvus, Qdrant or Pinecone, in PostgreSQL with the pgvector extension next to your regular data, or in a local library for small experiments. See the animated version.

Here is a complete pgvector example. The three-number vectors are only there to keep it short; real embeddings have hundreds of dimensions, and the vector(3) size would match your model's output.

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE docs (
  id        serial PRIMARY KEY,
  body      text,
  embedding vector(3)          -- use your model's size, e.g. vector(1536)
);

INSERT INTO docs (body, embedding) VALUES
  ('High latency on the web tier',   '[0.82, -0.11, 0.47]'),
  ('Packets dropped at the switch',  '[0.79, -0.08, 0.51]'),
  ('Quarterly revenue forecast',     '[-0.12, 0.55, 0.20]');

-- An index makes this fast at scale (HNSW with cosine distance):
CREATE INDEX ON docs USING hnsw (embedding vector_cosine_ops);

-- The 2 paragraphs closest in meaning to the question's vector.
-- <=> is cosine distance: smaller means closer.
SELECT body
FROM docs
ORDER BY embedding <=> '[0.80, -0.10, 0.49]'
LIMIT 2;

Put it together and a query looks like this: embed the question, ask the vector database for the nearest vectors, and get the matching paragraphs back.

Diagram: A question is turned into a vector by the embedding model, the vector database finds the nearest stored vectors, and the matching paragraphs come back ranked by closeness. See the animated version.

Coming Up Next

Day 13: Retrieval-Augmented Generation (RAG): chatting with proprietary documents.

VectorDatabase #DataScience #DataArchitecture #BackendEngineering #AI

Originally published at https://sureshpallapothu.in/blog/day-12-vector-databases, where this post includes animated diagrams.

Day 11: Embeddings and Vector Representations — How Machines Map Meaning

Suresh Kumar Pallapothu — Sun, 11 Oct 2026 13:47:50 +0000

To make an AI work with your own company data, you first have to turn that data into something a machine can measure. Words are not measurable. Numbers are. The bridge between the two is the embedding.

In plain terms: Imagine a giant map where every sentence in your company has a pin. Sentences about the same topic are pinned close together, and unrelated ones are far apart. An embedding is the pin's coordinates.

Diagram: An embedding model maps a chunk of text to a list of numbers. Texts about similar things, like network latency and packet drop, get similar numbers. Real embeddings have hundreds or thousands of values, and the ones shown are illustrative. See the animated version.

1. The mathematical map

An embedding model reads a chunk of text and outputs a dense vector: a list of numbers, usually 768 or 1,536 of them. You never read these numbers yourself. What matters is how they relate to each other. The numbers are produced so that text with a similar meaning gets a similar list.

2. Semantic closeness

Because similar meanings get similar numbers, they cluster together when you plot them. The vectors for "network latency" and "packet drop" sit close together. The vector for "financial forecast" sits far away from both. A question you type also becomes a vector, and it lands next to the text that answers it.

Diagram: Embeddings place texts on a map where meaning is distance. Network terms cluster together, finance terms cluster elsewhere, and a question lands near the text that answers it. The map is a flattened, illustrative view of a space with hundreds of dimensions. See the animated version.

3. The power of distance

Once all your documents are vectors, finding the right one becomes arithmetic. The common measure is cosine similarity, which compares the direction of two vectors. A score near 1 means they point the same way, so the texts mean much the same thing. A score near 0 means they are unrelated.

Diagram: Cosine similarity measures the angle between two vectors. Vectors pointing the same way score close to 1, meaning similar meaning, and unrelated ones score near 0. The scores shown are computed from the four-number example vectors, which are illustrative. See the animated version.

This is why embedding search beats keyword search. Ask "Why is the site slow?" and a keyword search finds nothing if no document contains those words. An embedding search finds the paragraph about high latency on the web tier, because the meaning is close.

Diagram: Keyword search needs the same words. Embedding search compares meaning, so a question about a slow site can find a paragraph about high latency on the web tier even though the words differ. See the animated version.

Here is the arithmetic in Python, using the same four-number example vectors from the diagrams. Real embedding models return far longer vectors, and you would call one from a library or an API.

import numpy as np

def cosine(a, b):
    a, b = np.array(a), np.array(b)
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

latency  = [0.82, -0.11, 0.47, 0.05]   # "network latency"
drop     = [0.79, -0.08, 0.51, 0.02]   # "packet drop"
forecast = [-0.12, 0.55, 0.20, 0.78]   # "financial forecast"

print(round(cosine(latency, drop), 2))      # 1.0   -> almost the same meaning
print(round(cosine(latency, forecast), 2))  # -0.03 -> unrelated

The numbers in this example are made up to show the idea. In a real system you embed every document once, store the vectors, and compare each new question against them.

Coming Up Next

Day 12: Vector databases: storing and searching enterprise knowledge.

VectorEmbeddings #MachineLearning #NLP #DataEngineering #AIArchitecture

Originally published at https://sureshpallapothu.in/blog/day-11-embeddings, where this post includes animated diagrams.