DEV Community: Karan Verma

[Boost]

Karan Verma — Mon, 08 Sep 2025 11:31:53 +0000

Karan Verma for Docker

May 3 '25

From Beginner to Pro: Docker + Terraform for Scalable AI Agents

#docker #terraform #aideployment #mlops

Comments

4 min read

From Zero to Kubernetes: A Beginner's Guide to Orchestrating Docker Containers

Karan Verma — Sat, 31 May 2025 12:30:28 +0000

Introduction

If you've ever built or deployed applications using Docker, you've likely hit a point where running containers on your laptop just isn’t enough. You need scaling, automation, recovery, and networking across machines. Enter Kubernetes, the container orchestrator trusted by startups and tech giants alike. In this beginner-friendly guide, we’ll walk you through what Kubernetes is, why it matters, and how Docker developers can start leveraging its power.

What is Kubernetes?

Kubernetes (also called K8s) is an open-source platform that automates deploying, scaling, and managing containerized applications. While Docker helps package your app into a container, Kubernetes helps run and scale it across many machines.

Kubernetes architecture explained: The Control Plane manages the cluster while Nodes run Pods, which host your Docker containers.

Why Use Kubernetes?

- Self-Healing: Restarts failed containers automatically.
- Scalability: Scale apps up or down automatically with a single command.
- Declarative Management: Define your infrastructure and app needs using YAML files.
- Portability: Run anywhere from your laptop with Minikube to cloud providers like AWS, GCP, or Azure.

How Kubernetes Works (for Docker Devs)

Kubernetes works on a cluster model. A cluster has:

- Master Node (Control Plane): Handles scheduling, scaling, and communication.
- Worker Nodes: Run your Docker containers inside Pods.

Pods and Deployments

A Pod is the smallest deployable unit in Kubernetes. It wraps your container(s) and runs on a node. You usually don’t run Pods directly, you use Deployments to manage them.

Exposing Your App with Services

Pods can come and go. You need a stable way to expose them; that’s where Services come in. A Service routes traffic to the right Pods and load-balances across them.

Kubernetes Service: Traffic from users is routed through a Service to reach the right Pods, ensuring balanced and reliable access to your app.

Step-by-Step: Try It Yourself with Minikube

Let’s get hands-on!

1. Install Minikube & kubectl

brew install minikube
minikube start
kubectl get nodes

2. Create a Deployment YAML

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-web-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80

3. Deploy it to Kubernetes

kubectl apply -f deployment.yaml
kubectl get pods

4. Expose Your Deployment as a Service

kubectl expose deployment my-web-app --type=NodePort --port=80

5. Access Your App

minikube service my-web-app

Bonus: Access a Pod Directly (Port Forwarding)

kubectl port-forward pod/my-web-app-xxxx 8080:80

📚 Further Reading

Here are some trusted, beginner-friendly resources to deepen your Kubernetes knowledge, especially curated for developers coming from Docker:

Kubernetes Official Documentation: The canonical source for Kubernetes knowledge, straight from the maintainers.
Docker + Kubernetes (Docker Docs): Docker’s own guide on moving from Docker CLI to Kubernetes orchestration.
Minikube Official Docs: Run Kubernetes locally in minutes, perfect for testing and dev environments.
kubectl Cheat Sheet: Bookmark this as your go-to for common Kubernetes CLI commands.
Build and Deploy Your First Image on DigitalOcean Kubernetes: A hands-on tutorial that ties together Docker image creation and Kubernetes deployment.
Kubernetes for Beginners (YouTube - TechWorld with Nana): A visual, practical walkthrough of key Kubernetes concepts is great for Docker users.

Conclusion

Kubernetes might seem complex at first, but if you’re already familiar with Docker, you’re well on your way to mastering it. In this guide, you took important first steps by deploying your app, scaling it, and exposing it with a service, all using tools on your own machine. With a bit of practice and curiosity, you’ll soon unlock the full power of Kubernetes to manage containers at scale, whether locally or in the cloud.

Keep experimenting, and enjoy the journey from zero to Kubernetes pro!🚀

Docker MCP Catalog & Toolkit: Building Smarter AI Agents with Ease

Karan Verma — Tue, 20 May 2025 11:43:38 +0000

Introduction: What Is Docker MCP and Why It Matters

The rise of agent-based AI applications, powered by ChatGPT, Claude, and custom LLMs, has created a demand for modular, secure, and standardized integrations with real-world tools. Docker’s Model Context Protocol (MCP), along with its Catalog and Toolkit, addresses this need.

Docker is positioning itself not just as a container platform but as the infrastructure backbone for intelligent agents. In this post, we’ll explore the MCP architecture, Catalog, and Toolkit, and demonstrate how to build your own MCP server.

Section 1: Understanding MCP: The Model Context Protocol

What it is:

MCP is an open protocol that allows AI clients (like agents) to call real-world services securely and predictably.
It's designed for tool interoperability, secure credential management (handling API keys and tokens), and container-based execution.

Why it matters:

Without standards like MCP, agents rely on brittle APIs or unsafe plugins.
Docker provides a secure, isolated runtime to host these services in containers.

Visual overview:

How an AI client communicates with containerized services via MCP

Section 2: MCP Catalog: Prebuilt, Secure MCP Servers

What it includes:

A growing library of 100+ Docker-verified MCP servers, including:

Stripe
LangChain
Elastic
Pinecone
Hugging Face

Key features:

Each MCP server runs inside a container and includes:

OpenAPI spec
Secure default config
Docker Desktop integration

Why developers care:

Plug-and-play tools for AI agents.
Consistent dev experience across services.

Visual overview:

MCP Catalog integration with Docker Desktop

Section 3: MCP Toolkit: Build Your Own Secure MCP Server

Toolkit CLI Features:

mcp init → Scaffolds new MCP server
mcp run → Runs local dev version
mcp deploy → Deploy to Docker Desktop

Security features:

Container isolation
OAuth support for credentials
Optional rate limiting and tracing

Demo walkthrough:

npm install -g @docker/mcp-toolkit
mcp init my-weather-api
cd my-weather-api
mcp run

Visual walkthrough:

MCP Toolkit Workflow: From CLI to Container

Section 4: Connecting MCP Servers to AI Clients

Supported clients:

Claude (Anthropic)
GPT Agents (OpenAI)
Docker AI (beta)
VS Code Extensions

How it works:

Agents call /invoke endpoint defined in MCP spec.
Secure token exchange handles identity.
Response returned to model for reasoning/action.

Use case example:

Claude uses a Docker MCP server to call a Stripe payment processing container during an e-commerce interaction.

Visual flow:

Shows how Claude securely calls a Stripe service via Docker MCP.

Section 5: Best Practices for MCP Server Developers

Security:

Never use root containers
Use docker scan and trivy for image vulnerability scanning
Store secrets with Docker's secret manager (or Vault)

Performance:

Keep containers lightweight (use Alpine or Distroless)
Use streaming responses for LLM interaction

Testing tips:

Use Postman + curl to test /invoke endpoint
Lint OpenAPI specs with swagger-cli

Section 6: The Future of MCP: What Comes Next?

Predictions:

Docker AI Dashboard integration
MCP orchestration (multiple services per agent)
AI-native DevOps (agents building infra with MCP servers)

Opportunities for devs:

Contribute to open MCP servers
Submit to Docker Catalog
Build agent tools for internal or public use

Closing Thoughts

Docker’s MCP Catalog and Toolkit are still in beta, but the path forward is clear: AI apps need real-world tool access, and Docker is building a secure, open ecosystem to power it.

Whether you’re building agent frameworks or just experimenting with tool-using LLMs, now’s the perfect time to get involved.

Got ideas for MCP servers you want to see? Or thinking about contributing your own? I’d love to hear from you! 😊

[Boost]

Karan Verma — Sun, 18 May 2025 10:46:20 +0000

Karan Verma for Docker

May 3 '25

From Beginner to Pro: Docker + Terraform for Scalable AI Agents

#docker #terraform #aideployment #mlops

Comments

4 min read

[Boost]

Karan Verma — Sun, 18 May 2025 08:15:29 +0000

Karan Verma for Docker

Apr 4 '25

Optimizing Docker Image Builds for Speed & Efficiency

#docker #devops #softwareoptimization #containerization

Comments

2 min read

[Boost]

Karan Verma — Sat, 17 May 2025 04:38:29 +0000

Karan Verma for Docker

May 3 '25

From Zero to GenAI Cluster: Scalable Local LLMs with Docker, Kubernetes, and GPU Scheduling

#genai #docker #kubernetes #llm

Comments

4 min read

From Beginner to Pro: Docker + Terraform for Scalable AI Agents

Karan Verma — Sat, 03 May 2025 10:49:32 +0000

Introduction

As AI and machine learning workloads grow more complex, developers and DevOps engineers are looking for reliable, reproducible, and scalable ways to deploy them. While tools like Docker and Terraform are widely known, many developers haven’t yet fully unlocked their combined potential, especially when it comes to deploying AI agents or LLMs across cloud or hybrid environments.

This guide walks you through the journey from Docker and Terraform basics to building scalable infrastructure for modern AI/ML systems.

Whether you’re a beginner trying to get your first container up and running or an expert deploying multi-agent LLM setups with GPU-backed infrastructure, this article is for you.

Docker 101: Containerizing Your First AI Model

Let’s start with Docker. Containers make it easier to package and ship your applications. Here’s a quick example of containerizing a PyTorch-based inference model.

Dockerfile:

FROM python:3.9-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
CMD ["python", "inference.py"]

Build & Run:

docker build -t ai-agent .
docker run -p 5000:5000 ai-agent

You now have a reproducible and portable AI model running in a container!

Terraform 101: Your Infrastructure as Code

Now let’s set up the infrastructure to run this container in the cloud using Terraform.

Basic Terraform Script:

provider "aws" {
  region = "us-east-1"
}

resource "aws_instance" "agent" {
  ami           = "ami-0abcdef1234567890"  # Choose a GPU-compatible AMI
  instance_type = "g4dn.xlarge"

  provisioner "remote-exec" {
    inline = [
      "sudo docker run -d -p 5000:5000 ai-agent"
    ]
  }
}

Deploy:

terraform init
terraform apply

Boom your container is live on an EC2 instance!

Integrating Docker + Terraform: Scalable AI Agent Setup

Now, we combine both tools to:

Auto-provision compute with Terraform
Pull and run your Docker images automatically
Scale agents dynamically by changing Terraform variables

Example:

variable "agent_count" {
  default = 3
}

resource "aws_instance" "agent" {
  count         = var.agent_count
  ami           = "ami-0abc123456"
  instance_type = "g4dn.xlarge"
  ...
}

This lets you spin up multiple Dockerized AI agents across your cloud fleet—perfect for inference APIs or retrieval-augmented generation (RAG) systems.

Advanced Use Case: AI Agents with Multi-GPU, CI/CD & Terraform

Imagine this setup:

Each agent runs an OpenAI-compatible LLM locally (e.g., Mistral, Ollama, LLaMA.cpp)
Terraform provisions GPU instances and networking
Docker builds include prompt routers and memory systems
GitHub Actions auto-triggers Terraform for deployments

Benefits:

Reproducibility across dev, staging, and prod
Cost savings via spot instances
Seamless rollback via Terraform state

This is modern MLOps, containerized.

☁️ Hybrid Multi-Cloud AI with Docker + Terraform

You can even expand this setup to support:

Azure or GCP compute targets
Multi-region failover
Local LLM agents in Docker Swarm clusters (home lab, edge)

Pro Tip: Use Terraform Cloud or Atlantis for remote state and team workflows.

Visual Overview: How Docker and Terraform Work Together to Deploy AI Agents

This diagram maps the full lifecycle from writing infrastructure-as-code, containerizing models, and deploying everything automatically.

Simulated Real-World Project: Structure, README & CLI

This structure outlines a robust setup designed for deploying and testing Docker + Terraform AI agents in hybrid cloud environments. It’s a scalable, reliable framework that can be leveraged for complex AI deployments.

📁 Project Structure

.
├── Dockerfile
├── terraform/
│   ├── main.tf
│   ├── variables.tf
│   └── outputs.tf
├── cloud-init/
│   └── init.sh
├── ai-model/
│   ├── inference.py
│   └── requirements.txt
└── README.md

Sample README.md (Private/Internal Repo Summary)

Title: Scalable AI Agent Deployment with Docker & Terraform

This project sets up a fully Dockerized AI inference agent that is deployed via Terraform on GPU-enabled EC2 instances. It demonstrates:

Docker container for model inference (PyTorch/Transformers)
Terraform to provision compute infra + networking
Cloud-init for auto-starting containers post-launch
Multi-agent scaling logic with variable interpolation

Basic Usage:

terraform init terraform apply

Run Docker Locally:

docker build -t ai-agent .
docker run -p 5000:5000 ai-agent

CLI Output Snapshot

Terraform:

> terraform apply

Apply complete! Resources:
 - aws_instance.agent[0]
 - aws_security_group.main

Public IP: 34.201.12.77

Docker:

> docker ps

CONTAINER ID   IMAGE       COMMAND                STATUS       PORTS
ae34c2f1c11b   ai-agent    "python inference.py"  Up 2 mins    5000/tcp

⚙️ Note: This setup has been tested with both local GPUs and AWS EC2 g4dn instances. The Docker + Terraform pipeline helped me cut down deployment effort by over 60% and simplified environment consistency across dev and test runs.

Simulated Real-World Project: Structure, README & CLI

For more information on Docker, you can refer to the official Docker documentation and explore relevant open-source projects on Docker's GitHub. Additionally, for Terraform-related resources, check out the official Terraform documentation and Terraform GitHub.

Final Takeaways

✅ Docker simplifies packaging AI/ML models
✅ Terraform provisions scalable infrastructure in minutes
✅ Together, they form a powerful pattern for reliable AI deployment

Whether you’re running LLMs locally, deploying agents in the cloud, or scaling across multi-cloud environments, this stack is your launchpad.

👋 Call to Action

If this guide helped you, share it with your team or community!

Thanks for reading. Happy hacking and may your containers always build clean! 🚀

[Boost]

Karan Verma — Sat, 03 May 2025 08:17:01 +0000

Karan Verma for Docker

Mar 14 '25

What If Your Containers Managed Themselves? Meet Docker GenAI Gordon

#docker #gordon #ai #devops

Comments

4 min read

From Zero to GenAI Cluster: Scalable Local LLMs with Docker, Kubernetes, and GPU Scheduling

Karan Verma — Sat, 03 May 2025 08:11:44 +0000

A practical guide to deploying fast, private, and production-ready large language models with vLLM, Ollama, and Kubernetes-native orchestration. Build your own scalable GenAI cluster with Docker, Kubernetes, and GPU scheduling for a fully private, production-ready LLM setup.

Prerequisites

Before we begin, ensure your system meets the following requirements:

A Kubernetes cluster with GPU-enabled nodes (e.g., via GKE, AKS, or bare-metal)
The NVIDIA device plugin installed on the cluster
Helm CLI installed and configured
Docker CLI and access to a GPU-compatible runtime (e.g., nvidia-docker2)

Introduction

Local LLMs are no longer a research luxury, they're a production need. But deploying them at scale, with GPU access, container orchestration, and real-time monitoring? That’s still murky territory for many.

In this article, I’ll walk you through how I built a fully operational GenAI cluster using Docker, Kubernetes, and GPU scheduling. It serves powerful language models like vLLM, Ollama, or HuggingFace TGI. We’ll make it observable with Prometheus and Grafana, and ready to scale when the real load hits.

This isn’t just another tutorial. It’s a battle-tested, experience-backed blueprint for real-world AI infrastructure, written for developers and DevOps engineers pushing the boundaries of what GenAI can do.

Why Local/Private LLMs Matter

Many teams today are realizing that hosted APIs like OpenAI and Anthropic, while convenient, come with serious trade-offs:

Cost grows fast when usage scales
Sensitive data can't always be sent to third-party clouds
Customization is limited to what the API provider allows
Latency becomes a bottleneck in low-connectivity environments

Self-hosting LLMs means freedom, control, and flexibility. But only if you know how to do it right.

What We'll Build

We’ll deploy a production-grade Kubernetes cluster featuring:

vLLM / Ollama / TGI model server containers
GPU scheduling and node affinity
Ingress with HTTPS via NGINX
Autoscaling using HPA or KEDA
Prometheus + Grafana for real-time insights
Declarative infrastructure using Helm or plain YAML

Architecture Overview

Figure: High-level architecture of a scalable GenAI Cluster using Docker, Kubernetes, and GPU scheduling.

This modular, observable cluster gives you full control over your LLM infrastructure, without vendor lock-in.

Step 1: Dockerizing the Model Server

Let’s start small: a single Docker container that wraps a model server like vLLM.

# Dockerfile.vllm
FROM nvidia/cuda:12.2.0-base-ubuntu20.04

RUN apt update && apt install -y git python3 python3-pip

RUN pip install vllm torch transformers

WORKDIR /app
COPY start.sh ./
CMD ["bash", "start.sh"]

start.sh:

#!/bin/bash
python3 -m vllm.entrypoints.openai.api_server --model facebook/opt-1.3b --port 8000

Then, build your container:

docker build -f Dockerfile.vllm -t vllm-server:v0.1 .

You can also use Ollama if you prefer pre-packaged models and a lower barrier to entry. vLLM is recommended for higher throughput and OpenAI-compatible APIs.

This is your first step toward building a modular, GPU-ready inference system.

Step 2: Kubernetes Deployment with GPU Scheduling

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm
  template:
    metadata:
      labels:
        app: vllm
    spec:
      containers:
        - name: vllm
          image: vllm-server:v0.1
          resources:
            limits:
              nvidia.com/gpu: 1
          ports:
            - containerPort: 8000
      nodeSelector:
        kubernetes.io/role: gpu
      tolerations:
        - key: "nvidia.com/gpu"
          operator: "Exists"
          effect: "NoSchedule"

And here’s the corresponding Service definition:

apiVersion: v1
kind: Service
metadata:
  name: vllm-service
spec:
  selector:
    app: vllm
  ports:
    - protocol: TCP
      port: 8000
      targetPort: 8000

This exposes your model server inside the cluster.

Step 3: Ingress and Load Balancing

Install NGINX Ingress Controller:

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm install nginx ingress-nginx/ingress-nginx

Then configure ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: vllm-ingress
spec:
  rules:
    - host: vllm.local
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: vllm-service
              port:
                number: 8000

Update your DNS or /etc/hosts to route vllm.local to your cluster.

Step 4: Autoscaling with KEDA (Optional)

helm repo add kedacore https://kedacore.github.io/charts
helm install keda kedacore/keda

With KEDA, you can scale your LLM pods based on GPU utilization, HTTP traffic, or even Kafka topic lag.

Step 5: Monitoring with Prometheus + Grafana

Install full-stack observability:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install monitoring prometheus-community/kube-prometheus-stack

Expose a /metrics endpoint from your container.

from prometheus_client import start_http_server, Summary
import time

REQUEST_TIME = Summary('request_processing_seconds', 'Time spent processing request')

@REQUEST_TIME.time()
def process_request():
    time.sleep(1)

if __name__ == '__main__':
    start_http_server(8001)
    while True:
        process_request()

Or use GPU exporters like dcgm-exporter. Grafana will pull all this into beautiful dashboards.

Step 6: Optional Components

Vector DB: Qdrant, Weaviate, or Chroma
Auth Gateway: Add OAuth2 Proxy or Istio
LangServe or FastAPI: Wrap your model with an API server or LangChain interface
Persistent Volumes / Object Store: Save fine-tuned models using PVCs or MinIO

Final Thoughts

This isn’t just code. It’s the story of how I learned to stitch together powerful AI infrastructure from open-source tools and make it reliable enough for real-world teams to trust.

Docker gave me modularity. Kubernetes gave me orchestration. GPUs gave me the muscle.

Put together, they gave me something every AI builder wants: freedom.

If you're tired of vendor lock-in and ready to roll up your sleeves, this cluster is your launchpad.

This is just the beginning. Start building your GenAI infrastructure today and take control of your AI stack. Share your progress, contribute to the community, and let’s push the boundaries of what’s possible together.

See you at the edge! 🌍