Docker with AI: A Practical Guide to Running LLMs, Agents and MCP

Harsh Manvar — Tue, 26 May 2026 05:39:35 +0000

If you've been searching for how to actually use Docker with AI not just spin up a demo but run models, agents and MCP servers in production here's what We have learned over the years and put into our new book.

If you typed "Docker with AI" into Google and landed here, you're in good company. That's the search I've watched explode over the last 18 months and it's also the question I get asked the most at meetups, on LinkedIn DMs.

People aren't asking "what is AI" anymore. They're asking something a lot more uncomfortable :

"I have a model, I have a notebook, I have a demo that worked once on my MacBook. Now what ?"

This post is for that person. I want to share how I think about Docker with AI today what the stack actually looks like in 2026, where most teams get stuck and the structured path @ajeetraina and I wrote down in our new book, Operational AI with Docker (links at the end).

No fluff. No "AI is changing the world." Just the parts I wish someone had handed me two years ago.

Docker Solved Software Packaging. AI Has the Same Problem, Just Heavier.

Think back to 2013. Before Docker, shipping a piece of software meant chasing dependencies, fighting OS differences and praying your requirements.txt lined up with the production box. Then containers showed up and the whole story compressed into three words: build, ship, run.

AI in 2026 looks suspiciously similar except the baggage is heavier :

A model file that's 4 to 70 GB
A tokenizer that needs to match the model exactly
GPU drivers that hate you specifically
Python environments that break the second you breathe on them
An agent that needs tools. Tools that need secrets. Secrets that need policies
A vector store. A retrieval pipeline. A prompt cache.

That's not a notebook problem. That's a packaging, isolation and runtime problem which is exactly what Docker has been solving for a decade in the software world.

So when I say "Docker with AI" I don't mean "let's stick a model inside a FROM python:3.11 and ship it." That's the cargo-cult version. The interesting version is:

How do we use Docker's primitives - images, runtimes, networking, secrets, orchestration to make AI workloads portable, reproducible, and operable ?

That's the whole question. Everything else is implementation detail.

What "Docker with AI" Actually Means in 2026

A lot has changed in the last year and most blog posts haven't caught up. So here's a quick map of the pieces I'm using daily right now.

Docker Model Runner (DMR)

This one trips people up. With Docker Model Runner, you don't put the model inside a container. DMR runs the model natively on the host, uses the GPU directly and exposes it through an OpenAI-compatible endpoint. What you get from Docker is the packaging story docker model pull, versioning and the same workflow you already know from images.

So when someone asks me "should I run my model inside a container?", the honest answer in 2026 is : probably not and you don't have to. Use DMR instead. That alone saves teams weeks of pain.

MCP and Docker MCP Gateway

The Model Context Protocol (MCP) exploded around December 2024. Within a few months, the ecosystem had over 3,000 MCP servers file systems, GitHub, Slack, databases, browsers you name it. If you're building an agent, MCP is how it talks to the outside world.

The problem? Running raw MCP servers is a security nightmare. Each one is a process with tool access, secrets and a giant trust surface.

The MCP Gateway fixes this with policy enforcement, secrets isolation, dynamic tool discovery and audit logs the boring infrastructure stuff that nobody blogs about because it's not flashy but that you absolutely need before you let an agent touch production.

Docker Sandboxes

Agents generate code. Sometimes they hallucinate rm -rf /.

Sometimes they pip-install something they shouldn't. Sandboxes give you a lightweight microVM to execute untrusted, agent-generated code without nuking your host. If you're running anything resembling an autonomous agent.

Agentic Compose and Docker Agents

Once you have models, tools and sandboxes, you need a way to wire them together that isn't 800 lines of glue code. Agentic Compose lets you declare agents, sub-agents and tools in YAML the same mental model you already have for docker-compose.yml just stretched to multi-agent workflows. It's versioned, reviewable and reproducible.

Kubernetes for GenAI

Eventually it leaves your laptop. When it does, you need autoscaling, cost-aware routing (because GPU minutes are expensive), observability that actually understands token usage and graceful failover when an upstream model API goes down. That's where Kubernetes comes back into the picture same patterns you know, with AI-specific twists.
That's the stack. Model Runner at the bottom, MCP and Sandboxes in the middle, Agents and Compose on top, Kubernetes wrapping the whole thing in production.

The Patterns I See on Real Production Calls

Most of my day job is sitting with platform and SRE teams that are trying to take an AI workload from "the data scientist's laptop" to "a service my on-call rotation can survive."

A few patterns repeat so often I now bring them up before the team does:

The model choice happens after the architecture is locked. Someone picks GPT-4 in week one because that's what the prototype used and six months later the bill is 5x the revenue. The chapter on choosing SLM vs MLM vs LLM is there for exactly this so you make the call when it's cheap to change, not after a board meeting.

Everything is one curl away from production. I've lost count of the agent demos I've reviewed where the tool call is a raw HTTP request to a database, with no auth scoping, no rate limiting and a secret pasted into an env var. That's a 2 AM incident waiting to happen. MCP Gateway exists for this exact reason and we spend a real amount of the book on how to put policy in front of your agents before they touch anything sensitive.

Nobody owns the GPU bill. When the workload was a prototype, it ran on someone's laptop for free. The moment it goes to a cluster, GPU costs land on a finance team that has no idea what an A100-hour even means. We walk through cost-aware routing, quantization and when running a smaller model locally is just objectively the right answer.
The observability story is "we'll add it later" And later never comes. Standard APM tools don't understand tokens, prompt caching or model failover. The Kubernetes chapters cover what observability for an LLM service actually needs to look like because if you can't see it, you can't operate it.

If any of these sound uncomfortably familiar, you're the reader we wrote this book for.

What's Inside the Book

Rather than copy the table of contents, here's the practical arc what you can actually do by the time you finish each part:

Run an LLM locally with Docker Model Runner, pull it like a container image, hit an OpenAI-compatible endpoint, swap models without changing a line of client code.

Build an AI agent with MCP, give it tools through the MCP Gateway, enforce which tools it can call, isolate its secrets, get an audit trail of every action.

Orchestrate a multi-agent system declaratively with Agentic Compose an orchestrator agent, a few specialist sub-agents, shared state and a clean way to version the whole topology.

Run agent-generated code safely inside Docker Sandboxes so a hallucinated shell command doesn't take down your laptop or your prod box.

Deploy on Kubernetes with autoscaling, cost-aware routing across multiple model backends and observability that actually measures the things that matter tokens, latency, error rates, $/request.

Every chapter is built on tools you can install today. Every example has working code in the companion repo, which we'll keep maintaining as the stack evolves.

Who I Wrote This For
Honestly, the same people who keep asking me these questions:

Developers who built an AI demo and now have to productionize it.
DevOps and platform engineers suddenly responsible for LLM workloads they didn't sign up for.
SREs trying to write runbooks for systems that hallucinate.
Architects sketching out an agentic AI roadmap and looking for a real operational reference.

You don't need an ML PhD. You don't need to have trained a model from scratch. If you're comfortable with containers and curious about how AI actually runs in real environments, you're the audience.
And if you come from the AI side and containers feel like a black box
the first few chapters will get you up to speed without talking down to you.

A Quick Word on Why It's Not Another Tutorial Series

Both @ajeetraina and I write a lot of blogs. He runs Collabnix; I've been publishing Docker and Kubernetes content for years. We could have just kept doing that.

But here's what we kept running into: tutorials age fast in this space. A post from 8 months ago is already half-wrong. People were reading 12 different tutorials, getting 12 contradictory answers and ending up more confused than when they started.

A book lets us tell a single, internally consistent story end to end. Pick the right model → run it locally → wrap it in an agent → secure the tools → sandbox the execution → ship to a cluster.

One narrative. One stack. One opinionated path that actually works.
It's a snapshot of where the practice stands in 2026 and a foundation you can build on as it keeps moving.

Get the Book

If any of this resonated, you can grab a copy here :

📘 Packt (Global): Operational AI with Docker
📦 Amazon (US): Paperback + Kindle
📦 Amazon (India): Paperback + Kindle
🔖 ISBN: 9781807301095

If you do pick it up, tag me on LinkedIn or X (@manvar_harsh) I read every message and I'd love to hear what's working for you and where you're still stuck.

Closing Thought

Two years ago, "Docker with AI" meant pulling a tensorflow:latest image and hoping for the best. Today it means a real, layered runtime Model Runner, MCP, Sandboxes, Agentic Compose, Kubernetes that lets you move from a working demo to a system you'd actually let your customers depend on.

That shift is what the book is about.

If you've been stuck somewhere on that path, I hope this helps you take the next step.
— Harsh

Free Geo location & IP API

Harsh Manvar — Fri, 04 Nov 2022 04:30:12 +0000

Overview

SaaS products are real game changers in today's cloud-based world, even if they might not be immediately evident. Cloud application development simplification trend and raised new phenomena software as a service.

Consider using Geolocation API to provide relevant goods, services, news, and notifications if your online business depends on the user's location.

There are two ways you can incorporate into your application.

Geolocation database: This is an IP database that you buy, host on your server, and use as a reference in your application. The problem with using the IP database is that it needs to be updated frequently in order for you to have the most recent information on IP location.

Geo location API: You don't have to worry about regularly maintaining and updating the database with the help of the geolocation API. You utilise the API within your application to get the pertinent IP data. You only pay for what you really use.

I'd like to offer one of the Cheap Geolocation APIs I found.

I liked it since they provide the 5k Free Geolocation request API access in the Free tier.

Response data have many different fields including Postal code, Country details, Time zone, ASN, Calling code etc.

The FindmeIP.com API has a good average response time of 25–30ms. Their free plan includes 5000 monthly queries with all modules like location, country, and ASN.

They offer 3 types of pricing models, you can choose the best one as per your requirement.

Free
5k requests/month - 30 Request / min

Basic $3.99/month
500k requests/month — 240 request / min

Pro $9.99/month
2M requests/month — No rate limit

Enterprise $14.99/month
5M requests/month — No rate limit

Check out documentation at : https://findmeip.com/documentation

DEV Community: Harsh Manvar

Docker with AI: A Practical Guide to Running LLMs, Agents and MCP

Free Geo location & IP API

Overview