DEV Community: Raveendiran RR

🚀Empowering Developers with Docker Model Runner: Run AI inference Models Locally with Enhanced Privacy and GPU Acceleration

Raveendiran RR — Thu, 03 Apr 2025 18:30:48 +0000

Hey there, tech enthusiasts! 👋

If you’ve ever thought:

“Wouldn’t it be cool if I could just run an AI model locally with zero setup pain?”

Well, let me introduce you to something magical: Docker Model Runner.

This tool is about to become your best friend — whether you’re a developer building ML apps, a DevOps engineer managing workflows, or a leader figuring out how to scale AI integration in your org.

Ready to roll? Let’s go!

🧠 What is Docker Model Runner?

In plain terms, Docker Model Runner lets you run open models like Llama, Mistral, or Gemma or even deepseek locally on your machine using Docker Desktop — without worrying about dependencies, GPU setup, or cloud costs.(Do keep in mind that local GPU can boost the performance)

It’s like giving your laptop a magic AI engine that works out of the box.

🔧 Why Should You Care? (Even if You’re Not a Dev)

Role What You Gain
Developer Run and test models locally in minutes
DevOps Integrate AI model runs into CI/CD pipelines
Manager Understand how teams can innovate faster, safely
Data Science Try models without wrangling Python environments
Product Lead Explore AI integration early in product lifecycle

✅ Prerequisites

• 🐳 Docker Desktop (v4.27 or later)
• 💻 macOS | Windows  | Linux (chipset :Apple Silicon or Intel)
• 🧠 Some curiosity about how AI models can power your tools
• Optional: An OpenAI API key or similar if you plan to do tool calling

🔥 Step-by-Step: Setting Up Docker Model Runner

Let’s do this.

1. Enable Model Runner in Docker Desktop

1.  Open Docker Desktop
2.  Navigate to Settings > Experimental Features
3.  Toggle ON “Model Runner”

2. Pull a Model

Docker makes this ridiculously easy. Open a terminal and run:

(refer [https://hub.docker.com/u/ai])

docker model pull <model name>

Check if the model has been downloaded

docker model list

3. Run the Model Locally

You’ll see it spin up a containerized AI model, ready to answer questions when you run the below command.However, this layer is abstracted by Docker

docker model run <model name>

AI models from Docker’s ai namespace:

🧠 Docker AI Models: At-a-Glance Comparison

Here’s a snapshot of the top models available via Docker's ai namespace, perfect for local GenAI experiments or production-grade setups.

Model Name	Description	Provider	Parameters	Quantization	Context Window	Key Features
`ai/llama3.1`	Meta's LLama 3.1: Chat-focused, benchmark-strong, multilingual-ready	Meta	8B, 70B	Q4_K_M, F16	128K	- Multilingual (EN, DE, FR, IT, PT, HI, ES, TH) - Text/code generation - Chat assistant - NLG - Synthetic data generation
`ai/llama3.3`	Newest LLama 3 release with improved reasoning and generation quality	Meta	N/A	N/A	N/A	- Improved reasoning - Better generation quality - Latest LLaMA release
`ai/smollm2`	Tiny LLM built for speed, edge devices, and local development	N/A	N/A	N/A	N/A	- Optimized for edge - Speed-focused - Local dev - Low resource footprint
`ai/mxbai-embed-large`	Text embedding model	N/A	N/A	N/A	N/A	- Text embedding - Large parameter size
`ai/qwen2.5`	Versatile Qwen update with better language skills	Qwen	N/A	N/A	N/A	- Improved language abilities - Versatile usage - Broader application support
`ai/phi4`	Microsoft’s compact model with strong reasoning and coding	Microsoft	N/A	N/A	N/A	- Compact - Strong reasoning - Code generation
`ai/mistral`	Efficient open model with top-tier performance	Mistral AI	N/A	N/A	N/A	- Fast inference - Top performance - Open model
`ai/mistral-nemo`	Mistral tuned with NVIDIA NeMo for enterprise	Mistral AI	N/A	N/A	N/A	- NVIDIA NeMo-optimized - Enterprise-grade - Smooth ops
`ai/gemma3`	Google’s small but powerful model for chat and gen	Google	N/A	N/A	N/A	- Compact yet strong - Chat-friendly - High-gen capabilities
`ai/qwq`	Experimental Qwen variant	Qwen	N/A	N/A	N/A	- Experimental - Lightweight - Fast
`ai/llama3.2`	Stable LLama 3 update for chat, Q&A, and coding	Meta	N/A	N/A	N/A	- Coding-friendly - Chat capable - Reliable Q&A
`ai/deepseek-r1-distill-llama`	Distilled LLaMA by DeepSeek for real-world tasks	DeepSeek	N/A	N/A	N/A	- Distilled version - Fast execution - Real-world optimization

⚠️ Note: Many models don’t list detailed specs (params, quant, etc.) publicly. Visit the Docker AI catalog and individual repos for the latest info.

🔁 Integration in Your Dev Lifecycle

Here’s where it gets interesting for teams and orgs.

👷 For Devs

• Add model runner commands in makefiles, test scripts, or runbooks.
• Prototype AI features before wiring them into your full app.

🔄 For CI/CD

• Spin up models in a container during testing.
• Validate AI model outputs in pull requests.

💼 For Management

• Encourage safe local testing without extra infra cost.
• Help teams build trust in GenAI adoption with repeatable environments.

🤔 Wait, Can This Replace the Cloud?

Not entirely. But it’s great for:

✅ Prototyping

✅ Demos

✅ Offline dev

✅ Local evaluation

✅ Privacy-sensitive tasks

You’ll still use the cloud for production workloads — but Model Runner is an amazing stepping stone.

🧪 Real Use Case Example

Imagine you’re building a customer support assistant. You could:
1. Run smollm2 locally via Docker
2. Feed it user queries
3. Use tool calling to fetch FAQs from your API
4. Iterate without pushing a line of code to prod

Dev speed just leveled up. 🚀

🗣️ Wrapping Up

Docker Model Runner is a game changer — not just for devs, but for anyone exploring GenAI.

It’s fast.
It’s local.
It’s powerful.
And best of all… it just works.

So go ahead — pull a model, ask it something, and blow your own mind.

⚠️ Finally .. a pinch of salt while using Docker Model Runner

Issue	Description	Workaround
No safeguard for oversized models	Docker Model Runner doesn’t prevent running models too large for your system, which can cause severe slowdowns or make the system unresponsive.	Make sure your machine has enough RAM/GPU before running large models.
`model run` drops into chat if pull fails	If a model pull fails (e.g., due to network/disk space), `docker model run` still enters chat mode, though the model isn't loaded, leading to confusion.	Manually retry `docker model pull` to confirm successful download before running.
No digest support in Model CLI	The CLI lacks reliable support for referencing models by digest.	Use model names (e.g., `mistralai/mistral-7b-instruct`) instead of digests for now.
Misleading pull progress after failure	If an initial `docker model pull` fails, a retry might misleadingly show "0 bytes downloaded" even though data is loading.	Wait—despite incorrect progress, the pull usually completes successfully in the background.

👋 Bonus: Run the Hello GenAI App Locally (In Under 5 Minutes)

If you’ve come this far, you’re probably itching to try a real-world app using Docker Model Runner. Good news: Docker has an awesome example project called hello-genai — and it’s the easiest way to see AI in action locally.

Here’s how to set it up:

🧰 Prerequisites

• Docker Desktop with Model Runner enabled ✅
• Git installed (or download the ZIP manually)
• Terminal access

🪜 Step-by-Step Setup

1. Clone the Repo

git clone https://github.com/docker/hello-genai.git
cd hello-genai

2. Pull the Required Model

docker model pull ai/smollm2:latest

You can replace the model if you want to try another supported one (like smollm2 or even deepseek).

some intersting commands

docker model list
docker model status
docker model inspect

3. Start the App

Just run the command:

./run.sh

This will start both the frontend and the model backend containers in Python | GO and Node.js on different ports

Docker Containers created for this simple chat app

4. Open in Your Browser

Navigate to http://localhost:8081 for python App and start chatting with your AI model right from the browser!

💡 What’s Going On Behind the Scenes?

The Hello GenAI app connects to your locally running model (via the Docker Model Runner) using python and React frontend. No cloud, no GPU setup — just local magic.

This is a great sandbox to:
• Prototype your own AI app
• Customize the frontend
• Try different models

🔄 Want to Stop It?

Simply hit Ctrl + C in the terminal and run:

docker compose down

🎯 Use Case Ideas With Hello GenAI

• Demo to your manager how quickly GenAI features can be spun up
• Test prompt flows before integrating into your real app
• Customize the UI and rebrand it for internal tools
• Hook it to a backend API for a tool-calling proof of concept

🏁 Wrapping This Up (For Real Now!)

Docker Model Runner + Hello GenAI = Your AI sandbox on steroids.
Now you’ve got the power to run, test, and innovate with open-source models without cloud costs or platform headaches.

Want me to create a follow-up walkthrough where we customize Hello GenAI for tool-calling or turn it into a Slack bot? Drop a comment or hit me up!

Let me know if you’d like this post packaged as a downloadable PDF or published directly on dev.to with formatting — happy to help you launch it 🚀

🙋‍♂️ What’s Next?

Have you tried Model Runner? Planning to use it in your product or workflow?

👉 Let me know in the comments, or share your experience!

👉 Love this content and want more like this -› Vote below!

:)Yes !! it counts a lot

📢 If You Loved This, Don’t Forget To:

• ❤️ Like this post
• 🔄 Share it with your team
• 📬 Follow for more dev-friendly AI tips

Keywords: Docker Model Runner setup, Docker for AI models, MLOps with Docker, running AI models locally, AI tool calling with Docker, Docker Desktop model integration

🚀 Exploring Docker Ask Gordon: Your AI-Powered Dev Companion!b

Raveendiran RR — Sun, 16 Feb 2025 19:59:05 +0000

As developers, we love automation, efficiency, and innovation. With Docker Ask Gordon, Docker has stepped into the AI-powered assistant space, making it easier than ever to interact with and optimize containerized workflows. Let’s dive into what it is, how it works, and how you can start using it today!

🔹

What is Docker and Why Use It?

Docker is a platform that enables developers to build, ship, and run applications in containers, ensuring consistency across different environments. It simplifies deployment, enhances scalability, and boosts productivity.

🌍

Generative AI & Model Context Protocol: The Evolution of Intelligent Development

Generative AI is revolutionizing how we interact with technology, from code generation to debugging. The Model Context Protocol (MCP) enables AI models to retain and utilize contextual understanding across different interactions, making AI assistants smarter and more efficient for developers.

🔄

Model Context Protocol: How It Powers Smart AI Assistants

MCP allows AI models to understand project-specific requirements, retain knowledge across sessions, and provide accurate recommendations, making tools like Ask Gordon more intuitive and responsive.

The MCP adds more context to your ask. It chooses the right MCP servers and the tools available in the server to complete the task you ask it

Imagine chatting with anything and getting things done. Like create a new file and write content into it by chatting with client IDE.Or query any DB and get insigths without writing a single line of code. The possibilities are limitless. Learn more on Model Context Protocol

🛠

Setting Up Docker Desktop

Download Docker Desktop from Docker’s official site.

Install it following platform-specific instructions.

Launch Docker Desktop and sign in to your Docker Hub account.

🎛

Activating Beta Features in Docker and Docker AI

Open Docker Desktop and navigate to Settings.

Under the Features in Development tab, enable Beta Features.

Restart Docker Desktop to apply changes.

Ask Gordon Activated

You can now ask any question to Ask Gordon and with the help of Docker AI, the questions are answered just like ChatGPT.

Notice that you have a coy button from where you can copy commands

🔍

How to get MCP working on Ask Gordon

Using MCP with Docker Desktop and Gordon

Create a .yml file called gordon-mcp.yaml in your working
directory from the terminal in docker desktop

touch gordon-mcp.yml

Add the MCP servers you want to activate for Docker AI. In this example, we have only listed the servers for file system and fetch ( webscraping)

services:
  fetch:
    image: mcp/fetch
  fs:
    image: mcp/filesystem
    command:
      - /rootfs
    volumes:
      - .:/rootfs

Now run this command to get the contents of the directory

$ docker ai list the contents of this file 
$ docker ai show me the content of https://docs.docker.com/desktop/features/gordon/mcp/

you can run either in the docker terminal or your system terminal . The ouput remains the same

Docker would create MCP server containers when it executes the MCP actions. Later these containers are stopped

🎯

Use Cases for Ask Gordon

Debugging Containers: Quickly diagnose and fix container-related issues.
Optimizing Dockerfiles: Get AI-driven suggestions to enhance performance.
Understanding Configurations: Learn how to set up volumes, networking, and security best practices.
CI/CD Integration: Ensure best practices in building efficient pipelines.

📚

References & Further Reading

Docker Desktop
Ask Gordon Overview
Model Context Protocol

Docker Ask Gordon is more than just an AI assistant—it’s a context-aware development companion. Try it out and supercharge your container workflows today! 🚀

I'm excited to share this information with you. Please Like | Share |Comment | follow me if this post has helped you