DEV Community

Cover image for Docker Model Runner: Run Local AI Models Like Containers
Anuj Tyagi
Anuj Tyagi

Posted on

Docker Model Runner: Run Local AI Models Like Containers

#ai #docker #modeldeploy

Running a local large language model often begins with excitement and ends with dependency conflicts.

You install Python.

Then PyTorch.

Then CUDA.

Then a model-serving framework.

Then the correct tokenizer.

Then a quantized model format.

Then you discover that one library requires a different version of another library.

Docker helped developers solve a similar problem for traditional applications by providing a consistent way to package, distribute, and run software.

Docker Model Runner brings that experience to AI models.

Instead of manually configuring an inference server, downloading model files, exposing an API, and managing runtime dependencies, you can use familiar commands such as:

docker model pull ai/smollm2
docker model run ai/smollm2
Enter fullscreen mode Exit fullscreen mode

Docker Model Runner manages the model artifact, selects an inference engine, loads the model, and exposes APIs that applications can call.

In this article, we will explore:

  • What Docker Model Runner is
  • How it differs from running a regular container
  • How to install and enable it
  • How to pull and run a local model
  • How to call the model through an API
  • How to use it with Python and Docker Compose
  • How model packaging and OCI artifacts work
  • Which inference engine to choose
  • Security, performance, and operational considerations

What Is Docker Model Runner?

Docker Model Runner, often abbreviated as DMR, is a Docker capability for managing, running, serving, and distributing AI models.

It allows developers to:

  • Pull models from Docker Hub
  • Pull supported models from Hugging Face
  • Run models from the Docker CLI
  • Interact with models through Docker Desktop
  • Expose models through compatible REST APIs
  • Use models from containerized applications
  • Package model files as OCI artifacts
  • Push custom models to OCI-compatible registries
  • Configure context size and runtime parameters
  • Run different inference engines for different workloads

Docker Model Runner currently supports OpenAI-, Anthropic-, and Ollama-compatible interfaces. It also supports llama.cpp, vLLM, and Diffusers as inference engines.

Docker introduced Model Runner in beta in April 2025 and announced its general availability in September 2025.

The Problem Docker Model Runner Solves

Consider what is normally required to run a local LLM.

You may need to:

  1. Find a compatible model.
  2. Download several gigabytes of model weights.
  3. Determine whether the model uses GGUF, Safetensors, or another format.
  4. Select an inference engine.
  5. Configure CPU, GPU, CUDA, Metal, or ROCm support.
  6. Start an inference server.
  7. Configure an API endpoint.
  8. Connect your application to that endpoint.
  9. Manage model versions and local storage.
  10. Reproduce the same environment on another machine.

Each task is manageable individually.

The difficulty comes from managing all of them together.

Docker Model Runner creates a consistent developer workflow around these concerns:

Application
    |
    | OpenAI, Anthropic, or Ollama-compatible API
    v
Docker Model Runner
    |
    | Selects and manages the inference runtime
    v
llama.cpp, vLLM, or Diffusers
    |
    v
Local model artifact
Enter fullscreen mode Exit fullscreen mode

The application does not need to manage the tokenizer, model process, inference server, or model file location directly.

It calls an API.

Docker Model Runner handles the runtime behind it.

Is an AI Model Really Running as a Container?

Not exactly.

This distinction is important.

A model is primarily a collection of weights, configuration files, tokenizer information, and metadata. It is not an application process by itself.

Docker Model Runner separates two concerns:

The model artifact

The artifact contains the model weights and related metadata.

Models can be packaged using the OCI artifact format and stored in registries using familiar repository names and tags.

The inference engine

The inference engine loads the model into memory and performs the actual computation.

Depending on your configuration, Docker Model Runner may use:

  • llama.cpp
  • vLLM
  • Diffusers

On Linux, Model Runner and its inference engines run within containers. On macOS and Windows, Docker uses platform-specific sandboxing for the inference engines rather than treating them as ordinary application containers.

The experience resembles running a container, but Docker is managing a specialized model-serving lifecycle underneath.

How Docker Model Runner Works

When an application requests a model, Docker Model Runner performs several operations.

1. Resolve the model

Docker locates the requested model in Docker Hub, another OCI-compatible registry, Hugging Face, or the local model cache.

2. Download the artifact

If the model is not available locally, Docker downloads it.

Because models may contain billions of parameters, the first download can take time.

3. Cache the model locally

Downloaded models remain cached so future requests do not require downloading the same artifact again.

4. Select an inference engine

Docker Model Runner determines which engine should serve the model.

For example:

  • A quantized GGUF model normally uses llama.cpp.
  • A Safetensors model intended for high-throughput serving may use vLLM.
  • A Stable Diffusion model may use Diffusers.

5. Load the model into memory

The model is loaded when it is requested rather than permanently consuming memory.

6. Serve the model through an API

Applications interact with the model through a compatible HTTP endpoint.

Models are loaded on demand and can be unloaded after inactivity to reduce resource usage.

Prerequisites

Docker Model Runner is available through Docker Desktop and Docker Engine.

At the time of writing, Docker’s documented minimum Desktop versions are:

  • Docker Desktop 4.40 or later on supported macOS systems
  • Docker Desktop 4.41 or later on supported Windows systems
  • Docker Engine with the Docker Model Runner plugin on Linux

Hardware support differs by operating system and inference engine. Docker’s current documentation includes Apple Silicon, supported NVIDIA and Qualcomm configurations on Windows, and CPU, CUDA, ROCm, and Vulkan options with Docker Engine.

Your model must also fit within the available RAM or GPU memory.

A small quantized model might run comfortably on a development laptop. A larger model may require significantly more memory or a dedicated GPU.

Step 1: Enable Docker Model Runner

Docker Desktop

Open Docker Desktop and navigate to:

Settings → AI
Enter fullscreen mode Exit fullscreen mode

Enable:

Docker Model Runner
Enter fullscreen mode Exit fullscreen mode

To call the model directly from applications running on your host, also enable:

Host-side TCP support
Enter fullscreen mode Exit fullscreen mode

The default host port is commonly:

12434
Enter fullscreen mode Exit fullscreen mode

Docker Desktop also provides a Models section where you can discover, download, run, and inspect models.

You can also enable TCP access from the command line:

docker desktop enable model-runner --tcp 12434
Enter fullscreen mode Exit fullscreen mode

Docker Engine on Ubuntu or Debian 

sudo apt-get update
sudo apt-get install docker-model-plugin
Enter fullscreen mode Exit fullscreen mode

Docker Engine on RPM-based Linux distributions 

sudo dnf update
sudo dnf install docker-model-plugin
Enter fullscreen mode Exit fullscreen mode

Verify the installation:

docker model version
Enter fullscreen mode Exit fullscreen mode

Docker Engine enables TCP access on port 12434 by default.

Step 2: Search for a Model

You can search Docker Hub’s AI namespace:

docker model search
Enter fullscreen mode Exit fullscreen mode

Search for a particular model family:

docker model search llama
Enter fullscreen mode Exit fullscreen mode

Search both Docker Hub and Hugging Face:

docker model search --source=all
Enter fullscreen mode Exit fullscreen mode

Docker Model Runner can search models available through Docker Hub and Hugging Face.

Step 3: Pull a Model

For a lightweight first experiment, pull SmolLM2:

docker model pull ai/smollm2
Enter fullscreen mode Exit fullscreen mode

You can also select a particular model variant:

docker model pull ai/smollm2:360M-Q4_K_M
Enter fullscreen mode Exit fullscreen mode

The tag tells us more about the model:

360M       Approximately 360 million parameters
Q4_K_M     A four-bit quantized GGUF variant
Enter fullscreen mode Exit fullscreen mode

Quantization reduces the memory needed to run a model by representing its weights with fewer bits.

A Q4_K_M model will normally use less memory than an F16 model, although some quality may be lost.

Docker’s inference-engine documentation recommends Q4_K_M as a practical balance between model quality and memory usage for many local llama.cpp workloads.

Pulling from Hugging Face

Docker Model Runner can also pull supported GGUF models directly from Hugging Face:

docker model pull \
  hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF
Enter fullscreen mode Exit fullscreen mode

You can request a particular quantization using a tag:

docker model pull \
  hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF:Q4_K_S
Enter fullscreen mode Exit fullscreen mode

When no quantization tag is supplied, Docker attempts to select an available GGUF variant according to its model-pull behavior.

Step 4: Run the Model

Start an interactive session:

docker model run ai/smollm2
Enter fullscreen mode Exit fullscreen mode

You can now enter prompts directly in the terminal.

For a single prompt:

docker model run ai/smollm2 \
  "Explain retrieval-augmented generation in simple terms."
Enter fullscreen mode Exit fullscreen mode

You can also preload the model without opening an interactive conversation:

docker model run --detach ai/smollm2
Enter fullscreen mode Exit fullscreen mode

Preloading can reduce the latency of the first application request because the model is already in memory.

Useful Docker Model Commands

List downloaded models:

docker model list
Enter fullscreen mode Exit fullscreen mode

List models currently loaded in memory:

docker model ps
Enter fullscreen mode Exit fullscreen mode

Inspect a model:

docker model inspect ai/smollm2
Enter fullscreen mode Exit fullscreen mode

Check Model Runner status:

docker model status
Enter fullscreen mode Exit fullscreen mode

View disk usage:

docker model df
Enter fullscreen mode Exit fullscreen mode

View Model Runner logs:

docker model logs
Enter fullscreen mode Exit fullscreen mode

View captured requests and responses:

docker model requests
Enter fullscreen mode Exit fullscreen mode

Unload a running model:

docker model unload ai/smollm2
Enter fullscreen mode Exit fullscreen mode

Remove a downloaded model:

docker model rm ai/smollm2
Enter fullscreen mode Exit fullscreen mode

The docker model CLI also includes commands for benchmarking, packaging, pushing, tagging, inspecting, and managing Model Runner contexts.

Calling the Model Through an OpenAI-Compatible API

Running a model in the terminal is useful for experimentation.

Applications generally communicate with models through APIs.

For software running directly on the host, Docker Model Runner exposes its OpenAI-compatible endpoint at:

http://localhost:12434/engines/v1
Enter fullscreen mode Exit fullscreen mode

Send a request with curl:

curl http://localhost:12434/engines/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ai/smollm2",
    "messages": [
      {
        "role": "system",
        "content": "You explain technical concepts clearly."
      },
      {
        "role": "user",
        "content": "What is an AI inference engine?"
      }
    ],
    "temperature": 0.3,
    "max_tokens": 300
  }'
Enter fullscreen mode Exit fullscreen mode

The response follows an OpenAI-compatible chat-completions structure.

Docker Model Runner supports common parameters such as:

  • model
  • messages
  • prompt
  • max_tokens
  • temperature
  • top_p
  • stream
  • stop
  • presence_penalty
  • frequency_penalty

Use the complete model identifier in API requests:

{
  "model": "ai/smollm2"
}
Enter fullscreen mode Exit fullscreen mode

Docker Hub models typically use identifiers such as:

ai/smollm2
ai/llama3.2
ai/qwen2.5-coder
Enter fullscreen mode Exit fullscreen mode

Custom models may use identifiers such as:

myorganization/my-model
Enter fullscreen mode Exit fullscreen mode

Docker documents chat completions, text completions, embeddings, model listing, JSON mode, multimodal input for compatible models, and function calling for compatible llama.cpp models.

Using Docker Model Runner with Python

Because the endpoint is OpenAI-compatible, an existing application using the OpenAI Python SDK can often be redirected to Docker Model Runner by changing its base URL.

Install the SDK:

pip install openai
Enter fullscreen mode Exit fullscreen mode

Create app.py:

from openai import OpenAI


def main() -> None:
    client = OpenAI(
        base_url="http://localhost:12434/engines/v1",
        api_key="not-needed",
    )

    response = client.chat.completions.create(
        model="ai/smollm2",
        messages=[
            {
                "role": "system",
                "content": (
                    "You are a concise technical assistant who explains "
                    "concepts using practical examples."
                ),
            },
            {
                "role": "user",
                "content": "Explain why model quantization is useful.",
            },
        ],
        temperature=0.2,
        max_tokens=300,
    )

    content = response.choices[0].message.content

    if not content:
        raise RuntimeError("The model returned an empty response.")

    print(content)


if __name__ == "__main__":
    main()
Enter fullscreen mode Exit fullscreen mode

Run it:

python app.py
Enter fullscreen mode Exit fullscreen mode

The API key value is a placeholder. Docker Model Runner does not require an API key for its local OpenAI-compatible endpoint.

Streaming Responses

For chat interfaces, users usually expect tokens to appear as they are generated.

Set stream to true:

curl http://localhost:12434/engines/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ai/smollm2",
    "stream": true,
    "messages": [
      {
        "role": "user",
        "content": "Give me five practical uses for local LLMs."
      }
    ]
  }'
Enter fullscreen mode Exit fullscreen mode

Streaming reduces perceived latency because the application does not need to wait for the complete response before showing output. Docker Model Runner supports streaming through its compatible API interfaces.

Anthropic-Compatible API

Applications designed for Anthropic-style APIs can call:

http://localhost:12434/v1/messages
Enter fullscreen mode Exit fullscreen mode

Example:

curl http://localhost:12434/v1/messages \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ai/smollm2",
    "max_tokens": 500,
    "messages": [
      {
        "role": "user",
        "content": "Explain Docker Model Runner in one paragraph."
      }
    ]
  }'
Enter fullscreen mode Exit fullscreen mode

This compatibility can help tools designed around Anthropic’s messages interface use a locally hosted model instead.

Ollama-Compatible API

Docker Model Runner also exposes endpoints compatible with tools that expect an Ollama-style interface.

Example:

curl http://localhost:12434/api/chat \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ai/smollm2",
    "messages": [
      {
        "role": "user",
        "content": "What is a model artifact?"
      }
    ]
  }'
Enter fullscreen mode Exit fullscreen mode

List local models through the Ollama-compatible endpoint:

curl http://localhost:12434/api/tags
Enter fullscreen mode Exit fullscreen mode

The compatibility layer makes it easier to connect existing local-AI clients without rewriting their entire integration.

Calling Model Runner from Another Container

Applications running in Docker Desktop containers can access Model Runner through:

http://model-runner.docker.internal
Enter fullscreen mode Exit fullscreen mode

Example:

curl \
  http://model-runner.docker.internal/engines/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ai/smollm2",
    "messages": [
      {
        "role": "user",
        "content": "Explain container networking."
      }
    ]
  }'
Enter fullscreen mode Exit fullscreen mode

For Docker Engine, the host endpoint is commonly:

http://localhost:12434
Enter fullscreen mode Exit fullscreen mode

Container networking may require mapping the host gateway:

extra_hosts:
  - "model-runner.docker.internal:host-gateway"
Enter fullscreen mode Exit fullscreen mode

The correct base URL depends on whether the client is running on the host, inside Docker Desktop, or inside a Docker Engine Compose network.

Using AI Models in Docker Compose

One of Docker Model Runner’s most useful features is its integration with Docker Compose.

A Compose application can declare a model as a dependency:

services:
  chat-app:
    build: .
    ports:
      - "8000:8000"
    models:
      llm:
        endpoint_var: AI_MODEL_URL
        model_var: AI_MODEL_NAME

models:
  llm:
    model: ai/smollm2
    context_size: 4096
Enter fullscreen mode Exit fullscreen mode

Docker Model Runner will:

  1. Pull the model when necessary.
  2. Run the model locally.
  3. Provide the model endpoint.
  4. Inject the endpoint into AI_MODEL_URL.
  5. Inject the model identifier into AI_MODEL_NAME.

Your application can read those environment variables:

import os

from openai import OpenAI


model_url = os.environ["AI_MODEL_URL"]
model_name = os.environ["AI_MODEL_NAME"]

client = OpenAI(
    base_url=model_url,
    api_key="not-needed",
)

response = client.chat.completions.create(
    model=model_name,
    messages=[
        {
            "role": "user",
            "content": "Summarize the purpose of Docker Compose.",
        }
    ],
)

print(response.choices[0].message.content)
Enter fullscreen mode Exit fullscreen mode

The Compose model declaration keeps the application and its model dependency in the same configuration without packaging the model weights inside the application image.

Configuring Context Size

The context size determines how many tokens a model can process in a request.

The limit includes:

  • System instructions
  • Conversation history
  • Retrieved documents
  • Tool results
  • User input
  • Generated output

A larger context window may support longer conversations or documents, but it also increases memory consumption.

Configure it in Compose:

models:
  llm:
    model: ai/qwen2.5-coder
    context_size: 8192
Enter fullscreen mode Exit fullscreen mode

Docker’s documented default behavior varies by inference engine and model. Its llama.cpp configuration commonly starts around 4,096 tokens, while vLLM may use the model’s trained maximum context length.

Do not automatically choose the largest possible context.

A better approach is to select the smallest context that reliably supports your application.

For example:

Workload Possible starting point
Short classification 2,048 tokens
General chat 4,096 tokens
Code or document analysis 8,192 tokens
Large-document workflows 16,384+ tokens

Actual requirements depend on the model, prompt, and hardware.

Configuring Runtime Parameters

Compose can also pass runtime flags to the inference engine:

models:
  llm:
    model: ai/qwen2.5-coder
    context_size: 4096
    runtime_flags:
      - "--temp"
      - "0.2"
      - "--top-p"
      - "0.9"
Enter fullscreen mode Exit fullscreen mode

Runtime parameters can influence:

  • Randomness
  • Response diversity
  • Token sampling
  • Memory usage
  • Batch behavior
  • CPU or GPU execution
  • Inference performance

Use low temperature for deterministic workloads such as extraction, classification, or code transformation.

Use higher temperature only where variation is desirable, such as brainstorming or creative generation.

Understanding the Inference Engines

Docker Model Runner supports multiple inference backends because no single runtime is ideal for every workload.

llama.cpp

Best suited for:

  • Local development
  • CPU inference
  • Apple Silicon
  • Quantized models
  • Memory-constrained environments
  • GGUF models

Example:

docker model run ai/smollm2 --backend llama.cpp
Enter fullscreen mode Exit fullscreen mode

llama.cpp is Docker Model Runner’s default inference engine and supports a broad range of hardware.

vLLM

Best suited for:

  • High-throughput inference
  • Concurrent requests
  • Server-oriented workloads
  • NVIDIA GPU deployments
  • Safetensors models

vLLM uses techniques designed to improve batching and inference throughput. Docker currently documents its Model Runner vLLM support for Linux x86-64 systems with NVIDIA CUDA GPUs.

Diffusers

Best suited for:

  • Text-to-image generation
  • Stable Diffusion models
  • GPU-backed image-generation workloads

The Diffusers backend currently requires supported Linux and NVIDIA CUDA environments.

Check active engines:

docker model status
Enter fullscreen mode Exit fullscreen mode

Install a particular backend with Docker Engine:

docker model install-runner \
  --backend vllm \
  --gpu cuda
Enter fullscreen mode Exit fullscreen mode

Model Formats: GGUF and Safetensors

GGUF

GGUF is commonly used with llama.cpp.

It supports quantization, making it suitable for running models on laptops and resource-constrained machines.

Example model tag:

ai/llama3.2:3B-Q4_K_M
Enter fullscreen mode Exit fullscreen mode

This indicates:

  • Llama 3.2 model family
  • Approximately three billion parameters
  • Q4 quantization

Safetensors

Safetensors is frequently used by frameworks such as vLLM and Diffusers.

It is generally more appropriate when serving full or less aggressively quantized models on dedicated GPU infrastructure.

Docker Model Runner can package GGUF and Safetensors model files as OCI artifacts.

Packaging Your Own Model

Suppose you have a local GGUF model:

./models/my-model.gguf
Enter fullscreen mode Exit fullscreen mode

Package it as an OCI artifact:

docker model package \
  --gguf ./models/my-model.gguf \
  myorganization/my-model:Q4_K_M
Enter fullscreen mode Exit fullscreen mode

Package and push it in one command:

docker model package \
  --gguf ./models/my-model.gguf \
  --push myorganization/my-model:Q4_K_M
Enter fullscreen mode Exit fullscreen mode

For a Safetensors model directory:

docker model package \
  --safetensors ./models/my-safetensors-model \
  --push myorganization/my-vllm-model
Enter fullscreen mode Exit fullscreen mode

Docker Model Runner can publish model artifacts to registries that support OCI artifacts, not only Docker Hub.

Why OCI Artifacts Matter for AI Models

Using OCI artifacts gives model distribution some of the same properties developers value in container workflows.

Versioning

Models can use explicit tags:

myorganization/support-model:1.0
myorganization/support-model:1.1
myorganization/support-model:Q4_K_M
Enter fullscreen mode Exit fullscreen mode

Distribution

Teams can pull approved models from a shared registry.

Reproducibility

An application can reference a particular model artifact instead of telling every developer to manually download a model from an external website.

Existing infrastructure

Organizations may reuse parts of their existing registry, access-control, and artifact-management workflows.

However, a model tag alone does not provide complete AI governance.

Teams still need to track:

  • Model licenses
  • Training-data restrictions
  • Evaluation results
  • Prompt templates
  • Safety tests
  • Quantization changes
  • Runtime configuration
  • Deployment approvals

Container-style distribution improves operational consistency, but it does not replace model governance.

Observing Requests and Responses

Docker Model Runner provides request inspection through Docker Desktop and the CLI.

Run:

docker model requests
Enter fullscreen mode Exit fullscreen mode

In Docker Desktop, open:

Models → Requests
Enter fullscreen mode Exit fullscreen mode

The request view can expose information such as:

  • Model name
  • Request time
  • Prompt payload
  • Response payload
  • Context usage
  • Token usage
  • Generation duration
  • Generation speed

This is useful when debugging:

  • Unexpected output
  • Context-window overflow
  • Incorrect parameters
  • Slow responses
  • Wrong model selection

Docker’s Model Runner interface also provides logs:

docker model logs
Enter fullscreen mode Exit fullscreen mode

Request and response inspection can help developers understand what their frameworks are actually sending to the model.

Be careful when enabling detailed request logging in applications that process confidential information.

Prompts may contain source code, personal information, documents, credentials, or retrieved enterprise data.

Security Considerations

Local execution does not automatically mean secure execution.

The Docker Model Runner API does not require authentication. Any client that can reach the endpoint may be able to send inference requests or perform supported model operations.

Therefore:

  • Do not expose port 12434 directly to the public internet.
  • Bind the service only to trusted interfaces.
  • Use firewall and network controls.
  • Avoid untrusted containers on the same accessible network.
  • Put an authenticated gateway in front of the service when remote access is required.
  • Do not assume the placeholder API key provides security.
  • Review model licenses before internal or commercial use.
  • Validate model output before triggering tools or business actions.
  • Sanitize prompts and retrieved data.
  • Treat model-generated commands as untrusted input.

The following code:

api_key="not-needed"
Enter fullscreen mode Exit fullscreen mode

exists only because some compatible SDKs require an API-key field.

It does not authenticate the request.

For an agentic application, additional controls are required around:

  • Tool permissions
  • Human approval
  • Data access
  • Prompt-injection defense
  • Output validation
  • Audit logging
  • Rate limits
  • Resource quotas

Docker Model Runner provides inference infrastructure. It does not automatically secure the complete AI application.

Performance Considerations

Model size

Larger models normally require more RAM or VRAM and may generate tokens more slowly on local hardware.

Start with a small model before moving to a seven-billion-parameter or larger model.

Quantization

Lower-bit quantization reduces memory requirements but may affect output quality.

Test the model on your actual application tasks rather than selecting it only from benchmark scores.

Context size

Increasing the context size increases memory use.

A model that runs successfully with 4,096 tokens may fail with a much larger context configuration.

Cold starts

The first request may be slower because Docker Model Runner must load the model into memory.

Preload latency-sensitive models:

docker model run --detach ai/smollm2
Enter fullscreen mode Exit fullscreen mode

Concurrent users

llama.cpp is useful for local and resource-efficient inference.

For larger numbers of simultaneous requests on compatible GPU infrastructure, vLLM may be a better fit because it is designed for higher-throughput serving.

Benchmark before deciding

Docker Model Runner includes a benchmark command:

docker model bench ai/smollm2
Enter fullscreen mode Exit fullscreen mode

Measure performance using workloads that resemble your real application.

Useful metrics include:

  • Time to first token
  • Tokens per second
  • End-to-end latency
  • Memory utilization
  • GPU utilization
  • Error rate
  • Throughput under concurrency
  • Output quality

The fastest model is not necessarily the most useful model.

When Should You Use Docker Model Runner?

Docker Model Runner is especially useful when:

You already use Docker

Your team can manage local AI models through tools and concepts it already understands.

You need private local experimentation

Prompts and responses can remain on the local machine when no external services are called.

You are developing offline

Previously downloaded models can support development without a continuous cloud-model connection.

You want reproducible development environments

Teams can reference a consistent model name and version rather than manually configuring separate inference servers.

You need an OpenAI-compatible local backend

An application can often switch between a cloud provider and Docker Model Runner by changing configuration rather than rewriting the integration.

You are building AI-enabled container applications

Compose can declare the application and model dependency together.

You want to distribute internal models

OCI artifacts provide a familiar approach for packaging and publishing approved model variants.

When Might It Not Be the Right Choice?

Docker Model Runner may not be the best option when:

  • The model cannot fit on your available hardware.
  • You need a large proprietary frontier model.
  • You require globally distributed inference.
  • You need enterprise-scale autoscaling immediately.
  • You need strict multi-tenant isolation.
  • You require managed uptime guarantees.
  • You need authentication, quotas, and billing built directly into the inference service.
  • Your workload depends on specialized serving features unavailable in the selected backend.

Local inference is valuable, but it is not always cheaper or simpler at scale.

For occasional development traffic, local models may reduce API costs.

For high-volume production traffic, the hardware, operations, observability, scaling, and reliability costs must be evaluated separately.

A Practical Development Pattern

A strong development pattern is to separate application code from model-provider configuration.

Use environment variables:

AI_BASE_URL=http://localhost:12434/engines/v1
AI_API_KEY=not-needed
AI_MODEL=ai/smollm2
Enter fullscreen mode Exit fullscreen mode

Then initialize the client dynamically:

import os

from openai import OpenAI


client = OpenAI(
    base_url=os.environ["AI_BASE_URL"],
    api_key=os.environ["AI_API_KEY"],
)

model_name = os.environ["AI_MODEL"]
Enter fullscreen mode Exit fullscreen mode

For local development:

AI_BASE_URL=http://localhost:12434/engines/v1
AI_API_KEY=not-needed
AI_MODEL=ai/smollm2
Enter fullscreen mode Exit fullscreen mode

For another compatible environment, change the configuration rather than the application architecture.

This pattern makes it easier to:

  • Test models locally
  • Compare different models
  • Run automated evaluations
  • Introduce routing later
  • Avoid provider-specific code throughout the application

API compatibility is not always perfect across providers, so test tool calling, structured output, token counting, streaming, and error behavior before switching environments.

Common Problems and Fixes

docker: 'model' is not a docker command

Confirm that Docker Desktop or the Docker Model Runner plugin is installed and current.

On some macOS installations, Docker documents creating a CLI-plugin symlink:

ln -s \
  /Applications/Docker.app/Contents/Resources/cli-plugins/docker-model \
  ~/.docker/cli-plugins/docker-model
Enter fullscreen mode Exit fullscreen mode

Then retry:

docker model version
Enter fullscreen mode Exit fullscreen mode

The API endpoint refuses the connection

Confirm that host-side TCP support is enabled:

docker desktop enable model-runner --tcp 12434
Enter fullscreen mode Exit fullscreen mode

Check the status:

docker model status
Enter fullscreen mode Exit fullscreen mode

The model is extremely slow

Possible causes include:

  • The model is too large for the hardware.
  • Inference is falling back to CPU.
  • The selected quantization is too large.
  • The context window is oversized.
  • The model is loading for the first time.
  • Available memory is being exhausted.

Try a smaller quantized model and reduce the context size.

Out-of-memory errors

Use a smaller model or more aggressive quantization.

For example, replace a large or full-precision variant with a Q4 GGUF model.

Also reduce the configured context size.

A container cannot reach Model Runner

For Docker Desktop, use:

http://model-runner.docker.internal
Enter fullscreen mode Exit fullscreen mode

For Docker Engine Compose environments, add:

extra_hosts:
  - "model-runner.docker.internal:host-gateway"
Enter fullscreen mode Exit fullscreen mode

Then call:

http://model-runner.docker.internal:12434
Enter fullscreen mode Exit fullscreen mode

The API cannot find the model

Use the complete namespace:

{
  "model": "ai/smollm2"
}
Enter fullscreen mode Exit fullscreen mode

Do not send only:

{
  "model": "smollm2"
}
Enter fullscreen mode Exit fullscreen mode

Final Thoughts

Docker Model Runner does not make model inference magically lightweight.

Large models still require memory.

GPU compatibility still matters.

Quantization still involves trade-offs.

AI applications still need evaluation, observability, security, governance, and output validation.

What Docker Model Runner changes is the developer experience.

It gives developers a consistent way to:

Instead of building a custom local-inference environment for every project, teams can use Docker workflows they already understand.

The simplest example captures the idea:

docker model pull ai/smollm2
docker model run ai/smollm2
Enter fullscreen mode Exit fullscreen mode

But the more important capability is what comes next:

That combination makes Docker Model Runner more than a local chat utility.

It provides a practical foundation for developing reproducible, private, and portable AI applications.

References

Top comments (0)