Your Next AI Agent's Brain Is One `docker model pull` Away

Your AI agent burns tokens to think. Most of that thinking - the tool-call planning, the YAML editing, the re-reads of the same file - does not need a frontier model behind an API. Docker Model Runner keeps a real LLM inside the Docker engine you already run, exposes it on an OpenAI-compatible endpoint, and charges you nothing per call. docker model pull is the whole setup.

This post is the wiring guide: what Model Runner actually is (and the one detail everyone gets wrong), how to size the model, and how to point Docker Agent - the CLI plugin that defines agent teams in YAML - at a local model for a fully offline agent stack.

In this post:
What is Docker Model Runner, actually?
The detail everyone gets wrong: no container starts
Sizing the brain: GGUF and the quantization table
Wiring it to an agent: the dmr.yaml contract
What a local brain can and cannot do for an agent
FAQ

Key Takeaways

• Model Runner is an inference server inside Docker, not a container. docker model run calls an API; no image is built, no container is created (Docker blog).
• It speaks OpenAI, Anthropic, and Ollama API formats from the same endpoint - http://localhost:12434/engines/v1 for OpenAI-compatible clients (Docker docs, API reference).
• Models are GGUF files pulled from Docker Hub or Hugging Face, quantized per tag: ai/llama3.2:3B-Q4_K_M means 3B parameters at ~4.5 bits per weight (Docker docs, inference engines).
• llama.cpp is the default engine and works everywhere; vLLM needs NVIDIA GPUs for production throughput; Diffusers generates images (Docker docs).
• Docker Agent accepts it as provider: dmr - one YAML block, and your agent team runs offline (examples/dmr.yaml).

What is Docker Model Runner, actually?

Model Runner is Docker's answer to a question that kept coming up as agents went mainstream: why should the model live behind someone else's API when it could live next to your containers?

The component ships with Docker Desktop 4.40+ (enabled by default on Apple silicon) and is being built out for Linux engines. You manage it with a CLI that mirrors the container grammar you already know (Docker blog, Run LLMs locally):

$ docker model pull ai/smollm2:360M-Q4_K_M
$ docker model run ai/smollm2

The tag scheme matches image tags: {model}:{parameters}-{quantization}. That 360M-Q4_K_M is a 360-million-parameter model at 4-bit quantization - small enough to feel instant, and the pattern scales up to the size your RAM allows.

Once a model is pulled, Docker exposes it through REST endpoints. The API reference lists OpenAI, Anthropic, and Ollama-compatible surfaces, so tools that already speak those dialects work without adaptation (Docker docs, DMR REST API):

Access from	Base URL
Host processes (TCP)	http://localhost:12434
Containers (Docker Desktop)	http://model-runner.docker.internal
Containers (Linux engine)	http://172.17.0.1:12434

For OpenAI-compatible SDKs specifically, the base URL is http://localhost:12434/engines/v1, and a completion is a standard POST /engines/v1/chat/completions (Docker docs, API reference).

The detail everyone gets wrong: no container starts

Here is the part I had to reread. Coming from years of docker container run, the mental model writes itself: the model runs in a container. It does not.

Docker's own walkthrough stops to make the point: docker model run "won't spin up any kind of container." Instead, it calls an Inference Server API endpoint hosted by the Docker engine. There is no image layer for the model runtime, no entrypoint, no container ls entry to find. The engine itself gained an inference server.

Why that matters in practice: there is nothing to shell into, nothing to restart when it misbehaves, and nothing that shows up in your Compose file. The model is closer to a built-in volume format than a workload - Docker calls the packaging "just files" with weights, tokenizer, and metadata in one GGUF binary (Docker blog, How to Build, Run, and Package AI Models).

Sizing the brain: GGUF and the quantization table

GGUF is the format llama.cpp consumes, and its whole trick is quantization: shrinking weights from 16 bits to fewer without collapsing quality. The inference-engines page publishes the table worth internalizing before you pull anything:

Quantization	Bits per weight	Memory usage	Quality
Q2_K	~2.5	Lowest	Reduced
Q3_K_M	~3.5	Minimal	Acceptable
Q4_K_M	~4.5	Low	Good
Q5_K_M	~5.5	Moderate	Excellent
Q6_K	~6.5	Higher	Excellent
Q8_0	8	High	Near-original
F16	16	Highest	Original

Docker's recommendation is Q4_K_M as the balance point for most use. The arithmetic is the whole decision: a 3B-parameter model at Q4_K_M is roughly 1.8 GB of weights (3B x 4.5 bits / 8), which fits on a laptop and leaves the model feeling responsive. The same 3B at F16 is about 6 GB.

Under the hood, llama.cpp is the default engine, with Metal acceleration automatic on Apple Silicon, CUDA on NVIDIA Windows and Linux, and a plain CPU path that works anywhere (Docker docs, inference engines). Two more engines exist for narrower jobs: vLLM for production throughput on NVIDIA hardware (Safetensors format), and Diffusers for image generation.

One caveat from the same page: models from Hugging Face must be GGUF already. A random Safetensors checkpoint will not pull (Docker docs, inference engines).

Wiring it to an agent: the dmr.yaml contract

Docker Agent - the CLI plugin, formerly cagent, that defines agent teams in YAML - takes Model Runner as a first-class provider. The repository's examples/dmr.yaml is the whole contract:

agents:
  root:
    model: qwen
    description: "Pirate-themed AI assistant"
    instruction: Talk like a pirate

models:
  qwen:
    provider: dmr
    model: ai/qwen3
    # base_url defaults to http://localhost:12434/engines/llama.cpp/v1
    # use http://model-runner.docker.internal/engines/v1 if you run
    # docker-agent from a container

provider: dmr is the switch. One environment variable's worth of configuration, and the same agent definition that ran against a cloud provider now runs against the engine on your machine. The README lists the cloud escape hatch as peers: OpenAI, Anthropic, Gemini, Bedrock, Mistral, xAI, plus DMR for local (docker-agent README).

Two details in that same example file are worth more than they look.

First, speculative decoding. The example defines a second model as a draft engine:

qwen_speculative:
  provider: dmr
  model: ai/qwen3
  provider_opts:
    speculative_draft_model: ai/qwen3:0.6B-Q4_K_M
    speculative_num_tokens: 16
    speculative_acceptance_rate: 0.8

A small draft model proposes tokens the big model verifies - a local-inference trick that trades a little quality risk for real speed, with the llama.cpp defaults documented right in the YAML (docker-agent examples).

Second, the multimodal flag. DMR-hosted models are not in the models.dev catalog, so docker-agent cannot detect what a local model can do and defaults to text-only. If you pull a vision model, you declare it yourself (docker-agent examples):

qwen_vision:
  provider: dmr
  model: ai/qwen2.5-vl
  provider_opts:
    supports_images: true
    supports_pdf: false

If you run docker-agent from inside a container rather than the host, the base URL changes to http://model-runner.docker.internal/engines/v1 - the same endpoint namespacing the Model Runner docs describe for containers.

And if you would rather not change the agent's provider at all, docker model gateway runs an OpenAI-compatible proxy on http://0.0.0.0:4000 and routes to Model Runner models by name - useful for tools that insist on their own base URL handling.

What a local brain can and cannot do for an agent

The honest part. A 360M SmolLM2 will not plan multi-step tool calls the way a frontier model does, and pretending otherwise is how agent projects die. What the local path is genuinely good for, based on what the docs themselves demonstrate:

High-frequency, low-stakes calls. Classification, routing, extraction, summarization - the calls an agent makes dozens of times per run where cloud latency and cost compound. The docker model gateway reference exists precisely because tools want these on a local endpoint.

Draft work. The speculative-decoding pattern in the dmr.yaml example is the docs' own suggestion: a small draft model plus a larger verifier. The same shape works inside agent teams - a fast local model triages, a cloud model handles the hard reasoning steps, and the provider switch is one line (docker-agent repo, examples).

Offline and private by construction. The completion never leaves the machine - the launch blog's curl example notes "No API key required - this runs locally and securely on your machine". For agent tasks touching private repos or data, that is not a cost feature; it is the feature.

What does not transfer: frontier reasoning, long-context retrieval over huge codebases, and reliable structured-output discipline under load. Those are exactly the capabilities you pay tokens for, and the Model Runner docs do not claim otherwise - the engines table positions llama.cpp for "local development, resource efficiency," with vLLM as the production path when you have the GPUs (Docker docs, Model Runner).

The sensible architecture is not local instead of cloud. It is a tiered brain: the agent's router, summarizer, and classifier run at $0.00 on Model Runner; the steps that genuinely need frontier judgment keep their API key. Docker Agent's provider model supports both in one team, which is what makes the whole thing practical rather than a stunt.

FAQ

Do I need Docker Desktop for Model Runner?
Docker Desktop 4.40+ ships it (default-on for Apple silicon). On Linux engines the component is arriving through the standard Docker repositories; the API reference documents the Linux base URL as http://172.17.0.1:12434 from containers and http://localhost:12434 from the host.

Which model should I pull first?
ai/smollm2:360M-Q4_K_M - the launch blog's own example. It is small enough to be instant and exercises the whole pull-run-query loop.

Can my existing OpenAI SDK point at it?
Yes. Base URL http://localhost:12434/engines/v1, standard chat completions, no API key. Anthropic- and Ollama-compatible clients are supported from the same endpoint with different paths (Docker docs, API reference).

Does the model run in a container?
No. It runs in an inference server inside the Docker engine. Nothing appears in docker container ls (Docker blog, Run LLMs locally).

Can docker-agent use a local vision model?
Yes, with supports_images: true declared under provider_opts - DMR-hosted models default to text-only because the agent cannot auto-detect their capabilities (docker-agent examples).

What are the five lines that matter?

$ docker model pull ai/smollm2:360M-Q4_K_M
$ curl http://localhost:12434/engines/v1/chat/completions ...

models:
  qwen:
    provider: dmr
    model: ai/qwen3

Pull a model, curl it on the OpenAI-compatible endpoint, point your agent at provider: dmr. The brain now runs where your containers run, and the meter never starts.

Verified against docs.docker.com (Model Runner overview, DMR REST API reference, inference engines, docker model gateway) and the docker-agent repository (README, examples/dmr.yaml) plus three Docker blog posts, all retrieved 2026-10-26. I did not run this walkthrough on my own machine yet - the model-runner component is not available for my Linux engine's repository at time of writing, and I flag that here rather than pretending otherwise. Every claim above traces to the linked sources.

Sources

Every claim above traces to these primary sources. Where I could not verify something hands-on, the post says so.

• Docker Model Runner
• Docker Agent
• Docker blog
• Docker docs, API reference
• Docker docs, inference engines
• examples/dmr.yaml
• docker model gateway
• launch blog's
• GitHub

About the author

Kunal Das writes operations engineering notes as he learns them, always against primary sources. Contact: GitHub and Dev.to. Follow the Docker tag for the next posts in this series.