Running AI Models with Docker Compose

#ai #docker #devops #programming

Docker Compose has completely changed the game in how we run and connect a multi-service application. Just run a single command, and everything is up and running, with all the services well interconnected.

When Docker introduced the Docker Model Runner (or DMR, as we call it internally at Docker), there was a missing piece (at least for me). To use an AI model with a Compose application, we had to run the model with DMR separately and then connect our Compose application service by passing the config of that running model.

Docker knew this, and sorted it out by adding the capability to describe an AI model right in compose.yml, so it can run and destroy the model on demand. Just like we configure services, networks, and volumes, we can now do the same for AI models with models.

Prerequisites

Docker and Docker Compose installed
A basic understanding of AI and LLMs

Getting Started

To make the concept easier to follow, I created a GitHub project: Pradumnasaraf/Saraf-AI (yes, it's my last name "Saraf", and I added "AI" to it :)). It's a Next.js chat application that talks to a Docker AI model using the OpenAI framework. You can clone it and keep it ready, since we will reference it many times.

The Docker Compose models component

First, let's look at the compose.yml. Alongside the familiar services, we now have models as a top-level element. This is the new element for defining AI models.

We define a service named saraf-ai that uses the model llm, and then define that llm model referencing the ai/smollm2 model image.

The complete config lives in compose.yml at the root of the repo:

services:
  saraf-ai:
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - 3000:3000
    # Models to run
    models:
      - llm

models:
  # Model name
  llm:
    # Model image
    model: ai/smollm2

Now we know how the config looks, but how does our app connect and communicate with this AI model? How do we set up environment variables like the model name, URL, and API key for the OpenAI specification?

This is where Docker shines.

When we add a model to a service, Docker auto-generates and injects two environment variables into the service based on the model name (in our case, llm):

LLM_URL: the model endpoint to communicate with.
LLM_MODEL: the model name.

We can reference these directly in our application. If that sounds confusing, you can read more about it here.

If you use multiple models and want to control the variable names, you can define them explicitly. For example, with two models below:

services:
  app:
    image: my-app
    models:
      llm:
        endpoint_var: AI_MODEL_URL
        model_var: AI_MODEL_NAME
      embedding-model:
        endpoint_var: EMBEDDING_URL
        model_var: EMBEDDING_NAME

models:
  llm:
    model: ai/smollm2
  embedding-model:
    model: ai/all-minilm

Now, instead of the default LLM_URL and LLM_MODEL, the application is injected with AI_MODEL_URL and AI_MODEL_NAME. And for embedding-model, it gets EMBEDDING_URL and EMBEDDING_NAME.

Now, let's look at our Next.js application.

Application config

We built a Next.js application that uses the OpenAI framework (the industry standard) to talk to the Docker AI model. It automatically picks up the environment variables that Docker injects.

We don't need an API key here, since this is a local model, not a cloud LLM with quotas.

Below is the complete code. You will also find it in src/app/api/chat/route.ts:

import OpenAI from 'openai';
import { NextResponse } from 'next/server';

const openai = new OpenAI({
  baseURL: process.env.LLM_URL || '',
  apiKey: 'key-not-needed',
});

const model = process.env.LLM_MODEL || '';

export async function POST(req: Request) {
  try {
    const { message, messages } = await req.json();

    // Validate input
    if (!message || typeof message !== 'string') {
      return NextResponse.json(
        { error: 'Message is required and must be a string' },
        { status: 400 }
      );
    }

    if (!Array.isArray(messages)) {
      return NextResponse.json(
        { error: 'Messages must be an array' },
        { status: 400 }
      );
    }

    const stream = await openai.chat.completions.create({
      messages: [...messages, { role: 'user', content: message }],
      model,
      stream: true,
      temperature: 0.7,
      max_tokens: 2000,
    });

    return new Response(
      new ReadableStream({
        async start(controller) {
          try {
            for await (const chunk of stream) {
              const text = chunk.choices[0]?.delta?.content || '';
              if (text) {
                controller.enqueue(new TextEncoder().encode(text));
              }
            }
          } catch (streamError) {
            console.error('Streaming error:', streamError);
            controller.error(streamError);
          } finally {
            controller.close();
          }
        },
      }),
      {
        headers: {
          'Content-Type': 'text/plain; charset=utf-8',
          'Cache-Control': 'no-cache',
          'Connection': 'keep-alive',
        },
      }
    );
  } catch (error: unknown) {
    console.error('OpenAI API error:', error);

    const errorMessage = error instanceof Error ? error.message : 'Unknown error';
    const errorStatus = (error as { status?: number })?.status || 500;

    return NextResponse.json(
      {
        error: 'Failed to get response from AI',
        details: errorMessage,
      },
      { status: errorStatus }
    );
  }
}

Dockerizing the application

Now, let's Dockerize the application with a Dockerfile. You will find it in the root of the project.

# Build stage
FROM node:24-alpine AS builder

WORKDIR /app

# Copy package files
COPY package*.json ./

# Install dependencies (including devDependencies needed to build)
RUN npm ci

# Copy source code
COPY . .

# Build the application
RUN npm run build

# Production stage
FROM node:24-alpine AS runner

WORKDIR /app

# Create a non-root user
RUN addgroup --system --gid 1001 nodejs
RUN adduser --system --uid 1001 nextjs

# Copy built application from builder stage
COPY --from=builder /app/public ./public
COPY --from=builder /app/.next/standalone ./
COPY --from=builder /app/.next/static ./.next/static

# Set ownership to the nextjs user
RUN chown -R nextjs:nodejs /app

USER nextjs

EXPOSE 3000

ENV PORT 3000
ENV HOSTNAME "0.0.0.0"

CMD ["node", "server.js"]

A couple of things worth calling out:

We use npm ci (not npm ci --only=production) in the build stage, because the build needs the devDependencies like tailwindcss, @tailwindcss/postcss, and typescript. The final image stays small because the runner stage only copies Next.js's standalone output, not node_modules.
We use a multi-stage build and a non-root user to keep the image smaller, faster, and more secure. The standalone output relies on output: 'standalone' in next.config.ts.

Once that's in place, run the Compose application with a single command:

docker compose up --build

Docker builds the image, pulls the ai/smollm2 model, wires the injected environment variables into the app, and starts everything. You will see output similar to the screenshot below.

Now head over to localhost:3000 in your browser and try it out. You get a chat window like ChatGPT: type a prompt and ask away.

Here is a short demo:

That's it. That's how you run AI models with Docker Compose.

As always, I'm glad you made it to the end. Thank you for your support and for reading. I regularly share tips on Twitter (it will always be Twitter ;)), so come connect with me there. Let me know if anything needs updating.