Building a Local AI Stack with Docker, Ollama, Mistral, Qdrant, and Open WebUI

Introduction

While developing MyZubster, I wanted an AI environment that was:

• Local and private
• Easy to reproduce
• Accessible from Docker containers
• Suitable for semantic search and RAG
• Independent from paid AI APIs during development

The resulting stack combines:

• Ollama for running local language models
• Mistral as the initial LLM
• Qdrant as a vector database
• Open WebUI for testing models from a browser
• Flask as the application API
• Docker Compose for service orchestration

In this article, I’ll show how the pieces fit together and share a working Compose configuration.

Architecture

The final request flow will look like this:

User
  ↓
MyZubster API
  ↓
Qdrant semantic search
  ↓
Relevant application context
  ↓
Mistral through Ollama
  ↓
Generated response

This is the foundation of a Retrieval-Augmented Generation, or RAG, system.

Instead of asking the language model to answer using only its pretrained knowledge, the application first retrieves relevant information from Qdrant and includes it in the prompt.

Why run Ollama outside Docker?

I already had Ollama running directly on Windows and listening on port 11434.

Running a second Ollama instance inside Docker would:

• Duplicate the model storage
• Consume additional memory
• Create a port conflict
• Make GPU configuration more complicated

Docker Desktop containers can reach services running on the host through:

host.docker.internal

Therefore, services inside the Compose network use this address:

http://host.docker.internal:11434

Testing Ollama from Docker

Before integrating the application, I verified that a temporary container could reach Ollama:

docker run --rm curlimages/curl:8.12.1 `
  http://host.docker.internal:11434/api/tags

The response showed the installed model:

{
  "models": [
    {
      "name": "mistral:latest",
      "parameter_size": "7.2B",
      "quantization_level": "Q4_K_M"
    }
  ]
}

This confirmed that Docker-to-host networking was working correctly.

Docker Compose configuration

Here is the complete docker-compose.yml:

services:
  api:
    build:
      context: .
    init: true
    ports:
      - "5000:5000"
    environment:
      MYZUBSTER_HOST: "0.0.0.0"
      MYZUBSTER_PORT: "5000"
      MYZUBSTER_OBSERVATIONS_FILE: "/data/observations.json"
      OLLAMA_BASE_URL: "http://host.docker.internal:11434"
      OLLAMA_MODEL: "mistral:latest"
      QDRANT_URL: "http://qdrant:6333"
      QDRANT_COLLECTION: "myzubster"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    volumes:
      - observations-data:/data
    depends_on:
      qdrant:
        condition: service_started
    restart: unless-stopped
    stop_signal: SIGTERM
    stop_grace_period: 15s

  qdrant:
    image: qdrant/qdrant:latest
    ports:
      - "127.0.0.1:6333:6333"
      - "127.0.0.1:6334:6334"
    volumes:
      - qdrant-data:/qdrant/storage
    restart: unless-stopped

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    ports:
      - "127.0.0.1:3000:8080"
    environment:
      OLLAMA_BASE_URL: "http://host.docker.internal:11434"
      WEBUI_AUTH: "true"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    volumes:
      - open-webui-data:/app/backend/data
    restart: unless-stopped

volumes:
  observations-data:
  qdrant-data:
  open-webui-data:

networks:
  default:
    name: myzubster-ai-network

Starting the stack

First, validate the configuration:

docker compose config

Pull the required images:

docker compose pull

Then rebuild and start everything:

docker compose down
docker compose up -d --build

Check the service status:

docker compose ps

The expected services are:

• api
• qdrant
• open-webui

Ollama does not appear in this list because it runs directly on the host.

Testing the connection from the API container

The following command sends a request to Ollama from inside the Flask container:

docker compose exec api python -c "import json,os,urllib.request; url=os.environ['OLLAMA_BASE_URL']+'/api/generate'; data=json.dumps({'model':os.environ['OLLAMA_MODEL'],'prompt':'Reply only with: MyZubster connected','stream':False}).encode(); req=urllib.request.Request(url,data=data,headers={'Content-Type':'application/json'}); result=json.loads(urllib.request.urlopen(req,timeout=120).read()); print(result['response'].strip())"

Receiving a generated response proves that:

1. The API container resolves host.docker.internal.
2. Ollama accepts requests from Docker.
3. The configured Mistral model is available.
4. The application can now integrate local generation.

Local service URLs

Once the stack is running, the services are available at:

• MyZubster API: http://localhost:5000
• Open WebUI: http://localhost:3000
• Qdrant dashboard: http://localhost:6333/dashboard
• Ollama API: http://localhost:11434

The Qdrant and Open WebUI ports are bound to 127.0.0.1, which prevents them from being exposed directly to the local network.

A common PowerShell mistake

A YAML configuration is not a PowerShell command.

If you paste this directly into the terminal:

services:
  api:
    build:
      context: .

PowerShell tries to execute words such as services, api, and build, producing errors like:

The term 'services:' is not recognized as the name of a cmdlet

The YAML must be saved inside docker-compose.yml before running Docker Compose.

You can use an editor such as Visual Studio Code:

code docker-compose.yml

Or create the file using a PowerShell here-string:

@'
services:
  # Compose configuration goes here
'@ | Set-Content -Encoding utf8 docker-compose.yml

What is still missing?

The infrastructure is ready, but infrastructure alone does not create an AI feature.

The Flask application still needs endpoints that:

1. Generate embeddings for application content.
2. Store vectors and metadata in Qdrant.
3. Convert user queries into embeddings.
4. Retrieve the most relevant records.
5. Build a prompt containing the retrieved context.
6. Send that prompt to Mistral through Ollama.
7. Return the generated answer to the client.

A possible endpoint could look like:

POST /api/ai/ask

With a request such as:

{
  "question": "Which products match sustainable packaging?"
}

The API would search Qdrant and then ask Mistral to answer using only the retrieved MyZubster data.

Lessons learned

A few practical lessons from this setup:

• Test container-to-host networking before changing application code.
• Avoid running duplicate Ollama instances unless isolation is required.
• Store Qdrant and Open WebUI data in named Docker volumes.
• Bind development dashboards to localhost.
• Validate every Compose file with docker compose config.
• A successful model response confirms connectivity, not application-level integration.
• LLMs do not always follow exact-output prompts, especially smaller local models. Application code should validate structured responses instead of trusting them blindly.

Next steps

The next phase for MyZubster is implementing the actual RAG pipeline:

• Add an embedding model to Ollama
• Create and configure the Qdrant collection
• Index MyZubster data
• Implement semantic retrieval
• Add an AI API endpoint
• Add automated integration tests
• Connect the feature to the user interface

This local stack gives us a reproducible foundation for developing those features without sending private application data to an external model provider.

If you are building a similar local AI stack, I’d be interested to hear which models and vector databases you are using.

---

MyZubster is currently an evolving MVP. This setup is intended for local development and experimentation, not as a production deployment.