DEV Community

Daniel Ioni
Daniel Ioni

Posted on

From Local Docker Stack to a Working RAG API with Ollama, Qdrant, and Mistral

#ai #python #docker #rag

Introduction

In the first phase of the MyZubster AI project, we prepared a local development stack containing:

  • A Python and Flask API
  • Ollama running on Windows
  • Mistral as the language model
  • Qdrant as the vector database
  • Open WebUI as the browser interface
  • Docker Compose for orchestration

The infrastructure was running, but the MyZubster application was not yet using it.

We have now completed the next step: MyZubster has a working Retrieval-Augmented Generation API.

The final workflow is:

Question
   ↓
MyZubster Flask API
   ↓
Embedding with nomic-embed-text
   ↓
Semantic search in Qdrant
   ↓
Relevant observations
   ↓
Answer generated by Mistral
Enter fullscreen mode Exit fullscreen mode

Everything runs locally.

What we implemented

We added the following endpoint:

POST /api/ai/ask
Enter fullscreen mode Exit fullscreen mode

It accepts a question:

{
  "question": "What was observed in the marketplace?"
}
Enter fullscreen mode Exit fullscreen mode

The endpoint then:

  1. Loads the observations stored by MyZubster.
  2. Generates embeddings for their descriptions.
  3. Creates the Qdrant collection when necessary.
  4. Indexes the observations in Qdrant.
  5. Generates an embedding for the question.
  6. Searches for the most relevant observations.
  7. Builds a prompt containing the retrieved context.
  8. Sends the prompt to Mistral through Ollama.
  9. Returns the generated answer and its sources.

The endpoint was implemented and merged through GitHub PR #10.

The models

The local environment now uses two different models.

Generation model 

mistral:latest
Enter fullscreen mode Exit fullscreen mode

Mistral generates the final natural-language response.

Embedding model 

nomic-embed-text
Enter fullscreen mode Exit fullscreen mode

This model converts observations and questions into numerical vectors.

Keeping generation and embedding separate is important. A conversational model is designed to generate text, while an embedding model is optimized for semantic similarity.

The API configuration

The API container receives the following environment variables:

environment:
  OLLAMA_BASE_URL: "http://host.docker.internal:11434"
  OLLAMA_MODEL: "mistral:latest"
  OLLAMA_EMBEDDING_MODEL: "nomic-embed-text"
  QDRANT_URL: "http://qdrant:6333"
  QDRANT_COLLECTION: "myzubster"
  AI_REQUEST_TIMEOUT: "120"
  AI_CONTEXT_LIMIT: "5"
Enter fullscreen mode Exit fullscreen mode

Ollama runs directly on Windows, so Docker containers reach it through:

host.docker.internal
Enter fullscreen mode Exit fullscreen mode

Qdrant runs inside Docker Compose, so the API reaches it by its service name:

http://qdrant:6333
Enter fullscreen mode Exit fullscreen mode

Installing the embedding model without the Ollama CLI

During setup, PowerShell returned:

ollama : The term 'ollama' is not recognized
Enter fullscreen mode Exit fullscreen mode

Ollama was running, but its executable was not available in the current PATH.

Instead of changing the system configuration, we used the Ollama HTTP API directly:

$body = @{
  name   = "nomic-embed-text"
  stream = $false
} | ConvertTo-Json

Invoke-RestMethod `
  -Uri "http://localhost:11434/api/pull" `
  -Method Post `
  -ContentType "application/json" `
  -Body $body
Enter fullscreen mode Exit fullscreen mode

The installed models can be verified with:

Invoke-RestMethod "http://localhost:11434/api/tags"
Enter fullscreen mode Exit fullscreen mode

The response should contain both:

mistral:latest
nomic-embed-text
Enter fullscreen mode Exit fullscreen mode

Docker networking issue

The Compose configuration originally used Docker’s default project network.

After adding a named network:

networks:
  default:
    name: myzubster-ai-network
Enter fullscreen mode Exit fullscreen mode

some existing containers were still associated with the previous network.

Docker returned:

container is not connected to the network myzubster-mvp_default
Enter fullscreen mode Exit fullscreen mode

We recovered without deleting persistent data:

docker compose down --remove-orphans
docker compose up -d --build
Enter fullscreen mode Exit fullscreen mode

If stale containers remain, they can be recreated explicitly:

docker rm -f `
  myzubster-mvp-api-1 `
  myzubster-mvp-qdrant-1 `
  myzubster-mvp-open-webui-1

docker compose up -d --build
Enter fullscreen mode Exit fullscreen mode

We deliberately avoided:

docker compose down -v
Enter fullscreen mode Exit fullscreen mode

because -v would delete the named volumes containing observations, Qdrant data, and Open WebUI state.

Git update conflict

The local docker-compose.yml had already been modified manually, while a newer version had been merged into the repository.

As a result, Git correctly blocked the pull:

Your local changes to docker-compose.yml would be overwritten by merge
Enter fullscreen mode Exit fullscreen mode

We first created a backup:

Copy-Item docker-compose.yml docker-compose.local.backup.yml
Enter fullscreen mode Exit fullscreen mode

Then restored the tracked version and pulled the merged changes:

git restore docker-compose.yml
git pull origin main
Enter fullscreen mode Exit fullscreen mode

This brought the local project to the merged commit while preserving a copy of the previous configuration.

Starting the complete stack

The environment can now be started with:

docker compose up -d --build
Enter fullscreen mode Exit fullscreen mode

We verify the services using:

docker compose ps
Enter fullscreen mode Exit fullscreen mode

The expected state is:

api          healthy
open-webui   healthy
qdrant       up
Enter fullscreen mode Exit fullscreen mode

The local services are available at:

  • MyZubster API: http://localhost:5000
  • Open WebUI: http://localhost:3001
  • Qdrant dashboard: http://localhost:6333/dashboard
  • Ollama API: http://localhost:11434

Creating a test observation

Before testing semantic retrieval, we created an observation:

$observation = @{
  description = "Recyclable packaging observed in the marketplace"
  latitude    = 44.0678
  longitude   = 12.5695
} | ConvertTo-Json

Invoke-RestMethod `
  -Method Post `
  -Uri "http://localhost:5000/api/observation" `
  -ContentType "application/json" `
  -Body $observation
Enter fullscreen mode Exit fullscreen mode

The API stored and returned the new observation.

Testing the RAG endpoint

PowerShell’s native command parsing caused an unexpected issue with curl.exe.

A normally quoted JSON command was split into several arguments. Curl then interpreted words from the question as hostnames:

Failed to connect to observations:80
Failed to connect to regarding:80
Enter fullscreen mode Exit fullscreen mode

The reliable solution was PowerShell’s stop-parsing token:

curl.exe --% -X POST http://localhost:5000/api/ai/ask -H "Content-Type: application/json" --data "{\"question\":\"What was observed in the marketplace?\"}"
Enter fullscreen mode Exit fullscreen mode

The endpoint returned:

{
  "answer": "Recyclable packaging was observed in the marketplace.",
  "embedding_model": "nomic-embed-text",
  "model": "mistral:latest",
  "sources": [
    {
      "description": "Recyclable packaging observed in the marketplace",
      "coordinates": {
        "lat": 44.0678,
        "lng": 12.5695
      }
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

This result confirmed that the complete pipeline was operational.

Why the sources field matters

Returning only the generated answer would make it difficult to understand why the model produced it.

The endpoint also returns:

{
  "sources": []
}
Enter fullscreen mode Exit fullscreen mode

with the observations retrieved from Qdrant.

This provides:

  • Basic traceability
  • Easier debugging
  • A foundation for citations in the user interface
  • A way to evaluate retrieval quality
  • Protection against completely ungrounded answers

The system prompt also instructs Mistral to answer only from the supplied context and to state when the available information is insufficient.

Error handling

The endpoint validates incoming requests and returns controlled errors.

Examples include:

400 — Missing or invalid question
502 — Invalid response from an AI component
503 — Ollama or Qdrant temporarily unavailable
Enter fullscreen mode Exit fullscreen mode

The question length is also limited to prevent unexpectedly large requests.

External calls have explicit timeouts because local inference can take longer than a typical HTTP request.

We also increased the Gunicorn timeout so the server does not terminate requests while Mistral is still generating a response.

Automated tests

The implementation includes tests for:

  • Missing JSON bodies
  • Missing questions
  • Successful grounded responses
  • Returned source observations
  • Interaction between retrieval and generation functions

The GitHub workflow completed successfully before the pull request was merged.

This gave us confidence that the new endpoint did not break the existing observation API.

What we learned

Several practical lessons came out of this implementation:

  1. A running Docker stack is infrastructure, not yet an application feature.
  2. Generation models and embedding models serve different purposes.
  3. Container-to-host networking must be tested independently.
  4. Persistent Docker volumes should never be deleted casually.
  5. PowerShell quoting behaves differently from Bash quoting.
  6. RAG responses should expose their retrieved sources.
  7. Local model inference requires realistic HTTP and server timeouts.
  8. Git should be allowed to block a pull when local changes may be lost.
  9. A placeholder such as “write the question here” is still treated as a real model instruction.
  10. Testing the full request path is essential.

Current status

The MyZubster local AI stack is now fully operational:

  • Flask receives the question.
  • Ollama generates embeddings.
  • Qdrant stores and searches observation vectors.
  • Mistral generates a context-grounded answer.
  • The API returns the answer and supporting sources.
  • Open WebUI remains available for direct model experiments.
  • Automated tests pass on GitHub.

The infrastructure has moved from “services are running” to a genuinely working application feature.

Next steps

The current implementation is appropriate for an MVP, but there is still room to improve it:

  • Index observations when they are created instead of on every question
  • Avoid recalculating unchanged embeddings
  • Add similarity-score thresholds
  • Add pagination and collection management
  • Protect the AI endpoint with authentication
  • Add rate limiting
  • Stream generated responses
  • Add source citations to the frontend
  • Measure retrieval quality
  • Add production secrets and Qdrant authentication
  • Evaluate multilingual embedding models

For now, MyZubster has achieved the important milestone: a complete local RAG pipeline using real application data.

What would you add next: streaming responses, hybrid search, or automatic indexing?

Repository: nicolaususnicola-lgtm/myzubster-mvp

Implementation: Pull Request #10

Top comments (0)