DEV Community: Gowsiya Syednoor Shek

Set Up Your Company for Success with Docker (Part 5)

Gowsiya Syednoor Shek — Sat, 14 Jun 2025 22:01:28 +0000

Docker isn't just for developers on personal projects, it's a powerful platform that, when configured correctly, can accelerate collaboration and strengthen security at the organizational level. Here’s how to set your company up for long-term success with Docker:

1. Enforce Sign-In for Docker Desktop

By default, Docker Desktop can be launched without requiring sign-in. This means users might bypass organizational policies and lose access to subscription benefits. Enforcing sign-in ensures tighter control:

Sign-in Prompt: Docker Desktop will block access unless the user signs in with an org-approved Docker ID.
Sign-out Behavior: Signed-out users are immediately blocked until they re-authenticate.

Enforcement Methods:

Platform	Method
Windows	Registry Key
macOS	Configuration Profiles or .plist
Cross-platform	`registry.json`

Note: Enforcing sign-in does not affect CLI access unless Single Sign-On (SSO) is also enforced.

2. Enforce Single Sign-On (SSO) for Centralized Authentication

Enforcing SSO ensures that all users authenticate through your company’s identity provider (e.g., Okta, Azure AD). Here is what this enables:

Centralized access policies (MFA, password rotation)
Automatic provisioning/de-provisioning (via SCIM)
Streamlined onboarding and offboarding

Enforcement Level	Description	Benefits
Sign-in Only	Requires Docker Hub account sign-in	Enables visibility & subscription usage
SSO Only	Forces sign-in via SSO	Aligns with enterprise identity governance
Both	Strongest option	Combines access control, policy enforcement, & visibility
Neither	Least secure	Not recommended for orgs

3. Create Organizations and Teams in Docker Hub

Using Organizations and Teams in Docker Hub, you can group users and define roles based on job functions.

Example Team Setup:

Team	Description
`frontendeng`	Front-end developers
`backendeng`	Back-end developers
`qaeng`	Quality Assurance testers
`devopseng`	DevOps / Infra team

You can assign Docker IDs to these teams and manage them via the Organizations > Teams tab in Docker Hub.

4. Set Repository-Level Access Permissions

After your teams are set up, configure fine-grained permissions for each Docker repository.

Example Access Table:

Repository	frontendeng	backendeng	qaeng	devopseng
`ui-build`	Admin	Read-only	Read-only	Admin
`api-build`	Read-only	Admin	Read-only	Admin
`ui-release`	Read-only	Read-only	Read & Write	Admin
`api-release`	Read-only	Read-only	Read & Write	Admin

Permissions are configured through Organizations > Teams > Permissions in Docker Hub.

5. Secure Image Delivery with Docker Content Trust & Scout

Enable Docker Content Trust (DCT)

DCT allows publishers to digitally sign images and enables consumers to verify image signatures before pulling or running them. This prevents tampering or unverified images from being used in production.

Trust is associated per image tag (e.g., myimage:latest)
Signed tags can coexist with unsigned ones under the same repo

# Enable DCT
export DOCKER_CONTENT_TRUST=1

# Generate key & sign image
docker trust key generate signer-name
docker trust sign registry.example.com/org/image:tag

# Inspect trust
docker trust inspect --pretty registry.example.com/org/image:tag

Back up your root key securely - it cannot be recovered if lost.

Activate Docker Scout for CVE Analysis and SBOMs

Docker Scout gives you real-time insights into the security status of your images:

Automatically scans images on Docker Hub (or other integrated registries)
Tracks vulnerabilities (CVEs) and severity
Uses SBOM (Software Bill of Materials) for in-depth package-level analysis

Enable & Analyze:

# Build with provenance + SBOM
docker build --push --tag myorg/myimage:tag --provenance=true --sbom=true .

# Analyze locally
docker scout quickview myorg/myimage:tag
docker scout cves --only-severity critical myorg/myimage:tag

CLI Tools:

docker scout quickview: Summary & base image comparison
docker scout cves: CVEs filtered by severity/type
docker scout compare: Compare two image versions for risk delta

Scout results update automatically as new CVEs emerge - no need to re-push.

Final Thoughts

By combining account enforcement, team-based access, image signing, and continuous vulnerability scanning, Docker can be transformed from a simple container platform into a secure and scalable foundation for modern DevOps workflows.

GenAI App with Prompt Templates and Role Switching (Part 4)

Gowsiya Syednoor Shek — Mon, 02 Jun 2025 01:19:57 +0000

In Part 3, we built a FastAPI backend that could talk to a local LLM using Docker Model Runner. In this Part 4, we are moving forward by adding:

Prompt Templates
Role Switching (system/user roles)
A Simple HTML UI - all running on your machine

Why This Matters

Prompt engineering isn't just about text, it's about structure. This app lets you:

Try different system roles like "You are a data science tutor" or "You are a sarcastic assistant"
Apply templates around your prompt
Experiment with how prompt structure changes output

Project Layout

docker-llm-fastapi-app/
├── app/
│   ├── main.py         
│   ├── templates/
│   │   └── index.html
├── Dockerfile
├── docker-compose.yml

How to Run

1. Pull & Run the Model (if not already)

docker model pull ai/mistral
docker model run ai/mistral

Ensure TCP host access is enabled in Docker Desktop’s Beta > Enable Docker Model Runner section.

2. Start the Backend API

docker compose up --build

Open from browser: http://localhost:8000
Full code is available here part4-code

UI Preview

The UI has:

A dropdown for system roles
A textbox for prompts
A Generate button to generate responses

All results appear below your prompt, in the same page.

Why Is It So Slow?

If your prompts are consistently taking more than a minute, it’s likely due to:

Model Size: Even relatively small LLMs like Mistral can take time to load into memory and start processing.
Cold Starts: Each prompt might be triggering a cold start if the model isn't staying warm between requests.
System Resources: Docker Model Runner currently doesn’t optimize for resource constraints, so if your machine lacks sufficient CPU/RAM or an NVIDIA GPU, performance may suffer.

Suggestions to improve speed:

Enable host-side TCP support.
Keep the model warm by sending periodic “ping” prompts every few minutes.
Try a smaller or quantized model (e.g., ai/smollm2 instead of ai/mistral).
Upgrade to a system with more memory or GPU support for Docker Desktop.

What’s Next

In Part 5, I’ll wrap up this mini-series with a broader, practical guide: "How to Set Up Your Company for Success with Docker"

This won’t be about just local demos, but will cover topics like:

Organizing Your Teams with Docker Organizations
Enforcing Sign-In and Enabling SSO
Standardizing Docker Desktop Configurations

After that, I will be exploring independent Docker topics in more depth, stay tuned!

Build a Local GenAI API with Docker Model Runner and FastAPI (Part 3)

Gowsiya Syednoor Shek — Sun, 25 May 2025 18:14:06 +0000

In Part 2, I ran an LLM locally using Docker Model Runner and connected to it through a Python script. Now in Part 3, we are wrapping that logic inside a FastAPI REST API - giving us a real, local GenAI backend we can use from Postman, web apps, or CLI tools.

Let’s dive in.

Goal

Build a FastAPI server that sends prompts to a locally running LLM (ai/mistral)
Expose a /generate endpoint
Run the API container and Docker Model Runner side-by-side

What We Built

A REST API (running in Docker) that talks to Docker Model Runner via an OpenAI-compatible endpoint. You send a prompt like:

{
  "prompt": ""Explain what is docker model runner in 3 points"
}

…and it responds with an AI-generated answer from a model running 100% on your machine.

Project Structure

docker-llm-fastapi-app/
├── app/
│   └── main.py         ← FastAPI logic
├── Dockerfile          ← API container
├── docker-compose.yml  ← Orchestration
├── README.md

Check out the code here part3-code

How to Run It

1. Pull and Start the Model

docker model pull ai/mistral
docker model run ai/mistral

If you have already pulled the model from previous tutorial, running pull again is not necessary.
You may see: Interactive chat mode started. Type '/bye' to exit.

That’s okay — the API is still active behind the scenes if TCP access is enabled.

2. Start the FastAPI Server

docker compose up --build

You’ll see:

Uvicorn running on http://0.0.0.0:8000

Call the Endpoint

Send a request from Postman or curl:

POST http://localhost:8000/generate
Content-Type: application/json

{
  "prompt": "What is MLOps in simple terms?"
}

Output:

{
  "response": "MLOps, short for Machine Learning Operations, is a practice for collaboration and..."
}

Things I Learned

1. Interactive Mode Still Enables API

Even though Docker says:

Interactive chat mode started. Type '/bye' to exit.

…the HTTP API is still available on localhost:12434. As long as TCP support is enabled in Docker Desktop, it works fine.

2. First Call Is Slow

The first request took ~2 minutes. Why?

The model is loaded into memory
Runtime warmup takes time

But after that, future prompts are little better.

What’s Next

In Part 4 I plan to build a - Prompt Templates + Role Options which adds a practical layer of prompt engineering to your GenAI app.

Stay tuned!

Run LLMs Locally with Docker Model Runner – A Real-World Developer Guide (Part 2)

Gowsiya Syednoor Shek — Sat, 17 May 2025 19:21:10 +0000

In Part 1 of this series, I explored how Docker Compose Watch helped accelerate development when working with Python-based AI apps. In this second part, I dive into Docker Model Runner – a powerful new feature that lets developers run large language models (LLMs) locally using OpenAI-compatible APIs, all powered by Docker.

But getting it to work wasn't as plug-and-play as I had hoped. So this tutorial is both a how-to and a real-world troubleshooting log for anyone trying to follow the same path.

What Is Docker Model Runner?

Docker Model Runner is a beta feature (from Docker Desktop 4.40+) that allows you to:

Pull open-source LLMs from Docker Hub or Hugging Face
Run them locally on your machine
Interact with them via OpenAI-compatible endpoints

It removes the need to use separate tools like Ollama or Hugging Face Transformers manually. And best of all: it plugs into Docker Desktop.

What I Tried to Build

I wanted a simple Python app that:

Sends a prompt to a local LLM using the /chat/completions API
Receives a response, all without touching OpenAI's cloud

Setup Troubles I Faced

Here are the real-world challenges I encountered:

1. Docker Desktop Version Was Too Old

At first, I was running Docker Desktop 4.39. This version showed experimental features like "Docker AI" and "Wasm," but didn't expose Docker Model Runner in the UI.

🔧 Fix: I upgraded to Docker Desktop 4.41, which finally showed "Enable Docker Model Runner" under the Beta Features tab.

2. Pulling the Model Fails Unless You Use the Right Name

Running this:

docker model pull mistral

resulted in:

401 Unauthorized

🔧 Fix: I checked Docker’s official documentation and found that model names are namespaced. I changed it to:

docker model pull ai/mistral

✅ This worked instantly.

3. TCP Port Refused Until I Enabled Host-Side Access

My Python script kept failing with:

ConnectionRefusedError: [WinError 10061] No connection could be made...

Even though the model was running in interactive mode.

🔧 Fix:I opened Docker Desktop → Features in Development → Enable Docker Model Runner
Then I scrolled down to find the checkbox: Enable host-side TCP support

Once enabled, the model ran with the HTTP API exposed on http://localhost:12434

Final Working Setup

Pull and Run the Model

docker model pull ai/mistral docker model run ai/mistral

This launches the model and exposes an OpenAI-compatible endpoint.

Test via Python

import requests

API_URL = "http://localhost:12434/engines/v1/chat/completions"
payload = {
    "model": "ai/mistral",
    "messages": [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Why is containerization important for deploying AI models?"}
    ]
}
headers = {"Content-Type": "application/json"}

response = requests.post(API_URL, headers=headers, json=payload)
print(response.json()["choices"][0]["message"]["content"])

Output: You’ll receive a well-formed response from the model — all running locally on your machine.

Final Thoughts

Docker Model Runner is incredibly promising — especially for developers who want to:

Build apps using LLMs without relying on cloud APIs
Save costs and protect sensitive data
Learn and prototype GenAI apps offline

But like many beta features, it takes some experimentation. If you're running into issues, check:

Docker version (must be 4.40+)
Correct model name (use the ai/ namespace)
TCP support enabled in the Model Runner settings

In the next article, I’ll explore integrating Docker Model Runner with a front-end GenAI UI.

👉 Stay tuned for Part 3

From Save to Serve: Boost LLM Dev Speed with Docker Compose Watch

Gowsiya Syednoor Shek — Sat, 10 May 2025 22:15:41 +0000

Next-Gen Docker for AI Developers: Series Intro

Welcome to my 6-part series exploring how Docker's latest features streamline AI and LLM-based application development.

Each part will show how to:

Accelerate development
Reduce manual steps
Improve containerized workflows for modern AI use cases

Part 1: Why Docker Compose Watch Is a Game Changer

As an AI developer working with LLM apps (LangChain, Flask APIs, etc.), I often deal with repetitive edit-rebuild-restart cycles. This gets especially frustrating while:

Tuning prompts
Testing temperature settings
Tweaking model configs
Updating dependencies

Even small changes can become time-consuming in a Dockerized workflow.

What Docker Compose Watch Promises

Docker Compose Watch solves this by:

Watching files or directories
Triggering live sync into the container (sync)
Or rebuilding when key files change (rebuild)
Supporting sync+restart for fast config reloading
Letting you stay in your flow — no manual restart or rebuilds

It sounded perfect for speeding up my LangChain + Flask API dev loop.

My Setup

I containerized a Flask API using LangChain to answer LLM queries.

Instead of manually rebuilding every time I changed a prompt or package, I enabled:

sync for Python code
rebuild for requirements.txt

Everything was orchestrated with docker-compose.override.yml using the new develop.watch feature.

🔗 GitHub Repo:

https://github.com/gowsiyabs/docker-llm-compose-watch

My Real-World Findings

Here’s what I observed after enabling Compose Watch for my LangChain + Flask setup:

Command used
docker compose -f docker-compose.yml -f docker-compose.override.yml up --build

Python code changes synced instantly

Once I saved the file, changes reflected in the running container without needing to restart. This was a big win for fast iteration.

Rebuild didn't work at first for requirements.txt

Initially, even after modifying requirements.txt, no rebuild was triggered. The console just kept showing the logs from the first run.

Fix: Enable watch support explicitly in your terminal

On Windows, I had to explicitly enable Docker’s watch support when using Git Bash or PyCharm's terminal. Once enabled, rebuilds started working as expected.

📌 Tip: Use a terminal that supports Docker watch triggers or set up your IDE to allow file system notifications to propagate correctly.

Lesson: Even though docker compose watch is powerful, it may require terminal or IDE-level configuration depending on your OS and environment.

Why This Matters for AI Developers

Compose Watch is ideal if you are:

Experimenting rapidly with LLM APIs
Testing prompts, chains, or model parameters
Tired of typing docker compose down && up every few minutes
Wanting a fast inner loop without sacrificing containerization

Summary

Docker Compose Watch takes us closer to hot-reload for containers, especially for interpreted languages like Python.

While rebuild actions still feel a bit rough around the edges, sync-based workflows are already a productivity booster for LLM, GenAI, and Flask-based AI apps.

🔜 Next in the Series

In Part 2, I will explore Docker Model Runner — a new way to run LLMs locally with Docker, without relying on cloud APIs or billing. Its available here now!

Follow me here on Dev.to to stay updated!