shashank ms

Posted on Sep 23

LLM Containerization Best Practices

#engineering #oxlo #ai

I recently shipped a containerized code review agent that reads git diffs and returns structured feedback via Oxlo.ai. In this tutorial, I will walk through the exact Dockerfile, compose setup, and agent code I use in production. If you want to run LLM workloads in reproducible, secure containers, this should save you a few hours of trial and error.

What you’ll need

Python 3.10 or newer
Docker Engine 24.0+
An Oxlo.ai API key from https://portal.oxlo.ai
The OpenAI SDK: pip install openai

Step 1: Scaffold the project

I keep the layout flat: one Python file, a requirements manifest, and a directory for input diffs. Below are the commands to create them.

mkdir oxlo.ai-code-review-agent
cd oxlo.ai-code-review-agent
python -m venv .venv
source .venv/bin/activate

cat > requirements.txt << 'EOF'
openai>=1.30.0
python-dotenv>=1.0.0
EOF

mkdir diffs

Step 2: Write the agent script and system prompt

The agent reads a diff file, sends it to Oxlo.ai, and prints the review. I use the OpenAI SDK with Oxlo.ai’s base URL so I can swap models later without touching client code.

Here is the system prompt I use:

SYSTEM_PROMPT = """You are a senior staff engineer performing code review.
Review the provided git diff for bugs, security issues, and style problems.
Output your findings as a concise bulleted list. If the diff looks good, say 'LGTM'.
"""

And the full agent script:

import os
import sys
from openai import OpenAI
from dotenv import load_dotenv

load_dotenv()

SYSTEM_PROMPT = """You are a senior staff engineer performing code review.
Review the provided git diff for bugs, security issues, and style problems.
Output your findings as a concise bulleted list. If the diff looks good, say 'LGTM'.
"""

def review_diff(diff_path: str) -> str:
    with open(diff_path, "r") as f:
        diff_content = f.read()

    if not diff_content.strip():
        return "No diff content found."

    client = OpenAI(
        base_url="https://api.oxlo.ai/v1",
        api_key=os.environ["OXLO_API_KEY"],
    )

    response = client.chat.completions.create(
        model="qwen-3-32b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": diff_content},
        ],
    )

    return response.choices[0].message.content

if __name__ == "__main__":
    if len(sys.argv) != 2:
        print("Usage: python agent.py <path-to-diff>")
        sys.exit(1)

    print(review_diff(sys.argv[1]))

Step 3: Add a hardened Dockerfile

I use a multi-stage build to keep the final image small, run as a non-root user, and avoid caching secrets in layers. Pin the base image to a specific digest in production.

# syntax=docker/dockerfile:1
FROM python:3.11-slim AS builder

WORKDIR /app
RUN python -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

FROM python:3.11-slim

RUN groupadd -r appgroup && useradd -r -g appgroup appuser

WORKDIR /app
COPY --from=builder /opt/venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"

COPY agent.py .

RUN chown -R appuser:appgroup /app
USER appuser

ENTRYPOINT ["python", "agent.py"]

Step 4: Configure secrets via environment variables

Never bake the Oxlo.ai API key into the image. I use a .env file for local testing and pass variables at runtime.

cat > .env.example << 'EOF'
OXLO_API_KEY=your_oxlo_key_from_portal_oxlo_ai
EOF

cp .env.example .env
# Edit .env and paste your real key from https://portal.oxlo.ai

Step 5: Add Docker Compose with a health check

Compose simplifies local runs and lets me mount diffs without rebuilding. I also add a basic health check so orchestrators can verify the container state.

cat > docker-compose.yml << 'EOF'
services:
  agent:
    build: .
    env_file:
      - .env
    volumes:
      - ./diffs:/app/diffs:ro
    healthcheck:
      test: ["CMD", "python", "-c", "import sys; sys.exit(0)"]
      interval: 30s
      timeout: 5s
      retries: 3
EOF

Step 6: Build and test the container

Create a sample diff that introduces a SQL injection risk, then build the image and run the agent. The container should exit after printing the review.

cat > diffs/sample.diff << 'EOF'
diff --git a/app.py b/app.py
index 123..456 789
--- a/app.py
+++ b/app.py
@@ -10,5 +10,5 @@ def handle(request):
-    password = request.args.get('password')
-    query = f"SELECT * FROM users WHERE pw = '{password}'"
+    password = request.args.get('password', '')
+    query = "SELECT * FROM users WHERE pw = ?"
     cursor.execute(query, (password,))
EOF

docker compose build
docker compose run --rm agent diffs/sample.diff

Run it

Here is the terminal output I see when running the sample diff through the container. Oxlo.ai returns a concise review that flags the security issue and confirms the fix.

$ docker compose run --rm agent diffs/sample.diff

- Lines 12-13: The original code is vulnerable to SQL injection because user input is interpolated directly into the query string.
- Line 13: Switching to a parameterized query fixes the injection vector. Good fix.
- Consider validating the password length before hitting the database.

Overall: LGTM after confirming tests cover the new query path.

Wrap-up

Next, wire this container into your CI pipeline so every pull request triggers a review automatically. After that, consider deploying to Kubernetes with an External Secrets Operator to manage the Oxlo.ai key outside the cluster.

DEV Community