LLM-Based CI/CD Pipeline Best Practices

#engineering #oxlo #ai

We are building a CI/CD gate script that sends git diffs to an LLM for code review, then blocks or allows the deploy based on structured feedback. It helps small teams ship faster without running a separate review service.

What you'll need

Python 3.10+
The OpenAI SDK: pip install openai
Git
An Oxlo.ai API key from https://portal.oxlo.ai

Step 1: Bootstrap the project

First, I need a way to capture the diff. I will use a Python script that shells out to git so it works in any repo without extra dependencies.

import subprocess
import sys

def get_diff(base_branch="main"):
    result = subprocess.run(
        ["git", "diff", f"{base_branch}...HEAD"],
        capture_output=True,
        text=True,
    )
    if result.returncode != 0:
        print("Failed to get diff. Are you in a git repo?")
        sys.exit(1)
    return result.stdout

if __name__ == "__main__":
    diff = get_diff()
    print(f"Diff length: {len(diff)} characters")

Step 2: Define the review agent

The system prompt is the contract with the model. I keep it in a constant so I can tune severity or JSON schema without touching the rest of the script.

SYSTEM_PROMPT = """You are a CI/CD code reviewer. Analyze the provided git diff and respond with a single JSON object. Do not wrap the JSON in markdown.

Rules:
- "verdict" must be "pass" or "fail".
- "issues" is a list of objects, each with "severity" (critical, warning, info), "file", and "description".
- "tests_needed" is a boolean indicating whether new unit tests are required.
- Be strict about security issues, race conditions, and unhandled exceptions.

Example output:
{
  "verdict": "fail",
  "issues": [
    {"severity": "critical", "file": "auth.py", "description": "Hardcoded API key introduced."}
  ],
  "tests_needed": true
}
"""

Step 3: Call Oxlo.ai for diff review

Now I send the diff to Oxlo.ai. I use llama-3.3-70b because it handles long context well, and with Oxlo.ai's flat per-request pricing a large diff costs the same as a small one. That matters when refactoring PRs touch dozens of files.

import json
from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

def review_diff(diff_text):
    if not diff_text.strip():
        return {"verdict": "pass", "issues": [], "tests_needed": False}

    user_message = f"Review this diff:\n\n{diff_text}"

    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
    )

    content = response.choices[0].message.content.strip()
    if content.startswith("

```"):
        content = content.split("\n", 1)[1]
    if content.endswith("```

"):
        content = content.rsplit("\n", 1)[0]
    return json.loads(content.strip())

Step 4: Parse the verdict and gate the build

The CI runner needs a pass/fail signal. I parse the JSON, print a summary, and exit with a non-zero code when the verdict is fail. This halts the pipeline immediately.

import sys

def gate_deploy(report):
    issues = report.get("issues", [])
    critical = sum(1 for i in issues if i.get("severity") == "critical")
    warnings = sum(1 for i in issues if i.get("severity") == "warning")

    print(f"Verdict: {report.get('verdict')}")
    print(f"Critical issues: {critical}")
    print(f"Warnings: {warnings}")

    for issue in issues:
        print(f"  [{issue['severity']}] {issue['file']}: {issue['description']}")

    if report.get("verdict") == "fail":
        print("Build gated. Fix issues before deploying.")
        sys.exit(1)

    print("Build cleared.")
    sys.exit(0)

Step 5: Generate missing tests

If the reviewer flags missing tests, I call Oxlo.ai again. I switch to qwen-3-32b here because it is strong at agentic coding tasks, then write the output to a file so the engineer can inspect it.

TEST_PROMPT = """You are a pytest expert. Given a git diff, generate focused unit tests for the new or changed functions. Output only valid Python test code. Use pytest. Mock external dependencies."""

def generate_tests(diff_text):
    user_message = f"Write tests for this diff:\n\n{diff_text}"

    response = client.chat.completions.create(
        model="qwen-3-32b",
        messages=[
            {"role": "system", "content": TEST_PROMPT},
            {"role": "user", "content": user_message},
        ],
    )

    return response.choices[0].message.content.strip()

if __name__ == "__main__":
    diff = get_diff()
    report = review_diff(diff)

    if report.get("tests_needed"):
        print("Generating tests...")
        tests = generate_tests(diff)
        with open("generated_tests.py", "w") as f:
            f.write(tests)
        print("Wrote generated_tests.py. Review before committing.")

    gate_deploy(report)

Run it

Save the assembled script as ci_gate.py, replace YOUR_OXLO_API_KEY, and run it against a feature branch.

$ python ci_gate.py

Diff length: 1243 characters
Generating tests...
Wrote generated_tests.py. Review before committing.
Verdict: fail
Critical issues: 1
Warnings: 0
  [critical] payment.py: Unvalidated user input passed directly to SQL formatter.
Build gated. Fix issues before deploying.

Next steps

Wire python ci_gate.py into a GitHub Actions job or GitLab CI script so it runs on every pull request. If you want to experiment at zero cost, swap the review model to deepseek-v3.2 on Oxlo.ai's free tier, then promote to llama-3.3-70b for main branch protections.

DEV Community