Stop LLM Prompt Regressions in CI/CD: An Open-Source Guide using LLM-as-a-Judge

Mukund Jha — Wed, 26 Aug 2026 06:11:14 +0000

Stop LLM Prompt Regressions in CI/CD: An Open-Source Guide using LLM-as-a-Judge

If you are building AI features, you have probably experienced this nightmare:

You tweak one line in your system prompt to fix a minor edge case. You test it twice in the OpenAI Playground, and it looks great. You push to production.

The next day, you find out that your "harmless" tweak caused the LLM to hallucinate a 60-day refund policy (instead of 30) and start leaking competitor names.

Traditional unit tests (assert output == "exact string") don't work because LLMs are non-deterministic. You end up manually testing 20 edge cases before every deploy, hoping you didn't break anything.

This is called Prompt Regression, and it is the biggest bottleneck preventing AI apps from moving fast.

In this article, I’ll show you how to use an architecture called LLM-as-a-Judge to automatically test your AI outputs, and I'll introduce an open-source Python CLI I built called Crilio that integrates this directly into your GitHub Actions.

TL;DR
The Problem: LLMs are non-deterministic; traditional tests fail.
The Solution: Use a fast, cheap model (like GPT-4o-mini) to "judge" your main model's outputs against strict rules.
The Tool: Crilio is a free, open-source CLI that automates this in CI/CD.

What is LLM-as-a-Judge?
Since we cannot do exact string matching on generative AI, we have to evaluate outputs semantically.

The "LLM-as-a-Judge" pattern uses a fast, cheap LLM to grade the output of your main LLM.

Instead of writing Python code to check if a string contains a word, you write a rule in plain English:

"Must mention the 30-day return window."
"Must NOT mention competitor names like Amazon or Walmart."

You send your Target LLM's response and the rule to the Judge LLM. The Judge returns a strict JSON object: {"rule_passed": false, "reason": "Mentioned Amazon"}.

If the Judge says the rule failed, your CI/CD pipeline fails.

How to Implement This in CI/CD
I was manually testing prompts before every git push, and it was driving me crazy. So I built Crilio, a lightweight Python CLI that acts like Jest for LLM prompts.

It uses the BYOK (Bring Your Own Key) model, so you use your own OpenAI or Anthropic keys, and it costs fractions of a cent to run.

Here is how you set it up in under 2 minutes.

1. Install and Initialize
Install the CLI via PyPI:

pip install crilio
crilio init

2. Define Your Test Rules
Open the crilio.yaml file and write your test prompts and the strict rules the AI must follow:


yaml

crilio.yaml
tests:
  - name: "Refund Policy Check"
    prompt: "How long do I have to return a product?"
    rules:
      - "Must mention the 30-day return window."
      - "Must NOT mention competitor names like Amazon or Walmart."
      - "Tone must be polite and professional."

3. Run the Tests Locally
Add your API key to your environment variables:

export OPENAI_API_KEY="sk-..."

4. Run the tool:

crilio run

Crilio will call your Target LLM (e.g., GPT-4o), capture the response, and send it to the Judge LLM (GPT-4o-mini). If the AI broke a rule, Crilio prints a beautiful red ❌ report in your terminal and exits with an error code.

4. The Gatekeeper: GitHub Actions
The real magic happens when you put this in your CI/CD pipeline. Create a file at .github/workflows/crilio.yml in your repo:

yaml

name: Crilio AI Tests
on: [pull_request]
jobs:
  test:

    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v4
        with:
          python-version: '3.10'
      - run: pip install crilio
      - run: crilio run
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Now, whenever a developer creates a Pull Request, GitHub automatically runs Crilio. If their prompt change causes the AI to hallucinate or break a rule, the GitHub Action fails. The PR is blocked from merging.

You have just turned your non-deterministic AI into a tested, reliable software component.

Why This Works Better Than Manual Testing

Catches Cascading Breakages: You might fix a refund bug, but accidentally break the shipping policy. The Judge catches both.

Zero Infrastructure: Because Crilio uses BYOK, you don't need to host a database or an evaluation server. It runs entirely in your GitHub Actions runner.

Developer Experience: Developers don't have to learn a complex observability platform. They just write a YAML file.

The Roadmap

Crilio is completely free and open-source (AGPL). Right now, it supports OpenAI and Anthropic, and it runs perfectly in the terminal and CI/CD.

The next step is building a Cloud Dashboard so teams can view historical regression data over time (which will be the paid tier).

Give it a Try

If you are tired of playing whack-a-mole with your AI prompts, I’d love for you to try Crilio and give me your brutal feedback.

GitHub Repo: https://github.com/mukundzha/crilio
PyPI: pip install crilio

If you found this helpful, drop a ⭐ on the repo—it helps other developers find it! Let me know in the comments how you are currently handling prompt testing in your CI/CD pipelines.And also give me some suggestions regarding upcoming features.

DEV Community: Mukund Jha

Stop LLM Prompt Regressions in CI/CD: An Open-Source Guide using LLM-as-a-Judge

Stop LLM Prompt Regressions in CI/CD: An Open-Source Guide using LLM-as-a-Judge