Flaky Tests Are Killing Your Merge Confidence: Build a Flake-Aware CI Gate

Flaky tests are kinetic energy for CI. They add minutes, hide regressions, and make every green check feel like a coin flip. You can't fix flakiness by retrying everything; you fix it by knowing which reds are noise and which are real.

As AI-generated code lands faster, the pipeline is the only safety net left. And flaky tests are tearing holes in it. So let's build a gate that tells the difference.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The idea is simple: when a test fails, send the failure log to a small model. Ask it to classify the failure as flake or regression, with a confidence score. If it's a flake, retry once with a seeded fixture. If the retry passes, mark the job as green but record the flake event. If the flake rate for that test goes over a threshold, block the merge until someone investigates.

This is a three-part workflow:

1. A GitHub Actions step that captures the failed test output.
2. A Python script that calls MonkeyCode's free model API and returns a structured verdict.
3. A tiny storage endpoint on MonkeyCode's free server that keeps the flake history.

You can copy the code below, swap in your own keys, and start taming your pipeline.

Step 1: Capture the failure context

GitHub Actions gives you the test report as an artifact. For this gate, we want the raw failure snippet and the test name. The simplest way is to run your test suite with --junitxml (pytest) or equivalent, then parse it.

Here's a reusable action step:

- name: Run tests
  id: tests
  run: pytest --junitxml=report.xml
  continue-on-error: true

- name: Extract failure context
  id: failure_context
  if: steps.tests.outcome == 'failure'
  run: |
    python extract_failure.py report.xml > failure.json
    echo "failure_file=failure.json" >> "$GITHUB_OUTPUT"

The extract_failure.py script pulls the first failure line, the test name, and the stack trace. Save it in your repo.

Step 2: Ask a model to classify the failure

Now we send that context to MonkeyCode's free model. This is the part that separates a "flaky" red from a "real" red. We don't need a huge model; a small one is fast and cheap for this classification task.

Here's classify_flake.py:

import json
import sys
import requests

MONKEYCODE_API_URL = "https://api.monkeycode.example/v1/classify"  # check docs
MONKEYCODE_API_KEY = sys.argv[1]  # from secrets
failure_data = json.load(open(sys.argv[2]))

payload = {
    "task": "classify_ci_failure",
    "input": {
        "test_name": failure_data["test_name"],
        "failure_line": failure_data["failure_line"],
        "stack_trace": failure_data["stack_trace"][:2000],
        "repo": failure_data["repo"],
        "branch": failure_data["branch"]
    }
}

resp = requests.post(MONKEYCODE_API_URL, json=payload, headers={
    "Authorization": f"Bearer {MONKEYCODE_API_KEY}"
})
verdict = resp.json()
print(json.dumps(verdict))  # {"is_flake": true, "confidence": 0.93, "reason": "Timeout at 30s, rerun passed"}

Yes, that's a placeholder URL. Check the current MonkeyCode docs for the exact endpoint and model names — free tiers change.

The model returns is_flake and confidence. If confidence is high and is_flake is true, we safely retry once.

Step 3: Retry or block with a gate

Now we glue it into the job. Here's a shell snippet that reads the verdict and decides:

VERDICT=$(python classify_flake.py "$API_KEY" failure.json)

if echo "$VERDICT" | grep -q '"is_flake": true'; then
  echo "Flake detected — retrying once with seed=42"
  pytest --seed 42 tests::FailedTest
else
  echo "Likely regression — failing the build"
  exit 1
fi

The seed is your fixture control. Enabling repeatable runs is the difference between gambling and gatekeeping.

But retrying means the green check is still a "maybe" — that's why we log the flake event.

Step 4: Record flakes on a free server

MonkeyCode's free server can host a tiny webhook that stores every flake verdict. You don't need to manage Postgres. Here's a minimal FastAPI app you can deploy in the free tier:

from fastapi import FastAPI, Request
import sqlite3

app = FastAPI()
conn = sqlite3.connect("flakes.db", check_same_thread=False)

@app.post("/flake")
async def record_flake(req: Request):
    data = await req.json()
    conn.execute(
        "INSERT INTO flakes (test_name, reason, confidence, repo) VALUES (?,?,?,?)",
        (data["test_name"], data["reason"], data["confidence"], data["repo"])
    )
    conn.commit()
    return {"ok": True}

@app.get("/flake/{test_name}")
def flake_count(test_name: str):
    cur = conn.execute("SELECT COUNT(*) FROM flakes WHERE test_name=?", (test_name,))
    return {"count": cur.fetchone()[0]}

Then in CI, after a flaky retry passes, send a POST:

curl -X POST https://your-free-server.example/flake \
  -H "Content-Type: application/json" \
  -d "{\"test_name\":\"test_user_can_login\",\"reason\":\"timeout?\",\"confidence\":0.93,\"repo\":\"$GITHUB_REPOSITORY\"}"

Now you have a per-test flake counter. Add a threshold in CI: if a test flakes more than 3 times in 24 hours, the merge fails until someone writes a fix.

COUNT=$(curl -s https://your-free-server.example/flake/test_user_can_login | jq .count)
if [ "$COUNT" -gt 3 ]; then
  echo "Too many flakes. Investigate before merging."
  exit 1
fi

Congratulations: you've turned retries from a blind habit into a measurable signal.

When you should not do this

This gate adds latency (one extra API call per failure) and a dependency on an external model. It's overkill for:

• Small projects where CI finishes in 2 minutes and flakes are rare.
• Teams that prefer to fix root causes immediately.
• Pipelines that run hundreds of tests per minute — the API call would be too slow.

Use this only when flaky failures are a daily occurrence and you're tired of manually scrolling logs.

The bottom line

Flaky tests won't disappear. But you can stop treating every red as the same category. By adding a free model to your CI hook and keeping history on a free server, you get a pipeline that is honest: green means the failure was understood, and red means someone actually needs to read it.

That's the green-to-merge path worth building.