Classify CI Failures Before You Read the Logs: A Free-Tier Webhook Setup

CI failures are not all equal. Some are real bugs. Many are flakes that eat your afternoon. You don't need to read every log to tell them apart — a small model can do a first pass. The trick is making that pass reliable and observable, not another black box.

The community has been talking about AI turning every developer into a reviewer without testing the reviewer. This article is the opposite: we build a tiny classifier, deploy it for free, and then test it in CI.

The workflow in one sentence

A failed test step posts its log tail to a webhook. The webhook asks a free-tier model to return flake or bug with a one-line reason. The result lands in your PR as a comment.

Why MonkeyCode for this?

MonkeyCode is an open-source project that gives you 10 million free tokens and a free server for small HTTP services. That combination fits this use case perfectly: low traffic, short logs, and no persistent state.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The free server means you don't have to touch AWS or heroku. You write one file, deploy, and get a URL. The free tokens mean this classifier costs you nothing for realistic experiments. The exact quota may change, so check the repo before you rely on it.

Step 1: Define the contract

The webhook accepts a JSON POST:

{"log_tail": "ERROR: connection reset by peer..."}

And returns:

{"verdict": "flake", "reason": "Network error outside test scope."}

Keep it tiny. The model only needs the last few KB of a log.

Step 2: Deploy the classifier

Here's a minimal Python service. It's deliberately short. The model endpoint is a placeholder because MonkeyCode's API is subject to change — check the docs.

# classifier.py
import os, json, requests
from fastapi import FastAPI, Request

app = FastAPI()
MODEL = os.getenv('MONKEYCODE_MODEL', 'default-free')

SYSTEM = 'You classify CI failure log tails. Respond ONLY with JSON like {"verdict":"flake|bug|unknown","reason":"short reason"}.'

@app.post('/classify')
async def classify(req: Request):
    body = await req.json()
    tail = (body.get('log_tail') or '')[-4000:]
    resp = requests.post(
        'https://api.monkeycode.dev/v1/chat/completions',  # verify in docs
        headers={'Authorization': 'Bearer ' + os.environ['MONKEYCODE_API_KEY']},
        json={
            'model': MODEL,
            'messages': [
                {'role': 'system', 'content': SYSTEM},
                {'role': 'user', 'content': 'Log tail: ' + tail}
            ],
            'temperature': 0.2,
        },
        timeout=20,
    )
    result = resp.json()['choices'][0]['message']['content']
    return json.loads(result)

This is not production-grade error handling. It's enough for a thin experiment.

Step 3: Hook it into GitHub Actions

You don't need a special action. A regular workflow step can capture the log and curl the webhook.

- name: Run tests
  run: npm test 2>&1 | tee test.log
  continue-on-error: true

- name: Triage
  if: failure()
  env:
    WEBHOOK: ${{ secrets.CLASSIFIER_URL }}
  run: |
    tail -c 4000 test.log > tail.txt
    python - <<'PY'
import json, os, urllib.request
payload = {'log_tail': open('tail.txt').read()}
req = urllib.request.Request(
    os.environ['WEBHOOK'] + '/classify',
    data=json.dumps(payload).encode(),
    headers={'Content-Type': 'application/json'},
)
with urllib.request.urlopen(req) as resp:
    print(resp.read().decode())
PY

That prints the verdict in CI logs. You can then send it to a PR comment using actions/github-script if you want.

Step 4: Test the tester

This is the part most people skip. You don't ship a function without unit tests. Why ship a model call without fixtures?

Add a small test that runs inside CI:

# test_classifier.py
import os, requests

FIXTURES = [
    ('''error: Error: connect ECONNREFUSED 127.0.0.1:5432
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1141:16)''', 'flake'),
    ('''TypeError: Cannot read properties of undefined (reading 'map')
        at renderDashboard (Dashboard.jsx:42)''', 'bug'),
]

def test_verdicts():
    for log, expected in FIXTURES:
        r = requests.post(os.environ['CLASSIFIER_URL'] + '/classify',
                          json={'log_tail': log}, timeout=30)
        assert r.json()['verdict'] == expected, f'{log}: {r.json()}'

Run this on every PR. If the model's behavior drifts, your CI catches it.

Limitations

A free model will misclassify. The unknown verdict exists for a reason. Keep humans in the loop: the classifier only proposes, it never blocks a merge.

The free server may go cold. First request could take a few seconds. That's fine for a PR comment, not for a 99th-percentile SLA.

Your logs go to an external service. If your codebase has secrets in test output, you shouldn't use this as-is. Did you redact log lines? No? Then don't ship it.

Who should not use this

• Teams with strict data-residency requirements.
• Projects where misclassification could cause a security incident.
• Developers who want a merge gate. Don't make a probabilistic model your only gate. Use it as a triage aid, not a judge.

Try it

The setup is small enough to test in a weekend. Grab a free MonkeyCode token allowance, deploy the tiny server, and point a CI workflow at it. You'll learn more about your failures than the model does.