I Built a CI Failure Digest on a Free Server. The Model Was the Easy Part.

The hard part of my little on-call bot was never the model. It was deciding what deserved to wake me up. Everything else — the free model, the free server, the cron loop — took an afternoon.

I run a small side project with a handful of users. CI fails. Slack fills up. After enough "build broken" notifications, I stopped reading them. That's worse than having no alerts, because now I'm ignoring the one signal that actually matters. What's the point of a triage system you've learned to tune out?

So I built a digest. One page of Python, one scheduled task, zero dollars a month. The bot pulls the last 24 hours of failed CI jobs, classifies each one, and posts a short summary. If nothing is urgent, it stays quiet. That last part was the real requirement: silence is a feature, not a bug.

The experiment had a second goal. I wanted to know whether free infrastructure changes what you build — not in theory, but in practice. I pointed the script at MonkeyCode, an open-source project that bundles free model access with a free server option, which is exactly what this experiment needed. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The point wasn't to benchmark anything. The point was to see if a project that costs nothing still earns its keep.

Here's the shape of it. The code is illustrative, so treat it as a starting point, not a template. Your CI and your model endpoint will differ.

import json, os, subprocess, urllib.request

def recent_failures():
    out = subprocess.check_output([
        "gh", "run", "list", "--status", "failure",
        "--limit", "20", "--json", "databaseId,workflowName,headBranch,createdAt"
    ])
    return json.loads(out)

KNOWN_NOISE = ("docs", "stale", "flaky", "timeout")

def is_noise(run):
    name = (run.get("workflowName") or "").lower()
    return any(k in name for k in KNOWN_NOISE)

def classify(run):
    prompt = (
        "You triage CI failures for a solo developer. "
        "Reply with exactly one word: URGENT or WAIT. "
        f"Failed workflow: {run['workflowName']} on branch {run['headBranch']}."
    )
    payload = {
        "model": os.environ.get("LLM_MODEL", "default-free-model"),
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0,
    }
    req = urllib.request.Request(
        os.environ["LLM_BASE_URL"] + "/chat/completions",
        data=json.dumps(payload).encode(),
        headers={
            "Authorization": "Bearer " + os.environ["LLM_API_KEY"],
            "Content-Type": "application/json",
        },
    )
    with urllib.request.urlopen(req, timeout=30) as resp:
        return json.load(resp)["choices"][0]["message"]["content"].strip()

def main():
    urgent, waiting = [], []
    for run in recent_failures():
        if is_noise(run):
            waiting.append(run)
            continue
        (urgent if classify(run) == "URGENT" else waiting).append(run)
    if urgent:
        print("FIX NOW:", ", ".join(r["workflowName"] for r in urgent))
    else:
        print("Nothing urgent. Staying quiet.")

if __name__ == "__main__":
    main()

And the schedule:

*/15 * * * * cd ~/digest && python3 digest.py >> digest.log 2>&1

The whole trick is the pre-filter. Most failures never touch the model. A workflow named "docs-stale-link-check" doesn't need a language model to tell you it's noise — a three-line tuple catches it before a single token is spent. You pay for judgment, not for reading logs.

I ran this for about two weeks. I'm not going to quote an accuracy number, because the sample was too small to mean anything, and anyone who gives you a percentage after two weeks is selling something. What I can tell you is qualitative. I logged every decision into a JSON file, because that's the only way I trust my own memory. The log showed the pre-filter doing the heavy lifting. The model was mostly confirming what a grep could have told me — which sounds disappointing until you realize that's exactly the division of labor you want. The cheap deterministic layer catches the obvious stuff. The model handles the ambiguous leftovers.

Lesson one: the deterministic layer earns its keep. Prompt engineering is fun. A tuple of known noise strings is boring. Boring wins, because boring never hallucinates and never costs a token.

Lesson two: a free server is not a free lunch. The instance goes to sleep when it's idle. My first version kept state in memory and lost a week of history on the first restart. That's how I learned the second rule: stateless or die. Write the state to a file, or accept that you'll start over.

Lesson three: the model is a summarizer, not a decision-maker. I stopped calling it a triage agent. It's a note-taker with good grammar. The decision stays with me. That's not a limitation of the model — it's a feature of the design. When the bot says "URGENT," I still open the log and look. The bot's job is to make me look at the right thing, not to make the call for me.

Who should not build this? Anyone with a real on-call rotation. If a human is accountable for a missed alert, a free tier is not your SLA. Don't build it if your CI fails twice a month, either — a cron job that emails you the raw log is simpler and more honest. And don't build it if you're hoping the model will replace your judgment. It won't. It'll just organize the noise so you can use yours.

A few caveats. Free quotas change. The numbers that were true when I wrote this are probably stale by the time you read it — check the current docs before you trust any allowance. The code above is illustrative, and the endpoint I used follows the OpenAI-compatible shape, so adapt it to whatever you're actually calling. Also, put the key in an environment variable, not in the script. I shouldn't have to say that, but I've seen worse.

Would I do it again? Yes, but with the pre-filter first and the prompt second. That's the order that made this project cost nothing and still feel useful. If you want to see how far free model access actually gets you, this is a cheap way to find out — fork it, run it for a week, and read your own log. The log will tell you more than I can.