DEV Community

Riley Li
Riley Li

Posted on

A Credit-Card-Free Triage Bot for Noisy CI Logs

#tutorial #ai #automation #devops

I used to start my week by deleting CI emails.

Not reading them—deleting. There were too many. A broken container here, a flaky UI test there, the same timeout from a third-party API. Somewhere in that pile was a real failure I actually needed to fix, but finding it felt like panning for gold in a river of noise.

So I wanted a small triage bot.

Not an autonomous agent that fixes my pipeline. I don't trust anything to start editing workflows after three failed runs. I wanted something much dumber: a service that reads a failed CI log and puts it into one of three buckets.

  • flake: the test failed on timing, an assertion order, or a retry would probably pass.
  • infra: the runner died, the network timed out, or a dependency couldn't be pulled.
  • code: the build or test surfaced something a developer should look at.

That's it. Three labels. No auto-remediation. No magic.

The problem was infrastructure. I didn't want to host another service, open a cloud account, or tie my personal credit card to something I'd use twice a week. My local machine could run a small model, but I didn't want to give up the RAM every time a webhook arrived.

That's where the free model and free server piece came in.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and a free server option are the two pieces I used to build this workflow.

I'm going to keep the model endpoint and server details generic. Free-tier names, tokens, and runtime limits change, and I'd rather show the pattern than tie it to a signup flow that might be different next month. If you want to reproduce this, plug in your own free model URL and free server host.

The shape of the workflow

Instead of paying for a monitoring dashboard, I wired up a tiny service.

  1. CI sends a webhook on failure.
  2. The free server receives the log excerpt.
  3. It scrubs obvious secrets before forwarding anything.
  4. The free model reads the log and returns one of the three labels.
  5. The server stores the result and exposes a simple /last endpoint.

The key is that each step is replaceable. The webhook can come from GitHub Actions, Jenkins, CircleCI, or a cron job that checks an API. The model doesn't need to be large. The task is classification on a short log, not code review.

The reusable part

Here's the server code I sketched. It's deliberately small, so you can paste it into app.py and run it with uvicorn app:app.

import os
from collections import defaultdict
import requests

from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse

app = FastAPI()

MODEL_URL = os.getenv('MODEL_URL', '')
MODEL_TOKEN = os.getenv('MODEL_TOKEN', '')
MODEL_NAME = os.getenv('MODEL_NAME', '')
ALLOWED_LABELS = {'flake', 'infra', 'code'}
SENSITIVE_MARKERS = ['api_key=', 'apikey=', 'token=', 'secret=', 'password=']

results = defaultdict(int)

def scrub(text: str) -> str:
    for marker in SENSITIVE_MARKERS:
        start = 0
        while True:
            idx = text.lower().find(marker, start)
            if idx == -1:
                break
            end = idx + len(marker)
            while end < len(text) and not text[end].isspace() and text[end] not in '&;':
                end += 1
            text = text[:idx + len(marker)] + '[REDACTED]' + text[end:]
            start = idx + len(marker) + len('[REDACTED]')
    return text

def classify(log: str) -> str:
    if not MODEL_URL:
        return 'unknown'
    prompt = (
        'You are triaging a failed CI log. Classify the failure as exactly one of: '
        'flake, infra, code. Return only the label. Log excerpt: ' + scrub(log)[:4000]
    )
    resp = requests.post(
        MODEL_URL,
        headers={'Authorization': f'Bearer {MODEL_TOKEN}'} if MODEL_TOKEN else {},
        json={'model': MODEL_NAME, 'prompt': prompt, 'max_tokens': 8},
        timeout=20,
    )
    resp.raise_for_status()
    label = resp.json().get('choices', [{}])[0].get('text', '').strip().lower()
    if label not in ALLOWED_LABELS:
        return 'unknown'
    return label

@app.post('/triage')
async def triage(request: Request):
    body = await request.json()
    log = body.get('log', '')
    label = classify(log)
    results[label] += 1
    return JSONResponse({'label': label, 'counts': dict(results)})

@app.get('/last')
async def last():
    return JSONResponse(dict(results))
Enter fullscreen mode Exit fullscreen mode

This is a sketch, not a copy-paste deployment. The payload format depends on the model provider, and some free servers force cold starts.

How I decided what counts as 'code'

A model will happily output code for any log if you let it, so a small decision table keeps the bot honest.

Signal in log Suggested label Why
Timed out, retry, flaky, expected X got Y on UI test flake Often a timing or DOM assertion issue, not a logic bug.
Docker pull, network unreachable, 503, memory limit, runner version infra The failure happened before your code could be judged.
TypeError, ImportError, AssertionError, stack trace in your package code The code itself broke.

You can turn this into a second pass: if the model says code but none of the code signals are present, drop the label to unknown.

The sanity check I'd run before trusting it

I would not deploy this on a real repo without a tiny ground-truth set.

  1. Take 20 old failed logs.
  2. Label them myself: 8 flake, 8 infra, 4 code.
  3. Run the triage endpoint and compare.
  4. Require 80%+ agreement before it gets to touch my inbox.
  5. Failing that, narrow the prompt or shorten the log excerpt.

Where this falls apart

Free model access and free servers are great for this, but they're not a production control plane.

  • Free servers may sleep, cold start, or get recycled. Your webhook can time out.
  • Free model quotas can be rate-limited, so a sudden burst of failing jobs may drop requests.
  • The model can still hallucinate a label. Bad classification is annoying; worse, it can teach you to ignore the bot.
  • Logs aren't free to share. Even with scrubbing, there can be repo names, usernames, IPs, and code snippets inside the trace.

I treat this as a noise filter, not a decision maker. It changes where I look first; it doesn't tell me what to ignore.

Who should skip this

  • If your logs contain regulated data, don't send them to a free model.
  • If you need an audit trail, make sure your logs and model calls are retained; a stateless free service won't do that by default.
  • If you already have a paid CI analytics tool, this probably duplicates part of it.
  • If you want an agent that auto-fixes failures, this is not that. It only labels.

If you have a free model endpoint and a free server lying around, start with the 20-log sanity check. It's the cheapest way to find out whether the signal is real before you bolt on more machinery.

Top comments (0)