Zero-Cost CI Flake Triage: Build Notes from a Free-Tier Weekend

Every AI-coding discussion this week keeps landing on the same bottleneck: the reviewer. More generated code reaches merge queues, and CI failures make the review signal noisier. A small webhook bot was built over one weekend to classify those failures automatically — running on a free server and free model tokens. The scope got cut hard, the demo ran, and the mislabels were more instructive than the hits.

Why flake triage became the test case

The current AI-tooling debate splits into two camps: teams that trust model output and teams that benchmark every claim. Both camps still share one pain point — flaky tests. A timeout, a leaked database state, an order-dependent test: each one interrupts a review and burns a human decision. A classifier that labels the flake type before a human opens the logs is a small but honest use of a language model.

The previous weekend project digested repositories. This one watches CI events instead. Two different bots, one shared lesson: free-tier capacity is enough for narrow tasks if the task stays narrow.

The scope cut, decided before the first line of code

The original plan had five features. The shipped version had two.

Planned	Shipped	Reason for the cut
Multi-repo webhooks	Single repo	Webhook auth config is its own project
Persistent history	In-memory queue	One weekend, no Redis
Slack alerting	PR label only	Fewer moving parts
Prompt versioning UI	Hardcoded prompt	Over-engineering
Model fallback on parse errors	Single retry	Free-tier constraint

The rule was simple: if a feature needed another database, another credential, or another UI, it did not ship.

Architecture: one server, two routes, one prompt

The whole bot is a Node process with two endpoints. POST /ci-event receives a test name and the tail of the failure log. The classifier maps the text to one of five buckets: TIMING, ORDER_DEPENDENT, NETWORK, STATE_LEAK, or UNKNOWN. A comment or label then lands on the PR.

The classification function is deliberately small:

// Sketch, not production code. Match the client to your endpoint's SDK.
const FLAKE_BUCKETS = ["TIMING", "ORDER_DEPENDENT", "NETWORK", "STATE_LEAK", "UNKNOWN"];

async function classifyFlake(failureText) {
  const prompt = [
    "You are a CI triage assistant. Put the failure below into exactly one bucket.",
    "TIMING: timeout or slow wait. ORDER_DEPENDENT: passes alone, fails in the suite.",
    "NETWORK: DNS, TLS, or external call. STATE_LEAK: shared DB or files, missing reset.",
    "Reply with one word only.",
    "",
    failureText.slice(0, 2000)
  ].join("\n");

  const reply = await model.chat({ prompt });
  return FLAKE_BUCKETS.includes(reply.trim()) ? reply.trim() : "UNKNOWN";
}

The route adds no cleverness:

app.post("/ci-event", async (req, res) => {
  const { testName, logTail } = req.body;
  const bucket = await classifyFlake(logTail);
  await labelPr(testName, bucket);
  res.json({ ok: true, bucket });
});

Two details matter. First, the prompt constrains the model to a fixed vocabulary, so parsing is a one-line check instead of regex wrangling. Second, any out-of-vocabulary reply becomes UNKNOWN, which keeps the bot honest instead of confident.

The free tier: where the server and the tokens came from

Hosting came from the free server option in MonkeyCode, so the webhook never touched a paid VM. The model calls ran against the free model tier, which lists 10 million tokens at the time of writing. A weekend of triage traffic for a single repo stays well inside that envelope, but the quota should be re-checked before pointing the bot at a busier pipeline.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The point of the exercise was not that free hosting exists. It was that a narrow task with a constrained prompt can run without card-on-file anxiety. The trade-offs showed up quickly: the free endpoint had no latency guarantee, and one malformed JSON reply required a retry. That is the realistic cost of a zero-dollar stack.

The demo: one request, one label

A real-looking failure from the local test run:

POST /ci-event
{
  "testName": "checkout_test",
  "logTail": "connect ETIMEDOUT 52.0.1.1:443\n    at TCPConnectWrap.afterConnect"
}

Response:

{ "ok": true, "bucket": "NETWORK" }

The obvious cases classify instantly. The interesting cases are the UNKNOWNs. Every unknown is a prompt bug or a genuinely weird failure, and both are worth a human look.

The evaluation plan worth copying

Model confidence is not evidence. The reproducible check from this project is a 30-sample manual comparison:

1. Pull 30 real failures from the repo's CI history.
2. Label each one by hand before running the classifier.
3. Run the bot over the same 30 logs.
4. Build a confusion matrix by bucket.
5. Keep the matrix somewhere visible.

This takes about an hour and produces a number the team can trust — unlike a marketing chart. For a small repo, 25–30 samples is enough to find systematic mislabeling in the two buckets that matter most.

What got skipped, and why that was correct

• No authentication on the webhook endpoint. Fine for a personal demo, wrong for anything shared.
• No durable queue. A server restart drops in-flight events.
• No metrics dashboard. The confusion matrix script replaced it.
• No multi-model comparison. The budget was one free endpoint, not a benchmark lab.
• No persistence beyond memory. Labels are re-derivable from CI history.

Who should not copy this setup

Teams with compliance constraints should not send failure logs to any third-party model endpoint without checking data classification rules. Repos with very few flakes do not need a classifier; the noise of a mislabel outweighs the time saved. Anyone who needs a response within a strict SLA should pay for a managed queue and a guaranteed tier instead. Free-tier tooling is a decision, not a default.

The narrowest version of this project is the useful one: a single repo, a constrained prompt, and an evaluation script. If the same 30-sample check is run on another CI, the flake distribution that comes back will likely surprise the team that owns it. That confusion matrix is the part of this weekend worth copying.