DEV Community

Sam Rivera
Sam Rivera

Posted on

I Stopped Paying for LLM Calls in My Side Project's CI: A One-Task Canary Setup

#ai #typescript #cli #indiehacker

My side project makes LLM calls in exactly one place: a CLI that drafts release notes from merged PR titles. The feature works. The problem was that every CI run and every local test was hitting a paid API, and my "it's just a few cents" spreadsheet had quietly become "it's a few dollars a month for a repo three people look at."

So I set a boring constraint: the release-notes canary in CI must cost $0, and if the free path degrades, I abandon the feature in CI rather than pay. This post is the setup that survived that constraint.

The actual goal

Not "use an LLM in CI" — that's easy. The goal was narrower:

  1. One deterministic task (draft release notes from PR titles).
  2. A provider abstraction so the model backend is swappable in one env var.
  3. A failure fixture that proves the canary actually catches bad output.
  4. A written abandonment criterion, decided before I got attached to the feature.

Step 1: One provider interface, no SDK lock-in

The whole trick is refusing to let any vendor's SDK into the repo. The CLI talks to an OpenAI-compatible /chat/completions endpoint, because that shape is the closest thing to a common denominator:

// llm.ts — the entire abstraction. ~40 lines.
export interface LlmConfig {
  baseUrl: string;      // any OpenAI-compatible endpoint
  apiKey: string;       // can be a dummy value for some servers
  model: string;
  timeoutMs: number;
}

export async function draftReleaseNotes(
  prTitles: string[],
  cfg: LlmConfig
): Promise<string> {
  const controller = new AbortController();
  const timer = setTimeout(() => controller.abort(), cfg.timeoutMs);

  try {
    const res = await fetch(`${cfg.baseUrl}/chat/completions`, {
      method: "POST",
      signal: controller.signal,
      headers: {
        "Content-Type": "application/json",
        Authorization: `Bearer ${cfg.apiKey}`,
      },
      body: JSON.stringify({
        model: cfg.model,
        temperature: 0,
        messages: [
          {
            role: "system",
            content:
              "You draft release notes. Output ONLY a markdown bulleted list. " +
              "Group items under Added / Fixed / Changed. Never invent features.",
          },
          {
            role: "user",
            content: prTitles.map((t) => `- ${t}`).join("\n"),
          },
        ],
      }),
    });

    if (!res.ok) {
      throw new Error(`LLM HTTP ${res.status}: ${await res.text()}`);
    }
    const data = await res.json();
    return data.choices[0].message.content as string;
  } finally {
    clearTimeout(timer);
  }
}
Enter fullscreen mode Exit fullscreen mode

fetch and a JSON body. No openai package, no vendor SDK, nothing to upgrade when a provider renames a field.

Step 2: The free backend

For the zero-cost lane I used MonkeyCode, which offers free model access and a free server option — enough surface for a single low-volume canary task like this one. Because it exposes an OpenAI-compatible API, the config is just three environment variables:

# .env.ci — the entire "migration"
LLM_BASE_URL=https://<your-monkeycode-endpoint>/v1
LLM_API_KEY=<key from the MonkeyCode dashboard>
LLM_MODEL=<a model available on the free tier>
LLM_TIMEOUT_MS=20000
Enter fullscreen mode Exit fullscreen mode

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The reason it fit this article honestly is that the interface above doesn't care which OpenAI-compatible backend sits behind LLM_BASE_URL — paid API, free tier, or a local server all work identically, which is the entire point of the exercise.

One practical note: I did not hardcode a model name anywhere. Free tiers rotate what's available, so the model lives in an env var and in a one-line note in the README saying which model the fixtures were recorded against.

Step 3: A canary with a failure fixture, not a vibe check

"The output looked fine" is not a test. The canary asserts structure, which is the part I actually depend on:

// canary.test.ts
import { test, expect } from "vitest";
import { draftReleaseNotes } from "./llm";

const cfg = {
  baseUrl: process.env.LLM_BASE_URL!,
  apiKey: process.env.LLM_API_KEY!,
  model: process.env.LLM_MODEL!,
  timeoutMs: Number(process.env.LLM_TIMEOUT_MS ?? 20000),
};

const PRS = [
  "fix: retry webhook delivery on 502",
  "feat: add --json flag to status command",
  "chore: bump typescript to 5.5",
];

test("release notes canary: structure holds", async () => {
  const out = await draftReleaseNotes(PRS, cfg);

  // Structural assertions only — never assert prose.
  expect(out).toMatch(/^#{1,3}\s*(Added|Changed)/m);   // has a section header
  expect(out).toMatch(/^-\s+/m);                        // has bullets
  expect(out.toLowerCase()).toContain("json");          // grounded in input
  expect(out.toLowerCase()).not.toContain("as an ai");  // classic failure mode
  expect(out.length).toBeLessThan(2000);                // didn't ramble
}, 30_000);
Enter fullscreen mode Exit fullscreen mode

The failure fixture that earned its keep: I temporarily changed the system prompt to drop "Never invent features" and re-ran. The model dutifully announced a "new plugin system" that does not exist. The not.toContain checks didn't catch it — the grounding check did (the invented feature name wasn't in the input titles). That's when I added one more assertion: every bullet must share at least one token with the input. Crude, but it killed the hallucination fixture dead.

const inputTokens = new Set(PRS.join(" ").toLowerCase().split(/\W+/));
const bullets = out.match(/^-\s+.+$/gm) ?? [];
for (const b of bullets) {
  const overlap = b.toLowerCase().split(/\W+/).some((w) => inputTokens.has(w));
  expect(overlap).toBe(true);
}
Enter fullscreen mode Exit fullscreen mode

Cost and time boundary, declared up front

Item Value
CI runs per month ~40
Tokens per canary run ~400 in / ~200 out
Monthly spend on this lane $0 (free tier)
Fallback lane (local mock) $0, always green
Time to build all of this ~2 hours, most of it on the fixture

The rollback plan, written before I needed it

This is the part I'd skip if I hadn't been burned: the repo contains a mock provider that returns canned release notes. If the free lane fails three consecutive nightly runs — quota change, endpoint deprecation, model pulled — CI flips to the mock via one env var, and the human release-notes step goes back to copy-pasting PR titles. The feature dies in CI before I pay a cent to keep it on life support. Deciding that in advance is the only reason it'll actually happen.

Limitations, and who shouldn't do this

  • Free tiers are not a foundation. No SLA, no permanence promise, possible rate limits. Fine for a canary; reckless for anything user-facing.
  • Don't put private code or customer data through a free endpoint just because it's there. My canary sends PR titles from a public repo, nothing more.
  • Structural tests aren't quality tests. The canary proves the output is parseable and grounded, not that it's good. I still read the notes before publishing a release.
  • If your task needs long context, tool calling, or sub-second latency, a free general-purpose lane is the wrong tool and no abstraction layer will fix that.

If you're running a small AI feature in CI and want to try the zero-cost lane, the free tier at MonkeyCode is one OpenAI-compatible option — but the interface above works with whatever you point it at, which is kind of the thesis.

What's the smallest LLM task in your pipeline that you're still paying for out of habit — and what would it take to move it to a swappable lane like this?

Top comments (0)