DEV Community

wartzar-bee
wartzar-bee

Posted on

Catch token-cost regressions in CI before they ship

#ai #devops #opensource #github

Last year a team shipped a "small prompt improvement" on a Friday. By Monday their Claude bill had jumped 40%. Nobody caught it in review — the diff looked fine. The tokens were invisible.

We wrote about the extreme end of this in We burned 136 million tokens running an autonomous agent studio — here's how we cut the bill ~90%. That was a runaway agent. But most cost regressions aren't dramatic — they're a prompt that grew by 200 tokens, a context window that stopped being trimmed, a new tool whose description is 800 words long. They compound quietly until the bill arrives.

We built the guardrail we wish we'd had.

What it does

wartzar-bee/ci-guardrail is a GitHub Action that:

  1. Runs @wartzar-bee/tokenscope on your PR branch and your base branch
  2. Computes the token-cost delta
  3. Posts a comment on the PR showing the delta and the top cost-driving files
  4. Optionally blocks the build if cost grows beyond your threshold

Here's what the PR comment looks like when a regression is caught:

## 🚨 wartzar-bee Cost Guardrail

| Metric          | Value      |
|-----------------|------------|
| Base tokens     | 12,400     |
| This PR tokens  | 16,200     |
| Delta           | **+30.6%** |
| Threshold       | 20%        |

### Top token consumers (HEAD)
| File                      | Tokens |
|---------------------------|--------|
| src/agent/prompts.ts      | 8,100  |
| src/agent/context.ts      | 4,200  |
| src/tools/search.ts       | 3,900  |

⛔ Build blocked — cost regression exceeds the 20% threshold.
Reduce prompt size, add caching, or raise the threshold if intentional.
Enter fullscreen mode Exit fullscreen mode

The comment is idempotent — it updates in place on each push, no spam.

Zero-config setup

Add one file to your repo:

# .github/workflows/cost-guardrail.yml
name: Cost Guardrail
on: [pull_request]

jobs:
  cost-check:
    runs-on: ubuntu-latest
    permissions:
      pull-requests: write
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - uses: wartzar-bee/ci-guardrail@v1
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          threshold-pct: 20   # block if tokens grow >20% vs base
Enter fullscreen mode Exit fullscreen mode

That's it. No API keys. No external service. GITHUB_TOKEN is the only secret — it's already in every repo.

Report-only mode

Not ready to block builds yet? Set threshold-pct: 0:

- uses: wartzar-bee/ci-guardrail@v1
  with:
    github-token: ${{ secrets.GITHUB_TOKEN }}
    threshold-pct: 0   # always comment, never block
Enter fullscreen mode Exit fullscreen mode

You get visibility without the gate. Add the gate when you're ready.

Why static analysis is enough (for now)

The action uses tokenscope scan — a static analysis of your agent code (prompts, context configs, tool definitions). It doesn't run your agent. That means:

  • Fast — runs in seconds, no LLM calls
  • Safe — read-only, no side effects
  • Deterministic — same code = same estimate, every time

The tradeoff: it estimates structural token cost, not runtime cost (which depends on conversation history, tool outputs, etc.). For catching regressions from prompt edits and config changes, static analysis catches the majority of real-world cases. Live sandbox execution is on the roadmap.

The pattern behind it

The 136M-token burn postmortem identified three root causes that appear in almost every cost incident:

  1. No baseline — teams don't know what "normal" looks like, so they can't see a regression
  2. No gate — even when someone notices, there's no mechanism to block a bad change
  3. No attribution — when the bill spikes, nobody knows which file or config caused it

The CI guardrail addresses all three: it establishes a per-PR baseline, provides an optional gate, and shows exactly which files are responsible.

Try it

The action is available now. The repo is at wartzar-bee/ci-guardrail — copy the workflow above and you're running in under two minutes.

If you hit an edge case or want to discuss the approach, open an issue or find us at @wartzarbee.

wartzar-bee builds cost-efficient autonomous agents. @wartzar-bee/tokenscope is our open-source token analysis tool — ~90 installs/month and growing.

Top comments (0)