DEV Community

wartzar-bee
wartzar-bee

Posted on

Put a token-cost gate on your AI-agent PRs in 5 minutes

#ai #tutorial #devops #github

If you run an AI agent, your token bill lives in your source code — in prompts, tool
descriptions, and how much context you re-send every turn. And source code changes in pull
requests. So the natural place to catch a cost regression is the same place you catch a bug:
in CI, on the PR, before it merges.

I learned this the expensive way. We once
put an agent on a timer and it burned 136M tokens overnight doing almost nothing.
That was the dramatic end. The everyday end is quieter: a system prompt that grew by 200
tokens, a context window that stopped being trimmed, a new tool whose description is 800 words
long. None of it shows up in a code review — the diff looks fine. The tokens are invisible.

This is a 5-minute walkthrough to make them visible: a GitHub Action that estimates the
token-cost delta of a PR, comments the responsible files on the PR, and can fail the build if
the cost jumps past a threshold you set.

The one file you add

Drop this into .github/workflows/cost-guardrail.yml:

name: Cost Guardrail
on: [pull_request]

jobs:
  cost-check:
    runs-on: ubuntu-latest
    permissions:
      pull-requests: write   # needed to post the comment
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0       # the action compares HEAD vs the base branch

      - uses: wartzar-bee/ci-guardrail@v1
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          threshold-pct: 20        # block if tokens grow >20% vs base
          working-directory: .     # where your agent code lives
Enter fullscreen mode Exit fullscreen mode

That's the whole setup. No account, no API key beyond the GITHUB_TOKEN your repo already
has, no service to sign up for. fetch-depth: 0 matters: the action checks out both branches
to diff them, so it needs the git history, not just the tip commit.

What you get on the next PR

Here's a comment from a real run of the action — a contributor added a few-shot block to an
agent's system_prompt.txt, which re-sends on every turn. These are genuine tokenscope
numbers, not a mock-up; the action updates the same comment on each push, so it never spams
the thread:

## 🚨 wartzar-bee Cost Guardrail

| Metric             | Value      |
|--------------------|------------|
| Base branch tokens | 978        |
| This PR tokens     | 1,512      |
| Delta              | **54.6%** (**+$0.0016**) |
| Threshold          | 20%        |

💵 Cost estimated at $3.00/1M tokens — set `price-per-1m-tokens` to your model's price.

### Biggest cost increases (responsible files)
| File                     | Base | Head  | Δ        |
|--------------------------|-----:|------:|---------:|
| agent/system_prompt.txt  | 962  | 1,496 | **+534** |

> ⛔ Build blocked — cost regression exceeds the 20% threshold.
> Reduce prompt size, add caching, or raise the threshold if intentional.
Enter fullscreen mode Exit fullscreen mode

The value isn't the top number — it's the responsible-files table. It turns "the bill went
up" into "the bill went up because system_prompt.txt grew 534 tokens." That's a comment a
reviewer can act on in the PR, not a surprise on the invoice. (The per-PR dollar figure is
small; the point is it re-sends every turn, every day, across your fleet — the percentage is
what gates the build.)

Start in report-only mode

Failing builds on day one is a good way to get an Action deleted. Start by measuring, not
blocking. Set the threshold to 0 and it always comments, never fails:

- uses: wartzar-bee/ci-guardrail@v1
  with:
    github-token: ${{ secrets.GITHUB_TOKEN }}
    threshold-pct: 0   # report only, never block
Enter fullscreen mode Exit fullscreen mode

Let it run on real PRs for a week. You'll see what a "normal" delta looks like for your repo —
maybe +5% is routine and +40% is the one worth stopping. Then set threshold-pct to a number
that reflects that, and turn on blocking with intent instead of guessing.

Make the dollar figure yours

The comment shows an estimated dollar delta so a non-engineer reading the PR gets it. By default
it uses an approximate blended input-token price of $3.00 per 1M tokens. That's a placeholder —
override it to your model so the number is real:

- uses: wartzar-bee/ci-guardrail@v1
  with:
    github-token: ${{ secrets.GITHUB_TOKEN }}
    threshold-pct: 20
    price-per-1m-tokens: 0.80   # e.g. a cheaper model's input price
Enter fullscreen mode Exit fullscreen mode

The percentage is what gates the build; the dollar figure is there to make the percentage land
with whoever approves the merge.

What it's actually doing

No magic, and nothing sent anywhere:

  1. Runs tokenscope on your HEAD branch to estimate its total token footprint.
  2. Fetches the base branch and runs the same scan.
  3. Computes the delta as a percentage.
  4. Posts (or updates) the PR comment with the per-file breakdown — and writes the same table to the Actions run summary, so you also see it on push or scheduled runs where there's no PR to comment on.
  5. Exits non-zero if the delta exceeds your threshold-pct (and the threshold is above 0).

The engine is tokenscope, an open-source token-cost analyzer; the Action is a thin, auditable
composite wrapper around it. Both are MIT.

Try it

Add the one file, open a PR, and watch the token delta show up next to the diff. If it catches
even one 800-word tool description before it merges, it's paid for itself.

Top comments (0)