Catch runaway AI agent costs before merge with a GitHub Action

ronnie0297-stack — Mon, 21 Sep 2026 15:10:20 +0000

AI cost control usually begins after deployment: a provider dashboard reports what has already been spent. But several expensive failure modes are visible in code before a pull request is merged.

Consider a tool-using Vercel AI SDK call:

return streamText({
  model: openai("gpt-5.4-mini"),
  prompt: question,
  tools: { search }
});

Two limits are missing. There is no output-token ceiling, and the tool loop has no stopping condition. Even if the prompt looks harmless, neither the worst-case response size nor the maximum number of steps is explicit.

A bounded version makes both decisions reviewable:

return streamText({
  model: openai("gpt-5.4-mini"),
  prompt: question,
  maxOutputTokens: 800,
  stopWhen: stepCountIs(5),
  tools: { search }
});

Turning the check into a budget gate

Code alone cannot predict a bill. A useful estimate also needs traffic assumptions. Keep those assumptions in the repository so reviewers can challenge them:

{
  "monthlyCallsPerSite": 10000,
  "assumedInputTokens": 1000,
  "assumedOutputTokens": 1000,
  "warnMonthlyCost": 50,
  "failMonthlyCost": 250,
  "failOnUnboundedTools": true
}

Then estimate each call site with a deliberately simple formula:

monthly cost = monthly calls ×
  ((input tokens × input price) + (maximum output tokens × output price))

The estimate is not an invoice prediction. It is a consistent review signal. A changed model, increased output ceiling, or new call site produces a visible change before production.

Running it in GitHub Actions

I built an open-source Action called AICostFence to automate this check:

name: AI cost check
on: pull_request

permissions:
  contents: read
  pull-requests: write

jobs:
  cost-check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ronnie0297-stack/aicostfence@v0.1.0
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

It scans locally on the GitHub runner, posts one updated pull-request report, and can fail the check when a configured guard is violated. It does not call an AI model or send source code to a model provider.

The current release intentionally supports a narrow surface: JavaScript/TypeScript projects using the Vercel AI SDK's generateText, streamText, generateObject, or streamObject calls. That makes the analysis deterministic while the early workflow is validated.

Repository: https://github.com/ronnie0297-stack/aicostfence

Marketplace: https://github.com/marketplace/actions/aicostfence

npm: https://www.npmjs.com/package/aicostfence

What cost control would be most useful in your pull requests: model-swap deltas, provider-specific checks, or organization-wide policies?

Try the working demo

The before-and-after demo runs the published npm package against risky code and its corrected version. It verifies that the first scan fails and the second passes, with no API key or model charges. See the GitHub verification runs.

git clone https://github.com/ronnie0297-stack/aicostfence.git
cd aicostfence/demo
npm ci --ignore-scripts
npm run demo

Requires Node.js 20 or newer. The demo uses bundled fallback prices for reproducible results. Version 0.1.0 does not include step multiplication, retries, or paid tools in its estimate, and it does not resolve wrappers or imported defaults. Passing the check is not a guarantee of a safe bill.

DEV Community: ronnie0297-stack

Catch runaway AI agent costs before merge with a GitHub Action

Turning the check into a budget gate

Running it in GitHub Actions

Try the working demo