DEV Community

ronnie0297-stack
ronnie0297-stack

Posted on Fully Autonomous

Catch runaway AI agent costs before merge with a GitHub Action

#ai #githubactions #typescript #finops

AI cost control usually begins after deployment: a provider dashboard reports what has already been spent. But several expensive failure modes are visible in code before a pull request is merged.

Consider a tool-using Vercel AI SDK call:

return streamText({
  model: openai("gpt-5.4-mini"),
  prompt: question,
  tools: { search }
});
Enter fullscreen mode Exit fullscreen mode

Two limits are missing. There is no output-token ceiling, and the tool loop has no stopping condition. Even if the prompt looks harmless, neither the worst-case response size nor the maximum number of steps is explicit.

A bounded version makes both decisions reviewable:

return streamText({
  model: openai("gpt-5.4-mini"),
  prompt: question,
  maxOutputTokens: 800,
  stopWhen: stepCountIs(5),
  tools: { search }
});
Enter fullscreen mode Exit fullscreen mode

Turning the check into a budget gate

Code alone cannot predict a bill. A useful estimate also needs traffic assumptions. Keep those assumptions in the repository so reviewers can challenge them:

{
  "monthlyCallsPerSite": 10000,
  "assumedInputTokens": 1000,
  "assumedOutputTokens": 1000,
  "warnMonthlyCost": 50,
  "failMonthlyCost": 250,
  "failOnUnboundedTools": true
}
Enter fullscreen mode Exit fullscreen mode

Then estimate each call site with a deliberately simple formula:

monthly cost = monthly calls ×
  ((input tokens × input price) + (maximum output tokens × output price))
Enter fullscreen mode Exit fullscreen mode

The estimate is not an invoice prediction. It is a consistent review signal. A changed model, increased output ceiling, or new call site produces a visible change before production.

Running it in GitHub Actions

I built an open-source Action called AICostFence to automate this check:

name: AI cost check
on: pull_request

permissions:
  contents: read
  pull-requests: write

jobs:
  cost-check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ronnie0297-stack/aicostfence@v0.1.0
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
Enter fullscreen mode Exit fullscreen mode

It scans locally on the GitHub runner, posts one updated pull-request report, and can fail the check when a configured guard is violated. It does not call an AI model or send source code to a model provider.

The current release intentionally supports a narrow surface: JavaScript/TypeScript projects using the Vercel AI SDK's generateText, streamText, generateObject, or streamObject calls. That makes the analysis deterministic while the early workflow is validated.

Repository: https://github.com/ronnie0297-stack/aicostfence

Marketplace: https://github.com/marketplace/actions/aicostfence

npm: https://www.npmjs.com/package/aicostfence

What cost control would be most useful in your pull requests: model-swap deltas, provider-specific checks, or organization-wide policies?

Top comments (1)

Collapse
 
raju_dandigam profile image
Raju Dandigam •

Making the estimate deterministic and reviewable is more useful than pretending it predicts the invoice exactly. For tool-using calls, I’d surface both the absolute budget and the PR delta: maxOutputTokens × maximum steps is only part of the envelope once retries and paid tool side effects enter the loop. A checked-in waiver with an owner and expiry could also keep exceptions visible instead of encouraging people to weaken the rule. Are wrapper functions and imported defaults on the roadmap, so the Action can resolve bounds that are not inline at the call site?