DEV Community

Cover image for My Agent Ran 15 Minutes of Tests to Fix a Comma. GitHub Cut Me Off Mid-Migration.
Phil Rentier Digital
Phil Rentier Digital

Posted on • Originally published at rentierdigital.xyz

My Agent Ran 15 Minutes of Tests to Fix a Comma. GitHub Cut Me Off Mid-Migration.

"You have used 100% of your included GitHub Actions minutes." When that mail arrived last Friday, right in the middle of a PostgreSQL migration, on the densest commit week this project ever had, I got worried. Private repo, solo dev, free plan, 2000 minutes. My finisher refuses to merge until CI is green, so from that mail on, nothing integrates anymore. Zero.

Argh.

1710 minutes billed in July. I stared at the number for a while... I was hallucinating. Where does that even come from.

It just piled up. 5 minutes per run, 3 runs per task, every task treated exactly the same.

A comma in a Markdown file costs the same as rewriting the whole distributor ingestion pipeline! Send help 😬 Same test suite, same wait, same bill.

And the question I asked myself that evening is the one that was missing everywhere in my setup: who decides a change deserves to be verified, if it is neither me nor the agent?

Fail-Closed on the Worst Possible Week

The timing was almost artistic. I was moving the product catalog from a hand-rolled SQLite thing to real PostgreSQL, which means every commit that week touched schema, migrations, or the distributor CSV importer. Exactly the kind of work where you want the tests to run.

And exactly the moment where I could no longer run them.

My delivery script is fail-closed on purpose. Branch merges into integration, integration merges into main, and nothing moves forward while the pipeline is red or unknown. Take the runner away and the whole thing does not degrade gracefully, it just stops. 4 branches sitting there, finished, tested locally, unmergeable.

The free tier resets on the 1st. That was 3 days away. 3 days of not shipping, on the week I had the most to ship.

I Rebuilt What GitHub Already Gave Me

My first reflex was not to ask why. My first reflex was to get more minutes.

So I spent a full day on it. Registering 2 self-hosted runners, the dev Mac as primary and the VPS as fallback. Rewriting the workflow files so jobs could target either one. Standing up an ephemeral PostgreSQL per job because the hosted service containers I was using are not free on a self-hosted setup. Then maintaining both lanes in parallel, because I did not trust the new one enough to delete the old one.

A full working day to rebuild for free what was already free. And it worked. My CI now literally runs on my machine, which is the one place a developer is never allowed to say something works.

Small aside while the runner was compiling: the Mac sits on a shelf in the living room, and under a full test suite the fan spins up to a noise you can hear from the couch. My youngest started calling it "the plane". For about 2 weeks I could tell whether a build was running without opening the laptop, just by whether the plane was taking off. I have since moved it, which I regret a bit, because that was honestly the best CI notification system I ever had.

Anyway. By evening, everything was green again and I was merging. Problem solved, or so it looked. Nothing in that whole day forced me to ask what had actually consumed 1710 minutes.

595 Runs, and My Guess Was Wrong

TITLE "Where 1710 Minutes Went" + subtitle "595 runs, 3 lanes, 1 empty quota". Metaphor: a fuel gauge dashboard mounted on a factory wall, with 3 pipes of different thickness feeding a single tank that reads EMPTY. Style: engineer blueprint, white technical linework on dark ground, dimension arrows, stencil lettering. Palette: navy #14213D, amber #FCA311, muted red #C1121F, pale grey #E5E5E5, black #111111. Content: 3 labeled pipes feeding the tank, TASK BRANCHES 633 min 37%, INTEGRATION 407 min 24%, MAIN 670 min 39%. A small side valve labeled NON-RUNTIME DIFFS 115 min 6.7% barely dripping. Below the tank, a caption block reading "5 min per run x 3 runs = 15 min per task, any size". Highlight: the INTEGRATION and MAIN pipes drawn with a red overlay and a bracket labeled "61% proving it twice", the bracket glowing amber. Legend: bottom-left stencil box, "thick pipe = billed compute / side valve = what I blamed first". Footer: © rentierdigital.xyz. NOT flat corporate vector, NOT minimalist tech startup aesthetic.


Time Distribution Analysis: 1710 Minutes Across Development Pipelines

The next morning I pulled the full July run history and classified every tested commit by its git diff.

Scope first, so the numbers make sense: 595 Actions runs on the month, of which 528 map to a commit I could classify. The rest are failures, cancellations, and runs with no usable diff.

Fact 1. A green run takes about 5 minutes whatever its job is. 5.2 on a task branch, 5.1 on integration, 5.0 on main. And a single integration is 3 sequential runs, so roughly 15 minutes of compute and of waiting per task, whatever the size of the task.

Fact 2. My starting hypothesis was wrong, and I should say it plainly because I was pretty sure of myself. I was convinced the documentation commits were eating the quota. Measured: pure non-runtime changes account for 115 minutes. 6.7% of the total, 54 runs out of 528. The absurdity is real, but it is unitary, not volumetric.

Fact 3, the one that holds. Task branches 633 minutes (37%). Integration branches 407 minutes (24%). Main 670 minutes (39%). The integration run tests the merge that main retests a few minutes later, and the task branch tested a nearly identical tree just before. 61% of my compute went into proving something another run was about to prove again.

Now, the obvious objection, and it is a good one. Testing before merge so you do not break main is the entire reason those 3 runs exist. It is a good practice, and the trees are not strictly identical either, main can move between the branch run and the merge. So this is not 61% of pure waste. It is 61% of compute whose justification depends entirely on what is inside the diff, billed without that diff ever being looked at once.

Which brings back the comma. 15 minutes of compute, and 15 minutes of my own waiting, to change a comma in a Markdown file. Same price as reworking the importer. It is the CI equivalent of saving 3 times in a row before walking into an empty room.

That redundancy was not new. It existed in June, in May, and it never emptied the quota once. So the number alone does not explain why the wall showed up in July.

The Pipeline Didn't Change. The Commit Rate Did.

The YAML has not moved in months. It was calibrated for a human commit rhythm, which in my case meant a handful of meaningful pushes a day, each one carrying enough work to justify 15 minutes of verification.

What changed is who pushes. My agent now produces 5 to 10 micro-tasks a day, each one landing as its own branch, each one perfectly formed, each one triggering the exact same ritual. The pipeline did not get more aggressive. It got more requests, and it obeys, and it obeys identically for a comma in a Markdown file and for a schema migration on the orders table.

Now the precise version, because the lazy one blames the agent. This is not Skynet quietly deciding to torch my quota, it is a very fast intern who was never told what anything costs. The agent is not launching redundant tests out of carelessness, it is pushing a branch exactly like I told it to, and the pipeline does the rest. What it lacks is not discipline, it is a question. "Is this worth verifying" is a question that is written down nowhere, not in the workflow file, not in the agent instructions, not in my head at the moment I set this up, and an agent will not invent a cost question you never gave it. So it never gets asked, 5 to 10 times a day, and each unasked question costs 15 minutes.

Execution got delegated. The cost decision never did.

There is academic work on the neighboring version of this, reruns triggered by precaution in industrial CI, where teams relaunch jobs hoping they eventually pass. The authors note the same accumulation effect, precautions that individually look reasonable and collectively eat CI resources and developer waiting time. Different trigger, same shape. No decision burned that budget. It just never had a stopping rule.

I have written before about why cheaper compute made me burn more of it, which is the price version of this problem. This one is not about price. It is about the ritual that fires whether or not anyone checked it was needed.

The Grid Solved This Without Discipline

Scale disclaimer up front, because the jump is big: a free plan with 2000 minutes is not an enterprise cloud budget. Same mechanism, wildly different orders of magnitude.

Flexera's 2026 State of the Cloud Report, 15th edition, 753 respondents: self-reported cloud waste climbs to 29%. That is the first increase in 5 years, after 28% in 2023 and 27% in both 2024 and 2025. Flexera's own reading is that AI adoption has outrun the tagging, attribution and governance practices that were built for predictable workloads.

2 honest caveats, and I am putting them here and not in a footnote. First, the measure is declarative. Respondents estimate themselves what share of their own spend they consider wasted, it is not an instrumented reading. Second, Flexera sells a FinOps and cloud cost management platform, so the company measuring the size of the problem also sells the cure. Same goes for most of the loud sources on this topic.

The detail that makes the curve interesting is who it happens to. At the moment waste turns back up, 63% of those organizations have a dedicated FinOps team and 71% have a Cloud Center of Excellence. The structure is there. The dashboards are there. Karen from Accounting has a monthly report with every name on it. And the number went up anyway.

Meanwhile the resource is being described as scarce by the people selling it. On the Meta earnings call last week, CFO Susan Li said the industry has historically underbuilt against the wave of AI adoption, which makes existing capacity highly valuable, and described the constraint as both demand and supply driven for the foreseeable period. Zuckerberg mentioned seeing demand for compute at premiums above what Meta itself pays for that capacity. So we have a resource declared scarce by its vendors, in the same week that nearly a third of what gets bought is declared wasted by its buyers.

Now the part I actually care about, because it is the only one that gives a rule.

The electrical grid solved this problem 60 years ago, and not with pedagogy. You don't run your dishwasher at midnight out of ecological conscience. The kWh is cheaper at night, and behavior follows. But the interesting number is not the price, it is the enrollment. In the US deployments documented by Brattle, a utility offering a dynamic rate as an opt-in signed up 1 customer in 7. Another utility applying the same rate as the default kept more than 80% of its customers on it, with 15 to 20% peak reduction during critical events. Same rate. Same customers, roughly. The lever was not virtue, it was which box was already ticked.

1 caveat so I do not oversell the analogy: a price signal moves consumption more than it reduces it. PG&E measured 14% peak reduction and 8.2% average bill savings under its critical pricing events, with no observable conservation effect, and a study on Jeju Island found total consumption rising across all periods after households switched, even though the shape of the curve improved.

Which is fair, and it applies to my case too. What I built next removes a wait and a run. It does not make the project frugal.

So the rule I took out of this is not a resolution to be more careful. It is that the arbitration has to sit in the default, at the moment the diff leaves.

A Two-Speed Cycle, Shipped Same Day

TITLE "1 Diff, 2 Speeds" + subtitle "the classifier decides, not the agent". Metaphor: a railway switch on a track, a single incoming train splitting onto 2 lines, one short spur ending fast, one long line with 3 signal towers. Style: engineer blueprint, white technical linework on dark ground, stencil lettering, dimension arrows. Palette: navy #14213D, amber #FCA311, muted red #C1121F, pale grey #E5E5E5, black #111111. Content: incoming track labeled DIFF vs ORIGIN/MAIN reaching a switch box labeled CLASSIFIER. Short spur labeled NON-RUNTIME, stations TYPE-CHECK then MERGE then WATCHER ON MAIN, marked "1 non-blocking run". Long line labeled RUNTIME, DB, CRONS, DELIVERY with 3 signal towers labeled BRANCH then INTEGRATION then MAIN, marked "3 runs, unchanged". Both lines end at a single red gate labeled DEPLOY GATE, FAIL-CLOSED. Highlight: the CLASSIFIER switch box drawn twice the size of other elements with an amber glow ring around it. Legend: bottom-left stencil box, "short spur = seconds / long line = tens of minutes". Footer: © rentierdigital.xyz. NOT flat corporate vector, NOT stock infographic.


Railway Switch Metaphor for Development Pipeline Classification

Same afternoon, 4 changes.

First, the finisher classifies the branch diff against origin/main before doing anything else.

SAFE='^(docs/|tests/|\.opencode/|\.claude/|\.github/|[^/]*\.md$|\.gitignore$)'

CHANGED=$(git diff --name-only origin/main...HEAD)
RISKY=$(echo "$CHANGED" | grep -Ev "$SAFE" || true)

if [ -z "$RISKY" ]; then
  echo "non-runtime diff, taking the short path"
  exit 0
fi
exit 1
Enter fullscreen mode Exit fullscreen mode

Second, if the diff is entirely non-runtime, short path. Merge and cleanup happen immediately, no blocking CI wait, local validation reduced to a type-check.

Third, the safety net on that short path, because a short path with no net is just skipping tests with extra steps. A detached watcher follows main's CI and mails me if it ends red or times out. The deployment gate stays fail-closed regardless, so nothing red reaches production with or without the short path.

Fourth, everything else keeps the full cycle, unchanged. Anything touching runtime code, the database, the crons, or the delivery scripts goes through the same 3 runs it always did. This is not a fast lane for me to sneak schema changes through when I am in a hurry.

Measured result: a documentation change integrates in seconds instead of tens of minutes, and consumes 1 non-blocking CI run instead of 3 expected ones. It also saves me from opening a second account to farm free minutes, which, yeah, I had thought about.

The classifier is to my pipeline what the default rate is to the grid. Not an option I have to remember to activate when I feel disciplined, a decision the system takes in my place, at the moment it matters.

A rule you have to remember is not a rule. It is a hobby.

The 3 Prompts That Found It

The whole thing came out of 1 session, and the shape of it is more reusable than the fix itself. 3 prompts, in order.

Prompt 1, the raw doubt. I did not propose a solution. I asked whether running the full suite on everything was not exaggerated, and whether it was normal to fire a whole commit cycle for a comma. That is it. Vague, unflattering, honest.

Prompt 2, the framing. State of the art on CI cost control, what is wrong with my current practice specifically, and what my options are. Not "fix it". I wanted something to compare myself against, because my own setup felt normal to me right up until it did not.

Prompt 3, GO plus capitalize. Decision made, implement it, and immediately write the retro up as a structured document in the repo.

The method lesson is in the order. Expressing a vague doubt instead of a formed solution is what let the audit contradict me, and it did contradict me, since I walked in blaming documentation commits and walked out with a structural redundancy. Asking for state of the art rather than a patch is what turned "my CI is expensive" into a classification problem. And documenting in the same session is what keeps the lesson from dying with the context window.

It is the same framing discipline as the Blueprint method in Vibe Coding, For Real, and roughly the same discipline as how framing prompts as contracts changed my shipping: a badly scoped task is expensive whatever tool you throw at it.

What I Still Don't Know

What is established: the short path has been live for a week. The documentation for this very retro went through it, merged in seconds, no blocking CI wait, watcher on main verified green.

What is not: 1 week of hindsight is 1 week. The watcher has caught nothing yet, which means it has never actually been tested, only observed not firing. And all of this runs in a solo context where I am the only one pushing. What the short path does when 3 humans work the same branches at once, I have no idea. My guess is it holds, since the deployment gate stays fail-closed either way, but that is a guess and could be I am reading my own setup too generously.

The bigger weakness is the classifier itself. Sorting risk by file path is a coarse proxy. A test helper that quietly gets imported by runtime code, a config file that ends up in the wrong folder, and the rule is confidently wrong, silently, on the fast lane. I think path-based classification is the right first version and the wrong long-term answer, and I do not currently know what the right one looks like.

So the question from the start has an answer, just not a comfortable one. The cost decision does not take itself, and it will not be taken by the thing executing the work either. It has to be written down, in a script, at the moment the diff leaves.

The short path has been running for a week. It changes everything, it is faster, just as effective, and it saved me from opening a second account to farm free minutes (I admit it...).

Sometimes you have to dare, open the hood, and look at what your AI coder has been quietly tinkering with in its corner.

Sources

This post may contain affiliate links. If you click them, I might earn a small commission (costs you nothing, and helps me keep shipping quality articles every day for your reading pleasure).

Top comments (0)