A few days ago, I used roughly 30% of my weekly ChatGPT Pro allowance in a short stretch of work on a pull request. CI/CD problems kept the work stuck, and the PR was still unresolved.
That was what pushed me to increase the number of CI jobs we could run at once.
After the change, the allowance seemed to last longer. I wanted to understand how much of that improvement came from the extra slots. The 30% figure is my reading of the weekly limit, not a reconstructed token count. Spending that much allowance without getting a PR through was a good reason to investigate.
We had already spent more than four months improving CI/CD. This latest change made a pattern much clearer: as agents got faster at writing code, more of our development speed depended on what happened after the code was written.
The bottleneck moved
Writing code used to take up a large part of our development time. Coding agents changed that for us. We could get to a first implementation much faster.
Then we had to read it.
More code was ready for review, but we still had to understand it, check it, and decide whether it was safe to deliver. Producing a large diff had become much faster than reviewing it carefully. An agent giving a convincing explanation was not enough.
We invested in testing harnesses: repeatable setups that exercised the code, checked its outputs, and made failures visible. We maintained high coverage requirements and used agents in pull request reviews and discussions. The agents could inspect a proposed change, raise a concern, and help work through a correction.
Those discussions were useful when they led to something we could check. A reviewer pointing out a possible failure mode gave us a reason to examine a caller, improve an assertion, or reproduce a problem. Another agent agreeing with the implementation offered much weaker evidence. The quality of the checks still mattered.
Coverage helped us identify code that had not been exercised. We also needed assertions that would fail when the behavior was wrong, realistic integration checks, and human judgment about requirements. We wanted each accepted change to come with evidence we could inspect.
As this process improved, more work reached CI. Every required check still had to pass. A correction discovered during review could send the change through validation again. An agent could finish its next piece of implementation while the previous change was waiting for an answer from the pipeline.
Fast feedback was already a core engineering concern. DORA's guidance on continuous integration emphasizes small changes, reliable automated tests, and feedback that arrives quickly enough to support frequent integration. Agents made the cost of slow feedback more visible in our daily work, because changes now reached CI faster.
Our response developed over months. First we made the delivery path more repeatable. We strengthened validation, separated work that could run independently, and made failures easier to investigate. Later we spent more time on repeated setup, test planning, scheduling restrictions, and the handoffs between successful validation and verified delivery.
These changes affected different parts of the process. A test optimization might shorten execution. A scheduling change might allow the same work to start sooner. Better deployment evidence could save us from reconstructing which version had reached the environment. Treating all three as a single CI speed number would hide where the benefit came from.
One investigation made that distinction particularly clear. We had jobs waiting, and the initial question was whether we needed more runners. The timing evidence pointed to a concurrency cap in the workflow: later jobs started as earlier ones finished and freed a slot. For that run, the cap was setting the pace.
A queue on a dashboard therefore needs a little investigation. It may mean a shortage of runners, a concurrency cap in the workflow, a dependency on another job, or contention inside the resources already running. Each explanation suggests a different change.
We also learned to be cautious when a benchmark of one component looked good. In one small local comparison, test planning fell from roughly 123 seconds to 24 seconds while producing the same plan. Running the selected shard of tests still took around 151 seconds. The planning comparison had only two runs per variant, but it showed a useful improvement in that step. It did not establish a comparable improvement in the whole pipeline.
Preserving verification was part of evaluating these changes. In one retained comparison, the coverage results matched before and after the optimization. More generally, we needed to know that the same required behavior had been checked and that failures still caused the run to fail. A shorter run obtained by silently losing tests would have defeated the purpose.
Capacity needed the same care. At one point, additional concurrency created memory pressure and had to be reduced. More simultaneous jobs can increase contention or produce failures that require another attempt. The useful setting is the one that improves completed, verified work within the resources available.
The number of agents adds another dimension. One agent can send a change into a pipeline that needs several CI slots at once. Four agents each starting eight independent jobs could create demand for 32 slots. In practice, that depends on when their work overlaps, which stages can run together, and what resources the jobs share. Adding agents can increase the amount of work waiting for verification unless CI capacity keeps up.
A narrower question we could test
The effect on our allowance was harder to measure. The work mix was changing. Models and settings could differ. Prompt caching and allowance resets complicated comparisons. More than four months of continuous pipeline improvements also made it difficult to isolate the effect of one recent change. A daily usage total would tell us how much we used that day without telling us how efficiently we delivered comparable work.
A slow pipeline can leave more work for an agent. While a pipeline is unresolved, an agent may check its status, interpret another log excerpt, reconsider a failure, retry an operation, or reconstruct context after an interruption. Those activities require additional model work. An avoidable infrastructure problem can create several rounds of it.
Waiting by itself does not necessarily consume more tokens. A tool can wait for completion and return the result when it is ready. The important question is what the agent does while it waits, and what additional work a delay or failure causes.
We built ci-cd-tokens-test to examine that narrower question with synthetic workloads. It measures the work of observing CI for an already-correct change. The agent writes no code and does no real debugging; one scenario includes a scripted failure and retry. Readers can reproduce the comparison without access to our projects, data, or infrastructure.
dnikolayev
/
ci-cd-tokens-test
Reproducible experiments on CI scheduling, agent observation, and token usage.
CI scheduling and agent token usage
A small, reproducible experiment: does faster CI reduce the work an agent does while waiting for verification?
Four identical GitHub Actions jobs run with a concurrency limit of either one or four. An agent observes them by repeatedly asking for status, or by letting a tool wait for completion. A separate scenario injects one predictable transient failure and permits one retry. Every condition keeps the same checks.
This measures observation overhead for an already-correct change. It does not measure coding quality, hardware scaling, or production deployment.
Read the article: AI Made Coding Faster. Our Delivery Pipeline Had to Catch Up.
The first pilot contains all twelve planned trials, including one failed verification, plus separately labelled preflight attempts. Faster CI did not produce a consistent token reduction in this small sample.
The larger comparison keeps 32 jobs fixed and tests caps of 4, 8, 16…
The first experiment runs four identical small jobs, each with the same assertions and a 15-second hold. A cap of 1 allows one job at a time; a cap of 4 allows up to four. We set the cap with GitHub's matrix max-parallel setting. Raising it changes a scheduling rule within the workflow. It does not add runners or guarantee that all four jobs start together, so we also measure actual job overlap.
The agent watches CI through an observer command. In the polling condition, each call returns a compact status result, and the agent calls again if the workflow is still running. In the tool-wait condition, one call waits until completion. The shell tool may hand control back while that command is running, so the agent still has to wait on its session. We count observer calls, completed commands, and tokens. The event stream does not reveal every model request.
The observer sleeps for 10 seconds before each check, with API and agent overhead between checks. Returning each polling result to the agent deliberately creates repeated observation work. That treatment does not represent every coding agent or every team's workflow.
A failure scenario makes one job fail on its first attempt and succeed on one permitted retry. Other jobs continue, and only the failed job reruns. The required assertions stay the same.
The runs on 12 October 2026 used Codex CLI 0.162.1, the gpt-6-astra model with reasoning set to max, and GitHub-hosted ubuntu-latest runners. Each trial starts a fresh agent session. Settings stay fixed, cap order is randomized within each comparison, and attempts are bounded. The methodology describes the protocol and its limits.
The pilot succeeded in 11 of 12 planned trials. Success requires the observer to verify the expected jobs and permitted retry, followed by the agent exiting successfully and reporting VERIFIED. One dispatch command returned an error after GitHub created the run. The harness cancelled it, and its usage remains in the report. Three earlier preflight attempts are retained separately.
A pair compares cap 1 and cap 4 under the same conditions and repetition. Five pairs succeeded at both caps. The table shows three with different token outcomes; the two remaining pairs, both tool-wait without an injected failure, appear in the chart. Agent time covers the whole session, including startup and the final response. Total tokens are input plus output, including cached input.
| Condition | Agent time, cap 1 | Agent time, cap 4 | Total tokens, cap 1 | Total tokens, cap 4 |
|---|---|---|---|---|
| Polling, no injected failure, first repetition | 106.9 s | 56.2 s | 383,550 | 101,314 |
| Polling, one failure and retry | 152.4 s | 80.6 s | 236,116 | 353,069 |
| Tool wait, one failure and retry | 153.1 s | 83.4 s | 128,201 | 127,844 |
Every row got faster at cap 4. Total tokens fell sharply in the first row, rose in the second, and barely changed in the third.
Cached input is input served through reuse of a previously computed prompt prefix. Uncached input is processed without that reuse; it can include repeated text as well as new material. In the first row, total tokens fell by about 282,000 while uncached input fell by about 3,200. In the tool-wait retry pair, uncached input rose by about 7,300 despite the almost unchanged total. The full results keep those counters separate.
Each line pairs trials without an injected failure, with two repetitions per monitoring approach. The solid polling line corresponds to the first table row. The dashed polling line includes the interrupted cap 4 trial, marked with a cross; its shorter time is not a speedup.
What happened at 8, 16, and 32 slots
Four jobs cannot show what happens above four slots, so we ran a separate experiment with 32 jobs in every condition. Each cap had one polling trial and one tool-wait trial, with no injected failure. All eight succeeded, and each reached its requested peak job overlap.
Account limits matter when reproducing this. GitHub currently documents default limits of 20 concurrent standard hosted jobs on Free and 40 on Pro. These are GitHub plans, separate from ChatGPT Pro. Other workflows can use some of that allowance too; check your account's runner limits before expecting 32 jobs to overlap.
| Cap | Agent time, polling | Agent time, tool wait | Total tokens, polling | Total tokens, tool wait |
|---|---|---|---|---|
| 4 | 194.9 s | 194.8 s | 448,226 | 179,956 |
| 8 | 114.8 s | 123.4 s | 378,660 | 102,658 |
| 16 | 74.7 s | 70.1 s | 127,924 | 123,233 |
| 32 | 81.7 s | 56.7 s | 226,808 | 76,358 |
Moving from 4 to 16 slots helped both approaches. Going from 16 to 32 helped the tool-wait trial, while the polling trial got slightly slower. One run per condition cannot tell us which setting is best.
Polling made 12, 7, 4, and 4 observer calls at caps 4, 8, 16, and 32. Each tool-wait session made one. The polling token increase from 16 to 32 therefore happened despite the same number of observer calls: total tokens rose by 98,884, mostly in cached input. Those counts alone do not explain the difference.
From cap 4 to cap 32, total tokens fell in both approaches. The reduction was dominated by cached input. Output also fell slightly, while uncached input rose. Both approaches had their lowest uncached count at cap 16, including an unusually low 4,922 for tool wait. With one trial per condition, we cannot tell variation from a trend. The larger experiment's records include all the counters and job timings. These token counts cannot tell us how much weekly allowance was saved.
Each point is one trial of the same 32-job workload. The middle panel shows the polling upturn at 32; the right panel shows the uncached-input dip at 16.
Across these trials, higher caps shortened agent time in all five complete pilot pairs and at every scaling step except polling from 16 to 32. Tool wait used fewer total tokens than polling at every scaling cap, although the gap at 16 was only 4,691 tokens.
The agent sessions ran sequentially in both experiments. We measured concurrency within a pipeline, so these results do not yet tell us what happens when several agents compete for shared CI capacity.
These experiments give a reason to investigate CI as part of agent efficiency. They changed a workflow cap on hosted runners and did not test renting additional machines. Establishing typical token savings would need more repetitions and more realistic tasks, including debugging and review corrections. Failed attempts belong in those results too: usage per successful verification must include the cost of attempts that did not succeed.
What I would change first
There is a budgeting decision here too. If machine capacity is the constraint, renting more CI servers may lower the combined cost of CI and agent work per completed, verified change. Faster, more reliable feedback could prevent enough repeated investigation and retries to pay for the additional machines. First check whether existing capacity is sitting behind a scheduling limit. Renting servers will not remove that limit.
For a hypothetical example, suppose extra capacity costs $60 a month, including operations and idle time, and supports 1,000 comparable verified changes a month. It would need to avoid six cents of agent spending per change to cover its cost. We did not measure that payback in these experiments. A real comparison would include the cost of failed attempts, runner maintenance, and unused capacity on both sides.
With a fixed subscription, the benefit may instead be more completed work within the same allowance. That does not automatically mean a smaller invoice. Delivery time can also improve even when token usage barely changes; it is worth measuring that benefit separately.
For engineers considering similar improvements, I would start by following a few changes all the way through delivery. Record when implementation was ready, when required jobs could start, when feedback arrived, and when the deployed behavior was verified. Count interventions and retries alongside elapsed time. That gives you a better chance of fixing the part that is actually slowing the work.
For teams with long queues, inspect scheduling rules and actual job overlap before choosing a capacity increase. On GitHub Actions, check max-parallel, concurrency groups, job dependencies through needs, and account limits. Then check memory, database load, and failure rates as concurrency rises. A faster successful run is encouraging; a higher rate of failed runs may erase the gain across a week of work.
For teams spending a lot of agent effort on CI, inspect the conversation around a run. Repeated status checks, large log dumps, and ambiguous results are opportunities to improve the interface. A concise failure summary, an exact link to the failing check, or a tool that waits for a terminal result can remove work without changing the tests.
Measure agent usage and CI costs against completed, verified changes, while keeping track of task difficulty, model settings, and the number of agents working at once. A usage total on its own cannot tell you whether comparable work was delivered more efficiently.
After opening more CI slots, my allowance seemed to go further. These experiments do not tell me how much of that improvement came from the change. They do show why CI deserves a place in the conversation about agent efficiency.
After more than four months of changes, we pay much closer attention to what happens after the first implementation is ready. We still need careful review and convincing tests. We also need the answers from those tests to arrive soon enough that the next change does not pile up behind them. That is where much of our development speed now comes from.
Top comments (0)