Recently, while working on a personal project, I stumbled upon an interesting feature in ChatGPT. Apparently, you can connect it to GitHub and give it access to your repositories. Nothing particularly groundbreaking at first glance, just another integration in a world already overflowing with AI tools. But it got me wondering how far I could push it. What if I used ChatGPT not just to analyze code, but as a proper coding agent capable of working directly with my repository?
By that point, I'd already been using Codex quite extensively, so I was familiar with both its capabilities and its biggest annoyance: usage limits. There's nothing quite like getting into the flow, breaking a large task into manageable pieces, and then hitting your five-hour quota just as things start getting interesting. That got me thinking: why spend my Codex allowance generating code if I could delegate some of that work to regular ChatGPT? Especially since I wasn't paying for Plus specifically for coding. I subscribed for everyday conversations, research, and all sorts of other things.
Of course, calling this approach completely free would be stretching the truth. ChatGPT Plus costs money, has its own limits, and computing power doesn't magically appear out of thin air. But if you're already paying for the subscription and your regular usage stays comfortably within its limits, having an additional way to generate code feels like a pretty nice bonus. At least in my case, I hadn't managed to hit ChatGPT's limits, even while using it for some fairly substantial development tasks.
The remaining question was whether this approach could do more than just dump code into a repository. Could it actually produce something useful without leaving me with hours of debugging afterward? There was only one way to find out. But first, I needed to set everything up.
Setting Up ChatGPT with GitHub
To get started, you'll need a ChatGPT account with access to the GitHub integration and, naturally, a repository to work with. Open the Plugins section in ChatGPT's settings, find GitHub, and start the connection process. You'll be redirected to GitHub, where you authorize the application and choose which repositories it can access. The setup is straightforward and doesn't require manually creating or configuring API tokens.
At this point, I'd strongly recommend taking a moment before clicking through the authorization screens. GitHub lets you choose between granting access to all repositories and the Only select repositories option, where you explicitly specify which repositories the application can access. I went with the latter, granting access only to the project I intended to work on. It's a straightforward application of the Principle of Least Privilege, something that's surprisingly easy to overlook when you're eager to try out a new tool.
It's equally important to review the permissions the application requests, especially if they include the ability to write to or modify repository contents. If an agent can change code, I wouldn't let it work directly on the main branch under any circumstances. It's much safer to create a separate branch for each independent task and review the results through a Pull Request. Besides making code reviews easier, this approach preserves the history of changes, lets you measure how much code the agent actually generated, and gives you a clean way to reject an implementation if things go sideways.
Once authorization is complete, make sure ChatGPT can access your repository and read its structure. There's one important detail, though: being able to read code doesn't necessarily mean your particular integration supports modifying it. Before getting started, check whether the agent can create branches, commits, and Pull Requests. In my case, all of these capabilities were available, so it was time for the interesting part: giving the agent a real task and seeing what happened.
Putting It to the Test: 20,000 Lines of Code in Half a Day
Back to my project. I'm working with a fairly large monorepo, where functionality is organized into separate packages with clearly defined responsibilities. While going about my usual development work, I picked out a few tasks I could delegate to ChatGPT without interfering with what I was doing locally. I wasn't particularly interested in starting with something trivial, like fixing lint errors or renaming variables. If I was going to test what this agent could do, I wanted to give it a proper architectural challenge: understand the existing codebase, work out an approach, and implement new functionality.
ChatGPT lets you choose the model's reasoning effort, so I decided to go straight for the strongest option available to me: High. I deliberately chose a setting intended for complex tasks rather than trying to save resources by using a less demanding mode. I don't know exactly how many tokens the agent burned through while reasoning, since I wasn't tracking that separately. But the amount of work it produced was substantial: in half a day, ChatGPT generated roughly 20,000 lines of code through GitHub, implementing new functionality in my monorepo.
But that wasn't even the most interesting part. Throughout the process, I kept an eye on my five-hour Codex quota, curious to see how much of it this kind of code generation would consume. And the number didn't budge. At least in my setup, regular ChatGPT with the GitHub integration could handle fairly substantial tasks on High reasoning without touching my Codex allowance. Of course, generating 20,000 lines of code doesn't automatically mean you've produced 20,000 lines of verified, production-ready software, but as a first test of the approach, the results were promising.
Slower, but You Can Go for a Walk
One thing became clear pretty quickly: generating code through ChatGPT was noticeably slower than using my usual local coding agent. ChatGPT also didn't have direct access to my computer, local runtime, or development tools, so its ability to validate the generated code was different from what Codex could do inside my IDE. That's a disadvantage, of course, since the code would still need to pass static analysis, build checks, and actual tests. On the other hand, if the agent doesn't need my computer to do its job, I don't necessarily need to be sitting in front of it either.
And that turned out to be one of my favorite parts of the whole experience. I actually launched some of those first tasks while taking my kid for a walk, with nothing but my phone and a mobile internet connection. Because I'd planned the tasks ahead of time and made them substantial enough, I didn't have to babysit the agent or intervene every few minutes. Checking my phone once every 10–15 minutes was usually enough to see how things were going and provide another instruction if needed. The rest of the time, I could focus on spending time with my kid while new commits quietly appeared in a GitHub repository somewhere in the cloud.
In a way, I'd found a way to turn time I wouldn't normally spend programming into actual development output. And it wasn't really about squeezing more working hours into my day, but about organizing the work differently. When a task is well-defined, being a developer doesn't necessarily mean sitting in front of an IDE and watching every step of the implementation. You can get on with your life and check in with the agent only when it actually needs your attention.
Another feature I particularly appreciated was ChatGPT's notifications when a conversation finished its work. As someone who constantly feels the urge to check whether a process is still running or if something has gone wrong, I found them surprisingly useful. They're especially handy when you have several conversations running at once, since you can rely on notifications instead of constantly switching between chats. That said, the notifications didn't always accurately reflect the actual state of the task, so I wouldn't recommend trusting them blindly.
Multiple Agents, Multiple Branches, Almost No Conflicts
Another interesting discovery was that independent tasks could run in parallel. If your repository is well-structured and packages have clearly defined responsibilities, there's little reason to make a single agent handle everything sequentially. I opened separate ChatGPT conversations, gave each agent its own task, and instructed it to work in a dedicated Git branch. As a result, several agents were modifying different parts of the same project simultaneously without getting in each other's way.
Of course, this approach requires a bit of discipline. I never allowed agents to modify the main branch directly, keeping their changes separate for later review and integration. The monorepo architecture helped a lot here: when packages have clear boundaries, it's much easier to distribute work without having two agents editing the same files. That doesn't eliminate the possibility of conflicts, but it certainly reduces the chances.
To give you an idea of how well this worked, merging the changes from those separate branches turned out to be surprisingly painless. I did run into a few conflicts, but resolving all of them took roughly two minutes. For me, that was a nice confirmation that parallel AI development is perfectly practical when tasks are properly decomposed. It also makes the slower generation speed of individual ChatGPT agents much less of a concern, since you can compensate for some of that delay by running multiple tasks at once.
Limitations and Unexpected Surprises
It would be suspicious if everything had worked perfectly, especially considering I was figuring things out through trial and error. ChatGPT has its own usage limits, and being able to launch several complex tasks simultaneously doesn't mean you have unlimited computing power. I didn't investigate the exact concurrency limits or track how many tokens each task consumed. What I can say is that three parallel conversations handling complex tasks on High reasoning worked reasonably well in my experience, and I didn't hit any usage limits during this initial experiment.
The actual problems came from somewhere else: occasionally, a task would stop unexpectedly. Some of these interruptions were probably related to unstable mobile internet, since I was using ChatGPT over 4G, including during air raid alerts here in Ukraine. Usually, once the connection recovered, I could continue without much trouble. But on several occasions, ChatGPT returned an error saying it couldn't verify the integrity of a response, and the agent effectively stopped working.
I can't say for sure what caused those errors. Perhaps a data stream was interrupted somewhere, or the system couldn't properly process the result of a tool call. Whatever the reason, the outcome was the same: the agent stopped, leaving some of its work already committed to the repository. That's when I discovered another quirk worth knowing about.
Simple commands like continue, go, or next didn't always help in these situations. The agent sometimes seemed to lose track of the interrupted operation or couldn't figure out exactly where to pick things up. What worked much better was giving it explicit instructions: you've already been working on this task, there are commits in your branch, so inspect the current state of the repository, figure out what's been implemented, and resume from where you left off. That usually got things moving again without having to restart the entire task.
After running into this a few times, I came away with a simple lesson: when working with a remote coding agent, Git history isn't just a way to store code. It's also a recovery mechanism. If a task is broken down into manageable stages and the agent regularly commits its progress, even a failed conversation doesn't necessarily mean losing the work that's already been done. This becomes especially important for long-running tasks, where starting over could be expensive in both time and resources.
Overall, my first experience turned out to be far more interesting than I'd expected. I generated roughly 20,000 lines of code, delegated several architectural tasks simultaneously, and barely touched my Codex quota in the process. But one question remained: how good was the generated code, and how much additional work would it take to make sure everything actually worked? That question led me to the next stage: a much more controlled experiment.
From First Impressions to a Controlled Experiment
The initial results were promising enough to make me want to take the idea further. Generating 20,000 lines of code without spending any Codex quota was certainly satisfying, but the number of lines alone says very little about the quality of the result. I was also curious how expensive it would be to bring that code to a working state if I delegated generation entirely to ChatGPT and used local Codex only for validation and bug fixing. So instead of running another AI coding demo, I decided to set up a controlled experiment on a real project.
The idea was simple: take the exact same task, give two agents as close to identical starting conditions as possible, and compare the total cost of getting a finished, working result. One agent would run locally through Codex in my IDE, while the other would work through ChatGPT's GitHub integration, without access to my computer. Both would use High reasoning, identical instructions, and the same test specifications. Once generation was complete, I'd subject both implementations to independent code reviews and then use the same local development environment to fix the identified problems and validate the results. This would allow me to compare not just how much code each agent produced, but also the effort required to bring its implementation into a working state.
Why Tests?
For the experiment, I chose automated tests for my backend API. My server has quite a few route handlers covering different business rules, request validation, authorization, and database interactions. I wanted a comprehensive test suite that would verify actual API behavior, support parallel execution, and run against an isolated test database. Ideally, the tests would spin up temporary Docker containers, prepare their own test data, and clean up after themselves.
Why tests instead of another product feature? Mainly because they give you a reasonably objective way to evaluate the result. For each test, you can check whether the required scenario was implemented, whether the assertions actually verify the intended behavior, and whether the test passes when executed. A green test suite doesn't automatically mean the tests are good, but combined with clearly defined behavioral requirements, it gives you a much more concrete definition of done than a subjective assessment of newly generated functionality.
There was another reason, too. I wasn't interested in comparing how well the agents could invent test scenarios, analyze coverage, or decide which edge cases deserved attention. That would have been a different experiment, where differences in the results might come down to how each agent interpreted the requirements rather than how it generated the code. What I wanted to measure was much simpler: how many resources would two different workflows need to implement the same, predefined amount of work?
Requirements First, Generation Second
I started by preparing the testing infrastructure so neither agent would waste time solving the same setup problems. Then I asked an independent agent to analyze all the route handlers in my server and produce a list of test scenarios. At this stage, it wasn't supposed to implement anything, just identify the behaviors that needed verification. The result was a set of Markdown documents containing references to specific route files and lists of required tests.
I tried to write the test names in a BDD style, making each one describe a specific behavioral requirement rather than the details of its implementation. For example, instead of something vague like should work correctly, a scenario such as should reject a request with an invalid identifier without modifying the database tells you much more. The name itself communicates the expected behavior and what the test needs to verify. At the same time, it leaves the agent free to decide which tools, fixtures, and code structure to use.
In total, I ended up with a specification covering 476 tests. I didn't want to throw all of them at the model at once, even with High reasoning enabled. A large context window doesn't necessarily mean the agent will give every requirement equal attention, and an excessively long task can make it harder to keep track of progress. So I split the specification into five parts, each containing roughly a hundred scenarios, and decided to feed them to the agents one at a time.
Same Prompts, Same Context
For each workflow, I created a separate Git branch from the same starting point in the repository. One branch was handled by local Codex in my IDE, while the other was handled by ChatGPT through GitHub. Both agents received the exact same initial prompt, which briefly described the task and the expected outcome. I deliberately kept it short instead of spelling out the project's architecture, since the repository already contained detailed instructions for AI agents, package conventions, and documentation of the existing infrastructure.
Both agents started with the same Markdown document containing the first batch of test scenarios. Once they completed it, I provided the next document, repeating the process until all five batches were done. The remaining four tasks didn't even come with any extra explanation: I simply attached the next document to the relevant conversation and hit Enter. I wanted to minimize the influence of additional hints and let each agent work with the project context already available to it.
Of course, I still managed to make a small mistake in the preparation. My initial prompt was missing one of the most important architectural requirements: the tests had to work against a real test database rather than verify the behavior of a mocked persistence layer. I realized this only after the first generation stage, so I had to introduce an additional task, which I called 1.1. Both agents received identical instructions to refactor the tests they'd already written to use isolated Docker containers with a test database.
So instead of the original five stages, I ended up with six tasks for each agent: the initial generation, a correction to the architectural requirements, and four more batches of test specifications. It wasn't exactly a flawless experiment from a preparation standpoint, but the important thing was that both agents received the same correction. Besides, this little mistake introduced an interesting variable of its own: I could now see how much extra effort each agent needed to rework existing code after a change in requirements.
Different Execution Environments
This brings us to one of the fundamental differences between the two approaches. Local Codex has access to my development environment, so it can execute commands, run TypeScript checks and tests, inspect failures, and fix problems immediately. ChatGPT with the GitHub integration, on the other hand, works primarily with the remote repository and doesn't have the same direct access to my IDE or local development tools. So even though both agents received the same task, they had very different options for validating their own work.
However, during generation, I noticed something interesting. The cloud agent discovered the GitHub Actions workflows already configured in my repository and started using CI to run checks. In other words, it wasn't just mentally simulating what the code might do. It found a way to get feedback from actual executions, even if that meant going through a much slower cycle of commits and workflow runs. That was a pleasant surprise, since it partially made up for the lack of a local runtime.
Of course, there's an economic side to this as well. GitHub Actions consumes computing resources, and the free CI allowance comes with limits depending on your plan, repository visibility, and runner configuration. I'm using a free GitHub account, and I didn't pay anything extra for those workflow runs. Still, it's worth accounting for this resource when comparing costs: not receiving a bill doesn't mean the computation itself is free.
How Do You Compare Results Across Different Environments?
The difference in available tools is exactly why I decided not to treat code generation as the finish line. Once both agents had completed the six generation tasks, I preserved their original revisions and conducted four independent, read-only audits of each implementation: ChatGPT Medium and High through GitHub, and local Codex Medium and High. All reviewers received the same five specification documents, identical evaluation requirements, and access to the exact same revision of the implementation they were reviewing. The local agents could run TypeScript checks, ESLint, Jest, and Docker, while the cloud agents relied on the GitHub integration and available CI results. Their task was to identify problems and provide evidence without modifying the code.
After completing all eight audits, I consolidated the findings into a separate report for each implementation. I removed duplicates and separated findings that actually described multiple independent defects. This gave me a more complete picture of the problems identified across all four reviews, rather than relying on the observations of a single agent. I also removed information about which reviewer had reported each issue, so the agent responsible for evaluating and fixing the problems wouldn't be influenced by the reviewer's identity.
For the remediation stage, I used local Codex with Medium reasoning for both implementations. At this point, the task was no longer about designing functionality from scratch, but about verifying the reported issues, implementing the necessary corrections, and running the tests. Both implementations were fixed in the same local development environment, under comparable conditions. For the final cost comparison, however, I counted only one High-reasoning audit per workflow: Local High for the Codex implementation and Cloud High for the ChatGPT implementation. The other independent reviews contributed to the consolidated remediation reports but were not included in that simplified cost calculation.
Counting More Than Just Tokens
Since the whole point of the experiment was to save Codex quota, I tracked its consumption after each generation and validation stage. This would show how much of the allowance a complete local Codex workflow consumed compared with an approach where ChatGPT handled code generation and Codex was brought in only for the final checks. But I also wanted to know how much code each agent had written before any fixes, and how extensive those fixes turned out to be.
To measure that, I organized both implementations into separate Pull Requests and divided their Git histories into two distinct ranges. The first covered the initial generation and showed how many lines had been added or deleted before the independent audit began. The second covered the remediation commits made by the local agent after reviewing the implementation. This allowed me to measure the amount of generated code and the subsequent rework separately, without confusing either with the final size of the Pull Request.
And I wasn't particularly interested in the raw line counts alone. If one agent wrote 5,000 lines of tests and the other wrote 7,000, that wouldn't necessarily tell me which approach was better. The number of missing scenarios, incorrect tests, architectural violations, and lines that needed to be added, removed, or rewritten after the audit would be far more informative. Especially when considered alongside the Codex quota consumed and the total time each workflow took.
Ultimately, I wanted to answer a very practical question: could I delegate most of the code generation to ChatGPT, use local Codex only for validation and fixes, and save a significant portion of my Codex quota in the process? And could I do that without ending up with hundreds of manual corrections or a pile of unreliable tests, but with a fully working implementation that met the exact same requirements in both workflows? If the approach actually worked, how much could I save compared with my usual local Codex workflow?
The Results: How Much Did Code Generation Actually Cost?
With both implementations finished, I finally had enough data to move beyond impressions and start comparing numbers. The first thing I wanted to measure was the cost of code generation itself, before bringing independent reviews and subsequent fixes into the picture. Both agents had received the same specifications, worked through the same six tasks, and used High reasoning. The difference was where they worked: one was running locally through Codex, while the other operated through ChatGPT and GitHub.
Generation Time and Codex Quota
I tracked the execution time of every task and, for local Codex, the percentage of my five-hour usage quota it consumed. The quota figures represent the amount spent on each individual task, not the remaining allowance shown afterward. I deliberately kept those two measurements separate because the entire experiment eventually stretched across more than one quota window. For this first comparison, though, the six generation tasks give us a straightforward picture.
| Task | Local Codex High | Codex Quota Used | ChatGPT High + GitHub |
|---|---|---|---|
| Task 1 | 6m 17s | 8% | 12m 44s |
| Task 1.1 | 10m 29s | 13% | 21m 02s |
| Task 2 | 11m 30s | 20% | 16m 04s |
| Task 3 | 4m 49s | 6% | 11m 54s |
| Task 4 | 6m 36s | 8% | 11m 15s |
| Task 5 | 6m 35s | 10% | 8m 59s |
| Total | 46m 16s | 65% | 1h 21m 58s |
The local Codex agent completed all six tasks in just over 46 minutes, consuming 65% of a five-hour Codex quota in the process. ChatGPT took almost 82 minutes to complete the same sequence, making it approximately 77% slower. That's a fairly substantial difference, and it confirms what I'd already noticed during my earlier experiments: local Codex is considerably faster at this kind of work. But the cloud workflow consumed none of my separate Codex quota.
That last distinction is the entire reason I started experimenting with this approach. I wasn't measuring whether ChatGPT somehow performed the computation for free, because it obviously didn't. I was measuring how much of my limited Codex allowance I could preserve by moving code generation into a service I was already using. And in this particular case, the difference was striking: 65% of my Codex quota versus no consumption of that quota at all, although ChatGPT remained subject to its own subscription limits.
How Much Code Did They Produce?
Execution time and quota consumption were only part of the picture. I also compared the Git changes produced by each agent before the independent audit and remediation stages began. Since the task involved both writing route tests and supporting them with real PostgreSQL infrastructure, I counted the complete implementation diff rather than test files alone. This gave me a consistent measure of how much code each workflow introduced.
| Metric | Local Codex High | ChatGPT High |
|---|---|---|
| Lines added | 5,032 | 6,214 |
| Lines deleted | 26 | 42 |
| Net lines changed | +5,006 | +6,172 |
Interestingly, the cloud agent produced 1,182 more added lines, roughly 23% more than local Codex. Both implementations accounted for the same 476 distinct required scenarios, although their test organization and supporting infrastructure differed. More code isn't automatically better code, of course, and at this stage I had no reason to assume the larger implementation was more complete or more reliable. That's precisely what the independent reviews were supposed to establish.
So the initial trade-off was clear. Local Codex delivered the implementation considerably faster, but at a substantial cost to its usage quota. ChatGPT took longer and generated a larger codebase, yet left that separate quota untouched. The remaining question was whether the savings would survive the next stages, when both implementations had to undergo code review, verification, and whatever fixes were necessary to make them genuinely reliable.
Who Does a Better Job Reviewing Code?
Writing several thousand lines of code in a relatively short time is certainly impressive. But the amount of code written tells us very little about how well it was implemented. So, after the generation phase, I decided to run another experiment: compare how AI agents with different reasoning levels and execution environments handle the analysis of a large, already implemented codebase.
For each of the two implementations, I ran four independent audits: ChatGPT Medium and High through GitHub, and local Codex Medium and High. All agents received the same review requirements, the five original specification documents, and access to the same revision of the implementation they were reviewing. However, their capabilities differed: local agents could independently run TypeScript, ESLint, Jest, and Docker, while cloud agents worked through the GitHub integration and could examine available CI results. Each agent was tasked with identifying problems, providing evidence, and producing a report without making any fixes.
Once the audits were complete, I consolidated all the reports into a separate findings registry for each implementation, removed duplicates, and split findings that actually described several independent defects. Another local agent running at Medium reasoning then examined each finding and attempted to fix the confirmed problems. It worked from the consolidated list without knowing which reviewer had originally reported each issue. This allowed me to evaluate not just the number of findings, but how well-founded they actually were.
Reviewing the Code Generated by ChatGPT
Let's start with the cloud agent's implementation. The four independent reviews produced a total of 31 initial findings, which were consolidated into 23 distinct defect candidates after normalization. Subsequent verification confirmed 19 problems, of which 16 were fixed and three remained unresolved. Another four cases were classified as Unverifiable: the agents had observed intermittent test failures, but their root causes could not be established.
| Metric | Cloud Medium | Cloud High | Local Medium | Local High |
|---|---|---|---|---|
| Review time | 4:13 | 15:12 | 5:56 | 10:41 |
| Codex quota consumed | 0% | 0% | 7% | 27% |
| Defect candidates identified | 5 | 11 | 6 | 18 |
| Confirmed findings | 5 | 11 | 5 | 15 |
| False Positives | 0 | 0 | 0 | 0 |
| Unverifiable | 0 | 0 | 1 | 3 |
| Unique confirmed findings | 0 | 3 | 0 | 4 |
| Known defects detected | 26.3% | 57.9% | 26.3% | 78.9% |
The results were quite revealing. Local Codex High found the most confirmed problems: 15 out of 19, although the review consumed 27% of its five-hour quota. ChatGPT High found 11 problems, taking more time but consuming none of the separate Codex quota. Interestingly, three confirmed problems were identified exclusively by Cloud High, while another four were found only by Local High.
None of the findings was ultimately classified as a False Positive, but that doesn't mean every finding was confirmed. Four cases remained unresolved due to insufficient evidence, so I excluded them from the confirmed defect count. It's also important to remember that passing tests don't guarantee those tests are correct: a significant proportion of the findings concerned insufficiently precise assertions rather than failing tests.
Reviewing the Code Generated by Local Codex
Now let's look at the review results for the local implementation. Here, the four reviewers produced 27 initial findings, which were consolidated into 17 distinct candidates after removing duplicates. Subsequent verification confirmed 13 problems, 12 of which were fixed. One finding turned out to be a false positive, while another three were classified as outside the agreed scope of work.
| Metric | Cloud Medium | Cloud High | Local Medium | Local High |
|---|---|---|---|---|
| Review time | 5:02 | 16:04 | 5:04 | 5:04 |
| Codex quota consumed | 0% | 0% | 6% | 7% |
| Defect candidates identified | 4 | 9 | 6 | 8 |
| Confirmed findings | 3 | 6 | 5 | 8 |
| False Positives | 1 | 1 | 0 | 0 |
| Out of Scope | 0 | 2 | 1 | 0 |
| Unique confirmed findings | 1 | 4 | 0 | 2 |
| Known defects detected | 23.1% | 46.2% | 38.5% | 61.5% |
Once again, Local Codex High found the most confirmed problems: eight out of thirteen. But this time, the difference in resource consumption was much smaller. Both local agents completed their reviews in approximately five minutes, consuming 6% and 7% of their quotas, respectively. Cloud High spent over 16 minutes and identified six confirmed problems, four of which no other agent detected. Meanwhile, Cloud Medium found only three genuine problems out of the four it reported, although one of those was its unique finding.
This time, false positives also appeared. Cloud Medium and Cloud High reported the same issue, which verification later revealed to be the result of an incorrect comparison of Git revisions. Another three findings were excluded from the confirmed defect statistics as Out of Scope. This clearly demonstrates why simply comparing the number of findings in each report would have been misleading.
Results Across Both Experiments
Combining the results of both implementations gives us 40 distinct defect candidates, 32 of which were confirmed during subsequent verification. Of those, 28 were fixed and four remained unresolved. Another four cases could not be sufficiently verified, one turned out to be a False Positive, and three were excluded as Out of Scope. Importantly, these findings concern two different implementations, so even similar defects are counted separately for each one.
| Metric | Cloud Medium | Cloud High | Local Medium | Local High |
|---|---|---|---|---|
| Defect candidates identified | 9 | 20 | 12 | 26 |
| True Positives | 8 | 17 | 10 | 23 |
| False Positives | 1 | 1 | 0 | 0 |
| Out of Scope | 0 | 2 | 1 | 0 |
| Unverifiable | 0 | 0 | 1 | 3 |
| Unique confirmed findings | 1 | 7 | 0 | 6 |
| Precision | 88.9% | 94.4% | 100% | 100% |
| Observed Recall | 25.0% | 53.1% | 31.3% | 71.9% |
Here, Precision measures the proportion of an agent's adjudicated findings that turned out to be genuine problems, excluding Out of Scope and Unverifiable cases. Observed Recall measures how many of the 32 known confirmed problems each agent managed to identify. This is not an absolute measure of review completeness, since some defects may have gone unnoticed by all four agents. Likewise, a Precision score of 100% doesn't mean the model is infallible: the sample is relatively small, and the final decisions were made by another AI agent, albeit one that also inspected the code and executed tests.
Across both implementations, Local Codex High ranked first in defect detection, identifying 23 of the 32 confirmed problems. ChatGPT High found 17, outperforming Local Codex Medium, which identified ten. At the same time, seven confirmed findings were unique to Cloud High, while six were unique to Local High. This means that even the most resource-intensive configuration could not replace all the others.
On the other hand, these results don't justify claiming that Local High will always review code better or that Cloud Medium will always perform worst. We compared only two specific implementations of the same requirements, and the execution environments had different capabilities for running checks. Still, an interesting trend emerges: increasing the reasoning level in both environments was accompanied by a substantial increase in the number of confirmed problems detected. ChatGPT High through GitHub also demonstrated that a high-quality audit doesn't necessarily require direct access to a local development environment.
How Much Did It Cost to Fix the Bugs?
After four independent audits of each implementation, it was time to fix the identified problems. And this is where I decided to simplify the experiment a little.
Instead of using a single agent's report, I combined the results of all four reviews, removed duplicates, and created one consolidated list of problems for each implementation. This way, the agent responsible for the fixes received the most complete picture possible, regardless of which reviewer had originally identified a particular defect.
For both implementations, I used local Codex with Medium reasoning. In my view, High was no longer quite as necessary at this stage: most of the investigative work had already been done during the audits. What remained was to verify the validity of the findings, implement the fixes, and ensure the tests passed.
Time, Quota, and Scope of Changes
| Metric | Local Codex Code | ChatGPT + GitHub Code |
|---|---|---|
| Time spent fixing bugs | 23m 01s | 12m 30s |
| Codex quota consumed | 44% | 24% |
| Initial lines of code added | 5,032 lines | 6,214 lines |
| Lines added during fixes | 297 lines | 346 lines |
| Lines deleted during fixes | 125 lines | 5 lines |
| Total volume of changes | 422 lines | 351 lines |
| Changes relative to initial code volume | 8.4% | 5.6% |
The percentage of changes is calculated by dividing the sum of added and deleted lines during remediation by the number of lines added during the initial implementation. This measures Git code churn, not the literal percentage of code that was rewritten.
And this is where I encountered a rather unexpected difference.
Fixing the code generated by local Codex took 23 minutes and consumed 44% of the quota. Meanwhile, the code created through ChatGPT was refined in just 12.5 minutes, consuming only 24%.
In other words, the local implementation required approximately 1.84 times as much time and 1.83 times as much quota, even though its initial codebase was smaller.
Interestingly, the ratios for time and quota consumption are almost identical. This suggests that the unusually high 44% figure is unlikely to be explained by some special cost associated with Medium reasoning on that particular task. A much simpler explanation is that the agent simply worked longer.
But why? One possible factor was flaky tests that occasionally failed and then passed when rerun. This may have forced the agent to spend additional time repeating checks and investigating failures. However, I can't determine exactly how much time was spent on those cases. Therefore, the 44% should be treated as the actual measured result of this particular run, not as a guaranteed cost for anyone attempting to reproduce the experiment.
The difference in the volume of changes was interesting too. In the local implementation, the agent added 297 lines and deleted 125, while in the cloud implementation it added 346 and deleted only five. Comparing the total volume of these changes against the original implementation size gives us 8.4% versus 5.6%.
However, it would be wrong to conclude that one implementation was better simply because it required a smaller percentage of changes. Fixes can involve either rewriting existing code or adding new checks. So these numbers are better understood as a measure of the amount of rework required, rather than a direct measure of the initial implementation's quality.
Final Results: Which Approach Was More Cost-Effective?
Now it's time to put all the numbers together and answer the main question behind this experiment: can you really save Codex quota by moving code generation into ChatGPT and using the local agent primarily for reviews and fixes?
For this comparison, I decided to include three main stages of work: code generation, independent Code Review, and fixing the identified problems.
For each approach, I'm counting the High-level review performed in its corresponding environment. That means Local High for the code generated by local Codex, and Cloud High for the code generated through ChatGPT. In both cases, the fixes were performed by local Codex Medium.
How Much Did the Full Cycle Cost?
| Stage | Local Codex: Time | Codex Quota | ChatGPT + GitHub: Time | Codex Quota |
|---|---|---|---|---|
| Generation — High | 46:16 | 65% | 1:21:58 | 0% |
| Code Review — High | 5:04 | 7% | 15:12 | 0% |
| Fixes — Local Medium | 23:01 | 44% | 12:30 | 24% |
| Total | 1:14:21 | 116% | 1:49:40 | 24% |
The quota percentages are summed across separate stages, which may have taken place in different five-hour usage windows. This does not mean that 116% was consumed within a single window. The zero values for ChatGPT mean zero consumption of the separate Codex quota, not an absence of all costs or usage limits.
And this is where things get really interesting.
Local Codex was approximately 35 minutes faster. The complete measured cycle took 1 hour and 14 minutes, compared with almost 1 hour and 50 minutes for ChatGPT. In other words, the cloud-based approach required approximately 47.5% more time.
But when we look at Codex quota consumption, the picture changes dramatically.
The local approach consumed a total of 116% of a five-hour quota, while the cloud-based approach consumed just 24%. This means that ChatGPT reduced Codex quota consumption by approximately 4.8 times, or 79%.
For me, this is the main result of the experiment. I obtained two functionally comparable results while spending almost five times less of the local agent's limited resources on one of them. The price of those savings was approximately 35 additional minutes of waiting.
Of course, this isn't a monetary saving in the literal sense. ChatGPT has its own pricing model and limits, and my calculations specifically concern Codex quota consumption. But if you already use both services and regularly run into Codex's five-hour usage limit, the difference can be quite significant.
But What About the Quality of the Generated Code?
Here, the results were less straightforward.
Both agents implemented the same 476 unique test scenarios. However, independent audits found different numbers of problems in their respective implementations.
| Metric | Local Codex | ChatGPT + GitHub |
|---|---|---|
| Confirmed problems | 13 | 19 |
| Time spent on fixes | 23:01 | 12:30 |
| Quota consumed by fixes | 44% | 24% |
| Changes relative to initial code volume | 8.4% | 5.6% |
Judging by the number of confirmed defects, the initial local Codex implementation appears to be better: reviewers found 13 problems in it, compared with 19 in ChatGPT's implementation. In other words, the cloud implementation had approximately 46% more confirmed findings.
But here's the paradox: fixing ChatGPT's code required almost half as much time and quota, and the relative volume of changes was smaller — 5.6% versus 8.4%.
This is a good example of why the number of defects and the difficulty of fixing them aren't always directly correlated. Some problems can be resolved by adding a few assertions, while others require substantial changes to the testing infrastructure or a different testing strategy altogether.
I wouldn't try to convert these numbers into something like "95% correct code." A Git diff measures the volume of changes, not the percentage of correct implementation. And the number of identified problems doesn't guarantee that reviewers found absolutely every defect.
So, in terms of initial code generation quality, I would favor local Codex because it received fewer confirmed findings. But when it comes to the cost of subsequent remediation, ChatGPT's implementation won this particular experiment.
Which Agent Is Better at Code Review?
Here, the winner was much clearer.
Combining the review results for both implementations gives us 32 confirmed problems. Here's how many of them the two High-reasoning agents managed to identify:
| Metric | Local High | Cloud High |
|---|---|---|
| Confirmed problems identified | 23 out of 32 | 17 out of 32 |
| Known problems detected | 71.9% | 53.1% |
| Unique confirmed findings | 6 | 7 |
| Codex quota consumed across two reviews | 34% | 0% |
Local Codex High proved to be the more effective reviewer, finding almost 72% of known problems compared with ChatGPT High's 53%. A difference of nearly 19 percentage points is quite substantial.
But the cloud agent's performance shouldn't be underestimated either. It found 17 confirmed defects — approximately 74% of the number identified by Local High. What's more, seven of those defects went unnoticed by every other reviewer.
And it achieved all of this without consuming any of the separate Codex quota.
So here's my conclusion: if I needed the most thorough Code Review possible and quota wasn't a concern, I would choose Local High. But if I needed to regularly review large amounts of code without burning through Codex limits, ChatGPT High with GitHub would be a very attractive alternative.
Better yet, I'd use them as independent reviewers. After all, the experiment showed that they don't simply find different numbers of problems — they also catch defects that the other agent misses.
My Final Verdict
After all these experiments, I've come to the conclusion that there's little reason to force a single tool to do absolutely everything.
Local Codex is faster, and its initial implementation received fewer confirmed findings in this experiment. At the High reasoning level, it also demonstrated the strongest Code Review performance.
But ChatGPT with GitHub delivered a functionally comparable result while consuming almost five times less Codex quota. Yes, it took approximately 47.5% more time, but for large tasks that don't require a developer's constant attention, that seems like a perfectly reasonable trade-off.
I'll probably adopt a hybrid approach going forward: ChatGPT for large-scale code generation and independent audits, and local Codex for tasks where speed is critical, direct code execution is necessary, or maximum review thoroughness matters.
The main idea is simple: you don't have to use the most quota-expensive tool at every stage to get a good result. Sometimes, all you need is to distribute the work intelligently between several agents.
Limitations of the experiment. The cost comparison presented here sums the recorded generation stage, one selected High-level review, and the measured remediation run. In reality, I used the combined results of four independent audits to prepare the remediation tasks, not just the single audit included in the cost table. There were also additional verification and remediation iterations after the initial fixes, for which time and quota consumption were not fully recorded. Therefore, the 116% and 24% figures should be interpreted as a comparison of selected, measured stages rather than the complete actual cost of every action performed during the experiment. Furthermore, several problems remained unresolved, and some intermittent test failures never received a definitive explanation.
A Small Clarification About "Free Codex"
I can already anticipate a perfectly fair question: if I promised Codex without consuming its quota, why did I end up using local Codex Medium to fix the bugs?
The thing is, I believe it's entirely possible to bring the code into a working state using ChatGPT with GitHub as well. But since the result still needs to be validated in a real execution environment, I decided to simplify the experiment and have a local agent fix both implementations under identical conditions.
Even with this approach, Codex quota consumption was reduced by approximately 4.8 times across the stages included in the comparison. And I suspect it's also realistic to avoid consuming the local Codex quota entirely: use ChatGPT for generation, analysis, and fixes, while running checks locally and feeding the results back into the chat. Of course, such a workflow would most likely require considerably more time and manual effort.
So "free" here means no consumption of the separate Codex quota, not the absence of a subscription, usage limits, or any other costs whatsoever. I didn't test a fully cloud-based development cycle in this experiment, but the results give me reason to believe that it's a promising approach.
Bonus: Does More Code Mean Better Code?
Toward the end of the experiment, I noticed something interesting: ChatGPT had generated 23.5% more code than local Codex — 6,214 added lines versus 5,032.
That raised another question: was the additional code actually justified, or had the cloud agent simply produced more boilerplate and duplication?
To find out, I asked an independent cloud-based AI reviewer to compare both original implementations, before any remediation. The reviewer examined functional correctness, requirements compliance, test reliability, architecture, decomposition, DRY, maintainability, and implementation complexity.
This time, I wasn't interested in how many individual bugs each implementation contained. I wanted an overall engineering assessment: which codebase would be better to maintain and extend?
Here's the verdict:
| Evaluation criterion | ChatGPT + GitHub | Local Codex |
|---|---|---|
| Functional correctness | ✓ | — |
| Requirements compliance | ✓ | — |
| Test reliability | ✓ | — |
| Architecture & Separation of Concerns | Tie | Tie |
| Decomposition | Tie | Tie |
| DRY & Reusability | Tie | Tie |
| Maintainability | Tie | Tie |
| Implementation complexity | — | ✓ |
| Code efficiency | — | ✓ |
| Score | 7.0 / 9 | 6.0 / 9 |
Scoring: Each implementation receives 1 point for a category win and 1 point for a tie. Scores are not zero-sum: a tie indicates comparable performance, so neither implementation is penalized. The maximum individual score is 9 points. This is a simplified summary of the reviewer's qualitative judgments, not a standardized software quality benchmark.
Interestingly, the reviewer gave ChatGPT the overall win, although with only moderate confidence.
Its implementation received stronger assessments for correctness, requirements compliance, and reliability. It featured more extensive database assertions, stricter test isolation, and a successful CI run on the exact revision being evaluated.
Local Codex, meanwhile, demonstrated better code efficiency and lower implementation complexity. Its centralized test infrastructure and greater use of parameterized tests allowed it to cover essentially the same requirements with substantially less code.
The reviewer found that approximately 86% of the difference in added lines came from the route tests themselves, rather than infrastructure. Some of ChatGPT's additional code provided meaningful assertions, while some came from repeated setup and less aggressive reuse.
There is one important qualification: the reviewer could verify a successful CI run for ChatGPT's original revision, but not for the exact local Codex revision. That gave ChatGPT stronger evidence of working behavior, without proving that the local implementation was broken. The final verdict was therefore conditional, not an absolute measurement of intrinsic code quality.
My takeaway? The additional 23.5% of code wasn't entirely wasted, but it wasn't entirely justified either. ChatGPT produced the implementation the independent reviewer preferred overall, while local Codex delivered a leaner solution with a similarly strong architectural foundation.
A rather fitting conclusion to this experiment: more code doesn't necessarily mean better code — but less code doesn't automatically mean better engineering either.
P.S. Thanks for Reading All the Way to the End!
If you found this article interesting or useful, I'd really appreciate a ❤️ and a follow!
I regularly run technical experiments, explore new approaches to software development, and share what I learn along the way. There's plenty more to come!
In the meantime, you might also enjoy my other articles:
- 💥 Easy way to save your AI coding agent a surprising amount of context — Learn how to optimize your AI-assisted development workflow and dramatically reduce token consumption.
- 100+ API Endpoints and Almost Zero Integration Bugs. Sharing the Framework — Discover my approach to managing complex API contracts and preventing integration issues.
Thanks for reading, and see you in the next experiment! 🚀
Top comments (0)