DEV Community: Mykola Mizhigurskiy

Codex at No Extra Cost: How I Turned ChatGPT into a Free AI Coding Agent

Mykola Mizhigurskiy — Fri, 16 Oct 2026 12:59:35 +0000

Recently, while working on a personal project, I stumbled upon an interesting feature in ChatGPT. Apparently, you can connect it to GitHub and give it access to your repositories. Nothing particularly groundbreaking at first glance, just another integration in a world already overflowing with AI tools. But it got me wondering how far I could push it. What if I used ChatGPT not just to analyze code, but as a proper coding agent capable of working directly with my repository?

By that point, I'd already been using Codex quite extensively, so I was familiar with both its capabilities and its biggest annoyance: usage limits. There's nothing quite like getting into the flow, breaking a large task into manageable pieces, and then hitting your five-hour quota just as things start getting interesting. That got me thinking: why spend my Codex allowance generating code if I could delegate some of that work to regular ChatGPT? Especially since I wasn't paying for Plus specifically for coding. I subscribed for everyday conversations, research, and all sorts of other things.

Of course, calling this approach completely free would be stretching the truth. ChatGPT Plus costs money, has its own limits, and computing power doesn't magically appear out of thin air. But if you're already paying for the subscription and your regular usage stays comfortably within its limits, having an additional way to generate code feels like a pretty nice bonus. At least in my case, I hadn't managed to hit ChatGPT's limits, even while using it for some fairly substantial development tasks.

The remaining question was whether this approach could do more than just dump code into a repository. Could it actually produce something useful without leaving me with hours of debugging afterward? There was only one way to find out. But first, I needed to set everything up.

Setting Up ChatGPT with GitHub

To get started, you'll need a ChatGPT account with access to the GitHub integration and, naturally, a repository to work with. Open the Plugins section in ChatGPT's settings, find GitHub, and start the connection process. You'll be redirected to GitHub, where you authorize the application and choose which repositories it can access. The setup is straightforward and doesn't require manually creating or configuring API tokens.

At this point, I'd strongly recommend taking a moment before clicking through the authorization screens. GitHub lets you choose between granting access to all repositories and the Only select repositories option, where you explicitly specify which repositories the application can access. I went with the latter, granting access only to the project I intended to work on. It's a straightforward application of the Principle of Least Privilege, something that's surprisingly easy to overlook when you're eager to try out a new tool.

It's equally important to review the permissions the application requests, especially if they include the ability to write to or modify repository contents. If an agent can change code, I wouldn't let it work directly on the main branch under any circumstances. It's much safer to create a separate branch for each independent task and review the results through a Pull Request. Besides making code reviews easier, this approach preserves the history of changes, lets you measure how much code the agent actually generated, and gives you a clean way to reject an implementation if things go sideways.

Once authorization is complete, make sure ChatGPT can access your repository and read its structure. There's one important detail, though: being able to read code doesn't necessarily mean your particular integration supports modifying it. Before getting started, check whether the agent can create branches, commits, and Pull Requests. In my case, all of these capabilities were available, so it was time for the interesting part: giving the agent a real task and seeing what happened.

Putting It to the Test: 20,000 Lines of Code in Half a Day

Back to my project. I'm working with a fairly large monorepo, where functionality is organized into separate packages with clearly defined responsibilities. While going about my usual development work, I picked out a few tasks I could delegate to ChatGPT without interfering with what I was doing locally. I wasn't particularly interested in starting with something trivial, like fixing lint errors or renaming variables. If I was going to test what this agent could do, I wanted to give it a proper architectural challenge: understand the existing codebase, work out an approach, and implement new functionality.

ChatGPT lets you choose the model's reasoning effort, so I decided to go straight for the strongest option available to me: High. I deliberately chose a setting intended for complex tasks rather than trying to save resources by using a less demanding mode. I don't know exactly how many tokens the agent burned through while reasoning, since I wasn't tracking that separately. But the amount of work it produced was substantial: in half a day, ChatGPT generated roughly 20,000 lines of code through GitHub, implementing new functionality in my monorepo.

But that wasn't even the most interesting part. Throughout the process, I kept an eye on my five-hour Codex quota, curious to see how much of it this kind of code generation would consume. And the number didn't budge. At least in my setup, regular ChatGPT with the GitHub integration could handle fairly substantial tasks on High reasoning without touching my Codex allowance. Of course, generating 20,000 lines of code doesn't automatically mean you've produced 20,000 lines of verified, production-ready software, but as a first test of the approach, the results were promising.

Slower, but You Can Go for a Walk

One thing became clear pretty quickly: generating code through ChatGPT was noticeably slower than using my usual local coding agent. ChatGPT also didn't have direct access to my computer, local runtime, or development tools, so its ability to validate the generated code was different from what Codex could do inside my IDE. That's a disadvantage, of course, since the code would still need to pass static analysis, build checks, and actual tests. On the other hand, if the agent doesn't need my computer to do its job, I don't necessarily need to be sitting in front of it either.

And that turned out to be one of my favorite parts of the whole experience. I actually launched some of those first tasks while taking my kid for a walk, with nothing but my phone and a mobile internet connection. Because I'd planned the tasks ahead of time and made them substantial enough, I didn't have to babysit the agent or intervene every few minutes. Checking my phone once every 10–15 minutes was usually enough to see how things were going and provide another instruction if needed. The rest of the time, I could focus on spending time with my kid while new commits quietly appeared in a GitHub repository somewhere in the cloud.

In a way, I'd found a way to turn time I wouldn't normally spend programming into actual development output. And it wasn't really about squeezing more working hours into my day, but about organizing the work differently. When a task is well-defined, being a developer doesn't necessarily mean sitting in front of an IDE and watching every step of the implementation. You can get on with your life and check in with the agent only when it actually needs your attention.

Another feature I particularly appreciated was ChatGPT's notifications when a conversation finished its work. As someone who constantly feels the urge to check whether a process is still running or if something has gone wrong, I found them surprisingly useful. They're especially handy when you have several conversations running at once, since you can rely on notifications instead of constantly switching between chats. That said, the notifications didn't always accurately reflect the actual state of the task, so I wouldn't recommend trusting them blindly.

Multiple Agents, Multiple Branches, Almost No Conflicts

Another interesting discovery was that independent tasks could run in parallel. If your repository is well-structured and packages have clearly defined responsibilities, there's little reason to make a single agent handle everything sequentially. I opened separate ChatGPT conversations, gave each agent its own task, and instructed it to work in a dedicated Git branch. As a result, several agents were modifying different parts of the same project simultaneously without getting in each other's way.

Of course, this approach requires a bit of discipline. I never allowed agents to modify the main branch directly, keeping their changes separate for later review and integration. The monorepo architecture helped a lot here: when packages have clear boundaries, it's much easier to distribute work without having two agents editing the same files. That doesn't eliminate the possibility of conflicts, but it certainly reduces the chances.

To give you an idea of how well this worked, merging the changes from those separate branches turned out to be surprisingly painless. I did run into a few conflicts, but resolving all of them took roughly two minutes. For me, that was a nice confirmation that parallel AI development is perfectly practical when tasks are properly decomposed. It also makes the slower generation speed of individual ChatGPT agents much less of a concern, since you can compensate for some of that delay by running multiple tasks at once.

Limitations and Unexpected Surprises

It would be suspicious if everything had worked perfectly, especially considering I was figuring things out through trial and error. ChatGPT has its own usage limits, and being able to launch several complex tasks simultaneously doesn't mean you have unlimited computing power. I didn't investigate the exact concurrency limits or track how many tokens each task consumed. What I can say is that three parallel conversations handling complex tasks on High reasoning worked reasonably well in my experience, and I didn't hit any usage limits during this initial experiment.

The actual problems came from somewhere else: occasionally, a task would stop unexpectedly. Some of these interruptions were probably related to unstable mobile internet, since I was using ChatGPT over 4G, including during air raid alerts here in Ukraine. Usually, once the connection recovered, I could continue without much trouble. But on several occasions, ChatGPT returned an error saying it couldn't verify the integrity of a response, and the agent effectively stopped working.

I can't say for sure what caused those errors. Perhaps a data stream was interrupted somewhere, or the system couldn't properly process the result of a tool call. Whatever the reason, the outcome was the same: the agent stopped, leaving some of its work already committed to the repository. That's when I discovered another quirk worth knowing about.

Simple commands like continue, go, or next didn't always help in these situations. The agent sometimes seemed to lose track of the interrupted operation or couldn't figure out exactly where to pick things up. What worked much better was giving it explicit instructions: you've already been working on this task, there are commits in your branch, so inspect the current state of the repository, figure out what's been implemented, and resume from where you left off. That usually got things moving again without having to restart the entire task.

After running into this a few times, I came away with a simple lesson: when working with a remote coding agent, Git history isn't just a way to store code. It's also a recovery mechanism. If a task is broken down into manageable stages and the agent regularly commits its progress, even a failed conversation doesn't necessarily mean losing the work that's already been done. This becomes especially important for long-running tasks, where starting over could be expensive in both time and resources.

Overall, my first experience turned out to be far more interesting than I'd expected. I generated roughly 20,000 lines of code, delegated several architectural tasks simultaneously, and barely touched my Codex quota in the process. But one question remained: how good was the generated code, and how much additional work would it take to make sure everything actually worked? That question led me to the next stage: a much more controlled experiment.

From First Impressions to a Controlled Experiment

The initial results were promising enough to make me want to take the idea further. Generating 20,000 lines of code without spending any Codex quota was certainly satisfying, but the number of lines alone says very little about the quality of the result. I was also curious how expensive it would be to bring that code to a working state if I delegated generation entirely to ChatGPT and used local Codex only for validation and bug fixing. So instead of running another AI coding demo, I decided to set up a controlled experiment on a real project.

The idea was simple: take the exact same task, give two agents as close to identical starting conditions as possible, and compare the total cost of getting a finished, working result. One agent would run locally through Codex in my IDE, while the other would work through ChatGPT's GitHub integration, without access to my computer. Both would use High reasoning, identical instructions, and the same test specifications. Once generation was complete, I'd subject both implementations to independent code reviews and then use the same local development environment to fix the identified problems and validate the results. This would allow me to compare not just how much code each agent produced, but also the effort required to bring its implementation into a working state.

Why Tests?

For the experiment, I chose automated tests for my backend API. My server has quite a few route handlers covering different business rules, request validation, authorization, and database interactions. I wanted a comprehensive test suite that would verify actual API behavior, support parallel execution, and run against an isolated test database. Ideally, the tests would spin up temporary Docker containers, prepare their own test data, and clean up after themselves.

Why tests instead of another product feature? Mainly because they give you a reasonably objective way to evaluate the result. For each test, you can check whether the required scenario was implemented, whether the assertions actually verify the intended behavior, and whether the test passes when executed. A green test suite doesn't automatically mean the tests are good, but combined with clearly defined behavioral requirements, it gives you a much more concrete definition of done than a subjective assessment of newly generated functionality.

There was another reason, too. I wasn't interested in comparing how well the agents could invent test scenarios, analyze coverage, or decide which edge cases deserved attention. That would have been a different experiment, where differences in the results might come down to how each agent interpreted the requirements rather than how it generated the code. What I wanted to measure was much simpler: how many resources would two different workflows need to implement the same, predefined amount of work?

Requirements First, Generation Second

I started by preparing the testing infrastructure so neither agent would waste time solving the same setup problems. Then I asked an independent agent to analyze all the route handlers in my server and produce a list of test scenarios. At this stage, it wasn't supposed to implement anything, just identify the behaviors that needed verification. The result was a set of Markdown documents containing references to specific route files and lists of required tests.

I tried to write the test names in a BDD style, making each one describe a specific behavioral requirement rather than the details of its implementation. For example, instead of something vague like should work correctly, a scenario such as should reject a request with an invalid identifier without modifying the database tells you much more. The name itself communicates the expected behavior and what the test needs to verify. At the same time, it leaves the agent free to decide which tools, fixtures, and code structure to use.

In total, I ended up with a specification covering 476 tests. I didn't want to throw all of them at the model at once, even with High reasoning enabled. A large context window doesn't necessarily mean the agent will give every requirement equal attention, and an excessively long task can make it harder to keep track of progress. So I split the specification into five parts, each containing roughly a hundred scenarios, and decided to feed them to the agents one at a time.

Same Prompts, Same Context

For each workflow, I created a separate Git branch from the same starting point in the repository. One branch was handled by local Codex in my IDE, while the other was handled by ChatGPT through GitHub. Both agents received the exact same initial prompt, which briefly described the task and the expected outcome. I deliberately kept it short instead of spelling out the project's architecture, since the repository already contained detailed instructions for AI agents, package conventions, and documentation of the existing infrastructure.

Both agents started with the same Markdown document containing the first batch of test scenarios. Once they completed it, I provided the next document, repeating the process until all five batches were done. The remaining four tasks didn't even come with any extra explanation: I simply attached the next document to the relevant conversation and hit Enter. I wanted to minimize the influence of additional hints and let each agent work with the project context already available to it.

Of course, I still managed to make a small mistake in the preparation. My initial prompt was missing one of the most important architectural requirements: the tests had to work against a real test database rather than verify the behavior of a mocked persistence layer. I realized this only after the first generation stage, so I had to introduce an additional task, which I called 1.1. Both agents received identical instructions to refactor the tests they'd already written to use isolated Docker containers with a test database.

So instead of the original five stages, I ended up with six tasks for each agent: the initial generation, a correction to the architectural requirements, and four more batches of test specifications. It wasn't exactly a flawless experiment from a preparation standpoint, but the important thing was that both agents received the same correction. Besides, this little mistake introduced an interesting variable of its own: I could now see how much extra effort each agent needed to rework existing code after a change in requirements.

Different Execution Environments

This brings us to one of the fundamental differences between the two approaches. Local Codex has access to my development environment, so it can execute commands, run TypeScript checks and tests, inspect failures, and fix problems immediately. ChatGPT with the GitHub integration, on the other hand, works primarily with the remote repository and doesn't have the same direct access to my IDE or local development tools. So even though both agents received the same task, they had very different options for validating their own work.

However, during generation, I noticed something interesting. The cloud agent discovered the GitHub Actions workflows already configured in my repository and started using CI to run checks. In other words, it wasn't just mentally simulating what the code might do. It found a way to get feedback from actual executions, even if that meant going through a much slower cycle of commits and workflow runs. That was a pleasant surprise, since it partially made up for the lack of a local runtime.

Of course, there's an economic side to this as well. GitHub Actions consumes computing resources, and the free CI allowance comes with limits depending on your plan, repository visibility, and runner configuration. I'm using a free GitHub account, and I didn't pay anything extra for those workflow runs. Still, it's worth accounting for this resource when comparing costs: not receiving a bill doesn't mean the computation itself is free.

How Do You Compare Results Across Different Environments?

The difference in available tools is exactly why I decided not to treat code generation as the finish line. Once both agents had completed the six generation tasks, I preserved their original revisions and conducted four independent, read-only audits of each implementation: ChatGPT Medium and High through GitHub, and local Codex Medium and High. All reviewers received the same five specification documents, identical evaluation requirements, and access to the exact same revision of the implementation they were reviewing. The local agents could run TypeScript checks, ESLint, Jest, and Docker, while the cloud agents relied on the GitHub integration and available CI results. Their task was to identify problems and provide evidence without modifying the code.

After completing all eight audits, I consolidated the findings into a separate report for each implementation. I removed duplicates and separated findings that actually described multiple independent defects. This gave me a more complete picture of the problems identified across all four reviews, rather than relying on the observations of a single agent. I also removed information about which reviewer had reported each issue, so the agent responsible for evaluating and fixing the problems wouldn't be influenced by the reviewer's identity.

For the remediation stage, I used local Codex with Medium reasoning for both implementations. At this point, the task was no longer about designing functionality from scratch, but about verifying the reported issues, implementing the necessary corrections, and running the tests. Both implementations were fixed in the same local development environment, under comparable conditions. For the final cost comparison, however, I counted only one High-reasoning audit per workflow: Local High for the Codex implementation and Cloud High for the ChatGPT implementation. The other independent reviews contributed to the consolidated remediation reports but were not included in that simplified cost calculation.

Counting More Than Just Tokens

Since the whole point of the experiment was to save Codex quota, I tracked its consumption after each generation and validation stage. This would show how much of the allowance a complete local Codex workflow consumed compared with an approach where ChatGPT handled code generation and Codex was brought in only for the final checks. But I also wanted to know how much code each agent had written before any fixes, and how extensive those fixes turned out to be.

To measure that, I organized both implementations into separate Pull Requests and divided their Git histories into two distinct ranges. The first covered the initial generation and showed how many lines had been added or deleted before the independent audit began. The second covered the remediation commits made by the local agent after reviewing the implementation. This allowed me to measure the amount of generated code and the subsequent rework separately, without confusing either with the final size of the Pull Request.

And I wasn't particularly interested in the raw line counts alone. If one agent wrote 5,000 lines of tests and the other wrote 7,000, that wouldn't necessarily tell me which approach was better. The number of missing scenarios, incorrect tests, architectural violations, and lines that needed to be added, removed, or rewritten after the audit would be far more informative. Especially when considered alongside the Codex quota consumed and the total time each workflow took.

Ultimately, I wanted to answer a very practical question: could I delegate most of the code generation to ChatGPT, use local Codex only for validation and fixes, and save a significant portion of my Codex quota in the process? And could I do that without ending up with hundreds of manual corrections or a pile of unreliable tests, but with a fully working implementation that met the exact same requirements in both workflows? If the approach actually worked, how much could I save compared with my usual local Codex workflow?

The Results: How Much Did Code Generation Actually Cost?

With both implementations finished, I finally had enough data to move beyond impressions and start comparing numbers. The first thing I wanted to measure was the cost of code generation itself, before bringing independent reviews and subsequent fixes into the picture. Both agents had received the same specifications, worked through the same six tasks, and used High reasoning. The difference was where they worked: one was running locally through Codex, while the other operated through ChatGPT and GitHub.

Generation Time and Codex Quota

I tracked the execution time of every task and, for local Codex, the percentage of my five-hour usage quota it consumed. The quota figures represent the amount spent on each individual task, not the remaining allowance shown afterward. I deliberately kept those two measurements separate because the entire experiment eventually stretched across more than one quota window. For this first comparison, though, the six generation tasks give us a straightforward picture.

Task	Local Codex High	Codex Quota Used	ChatGPT High + GitHub
Task 1	6m 17s	8%	12m 44s
Task 1.1	10m 29s	13%	21m 02s
Task 2	11m 30s	20%	16m 04s
Task 3	4m 49s	6%	11m 54s
Task 4	6m 36s	8%	11m 15s
Task 5	6m 35s	10%	8m 59s
Total	46m 16s	65%	1h 21m 58s

The local Codex agent completed all six tasks in just over 46 minutes, consuming 65% of a five-hour Codex quota in the process. ChatGPT took almost 82 minutes to complete the same sequence, making it approximately 77% slower. That's a fairly substantial difference, and it confirms what I'd already noticed during my earlier experiments: local Codex is considerably faster at this kind of work. But the cloud workflow consumed none of my separate Codex quota.

That last distinction is the entire reason I started experimenting with this approach. I wasn't measuring whether ChatGPT somehow performed the computation for free, because it obviously didn't. I was measuring how much of my limited Codex allowance I could preserve by moving code generation into a service I was already using. And in this particular case, the difference was striking: 65% of my Codex quota versus no consumption of that quota at all, although ChatGPT remained subject to its own subscription limits.

How Much Code Did They Produce?

Execution time and quota consumption were only part of the picture. I also compared the Git changes produced by each agent before the independent audit and remediation stages began. Since the task involved both writing route tests and supporting them with real PostgreSQL infrastructure, I counted the complete implementation diff rather than test files alone. This gave me a consistent measure of how much code each workflow introduced.

Metric	Local Codex High	ChatGPT High
Lines added	5,032	6,214
Lines deleted	26	42
Net lines changed	+5,006	+6,172

Interestingly, the cloud agent produced 1,182 more added lines, roughly 23% more than local Codex. Both implementations accounted for the same 476 distinct required scenarios, although their test organization and supporting infrastructure differed. More code isn't automatically better code, of course, and at this stage I had no reason to assume the larger implementation was more complete or more reliable. That's precisely what the independent reviews were supposed to establish.

So the initial trade-off was clear. Local Codex delivered the implementation considerably faster, but at a substantial cost to its usage quota. ChatGPT took longer and generated a larger codebase, yet left that separate quota untouched. The remaining question was whether the savings would survive the next stages, when both implementations had to undergo code review, verification, and whatever fixes were necessary to make them genuinely reliable.

Who Does a Better Job Reviewing Code?

Writing several thousand lines of code in a relatively short time is certainly impressive. But the amount of code written tells us very little about how well it was implemented. So, after the generation phase, I decided to run another experiment: compare how AI agents with different reasoning levels and execution environments handle the analysis of a large, already implemented codebase.

For each of the two implementations, I ran four independent audits: ChatGPT Medium and High through GitHub, and local Codex Medium and High. All agents received the same review requirements, the five original specification documents, and access to the same revision of the implementation they were reviewing. However, their capabilities differed: local agents could independently run TypeScript, ESLint, Jest, and Docker, while cloud agents worked through the GitHub integration and could examine available CI results. Each agent was tasked with identifying problems, providing evidence, and producing a report without making any fixes.

Once the audits were complete, I consolidated all the reports into a separate findings registry for each implementation, removed duplicates, and split findings that actually described several independent defects. Another local agent running at Medium reasoning then examined each finding and attempted to fix the confirmed problems. It worked from the consolidated list without knowing which reviewer had originally reported each issue. This allowed me to evaluate not just the number of findings, but how well-founded they actually were.

Reviewing the Code Generated by ChatGPT

Let's start with the cloud agent's implementation. The four independent reviews produced a total of 31 initial findings, which were consolidated into 23 distinct defect candidates after normalization. Subsequent verification confirmed 19 problems, of which 16 were fixed and three remained unresolved. Another four cases were classified as Unverifiable: the agents had observed intermittent test failures, but their root causes could not be established.

Metric	Cloud Medium	Cloud High	Local Medium	Local High
Review time	4:13	15:12	5:56	10:41
Codex quota consumed	0%	0%	7%	27%
Defect candidates identified	5	11	6	18
Confirmed findings	5	11	5	15
False Positives	0	0	0	0
Unverifiable	0	0	1	3
Unique confirmed findings	0	3	0	4
Known defects detected	26.3%	57.9%	26.3%	78.9%

The results were quite revealing. Local Codex High found the most confirmed problems: 15 out of 19, although the review consumed 27% of its five-hour quota. ChatGPT High found 11 problems, taking more time but consuming none of the separate Codex quota. Interestingly, three confirmed problems were identified exclusively by Cloud High, while another four were found only by Local High.

None of the findings was ultimately classified as a False Positive, but that doesn't mean every finding was confirmed. Four cases remained unresolved due to insufficient evidence, so I excluded them from the confirmed defect count. It's also important to remember that passing tests don't guarantee those tests are correct: a significant proportion of the findings concerned insufficiently precise assertions rather than failing tests.

Reviewing the Code Generated by Local Codex

Now let's look at the review results for the local implementation. Here, the four reviewers produced 27 initial findings, which were consolidated into 17 distinct candidates after removing duplicates. Subsequent verification confirmed 13 problems, 12 of which were fixed. One finding turned out to be a false positive, while another three were classified as outside the agreed scope of work.

Metric	Cloud Medium	Cloud High	Local Medium	Local High
Review time	5:02	16:04	5:04	5:04
Codex quota consumed	0%	0%	6%	7%
Defect candidates identified	4	9	6	8
Confirmed findings	3	6	5	8
False Positives	1	1	0	0
Out of Scope	0	2	1	0
Unique confirmed findings	1	4	0	2
Known defects detected	23.1%	46.2%	38.5%	61.5%

Once again, Local Codex High found the most confirmed problems: eight out of thirteen. But this time, the difference in resource consumption was much smaller. Both local agents completed their reviews in approximately five minutes, consuming 6% and 7% of their quotas, respectively. Cloud High spent over 16 minutes and identified six confirmed problems, four of which no other agent detected. Meanwhile, Cloud Medium found only three genuine problems out of the four it reported, although one of those was its unique finding.

This time, false positives also appeared. Cloud Medium and Cloud High reported the same issue, which verification later revealed to be the result of an incorrect comparison of Git revisions. Another three findings were excluded from the confirmed defect statistics as Out of Scope. This clearly demonstrates why simply comparing the number of findings in each report would have been misleading.

Results Across Both Experiments

Combining the results of both implementations gives us 40 distinct defect candidates, 32 of which were confirmed during subsequent verification. Of those, 28 were fixed and four remained unresolved. Another four cases could not be sufficiently verified, one turned out to be a False Positive, and three were excluded as Out of Scope. Importantly, these findings concern two different implementations, so even similar defects are counted separately for each one.

Metric	Cloud Medium	Cloud High	Local Medium	Local High
Defect candidates identified	9	20	12	26
True Positives	8	17	10	23
False Positives	1	1	0	0
Out of Scope	0	2	1	0
Unverifiable	0	0	1	3
Unique confirmed findings	1	7	0	6
Precision	88.9%	94.4%	100%	100%
Observed Recall	25.0%	53.1%	31.3%	71.9%

Here, Precision measures the proportion of an agent's adjudicated findings that turned out to be genuine problems, excluding Out of Scope and Unverifiable cases. Observed Recall measures how many of the 32 known confirmed problems each agent managed to identify. This is not an absolute measure of review completeness, since some defects may have gone unnoticed by all four agents. Likewise, a Precision score of 100% doesn't mean the model is infallible: the sample is relatively small, and the final decisions were made by another AI agent, albeit one that also inspected the code and executed tests.

Across both implementations, Local Codex High ranked first in defect detection, identifying 23 of the 32 confirmed problems. ChatGPT High found 17, outperforming Local Codex Medium, which identified ten. At the same time, seven confirmed findings were unique to Cloud High, while six were unique to Local High. This means that even the most resource-intensive configuration could not replace all the others.

On the other hand, these results don't justify claiming that Local High will always review code better or that Cloud Medium will always perform worst. We compared only two specific implementations of the same requirements, and the execution environments had different capabilities for running checks. Still, an interesting trend emerges: increasing the reasoning level in both environments was accompanied by a substantial increase in the number of confirmed problems detected. ChatGPT High through GitHub also demonstrated that a high-quality audit doesn't necessarily require direct access to a local development environment.

How Much Did It Cost to Fix the Bugs?

After four independent audits of each implementation, it was time to fix the identified problems. And this is where I decided to simplify the experiment a little.

Instead of using a single agent's report, I combined the results of all four reviews, removed duplicates, and created one consolidated list of problems for each implementation. This way, the agent responsible for the fixes received the most complete picture possible, regardless of which reviewer had originally identified a particular defect.

For both implementations, I used local Codex with Medium reasoning. In my view, High was no longer quite as necessary at this stage: most of the investigative work had already been done during the audits. What remained was to verify the validity of the findings, implement the fixes, and ensure the tests passed.

Time, Quota, and Scope of Changes

Metric	Local Codex Code	ChatGPT + GitHub Code
Time spent fixing bugs	23m 01s	12m 30s
Codex quota consumed	44%	24%
Initial lines of code added	5,032 lines	6,214 lines
Lines added during fixes	297 lines	346 lines
Lines deleted during fixes	125 lines	5 lines
Total volume of changes	422 lines	351 lines
Changes relative to initial code volume	8.4%	5.6%

The percentage of changes is calculated by dividing the sum of added and deleted lines during remediation by the number of lines added during the initial implementation. This measures Git code churn, not the literal percentage of code that was rewritten.

And this is where I encountered a rather unexpected difference.

Fixing the code generated by local Codex took 23 minutes and consumed 44% of the quota. Meanwhile, the code created through ChatGPT was refined in just 12.5 minutes, consuming only 24%.

In other words, the local implementation required approximately 1.84 times as much time and 1.83 times as much quota, even though its initial codebase was smaller.

Interestingly, the ratios for time and quota consumption are almost identical. This suggests that the unusually high 44% figure is unlikely to be explained by some special cost associated with Medium reasoning on that particular task. A much simpler explanation is that the agent simply worked longer.

But why? One possible factor was flaky tests that occasionally failed and then passed when rerun. This may have forced the agent to spend additional time repeating checks and investigating failures. However, I can't determine exactly how much time was spent on those cases. Therefore, the 44% should be treated as the actual measured result of this particular run, not as a guaranteed cost for anyone attempting to reproduce the experiment.

The difference in the volume of changes was interesting too. In the local implementation, the agent added 297 lines and deleted 125, while in the cloud implementation it added 346 and deleted only five. Comparing the total volume of these changes against the original implementation size gives us 8.4% versus 5.6%.

However, it would be wrong to conclude that one implementation was better simply because it required a smaller percentage of changes. Fixes can involve either rewriting existing code or adding new checks. So these numbers are better understood as a measure of the amount of rework required, rather than a direct measure of the initial implementation's quality.

Final Results: Which Approach Was More Cost-Effective?

Now it's time to put all the numbers together and answer the main question behind this experiment: can you really save Codex quota by moving code generation into ChatGPT and using the local agent primarily for reviews and fixes?

For this comparison, I decided to include three main stages of work: code generation, independent Code Review, and fixing the identified problems.

For each approach, I'm counting the High-level review performed in its corresponding environment. That means Local High for the code generated by local Codex, and Cloud High for the code generated through ChatGPT. In both cases, the fixes were performed by local Codex Medium.

How Much Did the Full Cycle Cost?

Stage	Local Codex: Time	Codex Quota	ChatGPT + GitHub: Time	Codex Quota
Generation — High	46:16	65%	1:21:58	0%
Code Review — High	5:04	7%	15:12	0%
Fixes — Local Medium	23:01	44%	12:30	24%
Total	1:14:21	116%	1:49:40	24%

The quota percentages are summed across separate stages, which may have taken place in different five-hour usage windows. This does not mean that 116% was consumed within a single window. The zero values for ChatGPT mean zero consumption of the separate Codex quota, not an absence of all costs or usage limits.

And this is where things get really interesting.

Local Codex was approximately 35 minutes faster. The complete measured cycle took 1 hour and 14 minutes, compared with almost 1 hour and 50 minutes for ChatGPT. In other words, the cloud-based approach required approximately 47.5% more time.

But when we look at Codex quota consumption, the picture changes dramatically.

The local approach consumed a total of 116% of a five-hour quota, while the cloud-based approach consumed just 24%. This means that ChatGPT reduced Codex quota consumption by approximately 4.8 times, or 79%.

For me, this is the main result of the experiment. I obtained two functionally comparable results while spending almost five times less of the local agent's limited resources on one of them. The price of those savings was approximately 35 additional minutes of waiting.

Of course, this isn't a monetary saving in the literal sense. ChatGPT has its own pricing model and limits, and my calculations specifically concern Codex quota consumption. But if you already use both services and regularly run into Codex's five-hour usage limit, the difference can be quite significant.

But What About the Quality of the Generated Code?

Here, the results were less straightforward.

Both agents implemented the same 476 unique test scenarios. However, independent audits found different numbers of problems in their respective implementations.

Metric	Local Codex	ChatGPT + GitHub
Confirmed problems	13	19
Time spent on fixes	23:01	12:30
Quota consumed by fixes	44%	24%
Changes relative to initial code volume	8.4%	5.6%

Judging by the number of confirmed defects, the initial local Codex implementation appears to be better: reviewers found 13 problems in it, compared with 19 in ChatGPT's implementation. In other words, the cloud implementation had approximately 46% more confirmed findings.

But here's the paradox: fixing ChatGPT's code required almost half as much time and quota, and the relative volume of changes was smaller — 5.6% versus 8.4%.

This is a good example of why the number of defects and the difficulty of fixing them aren't always directly correlated. Some problems can be resolved by adding a few assertions, while others require substantial changes to the testing infrastructure or a different testing strategy altogether.

I wouldn't try to convert these numbers into something like "95% correct code." A Git diff measures the volume of changes, not the percentage of correct implementation. And the number of identified problems doesn't guarantee that reviewers found absolutely every defect.

So, in terms of initial code generation quality, I would favor local Codex because it received fewer confirmed findings. But when it comes to the cost of subsequent remediation, ChatGPT's implementation won this particular experiment.

Which Agent Is Better at Code Review?

Here, the winner was much clearer.

Combining the review results for both implementations gives us 32 confirmed problems. Here's how many of them the two High-reasoning agents managed to identify:

Metric	Local High	Cloud High
Confirmed problems identified	23 out of 32	17 out of 32
Known problems detected	71.9%	53.1%
Unique confirmed findings	6	7
Codex quota consumed across two reviews	34%	0%

Local Codex High proved to be the more effective reviewer, finding almost 72% of known problems compared with ChatGPT High's 53%. A difference of nearly 19 percentage points is quite substantial.

But the cloud agent's performance shouldn't be underestimated either. It found 17 confirmed defects — approximately 74% of the number identified by Local High. What's more, seven of those defects went unnoticed by every other reviewer.

And it achieved all of this without consuming any of the separate Codex quota.

So here's my conclusion: if I needed the most thorough Code Review possible and quota wasn't a concern, I would choose Local High. But if I needed to regularly review large amounts of code without burning through Codex limits, ChatGPT High with GitHub would be a very attractive alternative.

Better yet, I'd use them as independent reviewers. After all, the experiment showed that they don't simply find different numbers of problems — they also catch defects that the other agent misses.

My Final Verdict

After all these experiments, I've come to the conclusion that there's little reason to force a single tool to do absolutely everything.

Local Codex is faster, and its initial implementation received fewer confirmed findings in this experiment. At the High reasoning level, it also demonstrated the strongest Code Review performance.

But ChatGPT with GitHub delivered a functionally comparable result while consuming almost five times less Codex quota. Yes, it took approximately 47.5% more time, but for large tasks that don't require a developer's constant attention, that seems like a perfectly reasonable trade-off.

I'll probably adopt a hybrid approach going forward: ChatGPT for large-scale code generation and independent audits, and local Codex for tasks where speed is critical, direct code execution is necessary, or maximum review thoroughness matters.

The main idea is simple: you don't have to use the most quota-expensive tool at every stage to get a good result. Sometimes, all you need is to distribute the work intelligently between several agents.

Limitations of the experiment. The cost comparison presented here sums the recorded generation stage, one selected High-level review, and the measured remediation run. In reality, I used the combined results of four independent audits to prepare the remediation tasks, not just the single audit included in the cost table. There were also additional verification and remediation iterations after the initial fixes, for which time and quota consumption were not fully recorded. Therefore, the 116% and 24% figures should be interpreted as a comparison of selected, measured stages rather than the complete actual cost of every action performed during the experiment. Furthermore, several problems remained unresolved, and some intermittent test failures never received a definitive explanation.

A Small Clarification About "Free Codex"

I can already anticipate a perfectly fair question: if I promised Codex without consuming its quota, why did I end up using local Codex Medium to fix the bugs?

The thing is, I believe it's entirely possible to bring the code into a working state using ChatGPT with GitHub as well. But since the result still needs to be validated in a real execution environment, I decided to simplify the experiment and have a local agent fix both implementations under identical conditions.

Even with this approach, Codex quota consumption was reduced by approximately 4.8 times across the stages included in the comparison. And I suspect it's also realistic to avoid consuming the local Codex quota entirely: use ChatGPT for generation, analysis, and fixes, while running checks locally and feeding the results back into the chat. Of course, such a workflow would most likely require considerably more time and manual effort.

So "free" here means no consumption of the separate Codex quota, not the absence of a subscription, usage limits, or any other costs whatsoever. I didn't test a fully cloud-based development cycle in this experiment, but the results give me reason to believe that it's a promising approach.

Bonus: Does More Code Mean Better Code?

Toward the end of the experiment, I noticed something interesting: ChatGPT had generated 23.5% more code than local Codex — 6,214 added lines versus 5,032.

That raised another question: was the additional code actually justified, or had the cloud agent simply produced more boilerplate and duplication?

To find out, I asked an independent cloud-based AI reviewer to compare both original implementations, before any remediation. The reviewer examined functional correctness, requirements compliance, test reliability, architecture, decomposition, DRY, maintainability, and implementation complexity.

This time, I wasn't interested in how many individual bugs each implementation contained. I wanted an overall engineering assessment: which codebase would be better to maintain and extend?

Here's the verdict:

Evaluation criterion	ChatGPT + GitHub	Local Codex
Functional correctness	✓	—
Requirements compliance	✓	—
Test reliability	✓	—
Architecture & Separation of Concerns	Tie	Tie
Decomposition	Tie	Tie
DRY & Reusability	Tie	Tie
Maintainability	Tie	Tie
Implementation complexity	—	✓
Code efficiency	—	✓
Score	7.0 / 9	6.0 / 9

Scoring: Each implementation receives 1 point for a category win and 1 point for a tie. Scores are not zero-sum: a tie indicates comparable performance, so neither implementation is penalized. The maximum individual score is 9 points. This is a simplified summary of the reviewer's qualitative judgments, not a standardized software quality benchmark.

Interestingly, the reviewer gave ChatGPT the overall win, although with only moderate confidence.

Its implementation received stronger assessments for correctness, requirements compliance, and reliability. It featured more extensive database assertions, stricter test isolation, and a successful CI run on the exact revision being evaluated.

Local Codex, meanwhile, demonstrated better code efficiency and lower implementation complexity. Its centralized test infrastructure and greater use of parameterized tests allowed it to cover essentially the same requirements with substantially less code.

The reviewer found that approximately 86% of the difference in added lines came from the route tests themselves, rather than infrastructure. Some of ChatGPT's additional code provided meaningful assertions, while some came from repeated setup and less aggressive reuse.

There is one important qualification: the reviewer could verify a successful CI run for ChatGPT's original revision, but not for the exact local Codex revision. That gave ChatGPT stronger evidence of working behavior, without proving that the local implementation was broken. The final verdict was therefore conditional, not an absolute measurement of intrinsic code quality.

My takeaway? The additional 23.5% of code wasn't entirely wasted, but it wasn't entirely justified either. ChatGPT produced the implementation the independent reviewer preferred overall, while local Codex delivered a leaner solution with a similarly strong architectural foundation.

A rather fitting conclusion to this experiment: more code doesn't necessarily mean better code — but less code doesn't automatically mean better engineering either.

P.S. Thanks for Reading All the Way to the End!

If you found this article interesting or useful, I'd really appreciate a ❤️ and a follow!

I regularly run technical experiments, explore new approaches to software development, and share what I learn along the way. There's plenty more to come!

In the meantime, you might also enjoy my other articles:

💥 Easy way to save your AI coding agent a surprising amount of context — Learn how to optimize your AI-assisted development workflow and dramatically reduce token consumption.
100+ API Endpoints and Almost Zero Integration Bugs. Sharing the Framework — Discover my approach to managing complex API contracts and preventing integration issues.

Thanks for reading, and see you in the next experiment! 🚀

Mykola Mizhigurskiy

Aug 10

💥 Easy way to save your AI coding agent a surprising amount of context.

#ai #agents #softwareengineering #productivity

2 min read

Mykola Mizhigurskiy

Aug 18

100+ API Endpoints and Almost Zero Integration Bugs. Sharing the Framework.

#api #testing #software #architecture

13 min read

100+ API Endpoints and Almost Zero Integration Bugs. Sharing the Framework.

Mykola Mizhigurskiy — Tue, 18 Aug 2026 14:19:48 +0000

Like most developers, I spent quite a long time suffering from integration issues that inevitably appear in any reasonably serious project: the backend changes a field, adds a new one, changes its format — and the frontend finds out only after something breaks.

So, some time ago, I decided I had had enough and built my own API contract testing framework for the project I was working on. In practice, the result turned out even better than I expected: after fully implementing it, the number of integration issues between the frontend and backend dropped to almost zero.

In this article, I want to share this framework with you: how I designed it, what problems I wanted to solve, how I organized schemas and contracts, and why it eventually became much more than just a collection of API tests.

The stack in my case is fairly standard for a modern frontend project: JavaScript, TypeScript, React, and Zod. But the idea itself is not tied to any particular library — Zod is simply a very convenient tool for implementing it.

Before building anything, I defined the primary goal of the framework: automatically validate real API responses and guarantee that the backend returns exactly what the frontend expects.

That means catching changes in field types and formats, missing required fields, newly introduced unexpected fields, validation rule violations, and changes to stable response structures.

And that is where the entire framework architecture begins.

QA Engineer vs. Software Developer

Before we begin, I want to address what I believe is a fairly common misconception about API contract testing: that it should be done by developers rather than QA engineers.

This framework can be implemented and successfully used by both a QA engineer and a software developer. Its basic implementation does not require deep knowledge of the application architecture: a QA engineer can describe API contracts, build validation schemas, and use them to automatically validate API responses. In this form, the framework already fully performs its primary task — detecting contract violations between the backend and its consumer.

However, when a software developer integrates this framework directly into the application architecture, the same resources can provide significantly more value. Properly designed schemas effectively become a single source of truth for several parts of the application:

TypeScript types;
contract tests;
runtime validation;
form validation;
rules and constraints for individual fields;
frontend elements that depend on those constraints, such as validation errors or character counters;
API documentation.

So instead of independently describing the same structure in types, tests, forms, UI, and documentation, we define it once and reuse it everywhere it is needed.

That is why, for QA, this framework can be a powerful API contract testing tool, while for a developer it can also become part of the application architecture itself.

Framework Organization

src/api/
  ├── endpoints  # Resources required for API interaction
  |   └── users  # Business-entity folder grouping related endpoints
  |       └── getProductById/  # Named endpoint folder
  |           ├── getProductById.api.ts  # Bare reusable API function
  |           ├── getProductById.hook.ts  # UI-ready hook
  |           └── getProductById.schemas.ts  # Validation schemas and types
  └── schemas  # Business-entity definitions
      ├── common.schemas.ts  # Schemas and types reused across business entities
      └── product.schemas.ts  # Schemas and types for a specific business entity

Pay attention to how the files inside each endpoint are organized. There are two reasons for this.

The first is that I build this framework as an AI-native architecture and try to make it as predictable and convenient for AI agents as possible. The more clearly responsibilities are separated and the more stable the naming patterns are, the easier it is for an agent to understand the structure, locate the correct file, and extend the existing implementation consistently.

The second reason is separation of concerns. I intentionally split the implementation into several suffix files, each with one clearly defined responsibility:

*.api.ts — a minimal function responsible for interacting with the API. It does not depend on React and contains only the logic required to make a request and receive a response. Because of this, it can be used independently in React hooks, contract tests, server-side code, or any other consumer.
*.hook.ts — the React layer on top of the API function. This is where loading and error states, caching, state management, transformations, and other UI-specific logic can live.
*.schemas.ts — the contract of a specific endpoint: Zod schemas for runtime validation and the TypeScript types associated with them.

This separation is particularly important for contract tests. A test does not need to initialize React or pull UI-specific logic along with it just to validate an API. It can call a function from *.api.ts directly and validate the response using a schema from *.schemas.ts.

Naming Conventions

I personally consider semantics one of the most underrated parts of software development, and I take it very seriously. A good name should immediately tell you what entity you are looking at, what role it performs, and where it is used, even before you open the file.

This becomes especially important in AI-native repositories: consistent naming allows AI to infer relationships between entities more accurately, anticipate their purpose, and find the right context. That is why naming conventions in this framework are not cosmetic — they are part of the architecture.

Endpoints

For endpoints, I use a simple rule:

<operation><entity><qualifier?>

The operation comes first, followed by the business entity, and then, when necessary, an additional qualifier describing the request.

For example:

getProductById
createProduct
modifyProduct
deleteProduct
getUserProduct

The same naming is used for the endpoint folder and its related files:

getProductById/
  ├── getProductById.api.ts
  ├── getProductById.hook.ts
  └── getProductById.schemas.ts

This way, all parts of a particular API request can easily be found using the same name.

Schemas & Types

For schemas and types, I use a separate set of standard suffixes describing the role of the data in a particular request:

QueryParams
RequestPayload
ResponsePayload

If the entity is a Zod schema, I simply add Schema to the name.

For example:

GetProductById_QueryParams
CreateProduct_RequestPayload
GetProductById_ResponsePayload
GetProductById_ResponsePayloadSchema

As a result, the name itself describes the structure:

<endpoint>_<data role><Schema?>

So when you see GetProductById_ResponsePayloadSchema, you immediately know three things:

which endpoint it belongs to;
which part of the API contract it describes;
that you are looking at a runtime schema rather than a TypeScript type.

Getting Started

Common Schemas

Let us start with the simplest file — common.schemas.ts.

Its purpose is to store small but reusable schemas that appear across many business entities. These are not standalone application models, but rather building blocks: IDs, timestamps, tracker fields, UUIDs, emails, and other standardized primitives.

For example, a minimal common.schemas.ts might look like this:

import { z } from 'zod'

// Unified ID type used across the application
export const Id_Schema = z.string().uuid()

// Unified set of fields used to track entity creation and modification
export const Tracker_Schema = z.object({
  createdOn: z.string().datetime(),
  modifiedOn: z.string().datetime(),
})

These schemas are not duplicated in every business entity. Instead, they are reused through extend() or directly as individual fields.

This allows us to change a rule in one place and automatically propagate it throughout the application.

Business Entity

Now let us move to product.schemas.ts — the file describing the Product business entity. For schemas like this, I use a simple naming rule: the entity name plus the Schema suffix — in our case, Product_Schema.

There is one fundamentally important point here: we describe the entity as the frontend sees it, not as it exists internally on the backend. The same entity can have different representations on the two sides of an API.

For example, the backend model of a product might contain a stock property with the exact number of units remaining in the warehouse. The store owners probably do not want to expose that information publicly, so instead of stock: 12345, the frontend might receive only inStock: true.

That frontend representation is what our contract should describe.

import { z } from 'zod'
import { Id_Schema, Tracker_Schema } from './common.schemas'

// Product business-entity schema for runtime validation.
// Extends the shared tracker fields.
export const Product_Schema = Tracker_Schema.extend({
  id: Id_Schema,
  name: z.string(),
  description: z.string().min(5).max(500),
  inStock: z.boolean(),
})

// Product type automatically inferred from the runtime validation schema.
export type Product = z.infer<typeof Product_Schema>

// Generic Response Schema for endpoints that return the same object shape.
export const Product_GenericResponseSchema = Product_Schema

An important detail here is that we do not define the Product type separately. It is automatically inferred from the same Zod schema through z.infer.

As a result, Product_Schema becomes both a runtime representation of the contract and the source of compile-time typing. If we change the structure or validation rules of Product_Schema, the TypeScript Product type changes with it.

Runtime validation and typing remain synchronized by design.

Also note Product_GenericResponseSchema. In real-world APIs, several endpoints often return the same representation of an entity.

Instead of describing that shape again in every endpoint-specific file, I suggest defining it at the higher business-entity level and reusing it from there.

One more important clarification about Product_Schema: a business-level schema should contain every field that may appear for this entity across any endpoint.

This is not an endpoint-specific contract. It is an entity-specific contract — effectively a single source of truth for all fields of the entity and their validation rules.

Individual endpoints then build their own contracts on top of this base schema using pick, omit, extend, or reusable subschemas.

API Contract

Once we have a canonical business entity, it is time to describe the contract of a specific endpoint. For this example, let us use createProduct.schemas.ts.

Suppose the get, create, and update endpoints all return Product in the same representation. In that case, there is no reason to redefine the schema every time.

We simply import the prepared Product_GenericResponseSchema from product.schemas.ts and use it as the basis for CreateProduct_ResponsePayloadSchema.

import { z } from 'zod'
import {
  Product_GenericResponseSchema,
  type Product,
} from '@/api/schemas/'

// Runtime validation schema for the Create Product response payload.
export const CreateProduct_ResponsePayloadSchema =
  Product_GenericResponseSchema

export type CreateProduct_ResponsePayload = z.infer<
  typeof CreateProduct_ResponsePayloadSchema
>

// Request payload type for Create Product.
// A Zod schema is intentionally not created because it is not used.
export type CreateProduct_RequestPayload = Pick<
  Product,
  'name' | 'description'
>

There is another important principle here: do not create a Zod schema just because you can.

If some data never goes through runtime validation and the schema is not used anywhere, it only creates additional code and another entity that must be maintained.

That is why a Zod schema makes sense for ResponsePayload, which we are going to validate in contract tests. But if compile-time typing is enough for RequestPayload, we can simply derive the required type from the existing Product type using Pick.

This way, even endpoint-specific contracts continue to derive from the same source of truth instead of creating parallel descriptions of the same structure.

Contract Tests

Once the business entities and endpoint contracts have been described, the tests themselves become almost boring.

All you need to do is call the API, receive the response, and validate it against the appropriate Zod schema.

import { getProductById_API } from '@/api/endpoints'
import { GetProductById_ResponsePayloadSchema } from '@/api/schemas'

describe('API contracts', () => {
  describe('Products', () => {
    it('GET product by ID', async () => {
      const data = await getProductById_API(12345)

      expect(() => {
        GetProductById_ResponsePayloadSchema.parse(data)
      }).not.toThrow()
    })
  })
})

All the complexity is already contained in properly designed schemas, so adding another check usually takes only a few lines of code.

Important Clarification: These Are Not API Unit Tests

It is important to understand the purpose of contract testing correctly.

We are not trying to test the API itself or verify all of its business logic. This is not the place for edge cases, attempts to “break” an endpoint, or checks of how the backend behaves when given invalid data.

Our task is much simpler: make a valid request, receive a successful response, and verify that it matches the contract our application relies on.

So keep contract tests as simple as possible:

call the API;
make sure the request succeeds;
validate the response against the appropriate schema.

Do not turn contract tests into a duplicate of the backend unit test suite — you do have one, right?

There is only one thing we care about here:

Does the API return what its consumer expects?

Pro Tips

#1. Array Validation

Keep in mind that Zod successfully validates an empty array even when a specific schema is defined for its elements:

z.array(Product_Schema)

For [], validation passes successfully because there is simply no element to which Product_Schema can be applied.

So if your test is supposed to validate not only the presence of an array but also the shape of its elements, make sure your test data contains at least one item.

For example:

expect(data.length).toBeGreaterThanOrEqual(1)

Otherwise, you risk getting a false positive: the test is green, but the structure of the objects inside the array has never actually been validated.

#2. Catching New Fields

In contract tests, it is important to detect not only missing or invalid fields, but also new unexpected fields appearing in the response.

That is why schemas should be validated in strict mode. Otherwise, the backend may add a new field and your test will never notice it.

There are two ways to solve this.

The first is to explicitly enable strict validation in every Zod object schema:

const Product_Schema = z
  .object({
    id: Id_Schema,
    name: z.string(),
  })
  .strict()

The second option, which I personally prefer, is to create a dedicated utility for contract tests.

It recursively traverses the entire provided Zod schema and applies strict behavior to all object schemas, including objects nested inside arrays, other objects, unions, and so on.

This keeps the schemas themselves universal while centralizing strict behavior exactly where it is needed — in contract tests.

#3. Show the Received Value

Another small detail that has a surprisingly large impact on debugging: the validation error should show the actual value that failed validation.

A message such as “the field does not match the expected format” is often not enough, especially when tests operate on large responses or deeply nested structures.

I recommend configuring error reporting so that, along with the field path and the reason for the failure, it also prints the value that triggered the error.

Instead of something abstract like:

Invalid format at products[3].id

it is much more useful to get something like:

Invalid format at products[3].id
Received: "abc-123"

It may seem like a small detail, but in practice it significantly reduces the time required to investigate failing tests.

#4. Stable Data

Sometimes an API contains endpoints that return stable data — values that are not expected to change regularly.

A typical example is a list of statuses for a business entity.

In this case, it can be useful not only to validate the general response shape but also to explicitly lock down the expected set of values using literals.

For example:

const ProductStatus_Schema = z.enum([
  'Pending',
  'In Progress',
  'Done',
])

At first glance, this may look like unnecessary overhead. In practice, however, it can be extremely useful — especially when frontend behavior depends directly on specific statuses.

For example, different statuses may display different UI, enable different actions, or trigger different business logic.

In such a case, introducing a new status or renaming an existing one is no longer just a data change. It is a contract change that can potentially affect the frontend.

If you cannot be completely sure that the backend team will always notify you about such changes in time, contract tests can serve as an additional safety net.

Bonuses

At the beginning of the article, I mentioned that a software developer can get much more from this framework than contract tests alone.

Since the Zod schemas have already become our single source of truth, we can reuse them directly in application code.

Here are a few practical examples.

#1. API Typing

Since TypeScript types are automatically inferred from our Zod schemas, the same types can be used directly in the API layer:

export const getProductByIdApi = async (
  id: string,
): Promise<GetProductById_ResponsePayload> => {
  return api.get(`/products/${id}`)
}

This means API typing and API validation are based on the same source of truth.

Change the schema, and the type used by the application code changes with it.

#2. Runtime Validation of Local Storage Data

Suppose a user is creating a product and we store a draft of the form in localStorage.

After the page reloads, we want to restore it — but before putting the data back into the form, we want to make sure it still conforms to our contract.

const rawDraft = localStorage.getItem('productDraft')

if (rawDraft) {
  try {
    const parsedDraft = JSON.parse(rawDraft)
    const result = ProductDraft_Schema.safeParse(parsedDraft)

    if (result.success) {
      form.reset(result.data)
    }
  } catch {
    // Invalid JSON — ignore the stored draft
  }
}

This gives the data from localStorage two validation layers: first, we ensure it is valid JSON; then Zod checks its structure and values before it is allowed back into the form.

#3. Form Validation

The same Zod schemas can be integrated directly with popular form libraries.

For example, React Hook Form provides a Zod resolver:

const form = useForm({
  resolver: zodResolver(Product_Schema),
})

As a result, rules such as:

description: z.string().min(5).max(500)

do not need to be described again in the form.

The schema that defines the API contract and TypeScript type also defines the form validation rules.

#4. Using Schema Constraints in the UI

A Zod schema contains not only information about the field type, but also its rules:

description: z.string().min(5).max(500)

If the UI displays a character counter:

127 / 500

there is no reason to hardcode 500 separately inside the React component.

We can get it directly from the schema:

const maxLength = Product_Schema.shape.description.maxLength

and use that value directly in the UI:

<CharacterCounter
  current={description.length}
  max={maxLength}
/>

This means 500 exists in exactly one place — Product_Schema.

Form validation, runtime validation, and the UI all use the same rule.

#5. Third-Party Service Integration

But why are we talking only about the classic frontend ↔ backend integration?

Let us look at this framework from another angle.

Suppose your application integrates with a third-party API. Your code directly depends on the responses returned by an external service, but you have neither a direct communication channel with its team nor any guarantee that you will receive timely notifications about API changes.

In this scenario, contract testing can become even more valuable.

Your tests effectively turn into an early-warning system: if the external service changes the response structure, removes a field, adds a new one, or changes its format, you can find out automatically — before that change turns into a problem for your users.

In other words, the same framework that helps reduce communication and integration issues in an internal frontend ↔ backend setup can, in the case of a third-party API, partially compensate for the complete absence of that communication.

#6. AI Compatibility

I put a lot of effort into standardization, structural consistency, and semantics while designing this framework.

And this is where we get to reap the benefits.

The more predictable the repository architecture is, the easier it is for AI to understand its patterns and extend them correctly.

If the framework is implemented consistently, AI no longer has to decide every time where to create files, how to name schemas, how to group endpoints, or how to structure contract tests.

The repository already contains a pattern that the agent only needs to recognize and reproduce.

In practice, this means that when a new API module appears, I can simply give the AI a link to the Swagger specification.

From there, it can parse the API specification, identify business entities and endpoints, and generate the module according to the existing structure: schemas, types, abstractions, and contract tests.

And yes, this has been tested in practice.

Implementing a new API module with around 10 endpoints, including testing, takes about 16 minutes.

So standardization here improves more than developer experience.

It makes the repository genuinely AI-native: instead of repeatedly explaining to an agent how another module should be implemented, we give it an architecture where the correct implementation pattern is already encoded in the structure and semantics of the code.

Conclusion

A few numbers.

While working with a large API containing more than 100 endpoints, this framework allowed me to reduce structural integration bugs to almost zero, cut the reaction time to API changes to within a single day, and catch around 100 integration issues in just three months — without requiring additional communication with the backend team.

I consider that a fairly strong result.

At this point, the framework solves one of the classic problems of software development quite well: the constant need to stay synchronized with API changes and detect situations where the actual contract starts diverging from what the application expects.

Overall, I am happy with the result.

The framework solves all the core problems I originally designed it for, requires relatively little additional code, and provides significantly more value than API validation alone.

I would be very interested to hear your thoughts.

Do you use anything similar in your projects?

Where do you think this approach works well — and where do you think it starts to break down?

I Tested Two AI Context Architectures: Context Discovery vs Prompt Engine. Here’s What I Learned.

Mykola Mizhigurskiy — Tue, 11 Aug 2026 18:51:23 +0000

For the last few months, I have been building my own AI framework for software development. It contains instructions for different engineering activities: implementing features, refactoring, writing tests, debugging, working with APIs, reviewing code, and many smaller tasks. As the framework grew beyond 100,000 characters of instructions, one problem became increasingly important: how do I give an AI agent exactly the instructions it needs without loading the entire framework into its context?

While working on this problem, I ended up testing two different architectures: Agent-Driven Context Discovery and a Prompt Engine with Context Compilation. Both allow you to keep instructions modular and load only what is relevant to the current task. The major difference is who is responsible for assembling that context.

Agent-Driven Context Discovery leaves this responsibility to the agent. You give it an entry point, usually a base instruction file, and that file references other instructions. The agent reads those files, follows their references, and gradually discovers the context required for the task.

Prompt Engine with Context Compilation moves most of this work into deterministic code. The agent still selects the type of activity it is about to perform, but once that decision is made, the Prompt Engine deterministically resolves the required instructions and compiles them into a ready-to-use context. The agent no longer has to manually discover and retrieve every instruction.

Shared Foundation: Intent Router / Playbook

Both architectures still need a mechanism that connects a task with the appropriate instructions. I call mine a playbook.

The playbook is essentially an intent router. It describes the types of activities the framework supports and maps those activities to the instruction sets they require. Implementing functionality, refactoring code, writing tests, debugging, and reviewing code are examples of activities that can have different instruction dependencies.

With Context Discovery, this routing can live inside the instruction graph itself. The base instruction tells the agent where to go, and the agent follows the appropriate path.

With my Prompt Engine, I use an MCP tool called getInstructions. It accepts typed activities and uses them to programmatically determine which instructions should be compiled. The agent therefore makes one high-level decision — what am I about to do? — while deterministic code handles the low-level question — which exact instructions does that activity require?

Approach 1: Agent-Driven Context Discovery

This was the first architecture I implemented. I split my framework into many small instruction files and created references between them. The agent received a base instruction as its entry point and was expected to follow those references whenever additional instructions were required.

Conceptually, it looked something like this:

Task → Base Instructions → Agent → Referenced Instructions → More Instructions → Work

I liked this approach initially because it was extremely easy to build. The instruction system remained modular, individual files could be reused, and adding a new rule usually meant adding or updating a reference. There was no separate infrastructure responsible for assembling prompts because the agent itself effectively acted as the resolver.

After using this architecture for some time, however, I started noticing several problems.

Problem 1: Reasoning Overhead

Every time the agent encounters a reference, it has to understand why that instruction exists, decide whether it should follow it, retrieve the file, process it, and potentially repeat the process. These are small decisions, but they are still decisions made by the model instead of deterministic operations performed by software.

I increasingly disliked spending the agent's reasoning capacity on this kind of plumbing. I want the model thinking about architecture, requirements, edge cases, and implementation decisions. Resolving an instruction dependency does not require intelligence if I can already express that dependency in code.

Problem 2: Reliability

I repeatedly saw agents skip files even when another instruction explicitly told them to read those files. It did not happen every time, but it happened often enough that I could not consider instruction delivery deterministic.

This creates an unpleasant failure mode. Your agent may produce perfectly reasonable code while silently missing one of the rules it was supposed to follow. The problem is not that the model misunderstood an instruction — it never loaded that instruction in the first place.

Problem 3: Tool Usage

Reading each instruction file requires a tool call. With more than 100,000 characters distributed across many files, the agent could make a surprising number of calls just to prepare itself for the actual task.

I repeatedly hit Copilot’s maximum tool-call limit while the agent was navigating instruction files. When that happened, Copilot stopped before the actual task was completed. That was a strong signal that I needed a different architecture.

Problem 4: Inefficient Context Usage

Another problem is that routing instructions remain in the agent's context long after they have served their purpose. Instructions such as "read this file," "for this activity, also read that file," or "if this condition applies, load these additional instructions" are useful during context discovery, but provide little value once the agent starts working on the actual task. From that point on, they simply consume space in the context window.

This problem also has the potential to grow over time. If the agent occasionally fails to load the right files for certain activities, the natural response is to make the routing instructions more explicit: add explanations, introduce more specific terminology, describe additional conditions, or repeat important references. That may improve routing reliability, but it also increases the amount of temporary information that remains in the context throughout the task.

In other words, the more effort you put into making agent-driven context discovery reliable, the more context you may end up spending on instructions that become useless immediately after discovery is complete.

Approach 2: Prompt Engine with Context Compilation

My second implementation moved instruction resolution out of the agent and into code.

The framework still contains dozens of small, reusable instruction files. The difference is that the agent no longer needs to navigate them one by one. A script resolves the dependencies, collects the required files, and returns the complete instruction set in a single operation.

The architecture now looks closer to this:

Task → Activity → Prompt Engine → Compiled Context → Agent → Work

The change may sound small, but the practical improvement was substantial. Once the agent selects an activity, the rest of the process is deterministic. If activity X requires instructions A, B, C, and D, the Prompt Engine delivers all four. There is no intermediate step where the agent can forget to open C or decide that D is unnecessary.

Context delivery also became much faster. Instead of making a sequence of tool calls, reading files one by one, and reasoning about what to retrieve next, the agent makes a single request. The Prompt Engine resolves the instruction graph programmatically and returns the compiled context in a single operation. This removes both the latency of repeated tool calls and the reasoning overhead involved in navigating the instruction structure.

Another important improvement was eliminating a practical limitation I repeatedly hit with Copilot: the maximum number of tool calls available to the agent. With Context Discovery, the agent regularly exhausted this limit while navigating and reading instruction files, which caused Copilot to stop before the task was completed. After moving instruction resolution into the Prompt Engine, instruction delivery requires only one tool call, so this failure mode effectively disappeared from my workflow.

In practice, I got three immediate benefits: the required instructions started reaching the context reliably, context assembly became much faster, and instruction discovery stopped consuming the agent's limited tool calls. Combined, these changes produced a very noticeable improvement in the overall speed and reliability of my workflow.

Problem 1: Your Prompt Engine Becomes a Mini-Product

The first downside is additional infrastructure. In my case, the Prompt Engine requires a custom MCP server that exposes the tools the agent uses to request its instructions. Once you introduce this layer, you effectively have another small software product living inside your repository.

Like any other software, it needs maintenance. You need to keep the implementation clean, handle edge cases, and ideally cover its critical behavior with automated tests. As the instruction system evolves, the Prompt Engine has to evolve with it.

This is simply the price of moving from a lightweight instruction structure to a more sophisticated architecture. You gain more deterministic and reliable context delivery, but you also introduce another component that you are responsible for maintaining.

Problem 2: You Can Accidentally Create Two Sources of Truth

Another challenge is instruction representation. The format that is convenient for programmatic processing is not necessarily the format that is convenient for a human. For example, the Prompt Engine may work better with structured JSON, while I would much rather read, write, review, and maintain the same instructions as Markdown.

The obvious solution is to keep both representations, but that immediately creates another problem: they can drift apart. If JSON says one thing and Markdown says another, you effectively have two sources of truth and can no longer be certain which representation describes the actual behavior of the system.

A better architecture needs one canonical source and a deterministic way to generate the other representation from it. Ideally, the two formats should also remain structurally similar enough that you can debug the agent-facing JSON while reading the human-friendly Markdown. Otherwise, the convenience of having two representations can quickly turn into another maintenance problem.

Problem 3: Compiled Instructions Can Pollute the Context

A Prompt Engine can also make context usage worse if the routing rules are not strict enough. Imagine that the current session already contains the instructions for API engineering, but the agent does not clearly understand that those instructions should not be requested again. It may call the instruction tool a second time and inject another full copy of the same compiled context into the conversation.

If your instruction sets are large, this can become surprisingly expensive. Instead of optimizing context usage, you may end up filling the context window with repeated copies of the same material. The more often the agent re-requests instructions, the faster this duplication compounds.

My recommendation is to add a hard guardrail at the infrastructure level.

Track which instruction sets have already been delivered during the current session, and if the agent requests the same one again, do not return the full instruction set.

Return a short message saying that the instruction is already present in the current context and should be reused.

I also make this failure mode explicit. If the agent believes the required instruction is actually missing despite the guardrail, it should treat that as a critical inconsistency, stop, and report the problem to the user rather than silently requesting or reconstructing the same context again.

This is one of those cases where a small deterministic safeguard can prevent a large amount of unnecessary context duplication.

Note: There Is Still One Non-Deterministic Step

This architecture does not make everything deterministic. The agent still needs to select the correct activity when it calls getInstructions, and that selection is an LLM decision.

For this reason, I think the tool interface should be strongly typed and the playbook should be designed carefully. Activity names and descriptions need to be clear enough that the model can reliably map real tasks to them. Otherwise, you simply move the failure point from instruction discovery to activity classification.

I would also recommend watching this routing closely when introducing such a system. Check which activities the agent selects for real tasks and adjust the playbook when you find ambiguous cases. In my framework, my subjective estimate is that incorrect activity selection happens in less than 1% of cases, possibly even below 0.5%, so this has not been a significant practical problem for me.

What I Learned

After working with both architectures, Prompt Engine with Context Compilation wins for me by a large margin.

The difference is not really about prompts or files. It is about deciding which responsibilities belong to an LLM and which belong to traditional software.

An LLM is excellent at understanding intent, interpreting ambiguous requirements, reasoning about code, and making decisions where the answer cannot be reduced to a simple algorithm. A script is excellent at following predefined dependencies, assembling files, validating inputs, and producing the same result every time.

Context Discovery asks the LLM to do some of both. A Prompt Engine lets me draw the boundary more deliberately.

The Core Takeaway: Whenever something can be reliably handled by deterministic software, move it out of the agent's reasoning loop.

That experiment eventually contributed to a broader principle I now use in my Semi-Automatic AI-Native Flow with a Human in the Loop.

For me, this is becoming one of the more interesting questions in AI-native development. We spend a lot of time asking how much more responsibility we can give to AI agents.

Maybe we should spend just as much time asking how much responsibility we can take away from them.

What edge cases did I miss here? How are you managing large instruction sets or context windows in your own AI frameworks? Let me know in the comments!

Microsoft GH-300: GitHub Copilot | Tips & Tricks

Mykola Mizhigurskiy — Mon, 10 Aug 2026 19:10:40 +0000

Today I passed the GitHub Copilot GH-300 exam and decided to write down my impressions right away while they are still fresh. First of all, this article is not so much a “how to pass” guide as a place where people who are currently preparing for the exam can ask questions to someone who literally took it today. If anything from my experience helps you prepare better, or at least understand what to expect, then this article has already served its purpose.

The gap between preparation materials and the actual exam questions

The first thing that genuinely surprised me was how different the information in the recommended preparation materials can be from what you actually see on the exam. A few times, my completely honest reaction was: “Where did this even come from?” Subjectively, I managed to answer some of those questions not because I had memorized the materials well, but because by the time I took the exam I had already been using Copilot quite intensively in real-world development. Sometimes practical experience helped, sometimes it was a general understanding of engineering processes, and sometimes it was simply common sense.

So I would not recommend treating the official materials as a complete map of the exam. They are useful and they provide the basic structure of the topics, but they are not enough to make you confident about every question. If you already use Copilot at work, that is a serious advantage, because part of the exam tests not only whether you know the wording from the documentation, but also whether you understand how the product behaves in common scenarios.

Copilot = Microsoft

Another thing worth keeping in mind: Copilot is GitHub, GitHub is Microsoft, and Microsoft also means Windows and PowerShell. If, like many developers, you have spent years working on macOS and barely touching the Windows ecosystem, some questions may come as an unpleasant surprise. For example, you may need at least a basic understanding of how something is installed or configured through PowerShell. I barely looked at things like that during my preparation because I relied too heavily on the recommended resources and hardly read any additional documentation. If I were preparing again, I would definitely close that gap.

Do questions from the preparation materials appear on the exam?

A separate story is my attempt to “outsmart the system.” During preparation, I collected all the questions I came across in the learning modules and the practice exam, put them into Anki, and simply memorized them. I thought it would be a decent safety net: even if the questions were not repeated word for word, at least I would have the relevant facts firmly memorized. In practice, my subjective impression is that I did not see a single question repeated exactly on the real exam, and only one or two vaguely resembled something I had learned this way. So I would not recommend memorizing the correct answers; it is much more useful to use those questions as a way to check whether you actually understand the topic.

Question formats

Another mistake in my expectations about the GH-300 was assuming that the exam would mostly follow the classic format: one question, four answers, one correct choice. In reality, there are more formats. Some questions require you to choose two correct answers out of four or three out of five, some ask you to match items between two columns, and some require you to arrange several steps in the correct order. Personally, I found the last type to be one of the trickiest because it is no longer enough to “roughly remember the correct answer” — you need to understand the process itself.

Statistically, you can feel the difference too. In a standard multiple-choice question with four options, you at least have a 25% chance of guessing correctly if you know absolutely nothing. But when you need to choose several items from a set and then put them in the correct order, the number of possible combinations grows very quickly. That is why I would pay particular attention to how GitHub Copilot handles basic workflows: authentication, starting a new session, resuming a previous one, working with context, and common interaction scenarios. It is important not only to know that a feature exists, but to understand what happens step by step.

A final gift :)

And finally, here is a small semi-joking life hack that I came up with for myself while preparing. Use this entirely at your own risk.

If you are looking at a question where you have absolutely no idea which answer is correct and you are going to guess anyway, take a closer look at the longest option. In the practice materials, I repeatedly noticed a pattern: the correct answer is often written in a complete, careful way with all the necessary qualifications, while the incorrect options sometimes look as if, after writing the correct one, the author got a little tired of coming up with alternatives.

If you don’t know which answer to choose, choose the longest one.

I do not know how consistently this works on the real exam, and I definitely would not recommend using it as a substitute for knowledge. But if you are already in a situation where you have to guess anyway, at least you will have some kind of heuristic. I saw this pattern often enough in the practice questions to notice it.

If I had to start preparing again today, I would spend less time memorizing ready-made answers and more time on three things:

actually working with Copilot
reading additional documentation
understanding the product workflow.

The learning path and the demo exam are useful, but I would treat them as a map of the topics rather than a simulation of the actual exam.

So,

if you are preparing for the GH-300 right now, leave your questions in the comments.

While my impressions are still as fresh as possible, I will try to answer everything I can.

Chasing Tokens: I Renamed My Entire Repository — Here’s What Happened

Mykola Mizhigurskiy — Mon, 10 Aug 2026 15:11:14 +0000

After optimizing my CI logs for AI agents, I found myself looking for tokens everywhere.

Then I had another idea.

Many of my files and directories used kebab-case.

My assumption was simple: if - increases token count, then switching everything to camelCase might reduce the amount of context an AI agent has to process.

So instead of guessing, I measured it.

I wrote a script to tokenize my entire repository before and after renaming every kebab-case path to camelCase.

Repository before: 376,553 tokens
Repository after: 376,204 tokens

Total reduction: 349 tokens.

Honestly? That was much smaller than I expected.

At first glance, this looks like a classic micro-optimization. But I don't think the story ends there.

Why File Paths Matter More Than You Think

Unlike source code, file and directory names have a very different lifecycle during an AI coding session.

An agent doesn't read them just once. They appear repeatedly in:

imports and exports
directory listings
search results
git diffs
stack traces
test output
formatter and linter logs
tool responses from the IDE

In other words, file paths are part of the agent's working context over and over again.

So while the repository itself became only 349 tokens smaller, the cumulative savings across a long coding session could be noticeably larger because those same paths are processed repeatedly.

I haven't measured that part yet, so I won't claim any specific numbers.

Conclusion

This experiment reminded me of something more important:

Measure first. Optimize second.

Some optimizations that sound brilliant turn out to be negligible. Others—like reducing unnecessary tool output—have a much larger impact than you'd expect.

AI-assisted development is creating a whole new class of performance optimizations. The only way to know which ones matter is to measure them.

What micro-optimizations have you tested for your AI coding agents? Did they actually move the needle, or did the data surprise you? Let me know in the comments!

💥 Easy way to save your AI coding agent a surprising amount of context.

Mykola Mizhigurskiy — Mon, 10 Aug 2026 15:07:28 +0000

One thing clicked for me only after working with AI agents for a while.

Like many teams, I have a set of mandatory checks that must pass before code is committed:

Prettier
ESLint
Type checking
Unit tests

I also have a rule for my coding agent: whenever it makes executable code changes, it automatically runs all of these checks before moving on.

Sounds obvious, right?

The problem was that all of these tools were configured for humans, not for AI.
For example:

Prettier printed every file it checked - even when nothing changed.
The type checker produced verbose success output.
The test runner listed every executed test file.
ESLint printed a lot of informational output even when everything passed.

Then it hit me:

Every line of that output becomes part of the agent's working context.
Those hundreds of log lines aren't free. They consume context, increase token usage, and make it harder for the model to focus on information that actually matters.

Fixing the Noise

So I changed the defaults:

Prettier runs silently unless there's a formatting issue.
ESLint only reports warnings and errors.
Tests use a dot reporter and print detailed output only for failures.
Successful checks produce only a short summary.

The result?

Exactly the same safety guarantees, but with dramatically less noise.

For humans, verbose logs are mostly harmless. For AI agents, they're context pollution.

And context isn't free. Every unnecessary token is one that could have been spent reasoning about your code instead.

Sometimes the best optimization isn't a faster model—it's simply giving the model less irrelevant information to read.

Appendix: For those who prefer numbers over opinions

I rolled the repository back and measured the raw output with a tokenizer:

Prettier output: ~15,000 tokens
Test runner output: ~35,000 tokens

After enabling silent output and compact reporting:

Prettier: ~500 tokens
Test runner: ~1,500 tokens

That is roughly 48,000 tokens removed from a single validation cycle—without removing a single check or weakening the pipeline.

Same guarantees. Far less context pollution.

What configuration tweaks have you made to keep your AI coding agents focused? Let me know in the comments!

AI Melting Your Brain? Here’s the Fix.

Mykola Mizhigurskiy — Mon, 10 Aug 2026 15:03:18 +0000

A few weeks ago, I came across a post from a developer who had spent six months working with five or six Claude Code terminals open at once. His description of the experience was funny, but also uncomfortably familiar: most of the time, he was simply waiting for agents to finish and pressing Enter.

His conclusion was blunt: “Claude Code is melting my brain.”

I’ve been seeing variations of this concern more and more often. Developers who use AI heavily start wondering whether they are becoming less sharp because the machine is doing too much of the actual coding. At the same time, another term has started appearing in discussions around AI-assisted development: comprehension debt.

I think that term captures the problem remarkably well.

Technical debt appears when a codebase accumulates decisions that will have to be dealt with later. Comprehension debt appears when the codebase evolves faster than the people responsible for it can maintain a mental model of what is happening inside it. The software keeps moving forward, but your understanding of it falls further and further behind.

Eventually, you can end up in a strange position: you are responsible for a system that you technically own, but no longer fully understand.

These look like two different problems:

AI making developers “dumber”
AI creating comprehension debt

... but I suspect they are mostly the same problem viewed from two angles. And, more importantly, I think they have the same solution.

The problem is not AI. The problem is you going full auto.

There is a peculiar productivity competition happening in software engineering right now. Every week I see another impressive number: hundreds of thousands of lines generated, thousands of commits, dozens of agents running in parallel, or thousands of pull requests produced by AI-assisted development pipelines.

Uber, for example, has publicly discussed an internal AI coding system producing roughly 1,800 pull requests per week (1 PR each 80 seconds).

That is certainly an impressive throughput number. But whenever I see metrics like this, I find myself asking a slightly uncomfortable question:

What exactly are we optimizing?

A pull request is a unit of software delivery. It is not a unit of product value.

A software product is, at some level, encoded business logic. Its purpose is to solve problems: make something easier for a user, automate a process, reduce costs, create revenue, satisfy a regulatory requirement, improve reliability, or enable some new capability.

So the interesting question is not how quickly an organization can manufacture changes to a repository. The interesting question is how quickly it can discover, validate and deliver meaningful changes to the product.

Imagine someone tells you that their AI workflow can generate a thousand PRs in a relatively short period of time. Fine. Now ask them a different question: can you name the next important feature your product needs—right now?

And I don’t mean something sitting in the backlog already. I mean a genuinely new requirement. Something that has been validated rather than hallucinated by AI. Something supported by customer feedback, user behavior, business analysis, stakeholder needs, market research or other evidence that suggests building it will actually create value.

That is much harder. And definitely not possible at the rate of 1 feature per 80 seconds.

The reason is simple: once implementation becomes extremely fast, implementation stops being the primary bottleneck. Product discovery does not suddenly become a hundred times faster because your coding agent did. Neither do requirements analysis, UX research, architectural reasoning, stakeholder alignment or understanding what customers actually need.

At some point, enough is enough.

There is a finite amount of meaningful change that a product can discover, validate, absorb and maintain. Once AI gives us effectively abundant implementation capacity, maximizing the amount of code produced becomes a strange goal in itself.

And this is where I think full-auto development becomes dangerous. Not necessarily because AI writes bad code, but because execution can begin moving faster than understanding.

That is where comprehension debt starts accumulating.

My approach: Semi-Automatic AI-Native Development

The workflow I use has a deliberately cumbersome name: Semi-Automatic AI-Native Development with Human in the Loop.

I described the broader workflow in another article, so I won’t repeat the whole thing here. The important part for this discussion is the human in the loop.

I almost never use AI in full-auto mode for meaningful development work. I use it aggressively, but I try to be very deliberate about which part of engineering I delegate.

For me, AI is an extremely fast electronic typewriter, an implementation engine, a research assistant, a second opinion and a tool for validating assumptions. What I do not want it to become is a replacement for engineering judgment.

Before I ask an agent to implement something, I try to define the system it is going to operate inside as precisely as the task justifies. Depending on the feature, that means thinking through types, validation rules, business requirements, functional behavior, file structure, data flow, relationships between modules, API contracts, architectural boundaries and test expectations.

And one thing I care about particularly strongly is semantics.

What does this entity actually represent? Who owns this state? Where does this transformation belong? What responsibility does this module have? What does this abstraction mean in terms of the domain? Which concepts are genuinely different, and which are merely different names for the same thing?

Those questions are not formatting preferences. They are architecture.

Once those constraints are clear, AI can do what it is extraordinarily good at: execute quickly inside them.

Recently I came across a post from Robert "Uncle Bob" Martin that described a surprisingly similar philosophy.

His approach is more extreme than mine. He says he does not read the code generated by his agents; instead, he surrounds those agents with extensive constraints: unit tests, Gherkin tests, QA procedures, quality metrics, mutation testing, coverage and many other forms of automated verification.

I do still read AI-generated code. But the underlying idea resonates with me: the engineer’s job increasingly becomes designing the environment in which automated implementation is allowed to happen.

You create the constraints. You define the architecture. You specify the semantics. Then you let the machine move quickly inside that system.

Why I still review the code

When people argue that developers should review AI-generated code, the usual explanation is safety: AI can hallucinate, misunderstand a requirement, introduce a security problem or simply produce incorrect code.

All of that is true. But for me, there is another reason that is at least as important.

The generated code is feedback on the architecture I designed.

Before implementation begins, an architecture is partly a hypothesis. I believe these responsibilities belong in these modules. I believe this abstraction is sufficiently general. I believe this data flow is clean. I believe this type model represents the domain correctly. I believe these boundaries will continue to make sense when the next requirement arrives.

Then the AI implements the feature at high speed, and that hypothesis collides with reality.

Perhaps the abstraction turns out to be awkward. Perhaps data starts leaking across boundaries that were supposed to be clean. Perhaps a type requires exceptions everywhere. Perhaps two concepts that looked independent are much more tightly coupled than expected. Perhaps the agent repeatedly struggles with one particular part of the design.

Those are not merely implementation problems. They are signals.

When I review the result, I am not only checking whether the agent wrote acceptable code. I am checking whether my model of the system survived implementation.

That creates an extraordinarily powerful learning loop.

AI can make you learn faster, not slower

This is the part of AI-assisted development that I think gets far less attention than it deserves.

Without AI, much of a developer’s working day is consumed by execution. You design something, then spend hours or days typing it into existence. You write boilerplate. You perform repetitive refactorings. You search through documentation. You implement the twentieth variation of a pattern you already understand.

AI can compress a huge percentage of that execution time.

If you keep yourself inside the reasoning loop, the time you save does not have to disappear. It can be reinvested into the parts of software engineering where the real intellectual work happens: architecture, modeling, semantics, constraints, trade-offs and validation.

My workflow increasingly looks like this:

Design → constrain → let AI execute → inspect the result → learn → adjust the design.

Then another requirement arrives and the loop starts again.

The interesting part is the speed. Suppose I design an architectural approach and implementing enough of it to discover its weaknesses would traditionally take several weeks. With an agent, I may get that feedback in hours. Sometimes much faster.

That means AI does not merely accelerate implementation. Used this way, it accelerates the architecture-feedback loop.

You make a design decision. Reality tests it almost immediately. You learn. Then you make another design decision.

Do that repeatedly and something interesting happens: instead of spending most of your time exercising your ability to type code, you spend much more of it exercising your ability to design systems.

That is almost the opposite of “AI melting your brain.”

It is closer to deliberate practice for software architecture, with an implementation engine attached.

Of course, this only works if you actually stay in the loop. If you delegate the decisions, delegate the implementation, accept the output and move directly to the next prompt, the mechanism reverses. AI keeps learning about your codebase while you gradually stop doing so.

That is comprehension debt.

I probably won’t generate 1,800 PRs a week. That’s fine.

A semi-automatic workflow with deliberate human involvement is obviously slower than telling a swarm of agents to modify a repository as quickly as possible.

That does not bother me.

I want to produce exactly as much code as meaningful product development requires. If AI makes me capable of implementing useful changes faster than a product organization can discover and validate those changes, I have already won. More implementation throughput beyond that point has rapidly diminishing value.

The objective of software engineering is not to maximize the amount of software produced. It is to solve useful problems with software.

AI gives us an extraordinary opportunity to remove a large amount of mechanical work from that process. I think the mistake would be to remove ourselves from the intellectual work at the same time.

So yes, review the code. It may contain bugs. It may contain security issues. It may misunderstand your requirements. Those are all perfectly good reasons.

But review it for another reason too.

Review it because it tells you whether your architecture was right.
Review it because it forces you to maintain a mental model of the system you are responsible for.
Review it because every implementation is an experiment, and every experiment gives you feedback.

And if AI can run those experiments dramatically faster than before, perhaps the technology that supposedly makes developers stop thinking can—used correctly—make us learn faster than we ever could before.

There is one line from Uncle Bob’s recent posts that summarizes the responsibility better than anything I could add:

That is the point.

You are not the engineer because you typed every line of code. You are not the engineer because you can explain every keystroke that created the repository. And you are certainly not the engineer because your AI generated more pull requests than somebody else’s.

You are the engineer because you are accountable for the system that comes out the other side.

What do you think?

How are you managing the balance between AI speed and system comprehension in your codebase? Do you use a human-in-the-loop workflow, or have you experimented with full-auto agents? Let me know in the comments below!

Stop asking AI to do a linter’s job.

Mykola Mizhigurskiy — Mon, 10 Aug 2026 14:56:28 +0000

I have developed a workflow that I call:

A semi-automated, AI-native development flow with a human in the loop.

It may sound like a collection of AI buzzwords, but each part represents a very practical principle.

1. Semi-automated

If a task can be performed reliably by a simple script, it should not be delegated to AI.

Want consistently formatted code? Configure Prettier.
Want custom naming conventions, import restrictions, or architectural constraints? Configure ESLint or write a custom rule.
Want consistent formatting across different editors? Add an EditorConfig file.

Do not make AI spend context and reasoning capacity checking indentation, semicolons, or naming conventions. Deterministic problems should have deterministic solutions.

Here is a less obvious example.

I have written several thousand unit tests for my current project. As a perfectionist and former QA engineer, I care about how elements are selected in tests.

Whenever possible, tests should use accessible selectors first, such as roles and accessible names. A test ID should only be used as a fallback.

Initially, I treated this as something AI should remember, while I would catch mistakes during code review.

But I noticed that I was correcting the same issue far too often.

So I asked myself:

Why am I repeatedly reviewing something that can be checked automatically?
I wrote a custom lint rule.

A few lines of code solved the problem permanently.

Now the linter knows the required selector priority and reports violations automatically.

The best part is that AI fixes these issues in virtually every case.

Why?

Because running the project’s validation procedures is mandatory in my workflow. Even when AI makes a mistake, it runs the linter, receives a precise error message, and corrects the code.

That is what “semi-automated” means to me:

AI handles tasks that require reasoning.
Scripts handle tasks that require consistency.

2. AI-native

A repository can be structured primarily for human convenience, or it can be structured so that both humans and AI can navigate it efficiently.

An AI-native repository treats context as a limited engineering resource.

Some developers are familiar with code golf: solving a problem using the fewest possible characters.

AI-native development is not exactly code golf, but the underlying constraint is similar:

Every token has a cost.

This does not mean compressing the code until it becomes unreadable. It means finding a balance between readability, consistency, and context efficiency.

Project standardization plays a major role here.

One component should not use a completely different structure, style, or paradigm from another component without a good reason.

Reusable abstractions should be clearly defined and used consistently.

This allows AI to find an existing implementation, understand it as the project standard, and use it as a reliable reference.

Another important principle is what I call context segregation.

A feature should be divided into meaningful files so that AI can load only the context required for the current task.

For example, a test implementation may have:

the test file itself
fixtures
mocks
setup utilities

Sometimes the setup is larger than the actual test suite.

When I only need AI to inspect test names or understand what behavior is covered, there is no reason to feed the entire setup into its context.

I use colocated suffix files for this purpose. A larger feature can have several supporting files placed next to it, with clear suffixes describing their roles.

Depending on the task, I can provide AI with exactly the files it needs—and nothing else.

And please, use barrel exports where they make architectural sense.

There is no reason to repeatedly feed AI long absolute import paths when a stable public module API can expose the same components with fewer tokens and less coupling.

3. Human in the loop

The internet currently seems obsessed with fully autonomous pipelines where an agent writes code around the clock without human participation.

Maybe I am old-fashioned, but I am highly skeptical of that approach.

I could list many examples of what AI misunderstands, what must be reviewed, and where autonomous implementations fail.

But there is one issue that matters more than all the others:

If you remove yourself from the process, you stop LEARNING.

You are no longer collecting feedback.
You are no longer examining what AI understands and what it misunderstands.
You do not see which instructions work, which create confusion, where AI is fast, where it struggles, or where an additional tool could eliminate an entire category of mistakes.

Most importantly, you lose the data required to improve your own development framework.

Here is my favorite example.

At one point, I created what I believed was the perfect AI development process.

I reviewed every instruction repeatedly. I refined every detail. I was convinced that the framework was exceptionally well designed.

Then I used it in production.

Within two months, the process went through five generations of fundamental changes—not counting dozens of smaller improvements.

Why?

Because real-world usage generated feedback.

I observed what failed, what created unnecessary work, what AI interpreted incorrectly, and what could be automated more effectively.

No matter how good your prompt, instruction set, agent configuration, or MCP setup appears, you do not have objective evidence that it works until you observe it under real production conditions.

That is why my workflow keeps a human in the loop.

Not merely to approve AI-generated code.

But to learn from every iteration and continuously improve the system producing that code.

A New AI Paradigm: The Billion-Dollar Blind Spot

Mykola Mizhigurskiy — Wed, 21 May 2025 11:55:36 +0000

A New AI Paradigm: The Billion-Dollar Blind Spot

Modern AI systems are excellent at detecting toxicity, abuse, radicalization.

But there's a blind spot — and it's costing us billions:

They can't detect brilliance.

Every day, millions of users interact with AI. Among them,

1 in 10,000 may hold a breakthrough idea.
1 in 100,000 may be modeling the future of society, science, or intelligence.

But all of that signal vanishes.

No system is designed to surface it.

Enter SVITLO

SVITLO (Signal of Visionary Ideas Through Latent Observation) is a protocol that proposes a new direction:

➡️ Not just detecting threats

➡️ But detecting insight

➡️ Surface-level agnostic, cognition-focused

It doesn't require new infrastructure.

It builds on existing analytics pipelines and reorients them from defense to discovery.

What if your model could say:

“This interaction shows signs of exceptional thought. Would you like to be seen?”

Use Cases

Identifying overlooked talent
Spotting founder-level thinkers
Building archives of visionary insight
Enhancing alignment, recruitment, and impact

📖 Full manifesto on GitHub:

🔗 SVITLO manifest

Feedback, forks, and discussion welcome.