Two Bugs, Zero Clues: How AI Saved Me Days of Debugging My API

Tomasz Jędrzejewski — Fri, 13 Nov 2026 09:15:42 +0000

Until recently, I thought the worst type of bugs were the ones that happen only in production. Now I know that there's something worse: bugs that happen only in production and the only clue you have is "there's an error" message. This is exactly what happened to me recently. I've been working on a side project: an API for city information and reverse geocoding. Everything went fine locally: creating spec, implementation, testing, dataset preparation, refinement... the API is nice, but it's not a rocket science, so I did not expect any further issues. The deployment should be quick. But it wasn't. "Internal Server Error, Happy Debugging, Good Luck".

How I use AI in this project

I'm using agentic coding and Spec-Driven Development (SDD) to build my API-s. That's my personal goal for it - I know how to write a code, I've been doing this for the last 20 years. What I'm exploring is how specs translate to the final product, how it affects engineering and engineering decisions.

Until approx. a year ago, I hated agentic coding. The quality of the generated code was so poor that I spent more time on fixing the code than I saved. If something wastes my time, I don't like it. Simple. But this has changed, when a new generation of models arrived. Actually, before I gave the agentic coding another try (this time successfully), I firstly noticed the change in the code reviews. One day, the agent found an actual issue in my PR. Then another one, and another. This matters for me - code review is a skill, and many experienced developers suffer, because they often work with people who don't have it, so in turn - nobody does proper code reviews for them :). And I also make bugs.

I mention code reviews on purpose, because they have a lot in common with debugging the actual production issues. That's why I decided to engage the agent into solving my production bugs. Let's get into them!

Bug 1: A problem with production database

I deployed the API to the production server and started testing. One endpoint worked as expected. The two others - did not:

{
   "error": "Internal Server Error"
}

I did not panic, because the endpoint was intentionally configured not to reveal error details. I obtained it from the detailed logs, and then I got really confused:

SQLSTATE[25000]: Invalid transaction state:
1207 Update locks cannot be acquired during
a READ UNCOMMITTED transaction

This is a read-only API. All queries are SELECT queries, so where did this "update" and "transactions" come from? And why only in production? The same code worked well locally. Obviously there was something wrong with the production database setup, but what?

I started with asking my coding agent:

Cities API - I deployed the Cities API to the production server, and the reverse + nearby endpoints respond with the following SQL error: (error here) The search endpoint works as expected. The error does not occur locally. What can be wrong there?

The agent responded:

No isolation level is set here — it inherits whatever the server's default transaction_isolation is. That's the smoking gun. What's happening: candidatesInBoundingBox() runs MBRContains(ST_GeomFromText(:envelope, 4326), c.location) — this is the only query path used by reverse/nearby that touches the location column's SPATIAL INDEX. search never touches it, which is exactly why search works and reverse/nearby don't. InnoDB's spatial (R-tree) indexes can't do a plain MVCC consistent read — a lookup against them needs to take predicate/page locks to keep the R-tree structurally consistent. Under READ UNCOMMITTED, InnoDB refuses to take any locks for a read, so it can't satisfy that requirement and MariaDB raises error 1207 ("Update locks cannot be acquired during a READ UNCOMMITTED transaction"). This is a documented MariaDB/InnoDB limitation: spatial indexes are only usable above READ UNCOMMITTED.

The agent suggested setting the transaction isolation level manually. It did not solve the issue, but the direction was good, so I kept grilling. The agent then focused on the server deployment. It knew the vendor behind the production server and came back with a story about MariaDB and Galera setup that - supposedly - broke the spatial index functionality. But I challenged that, knowing that other spatial queries work, including other bounding box queries:

Can you provide sources about Galera cluster and the issue details? The problem is that I successfully ran the bounding box query via phpMyAdmin in the production server.

I got the answer:

I was wrong about Galera — no verified source backs that, and I should have said so rather than asserting it. Retracting it. The real cause is a confirmed MariaDB bug. MDEV-26123 — Using Spatial Indexes results in "Update locks cannot be acquired during a READ UNCOMMITTED transaction". Affected versions: 10.2.39, 10.3.30, 10.4.20, 10.5.11, 10.6.1 — fix targeted for 10.11. (...) This maps onto our code: (...)

I cross-checked the verdict by pasting the error message in the Google, and I found references to the same bug on StackOverflow. The only thing that did not match was MariaDB actually did not fix it in version 10.11. But that was enough to spot the problem. The bug reference contained the list of workarounds to try out, until one works. The combination of upgrading MariaDB + adding SQL_NO_CACHE to the query did the job and the problem was gone.

Verdict

Hopeless without agent? Actually, no. If you paste the same error into Google, you quickly get the bug ID.
Added value: The agent combined the bug ID with the actual queries in the code, and correctly identified why one endpoint works, and two others do not. This is the place where I saved a lot of time.
Agent performance: The agent hallucinated a bit, but it was due to my actions. I intentionally did not give the agent the access to the production server, but I never explained the limitations and the rules.
Lesson learned:
- In such issues, understanding the production setup is important. Even if the agent does not have the access to the server, I do have such access and I can check/test something - I just need to know what, and I could have told the agent to ask me to do it. In human-to-human communication, we should avoid forcing others to guess our intents, the same applies to the agents.
- If some explanation sounds suspicious, ask for sources.

Bug 2: error while publishing the API

I list my API-s in one of the API marketplaces. The publishing process involves importing my OpenAPI Definition and generating:

An editable description of the API and endpoints in Markdown format.
The schemas of input and output request/response JSON-s, which look like JSON Schema with some custom extensions.

After a successful import and fixing some descriptions, I proceeded to testing the integration. And again: two endpoints worked, the third one did not:

Validation failed.

Yes, that was the whole error. No more clues, no access to the server or the source code. I made a mistake here: I tried to solve the problem on my own. Obviously I failed. There was certainly a bug on the gateway side, but without access to the source code, all I could do was writing to the support team. The next day, a weird idea came to my mind: why not asking the agent to solve it? What do am I losing? Nothing.

Just like before, the agent did not have the access to the server. But this time, I prepared the data samples and explained the limitations to the agent:

example.json - example response from my API
input-schema.json and output-schema.json - schemas copy-pasted from the publishing tool

Prompt:

I need help examining the bug on the API gateway. How it works: it uses custom input and output schema definitions for requests and responses, converted from our OpenAPI Definition. When I run the test for nearby endpoint, I constantly get validation failed error with no further details (tried out multiple times on multiple fix attempts). In /{directory}/ you can find input and output schema definitions and an example response. You have no details about the error and the schema structure is custom. Please analyze them - maybe you will spot some issue that I miss? I suspect that this is a bug on the gateway side, but if we could create a workaround, this would be great. Outcome: suggestions of possible issues to review. If I see that it may be the cause, I will ask you to create an updated schema, so that I can try it out.

The agent replied:

Confirmed: openapi/cities-v1.yaml defines Radius using the OpenAPI 3.0 / JSON Schema draft-4 boolean form — exclusiveMinimum: true paired with minimum: 0 — and that's copied verbatim into {directory}/input-schema.json. This is the only parameter across all three endpoints (nearby, reverse, search) that uses exclusiveMinimum at all; reverse and search have no equivalent field and don't exhibit this pattern.

That matters because JSON Schema 2019-09+/2020-12 (what most "custom" schema validators, including the one on the marketplace, actually implement) requires exclusiveMinimum to be a number (e.g. exclusiveMinimum: 0, with no separate minimum), not a boolean flag. A boolean exclusiveMinimum is structurally invalid there — depending on the validator, this either gets silently ignored (wrong behavior but not a failure) or trips the validator into a degenerate "can't satisfy this constraint" / fail-closed state, which matches your symptom exactly: every call to nearby fails validation with no detail, regardless of what radius value you send.

My top suspect, in order of confidence:

radius's exclusiveMinimum: true + minimum: 0 (draft-4/OpenAPI-3.0 boolean form) — the one schema quirk unique to nearby's input among the three endpoints, and a very plausible reason the gateway's validator chokes consistently.

Lower-confidence secondary suspect: the type/officialNames/translations nested objects use "nullable": true sitting alongside "type": "object" plus allOf — also an OpenAPI 3.0-ism that a strict JSON Schema validator won't interpret as "allow null." But this pattern is identical across nearby, search, and reverse output schemas, so it wouldn't explain why only nearby fails — unless the other two endpoints simply haven't been exercised yet or happen not to hit a null in your test data.

Workaround to try first, cheaply: rewrite radius in the input schema to the modern numeric form. If that alone makes nearby pass, it confirms the gateway's validator is draft 2019-09+/2020-12-only and doesn't back-translate OpenAPI 3.0 boolean exclusiveMinimum — which would also mean you should audit the OpenAPI source for other boolean-style exclusiveMinimum/exclusiveMaximum usages before they bite another endpoint.

Want me to produce the corrected input-schema.json (and the matching OpenAPI fix) for you to try?

Nailed it! Indeed, that was the issue. The fix worked, I sent my API for review, and I also notified the support team that they don't need to debug anything, because I found the root cause.

Verdict

Hopeless without agent? Yes - I tried, I failed.
Added value: Very few people have the patience to compare every attribute by hand against multiple versions of JSON Schema spec, looking for the needle in the haystack. The agent can do this in minutes.
Agent performance: The very first response found the root cause.
Lesson learned: Making manual, random changes to the schemas without clues was not a good idea.

Lessons learned & summary

AI agents are good at analyzing large amount of information in short time, looking for patterns and anomalies. This is essential in the debugging process. I turned the findings into a skill, so that I don't have to remember explaining restrictions every time I debug a production issue. And here's exactly what I learned:

The more the problem is a "needle in the haystack", the more the agent can outperform human.
Provide clear information, which parts of the system work, and which do not. In both cases, this helped the agent to triage the issue to the right parts of the code much quicker than humans could do.
Be clear about what we know, and what we don't know about the issue.
If you don't want to give the access to a certain resource (e.g. production server) as a safety measure, say it explicitly and explain, how the agent can ask you for additional information or test something.
Asking to back the explanations with sources helps.
Engineering skills still matter to assess the responses, and when we need to make the final decision: which of the fixes to apply, especially if we deal with a production system?

I hope that you will find the two stories useful also for you!

Used AI model and harness: Claude Sonnet 5.5 (medium), Claude Code

DEV Community: Tomasz Jędrzejewski

Two Bugs, Zero Clues: How AI Saved Me Days of Debugging My API

How I use AI in this project

Bug 1: A problem with production database

Verdict

Bug 2: error while publishing the API

Verdict

Lessons learned & summary