DEV Community: Jason Yakubu

I Built an AI Agent That Breaks Your API and Fixes It Automatically

Jason Yakubu — Mon, 06 Jul 2026 11:50:15 +0000

This is the third post in a series. The first covered
why APIs break in production. The second covered
the plan for building PatchFlow. This one covers
what actually happened.

The honest version of how this got built

I started this hackathon with a clear plan and spent the first three days
completely rebuilding it.

The original idea was a cloud operations monitoring system. Agents watching
infrastructure, detecting anomalies, remediating incidents. It was a solid idea
with one problem: Alibaba Cloud had just shipped their own native operations
monitoring product. I found out on day three.

So I scrapped it and started thinking about what I actually found annoying
as a developer.

The answer came quickly. Every production incident I had ever dealt with traced
back to the same thing: the API did not handle failure gracefully. Not because
the developer was careless, but because testing failure cases manually is slow,
repetitive, and easy to skip when you are trying to ship.

I had written about this exact problem in my first post. The irony of ignoring
my own premise was not lost on me.

That became PatchFlow.

What PatchFlow does

You give it your API. It breaks it in 18 different ways, finds every endpoint
that does not handle failure gracefully, reads your actual source code, and
opens a GitHub pull request with the fix already written.

The full pipeline runs through six agents:

Discovery parses your OpenAPI spec or Postman collection into a grouped endpoint list for you to review before anything runs
Chaos injects real failure modes via actual HTTP requests, not mocks
Analyst identifies patterns across all results and produces a risk score
Repo Context clones your repository and follows the call chain across files to find where the real problem lives
Fix writes a patch using your actual code as context, not a generic template
Review audits the fix before any PR is opened, rejects it if something is wrong, and sends it back to Fix with specific feedback

That last pair, Fix and Review, ended up being the most important thing I built.

The part that actually took the longest

The Repo Context agent.

The naive version of this problem sounds simple: find the file that handles
the failing endpoint, read it, write a fix. That works for about thirty percent
of real codebases.

The other seventy percent have route handlers that look like this:

@app.post("/payments/charge")
async def charge(body: ChargeRequest, db: Session = Depends(get_db)):
    return await payment_service.process(body, db)

The route itself has nothing wrong with it. The problem is three files away,
inside payment_service.process, which calls stripe_client.charge, which
has no timeout configured and no exception handling around the external API call.

Teaching an agent to follow that chain required building a set of tools the
agent could call iteratively: search_in_files, read_source_file, and a
reasoning loop that decided at each step whether it had found the actual
problem location or just a delegation point.

The Qwen ReAct loop was what made this possible. The agent would read a file,
reason about whether the bug lived there or further down, decide to follow an
import, read the next file, and repeat until it reached a conclusion. Each step
was a genuine reasoning decision, not a heuristic.

I used qwen3.7-plus via Qwen Cloud's OpenAI-compatible endpoint for every
agent in the pipeline. The tool-calling support was solid throughout. I did not
have to change any of the tool schemas between agents, which kept the shared
base agent class clean.

The Fix and Review loop

After Repo Context locates the actual code, Fix writes the patch. But the first
version of a generated fix is rarely the best version.

The Review agent runs independently and checks:

Does the fix address the actual failure mode that was found
Are all required imports present and correctly referenced
Does the fix introduce duplicates that already exist in the file
Does the style match the conventions of the surrounding codebase
Could this change break existing behaviour

If Review finds a problem, it rejects the fix with structured feedback and
sends it back to Fix. Fix revises. The loop continues until Review approves.

This mirrors what real code review actually does. The difference is that the
entire cycle happens before the pull request is opened. By the time a PR lands
in the developer's queue, it has already been written and reviewed by two
separate agents with different objectives.

In testing against my demo application, the rejection rate was around thirty
percent on the first attempt. Most rejections were for missing imports or
duplicate exception handlers. After one revision cycle, the approval rate was
close to a hundred percent.

The part I got wrong

Endpoint discovery.

I spent two days on approaches that did not work. Scanning the codebase for
route decorators worked reasonably well for FastAPI but broke on Express, missed
dynamically registered routes, and consumed a lot of tokens on large codebases.
Probing common URL patterns was worse.

The solution was embarrassingly obvious once I stopped overthinking it: use the
same inputs that Postman and Insomnia use. OpenAPI specs, Postman collections,
or manual entry. Every serious API team already has one of these. Stop guessing
and ask for what you need.

This also led to one of the better UX decisions in the project. Instead of
auto-selecting endpoints and running immediately, PatchFlow shows you every
endpoint grouped by tag, flags anything that looks risky (admin routes, delete
operations, webhooks) as unchecked by default, and waits for you to confirm
what to test. Nothing runs until you say so.

I borrowed this from Postman because it is the correct mental model: you see
the full inventory before executing anything.

What Qwen Cloud made possible

I want to be specific about this rather than generic.

The ReAct loop with tool calling is where Qwen's quality showed up most clearly.
Multi-step reasoning tasks where the agent had to gather information across
multiple tool calls, form a hypothesis, test it with another tool call, and then
reach a conclusion were handled well. The Repo Context agent in particular runs
four to six tool calls per finding before concluding, and the reasoning between
calls was coherent.

The OpenAI-compatible endpoint made it straightforward to build a shared base
agent class that all six agents inherit from. The tool schema format is identical
to what you would write for GPT-4, which meant I did not have to learn a new
interface while also building the product logic.

The one adjustment I made was being more explicit in system prompts about the
expected output format. Asking for JSON output with a specific schema and
including a concrete example in the prompt produced more consistent structured
responses than relying on implicit formatting expectations.

The numbers

18 failure modes across four categories (network, dependency, data, resource)
6 agents in the pipeline
6 database tables (sessions, endpoints, failure results, agent steps, reports, pull requests)
Every agent step streamed to the frontend in real time via WebSocket

Against my demo application, a typical run across six endpoints produces around
forty failure injection results, identifies three to five findings, and generates
two to four pull requests. The full pipeline takes four to six minutes to complete.

Where it stands

PatchFlow is live at the link below. You can point it at any API using an
OpenAPI spec URL, upload a spec file, or use a Postman collection.

The GitHub integration uses your own OAuth token, so every pull request it opens
appears under your account, not a service account. You review and merge from the
PatchFlow dashboard.

The three things I would build next if I had more time: scheduled scans triggered
on every deployment, broader language support beyond Python (the Fix and Review
agents currently work best on FastAPI codebases), and Review agent metrics
tracking which failure modes produce the highest fix rejection rates.

Closing thought

The most useful thing I learned building this was not technical. It was that
rejection is a feature.

The Fix agent's first output is almost never its best output. The Review agent
exists because first drafts have problems, and structured feedback improves them.
That principle applies equally to agent pipelines and to the code they are
fixing.

If you are building a multi-agent system and your agents only produce outputs,
consider adding one whose entire job is to push back.

Built for the Global AI Hackathon Series with Qwen Cloud.

Source code: github.com/jaytech504/chaos-agent

Live demo: PatchFlow

Building PatchFlow for the QwenCloud Hackathon

Jason Yakubu — Wed, 24 Jun 2026 10:19:30 +0000

Over the past few days, I've been building PatchFlow, an experiment in autonomous API reliability.
The idea started from a frustration I've experienced while building backend systems.
When an API fails, the workflow is often the same:

Reproduce the issue
Search logs
Trace the failure
Find the relevant code
Create a fix
Open a pull request Even with modern tooling, much of this process remains manual. I wanted to explore whether some of these steps could be automated.

The Idea
PatchFlow is designed to help developers move from:
Issue → Root Cause → Patch
The goal is not simply to detect failures but to assist with understanding and fixing them.
At a high level, the workflow looks like this:
Plain text
OpenAPI Specification
↓
Endpoint Discovery
↓
Reliability Testing
↓
Failure Analysis
↓
Root Cause Detection
↓
Patch Generation
↓
GitHub Pull Request

Early Challenges
One of the first technical challenges I encountered was endpoint discovery.
My initial idea was to scan repositories and automatically discover API routes regardless of framework.
The more I explored this approach, the more limitations I found.
Different frameworks expose routes differently:
FastAPI
Django
Express
Next.js
Spring Boot
Attempting to support every framework through repository scanning quickly became complex and inefficient.
Why I Switched to OpenAPI
After researching how existing API platforms operate, I realized that many tools rely on OpenAPI specifications rather than trying to discover endpoints from source code.
OpenAPI already provides:

Endpoint definitions
Request schemas
Response schemas
Methods and parameters

This allows PatchFlow to focus on reliability testing instead of route discovery.
The architecture immediately became cleaner and more scalable.

Building with QwenCloud
For this project, I'm using QwenCloud to power the reasoning and analysis components of the platform.
The ability to work through architectural tradeoffs, evaluate different approaches, and iterate quickly has been particularly valuable during development.
One example was evaluating multiple approaches for endpoint discovery before settling on an OpenAPI-first workflow.
That decision significantly simplified the overall system design.
Current Progress
So far I have:

✅ Defined the product architecture
✅ Chosen an OpenAPI-first testing workflow
✅ Planned GitHub integration for automated pull requests
✅ Started frontend development
Next, I'll focus on:

Reliability testing engine
Failure analysis workflows
GitHub patch generation
End-to-end demo preparation

What I'm Learning

One lesson from this project is that many successful developer tools are effective because they reduce complexity rather than add more intelligence
.
My original design attempted to automatically discover every endpoint in every framework.
The current design is simpler, more practical, and closer to how developers already work.
Sometimes the best solution is not the most automated one.
Looking Ahead
The goal for the remainder of the hackathon is to build a working prototype that can:

Read an OpenAPI specification
Test critical endpoints
Identify failures
Suggest a fix
Generate a GitHub pull request

I'll continue sharing updates as PatchFlow evolves

Why APIs Break in Production and Why We Still Test Them Manually

Jason Yakubu — Tue, 16 Jun 2026 07:52:55 +0000

Modern software is built on APIs.

Every login, every payment, every user action — it all flows through backend APIs. And yet, despite how critical they are, API reliability is still surprisingly fragile.

We still rely on manual testing tools, handwritten scripts, and fragmented debugging workflows to ensure systems don’t fail in production.

And somehow, that hasn’t changed in years.

The Hidden Reality of Backend Systems

On the surface, APIs look stable:

They have defined routes
They return structured responses
They pass unit tests
They work in staging

But in real-world usage, things break in ways that are not always obvious:

A missing null check crashes a login flow only under specific inputs
A slow database query causes intermittent checkout failures
A malformed payload passes validation in staging but fails in production
A refactored service silently breaks a dependent endpoint
A race condition only appears under load

These are not rare edge cases — they are everyday production issues.

And the worst part is:

Most of them are discovered by users, not developers.

Why Current API Testing Is Not Enough

Today’s API testing ecosystem is powerful, but fundamentally limited in scope.

Postman and manual testing

Postman is widely used, but it relies on manually defined requests.

You must already know:

Which endpoint exists
What payload it expects
What scenarios to test

It is a tool for executing tests, not discovering system weaknesses.

CI-based test suites

Automated tests in CI pipelines help catch regressions, but they suffer from one core limitation:

They only test what developers explicitly wrote.

If a scenario was never imagined, it is never tested.

Observability tools (Datadog, etc.)

Monitoring tools are excellent at telling you what is broken in production.

But they operate after the fact.

They answer:

“What failed?”
“Where did it fail?”

But not:

“Why did this happen in the first place?”
“Could this have been prevented?”
“How do we fix it automatically?”
The Gap Between Testing and Fixing

Across the entire software lifecycle, there is a missing layer:

We test APIs manually or partially automatically
We observe failures in production
We debug and fix issues manually

But there is no system that closes the loop.

There is no system that:

Understands backend APIs, tests them intelligently, identifies failures, and fixes them automatically.

That gap is where most production bugs live.

A Different Approach: Treat APIs as Systems, Not Endpoints

The real problem is not testing individual endpoints.

The real problem is understanding how the system behaves under failure.

Instead of asking:

“Does this endpoint work?”

We should be asking:

“What happens when this system is stressed, misused, or given unexpected input?”

This requires moving beyond manual test cases and static assumptions.

It requires an autonomous approach.

Toward Autonomous API Reliability

A more complete system would work like this:

Understand the API surface

Instead of manually defining tests, the system reads structured API definitions (such as OpenAPI specifications) or derives them from the codebase.

Generate intelligent test scenarios

It does not just test “happy paths”, but also:

invalid inputs
missing fields
type mismatches
unexpected payload structures
failure conditions under stress

Simulate real-world failures

It intentionally breaks assumptions:

slow responses
partial failures
incorrect sequencing of requests

Analyze and trace failures

When something breaks, it does not stop at reporting an error. It traces the failure back to the likely source in the codebase.

Propose and apply fixes

Instead of stopping at detection, it generates code patches and opens pull requests for review.

This closes the loop between:

detection → diagnosis → correction

Why This Matters

Software complexity is increasing faster than our ability to manually test it.

Modern backend systems are:

distributed
event-driven
dependent on multiple services
deployed continuously

In such environments, manual testing does not scale.

Even traditional automation struggles because it assumes we can predict all failure cases in advance.

But real systems don’t fail in predictable ways.

The Direction Forward

The future of backend reliability is not more dashboards.

It is not more manual test scripts.

It is systems that can:

understand APIs automatically
stress-test intelligently
identify root causes
and propose fixes in real time

In other words:

Software that participates in its own debugging process.

Closing Thought

We have spent years building better tools to write code faster.

Now the next step is clear:

We need tools that help us trust code more, not just write it faster.

Because in production systems, speed of development means nothing without reliability.

And reliability should not be entirely manual anymore.