This is the third post in a series. The first covered
why APIs break in production. The second covered
the plan for building PatchFlow. This one covers
what actually happened.
The honest version of how this got built
I started this hackathon with a clear plan and spent the first three days
completely rebuilding it.
The original idea was a cloud operations monitoring system. Agents watching
infrastructure, detecting anomalies, remediating incidents. It was a solid idea
with one problem: Alibaba Cloud had just shipped their own native operations
monitoring product. I found out on day three.
So I scrapped it and started thinking about what I actually found annoying
as a developer.
The answer came quickly. Every production incident I had ever dealt with traced
back to the same thing: the API did not handle failure gracefully. Not because
the developer was careless, but because testing failure cases manually is slow,
repetitive, and easy to skip when you are trying to ship.
I had written about this exact problem in my first post. The irony of ignoring
my own premise was not lost on me.
That became PatchFlow.
What PatchFlow does
You give it your API. It breaks it in 18 different ways, finds every endpoint
that does not handle failure gracefully, reads your actual source code, and
opens a GitHub pull request with the fix already written.
The full pipeline runs through six agents:
- Discovery parses your OpenAPI spec or Postman collection into a grouped endpoint list for you to review before anything runs
- Chaos injects real failure modes via actual HTTP requests, not mocks
- Analyst identifies patterns across all results and produces a risk score
- Repo Context clones your repository and follows the call chain across files to find where the real problem lives
- Fix writes a patch using your actual code as context, not a generic template
- Review audits the fix before any PR is opened, rejects it if something is wrong, and sends it back to Fix with specific feedback
That last pair, Fix and Review, ended up being the most important thing I built.
The part that actually took the longest
The Repo Context agent.
The naive version of this problem sounds simple: find the file that handles
the failing endpoint, read it, write a fix. That works for about thirty percent
of real codebases.
The other seventy percent have route handlers that look like this:
@app.post("/payments/charge")
async def charge(body: ChargeRequest, db: Session = Depends(get_db)):
return await payment_service.process(body, db)
The route itself has nothing wrong with it. The problem is three files away,
inside payment_service.process, which calls stripe_client.charge, which
has no timeout configured and no exception handling around the external API call.
Teaching an agent to follow that chain required building a set of tools the
agent could call iteratively: search_in_files, read_source_file, and a
reasoning loop that decided at each step whether it had found the actual
problem location or just a delegation point.
The Qwen ReAct loop was what made this possible. The agent would read a file,
reason about whether the bug lived there or further down, decide to follow an
import, read the next file, and repeat until it reached a conclusion. Each step
was a genuine reasoning decision, not a heuristic.
I used qwen3.7-plus via Qwen Cloud's OpenAI-compatible endpoint for every
agent in the pipeline. The tool-calling support was solid throughout. I did not
have to change any of the tool schemas between agents, which kept the shared
base agent class clean.
The Fix and Review loop
After Repo Context locates the actual code, Fix writes the patch. But the first
version of a generated fix is rarely the best version.
The Review agent runs independently and checks:
- Does the fix address the actual failure mode that was found
- Are all required imports present and correctly referenced
- Does the fix introduce duplicates that already exist in the file
- Does the style match the conventions of the surrounding codebase
- Could this change break existing behaviour
If Review finds a problem, it rejects the fix with structured feedback and
sends it back to Fix. Fix revises. The loop continues until Review approves.
This mirrors what real code review actually does. The difference is that the
entire cycle happens before the pull request is opened. By the time a PR lands
in the developer's queue, it has already been written and reviewed by two
separate agents with different objectives.
In testing against my demo application, the rejection rate was around thirty
percent on the first attempt. Most rejections were for missing imports or
duplicate exception handlers. After one revision cycle, the approval rate was
close to a hundred percent.
The part I got wrong
Endpoint discovery.
I spent two days on approaches that did not work. Scanning the codebase for
route decorators worked reasonably well for FastAPI but broke on Express, missed
dynamically registered routes, and consumed a lot of tokens on large codebases.
Probing common URL patterns was worse.
The solution was embarrassingly obvious once I stopped overthinking it: use the
same inputs that Postman and Insomnia use. OpenAPI specs, Postman collections,
or manual entry. Every serious API team already has one of these. Stop guessing
and ask for what you need.
This also led to one of the better UX decisions in the project. Instead of
auto-selecting endpoints and running immediately, PatchFlow shows you every
endpoint grouped by tag, flags anything that looks risky (admin routes, delete
operations, webhooks) as unchecked by default, and waits for you to confirm
what to test. Nothing runs until you say so.
I borrowed this from Postman because it is the correct mental model: you see
the full inventory before executing anything.
What Qwen Cloud made possible
I want to be specific about this rather than generic.
The ReAct loop with tool calling is where Qwen's quality showed up most clearly.
Multi-step reasoning tasks where the agent had to gather information across
multiple tool calls, form a hypothesis, test it with another tool call, and then
reach a conclusion were handled well. The Repo Context agent in particular runs
four to six tool calls per finding before concluding, and the reasoning between
calls was coherent.
The OpenAI-compatible endpoint made it straightforward to build a shared base
agent class that all six agents inherit from. The tool schema format is identical
to what you would write for GPT-4, which meant I did not have to learn a new
interface while also building the product logic.
The one adjustment I made was being more explicit in system prompts about the
expected output format. Asking for JSON output with a specific schema and
including a concrete example in the prompt produced more consistent structured
responses than relying on implicit formatting expectations.
The numbers
- 18 failure modes across four categories (network, dependency, data, resource)
- 6 agents in the pipeline
- 6 database tables (sessions, endpoints, failure results, agent steps, reports, pull requests)
- Every agent step streamed to the frontend in real time via WebSocket
Against my demo application, a typical run across six endpoints produces around
forty failure injection results, identifies three to five findings, and generates
two to four pull requests. The full pipeline takes four to six minutes to complete.
Where it stands
PatchFlow is live at the link below. You can point it at any API using an
OpenAPI spec URL, upload a spec file, or use a Postman collection.
The GitHub integration uses your own OAuth token, so every pull request it opens
appears under your account, not a service account. You review and merge from the
PatchFlow dashboard.
The three things I would build next if I had more time: scheduled scans triggered
on every deployment, broader language support beyond Python (the Fix and Review
agents currently work best on FastAPI codebases), and Review agent metrics
tracking which failure modes produce the highest fix rejection rates.
Closing thought
The most useful thing I learned building this was not technical. It was that
rejection is a feature.
The Fix agent's first output is almost never its best output. The Review agent
exists because first drafts have problems, and structured feedback improves them.
That principle applies equally to agent pipelines and to the code they are
fixing.
If you are building a multi-agent system and your agents only produce outputs,
consider adding one whose entire job is to push back.
Built for the Global AI Hackathon Series with Qwen Cloud.
Source code: github.com/jaytech504/chaos-agent
Live demo: PatchFlow
Top comments (0)