Building AutoFix-Agent: How I Taught an AI to Fix Broken CI/CD Pipelines in 38 Seconds
Event: The Agent Harness Hackathon — WeMakeDevs × TrueFoundry × Qodo
Author: Priyanshu Singh · @Priyanshu123-coder
Stack: TrueForge · Qodo AI · Model Context Protocol (MCP) · TypeScript · Node.js
Links: GitHub · Live Dashboard · Demo Video
You push code at 11 PM. GitHub Actions fires. Two minutes later, you get this:
FAIL test/calculator.test.js
✕ Division operations mismatch (Expected 5, Received Infinity)
You open the logs. You download the runner output. You try to reproduce it locally. You figure out the root cause, write the fix, run the tests again, and open a PR — all while half asleep.
That entire process now takes 38 seconds. Automatically. Without you.
That's AutoFix-Agent.
Why I Built This
Modern engineering teams lose 20–30% of their sprint velocity manually triaging broken pipelines. The problem isn't lack of intelligence — it's that current tools don't go far enough.
Raw LLMs can write code. But they fail catastrophically at autonomous engineering because of three hard limitations:
1. They can't reach your tools.
A model in a chat window can't pull live GitHub Actions logs, inspect runner states, or query the actual CI failure without structured connectors.
2. They can't safely run what they generate.
If an LLM synthesizes a fix, there's no built-in mechanism to execute it in an isolated environment and verify that no regressions were introduced. It just guesses.
3. They can't be stopped before they do damage.
Without a governance layer, an agent will happily push untested, hallucinated code directly to your main branch.
This is exactly the gap AutoFix-Agent was built to close — using TrueForge as its runtime harness.
What TrueForge Actually Does (And Why It Matters)
TrueForge is TrueFoundry's open-source agent harness. It's the runtime layer that sits between a foundation model and everything it needs to interact with — your tools, your sandbox, your governance controls.
Here's why this changes everything:
A chatbot answers questions. An agent acts on them.
TrueForge turns a model into a working agent by giving it four things it doesn't natively have:
- Tool connectivity via Model Context Protocol (MCP)
- Safe code execution via isolated sandboxing
- State persistence across crashes and reconnects
- Human-in-the-loop governance before irreversible actions
Without these, you have a chatbot. With them, you have AutoFix-Agent.
The Architecture: A 6-Stage Autonomous Loop
AutoFix-Agent structures every remediation into a deterministic state machine orchestrated by the TrueForge runtime:
ANALYZING → REPRODUCING → PATCHING → VERIFYING → AWAITING_APPROVAL → COMPLETED
Here's what happens at each stage:
Stage 1 — Log Parsing & Ingestion
The agent fetches the raw GitHub Actions workflow run via the github_fetch_ci_logs MCP tool. It parses the stack trace, isolates the failing assertion, and identifies the suspect source file — all without any human input.
Stage 2 — Sandbox Reproduction
Before writing a single line of fix, the agent mounts the target workspace in an isolated sandbox (demo-repo/) and runs the test suite to confirm the failure reproduces. Exit code 1 — confirmed. This is the safety check that raw LLMs skip entirely.
Stage 3 — Patch Synthesis
The LLM core synthesizes a targeted fix. In the live demo, this means converting a multiplication operator bug into a proper division implementation with a zero-division guard. The patch is staged — not committed.
Stage 4 — Sandbox Verification
The patched code runs again inside the same isolated sandbox. The agent checks for two things: all tests must pass (exit code 0), and zero regressions must be introduced. If either check fails, it loops back to Stage 3 automatically.
Only when 5/5 tests pass with 0 regressions does the agent proceed.
Stage 5 — The Governance Gate 🔴
This is the critical moment.
Before doing anything irreversible, the TrueForge harness halts completely. It presents the unified diff to the operator and requires explicit sign-off. You either approve or reject. Nothing moves until a human decides.
This isn't optional. This isn't a prompt. This is a hard architectural stop built into the harness runtime — exactly what "control and safety" means in a production AI agent.
Stage 6 — PR Dispatch & Qodo Review
Once approved, the agent creates a feature branch, pushes the verified patch, and opens a GitHub Pull Request. Qodo's /agentic_review is triggered automatically and audits the code before it can be merged.
The Benchmark Numbers
| Metric | Manual Triage | Raw LLM Chat | AutoFix-Agent |
|---|---|---|---|
| Mean Time to Remediate | 14.5 minutes | 8.2 minutes | 38.2 seconds |
| Sandbox Regression Rate | 8.3% | 34.0% | 0.0% |
| Governance Control | Manual | None | Enforced gate |
| Automated PR Review | Manual | None | Qodo /agentic_review |
| Cross-Platform Support | Variable | N/A | Windows, Linux, macOS |
The 0.0% regression rate is the number I'm most proud of. Every patch is verified before it ever leaves the sandbox.
How Qodo Made the Code Better
Every feature in this project went through a GitHub Pull Request reviewed by Qodo before it was merged. That's not just a hackathon rule — it's genuinely how the code got better.
On PR #1, Qodo's /agentic_review surfaced a real cross-platform defect: executing sandbox test commands without an absolute binary reference would throw ENOENT on Windows hosts.
We refactored SandboxTools to use execFile with process.execPath — an explicit Node.js binary path that works consistently across operating systems. A follow-up review passed clean with zero high-severity findings.
The Qodo review trail is public: PR #1 on GitHub
What Broke Along the Way
No engineering story is complete without what went wrong. Here's what actually broke:
The Windows ENOENT bug — the one Qodo caught. On Linux everything worked. On Windows, the sandbox execution failed silently because I hardcoded "node" as the executable instead of using process.execPath. Never assume a binary is on PATH.
Session state persistence — early versions lost context after reconnection. TrueForge's session store solved this, but wiring .trueforge/sessions.json correctly across the state machine took several iterations.
The approval gate UX — the first version showed the diff in the terminal. That's useful for engineers. It's terrible for a demo video. The final version surfaces it as a prominent UI component with a hard approve/reject decision — the "money shot" of the demo.
The Bigger Lesson
This hackathon answered a question I'd been thinking about for a while: what's actually missing between LLMs and production-grade autonomous agents?
The answer isn't intelligence. Foundation models are already capable enough to synthesize correct code patches.
The answer is infrastructure — the runtime layer that gives models:
- Real tool access (not mocked)
- Safe execution environments
- Persistent state
- Human oversight at the right moment
TrueForge provides exactly that. AutoFix-Agent shows what's possible when you stop treating LLMs as chatbots and start treating them as the core reasoning engine inside a properly engineered harness.
The harness is the product. The model is the engine.
Try It Yourself
Live Dashboard (no setup needed):
👉 https://priyanshu123-coder.github.io/autofix-agent/
Run it locally:
git clone https://github.com/Priyanshu123-coder/autofix-agent.git
cd autofix-agent
cp .env.example .env
# Add your ANTHROPIC_API_KEY or OPENAI_API_KEY
node dist/demo_runner.js
Demo Video: https://youtu.be/smTV2T7QwJY
GitHub Repo: https://github.com/Priyanshu123-coder/autofix-agent
Thank You
Built in 7 days for The Agent Harness Hackathon, organised by WeMakeDevs in collaboration with TrueFoundry and Qodo.
If you're building agents, use a harness. If you're reviewing code, use Qodo. And if your CI is broken at 11 PM, there's an agent for that now.
If this resonated, follow along — I write about building production AI agents, developer tooling, and the infrastructure that makes AI actually useful.
#AI #AgenticAI #TrueForge #Qodo #OpenSource #DevOps #MachineLearning #WeMakeDevs #Hackathon #TypeScript #GitHub
Top comments (0)