Everyone building an autonomous agent right now is optimizing for the same thing: how fast can it act. I spent most of this hackathon building the opposite instinct into mine.
This is the story of NeuroScale Autopilot, a five-agent Kubernetes incident-response system built for the Qwen Cloud Global AI Hackathon (Track 4: Autopilot Agent), and the specific design decision that I think actually matters: it doesn't just fix your cluster, it proves the fix is safe before it acts, and knows when to stop and ask a human instead.
The problem with "fast" as the only metric
I've worked as a platform engineer on real production Kubernetes clusters. The incidents that actually hurt weren't the slow ones. They were the fast, confident, wrong ones: an automated rollback that hit the wrong revision, an auto-scaler that scaled the wrong deployment to zero. Speed without judgment is how a small incident becomes a large outage.
So instead of building "an agent that fixes things fast," I built an agent that has to earn the right to act automatically, incident by incident, by passing an explicit set of checks first.
What NeuroScale Autopilot actually does
The pipeline is five agents:
- Detector: watches the Kubernetes event stream directly (pod events, OOMKills, CrashLoopBackOff, image pull failures)
- Analyzer (Qwen-Max): root cause analysis, with a confidence score and a risk level
- Planner (Qwen text-embedding-v3): semantic search over a runbook library via FAISS, returns the best-matching remediation with a similarity score
- Executor: runs the approved remediation with circuit-breaker protection
- Escalation (Qwen-Turbo): compresses the full incident context into a human-readable approval request
Here's the part that's actually the point of the whole project:
Between the Planner and the Executor sits a Trust Layer gate. Before anything executes automatically, three independent signals all have to clear a bar at the same time:
- Is the Analyzer's confidence high?
- Does the Planner's retrieved runbook similarity score clear 0.65?
- Is the risk level low?
If even one of those fails, the system doesn't guess. It holds the incident open, attaches the exact rollback command it would have used, and waits for a human to approve or reject it.
The incident that proved the design was right
I deployed NeuroScale on a real Alibaba Cloud ECS instance running a real k3s cluster (not a local demo; more on that below), and during testing I pushed a broken image tag to a sample checkout service to see what would happen.
Qwen-Max diagnosed it correctly in under three seconds. It even noted the tag looked like a mistake accidentally committed. High confidence, low risk. A textbook case for auto-remediation.
But the runbook retrieval score came back at 0.59, just under the 0.65 threshold, with the second-best match only 0.03 behind it. The system held. It didn't execute a plausible-looking fix on a shaky match. It escalated, with the rollback command already prepared, and waited for a human.
That's the entire thesis of the project in one incident: a confident answer is not, by itself, permission to act.
Why three different Qwen models, not one
I used Qwen for three distinct jobs, deliberately not the same model everywhere:
- Qwen-Max for the Analyzer, where deep reasoning actually matters and latency is less critical
- text-embedding-v3 for the Planner's RAG retrieval, where you want fast, cheap vector search over a runbook library
- Qwen-Turbo for Escalation, where you're compressing context into a short human-readable summary and speed matters more than depth
Routing different reasoning loads to differently-sized models kept the pipeline fast without sacrificing quality where it counts.
Real deployment, not a local demo
I wanted a URL a judge (or you, reading this) could open right now, not a description to take on faith. NeuroScale is deployed on a real Alibaba Cloud ECS instance in Singapore, running a real k3s cluster:
And the dashboard, live, reacting to a real incident on that cluster:
Numbers measured directly from that deployment:
| Metric | Result |
|---|---|
| Full pipeline latency, alert to human-ready decision (real Qwen inference included) | Under 5 seconds |
| Remediation plans with an explicit rollback command attached | 100%, zero exceptions |
| Behavior when Qwen calls failed during infrastructure testing | Escalated to human every time (never guessed) |
Two problems that taught me the most
The account configuration problem. My Alibaba Cloud account couldn't provision an ECS instance in mainland China (real-name authentication requirement), which I fixed by switching regions to Singapore. Later, every Qwen API call failed with an access-denied error even with a valid key. It turned out my API key belonged to a different Model Studio workspace than the one where I'd activated my models. A valid key and an activated model are two separate states, and they only work together if they're in the same workspace. Once I generated a new key from inside the correct workspace, everything worked.
The detector replay bug. While preparing the live demo, I found a real bug: the Kubernetes events watch had no resourceVersion continuation, so every ~60-second reconnect would re-list recent events instead of resuming where it left off. That meant a single ongoing issue could re-trigger a brand-new analyze-plan-escalate pipeline run (including a fresh Qwen API call) several times for the same problem. I fixed it by tracking resourceVersion across reconnects and adding a suppression window, then wrote four regression tests specifically covering that behavior.
What I'd tell someone starting the same build
Don't add more agents, more tools, or more scenarios to feel more impressive. I was tempted to keep expanding NeuroScale's surface area throughout the build. What actually made the project stronger was the opposite: cutting planned expansion and spending that time proving the one workflow that already existed was real, safe, and honestly measured, down to capturing a uniquely timestamped test incident specifically so the exact string could be cross-checked against the raw server logs.
Try it yourself
- Live dashboard: http://43.98.177.117:3000
- Code: github.com/sodiq-code/neuroscale-autopilot
- How the Trust Layer scoring actually works: TRUST_LAYER.md
- Full deployment verification, including live curl commands: PROOF_OF_DEPLOYMENT.md
Everyone's agent acts. This one proves it's safe first.
Top comments (0)