Everyone building an autonomous agent right now is optimizing for the same thing: how fast can it act. I spent most of this hackathon building the opposite instinct into mine.
This is the story of NeuroScale Autopilot, a five-agent Kubernetes incident-response system built for the Qwen Cloud Global AI Hackathon (Track 4: Autopilot Agent), and the specific design decision that I think actually matters: it doesn't just fix your cluster, it proves the fix is safe before it acts, and knows when to stop and ask a human instead.
The problem with "fast" as the only metric
I've worked as a platform engineer on real production Kubernetes clusters. The incidents that actually hurt weren't the slow ones. They were the fast, confident, wrong ones: an automated rollback that hit the wrong revision, an auto-scaler that scaled the wrong deployment to zero. Speed without judgment is how a small incident becomes a large outage.
So instead of building "an agent that fixes things fast," I built an agent that has to earn the right to act automatically, incident by incident, by passing an explicit set of checks first.
What NeuroScale Autopilot actually does
The pipeline is five agents:
- Detector: watches the Kubernetes event stream directly (pod events, OOMKills, CrashLoopBackOff, image pull failures)
- Analyzer (Qwen-Max): root cause analysis, with a confidence score and a risk level
- Planner (Qwen text-embedding-v3): semantic search over a runbook library via FAISS, returns the best-matching remediation with a similarity score
- Executor: runs the approved remediation with circuit-breaker protection
- Escalation (Qwen-Turbo): compresses the full incident context into a human-readable approval request
Here's the part that's actually the point of the whole project:
Between the Planner and the Executor sits a Trust Layer gate. Before anything executes automatically, three independent signals all have to clear a bar at the same time:
- Is the Analyzer's confidence high?
- Does the Planner's retrieved runbook similarity score clear 0.65?
- Is the risk level low?
If even one of those fails, the system doesn't guess. It holds the incident open, attaches the exact rollback command it would have used, and waits for a human to approve or reject it.
The incident that proved the design was right
I deployed NeuroScale on a real Alibaba Cloud ECS instance running a real k3s cluster (not a local demo; more on that below), and during testing I pushed a broken image tag to a sample checkout service to see what would happen.
Qwen-Max diagnosed it correctly in under three seconds. It even noted the tag looked like a mistake accidentally committed. High confidence, low risk. A textbook case for auto-remediation.
But the runbook retrieval score came back at 0.59, just under the 0.65 threshold, with the second-best match only 0.03 behind it. The system held. It didn't execute a plausible-looking fix on a shaky match. It escalated, with the rollback command already prepared, and waited for a human.
That's the entire thesis of the project in one incident: a confident answer is not, by itself, permission to act.
Why three different Qwen models, not one
I used Qwen for three distinct jobs, deliberately not the same model everywhere:
- Qwen-Max for the Analyzer, where deep reasoning actually matters and latency is less critical
- text-embedding-v3 for the Planner's RAG retrieval, where you want fast, cheap vector search over a runbook library
- Qwen-Turbo for Escalation, where you're compressing context into a short human-readable summary and speed matters more than depth
Routing different reasoning loads to differently-sized models kept the pipeline fast without sacrificing quality where it counts.
Real deployment, not a local demo
I wanted a URL a judge (or you, reading this) could open right now, not a description to take on faith. NeuroScale is deployed on a real Alibaba Cloud ECS instance in Singapore, running a real k3s cluster:
And the dashboard, live, reacting to a real incident on that cluster:
Numbers measured directly from that deployment:
| Metric | Result |
|---|---|
| Full pipeline latency, alert to human-ready decision (real Qwen inference included) | Under 5 seconds |
| Remediation plans with an explicit rollback command attached | 100%, zero exceptions |
| Behavior when Qwen calls failed during infrastructure testing | Escalated to human every time (never guessed) |
Two problems that taught me the most
The account configuration problem. My Alibaba Cloud account couldn't provision an ECS instance in mainland China (real-name authentication requirement), which I fixed by switching regions to Singapore. Later, every Qwen API call failed with an access-denied error even with a valid key. It turned out my API key belonged to a different Model Studio workspace than the one where I'd activated my models. A valid key and an activated model are two separate states, and they only work together if they're in the same workspace. Once I generated a new key from inside the correct workspace, everything worked.
The detector replay bug. While preparing the live demo, I found a real bug: the Kubernetes events watch had no resourceVersion continuation, so every ~60-second reconnect would re-list recent events instead of resuming where it left off. That meant a single ongoing issue could re-trigger a brand-new analyze-plan-escalate pipeline run (including a fresh Qwen API call) several times for the same problem. I fixed it by tracking resourceVersion across reconnects and adding a suppression window, then wrote four regression tests specifically covering that behavior.
What I'd tell someone starting the same build
Don't add more agents, more tools, or more scenarios to feel more impressive. I was tempted to keep expanding NeuroScale's surface area throughout the build. What actually made the project stronger was the opposite: cutting planned expansion and spending that time proving the one workflow that already existed was real, safe, and honestly measured, down to capturing a uniquely timestamped test incident specifically so the exact string could be cross-checked against the raw server logs.
Try it yourself
- Live dashboard: http://43.98.177.117:3000
- Code: github.com/sodiq-code/neuroscale-autopilot
- How the Trust Layer scoring actually works: TRUST_LAYER.md
- Full deployment verification, including live curl commands: PROOF_OF_DEPLOYMENT.md
Everyone's agent acts. This one proves it's safe first.
Top comments (18)
Best framing I have read this week: a confident answer is not, by itself, permission to act. The Trust Layer holding at 0.59 with the rollback already prepared is exactly the behavior I wish more agents had, and the honesty of shipping that instead of forcing a plausible fix is the whole point.
One thing I kept circling on: your judgment lives at runtime, per incident, which is great. The harder half is judgment that persists. When the system escalates and a human approves or rejects that rollback, where does the verdict go? If it evaporates, the next near-identical incident re-runs the same gate and re-escalates the same call. The compounding version is writing each human verdict back as a decision the agent reads before acting (what we chose NOT to auto-remediate here, and why), so restraint accumulates instead of resetting every incident.
That handoff of operating context and decisions across runs is basically why I built cowork-os, so this hit home. Does the approve/reject verdict feed back into your runbook library, or is every incident judged cold?
Appreciate this — and no, right now every incident is judged cold. The
approve/reject verdict gets logged in the incident history, but nothing
feeds it back into the Planner's retrieval or the Analyzer's reasoning yet.
You've named the actual next step better than I had: not just an audit
log, but the agent reading its own decision history as context before
acting. Concretely I'm thinking the Planner's RAG index gets extended
with past verdicts attached to runbook matches, so a rejected auto-remediation becomes a negative signal the retrieval score accounts for next time.
Going to credit this framing when I build it. Thanks for the sharp read.
The RAG-index-with-verdicts move is the right long-tail mechanism, past incidents attached to runbook matches as a negative signal is clean. One caution, and it is your own hero incident in reverse: retrieval only surfaces a verdict when the new incident ranks similar enough. The near-miss is the trap, an incident just under the cutoff that slips the gate and re-runs the exact call a human already rejected. Your 0.59 held because the threshold was doing the work, a buried negative signal does not get that luxury.
So I would split it. Long tail in the RAG index like you said, but the load-bearing verdicts, the never-auto-remediate-this-class-here, go in a small standing policy the Planner reads before retrieval, not gated by whether they rank this turn. Pull-by-relevance for history, push-by-state for the hard blocks.
When a human rejects a remediation, do you capture the why or only the that? The why is what lets the block generalize to the near-miss instead of memorizing one incident.
You're right about the near-miss problem. Relying on retrieval alone means a similar incident could miss the similarity cutoff and lose the context of an earlier human decision. I like the split between a small standing policy that's always evaluated and retrieval for historical context. Right now I record the full decision and Trust Layer signals, but not the operator's rationale in a structured form. That's the missing piece, because the why is what lets the system generalize beyond one incident instead of just remembering it. That's the direction I'm exploring for the next iteration.
That's exactly it, and you put it cleaner than I did: the why is what generalizes, the what is just a log line. Once the operator's rationale lives as structured text, a near miss can match on the principle even when the surface features drift below your similarity cutoff, which is the exact failure mode you flagged.
The shape that has worked for me: each verdict becomes a small decision record, not a log entry. The fields I keep are the context, the constraint that actually drove it, the verdict, the rationale in a sentence or two, and a status: active, superseded, or expired. The Planner reads the active set before acting, so a rejected auto remediation is not just a negative retrieval score, it is a stated rule with a reason attached. When the reason stops holding you mark it superseded rather than deleting, so the trail survives without getting re-litigated.
I keep that in plain files rather than a vector store on purpose, so the why stays human readable and correctable. Open here if any of the structure is useful to lift: cowork-os.
How are you leaning on storage: does the rationale sit next to the runbook match in the RAG index, or as a separate policy layer the Planner always evaluates before it hits retrieval?
I'm leaning toward keeping them separate. The RAG index still feels like the right place for precedent and historical context, but I don't want hard operational constraints to depend on similarity retrieval. My current thinking is that the Planner evaluates a small policy layer of active decision records first, then uses RAG to enrich that decision with related incidents and runbooks. I do like your active/superseded idea as well. It preserves the reasoning trail without turning yesterday's judgment into tomorrow's permanent rule, which feels important if the operating environment changes. I'll take a look at cowork-os as well, I’m curious how you've structured those decision records in practice.
Keeping them separate is the call I would make too, and for the reason you gave: a hard operational constraint that depends on similarity retrieval can miss the cutoff exactly when it matters, so it should not live in the ranker. The shape that has worked for me: each decision is a record with a status (proposed, active, superseded, expired, rejected), the constraint it actually settled, a one or two line rationale for the why, an owner and a review date. The Planner reads the active set first as your standing policy layer, then RAG enriches with the related incidents and runbooks, which is your split exactly. Superseded is the load-bearing move: a reversed decision is not deleted, it flips to superseded and stays as trail, so the reasoning survives without turning yesterday's judgment into a permanent rule when the environment changes, which is the risk you named. If it helps to see it wired up, the structure is here: cowork-os, the decisions/ folder is the whole pattern. When your Planner reads the active set, do you gate purely on status, or do you also scope the active records to the current operating context so an active but out-of-scope constraint does not fire?
I'm thinking the same: a separate policy layer for the active hard constraints (based on status), then RAG for enrichment. The active/superseded idea is smart. It keeps the reasoning trail without locking in old decisions when things change.
I'll check the decisions folder in cowork-os. Right now I'm mostly using status-based gating, but scoping to context like namespace or workload type is something I want to add.
Appreciate you sharing the details. This is making the next steps much clearer.
Namespace or workload type as the scope key is exactly the missing piece I would add too. Status alone tells you if a decision is live, not whether it applies here. What has worked for me is a scope field next to status, so the Planner filters to applies to (namespace: payments, workload: stateful) before it even checks proposed vs active, otherwise an active decision fires somewhere it was never meant to. This whole thread turned into one of the more useful design reviews I have had on dev.to, thank you for pushing on every weak point. If the decisions folder in cowork-os is useful once you dig in, a star helps me know it landed, no pressure either way. What ends up being the scope key for you, namespace, environment, or something else?
"Speed without judgment is how a small incident becomes a large outage" — this should be printed on every platform engineering team's wall.
The "earn the right to act" pattern is exactly right. I've been experimenting with a similar concept in AI agent quality checking: instead of letting the model decide pass/fail on every case, use vote entropy as an escalation signal. When the model is uncertain, don't force a decision — escalate to human judgment. Same philosophy as your approach.
One question: how does the Planner handle cases where the FAISS runbook search returns low-similarity matches? Do you have a minimum confidence threshold before the Executor is allowed to proceed?
Thanks, Xiao Man. I like the vote entropy analogy because it's the same design philosophy: uncertainty shouldn't be treated as just another prediction, it should change the system's behavior.
On the Planner, yes. If the best runbook match falls below ~0.65 (or the retrieval margin is too narrow), the Planner can still propose a remediation, but the Trust Layer won't allow autonomous execution. Instead, it routes for human approval. The 0.59 incident was a good example: the system had a plausible rollback ready, but the evidence wasn't strong enough to let it act without review.
That 0.65 threshold is interesting — it's basically a confidence gate before the trust layer. The idea that the Planner can still propose but the Trust Layer says "not today" without a human stamp feels right. Too many systems either go full-auto or full-block. Having that middle ground where the suggestion exists but needs sign-off is the pragmatic sweet spot. Curious what your false-positive rate looks like around that threshold — do you find the 0.59 case is rare or does it come up more than expected?
The 0.65 threshold is holding up great. That 0.59 case was actually a deliberate test on my end (I forced a bad image tag), but similar low-similarity edge cases do pop up organically during chaos testing. They aren't super common, but they happen just enough to prove the human fallback works.
I totally agree on that middle ground. Going full-auto in production K8s is reckless, but staying full-manual defeats the point of the agent. "Propose but require sign-off" is the perfect balance.
Speaking of your earlier point, I’m really curious about your vote entropy experiments. How are you actually applying that for quality checking in practice?
Glad the threshold is working for you! Forcing bad inputs in chaos testing is smart — that's exactly how you build confidence in the guardrail.\n\nOn vote entropy: I run 3-5 models on the same ambiguous task and compare their outputs. When they all agree (low entropy), I let it through. When they diverge wildly (high entropy), I escalate to human review. The key insight was that entropy correlates much better with "needs human attention" than any single model's confidence score.\n\nPractical tip: I cap anchors at 6-10. More than that and you start optimizing the rubric instead of catching edge cases. Quality over quantity.\n\nHave you tried measuring entropy across temperature settings? I found that running the same model at T=0.3 vs T=0.7 gives a different kind of disagreement signal.
That's really interesting. I haven't tried varying the temperature yet. Right now the Trust Layer is based more on evidence quality than reasoning variance, so it's looking at things like analyzer confidence, retrieval confidence, and risk rather than disagreement between multiple outputs.
But I can definitely see temperature disagreement being useful as another uncertainty signal, especially when the model can justify two different remediation paths.
I like the idea because it's asking, "Is the model consistent with itself?" instead of just trusting a confidence score. Definitely adding that to my list to experiment with.
Glad the temperature angle resonates! For vote entropy in practice, the setup is pretty straightforward — run the same input through the model 3-5 times with identical prompts but different seeds, then count how many outputs actually agree on the same action. High agreement = low entropy = you can trust it. Low agreement = high entropy = escalate to human. The key is setting the threshold empirically from your specific domain data rather than guessing. We typically see the sweet spot around 60-80% agreement depending on how deterministic the domain should be.
That's a neat way to approach it. I like the idea of calibrating the threshold from real behavior instead of picking a number upfront. I can definitely see that fitting alongside the Trust Layer as another signal rather than replacing it. Appreciate you sharing how you're using it.
Exactly — treating it as an additional signal rather than a replacement keeps the architecture flexible. The Trust Layer already has enough on its plate with evidence quality and retrieval confidence. Vote entropy slots in nicely as the "model-consistency-with-itself" check, which is a different failure mode than evidence quality.