#kubernetes #devops #sre #opensource
Eight months ago I published an article here called "What a 60-Second War-Room Scan Revealed in Production."
The article turned out to be right about one thing — but I realized I was solving the wrong problem.
The scan found things. Engineers who ran it found crash-looping pods with thousands of restarts, privileged containers in production, namespaces idle for weeks. The comments confirmed the pattern was real and widespread.
But the most common follow-up wasn't "great tool." It was: "I found the problem. Now what do I do first?"
That question is harder than it looks.
Finding Problems Was Never the Hard Part
Here's what actually happens after a scan surfaces ten issues:
You have a crash loop in payments. An ImagePullBackOff in inventory. A privileged container in auth. Three orphaned PVCs. Two namespaces missing NetworkPolicy.
Which one do you touch first?
The scan gave you a list. It didn't give you a rank.
I kept running into this in practice — the scan was useful, but it handed engineers a flat list of findings and walked away. The decision about where to start was still entirely manual. For a senior engineer with years of cluster experience, that's manageable. For everyone else, it's another form of paralysis.
I spent the next eight months trying to fix that.
What I Built
Not another dashboard. A triage layer.
I didn't want another page full of Kubernetes objects. I wanted the dashboard I wished I'd had during production incidents — the one that tells me where to spend the next hour instead of asking me to inspect another hundred resources.
The dashboard looks across reliability, security, and operational efficiency, then turns dozens of findings into one prioritized list. The result is OpsCart Watcher — an open-source operational triage dashboard for Kubernetes.
Overview page showing Incident Score 41/100 with Top 5 Things to Fix
The first screen shows one number: the Incident Score. Zero to 100, lower is worse. Below it, the Top 5 things that deserve attention first — not every warning in the cluster, just the ranked work that matters most.
Four crash-looping pods collapse into one ranked item. The ImagePullBackOff failures become another. The privileged container. The missing NetworkPolicy. The orphaned storage.
Five prioritized problems. One clear starting point.
The War Room
War Room showing critical issues with visual differentiation per type
Every critical issue in one view. Different issue types use different layouts because a CrashLoopBackOff, an orphaned PVC, and a privileged container require different information to make a decision. Crash loops show restart count and age. Image pull failures show the registry error. Privileged containers show which pod and namespace.
Every card carries the exact kubectl command to start investigating, with a copy button. Instead of jumping between namespaces, one prioritized view. At 2 AM, this is the difference between paralysis and progress.
The Part That Surprised Me
I thought the ranking was the hard part. It wasn't.
The hard part was what came after finding an issue. Engineers knew something was broken. They didn't know what to run next.
So I built the investigation page.
Investigation page showing OpsCart Assessment, Evidence, and Recommended Investigation
One click from any triage finding opens a dedicated investigation view. At the top: an OpsCart Assessment.
For a crash-looping pod with 1810 restarts over 6 days:
"The restart rate appears stable, suggesting a deterministic configuration or application failure. Investigation should begin with previous container logs. Estimated time: 5–10 minutes."
That paragraph is rules-based. No AI. It reads restart count, failure pattern, and referenced configuration objects. The output is deterministic and auditable.
Below it: confidence-rated investigation steps. High confidence: check previous logs. Medium: verify Secrets exist. Low: check for OOMKill in events. Each step has the exact kubectl command, ready to copy.
This builds on kubectl rather than replacing it. The goal is to tell you which command to run first, not hide the underlying Kubernetes primitives.
The five-minute walkthrough at the top of this article shows the complete workflow end to end.
What Eight Months Actually Looked Like
I want to be honest about the process because nobody talks about this part.
It was messy.
The bash script became a CLI scanner. The CLI became a dashboard. The dashboard grew pages. main.go grew to 3,100 lines before I finally admitted it needed to be refactored. I shipped features, took them down, rewrote them. The original article used production data; I had to take it down when I realized colleagues could recognize the cluster.
The SQLite operational memory — which now powers trend arrows and incident age tracking — was the last major piece. It changed the tool from something that answers "what is broken?" to something that answers "what changed?" An incident that has existed for three days deserves different attention than one that appeared five minutes ago.
Every version taught me something the previous version couldn't.
Lessons Learned
Three things I'd do differently:
Build the investigation page first. Detection without investigation is half a tool. The most useful feedback I received was always about what to do next, not what was broken.
Add operational memory earlier. Trend direction changes urgency. "8 critical issues, up 6 since yesterday" is different information from "8 critical issues."
Ship earlier and smaller. I spent months building before anyone else ran it. The bash script got 5,500 readers because I shipped it when it did one thing well. The dashboard took longer to reach people because I kept adding features before publishing.
Try It
Helm (recommended):
helm install opscart-watcher ./helm/opscart-watcher \
--namespace opscart-system \
--create-namespace
kubectl port-forward -n opscart-system svc/opscart-watcher 8080:80
kubectl:
kubectl apply -f https://raw.githubusercontent.com/opscart/opscart-k8s-watcher/main/deploy/dashboard.yaml
kubectl port-forward -n opscart-system svc/opscart-watcher 8080:80
Read-only. No agents. No cloud credentials.
What's included:
- Operational triage dashboard with Incident Score
- War Room for critical incidents
- Investigation workflow with confidence-rated recommendations
- CLI mode (JSON, HTML, and Markdown reports)
- Helm chart deployment
- SQLite operational memory for trend arrows and incident history
- Read-only RBAC — get and list only
- No agents, no cloud credentials, no telemetry
Kubernetes already gives us the data. The missing piece is deciding what deserves a human's attention first. That's the problem OpsCart is trying to solve.
GitHub: opscart/opscart-k8s-watcher
Previous articles in this series:
- What a 60-Second War-Room Scan Revealed in Production — January 2026
- From Bash Script to Operational Triage — July 2026 (DZone)
Top comments (0)