🚀 One of my favorite parts of building in the open is the feedback I receive from the community.
A few days ago, someone asked me a simple but important question: If DevOps Open Agent relies on one LLM for its recommendations, how do we know those recommendations are actually correct? Shouldn't another model verify them?"
That was a great point.
Today I've added LLM-as-a-Judge to DevOps Open Agent.
💡 Instead of blindly trusting the first AI response, you can now have a second, independent LLM review the investigation.
Here's what happens:
🔍 The primary LLM analyzes the Kubernetes issue and generates:
• Root cause
• Recommended fix
• Suggested kubectl commands
• Confidence score
🧑⚖️ A second LLM then receives both:
• The complete investigation evidence
• The first model's diagnosis
It independently checks:
✅ Is the diagnosis supported by the evidence?
✅ Are the recommended commands safe?
✅ Was any important evidence overlooked?
✅ Is the suggested solution complete and actionable?
Finally, it returns a verdict:
🟢 Agree
🟡 Partially Agree
🔴 Disagree
One thing I want to emphasize: the judge is advisory, not authoritative.
The goal isn't to replace engineers with another AI. The goal is to give engineers another perspective before making changes to production systems.
Even better, you're free to mix providers.
➡️ Diagnose with OpenAI and validate with Anthropic.
➡️ Diagnose with Anthropic and validate with Gemini.
➡️ Use whichever combination makes the most sense for your environment.
Everything can be configured directly from the UI, or you can define default providers using environment variables.
I genuinely believe that if we're going to use AI in production operations, trust has to be built, not assumed. Features like this are a small step toward making AI-assisted DevOps more reliable.
If you're interested in AI for DevOps, Kubernetes, and SRE, I'd love your feedback.
⭐ If you find the project useful, please consider giving it a star on GitHub.
GitHub: https://lnkd.in/giri6MF5
🎉 August enrollment is now open at IdeaWeaver AI Labs!
We have 3 programs starting this month, pick the one that fits where you are:
🔵 GenAI for DevOps Engineers
👉 https://lnkd.in/g8EUki-E
🟢 GenAI for Beginners
👉 https://lnkd.in/gV3byVfh
🟡 NVIDIA-Certified Associate: AI Infrastructure and Operations
👉 https://lnkd.in/g-ByqQF4
Multiple batch timings available: weekday and weekend, morning and evening.
🇮🇳 India folks: UPI/Paytm payment option available on the enrollment page.
Top comments (0)