The Problem
Manual security code review costs roughly $85/hour of engineer time and doesn't scale with commit velocity. Most AI code-scanners stop at "here's a vulnerability" — they don't verify their own fixes, and they're completely opaque about what they're actually doing under the hood.
DevGuard AI is my attempt at fixing both problems for the "Agents of SigNoz" hackathon: an autonomous pipeline that detects vulnerabilities, patches them, adversarially reviews its own work, and — critically — observes itself through SigNoz closely enough to change its own behavior mid-request.
Architecture
Request flow: Browser → POST /scan → FastAPI backend → circuit-breaker-wrapped pipeline → Scanner Agent (RAG-augmented) → Fixer/Validator reflection loop (up to 3 attempts) → response.
Three agents do the actual security work:
- Scanner — RAG-augmented vulnerability detection against a CWE/OWASP knowledge base, always on the strongest available model (a missed critical vulnerability is far more expensive than the token cost of using a bigger model).
- Fixer — generates a patch, with prior reviewer feedback folded in on retries.
- Validator — an adversarial reviewer whose entire job is finding reasons the fix is inadequate. It only passes a fix it would "personally ship."
If the Validator rejects a fix, its feedback goes straight back into the Fixer's next prompt. This loops up to 3 times before the pipeline gives up and returns its best attempt.
Where SigNoz Comes In
Every one of those steps — Scanner, each Fixer/Validator retry, the circuit breaker's state transitions — is wrapped in an OpenTelemetry span via a custom @traced decorator. Open a trace for a single scan in SigNoz and you see the entire reflection loop as a flame graph: which attempt failed, how long each agent took, and exactly where an LLM call retried after a transient failure.
On top of tracing, I built custom OTel metrics feeding a "DevGuard AI Command Center" dashboard in SigNoz:
-
devguard.llm.tokens_total/devguard.llm.cost_total— real token and cost accounting per scan, tagged by agent and model -
devguard.scan.latency— a histogram driving p50/p95/p99 panels -
devguard.cache.hit_total/miss_total— Redis cache efficiency -
devguard.circuit_breaker.state_changes_total— how often the resilience layer trips
I also configured three SigNoz Alert Rules so the pipeline monitors itself in production terms, not just in a dashboard someone has to remember to check:
- Cost Budget Exceeded — fires if cumulative LLM spend crosses a threshold
- SLO Degradation - High Latency — fires if p99 scan latency exceeds 15s
- Circuit Breaker Stuck Open — fires if the breaker trips repeatedly, signalling the upstream LLM provider is struggling
The Differentiator: A Self-Observing Agent Layer
This is the part I'm most excited about. Most "observability for AI agents" projects are one-directional — the agent runs, SigNoz watches. DevGuard closes the loop: the pipeline reads its own recent telemetry back out through SigNoz's MCP server before making key decisions, via a small mcp_client.py that speaks the real MCP protocol (streamable-HTTP, authenticated) against SigNoz's own MCP server — confirmed working with a live session.initialize() + list_tools() handshake returning real SigNoz tool names.
Two adaptations run off that telemetry today:
- Telemetry-aware routing — if recent LLM spend is trending high, the Fixer can be routed to a cheaper model tier — except for critical-severity findings, which always get the strongest model regardless of cost pressure.
- Cost Guardian — batches cumulative-cost checks and flips a global "conservation mode" flag when a session's spend crosses a budget.
Every adaptation is (a) returned in the API response so the frontend can show it, and (b) stamped onto the active OpenTelemetry span as an attribute — so a routing override is visible both to the end user and inside the SigNoz trace that produced it. And the whole layer is deliberately fail-safe: if SigNoz or its MCP server is unreachable, every function degrades to "behave exactly as if this layer didn't exist" rather than blocking a scan.
Resilience
A hand-rolled circuit breaker (CLOSED/OPEN/HALF_OPEN) sits between the pipeline and the LLM provider. If the provider starts failing, the breaker trips, the pipeline falls back to a cheaper model automatically, and a state transition event gets logged onto the active span — visible directly in the SigNoz trace, not buried in application logs.
What I'd Do With More Time
- Wire the MCP-informed routing decision to trigger visibly on every scan, not just under cost pressure, so it's easier to demo end-to-end
- Add a full postmortem-generation flow: when the circuit breaker trips, have a small LLM call write a plain-English root-cause summary automatically
- Push the accuracy benchmark (14 hand-labeled OWASP snippets, tracking precision/recall) into the dashboard as a live panel
Closing Thoughts
Building this for the hackathon reinforced something I didn't fully appreciate going in: instrumenting an AI pipeline for observability is the easy 80%. The hard 20% — and the actually interesting part — is closing the loop so the system can act on what it's observing about itself, without that self-observation ever being allowed to break the thing it's trying to help.
Repo: github.com/akashbichukale0111/devguard-ai
Built for the SigNoz "Agents of SigNoz" hackathon by WeMakeDevs.
Top comments (0)