At 22:57:58 in one of our test runs, our AI controller looked at a rate-limit cascade, decided the fix was to add worker threads, and confidently proposed MAX_WORKER_CONCURRENCY=999. Our guardrail engine rejected it in the same second, forced the agent to re-plan, and it came back with 100 — the actual safe ceiling. That single log line is the reason this whole project exists.
We built AegisMesh for the Agents of SigNoz hackathon. The idea started from a familiar 3 a.m. problem: in a cloud-native stack, one slow downstream service — a GenAI inference model, a vector DB — quietly triggers a cascade. Worker pools saturate holding open connections, the API gateway breaches its latency budget, and users get HTTP 504s. Manually untangling that means hopping between trace dashboards, log tools, and terminals while half awake.
The obvious 2026 answer is "put an LLM in the remediation loop." The obvious problem is that an LLM in the remediation loop can also cheerfully set a thread pool to 999 or run a DROP TABLE. So AegisMesh is really two things: an autonomous SRE loop that SigNoz can watch end to end, and a zero-trust guardrail that treats the AI as an untrusted operator. SigNoz is the eyes; the guardrail is the seatbelt.
The system we're actually watching
The target is a four-tier chain of FastAPI services, each on its own port: api-gateway (8001) → worker-queue (8002) → ai-inference (8003) → vector-db (8004). Every service is instrumented with the OpenTelemetry SDK and exports OTLP straight to our self-hosted SigNoz collector. We inject trace_id and span_id into every JSON log line so a log is one click from its trace, and we emit a GenAI token-usage histogram following OpenTelemetry's GenAI semantic conventions (gen_ai.client.token.usage, with bounded labels for token type and model — more on why "bounded" matters later).
The one thing that bit us early: the hop from worker-queue into the downstream call is asynchronous, and if you don't propagate context across it, the trace silently splits into two. Instrumenting the web framework isn't enough — you have to instrument the outbound client too. Once we did, a single request shows up as one connected waterfall across all four services.
Watching the cascade happen
We trigger a fault from the dashboard — say, a token rate limit on ai-inference (8003). What follows isn't scripted: worker-queue threads block on retries, the connection pool of 5 saturates, and api-gateway breaches its timeout budget and starts returning 504s. In SigNoz you can watch it in three signals at once — latency on the gateway climbs past 4,800 ms, TokenQuotaExceeded errors appear in the logs, and the token histogram spikes from ~110/s to ~1,450/s. On our topology map, the failing node lights up red.
The autonomous loop
When the incident fires, the controller queries SigNoz over the MCP server using JSON-RPC tool calls — get_trace_spans(trace_id) to isolate the root-cause span, get_correlated_logs(span_id) for the exact stack trace, and get_metric_aggregates() for the token-usage trend. It bundles those three signals into one telemetry payload and hands it to Gemini 1.5 Flash under a strict JSON schema contract, so the model can't ramble, it has to return a structured diagnosis:
*{
"failing_service": "ai-inference-service",
"failure_mode": "TOKEN_RATE_LIMIT_EXHAUSTION",
"recommended_action": "REDUCE_SAMPLING_RATE",
"target_param": "LLM_SAMPLING_RATE",
"recommended_value": 0.5,
"confidence": 0.98
}
*
Forcing the schema was the single biggest reliability win. Free-text answers were unusable; a typed contract made the output something the rest of the system could trust or reject.
The guardrail engine (the part I'd actually keep)
Every proposal passes through five checks before anything touches a live service: a security blacklist (instant reject on rm -rf, DROP TABLE, SUDO), a parameter whitelist, scope authorization (e.g. LLM_SAMPLING_RATE is only ever allowed on ai-inference), range bounds (MAX_WORKER_CONCURRENCY must land in [10, 100]), and per-service cooldowns so it can't thrash. When a proposal fails a bounds check, an adaptive re-planner recomputes the nearest safe value instead of just giving up.
That's what caught the 999. The model wasn't wrong to want more concurrency — it was wrong about the ceiling. The guardrail rejected the value, the re-planner clamped it to 100, and only then did the patch proceed. If you take one thing from this post: an LLM will propose out-of-range actions, and the guardrail is not optional decoration — it's the load-bearing wall.
Hot-patching without a reboot
An approved fix becomes an HTTP POST to the target service's /admin/config endpoint, updating the runtime parameter in memory — no container restart, no dropped in-flight requests. Within a few seconds the gateway's latency falls back under its 200 ms SLA and the topology node goes green again.
Writing the lesson back into SigNoz and why the second time is far faster
Here's the part I'm proudest of. After a successful remediation, the controller makes an MCP write call — create_alert_rule() — that writes a permanent alert rule back into SigNoz, keyed to the metric signature of the incident it just solved.
That changes the economics of the next occurrence completely. The expensive step in the whole loop is the LLM diagnosis — pulling three signals, reasoning over them, scoring confidence. The first time we see a fault, we pay that cost: in our local runs the full first-time loop lands around 9 seconds end to end. But once the permanent rule exists, a recurrence of the same fault is matched instantly by SigNoz, and the controller skips diagnosis entirely — it already knows the remediation, so it goes straight to the guarded hot-patch. In our runs that fast path completes in about half a second.
So the system gets faster at incidents it has seen before, because it caches the hard-won mapping from "this telemetry signature" to "this safe fix" as a SigNoz alert rule. First time: reason it out. Every time after: recognize and act.
What I'd tell my past self
A few honest lessons, including the ones that cost us hours:
1) Instrument the client, not just the server. Our trace kept splitting at the async queue hop until we propagated context across the outbound call. This is the classic OTel gotcha and we walked straight into it.
2) Keep high-variety IDs off metric labels. We nearly put a per-request ID on the token metric. That's a cardinality bomb, it belongs on span attributes, where it's cheap, not on a metric, where it multiplies your time series. Traces and metrics index differently for a reason.
3) A typed schema beats a clever prompt. The JSON contract did more for reliability than any prompt wording.
4) The numbers here are demo measurements on a laptop, not a production benchmark. The ~9s and ~0.5s figures are what we observed in our own runs; the "45 minutes of manual triage" is the context we're replacing, not a controlled comparison. I'd rather say that plainly than dress it up.
5) The MCP write path has a version floor. Creating alert rules via MCP needs a recent SigNoz version — confirm yours before you build the whole feature around it, or the write call will 404.
Wrapping up
AegisMesh isn't "AI magically fixes production." It's a narrow, observable loop detect in SigNoz, reason with an LLM, gate every action through a zero-trust policy, patch live, and write the lesson back as an alert rule so the next occurrence is cheap. The interesting engineering wasn't making the AI act; it was making it safe to let act, and making every step of that visible in SigNoz so a human can audit exactly what it did and why.
Repo, demo video, and the four-service test harness are linked below the guardrail policy file is worth a read if you're thinking about putting an agent anywhere near your config.
— Built for Agents of SigNoz 2026.
Top comments (0)