How I Used SigNoz to Prove My AI Incident-Triage Fallback Actually Worked

James Gurung — Sat, 18 Jul 2026 11:38:46 +0000

A FastAPI service, OpenTelemetry, self-hosted SigNoz, and one uncomfortable truth: HTTP 200 can lie.

My API returned HTTP 200. But the LLM it depended on had just failed.

No error. No exception. The user got a response — just not the AI one. A three-sentence local fallback ran instead, and without telemetry, nobody would ever know the difference.

This article shows how I built an AI incident triage agent, connected it to self-hosted SigNoz using OpenTelemetry, and used traces, logs, metrics, a dashboard, and an alert to prove exactly when the fallback ran and why.

What I Built

The app receives an error message and triages it into a structured incident report.
Request:

Request:

Response:

Three internal steps:

1.Classify the error with keyword matching
2.Load a matching JSON runbook
3.Call OpenRouter to generate an AI explanation

If the LLM is unavailable, a local fallback generates a shorter explanation from the runbook. Both paths return the same five-field response — that is the hidden problem. You cannot tell which path ran just by looking at the response.

Architecture

The service name incident-triage-warmup ties all three signals together in SigNoz. One name, three views of the same application.

Testing Normal Mode — LLM Works

Start the app:

cd blog set -a; source .env; set +a export FORCE_LLM_FAILURE=false .venv/bin/uvicorn app.main:app --host 0.0.0.0 --port 8001

Health check:

curl http://127.0.0.1:8001/health
{"status":"healthy"}

Send a real request:

curl -i -X POST 'http://127.0.0.1:8001/analyze' \ -H 'Content-Type: application/json' \ --data '{"error":"Database timeout while loading customer orders"}'

Server log:

INFO app.llm LLM request started INFO httpx HTTP Request: POST https://openrouter.ai/api/v1/chat/completions "HTTP/1.1 200 OK" INFO app.llm LLM explanation generated INFO app.main Analysis completed

Response: HTTP/1.1 200 OK with a 702-character AI explanation. The log line confirms the real provider was called and responded.

Generate 22 mixed requests:

AGENT_URL=http://127.0.0.1:8001 \ REQUEST_COUNT=22 \ bash blog/scripts/generate_normal_traffic.sh

Each LLM call took several seconds, so 22 requests took a few minutes. All 22 returned HTTP 200. Zero fallback activations in the log.

Three Mistakes Worth Knowing

These are the real problems I hit. I am including them because they only show up when you actually run the code.

Mistake 1 :- The fallback script does not enable fallback mode.

generate_fallback_traffic.sh is a curl loop. It sends HTTP requests and labels the output as "fallback scenario." It cannot reach inside a running Uvicorn process and change an environment variable. I ran it, saw 22 successes, and assumed I had proven fallback. I had proven nothing. The server was still calling OpenRouter normally.

Fix: stop the server with Ctrl+C, set FORCE_LLM_FAILURE=true, restart.

Mistake 2 :- Pasted a curl command without proper line breaks.

I got HTTP/1.1 422 Unprocessable Entity, then -H: command not found, then -d: command not found. FastAPI rejected the request because the body was missing. The shell then tried to run -H and -d as commands.

The correct version with proper escaping:

curl -i -X POST 'http://127.0.0.1:8001/analyze' \ -H 'Content-Type: application/json' \ --data '{"error":"Database timeout while loading customer orders"}'

Mistake 3 :- Used List View to understand one request.

I saw llm.request and fallback.generate in the same List View and assumed they belonged together. They came from different requests at different timestamps. List View searches individual spans. Trace View shows the parent-child structure of one complete request. That distinction matters when you are trying to prove causality.

Testing Fallback Mode LLM Fails, API Survives
Stop the server. Restart with the failure flag:

cd blog set -a; source .env; set +a export FORCE_LLM_FAILURE=true .venv/bin/uvicorn app.main:app --host 0.0.0.0 --port 8001

Send the same request:

curl -i -X POST 'http://127.0.0.1:8001/analyze' \ -H 'Content-Type: application/json' \ --data '{"error":"Database timeout while loading customer orders"}'

Response: HTTP/1.1 200 OK

The ai_explanation was a short local sentence — not the LLM response. Server log:

WARNING app.llm LLM unavailable; local fallback activated
INFO app.main Analysis completed
No openrouter.ai line appeared. The provider was never called.

Generate 22 fallback requests:

AGENT_URL=http://127.0.0.1:8001 \
REQUEST_COUNT=22 \
bash blog/scripts/generate_fallback_traffic.sh
Output:

Requests attempted : 22
Requests successful: 22
Requests failed : 0
Scenario : fallback (FORCE_LLM_FAILURE=true on server)
Log confirmed: 24 fallback activations, 0 provider calls.

My Favourite Feature: Distributed Tracing

A span is one named operation. incident.classify is a span. llm.request is a span. A trace connects all the spans for one request, in order, with parent-child relationships.

What I needed to see — and what Screenshot 1 shows — is this structure:

llm.request failed. The application caught it, ran fallback.generate, and the complete HTTP request returned successfully. The parent span is green because the overall request completed without an unhandled exception.

That is graceful degradation made visible. Without tracing, you see one HTTP 200. With tracing, you see the exact decision tree that produced it — every step, every status, every duration.

This is why distributed tracing is my favourite SigNoz feature. It transforms a black box into a visible sequence of named operations. When something goes wrong inside a successful response, tracing is the only signal that shows it.

Logs with Trace IDs

The warning log tells me fallback happened. The trace tells me what happened around it. The trace_id field connects them.

Searching SigNoz Log Explorer for LLM unavailable; local fallback activated returns warning entries with service.name = incident-triage-warmup and a trace_id field. One click on that ID opens the exact trace for that request.

Without trace IDs, correlating a warning to its request means manually matching timestamps across two tools. With them, it is one click.

Metrics: How Often Did This Happen?

Traces answer one request. Logs answer one event. Metrics answer the whole window.

The key counter is triage_fallback_total — it increments exactly once per fallback request. After 22 fallback requests, it shows a clear spike. After 22 normal requests, it stays flat. That difference is the measurement that proves the experiment worked at scale, not just for one manual curl.

Other instruments I created:

One practical note: metrics export every 15 seconds. If you check SigNoz immediately after sending requests, data may not appear yet. Wait one export interval before concluding the metric is missing.

Dashboard: One View for Everything

I created a dashboard named AI Triage Reliability with five panels.

The P95 duration panel tells the most interesting story. In normal mode, every request waits several seconds for the OpenRouter API. In fallback mode, the pipeline completes in milliseconds because no external call is made. That difference shows up as two completely separate bands on the same chart — without reading a single log.

Alert: Don't Let Fallback Hide an Outage

Fallback protects the user. But it must not silently hide a broken LLM provider. The engineering team needs to know when the AI layer is unhealthy.

Alert configuration:

Name: AI Incident Agent Using Local Fallback
Metric: triage_fallback_total
Condition: greater than 0
Evaluation window: 5 minutes

Trigger it:

AGENT_URL=http://127.0.0.1:8001 \ REQUEST_COUNT=10 \ bash blog/scripts/generate_fallback_traffic.sh Wait one evaluation window, then check the Alerts page.

What I Learned

HTTP 200 does not mean every internal step succeeded. It means the HTTP layer succeeded. The AI layer can fail silently behind it.

A fallback script cannot change a running server's environment. You must stop and restart with the new variable set. This cost me real time.

List View finds spans. Trace View explains a request. They are different tools for different questions.

A parent span can succeed while a child span fails. That is graceful degradation — not a telemetry bug.

Logs with trace IDs are worth far more than logs without them. One field turns a search into a one-click jump to the exact request.

Metrics show trends. Traces show causes. After the fallback ran 24 times, metrics told me the scale. Traces told me why each one happened.

An alert on triage_fallback_total > 0 turns silent degradation into a visible incident.

Conclusion

A reliable AI application needs more than fallback code. It needs telemetry that proves when the fallback was used, why it was used, and whether the user remained protected.

Every piece of that story was visible in SigNoz. The trace showed the failed LLM call, the successful fallback, and the successful HTTP response together. The log confirmed the warning with a trace ID attached. The metric showed the count growing across 22 requests. The alert fired when the threshold crossed.

Distributed tracing was my favourite SigNoz feature. It transformed "the API returned 200" into "here is exactly what happened inside that request, in order, with a status for every step." That is the difference between assuming the system worked and proving it.

Code:- https://github.com/jmass-ggg/signoz-silent-llm-fallback.git

DEV Community: James Gurung

How I Used SigNoz to Prove My AI Incident-Triage Fallback Actually Worked

What I Built

Testing Normal Mode — LLM Works

Three Mistakes Worth Knowing

My Favourite Feature: Distributed Tracing

Logs with Trace IDs

Metrics: How Often Did This Happen?

Dashboard: One View for Everything

Alert: Don't Let Fallback Hide an Outage

What I Learned

Conclusion