How I Used SigNoz to Stop Debugging in the Dark

In 2026, AI can generate applications in minutes, but debugging them is still one of the biggest challenges in software development. While building a small microservice sandbox to explore SigNoz for the hackathon, I realized that writing the application was the easy part—understanding what was happening inside it was much harder. That's when I discovered how much easier observability makes debugging.

In this post, I’ll walk you through how I set up a local microservice sandbox to test SigNoz, the roadblocks I hit, and the hidden performance bottlenecks I uncovered along the way.

The Goal: Exploring the Pillars of Observability

To prepare for the hackathon, I wanted to do more than just test one tool. I aimed to fully explore how SigNoz handles the complete observability suite, including:

• 1. Traces: Mapping requests across microservices.
• 2. Logs: Viewing real-time text events.
• 3. Metrics & Dashboards: Plotting custom business metrics.
• 4. Alerts: Receiving automatic notifications when things go wrong.

The Sandbox Architecture

I believe hands-on development is the best way to learn rather than simply reading official documentation. So, I built a simple two-microservice architecture to evaluate SigNoz in a realistic environment. This setup allowed me to observe how requests flowed across services, how failures propagated between them, and how traces, logs, and metrics worked together during debugging.

Generating the Data: Simulating Real-World Traffic

An observability platform is only as useful as the data flowing through it. To properly evaluate SigNoz, I needed a mix of successful requests, failures, and realistic traffic patterns rather than a few manual API calls.

Observability dashboards are useless without data, so I created a PowerShell script (generate_traffic.ps1) to simulate live traffic automatically.

The script fires 25 successful orders and 15 failure-inducing requests, adding randomized delays to mimic real user behavior. This populates the SigNoz charts with a realistic stream of data.

Exploring the SigNoz Dashboard

With traffic flowing, I explored SigNoz’s monitoring features. Here are the five major ones I used:

1: Services APM & Latency (The Gateway Overhead Mystery)

The Services Overview page immediately revealed something unexpected—a large latency gap between my two services.

At first, I assumed the web-service itself was poorly optimized. However, the latency breakdown in SigNoz quickly showed that the application code wasn't the real bottleneck.

web-service averaged 800ms per order.
worker-service only took 148ms.
Tracing further showed:

• web-service spent 414ms calling worker-service.
• worker-service processed in 148ms. The missing 266ms was network lag caused by Windows-to-WSL2 bridging since SigNoz ran in Docker inside WSL2. Without SigNoz, I’d have wasted hours optimizing the wrong part!

2. Flamegraph & Traces (The Domino Effect)

Triggering the /fail endpoint revealed a domino effect in the trace view:

Instead of treating the failure as a single error, the trace visualized exactly how it propagated through each service, making the root cause immediately obvious.

The database INSERT failed.
This broke the worker’s handler.
Then the gateway call failed.
Watching this chain reaction in real-time helped me pinpoint the exact failure.

3. Exceptions Explorer & Log Correlation (No More Log Hunting)

The trace pointed me to the failed request, while the Exceptions Explorer and correlated logs helped me understand exactly why it failed.

When multiple requests fail simultaneously, traditional logs can quickly become overwhelming. SigNoz's Exceptions Explorer grouped errors cleanly, showing sqlite3.OperationalError along with the number of occurrences. Instead of scanning hundreds of log lines to find the first failure, I could immediately identify the recurring exception and focus on investigating a single issue. This saved me from chasing symptoms and let me concentrate on the actual root cause.

Log Correlation linked logs and traces using the same trace_id, allowing me to jump from the exception directly to every related log for that request. Instead of switching between multiple terminal windows and manually matching timestamps, I could follow the complete request from the gateway to the worker service in one place. With one click, I could see the exact sequence of events leading to the crash, making the investigation much faster.

4. Apdex Score: Tracking My User Experience

SigNoz reported an Apdex score of 0.5 for my web-service, even though there were no application errors.

At first, this confused me because every request was completing successfully. Looking deeper, I realized the issue wasn't reliability—it was response time. Since my average latency exceeded the 500 ms threshold, many requests were classified as Tolerating, reducing the overall Apdex score.

This was an important lesson for me: an application can appear healthy from an error-rate perspective while still delivering a poor user experience. The Apdex score helped me look beyond crashes and focus on how responsive the application actually felt to users.

5. Trace Funnels: Tracking the User Journey

I configured a Trace Funnel to track the complete order processing flow across both microservices:

• web-service: GET /order
• worker-service: POST /process-order

Since every request completed successfully, the funnel reported a 100% conversion rate, confirming that each request reached the final processing stage without dropping off.

Although my sandbox didn't expose any drop-offs, this feature showed me how easy it would be to identify exactly where users abandon a workflow in a real-world application. Instead of manually tracing requests across multiple services, the funnel would immediately highlight the stage where failures or interruptions occur.

My Favorite Feature: Log Correlation

Out of all the features I explored, Log Correlation is my absolute favorite because it solved the biggest headache I faced while testing this project—manually hunting down logs across different terminals.

Although traces helped me understand the overall request flow, Log Correlation was the feature I found myself using the most because it connected every piece of evidence in one place.

• No More Terminal Switching: Before SigNoz, I had to keep separate terminal windows open on my monitor (one for the gateway on port 3000 and another for the worker on port 4000) and manually line up timestamps to guess what happened.
• Unified Timelines: With Log Correlation, clicking a single error log instantly reveals the full request timeline and related database queries across both microservices.
• Automatic Stitching: It automatically links the gateway's warning (Sending failure request...), the worker's processor logs, and the final SQLite database error under a single Trace ID, making debugging ten times faster.

The Sentry-Style Exception Tracker (But Free and Integrated)*

Unlike other tools requiring separate signups (like Sentry), SigNoz offers built-in crash tracking for free in the same dashboard.

When my /fail database query crashed, I saw the exact Python line causing it—without extra configuration or juggling multiple services. This simplicity was a huge relief.

The Roadblocks I Hit Along the Way

Setting up SigNoz and OpenTelemetry wasn't just about running a few commands—it involved solving several small but important issues before everything worked together. Interestingly, fixing these problems also helped me better understand how the observability pipeline works under the hood.

• Silent Startup Freeze: Uvicorn and OpenTelemetry clashed on logging threads. Switching to synchronous logging solved the freeze.

• Windows vs. Docker Networking: My Python code running on Windows couldn't initially communicate with SigNoz inside WSL2 Docker. After adjusting my .env configuration, telemetry started flowing correctly.

• Missing Import Hunt: The Python SDK's log exporter required a less obvious import path (_log_exporter), which took time to discover but ultimately completed the logging setup.

With SigNoz’s full observability suite tested, I’m confident for the hackathon. Now, I can focus on building instead of debugging blindly.

The prep is done. It’s time to build! 🚀

Resources & Links to Learn More

Check out the SigNoz Documentation to set up your own dashboard.

• Explore the OpenTelemetry Python SDK to learn how auto-instrumentation works.
• Join the SigNoz Slack Community to connect with other developers.