DEV Community

Cover image for Visualizing Deep Trace Hierarchies: Self-Hosting SigNoz via Foundry on Windows
Rahul Chandra Padamuttam
Rahul Chandra Padamuttam

Posted on

Visualizing Deep Trace Hierarchies: Self-Hosting SigNoz via Foundry on Windows

If you’ve ever had a night of going through a disorganized pile of flat text logs which had you searching for the reason a single API endpoint performance slowed to a crawl, you know the issue. Distributed systems require distributed tracing but often by the time you get that cloud-based observability platform up and running you end up dealing with issues of setup friction or unexpected costs. In this piece I’ll be talking about how I was able to get around the cloud completely and self-host a full local SigNoz backend on Windows with Foundry which in turn allowed me to trace out complex microservice call stacks with microsecond precision at no cost.

Why Local Observability Matters

While working on modern applications, it's becoming increasingly important to not only know which calls are failing but to identify the sources of latency in the whole call graph. Terminal logs are nice and all, but the moment you have to deal with something more involved than a simple syntax error you're on your own.

I set up this zero-cost sandbox to be able to generate some interesting structured telemetry and to be able to play around with OTel ingestion pipelines without incurring costs and without having to faff around with a staging environment. By running the 12-container SigNoz backend that I deployed via Foundry, I've turned my Windows machine into a self-contained observability playground.

This is how I got it to work, what gotchas I had to deal with, and what the data looks like behind the scenes.

Setting Up the Pipeline & Showing the Work

Setting up a distributed observability platform involves a number of networking gotchas in a local environment. If you launch a load-generator (such as the one provided by the OpenTelemetry project as telemetrygen) in a detached container, it needs to learn how to route the traffic to the SigNoz OTLP receiver running on the host machine.

Here’s how I fixed that on Windows using good old Docker’s --add-host flag to bind the host’s internal IP address in the container’s network namespace. Here’s a PowerShell command I’ve used to start generating some synthetic telemetry and forcing a complex 15-level deep execution tree:

docker run --rm --add-host host.docker.internal:host-gateway ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:latest traces --otlp-endpoint host.docker.internal:4317 --otlp-insecure --rate 5 --duration 300s --child-spans 15
Enter fullscreen mode Exit fullscreen mode

Once the signal generator started sending messages over port 4317, the local ClickHouse started indexing information in real-time. Here is what the whole observability stack looked like while working with the actual request flow:

Step 1: Working with the Traces Explorer and Waterfall View
Upon launching the application, the initial window that appeared to me was the Traces tab. First of all, I was struck by how easy it is to get an overall idea of all distributed transactions using the timeline view. It is possible to sort transactions using the service names or particular operations:

SigNoz Traces Explorer showing a list of distributed transactions from telemetrygen

Opening any one of the transactions shows the power of distributed tracing. Instead of seeing one simple line of the log, the custom --child-spans 15 parameter actually prompted OpenTelemetry to generate a nicely nested tree of execution spans.

15-level deep execution tree waterfall view in SigNoz with OpenTelemetry span attributes

This perspective allows you to identify performance issues at a glance. The parent span is waiting for a cascading staircase of child microservices. On the right-side, attributes pane, you can see the critical OpenTelemetry semantic conventions discovered by SigNoz - including the trace kind Server, status code Unset, and the network routing endpoint:

{
  "attributes": {
    "network.peer.address": "1.2.3.4",
    "service.peer.name": "telemetrygen-client"
  },
  "duration_nano": 123000,
  "kind_string": "Server",
  "name": "okey-dokey-9"
}
Enter fullscreen mode Exit fullscreen mode

Step 2: Tracking Application Metrics & Logs
In order to observe the interaction between the components, it is essential to correlate multiple telemetry sources. To see the connection between this stream of information and the current state of the system, I proceeded to the APM dashboard:

SigNoz APM dashboard displaying live request rate and low P99 latency metrics

The graphs were simple to read, clearly showing our load metrics. By looking at the graphs, I could see that our application worked stably at a certain level, handling 1.75 ops/s with low P99 latency that rarely peaked above 1.85 ms.

I proceeded to the Logs Explorer tab to see the actual requests. The generator sends rather simple-looking messages to the backend in the format of JSON arrays:

{
  "timestamp": "2026-07-12 19:10:02.250",
  "body": "the message",
  "attributes": {
    "app": "server",
    "service.name": "telemetrygen"
  }
}
Enter fullscreen mode Exit fullscreen mode

The Log Details interface parsed these JSON attributes accurately, indexing the custom app: server and transaction tags dynamically right alongside the string message:

SigNoz Logs Explorer displaying structured JSON log attributes correlated with trace tags

Step 3: Building Dashboards and Alerts Out of the Box

To conclude the exploration, I attempted to create a custom Dashboard with a specific query from scratch. The native Dashboard builder was employed, and the UI Query Builder allowed me to track the chosen internal metric over the 30-minute period:
Custom dashboard panel in SigNoz tracking average telemetry rate over time using Query Builder

  • Panel Type: Time Series

  • Metric Filter Selection: gen

  • Aggregation: Avg within time series (Every Auto Seconds)

  • Cross-Series Aggregation: Avg by Everything (no breakdown)

Next, I wanted to turn this monitoring graph into an active safety net by configuring a custom alert rule:
SigNoz Alert Rules interface configuring a metric-based threshold alert for error rates

Using the Query Builder interface, create a condition for which you want to be alerted. For example, “Alert me if the error rate is more than 5%”. Then, by analyzing data stream A, set a value of critical threshold near zero. Thus, in case our containerized environment encounters any errors or network issues, the defined rule will react to it, instead of forcing our developers to watch the dashboard in real-time.

What I Learned & Key Takeaways

The process of installing and configuring the local observability pipeline was educational for me, as I've got several crucial takeaways from the activity:

  1. The telemetry data is decoupled: one of the first insights I had while working on the project was that sending traces does not automatically make them available as logs and application metrics. These are three separate data streams handled by OpenTelemetry, and they should be correlated in the unified interface for proper analysis (in this case, the ClickHouse database).

  2. Visualization is the ultimate productivity booster: having a visual representation of my request processing flow in the form of the layered waterfall makes analyzing my application's behavior significantly more intuitive than poring over the raw text logs in the terminal.

  3. The local development loop is critical: having an observability setup that I can experiment with at will without fear of impacting production systems and wasting precious cloud resources on unneeded expensive instances is invaluable. With this approach, I can stress-test my own application to find what metrics thresholds are needed to trigger alerts.

Conclusion

By building this local observability sandbox I have gained valuable insight into the workings of a full-stack observability system. Now I am looking forward to applying my understanding of OpenTelemetry to my hackathon projects so that all components are observable, understandable, and high-performing.

You can view my complete deployment manifests, including the environment-specific casting.yaml files, in my public GitHub repository.

Top comments (0)