DEV Community

Lochan Gowda T M
Lochan Gowda T M

Posted on

Getting Started with SigNoz: A Beginner's Journey into Observability

#ai #signoz #observability #devops

Getting Started with SigNoz: A Beginner's Journey into Observability

Last week, I spent over three hours trying to figure out why my AI agent was taking a sluggish five seconds to respond to simple queries. I kept wishing I could somehow see exactly what it was doing under the hood, rather than just staring at vague terminal logs.

That frustration led me to join the "Agents of SigNoz" hackathon. As a complete beginner in the world of observability, I wanted to understand how tools like SigNoz actually help developers monitor and debug complex AI applications in real-world scenarios.

My biggest realization this weekend? Tracing transforms debugging from a guessing game into a data-driven process. You no longer need to rely on logs or intuition; you can see exactly what happened, when, and where.

Here is a look at my journey setting up SigNoz from scratch and the insights I gained along the way.

The Setup: Self-hosting with Docker Compose

I decided to self-host SigNoz on my local machine using Docker Compose. The process is surprisingly straightforward, though it does require a bit of patience as the initial databases spin up.

After installing Docker Desktop, I simply cloned the official repository and ran the setup script:

git clone -b main https://github.com/SigNoz/signoz.git
cd signoz/deploy/
docker compose -f docker/clickhouse-setup/docker-compose.yaml up -d
Enter fullscreen mode Exit fullscreen mode

Troubleshooting My Setup

As a Windows user, things didn't work perfectly on the very first try. I ran into a specific issue with port conflicts that I had to troubleshoot and fix:

  1. Port 3000 Conflict: When I tried to start the containers, the frontend service failed because port 3000 was already in use on my system.
  2. The Fix: Instead of stopping my existing services, I opened docker-compose.yaml and changed the mapped port for the frontend service from 3000:3301 so that it would run on an available port.

With that quick tweak, all the containers successfully spun up!

Once the containers were healthy, I accessed the UI at http://localhost:3301 and created my admin account.

The Overview Dashboard provides a quick health check of your monitored applications, showing total traces, error rates, and duration distributions.

Exploring the Platform: Beyond the Basics

SigNoz provides a unified interface for metrics, traces, and logs. This is incredibly valuable because it reduces "context-switching." Instead of using Prometheus for metrics, Jaeger for traces, and ELK for logs, everything is available in a single dashboard.

For AI applications specifically, this means you can track token usage, LLM latency, and agent reasoning loops without jumping between different tabs.

SigNoz comes with customizable dashboards allowing you to build specific views for your infrastructure or AI models.

The "Aha!" Moment: Deep Dive into Tracing

While metrics are great for high-level overviews, tracing was by far the most intuitive and powerful feature I explored. Tracing shows exactly how requests flow through your application, revealing hidden latency bottlenecks and service dependencies.

To test this out, I looked at traces generated by HotROD, a sample application bundled with the installation.

The trace timeline view visually maps out the duration of each span and their parent-child relationships across multiple services.

I selected a trace that passed through several microservices: frontend, route, customer, driver, mysql, and redis. By simply looking at the colorful timeline (the Gantt chart), I could immediately identify service boundaries and see exactly which operation was taking the most time.

If an AI agent was running this request, I would be able to see the exact breakdown: input parsing → tool selection → API call → output generation. I would know instantly if the agent was stuck waiting on a database query or an external search API.

Why This Matters for AI Developers in Production

My weekend experimenting with SigNoz took observability from being an abstract "devops concept" to a concrete tool I can use daily.

Here is why this is game-changing for AI applications in production:

  1. Faster Debugging: Instead of guessing which service is slow, you can see exactly which span is the outlier.
  2. LLM Latency Analysis: You can trace each step of an LLM call to identify bottlenecks in network transfer or model inference.
  3. Proactive Issue Detection: By setting up alerts on key metrics, teams can be notified of issues before they impact users.

If you are building AI agents, starting small by implementing distributed tracing can save you countless hours of pulling your hair out over mysterious latency spikes.

Credits: A huge thank you to Claude Code and Antigravity for the guidance and assistance throughout this hackathon project!

Top comments (0)