Observability AI Project Using SigNoz

I Built My Observability AI Project Using SigNoz

Before this project, I honestly didn't understand observability.

I thought it was just dashboards and logs.

After building an AI project from scratch, I realized I had no idea what was happening inside my application after it started running. If something failed, I could only print variables or add more logging.

That became really frustrating.

So I decided to try SigNoz.

What I was building

I built a small AI system that learns from repeated SQL requests. It keeps track of user behavior, increases confidence over time, and eventually starts applying learned rules to future requests.

The important part wasn't the AI logic.

The important part was answering questions like:

What happened during this request?
Which component took the most time?
Was a rule retrieved?
How did confidence change?
What spans were created?
Can I replay exactly what happened?

I couldn't answer those questions with print statements.

Adding OpenTelemetry

I added OpenTelemetry instrumentation to my FastAPI application.

Instead of only logging messages, I started creating spans for every important operation.

For example:

request received
SQL evaluation
rule retrieval
guidance injection
evidence creation
confidence update
lifecycle completion

Each request generated its own trace.

Seeing the entire request as one timeline was much easier than scrolling through logs.

Custom Metrics

I also exported my own metrics.

Some examples:

total requests
rule confidence
evidence count
SQL safety ratio

These metrics appeared automatically inside SigNoz.

Watching confidence change after multiple requests was much more interesting than looking at static logs.

The most useful feature

The trace explorer became my favorite part.

When I clicked a trace, I could see every span and every attribute attached to it.

Things like:

classification
confidence
detected SQL patterns
duration
service name

Instead of guessing what happened, I could actually inspect the request.

That saved me a lot of debugging time.

What surprised me

I expected observability to be something you add after finishing a project.

Instead, I found it much more useful while building.

Whenever something looked wrong, the traces usually told me where to start looking.

I also realized that dashboards are only a small part of observability.

The traces were where I spent most of my time.

What I'd do differently

If I started again, I'd add OpenTelemetry from the first day instead of waiting until the project was mostly complete.

Adding observability later meant I had to revisit many parts of the code to instrument them properly.

Doing it early would have made development easier.

Final thoughts

This project changed how I think about debugging.

Instead of asking "Why did my code fail?", I can now ask "What exactly happened during this request?"

That small change made debugging much less frustrating.

If you're building anything more than a simple API, I'd recommend trying observability much earlier than I did.