I wasn't hunting for a privacy problem. I just wanted to see how slow my AI agent really was.
For the "Agents of SigNoz" hackathon I built a small tool-using agent, wired it up to SigNoz with OpenTelemetry, and started poking at what got recorded every time it ran. Somewhere in the middle of that I found myself staring at a database row with the exact text a user had typed to my agent, sitting there in plaintext, forever. This post is what I built, what actually caught me off guard, and what I did after I saw that row.
Where I'm coming from
I'm not a devops person. During my internship I built factory monitoring dashboards, MQTT sensors going into Node-RED going into Grafana, watching machine temperatures and status codes. So I already knew what a metrics dashboard looks like. What I didn't know was distributed tracing, or what "observability" even means for something that isn't a sensor, like an AI agent making LLM calls.
I made myself one rule: I wouldn't write about anything I hadn't actually run. So everything below, I ran it myself.
The agent is small. It's OpenAI-powered with four tools: a weather lookup, a calculator, a fake "web search" that sleeps for 2.5 seconds on purpose, and a fake "API call" that throws an error on purpose. I instrumented it by hand with OpenTelemetry (traces, logs, metrics) and pointed it at a SigNoz instance I self-hosted locally with Docker.
Getting it running
One thing worth knowing first: SigNoz recently dropped its old docker-compose install in favor of a new CLI called Foundry (foundryctl). If you find an older tutorial it won't match what you actually get today, which cost me some confusion. I was on SigNoz v0.133.0 running ClickHouse 25.12.5.44.
The only real snag before I even reached SigNoz was Python. Homebrew's python@3.14 build ships without a working ssl module, so pip couldn't even reach PyPI ("Can't connect to HTTPS URL because the SSL module is not available"). I gave up on it and built my virtualenv against system Python (3.9.6) instead, which just worked. Small thing, but it's the kind of thing you only hit by actually sitting down and doing it.
The thing that changed how I think about "slow"
Once traces were flowing I opened the waterfall view for one of my agent's runs for the first time. That view takes a single request and breaks it into every step it took, laid out left to right by how long each step lasted. It was new to me. My old Grafana dashboards would tell me a number crossed a threshold. This told me which exact step in which exact request was the reason.
And it showed me something I wasn't expecting. In a normal run, my own tool code (the calculator, the weather lookup) ran in under a tenth of a millisecond. Every single OpenAI call took somewhere between 1200 and 3700 milliseconds. For an AI agent the model calls basically are the latency, and my own code is close to free by comparison. I would never have gone looking for that number on my own. The waterfall just put it in front of me.
Breaking it on purpose
I broke the agent two different ways to see how each one shows up in SigNoz.
First, a bad API key. That crashed the whole request, and both the failing LLM call and the parent trace came back marked as errored. Second, I made one tool throw an exception internally but caught it in my own code so the agent could recover and keep going. That time only the one tool's span went red. The overall run wasn't marked as failed, because it genuinely wasn't, the agent handled it and told the user something went wrong. So the shape of the trace told me which kind of failure I was looking at before I read a single line of text.
That second failure is also what I used to test alerts for real. I set up a Slack webhook, added it to SigNoz as a notification channel, and built a trace-based alert on "more than zero error spans in a rolling 5-minute window." Then I ran my broken tool six times in a row on purpose and watched the alert flip to Firing, with a real message landing in Slack a few seconds later. I didn't just configure it and trust that it works. I forced it to go off.
The dashboard side was messier, and honestly that's the part I most wanted to write about. My first panel, a P90 latency chart grouped by tool name, failed with a generic "something went wrong" error in the UI. Instead of giving up I went into SigNoz's own backend container logs and found the real cause: the query builder had generated a ClickHouse SQL query calling a function named histogramQuantile, which is a PromQL function, not a ClickHouse one. It doesn't exist in that engine. That's a real, reproducible bug in this version, not me holding it wrong. Switching the aggregation from a percentile to a plain average got around it.
Then I hit a second, dumber problem. The chart said "No Data" because ClickHouse stores timestamps in UTC and I was checking against my local time, eight hours off, and on top of that I hadn't re-triggered one of my tools since adding metrics. Once I fixed both, the chart was genuinely satisfying: one sharp 2.6-second spike for my slow tool, flat near-zero for everything else. Same insight the waterfall gave me, now as a time-series chart. It just took two real bugs to get there.
What I actually found
Here's the part I didn't see coming.
I queried what SigNoz had actually stored about one of my agent's runs, expecting spans, timings, maybe some metadata. Instead I found the complete, word-for-word text of a user's request and the model's full reply, sitting in plaintext in the trace data:
gen_ai.prompt.0.content: "Fetch live data by calling the /orders API endpoint."
gen_ai.completion.0.content: "It seems that the /orders API endpoint is currently
unavailable, and I couldn't fetch the required live data. If there's anything
else you need or if you'd like to try again later, please let me know!"
This isn't a SigNoz problem exactly. SigNoz just stores whatever spans it receives. The instrumentation library I was using (opentelemetry-instrumentation-openai, part of the OpenLLMetry ecosystem) is the one deciding to capture that content. But the result is the same either way. In a real product that's every user's prompt, and whatever they put inside it, sitting in your observability backend by default, with nothing warning you it's happening.
I didn't want to just claim that and move on, so I checked it two ways.
First I opened the actual installed package source and found the switch: a function called should_send_prompts(), gated by an environment variable TRACELOOP_TRACE_CONTENT that defaults to "true" if you never set it. Then I checked it against the official OpenTelemetry GenAI semantic conventions spec, not the library's own docs but the actual OTel spec. It explicitly calls out gen_ai.input.messages and gen_ai.output.messages as "likely to contain sensitive information including user/PII data," and recommends that capturing them should be opt-in. So this isn't only my opinion. The people who wrote the spec already flagged this exact risk. The library just still defaults the other way.
Then I tested the fix, not only the setting. I set TRACELOOP_TRACE_CONTENT=false, ran the same kind of request again, and looked at the new span. The content fields were completely gone. The token counts (input, output, total) were all still there, accurate, untouched. So you get to keep the number you actually need for cost and debugging, and drop the part that's a liability.
Fair is fair, though. ClickHouse and Postgres, where all this data actually lives, had zero ports exposed to my host machine in this setup, locked down correctly by default. The OTLP port that receives telemetry was the opposite, open with no authentication in front of it. Neither of those is a reason to panic. They're both just things worth knowing before you point any of this at something real.
What I'd tell myself starting over
The biggest shift for me wasn't a SigNoz feature, it was the mental model. Coming from dashboards built for physical sensors, I already understood metrics. I didn't understand tracing, and tracing turned out to be the thing that actually answers "why was this slow" for anything that isn't a sensor.
The second one is more of a warning than a lesson. If you bolt observability onto an AI feature just to debug it, go check what your instrumentation captures by default. "Observability" and "a permanent record of what your users typed" can quietly be the same thing, and nothing is going to warn you.
And two small practical ones: check your timezones before you assume "no data" means something is broken, and check the actual backend logs before you assume a UI error is your fault.
Where this leaves me
I only wrote about things I actually ran, so let me be clear about what that covers: traces, logs, metrics, a working alert, a dashboard panel, and one real, tested privacy behavior with a fix I verified myself. I didn't load-test any of this, and I haven't touched SigNoz Cloud. So take it as what happened on my one self-hosted setup, not a universal claim.
If you're building with LLMs and adding observability, it's worth five minutes to go check what your traces are actually holding onto. Mine had a lot more in them than I expected.
Top comments (0)