Everything Was Green. The AI Was Still Wrong.
Three days before I wanted to call this done, I was staring at the SigNoz Logs Explorer trying to figure out why my OTLP logs weren't showing up. Traces had landed fine from the first run. Logs? Nothing. Not even an error, which honestly is worse than an error — at least an error gives you something to grep for.
That's kind of the whole point of this post, actually. The agent can be dead wrong and none of your dashboards will notice, because they were never built to notice.
The gap I kept running into
Standard observability tells you three things: is it up, is it fast, did it throw a 500. That's it. An AI agent can pass all three checks and still hallucinate a fact, retrieve zero relevant documents and answer anyway, or execute a plan that was broken from the first step. None of that shows up as an error. It shows up as a decision — and decisions don't come with status codes attached.
I kept coming back to one rule while building Veritas: it can't just be a black box scoring another black box. If it flags a run and I can't explain why in plain English, that's not accountability, that's just a second opinion nobody can argue with. So everything the score is based on has to already exist on the trace, and every point has to be something a human could look at and say "yeah, that's fair" or "no, that's wrong."
That's also why nothing in the risk engine is machine-learned. Not because ML is bad, just — for this specific job, I wanted something I could point at and explain, not something I'd have to trust.
What it actually does
Veritas reads OpenTelemetry data out of a self-hosted SigNoz instance and turns it into a Semantic Risk score from 0–100 for each agent run. Then it builds a Replay Timeline so you can jump straight to the step where things fell apart instead of scrolling through logs like you're doing archaeology. It doesn't store any telemetry itself — SigNoz stays the source of truth, Veritas just reads off it.
My FastAPI agent has five stages: planner, retriever, tool, memory, response. Each one is its own OTel span nested under a root agent.run span, and each carries attributes like tool.success, retrieval_score, response.confidence. Six heuristics turn those attributes into the score, with weights that live in one file:
RETRIEVAL_ZERO_DOCS = 25
TOOL_FAILURE = 30
PLANNER_ZERO_TASKS = 25
CONFIDENCE_VERY_LOW_PENALTY = 30
LATENCY_HIGH_PENALTY = 10
I ran it against five scenarios through the actual engine (not typed in by hand):
| Scenario | Score | Level |
|---|---|---|
| Healthy | 5 | LOW |
| Retriever issue | 35 | MODERATE |
| Tool timeout | 65 | HIGH |
| Planner failure | 75 | HIGH |
| Multiple failures | 100 | CRITICAL |
Small thing, but every weight is a multiple of 5, so the engine can only ever spit out multiples of 5. I'd written an early planning doc with "illustrative" scores like 48 and 84 in it, and when I noticed that months later I realized the engine literally cannot produce those numbers. I left the real output alone instead of fudging it to match the doc — a demo that's quietly tuned to look verified is exactly the thing this whole project is supposed to catch, so it would've been a bit much to do that here.
Here's the Replay Timeline on a real tool-timeout run, score 75, HIGH. Retriever comes back with one document instead of several, the policy lookup times out at 330ms, confidence drops to 0.41 — each of those shows up as its own line instead of getting buried somewhere in a log file.
Click into a flagged run and you get a full incident report: root cause, what was expected vs. what happened, a plain-English chain of why, and the business impact. All of it comes straight out of the same risk breakdown — no extra model call, no second AI judging the first one.
Why SigNoz
Before any of the risk-scoring stuff, I had a much more boring problem: I needed somewhere to actually put the telemetry. I'd built agent pipelines before without any real observability layer — just print statements and a prayer — and this time I wanted to do it properly, which meant OpenTelemetry, which meant picking a backend to send it to.
My first instinct was the usual stack everyone reaches for: Prometheus for metrics, Loki for logs, Tempo for traces, Grafana to glue it all together and pretend it's one tool. I got about an hour into setting that up before I realized I was going to spend the whole weekend wiring four services together and configuring their scrape intervals and label schemas, and none of that time would go toward the actual idea, which was scoring agent runs, not learning YAML. So I went looking for something that treated traces, logs, and metrics as one thing instead of three things you correlate by hand, and SigNoz kept coming up. Self-hosted, OTLP-native, one docker-compose to get a working instance, and — this is the part that mattered most — logs and traces share the same trace_id in the same UI, so you can pivot from a span straight to the log lines that happened during it without switching tools or copy-pasting an ID between two different dashboards.
That single decision is basically the reason Veritas works the way it does. The Replay Timeline only exists because I could pull a trace and its logs together as one query instead of stitching two systems' outputs together after the fact. If I'd gone with the four-tool stack, I think I would've spent the entire build just writing glue code to keep Tempo's trace IDs and Loki's log labels in sync, and the actual risk engine — the part I actually cared about — would've been the last 10% of the weekend instead of the first 80%. SigNoz didn't give me the scoring logic, that's still just Python and a handful of if statements, but it gave me a place to point that logic at where everything I needed was already sitting next to everything else.
Here's the raw Logs Explorer mid-debug, with risk_score and risk_level sitting right there as structured fields on the log line, not something I had to reconstruct from a separate metrics panel:
The stuff that actually broke
Logs that went nowhere. Traces worked from run one. Logs didn't. I wired up the OTLP log handler, hit the endpoint, checked the explorer — nothing. No error, no warning. My first guess was propagate = False somewhere blocking records before they hit the root logger's handler, so I grepped the whole logging setup for the word propagate. Zero hits. Theory dead, and no error message to chase down instead. What ended up working was attaching the handler directly to each veritas.agent.* logger by name instead of the root logger. Logs started flowing immediately. Downside: lines duplicate now, since both handlers fire on the same record.
I went back and checked this against the actual OpenTelemetry Python docs afterward, rather than just trusting my memory of why the fix worked, and it turns out the documented setup is the opposite of what I did — you're supposed to attach the handler once, to the root logger, and named child loggers should reach it automatically through Python's default propagate=True. The OTel docs also mention a much more common cause for this exact symptom: if logging.basicConfig() gets called anywhere before the OTel logging integration is set up, the integration's formatting silently never takes effect. So if you hit this same wall — logs go nowhere, nothing in the console — check your init order first. That's the documented issue. Mine might have just been working around the same root cause from a different angle, honestly not 100% sure. Leaving the account here as it happened rather than rewriting it to sound like I knew what I was doing at the time.
A route that didn't exist. The Replay Timeline kept throwing 404s and every instinct said "broken import somewhere." It wasn't that — app/trace/[id]/page.tsx genuinely hadn't been created yet. Components were built, the route just wasn't wired up.
Once that got fixed, I found the trace lookup only recognized two hardcoded keys — high-risk and low-risk. Five other demo fixtures were sitting fully generated on disk and completely invisible to the UI.
SigNoz's percentile gap. A P50/P90 latency panel threw Function with name 'histogramQuantile' does not exist. Turned out to be a real version mismatch in the specific SigNoz/ClickHouse build I'd self-hosted — nothing wrong with my config. I ended up rebuilding every percentile panel as a simple sum-divided-by-count average instead.
Zooming out to the fleet view
Past a single run, there's a fleet-level dashboard: risk distribution, latency by stage, which agents are failing most, and some rule-based recommendations keyed off whatever's breaking most often right now.
Worth being upfront about: the numbers on that page are simulated 24-hour traffic across three agents, and it says so right on the page. Only the Refund Agent card links to a real, clickable trace — Research and Support Agent don't yet. Didn't want someone to click around and find that out on their own without me having said it first.
What I'd tell myself three weeks ago
"No errors" is not the same as "working." I know that now, but at the time it just felt like relief — like, okay, good, nothing's on fire. Took me embarrassingly long to realize that silence is its own red flag, and arguably a worse one than an actual error, because at least an error gives you a line number.
The planning doc thing I found almost by accident — I had it open in another tab while writing this up, not even looking for anything, and just noticed 48 and 84 sitting in there. Had this small "wait, my engine literally can't make that number" moment. Every weight's a multiple of 5. It can't output 48. Those were just numbers I'd typed in early on because they sounded roughly right, from before any of this was actually running. Nobody would have noticed if I'd just quietly loosened the engine to allow numbers like that instead.
One line
I don't think AI agents need less autonomy. I think they need the same kind of accountability every other system had to earn before anyone trusted it with production traffic.
Veritas is open source: https://github.com/ishantgupta30/veritas-flight-recorder
Top comments (0)