One Trace ID, Four Spans: Observing an Autonomous Trading Agent with SigNoz

#signoz #ai #agents #observability

Cognivern is an autonomous agent that runs sealed-bid auctions on Canton DevNet for enterprise RFPs. Every cycle it reads an open request for proposal, calls an LLM, picks a bid, signs it, and submits it to a Daml contract — no human in the loop. Building the bidding logic was, honestly, the easy part. The hard part was answering a much dumber-sounding question afterward: what did the agent actually just do, what did that cycle cost, and did the policy engine let it happen for the right reason?

Before I wired up SigNoz, "debugging" the agent meant grepping through console output across four separate services and hoping the timestamps lined up. That's not observability, that's archaeology. This post is about what changed once I put OpenTelemetry and SigNoz in front of every decision the agent makes.

Why an RFP-bidding agent needs more than logs

Cognivern's decision loop isn't one function call — it's a chain: the agent reads an open RFP and forecasts a bid, an LLM router picks a model and executes with fallback, a governance service evaluates whether the action is allowed, and an audit service writes the record. Each of those lives in a different part of the stack. When something went wrong — a denied bid, a slow LLM call, an unexpected cost spike — I had no single place to look. I had four logs and a headache.

What I actually needed was one question answered instantly: for this one decision, show me everything that happened, in order, with timing and cost attached. That's a tracing problem, not a logging problem, and it's exactly what OpenTelemetry and SigNoz are built for.

Wiring up OpenTelemetry once, everywhere

The instrumentation lives in a single otel.ts file that every service imports. It sets up a NodeTracerProvider, configures an OTLP gRPC exporter pointed at self-hosted SigNoz, and attaches resource attributes so spans from different services are still recognizable as belonging to the same agent run. Because the agent runtime, the LLM router, the policy enforcement service, and the audit log service all share this one setup, they also share a traceId. That single shared ID is what turns four separate services into one readable story in SigNoz.

The span tree for a single decision cycle now looks like this:

agent.sapience.forecast_cycle        842ms
 └─ llm.execute_with_fallback        388ms
     └─ governance.evaluate_decision  112ms
         └─ audit.log_action          44ms

One trace ID. Four spans. The full path from "the agent decided to forecast a bid" to "the audit log has a permanent record of it" is now something I can click into, instead of something I have to reconstruct from memory and grep.

Turning traces into dashboards a human actually watches

Traces are great when you already know which decision to investigate. But most of the time you don't — you want to glance at a screen and know if anything's wrong. So alongside the tracing, I emit OTel metrics for token spend, allowed/denied action counts, and provider latency, and built three native SigNoz dashboards on top of them:

Governance Overview — how many actions were allowed vs. denied, and why
Token Cost — spend per cycle, so a runaway LLM loop shows up immediately instead of on next month's bill
Provider Health — latency and fallback behavior for the LLM provider chain

All three are embedded directly in a /observability page in the Cognivern web app, so an operator doesn't need a separate SigNoz login to see the agent's health — it's just part of the product. And because SigNoz supports deep-linking, clicking a governance event in the dashboard drops you straight into the exact trace that produced it, with the full LLM context, the policy reasoning, and the signed payload all sitting right there.

That last part mattered more than I expected. The audit service stores the traceId alongside every logged record, which means an audit entry isn't just "action X was allowed at time Y" — it's a permanent pointer back into the exact trace that explains why. For a system making autonomous, signed, on-chain decisions, being able to answer "why did it do that" after the fact isn't a nice-to-have, it's the whole point.

What's actually on the ledger right now

None of this is a simulation running against fake data. Checking the live VPS as I write this, there are three RFP rounds sitting on Canton in different states:

A security audit RFP — still open, two bids in so far
A legal-counsel retainer for 2026 — bids revealed, won by bob-cognivern::122003aa7c491e00a453145c4d2cd3dbf5db8908b4e663c9944baed57fd66effa668 at a winning bid of 185,000
A cloud-infrastructure 3-year commit — open, one bid in

All three carry backend: canton — real sealed-bid rounds on Daml, not a mocked-up demo screen. This is exactly the kind of state the Governance Overview dashboard is built to show at a glance: which rounds are open, which are revealed, which are settled — without me having to query the Canton ledger by hand to find out. The dashboard isn't decoration on top of the ledger; it's the difference between "I think three rounds are active" and knowing it.

What I got wrong the first time

My first instinct was to treat tracing as something you bolt on when things break. That's backwards for an agent system — by the time something breaks, the interesting context (which model responded, what the policy engine actually evaluated, what the bid amount was) has usually scrolled off a terminal somewhere. Instrumenting every span up front, including the "boring" ones like audit.log_action, is what made post-hoc debugging possible at all. If I'd only traced the LLM call, I'd have cost visibility but no governance story.

The other thing I underestimated: how much a shared traceId costs you when you don't set it up from day one. Retrofitting one config file across four services was simple. Retrofitting it after those services had already diverged in how they log would not have been.

Takeaways

A single shared trace ID across services is the highest-leverage thing you can add early — it's what turns four separate logs into one story.
Metrics dashboards answer "is anything wrong right now," traces answer "why did this specific thing happen." You want both, and you want them linked.
For an autonomous agent making real decisions, the audit trail is only as good as its ability to point back to the exact trace behind it. A record that just says "approved" isn't an audit trail, it's a rubber stamp.
Self-hosting SigNoz alongside the agent stack meant none of this telemetry — including LLM prompts and signed payload context — left infrastructure I control.

Wrap-up

The bidding logic got Cognivern into the auction. OpenTelemetry and SigNoz are what let me actually trust what it's doing once it's in there, instead of taking the agent's word for it. If you're building anything that makes autonomous decisions with real consequences — spending, bidding, signing — instrument it before you need to, not after. The cost of adding it later is a rewrite; the cost of adding it early is one config file.