DEV Community: Jesutofunmi

We Gave Our AI Code Review Agent a Dashboard — Here's What SigNoz Showed Us

Jesutofunmi — Sun, 26 Jul 2026 00:23:34 +0000

Every AI agent shouldn't be a black box

While building Jargons, our AI-powered GitHub code review agent, we noticed something strange. We could see the comments it posted on pull requests, but We couldn't answer simple questions about what happened behind the scenes.

How long did a review actually take?
How many tokens did it consume?
How much did each review cost?
Which step was slowing everything down?
Did the agent quietly fail halfway through?

For software that runs automatically on every pull request, that's a surprising amount of trust to place in something you can't actually observe. So instead of treating observability as an afterthought, We decided to build it into the agent from day one. That decision ended up uncovering bugs We probably never would have found by reading the code alone.

Meet Jargons

Jargons is an AI-powered GitHub code review and security scanning agent. Given a pull request or an entire repository, it can:

Review code changes
Detect bugs and security issues
Highlight architectural problems
Suggest improvements
Automatically open a companion pull request containing fixes whenever possible

For the Agents of SigNoz Hackathon (Track 1), We wanted to go one step further. Instead of simply letting the agent review code, We wanted to observe everything it was doing in real time. That meant instrumenting the entire review pipeline with OpenTelemetry and sending every trace to SigNoz, allowing us to visualize every review from start to finish.

The stack looks like this:

TanStack Start (React 19) running on Nitro
Drizzle ORM + PostgreSQL
GitHub App for webhook events and repository access
Google Gemini as the LLM
OpenTelemetry
Self-hosted SigNoz

Getting SigNoz running

One thing We appreciated was how quick it was to get SigNoz running locally. The hackathon uses SigNoz Foundry, so the deployment is defined in a small declarative configuration file committed directly into the repository.

apiVersion: v1alpha1
metadata:
  name: signoz
spec:
  deployment:
    mode: docker
    flavor: compose

Starting everything only required:

curl -fsSL https://signoz.io/foundry.sh | bash
foundryctl cast -f casting.yaml

Foundry also generates a lock file, making the deployment reproducible for anyone cloning the project — including the hackathon judges. On our setup, running Docker Compose through Windows + WSL, everything worked smoothly.

Every review becomes a trace

Instead of viewing a code review as one giant operation, We treated every major step as its own span. A typical review looks something like this:

review.run
├── github.fetch_diff
├── llm.review
├── db.write_findings
├── github.open_fix_pr
│   └── llm.autofix
└── github.post_review

That means every pull request becomes one distributed trace inside SigNoz. Instead of wondering where time was spent, We could immediately see which step was responsible. The LLM spans also follow OpenTelemetry's GenAI semantic conventions, allowing SigNoz to automatically surface information such as model usage, token consumption, and estimated cost. Rather than manually piecing together logs, We could simply open a trace and understand the entire lifecycle of a review.

Building an observability dashboard

Tracing was only part of the story. We also wanted a high-level view of how the agent was behaving over time. Using custom OpenTelemetry metrics, Jargons records information such as:

Reviews processed
Review duration
LLM token usage
Estimated model cost

Those metrics power a dedicated dashboard showing the overall health of the agent. Instead of watching logs scroll by, We could immediately answer questions like:

Is the agent healthy?
Are reviews getting slower?
Is token usage increasing?
Is one workspace consuming significantly more resources?

Bringing observability into the product

One feature we're particularly happy with is that observability isn't limited to the SigNoz dashboard. Jargons includes its own Agent Health page. Some information — such as review counts and historical statistics — comes directly from PostgreSQL because those numbers need to be exact. Live observability data, including token usage and model cost, comes directly from SigNoz through its query API.

The page also displays whether the application can currently communicate with SigNoz. If SigNoz becomes unavailable, the page degrades gracefully instead of failing entirely. That separation keeps operational monitoring independent from application data while still presenting everything in one place for users.

Observability without exposing user code

One design decision we were careful about involved privacy. The spans only contain operational metadata such as:

Repository name
Pull request number
Model used
Token counts
Latency
Cost
Number of findings

They never include source code, pull request diffs, or generated review content. Observability shouldn't become another way to leak sensitive application data.

The bugs tracing helped us find

This ended up being our favorite part of the project.

The first issue involved throughput metrics. Initially, the dashboard appeared broken because throughput stayed at zero immediately after startup. The traces clearly showed reviews completing successfully. The problem wasn't the agent — it was the metric. Since cumulative counters need multiple samples before functions like increase() become meaningful, a brand-new workspace naturally reported zero throughput. The fix was straightforward: use PostgreSQL for exact counts while letting SigNoz focus on telemetry such as latency, token usage, and cost.

The second bug was even more interesting. After watching the traces for a while, we noticed something unusual. The same repository kept producing nearly identical review.run traces. Looking closer, we realized Jargons had started reviewing the pull requests it created itself. Nothing crashed. No exceptions were thrown. The agent had quietly entered a review loop. A single guard preventing reviews on jargons/fix-* branches solved the problem immediately. Without distributed tracing, that bug could have taken much longer to discover.

What we learned

If we could give our past selves one piece of advice, it would be this: instrument your AI agent before you think you need to.

Observability isn't just about debugging production issues. It changes how you build. Instead of guessing where time goes, you can measure it. Instead of wondering why costs increase, you can see exactly when they happen. Instead of assuming your workflows behave correctly, you can watch every step execute. That visibility changes the way you design software.

Final thoughts

Building Jargons was an exciting engineering challenge. Making it observable made it trustworthy. Today, every review generates a trace. Every AI call has measurable latency. Every token consumed can be monitored. Every workflow can be inspected. And the health of the agent lives on a dashboard instead of disappearing into log files.

For us, that's what observability adds to AI systems: confidence.

Built with AI assistance (Anthropic's Claude); all architecture, integration, and testing decisions were directed and reviewed by our team, in accordance with the hackathon rules.

Repository: https://github.com/devtofunmi/jargons
Live Demo: https://jargons.vercel.app

The 500 That Told Me Nothing — Until I Self-Hosted SigNoz

Jesutofunmi — Thu, 16 Jul 2026 20:08:04 +0000

A few days ago, a route in one of my side projects started returning this:

{"status":500,"unhandled":true,"message":"HTTPError"}

That's it. That's the whole error. No stack trace in the response, nothing useful in the handler — my own try/catch never even fired. The browser got a 500, my code swore it did nothing wrong, and I spent longer than I'd like to admit staring at both.

I eventually found the cause (more on that later — it's a genuinely sneaky one), but the experience exposed a bigger problem: I could only see what happened inside my handler. Once the request left my code and passed back through the framework, I was blind. If an error can happen outside my own code, console.log will never be enough.

What I needed was a way to follow the entire request — through my handler, the framework, and every operation in between — and see exactly where it failed. That is what distributed tracing is built for. To try it properly, I chose SigNoz: an open-source, OpenTelemetry-native observability platform I could self-host for free. Then I built a tiny reproduction app, instrumented it, and pointed it at my local SigNoz instance.

This post is what actually happened — including the parts where things didn't work, because those turned out to be the most instructive bits.

The bug, revisited — what tracing could and couldn't see

Once the reproduction app was sending traces, I went straight back to the problem that started all of this: what was my opaque 500? In TanStack Start, this innocent-looking handler is broken:

GET: async ({ request }) => {
  setCookie('demo_session', crypto.randomUUID(), { httpOnly: true, path: '/' })
  return Response.redirect(new URL('/', new URL(request.url).origin), 302)
}

Response.redirect() returns a Response whose headers are immutable — that's the Fetch spec, not a framework quirk. The framework then tries to append the Set-Cookie header to the returned response after the handler has returned. Appending to immutable headers throws TypeError: immutable, the framework converts it to a generic 500, and your try/catch never sees a thing, because the explosion happens outside your code. (The fix: build the redirect by hand with new Response(null, { status: 302, headers: { location } }).)

Here's the part that taught me something about observability. In SigNoz, the /bug requests show up as error spans — they're why the error rate chart works. But the Exceptions tab stayed empty. Why? Because nothing ever called recordException. The framework swallowed the TypeError and returned a 500; the span was marked as an error by its status code, but no exception event was ever attached to it.

Status-code errors and recorded exceptions are different signals. If your code never sees the error, neither does your exception tracking — only the trace does.

To prove the contrast, I added /crash, a failure that is instrumented the way failures should be:

try {
  throw new TypeError("Cannot read properties of undefined (reading 'amount')")
} catch (error) {
  span.recordException(error)
  span.setStatus({ code: SpanStatusCode.ERROR, message: error.message })
  return new Response(JSON.stringify({ error: 'payment processing failed' }), { status: 500 })
}

The trace view shows it as a red, failed span:

And this one shows up in the Exceptions tab, grouped by type, with a count and timeline:

Same status code as /bug. Completely different visibility. That contrast — an invisible failure next to an instrumented one, in the same service, on the same screen — is the clearest argument for tracing I've come across.

What else SigNoz showed me

Once I understood the original bug, I explored what the same instrumentation revealed across the rest of the app. A few minutes of clicking around, and the Services page already had opinions:

P99 latency, throughput, and an error rate — computed entirely from auto-instrumented spans. I never wrote separate metrics code. The 500s from /bug were already there as a percentage on a chart, which made the service-level impact of the failure visible before I opened an individual trace.

The service detail page also has a Key Operations table that ranks every span by latency — including my hand-written query-database and load-user-session spans, sitting alongside the auto-captured HTTP and TLS spans, with P50/P95/P99 for each. Zero dashboard configuration.

Following a slow request

Opening a single /slow request shows the anatomy of the whole 3.56 seconds:

The GET request contains load-user-session (120ms), then query-database (340ms), then the TCP + TLS handshake and the outbound GitHub API call. Every "why is this page slow?" conversation I've ever had would have been shorter with this view.

A detail I didn't expect: each trace contained two GET server spans. Vite's dev mode runs application code in a worker process behind a small internal proxy — and OpenTelemetry's context propagation stitched the proxy process and the worker process into a single coherent trace, automatically, across process boundaries. That's the machinery working exactly as designed, visible on a laptop.

The feature I like most

After a few hours of poking at dashboards, alerts, logs, and explorers, the feature I keep coming back to is the trace detail view — the flamegraph plus span waterfall.

Everything else in SigNoz is aggregate: charts, percentiles, rates. All useful, all things other tools also do. But the trace view answers the question aggregates can't: what happened to this one request? It turned my invisible framework bug into a red span with a timestamp and a duration. It showed me my dev server was secretly two processes. It put my hand-written spans, the framework's HTTP handling, and a third-party API call on one timeline with no effort beyond starting the SDK.

And because SigNoz is OpenTelemetry-native, getting there required nothing proprietary — one exporter URL. If I ever want to point the same app somewhere else, it's a one-line change. That's the right kind of dependency.

What I'd tell you if you're trying this

Create the SigNoz account before debugging "broken" OTLP ports. The collector's receivers don't start until an org exists.
Filter dev-server noise with ignoreIncomingRequestHook, or your traces are unreadable.
recordException is not optional if you want the Exceptions view to mean anything. Span status alone won't do it.
Windows works fine via the Docker Compose files, even though the shiny new installer says otherwise.

The whole demo app is a handful of files — an instrumentation.mjs, four routes, and a docker-compose you didn't have to write. For a free, self-hosted stack, the distance between "nothing" and "I can see every request my app makes" turned out to be about an afternoon — most of which was me learning these pitfalls so you don't have to.

Those are the lessons I would carry into the next project. If you want to reproduce the experiment yourself, here is the exact setup that produced them.

The setup

Windows 11, Docker Desktop (WSL2 backend)
A minimal TanStack Start app (React + file-based routing with server routes), Node 24
SigNoz v0.99.0, self-hosted with Docker Compose

The demo app has four routes, each designed to produce a different kind of telemetry:

/slow — a deliberately multi-step request: two fake I/O waits wrapped in manual spans, plus a real outbound call to the GitHub API
/bug — a faithful reproduction of my opaque 500
/fixed — the same logic as /bug, done correctly
/crash — a failure that is properly instrumented

Self-hosting SigNoz (the Windows detour)

SigNoz's current docs push a new installer called Foundry — which plainly states that Windows is not officially supported. The older Docker Compose files still ship in the repo though, and the last compose-based release is the current release, so on Windows this works fine:

git clone --depth 1 --filter=blob:none --sparse https://github.com/SigNoz/signoz.git
cd signoz
git sparse-checkout set deploy
git checkout v0.99.0
cd deploy/docker
docker compose up -d

First run pulls a few gigabytes (ClickHouse, ZooKeeper, the SigNoz server, and the OTel collector), then the stack boots in order: ClickHouse → schema migrator → SigNoz server → collector. A few minutes later the UI is up at http://localhost:8080.

One honest note from the logs: the server warns loudly if SIGNOZ_JWT_SECRET isn't set. Fine for a laptop demo, but set it if you deploy this anywhere real.

Setup pitfall: the OTLP ports are dead until you create an account

This one cost me actual debugging time, so I'm writing it down for the next person.

After the stack came up, the collector logged Everything is ready. Begin running and processing data. — and yet nothing could connect to the OTLP endpoints on ports 4317/4318. Connection refused, even container-to-container. The ports were mapped, the config said 0.0.0.0:4318, the collector claimed to be alive.

The answer was in the SigNoz server's logs, not the collector's:

Failed to find or create agent ... cannot create agent without orgId

SigNoz's collector runs in managed mode: it fetches its live configuration from the SigNoz server over OpAMP, and it won't start its receivers until it registers. Registration requires an organization to exist — and the organization is only created when you set up the first admin account in the UI.

So the fix for "my OTLP ports won't open" was: open localhost:8080, create the login. Thirty seconds later the collector registered and ports 4317/4318 came alive. Every span my app had sent before that moment was silently dropped.

Instrumenting a TanStack Start app

With SigNoz running, the last step is connecting the application to it. The Node OpenTelemetry SDK gets preloaded into the dev server via NODE_OPTIONS. The whole setup is one file:

// instrumentation.mjs
import { NodeSDK } from '@opentelemetry/sdk-node'
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node'

const NOISY_PATHS = [/^\/@/, /node_modules/, /\.vite/, /favicon/, /\.css/, /\.js\b/]

const sdk = new NodeSDK({
  serviceName: 'otel-playground',
  traceExporter: new OTLPTraceExporter({ url: 'http://localhost:4318/v1/traces' }),
  instrumentations: [
    getNodeAutoInstrumentations({
      '@opentelemetry/instrumentation-fs': { enabled: false },
      '@opentelemetry/instrumentation-http': {
        ignoreIncomingRequestHook: (request) =>
          NOISY_PATHS.some((pattern) => pattern.test(request.url ?? '')),
      },
    }),
  ],
})

sdk.start()

// package.json
"dev": "cross-env NODE_OPTIONS=--import=./instrumentation.mjs vite dev --port 3001"

Two practical notes:

The ignoreIncomingRequestHook matters in dev. Vite serves a storm of module and asset requests; without the filter, your traces drown in /@vite/client noise.
Custom spans are just a few lines inside a server function:

const tracer = trace.getTracer('otel-playground')

await tracer.startActiveSpan('query-database', async (span) => {
  await simulateDbCall()
  span.end()
})

That's the entire integration. No agent daemon, no vendor SDK — the OTLP endpoint is the only SigNoz-specific thing in the app, which means none of this is lock-in.

Links

Demo code (the app, all four routes, and the instrumentation file): github.com/devtofunmi/otel-playground
SigNoz self-hosting docs: signoz.io/docs/install/self-host
OpenTelemetry JS getting started: opentelemetry.io/docs/languages/js
Written for the Agents of SigNoz hackathon early-blog challenge.

Transparency note: I used an AI assistant (Claude) to help draft and edit this post. Every install step, error message, trace, and screenshot is from my own machine — including the hours those pitfalls cost me.