Every AI agent shouldn't be a black box
While building Jargons, our AI-powered GitHub code review agent, we noticed something strange. We could see the comments it posted on pull requests, but We couldn't answer simple questions about what happened behind the scenes.
- How long did a review actually take?
- How many tokens did it consume?
- How much did each review cost?
- Which step was slowing everything down?
- Did the agent quietly fail halfway through?
For software that runs automatically on every pull request, that's a surprising amount of trust to place in something you can't actually observe. So instead of treating observability as an afterthought, We decided to build it into the agent from day one. That decision ended up uncovering bugs We probably never would have found by reading the code alone.
Meet Jargons
Jargons is an AI-powered GitHub code review and security scanning agent. Given a pull request or an entire repository, it can:
- Review code changes
- Detect bugs and security issues
- Highlight architectural problems
- Suggest improvements
- Automatically open a companion pull request containing fixes whenever possible
For the Agents of SigNoz Hackathon (Track 1), We wanted to go one step further. Instead of simply letting the agent review code, We wanted to observe everything it was doing in real time. That meant instrumenting the entire review pipeline with OpenTelemetry and sending every trace to SigNoz, allowing us to visualize every review from start to finish.
The stack looks like this:
- TanStack Start (React 19) running on Nitro
- Drizzle ORM + PostgreSQL
- GitHub App for webhook events and repository access
- Google Gemini as the LLM
- OpenTelemetry
- Self-hosted SigNoz
Getting SigNoz running
One thing We appreciated was how quick it was to get SigNoz running locally. The hackathon uses SigNoz Foundry, so the deployment is defined in a small declarative configuration file committed directly into the repository.
apiVersion: v1alpha1
metadata:
name: signoz
spec:
deployment:
mode: docker
flavor: compose
Starting everything only required:
curl -fsSL https://signoz.io/foundry.sh | bash
foundryctl cast -f casting.yaml
Foundry also generates a lock file, making the deployment reproducible for anyone cloning the project — including the hackathon judges. On our setup, running Docker Compose through Windows + WSL, everything worked smoothly.
Every review becomes a trace
Instead of viewing a code review as one giant operation, We treated every major step as its own span. A typical review looks something like this:
review.run
├── github.fetch_diff
├── llm.review
├── db.write_findings
├── github.open_fix_pr
│ └── llm.autofix
└── github.post_review
That means every pull request becomes one distributed trace inside SigNoz. Instead of wondering where time was spent, We could immediately see which step was responsible. The LLM spans also follow OpenTelemetry's GenAI semantic conventions, allowing SigNoz to automatically surface information such as model usage, token consumption, and estimated cost. Rather than manually piecing together logs, We could simply open a trace and understand the entire lifecycle of a review.
Building an observability dashboard
Tracing was only part of the story. We also wanted a high-level view of how the agent was behaving over time. Using custom OpenTelemetry metrics, Jargons records information such as:
- Reviews processed
- Review duration
- LLM token usage
- Estimated model cost
Those metrics power a dedicated dashboard showing the overall health of the agent. Instead of watching logs scroll by, We could immediately answer questions like:
- Is the agent healthy?
- Are reviews getting slower?
- Is token usage increasing?
- Is one workspace consuming significantly more resources?
Bringing observability into the product
One feature we're particularly happy with is that observability isn't limited to the SigNoz dashboard. Jargons includes its own Agent Health page. Some information — such as review counts and historical statistics — comes directly from PostgreSQL because those numbers need to be exact. Live observability data, including token usage and model cost, comes directly from SigNoz through its query API.
The page also displays whether the application can currently communicate with SigNoz. If SigNoz becomes unavailable, the page degrades gracefully instead of failing entirely. That separation keeps operational monitoring independent from application data while still presenting everything in one place for users.
Observability without exposing user code
One design decision we were careful about involved privacy. The spans only contain operational metadata such as:
- Repository name
- Pull request number
- Model used
- Token counts
- Latency
- Cost
- Number of findings
They never include source code, pull request diffs, or generated review content. Observability shouldn't become another way to leak sensitive application data.
The bugs tracing helped us find
This ended up being our favorite part of the project.
The first issue involved throughput metrics. Initially, the dashboard appeared broken because throughput stayed at zero immediately after startup. The traces clearly showed reviews completing successfully. The problem wasn't the agent — it was the metric. Since cumulative counters need multiple samples before functions like increase() become meaningful, a brand-new workspace naturally reported zero throughput. The fix was straightforward: use PostgreSQL for exact counts while letting SigNoz focus on telemetry such as latency, token usage, and cost.
The second bug was even more interesting. After watching the traces for a while, we noticed something unusual. The same repository kept producing nearly identical review.run traces. Looking closer, we realized Jargons had started reviewing the pull requests it created itself. Nothing crashed. No exceptions were thrown. The agent had quietly entered a review loop. A single guard preventing reviews on jargons/fix-* branches solved the problem immediately. Without distributed tracing, that bug could have taken much longer to discover.
What we learned
If we could give our past selves one piece of advice, it would be this: instrument your AI agent before you think you need to.
Observability isn't just about debugging production issues. It changes how you build. Instead of guessing where time goes, you can measure it. Instead of wondering why costs increase, you can see exactly when they happen. Instead of assuming your workflows behave correctly, you can watch every step execute. That visibility changes the way you design software.
Final thoughts
Building Jargons was an exciting engineering challenge. Making it observable made it trustworthy. Today, every review generates a trace. Every AI call has measurable latency. Every token consumed can be monitored. Every workflow can be inspected. And the health of the agent lives on a dashboard instead of disappearing into log files.
For us, that's what observability adds to AI systems: confidence.
Built with AI assistance (Anthropic's Claude); all architecture, integration, and testing decisions were directed and reviewed by our team, in accordance with the hackathon rules.
Repository: https://github.com/devtofunmi/jargons
Live Demo: https://jargons.vercel.app
Top comments (0)