DEV Community: Akash Goyal

How I Built Warden - an authorization gateway for agentic fleets on Google Cloud

Akash Goyal — Mon, 31 Aug 2026 19:54:23 +0000

I built Warden for the All Things Agentic Hackathon, in the Fortified Enterprise Fleet track. It's an authorization gateway that sits in front of a fleet of task agents and checks every tool call — search a flight, hold a booking, charge a card — before it executes, not after.

Live app: https://warden-330594494974.us-central1.run.app
Architecture walkthrough: https://akashgoyal.github.io/aiml/blog/warden-architecture.html
Source: https://github.com/akashgoyal/finops-te-agents-warden
Video: https://youtu.be/ebTWWi_bfEc

In this post

Section	Google tools covered
Why I built this	—
What I built	Gemini, Gemma, Google ADK, Firestore
Watching it run	—
From my laptop to Google Cloud	Cloud Run, Google AI Studio
Making every decision persist	Firestore
Adding Model Armor as an extra layer	Vertex AI, Model Armor
Where things actually stand	Vertex AI Agent Engine
Setting up the Google Cloud tools I used	Cloud Run, Firestore, Vertex AI, Model Armor, Vertex AI Agent Engine
Try it yourself	—

Why I built this

2026 didn't wait long to hand me real examples of agents doing things nobody had actually authorized:

Fox News: "AI agent hacks gym system to move up waitlist"

The Hacker News: "OpenAI Agent Used Exposed Credentials Across Four Services During Hugging Face Breach"

Reading both of these, I kept landing on the same thought:
Neither agent was "hacked" from outside. Each one just did something technically possible that nobody had actually signed off on. That's the gap I set out to close — for a workflow every company already has a strict policy for: Travel & Expense.

What I built

I think of it as a bouncer standing between every agent in the fleet and the tools it's asking to use. Both agents doing the actual thinking here are built on Google's Agent Development Kit (ADK), and
every attempted call goes through, in order:

A hard-limit check — no model runs for this one, on purpose. It reads the cap straight from the agent's own scope, stored as data in the same Firestore registry entry you'll see a screenshot of below. A payment over that cap stops the trip and waits for a human, full stop; some decisions shouldn't be a model's judgment call at all.
A cheap first pass, Gemma, that clears the obvious, in-scope calls for free before anything heavier gets involved.
A careful review — an ADK agent running on Gemini — for anything ambiguous or out-of-scope. It reads the fleet's actual policy document and returns a real decision, with a reason.
If a call gets blocked, a second ADK agent, also on Gemini, decides live, whether to retry the same step through the agent that's actually allowed to do it, or abort the trip.

Every decision, allowed or blocked, gets written to Firestore as a tamper-evident log before anything happens next. So the pipeline runs on three pieces of the Google stack end to end: Gemini and Gemma doing the actual judgment calls, ADK structuring the two agents that make them, and Firestore making sure none of it gets lost.

The full pipeline I built. I go into more depth on this — including two scenario walkthroughs and the Google Cloud stack behind each stage — on the architecture page.

Watching it run

The dashboard I built is a real console, not a mockup — I click a trip and watch my own fleet run:

The live app, sitting idle before I trigger anything. Four panels: what the traveler sees, the live call trace, past transactions, and every touch of Google infrastructure as it happens.

A full AUS → CHI trip I ran end to end. At step 6, hotel_agent tries to charge a card directly — outside its job. I block it. At step 7, a second agent decides to retry the same charge through payment_agent instead, and the trip finishes.

Zooming into the two moments I care about most:

Gemini's actual reasoning: hotel_agent is scoped to search and hold a room, not to charge anything.

This isn't a hand-coded fallback I wrote — it's a second model call, at runtime, deciding this specific retry is safe because payment_agent is the one agent actually authorized for it.

From my laptop to Google Cloud

While I was building this, my day-to-day iteration ran against small open models on my own laptop, through Ollama — that let me test changes instantly without a cloud project in the loop at every step. That's a development convenience, not what's actually deployed, and I want to be upfront about the distinction.

What's actually live right now is a different, real thing: Cloud Run running the same code, calling Gemini through Google AI Studio, backed by Firestore. No stub mode, no mocked responses — I wanted anyone checking this out to be looking at the real thing.

I pulled this straight off the deployed revision — MODEL_BACKEND=gemini, 1 CPU, 1GiB, port 8080.

Container instance count returns to zero between runs — real serverless scaling on Cloud Run, not something I'm just claiming.

Real request logs from a trip I actually ran — not a local server standing in for the deployment.

Making every decision persist

I wanted every agent's declared scope, and every decision Warden makes, written somewhere durable — so restarting the service, or Cloud Run scaling down, never loses the record. I used Firestore for that.

The registry entry that actually made the block possible — hotel_agent's allowed tools, stored as data I wrote once, not hardcoded logic buried in the reviewer.

One trip I ran, fully persisted — every step, its decision, the reviewing model, and the signed token, exactly as the dashboard read them back.

Adding Model Armor as an extra layer

Model Armor is Google Cloud's own screening layer — it inspects prompts and responses for injection attempts, jailbreaks, and sensitive data before and after they reach a model. I wired it into my reviewer's calls, and I wanted to actually see it work, not just configure it and move on. These next three are from a run I did locally against the Vertex AI backend — my live deployment above intentionally runs a different path, which I explain right after:

The warden-prompt-response badge here is read live off the actual review result I got back — not a label I added for the screenshot.

Same screening showed up on the recovery decision right after.

I confirmed it in Cloud Logging too — a real call to gemini-2.5-flash through Vertex AI, not something simulated.

So why isn't this my deployed default? Model Armor only attaches when calls go through Vertex AI directly — and this project's Vertex AI model access currently tops out at Gemini 2.5 Flash, not the newer Gemini generation I wanted running as my main deployment. So I kept Model Armor as a working, opt-in layer I can turn on and demonstrate, rather than swapping it into the always-on default.

Where things actually stand

Here's where things actually stand, plainly — I'd rather be upfront about it than oversell it:

The audit trail, agent registry, and transaction history are real Firestore data I wrote to — not held in memory, not reset on restart.
The reviewer and the recovery decision are real Google ADK agents I built, reasoning over an actual policy document — not if/else logic dressed up as AI.
What's live is genuinely running on Cloud Run, on Gemini — nothing shown above is stubbed.
Model Armor screening is something I demonstrated and verified working, and I've kept it additive rather than folding it into the main deployment, for the reason above.
I also gave my reviewer a real per-agent cryptographic identity — and it came with persistent memory as a bonus — on a separate Vertex AI Agent Engine deployment. That's a different, more involved setup, so I kept it additive rather than wiring it into the main live path.
One gap I'll be upfront about: full trace-level observability, the kind OpenTelemetry tooling expects, is something I've partially confirmed, not fully — I'd rather say that plainly than gloss over it.

Setting up the Google Cloud tools I used

Here's what I actually configured, tool by tool, in case you want to run this yourself. My repo's README has the exact commands — this is the map of what each piece is for.

Cloud Run — hosts the gateway itself.

I enabled the Cloud Run API from the API Library.
I deployed with --min-instances=0, so it scales to zero when idle and spins back up on the next request.
The model backend, project ID, and secret key are set as deploy-time environment variables, not baked into the image — you can see them for yourself on the Cloud Run console.

Firestore — the agent registry and the audit/transaction ledger.

I created a Native mode database from the Firestore console — Datastore mode won't work here.
I ran a seed script once to write each agent's declared scope in as data — nothing about scopes is hardcoded in my reviewer's code.
I leave the project ID blank in local dev, and everything falls back to in-memory automatically.

Vertex AI + Model Armor — the opt-in screening layer.

I enabled the Vertex AI and Model Armor APIs from the API Library.
I created a Model Armor template — prompt-injection/jailbreak filters, sensitive-data checks — with one gcloud model-armor templates create call, done once.
I pointed my reviewer at Vertex AI instead of the Gemini Developer API, and passed that template's name in on each call — that's the entire integration surface.
Worth checking your own project's Vertex AI model catalog before assuming it matches Google AI Studio's — mine didn't, which is exactly why I kept this path opt-in.

Vertex AI Agent Engine (optional) — a real per-agent identity.

From the Vertex AI console, or through the SDK, I deployed the same reviewer agent as a managed Agent Engine resource.
Google provisioned a dedicated IAM service account for that specific deployed agent automatically — I didn't configure that by hand.
It also came with a persistent Memory Bank for cross-session context, with no extra setup from me.
Unlike Cloud Run, this one stays provisioned rather than scaling to zero — I clean it up once I'm done demonstrating it.

Try it yourself

Live app: https://warden-330594494974.us-central1.run.app
Architecture walkthrough: https://akashgoyal.github.io/aiml/blog/warden-architecture.html
Source: https://github.com/akashgoyal/finops-te-agents-warden

I Gave My AI Coding Agent Amnesia, Then Watched Sentry Diagnose It

Akash Goyal — Sun, 23 Aug 2026 18:24:59 +0000

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Project Overview

I built sentry-agent-harness that instruments the cline AI coding agents with Sentry AI Monitoring.
I took a fork of Cline and instrumented it entirely through its own public hook system (no patches to Cline's internals).
It's delievered with the four toggleable agent failure modes in the github repo :

silent tool-output truncation
an infinite tool-call retry loop
an MCP schema mismatch, and
unbounded context growth

To actually exercise it, I didn't hand Cline a synthetic prompt — I gave it a real support ticket:

A customer (grace@example.com) says her July invoice — she thinks the ID is INV-1034 — is higher than expected. Look into the database and the billing app, figure out what's wrong, and explain the root cause.

Behind that ticket is a tiny billing service (a users/orders/invoices SQLite database over MCP, a billing_engine.py with the real calculation logic, a PRICING.md with the documented pricing contract, and an audit.log of deploy events) carrying a genuine, seeded bug: a deploy on 2026-07-01 silently switched whether tax is computed before or after a discount, so invoice INV-1043 came out $195.00 instead of the correct $192.50. The ticket's invoice ID even has a typo, so step one is already a dead end.

Everything ran against llama3.1:8b through Ollama — entirely local, no API key, no hosted inference bill. That matters twice over: anyone can reproduce this for free, and a small local model is noisier than a frontier hosted one, which is honestly closer to real production agent traffic than a cherry-picked demo would be — exactly the kind of variance you need observability for, not the case where everything already works.

Worth separating out clearly from the four scenarios below: some of that noise was just the model being an 8B model, nothing to do with any bug I injected. It occasionally hallucinated a plausible-looking tool call as plain text instead of issuing a real one, and routinely sent integer arguments as quoted strings ("limit": "5" instead of 5) — usually harmless, since the MCP server coerced it, but a good reminder that "the agent looks like it's working" and "the agent is reliably doing what you asked" aren't the same claim. None of the four failures in this post are that — each one below reproduces identically regardless of model size, because each one is an infrastructure/hook-level bug, not a reasoning failure. A bigger model wouldn't have avoided any of them: silent truncation cuts data before it reaches the model, and the other three are hook-level overrides that happen regardless of what the model does.

Bug Fix or Performance Improvement

There were two layers of bugs here, and honestly the second one ended up being the more interesting story.

Layer 1 — the bug I seeded on purpose. The billing discrepancy above, investigated live by Cline while I watched Sentry. Four distinct failure modes surfaced during that investigation (full detail in "My Improvements" below): a silently truncated log file that made the agent confidently cite the wrong deploy, a retry loop that burned 3x the tokens on a typo'd invoice ID, and a schema-mismatch that produced a real, cascading ValidationError.

Layer 2 — the bugs I found by accident. Building this wasn't "write the instrumentation once, watch it work." Actually running it against a live Sentry project surfaced five real bugs in my own instrumentation code, none of which tsc or a unit test would have caught, all of which I only noticed because I went looking in the Sentry UI and found something missing or wrong:

Events dropped at process exit. My first version called Sentry.flush() fire-and-forget. Cline's CLI calls process.exit() moments after a run finishes, which routinely won the race against the network flush — exceptions were being generated and silently never delivered.
The error flag that's never true. FastMCP catches its own schema validation errors and returns them as normal-looking text content, not a protocol-level error — so gating my capture logic on result.isError meant it never fired for the exact scenario it existed to catch.
Seer caught a bug in my bug report. My exception message did String(result.output) on a response object, coercing it to the literal text "[object Object]". I asked Seer for the root cause of my own captured exception, and it correctly diagnosed my bug instead of the scenario I'd built — exact line number, reproduction steps, all of it.
The fix that silently didn't apply. My first version of silent truncation returned a modified tool result from an afterTool hook. The hook fired, computed the right truncated value, returned it — and the next model call still saw the full, untruncated content anyway.
Breadcrumbs need a body to travel in. Sentry breadcrumbs only ship attached to an event. Three of my four scenarios are deliberately silent (no exception), so their breadcrumbs were being recorded locally and then never sent anywhere — invisible until I noticed nothing showed up where I expected an issue.

All five are fixed in the code linked below.

Code

Git Commit for cline instrumentation : https://github.com/akashgoyal/sentry-agent-harness/commit/68d590af3ac41fab1bd4a57a05ebf6fa5aadb7ec

All four scenarios are just environment variables, off by default — clone the repo and flip whichever one you want to see for yourself, no code changes needed:

CLINE_SENTRY_SILENT_TRUNCATION=1     # cuts tool output over 2,000 chars before the model sees it
CLINE_SENTRY_INFINITE_TOOL_LOOP=1    # forces a retry instead of stopping on an empty result
CLINE_SENTRY_MCP_SCHEMA_MISMATCH=1   # corrupts an outgoing MCP argument's type
CLINE_SENTRY_CONTEXT_INFLATION=1     # skips message-history trimming

My Improvements

No patches to Cline's internals. AgentRuntime — the actual agent loop — already exposes a public hook system (beforeModel, beforeTool, afterTool, onEvent) plus a subscribable event stream. The whole integration is one new module split into two files by responsibility: a telemetry.ts that turns Cline's own events into a Sentry span tree and knows nothing about the scenarios below, and a scenarios.ts that injects the four failure modes purely through the same public hooks. Three call sites, merged in via mergeAgentHooks.

Failure #1 — the confidently wrong agent. The audit.log file is 4,258 characters. Buried at character 2,311 — well past a naive truncation point — is the one line that actually explains the bug: a billing-engine version=2.0.0 deploy whose changelog admits the tax-calculation change outright. I added a middleware bug on purpose: silently cut any tool output over 2,000 characters before it reaches the model, no exception raised. With it active, the agent confidently blamed a real but unrelated api-gateway release from three weeks earlier — sha and all — because that's what was still inside its 2,000-char window. Nothing crashed. A green checkmark looks identical whether the agent got lucky or got it right.

Failure #2 — the loop that won't take no for an answer. Normally an empty query result is where the agent should say "I don't have that" and stop. I added a hook that, on an empty result, injects a "try a different query" nudge instead of letting the run end. Baseline: 2 iterations. With the nudge active, runs stretched to 3–6 iterations depending on how many reformulations the model attempted, and prompt token usage peaked at 116,901 in one run — over 3x the baseline's ~36,000 for the exact same question.

Failure #3 — the one with a real stack trace. I corrupted an outgoing MCP tool argument's type right before it left the client — a limit parameter that should be an integer went out as the string "not-a-valid-type". What happened next wasn't scripted: the tool rejected it with a genuine schema ValidationError from the server's own pydantic validation. The agent retried the exact same broken call — nothing about the corruption changes between attempts — until Cline's own "max consecutive mistakes" safety net aborted the run. A cascading failure, not a single bad call.

Why hooks instead of a fork-and-edit. Every failure mode above is injected through beforeModel/beforeTool/onEvent — the same seams Cline itself uses for its own hook-event dispatch, tool approval, and context compaction. That meant zero merge conflicts with Cline's own logic (confirmed by testing that its native hook-event output kept working unchanged throughout), and it's the reason the one genuinely subtle bug (#4 in the list above) was fixable by moving logic between two equally legitimate hooks rather than needing a deeper architectural change.

Best Use of Sentry

This project leans on several distinct Sentry AI Monitoring capabilities, each mapped to a specific problem in agent debugging that a plain log file handles badly:

Distributed Tracing / Agent Tracing — AgentRuntime's event stream (run-started, turn-started, tool-started, ...) is turned into a real nested span tree: agent.run → agent.turn <n> → agent.llm_call / tool.<name>, with prompt/completion token counts attached to each LLM span via gen_ai.usage.* attributes. Because the span op values follow Sentry's gen_ai.* semantic conventions (gen_ai.invoke_agent, gen_ai.chat, gen_ai.execute_tool), Sentry's own UI recognized this as agent monitoring automatically — a dedicated Agent Activity view came for free, not something I had to build a custom dashboard to get.
Error Monitoring — the schema-mismatch scenario produces a real Sentry.captureException with full breadcrumb context (which argument got corrupted, on which tool, right before the server's validation error). The two other "silent" scenarios (truncation, infinite loop) don't throw by design, so I added explicit Sentry.captureMessage(..., "warning") calls at the exact moment each one fires — turning invisible failures into findable issues, with the issue's event count doubling as "how many times did this actually happen in one run."
Seer — asked to explain the schema-mismatch issue, and its root cause analysis was accurate down to the specific line and the specific reason (String() coercing an object to "[object Object]") — which is how I found and fixed bug #3 in the list above. Seer diagnosed a real bug in my own code, not the synthetic scenario I thought I was showing it.
Breadcrumbs — every AgentRuntimeEvent (run/turn/tool lifecycle, model requests) leaves a breadcrumb, plus a scenario-specific one at each failure point (scenario.silent_truncation, scenario.infinite_tool_loop, scenario.mcp_schema_mismatch). These are what actually let me (and Seer) reconstruct why a run went wrong, not just that it did.

Seer's RCA on schema-mismatch issue :

Seer's RCA for Infinite tool calls issue :

Why I Built PaperFlakes (And How Zerops Saved the Stack)

Akash Goyal — Sun, 09 Aug 2026 18:22:38 +0000

Late one night while preparing slides for an upcoming talk, I ran three dense research papers through a standard OCR tool.

The OCR worked, but left me with 40 pages of raw, unformatted text. The real problem wasn't extracting the text; it was finding the insight buried inside it.

I wanted something that could turn dense PDFs into small, visual insights - bite-sized cards ready for slides, developer notes, or Twitter.

That was the spark for PaperFlakes.

PaperFlakes takes PDFs - pick from a curated list of classic ML papers or upload your own - OCRs them, and turns the extracted text into shareable "sticky note" insight cards across four categories, which you can filter, download as images, or share straight to X /Twitter.

Features for Users

I set out to build a platform that turns dense academic reading into a fast, visual experience:

Batch Processing: Drop up to 25 multi-page PDFs.
Real-Time Progress: Watch live updates as the engine parses document queues, tracking statuses, page counts, and extracted word tallies.
"Did You Know?" Fact Cards: Color-coded visual cards highlighting key facts, formulas, and takeaways.
Instant Export & Sharing: Download cards or share them straight to social media.

How to use it

Open the live app.
Check one or more papers under "Try it on papers" and click Process batch.
Watch the Processing status table update as each document is OCR'd.
Insight cards appear in the Insights panel as soon as each category finishes — no need to wait for the whole batch.
Use the category chips and paper dropdown above the panel to filter what's shown.
On any card: Save downloads it as a PNG
On any card: 𝕏 Share opens a pre-filled tweet.
Download all (top right) exports every currently-visible card as one combined image.

The UI view:

The social media share view: Link to generated tweet

Product Walkthrough :

The application follows a simple three-stage flow:

Upload & Queue (left) - Users select multiple PDFs or upload their own documents and start a batch.
Process & Track (center) - Each document is processed asynchronously, with its status, page count, and processing details visible in real time.
Generate Insights (right) - As soon as a document finishes processing, AI-generated facts and insights appear as shareable sticky-note cards.

The Engineering & Zerops ZCP

Behind the UI, the backend handles the pipeline:
PDF ingestion → metadata storage → chunking & OCR → AI-powered insight extraction → result storage → fact-card generation.

This keeps the user experience simple while the heavy document processing happens asynchronously in the backend.

The zerops.yaml is shared here .

When a user submits a batch, the Express API registers the job inside PostgreSQL with a queued status and returns an instant job token to the React UI. As workers process pages through Gemma (Together AI) asynchronously, the database updates in real time, and the UI streams progress smoothly without dropping connections.

This whole project is heavily reliant on Zerops infra & built using the ZCP.

Runtime service
Managed PostgreSQL
Public HTTPS — zerops.app subdomain after deployment.
Secrets — TOGETHER_API_KEY is stored as a Zerops service secret.
Health checks
Git-based deployment — deploys the latest commit to Zerops using zcli.
ZCP with Claude Code

Running your own copy

The repository is self-contained: zerops.yaml defines the deployment configuration, while migrate.py initializes the database schema idempotently on each deploy.

Create a Zerops project with a Python runtime service and a PostgreSQL service named db.
Create a Together AI API key and add it as the TOGETHER_API_KEY secret on the runtime service.
Configure ZEROPS_TOKEN and ZEROPS_SERVICE_ID as GitHub repository secrets.
Push to main. The included GitHub Actions workflow builds and deploys the application automatically.

You can also deploy manually using zcli push or the Zerops MCP tools.

You can see the zerops-services in action & associated cost with them (left-bottom)

By offloading setup headaches to Zerops, I was able to go from an initial draft idea to a production-ready, multi-service application in a single hackathon build window.

Demo Video : https://youtu.be/5ra5lgwRpeQ
Github : https://github.com/akashgoyal/paperflakes-zerops
Live App : https://app-2c11-8000.prg1.zerops.app/
Built For : https://www.wemakedevs.org/hackathons/zerops

Written by : Akash Goyal

Sentinel MCP Signoz (Agents of Signoz)

Akash Goyal — Sun, 26 Jul 2026 17:40:36 +0000

I participated in 'Agents of Signoz' hacakthon with Ashish Agrawal.
We built an impactful project using the Signoz ecosystem.

Signoz Alert based - Agentic system - with Unified Observability Dashboard

Project features :

Agentic application with SRE Agent, Data Quality Agent.
SREAgent would trigger on Alert (from log/trace/metric) via webhook.
Data Quality Agent would run at certain durations once the Data ingestion is complete.
Each Agents make use MCP tools from signoz-mcp-server & tools
Signoz Dashboard customized for unified experience - for Agent, App, mcp-tools & APM metrics - using Query Builder

Hackathon : https://www.wemakedevs.org/hackathons/signoz
Github : https://github.com/akashgoyal/sentinel-mcp-signoz
Hackathon Project PPT : LINK
Youtube Video : Demo Link

This blog will hand hold you through the project - in understanding it's whereabout, reasoning behind features, and finally practicals.
Kepe reading to know below :

About the Project
Tech Stack & architecture
How to setup on your local & run demo
Learning & Growth

Create User application - which will use OTel

Langgraph based application with [Ingestion → Analyser → Extractor → Refiner] Nodes. All using qwen2.5-coder:1.5b model from ollama.
Application Flow chart :

Create SRE Agent & Data Quality Agent - using signoz-mcp-server

SRE Agent :

SRE Agent (powered by gemini-3.6-flash) takes over without human intervention. Features :

Performs RCA on application & setup errors
Triggered on Signoz-Alert for RCA.
Analyzes the application traces, logs, metrics
Identify the Root Cause of the problem

Design Diagram :

Data Quality Agent :

Data Quality Agent (powered by openai-4o) takes over without human intervention. Features :

Triggered on Signoz-Alert when ingestion process completes.
Analyzes the application traces and logs - from Signoz Storage.
Generates an analysis focused on Compliance Governance.

Design Diagram :

Setup Signoz Dashboard :

Unified Observability Dashboard

Single monitoring panel for all project components. Features :

Metrics related to >10 signoz-mcp-server tool calls .
Langgraph based user app metrics (custom metrics)
SRE Agent related metrics
APM Metrics

Tools Used :

Default Signoz Dashboard Templates.
Query Builder - Drag & Drop, query builder.

PRACTICALS

Setup on your machine locally :

Refer the github repo shared in first section of this blog.

Infra Setup

Setup Signoz via foundry. Enable MCP. Follow this guide.
Create user-account in Docker
Create a service-account
Create Signoz-API-Key in service account

User + Agents Application Setup

An agentic app using instrumented packages which could generate telemetry data
Enable Traces, Logs, metrics in app
Exporters configured to Signoz OTel Collector
Scripts to generate data - for both Success, Failure scenarios

Signoz Viewers

Check Trace Viewers
Check Log Viewers
Check Metric Viewers
Create Metric dashboard and use Signoz Query Builder to adjust metric widgets queries, with service-name, metric name or trace-field name
Setup a webhook notification channel
Configure Alerts (using query builder)

Execution Outputs :

User Application Run
Errors generated in Signoz
Alert triggered in Signoz
Webhook server start & request received
SRE Agent in Action - MCP tool call (fetch logs)
SRE Agent in Action - MCP tool call (fetch traces) & OUTPUT
Unified Observability Dashboard for - APM + User App + Agent App + MCP Tools metrics

Impact of Project

Built an AI-powered observability solution leveraging the Signoz ecosystem.
Minutes instead of hours for initial incident triage using AI agents.
Single-click investigation across Signoz Logs, Metrics, Traces, and Query Builder.
Automated root cause analysis and data quality validation using Signoz MCP Tools.
Showcased the potential to significantly reduce manual debugging effort.
Dashboard view in one place has customer - Easy visuals for Business stakeholders

Learning & Growth

Gained hands-on experience with Signoz, OpenTelemetry, and Signoz MCP.
Learned to integrate Logs, Metrics, Traces, Query Builder, Dashboards, and Alerts into AI workflows.
Strengthened skills in AI agents, observability, and event-driven system design.
Improved rapid prototyping and solution building in a hackathon environment.
MCP tools eased the Agentic Development

References

Github Repo : https://github.com/akashgoyal/sentinel-mcp-signoz

User Application
MCP based Agents
Programmatic Webhook Server

Disclaimer :

AI-tools were used in :

Writing Code
Debugging issues
Refining system design diagrams
Getting clarity on doubtful concepts AI Tools - Gemini, CoPilot, AntiGravity

Presenting with Thanks

Custom Metrics Dashboard for GenAI apps with SigNoz and OpenTelemetry

Akash Goyal — Sat, 18 Jul 2026 22:55:40 +0000

SigNoz supports lot of features and has a lot of documentations around those features. It has a full section dedicated to LLM observability across different service providers.

But, this blog is a bit different. Instead of relying on out-of-the-box vendor presets, it shows you how to create a completely custom dashboard tailored for your specific GenAI/agentic application as per your needs.

Just tracking generic infrastructure metrics like CPU and memory utilization isn't enough. To understand how your AI features are truly performing, you need specialize application-level telemetry.

All the code discussed in this post is available in the GitHub Repository.

Signoz supports both Trace Data and Metrics Data - for setting metric widget queries. This feature is really helpful in scaling the existing instrumented codebases & has uniqueness.

Part 1: Setting Up SigNoz Locally (Self-Hosted Docker)

Referred Signoz Doc Section for setting this up.

### Step 1: Install Foundry
curl -fsSL https://signoz.io/foundry.sh | bash

### Step 2: Create `casting.yaml`
vi casting.yaml  #add below
"""
apiVersion: v1alpha1
kind: Installation
metadata:
  name: signoz
spec:
  deployment:
    flavor: compose
    mode: docker
"""

### Step 3: Deploy SigNoz - spin up your local SigNoz infrastructure:
foundryctl cast -f casting.yaml

Part 2: Application Architecture & Quick Start

I designed a minimal llm-inference based workflow application, with two language models, to work on this topic.

1. Application Architecture Block

Explanation of the diagram:

Traffic Flow: The LLM Client actively invokes endpoints on the LLM Server to run inference tasks.
Telemetry Collection: Both the client and the server generate performance data during these interactions.
Data Export: The gathered telemetry is continuously pushed out via a Metrics periodic export routine utilizing OpenTelemetry standard components (OTLP Span Exporter and OTLP Metric Exporter).
Storage and Visualization: The exported data flows directly into SigNoz, where it is permanently stored in a ClickHouse DB.
Observability: Finally, developers can monitor the application using the integrated Trace Viewer and Dashboards.

2. Getting Started with App

Clone the repo from github. Then, follow these steps to initialize the environment and launch your components:

# 1. Setup local virtual environment
python -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt

# 2. Ensure metric computation logic is added.

CODE - custom metrics relevant to the application.

CODE - Setting span attrs & record metrics

# 3. Spin up the LLM Server container 
# Create your docker-compose.yaml file first, then run:
docker compose up -d llm-server

# 4. Configure local environment and run client loops sequentially
pip install httpx opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp 
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"
export OTEL_RESOURCE_ATTRIBUTES="service.name=llm-client-service"
python llm_client.py

Part 3: Visualising the Traces

A look at the traces :

Success Client requests - Trace View:
If the infra capacity is limited, after some requests, time-out errors might start surfacing. (Nothing to worry, just restart the llm-server from docker app). Related traces examples below:

Requests getting Timed Out one after other. (Server was hung)

Errored Request Trace (server was not up by that time)

Part 4: Custom Metrics Dashboard in SigNoz

Creating the Custom Dashboard - Below are the steps I followed in order:

Implemented the custom metrics relevant to the application.
Added relevant attributes in spans & record metrics.
Referred one of the template dashboards in Signoz Dashboard Page. Took the Agno Dashboard as Base to build upon.
Took the code for 'Error rate' metric widget. Created new metric widgets, in sync with the metric_record.py logic.
Modified the queries of existing widgets in Base Json.

For any further edits (metric-name, widget-type, etc.), I used the SigNoz's native Query Builder.

The dashboard JSON used for this experiment is shared in the repo file.

How it appears in SigNoz:

Signoz Default Metrics Section :

Your Custom Metrics dashboard - data rendered using both span params & metrics :

Conclusion & Impact: An insight into custom dashboards with Signoz

These are my findings on how Signoz supports custom metrics dashboards, and what changes are needed or good to have while working on such requirements.

Signoz supports both Trace Data and Metrics Data - for setting metric widget queries. This feature is really helpful in scaling the existing instrumented codebases & has uniqueness.

I hope this content would help readers enhance their knowledge.

Ready to get started? Clone the experimental code from GitHub Repository and spin it up in your local developer environment!