DEV Community

Akash Goyal
Akash Goyal

Posted on

Sentinel MCP Signoz (Agents of Signoz)

I participated in 'Agents of Signoz' hacakthon with Ashish Agrawal.
We built an impactful project using the Signoz ecosystem.

Signoz Alert based - Agentic system - with Unified Observability Dashboard

Project features :

  1. Agentic application with SRE Agent, Data Quality Agent.
  2. SREAgent would trigger on Alert (from log/trace/metric) via webhook.
  3. Data Quality Agent would run at certain durations once the Data ingestion is complete.
  4. Each Agents make use MCP tools from signoz-mcp-server & tools
  5. Signoz Dashboard customized for unified experience - for Agent, App, mcp-tools & APM metrics - using Query Builder

Hackathon : https://www.wemakedevs.org/hackathons/signoz
Github : https://github.com/akashgoyal/sentinel-mcp-signoz
Hackathon Project PPT : LINK
Youtube Video : Demo Link

Tech Stack

This blog will hand hold you through the project - in understanding it's whereabout, reasoning behind features, and finally practicals.
Kepe reading to know below :

  1. About the Project
  2. Tech Stack & architecture
  3. How to setup on your local & run demo
  4. Learning & Growth

Create User application - which will use OTel

  1. Langgraph based application with [Ingestion → Analyser → Extractor → Refiner] Nodes. All using qwen2.5-coder:1.5b model from ollama.
  2. Application Flow chart :

sentinel-mcp-signoz project overview

MCP agents in project

Create SRE Agent & Data Quality Agent - using signoz-mcp-server

SRE Agent :

SRE Agent (powered by gemini-3.6-flash) takes over without human intervention. Features :

  1. Performs RCA on application & setup errors
  2. Triggered on Signoz-Alert for RCA.
  3. Analyzes the application traces, logs, metrics
  4. Identify the Root Cause of the problem

Design Diagram :
SRE Agent

Data Quality Agent :

Data Quality Agent (powered by openai-4o) takes over without human intervention. Features :

  1. Triggered on Signoz-Alert when ingestion process completes.
  2. Analyzes the application traces and logs - from Signoz Storage.
  3. Generates an analysis focused on Compliance Governance.

Design Diagram :
Data Quality Agent

Setup Signoz Dashboard :

Unified Observability Dashboard

Single monitoring panel for all project components. Features :

  1. Metrics related to >10 signoz-mcp-server tool calls .
  2. Langgraph based user app metrics (custom metrics)
  3. SRE Agent related metrics
  4. APM Metrics

Tools Used :

  1. Default Signoz Dashboard Templates.
  2. Query Builder - Drag & Drop, query builder.

Unified Observability Dashboard

PRACTICALS

Setup on your machine locally :

Refer the github repo shared in first section of this blog.

Infra Setup

  1. Setup Signoz via foundry. Enable MCP. Follow this guide.
  2. Create user-account in Docker
  3. Create a service-account
  4. Create Signoz-API-Key in service account

User + Agents Application Setup

  1. An agentic app using instrumented packages which could generate telemetry data
  2. Enable Traces, Logs, metrics in app
  3. Exporters configured to Signoz OTel Collector
  4. Scripts to generate data - for both Success, Failure scenarios

Signoz Viewers

  1. Check Trace Viewers
  2. Check Log Viewers
  3. Check Metric Viewers
  4. Create Metric dashboard and use Signoz Query Builder to adjust metric widgets queries, with service-name, metric name or trace-field name
  5. Setup a webhook notification channel
  6. Configure Alerts (using query builder)

Execution Outputs :

  1. User Application Run
    User Application Run

  2. Errors generated in Signoz
    Errors generated in Signoz

  3. Alert triggered in Signoz
    Alert triggered in Signoz

  4. Webhook server start & request received
    Webhook server logs

  5. SRE Agent in Action - MCP tool call (fetch logs)
    SREAgent-log-fetch

  6. SRE Agent in Action - MCP tool call (fetch traces) & OUTPUT
    SREAgent-trace-fetch

  7. Unified Observability Dashboard for - APM + User App + Agent App + MCP Tools metrics
    unified-observability-dashboard

Impact of Project

  1. Built an AI-powered observability solution leveraging the Signoz ecosystem.
  2. Minutes instead of hours for initial incident triage using AI agents.
  3. Single-click investigation across Signoz Logs, Metrics, Traces, and Query Builder.
  4. Automated root cause analysis and data quality validation using Signoz MCP Tools.
  5. Showcased the potential to significantly reduce manual debugging effort.
  6. Dashboard view in one place has customer - Easy visuals for Business stakeholders

Learning & Growth

  1. Gained hands-on experience with Signoz, OpenTelemetry, and Signoz MCP.
  2. Learned to integrate Logs, Metrics, Traces, Query Builder, Dashboards, and Alerts into AI workflows.
  3. Strengthened skills in AI agents, observability, and event-driven system design.
  4. Improved rapid prototyping and solution building in a hackathon environment.
  5. MCP tools eased the Agentic Development

References

Github Repo : https://github.com/akashgoyal/sentinel-mcp-signoz

  1. User Application
  2. MCP based Agents
  3. Programmatic Webhook Server

Disclaimer :

AI-tools were used in :

  1. Writing Code
  2. Debugging issues
  3. Refining system design diagrams
  4. Getting clarity on doubtful concepts AI Tools - Gemini, CoPilot, AntiGravity

Presenting with Thanks

  1. Akash Goyal
  2. Ashish Agrawal

Top comments (0)