DEV Community

Aniket
Aniket

Posted on

My AI agent kept hallucinating metric names, so I built it a Prometheus MCP server

If you've pointed an AI agent at a live incident, you've seen the failure mode: it invents a plausible-sounding metric, queries it, gets an empty result, and confidently invents another. The Prometheus HTTP API is a bad interface for an LLM — it's powerful, but nothing helps the agent discover what's actually there before it starts guessing.

I work in production engineering, so I built the interface I wanted the agent to have: prom-mcp, an MCP (Model Context Protocol) server for Prometheus written in Go.

What it gives the agent

Three capabilities, shaped for triage rather than raw API access:

  1. PromQL queries with readable output. Instant and range queries, formatted like a triage note instead of a JSON wall.
  2. Alert explanations. When an alert is firing, the agent gets the alert joined with the rule definition that produced it — the expression, the labels, the annotations — so it can reason about the real condition.
  3. Discovery before querying. Series and label discovery tools let the agent check which metrics actually exist, which kills the hallucination loop at the source.

Plus a triage prompt that walks the agent through an incident in order: what's firing, since when, what's the blast radius.

Install

Single Go binary, zero dependencies, stdio transport:

claude mcp add prom -- prom-mcp
Enter fullscreen mode Exit fullscreen mode

Set PROM_URL if your Prometheus isn't at http://localhost:9090. No server handy? prom-mcp demo runs against fixtures so you can see the tool shapes in action. Claude Desktop users can grab the MCPB bundle from the GitHub releases, and it's listed on the official MCP Registry as io.github.aniketatgithub/prom-mcp.

Repo: https://github.com/aniketatgithub/prom-mcp

What I'd like feedback on

The tool shapes. If you run agents against real infrastructure, what questions do you wish yours could ask your metrics stack mid-incident? That list is my roadmap.

Top comments (1)

Collapse
 
arhancanli profile image
Arhan Canli •

Discovery-before-query is the right fix, and I'd test one case it can still miss: a metric that exists in the index but has stopped reporting. Series discovery over a time range will list it, an instant query returns nothing once the default lookback (5 minutes, if I remember the default right) passes, and the agent sees the same empty result it got from a hallucinated name, then guesses again.

Two cheap things for the tool output: return "exists, last sample N minutes ago" separately from "no such metric", and when a name isn't found, return the closest real names (edit distance over the metric list) so the agent corrects itself instead of inventing a second one. A test with a fixture that has one dead series and one typo'd name would show whether the loop really ends.

On your feedback question: the one I'd want mid-incident is "what changed in the last 30 minutes", i.e. series whose rate shifted most against the previous hour. Does prom-mcp demo include a fixture with a stale series?