DEV Community

Cover image for Give Your AI Agent Eyes on GPUs: Introducing gpu-mcp-server
Pavan Madduri
Pavan Madduri

Posted on

Give Your AI Agent Eyes on GPUs: Introducing gpu-mcp-server

#agents #ai #mcp #monitoring

Your AI agent can write code, search the web, and query databases. But ask it what's happening on your GPUs right now utilization, memory pressure, thermals and it draws a blank. That's because there's been no standard way for agents to access hardware telemetry.

gpu-mcp-server changes that. It's an open source MCP server that gives any AI agent real-time access to NVIDIA GPU metrics through the Model Context Protocol.

The Problem

If you're running GPU workloads inference servers, training jobs, fine-tuning pipelines you probably check GPU status the same way everyone does:

nvidia-smi
Enter fullscreen mode Exit fullscreen mode

That works for humans. It doesn't work for agents.

When an agent needs to decide whether to scale up replicas, diagnose a slow inference endpoint, or figure out why a training job OOM'd, it has no way to see the hardware layer. It's flying blind.

Monitoring stacks like Prometheus + dcgm-exporter solve this for dashboards, but they're heavy infrastructure that agents can't easily query through MCP.

What gpu-mcp-server Does

It's a single Go binary that runs alongside your agent and exposes three tools over the Model Context Protocol:

Tool What It Does
list_gpus Lists every GPU on the machine with utilization and memory
get_gpu_metrics Deep dive on one GPU temp, power, PCIe/NVLink throughput
gpu_summary Aggregate stats across all devices

The server talks to GPUs through NVIDIA's NVML C library directly - no Prometheus, no metric pipelines, no extra infrastructure.

MIG Support Built In

If you're running Multi-Instance GPU (MIG) on A100s or H100s, each MIG instance shows up as a separate device. Shared metrics (temperature, power, PCIe) are pulled from the physical GPU. Per-instance metrics (utilization, memory) come from the MIG slice.

Why MCP?

The Model Context Protocol is an open standard now hosted by the Agentic AI Foundation at the Linux Foundation for connecting AI agents to tools and data sources. Every major AI platform supports it: Claude, Goose, Cursor, Windsurf, and more.

By implementing GPU metrics as an MCP server, any of these agents can access your GPU state without custom integrations.

Setup

Build 

git clone https://github.com/pmady/gpu-mcp-server.git
cd gpu-mcp-server
make build   # requires CGO + NVIDIA drivers
Enter fullscreen mode Exit fullscreen mode

Claude Desktop

Add to claude_desktop_config.json:

{
  "mcpServers": {
    "gpu": {
      "command": "/path/to/gpu-mcp-server"
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

Goose 

extensions:
  gpu-metrics:
    type: stdio
    cmd: /path/to/gpu-mcp-server
Enter fullscreen mode Exit fullscreen mode

Once connected, you can ask your agent things like:

  • "Which GPU has the most memory available?"
  • "Are any GPUs thermal throttling right now?"
  • "What's the total power draw across all devices?"
  • "Show me the MIG instances on GPU 0"

The agent calls the right tool, gets structured JSON back, and reasons over it.

Architecture 

Agent (Claude / Goose / Cursor)
          │
          │ MCP (stdio)
          ▼
    gpu-mcp-server
          │
          │ NVML (cgo)
          ▼
    NVIDIA GPU Hardware
Enter fullscreen mode Exit fullscreen mode

No sidecar. No network hops. No metric pipeline. The server runs as a local process and calls NVML directly through cgo. If you can run nvidia-smi, you can run this.

Who Is This For?

  • ML engineers who want agents to diagnose GPU workload issues
  • Platform teams building agentic infrastructure on GPU clusters
  • Anyone running inference servers (vLLM, Triton, SGLang) who wants agents to understand hardware utilization

Relationship to keda-gpu-scaler

If you've seen keda-gpu-scaler - my KEDA external scaler for GPU-based autoscaling — this shares the same NVML foundation. Where keda-gpu-scaler uses GPU metrics to drive Kubernetes autoscaling decisions, gpu-mcp-server exposes those same metrics to AI agents through MCP.

Same GPU telemetry engine, two different consumers: orchestrators and agents.

What's Next

  • Streamable HTTP transport - run as a network service, not just stdio
  • Resource endpoints - expose GPU info as MCP resources for agent context
  • Process-level GPU usage - per-PID memory and compute attribution
  • AAIF project proposal - working toward official AAIF hosted project status

Get Involved

The project is Apache 2.0 and contributions are welcome:

If you're running GPU workloads and using MCP-compatible agents, give it a try.
Star the repo if it's useful. Open an issue if it's not.

Pavan Madduri is a platform engineer focused on GPU-aware AI infrastructure.
He maintains keda-gpu-scaler and
contributes to CNCF projects including Dragonfly, Volcano, and KEDA.

Top comments (0)