How AI Changed Our Hiring: An AI SRE on a Spare GPU

We were about to hire an SRE.

Instead, every alert in our Grafana now gets investigated by an AI before a human even opens it. It runs on an old graphics card that was collecting dust, costs nothing per use, and none of our data leaves our network.

This is the story of how we got there, and what I learned along the way.

First, we needed to see

Our services and infrastructure grew fast. At some point "just check the logs on the server" stopped working. There were too many services, too many machines, too many places for things to go wrong.

So I put observability first. Over time we built it up piece by piece:

• ClickStack (HyperDX on top of ClickHouse) for logs, traces and OpenTelemetry metrics
• Two Prometheus servers: a central one for our machines and infrastructure, and one inside Kubernetes
• Grafana for dashboards and alerting
• Graylog on Elasticsearch for our older logs
• all of it around Kubernetes

And it worked. For the first time, we could actually see what was happening.

Then someone had to look

Here's the part nobody tells you: good observability creates a new job.

Alerts started firing. Every single one meant the same routine. Open Grafana. Read the alert rule. Jump to the logs. Check the pods. Open another dashboard. Try to remember which Prometheus has which metrics.

None of it is hard. It's just repetitive, it takes time, and at 3 a.m. it's painful.

The obvious answer was to hire someone, an SRE whose job is to watch all of this.

I thought about it seriously. But when I looked at what that person would spend their first hour on, for every alert, it was mostly reading. Reading logs, reading metrics, reading charts, and connecting the dots.

That's exactly what AI is good at.

So I gave the first hour to an AI

I didn't want to replace people. I wanted whoever picks up an alert to start with answers instead of a blank screen.

So I built an AI "first responder". Today, when an alert fires:

1. A small service notices it within a minute.
2. It hands the alert to an AI agent, together with the alert's actual query and threshold, so the agent knows what is being measured, not just the alert's name.
3. The agent goes digging: logs, traces, metrics, and the state of our Kubernetes cluster.
4. A few minutes later, a card lands in our Teams channel: what's happening, where, and what it looked at.

I had three rules from day one:

• No cloud AI. Our logs can contain sensitive things. Nothing leaves our network.
• Don't touch production. We already run a vLLM server for other workloads, and this had to stay away from it.
• Read-only. The AI can look at everything and change nothing. Our existing alert rules and notifications work exactly as before.

Under the hood

No configs here, just the pieces, in case you want to build something similar:

• The agent: HolmesGPT. An open-source (CNCF) SRE agent from Robusta. It already knows how to query Prometheus, Grafana, Elasticsearch and Kubernetes, and it can use MCP servers for anything else. We deploy it with Helm through Argo CD.
• The model: Qwen3.5-9B, a small open model, served by llama.cpp on a GTX 1080 Ti (11 GB). The card is too old for vLLM, but llama.cpp runs happily on it and offers an OpenAI-compatible API, so HolmesGPT treats it like any other provider. It uses about 7.5 GB of VRAM, reads at about 390 tokens/s and writes at about 21 tokens/s.
• The bridge: a small Python service with no dependencies. It polls Grafana's API every minute with a read-only token, runs one investigation at a time, and waits 6 hours before investigating the same alert again. If an investigation fails, it retries. It never posts a broken result.
• A read-only proxy for ClickStack. ClickStack has its own MCP server, which lets the agent search logs and run SQL on ClickHouse. A tiny proxy in front of it only shows and allows the read-only tools.
• Teams: the results go out as Adaptive Cards through a Power Automate flow, into a private channel.

What I got wrong (and what I learned)

Bigger is not smarter

My first instinct was to use the biggest model the card could hold: a 27B model, compressed down to 2 bits to fit in 11 GB. It was a disaster.

On one alert, it went in circles for 48 tool calls, running almost the same query again and again. I capped the agent at 15 steps, but even then it was too slow: investigations took over 25 minutes and many timed out.

Then I switched to the 9B model, about a third of the size but with much lighter compression:

	27B, 2-bit	Qwen3.5-9B
Reading speed	~160 tokens/s	~390 tokens/s
Typical investigation	25+ min, often timed out	3-4 min

The answers also got better. The lesson: pick the model that fits the job, not the biggest one you can squeeze in.

Give it fewer tools, not more

ClickStack's MCP server offers 31 tools. Nine of them can change things: create or delete dashboards, alerts and webhooks.

I hid all nine behind the read-only proxy. That was the obvious safety move. The surprise was speed: the descriptions of those nine tools alone were about 45,000 tokens, sent with every single request. Removing them made everything much faster.

Two more tools took deeply nested inputs that the small model never got right. I hid those too and told the agent to just write plain SQL. A typical question now takes 3 tool calls, all successful.

Less choice made it better.

Labels matter more than clever prompts

I spent a lot of time rewriting instructions for the AI. What actually made the difference was renaming things.

The agent knows each Prometheus server only by its name and URL. With generic names, it kept looking in the wrong one. When I renamed them to say what's inside (machines and infrastructure vs. pods and workloads), it started picking the right one.

Along the way I found that our machines' metrics lived in the central Prometheus, not the Kubernetes one, because the machines already ran node_exporter on their own. Obvious to anyone on the team. Invisible to the AI until the name said so.

If a new teammate would be confused by a name, the AI will be too.

Slow and right beats fast and wrong

The model has a "thinking" mode. With it on, it's about twice as slow, but it makes far fewer broken tool calls.

I kept it on. Nobody is waiting in a chat box for an alert summary. A correct answer in 4 minutes beats a wrong one in 2.

It found a bug in our own work

When I asked "which services have the most errors?", the answer looked off. It turned out some of our services log ERROR in capitals and others log error, and our query was only counting one of them.

The AI wasn't wrong. Our query was. Making the match case-insensitive changed the ranking.

Small things that bit me

• A curl header broke every tool call. HolmesGPT (sensibly) refuses tool calls with shell special characters in them. It was also passing my HTTP headers through to the tools, and curl's default Accept: */* contains a *. So every test from my terminal was quietly refused, until I stopped passing headers through.
• The tool list alone took ~30,000 tokens, before the model read a single log line. I had to let one conversation use the full context window instead of splitting it across parallel slots.
• Two alerts at once broke it. Parallel investigations shared one memory pool and failed with "context size exceeded". Now two can run side by side and a third waits. This model needs very little memory per token, so that cost under 1 GB.
• The server was killed for using too much RAM. llama.cpp caches prompts in normal memory, up to 8 GB by default. On a busy machine, that got it OOM-killed. Limiting it to 1 GB fixed it.

What it can't do

I want to be honest here.

This is a 9B model. It still gets numbers wrong sometimes. It occasionally mixes up a server with a similarly named Kubernetes node.

So we treat every card as a first look, not a verdict. It tells you where to start. A human still decides what to do.

So, did AI change our hiring?

Yes, but not the way people fear.

We didn't replace anyone. What changed is that the most repetitive part of the job, the first hour of every alert, is now done automatically, in a few minutes. The people on our team start from a summary instead of from zero.

And the whole thing runs on an old gaming GPU, inside our own network, with zero cost per token.

If you have alerts piling up and an old GPU somewhere, you can do this too. Start small, keep it read-only, and spend more time on what the AI can see than on what you tell it.

I'd love to hear how others are handling alert fatigue. Are you using AI for it, or still doing it by hand?