How I Designed an AI SRE Agent to Cut LLM Costs by Up to 100x — With Architecture, Not Prompts

shafivullashaik-tech-ops — Mon, 13 Jul 2026 13:14:44 +0000

The fastest way to kill an AI project isn't a bad model. It's the first cloud bill.

When I set out to build an AI SRE agent for Kubernetes — an agent that detects incidents, reads live cluster state, and drafts proposed fixes either proactively or when an engineer requests a diagnosis — the scariest number wasn't accuracy. It was this:

What happens when an LLM sits inside a loop that runs every 5 minutes, 24/7, across a whole cluster?

Do the math. In a naive architecture where every scan invokes the model, a scan every five minutes results in approximately 8,640 LLM invocations per 30-day month. Per cluster. Before a single user asks a single question. That's how a modest system quietly becomes a very expensive one — and how many AI-ops projects die in their first budget review.

This article is about how I designed the invocation architecture so the model participates only at the steps where reasoning provides meaningful value. Almost none of it involves prompts. All of it involves architecture.

Here's the shape of the difference, comparing the naive design (LLM inside the scan loop) with the final design::

Metric	Naive design	Final design
Scheduled scans / month	8,640	8,640
LLM calls from the scan loop	8,640	0
LLM calls per repeated incident	1 per occurrence	1 per fingerprint
When the model runs	every scan	once per materially distinct incident

Depending on your incident volume, that's a reduction in LLM invocations of one to two orders of magnitude — the "up to 100x" in the title comes from the invocation count; your exact dollar savings will depend on model pricing and incident rate. The point of this article is the architecture that makes the reduction structural rather than accidental.

The system in one paragraph

The agent is an AI SRE for Kubernetes, built on AWS: the agent runtime lives on Amazon Bedrock AgentCore with Claude as the reasoning model, every capability is a governed tool behind an MCP gateway (live K8s API reads, runbook retrieval from a Bedrock Knowledge Base), and detection runs on a serverless pipeline — EventBridge → Lambda → DynamoDB. The agent's runtime identity has read-only Kubernetes permissions, every tool invocation is audited, and remediation is always human-approved: the agent produces the root cause, a YAML diff, and the exact kubectl commands as text; an authenticated engineer reviews and applies them through the organization's existing access controls.

With that picture in mind, here are the five architectural decisions that shaped the cost model.

1. Rules detect, the LLM diagnoses

The single most important decision in the whole system: the model never enters the detection loop.

The 5-minute scan is pure rule-based logic. It reads live cluster state and looks for well-known failure signatures:

CrashLoopBackOff
OOMKilled
ImagePullBackOff
NodeNotReady

Known Kubernetes states like these can be detected deterministically — there's nothing for a model to interpret at the detection step. Detection is deterministic, auditable, and low-cost compared with invoking an LLM on every scan:

# detection Lambda — no LLM anywhere in this path
DETECTION_RULES = {
    "CrashLoopBackOff": lambda pod: any(
        cs.state.waiting and cs.state.waiting.reason == "CrashLoopBackOff"
        for cs in (pod.status.container_statuses or [])
    ),
    "OOMKilled": lambda pod: any(
        cs.last_state.terminated and cs.last_state.terminated.reason == "OOMKilled"
        for cs in (pod.status.container_statuses or [])
    ),
    # ... ImagePullBackOff, NodeNotReady
}

def scan(cluster_state) -> list[Incident]:
    incidents = []
    for pod in cluster_state.pods:
        for rule_name, rule in DETECTION_RULES.items():
            if rule(pod):
                incidents.append(make_incident(rule_name, pod))
    return incidents

A note on connectivity, because it's the first question any AWS/K8s architect asks: the detection Lambda authenticates to EKS through an IAM role mapped to a read-only Kubernetes identity (via EKS access entries / RBAC). For clusters with private API endpoints, the function runs in approved VPC subnets with security-group connectivity restricted to the EKS control-plane endpoint. In multi-cluster setups, each cluster gets its own scoped role — credentials are never shared across clusters.

The LLM enters only where reasoning actually adds value: diagnosis. Why is this pod OOMKilled — bad memory limit, a leak after the last deploy, a node problem? That's a reasoning task, grounded in logs, events, and runbooks. That's worth a model invocation.

The rule I'd carry into any agentic system: an LLM call inside a scheduled loop is a design smell. If a rule can handle it, a rule should handle it. Every time I was tempted to add "intelligence" somewhere, the right question was: does this step need reasoning, or just reliability?

2. One incident fingerprint = one diagnosis

Even after moving the LLM out of the loop, there's a second cost trap: the same incident firing over and over. A pod crash-looping all night can generate the same incident 50+ times. Diagnosing it 50 times is 49 wasted invocations — the answer didn't change.

The fix is incident fingerprinting with deduplication. Each incident gets a stable identity built from enough dimensions to distinguish materially different failures — not just "same workload, same reason," because the same deployment can be OOMKilled today for a completely different cause than yesterday:

def incident_fingerprint(incident) -> str:
    key = ":".join([
        incident.cluster,
        incident.namespace,
        incident.workload,
        incident.container,
        incident.reason,
        incident.deployment_revision or "unknown",
        incident.image_digest or "unknown",
        incident.error_signature or "unknown",  # normalized from logs/events
    ])
    return hashlib.sha256(key.encode()).hexdigest()[:20]

The scan writes incidents to DynamoDB with a conditional write: if the fingerprint already exists, it just bumps the occurrence count and last-seen timestamp — no new diagnosis. If the fingerprint is new or materially changed, that's when a diagnosis is triggered:

def handle_incident(incident):
    fp = incident_fingerprint(incident)
    existing = incidents_table.get_item(Key={"fingerprint": fp}).get("Item")

    if existing and is_fresh(existing):
        bump_occurrence(fp)                      # 0 LLM calls
        return

    # new or materially changed incident:
    # high-severity -> diagnose immediately (result waits for the engineer)
    # lower-severity -> diagnose on first human open
    if incident.severity == "high":
        diagnosis = invoke_agent(incident)       # 1 LLM call
        store_diagnosis(fp, diagnosis, ttl_hours=24)

And the lower-severity path, triggered when an engineer opens the incident in the UI:

def diagnose_on_open(fingerprint: str):
    incident = get_incident(fingerprint)

    if incident.get("diagnosis") and is_fresh(incident):
        return incident["diagnosis"]             # cached — 0 LLM calls

    diagnosis = invoke_agent(incident)           # first open — 1 LLM call
    store_diagnosis(fingerprint, diagnosis, ttl_hours=24)
    return diagnosis

Two details that matter here:

When the diagnosis runs. High-severity incidents trigger one diagnosis on the first occurrence of a new fingerprint — so by the time an engineer opens the incident, the root cause and proposed fix are already waiting. Lower-severity incidents are diagnosed on first human open. Either way, repeated occurrences reuse the cached result. Same crash looping 50 times with the same revision and error signature? One LLM call, not 50.

When the cache dies. Diagnoses are cached for up to 24 hours, but invalidated earlier when the workload revision, image digest, configuration, or normalized error signature changes. A fixed TTL alone would happily serve yesterday's diagnosis for today's different failure — freshness has to be event-aware, not just time-based.

What surprised me: this wasn't an optimization I bolted on. Deciding that one fingerprint = one diagnosis ended up shaping the DynamoDB schema, the cost model, and even the UX (incidents show a "diagnosed" state with cache freshness). If I built this again, I'd design the fingerprinting and invalidation strategy on day one. Caching is product design, not optimization.

3. Right-size the model

Most routine diagnoses do not require the largest available model. They need a fast, cost-efficient model with good tool-calling, grounded in live cluster data and runbooks.

This is the part most cost advice gets backwards. The instinct is "harder problem → bigger model." But for operational diagnosis, the biggest quality lever isn't model size — it's grounding. A smaller, well-grounded model that can see this pod's actual events, the last N log lines, and the matching runbook can outperform a larger model that lacks relevant operational context.

Garbage context in, confident garbage out — at any model size.

Save the frontier model for where reasoning depth genuinely pays: complex multi-service correlation, ambiguous novel failures. For the common case — "why is this pod OOMKilled" — a well-grounded mid-tier model is cheaper and often just as good or better.

4. Serverless orchestration and managed agent runtime

The runtime cost model matters as much as the invocation cost model.

To be precise about scope: the detection, incident-storage, and diagnosis-orchestration path is what's serverless — EventBridge, Lambda, DynamoDB, plus a managed agent runtime (Bedrock AgentCore). The Kubernetes cluster itself, networking, and observability stack obviously exist independently; the point is that the AI control plane adds no always-on compute of its own:

No idle compute for the agent between incidents.
No GPU sitting warm waiting for a question.
No dedicated "AI infrastructure" cluster to justify, patch, and pay for at 3 a.m.

The AI layer's cost scales with incidents, not with time. A healthy cluster incurs only the relatively small cost of scheduled checks, API access, logging, and incident-state storage. That's exactly the cost curve you want for an ops tool: the expensive weeks are the weeks it earned its keep.

5. Tools return data, not dumps

The quietest cost (and quality) killer: tool responses that dump everything.

Each tool behind the MCP gateway returns only what the diagnosis needs:

this pod's events — not the namespace's
the last N log lines — not the full log stream
the relevant runbook section — not the whole knowledge base

# tool: k8s_data_lookup — scoped by design
def get_pod_context(namespace: str, pod: str) -> dict:
    return {
        "pod_spec": get_pod_spec(namespace, pod),
        "events": get_events(namespace, pod, limit=20),        # this pod only
        "logs": get_logs(namespace, pod, tail_lines=100),      # last 100 lines
        "restarts": get_restart_summary(namespace, pod),
    }

Token discipline at the tool layer is invisible in demos and decisive in production. It cuts input tokens per invocation dramatically — and, as a bonus, it made diagnosis quality go up, because the model stopped drowning in irrelevant context. I got more quality improvement from making tools return clean, scoped data than from any prompt rewrite I ever did.

The trust boundary (because someone will ask)

A cost article isn't the place for the full security design, but two points are inseparable from the architecture above:

The agent's identity is read-only. Its runtime identity maps to a read-only ClusterRole. The suggested remediation — YAML diff, kubectl commands — is rendered as text only. Any execution happens separately, by an authenticated engineer, through the organization's existing access controls and approval process. (For stronger governance, generated commands can additionally be validated against an allowlist or policy engine before display.)

Everything is audited. Every tool invocation is recorded with the requesting identity, incident fingerprint, tool name, parameters, timestamp, and result status — searchable through the observability platform and retained in S3 per the audit-retention policy, with secrets and sensitive log content redacted before storage. When someone asks "what did the AI actually do?", the answer is a log, not a shrug.

The principle underneath all five

Put the LLM at the point of maximum value — and nowhere else.

Detection? Rules. (Reliability, not reasoning.)
Repeated incidents? Fingerprint + cache. (The answer didn't change.)
Routine diagnosis? Smaller grounded model. (Grounding beats size.)
Idle time? Serverless AI control plane. (Pay for incidents, not hours.)
Context? Scoped tools. (Tokens are a budget.)

The result is a system where the expensive component — the model — runs once per materially distinct incident, sees only the context it needs, and never repeats work it already did.

Most LLM cost advice is about prompt compression and model discounts. Those are real, but they're 2x levers. In my experience, the largest savings often live one level up — in the invocation architecture, in deciding when the model is allowed to run at all.

What this looks like in practice

The full flow, end to end:

EventBridge scheduled scan (every 5 min)
        ↓
Lambda performs deterministic detection          ← no LLM
        ↓
Normalize + fingerprint incident
        ↓
DynamoDB conditional write
        ↓
New or materially changed incident?
   ├── No  → bump occurrence count + last-seen   ← no LLM
   └── Yes → diagnosis (by severity or on human open)
                    ↓
            Retrieve scoped evidence (tools)
                    ↓
            Retrieve relevant runbook sections
                    ↓
            Invoke appropriately sized model      ← the one LLM call
                    ↓
            Store diagnosis + evidence references
                    ↓
            Engineer reviews proposed remediation

By the time an engineer opens a high-severity incident, the root cause and a proposed fix — YAML diff plus exact kubectl commands — are already waiting, generated from live cluster data and runbooks, on a read-only credential path, fully audited. The agent turns a repetitive 30-minute investigation into a focused human review. And the bill stays boring.

This is part of a series on building a production-grade agentic AI SRE — upcoming posts cover the read-only trust architecture (why my agent can diagnose anything but can't restart a single pod) and the lessons learned building it. Follow me here or on LinkedIn if you want the rest.

How have LLM costs surprised your team — and did the biggest issue come from prompts, excessive context, or the invocation loop?

DEV Community: shafivullashaik-tech-ops