DEV Community: Oleksandr Kuryzhev

RAG Pipeline for SRE Runbooks: 7 Vector Search Tips That Work

Oleksandr Kuryzhev — Mon, 15 Jun 2026 07:03:02 +0000

Originally published on kuryzhev.cloud

Your on-call engineer gets paged at 2 AM and your RAG system confidently surfaces a runbook from six months ago — deprecated after the last migration, full of references to services that no longer exist. The engineer follows it anyway. That's the failure mode nobody talks about when they say "we RAG-ified our runbooks." Building a RAG pipeline for SRE runbooks that actually works in production means getting the embedding model, the index structure, the ingestion loop, and the retrieval quality all right at the same time. These seven tips are what I wish I'd known before our first on-call integration went sideways.

Tip 1: Choose the Right Embedding Model for Runbook Content

Generic embedding models misread SRE jargon — domain matters more than benchmark scores.

Terms like OOMKilled, CrashLoopBackOff, HighMemoryUsage, or your internal alert names are essentially invisible to models trained on general web text. They get embedded close to random technical noise rather than clustering with semantically related runbook content. I learned this after watching text-embedding-ada-002 confidently return a Kubernetes networking runbook for a PostgreSQL replication alert because both happened to mention "connection timeout."

My current preference is BAAI/bge-small-en-v1.5 via sentence-transformers>=2.7.0. It produces 384-dimensional vectors, runs about 5x faster than ada-002 at inference time, and handles technical prose significantly better in practice. A single t3.medium can push roughly 50 embed requests per second — more than enough for alert-driven RAG queries, though you'll need batching for bulk re-indexing. If you need a hosted option and ada-002 is already in your stack, it's usable, but use distance: Dot in your Qdrant collection config for OpenAI vectors rather than Cosine — they're not interchangeable.

One chunking detail that trips people up: don't split runbooks by fixed token count without respecting procedural step boundaries. Splitting "Step 3: drain the node" across two chunks destroys the procedural context the retriever needs. Use 512-token chunks with 64-token overlap as a starting point — the overlap preserves continuity across step boundaries without ballooning your index size.

Tip 2: Structure Your Vector Store Index Around Incident Taxonomy

Metadata filtering before semantic search cuts irrelevant results by ~60% — don't skip it.

A pure vector search across your entire runbook corpus will always surface some plausible-but-wrong results. The fix isn't a better model — it's filtering. Before the semantic ranking even runs, filter by structured metadata fields that you already have: alert_name, service, severity, on_call_team, and critically, last_updated. That last field is the one most teams forget to store, and it's what lets you warn engineers when the best matching runbook is eight months stale.

For the vector store itself, I use Qdrant in production. Version 1.9.x added native sparse+dense hybrid search via the sparse_vectors config, which gives you BM25 keyword matching combined with semantic similarity in a single query — genuinely useful when alert names are exact-match keywords. If you're evaluating alternatives: Weaviate v1.24+ has the generative-openai module built in, which is tempting, but it couples your retrieval and generation layers tightly and makes model swaps painful. Pinecone namespaces work well if you're already in that ecosystem and don't need hybrid search.

Watch out for: Qdrant's default Docker image ships with zero authentication enabled. Always set the QDRANT_SERVICE_API_KEY environment variable and keep port 6333 inside a private subnet. I've seen this misconfiguration in three separate internal tooling audits.

Tip 3: Ingest Runbooks from Confluence or Git with a Lightweight Pipeline

Hash-based change detection keeps your vector store fresh without re-embedding everything on every run.

The ingestion pipeline is where most RAG implementations get lazy and end up paying for it — either in stale data or in runaway embedding API costs. The pattern I use: store a sha256 of each document's content in Redis. On every pipeline run, compare the current hash. If it matches, skip re-embedding entirely. Only new or changed content hits the embedding model.

For Git-based runbooks, enforce a path convention: docs/runbooks/{service}/{alert_name}.md. This lets you extract service and alert_name metadata directly from the file path without parsing file content — simpler and less error-prone. For Confluence, the REST API endpoint /wiki/rest/api/content?type=page&spaceKey=SRE works, and LangChain's ConfluenceLoader (requires atlassian-python-api>=3.41.0) gets you started fast. That said, I moved off it to a custom fetch — you get better metadata control and don't inherit LangChain's chunking decisions.

Here's the full ingestion pipeline with hash-based deduplication and Redis embedding cache:


# rag_ingest.py — Runbook ingestion pipeline with hash-based deduplication
# Deps: qdrant-client>=1.9.0, sentence-transformers>=2.7.0, python-dotenv, redis, tiktoken

import os
import hashlib
import json
from pathlib import Path
from dotenv import load_dotenv
import redis
from qdrant_client import QdrantClient
from qdrant_client.models import (
    Distance, VectorParams, PointStruct, Filter,
    FieldCondition, MatchValue
)
from sentence_transformers import SentenceTransformer

load_dotenv()

# --- Config ---
QDRANT_URL = os.getenv("QDRANT_URL", "http://localhost:6333")
QDRANT_API_KEY = os.getenv("QDRANT_API_KEY")
COLLECTION_NAME = "sre_runbooks"
EMBED_MODEL = "BAAI/bge-small-en-v1.5"   # 384-dim, fast, good on technical text
CHUNK_SIZE = 512        # tokens
CHUNK_OVERLAP = 64      # token overlap to preserve step continuity
SCORE_THRESHOLD = 0.78  # minimum cosine similarity to surface a result

# --- Clients ---
redis_client = redis.Redis(host="localhost", port=6379, decode_responses=True)
qdrant = QdrantClient(url=QDRANT_URL, api_key=QDRANT_API_KEY)
model = SentenceTransformer(EMBED_MODEL)

def chunk_text(text: str, size: int = CHUNK_SIZE, overlap: int = CHUNK_OVERLAP) -> list[str]:
    """Split on word boundaries respecting overlap — avoids mid-step cuts."""
    words = text.split()
    chunks, i = [], 0
    while i < len(words):
        chunk = " ".join(words[i:i + size])
        chunks.append(chunk)
        i += size - overlap  # slide with overlap
    return chunks

def embed_with_cache(text: str) -> list[float]:
    """Return cached embedding or compute and store it."""
    key = f"emb:v1:{hashlib.sha256(text.encode()).hexdigest()}"
    cached = redis_client.get(key)
    if cached:
        return json.loads(cached)
    vector = model.encode(text, normalize_embeddings=True).tolist()
    redis_client.setex(key, 604800, json.dumps(vector))  # TTL: 7 days
    return vector

def ingest_runbook(filepath: Path):
    """Parse path for metadata, chunk content, upsert to Qdrant."""
    # Expected path: docs/runbooks/{service}/{alert_name}.md
    parts = filepath.parts
    service = parts[-2] if len(parts) >= 2 else "unknown"
    alert_name = filepath.stem  # filename without .md

    content = filepath.read_text(encoding="utf-8")
    doc_hash = hashlib.sha256(content.encode()).hexdigest()

    # Fast change detection via Redis — skip unchanged docs entirely
    hash_key = f"doc_hash:{filepath}"
    if redis_client.get(hash_key) == doc_hash:
        print(f"[SKIP] {filepath} unchanged")
        return

    chunks = chunk_text(content)
    points = []
    for idx, chunk in enumerate(chunks):
        vector = embed_with_cache(chunk)
        point_id = int(hashlib.sha256(f"{filepath}:{idx}".encode()).hexdigest()[:8], 16)
        points.append(PointStruct(
            id=point_id,
            vector=vector,
            payload={
                "service": service,
                "alert_name": alert_name,
                "chunk_index": idx,
                "source_path": str(filepath),
                "doc_hash": doc_hash,
                "text": chunk,
            }
        ))

    qdrant.upsert(collection_name=COLLECTION_NAME, points=points)
    redis_client.set(hash_key, doc_hash)  # update change-detection cache
    print(f"[OK] Ingested {len(points)} chunks from {filepath}")

def ensure_collection():
    """Create collection if it doesn't exist."""
    existing = [c.name for c in qdrant.get_collections().collections]
    if COLLECTION_NAME not in existing:
        qdrant.create_collection(
            collection_name=COLLECTION_NAME,
            vectors_config=VectorParams(size=384, distance=Distance.COSINE),
        )
        print(f"[INIT] Created collection: {COLLECTION_NAME}")

if __name__ == "__main__":
    ensure_collection()
    runbook_dir = Path("docs/runbooks")
    for md_file in runbook_dir.rglob("*.md"):
        ingest_runbook(md_file)

Tip 4: Wire the RAG Query into Your Alerting Workflow

Surface runbook context automatically when an alert fires — not only when someone thinks to ask.

The real value of a RAG pipeline for SRE runbooks isn't a chat interface. It's injecting relevant procedure context into the incident notification itself, before the engineer even opens a terminal. The integration point is your Alertmanager or PagerDuty webhook. When a webhook fires, extract the alertname label (Alertmanager v2 path: .alerts[0].labels.alertname) and use it as the query string to your RAG endpoint.

One PagerDuty-specific gotcha: webhook v3 sends event.data.title as the incident name. Map this field, not event.id, to your query — I've seen this wired wrong in three different integrations and the resulting queries return garbage.

Set a similarity score threshold of 0.78 with cosine distance as your starting point. Below that, return a "matched": false signal so your Slack notification can still fire — just without a runbook attachment. A "no confident match" message is far safer than surfacing a low-confidence wrong runbook. Return the top-3 chunks maximum; more than that and engineers stop reading them.

Here's the FastAPI query endpoint wired to an Alertmanager webhook payload:


# rag_query.py — Query endpoint wired to Alertmanager webhook
# Receives alert payload, returns top-3 runbook chunks above threshold

import os
from fastapi import FastAPI, Request, HTTPException
from qdrant_client import QdrantClient
from qdrant_client.models import Filter, FieldCondition, MatchValue
from sentence_transformers import SentenceTransformer

QDRANT_URL = os.getenv("QDRANT_URL", "http://localhost:6333")
QDRANT_API_KEY = os.getenv("QDRANT_API_KEY")
COLLECTION_NAME = "sre_runbooks"
SCORE_THRESHOLD = 0.78
TOP_K = 3

app = FastAPI()
qdrant = QdrantClient(url=QDRANT_URL, api_key=QDRANT_API_KEY)
model = SentenceTransformer("BAAI/bge-small-en-v1.5")

@app.post("/query/alert")
async def query_from_alert(request: Request):
    """
    Accepts Alertmanager webhook JSON.
    Extracts alertname + service label, runs filtered vector search.
    Returns top-K chunks or a no-match signal.
    """
    body = await request.json()

    try:
        # Alertmanager v2 webhook schema
        alert = body["alerts"][0]
        alert_name = alert["labels"]["alertname"]       # e.g. "HighMemoryUsage"
        service = alert["labels"].get("service", None)  # optional label
    except (KeyError, IndexError):
        raise HTTPException(status_code=400, detail="Invalid Alertmanager payload")

    query_text = f"{alert_name} {service or ''}".strip()
    query_vector = model.encode(query_text, normalize_embeddings=True).tolist()

    # Pre-filter by alert_name metadata before semantic ranking
    search_filter = Filter(
        must=[FieldCondition(key="alert_name", match=MatchValue(value=alert_name))]
    ) if alert_name else None

    results = qdrant.search(
        collection_name=COLLECTION_NAME,
        query_vector=query_vector,
        query_filter=search_filter,
        limit=TOP_K,
        score_threshold=SCORE_THRESHOLD,  # drop low-confidence results
        with_payload=True,
    )

    if not results:
        # Fallback: no confident match — Slack still pages, just without runbook
        return {"matched": False, "alert_name": alert_name, "chunks": []}

    return {
        "matched": True,
        "alert_name": alert_name,
        "chunks": [
            {
                "text": r.payload["text"],
                "source": r.payload["source_path"],
                "score": round(r.score, 4),
                "chunk_index": r.payload["chunk_index"],
            }
            for r in results
        ],
    }

# Example response:
# {
#   "matched": true,
#   "alert_name": "HighMemoryUsage",
#   "chunks": [
#     {"text": "Step 1: check OOMKilled pods with kubectl describe...",
#      "source": "docs/runbooks/api/HighMemoryUsage.md",
#      "score": 0.8912, "chunk_index": 2}
#   ]
# }

For Slack delivery, use Block Kit's section block with a mrkdwn text field to render the runbook chunk inline alongside the alert details. Include the source_path and score so engineers immediately know where it came from and how confident the match is.

Tip 5: Evaluate Retrieval Quality Before You Trust It in Production

The silent failure mode is a RAG that returns plausible-but-wrong runbook steps with high confidence.

Most teams evaluate their RAG pipeline by asking "does the LLM answer look right?" That's the wrong question. You need to evaluate whether the retrieved chunks were actually the correct runbook sections before any LLM even sees them. A well-phrased wrong answer is worse than an obvious failure.

Build a golden dataset: 20-30 pairs of (alert_name, expected_runbook_section). Run recall@3 checks — does the correct chunk appear in the top 3 results? That's your baseline metric. For a more structured eval, the ragas library (v0.1.x) provides context_recall and answer_relevancy metrics. Note that ragas requires openai>=1.0.0 and makes separate LLM calls for scoring — budget for that API cost in your eval pipeline, it's not free.

Run this eval gate on every significant change to the runbook corpus or after swapping embedding models. I caught a 15% recall drop after a Confluence space reorganization that changed page titles — the metadata-extracted alert_name fields shifted, and the pre-filter was excluding correct results. Without the eval gate, that would have silently degraded on-call for weeks.

Tip 6: Secure the Pipeline — Runbooks Contain Sensitive Operational Detail

Your vector store holds internal hostnames, escalation contacts, and credential patterns — treat it like production infrastructure.

This is the access control gap I see most often. Teams move runbooks into a vector DB, wire up a query API, and mark it "internal only" as if that's sufficient. Runbooks regularly contain things like internal service hostnames, credential rotation procedures, escalation phone trees, and network topology details. If a service account with access to your RAG query API is compromised, an attacker can enumerate your entire operational playbook through semantic search.

Enforce collection-level ACLs in Qdrant using per-collection API keys. In Weaviate, use RBAC to scope read access by team. Never expose the RAG query endpoint without authentication, even on an internal network — lateral movement from a compromised service is a real threat model, not a theoretical one.

Watch out for: the Redis embedding cache also needs protection. Those cached vectors can be used to reconstruct approximate source text. Keep Redis on a private interface, require requirepass, and set appropriate bind directives. I stopped treating the cache layer as "just an optimization" after reading about embedding inversion attacks — they're not academic anymore.

Also store last_updated as a metadata field on every point. Without it, you have no way to surface a staleness warning to the on-call engineer when the best matching runbook is months old. This is a cheap field to add and an expensive oversight to fix after the fact. For more on securing internal tooling pipelines, see the patterns we cover at kuryzhev.cloud.

Tip 7: Control Costs by Caching Embeddings and Limiting Re-Indexing Frequency

Naive re-indexing pipelines multiply embedding costs fast — cache aggressively and schedule smart.

At first glance, embedding costs look trivial. Five hundred runbook pages at roughly 10 chunks each, priced at text-embedding-ada-002's $0.0001 per 1K tokens, works out to about $0.25 per full re-index. That sounds fine. But a naive pipeline that re-embeds everything on every CI merge, or that re-indexes when Confluence sends a webhook for a minor edit, turns that $0.25 into a daily charge. At scale with a self-hosted GPU model, it becomes compute time you're burning for no reason.

The fix is two-layered. First, the Redis embedding cache with key pattern emb:v1:{sha256(chunk_text)} — identical chunk content across different documents or pipeline runs hits the cache, not the model. Include a version prefix (v1) so that when you upgrade your embedding model, you can invalidate the entire cache cleanly by bumping to v2 without touching cache logic. Second, schedule full re-indexes weekly. Run incremental re-indexing (changed documents only, via hash comparison) on every merge to main. This keeps the index current without re-embedding stable content.

One more cost lever: use gRPC instead of HTTP for Qdrant batch upserts. The default HTTP port is 6333, gRPC is 6334. Switching to gRPC gives approximately 30% lower latency on batch operations — not a cost saving directly, but it reduces the wall-clock time your ingestion job runs, which matters if you're paying for the compute that runs it.

MCP Servers for DevOps: Build vs Pre-Built — What to Choose

Oleksandr Kuryzhev — Sun, 14 Jun 2026 07:02:35 +0000

Originally published on kuryzhev.cloud

You wired an LLM into your incident workflow, gave it kubectl access via an MCP server you found on GitHub, and only later realized it was running against your production cluster with your personal kubeconfig. That's the kind of mistake that happens when you move fast with MCP server DevOps tooling without thinking through the operational model first. I've been there. This post is the comparison I wish I had before we started.

The Model Context Protocol (MCP) is a JSON-RPC 2.0 protocol that gives LLMs like Claude or GPT-4 structured, typed access to external tools — kubectl, Terraform, PagerDuty, your internal CMDB — with session state and proper error handling. It's not just another API wrapper. It's the difference between prompt-engineering your way to fragile shell commands and giving your AI assistant a real, auditable toolchain. But the moment you go from a weekend demo to a team-shared production deployment, you face a real architectural decision: do you use community-maintained pre-built MCP servers, or do you build your own?

The wrong answer costs you either brittle, unmaintainable hacks or an over-engineered custom server nobody on your team wants to touch at 3 AM. Let me walk you through both options with the honest trade-offs.

When You Face This Choice

This decision usually surfaces when you're wiring LLMs into something that actually matters: incident triage, infra query automation, or CI/CD pipeline introspection. You need structured tool access — not raw API calls buried in a system prompt, but proper MCP tools with typed inputSchema, error normalization, and the ability to maintain context across multi-step operations like "get the failing pods, look up the runbook, then check recent deploys."

The MCP spec (currently at version 2025-03-26, which introduced breaking changes to the resources/list response schema versus the earlier 2024-11-05 version) defines a protocolVersion field in the initialize handshake. If your server and client are on different spec versions, tool calls silently misbehave. That's your first hint that the ecosystem is still moving fast and your tooling choices have real consequences.

The core question: does your team need to expose internal tools that will never have a community server, and does this MCP setup need to serve more than one engineer? If yes to either, you're already leaning toward custom. If you're running a proof-of-concept for one person against non-production systems, pre-built is the right starting point. Let's go through both properly.

Option A — Pre-Built Community MCP Servers

Pre-built servers are genuinely impressive for getting started. uvx mcp-server-kubernetes gives you pod listing, log fetching, and resource inspection in under ten minutes. The official GitHub MCP server (github.com/github/github-mcp-server, distributed as a Go binary) supports fine-grained PATs and is the only community server I know of with a published security advisory process as of mid-2025. mcp-server-prometheus handles instant PromQL queries with authentication. You get community-maintained tool schemas that already handle pagination, error normalization, and auth flows — things that take real time to get right when you build from scratch.

Pros: Zero-to-running in under 10 minutes. npx @modelcontextprotocol/server-github or uvx mcp-server-kubernetes works out of the box. The tool schemas are already battle-tested for common operations. You don't own the spec upgrade burden on day one.

Cons: The tool surface is fixed. You cannot expose your internal Backstage catalog, your custom SLO dashboard API, or your Spinnaker deployment history. Versioning is genuinely chaotic — mcp-server-kubernetes v0.1.x vs v0.2.x broke tool signatures in March 2025, and if you're pinned to an old version in a team deployment, you'll spend a debugging session figuring out why tool calls are returning MCP error -32602: Invalid params. You also inherit their security model entirely.

Watch out for this: uvx mcp-server-kubernetes requires kubectl in PATH and uses your active kubeconfig context. It will happily point at production if that's your current context. Always set KUBECONFIG explicitly in your MCP server config to a context-specific kubeconfig file, not your default ~/.kube/config.

Watch out for this too: Most community servers run as stdio transport by default — one process per session, zero shared state, fine for local Claude Desktop usage. The moment you want a team-shared deployment, you need a proxy layer. mcp-proxy (available on PyPI) bridges stdio servers to SSE: mcp-proxy --port 9000 -- uvx mcp-server-kubernetes. That's an extra operational component you now own. And mcp-server-prometheus does not support PromQL range queries out of the box — only instant queries. If your on-call workflow needs query_range, you're building a custom tool regardless.

Option B — Custom MCP Servers with the Python or TypeScript SDK

Building your own MCP server with the official Python SDK (pip install mcp==1.6.0) or TypeScript SDK (npm install @modelcontextprotocol/sdk@1.10.0) gives you complete control. You define exactly what tools exist, what their schemas look like, how auth works (mTLS, OIDC tokens, Vault-sourced secrets), and critically — what the LLM is actually allowed to touch. You can wrap internal APIs that will never have a community server: your CMDB, your internal runbook API, your custom SLO dashboards, your Spinnaker deployment history.

Pros: Full control over tool schema, authentication model, and blast radius. You can implement tool-level allow-lists in handler middleware. You can log every single tool call to your SIEM. You can truncate verbose responses server-side before they hit the LLM context window — because full kubectl get pods -A JSON output runs 8,000–15,000 tokens per call, which is both expensive and context-window-polluting.

Cons: You own the maintenance burden. The MCP spec moved from 2024-11-05 to 2025-03-26 with breaking changes to resources and sampling. The Python and TypeScript SDKs are still effectively pre-1.0 stable in practice despite their version numbers. You will be doing spec upgrades on a Friday afternoon at some point.

Critical gotcha that almost everyone gets wrong: The MCP SDK does NOT automatically validate tool.inputSchema against the arguments the LLM passes. If the LLM sends malformed args — or if a prompt injection in a log file triggers an unexpected tool call with bad parameters — your handler receives them as-is. You will get a confusing Python KeyError or TypeError deep in your handler instead of a clean error message. You must add Pydantic (Python) or Zod (TypeScript) validation yourself, at the top of every tool handler. This is non-negotiable for production.

Also: do not register more than 15–20 tools in a single MCP server. LLMs degrade in tool selection accuracy above roughly 15 tools. Split by domain: infra-tools, observability-tools, incident-tools. Separate servers, separate concerns.

Here's a minimal but production-pattern custom MCP server in Python. It wraps kubectl for pod listing and an internal runbook API, with proper Pydantic validation and response truncation:


# custom_mcp_server.py
# Minimal production-pattern MCP server wrapping kubectl + internal runbook API
# Requires: pip install mcp==1.6.0 pydantic==2.7.0 httpx==0.27.0

import asyncio
import subprocess
import json
import httpx
from mcp.server import Server
from mcp.server.stdio import stdio_server
from mcp.types import Tool, TextContent, CallToolResult
from pydantic import BaseModel, ValidationError

# --- Input schema models (SDK does NOT validate these automatically) ---

class KubectlGetPodsInput(BaseModel):
    namespace: str
    label_selector: str = ""          # optional label filter e.g. "app=nginx"
    max_results: int = 20             # hard cap to control token usage

class RunbookLookupInput(BaseModel):
    service_name: str
    alert_name: str

# --- Server init ---

app = Server("sre-mcp-server")

RUNBOOK_API_BASE = "https://runbooks.internal.example.com/api/v1"
RUNBOOK_API_TOKEN = "REPLACE_WITH_VAULT_SOURCED_SECRET"   # inject via env in prod

# --- Tool definitions ---

@app.list_tools()
async def list_tools() -> list[Tool]:
    return [
        Tool(
            name="kubectl_get_pods",
            description="List pods in a namespace. Read-only. Max 20 results.",
            inputSchema=KubectlGetPodsInput.model_json_schema(),
        ),
        Tool(
            name="runbook_lookup",
            description="Fetch the runbook for a given service and alert name.",
            inputSchema=RunbookLookupInput.model_json_schema(),
        ),
    ]

# --- Tool handlers ---

@app.call_tool()
async def call_tool(name: str, arguments: dict) -> CallToolResult:

    if name == "kubectl_get_pods":
        try:
            args = KubectlGetPodsInput(**arguments)   # validate here, not in SDK
        except ValidationError as e:
            return CallToolResult(
                content=[TextContent(type="text", text=f"Invalid args: {e}")]
            )

        cmd = ["kubectl", "get", "pods", "-n", args.namespace, "-o", "json"]
        if args.label_selector:
            cmd += ["-l", args.label_selector]

        result = subprocess.run(cmd, capture_output=True, text=True, timeout=15)

        if result.returncode != 0:
            return CallToolResult(
                content=[TextContent(type="text", text=f"kubectl error: {result.stderr}")]
            )

        pods_json = json.loads(result.stdout)
        # Truncate to max_results and strip managed fields to reduce token usage
        items = pods_json.get("items", [])[:args.max_results]
        slim = [
            {
                "name": p["metadata"]["name"],
                "phase": p["status"].get("phase"),
                "ready": all(
                    c["ready"] for c in p["status"].get("containerStatuses", [])
                ),
                "restarts": sum(
                    c.get("restartCount", 0)
                    for c in p["status"].get("containerStatuses", [])
                ),
            }
            for p in items
        ]
        return CallToolResult(
            content=[TextContent(type="text", text=json.dumps(slim, indent=2))]
        )

    elif name == "runbook_lookup":
        try:
            args = RunbookLookupInput(**arguments)
        except ValidationError as e:
            return CallToolResult(
                content=[TextContent(type="text", text=f"Invalid args: {e}")]
            )

        async with httpx.AsyncClient() as client:
            resp = await client.get(
                f"{RUNBOOK_API_BASE}/runbooks",
                params={"service": args.service_name, "alert": args.alert_name},
                headers={"Authorization": f"Bearer {RUNBOOK_API_TOKEN}"},
                timeout=10,
            )
            if resp.status_code != 200:
                return CallToolResult(
                    content=[TextContent(type="text", text=f"Runbook API error: {resp.status_code}")]
                )
            return CallToolResult(
                content=[TextContent(type="text", text=resp.text)]
            )

    return CallToolResult(
        content=[TextContent(type="text", text=f"Unknown tool: {name}")]
    )

# --- Entry point ---

async def main():
    async with stdio_server() as (read_stream, write_stream):
        await app.run(read_stream, write_stream, app.create_initialization_options())

if __name__ == "__main__":
    asyncio.run(main())

And here's how to wire both a custom server and community servers together in your Claude Desktop config. This is the hybrid pattern I actually run:


{
  "mcpServers": {

    "sre-tools": {
      "command": "python",
      "args": ["/opt/mcp-servers/custom_mcp_server.py"],
      "env": {
        "KUBECONFIG": "/home/sre/.kube/config-staging",
        "RUNBOOK_API_TOKEN": "${RUNBOOK_API_TOKEN}"
      }
    },

    "prometheus": {
      "command": "uvx",
      "args": ["mcp-server-prometheus"],
      "env": {
        "PROMETHEUS_URL": "https://prometheus.internal.example.com",
        "PROMETHEUS_USERNAME": "mcp-readonly",
        "PROMETHEUS_PASSWORD": "${PROM_PASSWORD}"
      }
    },

    "github": {
      "command": "/usr/local/bin/github-mcp-server",
      "args": ["stdio"],
      "env": {
        "GITHUB_PERSONAL_ACCESS_TOKEN": "${GITHUB_PAT}"
      }
    }

  }
}

Note the explicit KUBECONFIG path pointing to a staging context. Note that the API token comes from a shell environment variable, not hardcoded. Small details, but they matter when this runs unattended.

You can test any MCP server locally without a full LLM client using the mcp dev CLI command, which launches the MCP Inspector UI at http://localhost:5173. Run it before you ever connect Claude to a new server. It saves a lot of confusion.

Decision Matrix

Here's how these two options actually stack up across the dimensions that matter for real DevOps teams — not theoretical ones.

Dimension	Pre-Built (Option A)	Custom (Option B)
Setup time	Under 10 minutes	Half a day minimum
Tool customization	Fixed, community-defined	Full control
Internal API access	Not possible	First-class
Security posture	Inherited, opaque	Explicit, auditable
Maintenance burden	Low initially, chaotic on upgrades	Ongoing, owned by you
Multi-user / team deployment	Requires mcp-proxy, extra ops	SSE transport, proper auth
Production readiness	Proof-of-concept to small teams	Production-grade with work
SIEM / audit logging	Not available	Implement in handler middleware

The inflection point is clear. If your team has fewer than three internal-only tool integrations and this is a proof-of-concept, use pre-built. If you're running MCP in production for more than five engineers, touching real infrastructure, you need custom servers with proper auth and audit logging. There's no middle ground that scales.

The transport layer is another forcing function. Stdio transport spawns a new process per Claude Desktop session — zero shared state, which breaks any tool that needs to maintain a Terraform plan or a multi-step workflow between calls. SSE transport (http://localhost:8000/sse by default, configurable with --host, --port, or MCP_HOST/MCP_PORT env vars) is HTTP-based and supports multiple concurrent clients. The moment you need SSE, you're in custom server territory anyway, because most community servers don't ship with SSE support configured for team use.

MCP has no built-in authorization at the tool level. This is the security gap that matters most. If your server exposes both kubectl_get and kubectl_delete, the LLM — or a prompt injection hiding in a log file your LLM just read — can call either. Implement tool-level allow-lists in your handler middleware. The official MCP documentation covers the protocol spec in detail at modelcontextprotocol.io. For Kubernetes RBAC, bind your MCP server's service account to a ClusterRole with only get, list, and watch verbs. See the Kubernetes RBAC documentation for the ClusterRole pattern. Never bind create, delete, or patch unless you have a separate, explicitly audited use case for it.

My Pick

I prefer the hybrid approach, and I'm not hedging on this. Start with pre-built for your proof-of-concept — specifically the mcp-server-kubernetes plus mcp-server-prometheus combination. Get your team comfortable with what MCP-powered workflows actually feel like before you write a line of server code. Then, once you know which internal APIs you actually need and what the real operational requirements are, build a thin custom Python MCP server that wraps those internal integrations.

Do not fork community servers. That's the trap. You end up owning their entire codebase plus your customizations, and when the spec upgrades, you're doing a merge instead of just bumping your SDK version.

The pattern I run: community servers (Prometheus, GitHub) run as separate stdio processes. My custom server handles everything internal. Both are registered in Claude Desktop config as separate mcpServers entries. The LLM picks tools from both. Clean separation, minimal blast radius per server.

One non-negotiable: any MCP server touching production infrastructure runs with a dedicated service account, scoped RBAC, and logs every tool call to the SIEM. Treat it like a privileged CI runner, not a chatbot plugin. Store credentials in Vault or your secrets manager, inject them as environment variables at runtime, and rotate them on the same schedule as your other service accounts. The mcp dev inspector tool at http://localhost:5173 is your friend for validating tool schemas before you ever point a real LLM at a server.

MCP server DevOps tooling is moving fast. The spec will break again. But the operational principles — scoped access, audit logging, input validation, response truncation — those don't change. Get those right and you can absorb the spec churn without a production incident. You can find more patterns for building secure, automated infrastructure tooling at kuryzhev.cloud.

Auto-Summarize On-Call Incidents with n8n AI Agents

Oleksandr Kuryzhev — Sat, 13 Jun 2026 07:03:03 +0000

Liquid syntax error: Variable '{{ {% raw %}' was not properly terminated with regexp: /\}\}/

AI Code Review for Terraform PRs: CI Checklist and Automation

Oleksandr Kuryzhev — Fri, 12 Jun 2026 07:03:08 +0000

Originally published on kuryzhev.cloud

Your Terraform PR passed tflint and checkov — but the AI reviewer just flagged that you're about to delete a production RDS instance and nobody noticed. That's the exact scenario that pushed our team to build a structured AI Terraform PR review process into CI. Not as a replacement for human review. As a safety net that catches what humans miss when they're tired, rushed, or reviewing their twelfth PR of the day.

This checklist is for teams running Terraform 1.5+ in GitHub Actions, GitLab CI, or Atlantis pipelines. Every item here came from either a real incident or a near-miss. Some of it will be obvious. A few items will surprise you.

Why This Checklist

A typical IaC pull request touches anywhere from 10 to 50 resources. Under time pressure — and reviewers are almost always under time pressure — human engineers miss roughly 30% of policy violations. I've seen it happen on our own team. A reviewer approves a PR with an unencrypted S3 bucket because the checkov annotation looks fine at a glance, but the skip reason references a deprecated ADR. Nobody catches it until a compliance scan runs two weeks later.

AI code review tools like CodeRabbit, custom GPT-4o scripts, and Atlantis LLM hooks analyze the full plan diff in seconds. They don't get tired. They don't skip the last file because standup starts in five minutes. But they also hallucinate. They confuse AWS provider v4 attributes with v5 changes. They miss secrets buried in .auto.tfvars files. That's exactly why you need a checklist — not to trust the AI blindly, but to define what it must check and what it cannot be trusted to catch alone.

One more thing worth saying upfront: adding AI review as a blocking CI gate on day one is a mistake. Run it as an advisory check for two weeks first. Let the team calibrate signal versus noise. Then promote it to blocking. I learned this the hard way after we blocked three legitimate PRs because the model flagged a lifecycle { prevent_destroy = true } block as "suspicious" on a stateful resource it didn't recognize.

The AI Code Review Checklist for Terraform PRs

Each item below is something you configure once and enforce on every PR. The first four are static analysis gates. Items five through eight are AI-specific prompt checks. Nine through twelve cover plan diff review. The final three address state hygiene.

Run tflint v0.50+ with --format=json. Earlier versions produce text output that's unparseable by downstream scripts. Pin the version explicitly in your CI — don't pull latest.
Wire tflint exit codes correctly. Exit code 1 means violations found. Exit code 2 means the tool itself errored. Most pipelines treat both as the same failure. They are not. A tool error should page someone; a lint violation should block the PR.
Run checkov 3.x on changed files only. Running checkov on the entire repo on every PR is slow and noisy. Use git diff to scope it to changed .tf files.
Distinguish checkov exit code 1 vs 2. Same issue as tflint — exit code 2 is a tool error, not a policy violation. Wire them separately in your CI logic.
Prompt the AI to flag hardcoded credentials. Static tools miss credentials embedded in locals blocks or passed as default values in variable declarations. Include this explicitly in your system prompt.
Prompt for missing lifecycle blocks on stateful resources. RDS instances, ElastiCache clusters, and S3 buckets without prevent_destroy = true are a silent risk. The AI catches this consistently when you ask for it explicitly.
Prompt for untagged resources. Define your required tags (env, owner, cost-center) in the system prompt. The model will flag resources missing any of them.
Prompt for overly permissive IAM. Wildcard actions on wildcard resources. Effect: Allow on *. The AI is genuinely good at spotting these patterns across multiple policy documents simultaneously.
Generate a JSON plan and feed it to the AI. Run terraform plan -out=tfplan followed by terraform show -json tfplan. This gives the model the actual planned state, not just HCL syntax — which is a fundamentally different and more accurate input.
Extract only changed resources before sending to the AI. A full JSON plan for 300 resources hits 80k–110k tokens. GPT-4o's context window is 128k tokens — you'll overflow it on large repos. Filter to changed resources only using jq.
Ask the AI to produce a risk score. LOW, MEDIUM, HIGH, CRITICAL. This gives reviewers a fast signal before they read the full comment. It also makes the output machine-parseable if you want to auto-block CRITICAL findings later.
Flag destructive changes explicitly. Use the jq filter below to isolate deletes and replacements. Send this subset to the AI with elevated attention in the prompt. A delete on a database should always surface as CRITICAL.
Verify remote backend configuration is present. No local .tfstate files committed. State locking enabled via DynamoDB. Backend config not hardcoded with account IDs.
Check for state lock contention risk. Running terraform plan without -lock=false in CI causes lock contention when multiple PRs run simultaneously. This one has burned us twice.
Pin provider versions. version = "~> 5.0" in the required_providers block. AI models trained before mid-2023 may suggest deprecated constraint syntax — validate suggestions against the official Terraform provider requirements docs.

Here's the GitHub Actions workflow that runs this entire checklist. It handles tflint, checkov, plan generation, resource filtering, and the OpenAI API call in a single job:

# .github/workflows/terraform-ai-review.yml
# Requires: OPENAI_API_KEY secret, AWS credentials for plan, terraform 1.5+

name: Terraform AI Code Review

on:
  pull_request:
    paths:
      - '**.tf'
      - '**.tfvars'

jobs:
  ai-review:
    runs-on: ubuntu-22.04
    permissions:
      pull-requests: write   # needed to post review comments
      contents: read

    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Setup Terraform 1.7.x
        uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: "1.7.5"
          terraform_wrapper: false   # wrapper breaks JSON output parsing

      - name: Setup tflint v0.50.3
        run: |
          curl -s https://raw.githubusercontent.com/terraform-linters/tflint/master/install_linux.sh | \
            TFLINT_VERSION=v0.50.3 bash

      - name: Run tflint (JSON output for downstream parsing)
        run: |
          tflint --format=json --recursive > tflint-results.json || true
          # exit code 2 = error, exit code 1 = lint violations — handle separately
          EXIT=$?; if [ $EXIT -eq 2 ]; then echo "tflint tool error" && exit 2; fi

      - name: Run checkov on changed files only
        run: |
          pip install checkov==3.2.0 --quiet
          # get list of changed .tf files from git diff
          CHANGED=$(git diff --name-only origin/${{ github.base_ref }}...HEAD | grep '\.tf$' | tr '\n' ',')
          checkov -f "$CHANGED" \
            --output json \
            --compact \
            --skip-check CKV2_AWS_5 \
            > checkov-results.json || true

      - name: Terraform Init + Plan (generate JSON plan)
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        run: |
          terraform init -input=false -backend-config="key=pr-${{ github.event.pull_request.number }}.tfstate"
          terraform plan -lock=false -input=false -out=tfplan
          terraform show -json tfplan > tfplan.json

      - name: Extract changed resources only (reduce token usage)
        run: |
          jq '[.resource_changes[] | select(.change.actions != ["no-op"]) |
            {address, actions: .change.actions, before: .change.before, after: .change.after}]' \
            tfplan.json > tfplan-diff.json
          # log token estimate: ~4 chars per token
          CHARS=$(wc -c < tfplan-diff.json)
          echo "Estimated tokens: $((CHARS / 4))"

      - name: Send to OpenAI for review + post PR comment
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          PR_NUMBER: ${{ github.event.pull_request.number }}
          REPO: ${{ github.repository }}
        run: |
          python3 .github/scripts/ai_review.py \
            --plan tfplan-diff.json \
            --tflint tflint-results.json \
            --checkov checkov-results.json \
            --pr "$PR_NUMBER" \
            --repo "$REPO"

And here's the Python script that calls GPT-4o and posts the structured review comment back to the PR. It includes exponential backoff via tenacity to handle OpenAI rate limits — tier-1 accounts cap at 500 RPM on GPT-4o, which a busy monorepo with 20 simultaneous PRs will absolutely hit:

# .github/scripts/ai_review.py
# Posts structured AI review comment to GitHub PR
# Requires: openai>=1.14.0, requests, tenacity

import argparse, json, os, sys
import openai
import requests
from tenacity import retry, stop_after_attempt, wait_exponential

SYSTEM_PROMPT = """You are a senior DevOps engineer reviewing Terraform infrastructure changes.
Analyze the provided plan diff and linter results. Return a structured review with:
1. RISK_LEVEL: LOW | MEDIUM | HIGH | CRITICAL
2. DESTRUCTIVE_CHANGES: list any resource deletions or replacements
3. SECURITY_FINDINGS: IAM over-permissions, open security groups, missing encryption
4. MISSING_TAGS: resources lacking required tags (env, owner, cost-center)
5. RECOMMENDATIONS: max 5 bullet points, actionable only
Do NOT hallucinate resource attributes. If unsure, say 'verify in provider docs'."""

@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=4, max=30))
def call_openai(plan_content: str, lint_content: str) -> str:
    client = openai.OpenAI(api_key=os.environ["OPENAI_API_KEY"])
    response = client.chat.completions.create(
        model="gpt-4o",
        max_tokens=1500,
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": f"PLAN DIFF:\n{plan_content}\n\nLINTER FINDINGS:\n{lint_content}"}
        ]
    )
    return response.choices[0].message.content

def post_pr_comment(repo: str, pr: str, body: str):
    url = f"https://api.github.com/repos/{repo}/issues/{pr}/comments"
    headers = {
        "Authorization": f"Bearer {os.environ['GH_TOKEN']}",
        "Accept": "application/vnd.github+json"
    }
    resp = requests.post(url, json={"body": f"## 🤖 AI Terraform Review\n\n{body}"}, headers=headers)
    resp.raise_for_status()

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--plan"); parser.add_argument("--tflint")
    parser.add_argument("--checkov"); parser.add_argument("--pr"); parser.add_argument("--repo")
    args = parser.parse_args()

    plan = json.load(open(args.plan))
    tflint = json.load(open(args.tflint))

    # Truncate if over ~80k tokens (~320k chars) — keep first 300k chars
    plan_str = json.dumps(plan)[:300_000]
    lint_str = json.dumps(tflint)[:20_000]

    review = call_openai(plan_str, lint_str)
    post_pr_comment(args.repo, args.pr, review)
    print("AI review posted successfully")

if __name__ == "__main__":
    main()

Commonly Missed Items

These are the gaps that bite teams six months after they think the setup is complete.

AI models hallucinate provider attributes. This is the biggest one. AWS provider v5.x introduced breaking changes — renamed attributes, removed arguments, new required fields. Models trained before mid-2023 don't know about these. When the AI tells you to add acl = "private" to an S3 bucket resource, it's wrong — that argument was removed in AWS provider v4.0. The fix: always pin provider versions with ~> 5.0 in your required_providers block, and instruct the AI in the system prompt to validate suggestions against the actual plan output, not just HCL syntax. If the plan succeeds, the attributes are valid. If the AI contradicts the plan, trust the plan.

Watch out for: .tfvars and .auto.tfvars files being excluded from AI context. This is where secrets and environment-specific values live. Database passwords passed as variables, account IDs, CIDR blocks that reveal internal network topology. These files are often excluded from the AI context window either by accident (the workflow only globs *.tf) or intentionally (to avoid sending secrets to the API). The problem is that the AI then reviews IAM policies and security group rules without knowing the actual values they'll be populated with. The solution is to redact sensitive values with sed or sops --decrypt | sed 's/=.*/=REDACTED/' before including them in the prompt, not to exclude the files entirely.

Module version pinning is silently skipped by most AI tools. A Terraform module sourced as source = "terraform-aws-modules/vpc/aws" without a version constraint will pull whatever is latest at plan time. This causes silent drift between environments. Static tools like tflint and checkov don't flag this by default. The AI won't flag it either unless you explicitly include "check for unpinned module versions" in your system prompt. Add it. I've seen a VPC module minor version bump change subnet behavior in production because nobody pinned it.

Watch out for: tfsec references in older pipelines. tfsec was merged into trivy as of v0.21. If your CI image still references the tfsec binary directly and it's been upgraded to a newer image, you'll get a silent no-op — the command exits 0 because the binary isn't found and the shell swallows the error. Check your CI logs for actual tfsec output, not just a green checkmark.

The checkov skip annotation is itself a finding. When engineers add #checkov:skip=CKV_AWS_20:reason to suppress a finding, some AI configurations will flag the skip annotation as a new security concern without reading the reason. This creates noise. Include in your system prompt: "Treat checkov skip annotations as accepted risks if a reason is provided. Do not re-flag them as findings."

For more on securing your Terraform CI pipelines and managing infrastructure secrets, see the related posts at kuryzhev.cloud.

Automation Ideas

Once the checklist is solid, the next step is removing every manual touch from the process.

GitHub Actions with CodeRabbit or a custom OpenAI script. The workflow above handles the custom script path. For CodeRabbit, the setup is simpler — install the GitHub App, add a .coderabbit.yaml config file, and it hooks into PR events automatically. CodeRabbit free tier covers 200 files per month; paid starts at $12/user/month. For most teams under 10 engineers, the free tier is sufficient for Terraform-only reviews. For larger teams or monorepos, the custom OpenAI script gives you more control over the system prompt and cost per call.

Atlantis pre/post-plan hooks. Atlantis added pre_workflow_hooks in v0.19.0. You can wire checkov as a pre-plan hook and the Python AI review script as a post-plan hook. The Atlantis config in atlantis.yaml at repo root controls this. Post-plan hooks receive the plan output path as an environment variable — pipe it through jq to extract changed resources, then pass to the AI script. This approach keeps everything server-side and avoids GitHub Actions runner costs for plan generation.

Cost management for large plans. OpenAI API calls on large JSON plan files with 500+ resources cost between $0.08 and $0.40 per PR. That adds up fast on active repos. Two mitigations: first, use the jq filter to send only changed resources, not the full plan. Second, cache plan outputs by content hash — if the plan JSON hasn't changed between two PR pushes (force-push with no IaC changes), skip the API call entirely. Implement this with a sha256sum check in your workflow before the OpenAI step.

Security note on API calls. Sending your infrastructure topology to a third-party LLM is a real concern for regulated environments. Your plan JSON contains resource names, CIDR blocks, account IDs, and IAM policy structures. For HIPAA or PCI environments, use a self-hosted model — Ollama with CodeLlama 34B works reasonably well — or Azure OpenAI with data residency guarantees. Store OPENAI_API_KEY and REVIEWPAD_API_KEY as encrypted CI secrets, not repo variables. Repo variables are readable by all contributors in GitHub Actions. That's not a theoretical risk — it's a common misconfiguration.

Calibration period before blocking. Run AI review as a non-blocking advisory check for two weeks. Export the AI findings to a spreadsheet. Categorize each as true positive, false positive, or noise. Tune your system prompt to eliminate the false positive patterns. Then promote to blocking. Skipping the calibration period is how you end up with a team that ignores AI review comments because they've been burned by too many false alarms. The 45–90 second CI overhead is worth it — but only if the signal is trusted.

LLM Log Triage With Loki and CloudWatch Insights

Oleksandr Kuryzhev — Thu, 11 Jun 2026 08:31:39 +0000

Liquid syntax error: Variable '{{app="{app}' was not properly terminated with regexp: /\}\}/

Jenkins Pipeline: Build, Test, and Deploy to AWS EC2 with ECR

Oleksandr Kuryzhev — Wed, 10 Jun 2026 07:02:50 +0000

Originally published on kuryzhev.cloud

If your Jenkins pipeline deploy to AWS is still using image:latest and a hardcoded ECR password stored as a plain Username/Password credential, you are one 12-hour token expiry away from a broken production deploy at exactly the wrong moment. I have seen this happen twice on teams I joined mid-incident. The fix is not complicated — but it requires wiring up credentials, image tagging, and the deploy stage in the right order, which most tutorials skip entirely.

This post walks through a complete, production-hardened setup: GitHub push triggers Jenkins, Jenkins builds and pushes a versioned Docker image to ECR, runs unit tests inside the container, then deploys to an EC2 instance over SSH. No Ansible, no manual steps, no latest tags in production.

The Scenario

We have a Node.js microservice living on GitHub. Every push to main should automatically build a Docker image, run the test suite, push the image to Amazon ECR, and deploy it to a target EC2 instance — zero manual SSH required. The same pipeline handles staging and production, gated by a parameter so nobody accidentally ships to prod from a feature branch.

The full pipeline shape looks like this: Source → Build → Test → Push to ECR → Deploy to EC2 via SSH. When everything works correctly, you get a green Jenkins build, a live endpoint returning HTTP 200, and a deployment audit trail in the Jenkins console that shows exactly which image tag is running in which environment. That audit trail matters more than most people realize — it is the first thing you reach for during a 2 AM incident.

We are not doing anything exotic here. No ECS, no Kubernetes, no blue-green infrastructure. Just a real-world pattern that a lot of teams actually run, done properly. Once this is solid, migrating to ECS update-service for zero-downtime rolling deploys is a natural next step — and the Jenkinsfile barely changes.

Prerequisites

Before writing a single line of pipeline code, make sure every dependency is in place at the right version. Environment drift mid-tutorial wastes hours.

Jenkins side: Jenkins 2.440 LTS running on a dedicated EC2 instance — t3.small minimum, t3.medium recommended if you are running Docker builds on the same host. Required plugins: Pipeline, Git, AWS Credentials, Docker Pipeline (563.vd5d2e5c4007f), SSH Agent (295.v9ca_a_1c7cc3a_a_). Install all of them via Manage Jenkins → Plugin Manager → Available. If the Docker Pipeline plugin version is older than 563.x, the ecr: credential helper prefix will not be recognized.

AWS side: An ECR repository already created, and an IAM user (or preferably an EC2 instance profile on the Jenkins host — more on that below) with permissions scoped to ECR push operations and EC2 describe. Store the IAM access key in Jenkins as a credential of type AWS Credentials with the ID aws-credentials. The ECR URI format is <account_id>.dkr.ecr.<region>.amazonaws.com/<repo-name> — the region must match the AWS region configured on the Jenkins agent or the ecr get-login-password call will silently fail with a generic 401.

Local and target tooling: AWS CLI v2.15+ (v1 uses the deprecated aws ecr get-login command which pipes differently and will break the login step), Docker 25+, Git 2.40+. The target EC2 runs Amazon Linux 2023 with the Docker daemon already running and ec2-user in the docker group. Also confirm the jenkins OS user on the Jenkins host is in the docker group — if not, you will hit Cannot connect to the Docker daemon at unix:///var/run/docker.sock immediately. Fix: usermod -aG docker jenkins, then restart Jenkins.

See the official Jenkins Pipeline with Docker documentation for plugin compatibility details.

Step 1 — Configure Jenkins Credentials and ECR Access

This is the step most tutorials either skip or get wrong. Wire up secrets before touching the Jenkinsfile.

Navigate to Manage Jenkins → Credentials → System → Global credentials and add three entries:

The AWS IAM key pair — type: AWS Credentials, ID: aws-credentials
The EC2 SSH private key — type: SSH Username with private key, ID: ec2-ssh-key, username: ec2-user
A GitHub personal access token — type: Username with password, ID: github-token (used by the Multibranch Pipeline source)

Now set up the ECR credential helper on the Jenkins agent. Install amazon-ecr-credential-helper and add the following to /root/.docker/config.json (or /var/lib/jenkins/.docker/config.json depending on how Jenkins runs):

{
  "credsStore": "ecr-login"
}

This is critical. Without the credential helper, you have two bad options: store the ECR password as a plain credential (it expires after 12 hours, causing mysterious 401 errors mid-pipeline) or call aws ecr get-login-password manually in every stage. The ecr: prefix in docker.withRegistry() handles token refresh automatically when the helper is configured.

Watch out for this: The most common mistake I see is teams storing ECR credentials as Username/Password type in Jenkins. It works fine for the first 12 hours, then starts failing overnight with no clear error message. Use the credential helper. It takes five minutes to set up and you never touch it again.

Step 2 — Write the Jenkinsfile

The Jenkinsfile lives at the repo root. Jenkins detects it automatically when the job type is Multibranch Pipeline or Pipeline from SCM. Here is the full declarative pipeline — I will walk through each block below.

// Jenkinsfile — Declarative pipeline: build, test, deploy to AWS EC2 via ECR
// Assumes: Jenkins 2.440+, Docker Pipeline plugin, SSH Agent plugin, AWS Credentials plugin

pipeline {
    agent any  // replace with a labeled agent node in production: agent { label 'docker-agent' }

    environment {
        AWS_REGION      = 'us-east-1'
        ECR_ACCOUNT_ID  = '123456789012'
        ECR_REPO        = 'my-app'
        IMAGE_TAG       = "${BUILD_NUMBER}-${GIT_COMMIT[0..7]}"  // e.g. 42-a3f9c12
        ECR_URI         = "${ECR_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/${ECR_REPO}"
        FULL_IMAGE      = "${ECR_URI}:${IMAGE_TAG}"
    }

    parameters {
        choice(
            name: 'DEPLOY_ENV',
            choices: ['staging', 'production'],
            description: 'Target deployment environment'
        )
    }

    stages {

        stage('Checkout') {
            steps {
                // Checks out the triggering branch; Multibranch Pipeline sets GIT_COMMIT automatically
                checkout scm
            }
        }

        stage('Build & Push Image') {
            steps {
                script {
                    // ecr: prefix triggers amazon-ecr-credential-helper — no password stored in Jenkins
                    docker.withRegistry("https://${ECR_URI}", "ecr:${AWS_REGION}:aws-credentials") {
                        def appImage = docker.build("${FULL_IMAGE}", "--no-cache .")
                        appImage.push()
                        // Also push a human-readable env tag for traceability in ECR console
                        appImage.push("${DEPLOY_ENV}-latest")
                    }
                }
            }
        }

        stage('Run Tests') {
            steps {
                // Run tests inside the freshly built image — same environment as production
                sh """
                    docker run --rm \\
                        -e NODE_ENV=test \\
                        --name test-runner \\
                        ${FULL_IMAGE} \\
                        npm test -- --reporters=jest-junit --outputFile=test-results/results.xml
                """
            }
            post {
                always {
                    // Publish JUnit results regardless of pass/fail so Jenkins tracks test trends
                    junit allowEmptyResults: true, testResults: 'test-results/**/*.xml'
                }
            }
        }

        stage('Deploy') {
            // Gate: only deploy to production from the main branch
            when {
                expression {
                    return params.DEPLOY_ENV == 'staging' || env.BRANCH_NAME == 'main'
                }
            }
            environment {
                // Resolve the correct EC2 host based on chosen environment
                EC2_HOST = "${params.DEPLOY_ENV == 'production' ? env.PROD_HOST : env.STAGING_HOST}"
            }
            steps {
                // sshagent injects the private key for the duration of this block only
                sshagent(credentials: ['ec2-ssh-key']) {
                    sh """
                        ssh -o StrictHostKeyChecking=no ec2-user@${EC2_HOST} '
                            aws ecr get-login-password --region ${AWS_REGION} \\
                                | docker login --username AWS --password-stdin ${ECR_URI} && \\
                            docker pull ${FULL_IMAGE} && \\
                            docker stop app 2>/dev/null || true && \\
                            docker rm   app 2>/dev/null || true && \\
                            docker run -d --name app --restart unless-stopped \\
                                -p 80:3000 \\
                                -e NODE_ENV=${DEPLOY_ENV} \\
                                ${FULL_IMAGE}
                        '
                    """
                }
            }
        }
    }

    post {
        failure {
            // Notify team on any failure — replace with slackSend or emailext as needed
            echo "Pipeline FAILED for ${FULL_IMAGE} targeting ${params.DEPLOY_ENV}"
        }
        always {
            // Prevent disk exhaustion on the Jenkins agent over repeated builds
            cleanWs()
        }
    }
}

A few things worth calling out explicitly. The IMAGE_TAG uses ${BUILD_NUMBER}-${GIT_COMMIT[0..7]} — something like 42-a3f9c12. This is intentional. Using latest in the deploy command is the single most common mistake that causes "works on my machine" deploys. Jenkins pulls the cached local latest instead of the newly pushed image, and you spend 45 minutes wondering why your code change is not live.

The docker.withRegistry() call requires the full https:// prefix. Omitting it causes unauthorized: authentication required with no further detail — one of the more frustrating silent failures in the Docker Pipeline plugin.

The cleanWs() post step is not optional on small instances. On a t3.small with an 8 GB root volume, Docker layer caches from repeated builds fill the disk in roughly 20 builds. I stopped using shared Jenkins masters for Docker builds entirely after we killed a production deploy because the agent ran out of disk space at the image push step.

Step 3 — Scope the IAM Policy

The pipeline needs AWS permissions to push to ECR. Here is the minimal IAM inline policy — scope it to the single repository, not the entire account.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ECRAuthToken",
            "Effect": "Allow",
            "Action": "ecr:GetAuthorizationToken",
            "Resource": "*"
        },
        {
            "Sid": "ECRPushToSpecificRepo",
            "Effect": "Allow",
            "Action": [
                "ecr:BatchCheckLayerAvailability",
                "ecr:CompleteLayerUpload",
                "ecr:InitiateLayerUpload",
                "ecr:PutImage",
                "ecr:UploadLayerPart",
                "ecr:DescribeImages",
                "ecr:ListImages"
            ],
            "Resource": "arn:aws:ecr:us-east-1:123456789012:repository/my-app"
        },
        {
            "Sid": "EC2DescribeForHealthChecks",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "ec2:DescribeInstanceStatus"
            ],
            "Resource": "*"
        }
    ]
}

Note that ecr:GetAuthorizationToken cannot be scoped to a specific resource — that is an AWS API constraint, not a mistake in the policy. Everything else is locked to the single ECR repository.

Security consideration worth repeating: The IAM user approach works, but it means long-lived access keys sitting in Jenkins credentials. The better path is to attach this policy to an EC2 instance profile on the Jenkins host. The Jenkins AWS Credentials plugin picks up the instance metadata automatically — no key ID, no secret, no rotation burden. I made this switch on every team I have worked with after the second time a leaked key caused an incident. See the AWS IAM roles for EC2 documentation for setup details.

Also: StrictHostKeyChecking=no in the SSH command is acceptable for known internal hosts, but if your security posture requires it, replace it with a known_hosts file pre-populated with the EC2 host fingerprint. On a Multibranch Pipeline, store it as a Jenkins file credential and copy it into ~/.ssh/known_hosts before the SSH step.

Verify and Test

A green Jenkins build is not proof the pipeline works. Here is how we actually verify the full end-to-end flow.

Trigger a build manually first via Build with Parameters, choosing staging. Watch the Console Output in real time. You are looking for two specific lines: Login Succeeded in the Build & Push stage, and the new container ID printed by docker run in the Deploy stage. If the login line is missing, the credential helper is not configured correctly on the agent.

Once the build is green, SSH into the target EC2 and run this to confirm the right image is actually running:

# Confirm the new versioned image tag is running, not a stale container
docker ps --format "table {{.Image}}\t{{.Status}}\t{{.Ports}}"

You should see your versioned tag — something like 123456789012.dkr.ecr.us-east-1.amazonaws.com/my-app:42-a3f9c12 — with status Up X seconds and port 0.0.0.0:80->3000/tcp. If you see my-app:latest, the deploy command is using the wrong image reference.

Hit the health endpoint directly:

curl -I http://<ec2-public-ip>/health

Expect HTTP/1.1 200 OK. Then do the real gate test: intentionally break a unit test, push to main, and confirm the pipeline fails at the Run Tests stage without reaching Deploy. This is the proof the quality gate actually works — not just that the happy path is green. A lot of teams skip this step and find out the hard way that their test stage was misconfigured to always exit 0.

Check the Jenkins workspace path at /var/lib/jenkins/workspace/<job-name> after a few builds to confirm cleanWs() is doing its job. The directory should be empty or absent between runs.

What we have built here is a fully automated Jenkins pipeline deploy to AWS: GitHub push triggers a versioned Docker build, tests run inside the same image that ships to production, the image lands in ECR with a traceable tag, and the EC2 deploy happens over SSH with credentials scoped correctly at every stage. Slack alerting fires on failure, and the workspace cleans itself up. The natural next steps from here are migrating the deploy stage from direct SSH to ECS update-service for zero-downtime rolling deploys, adding a SonarQube stage for static analysis between Build and Test, and replacing the IAM user entirely with an EC2 instance profile on the Jenkins host to eliminate long-lived key management. Each of those is a one-stage change to the Jenkinsfile — which is exactly the point of building the foundation right the first time. More CI/CD patterns we use in production are documented at kuryzhev.cloud.

How to Build a Jenkins Pipeline That Deploys to AWS ECS

Oleksandr Kuryzhev — Tue, 09 Jun 2026 09:49:24 +0000

Originally published on kuryzhev.cloud

Most Jenkins pipelines that deploy to AWS ECS work fine in a demo and silently leak credentials, orphan ECR images, and block build agents for hours the moment they hit real production load. I've inherited three of these pipelines in the last two years. The problems are always the same. The fixes are not complicated — but you have to understand why the mistakes happen in the first place.

This is a deep-dive into how Jenkins pipelines actually work internally, where teams consistently get the architecture wrong, and what a production-grade Jenkinsfile looks like when you need reliable build, test, and AWS deploy stages without the hidden costs.

What a Jenkins Pipeline Actually Does End-to-End

A Jenkinsfile is not a shell script with YAML on top. It is a declarative DSL that Jenkins compiles into a pipeline execution graph — closer to a DAG than a linear sequence of commands. Understanding this distinction matters because it changes how you reason about failures, retries, and resource allocation.

The three core primitives — agent, stage, and steps — have completely separate lifecycles. An agent directive provisions a workspace and an executor. A stage is a logical grouping that appears in the UI and can carry its own agent declaration. steps are the actual commands that run inside that agent context. When you declare agent none at the top level and then specify an agent per stage, Jenkins allocates and releases executors independently for each stage. This is intentional isolation — and most teams skip it entirely.

Environment variables declared in the global environment block propagate into every stage automatically. Variables declared inside a stage are scoped to that stage only. The SCM checkout step that Jenkins runs automatically on the default agent is not the same as running git clone manually — it sets GIT_COMMIT, GIT_BRANCH, and other built-in variables that you can reference downstream.

One thing that trips people up constantly: mixing declarative and scripted pipeline syntax. You can embed a script {} block inside declarative stages for Groovy logic, but you cannot use declarative directives like when or post inside a fully scripted pipeline. Mixing them in the wrong direction causes silent failures where Jenkins simply skips the directive without throwing an error. Jenkins LTS 2.440.3 is the current stable release — pipelines written for 2.3xx may fail on agent directive syntax changes, so check your version before debugging mysterious parse errors.

How Teams Use Jenkins Pipelines Wrong

I've seen the same three mistakes across every team that hasn't gone through a pipeline audit. Each one is invisible until it causes an incident.

Mistake 1: Hardcoding AWS credentials as environment variables. This pattern appears in a surprising number of production Jenkinsfiles:

environment {
    AWS_ACCESS_KEY_ID = credentials('aws-key')  // WRONG — exposes key ID in UI
}

Even when using the credentials() helper, setting credentials directly in the environment block exposes the key ID in the Jenkins "Environment Variables" panel, which is visible to every user with read access to the job. The correct approach uses withCredentials with AmazonWebServicesCredentialsBinding — scoped to the exact steps that need AWS access. Watch out for this: the Credentials Binding Plugin version 657.v2b_19db_7d6d6d or later is required for the AmazonWebServicesCredentialsBinding class. Earlier versions fail silently and fall back to injecting the raw string.

Mistake 2: Running all stages on the master node. When every stage uses agent any without specifying a label or Docker image, Jenkins defaults to the controller. On a team running 10 microservices, this turns a 4-minute build into a 22-minute queue. The controller is not an execution node — it is an orchestration node. Running builds on it also means a runaway build can starve the Jenkins UI of threads.

Mistake 3: Skipping the post block entirely. Without a post { always { cleanWs() } } block, workspaces accumulate on persistent agents. I've seen a 200GB disk fill up over a weekend from a pipeline that ran every 15 minutes and never cleaned up. Beyond disk exhaustion, skipping the post block means failed deployments leave ECR images untagged and S3 artifacts orphaned — storage costs compound at $0.10/GB/month with no upper bound. Jenkins does not clean workspaces automatically between builds. This is not a default you can rely on.

There's also a concurrency issue worth calling out separately: not setting disableConcurrentBuilds() in the options block. Concurrent deploys to the same ECS service cause task definition version conflicts and rollback failures that are genuinely hard to debug after the fact.

The Correct Approach: Jenkinsfile for Build, Test, and AWS Deploy

Here is the full production-grade Jenkinsfile we use for a Java/Maven service deploying to AWS ECS. Every design decision is intentional — I'll explain the non-obvious ones inline.

This pipeline uses per-stage Docker agents to isolate environments, stash/unstash to pass artifacts between stages, and withCredentials scoped tightly around AWS CLI calls. The IMAGE_TAG uses the short Git SHA for traceability — you can look at any running ECS task and trace it back to an exact commit.

// Jenkinsfile — declarative pipeline for build, test, and deploy to AWS ECS
// Requires: Docker Pipeline plugin, AWS Credentials Binding plugin, JUnit plugin
// Jenkins LTS 2.440.3+, AWS CLI v2 pre-installed on agent image

pipeline {
    agent none  // No global agent — each stage declares its own to isolate environments

    options {
        disableConcurrentBuilds()           // Prevent parallel deploys to same ECS service
        timeout(time: 30, unit: 'MINUTES')  // Kill runaway builds; protects agent pool
        buildDiscarder(logRotator(numToKeepStr: '20'))
    }

    environment {
        AWS_REGION      = 'us-east-1'
        ECR_REGISTRY    = '123456789012.dkr.ecr.us-east-1.amazonaws.com'
        ECR_REPO        = 'my-app'
        IMAGE_TAG       = "${env.GIT_COMMIT[0..7]}"  // Short SHA for traceability
        ECS_CLUSTER     = 'prod-cluster'
        ECS_SERVICE     = 'my-app-service'
    }

    stages {
        stage('Build') {
            agent {
                docker {
                    image 'maven:3.9.6-eclipse-temurin-21'
                    args  '-v $HOME/.m2:/root/.m2'  // Cache local Maven repo across builds
                }
            }
            steps {
                sh 'mvn clean package -DskipTests -q'  // Tests run in dedicated stage
                stash name: 'build-artifact', includes: 'target/*.jar'
            }
        }

        stage('Test') {
            agent {
                docker {
                    image 'maven:3.9.6-eclipse-temurin-21'
                    args  '-v $HOME/.m2:/root/.m2'
                }
            }
            steps {
                unstash 'build-artifact'
                sh 'mvn test -q'
            }
            post {
                always {
                    // Publish results even on test failure so failures are visible in UI
                    junit 'target/surefire-reports/**/*.xml'
                }
            }
        }

        stage('Docker Build & Push') {
            agent { label 'docker-agent' }  // Agent with Docker daemon access
            steps {
                unstash 'build-artifact'
                withCredentials([[
                    $class:             'AmazonWebServicesCredentialsBinding',
                    credentialsId:      'aws-ecr-credentials',  // Stored in Jenkins Credentials
                    accessKeyVariable:  'AWS_ACCESS_KEY_ID',
                    secretKeyVariable:  'AWS_SECRET_ACCESS_KEY'
                ]]) {
                    sh """
                        # Authenticate to ECR — token valid 12 hours
                        aws ecr get-login-password --region ${AWS_REGION} \
                          | docker login --username AWS --password-stdin ${ECR_REGISTRY}

                        # Build with BuildKit for parallel layer execution
                        DOCKER_BUILDKIT=1 docker build \
                          --cache-from ${ECR_REGISTRY}/${ECR_REPO}:latest \
                          -t ${ECR_REGISTRY}/${ECR_REPO}:${IMAGE_TAG} \
                          -t ${ECR_REGISTRY}/${ECR_REPO}:latest .

                        docker push ${ECR_REGISTRY}/${ECR_REPO}:${IMAGE_TAG}
                        docker push ${ECR_REGISTRY}/${ECR_REPO}:latest
                    """
                }
            }
        }

        stage('Deploy to ECS') {
            agent { label 'docker-agent' }
            when {
                branch 'main'  // Only deploy from main branch; feature branches stop here
            }
            steps {
                // Manual approval gate — times out after 15 min to release the executor
                timeout(time: 15, unit: 'MINUTES') {
                    input message: "Deploy ${IMAGE_TAG} to production ECS?", ok: 'Deploy'
                }
                withCredentials([[
                    $class:             'AmazonWebServicesCredentialsBinding',
                    credentialsId:      'aws-ecr-credentials',
                    accessKeyVariable:  'AWS_ACCESS_KEY_ID',
                    secretKeyVariable:  'AWS_SECRET_ACCESS_KEY'
                ]]) {
                    sh """
                        aws ecs update-service \
                          --region ${AWS_REGION} \
                          --cluster ${ECS_CLUSTER} \
                          --service ${ECS_SERVICE} \
                          --force-new-deployment
                    """
                }
            }
        }
    }

    post {
        always {
            cleanWs()  // Mandatory: prevents disk exhaustion on persistent agents
        }
        failure {
            echo "Pipeline failed on branch ${env.BRANCH_NAME} at commit ${IMAGE_TAG}"
        }
    }
}

One gotcha worth calling out: the JUnit glob pattern matters. Using target/surefire-reports/**/*.xml scopes the search correctly. If you use **/surefire-reports/**/*.xml without the target/ prefix, Jenkins scans the entire workspace and slows test result parsing by 3–5x on large repos. I made this mistake on a monorepo with 40 modules and spent an afternoon wondering why the post-build step was taking longer than the tests themselves.

Also watch out for the AWS CLI path issue. On Amazon Linux 2, AWS CLI v2 installs to /usr/local/bin/aws. The v1 path is /usr/bin/aws. If your Docker agent image has both installed, the wrong one can silently take precedence depending on PATH ordering — and the ECR login command syntax differs between versions in ways that produce confusing authentication errors rather than a clear "wrong version" message.

Advanced Patterns: Shared Libraries, Approvals, and Multi-Environment Promotion

Once you have one pipeline working correctly, the next problem is copy-paste drift across 20 microservices. Every team I've worked with hits this around service number five or six. The answer is Jenkins Shared Libraries.

A Shared Library lives in a separate Git repository with a specific directory structure: vars/ for global step functions, src/ for Groovy classes, and resources/ for static files. Deviating from this structure causes No such DSL method errors that are not immediately obvious. The library is loaded at the top of any Jenkinsfile with @Library('jenkins-shared-libs@main') _ — the trailing underscore is required and easy to forget.

Here is a real shared library helper we use for ECS deployments. The key addition over the inline approach is the aws ecs wait services-stable call, which blocks until the ECS service reaches steady state or fails the pipeline. Without this, a deployment that pushes a broken image appears successful in Jenkins — ECS just quietly fails the task and rolls back, and you find out from an alert 10 minutes later.

// vars/ecsDeployHelper.groovy — Shared Library example
// Stored in: https://github.com/org/jenkins-shared-libs (loaded via @Library annotation)
// Load in Jenkinsfile with: @Library('jenkins-shared-libs@main') _

/**
 * Deploys a Docker image to an ECS service and waits for stability.
 * Usage: ecsDeployHelper.deploy(cluster: 'prod-cluster', service: 'my-app', region: 'us-east-1')
 */
def deploy(Map config) {
    // Validate required keys — fail fast with a clear message rather than cryptic AWS error
    ['cluster', 'service', 'region'].each { key ->
        if (!config[key]) error("ecsDeployHelper.deploy: missing required param '${key}'")
    }

    echo "Deploying to ECS cluster=${config.cluster} service=${config.service}"

    sh """
        aws ecs update-service \
          --region ${config.region} \
          --cluster ${config.cluster} \
          --service ${config.service} \
          --force-new-deployment

        # Wait up to 10 minutes for service to reach steady state
        # Fails pipeline if deployment does not stabilize — catches bad images early
        aws ecs wait services-stable \
          --region ${config.region} \
          --cluster ${config.cluster} \
          --services ${config.service}
    """

    echo "ECS service ${config.service} reached stable state successfully"
}

return this

For multi-environment promotion, we use a choice parameter — params.ENVIRONMENT set to dev/staging/prod — combined with environment-specific credential blocks. The pattern keeps a single Jenkinsfile per service while allowing environment-specific IAM roles and regions. The IAM role attached to the Jenkins EC2 instance should use sts:AssumeRole to access cross-account resources — never store long-lived access keys on the instance itself. This is not just a best practice; it is the only approach that survives a credential rotation without pipeline downtime.

The input step timeout deserves its own warning. If you write input message: 'Deploy to production?' without wrapping it in timeout(time: 15, unit: 'MINUTES'), the executor thread hangs indefinitely waiting for a human. That executor is blocked. On a small Jenkins instance with two executors, one forgotten approval gate can stall your entire CI system. I stopped using bare input steps entirely after this happened during an on-call weekend.

Performance Notes: Build Time, Agent Cost, and ECR Storage

The performance decisions in a Jenkins pipeline have real dollar amounts attached to them. These are not theoretical optimizations.

Docker layer caching is the single highest-impact change you can make to build time. Without the --cache-from flag pulling a warm cache image from ECR, every build rebuilds all layers from scratch. With it, and with a well-structured Dockerfile that copies dependency files before source code, average build time drops from roughly 8 minutes to around 90 seconds for a typical Java service. The requirement is that you push the :latest tag after every successful build — which the pipeline above does — so the cache is always warm for the next run. Also enable BuildKit: DOCKER_BUILDKIT=1 enables parallel execution of independent RUN instructions. Without it, they execute sequentially and add 2–4 minutes to image build time depending on your Dockerfile structure.

Spot instance agents via the EC2 Fleet Plugin cut compute cost by 60–70% for non-production builds. The configuration that matters most is idleTerminationMinutes: 5. Without it, idle agents accumulate overnight and you discover a surprisingly large EC2 bill the next morning. Five minutes is aggressive but safe for most build patterns — agents provision in under 60 seconds on modern AMIs.

ECR lifecycle policies are not optional if you run pipelines frequently. Every push creates a new image. Without an explicit lifecycle policy, ECR retains every image indefinitely at $0.10/GB/month. For a service with a 500MB image running 20 deploys per day, that compounds quickly. Set a policy that retains the last 30 tagged images and expires all untagged images after 1 day. The AWS CLI command is straightforward:

aws ecr put-lifecycle-policy \
  --repository-name my-app \
  --lifecycle-policy-text '{
    "rules": [
      {
        "rulePriority": 1,
        "description": "Expire untagged images after 1 day",
        "selection": {
          "tagStatus": "untagged",
          "countType": "sinceImagePushed",
          "countUnit": "days",
          "countNumber": 1
        },
        "action": { "type": "expire" }
      },
      {
        "rulePriority": 2,
        "description": "Keep last 30 tagged images",
        "selection": {
          "tagStatus": "tagged",
          "tagPrefixList": ["v"],
          "countType": "imageCountMoreThan",
          "countNumber": 30
        },
        "action": { "type": "expire" }
      }
    ]
  }'

One last performance note: the stash/unstash mechanism has a 100MB default limit. Artifacts larger than this need to go through S3 using the S3 plugin or explicit AWS CLI copy commands. Hitting this limit produces an error that is not always immediately obvious about the cause — hudson.remoting.ChannelClosedException is the error you'll see if the agent JVM also runs out of heap during a large stash operation. Set -Xmx512m in agent JVM arguments via the EC2 plugin configuration if you see this.

For the full Jenkins pipeline deploy AWS reference and related CI/CD patterns, see the kuryzhev.cloud DevOps_DayS archive. The official Jenkins Pipeline Syntax documentation and AWS ECS service update reference are the two external sources worth bookmarking alongside this.

AWS Lambda S3 Trigger Python: 3 Production Mistakes We Fixed

Oleksandr Kuryzhev — Sat, 06 Jun 2026 10:24:03 +0000

Originally published on kuryzhev.cloud

We lost 6 hours of vendor CSV data because our AWS Lambda S3 trigger Python function silently crashed on a test event payload — and we had no DLQ to catch it. Nobody noticed until a vendor called asking why their upload hadn't been processed. That incident kicked off a proper post-mortem, and what we found was embarrassing: three compounding mistakes that any team new to event-driven Lambda patterns can fall into. This is the honest account of what went wrong and exactly how we hardened the pipeline afterward.

Context: Why We Chose Lambda and S3 for This Pipeline

The setup was straightforward on paper. Third-party vendors drop CSV files into an S3 bucket. A Lambda function picks up the S3 event notification, parses the file, and inserts rows into RDS PostgreSQL. We chose Lambda over ECS for a few practical reasons: invocations were sporadic (dozens per day, not continuous), we didn't want to manage a running container 24/7, and the per-invocation billing model made cost predictable at low volume.

The stack: Python 3.11, boto3 1.34.x, Terraform for all infra, and S3 event notifications wired directly to Lambda — not EventBridge, which we weren't using at the time. The team had solid Python experience but relatively limited exposure to Lambda's async invocation model and its failure semantics. We assumed it would behave like a normal function call. It doesn't. That assumption is where the trouble started.

If you want broader context on event-driven failure patterns we've hit before, the kuryzhev.cloud DevOps archive covers several related post-mortems worth reading alongside this one.

Mistake 1: We Trusted the S3 Event Payload Without Validating It

Our original handler accessed the S3 object key like this:

key = event["Records"][0]["s3"]["object"]["key"]

No guard. No validation. Direct dictionary access assuming the payload always matched the documented S3 event schema. That worked fine — until someone clicked "Send test event" from the Lambda console. The S3 test event uses a different schema. It sends a fake key value of AmazonS3ExampleObject and the overall structure is close enough to look real, but critically it doesn't match what a live S3 notification sends. The function threw a KeyError, Lambda logged it to CloudWatch, and then... nothing. The invocation was gone.

Watch out for this: S3 event notifications are asynchronous and do not retry by default on their own — Lambda handles retry behavior, and with no DLQ configured, failed invocations simply disappear. We had no dead-letter queue. We had no alarm on Lambda errors. We had CloudWatch Logs set to "Never expire" with no metric filter watching for ERROR lines. Six hours passed before the vendor noticed.

The second gotcha here: S3 URL-encodes object keys. A filename like vendor upload Q1.csv arrives as vendor+upload+Q1.csv. If you access the key and pass it directly to s3_client.get_object(), you'll get a NoSuchKey error that looks like the file doesn't exist — when it's right there in the bucket. Always call urllib.parse.unquote_plus(key) immediately after extracting the key. We weren't doing that either.

Here's the validated event parsing function we use now. It extracts the bucket and key safely, decodes the URL-encoding, and explicitly rejects console test events before any processing begins:

# lambda_function.py
# Python 3.11 | boto3 1.34.x
# S3-triggered Lambda with structured error handling and DLQ support

import json
import csv
import io
import logging
import urllib.parse
import boto3
from botocore.exceptions import ClientError

logger = logging.getLogger()
logger.setLevel(logging.INFO)

s3_client = boto3.client("s3")


def parse_s3_event(event: dict) -> tuple[str, str]:
    """
    Extract and validate bucket name and object key from S3 event payload.
    URL-decodes the key — critical for filenames with spaces or special chars.
    Raises ValueError on malformed or test event payloads.
    """
    try:
        record = event["Records"][0]
        bucket = record["s3"]["bucket"]["name"]
        # S3 encodes object keys — unquote_plus handles spaces encoded as '+'
        key = urllib.parse.unquote_plus(record["s3"]["object"]["key"])
    except (KeyError, IndexError) as e:
        raise ValueError(f"Malformed S3 event payload: {e}") from e

    # Reject S3 console test events — they send a fake key
    if key == "AmazonS3ExampleObject":
        raise ValueError("Received S3 test event — skipping processing")

    return bucket, key

Mistake 2: Unhandled Exceptions Caused Silent Duplicate Processing

Our original handler raised raw exceptions when something went wrong. The thinking was: Lambda will log it, we'll see it in CloudWatch, and we'll fix it. That reasoning misses something important about how Lambda handles async invocations.

When a Lambda function invoked asynchronously raises an unhandled exception, Lambda retries it. The default is two additional attempts — three total. You can verify this in the AWS Lambda async invocation docs. We didn't know that. So when a malformed CSV arrived and our parser threw an exception, Lambda dutifully tried again. And again. The same broken file was processed three times before the retries exhausted.

The damage: we had no idempotency check. There was no record of which S3 keys had already been processed. The first invocation partially inserted rows before failing mid-loop. The retries inserted some of those rows again. We ended up with duplicate records in RDS that took an afternoon to identify and clean up.

Watch out for this: The distinction between retryable and non-retryable errors matters enormously in async Lambda. A transient ClientError from boto3 (throttling, service unavailable) is worth retrying. A malformed CSV or a validation failure is not — retrying it three times just creates three times the mess.

The fix has two parts. First, structured exception handling in the handler itself: catch ValueError for fatal non-retryable conditions, log it, and return a 200 so Lambda doesn't retry. Re-raise only for genuinely transient errors. Second, set MaximumRetryAttempts to 0 in Terraform for any Lambda handling non-idempotent operations:

def fetch_and_parse_csv(bucket: str, key: str) -> list[dict]:
    """
    Fetch CSV from S3 and parse with stdlib csv — no pandas dependency.
    Raises ClientError for missing objects (non-retryable after DLQ setup).
    """
    try:
        response = s3_client.get_object(Bucket=bucket, Key=key)
    except ClientError as e:
        error_code = e.response["Error"]["Code"]
        if error_code == "NoSuchKey":
            # Fatal — do not retry, let DLQ catch it
            raise ValueError(f"Object not found: s3://{bucket}/{key}") from e
        # Other ClientErrors (throttling, etc.) are retryable — re-raise
        raise

    body = response["Body"].read().decode("utf-8")
    reader = csv.DictReader(io.StringIO(body))
    return list(reader)


def lambda_handler(event: dict, context) -> dict:
    """
    Main Lambda handler. Structured error handling:
    - ValueError → fatal, log and return 200 to prevent retry
    - ClientError (non-NoSuchKey) → re-raise to trigger retry / DLQ
    """
    logger.info("Received event: %s", json.dumps(event))

    try:
        bucket, key = parse_s3_event(event)
        logger.info("Processing s3://%s/%s", bucket, key)

        rows = fetch_and_parse_csv(bucket, key)
        logger.info("Parsed %d rows from %s", len(rows), key)

        # TODO: insert rows into RDS or forward downstream
        # process_rows(rows)

        return {"statusCode": 200, "body": f"Processed {len(rows)} rows"}

    except ValueError as e:
        # Non-retryable: log, do not re-raise (prevents unnecessary retries)
        logger.error("Fatal validation error: %s", str(e))
        return {"statusCode": 400, "body": str(e)}

    except Exception as e:
        # Retryable or unexpected: re-raise so Lambda/DLQ handles it
        logger.exception("Unexpected error processing event")
        raise

Mistake 3: Broad IAM Permissions and a Bloated Deployment Package

This one is actually two mistakes that compounded each other, and I'm grouping them because they both came from the same root cause: we shipped fast and didn't review the details.

The IAM execution role we originally attached had s3:* on *. It passed our security review because the reviewer was focused on the VPC config and missed the policy. AWS Security Hub flagged it later under rule S3.6, but by then it had been in production for three weeks. Least-privilege for an S3-reading Lambda is simple: s3:GetObject scoped to arn:aws:s3:::your-bucket-name/*. Nothing else. If the function never needs to list or write, those permissions shouldn't exist on the role.

The second mistake: we bundled the entire pandas library into the deployment package because one engineer was comfortable with it for CSV parsing. Pandas compressed is roughly 45 MB. Our deployment package hit the size limit warnings and our cold start went from around 800ms to over 4200ms. That's a 5x regression for functionality that Python's stdlib csv module handles in 20 lines with zero dependencies. Lambda has a 250 MB unzipped deployment package limit — we were well under it, but cold start latency scales with package size regardless. We measured this directly: same function logic, stdlib csv vs. pandas. The difference was consistent across 20 test invocations.

I stopped using pandas in Lambda entirely after this. For anything beyond simple parsing — aggregations, complex transforms — I'd rather push that work into a proper compute layer (ECS, Glue) than fight Lambda's package constraints. The AWS Lambda Python packaging docs explain dependency management and layer options if you genuinely need heavy libraries.

What We Do Differently Now

The full Terraform configuration below captures the hardened setup: DLQ attached, retry attempts set to zero, IAM scoped to the specific bucket ARN, and the source_account condition on the Lambda permission resource to prevent confused deputy attacks when S3 is the invoking principal.

# terraform/lambda.tf
# Terraform ~> 5.x | AWS provider ~> 5.x
# Lambda function with scoped IAM, DLQ, and retry config

resource "aws_sqs_queue" "lambda_dlq" {
  name                      = "csv-processor-dlq"
  message_retention_seconds = 1209600 # 14 days
}

resource "aws_lambda_function" "csv_processor" {
  function_name = "csv-processor"
  runtime       = "python3.11"
  handler       = "lambda_function.lambda_handler"
  filename      = "lambda.zip"
  role          = aws_iam_role.lambda_exec.arn
  timeout       = 30
  memory_size   = 256

  # Avoid bundling heavy libs — keep package lean for cold start
  ephemeral_storage {
    size = 512 # MB, default; increase only if /tmp needed
  }

  dead_letter_config {
    target_arn = aws_sqs_queue.lambda_dlq.arn
  }
}

resource "aws_lambda_function_event_invoke_config" "csv_processor" {
  function_name          = aws_lambda_function.csv_processor.function_name
  maximum_retry_attempts = 0 # Disable async retries — we handle idempotency manually
}

# Least-privilege IAM — scoped to specific bucket, not s3:*
resource "aws_iam_role_policy" "lambda_s3_policy" {
  name = "csv-processor-s3-policy"
  role = aws_iam_role.lambda_exec.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect   = "Allow"
        Action   = ["s3:GetObject"]
        Resource = "arn:aws:s3:::your-bucket-name/*" # Scope to bucket, not *
      },
      {
        Effect   = "Allow"
        Action   = ["sqs:SendMessage"]
        Resource = aws_sqs_queue.lambda_dlq.arn
      }
    ]
  })
}

# Allow S3 to invoke Lambda — include SourceAccount to prevent confused deputy
resource "aws_lambda_permission" "allow_s3" {
  statement_id   = "AllowS3Invoke"
  action         = "lambda:InvokeFunction"
  function_name  = aws_lambda_function.csv_processor.function_name
  principal      = "s3.amazonaws.com"
  source_account = var.aws_account_id # Prevents confused deputy attack
  source_arn     = "arn:aws:s3:::your-bucket-name"
}

Beyond the code, we added three process gates that must pass before any async Lambda merges to main. First: the function must be tested with both a real S3 event payload and an S3 console test event — the test event should be handled gracefully, not crash. Second: cold start must be measured and confirmed under 1 second on a clean invocation; if it's over that, the package is too large. Third: a CloudWatch alarm on DLQ message count must exist and be confirmed active. No alarm, no merge. We also set all Lambda log groups to 14-day retention — "Never expire" is not a default we accept anymore after seeing what log storage costs compound to across dozens of functions.

The AWS Lambda S3 trigger Python pattern is genuinely simple when it works. The failure modes are subtle and the consequences — silent data loss, duplicate writes, security exposure — are disproportionately severe for how small the mistakes are. Every one of these was a one-line fix. The cost was not catching them before production.

Docker BuildKit Cache Optimization for Faster CI Pipelines

Oleksandr Kuryzhev — Sat, 06 Jun 2026 06:11:05 +0000

Docker BuildKit cache optimization for faster CI pipelines is one of those infrastructure improvements that pays dividends immediately — every pipeline run after the first one benefits from layers that were already built and pushed. This post walks through the full setup: enabling BuildKit correctly, choosing between inline and registry cache modes, and writing a GitHub Actions workflow that actually reuses layers instead of rebuilding from scratch on every push.

The performance gap between a cold build and a warm cache hit can be dramatic. A Node.js or Python image that takes four minutes to install dependencies cold can complete that same stage in under ten seconds when the cache layer is pulled from a registry. Getting there requires more than just adding a flag — it requires understanding how BuildKit stores, retrieves, and validates cache manifests.

Requirements

Before touching a single YAML file, confirm your environment meets these prerequisites. Missing any one of them is the most common reason cache configurations appear to work but silently miss on every run.

Docker 20.10+ with BuildKit support — either set DOCKER_BUILDKIT=1 as an environment variable or add "features": {"buildkit": true} to /etc/docker/daemon.json on self-hosted runners.
Docker Buildx plugin — required for type=registry and type=gha cache backends. Plain docker build does not support these modes.
GitHub Actions runner on ubuntu-22.04 or later — the docker/setup-buildx-action works reliably on this image.
A container registry with write access — GitHub Container Registry (ghcr.io) works well here because the GITHUB_TOKEN handles authentication without managing separate credentials.
A Dockerfile with dependency installation separated from application code — if your COPY . . instruction appears before RUN pip install or RUN npm ci, every code change invalidates the dependency layer and the cache never helps where it matters most.
Network access from the runner to the registry — this sounds obvious, but using --cache-from without prior registry authentication causes a silent cache miss, not a build failure. You will not see an error; you will just see a slow build. If you are running self-hosted runners in a private network, review the BuildKit cache backend documentation for proxy and authentication considerations before proceeding.

Implementation
There are two cache strategies worth understanding before writing any configuration: inline cache and registry cache. Inline cache embeds cache metadata directly into the image manifest using the BUILDKIT_INLINE_CACHE=1 build argument. It requires no separate cache tag, but it only captures the final stage — useless for multi-stage builds where you want to cache the builder stage independently. Registry cache, by contrast, stores cache data as a separate manifest tag and supports mode=max, which pushes every intermediate layer rather than just the final one.

For most CI pipelines, registry cache with mode=max is the right choice. The difference between mode=min and mode=max is significant: min caches only the final stage output, while max caches every intermediate layer including builder stages, test runners, and dependency installers. If your Dockerfile has a dedicated build stage that compiles binaries, mode=max is what makes that stage reusable across runs.

Here is the complete GitHub Actions workflow implementing registry cache with BuildKit. Note that DOCKER_BUILDKIT: "1" is set at the job-level env block, not inside a single step — this ensures it is available to all steps that invoke Docker, including setup actions.

# .github/workflows/docker-build.yml
name: Docker Build with BuildKit Cache

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

env:
  DOCKER_BUILDKIT: "1"
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

jobs:
  build:
    runs-on: ubuntu-22.04
    permissions:
      contents: read
      packages: write

    steps:
      - name: Checkout source
        uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Log in to GitHub Container Registry
        uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Extract metadata for Docker
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}

      - name: Build and push with registry cache
        uses: docker/build-push-action@v5
        with:
          context: .
          file: ./Dockerfile
          push: ${{ github.event_name != 'pull_request' }}
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          # Pull existing cache layers from registry before building
          cache-from: |
            type=registry,ref=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:cache
          # Push all intermediate layers (mode=max) for maximum cache reuse
          cache-to: |
            type=registry,ref=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:cache,mode=max
          build-args: |
            BUILDKIT_INLINE_CACHE=1

      - name: Verify cache hit in build log
        run: |
          echo "Check above build step output for 'CACHED' layer lines."
          echo "Re-run this workflow without code changes to confirm cache is working."

A few implementation details worth highlighting. The cache tag (:cache) is stored as a separate manifest in the registry — deleting your application image tag does not remove the cache tag. Manage them independently if you have registry retention policies. Also note that pull requests use cache-from but the workflow skips push on PR runs (push: ${{ github.event_name != 'pull_request' }}), which means PR builds read from cache but do not write back. This is intentional — you do not want untested branches polluting the cache used by main.

For teams already using GitHub Actions cache storage rather than a registry, switching to type=gha is straightforward: replace type=registry,ref=... with type=gha in both cache-from and cache-to. The docker/setup-buildx-action handles the necessary token injection automatically. That said, registry cache is more portable and works identically on self-hosted runners without GitHub-specific configuration.

For more on structuring CI pipelines around reusable patterns, see the DevOps_DayS pipeline guides on kuryzhev.cloud, which cover related topics including Jenkins shared library design and GitLab CI artifact passing between stages.

Test the Setup
Verifying that cache is actually being used — not just configured — requires inspecting build output directly. The key signal is the CACHED prefix on layer lines in the build log.

Run the workflow twice without changing any application code. On the second run, add --progress=plain to your build command (or check the raw step output in GitHub Actions) and look for lines like this:

#8 [builder 3/5] RUN pip install --no-cache-dir -r requirements.txt
#8 CACHED

#9 [builder 4/5] COPY src/ /app/src/
#9 CACHED

If you see CACHED on dependency installation layers, the registry cache is working. If every layer shows a timestamp and byte count instead, the cache is being rebuilt. The most common causes are: the registry authentication step running after the buildx setup step (reorder them), the cache tag not yet existing on first run (expected — second run will be warm), or a COPY instruction ordering problem in the Dockerfile invalidating the cache before the expensive layers.

To confirm the cache manifest exists in the registry independently of your application image, run the following against your registry:

# Inspect the cache manifest tag directly
docker manifest inspect ghcr.io/your-org/your-repo:cache

# Expected output includes mediaType for BuildKit cache manifest
# "mediaType": "application/vnd.oci.image.manifest.v1+json"
# with layers referencing cached build stages

If the manifest inspect returns a standard image manifest rather than a BuildKit cache manifest, the cache-to step did not complete successfully — check that the runner has write permissions to the packages scope and that the registry login step preceded the build step in your workflow.

Timing comparison is the most practical validation. Record the build duration from the Actions summary on run one (cold) and run two (warm). A well-structured Dockerfile with stable dependency layers should show a 60–80% reduction in build time on cache hits for dependency-heavy images.

Set DOCKER_BUILDKIT=1 at the job env level in GitHub Actions, not just inside a single step — it must be present for all Docker-related actions in the job.
Use mode=max for multi-stage Dockerfiles; mode=min only caches the final stage and misses most of the value in complex builds.
The registry cache tag (:cache) is independent of your application image tag — manage its lifecycle separately if you have registry cleanup automation.
A missing or failed registry login before cache-from causes a silent miss — verify authentication order in your workflow steps.
Structure your Dockerfile so stable RUN dependency installation steps appear before frequently-changing COPY instructions — no amount of cache configuration compensates for poor layer ordering.
Inline cache (BUILDKIT_INLINE_CACHE=1) is useful as a fallback for single-stage images but should not replace registry cache in production CI pipelines.

More interesting topics I will touch on in my blog.

DEV Community: Oleksandr Kuryzhev

RAG Pipeline for SRE Runbooks: 7 Vector Search Tips That Work

Tip 1: Choose the Right Embedding Model for Runbook Content

Tip 2: Structure Your Vector Store Index Around Incident Taxonomy

Tip 3: Ingest Runbooks from Confluence or Git with a Lightweight Pipeline

Tip 4: Wire the RAG Query into Your Alerting Workflow

Tip 5: Evaluate Retrieval Quality Before You Trust It in Production

Tip 6: Secure the Pipeline — Runbooks Contain Sensitive Operational Detail

Tip 7: Control Costs by Caching Embeddings and Limiting Re-Indexing Frequency

Related

MCP Servers for DevOps: Build vs Pre-Built — What to Choose

When You Face This Choice

Option A — Pre-Built Community MCP Servers

Option B — Custom MCP Servers with the Python or TypeScript SDK

Decision Matrix

My Pick

Related

Auto-Summarize On-Call Incidents with n8n AI Agents

AI Code Review for Terraform PRs: CI Checklist and Automation

Why This Checklist

The AI Code Review Checklist for Terraform PRs

Commonly Missed Items

Automation Ideas

Related

LLM Log Triage With Loki and CloudWatch Insights

Jenkins Pipeline: Build, Test, and Deploy to AWS EC2 with ECR

The Scenario

Prerequisites

Step 1 — Configure Jenkins Credentials and ECR Access

Step 2 — Write the Jenkinsfile

Step 3 — Scope the IAM Policy

Verify and Test

Related

How to Build a Jenkins Pipeline That Deploys to AWS ECS

What a Jenkins Pipeline Actually Does End-to-End

How Teams Use Jenkins Pipelines Wrong

The Correct Approach: Jenkinsfile for Build, Test, and AWS Deploy

Advanced Patterns: Shared Libraries, Approvals, and Multi-Environment Promotion

Performance Notes: Build Time, Agent Cost, and ECR Storage

Related

AWS Lambda S3 Trigger Python: 3 Production Mistakes We Fixed

Context: Why We Chose Lambda and S3 for This Pipeline

Mistake 1: We Trusted the S3 Event Payload Without Validating It

Mistake 2: Unhandled Exceptions Caused Silent Duplicate Processing

Mistake 3: Broad IAM Permissions and a Bloated Deployment Package

What We Do Differently Now

Related

Docker BuildKit Cache Optimization for Faster CI Pipelines