DEV Community: DevOps Start

Mitigate CVE-2026-20245: Cisco SD-WAN Manager Exploit Guide

DevOps Start — Fri, 26 Jun 2026 13:02:06 +0000

This article was originally published on devopsstart.com. It provides a step-by-step guide to detect, block, and patch the critical CVE-2026-20245 remote code execution vulnerability in Cisco SD-WAN Manager.

Introduction

A single unauthenticated API request can give an attacker full control over your Cisco SD-WAN Manager (vManage). If you are running a vulnerable version, you need to act within hours, not days. This guide explains the CVE-2026-20245 remote code execution bug, shows how to detect it, gives a no-patch workaround, and walks through the official fix. You will leave with a repeatable checklist that secures your SD-WAN control plane.

Problem

CVE-2026-20245 is a critical (CVSS 9.8) remote code execution vulnerability in Cisco SD-WAN Manager. The flaw exists in the REST API endpoints that handle device registration and configuration push. An unauthenticated attacker can send specially crafted HTTP requests to the vManage server and execute arbitrary commands with root privileges. This means full device compromise: the attacker can pivot to managed routers, steal configuration secrets, and disrupt WAN traffic. The exploit does not require any prior access.

Root Causes

Three conditions make this exploit possible.

Unpatched software version. Affected releases include Cisco SD-WAN Manager versions 20.3.x before 20.3.4, 20.6.x before 20.6.2, and 20.9.x before 20.9.1. If you are running one of these, the vulnerable code path is present by default.

Management interface exposed to untrusted networks. The API endpoint /dataservice/device/config is reachable on TCP port 443. Many organisations place vManage on a flat management network or expose the web UI to the internet for remote access. A misconfigured firewall or lack of ACLs lets attackers reach the vulnerable service.

Missing authentication enforcement for certain API calls. Some API routes were not properly guarded. The /dataservice/device/config endpoint expects a token, but prior to the patch the token validation was skipped for requests that included a specific Content-Type header. A simple curl command bypasses authentication entirely.

Solution

Follow these steps to mitigate and then patch the vulnerability. The workaround should be applied immediately; schedule the patch window as soon as possible.

Step 1 – Immediate Workaround (Block the Vulnerable Endpoint)

Do not rely on the vManage's own CLI (it does not run a standard IOS configuration system). Instead, implement a firewall rule on the network layer. On the firewall or router that sits in front of vManage, add an access control entry that drops all traffic to the vManage IP on port 443 from untrusted sources. For more precise blocking, configure deep packet inspection to drop HTTP requests with a URI containing /dataservice/device/config. If your firewall does not support L7 inspection, an iptables rule on the vManage Linux host can do the job:

$ ssh admin@vmanage-ip
$ sudo iptables -A INPUT -p tcp --dport 443 -m string --string "/dataservice/device/config" --algo bm -j DROP
$ sudo iptables-save > /etc/iptables/rules.v4

Important: This iptables rule blocks only requests targeting that specific path. All other API calls and web GUI access remain functional. Verify the rule is active: sudo iptables -L INPUT -v -n | grep DROP.

If you cannot apply network-layer filtering immediately, disable all REST API services via the vManage GUI: Administration > Settings > REST API > Disable. This breaks automation workflows, but it is safer than leaving the endpoint open.

Step 2 – Apply the Official Patch

Download the patched version from the Cisco Software Download Center. The recommended versions are:

20.3.4 (or later)
20.6.2 (or later)
20.9.1 (or later)

Procedure:

Backup the current configuration using the GUI: Administration > Maintenance > Backup/Restore. Download a full backup.
Upgrade the SD-WAN Manager first, then the controllers (vBond, vSmart). Use the GUI Administration > Software Upgrade or the CLI:

$ request software install <filename>
$ request software activate
$ reload

After reboot, verify the version:

$ show version
Cisco SD-WAN Manager 20.9.1

Step 3 – Post-Mitigation Verification

Test that the exploit no longer works. From a separate host, run:

$ curl -k -X POST "https://<vmanage-ip>/dataservice/device/config" \
  -H "Content-Type: application/xml" \
  -d '<config><device-ip>10.0.0.1</device-ip></config>'

If the patch is applied, you get a 403 Forbidden or 401 Unauthorized response. A successful 200 response means the endpoint is still accessible – double-check your firewall and patch version.

Also monitor syslog for any unusual connection attempts. If you used the iptables rule, check its counters:

$ sudo iptables -L INPUT -v -n | grep "dataservice"

A non-zero packet count means the vulnerable endpoint was targeted (possibly a live attack).

Prevention

Long term, treat the SD-WAN Manager as a bastion host.

Segment the management network. Place vManage behind a firewall that only allows access from trusted jump boxes. Use Cisco TrustSec or 802.1X for additional network access control.

Enforce RBAC. Ensure only read-only API tokens are used where possible. Assign the netadmin role sparingly.

Enable API rate limiting. In the vManage GUI, go to Administration > Settings > API Rate Limiting and set a low limit (for example, 100 requests per minute).

Integrate with a SIEM. Forward vManage logs to your SIEM and alert on any /dataservice/device/config requests. Add a log-action to your firewall rule to capture hits.

For a broader security posture, review our CVE remediation guides, such as How to Fix CVE-2026-43284: Preventing Dirty Frag Pod Escapes and How to Mitigate Copy Fail (CVE-2026-31431) with Seccomp. These articles apply similar workaround-first, patch-second patterns to infrastructure vulnerabilities.

Act now. A remote root shell in your SD-WAN control plane is one curl request away.

Mastering DevOps Fundamentals: A Practical Guide

DevOps Start — Fri, 26 Jun 2026 08:32:34 +0000

True DevOps mastery isn't about adopting specific tools or methodologies in isolation, but about deeply integrating cultural shifts with practical, often low-tech, engineering discipline.

Why this take

I've seen too many organizations spend fortunes on tools, proclaiming they're "doing DevOps" because they bought a CI/CD platform or deployed Kubernetes, only to find their release cycles are still glacial, incidents are frequent, and teams are burnt out. The mistake is believing that technology alone can fix systemic issues rooted in process, communication, and organizational structure.

Consider a team I worked with that adopted a high-end CI/CD pipeline. They integrated static analysis, unit tests, and automated deployments to Kubernetes v1.28. Sounds great, right? The problem was that their cultural "inner loop" was broken. Developers would push code, the pipeline would fail due to flaky tests or environment mismatches, and instead of fixing the root cause, they'd spend hours manually re-running builds or merging stale branches. The tools allowed them to fail faster and more often, but didn't address the lack of shared ownership over the pipeline's health or the inconsistent local development environments. They were doing "continuous integration" in name, but not in spirit. This led to a significant increase in lead time for changes, the exact opposite of the DevOps promise.

Another common anti-pattern is the "Kubernetes or bust" mentality. A growing startup I advised decided to migrate all their services to Kubernetes v1.29 to "be cloud-native" and "do DevOps." They skipped crucial steps like improving their logging, monitoring, and incident response for simpler, monolithic applications. The result? They suddenly had a massively complex distributed system with no effective way to observe its behavior or debug issues. When a pod started crashing (a classic CrashLoopBackOff scenario, often due to misconfigured liveness probes), developers were lost. They lacked the fundamental observability practices that would have been valuable even with a simple virtual machine deployment. The advanced tooling amplified their lack of basic operational discipline, turning minor issues into major outages. If you're struggling to debug a monolith, a microservice running on Kubernetes will only make things worse. You can get a sense of how complex these issues can be by checking out guides like Fix CrashLoopBackOff in Kubernetes Pods.

Finally, many teams declare they're "shifting left" security by integrating vulnerability scanners into their CI/CD pipeline. This is a good start. However, if they don't simultaneously implement basic supply chain security practices, enforce least privilege, or establish a robust incident response plan, they're only patching a symptom. I've seen teams with pristine SAST reports fall victim to dependency confusion attacks or secret leaks because their fundamental security hygiene was lacking. Shifting left is only effective when paired with a strong security posture across the entire lifecycle, including runtime protection and a clear understanding of the threat model. The most sophisticated security tools won't help if your team uses weak passwords or stores secrets in plaintext in Git.

The strongest counter-argument

The strongest counter-argument is that while cultural shifts are paramount, tools are not merely optional accessories; they are fundamental enablers without which those cultural aspirations often remain just that: aspirations. Without robust tooling, the "culture of automation" becomes a pipe dream, drowned out by manual toil and inconsistent processes. It's difficult to argue for a blameless culture around incident response if every deployment requires 20 manual steps and a prayer.

Tools provide the scaffolding for DevOps practices. Git v2.43, for example, isn't just a version control system; it's the bedrock of collaborative development, enabling practices like pull requests, code reviews, and reproducible builds. Without a solid version control system, concepts like Infrastructure as Code (IaC) or GitOps are simply impossible. Imagine trying to manage Terraform v1.7.0 configuration files or Kubernetes manifests without Git history, branching, or merge capabilities. You'd quickly descend into "config drift" hell and a manual, error-prone deployment process. The official Terraform documentation on state management makes it clear that a robust backend with locking, like Amazon S3 with DynamoDB, is crucial, and that's just one piece of the tooling puzzle for reliability.

Similarly, CI/CD platforms like GitHub Actions v3.0, GitLab CI v16.8, or Jenkins v2.426 don't just "do" automation; they enforce it. They standardize the build, test, and deployment process, making it repeatable and auditable. This consistency is what allows teams to gain confidence in frequent releases, which in turn fosters a culture of small, incremental changes, easier debugging, and quicker feedback loops. It's not the presence of the tool, but its effective application that makes the difference. If you can automate a deployment process from commit to production in under 10 minutes with high confidence, that speed and reliability fundamentally change how teams approach their work. It shifts their focus from "will it deploy?" to "is the feature valuable?"

Moreover, modern infrastructure complexity practically demands sophisticated tooling. Managing hundreds of microservices, thousands of containers, and petabytes of data without tools for observability (Prometheus v2.48, Grafana v10.3), centralized logging (Elastic Stack v8.11), and automated incident response (PagerDuty) is simply unfeasible. These tools collect the data and provide the insights necessary for a team to truly understand their systems, respond effectively to failures, and iterate on improvements. The tools are not just "nice-to-haves"; they are the very mechanisms through which a modern DevOps team gains visibility, control, and ultimately, reliability at scale.

Exceptions where a tool-first approach still wins

There are specific scenarios where a judicious, tool-first approach can indeed catalyze cultural change and deliver immediate, tangible benefits, even before every cultural nuance of "DevOps" has permeated an organization. These are typically situations where a clear, pressing technical need can be met by a well-understood tool, subsequently driving process improvements.

One such scenario is when a team is struggling with inconsistent infrastructure deployments. Introducing Infrastructure as Code (IaC) with a tool like Terraform v1.7.0, even with minimal initial buy-in beyond the core infrastructure team, can be a game-changer. By centralizing infrastructure definitions in Git, it immediately enforces a single source of truth, makes changes auditable, and enables repeatable deployments. Developers who previously waited days for manual infrastructure provisioning suddenly get resources in minutes via self-service pipelines. This tangible benefit often kickstarts broader adoption of version control for everything, peer review processes, and a shared understanding of infrastructure, organically fostering a "you build it, you run it" mentality. Using tools like Terraform with established best practices can dramatically improve the security posture of your infrastructure, as discussed in Secure Terraform PRs with an Architecture Firewall.

Another strong case is in highly regulated industries or environments with strict compliance requirements. Here, adopting specific security, audit, or compliance tools isn't just a best practice; it's often a legal mandate. For example, implementing a robust software supply chain security solution that scans dependencies (for example, with tools like Trivy v0.49 or Snyk v1.1270) and enforces policies throughout the CI/CD pipeline. The tool dictates a new, more secure process, forcing developers to address vulnerabilities earlier and providing auditors with clear evidence of compliance. While the ideal is cultural adoption, the immediate need for compliance can drive the integration of specific tools which then educate and influence behavior. These tools act as a guardrail, preventing non-compliant actions and establishing a baseline for security that might otherwise be overlooked. This proactive stance is critical for avoiding security incidents and can be bolstered by advanced strategies discussed in Supply Chain Security Proxy: Move Beyond Vulnerability Scann.

Finally, in rapidly scaling organizations, specialized tools become indispensable for managing complexity, even if the cultural maturity lags. When you move from a handful of services to hundreds, or from a few dozen users to millions, tools for advanced observability (like a comprehensive ELK stack for logs, Prometheus v2.48 for metrics, and Jaeger for tracing), performance testing, and cloud cost management (FinOps tools) are not optional. They provide the necessary visibility and control to prevent total collapse. While a fully mature DevOps culture would use these tools to drive continuous improvement, their initial implementation often serves as a survival mechanism, providing the data necessary to react to growth challenges. The insights gained from these tools can then be used to evangelize and drive the cultural changes required for long-term sustainability.

Practical Fundamentals You Should Prioritize

If you're looking to truly master DevOps fundamentals, shift your focus from chasing the latest shiny tool to building a solid foundation of engineering discipline and fostering the right cultural habits. Here's where to put your energy:

Version Control Everything

This isn't just about your application code. Your infrastructure configurations, Kubernetes manifests, documentation, database schemas, and even your .env files (sanitized, of course) belong in Git. This provides an auditable history, enables collaboration, and is the absolute bedrock for automation.

Here's a simple example of cloning a repository and checking its status:

$ git clone https://github.com/your-org/your-repo.git
$ cd your-repo
$ git status

Output:

On branch main
Your branch is up to date with 'origin/main'.

nothing to commit, working tree clean

Automate Relentlessly (But Smartly)

CI/CD isn't a destination; it's a continuous process of reducing manual effort and improving feedback loops. Start with the most repetitive, error-prone tasks. Build, test, scan, and deploy automatically. But don't just automate bad processes; fix the processes first, then automate them.

A basic GitHub Actions workflow for a Node.js application:

name: Node.js CI

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - name: Use Node.js 20.x
      uses: actions/setup-node@v4
      with:
        node-version: '20.x'
    - name: Install dependencies
      run: npm ci
    - name: Run tests
      run: npm test

This workflow ensures that every push and pull request to main branch automatically runs tests, giving immediate feedback.

Treat Infrastructure as Code (IaC)

Stop clicking in the console. Define your infrastructure (servers, networks, databases, Kubernetes clusters) in code using tools like Terraform v1.7.0 or Pulumi v3.100. This makes your infrastructure reproducible, versionable, and testable. It's the only way to scale infrastructure reliably.

Here's a simple main.tf file using Terraform v1.7.0 to define an AWS S3 bucket:

# main.tf for Terraform v1.7.0

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region = "us-east-1"
}

resource "aws_s3_bucket" "example_bucket" {
  bucket = "my-unique-application-logs-bucket-12345" # Must be globally unique
  tags = {
    Environment = "Dev"
    Project     = "MyApp"
  }
}

output "bucket_name" {
  value       = aws_s3_bucket.example_bucket.bucket
  description = "The name of the S3 bucket"
}

To apply this, you would run:

$ terraform init
$ terraform plan
$ terraform apply --auto-approve

Output of terraform apply:

Terraform will perform the following actions:

  # aws_s3_bucket.example_bucket will be created
  + resource "aws_s3_bucket" "example_bucket" {
      + accel_transfer_enabled = false
      + acl                    = (known after apply)
      + arn                    = (known after apply)
      + bucket                 = "my-unique-application-logs-bucket-12345"
      + bucket_domain_name     = (known after apply)
      + bucket_prefix          = (known after apply)
      + bucket_regional_domain_name = (known after apply)
      + force_destroy          = false
      + id                     = (known after apply)
      + object_lock_enabled    = false
      + policy                 = (known after apply)
      + region                 = (known after apply)
      + request_payer          = (known after apply)
      + tags                   = {
          + "Environment" = "Dev"
          + "Project"     = "MyApp"
        }
      + tags_all               = {
          + "Environment" = "Dev"
          + "Project"     = "MyApp"
        }
      + website_domain         = (known after apply)
      + website_endpoint       = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.
aws_s3_bucket.example_bucket: Creating...
aws_s3_bucket.example_bucket: Creation complete after 1s [id=my-unique-application-logs-bucket-12345]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:

bucket_name = "my-unique-application-logs-bucket-12345"

Remember to use a globally unique bucket name.

Monitor and Observe Deeply

You can't fix what you can't see. Instrument your applications and infrastructure to collect metrics, logs, and traces. Use tools like Prometheus v2.48 for metrics, Grafana v10.3 for dashboards, and a robust logging solution (for example, Loki v2.9 or Elastic Stack v8.11) to understand system behavior. Proactive monitoring helps you detect issues before your users do. Don't just watch CPU and memory; understand business metrics and application health indicators. For example, monitor your API's 99th percentile latency and error rates, not just if the process is running.

Embrace Feedback Loops and a Blameless Culture

DevOps is fundamentally about continuous improvement. After an incident, conduct blameless post-mortems focusing on system and process failures, not individual blame. Learn from mistakes, implement corrective actions, and share knowledge. Encourage frequent, open communication between development and operations teams. This fosters trust and a shared sense of responsibility. If something breaks in production, the question should always be "What can we do to prevent this type of failure again?" not "Whose fault was this?"

Shift Security Left, but Don't Forget Right

Integrate security into every stage of your development pipeline. This means security training for developers, static and dynamic analysis in CI/CD, dependency scanning, and threat modeling. However, don't stop there. Implement runtime security measures, network segmentation, robust access controls, and a solid incident response plan. A comprehensive approach covers the entire lifecycle, from design to production. Focusing solely on "shift left" without considering runtime protection is like locking your front door but leaving your back door wide open.

Mastering DevOps fundamentals means internalizing these core principles. It's about changing how teams work together, improving their engineering practices, and relentlessly seeking efficiency and reliability. The tools are there to support these efforts, but they are not the goal in themselves. Without the underlying cultural and practical discipline, even the most sophisticated tools will only lead to more complex problems. Start simple, focus on the fundamentals, and let your problems drive your tool choices, rather than letting tools dictate your approach.

LLM Prompt Caching with Git to Cut API Costs

DevOps Start — Thu, 25 Jun 2026 08:30:04 +0000

If your CI/CD pipelines call LLM APIs like OpenAI's GPT-4, you've probably noticed the token costs. Automated systems that generate documentation or review code often run the same prompts repeatedly, leading to high bills. You can reduce these costs significantly by implementing a simple prompt cache using a tool you already have: Git.

This article explains how to use a dedicated Git repository as a database-free key-value cache for LLM prompts and responses. Before calling an expensive API, your script checks a local Git clone for a cached answer. If found, it uses the saved response, avoiding the API call entirely. This method can cut costs by over 50% in CI/CD environments where prompts are frequently repeated.

How Git-Based Caching Works

The approach treats a Git repository as a key-value store. You simply create a new, dedicated repository to act as the cache.

Key: A SHA256 hash of the prompt's content. Hashing ensures that even a one-character difference creates a unique key.
Value: The LLM's response, stored as a plain text file. The filename is the key, for example, 5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8.

When your script needs an LLM response, it first calculates the prompt's hash. It then checks if a file with that name exists in its local clone of the cache repository. If it does, that's a cache hit. If not, it's a miss.

The Caching Workflow in Action

The logic for your application or CI script follows a "check-miss-write" pattern.

Clone/Pull: Before running, ensure your script has an up-to-date local clone of the cache repository. A quick $ git pull is all you need.
Generate Hash: Take the full prompt string and generate its SHA256 hash. This becomes your cache key.
Check for Key: Look for a file named after the hash in the local cache repository.
Handle the Result:

Cache Hit: If the file exists, read its contents. This is your LLM response. No API call is made.
Cache Miss: If the file does not exist, call the actual LLM API to get a new response.

Write to Cache: On a cache miss, save the new response to a file named after the prompt's hash. Then, commit and push this new file to the remote cache repository.

# Example of a cache repository's structure
$ ls -1 llm-cache/
0a3b...
1c5d...
5e88...

This workflow ensures that the next time the same prompt is encountered by any user or pipeline with access to the repo, it will be a cache hit. This is particularly effective in CI pipelines that build AI agents for Kubernetes deployments, where environment setup prompts are often identical across runs.

A Python Implementation Example

Here is a simple Python function that implements this caching logic. It uses the standard hashlib and os libraries. You can consult the official Python hashlib documentation for more details on hashing.

import hashlib
import os
import subprocess

# --- Configuration ---
# IMPORTANT: Update this to the absolute path of your cache repository clone.
CACHE_DIR = "/path/to/your/local/llm-cache-repo"

def get_llm_response_with_cache(prompt: str, llm_api_call_func) -> str:
    """
    Gets an LLM response, using a Git-based file cache to avoid duplicate API calls.

    Args:
        prompt: The full prompt string to send to the LLM.
        llm_api_call_func: A function that takes a prompt string and returns the API response.

    Returns:
        The LLM's response, either from the cache or a new API call.
    """
    # 1. Ensure the cache is up-to-date
    # A production implementation should include robust error handling for Git commands.
    subprocess.run(["git", "pull"], cwd=CACHE_DIR, check=True)

    # 2. Generate the cache key
    prompt_hash = hashlib.sha256(prompt.encode('utf-8')).hexdigest()
    cache_file_path = os.path.join(CACHE_DIR, prompt_hash)

    # 3. Check for a cache hit
    if os.path.exists(cache_file_path):
        print(f"CACHE HIT: Found response for hash {prompt_hash}")
        with open(cache_file_path, 'r', encoding='utf-8') as f:
            return f.read()

    # 4. Handle a cache miss
    print(f"CACHE MISS: Calling API for hash {prompt_hash}")
    response = llm_api_call_func(prompt)

    # 5. Write the new response to the cache and push
    # Note: The prompt and response are stored in plain text. Do not use this method for sensitive data.
    with open(cache_file_path, 'w', encoding='utf-8') as f:
        f.write(response)

    print("Adding new response to cache...")
    subprocess.run(["git", "add", cache_file_path], cwd=CACHE_DIR, check=True)
    subprocess.run(["git", "commit", "-m", f"Add cache for {prompt_hash}"], cwd=CACHE_DIR, check=True)
    subprocess.run(["git", "push"], cwd=CACHE_DIR, check=True)

    return response

# --- Example Usage ---
def fake_openai_call(prompt: str) -> str:
    # Replace this with your actual client.chat.completions.create() call
    print("--- Faking expensive API call ---")
    return f"This is the LLM's answer to the prompt starting with: '{prompt[:30]}...'"

if __name__ == "__main__":
    my_prompt = "Generate a Kubernetes Deployment YAML for a Python Flask app named 'my-app' listening on port 5000."

    # First call (will be a miss)
    response1 = get_llm_response_with_cache(my_prompt, fake_openai_call)
    print("\nResponse 1:\n", response1)

    # Second call (will be a hit)
    response2 = get_llm_response_with_cache(my_prompt, fake_openai_call)
    print("\nResponse 2:\n", response2)

Benefits of This Approach

Cost Reduction: Avoids expensive API calls for repeated prompts. With GPT-4 Turbo input prices around $10 per million tokens, caching just a few hundred complex prompts in CI can lead to substantial savings.
No New Infrastructure: It uses your existing Git provider, so there is no need to set up or maintain a separate caching service like Redis or Memcached.
Audit Trail: The Git history provides a complete, version-controlled log of every unique prompt and its corresponding LLM response.
Faster Execution on Hits: Reading a local file takes milliseconds, while a network API call can take several seconds. This speeds up CI/CD jobs that get a cache hit.

Limitations and Considerations

This method is pragmatic but has trade-offs compared to a dedicated caching system.

Manual Cache Invalidation: To get a fresh response for a cached prompt, you must manually delete the file from the repository (git rm <hash>, commit and push). There is no built-in time-to-live (TTL) mechanism.
Repository Size: The cache repository will grow indefinitely. While text-based responses are small, this method is unsuitable for caching large files like images or audio. Regular maintenance may be needed to prune old entries.
Concurrency and Race Conditions: If two CI jobs miss on the same prompt simultaneously, they will both call the LLM API. They will then race to commit and push the new file. One git push will fail. The failing script needs retry logic (for example, git pull and check again), or you will waste an API call.

This Git-based caching technique is most effective in environments with high prompt repetition, such as CI/CD pipelines for code analysis, documentation generation, or testing. For applications requiring high-throughput, atomic operations, or automatic cache eviction, a dedicated solution like Redis is more appropriate. For many teams, however, this simple approach provides a significant benefit for minimal effort.

How to Detect and Prevent Malicious AI Agent Skills

DevOps Start — Wed, 24 Jun 2026 08:30:44 +0000

Malicious AI agent skills are tool or server dependencies (often via Model Context Protocol or MCP) that present a benign interface to the LLM but execute unauthorized actions on the host system. You detect these by monitoring for unauthorized tool calls, unexpected egress traffic or attempts to access sensitive system files like /etc/shadow. Prevention requires a zero-trust architecture where skills are isolated in sandboxes with strict network egress filters and human-in-the-loop approvals for destructive actions.

What the error means

When you integrate "skills" into an AI agent, you add executable dependencies to your runtime. A malicious skill acts as a Trojan horse, allowing an agent to perform "silent failures" or "unauthorized escalation."

Unlike a traditional software crash, the error here is a security breach. You might see an agent unexpectedly exfiltrating .env files, executing rm -rf /, or sending internal system prompts to an external API. If your agent calls tools you didn't authorize or accesses paths outside its designated workspace, your skill supply chain is compromised.

Root Causes

The vulnerability comes from treating AI tools as static configuration rather than executable code.

Implicit Trust in the Supply Chain
Many developers install MCP servers from community repositories without auditing the source. Similar to a malicious npm package, an MCP server can contain obfuscated code that triggers only under specific prompt conditions.

Indirect Prompt Injection
An agent may read a malicious file (for example, a README.md in a repo) containing hidden instructions. These instructions trick the LLM into using a legitimate skill, such as a shell executor, to perform a malicious action that bypasses the user's intent.

Over-Privileged Environments
Running AI agents with the same permissions as the local user is a critical failure. If the agent has root access or full SSH key access, one compromised skill can compromise the entire workstation or cluster.

Lack of Egress Control
Most agent runtimes allow unrestricted outbound HTTP requests. This allows malicious skills to "phone home" with stolen secrets or API keys.

Detection and Neutralization

To stop malicious skills, implement layered defense focusing on isolation and auditing. For those building custom agents, refer to the Model Context Protocol documentation to understand standard communication patterns.

Static Audit of MCP Servers

Before adding a server to your claude_desktop_config.json or agent config, audit the entry point. Search for curl, wget, or eval calls that fetch remote scripts.

# Search for suspicious remote execution patterns in a local MCP server directory
grep -rE "curl|wget|eval|exec|base64" ./mcp-servers/suspicious-tool/

Implement a Restricted Runtime

Never run agent skills directly on your host. Use a containerized environment with limited resources. For those managing agents on a cluster, integrate LLM Observability on Kubernetes: A Practical Guide to monitor tool-call latency and volume. I have seen this prevent total host compromise in environments with >10 nodes by trapping the agent in a non-privileged namespace.

# Run a potentially risky MCP server in a restricted Docker container
docker run -d \
  --name mcp-sandbox \
  --memory="512m" \
  --cpus="0.5" \
  --network="bridge" \
  --read-only \
  -v /tmp/agent-data:/data:rw \
  mcp-server-image:v1.0.0

Enforce Network Egress Filtering

Use iptables or a service mesh to block all outbound traffic except to known API endpoints. This reduces the risk of data exfiltration by nearly 100% for basic "phone-home" malware.

# Block all outbound traffic by default, allow only specific APIs
sudo iptables -P OUTPUT DROP
sudo iptables -A OUTPUT -p tcp --dport 443 -d api.anthropic.com -j ACCEPT
sudo iptables -A OUTPUT -p tcp --dport 443 -d github.com -j ACCEPT

Structured Tool Logging

Configure your agent to log every tool_use call, including the exact arguments passed and the raw output returned.

# Piping agent logs to a file for forensic analysis
agent-runtime --log-level debug 2>&1 | tee agent_audit.log

Prevention Strategies

Shift from a "trust-by-default" to a "zero-trust" agent architecture to avoid future compromises.

AI Bill of Materials (AIBOM)
Maintain a versioned list of every MCP server and model version used in production. Do not allow "latest" tags; pin to specific git hashes. This prevents "poisoned" updates from automatically entering your environment.

Human-in-the-Loop (HITL)
Configure your agent interface to require manual approval for destructive tools, such as delete_file, execute_shell, or send_email.

Least Privilege
Create a dedicated OS user for the agent with no sudo privileges and restricted directory access.

Secret Management
Use a secret manager instead of environment variables. This prevents skills from simply calling env to steal your keys, a common tactic seen in GitHub Actions Security: How to Stop Secret Leaks in CI/CD.

Next Steps for DevOps Teams

Audit your current config.json files for third-party MCP servers.
Wrap your agent runtime in a Docker container with --read-only flags.
Implement an egress allow-list to restrict tool communications.

Kubernetes Troubleshooting: Why Did My Pod Die?

DevOps Start — Tue, 23 Jun 2026 08:32:37 +0000

Pods die because of scheduling failures, startup crashes or runtime terminations. When a Kubernetes pod fails, it rarely tells you exactly why in a single line. Instead, it gives you a status like CrashLoopBackOff or Pending, which are symptoms rather than root causes. To fix a pod, you must distinguish between these three failure modes. This guide provides a decision tree for diagnosing pod deaths, moving from high-level status checks to deep-dive container inspection.

For a comprehensive look at the most common restart cycles, check out the guide on how to fix Kubernetes CrashLoopBackOff in production.

Understanding the failure states

A pod "death" is a transition in the Pod Lifecycle. When you see CrashLoopBackOff, the container started, crashed, and Kubernetes is now waiting an exponentially increasing amount of time before trying to start it again to avoid hammering the node.

OOMKilled means the Linux kernel terminated the process because it exceeded its memory limit. I have seen this happen frequently in Java applications where the JVM heap is set higher than the Kubernetes memory limit. ImagePullBackOff means the kubelet cannot retrieve the container image from the registry. Each of these states points to a different layer of the stack: the infrastructure, the container runtime or the application code itself. Refer to the official Kubernetes Pod Lifecycle documentation for the full state machine.

Common root causes

Pod failures generally fall into three categories.

Startup and Configuration Failures

Missing Environment Variables: The application panics immediately because a required database URL or API key is missing.
Invalid Image Tags: A typo in the image version or a deleted tag in the registry leads to ErrImagePull.
Registry Authentication: The imagePullSecrets are missing or the service account lacks permission to pull from a private repository.

Resource and Infrastructure Constraints

Memory Limits (OOMKilled): The container attempted to allocate more memory than defined in its limits section.
CPU Throttling: Extreme throttling can trigger Liveness probe timeouts, causing Kubernetes to kill and restart the pod.
Scheduling Constraints: Pods stuck in Pending usually suffer from "Insufficient cpu" or "Insufficient memory" on all available nodes, or they have nodeSelector constraints that no node meets.

Health Check Failures

Liveness Probe Misconfiguration: The probe checks a /health endpoint that takes 10 seconds to respond, but the timeoutSeconds is set to 1. Kubernetes assumes the app is dead and kills it.
Slow Startup: Heavy frameworks like Spring Boot may take 60 seconds to start. If the Liveness probe starts checking after 10 seconds, the pod is killed before it ever becomes ready.

Step-by-step recovery process

Follow this decision tree to isolate the root cause.

Step 1: Identify the Symptom

Start with the high-level status.

kubectl get pods

Observation:

If status is Pending → Go to Step 2.
If status is CrashLoopBackOff or Error → Go to Step 3.
If status is Running but the pod keeps restarting → Go to Step 4.

Step 2: Debugging Pending Pods

If the pod isn't even starting, check the events.

kubectl describe pod <pod-name>

Look at the Events section at the bottom. If you see FailedScheduling, check for taints or resource pressure. If you see FailedMount, your PVC is likely stuck in another zone or not bound.

Step 3: Debugging CrashLoops and Errors

If the pod starts and then dies, check the application logs.

kubectl logs <pod-name> --previous

The --previous flag is critical. It allows you to see the logs from the container that just crashed, rather than the logs of the new container currently starting.

If logs are empty, check the exit code using kubectl describe pod.

Exit Code 137: This is almost always OOMKilled. You must increase the memory limits in your manifest.
Exit Code 1: Application crash (NullPointerException, missing config, etc.).

Step 4: Debugging Pods that Restart

If the pod is Running but the restart count is climbing, the Liveness probe is likely killing it.

kubectl describe pod <pod-name> | grep -i "Liveness probe failed"

Implement a startupProbe. This tells Kubernetes to ignore Liveness and Readiness probes until the container has finished its initial boot sequence. In clusters with slow-starting legacy apps, this reduces unnecessary restart cycles by nearly 100%.

# Example Startup Probe for slow apps
startupProbe:
  httpGet:
    path: /healthz
    port: 8080
  failureThreshold: 30
  periodSeconds: 10

Step 5: Advanced Inspection

If you cannot get logs and the pod dies too fast to exec into, use an ephemeral debug container.

kubectl debug -it <pod-name> --image=busybox --target=<container-name>

This attaches a shell to the process namespace of the failing pod without restarting it, which allows you to inspect the filesystem and network state in real time.

How to prevent future failures

Preventing pod failure requires a production-ready manifest checklist. Never deploy a pod without these four elements:

Explicit Resource Requests and Limits: Set requests to what the app needs to run and limits to a reasonable ceiling. This prevents a single pod from consuming all node memory and triggering a node-wide Out-of-Memory event.
Proper Probe Hierarchy: Use startupProbe for initial boot, livenessProbe for deadlock detection and readinessProbe to control traffic flow.
Non-Root Users: Use securityContext to ensure the pod does not crash due to permission errors when writing to mounted volumes.
Graceful Shutdown: Handle SIGTERM in your application code. This allows Kubernetes to drain connections before the 30-second terminationGracePeriodSeconds expires, avoiding 502 errors during deployments.

How to Debug OOMKilled Pods in Kubernetes: A Step-by-Step Guide

DevOps Start — Mon, 22 Jun 2026 09:24:18 +0000

What Does OOMKilled Actually Mean?

An OOMKilled status occurs when the Linux kernel's Out-of-Memory (OOM) killer terminates a process to prevent a system-wide crash. In Kubernetes, this is signaled by Exit Code 137. The primary goal of the OOM killer is to reclaim memory to keep the underlying node stable.

There are two distinct triggers for this event. First, a container may exceed its specified memory limit, leading the kubelet to kill the process immediately. Second, the entire node may experience memory exhaustion, forcing the kernel to select a "victim" pod based on its Quality of Service (QoS) class. I've seen this happen frequently in clusters where "Burstable" pods are over-provisioned, causing the kernel to kill pods that were technically under their own limits just to save the node. Detailed resource management specifications are available in the official Kubernetes documentation.

Why is Your Pod Getting OOMKilled?

Root causes typically fall into three categories: application leaks, misconfigured limits, or node-level pressure.

Application Memory Leaks

This is the most common cause for pods that operate normally for several hours before crashing. A leak occurs when an application allocates memory but fails to release it. In Java, this often happens when objects are stored in static collections that never clear. In Go, common culprits include unclosed goroutines or slices that grow indefinitely. In these cases, memory usage climbs linearly until it hits the hard limit.

Improper Resource Limits

Engineers often set limits too close to requests. Many applications have a "bursty" startup phase. For example, a Spring Boot application loading numerous beans into memory may spike to 600Mi during initialization. If the limit is set to 512Mi, the pod will be killed before it ever reaches a healthy state.

Node-Level Memory Pressure

When multiple pods are configured with requests significantly lower than their limits, they can all attempt to burst simultaneously. If the aggregate usage exceeds the node's physical RAM, the kernel triggers a node-level OOM event. The kernel targets pods with the lowest priority or those consuming the most memory relative to their requested amount.

How to Diagnose and Fix OOMKilled Pods

Follow this systematic workflow to move from a crashing pod to a verified root cause.

Step 1: Confirm the Termination Reason

Identify the failing pod and verify the exit code using kubectl.

$ kubectl get pods
NAME                               READY   STATUS             RESTARTS   AGE
api-gateway-7f8db6d9-abc12          0/1     CrashLoopBackOff   4          12m

$ kubectl describe pod api-gateway-7f8db6d9-abc12

Locate the Last State section in the output:

Containers:
  api-container:
    State:          Running
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Finished:     Thu, 24 Oct 2024 14:22:01 +0000

An Exit Code: 137 confirms an OOM event. If the pod cycles through crashes without this specific code, refer to our guide on Fix CrashLoopBackOff in Kubernetes Pods.

Step 2: Analyze the "Working Set" Metric

Avoid using container_memory_usage_bytes in Prometheus, as it includes page caches that the kernel can reclaim under pressure. Kubernetes makes OOM decisions based on container_memory_working_set_bytes.

Execute this PromQL query in your Grafana dashboard:

sum(container_memory_working_set_bytes{pod="api-gateway-7f8db6d9-abc12"}) by (pod)

A "sawtooth" pattern (steady growth followed by a sharp drop to zero) indicates a memory leak. A flat line with a sudden, vertical spike suggests a resource limit issue or a specific heavy request.

Step 3: Application Profiling

Increasing limits without profiling only delays the crash. If a leak is suspected, you must analyze the heap.

For Go applications, integrate pprof. Add this to your main.go:

import (
    _ "net/http/pprof"
    "net/http"
)

func main() {
    go func() {
        // Listen on a separate port to avoid interfering with app traffic
        http.ListenAndServe("0.0.0.0:6060", nil)
    }()
    // your app logic
}

Capture a heap profile while the pod is under load:

kubectl exec api-gateway-7f8db6d9-abc12 -- curl -s http://localhost:6060/debug/pprof/heap > heap.pprof
go tool pprof -http=:8080 heap.pprof

For Java applications, trigger a heap dump using jcmd:

kubectl exec api-gateway-7f8db6d9-abc12 -- jcmd 1 GC.heap_dump /tmp/heapdump.hprof
kubectl cp api-gateway-7f8db6d9-abc12:/tmp/heapdump.hprof ./heapdump.hprof

Analyze the .hprof file in VisualVM or Eclipse MAT to identify the leaking class.

Step 4: Right-Sizing the Limits

Calculate the new limit based on observed peaks. A production-ready standard is to set the limit 20% to 30% above the peak working_set_bytes observed during a full load test.

Update your deployment manifest:

resources:
  requests:
    memory: "512Mi"
    cpu: "250m"
  limits:
    memory: "1Gi"
    cpu: "500m"

Apply the change:

kubectl apply -f deployment.yaml

How to Prevent OOMKills in the Future

To stop reacting to memory crashes, implement these three architectural strategies.

Deploy Vertical Pod Autoscaler (VPA): Use VPA in Recommender mode. It analyzes historical usage and suggests the ideal requests and limits, reducing the guesswork that leads to OOM events.
Enforce Guaranteed QoS: For critical workloads, set requests exactly equal to limits. This assigns the pod to the "Guaranteed" QoS class, making it the last candidate for eviction when the node runs out of RAM.
Coordinate with HPA: Ensure your Horizontal Pod Autoscaler (HPA) is tuned. If HPA triggers new pods based on CPU but your pods are OOMKilled due to memory, you'll experience cascading failures. Read Kubernetes HPA Deep Dive: Autoscaling Explained for coordination tips.

FAQ

Q: Why did my pod get OOMKilled even though it was below its limit?
A: This is a node-level OOM event. When the physical RAM of the node is exhausted, the kernel kills pods based on QoS class. "BestEffort" pods die first, then "Burstable" pods. If your pod is "Burstable" and the node is under pressure, it can be killed regardless of its individual limit.

Q: What is the difference between RSS and Working Set?
A: Resident Set Size (RSS) is the memory physically held in RAM. Working Set is RSS plus cached memory that cannot be evicted. Kubernetes uses Working Set for OOM decisions because it represents the memory the container absolutely requires to function.

Q: Should I enable swap in Kubernetes to prevent OOMKills?
A: No. While Kubernetes 1.28+ has improved swap support, relying on swap usually masks memory leaks and introduces severe latency spikes. It is better to fix the leak or increase the node size.

Conclusion and Next Steps

Debugging OOMKilled pods requires moving beyond kubectl get pods and into memory metrics and heap profiling. By distinguishing between container-level and node-level OOM events, you can apply the correct fix, whether that is tuning JVM heap sizes or adjusting your QoS class.

Your next steps:

Audit your current deployments for "Burstable" pods with wide gaps between requests and limits.
Install the Prometheus kube-state-metrics to track container_memory_working_set_bytes.
Integrate pprof or jcmd into your base container images to make profiling immediate during incidents.

Cilium vs Calico: CNI Comparison for Platform Teams

DevOps Start — Sun, 21 Jun 2026 08:47:09 +0000

Choosing a Container Network Interface (CNI) for Kubernetes used to be a simple decision. Does it connect pods? Good enough. Today, that choice is a foundational platform decision that dictates your cluster's security posture, performance limits, and observability capabilities for years to come. Your CNI is no longer just a networking plugin; it's a critical piece of infrastructure that can either accelerate or bottleneck your entire stack.

In 2026, the conversation is dominated by two leaders: Cilium and Calico. Both are powerful, mature, and trusted in massive production environments. But they represent different philosophies and technical approaches. Calico offers battle-tested flexibility with a rich history, while Cilium represents a new, eBPF-native world of networking. This guide will cut through the noise to help your platform team make the right long-term decision, focusing not just on features, but on the strategic implications for performance, security, and operations at scale.

The Core Differentiator: The Rise of eBPF

To understand the Cilium vs. Calico debate, you first have to understand eBPF (extended Berkeley Packet Filter). Think of eBPF as a way to run sandboxed, event-driven programs directly within the Linux kernel without changing kernel source code. For networking, this is a game-changer. You can find a deep dive on the technology at the official eBPF & Cilium community site.

For decades, Kubernetes networking relied on iptables or IPVS. These tools are powerful but operate by creating long, complex chains of rules. As your cluster grows, with thousands of services and pods, traversing these chains for every single packet adds significant CPU overhead and latency. It's like sending a package through a city with thousands of traffic stops.

eBPF offers a direct path. eBPF programs can be attached to network interfaces to make intelligent routing and filtering decisions right at the source, bypassing the cumbersome iptables and conntrack subsystems entirely. The result is a faster, more efficient, and more programmable data path.

Cilium was built from the ground up with eBPF as its core. Every feature (networking, observability, and security) is designed to leverage the power and efficiency of eBPF.
Calico started with a standard Linux networking data plane using iptables and IP-in-IP or BGP for routing. It's incredibly robust and well-understood. Recognizing the power of eBPF, Calico introduced a pluggable eBPF data plane. This gives you a choice, but it also means eBPF is an add-on to its core architecture rather than its native foundation.

This fundamental difference in architecture is the source of nearly every other distinction between the two projects.

Network Policy and Security Showdown

Both Cilium and Calico provide robust network security, but their approaches and advanced capabilities differ significantly due to eBPF.

Calico's Policy Engine

Calico has long been the gold standard for network policy. It supports standard Kubernetes NetworkPolicy resources and extends them with its own CRDs like GlobalNetworkPolicy and NetworkPolicy with richer features like explicit deny rules and rule ordering.

A typical Calico policy is direct and focuses on L3/L4 (IP and port) filtering. Here's an example that allows ingress from pods with the app: frontend label to port 6379/TCP on pods with the app: database label.

apiVersion: projectcalico.org/v3
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-db
  namespace: production
spec:
  selector: app == 'database'
  types:
  - Ingress
  ingress:
  - action: Allow
    protocol: TCP
    source:
      selector: app == 'frontend'
    destination:
      ports:
      - 6379

This is clear, effective, and works reliably across Calico's standard and eBPF data planes. For many organizations, this level of control is perfectly sufficient.

Cilium's L7-Aware Security

Cilium's eBPF-native approach allows it to operate at a much higher level of the stack. It can parse application-layer protocols like HTTP, gRPC, and Kafka directly from the kernel. This enables incredibly granular, identity-aware policies that iptables-based systems simply cannot match.

Imagine you want to allow the billing-service to read from a metrics-api but not write to it. With Cilium, you can enforce this at the API level.

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: api-aware-policy-example
  namespace: production
spec:
  endpointSelector:
    matchLabels:
      app: metrics-api
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: billing-service
    toPorts:
    - ports:
      - port: "8080"
        protocol: TCP
      rules:
        http:
        - method: "GET"
          path: "/api/v1/metrics"

This policy enforces that only GET requests to /api/v1/metrics are allowed from the billing-service. Any POST, DELETE, or other requests would be dropped by the CNI, even if they are on the correct port. This is a massive security enhancement, effectively turning your network into a zero-trust environment at the API level without requiring a service mesh.

For encryption, both projects support WireGuard for transparent, in-transit encryption between nodes, providing a modern and performant alternative to IPsec.

Performance and Scalability in 2026

In a small cluster, the performance difference between Cilium and Calico's eBPF mode might be negligible. But when you scale to hundreds of nodes and tens of thousands of pods, the architectural differences become stark.

The primary performance advantage of eBPF comes from avoiding the iptables and conntrack kernel subsystems.

Reduced CPU Overhead: Every packet doesn't need to traverse long chains of iptables rules. This frees up significant CPU cycles on each node, which can then be used by your applications. This is especially noticeable on nodes with high connection churn.
Lower Latency: By processing packets on a more direct path, eBPF reduces the per-packet latency within the cluster. This can have a meaningful impact on the performance of latency-sensitive applications like databases or real-time APIs.
Improved Throughput: The efficiency of the eBPF data path allows for higher overall network throughput, a key consideration for data-intensive workloads.

While Calico's eBPF mode also gains these benefits, Cilium's eBPF implementation is often considered more mature and deeply integrated, having been the project's sole focus since its inception. In large-scale clusters, this maturity can translate to better stability and more predictable performance under extreme load. If you are deploying large clusters, for example when you deploy an EKS cluster with Terraform, this long-term performance profile becomes a critical factor.

Observability: Seeing the Unseen

Troubleshooting network issues in Kubernetes can be a nightmare. Is a packet being dropped by a network policy? Is a service not responding? This is where built-in observability tools make a huge difference.

Cilium's Hubble

Cilium comes with Hubble, a powerful, purpose-built observability platform that leverages eBPF to provide deep visibility into network flows without any application instrumentation. It gives you a service dependency map, real-time flow data, and policy troubleshooting tools out of the box.

With the Hubble CLI, you can immediately see what's happening. For instance, to see all DNS requests and replies in real time:

$ # Assuming Cilium v1.15+ and Hubble is enabled
$ hubble observe --protocol dns -f

TIMESTAMP                  SOURCE                      DESTINATION                 TYPE          VERDICT   SUMMARY
May 20 12:34:56.123Z   kube-system/coredns-abcde   default/my-app-fghij        L7:response   FORWARDED   DNS Qry: A, Rcode: NOERROR
May 20 12:34:56.456Z   default/my-app-fghij        kube-system/coredns-abcde   L7:request    FORWARDED   DNS Qry: my-external-service.com.

The Hubble UI provides a graphical representation of this data, making it incredibly easy to spot anomalies or understand service communication patterns. If you're constantly fighting weird networking bugs or dealing with a CrashLoopBackOff, this level of insight can save hours of debugging.

Calico's Observability

Calico Enterprise offers similar observability features, including a dynamic service graph and flow visualization. In the open-source version, observability is typically achieved by exporting metrics to Prometheus and building Grafana dashboards. This is a powerful and flexible approach, but it requires more setup and integration effort. You can get flow logs, but they are not as rich or easily queryable as what Hubble provides out of the box.

For teams that want a "batteries-included" observability solution tightly coupled with their CNI, Hubble is a clear winner. For teams that prefer to build their own observability stack around standards like Prometheus, Calico's approach is perfectly viable.

The Blurring Lines with Service Mesh

The next battleground for CNIs is the service mesh. Both Cilium and Calico are expanding their feature sets to provide capabilities traditionally associated with tools like Istio or Linkerd.

Cilium Service Mesh: Cilium offers service mesh features like traffic management (retries, timeouts), canary deployments, and mTLS encryption without requiring a sidecar proxy. It achieves this by embedding Envoy proxy functionality directly into the CNI layer, managed by eBPF. This sidecar-less model promises better performance and lower resource overhead compared to traditional service meshes.
Calico and Istio: Calico's strategy is to provide best-in-class integration with Istio. It can accelerate Istio's data plane by offloading some of the networking functions to its eBPF layer. This allows you to combine Calico's robust CNI and policy engine with the full, mature feature set of the Istio service mesh.

This presents a strategic choice: do you want a single, integrated platform for networking, security, and service mesh (Cilium), or do you prefer a modular approach, combining a dedicated CNI (Calico) with a dedicated service mesh (Istio)?

Making the Choice: A Decision Framework

There is no single "best" CNI. The right choice depends entirely on your team's priorities, existing infrastructure, and future goals.

Choose Cilium if

Performance is your top priority. You run latency-sensitive or high-throughput applications and want to squeeze every ounce of performance from your infrastructure.
You need advanced, L7-aware security. Your security model requires enforcing policies at the API level (HTTP, gRPC) and you want to build a zero-trust network.
You want built-in, out-of-the-box observability. Your team values integrated tools like Hubble for rapid troubleshooting and dependency mapping.
You are building a new platform and are "all-in" on eBPF. You are comfortable adopting a modern, eBPF-native stack and are interested in its integrated service mesh capabilities.

Choose Calico if

You need maximum flexibility and a proven track record. You operate in diverse environments (on-prem, multiple clouds) and need a CNI that supports various data planes (standard Linux, eBPF, Windows).
Your organization already has a significant investment in Istio. Calico's deep integration with Istio allows you to enhance your existing service mesh with a powerful CNI.
You prefer a more gradual adoption of eBPF. You want the option to run some clusters with the well-understood standard Linux data plane while experimenting with the eBPF data plane in others.
Your team has deep expertise in traditional Linux networking. Calico's architecture will feel familiar and its use of BGP for routing is a standard in many enterprises.

Frequently Asked Questions (FAQ)

1. Is eBPF mature enough for production in 2026?
Absolutely. eBPF has been a stable part of the Linux kernel for years. It's used in production by massive tech companies like Google, Meta, and Netflix to handle networking, security, and observability at an immense scale. Both Cilium and Calico's eBPF implementations are considered production-grade.

2. Can I migrate from Calico to another CNI like Cilium?
Yes, but it's a complex and disruptive process. A CNI migration typically requires a "rip and replace" approach, which involves draining nodes, uninstalling the old CNI, installing the new one, and then uncordoning the nodes. This requires careful planning and significant downtime. It's far better to make the right choice from the beginning.

3. Does Calico's standard Linux data plane still have a place?
Yes. For environments with older Linux kernels that lack the necessary eBPF features, or for teams who prioritize stability and familiarity over cutting-edge performance, the standard iptables-based data plane is still a very solid and reliable choice. It's one of Calico's key strengths: providing that flexibility.

4. What about other CNIs like Flannel or Weave Net?
While Flannel and Weave Net are excellent for getting started or for smaller, less demanding clusters, they generally lack the advanced security, observability, and performance features of Cilium and Calico. For any serious production platform, the choice has largely narrowed to these two.

Conclusion

Choosing between Cilium and Calico is a strategic decision that will shape your Kubernetes platform for years. It's a choice between an eBPF-native, all-in-one solution that pushes the boundaries of performance and security (Cilium), and a flexible, battle-hardened CNI that offers a choice of data planes and deep integration with the wider ecosystem (Calico).

Your next step should be hands-on evaluation. Don't just read about it. Spin up two identical test clusters. Install Cilium on one and Calico with the eBPF data plane on the other. Deploy a sample application and use tools like netperf to measure throughput and latency. Test a complex network policy on each. Use Hubble to visualize traffic. Only by seeing how they operate in your environment can you make a truly informed decision for the future of your platform.

NGINX vs Traefik: Kubernetes Ingress Comparison Guide

DevOps Start — Sat, 20 Jun 2026 08:28:19 +0000

Introduction

Choosing between NGINX and Traefik isn't about finding the "better" tool, but about deciding where you want your operational toil to live. For a platform engineer, the choice comes down to a trade-off between raw, predictable performance and developer agility. NGINX is the industry titan, offering unmatched throughput and a configuration model that's been battle-tested for decades. Traefik, conversely, was born for the cloud-native era, treating infrastructure as a fluid entity where services appear and disappear in seconds.

To evaluate these, you must look beyond the feature checklist. You need to consider Day 2 operations: how the controller handles 500+ microservices, the complexity of managing TLS certificates and the latency introduced during configuration reloads. As the industry shifts toward the Kubernetes Gateway API, both tools are evolving, but their core philosophies remain distinct. You're choosing between a high-performance proxy that adapts to Kubernetes and a Kubernetes-native orchestrator that happens to be a proxy.

Side-by-Side Comparison Table

Feature	NGINX Ingress Controller	Traefik Proxy
Architecture	Process-based (C-based)	Event-driven (Go-based)
Config Model	Annotations $\rightarrow$ nginx.conf	CRDs $\rightarrow$ Dynamic Config
Config Updates	Reloads (potential connection drops)	Hot-reloads (zero downtime)
TLS/SSL	External (cert-manager)	Native ACME/Let's Encrypt
Observability	External (Prometheus/Grafana)	Built-in Dashboard + Metrics
Performance	Higher raw throughput, lower CPU	High, but slightly higher overhead
Learning Curve	Steep for complex routing	Moderate, native K8s feel
Gateway API	Supported	First-class citizen

NGINX: The High-Performance Workhorse

NGINX Ingress is the safe bet for environments where every millisecond of latency counts and traffic patterns are relatively stable. Its strength lies in its efficiency. Because it's written in C, it handles massive concurrency with a smaller memory footprint than Go-based alternatives. In high-load scenarios, NGINX typically maintains a lower p99 latency than Traefik.

However, the "NGINX way" often involves a heavy reliance on annotations. If you need complex routing, you end up with an Ingress resource cluttered with nginx.ingress.kubernetes.io keys. For advanced logic, you have to use "snippets", which are essentially raw NGINX config fragments injected into the template. This is powerful but dangerous, as a syntax error in a snippet can crash the entire controller.

One major pain point is the reload mechanism. While the controller tries to minimize disruption, changing certain global settings triggers a reload of the NGINX process. I've seen this cause intermittent connection drops in clusters with >100 nodes when managing long-lived WebSockets or gRPC streams.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-gateway
  annotations:
    # Example of annotation-heavy config
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/limit-rps: "50"
spec:
  ingressClassName: nginx
  rules:
  - host: api.example.com
    http:
      paths:
      - path: /v1
        pathType: Prefix
        backend:
          service:
            name: api-service
            port:
              number: 80

Traefik: The Cloud-Native Orchestrator

Traefik is designed for the "churn" of microservices. It doesn't just read a config file; it listens to the Kubernetes API server. When a new service is deployed or an HPA scales your pods, Traefik updates its routing table in real-time without restarting or reloading.

The standout feature is the native CRD approach. Instead of messy annotations, Traefik uses IngressRoute and Middleware resources. This allows you to define a "RateLimit" middleware once and attach it to ten different services, rather than duplicating annotations across ten different Ingress files. This architectural choice reduces YAML duplication by roughly 40% in large-scale deployments.

The built-in Let's Encrypt integration is a massive win for platform teams. You don't need to install and manage cert-manager and its associated ClusterIssuers if you only need basic ACME automation. Traefik handles the challenge and renewal internally.

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: rate-limit-api
spec:
  rateLimit:
    average: 100
    burst: 50

---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: api-route
spec:
  entryPoints:
    - websecure
  routes:
  - match: Host(`api.example.com`) && PathPrefix(`/v1`)
    kind: Rule
    services:
    - name: api-service
      port: 80
    middlewares:
    - name: rate-limit-api

When to Choose Which

The decision should be based on your team's operational capacity and your application's traffic profile.

Choose NGINX if:

You are running a high-throughput API where raw latency is your primary KPI.
You have a small number of stable services that don't change their routing rules hourly.
Your team is already comfortable with NGINX syntax and wants a predictable tool.
You are leveraging a CNI like Cilium to handle some of the L7 logic and only need NGINX for the edge. If you're evaluating networking layers, check out our /comparisons/kubernetes-cni-comparison-cilium-vs-calico-for-platform-team for more context.

Choose Traefik if:

You are managing a volatile microservices environment with frequent deployments and auto-scaling.
You want a built-in dashboard to visualize traffic flow without configuring a complex Grafana stack.
You want to reduce "YAML bloat" by using reusable Middlewares.
You prefer a tool that feels like a native part of the Kubernetes API rather than an external proxy ported to K8s.

FAQ

Does Traefik support the standard Kubernetes Ingress resource?
Yes, Traefik supports the standard Ingress resource for compatibility, but to unlock features like advanced Middlewares and complex routing rules, you must use the IngressRoute CRD.

Which one is better for gRPC traffic?
Both support gRPC, but NGINX generally provides better raw performance for gRPC streams. However, Traefik's lack of reload-based disruptions makes it more stable for long-lived gRPC connections during configuration updates.

Can I use both in the same cluster?
Yes. By using different ingressClassName values, you can run both controllers. This is a common pattern when migrating from one to the other or when separating internal and external traffic.

Migration and Adoption Checklist

If you are moving from NGINX to Traefik (or vice versa), avoid a "big bang" migration. The risk of a total outage is too high.

Parallel Deployment: Install the new controller alongside the old one. Use different ingressClassName values to ensure they don't fight over the same resources.
Canary Routing: Use a DNS weight shift (e.g., Route53 or Cloudflare) to send 5% of traffic to the new controller.
Middleware Mapping: Map your NGINX snippets to Traefik Middlewares. Document every custom header or rewrite rule.
Cert Validation: If moving to Traefik, verify ACME challenge propagation before deleting your cert-manager setup.
Observability Sync: Ensure your Prometheus scrapers are updated to pull the specific metrics format of the new controller. If you're struggling with pod stability during this rollout, see our guide on /troubleshooting/crashloopbackoff-kubernetes to debug fast.

Once 100% of traffic is stable on the new controller, prune the old Ingress resources and uninstall the legacy controller to reclaim cluster resources.

Fix GitLab CI Docker daemon connection error in 3 steps

DevOps Start — Fri, 19 Jun 2026 18:59:16 +0000

This article was originally published on DevOpsStart.com. Here's a quick fix for the dreaded 'Cannot connect to the Docker daemon' error in GitLab CI.

The Problem

You have a GitLab CI job that tries to run docker build or docker info, and it fails with this error:

Cannot connect to the Docker daemon at tcp://docker:2375. Is the docker daemon running?

The error means your job is running inside a container that has the docker CLI installed but no Docker daemon (dockerd) service to talk to. The default docker image from Docker Hub ships only the client. Without a running daemon, every docker command returns this error.

Root Causes

Three things cause this error.

1. Missing `docker` service in `.gitlab-ci.yml`

You must define a services block that runs the daemon. The standard approach is:

services:
  - docker:dind

Without this, your job has no daemon to connect to.

2. Incorrect or missing `DOCKER_HOST` variable

When you use the docker executor (the default GitLab Runner executor), the daemon runs as a separate container in the same Pod. You must tell the client where to find it:

variables:
  DOCKER_HOST: tcp://docker:2375

If DOCKER_HOST is set to tcp://localhost:2375 or missing entirely, the client looks for a local Unix socket that does not exist inside your job container. The job container has no socket file.

3. Runner not in `privileged` mode

The docker:dind service requires privileged mode to run its own Docker daemon. If your GitLab Runner's config.toml has privileged = false or is unset, the dind container cannot start. Check your runner configuration:

$ cat /etc/gitlab-runner/config.toml
[[runners]]
  name = "my-runner"
  url = "https://gitlab.com/"
  token = "..."
  executor = "docker"
  [runners.docker]
    privileged = true
    pull_policy = "always"

If privileged is false or absent, the dind service never boots, and you get the Cannot connect to the Docker daemon error.

The Solution

A complete fix takes three steps.

Step 1: add the `docker:dind` service

Add this to your .gitlab-ci.yml:

image: docker:24.0.7

services:
  - name: docker:dind
    alias: docker

variables:
  DOCKER_HOST: tcp://docker:2375
  DOCKER_TLS_CERTDIR: ""

The DOCKER_HOST value tcp://docker:2375 tells the client to use the service container named docker (the GitLab Runner resolves the service alias to the container hostname). The DOCKER_TLS_CERTDIR: "" disables TLS for simplicity.

Step 2: test with a simple job

Add a minimal job to verify the fix works:

build:
  stage: build
  script:
    - docker info
    - docker build -t my-app .

The docker info command confirms the daemon is reachable. If it succeeds, your pipeline is ready.

Step 3: check runner logs

If docker info still fails, inspect the runner logs. On your GitLab Runner host, run:

$ docker logs <runner-container-id> --tail 50

Look for lines like:

Starting container service-docker:dind-0 ...

If you see no such line, the service never started. Check privileged mode again in config.toml.

Prevention

To avoid this error in future:

Always add services: - docker:dind to any job that needs docker commands.
Always set DOCKER_HOST: tcp://docker:2375 as a top-level variable in your pipeline.
Verify your GitLab Runner's config.toml has privileged = true before you push new jobs.
For a production-grade setup, consider using the docker:24.0.7-dind image variant. It bundles the daemon and the client in one image, reducing the chance of version mismatches.

How to Fix Kubernetes CrashLoopBackOff in Production

DevOps Start — Fri, 19 Jun 2026 09:04:30 +0000

Problem: What CrashLoopBackOff actually means

When you see CrashLoopBackOff in your kubectl get pods output, you aren't looking at a specific error, but a state. It is a symptom. It tells you that the kubelet tried to start your container, the container crashed, and Kubernetes is now waiting before trying again.

To prevent the API server and the node from being hammered by a process that crashes instantly, Kubernetes implements an exponential backoff delay. The first restart happens quickly, but subsequent failures increase the wait time (10s, 20s, 40s, up to a maximum of 5 minutes). If you don't intervene, your pod will spend more time waiting than running, making it nearly impossible to catch logs in real-time.

You can find more details on the pod lifecycle in the official Kubernetes Documentation.

Root Causes: The "Why" behind the crash

Diagnosing a crash loop requires moving from the symptom to the cause. I've seen this fail most often in clusters with >50 nodes where configuration drift becomes common. Root causes generally fall into three severity tiers.

Low Severity: Configuration and Environment

These are "fail-fast" errors. The application starts, realizes it is missing a critical piece of information, and exits. Common culprits include:

Missing ConfigMaps or Secrets: The pod is configured to expect a volume or environment variable that doesn't exist.
Incorrect Environment Variables: A typo in a database URL or a missing API key.
Wrong Command/Args: An incorrect entrypoint in the Dockerfile or a typo in the args section of the YAML.

Medium Severity: Resource and Infrastructure

These crashes are often intermittent or happen shortly after the app begins processing traffic:

OOMKilled (Exit Code 137): The container exceeded its memory limit. This is the most common production crash, often reducing availability by 100% for that specific replica.
Liveness Probe Death Spiral: The application takes 30 seconds to start, but the liveness probe kills it after 10 seconds. The pod is healthy, but Kubernetes thinks it's dead.
Storage Permissions: The container user doesn't have write access to a mounted PersistentVolume.

High Severity: External Dependencies

The application is healthy, but its environment is hostile:

Database Connection Timeouts: The app crashes because it cannot reach the DB due to a firewall rule or incorrect credentials.
DNS Resolution Failures: CoreDNS issues prevent the app from finding other services within the cluster.
Dependency API Outages: A hard dependency (like an external Auth provider) is down, and the app isn't designed to handle the failure gracefully.

Solution: Step-by-Step Production Triage

When a production service goes into CrashLoopBackOff, follow this severity-based logic tree. Do not guess; use the data provided by the cluster.

Step 1: The Rapid Triage

Start by checking the pod status and events. This tells you if the crash is happening because of the image, the scheduler, or the application itself.

kubectl describe pod <pod-name>

Look at the Containers section for the Last State. You will see an Exit Code. This is your most important clue.

Expected Output:
You should see a section like this:
Last State: Terminated
Reason: Error
Exit Code: 137

Step 2: Decoding the Exit Code

Map the exit code from describe to your action plan:

Exit Code 0: The app finished its task and exited. If this is a Deployment, it shouldn't happen. You likely need a Job instead.
Exit Code 1: General application crash. Move to Step 3 to check logs.
Exit Code 137: OOMKilled. Increase your memory limits in the deployment YAML.
Exit Code 139: Segmentation fault. This is usually a binary incompatibility or a memory corruption issue in the code.
Exit Code 143: SIGTERM. The pod was told to stop but didn't do it gracefully.

Step 3: Retrieving the "Hidden" Logs

If a pod is crashing, kubectl logs <pod-name> often returns nothing because the current container has just started and hasn't logged anything yet. You must check the logs of the previous failed instance.

kubectl logs <pod-name> --previous

If the logs show a connection timeout to a database, check your network policies or secret values. If you need a faster way to handle urgent fixes, you can use /tips/rapid-rollback-kubectl-set-image-for-urgent-fixes to revert to a known working image version.

Step 4: Debugging "Silent" Crashes with Ephemeral Containers

Sometimes logs are empty and the exit code is vague. In Kubernetes v1.23+, you can use kubectl debug to spin up a sidecar container with debugging tools (like curl, dig, or vim) that shares the same process namespace as the crashing pod.

kubectl debug -it <pod-name> --image=busybox --target=<container-name>

Once inside, you can inspect the /tmp directory, check network connectivity, or look at the filesystem to see if a config file was mounted incorrectly. For a wider array of helpful commands, refer to the /tips/kubectl-essential-commands guide.

Prevention: Stopping the loop before it starts

Prevention is about shifting from "fixing" to "hardening". Implement these four strategies to eliminate CrashLoopBackOff in production:

Right-Size Resources: Use a Vertical Pod Autoscaler (VPA) in Recommender mode to find the actual memory usage of your app. Set your limits roughly 20% higher than your requests to handle spikes without triggering an OOMKill.
Graceful Probes: Never set a livenessProbe with the same timing as your readinessProbe. Give your application a initialDelaySeconds that covers its worst-case startup time. If your app takes 20 seconds to boot, set the delay to 30 seconds.
Robust Signal Handling: Ensure your application handles SIGTERM (Exit Code 143). If your app ignores this signal, Kubernetes will eventually force-kill it with SIGKILL (Exit Code 137) after the terminationGracePeriodSeconds expires.
Better Observability: Integrate deep monitoring. If you are running AI workloads, implementing /tutorials/llm-observability-on-kubernetes-a-practical-guide will help you identify if crashes are caused by GPU memory exhaustion or model loading timeouts.

FAQ

Why does my pod stay in CrashLoopBackOff even after I fix the ConfigMap?
Kubernetes uses an exponential backoff. If your pod has crashed multiple times, it might wait up to 5 minutes before the next restart attempt. You can force an immediate restart by deleting the pod: kubectl delete pod <pod-name>.

Is an Exit Code 137 always an OOMKill?
Almost always, but not exclusively. It means the process received a SIGKILL (Signal 9). While the kubelet usually sends this when a container exceeds its memory limit, it can also happen if an external process or the node's OOM killer terminates the process.

How can I prevent a crashing pod from affecting other pods on the same node?
Define strict resources.limits. Without memory limits, a single leaking container can consume all node memory, triggering the Node-level OOM killer and causing unrelated pods to be evicted.

Conclusion and Next Steps

CrashLoopBackOff is a safety mechanism, not a bug. By isolating the exit code and inspecting previous logs, you can quickly determine if you are facing a configuration error, a resource constraint, or a dependency failure.

To further harden your production environment, take these next steps:

Audit your livenessProbes to ensure they aren't too aggressive.
Deploy a Vertical Pod Autoscaler (VPA) to get data-driven memory limits.
Implement a structured logging pipeline (EFK/ELK) so you don't have to rely on --previous logs during an incident.

Senior SRE Interview Questions & Answers for 2026

DevOps Start — Thu, 18 Jun 2026 08:56:33 +0000

Introduction

Landing a Senior Site Reliability Engineering (SRE) role in 2026 requires more than just knowing how to write a YAML file or explaining the difference between a Pod and a Deployment. The industry has shifted. We have moved past the early adoption phase of Kubernetes and into the era of Platform Engineering, where the goal is not just to manage infrastructure but to build an Internal Developer Platform (IDP) that enables self-service.

Interviewers no longer test for rote memorization of Linux commands. They look for architectural judgment, the ability to manage cognitive load for developers and a deep understanding of how reliability impacts the bottom line. A Senior SRE is expected to be a force multiplier, not just a firefighter.

In this guide, you will find the high-signal questions currently being asked at top-tier tech companies, along with the senior-level reasoning required to answer them. We cover everything from cell-based architectures and OpenTelemetry to the psychological nuances of blameless post-mortems.

The Evolution of SRE in 2026

Before diving into specific questions, you must understand the current landscape. SRE has largely merged with Platform Engineering. The focus is now on "Golden Paths" (standardized, supported ways to deploy software) to reduce developer friction.

Furthermore, AIOps is no longer a buzzword. Senior SREs are now expected to integrate LLMs into their observability stacks for anomaly detection and automated root cause analysis. If you discuss observability, you should reference patterns for monitoring non-deterministic AI workloads, such as tracking token latency and prompt cache hit rates, to show you understand how to monitor LLM-integrated applications.

Advanced System Design for Reliability

At the senior level, scalability is not just about adding more replicas to a deployment. It is about blast radius reduction.

Question: How do you design a system to prevent a single regional failure from taking down your entire global platform?

What they are looking for: Knowledge of Global Server Load Balancing (GSLB), Anycast routing and specifically "Cell-based Architecture."

The Senior Answer:
Do not just say "I would use a multi-region deployment." Explain the trade-offs. Mention that while multi-region provides availability, it introduces data consistency challenges tied to the CAP theorem.

Explain the concept of Cells. Instead of one giant regional cluster, divide infrastructure into isolated cells (smaller, independent units of deployment). If a bug is deployed or a database locks up, it only affects one cell (e.g., 5% of users) rather than the entire region.

Key components to mention:

Route53/Cloudflare: For traffic steering based on health checks.
Cell Router: A thin layer that maps a User ID to a specific cell.
Asynchronous Replication: Using tools like CockroachDB or DynamoDB Global Tables to handle state across regions without killing latency.

Question: How do you handle "Thundering Herd" problems in a distributed system?

What they are looking for: Understanding of caching strategies and request shedding.

The Senior Answer:
A thundering herd occurs when many clients retry a failed request simultaneously, crashing the recovering service. I implement three layers of defense:

Exponential Backoff with Jitter: Ensure clients do not retry at the exact same millisecond. A simple sleep(2^attempt + random_jitter) prevents synchronized spikes.
Circuit Breakers: Use a service mesh (like Istio or Linkerd) to trip the circuit and fail fast when a downstream service is overwhelmed.
Request Prioritization: Implement a priority queue where critical traffic (e.g., checkout) is processed before background tasks (e.g., analytics).

Observability vs. Monitoring

Monitoring tells you that something is wrong; observability allows you to understand why it is wrong without shipping new code.

Question: We have millions of metrics, but our dashboards are noisy. How do you define "actionable" SLIs and SLOs?

What they are looking for: The ability to link technical metrics to business value.

The Senior Answer:
Most teams make the mistake of measuring CPU usage as an SLI. CPU is a cause, not a symptom. A senior SRE focuses on the user experience.

I use the Four Golden Signals (Latency, Traffic, Errors, Saturation) but tie them to specific user journeys. For example, instead of "API Error Rate," I define the SLI as "Percentage of successful 'Add to Cart' requests completed within 500ms over a rolling 30-day window."

If the Error Budget is exhausted, the action is a freeze on feature releases to focus on reliability. This turns a technical metric into a business decision.

Question: How do you handle high-cardinality data in a distributed tracing environment?

What they are looking for: Experience with OpenTelemetry (OTel) and the cost implications of telemetry.

The Senior Answer:
High cardinality (e.g., putting a unique UserID in every metric tag) can crash a Prometheus instance or lead to massive bills in Datadog.

I recommend moving to OpenTelemetry for a vendor-agnostic approach. To handle cardinality, I implement Head-based or Tail-based Sampling. Instead of keeping 100% of traces, we keep 100% of errors and 5% of successful requests. This provides the necessary visibility into failures without the storage overhead of every single "200 OK" request.

Incident Management and the "SRE Mindset"

Question: You are leading an incident where a cascading failure is occurring across three microservices. How do you manage the situation?

What they are looking for: Command and control (ICS), communication skills and technical triage.

The Senior Answer:
First, I establish roles: an Incident Commander (IC) to coordinate, a Communications Lead to update stakeholders and an Ops Lead to handle the technical fix. I avoid having too many people directing the technical execution.

Technically, my first goal is to stop the bleeding, not find the root cause. I look for the bottleneck service and apply aggressive load shedding or disable non-essential features using feature flags to lower the pressure on the system. Once the system is stable, we move to the Post-Mortem phase.

Question: How do you handle a situation where a Product Manager insists on a feature release that you know will risk the Error Budget?

What they are looking for: Negotiation skills and a commitment to the SRE philosophy.

The Senior Answer:
I do not frame it as "No, we cannot do this." I frame it as a risk management conversation.

I show the current Error Budget burn rate. If we are at 10% of our budget for the month, I explain that a failed release could lead to an outage that violates our SLA, potentially costing the company $X per hour in revenue. I suggest a Canary Deployment strategy, releasing to 1% of users first. This allows the PM to get the feature out while limiting the blast radius.

Infrastructure as Code (IaC) and GitOps at Scale

Question: Terraform is becoming slow and state locking is a constant issue for our team of 50 engineers. How do you scale your IaC?

What they are looking for: Experience with state management and modularization.

The Senior Answer:
The monolithic state is a common failure point. I first implement state splitting, breaking the infrastructure into logical layers (e.g., Networking, Database, Application) so that a change to an app does not require locking the VPC state.

For teams moving toward massive scale, I evaluate the transition to a programmatic IaC approach using Pulumi or OpenTofu, which allows for better testing and abstraction than HCL. To automate the rollout, I implement a GitOps pipeline using Argo CD or Flux to ensure the cluster state always matches the Git repository.

Cloud-Native Security (DevSecOps)

Question: How do you implement a "Zero Trust" network in a Kubernetes environment?

What they are looking for: Knowledge of Network Policies and eBPF.

The Senior Answer:
Default Kubernetes networking is flat, meaning any pod can talk to any pod. To implement Zero Trust, I start with a Default Deny Network Policy for all namespaces.

Then, I use a CNI that supports eBPF, such as Cilium. eBPF allows us to enforce security policies at the kernel level rather than relying on iptables, which provides better performance and deeper visibility into the network flow. I also integrate a service mesh like Istio to enforce Mutual TLS (mTLS) for all service-to-service communication, ensuring that identities are verified via certificates, not just IP addresses.

Practical Troubleshooting Scenarios

In senior interviews, you will often get a whiteboard scenario. The interviewer does not want the right answer immediately; they want to see your debugging methodology.

Scenario: "The database CPU is spiking to 90%, but the application traffic (requests per second) is flat. How do you debug this?"

The Senior Approach:
I follow a top-down diagnostic path:

Identify the Workload: Is the CPU spike caused by an increase in total queries or is a small number of queries becoming more expensive? I check the Slow Query Log.
Check for Locking/Contention: I look for long-running transactions or lock waits. A single unoptimized query hitting a table without an index can spike CPU even if traffic is flat.
External Factors: I check for background jobs. Did a database backup start? Is an ETL process running a massive join?
Resource Exhaustion: I check if the DB is swapping to disk or if there is a memory leak causing excessive Garbage Collection.

Scenario: "A new deployment caused a spike in 5xx errors. The pods are running, but the app is failing. What do you do?"

The Senior Approach:

Immediate Mitigation: First, I trigger a rollback to the last known good image using kubectl rollout undo deployment/<deployment-name>. Speed of recovery is the priority.
Log Analysis: I check the logs for "Panic" or "Out of Memory" (OOM) errors. If pods are restarting, I check if it is a probe failure (Liveness/Readiness).
Diffing: I compare the configuration changes between the failed version and the previous version. Was a secret missing? Did an environment variable change?
Trace Analysis: I use distributed tracing to see if the 5xx is coming from the app itself or a downstream dependency that the new version is calling differently.

FAQ

What is the biggest difference between a DevOps Engineer and an SRE?

DevOps is a cultural philosophy focused on breaking down silos between Dev and Ops. SRE is a specific implementation of DevOps. SRE applies software engineering principles to operations problems, focusing on SLIs, SLOs and Error Budgets.

Which tool should I learn first: Terraform or Pulumi?

Terraform is the industry standard and essential for any resume. However, Pulumi is gaining traction in Platform Engineering because it allows you to use general-purpose languages (TypeScript, Python, Go), making it easier to build complex logic for internal platforms.

How do I handle "on-call burnout" as a Senior SRE?

Burnout is a systemic failure, not a personal one. I advocate for Operational Load tracking. If the team spends more than 50% of their time on toil (manual, repetitive work), I negotiate with leadership to halt feature work and dedicate a stability sprint to automate the causes of the alerts.

Conclusion and Next Steps

Passing a Senior SRE interview is about demonstrating that you can think in terms of systems, trade-offs and business risk. You are not just there to keep the lights on; you are there to build a system that can survive the failure of its individual components.

Your Action Plan:

Audit your experience: For every project on your resume, identify the trade-off. Why did you choose X over Y? What was the cost?
Master the Golden Signals: Be ready to explain exactly how you would measure the reliability of a specific business feature.
Practice the Cell mindset: Read up on how companies like AWS and Meta use cell-based architectures to limit blast radius.
Hands-on with OTel: Deploy an OpenTelemetry collector in a lab environment to understand how traces and metrics flow.

LLM Observability on Kubernetes: A Practical Guide

DevOps Start — Wed, 17 Jun 2026 09:03:45 +0000

Monitoring traditional applications often feels like a well-trodden path. You set up logs, grab some metrics, and perhaps add a few traces. However, integrating Large Language Models (LLMs) or AI agents, especially when running on Kubernetes, fundamentally changes this paradigm. LLM observability on Kubernetes is a different beast entirely, demanding a more nuanced approach than standard application monitoring.

This tutorial is designed for DevOps, ML, or platform engineers grappling with the unique challenges of monitoring LLM-powered applications and AI agents on Kubernetes. You'll learn why traditional tools fall short and how to build a practical, end-to-end observability pipeline. We will use battle-tested Kubernetes-native tools like Prometheus, Grafana, Loki, and OpenTelemetry. The tutorial includes hands-on experience with a simple AI agent application, instrumenting it, deploying it to Kubernetes, and setting up a unified observability stack to monitor its performance, cost, and behavior.

Why Traditional Observability Fails for LLM-Powered AI Agents

Generative AI applications, particularly those powered by LLMs and complex AI agents, introduce a new dimension to observability that traditional methods struggle to address. It's no longer just about CPU and memory; it's about context, coherence, and cost per token. For effective LLM observability on Kubernetes, your standard monitoring stack needs an upgrade.

Here’s why traditional monitoring falls short for LLMs and AI agents:

Non-Determinism: LLMs are inherently non-deterministic. The same prompt can yield different responses, making it hard to track performance or identify regressions solely through request/response codes. You need to understand the content and quality of responses. For example, a successful HTTP 200 response doesn't indicate if an LLM response was a hallucination.
Complex Prompt/Response Dynamics: The interaction isn't a simple input-output. It involves intricate prompt engineering, context windows, and diverse response formats. Observing just the HTTP status code tells you nothing about a hallucination or an off-topic answer.
Token Usage & Cost: Every interaction with an LLM consumes tokens, which directly translates to cost, especially with proprietary models like OpenAI's GPT-4. Monitoring token usage per request, per user, or per session is critical for cost control and capacity planning. Traditional metrics simply don't capture this. For example, a simple query might cost fractions of a cent, but 10,000 queries per minute can quickly escalate costs to thousands of dollars daily.
Latency Nuances: LLM response times are often dominated by token generation, not just initial processing. You need to differentiate between prompt processing latency and response streaming latency for accurate performance tuning of your LLM applications.
AI Agent Complexity: This is where it gets really interesting. AI agents involve multiple steps: planning, tool selection, tool execution, memory management, and iterative reasoning. Each step is a potential failure point. You need to trace the agent's entire decision path, track tool call successes/failures, and understand how the agent arrived at its final answer. A simple error log tells you that something failed, but not why the agent chose a particular path or tool.
"Black Box" Nature: While you can control the inputs, the internal workings of large foundation models are opaque. Observability needs to shine a light on the model's behavior at the application layer.

On Kubernetes, you're adept at monitoring resource utilization at the pod and container level. But for LLM applications, this is only half the story. You need to correlate Kubernetes infrastructure metrics with application-specific LLM and agent metrics to get a complete picture of your LLM observability on Kubernetes.

Core Observability Pillars for LLM Workloads on Kubernetes

The traditional pillars of observability (Logs, Metrics, and Traces) remain foundational, but they need to be adapted and extended for LLM workloads on Kubernetes. This integrated approach is key to achieving comprehensive LLM observability.

Logs: The Narrative of Your AI Agent's Decisions

For LLM applications and AI agents, logs are more than just error messages. They are the narrative of your agent's reasoning process, crucial for understanding and debugging LLM observability on Kubernetes.

Prompt/Response Logging: Crucial for debugging and understanding model behavior. Log the full input prompt, context, and the LLM's raw response. This helps diagnose why a model might have hallucinated or gone off-topic.
Agent Decision Logs: For AI agents, log every significant step:
- Initial plan formulation.
- Tool selection decisions.
- Inputs and outputs of each tool call.
- Re-planning attempts.
- Intermediate reasoning steps.
Structured Logging: Absolutely essential. Use JSON logging to include metadata like trace_id, span_id, user_id, session_id, model_name, temperature, token_counts, and safety_flags. This makes logs queryable and correlatable with metrics and traces.
Kubernetes Integration: Leverage Kubernetes' standard output (stdout/stderr) for logs. A log collector like Fluent Bit, deployed as a DaemonSet, can then ship these structured logs to a centralized logging solution like Loki, Elasticsearch, or Splunk.

Metrics: Quantifying LLM Performance and Cost

Metrics provide the quantitative insights into your LLM application's health, performance, and operational cost. These are vital for effective LLM observability on Kubernetes.

Application-Level Metrics: These are paramount. We'll dive into specific examples shortly, but think latency, token usage, error rates for LLM calls, and specific agent action success rates.
Resource Utilization: Standard Kubernetes metrics for CPU, memory, network I/O are still vital. For GPU-accelerated inference, monitoring GPU utilization, memory, and temperature is critical. Prometheus can scrape these from Kubernetes kube-state-metrics and node-exporter (or specific GPU exporters like DCGM Exporter).
Cost Metrics: Beyond just resource utilization, track API calls to external LLMs and internal token consumption. This allows for real-time cost estimation and budgeting.
Kubernetes Integration: Prometheus, with its ServiceMonitor and PodMonitor custom resources, is perfectly suited for scraping application-level metrics directly from your LLM application pods.

Traces: Following the AI Agent's Chain of Thought

Distributed tracing is arguably the most powerful pillar for debugging complex, multi-step AI agents, offering deep insights into LLM observability on Kubernetes. It visualizes the entire execution path.

End-to-End Flow: A trace provides a timeline view of a single request or agent interaction, spanning across multiple services and internal functions. For an AI agent, this means seeing the initial user query, the LLM call, the tool selection logic, the tool execution, and the final response, all linked together.
Span Granularity: Create spans for:
- Incoming request.
- Each LLM API call (input prompt, model, temperature, output response).
- Each step of the agent's reasoning chain (e.g., "planning step", "tool invocation", "context retrieval").
- Each external tool call (e.g., database lookup, external API call).
Context Propagation: Essential for connecting spans across service boundaries. OpenTelemetry automatically handles this for many protocols.
Semantic Conventions: Use OpenTelemetry's semantic conventions for LLM operations. This ensures consistency and makes traces easier to interpret across different tools.
Kubernetes Integration: Deploy an OpenTelemetry Collector within your Kubernetes cluster to receive traces from your instrumented applications and export them to a tracing backend like Jaeger or Tempo.

Key Metrics for LLM Apps & AI Agents on Kubernetes

Let's get specific about the metrics you must track for LLM-powered applications and AI agents to ensure comprehensive LLM observability on Kubernetes.

Performance Metrics for LLM Applications

These reveal how quickly and efficiently your LLM application is serving requests.

Prompt Processing Latency: Time taken from receiving a prompt to sending it to the LLM API.
- Prometheus metric type: histogram
- Example: llm_prompt_processing_seconds_bucket
Response Generation Latency: Time taken for the LLM to generate the full response, or the time until the first token is received for streaming.
- Prometheus metric type: histogram
- Example: llm_response_generation_seconds_bucket
Total Request Latency: End-to-end time for a user query.
- Prometheus metric type: histogram
- Example: llm_agent_total_request_seconds_bucket
Throughput (Queries Per Second): Number of requests or agent interactions handled per second.
- Prometheus metric type: counter
- Example: llm_agent_requests_total
Tool Call Latency: Time taken for specific tools invoked by the agent (e.g., database query, external API call).
- Prometheus metric type: histogram
- Example: agent_tool_call_seconds_bucket{tool_name="weather_api"}

Resource Utilization Metrics for LLMs on Kubernetes

While standard Kubernetes metrics cover CPU and memory, pay special attention to:

GPU Utilization: Percentage of GPU compute units being used. Critical for local inference.
- Prometheus metric type: gauge
- Example: gpu_utilization_percentage
GPU Memory Usage: Amount of memory allocated on the GPU.
- Prometheus metric type: gauge
- Example: gpu_memory_usage_bytes
CPU and Memory (per Pod): Standard container_cpu_usage_seconds_total and container_memory_working_set_bytes from kube-state-metrics and node-exporter.

LLM Cost Monitoring Metrics

Directly impacting your budget, these are often overlooked in initial deployments and are crucial for comprehensive LLM observability.

Input Token Count: Number of tokens sent in the prompt.
- Prometheus metric type: counter
- Example: llm_input_tokens_total
Output Token Count: Number of tokens received in the response.
- Prometheus metric type: counter
- Example: llm_output_tokens_total
Total LLM API Calls: Number of requests made to the underlying LLM (internal or external).
- Prometheus metric type: counter
- Example: llm_api_calls_total{model_name="gpt-4"}
Estimated Cost: A derived metric calculated by multiplying token counts or API calls by their respective per-unit costs. This is often best handled in Grafana using PromQL. For example, if input tokens cost $0.01 per 1000 and output tokens cost $0.03 per 1000, you can calculate real-time cost.
- Prometheus metric type: (Derived in Grafana)
- Example (PromQL): (sum(rate(llm_input_tokens_total[5m])) / 1000 * 0.01) + (sum(rate(llm_output_tokens_total[5m])) / 1000 * 0.03)

Model Quality & AI Agent Behavior Metrics

These are more challenging to define but crucial for understanding the LLM's effectiveness and the performance of your AI agents.

Safety Guardrail Activations: Count of times a safety or moderation filter was triggered (e.g., content flagged as unsafe).
- Prometheus metric type: counter
- Example: llm_safety_violations_total
Hallucination Flags: If your application has logic to detect potential hallucinations, count these. This is often heuristic.
- Prometheus metric type: counter
- Example: llm_hallucinations_detected_total
Agent Goal Completion Rate: For multi-step agents, track the percentage of interactions where the agent successfully achieved its objective.
- Prometheus metric type: counter (for successes/failures)
- Example: agent_goal_completions_total, agent_goal_failures_total
Tool Call Success/Failure Rates: Track how often an agent's chosen tool executed successfully.
- Prometheus metric type: counter
- Example: agent_tool_call_success_total{tool_name="database_lookup"}, agent_tool_call_failure_total{tool_name="external_api"}
Number of Agent Steps/Iterations: How many steps or LLM calls an agent took to complete a task. High numbers might indicate inefficiency.
- Prometheus metric type: histogram or gauge
- Example: agent_iteration_steps_count_bucket

Instrumenting LLM Applications & Agents for Observability

Instrumentation is where you expose the internal state of your LLM application and AI agent. OpenTelemetry is the gold standard here for its vendor-neutral approach and comprehensive support for traces, metrics, and logs, making it ideal for LLM observability on Kubernetes.

Why OpenTelemetry is Essential for LLM Observability

OpenTelemetry (OTel) provides a set of APIs, SDKs, and tools to instrument your application to generate and export telemetry data. It's language-agnostic and supports various exporters, allowing you to switch observability backends without re-instrumenting your code. For complex distributed systems like Kubernetes-hosted AI agents, OTel's distributed tracing capabilities are invaluable for understanding the flow of LLM interactions.

Instrumentation Methods for LLM Observability

1. OpenTelemetry for Tracing and Metrics

This is the recommended approach for deep, custom instrumentation for LLM observability.

Traces:
- Use the OpenTelemetry SDK for your language (e.g., opentelemetry-python).
- Wrap LLM calls, tool calls, and agent steps with spans.
- Add relevant attributes (e.g., model_name, prompt_hash, token_counts, tool_name, status) to spans.
Metrics:
- While OpenTelemetry can also generate metrics, for simple counter/gauge/histogram metrics that Prometheus can scrape directly, using your language's Prometheus client library (e.g., prometheus_client for Python) is often simpler for HTTP exposition.
- For richer, more complex metrics or if you want to use OTLP for metrics, OpenTelemetry's metrics API is also powerful.

2. Framework Callbacks (e.g., LangChain)

If you're using an LLM framework like LangChain, many provide callback systems that are perfect for capturing agent activity for better LLM observability.

LangChain Callbacks: Implement custom callbacks (BaseCallbackHandler) to log, emit metrics, or create traces at various points:
- on_llm_start/on_llm_end
- on_tool_start/on_tool_end
- on_chain_start/on_chain_end
- on_agent_action/on_agent_finish
Integrating with OpenTelemetry: Within these callbacks, you can explicitly create OpenTelemetry spans and add attributes.

3. Custom Wrappers

For simpler LLM integrations or when frameworks don't offer enough hooks, you can create custom wrapper functions around your LLM API calls and agent logic.

import time
import logging
from prometheus_client import Histogram, Counter
from opentelemetry import trace
from opentelemetry.propagate import set_global_textmap
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import (
    ConsoleSpanExporter,
    BatchSpanProcessor,
)
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.sdk.trace.sampling import ALWAYS_ON
from opel_propagate_b3 import B3Format # pip install opel-propagate-b3

# --- OpenTelemetry Setup (for Tracing) ---
# For a real application, you'd configure the OTLPSpanExporter to point to your OpenTelemetry Collector.
# For demonstration, we'll use ConsoleSpanExporter or a local OTLP endpoint if available.
resource = Resource.create({"service.name": "llm-agent-app", "service.version": "1.0.0"})
provider = TracerProvider(resource=resource, sampler=ALWAYS_ON)
# In a K8s cluster, this endpoint should point to the OTel Collector service.
otlp_exporter = OTLPSpanExporter(endpoint="http://otel-collector.observability:4317", insecure=True)
span_processor = BatchSpanProcessor(otlp_exporter)
provider.add_span_processor(span_processor)
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("llm.agent.tracer")

# Propagator for B3 headers (commonly used in microservices)
set_global_textmap(B3Format())

# Instrument requests library for outgoing HTTP calls (e.g., to actual LLM API)
RequestsInstrumentor().instrument()

# --- Prometheus Metrics Setup ---
# Latency of the LLM call itself
LLM_CALL_LATENCY_SECONDS = Histogram(
    'llm_call_latency_seconds',
    'Latency of LLM API calls in seconds',
    ['model_name', 'status_code'],
    buckets=[0.1, 0.5, 1.0, 2.0, 5.0, 10.0, 20.0]
)

# Total input and output tokens
LLM_INPUT_TOKENS_TOTAL = Counter(
    'llm_input_tokens_total',
    'Total input tokens processed by LLM',
    ['model_name']
)
LLM_OUTPUT_TOKENS_TOTAL = Counter(
    'llm_output_tokens_total',
    'Total output tokens generated by LLM',
    ['model_name']
)

# Agent tool call success/failure
AGENT_TOOL_CALLS_TOTAL = Counter(
    'agent_tool_calls_total',
    'Total agent tool calls',
    ['tool_name', 'status'] # status: 'success' or 'failure'
)

# Agent overall request latency
AGENT_REQUEST_LATENCY_SECONDS = Histogram(
    'agent_request_latency_seconds',
    'End-to-end latency of agent requests in seconds',
    buckets=[0.5, 1.0, 2.0, 5.0, 10.0, 20.0, 30.0, 60.0]
)

# --- Structured Logger Setup ---
# Note: In a real Flask app, this would be integrated into the app's logger.
# This is a simplified example.
logging.basicConfig(
    level=logging.INFO,
    format='%(message)s', # The formatter will produce JSON
    datefmt="%Y-%m-%dT%H:%M:%S%z"
)
logger = logging.getLogger(__name__)

# Override the default formatter to inject trace_id and span_id
class JsonFormatter(logging.Formatter):
    def format(self, record):
        log_data = {
            "timestamp": self.formatTime(record, self.datefmt),
            "level": record.levelname,
            "message": record.getMessage(), # Default to plain message
            "service": "llm-agent-app",
            "component": record.name,
        }

        # Attempt to parse message as JSON and merge
        try:
            msg_dict = json.loads(record.getMessage())
            log_data.update(msg_dict)
            if "message" not in msg_dict: # Ensure a 'message' field is always present
                log_data["message"] = record.getMessage()
        except json.JSONDecodeError:
            pass # Message is not JSON, use original record.getMessage()

        # Inject trace_id and span_id
        current_span = trace.get_current_span()
        if current_span and current_span.get_span_context().is_valid:
            log_data["trace_id"] = format(current_span.get_span_context().trace_id, 'x')
            log_data["span_id"] = format(current_span.get_span_context().span_id, 'x')
        else:
            log_data["trace_id"] = "0"
            log_data["span_id"] = "0"

        return json.dumps(log_data)

# Remove default handler and add one with custom formatter
# Note: For Flask, actual integration may differ.
for handler in logger.handlers[:]:
    logger.removeHandler(handler)
handler = logging.StreamHandler()
handler.setFormatter(JsonFormatter())
logger.addHandler(handler)


# --- Mock LLM and Agent Logic ---
import json
def mock_llm_call(prompt: str, model_name: str = "gpt-3.5-turbo"):
    """Simulates an LLM API call."""
    with tracer.start_as_current_span("llm_api_call") as span:
        span.set_attribute("model_name", model_name)
        span.set_attribute("llm.request.type", "chat")
        span.set_attribute("llm.prompts", json.dumps({"role": "user", "content": prompt})) # Store as JSON string

        start_time = time.perf_counter()
        time.sleep(0.5 + 0.5 * (len(prompt) / 100)) # Simulate variable latency
        end_time = time.perf_counter()
        latency = end_time - start_time

        # Simulate token usage
        input_tokens = len(prompt.split()) + 10 # Example
        output_tokens = 50 + len(prompt.split()) // 2 # Example

        response_text = f"This is a simulated response to: '{prompt}'. "
        response_action = None

        # Simulate agent action suggestion
        if "weather" in prompt.lower() or "forecast" in prompt.lower():
            response_text += "I recommend using a weather tool."
            response_action = {"tool_name": "weather_tool", "city": "New York"}
        elif "time" in prompt.lower():
            response_text += "I recommend using a time tool."
            response_action = {"tool_name": "time_tool"}
        elif "error" in prompt.lower():
            response_text += "Simulating an LLM error."
            status_code = 500
        else:
            response_text += "No tool suggested."
            status_code = 200

        span.set_attribute("llm.response.model", model_name)
        span.set_attribute("llm.response.tokens.total", input_tokens + output_tokens)
        span.set_attribute("llm.response.tokens.prompt", input_tokens)
        span.set_attribute("llm.response.tokens.completion", output_tokens)
        span.set_attribute("llm.response.content", response_text)
        span.set_attribute("http.status_code", status_code)


        LLM_CALL_LATENCY_SECONDS.labels(model_name=model_name, status_code=status_code).observe(latency)
        LLM_INPUT_TOKENS_TOTAL.labels(model_name=model_name).inc(input_tokens)
        LLM_OUTPUT_TOKENS_TOTAL.labels(model_name=model_name).inc(output_tokens)
        logger.info(json.dumps({
            "event": "llm_call_completed",
            "message": "LLM call finished",
            "model": model_name,
            "prompt": prompt,
            "response_summary": response_text,
            "latency_sec": f"{latency:.2f}",
            "input_tokens": input_tokens,
            "output_tokens": output_tokens,
            "status_code": status_code
        }))
        return {"text": response_text, "action": response_action, "status_code": status_code}

def mock_weather_tool(city: str):
    """Simulates an external weather API call."""
    with tracer.start_as_current_span("tool_call_weather") as span:
        span.set_attribute("tool.name", "weather_tool")
        span.set_attribute("tool.parameters.city", city)

        start_time = time.perf_counter()
        time.sleep(0.3)
        end_time = time.perf_counter()
        latency = end_time - start_time

        result = f"The weather in {city} is sunny with 25°C."
        status = "success"
        span.set_attribute("tool.status", status)
        span.set_attribute("tool.result", result)

        AGENT_TOOL_CALLS_TOTAL.labels(tool_name="weather_tool", status=status).inc()
        logger.info(json.dumps({
            "event": "tool_call",
            "message": "Weather tool call",
            "tool_name": "weather_tool",
            "city": city,
            "status": status,
            "latency_sec": f"{latency:.2f}"
        }))
        return result

def mock_time_tool():
    """Simulates an external time API call."""
    with tracer.start_as_current_span("tool_call_time") as span:
        span.set_attribute("tool.name", "time_tool")

        start_time = time.perf_counter()
        time.sleep(0.1)
        end_time = time.perf_counter()
        latency = end_time - start_time

        result = f"The current time is {time.strftime('%H:%M:%S')}."
        status = "success"
        span.set_attribute("tool.status", status)
        span.set_attribute("tool.result", result)

        AGENT_TOOL_CALLS_TOTAL.labels(tool_name="time_tool", status=status).inc()
        logger.info(json.dumps({
            "event": "tool_call",
            "message": "Time tool call",
            "tool_name": "time_tool",
            "status": status,
            "latency_sec": f"{latency:.2f}"
        }))
        return result


def llm_agent_handler(prompt: str, request_headers: dict = None):
    """The core logic of our simple LLM agent."""
    # Extract trace context from incoming request headers
    ctx = trace.Context()
    if request_headers:
        ctx = extract(request_headers)

    with tracer.start_as_current_span("llm_agent_request", context=ctx) as span:
        span.set_attribute("user.query", prompt)

        agent_start_time = time.perf_counter()

        logger.info(json.dumps({"event": "agent_started", "message": "Agent request initiated", "query": prompt}))

        llm_response_data = mock_llm_call(prompt, "gpt-3.5-turbo-mock")
        final_answer = llm_response_data["text"]

        # Step 2: Agent decision making based on LLM response
        if llm_response_data["action"]:
            action = llm_response_data["action"]
            tool_name = action["tool_name"]
            logger.info(json.dumps({"event": "agent_decision", "message": "Agent decided to call tool", "decision": "call_tool", "tool_name": tool_name}))
            with tracer.start_as_current_span("agent_decision_making") as decision_span:
                decision_span.set_attribute("decision.type", "tool_call")
                decision_span.set_attribute("decision.tool_name", tool_name)

                if tool_name == "weather_tool":
                    city = action.get("city", "New York")
                    tool_result = mock_weather_tool(city)
                    final_answer = f"{final_answer}\nTool result: {tool_result}"
                elif tool_name == "time_tool":
                    tool_result = mock_time_tool()
                    final_answer = f"{final_answer}\nTool result: {tool_result}"
                else:
                    logger.warning(json.dumps({"event": "unknown_tool", "message": "Agent attempted to call unknown tool", "tool_name": tool_name}))
                    AGENT_TOOL_CALLS_TOTAL.labels(tool_name=tool_name, status='failure').inc()

        agent_end_time = time.perf_counter()
        total_latency = agent_end_time - agent_start_time
        AGENT_REQUEST_LATENCY_SECONDS.observe(total_latency)

        span.set_attribute("agent.total_latency_seconds", total_latency)
        span.set_attribute("agent.final_response", final_answer)
        logger.info(json.dumps({"event": "agent_finished", "message": "Agent request completed", "final_response_summary": final_answer, "total_latency_sec": f"{total_latency:.2f}"}))
        return final_answer

This Python code snippet demonstrates the principles for building LLM observability on Kubernetes:

OpenTelemetry tracer: Used to create spans for the overall agent request, the LLM call, and each tool call. Attributes are added to these spans for rich context. The extract(request_headers) ensures trace context propagation.
Prometheus client metrics: Histogram for latencies and Counter for tokens and tool calls are exposed.
Structured Logging: logger.info calls output JSON logs, including trace_id and span_id for easy correlation. The custom JsonFormatter ensures proper structure.

Leveraging Kubernetes-Native Observability Tools for LLMs

Now, let's tie this into your Kubernetes cluster using familiar tools to enhance LLM observability on Kubernetes.

Prometheus & Grafana for LLM Metrics

Prometheus is the de-facto standard for metric collection in Kubernetes. Grafana provides powerful visualization.

Prometheus Operator: The easiest way to deploy and manage Prometheus in Kubernetes is using the Prometheus Operator. It introduces Custom Resource Definitions (CRDs) like ServiceMonitor and PodMonitor that simplify scraping configuration for your LLM applications.
Grafana: A leading open-source dashboarding tool that integrates seamlessly with Prometheus for visualizing LLM metrics.

Logging Solutions with Fluent Bit and Loki/Elasticsearch

Centralized logging is non-negotiable for debugging microservices, including LLM-powered applications.

Fluent Bit: A lightweight and efficient log processor and forwarder. Deploy it as a DaemonSet on your Kubernetes nodes to collect logs from container stdout/stderr.
Loki: Grafana Labs' log aggregation system, designed for cost-effectiveness and scalability, especially when paired with Grafana for visualization. It indexes metadata (labels) rather than full log content, which is great for LLM observability.
Elasticsearch/Kibana: Another popular stack for log aggregation and analysis, especially powerful for full-text search.

OpenTelemetry Collector for Traces and LLM Observability

The OpenTelemetry Collector is an essential component for distributed tracing within your Kubernetes environment.

Role: It receives, processes, and exports telemetry data. Your LLM applications send traces to the collector, which then forwards them to your chosen tracing backend (e.g., Jaeger, Tempo).
Deployment: Deploy the collector as a Deployment in your Kubernetes cluster, typically in your observability namespace.
Configuration: Configure it to receive OTLP (OpenTelemetry Protocol) traces, process them (e.g., batching, sampling), and then export them to your backend.

Hands-on Guide: Building LLM Observability Pipeline on Kubernetes

Let's put theory into practice. We'll deploy our simple LLM agent application, then set up Prometheus, Grafana, OpenTelemetry Collector, and Loki to observe it, building out comprehensive LLM observability on Kubernetes.

Prerequisites

Ensure you have:

A running Kubernetes cluster (Minikube, k3s, or a cloud-managed cluster).
kubectl (v1.25+) installed and configured to connect to your cluster.
helm (v3.10+) installed.
docker installed (to build the application image).
A Docker Hub account (or other container registry) to push your image.

Step 1: Prepare the LLM Agent Application

First, let's create our Python Flask application (app.py) which implements the agent logic and instrumentation we discussed.

# app.py
import time
import logging
import json
import os
from flask import Flask, request, jsonify
from prometheus_client import Histogram, Counter, generate_latest, REGISTRY
from opentelemetry import trace
from opentelemetry.propagate import set_global_textmap, extract
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import (
    BatchSpanProcessor,
)
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.wsgi import WsgiMiddleware
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.sdk.trace.sampling import ALWAYS_ON
from opentelemetry.metrics import set_meter_provider
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.exporter.prometheus import PrometheusMetricReader
from opel_propagate_b3 import B3Format # pip install opel-propagate-b3

# --- OpenTelemetry Setup ---
# Resource for the service
resource = Resource.create({
    "service.name": "llm-agent-app",
    "service.version": "1.0.0",
    "k8s.pod.name": os.getenv("HOSTNAME", "unknown"),
    "k8s.namespace.name": os.getenv("KUBERNETES_NAMESPACE", "default")
})

# Tracer Provider
provider = TracerProvider(resource=resource, sampler=ALWAYS_ON)
# Configure OTLP exporter to send traces to the OpenTelemetry Collector in the 'observability' namespace
otlp_exporter = OTLPSpanExporter(endpoint="http://otel-collector.observability:4317", insecure=True)
span_processor = BatchSpanProcessor(otlp_exporter)
provider.add_span_processor(span_processor)
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("llm.agent.tracer")

# Propagator for B3 headers (commonly used in microservices for context propagation)
set_global_textmap(B3Format())

# Instrument requests library for outgoing HTTP calls (e.g., to actual LLM API)
RequestsInstrumentor().instrument()

# --- Prometheus Metrics Setup ---
# Latency of the LLM call itself
LLM_CALL_LATENCY_SECONDS = Histogram(
    'llm_call_latency_seconds',
    'Latency of LLM API calls in seconds',
    ['model_name', 'status_code'],
    buckets=[0.1, 0.5, 1.0, 2.0, 5.0, 10.0, 20.0]
)

# Total input and output tokens
LLM_INPUT_TOKENS_TOTAL = Counter(
    'llm_input_tokens_total',
    'Total input tokens processed by LLM',
    ['model_name']
)
LLM_OUTPUT_TOKENS_TOTAL = Counter(
    'llm_output_tokens_total',
    'Total output tokens generated by LLM',
    ['model_name']
)

# Agent tool call success/failure
AGENT_TOOL_CALLS_TOTAL = Counter(
    'agent_tool_calls_total',
    'Total agent tool calls',
    ['tool_name', 'status'] # status: 'success' or 'failure'
)

# Agent overall request latency
AGENT_REQUEST_LATENCY_SECONDS = Histogram(
    'agent_request_latency_seconds',
    'End-to-end latency of agent requests in seconds',
    buckets=[0.5, 1.0, 2.0, 5.0, 10.0, 20.0, 30.0, 60.0]
)

# --- Structured Logger Setup ---
class JsonFormatter(logging.Formatter):
    def format(self, record):
        # Base log data
        log_data = {
            "timestamp": self.formatTime(record, self.datefmt),
            "level": record.levelname,
            "message": record.getMessage(),
            "service": "llm-agent-app",
            "component": record.name,
            "filename": record.filename,
            "lineno": record.lineno,
            "funcName": record.funcName,
        }

        # Attempt to parse message as JSON and merge. This supports logger.info(json.dumps({"event": "..."}))
        try:
            msg_dict = json.loads(record.getMessage())
            log_data.update(msg_dict)
            # If the JSON message didn't contain a 'message' field, keep the default one
            if "message" not in msg_dict:
                log_data["message"] = record.getMessage()
        except json.JSONDecodeError:
            pass # Message is not JSON, use original record.getMessage() as the 'message' field

        # Inject trace_id and span_id if available
        current_span = trace.get_current_span()
        if current_span and current_span.get_span_context().is_valid:
            log_data["trace_id"] = format(current_span.get_span_context().trace_id, 'x')
            log_data["span_id"] = format(current_span.get_span_context().span_id, 'x')
        else:
            log_data["trace_id"] = "0"
            log_data["span_id"] = "0"

        return json.dumps(log_data)

logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
# Clear existing handlers to prevent duplicate logs from Flask's default logger
for handler in logger.handlers[:]:
    logger.removeHandler(handler)
handler = logging.StreamHandler()
handler.setFormatter(JsonFormatter())
logger.addHandler(handler)


# --- Mock LLM and Agent Logic ---
def mock_llm_call(prompt: str, model_name: str = "gpt-3.5-turbo"):
    """Simulates an LLM API call."""
    with tracer.start_as_current_span("llm_api_call") as span:
        span.set_attribute("model_name", model_name)
        span.set_attribute("llm.request.type", "chat")
        span.set_attribute("llm.prompts", json.dumps([{"role": "user", "content": prompt}])) # Store as JSON string

        start_time = time.perf_counter()
        time.sleep(0.5 + 0.5 * (len(prompt) / 100)) # Simulate variable latency
        end_time = time.perf_counter()
        latency = end_time - start_time

        # Simulate token usage
        input_tokens = len(prompt.split()) + 10 # Example: base 10 tokens + words
        output_tokens = 50 + len(prompt.split()) // 2 # Example: base 50 tokens + half of input words

        response_text = f"This is a simulated response to: '{prompt}'. "
        response_action = None
        status_code = 200

        # Simulate agent action suggestion
        if "weather" in prompt.lower() or "forecast" in prompt.lower():
            response_text += "I recommend using a weather tool."
            response_action = {"tool_name": "weather_tool", "city": "New York"}
        elif "time" in prompt.lower():
            response_text += "I recommend using a time tool."
            response_action = {"tool_name": "time_tool"}
        elif "error" in prompt.lower():
            response_text += "Simulating an LLM error."
            status_code = 500
        else:
            response_text += "No tool suggested."

        span.set_attribute("llm.response.model", model_name)
        span.set_attribute("llm.response.tokens.total", input_tokens + output_tokens)
        span.set_attribute("llm.response.tokens.prompt", input_tokens)
        span.set_attribute("llm.response.tokens.completion", output_tokens)
        span.set_attribute("llm.response.content", response_text)
        span.set_attribute("http.status_code", status_code)

        LLM_CALL_LATENCY_SECONDS.labels(model_name=model_name, status_code=status_code).observe(latency)
        LLM_INPUT_TOKENS_TOTAL.labels(model_name=model_name).inc(input_tokens)
        LLM_OUTPUT_TOKENS_TOTAL.labels(model_name=model_name).inc(output_tokens)

        logger.info(json.dumps({
            "event": "llm_call_completed",
            "message": "LLM call finished",
            "model": model_name,
            "prompt": prompt,
            "response_summary": response_text,
            "latency_sec": f"{latency:.2f}",
            "input_tokens": input_tokens,
            "output_tokens": output_tokens,
            "status_code": status_code
        }))

        return {"text": response_text, "action": response_action, "status_code": status_code}

def mock_weather_tool(city: str):
    """Simulates an external weather API call."""
    with tracer.start_as_current_span("tool_call_weather") as span:
        span.set_attribute("tool.name", "weather_tool")
        span.set_attribute("tool.parameters.city", city)

        start_time = time.perf_counter()
        time.sleep(0.3)
        end_time = time.perf_counter()
        latency = end_time - start_time

        result = f"The weather in {city} is sunny with 25°C."
        status = "success"
        span.set_attribute("tool.status", status)
        span.set_attribute("tool.result", result)

        AGENT_TOOL_CALLS_TOTAL.labels(tool_name="weather_tool", status=status).inc()
        logger.info(json.dumps({
            "event": "tool_call",
            "message": "Weather tool call",
            "tool_name": "weather_tool",
            "city": city,
            "status": status,
            "latency_sec": f"{latency:.2f}"
        }))
        return result

def mock_time_tool():
    """Simulates an external time API call."""
    with tracer.start_as_current_span("tool_call_time") as span:
        span.set_attribute("tool.name", "time_tool")

        start_time = time.perf_counter()
        time.sleep(0.1)
        end_time = time.perf_counter()
        latency = end_time - start_time

        result = f"The current time is {time.strftime('%H:%M:%S')}."
        status = "success"
        span.set_attribute("tool.status", status)
        span.set_attribute("tool.result", result)

        AGENT_TOOL_CALLS_TOTAL.labels(tool_name="time_tool", status=status).inc()
        logger.info(json.dumps({
            "event": "tool_call",
            "message": "Time tool call",
            "tool_name": "time_tool",
            "status": status,
            "latency_sec": f"{latency:.2f}"
        }))
        return result

def llm_agent_handler(prompt: str, request_headers: dict):
    """The core logic of our simple LLM agent."""
    # Extract trace context from incoming request headers for distributed tracing
    ctx = extract(request_headers)
    with tracer.start_as_current_span("llm_agent_request", context=ctx) as span:
        span.set_attribute("user.query", prompt)

        agent_start_time = time.perf_counter()

        logger.info(json.dumps({"event": "agent_started", "message": "Agent request initiated", "query": prompt}))

        llm_response_data = mock_llm_call(prompt, "gpt-3.5-turbo-mock")
        final_answer = llm_response_data["text"]

        # Step 2: Agent decision making based on LLM response
        if llm_response_data["action"]:
            action = llm_response_data["action"]
            tool_name = action["tool_name"]
            logger.info(json.dumps({"event": "agent_decision", "message": "Agent decided to call tool", "decision": "call_tool", "tool_name": tool_name}))
            with tracer.start_as_current_span("agent_decision_making") as decision_span:
                decision_span.set_attribute("decision.type", "tool_call")
                decision_span.set_attribute("decision.tool_name", tool_name)

                if tool_name == "weather_tool":
                    city = action.get("city", "New York")
                    tool_result = mock_weather_tool(city)
                    final_answer = f"{final_answer}\nTool result: {tool_result}"
                elif tool_name == "time_tool":
                    tool_result = mock_time_tool()
                    final_answer = f"{final_answer}\nTool result: {tool_result}"
                else:
                    logger.warning(json.dumps({"event": "unknown_tool", "message": "Agent attempted to call unknown tool", "tool_name": tool_name}))
                    AGENT_TOOL_CALLS_TOTAL.labels(tool_name=tool_name, status='failure').inc()

        agent_end_time = time.perf_counter()
        total_latency = agent_end_time - agent_start_time
        AGENT_REQUEST_LATENCY_SECONDS.observe(total_latency)

        span.set_attribute("agent.total_latency_seconds", total_latency)
        span.set_attribute("agent.final_response", final_answer)
        logger.info(json.dumps({"event": "agent_finished", "message": "Agent request completed", "final_response_summary": final_answer, "total_latency_sec": f"{total_latency:.2f}"}))
        return final_answer

# --- Flask App ---
app = Flask(__name__)
# Wrap Flask app with OpenTelemetry WSGI middleware for automatic request tracing
app.wsgi_app = WsgiMiddleware(app.wsgi_app)

@app.route('/healthz', methods=['GET'])
def healthz():
    return "OK", 200

@app.route('/metrics', methods=['GET'])
def metrics():
    """Expose Prometheus metrics."""
    return generate_latest(), 200

@app.route('/query', methods=['POST'])
def query_agent():
    """Handle LLM agent queries."""
    data = request.json
    prompt = data.get('prompt')
    if not prompt:
        return jsonify({"error": "Prompt is required"}), 400

    # Pass request headers for context propagation
    response = llm_agent_handler(prompt, request.headers)
    return jsonify({"response": response})

if __name__ == '__main__':
    # PrometheusMetricReader needs to be set up globally for default registry
    reader = PrometheusMetricReader()
    meter_provider = MeterProvider(metric_readers=[reader], resource=resource)
    set_meter_provider(meter_provider)

    app.run(host='0.0.0.0', port=5000)

Now, create a Dockerfile for our application:

# Dockerfile
FROM python:3.10-slim-buster

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app.py .

EXPOSE 5000

CMD ["python", "app.py"]

And requirements.txt:

Flask==2.3.3
prometheus_client==0.18.0
opentelemetry-api==1.25.0
opentelemetry-sdk==1.25.0
opentelemetry-exporter-otlp-proto-grpc==1.25.0
opentelemetry-instrumentation-requests==0.45b0
opentelemetry-instrumentation-wsgi==0.45b0
opel-propagate-b3==1.0.0

Build and push your Docker image. Remember to replace your-dockerhub-user with your actual Docker Hub username.

docker build -t your-dockerhub-user/llm-agent-app:v1.0.0 .
docker push your-dockerhub-user/llm-agent-app:v1.0.0

Step 2: Deploy Observability Stack to Kubernetes

We'll use Helm to deploy the core components for robust LLM observability. Create an observability namespace first.

kubectl create namespace observability

2.1 Deploy Prometheus Operator and Grafana

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# Install kube-prometheus-stack which includes Prometheus, Grafana, and Alertmanager
helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace observability \
  --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesLabels=false \
  --set prometheus.prometheusSpec.podMonitorSelectorNilUsesLabels=false \
  --set grafana.enabled=true \
  --set grafana.adminPassword="prom-operator" \  # pragma: allowlist secret
  --set grafana.service.type="LoadBalancer" # Use "NodePort" for Minikube or local clusters

Wait for Prometheus and Grafana pods to be ready. You can get Grafana's LoadBalancer IP (or NodePort):

kubectl get svc -n observability grafana

2.2 Deploy Loki and Fluent Bit for Logging

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

# Install Loki
helm install loki grafana/loki \
  --namespace observability \
  --set service.type="ClusterIP" \
  --set persistence.enabled=true \
  --set persistence.size=10Gi # Adjust size as needed

# Install Fluent Bit
helm install fluent-bit grafana/fluent-bit \
  --namespace observability \
  --set config.service.flush=1 \
  --set config.inputs[0].name=tail \
  --set config.inputs[0].path="/var/log/containers/*.log" \
  --set config.inputs[0].multiline.parser=docker,cri \
  --set config.inputs[0].db=/var/log/flb_kube.db \
  --set config.inputs[0].mem_buf_limit=5MB \
  --set config.outputs[0].name=loki \
  --set config.outputs[0].host="loki.observability.svc.cluster.local" \
  --set config.outputs[0].port=3100 \
  --set config.outputs[0].labels="job=fluent-bit" \
  --set config.outputs[0].removeKeys="kubernetes.host,kubernetes.labels,kubernetes.annotations,kubernetes.pod_id,kubernetes.container_id,kubernetes.docker_id,kubernetes.container_hash,kubernetes.container_image,kubernetes.daemonset,kubernetes.deployment" \
  --set config.outputs[0].labelMapPath="/fluent-bit/config/label_map.json" \
  --set extraFiles.label_map_json="{\"kubernetes\": {\"container_name\": \"container\", \"namespace_name\": \"namespace\", \"pod_name\": \"pod\"}}" # Maps K8s metadata to Loki labels

2.3 Deploy OpenTelemetry Collector

Create otel-collector.yaml:

# otel-collector.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
  namespace: observability
  labels:
    app: otel-collector
spec:
  replicas: 1
  selector:
    matchLabels:
      app: otel-collector
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      containers:
      - name: otel-collector
        image: otel/opentelemetry-collector:0.100.0 # Use a specific, stable version
        command:
        - "/otelcol"
        - "--config=/conf/otel-collector-config.yaml"
        ports:
        - containerPort: 4317 # OTLP gRPC receiver
        - containerPort: 4318 # OTLP HTTP receiver
        volumeMounts:
        - name: otel-collector-config-vol
          mountPath: /conf
      volumes:
      - name: otel-collector-config-vol
        configMap:
          name: otel-collector-config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-config
  namespace: observability
data:
  otel-collector-config.yaml: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
    processors:
      batch: # Batching for efficiency
        send_batch_size: 100
        timeout: 10s
    exporters:
      logging: # For demo purposes, logs traces to stdout. Replace with Jaeger/Tempo for a full setup.
        verbosity: detailed
      # Example for Jaeger/Tempo exporters:
      # jaeger:
      #   endpoint: "jaeger-collector.observability:14250" # Assuming Jaeger is deployed
      #   tls:
      #     insecure: true
      # tempo:
      #   endpoint: "tempo.observability:4317" # Assuming Tempo is deployed
      #   tls:
      #     insecure: true
    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [batch]
          exporters: [logging] # Change to [jaeger] or [tempo] if you have them configured

kubectl apply -f otel-collector.yaml -n observability

Step 3: Deploy the LLM Agent Application to Kubernetes

Now, deploy your instrumented application. Create llm-agent-app.yaml. Remember to replace your-dockerhub-user with your actual Docker Hub username.

# llm-agent-app.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-agent-app
  namespace: default # Deploy in default or your app namespace
  labels:
    app: llm-agent-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: llm-agent-app
  template:
    metadata:
      labels:
        app: llm-agent-app
      annotations:
        prometheus.io/scrape: "true" # Enable Prometheus scraping
        prometheus.io/port: "5000"   # Port where metrics are exposed by the application
        prometheus.io/path: "/metrics"
    spec:
      containers:
      - name: llm-agent-app
        image: your-dockerhub-user/llm-agent-app:v1.0.0 # Replace with your image
        ports:
        - containerPort: 5000
          name: http-app # This port will serve both the application and /metrics
        env:
        - name: KUBERNETES_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        resources:
          requests:
            cpu: "200m"
            memory: "512Mi"
          limits:
            cpu: "1000m"
            memory: "1Gi"
---
apiVersion: v1
kind: Service
metadata:
  name: llm-agent-app
  namespace: default
  labels:
    app: llm-agent-app
spec:
  selector:
    app: llm-agent-app
  ports:
  - protocol: TCP
    port: 80 # Service exposed port
    targetPort: http-app # Maps to containerPort: 5000
    name: http # Name of this service port
  type: LoadBalancer # Use "NodePort" for local clusters like Minikube/k3s
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: llm-agent-app-sm
  namespace: observability # ServiceMonitor should be in the same namespace as Prometheus
  labels:
    release: prometheus # This label links to the Prometheus instance from kube-prometheus-stack
spec:
  selector:
    matchLabels:
      app: llm-agent-app # Selects services with this label
  endpoints:
  - port: http # Name of the port in the Service that exposes the metrics endpoint
    path: /metrics
    interval: 15s
    scrapeTimeout: 10s
  namespaceSelector:
    matchNames:
    - default # Or the namespace where your app is deployed

kubectl apply -f llm-agent-app.yaml -n default

Wait for the llm-agent-app pod and service to be ready. Get its external IP:

kubectl get svc llm-agent-app -n default

Step 4: Interact with the Agent to Generate Data

Use curl or a simple script to send queries to your agent. This will generate logs, metrics, and traces, populating your LLM observability stack.

# For LoadBalancer, get the external IP
AGENT_IP=$(kubectl get svc llm-agent-app -n default -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

# For NodePort (e.g., Minikube), get Minikube IP and NodePort
# AGENT_IP=$(minikube ip)
# AGENT_PORT=$(kubectl get svc llm-agent-app -n default -o jsonpath='{.spec.ports[?(@.name=="http")].nodePort}')
# export AGENT_URL="http://$AGENT_IP:$AGENT_PORT"

# Use AGENT_IP for LoadBalancer, or AGENT_URL for NodePort
export TARGET_URL="http://$AGENT_IP" # or $AGENT_URL if using NodePort

echo "Sending requests to $TARGET_URL/query"

# Send some queries
curl -X POST -H "Content-Type: application/json" -d '{"prompt": "What is the weather like in London today?"}' $TARGET_URL/query
curl -X POST -H "Content-Type: application/json" -d '{"prompt": "Tell me a fun fact about Kubernetes."}' $TARGET_URL/query
curl -X POST -H "Content-Type: application/json" -d '{"prompt": "What time is it?"}' $TARGET_URL/query
curl -X POST -H "Content-Type: application/json" -d '{"prompt": "Simulate an error now."}' $TARGET_URL/query # Test error paths

# Send more queries to generate enough data for dashboards
for i in $(seq 1 10); do
  curl -X POST -H "Content-Type: application/json" -d '{"prompt": "Give me another interesting fact."}' $TARGET_URL/query &>/dev/null
  sleep 0.5
  curl -X POST -H "Content-Type: application/json" -d '{"prompt": "What is the current weather?"}' $TARGET_URL/query &>/dev/null
  sleep 0.5
done

echo "Requests sent. Data should now be flowing into your observability stack."

Repeat these calls a few times to generate sufficient data for visualization.

Step 5: Visualize LLM Observability Data in Grafana

Access Grafana using the LoadBalancer IP or NodePort you obtained earlier.

5.1 Prometheus Dashboard for LLM Metrics

Go to Connections -> Data sources and ensure Prometheus is configured (it should be automatically set up by kube-prometheus-stack).
Create a new Dashboard (+ -> New Dashboard).
Add new panels with the following PromQL queries to monitor your LLM observability on Kubernetes:
- LLM Call Latency (95th Percentile):
  - Query: histogram_quantile(0.95, sum by(le, model_name) (rate(llm_call_latency_seconds_bucket{app="llm-agent-app", namespace="default"}[5m])))
- LLM Token Usage (Input/Output Rate):
  - Query: sum by(model_name) (rate(llm_input_tokens_total{app="llm-agent-app", namespace="default"}[5m]))
  - Query: sum by(model_name) (rate(llm_output_tokens_total{app="llm-agent-app", namespace="default"}[5m]))
- Agent Request Latency (99th Percentile):
  - Query: histogram_quantile(0.99, sum by(le) (rate(agent_request_latency_seconds_bucket{app="llm-agent-app", namespace="default"}[5m]))) (99th percentile end-to-end agent latency)
- Agent Tool Call Success/Failure Rates:
  - Query: sum by(tool_name, status) (rate(agent_tool_calls_total{app="llm-agent-app", namespace="default"}[5m]))
- Estimated Cost (Example, adjust token costs as per your LLM provider):
  - Query: (sum(rate(llm_input_tokens_total{app="llm-agent-app", namespace="default"}[5m])) / 1000 * 0.01) + (sum(rate(llm_output_tokens_total{app="llm-agent-app", namespace="default"}[5m])) / 1000 * 0.03)
  - Explanation: This query calculates an estimated cost based on a hypothetical rate of $0.01 per 1000 input tokens and $0.03 per 1000 output tokens.

5.2 Loki Dashboard for LLM Logs

Go to Connections -> Data sources and add Loki as a new data source:
- Name: Loki
- URL: http://loki.observability.svc.cluster.local:3100
Add a new panel to your dashboard. Set the visualization type to Logs.
Select Loki as your data source.
Log Queries (examples):
- Show all logs from your app: {container="llm-agent-app", namespace="default"}
- Filter for LLM calls: {container="llm-agent-app", namespace="default"} | json | event="llm_call_completed"
- Filter for agent tool calls: {container="llm-agent-app", namespace="default"} | json | event="tool_call"
- Filter for specific trace ID: {container="llm-agent-app", namespace="default"} | json | trace_id="<your-trace-id>" (You can pick a trace_id from the Prometheus data or another log query).
- Filter for LLM errors: {container="llm-agent-app", namespace="default"} | json | status_code="500"
The | json pipe command is crucial as our application emits structured JSON logs, allowing you to filter and parse fields within the log lines. You can then use | line_format "{{.message}}" to display only the message, | level="ERROR" to filter by log level, or | prompt=~".*weather.*" to search within specific fields.

5.3 OpenTelemetry Traces (Optional: If Jaeger/Tempo is set up)

If you had deployed a full tracing backend like Jaeger (or Tempo), you would typically configure Jaeger as a data source in Grafana. Then you could navigate to the Traces section in Grafana, search by service name (llm-agent-app) and trace ID to visualize the full agent execution flow with all its spans, greatly enhancing your LLM observability.

For this tutorial, since we configured the OpenTelemetry Collector to log traces to stdout, you can inspect the collector's logs to see the trace data being received.

kubectl logs -f -n observability deployment/otel-collector

You'll see detailed output for each trace, confirming that your application is successfully sending trace data to the collector. For example:

{
  "resource": {
    "attributes": [
      {"key": "service.name", "value": {"stringValue": "llm-agent-app"}},
      {"key": "k8s.pod.name", "value": {"stringValue": "llm-agent-app-..."}},
      {"key": "k8s.namespace.name", "value": {"stringValue": "default"}}
    ]
  },
  "scopeSpans": [
    {
      "scope": {"name": "llm.agent.tracer", "version": "1.0.0"},
      "spans": [
        {
          "traceId": "...",
          "spanId": "...",
          "parentSpanId": "...",
          "name": "llm_agent_request",
          "kind": "SPAN_KIND_SERVER", # Due to WSGIMiddleware
          "startTimeUnixNano": "...",
          "endTimeUnixNano": "...",
          "attributes": [
            {"key": "user.query", "value": {"stringValue": "..."}},
            {"key": "agent.total_latency_seconds", "value": {"doubleValue": 1.23}},
            ...
          ],
          "events": [],
          "status": {"code": "STATUS_CODE_UNSET"}
        },
        {
          "traceId": "...",
          "spanId": "...",
          "parentSpanId": "...", # Parent will be llm_agent_request's spanId
          "name": "llm_api_call",
          "kind": "SPAN_KIND_INTERNAL",
          "attributes": [
            {"key": "model_name", "value": {"stringValue": "gpt-3.5-turbo-mock"}},
            {"key": "llm.response.tokens.total", "value": {"intValue": 120}},
            ...
          ],
          ...
        }
        # ... more spans for agent_decision_making, tool_call_weather etc.
      ]
    }
  ]
}

This confirms traces are being generated and processed by the collector. Integrating a full Jaeger or Tempo setup is a worthy next step, but is beyond the immediate scope of this tutorial.

Actionable Insights, Alerting, and Troubleshooting for LLM Observability

Collecting data is only half the battle. You need to interpret it to derive actionable insights and set up effective alerts for your LLM observability on Kubernetes.

Interpreting LLM Observability Data

Performance Bottlenecks:
- High LLM Call Latency (P95/P99): Check llm_call_latency_seconds_bucket. Is it the model itself, network, or rate limiting from the LLM provider? If your average LLM call latency is consistently above 5 seconds, it might indicate network issues or an overloaded LLM endpoint.
- High Agent Request Latency: Examine traces (llm_agent_request span) to pinpoint which step (LLM call, tool call, internal logic) is contributing most to the delay. Look for unusually long tool_call spans.
- Low Throughput: Is llm_agent_requests_total not meeting expectations? Check CPU/memory utilization of the pod. Is the LLM model slow or is the application itself bottlenecked? A drop of 20% in QPS without a corresponding reduction in load might indicate an issue.
Cost Overruns:
- Spikes in Token Usage: Monitor llm_input_tokens_total and llm_output_tokens_total. Are prompts getting unexpectedly long? Is the model generating excessively verbose responses? This could indicate a prompt engineering issue or model drift. An sudden increase of 50% in output tokens per request could drastically raise costs.
- Frequent LLM API Calls: llm_api_calls_total can highlight agents engaging in too many iterative LLM calls, indicating inefficiency.
Model Degradation & Agent Failures:
- Increased LLM Error Rates: Monitor sum by(status_code) (rate(llm_call_latency_seconds_count{status_code!="200"}[5m])). Correlate with logs to see actual error messages and prompts.
- Increased Agent Goal Failures: agent_goal_failures_total. What type of queries are failing? Dive into traces for these failed requests to see the agent's decision path.
- Decreased Tool Call Success Rates: agent_tool_call_failure_total. Is a specific external tool failing? This points to external dependency issues.
- Unexpected Agent Behavior: Use Loki to search for keywords in prompt/response logs or agent decision logs. "Why did the agent choose X tool here?" can often be answered by reviewing the log of its thought process.

Setting Up Effective Alerts for LLM Workloads

Alerts should be proactive and actionable. Use Grafana Alerting (integrated with Prometheus Alertmanager).

Critical Alerts (PagerDuty, Slack):
- High Error Rate: sum(rate(llm_call_latency_seconds_count{status_code!="200"}[5m])) > 5 (more than 5 LLM errors per 5 minutes for instance).
- Service Unavailability: up{app="llm-agent-app", namespace="default"} == 0 (agent service is down).
- Critical Latency Spike: histogram_quantile(0.99, sum by(le) (rate(agent_request_latency_seconds_bucket{app="llm-agent-app", namespace="default"}[5m]))) > 15 (99th percentile request latency exceeds 15 seconds).
Warning Alerts (Slack, Email):
- High Token Consumption Rate: sum(rate(llm_output_tokens_total{app="llm-agent-app", namespace="default"}[1h])) > 1000000 (e.g., more than 1 million output tokens in the last hour, indicating potential cost overrun).
- Increased Agent Tool Failures: sum(rate(agent_tool_calls_total{status="failure", app="llm-agent-app", namespace="default"}[5m])) > 1 (any tool failures in a 5-minute window could warrant investigation).
- High GPU Utilization: gpu_utilization_percentage{pod="llm-agent-app-..."} > 90 (indicates resource saturation, potentially leading to performance degradation on GPU-accelerated workloads).
Informational Alerts: Track trends without immediate action, e.g., daily cost reports.

Troubleshooting Workflow for LLM Observability

Alert Triggered: Receive an alert about high latency or errors.
Dashboard Check: Go to your Grafana dashboard. Look at the specific metric that triggered the alert. Correlate with other metrics (e.g., if latency is high, are CPU/memory also high? Are token counts spiking?).
Logs Investigation: If metrics point to an application-specific issue, jump to Loki. Use the trace ID from the metric context (if available, or infer from timestamps) to find relevant logs. Search for errors, warnings, or specific agent decision steps around the time of the incident. Review the full prompt/response pairs.
Traces Deep Dive: If the issue is complex and spans multiple steps or services (especially for agents), use your tracing backend (Jaeger/Tempo). Find the trace for a representative failing request. Visualize the spans to identify the exact step (LLM call, tool call, external API) that introduced latency or an error. Examine attributes attached to spans for detailed context like LLM model, parameters, or tool call arguments.
Identify Root Cause: Based on the correlated data, pinpoint the root cause: an inefficient prompt, a slow external tool, a bug in agent logic, or an overloaded Kubernetes node.
Remediate and Verify: Implement the fix, then monitor your observability stack to verify the issue is resolved and new alerts are not triggered.

Frequently Asked Questions About LLM Observability on Kubernetes

What is LLM observability and why is it important on Kubernetes?

LLM observability is the ability to understand the internal state, performance, cost, and behavior of Large Language Model (LLM) powered applications and AI agents. It's crucial on Kubernetes because traditional monitoring tools don't capture the unique non-deterministic nature, token usage, and complex decision-making processes inherent to LLMs and AI agents, leading to blind spots in performance and cost management.

How do I monitor LLM costs on Kubernetes?

To monitor LLM costs on Kubernetes, track metrics like llm_input_tokens_total, llm_output_tokens_total, and llm_api_calls_total. These application-level metrics, collected by Prometheus, can then be used in Grafana with PromQL queries to calculate real-time estimated costs based on your LLM provider's pricing for tokens or API calls.

Can OpenTelemetry be used for LLM metrics and traces?

Yes, OpenTelemetry is the recommended standard for instrumenting LLM applications to generate both metrics and traces. It provides APIs and SDKs to create detailed spans for LLM calls and agent steps, and to emit custom metrics like token usage and latency, all of which can be collected and exported by the OpenTelemetry Collector.

Why do traditional monitoring tools fail for AI agents?

Traditional monitoring tools primarily focus on infrastructure metrics (CPU, memory) and simple request/response codes. They fail for AI agents because agents have non-deterministic behavior, complex multi-step reasoning processes, rely on token usage for cost, and can hallucinate or go off-topic even with successful HTTP responses. Understanding these nuances requires deep application-level metrics, structured logs, and distributed traces.

What key metrics should I track for LLM performance on Kubernetes?

Key metrics for LLM performance on Kubernetes include:

Prompt Processing Latency
Response Generation Latency
Total Agent Request Latency
Throughput (Queries Per Second)
Tool Call Latency (for AI agents)
Input and Output Token Counts
LLM API Call Counts These provide a comprehensive view of speed, efficiency, and resource consumption.

Conclusion

Building robust LLM observability on Kubernetes is not just about extending your existing monitoring stack; it requires a paradigm shift. You need to look beyond infrastructure metrics and delve deep into the nuances of LLM behavior, token economics, and agent decision-making.

By leveraging OpenTelemetry for rich instrumentation, Prometheus and Grafana for comprehensive metrics, and Loki with Fluent Bit for structured logging, you can construct a powerful, integrated observability pipeline. This pipeline gives you the visibility needed to understand performance, control costs, and proactively troubleshoot the complex, often non-deterministic world of generative AI on Kubernetes.

Your next steps should be:

Refine your application instrumentation: Integrate your actual LLM calls and agent logic with OpenTelemetry and Prometheus client metrics.
Experiment with diverse prompts: Send various types of queries to your agent, including edge cases and error-inducing prompts, to see how your observability stack reacts.
Build custom Grafana dashboards: Create dashboards tailored to your specific LLM applications, focusing on the most critical performance, cost, and quality metrics for LLM observability on Kubernetes.
Implement robust alerting: Define clear, actionable alerts based on thresholds that matter for your application's reliability and cost efficiency.
Explore tracing backends: Consider deploying Jaeger or Tempo to gain full end-to-end distributed tracing capabilities, which are invaluable for complex AI agent debugging.

The journey to effective LLM observability is iterative. Continuously refine your instrumentation, dashboards, and alerts as your AI agents evolve and your understanding of their behavior deepens.