DEV Community

AWS Community Builders profile image piyushsachdeva
piyushsachdeva for AWS Community Builders

Posted on • Edited on

Build an AI Observability Pipeline on EKS Using Elasticsearch

It's 2am and your pager is going off. You're on call, you were asleep, and now there's a pod stuck in CrashLoopBackOff. You know the drill: tail the logs, kubectl describe, scroll Git history, ask in Slack whether anyone shipped anything in the last hour.

Forty minutes later you've found it, and you've woken up your tech lead to confirm.

This walkthrough is about not doing that anymore.

We'll build an AI observability pipeline on Amazon EKS with Elastic Cloud and OpenTelemetry, end to end. The whole thing rests on one move: join two streams of data, your crash logs and your deployment history, and let a language model reason across the join.

Then you ask a plain question, "Why is paymentservice crashing?", and instead of another dashboard to squint at, you get the root cause:

paymentservice went OOMKilled three minutes after commit a1b2c3 by xyz, which dropped the memory limit from 128Mi to 24Mi.

The commit SHA, the author, what changed, what to revert. Two indices and one agent is all it takes. Over the next sections you'll wire OpenTelemetry, Elasticsearch, and ArgoCD into your EKS cluster until that 2am question answers itself.

Table of contents

Architecture

Architecture diagram of AI observability setup on EKS using Elasticsearch

Two data flows meet inside Elasticsearch.

On the infrastructure side, you push code, Argo CD syncs it to EKS, a pod crashes, and OpenTelemetry ships the crash event to the logs-* index.

On the metadata side, the same push fires a GitHub Action that POSTs the commit details (SHA, author, diff URL, and message) to the github-deploymentsindex.

The Elastic AI agent owns both queries. Ask it a question, and it runs one ES|QL query against the logs to find the failure and a second against the deployments to find the most recent push and stitches the two into a plain-English answer.

A few things to keep in mind before you start.

  • Push your Kubernetes manifests to GitHub before you create the Argo CD app. Argo CD syncs whatever is in the repo on first sync, so an empty repo sends you down a long debugging detour.
  • Setup Elastic Cloud before you install OpenTelemetry. The OTel chart wants the endpoint and API key at install time.
  • Create the github-deploymentsindex before you push to a repo that already has the workflow file, because the first push tries to write straight away.
  • Both indices need real data before you switch on the agent. It cannot reason over nothing.

Prerequisites

You'll need these tools installed locally:

  • AWS CLI (aws)
  • Kubernetes CLI (kubectl)
  • eksctl (for EKS cluster management)
  • Helm
  • Argo CD CLI
  • GitHub CLI (gh) and Git

Connect kubectl to your EKS cluster:

aws eks update-kubeconfig \
  --region us-east-1 \
  --name <CLUSTER_NAME>
Enter fullscreen mode Exit fullscreen mode

Verify nodes are healthy:

kubectl get nodes
# Expect: 3 nodes, Status: Ready
Enter fullscreen mode Exit fullscreen mode

One sizing note that will save you a Slack message later: use m5.xlarge instances, not t3.medium. You're running microservices, OTel DaemonSets, and Argo CD on the same cluster. Two vCPUs per node won't cut it. Three m5.xlargenodes is the sweet spot.

If you don't have an EKS cluster yet, spin one up quickly:

eksctl create cluster \
  --name demo-cluster \
  --region us-east-1 \
  --nodegroup-name standard-workers \
  --node-type m5.xlarge \
  --nodes 3 \
  --nodes-min 3 \
  --nodes-max 5 \
  --managed
Enter fullscreen mode Exit fullscreen mode

Step 1: Provision Elasticsearch from AWS Marketplace

You could self-host Elasticsearch; however, the Elastic Cloud takes care of managing the service for you, which is another way to get peace of mind. The Marketplace deployment lands on the same AWS invoice and handles upgrades for you. (A free trial is enough for the demo.)

  1. Open the AWS Console and navigate to AWS Marketplace.

  2. Search for Elastic Cloud (Elasticsearch Service).
    Elastic Cloud (Elasticsearch Service) on AWS Marketplace

  3. Subscribe, then click Set up your account. That kicks you over to the Elastic Cloud console.

  4. The setup wizard will ask a few basic questions, along with "Which use case are you looking to try first?". Choose Elastic for Observability. This template pre-enables the bits you need: log aggregation, APM, and AIOps. The plain Elasticsearch template means turning those on later by hand.
    Elastic for observability

  5. The next question it will ask is "Which Deployment option would you like to try?" Click Create Cloud Serverless.
    Elastic Cloud Serverless Deployment

  6. Then it will ask for the configuration options; select "Observability complete."
    Observability complete

  7. Lastly, it will ask the preferred region for cloud data storage, you can keep the default as cloud provider: Amazon Web Services and region to us-east-1.

Provisioning takes about five minutes. When the credentials screen appears, copy the elastic user password.

Get your endpoints

From the Elastic Cloud console, open your deployment and grab these two:

  1. Elasticsearch → Copy endpoint — used with GitHub Actions secret ES_ENDPOINT and curl commands.
  2. Integrations → Manage → APM → copy OTLP endpoint — used as the OTel kube-stack secret elastic_otlp_endpoint.

Step 1b: Generate a Kibana API key

You'll need one API key in two places (the OTel install and your GitHub repo secrets), so generate it now.

Navigate to:

https://<YOUR-KIBANA-URL>/app/management/security/api_keys
Enter fullscreen mode Exit fullscreen mode

Or via the menu: dashboard -> settings -> Access -> Api Keys

Create Kibana API keys

Click Create API key, name it something obvious like github-deploy-key, and click Create.

Kibana API key creation screen

Copy the key. Same rule as the password: shown once. You'll reuse it as:

  • elastic_api_key in the OTel secret (Step 2)
  • ES_API_KEY in your GitHub repo secrets (Step 3) Verify the key works: 
curl -s -w "\nHTTP:%{http_code}" \
  -X GET "<YOUR-ES-ENDPOINT>" \
  -H "Authorization: ApiKey <YOUR-API-KEY>"
# Expect: HTTP:200
Enter fullscreen mode Exit fullscreen mode

If you don't get a 200, fix that now. A bad key here causes OTel to silently drop logs later, and you'll spend an hour wondering why Discover is empty.

Step 2: Configure OpenTelemetry to ship Kubernetes logs

OpenTelemetry is the bridge between your cluster and Elasticsearch. Once it's installed as a DaemonSet, every pod log in the cluster ships automatically. You don't have to instrument individual services for this tutorial.

  1. In Elastic, open your deployment and click Add Observability Data.
  2. Choose Kubernetes.
  3. Under Monitor your Kubernetes cluster using, pick OpenTelemetry: Full Observability. Elastic pre-fills the Helm commands with your endpoint and credentials.

Add the Helm repo:

helm repo add open-telemetry \
  https://open-telemetry.github.io/opentelemetry-helm-charts --force-update
Enter fullscreen mode Exit fullscreen mode

Create the namespace, secret, and chart:

kubectl create namespace opentelemetry-operator-system

kubectl create secret generic elastic-secret-otel \
  --namespace opentelemetry-operator-system \
  --from-literal=elastic_otlp_endpoint='<YOUR-OTLP-INGEST-ENDPOINT>' \
  --from-literal=elastic_api_key='<YOUR-API-KEY>'

helm upgrade --install opentelemetry-kube-stack \
  open-telemetry/opentelemetry-kube-stack \
  --namespace opentelemetry-operator-system \
  --values 'https://raw.githubusercontent.com/elastic/elastic-agent/refs/tags/v9.3.3/deploy/helm/edot-collector/kube-stack/managed_otlp/values.yaml' \
  --version '0.12.4'
Enter fullscreen mode Exit fullscreen mode

If Helm complains about a conflict, run helm uninstall opentelemetry-kube-stack -n opentelemetry-operator-system and reinstall. This happens to everyone the first time.

Verify all pods came up:

kubectl get all -n opentelemetry-operator-system
Enter fullscreen mode Exit fullscreen mode

All pods running in the opentelemetry-operator-system namespace

Step 2b: Verify logs are flowing in Kibana

Before you wire up anything else, confirm OTel is actually shipping logs. Open Kibana and go to Discover. The Kibana URL is on the Elastic Cloud deployment overview page, listed next to the Elasticsearch endpoint.

In Discover, pick the All logs data view, set the time range to Last 15 minutes, and look for documents arriving from your EKS pods. You should see system components, OTel collectors, and any applications running on the cluster.

Kibana Discover showing logs flowing from cluster pods

If you see documents, you're good. If not, double-check the API key and the OTLP endpoint in the secret.

EKS-specific note: If you're using a private EKS cluster (no public endpoint), make sure the OTel collector pods can reach the Elastic endpoint. You may need to add a NAT Gateway or VPC endpoint for outbound HTTPS traffic on port 443.

Step 3: Index Git commit metadata via GitHub Actions

The agent answers "Who deployed this?" by querying a second index, and that index needs data in it. A GitHub Action handles that automatically on every push. Nothing runs inside your cluster for this part; Actions runs in GitHub's cloud and POSTs straight to Elasticsearch.

3a. Create the index 

curl -X PUT "<YOUR-ES-ENDPOINT>/github-deployments" \
  -H "Authorization: ApiKey <YOUR-API-KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "mappings": {
      "properties": {
        "timestamp":   { "type": "date" },
        "commit_sha":  { "type": "keyword" },
        "author":      { "type": "keyword" },
        "service":     { "type": "keyword" },
        "image_tag":   { "type": "keyword" },
        "change":      { "type": "text" },
        "diff_url":    { "type": "keyword" }
      }
    }
  }'
# Expect: {"acknowledged":true}
Enter fullscreen mode Exit fullscreen mode

3b. Add GitHub repository secrets 

gh secret set ES_ENDPOINT \
  --repo <YOUR-GITHUB-USERNAME>/<YOUR-REPO> \
  --body "<YOUR-ES-ENDPOINT>"

gh secret set ES_API_KEY \
  --repo <YOUR-GITHUB-USERNAME>/<YOUR-REPO> \
  --body "<YOUR-API-KEY>"
Enter fullscreen mode Exit fullscreen mode

Step 3c: The workflow file

Create .github/workflows/index-deploy.yml. On every push to main, it figures out which service changed (by diffing release/kubernetes-manifests.yaml) and POSTs one document to the github-deployments index.

name: Index deployment to Elasticsearch
on:
  push:
    branches: [main]

jobs:
  index:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 2

      - name: Detect changed service
        id: changed
        run: |
          CHANGED=$(git diff HEAD~1 HEAD -- release/kubernetes-manifests.yaml \
            | grep '^+' \
            | grep -oP 'name:\s+\K[a-z][a-z0-9-]+service' \
            | head -1)
          if [ -z "$CHANGED" ]; then
            CHANGED=$(git diff HEAD~1 HEAD -- release/kubernetes-manifests.yaml \
              | grep -oE 'paymentservice|cartservice|frontend|recommendationservice|adservice|checkoutservice|productcatalogservice|currencyservice|shippingservice|emailservice|redis-cart|loadgenerator' \
              | head -1)
          fi
          echo "service=${CHANGED:-unknown}" >> $GITHUB_OUTPUT

      - name: Push deploy event to Elasticsearch
        run: |
          curl -X POST "${{ secrets.ES_ENDPOINT }}/github-deployments/_doc" \
            -H "Authorization: ApiKey ${{ secrets.ES_API_KEY }}" \
            -H "Content-Type: application/json" \
            -d "{
              \"timestamp\":  \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",
              \"commit_sha\": \"${{ github.sha }}\",
              \"author\":     \"${{ github.actor }}\",
              \"service\":    \"${{ steps.changed.outputs.service }}\",
              \"image_tag\":  \"${{ github.sha }}\",
              \"change\":     \"${{ github.event.head_commit.message }}\",
              \"diff_url\":   \"${{ github.event.compare }}\"
            }"
Enter fullscreen mode Exit fullscreen mode

Step 3d: Verify it works 

git commit --allow-empty -m "test: verify ES indexing"
git push origin main

gh run list --limit 3
# Expect: "Index deployment to Elasticsearch" → completed success

curl -s "<YOUR-ES-ENDPOINT>/github-deployments/_count" \
  -H "Authorization: ApiKey <YOUR-API-KEY>"
# Expect: {"count":1,...}
Enter fullscreen mode Exit fullscreen mode

Step 4: Set up Argo CD for GitOps deployment

Argo CD watches your repo and syncs changes to EKS on a schedule. The whole point of the demo is to push code and have it deploy on its own, and Argo CD is what makes that real.

# Install Argo CD
kubectl create namespace argocd
kubectl apply -n argocd \
  -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
kubectl rollout status deployment/argocd-server -n argocd --timeout=180s

# Expose publicly via LoadBalancer (creates an AWS ELB)
kubectl patch svc argocd-server -n argocd \
  -p '{"spec":{"type":"LoadBalancer"}}'
kubectl get svc argocd-server -n argocd
# Note the EXTERNAL-IP (AWS ELB hostname)

# Get admin password
kubectl get secret argocd-initial-admin-secret -n argocd \
  -o jsonpath='{.data.password}' | base64 -d && echo ""

# Log in
argocd login <EXTERNAL-IP> --username admin --password <PASSWORD> --insecure
Enter fullscreen mode Exit fullscreen mode

Note: On EKS, the EXTERNAL-IP for the LoadBalancer will be an AWS ELB DNS hostname (e.g., xxxxxx.us-east-1.elb.amazonaws.com), not a plain IP. Use that hostname wherever <EXTERNAL-IP> appears.

Feel free to use this sample repo as your application.

Create the Argo CD application pointing at your repo:

kubectl apply -f - <<EOF
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: online-boutique
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/<YOUR-USERNAME>/microservices-demo
    targetRevision: main
    path: release
  destination:
    server: https://kubernetes.default.svc
    namespace: online-boutique
  syncPolicy:
    automated:
      prune: true
    syncOptions:
      - CreateNamespace=true
EOF
Enter fullscreen mode Exit fullscreen mode

Trigger the first sync. The default Argo CD poll interval is three minutes, which is fine in production and brutal in a demo. Drop it to thirty seconds so you can actually watch things happen.

argocd app sync online-boutique --force --prune

kubectl patch configmap argocd-cm -n argocd --type merge \
  -p '{"data": {"timeout.reconciliation": "30s"}}'
kubectl rollout restart deployment/argocd-repo-server -n argocd

# Verify all 12 pods are running
kubectl get pods -n online-boutique
Enter fullscreen mode Exit fullscreen mode

Step 5: Build the AI diagnostic agent in Kibana

Both indices are live. Now build the agent that queries them.

Step 5a: LLM setup

The agent needs a language model behind it. Use Elastic Inference Service (EIS).

EIS ships with Elastic for Observability deployments and Marketplace trials, so the model dropdown is already populated when you create an agent. You'll see Claude for reasoning, JINA embeddings for vectorization, and the JINA reranker for retrieval ordering, all billed through the same Elastic invoice. Pick Claude (or whichever reasoning model you prefer) for the agent itself.

You'll want the JINA models later, if you extend this pattern to semantic search over logs or RAG over runbooks. For the diagnostic agent here, the reasoning model is all you need.

If EIS isn't available on your deployment, fall back to an external connector:

Kibana → ☰ → Stack Management → Connectors → Create connector → OpenAI

Paste an API key from platform.openai.com, set the model to gpt-4o, click Save & test, then pick the connector in Agent Builder.

Anthropic, Google Vertex, and AWS Bedrock connectors follow the same shape. Pick whichever you already have billing set up for.

Step 5b: Create the ES|QL tools

Kibana → ☰ → Search → Tools → Create tool.

Tool 1: get_crash_logs

  • Type: Elasticsearch query
  • Index: logs-* 
FROM logs-*
| WHERE resource.attributes.k8s.namespace.name == "online-boutique"
| WHERE body.text LIKE "*OOMKill*" OR body.text LIKE "*memory*"
| SORT @timestamp DESC
| LIMIT 20
| KEEP @timestamp, resource.attributes.k8s.deployment.name, resource.attributes.k8s.pod.name, body.text
Enter fullscreen mode Exit fullscreen mode

Elastic Agent Builder create tool wizard

Tool 2: get_deploy_history

  • Type: Elasticsearch query
  • Index: github-deployments 
FROM github-deployments
| SORT timestamp DESC
| LIMIT 5
| KEEP timestamp, author, commit_sha, service, change, diff_url
Enter fullscreen mode Exit fullscreen mode

Step 5c: Create the agent

Kibana → ☰ → Search → Agent Builder → Create agent.

  • Name: blame-the-deploy
  • Model: Claude, or your configured connector

Paste these instructions:

You are an SRE assistant. When asked why a service is crashing:
1. Use get_crash_logs to find OOMKill or memory events in logs-*
2. Use get_deploy_history to find the most recent GitHub deployment to that service
3. If the deploy happened shortly before the crash, that is the likely cause
4. Report: crash time, commit SHA, author, what changed, and how to fix it
Always cite the commit SHA and author name in your answer.
Enter fullscreen mode Exit fullscreen mode

Under Tools, add get_crash_logs and get_deploy_history and click save.

Step 6: Simulate and diagnose a crash

The whole reason you built this is to ask the agent a hard question. So break something on purpose.

Introduce the bad commit

Open release/kubernetes-manifests.yaml. Find the paymentservice deployment (around line 631) and squeeze the memory limits below what the service actually needs at startup:

resources:
  requests:
    memory: "24Mi"   # was 64Mi
  limits:
    memory: "24Mi"   # was 128Mi
Enter fullscreen mode Exit fullscreen mode 
git add release/kubernetes-manifests.yaml
git commit -m "perf: tune paymentservice memory limits for cost optimisation"
git push origin main
Enter fullscreen mode Exit fullscreen mode

That commit message is the kind of thing a tired engineer actually writes at 4pm on a Friday. Argo CD picks it up within thirty seconds and applies the manifest. Because 24Mi isn't enough, the pod dies on startup.

kubectl get pods -n online-boutique -w
# paymentservice: Running → OOMKilled → CrashLoopBackOff
Enter fullscreen mode Exit fullscreen mode

Confirm which pod is unhappy:

kubectl describe pod -n online-boutique <pod-name> | grep -A5 'Last State\|OOM'
Enter fullscreen mode Exit fullscreen mode

Ask the agent

Open the Elastic Agent chat:

Why is paymentservice crashing? Check the logs and recent deployments.

Elastic Agent chat interface

The agent runs get_crash_logs, sees the OOMKill, runs get_deploy_history, finds the commit that lowered the memory limit, and reports back with the SHA, the author, the commit message, the diff URL, and a fix suggestion.

Elastic Agent response showing root cause analysis

Fix it 

git revert HEAD --no-edit
git push origin main
# Pod recovers in about 30 seconds
Enter fullscreen mode Exit fullscreen mode

Ask the agent again:

Is paymentservice healthy now?

It checks the logs, finds no recent OOMKills, and confirms the service is stable.

Why this works

It comes down to the join. Kubernetes logs tell you what crashed and when. The github-deployments index tells you what was deployed and by whom. Either index alone leaves you guessing. Stitch them together behind an agent and you skip the part of the incident where you're bouncing between CloudWatch, GitHub, Slack, and kubectl.

The pattern extends. Want to diagnose CPU throttling? Add a tool that searches logs-* for CPUThrottlingHigh events. Want to catch failed health checks? Same idea with a different ES|QL clause. The tools are just saved queries, and the agent picks up a new diagnostic capability the next time you chat with it.

One caveat before you put this in front of the rest of your team. The agent is good at correlation, not causation. If two services deploy in the same five-minute window and one crashes, the agent will cheerfully blame the wrong commit. Treat the output as a strong lead, not a verdict.

Quick reference

Commands you will reuse 

kubectl get pods -n online-boutique        # check all pods
kubectl get pods -n online-boutique -w     # watch live
kubectl get pods -n opentelemetry-operator-system  # check OTel
argocd app get online-boutique             # Argo CD sync status
argocd app sync online-boutique --force    # force sync
Enter fullscreen mode Exit fullscreen mode

ES|QL: find OOMKill events 

FROM logs-*
| WHERE resource.attributes.k8s.namespace.name == "online-boutique"
| WHERE body.text LIKE "*OOMKill*"
| SORT @timestamp DESC
| LIMIT 20
| KEEP @timestamp, resource.attributes.k8s.deployment.name, body.text
Enter fullscreen mode Exit fullscreen mode

ES|QL: find recent deploys 

FROM github-deployments
| SORT timestamp DESC
| LIMIT 10
| KEEP timestamp, author, commit_sha, service, change
Enter fullscreen mode Exit fullscreen mode

FAQs

What is an AI observability pipeline?

An AI observability pipeline is a system that ships infrastructure telemetry and deployment metadata into the same data store, then puts an LLM in front of it so you can ask plain-English questions about failures. The model runs structured queries against your indices instead of asking you to remember which dashboard goes with which symptom.

Can I run this without AWS?

Yes. The architecture is portable. Swap EKS for GKE, AKS, or any conformant Kubernetes cluster. Elasticsearch has hosted offerings on GCP and Azure Marketplace, and OpenTelemetry doesn't care which cloud it runs on. The only AWS-specific steps are the Marketplace billing flow and the aws eks update-kubeconfig command, and both have direct equivalents on the other clouds.

Why use Elasticsearch for this use case?

Elasticsearch lets you keep your logs, metrics, deployment events, and vector embeddings in one place. The Kibana AI agent is built in. You define a tool, point it at an index, and chat with it. No LangChain wrapper, no FastAPI orchestrator, no model gateway to keep alive. And ES|QL is one query language across logs, metrics, and structured JSON metadata, which is the only reason the join in this tutorial (logs-* against github-deployments) is five lines instead of a Python service.

How much does this cost to run?

A small Elastic Cloud deployment on us-east-1 runs about $95 per month at the time of writing. EKS costs depend on node count; three m5.xlarge nodes are roughly $210 per month before Savings Plans or Reserved Instance discounts. The EKS control plane itself is $0.10/hour (~$73/month). If you're running this as a learning exercise, tear the cluster down between sessions with eksctl delete cluster --name demo-cluster --region us-east-1.

My GitHub action fires, but no document lands in Elasticsearch. What now?

Check three things in this order. First, do the repo secrets resolve? Add echo "endpoint length: ${#ES_ENDPOINT}" to the workflow to verify. Second, does the API key have write permission on github-deployments? Test with curl -X POST from your laptop using the same key. Third, is your ES_ENDPOINT value missing the https:// prefix? That one bites everyone.

Can the Kibana AI agent fix the bug or just identify it?

For this tutorial, it identifies. It doesn't open a pull request or call kubectl rollout undo on your behalf. You can extend it to do that with a custom tool that wraps the GitHub API or kubectl, but I'd rather you read the diagnosis first and decide. Autonomous remediation in production is a separate conversation.

Does this replace my on-call runbooks?

No. It collapses the first ten minutes of an incident, the part where you're hunting for the right tab. The runbook still tells you what to do once you know what broke. Think of the agent as the on-call's first responder, not the surgeon.

What IAM permissions does my EKS cluster need?

The OTel DaemonSet only needs outbound HTTPS to the Elastic endpoint; no AWS-specific IAM permissions are required. If your EKS nodes use an instance profile, make sure the security group allows outbound 443. Argo CD similarly needs no special AWS permissions, since it operates against the Kubernetes API only.

References

Top comments (0)