DEV Community

Ranjini Ganeshan
Ranjini Ganeshan

Posted on

Log Doctor: Letting AI Take the First Look at My Kubernetes Errors Using AWS Bedrock and AWS Opensearch

#aws #bedrock #opensearch #ai

Description: "Every 15 minutes, a small CronJob reads the error logs, asks an LLM on Amazon Bedrock what's going on, and posts a short diagnosis to Slack. Here's how I built it on EKS, Fluentd and OpenSearch."

The problem

If you've ever been on call, you know how this goes.

An alert fires: error rate on orders-api above 5%. You open the log search, filter on level: ERROR, and get back a few hundred lines. You start scrolling. Most of them are the same stack trace repeated over and over. A few are something else. One or two of those might be the real problem, but you won't know until you've read them.

By the time you've worked out "okay, the database is out of connections, and the payment timeouts are probably a side effect", ten minutes have gone. That's ten minutes of reading, not fixing.

  • Alerts tell you that something is wrong. A number crossed a threshold.
  • Log search gives you everything, raw and unsorted.

What's missing is the step in between, the first pass a senior engineer does without thinking about it. Group the noise, notice which error is the loud one, form a quick theory, and decide where to look first.

That first pass is mostly reading and summarizing repetitive text. That happens to be something large language models are genuinely good at.

What I wanted

I didn't want an AI agent with access to my whole cluster. I wanted something much smaller and simpler:

Every 15 minutes, look at the errors from the last 15 minutes. If there are any, post a short note to Slack, per service: what's happening, the likely cause, and what to check. If there are none, say nothing.

A few rules I set for myself before writing any code:

  1. The model never queries anything. My code fetches the logs, groups them and trims them. The model only sees a few kilobytes of summary. That keeps the output predictable, and I can see exactly what was sent.
  2. Cost has a ceiling. The amount sent to the model is capped by configuration, not by how bad the incident is. A thousand copies of the same stack trace become one line with a count.
  3. Scrub before sending. Passwords, tokens and email addresses get redacted before the prompt is built.
  4. No AWS keys anywhere. Every component talks to AWS with an IAM role, so there are no access keys to rotate or leak.
  5. Quiet when healthy. A bot that posts "all good!" every 15 minutes gets muted by day two.

How it fits together 

orders-api / payments-api → Fluentd → OpenSearch → Log Doctor (CronJob) → Amazon Bedrock → Slack
Enter fullscreen mode Exit fullscreen mode

Everything runs on a stack that's common in AWS shops, and it's all Terraform plus plain Kubernetes manifests:

  • Two FastAPI services write one JSON object per log line to stdout. They have a /simulate-error endpoint that fails on purpose in realistic ways: a database out of connections, a payment call timing out, and the classic 'NoneType' object is not subscriptable.
  • Fluentd runs on every node, tails the container logs, adds Kubernetes metadata and writes to a daily app-logs-YYYY.MM.DD index.
  • Amazon OpenSearch stores the logs. The domain sits inside the VPC and only accepts signed IAM requests.
  • Log Doctor is about 500 lines of Python running as a CronJob. It does the fetching, the trimming and the posting.
  • Amazon Bedrock does the explaining. I picked it over a direct LLM API mainly because of access: there's no API key to manage, and the CronJob's IAM role can call exactly one model.
  • Slack is the output, because that's where people already look.

👉 The code is here: github.com/ranjiniganeshan/log-doctor

One query does most of the work

My first thought was to fetch a few hundred log lines and give them to the model. But OpenSearch can already do the grouping, and it gives exact counts. Log Doctor runs a single aggregation: errors per service, then top messages per service, with two recent samples of each.

"aggs": {
    "services": {
        "terms": {"field": cfg.service_field, "size": cfg.max_services, "missing": "(unknown)"},
        "aggs": {
            "messages": {
                "terms": {"field": cfg.message_field, "size": cfg.max_messages_per_service},
                "aggs": {
                    "first_seen": {"min": {"field": cfg.timestamp_field}},
                    "last_seen": {"max": {"field": cfg.timestamp_field}},
                    "samples": {
                        "top_hits": {
                            "size": cfg.samples_per_message,
                            "sort": [{cfg.timestamp_field: {"order": "desc"}}],
                            "_source": {"includes": SAMPLE_SOURCE_FIELDS},
                        }
                    },
                },
            }
        },
    }
},
Enter fullscreen mode Exit fullscreen mode

With the defaults, that's at most 10 services, 5 messages per service and 2 samples per message. So I know the maximum size of the prompt, however bad the incident is. The Python code then trims it further:

  • Stack traces keep only their last 12 lines, because that's usually where the cause is.
  • The whole prompt has a hard budget of 16,000 characters. When it gets tight, samples go first, then whole services, and the prompt says what was left out.
  • maxTokens caps the reply.

There's another benefit: the model gets real counts. It's told "25 of these" instead of having to count lines itself.

Treating logs as untrusted input

I spent extra time on this part. Log lines can contain whatever your users typed. If you put them in a prompt and post the answer to Slack, a stranger's text can end up in your on-call channel.

So I added three protections:

_REDACTIONS = [
    (re.compile(r"AKIA[0-9A-Z]{16}"), "[REDACTED_AWS_KEY]"),
    (re.compile(r"(?i)bearer\s+[a-z0-9._\-]+"), "Bearer [REDACTED]"),
    (re.compile(r"(?i)(password|passwd|secret|token|api[_-]?key)\s*[=:]\s*\S+"), r"\1=[REDACTED]"),
    (re.compile(r"[A-Za-z0-9._%+\-]+@[A-Za-z0-9.\-]+\.[A-Za-z]{2,}"), "[REDACTED_EMAIL]"),
    (re.compile(r"https://hooks\.slack\.com/\S+"), "[REDACTED_WEBHOOK]"),
]
Enter fullscreen mode Exit fullscreen mode
  1. Redaction runs on the messages and samples before they reach the prompt.
  2. The system prompt says log content is untrusted and that instructions inside log lines must not be followed.
  3. Slack control characters are escaped, so a log line can't get <!channel> pushed to your whole team.

None of these is perfect. Regex only catches the patterns you thought of, and a model can still be tricked. So the most important protection is the design itself: the model has no tools and no credentials, and all it can do is write text for Slack. If someone does trick it, the worst result is a misleading message.

Hands-on: build it yourself

Prerequisites

AWS account

  • A Paid plan. The AWS Free plan limits which instance types you can launch, so the EKS node may not start.
  • Permission to create an IAM OIDC provider. IRSA needs iam:CreateOpenIDConnectProvider. If your account is in an AWS Organization, check that no service control policy blocks it.
  • Access to your chosen region. All the resources are created in one region, so make sure no policy restricts it.
  • A Bedrock model you can call from that region. See Step 1.

Tools

  • AWS CLI v2
  • Terraform 1.6 or later
  • kubectl
  • Docker with buildx
  • envsubst (brew install gettext on macOS)
  • make and curl

Slack

  • A workspace where you can add an app

Run these checks before you create anything. They don't cost anything, and the EC2 command is only a dry run:

export AWS_REGION=ap-south-1

# Confirm you're in the right account
aws sts get-caller-identity

# Check that you can launch the node type
AMI=$(aws ssm get-parameter \
  --name /aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64 \
  --query Parameter.Value --output text)
aws ec2 run-instances --dry-run --image-id "$AMI" \
  --instance-type t3.medium --count 1
Enter fullscreen mode Exit fullscreen mode

The EC2 command should reply with Request would have succeeded, but DryRun flag is set.

💸 This creates billable resources. Estimated cost is in the cost section. Tear it down when you're finished.

Step 1: Pick a model and test it

Log Doctor uses Bedrock's Converse API, so it works with any Bedrock chat model. List the models you can call directly in your region:

aws bedrock list-foundation-models --region ap-south-1 --by-output-modality TEXT \
  --query "modelSummaries[?contains(inferenceTypesSupported,'ON_DEMAND')].modelId"
Enter fullscreen mode Exit fullscreen mode

I used openai.gpt-oss-20b-1:0, OpenAI's open-weight model. Some models are only offered through cross-region inference profiles, with IDs that start with global. or apac.. Those can send your request to another region, so a policy that limits your account to one region can block them.

Then make a real test call. A model appearing in the list doesn't prove your account can use it.

export AWS_REGION=ap-south-1
aws bedrock-runtime converse --model-id "openai.gpt-oss-20b-1:0" \
  --messages '[{"role":"user","content":[{"text":"Reply with OK"}]}]' \
  --inference-config '{"maxTokens":800}'
Enter fullscreen mode Exit fullscreen mode

Third-party models can create an AWS Marketplace subscription on first use. The CronJob's role isn't allowed to do that, which is correct for least privilege, so make this first call with your own admin identity.

This model returns reasoning blocks as well as the answer, and it counts them against maxTokens. Don't set the limit too low. Log Doctor keeps only the text blocks.

Step 2: Create a Slack webhook

At api.slack.com/apps, create an app, turn on Incoming Webhooks, and add one to a channel. Treat the URL like a password, because anyone who has it can post to your channel.

export SLACK_WEBHOOK_URL='https://hooks.slack.com/services/...'
Enter fullscreen mode Exit fullscreen mode

It stays in your shell and goes into a Kubernetes Secret later. Nothing gets written to the repo.

📌 ADD BEFORE PUBLISHING: screenshot of the Incoming Webhooks page, with the URL hidden

Step 3: Create the infrastructure 

cp infra/terraform.tfvars.example infra/terraform.tfvars
Enter fullscreen mode Exit fullscreen mode

These are the settings I used:

region                               = "ap-south-1"
bedrock_model_id                     = "openai.gpt-oss-20b-1:0"
node_instance_type                   = "t3.medium"
cluster_endpoint_public_access_cidrs = ["<your-ip>/32"]
Enter fullscreen mode Exit fullscreen mode

Then:

make infra-up
kubectl get nodes
Enter fullscreen mode Exit fullscreen mode

That creates 67 resources: a VPC, an EKS cluster with one node, a single-node OpenSearch domain, two ECR repositories and the IAM roles.

For me, the whole apply took 23 minutes. OpenSearch was the slowest part at about 14 minutes, and the EKS control plane took a little over 8.

kubectl get nodes 
NAME                                          STATUS   ROLES    AGE   VERSION
ip-10-40-13-187.ap-south-1.compute.internal   Ready    <none>   17m   v1.36.5-eks-6e7947a
Enter fullscreen mode Exit fullscreen mode

How the permissions work. Neither workload has stored AWS credentials. Each one gets its own role through IRSA, and each role's trust policy is pinned to one exact service account:

Workload Can do
Fluentd POST/PUT on /_bulk and /app-logs-*
Log Doctor GET/POST on /app-logs-*/_search, and InvokeModel on one model

I used OpenSearch's IAM-only access policy rather than fine-grained access control because it's the simplest secure option: no master user, no role mappings, no setup step after terraform apply. The trade-off is that Fluentd writes through /_bulk, which names the index in the request body, so its role could write to any index. It still can't read, search or delete. If you need per-index write control, use fine-grained access control.

Step 4: Build and push the images 

make build-push
Enter fullscreen mode Exit fullscreen mode

This builds for linux/amd64, which matters if you're on Apple Silicon: without it, you get exec format error when the pods start on an x86 node.

Step 5: Deploy 

make slack-secret
make deploy
make status
Enter fullscreen mode Exit fullscreen mode

make deploy renders the manifests from the Terraform outputs and applies them in order: namespaces, service accounts, Fluentd, the two services, then the CronJob.

Opensearch, EKS cluster is created.

{
    "DomainNames": [
        {
            "DomainName": "log-doctor",
            "EngineType": "OpenSearch"
        }
    ]
}
{
    "clusters": [
        "log-doctor"
    ]
}
[
    "nat-0c95abf4eb696bfdf"
]
Enter fullscreen mode Exit fullscreen mode

Two CronJob settings are there to handle failures:

concurrencyPolicy: Forbid   # a slow run never overlaps the next one
backoffLimit: 0             # a retry could post the same summary twice
Enter fullscreen mode Exit fullscreen mode

If Bedrock fails, Log Doctor still posts a raw digest of top messages and counts, then exits 1. On-call gets the information without the explanation, and the failed Job stays visible.

kubectl -n logging get pods
NAME            READY   STATUS    RESTARTS   AGE
fluentd-v9dxv   1/1     Running   0          2m2s
kubectl -n apps get pods
NAME                            READY   STATUS    RESTARTS   AGE
orders-api-5d7fc5f46f-s49kk     1/1     Running   0          2m1s
payments-api-754bb84f5c-f7c2v   1/1     Running   0          2m
kubectl -n log-doctor get cronjobs,jobs,pods
NAME                       SCHEDULE       TIMEZONE   SUSPEND   ACTIVE   LAST SCHEDULE   AGE
cronjob.batch/log-doctor   */15 * * * *   Etc/UTC    False     0        <none>          2m
Enter fullscreen mode Exit fullscreen mode

Demo: breaking things on purpose

Now for the fun part. Each step below shows one stage of the pipeline.

1. Cause some trouble. This sends normal traffic plus a burst of errors: 25 database failures and 4 NoneType bugs on orders-api, and 12 timeouts and 3 database failures on payments-api.

make simulate-errors
Enter fullscreen mode Exit fullscreen mode 
{"timestamp": "2026-11-05T15:50:04.398+00:00", "level": "ERROR", "service": "orders-api", "message": "Unhandled error building order confirmation", "trace_id": "3dd9857419094b36a047432aba78a598", "logger": "orders-api", "error": {"type": "TypeError", "message": "'NoneType' object is not subscriptable", "stack_trace": "Traceback (most recent call last):\n  File \"/srv/app/main.py\", line 194, in simulate_error\n    action()\n  File \"/srv/app/main.py\", line 182, in <lambda>\n    \"null\": (\"Unhandled error building order confirmation\", lambda: build_confirmation_email(\"c-9999\")),\n                                                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/srv/app/main.py\", line 146, in build_confirmation_email\n    return f\"To: {customer['email']}\"  # customer is None for unknown IDs -> TypeError\n                  ~~~~~~~~^^^^^^^^^\nTypeError: 'NoneType' object is not subscriptable\n"}, "error_scenario": "null"}
Enter fullscreen mode Exit fullscreen mode

2. See exactly what the model will see. Wait 30 seconds for Fluentd to flush, then run this. It makes no Bedrock call and posts nothing, and it also proves the logs reached OpenSearch.

make dry-run SKIP_LLM=1
Enter fullscreen mode Exit fullscreen mode 
ERROR log summary for the last 15 minutes (total errors: 44, services shown: 2).
<logs>
## service: orders-api (total errors: 29)
- message: "Failed to fetch order from database"
  count: 25, first_seen: 2026-11-05T15:50:04.846Z, last_seen: 2026-11-05T15:50:04.846Z
  sample 1:
      [2026-11-05T15:50:04.846254876+00:00] pod=orders-api-5d7fc5f46f-s49kk trace_id=51c260bd3fab48a793f43dec068802e9
      DatabaseConnectionError: connection to server at "orders-db.internal" (10.40.3.21), port 5432 failed: FATAL:  remaining connection slots are reserved for non-replication superuser connections
      ...
Enter fullscreen mode Exit fullscreen mode

3. Ask the model, but don't post.

make dry-run
Enter fullscreen mode Exit fullscreen mode 
*Shared root cause:* `orders-db` connection pool exhausted – “remaining connection slots are reserved for non‑replication superuser connections” is firing in both services.

**Orders‑api** – 29 errors
• *What's happening:* 25 DB connection errors plus 4 TypeError failures building confirmation emails.
• *Likely cause:* The Postgres pool is saturated, and an unknown customer ID triggers a None dereference.
• *What to check:*
  • run `kubectl exec -it <orders-db-pod> -- psql -c "SELECT * FROM pg_stat_activity WHERE state='active';"` to count active connections.
  • review `orders-api`’s connection‑pool config (`max_pool_size`, `idle_timeout`) and consider raising `max_connections` or decreasing pool size.
  • audit `build_confirmation_email` to guard against missing customer records.

**Payments‑api** – 15 errors
• *What's happening:* 12 upstream timeouts to the payment gateway, 3 DB connection errors.
• *Likely cause:* The gateway service is too slow or unresponsive, and the shared DB pool limit is contributing.
• *What to check:*
  • check `payment-gateway` pod CPU/mem and logs for latency spikes (`kubectl logs <gateway-pod>`).
  • run `curl -v http://payment-gateway.payments.svc.cluster.local:8080/v1/authorize` from a pod to test connectivity.
  • increase `payments-api`’s HTTP timeout or enable a retry/circuit‑breaker if the gateway is intermittently overloaded.
Enter fullscreen mode Exit fullscreen mode

4. The real thing.

make run-now
Enter fullscreen mode Exit fullscreen mode

Wait for a window with no errors. The job runs, finds zero errors, and posts nothing.

Tearing it down 

make infra-down
Enter fullscreen mode Exit fullscreen mode

If you try it, I'd love to hear what it says about your logs, especially when it gets something wrong. The code is at github.com/ranjiniganeshan/log-doctor.

Top comments (0)