DEV Community

Sergey Zhekpisov
Sergey Zhekpisov

Posted on • Edited on

1

Memorious Prometheus

Recently, we received alerts in Alertmanager, deployed with a kube-stack-prometheus Helm chart. The alert stated that 50% of the EKS endpoints for "apiserver/kubernetes" were down.

50% of the apiserver/kubernetes targets in the default namespace are down.

A brief look at Prometheus revealed that there were four(!) targets for the serviceMonitor/monitoring/prometheus-operator-monito-apiserver/0 endpoint - two were down, and two were up. Upon examining other clusters, it became clear that there are normally only two targets for each cluster.

So, it turns out that the EKS Control Plane was updated during the night, and the apiserver endpoints received new IP addresses. However, the Prometheus scraper retained old IP addresses in its database.

Solution was simple:

kubectl rollout restart statefulset prometheus-prometheus-operator-monito-prometheus -n monitoring

...and the old targets that were "down" disappeared, and the alert was resolved.

Image of AssemblyAI

Automatic Speech Recognition with AssemblyAI

Experience near-human accuracy, low-latency performance, and advanced Speech AI capabilities with AssemblyAI's Speech-to-Text API. Sign up today and get $50 in API credit. No credit card required.

Try the API

Top comments (0)

A Workflow Copilot. Tailored to You.

Pieces.app image

Our desktop app, with its intelligent copilot, streamlines coding by generating snippets, extracting code from screenshots, and accelerating problem-solving.

Read the docs

👋 Kindness is contagious

Dive into an ocean of knowledge with this thought-provoking post, revered deeply within the supportive DEV Community. Developers of all levels are welcome to join and enhance our collective intelligence.

Saying a simple "thank you" can brighten someone's day. Share your gratitude in the comments below!

On DEV, sharing ideas eases our path and fortifies our community connections. Found this helpful? Sending a quick thanks to the author can be profoundly valued.

Okay