Monitoring AWS EKS with Prometheus and Grafana

In my previous articles, I shared how I moved my application from Docker to AWS EKS and how I used Terraform to build the infrastructure.

Once everything was deployed and running, I had another question:

How do I know if my application and Kubernetes cluster are actually healthy?

Seeing pods in a Running state is useful, but it doesn't tell the whole story.

That's where observability came in.

For my project, I implemented a monitoring stack based on Prometheus, Grafana, and Alertmanager.

Here's what I built and, more importantly, what I learned from it.

The Monitoring Stack

I deployed kube-prometheus-stack on my EKS cluster using Helm.

It provides several components out of the box:

• Prometheus to collect and store metrics
• Grafana to visualize them
• Alertmanager to manage alerts
• node-exporter for node-level metrics
• kube-state-metrics for Kubernetes object metrics

Instead of installing and configuring every component separately, this gave me a solid monitoring foundation for Kubernetes.

The simplified flow looks like this:

EKS Cluster
    │
    ├── Application Pods
    ├── Kubernetes Metrics
    └── Worker Nodes
           │
           ▼
      Prometheus
           │
      ┌────┴────┐
      ▼         ▼
   Grafana   Alertmanager

Prometheus collects the metrics, Grafana turns them into dashboards, and Alertmanager handles situations that require attention.

What Am I Actually Monitoring?

Installing Prometheus is easy.

Knowing what is worth monitoring is more interesting.

For my EKS environment, I wanted visibility at several levels.

Cluster and nodes

I can monitor things such as:

• CPU usage
• Memory consumption
• Node availability
• Kubernetes resource usage

This helps answer questions like:

Is the cluster running normally, or are resources becoming saturated?

Kubernetes workloads

At the Kubernetes level, I can observe:

• Number of running pods
• Pod restarts
• Deployment status
• Resource consumption per pod

This is particularly useful when troubleshooting.

A service may still be reachable while one of its pods is constantly restarting in the background.

Without monitoring, that kind of behavior can easily go unnoticed.

Grafana: Turning Metrics into Something Useful

Prometheus collecting thousands of metrics isn't very useful if I can't easily interpret them.

That's where Grafana comes in.

I use Grafana dashboards to get a visual overview of the cluster and quickly identify unusual behavior.

For example, instead of repeatedly running:

kubectl top pods

I can observe CPU and memory usage over time.

That “over time” part is important.

A command tells me what is happening now.

Monitoring helps me understand what happened before the problem appeared.

Accessing Grafana

I also wanted Grafana to be accessible without relying on local port forwarding.

I exposed it through an AWS Application Load Balancer, using a dedicated subdomain and its own ACM certificate for HTTPS.

Conceptually:

Browser
   │
   ▼
Grafana subdomain
   │
   ▼
AWS ALB + HTTPS
   │
   ▼
Grafana on EKS

This made the monitoring interface accessible in a way that was closer to how I would expose an internal platform service in a real environment.

Monitoring vs Alerting

Dashboards are useful when someone is looking at them.

But nobody should have to watch Grafana all day.

That's why the stack also includes Alertmanager.

Prometheus can evaluate alerting rules based on collected metrics. When a condition is triggered, Alertmanager is responsible for processing the alert and routing the notification.

This introduced an important distinction for me:

Monitoring tells me what's happening.
Alerting tells me when I need to care.

A good monitoring system shouldn't generate noise for every small variation. The challenge is deciding which conditions actually require action.

Observability Changed How I Look at Kubernetes

One of the biggest lessons from this project was that:

Running is not the same as healthy.

Kubernetes can tell me that a pod exists.

Prometheus can tell me that its memory consumption has been increasing for the last two hours.

Grafana can make that trend immediately visible.

Alertmanager can tell me when that trend crosses a threshold that requires attention.

These tools complement Kubernetes rather than replace its built-in health mechanisms.

What I Learned

Before implementing this stack, I mostly thought about monitoring as dashboards.

Now I see observability as part of the architecture itself.

When deploying an application, I should be able to answer questions like:

Is it available?

Is it performing normally?

Are resources becoming saturated?

Did something change before the incident happened?

And most importantly:

Will I know there's a problem before a user tells me?

Building the monitoring stack made the EKS environment feel much less like a collection of running resources and much more like a system I could actually operate.

What's Next?

Monitoring EKS also forced me to understand more about what happens underneath Kubernetes.

How does traffic reach my application?

Why are my worker nodes in private subnets?

Why do they need NAT Gateways?

And where exactly does the Application Load Balancer fit into all of this?

That's what I'll cover next:

Understanding EKS Networking: ALB, Private Subnets & NAT Gateway

If you're running Kubernetes, what are the first metrics or alerts you usually configure?