If you don't know what is happening inside your infrastructure and applications, you are not really operating them — you are guessing.
Modern applications run across cloud infrastructure, containers, Kubernetes clusters, databases, microservices, load balancers, and external services.
When something goes wrong in production, a DevOps engineer should be able to answer questions like:
- Is the server healthy?
- Is the application available?
- Are requests becoming slower?
- Are errors increasing?
- Is the database overloaded?
- Are Kubernetes pods restarting?
- Is the infrastructure running out of resources?
- Which service is causing the problem?
- Did something change before the problem started?
This is where monitoring and observability become fundamental DevOps skills.
- What Is Monitoring?
Monitoring is the continuous collection and analysis of information about infrastructure, applications, and services.
A simple way to think about monitoring is:
System
↓
Metrics / Logs / Events
↓
Collection
↓
Storage
↓
Visualization
↓
Alerting
↓
Engineer
↓
Action
The goal is not simply to create dashboards.
The real goal is:
«Detect problems early, understand their impact, and respond before they become major incidents.»
- What Should a DevOps Engineer Monitor?
Monitoring can be divided into several important areas.
Infrastructure Monitoring
For Linux servers, EC2 instances, VMs, and physical servers, monitor:
- CPU utilization
- Memory utilization
- Disk utilization
- Disk I/O
- Network traffic
- Network errors
- Load average
- Filesystem usage
- Processes
- System uptime
For example:
EC2 Server
CPU = 92%
Memory = 88%
Disk = 91%
Load = 8.5
Network = 850 Mbps
This immediately tells us that the server may be under significant pressure.
- Application Monitoring
Infrastructure can be healthy while the application is completely broken.
Therefore, application monitoring is equally important.
Monitor:
- Request count
- Response time
- Latency
- Error rate
- HTTP 4xx errors
- HTTP 5xx errors
- Timeouts
- Application exceptions
- Thread count
- JVM heap
- Garbage collection
- Database connection pools
Example:
Requests = 10,000/min
Average latency = 250 ms
P95 latency = 700 ms
5xx errors = 4%
The server may have only 40% CPU utilization, but users may still be experiencing failures.
- Database Monitoring
Databases are often critical bottlenecks.
Monitor:
- CPU
- Memory
- Storage
- Connections
- Connection pool
- Query latency
- Slow queries
- Locks
- Transactions
- Replication lag
- Connection failures
Example:
DB Connections = 95/100
Slow Queries = 120
Replication Lag = 30 seconds
CPU = 90%
This should immediately attract the attention of the operations team.
- Kubernetes Monitoring
For Kubernetes and EKS, monitoring needs to happen at multiple levels.
Think of Kubernetes monitoring like this:
Cluster
↓
Node
↓
Pod
↓
Container
↓
Application
Cluster
Monitor:
- Cluster health
- Node count
- API server
- Scheduler
- Controller Manager
- Resource capacity
Nodes
Monitor:
- CPU
- Memory
- Disk
- Disk pressure
- Network
- Node readiness
Pods
Monitor:
- Pod status
- CPU
- Memory
- Restarts
- OOMKilled
- Pending pods
- CrashLoopBackOff
Application
Monitor:
- Requests
- Latency
- Errors
- Availability
This layered approach becomes extremely important when troubleshooting production Kubernetes environments.
- The Four Golden Signals
One of the most important concepts for DevOps and SRE engineers is the Four Golden Signals.
They are:
- Latency
- Traffic
- Errors
- Saturation
Let's understand them.
6.1 Latency
Latency tells us:
«How long does the system take to respond?»
Example:
API Request
↓
Response = 200 ms
Don't look only at average latency.
Important percentiles include:
P50 = 100 ms
P95 = 400 ms
P99 = 1.2 sec
P95 and P99 can reveal slow requests that an average may hide.
- Traffic
Traffic tells us:
«How much workload is the system receiving?»
Examples:
Requests/sec
Transactions/sec
Messages/sec
Users/sec
Example:
Normal traffic = 500 requests/sec
Current traffic = 5,000 requests/sec
This could indicate:
- Traffic spike
- Marketing campaign
- DDoS attack
- Application bug
- Retry storm
Traffic must always be considered in context.
- Errors
Errors tell us:
«How many requests are failing?»
Example:
Total requests = 10,000
Failed requests = 500
Error rate = 5%
Monitor:
HTTP 4xx
HTTP 5xx
Timeouts
Exceptions
Connection failures
A sudden increase in HTTP 5xx errors can indicate a serious application or infrastructure problem.
- Saturation
Saturation tells us:
«How close is the system to its limit?»
Examples:
CPU = 90%
Memory = 95%
Disk = 92%
DB connections = 98%
Thread pool = 95%
Saturation is extremely important because it can help you detect problems before the system completely fails.
- Metrics vs Logs vs Traces
This is one of the most important concepts in modern observability.
Metrics
Metrics are numerical measurements over time.
Examples:
CPU = 80%
Memory = 70%
Requests = 500/sec
Latency = 200 ms
Error rate = 2%
Metrics are excellent for:
- Dashboards
- Alerts
- Trends
- Capacity planning
Metrics usually tell you:
«Something is wrong.»
- Logs
Logs provide detailed information about events.
Example:
2026-08-17 12:10:01
ERROR PaymentService
Database connection timeout
Logs can help answer:
«What happened?»
For example:
Metric:
5xx errors increased
↓
Log:
Database connection timeout
↓
Root cause:
Database connection pool exhausted
- Distributed Tracing
Tracing becomes extremely useful in microservices.
Imagine this request:
User
↓
API Gateway
↓
Order Service
↓
Payment Service
↓
Database
The complete request takes:
Total = 2 seconds
Tracing can show:
API Gateway = 50 ms
Order Service = 300 ms
Payment Service = 1.5 sec
Database = 100 ms
Now we immediately know:
«Payment Service is the major contributor to the latency.»
This is why tracing is especially valuable in distributed systems.
- The Four Observability Signals
Modern observability goes beyond basic infrastructure monitoring.
A commonly used model includes:
Metrics
Logs
Traces
Profiles
Metrics provide the high-level picture.
Logs provide detailed event information.
Traces show request flow across services.
Profiles help identify code-level CPU and memory bottlenecks.
The important point is that these signals should not be treated as isolated systems.
You want to correlate them.
For example:
Metric
↓
Latency increased
↓
Trace
↓
Payment Service is slow
↓
Log
↓
Database timeout
↓
Root Cause
↓
Database connection exhaustion
That is the real power of observability.
- Monitoring vs Observability
These terms are often used interchangeably, but they are not exactly the same.
Monitoring
Monitoring generally focuses on known questions.
For example:
Is CPU > 80%?
Is disk > 90%?
Is error rate > 5%?
Observability
Observability helps you investigate unexpected behavior and understand the internal state of complex systems from their telemetry.
A useful way to remember it:
Monitoring:
"Something is wrong."
Observability:
"Something is wrong.
Where is it?
Why is it happening?
What changed?
Which services are affected?"
Modern distributed systems make this distinction increasingly important.
- Prometheus
Prometheus is one of the most important open-source tools for DevOps monitoring.
A simplified architecture is:
Linux Server
↓
Node Exporter
↓
Prometheus
↓
PromQL
↓
Grafana
Prometheus collects and stores time-series metrics.
For example:
CPU
Memory
Disk
Network
Application metrics
Kubernetes metrics
- What Is an Exporter?
An exporter exposes metrics that Prometheus can collect.
For Linux:
Linux Server
↓
Node Exporter
↓
Prometheus
Node Exporter can expose system-level metrics such as:
- CPU
- Memory
- Filesystem
- Disk
- Network
- Load
This gives Prometheus visibility into the server.
- What Is PromQL?
PromQL is Prometheus's query language.
It allows engineers to ask questions about metrics.
For example:
CPU utilization
or:
Which servers have high CPU?
or:
What is the request rate?
or:
What is the error rate over the last 5 minutes?
PromQL is therefore a critical skill if you want to become strong in Prometheus-based monitoring.
- What Is Grafana?
Grafana is used to query, visualize, explore, and alert on observability data.
A typical dashboard might look conceptually like:
+----------------------------------+
| Production Dashboard |
+----------------------------------+
CPU 72%
Memory 68%
Disk 81%
Requests/sec 850
Error Rate 0.5%
P95 Latency 320 ms
Active Pods 24
Pod Restarts 3
DB Connections 72%
Grafana dashboards allow engineers to see system behavior at a glance.
- What Is Alerting?
A dashboard tells you what is happening.
An alert tells you:
«Something needs attention.»
For example:
IF disk_usage > 80%
FOR 10 minutes
THEN
Generate alert
A typical flow might be:
Prometheus
↓
Alert Rule
↓
Alertmanager
↓
Email / Slack / PagerDuty / SNS
↓
DevOps Engineer
- Avoid Bad Alerts
Consider this alert:
CPU > 80%
This could create many unnecessary alerts.
CPU might temporarily reach 80% during a legitimate workload spike.
A better rule could be:
CPU > 80%
FOR 15 minutes
Even better, consider the user impact:
High CPU
+
High latency
+
Increasing error rate
This gives you stronger evidence that there is an actual service problem.
Grafana's dashboard guidance similarly recommends focusing alerts on symptoms/user impact rather than simply alerting on every underlying cause.
- A Real Production Scenario
Imagine your users report:
«"The application is very slow."»
Don't randomly SSH into servers.
Use a systematic approach.
Step 1 — Check availability
Is the application UP?
Step 2 — Check traffic
Did traffic suddenly increase?
Step 3 — Check latency
P95 = ?
P99 = ?
Step 4 — Check errors
4xx = ?
5xx = ?
Timeouts = ?
Step 5 — Check infrastructure
CPU
Memory
Disk
Network
Step 6 — Check Kubernetes
Pod restarts
OOMKilled
Pending pods
Node pressure
Step 7 — Check database
Connections
Slow queries
CPU
Locks
Latency
Step 8 — Check traces
Find which microservice is taking the most time.
Step 9 — Check logs
Find the detailed error.
Step 10 — Find the root cause
For example:
Traffic spike
↓
More requests
↓
Payment Service overloaded
↓
Thread pool exhausted
↓
Requests waiting
↓
Latency increased
↓
Timeouts
↓
HTTP 5xx
↓
Users experienced failures
This is how monitoring becomes a troubleshooting tool, not just a dashboard.
- A Practical DevOps Monitoring Architecture
A typical environment could look like:
USERS
|
↓
Load Balancer
|
↓
Kubernetes
|
+--------------+--------------+
| | |
↓ ↓ ↓
Service A Service B Service C
| | |
+--------------+--------------+
|
Metrics / Logs / Traces
|
+--------------+--------------+
| | |
↓ ↓ ↓
Prometheus Loki OpenTelemetry
| | |
+--------------+--------------+
|
↓
Grafana
|
↓
Alerting
|
+-----------+-----------+
| |
SNS Slack
Grafana can connect to multiple data sources and visualize metrics, logs, traces, and other data from a unified interface.
- What Should You Learn as a DevOps Engineer?
I recommend learning monitoring in this order:
- Linux monitoring ↓
- Infrastructure metrics ↓
- Application metrics ↓
- Logs ↓
- Alerting ↓
- Prometheus ↓
- PromQL ↓
- Grafana ↓
- Alertmanager ↓
- JVM monitoring ↓
- Kubernetes monitoring ↓
- EKS monitoring ↓
- Distributed tracing ↓
- OpenTelemetry ↓
- SLI / SLO / SLA ↓
- Incident response ↓
- SRE practices
- The Most Important Mental Model
Whenever you troubleshoot production, ask these questions:
Is the system available?
↓
Is it fast?
↓
Is it failing?
↓
Is it reaching its limits?
↓
Where is the problem?
↓
Why is it happening?
↓
What changed?
↓
What should we do?
This progression takes you from basic monitoring toward observability and SRE.
- Final Takeaway
Monitoring is not:
Install Prometheus
+
Install Grafana
+
Create dashboards
That is only the tooling.
Real monitoring means understanding:
Infrastructure
+
Applications
+
Databases
+
Kubernetes
+
Metrics
+
Logs
+
Traces
+
Alerts
+
Troubleshooting
+
SRE practices
The ultimate objective is simple:
«Know that something is going wrong before your users tell you — and have enough information to understand why.»
That is what separates a DevOps engineer who merely operates tools from an engineer who can actually operate production systems.
What's Next?
This article establishes the fundamentals.
The next practical step is to build a complete monitoring lab:
AWS EC2
↓
Node Exporter
↓
Prometheus
↓
Grafana
↓
CPU / Memory / Disk / Network
↓
PromQL
↓
Alert Rules
↓
Alertmanager
↓
SNS / Email
Then extend the same architecture to:
Spring Boot
↓
Actuator
↓
Prometheus
↓
Grafana
Kubernetes / EKS
↓
kube-state-metrics
↓
Prometheus
↓
Grafana
That progression will take you from monitoring fundamentals → production monitoring → Kubernetes monitoring → observability → SRE.
Top comments (0)