You can't fix what you can't see. As systems grow more distributed — microservices, serverless functions, event-driven flows — the gap between "deployed" and "observable" becomes the difference between resolving incidents in minutes versus hours.
AWS's observability stack has evolved significantly. X-Ray SDK entered maintenance mode in February 2026, replaced by OpenTelemetry. Application Signals introduced SLO-based monitoring. CloudWatch absorbed capabilities that previously required third-party tools.
This post covers the complete AWS observability architecture for modern applications — what to collect, where to send it, and how to build alerting that actually works.
The Three Pillars of Observability
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ METRICS │ │ LOGS │ │ TRACES │
│ │ │ │ │ │
│ What's │ │ What │ │ Where in │
│ happening? │ │ happened? │ │ the path? │
│ │ │ │ │ │
│ CPU, memory │ │ Error msgs │ │ Request │
│ request rate│ │ debug info │ │ flow across │
│ error count │ │ audit trail │ │ services │
│ latency p99 │ │ │ │ │
└─────────────┘ └─────────────┘ └─────────────┘
│ │ │
└───────────────────┼───────────────────┘
│
┌─────────────┐
│ CORRELATION │
│ Trace ID → │
│ connects │
│ all three │
└─────────────┘
Metrics tell you WHAT is wrong (error rate spiked).
Logs tell you WHY it's wrong (NullPointerException at line 42).
Traces tell you WHERE in the request path it went wrong (payment-service → database timeout).
All three need a common correlation key (trace ID) to be useful together.
The AWS Observability Stack
┌─────────────────────────────────────────────────────────────────┐
│ COLLECTION (Instrumentation) │
│ ADOT (OpenTelemetry) | CloudWatch Agent | SDKs │
├─────────────────────────────────────────────────────────────────┤
│ PROCESSING & STORAGE │
│ CloudWatch (Metrics + Logs) | X-Ray (Traces) | S3 (Archive) │
├─────────────────────────────────────────────────────────────────┤
│ ANALYSIS │
│ Log Insights | Metrics Insights | Trace Analytics | ServiceLens │
├─────────────────────────────────────────────────────────────────┤
│ SLO & ALERTING │
│ Application Signals | CloudWatch Alarms | Composite Alarms │
├─────────────────────────────────────────────────────────────────┤
│ VISUALIZATION │
│ CloudWatch Dashboards | ServiceLens Map | Managed Grafana │
└─────────────────────────────────────────────────────────────────┘
Metrics: CloudWatch Metrics + EMF
Default Metrics (Free)
AWS services automatically emit metrics to CloudWatch:
- EC2: CPU, network, disk, status checks
- ECS/Fargate: CPU, memory utilization per task
- Lambda: invocations, duration, errors, throttles, cold starts
- ALB: request count, latency, HTTP 4xx/5xx, healthy host count
- RDS: connections, IOPS, replication lag, free storage
Custom Metrics
For application-specific metrics, use Embedded Metric Format (EMF):
{
"_aws": {
"Timestamp": 1692286800000,
"CloudWatchMetrics": [{
"Namespace": "MyApp/Orders",
"Dimensions": [["Service", "Environment"]],
"Metrics": [
{"Name": "OrdersProcessed", "Unit": "Count"},
{"Name": "ProcessingTime", "Unit": "Milliseconds"}
]
}]
},
"Service": "order-service",
"Environment": "production",
"OrdersProcessed": 47,
"ProcessingTime": 230
}
Why EMF over PutMetricData: EMF lets you log structured JSON that CloudWatch automatically extracts as metrics AND preserves as log entries — one write, two outputs. No PutMetricData API calls (cheaper, lower latency).
Container Insights
For ECS and EKS, Container Insights provides:
- Per-container CPU and memory
- Per-pod/task network I/O
- Cluster-level resource utilization
- Kubernetes-aware dimensions (namespace, deployment, pod)
Enable with: ADOT collector as DaemonSet (EKS) or sidecar (ECS).
Logs: CloudWatch Logs + Log Insights
Structured Logging (Essential)
Unstructured logs are unsearchable at scale. Always log structured JSON:
{
"timestamp": "2026-08-13T10:30:00Z",
"level": "ERROR",
"service": "payment-service",
"traceId": "1-abc123-def456",
"requestId": "req-789",
"message": "Payment failed",
"error": "TimeoutException",
"customerId": "cust-123",
"amount": 99.99,
"duration_ms": 5002
}
Log Insights Queries
CloudWatch Log Insights provides SQL-like querying across log groups:
# Find slowest requests in the last hour
fields @timestamp, service, duration_ms, requestId
| filter duration_ms > 1000
| sort duration_ms desc
| limit 20
# Error rate per service
filter level = "ERROR"
| stats count(*) as errors by service
| sort errors desc
# Trace specific request across services
filter traceId = "1-abc123-def456"
| sort @timestamp asc
Log Architecture Patterns
| Pattern | When |
|---|---|
| Direct to CloudWatch | Lambda (automatic), ECS (awslogs driver), EKS (Fluent Bit) |
| CloudWatch → S3 | Long-term retention, compliance archives (subscription filter) |
| CloudWatch → OpenSearch | Need full-text search, complex aggregations, Kibana dashboards |
| ADOT → CloudWatch | OpenTelemetry-based collection with CloudWatch backend |
Retention and Cost
- Set retention policies per log group (don't default to "never expire")
- Production: 30-90 days in CloudWatch, archive to S3 Glacier after
- Dev/staging: 7-14 days (no archival)
- Infrequent Access class: 50% cheaper for logs you rarely query
Traces: X-Ray + OpenTelemetry (ADOT)
The Migration: X-Ray SDK → OpenTelemetry
Important: X-Ray SDK entered maintenance mode in February 2026. AWS now recommends OpenTelemetry for all new instrumentation.
| Old Approach | New Approach |
|---|---|
| X-Ray SDK (language-specific) | AWS Distro for OpenTelemetry (ADOT) |
| Proprietary trace format | OpenTelemetry (OTel) standard |
| AWS-only export | Export to X-Ray, Jaeger, Zipkin, Grafana Tempo, etc. |
| Auto-instrumentation (limited) | Auto-instrumentation (comprehensive) |
ADOT Architecture
┌────────────┐ ┌──────────────┐ ┌─────────────┐
│Application │────→│ ADOT Collector│────→│ X-Ray │
│(OTel SDK) │ │ (sidecar or │ │ CloudWatch │
│ │ │ DaemonSet) │ │ Prometheus │
└────────────┘ └──────────────┘ └─────────────┘
ADOT Collector receives traces/metrics via OTLP protocol and exports to one or more backends. You can send to X-Ray AND Grafana Tempo simultaneously.
Auto-Instrumentation (Zero Code Changes)
For Java, Python, Node.js, and .NET — ADOT auto-instrumentation captures:
- HTTP requests (incoming and outgoing)
- Database queries (SQL, DynamoDB, Redis)
- AWS SDK calls (S3, SQS, SNS, Lambda invocations)
- gRPC calls
EKS: Deploy ADOT auto-instrumentation as a Kubernetes operator — injects instrumentation into pods automatically.
Lambda: Enable Lambda X-Ray Active Tracing — one toggle, zero code.
Trace Anatomy
Trace: 1-abc123-def456 (entire request lifecycle)
│
├── Span: API Gateway (12ms)
│ └── Span: Lambda: order-handler (450ms)
│ ├── Span: DynamoDB: GetItem (23ms)
│ ├── Span: HTTP: payment-service (380ms)
│ │ └── Span: RDS: INSERT (45ms) ← SLOW?
│ └── Span: SQS: SendMessage (15ms)
Each span shows: service name, duration, status, metadata. Find the bottleneck instantly.
Application Signals: SLO-Based Monitoring
Application Signals (GA 2025) is CloudWatch's answer to "monitor what matters to users" — automatically tracks SLIs (Service Level Indicators) and lets you define SLOs (Service Level Objectives).
What It Auto-Discovers
Without any configuration, Application Signals detects:
- Services and their dependencies (service map)
- Call volume between services
- Latency (p50, p90, p99)
- Error rate and fault rate
- Availability
Defining SLOs
SLO: "Payment Service Availability"
├── SLI: Success rate (HTTP 2xx / total requests)
├── Target: 99.9% over 30-day rolling window
├── Error budget: 0.1% (43 minutes/month of allowed errors)
└── Alert: When burn rate exceeds 10x normal → page on-call
Why SLOs > Threshold Alarms
| Traditional Alarm | SLO-Based Alert |
|---|---|
| "CPU > 80%" → alert | "Error budget burning too fast" → alert |
| Noisy, often false positive | Only fires when users are impacted |
| Measures infrastructure health | Measures user experience |
| Doesn't account for context | Accounts for error budget remaining |
Alerting That Works
Alarm Strategy
| Layer | What to Alert On | Action |
|---|---|---|
| SLO breach | Error budget burn rate > threshold | Page on-call immediately |
| Service health | Error rate > 5% for 5 minutes | Page on-call |
| Saturation | CPU > 85%, memory > 90%, disk > 80% | Auto-scale + notify |
| Dependencies | Upstream latency > SLA | Notify (not page) |
| Business metrics | Orders/min drops > 50% | Alert business + engineering |
Composite Alarms
Reduce noise by combining related alarms:
CompositeAlarm: "Payment Service Degraded"
├── AND: Error rate > 5%
├── AND: Latency p99 > 2000ms
└── AND: NOT in maintenance window
→ Only fires when BOTH error rate AND latency are degraded
→ Eliminates false positives from single-metric spikes
Alert Routing
CloudWatch Alarm → SNS Topic → Multiple targets:
├── PagerDuty/Opsgenie (critical: pages on-call)
├── Slack channel (warning: notification only)
├── Lambda (auto-remediation for known issues)
└── ITSM (ServiceNow incident creation)
Dashboards: What to Show
Per-Service Dashboard (Auto-Generated with Application Signals)
- Request rate (rpm)
- Error rate (%)
- Latency (p50, p90, p99)
- Dependency health (downstream services)
- Recent deployments (correlate changes with metrics)
Platform Dashboard (SRE/Platform Team)
- SLO status across all services (green/yellow/red)
- Error budget remaining per service
- Top 5 highest-latency services
- Recent alerts and resolution time
- Deployment frequency and failure rate
Cost Dashboard (FinOps)
- Compute spend trend (CloudWatch billing metric)
- Data transfer costs by service
- Lambda invocations/cost correlation
- Over-provisioned resources (Compute Optimizer findings)
Observability for Different Architectures
Serverless (Lambda + API Gateway + DynamoDB)
| Pillar | Tool | Setup |
|---|---|---|
| Metrics | CloudWatch (auto) + EMF for custom | Lambda Powertools library |
| Logs | CloudWatch Logs (auto) | Structured JSON, set retention |
| Traces | X-Ray Active Tracing (one toggle) | Zero code |
| SLOs | Application Signals | Auto-discovers Lambda services |
Containers (ECS/EKS + ALB + RDS)
| Pillar | Tool | Setup |
|---|---|---|
| Metrics | Container Insights + EMF | ADOT collector as sidecar/DaemonSet |
| Logs | CloudWatch (awslogs/Fluent Bit) | Structured JSON, per-container log groups |
| Traces | ADOT (OpenTelemetry) | Auto-instrumentation operator (EKS) |
| SLOs | Application Signals | Service map auto-discovery |
Hybrid (Containers + Lambda + Step Functions)
| Pillar | Tool | Setup |
|---|---|---|
| Metrics | Container Insights + Lambda metrics | Unified CloudWatch namespace |
| Logs | CloudWatch Logs (all services) | Common trace ID in all log entries |
| Traces | ADOT + X-Ray Active Tracing | Trace propagation across Lambda→ECS→SQS |
| SLOs | Application Signals | End-to-end service map |
Cost Optimization for Observability
Observability itself can be expensive. Control costs:
| Lever | Savings |
|---|---|
| Log retention policies (don't keep forever) | 50-80% on log storage |
| Infrequent Access log class | 50% for rarely-queried logs |
| EMF instead of PutMetricData API | Avoid per-metric API charges |
| Sampling traces (e.g., 10% of requests) | 90% trace storage savings |
| Metric filters instead of full log queries | Reduce Log Insights costs |
| Archive to S3 after 30 days | CloudWatch → S3 Glacier (95% cheaper) |
Summary
AWS observability in 2026 centers on three shifts:
- X-Ray SDK → OpenTelemetry (ADOT) — industry-standard instrumentation, multi-backend export, auto-instrumentation for zero code changes
- Threshold alarms → SLO-based alerting — Application Signals measures what users experience, not just infrastructure health
- Separate tools → unified correlation — trace IDs connect metrics, logs, and traces so you move from "something's wrong" to "here's the failing span" in seconds
The architecture: Instrument with ADOT (OTel), store in CloudWatch + X-Ray, analyze with Log Insights + Trace Analytics, alert on SLOs via Application Signals, visualize in CloudWatch Dashboards or Managed Grafana.
Start with auto-instrumentation and structured logs. Add custom metrics and SLOs as your services mature. Don't over-instrument day one — observability should grow with your system's complexity.
Alpesh Kumbhare is an AWS Architect at Atos, specializing in AWS observability, infrastructure automation, and cloud architecture. Connect on LinkedIn.
Top comments (0)