AWS & SRE Field Manual (Part 8): Amazon CloudWatch Architecture, Telemetry Pipelines & Production Observability

1. TL;DR & Problem Statement

• Definition: A native, serverless observability, telemetry, and monitoring suite within the AWS ecosystem that consolidates logs aggregation, time-series metrics collection, automated threshold alarms, and performance dashboards into a unified control plane.
• Problem Solved: Eliminates the operational overhead and single-points-of-failure associated with deploying and maintaining self-hosted monitoring stacks (e.g., standalone Prometheus for metrics, Loki/ELK for logs, Alertmanager for notifications) while providing native, out-of-the-box visibility into AWS managed services.
• Category: Management & Governance / Observability & Telemetry

---

2. Core Architecture & Key Components

                     ┌───────────────────────────────────────────────┐
                     │           AWS Infrastructure Resources        │
                     │   (EC2 Instances / EKS Pods / RDS / ALB)      │
                     └───────────────────────┬───────────────────────┘
                                             │
                                   Telemetry Pipeline
                                             │
        ┌────────────────────────────────────┼────────────────────────────────────┐
        ▼                                    ▼                                    ▼
┌─────────────────┐                  ┌─────────────────┐                  ┌─────────────────┐
│ CloudWatch Logs │                  │CloudWatch Metric│                  │ Custom App Data │
│  (Raw Streams)  │                  │ (System Stats)  │                  │ (Business KPIs) │
└────────┬────────┘                  └────────┬────────┘                  └────────┬────────┘
         │                                    │                                    │
         ▼                                    ▼                                    ▼
┌─────────────────┐                  ┌─────────────────┐                  ┌─────────────────┐
│ Logs Insights   │                  │CloudWatch Alarms│                  │   Dashboards    │
│  (SQL Querying) │                  │(SNS / AutoScale)│                  │ (Visual Graphs) │
└─────────────────┘                  └─────────────────┘                  └─────────────────┘

2.1. CloudWatch Logs (Log Groups & Streams)

• Ingests, processes, and stores raw application and system logs from EC2 (via CloudWatch Agent), Amazon EKS containers (via Fluent Bit / AWS Distro for OpenTelemetry), and native AWS audit/access trails (RDS logs, ALB access logs, Lambda execution logs).
• Log Groups & Streams: Log events reside inside distinct streams grouped hierarchically by application or microservice boundaries.

2.2. CloudWatch Logs Insights (Interactive Log Analytics)

• A purpose-built, high-speed distributed query engine that executes interactive, SQL-like queries to parse, aggregate, and troubleshoot petabyte-scale log streams without standing up Elasticsearch clusters.

2.3. CloudWatch Metrics (System vs. Custom Telemetry)

• Hypervisor-Level Metrics: AWS services emit out-of-the-box infrastructure metrics at 1-minute to 5-minute resolutions (e.g., EC2 CPU utilization, network packet counters, EBS volume operations).
• OS-Level Metrics: Memory utilization (RAM) and disk capacity are managed within the guest OS and require the unified CloudWatch Agent to be installed.
• Custom Metrics: Workloads can publish proprietary business domain metrics (e.g., CheckoutLatency, OrdersProcessedPerMinute) using the AWS SDK, CLI, or Embedded Metric Format.

2.4. CloudWatch Alarms & Composite Alarms

• Metric Alarms: Continuously evaluate metric data points across rolling statistical windows (p95, p99, average) against explicit thresholds or dynamic machine-learning anomaly detection bands.
• Composite Alarms: Combine multiple discrete alarms using boolean operators (AND, OR, NOT) to eliminate false-positive alert storms (e.g., "Trigger PagerDuty ONLY if CPU > 85% AND 5xx Error Rate > 5%").
• Action Handlers: Dispatch webhook events to Amazon SNS (routing alerts to Slack/PagerDuty), trigger EC2 Auto Scaling policies, or invoke AWS Systems Manager Automation runbooks.

---

3. Deep Dive Engineering & Advanced Patterns

Embedded Metric Format (EMF) & High Cardinality

Publishing custom metrics individually via the PutMetricData API incurs substantial costs at scale and introduces strict API throttling/rate-limiting risks.

• Mechanism: Applications output structured JSON logs containing a special _aws directive directly to stdout. CloudWatch Logs ingests the log stream, parses the EMF payload asynchronously, and extracts custom metrics in the background with zero PutMetricData API costs.

Container Insights with AWS Distro for OpenTelemetry (ADOT)

• Rather than deploying legacy vendor-locked agents, modern EKS clusters deploy the ADOT Collector (OpenTelemetry standard).
• ADOT collects distributed traces, container resource telemetry, and application metrics using vendor-neutral protocols (OTLP), exporting directly to CloudWatch, Amazon Managed Prometheus (AMP), or Jaeger.

Metric Math (Derived Expressions)

• Enables dynamic calculations across multiple raw metrics on dashboards and alarms without deploying intermediate data-processing microservices (e.g., calculating percentage ratios: (ErrorCount / TotalRequests) * 100).

---

4. Practical Notes & Configuration Snippets

CloudWatch Logs Insights: Aggregating API 5xx Errors by Endpoint

fields @timestamp, @message, status, path
| filter status >= 500
| stats count(*) as error_count by path
| sort error_count desc
| limit 20

Embedded Metric Format (EMF) Structured Log Output

{
  "_aws": {
    "Timestamp": 1718000000000,
    "CloudWatchMetrics": [
      {
        "Namespace": "Production/PaymentService",
        "Dimensions": [["Environment", "Operation"]],
        "Metrics": [
          {
            "Name": "ProcessingLatencyMs",
            "Unit": "Milliseconds"
          },
          {
            "Name": "SuccessfulTransactions",
            "Unit": "Count"
          }
        ]
      }
    ]
  },
  "Environment": "Production",
  "Operation": "StripeCheckout",
  "ProcessingLatencyMs": 84.5,
  "SuccessfulTransactions": 1,
  "RequestId": "req-9923847-acdf"
}

Terraform: Composite Alarm Definition

resource "aws_cloudwatch_composite_alarm" "high_severity_incident" {
  alarm_name        = "HighSeverity-PaymentService-Outage"
  alarm_description = "Fires only when high latency correlates with elevated 5xx error rate"

  alarm_rule = "ALARM(${aws_cloudwatch_metric_alarm.high_latency.alarm_name}) AND ALARM(${aws_cloudwatch_metric_alarm.elevated_5xx_errors.alarm_name})"

  alarm_actions = [aws_sns_topic.pagerduty_alerts.arn]
  ok_actions    = [aws_sns_topic.pagerduty_alerts.arn]
}

---

5. Gotchas & Common Pitfalls

• Default Log Retention (Never Expire): By default, CloudWatch Log Groups retain ingested logs indefinitely with no expiration date. Over months, unmanaged debug and access log groups accumulate silent, compounding storage costs. Always enforce an explicit retention policy (14–90 days) via Infrastructure-as-Code.
• PutMetricData API Cost & Throttling: Sending thousands of high-frequency custom metrics synchronously via PutMetricData can generate unexpectedly large AWS bills and trigger API rate-limiting errors under peak traffic. Always buffer metrics or migrate to Embedded Metric Format (EMF).
• Missing Standard RAM/Disk Metrics on EC2: Basic CloudWatch monitoring does not track guest operating system memory allocation or disk fill rates. Relying on default EC2 metrics for database or caching hosts will leave out-of-memory (OOM) conditions undetected.

---

6. Production Best Practices

• Subscription Filters & Cross-Account Streaming: Stream high-throughput log groups in real-time to Amazon Kinesis Data Firehose or AWS Lambda to forward logs into cold S3 Parquet data lakes or external SIEM platforms (e.g., Splunk, Datadog), minimizing expensive long-term CloudWatch storage fees.
• Log Anomaly Detection: Enable automated CloudWatch Log Anomaly Detection on critical log groups. The service uses machine learning to establish behavioral baselines and proactively flags unexpected error spikes or new unknown log formats without requiring static regex rules.
• High-Resolution Metrics (1-Second Interval): For latency-critical financial or real-time trading services, configure custom metrics with StorageResolution: 1 to publish sub-minute (1-second) metrics for immediate incident triage.
• CloudWatch Synthetics (Canaries): Deploy synthetic canary scripts (Node.js/Python headless Chromium) to continuously execute end-to-end user journeys (e.g., login, checkout flow) from global edge locations, verifying system availability even when real traffic is low.