AWS Observability & Automation — CloudWatch, CloudTrail & EventBridge

📋 Topics Covered

#	Topic	Type
1	The Big Picture — Why Three Different Services	Concept
2	CloudWatch — Monitoring & Architecture	Concept
3	CloudWatch Metrics — Namespace, Dimensions, Statistic, Period	Concept + Cert
4	High-Resolution Metrics	Concept
5	CloudWatch Logs — Groups, Streams, Events, Retention	Concept
6	CloudWatch Logs Insights	Concept + DevOps
7	CloudWatch Dashboards	Concept
8	CloudWatch Alarms — States, Evaluation, Actions	Concept + Cert
9	CloudWatch Logs → S3 Export	Concept + DevOps
10	Multi-Account and Multi-Region Monitoring	Concept + DevOps
11	EC2 Instance Recovery	Concept + Cert
12	CloudTrail — Management Events	Concept + Interview
13	CloudTrail — Data Events	Concept + Interview
14	CloudTrail Insights	Concept
15	Network Activity Events vs VPC Flow Logs	Concept + Interview
16	EventBridge — Event Bus & Rules	Concept + Cert
17	EventBridge Scheduler	Concept
18	CloudWatch vs CloudTrail vs EventBridge	Interview
19	How the Services Work Together	Concept + DevOps
20	Production Mental Model	DevOps
21	Practice Tasks 1–5	Practice

---

The Big Picture

AWS environments continuously generate operational and audit data. The challenge isn't collecting it — it's turning it into something useful before it turns into an incident:

Collect → Monitor → Store → Analyze → Act

Each of the three services in this post answers a different question, and mixing them up is one of the most common gaps in AWS troubleshooting interviews and real incident response alike:

• CloudWatch → How is the system behaving?
• CloudTrail → What happened in the AWS account, and who did it?
• EventBridge → An event happened — what should happen next?

They're rarely used in isolation. A real incident usually touches all three: CloudWatch tells you something's wrong, CloudTrail tells you what changed, and EventBridge is what automatically reacts the next time it happens.

---

Amazon CloudWatch

What Is Monitoring?

Scenario: It's 2 AM. Checkout requests on an e-commerce site start failing. Nobody manually noticed — an alarm did, and paged the on-call engineer with the exact metric that crossed its threshold, five minutes before customers started tweeting about it.

That's what monitoring is for: continuously observing systems and applications so abnormal behavior gets detected, investigated, and acted on — ideally before a human notices the hard way. Common signals worth watching include EC2 CPU utilization and network traffic, RDS connections, SQS queue depth, Lambda errors, application error rates, and request latency.

The operational loop behind this: a problem occurs → telemetry is collected → CloudWatch ingests it → the anomaly is detected and investigated → an alarm, automation, or remediation responds. The production goal is never "watch a dashboard all day" — it's catching meaningful changes early and responding to them, ideally with as little human involvement as the situation allows.

CloudWatch Architecture

                    DATA SOURCES
                         │
        ┌────────────────┼─────────────────┐
        │                │                 │
   AWS Resources    Applications      Multiple
   EC2/ECS/RDS/     & Services        Accounts/Regions
   Lambda/S3/etc.
        │                │                 │
        └────────────────┼─────────────────┘
                         ▼
                  AMAZON CLOUDWATCH
                         │
          ┌──────────────┼──────────────┐
          ▼              ▼              ▼
       Metrics          Logs          Alarms
          │              │              │
          ▼              ▼              ▼
     Dashboards     Logs Insights     Actions
                                     / Integrations
                                          │
                    ┌─────────────────────┼─────────────────┐
                    ▼                     ▼                 ▼
                   SNS                 Lambda          Auto Scaling
                    │
                    ▼
               Notifications

A useful mental model to keep in mind for everything that follows: metrics measure behavior, logs describe events and details, alarms detect a condition, and actions respond to that condition.

CloudWatch Metrics

Scenario: Your monitoring dashboard shows CPUUtilization climbing from 8% at 10:00, to 11% at 10:01, to 37% at 10:02, to 82% at 10:03 — a metric is exactly that: a numerical measurement observed over time, and that climb is what your alarm will eventually key off of.

Common AWS metrics worth knowing by heart: EC2's CPUUtilization, NetworkIn/NetworkOut, and disk I/O; SQS's ApproximateNumberOfMessagesVisible; Lambda's Invocations, Errors, and Duration; and RDS's CPUUtilization and DatabaseConnections.

Every metric is identified by a combination of parts, and understanding each one is what separates "I set up a metric" from "I understand what I'm actually alarming on":

• Namespace — groups related metrics, e.g. AWS/EC2, AWS/SQS, AWS/Lambda, AWS/RDS
• Metric name — what's being measured, e.g. CPUUtilization, Errors, Duration
• Dimensions — identify the specific resource, e.g. InstanceId=i-0123456789
• Statistic — how CloudWatch summarizes the raw values: Average, Minimum, Maximum, Sum, SampleCount
• Period — the time interval per datapoint, e.g. 1 minute, 5 minutes, 1 hour

Put together, a complete metric reference looks like: namespace AWS/EC2, metric CPUUtilization, dimension InstanceId=i-0123456789.

The period choice genuinely changes alarm behavior — CPU > 80% for 1 minute will fire far more readily than CPU > 80% for 30 minutes. Alarm design should match the failure condition that actually matters to the business, not just "make the number go away."

High-Resolution Metrics

CloudWatch also supports high-resolution custom metrics for workloads that need finer-grained monitoring than the standard 1-minute granularity. The production decision here isn't "higher resolution is always better" — it's "use the smallest monitoring interval that's actually useful for the problem." Higher resolution helps with fast-changing systems, but it increases telemetry volume and cost, so it should be a deliberate choice, not a default.

CloudWatch Logs

Scenario: A metric shows Lambda Errors = 37 — useful, but it doesn't tell you why. The corresponding log line does: ERROR Database connection timeout, requestId=..., service=order-service. That's the core distinction — a metric tells you what level or how many, a log tells you what exactly happened. For real troubleshooting, you need both.

CloudWatch Logs is organized hierarchically:

/aws/lambda/order-service
    │
    ├── Stream A
    │     ├── Event
    │     ├── Event
    │     └── Event
    │
    └── Stream B
          ├── Event
          └── Event

A Log Group is a logical collection of related logs (e.g. /aws/lambda/order-service). A Log Stream is a sequence of log events from a particular source or execution context within that group. A Log Event is a single entry, like 2026-09-04 10:31:12 ERROR Database timeout.

Log retention shouldn't default to "forever." A production logging strategy decides how long each log group should be kept, weighing troubleshooting needs, compliance, cost, security requirements, and audit requirements — and operational logs often warrant a different retention period than security/audit logs.

CloudWatch Logs Insights

Scenario: An incident just happened, and someone asks "how many AccessDenied errors did we see in the last hour, and from which service?" Scrolling through raw logs isn't an option at scale — this is exactly what Logs Insights is for: querying and aggregating large volumes of log data to turn raw logs into searchable operational evidence.

fields @timestamp, @message
| filter @message like /ERROR/
| sort @timestamp desc
| limit 50

Typical questions it answers well: which requests generated 500 errors, how many AccessDenied errors happened, which service is producing the most failures, and what happened during a specific incident window.

CloudWatch Dashboards

A dashboard provides a centralized operational view. A production application dashboard might track request rate, latency, 5xx errors, EC2 CPU, RDS connections, SQS queue depth, and Lambda errors — but the goal should be decision-making, not visualizing everything you possibly can. A genuinely useful dashboard answers one clear question: can users currently use the application normally?

CloudWatch Alarms

Scenario: CPUUtilization > 80% is set as the alarm condition. At 30% CPU, the alarm sits in OK. At 92% CPU, it flips to ALARM. If there isn't yet enough usable data — a brand-new alarm, or a metric that hasn't produced enough datapoints — it shows INSUFFICIENT_DATA instead of guessing.

The basic flow: a metric is evaluated against a condition → the alarm enters a state → that state triggers an action.

Alarm evaluation settings matter more than they look. Behavior depends on the metric, statistic, period, threshold, evaluation periods, and datapoints-to-alarm — because production alerting should avoid turning every temporary spike into an unnecessary page. CPU > 80% for 1 datapoint is far more trigger-happy than CPU > 80% for several consecutive evaluation periods. Good alerting detects meaningful failure conditions, not just unusual-looking numbers.

Alarms and actions are two separate concerns. A CloudWatch alarm can trigger SNS notifications, Auto Scaling actions, EC2 recovery, or other supported integrations — for example, CPU > threshold → CloudWatch Alarm → SNS → Email/other subscribers. The alarm makes the decision; SNS (or whatever the target is) handles distribution. Keeping that distinction clear avoids a lot of confused debugging later ("the alarm fired but nobody got notified" is almost always an SNS/target problem, not an alarm problem).

CloudWatch Logs → S3 Export

CloudWatch Logs is built for active operations and investigation; S3 is built for durable long-term storage and archival. A logging design should intentionally decide what stays "hot" in CloudWatch versus what gets exported to S3 (CloudWatch Logs → export/archive → Amazon S3) — common reasons include long-term retention, compliance, archival, external analytics, and central data storage.

Multi-Account and Multi-Region Monitoring

Scenario: Your organization runs Production, Development, Staging, and Security in separate AWS accounts, spread across two Regions. Asking an operator to individually check each account during an incident doesn't scale.

Account A ─────┐
Account B ─────┤
Account C ─────┼──→ Central Monitoring Account
Region 1 ──────┤
Region 2 ──────┘
                         │
                         ▼
                 Dashboards / Alarms

A centralized monitoring approach provides visibility from a single monitoring account instead — this is the basis of centralized observability at organizational scale.

EC2 Instance Recovery

EC2 can be configured to recover automatically from certain underlying system/host failures using a CloudWatch alarm — commonly based on StatusCheckFailed_System, which flows StatusCheckFailed_System → CloudWatch Alarm → EC2 Recovery.

The important distinction: instance recovery targets underlying host/system problems, not arbitrary application failures. An underlying host issue is exactly what recovery is for — but Apache crashing, or an application bug, is a completely different failure mode that instance recovery does nothing to fix.

---

AWS CloudTrail

What Is CloudTrail?

Scenario: Someone deletes an S3 bucket that shouldn't have been deletable. CloudWatch won't tell you who did it — that's not what it's for. CloudTrail is: it answers what API call happened, who performed it, when it happened, which resource was involved, and what request context was recorded.

The cleanest way to keep CloudWatch and CloudTrail straight: CloudWatch asks "how is the system behaving?" CloudTrail asks "what activity happened in AWS?" CloudTrail is therefore heavily used for auditing, security investigation, change tracking, incident investigation, governance, and compliance.

CloudTrail Management Events

Management events cover control-plane operations that change or inspect AWS resources — things like CreateBucket, RunInstances, CreateRole, AttachRolePolicy, CreateSubnet, or DeleteSecurityGroup. Mental model: management events are mainly about AWS resource and configuration activity. If the question is "who changed this security configuration?", CloudTrail's management events are the natural place to look.

CloudTrail Data Events

Data events focus on activity against supported resources' actual data — for S3, that means things like GetObject, PutObject, and DeleteObject. The key distinction: a management event asks "what happened to the AWS resource/configuration?" while a data event asks "what happened to the underlying data?" Data events can be high-volume and are configured separately from management events for exactly that reason.

CloudTrail Insights

Scenario: Your S3 bucket normally sees about 10 DeleteObject calls a minute. Suddenly it's 2,000 a minute. That kind of deviation from a learned baseline is exactly what CloudTrail Insights is built to surface. Mental model: CloudTrail records activity; CloudTrail Insights detects unusual behavior within that activity.

Network Activity Events vs VPC Flow Logs

CloudTrail also supports network activity event logging for supported AWS network-related operations — but don't confuse this with VPC Flow Logs. They are not the same thing: CloudTrail covers service/account activity, while Flow Logs cover network traffic metadata. If you need to know "who called this API," that's CloudTrail. If you need to know "what packets moved between these two hosts," that's Flow Logs.

---

Amazon EventBridge

What Is EventBridge?

Scenario: An EC2 instance terminates, and instead of someone noticing it's gone twenty minutes later, an automated notification hits the operations team's Slack the moment it happens — no polling, no cron job checking instance state every few minutes. That's the value EventBridge adds: it's primarily an event routing and automation service.

Its basic model: an event arrives on an event bus, a rule examines the event and decides whether it matches, and if it does, the event is routed to a target. Events can originate from AWS services, your own applications, custom applications, or supported SaaS integrations.

Event Bus

                          Event Bus
                    ┌────────┼────────┐
                    ▼        ▼        ▼
                 Rule A   Rule B   Rule C
                    │        │        │
                    ▼        ▼        ▼
                 Lambda     SNS      SQS

The event bus is where events land and get routed. This is what keeps producers and consumers loosely coupled — the service emitting the event doesn't need to know or care what eventually consumes it.

EventBridge Rules

A rule means: when an event matches this pattern, take this action. For example, a rule that watches for EC2 terminations:

{
  "source": ["aws.ec2"],
  "detail-type": ["EC2 Instance State-change Notification"],
  "detail": {
    "state": ["terminated"]
  }
}

The resulting flow: EC2 terminated → EventBridge → rule matches → SNS / Lambda / SQS. This is a genuinely production-relevant event-driven automation pattern — the exact one behind the "someone gets notified automatically" scenario above.

EventBridge Scheduler

Scheduler answers a fundamentally different question than a rule does: not "when an event happens, react," but "when should something happen?" Instead of waiting on an event trigger (Event happens → EventBridge Rule), you schedule the action directly (2:00 AM every day → EventBridge Scheduler → Lambda). Common uses: running nightly cleanup, generating a morning report, invoking a Lambda periodically, or triggering a one-time task at a specific moment.

The key distinction to keep straight: event-driven means something happens, then react; time-driven means a scheduled time arrives, then execute.

---

CloudWatch vs CloudTrail vs EventBridge

Service	Primary Question	Typical Use
CloudWatch	How is my system behaving?	Metrics, logs, alarms, dashboards
CloudTrail	What AWS activity happened?	API auditing, investigation, governance
EventBridge	What should happen after an event?	Routing, automation, event-driven workflows

A handful of quick scenario-to-service mappings worth having memorized:

• EC2 CPU is too high → CloudWatch
• Someone deleted an S3 object → CloudTrail
• EC2 termination should notify an operations team → EventBridge → SNS
• CPU breach should send an email → CloudWatch Alarm → SNS → Email
• Every night at 2 AM, run cleanup → EventBridge Scheduler → Lambda

---

How the Services Work Together

Real production systems rarely use just one of these — they're stitched together:

                  Application
                      │
             ┌────────┼─────────┐
             ▼        ▼         ▼
          Metrics    Logs    API Activity
             │        │         │
             ▼        ▼         ▼
         CloudWatch CloudWatch CloudTrail
             │        │
             ▼        ▼
           Alarm   Logs Insights
             │
             ▼
            SNS


CloudTrail / AWS event
          ↓
      EventBridge
          ↓
     Lambda / SNS / SQS

This creates a continuous operational loop: observe → detect → investigate → decide → act.

---

Production Mental Model

Scenario: An incident just started. Instead of randomly clicking through the AWS Console, work through these five questions in order:

1. What is the system doing? → CloudWatch metrics/logs
2. What exactly happened in the AWS environment? → CloudTrail
3. What should automatically happen because of an event? → EventBridge
4. How should people or systems be notified? → SNS / other targets
5. What remediation should occur? → Auto Scaling / Lambda / Systems Manager / recovery actions

That maps to a practical investigation flow:

                Incident
                   │
        ┌──────────┼──────────┐
        ▼          ▼          ▼
   CloudWatch   CloudTrail  EventBridge
   telemetry     audit       automation
        │          │          │
        └──────────┼──────────┘
                   ▼
             Investigation
                   │
                   ▼
                Action

---

Key Takeaways

• CloudWatch is the core operational monitoring and observability service: metrics, logs, dashboards, alarms, and related actions.
• Metrics show numerical system behavior; logs provide detailed event-level context.
• Log Groups → Log Streams → Log Events is the basic CloudWatch Logs hierarchy.
• Logs Insights is for querying and aggregating large volumes of log data.
• Alarms evaluate metric conditions and can trigger actions such as SNS notifications, scaling, or recovery.
• CloudTrail records AWS account/API activity and is central to auditing and investigation.
• Management Events focus on control-plane activity; Data Events capture supported data-plane activity such as S3 object operations.
• CloudTrail Insights helps detect unusual API activity against established baselines.
• EventBridge routes events to targets and enables event-driven automation.
• EventBridge Scheduler is for time-driven execution.
• Production observability often combines all three services rather than choosing only one.

Remember the three questions:

CloudWatch — How is it behaving?

CloudTrail — What happened?

EventBridge — What should happen next?

---

Practice Tasks

Practice Task 1 — Custom Metric, Alarm & Notification

Publish a custom CloudWatch metric from an application (or a simple script) representing something business-relevant — e.g. failed login attempts or order-processing time — and configure an alarm on it that notifies an SNS topic when the threshold is breached. Pick a period and evaluation-periods combination deliberately, and be able to justify why it won't fire on a harmless single-datapoint spike.

Practice Task 2 — Investigate an Incident with Logs Insights

Given a Lambda function producing a realistic mix of successful and failing invocations, write Logs Insights queries to answer: how many errors occurred in the last hour, which error message is most frequent, and what the request pattern looked like in the five minutes immediately before the first failure. Produce a short summary of what the queries reveal, as if handing it to an incident commander.

Practice Task 3 — Trace a Change with CloudTrail

Make a deliberate, traceable change in your account — modify a security group rule, delete an S3 object, or alter an IAM policy — then use CloudTrail to answer who made the change, exactly when, from what source IP, and using which API call. Separately, identify whether the change would show up as a management event or a data event, and explain why.

Practice Task 4 — Event-Driven Automation with EventBridge

Build an EventBridge rule that reacts to a real AWS event (e.g. an EC2 instance state change or an S3 object creation) and routes it to a target of your choice (SNS, Lambda, or SQS). Trigger the underlying event for real, and confirm the target actually received and processed it — not just that the rule matched in the console.

Practice Task 5 — Full-Stack Incident Response Scenario

Simulate a small production incident end to end: use CloudWatch to detect a threshold breach (e.g. elevated errors or CPU), use CloudTrail to confirm whether any recent account change correlates with the timing of the incident, and use EventBridge to automatically trigger a remediation action (e.g. invoking a Lambda function or notifying a team) the moment the CloudWatch alarm changes state. Document the full chain — metric → alarm → event → automation — as a single incident timeline.