Introducing Today's Project!
In this project, I am building an AWS Security Monitoring System to demonstrate how cloud-native tools can be configured to detect, alert, and respond to threats. My goal is to show how services like CloudWatch, GuardDuty, and Security Hub can strengthen visibility and resilience in a cloud environment. I am also doing this project to demonstrate hands-on experience in cloud security, reflecting on my strengths and weaknesses, and investing in my personal growth.
Tools and concepts
Services I used were AWS Secrets Manager, CloudTrail, CloudWatch Logs, CloudWatch Metrics & Alarms, and SNS. Key concepts I learnt include how CloudTrail records API activity for auditing, how CloudWatch Logs and Metrics enable monitoring and alerting, the role of thresholds in alarms to trigger notifications, the importance of SNS topics and confirmed subscriptions for reliable message delivery, and how these services integrate to create a security monitoring pipeline that detects, logs, and alerts on secret access events.
Project reflection
This project took me approximately two hours. The most challenging part was tuning the CloudWatch alarm threshold setting it to 1 didn’t reliably detect a single secret access, so I adjusted it to 0.01 to trigger as expected. It was most rewarding to see the full pipeline work end‑to‑end: CloudTrail logged the access, CloudWatch recorded the metric and fired the alarm, and SNS delivered the email notification.
Create a Secret
Secrets Manager is an AWS cloud service that securely stores and manages sensitive data like passwords, API keys, and database credentials, enabling applications to access them safely at runtime with encryption, access control, and automatic rotation instead of hardcoding secrets in code.
To set up for my project, I created a secret called TopSecretInfo that contains the the key and value needed by my AWS security monitoring system.
Set Up CloudTrail
CloudTrail is an AWS service that records and monitors all API calls and account activity, providing visibility into actions taken by users, roles, and services for auditing and security purposes. I set up a trail to capture these events and deliver them to a designated S3 bucket, ensuring that logs are securely stored, easily accessible for analysis, and available for compliance or incident investigation.
CloudTrail events include management events that log control‑plane actions like creating users or modifying security groups, data events that capture resource‑level activity such as reading or writing S3 objects or invoking Lambda functions, network activity events that track API calls through VPC endpoints, and insights events that detect anomalies or unusual patterns, together providing full visibility into AWS account activity for auditing and security monitoring.
Read vs Write Activity
Read API activity involves retrieving or viewing information without making changes, such as listing users, describing resources, or getting object data, while Write API activity involves creating, modifying, or deleting resources, such as launching instances, updating configurations, or deleting objects. For this project, we need to capture both Read and Write activities to ensure complete visibility into account actions, since monitoring only one type would leave gaps in detecting potential security issues.
Verifying CloudTrail
I retrieved the secret in two ways: first through the AWS Management Console, where I opened the secret directly to view its stored values, and second using the AWS CLI, where I ran a command to programmatically fetch the secret, ensuring both manual and automated access methods were tested for logging and monitoring.
To analyze my CloudTrail events, I visited the CloudTrail console and reviewed the event history; I found entries showing both the retrieval of my secret from Secrets Manager and the corresponding API activity recorded with details like the user identity, time, and access method; this tells me that CloudTrail is successfully capturing and logging sensitive operations, providing the visibility and accountability needed for auditing and security monitoring.
CloudWatch Metrics
CloudWatch Logs is an AWS service that collects, stores, and manages log data from resources and applications, making it possible to monitor activity in real time. It’s important for monitoring because it centralizes logs, enables the creation of metrics and alarms, and provides visibility into system behavior, which helps detect issues, track security events, and ensure compliance.
CloudTrail’s Event History is useful for quickly viewing recent account activity and auditing specific API calls, while CloudWatch Logs are better for continuous monitoring and analysis because they store detailed log data, allow you to create metrics and alarms, and provide real‑time visibility into patterns or anomalies across your AWS environment.
A CloudWatch metric is a measurement that tracks specific activity or performance in your AWS environment. When setting up a metric, the metric value represents the actual data point being recorded, such as the number of times a secret was accessed. The default value is used when no data is reported for a given period, ensuring the metric still has a baseline reading so alerts or dashboards remain consistent even if no events occur.
CloudWatch Alarm
A CloudWatch alarm is a monitoring tool that watches a specific metric and triggers an action when the metric crosses a defined threshold. I set my CloudWatch alarm threshold to 1 on the SecretIsAccessed metric so the alarm will trigger whenever a secret is accessed at least once, ensuring I immediately receive a notification through SNS whenever sensitive data is retrieved.
I created an SNS topic along the way. An SNS topic is a communication channel that groups multiple subscribers together so messages can be delivered to all of them at once. My SNS topic is set up to send notifications whenever the CloudWatch alarm detects secret access, and since I subscribed my email address to it, I’ll immediately receive an alert in my inbox whenever the alarm is triggered.
AWS requires email confirmation for SNS subscriptions because it ensures that the recipient explicitly agrees to receive notifications, preventing accidental or unauthorized subscriptions. This helps prevent spam, protects users from unwanted messages, and guarantees that alerts only go to verified and consenting email addresses.
Troubleshooting Notification Errors
To test my monitoring system, I deliberately retrieved the secret value again to trigger the CloudWatch alarm. The results were mixed, while the event was successfully logged in CloudTrail, I didn’t receive the expected notification in my inbox. This showed that the alarm was firing but the SNS subscription or email delivery wasn’t configured correctly. The test highlighted the importance of end-to-end validation, ensuring not just that events are captured but also that alerts reach the intended recipients.
When troubleshooting the notification issues I checked five areas: first, I verified that the CloudWatch alarm was actually firing when the secret was retrieved. Second, I confirmed that the alarm was correctly linked to the SNS topic. Third, I checked that my email subscription to the SNS topic was active and confirmed. Fourth, I reviewed IAM permissions to ensure CloudWatch and SNS could publish and deliver messages. Finally, I inspected spam filters and delivery settings to rule out blocked or misrouted emails.
I initially didn't receive an email because the alarm used Average (and a long period), so a single 1 was diluted or treated as missing; metric/dimension mismatch or an unconfirmed SNS subscription can also block alerts.
Key solution: switch the alarm to Sum or Maximum, use a short period (e.g., 60s) with EvaluationPeriods=1, verify metric/dimensions, set missing‑data handling, and confirm the SNS email subscription.
Success!
To validate my monitoring system, I checked the CloudWatch alarm tied to the SecretIsAccessed metric after retrieving a secret, and I confirmed that the alarm was triggered. I then verified that the SNS topic delivered an email notification to my subscribed address. I received the alert exactly as expected, which tells me that the monitoring pipeline from CloudTrail logging to CloudWatch metrics and alarms, through to SNS notifications is working correctly and will reliably inform me of secret access events.
Comparing CloudWatch with CloudTrail Notifications
I updated my CloudTrail configurations because I needed to capture additional management events across all regions, ensure logs were encrypted with KMS keys, and integrate with CloudWatch for real-time alerts. This allowed me to enhance security monitoring, meet compliance standards, and reduce blind spots in activity tracking.
After enabling CloudTrail SNS notifications, my inbox quickly filled with hundreds of event emails per day. In terms of usefulness, I thought the notifications were initially helpful for confirming that logging was working, but the sheer volume made them impractical for daily monitoring. Instead, I realized it was better to route these events into CloudWatch or a centralized logging system, where I could filter, aggregate, and set alerts for only the truly critical activities.
Key Takeaways
- End-to-End Validation is Crucial: Building a security pipeline isn't just about turning services on. Testing the flow from the initial event (accessing the secret) all the way to the final alert (receiving the email) is the only way to guarantee your system works when a real threat occurs.
- Signal vs. Noise matters: Raw CloudTrail notifications can quickly overwhelm your inbox with hundreds of events. Routing these logs through CloudWatch allows you to filter, aggregate, and set alarms only for critical activity, saving you from alert fatigue.
- Metrics Require Tuning: Default configurations don't always catch single events. Changing metric math from Average to Sum or Maximum, shortening the evaluation period, and properly handling missing data are essential steps to ensure single occurrences (like reading a secret) trigger an immediate alarm.
- Complete Visibility Requires Both Read and Write Logging: Monitoring only "Write" activities leaves massive security gaps. Capturing "Read" events is equally critical, especially when protecting sensitive data like passwords and API keys stored in Secrets Manager.
- Security is a Pipeline: No single AWS tool provides complete security on its own. It requires a layered integration: CloudTrail for auditing, CloudWatch for monitoring and alerting, and SNS for reliable delivery to the right people.
🤝This is the end of the series on Security.
Next series will be Three-Tier Architecture
Top comments (0)