Imagine this: you come home after a long day and find that your house is a complete mess. You have absolutely no clue what happened. Who created the mess in your house? How and when did they get access to your house? What did the intruder take? What did they displace/destroy in the house?
If you’re lucky, you may have a security system in place, e.g., cameras that recorded the incident and that you can use to trace back and identify how and when the destruction happened. But if not, then it becomes a complete nightmare to try to figure out what happened. And this is exactly what it feels like to try to troubleshoot an event in AWS without CloudTrail.
Think of CloudTrail as that indoor security camera that captures who came into your house, what they touched, changes, added, or even removed from the environment, and the exact day/time that all these activities occurred. So, just like reviewing the security footage will help you understand the break-in event, reviewing CloudTrail logs will provide the vital evidence you need to investigate and resolve any issues within your AWS environment.
The good news is that a lot of people know exactly WHAT CloudTrail is and what it is meant to be used for. The bad news is that not enough people know HOW to use CloudTrail to derive useful insights from the captured logs. So, let’s dive into how exactly you use AWS CloudTrail to Investigate Cloud Conundrums.
Some Basics
- CloudTrail is enabled by default for your account, which means you automatically have access to CloudTrail Event History.
- CloudTrail Event History provides an immutable record of events from the past 90 days. These are events captured from the Console, CLI, SDKs, and APIs.
- You are not charged for viewing CloudTrail Event History.
- CloudTrail events are regional.
Layout
Let’s start with the standard layout of the CloudTrail Console:
Note: For this piece, we are focusing on the Event History Section of CloudTrail on the Console.
Display Customization
The settings icon at the far right [3] allows you to customize the fields that are displayed. Options are as follows:
You can read about what each of the fields represents here.
Event Filtering
CloudTrail tracks every API call within your AWS account, resulting in a historical record that can grow significantly, even for small accounts with limited users. Of course, the more activity in the account, the larger the volume of events recorded. This can make investigating a singular event a nightmare, as you’d need to comb through hundreds of records to get to the record of interest.
This is where fields [1] and [2] come in. [1] provides a range of parameters you can use to filter through your events, as below:
For instance, let’s say you want to investigate who created a new user account and when the user was created, you can apply a filter as follows:
For the case above, we apply a filter based on the Event name and search specifically for CreateUser events.
The User name will typically provide the name of the IAM principal that performed the CreateUser action, i.e., an IAM user or a service. The Resource Name, on the other hand, is the actual AWS resource that the action was performed on, i.e., for the case above, this would be the name of the user that was created.
Another good example would be if you want to see what activities a specific user or service performed in the account in a given period, you can apply a filter such as below:
Note: The date/time filter supports both a relative and an absolute range.
Event Types
In my investigations, I found that I lean more towards filtering by Event name as I often find myself looking for a specific event from a specific service. The challenge I found with this, though, was in identifying what events are recorded for each service and what the event names are.
For starters, it is important to note that CloudTrail Event History only supports management events. Note that to capture data events, you must create a trail and explicitly add each resource type for which you want to collect data plane activity. Second, AWS provides a comprehensive list of API Actions for each service through the API Reference documentation.
The API Reference documentation for a service provides descriptions, API request parameters, and the XML response for the service’s API actions. For example, from the EC2 API Reference documentation, we see the following example EC2 API Actions:
So now, from the CloudTrail Event History console, I can easily investigate an event such as when an Elastic IP Address was created using the below filter:
Note: to view the API reference for an AWS Service, simply search for API Reference on your favorite browser😊
Event Sources
This filter comes in handy if you want to view a general record of all activity performed on a specific AWS service, e.g., S3.
If you select the Event source filter on the CloudTrail history console, you can view a list of all the service names available in AWS that you can filter by.
Note that the event name takes the form of . amazonaws.com so simply type the service in your search, and the full service name is autocompleted for you.
Examples
Now let’s have a look at a few real-world use cases and how you’d typically use CloudTrail to investigate.
Scenario 1: An administrator receives a notification for a sudden spike in failed login attempts for a critical IAM user account.
Troubleshooting:
The above scenario is a ConsoleLogin Event. What we’d want to determine here is if this is a legitimate login attempt from the user or if it could potentially be a brute force attack on the account. So essentially, we are looking for details such as:
Source IP Address: Multiple login attempts originating from a single, unexpected IP address can be a red flag.
- User Agent: this is information relating to the device used for the login attempt e.g., browser and OS. This could reveal inconsistencies compared to the expected login patterns for the user (hint: check previous successful login attempts to identify the usual patterns)
- Timestamp: A rapid succession of failed login attempts within a short timeframe is a strong indicator of a brute-force attack.
- Number of failed login attempts: A high number of failed login attempts within a short period is a strong indicator of brute-force attempts.
To get the above information, we can proceed as follows:
i. Filter
We can apply a filter as follows:
ii. View Details
From the list, select the event of interest. For this case, we’d identify this event using the User name field.
Once you click on the event, you get access to a more detailed event record as follows:
From here, you can pick out the relevant details and determine if this is a security event.
Scenario 2: An application deployed on EC2 instances suddenly experiences slow loading times and high error rates. You suspect a recent configuration change might be to blame.
Troubleshooting:
i. Identify the timeframe: Narrow down when the application issues began. We need to apply a time filter for this period.
ii. Filter Events: Depending on your architecture, you want to check for a couple of things, e.g.,
- Configuration changes made to the instance itself i.e., instance resources, could reveal why the slow loading times occur.
- Configuration changes made to the instance’s networking, e.g., changes to security group, route table, NACLs,s and any applicable policies.
- Configuration changes made to the other service that the application is communicating with, e.g., if the application is picking/putting data into an S3 bucket, it could be that the instance profile was changed.
Event filters can then be applied as follows:
Check for configuration changes made to the security group.
Check for configuration changes made to the EC2 instance.
Check for configuration changes made to the EC2 instance role.
Limitations
While the CloudTrail console offers a convenient way to investigate AWS events, its filtering capabilities are currently limited to a single field at a time. This can be restrictive when you need to refine your search based on multiple criteria.
For more comprehensive filtering, you can:
- View history via the AWS CLI
- Download your CloudTrail events as a CSV file and leverage the powerful filtering and analysis features of tools like Excel.
- Save logs to an S3 bucket and use an Athena table for filtering.
Stop Guessing, Start Tracking: Enable CloudTrail today and gain visibility into your AWS activity.
Top comments (1)
nice post