Note: This content was originally published at the Simple AWS newsletter.
Imagine this scenario: You're careful with security, and you set up Multi-Factor Authentication for your AWS IAM or IAM Identity Center user. At one point, a malicious agent of evilness figures out your password, either via phishing, keylogging, or any other technique. The only thing standing between them and $50k in bitcoin mined on your AWS account (and the $500k AWS bill paid with your credit card) is your MFA device. They don't have access it (yet!), so you're safe (for now!).
The correct response is obvious: Change your password, so they'll be back to square one. The problem with this is that most of the times you don't know that a password has become known until someone uses it. This is the reason why we should rotate passwords regularly!
Let me propose an extra layer of security: Get notified every time a login attempt fails because of a failed MFA check.
Understanding CloudTrail event logs
AWS CloudTrail is a service that logs all requests performed against AWS in your account. This includes all actions and requests (even unauthorized ones) done through the Console, CLI, AWS SDKs, and APIs.
By default (already enabled when you create your account) and for free, CloudTrail offers you a viewable, searchable, downloadable, and immutable record of all events that happened in the past 90 days, called Event History.
You can also create Trails, which let you export to S3 or CloudWatch Logs all or a subset of CloudTrail events. This way, you can use other AWS services like Athena and OpenSearch, or external tools like ElasticSearch, to analyze CloudTrail events. Event history is per region, while Trails can be created for all regions in an account, and even for all accounts in an Organization.
Events are JSON objects, which contain all the information of what was attempted, when, by whom, some other details on the request, what was the result, and sometimes some other details on the response, such as the reason.
For example, this is what a ConsoleLogin event that fails looks like (I hid a few details replacing them with HIDDEN):
{
"eventVersion": "1.08",
"userIdentity": {
"type": "IAMUser",
"principalId": "HIDDEN",
"accountId": "HIDDEN",
"accessKeyId": "HIDDEN",
"userName": "HIDDEN"
},
"eventTime": "2023-08-24T21:07:08Z",
"eventSource": "signin.amazonaws.com",
"eventName": "ConsoleLogin",
"awsRegion": "us-east-1",
"sourceIPAddress": "HIDDEN",
"userAgent": "HIDDEN",
"errorMessage": "Failed authentication",
"requestParameters": null,
"responseElements": {
"ConsoleLogin": "Failure"
},
"additionalEventData": {
"LoginTo": "https://console.aws.amazon.com/console/home?HIDDEN",
"MobileVersion": "No",
"MFAUsed": "Yes"
},
"eventID": "HIDDEN",
"readOnly": false,
"eventType": "AwsConsoleSignIn",
"managementEvent": true,
"recipientAccountId": "HIDDEN",
"eventCategory": "Management",
"tlsDetails": {
"tlsVersion": "TLSv1.3",
"cipherSuite": "TLS_AES_128_GCM_SHA256",
"clientProvidedHostHeader": "signin.aws.amazon.com"
}
}
How are IAM and IAM Identity Center events logged in CloudTrail?
Naturally, CloudTrail logs events when an IAM or IAM IC user logs in to AWS. However, it's not just one event. Let me show you what happens behind the scenes with IAM IC and IAM.
CloudTrail Events for IAM Identity Center users logging in
These are the events that CloudTrail logs when an IAM or IAM IC user logs in:
CredentialChallenge: AWS requested some form of credential, such as password or MFA device. Each of these is followed by UserAuthentication and one CredentialVerification event, and this sequence of three is repeated until either all necessary credentials are provided, or CredentialVerification fails.
UserAuthentication: AWS receives the requested credentials.
CredentialVerification: AWS checks whether the received credentials are valid. If they are, this event contains:
"serviceEventDetails":{ "CredentialChallenge":"Success" }, and the process continues either by requesting the next credential or by authenticating. If the credentials received are invalid, this event contains"serviceEventDetails":{ "CredentialChallenge":"Failure" }and the process stops. The type of credential requested can be found in"additionalEventData":{ "CredentialType":"PASSWORD" }, the value of which can bePASSWORDfor the regular password,TOTPfor MFA devices that produce temporary codes,WEBAUTHNfor web apps using WebAuthn,EXTERNAL_IDPfor external identity providers, orRESYNC_TOTPto re-synchronize TOTP devices.Authenticate: Once CredentialVerification succeeds and no more credentials are required, this event is logged. This means the user successfully authenticated to IAM Identity Center. If you're authenticating with an external Identity Provider such as Google Workspaces, Microsoft AD or Okta, this is the only event you'll see.
Federate: This event means the IAM IC user assumed an IAM role in an AWS account.
ConsoleLogin: This event means the IAM IC user logged in to the AWS Console using the assumed IAM role.
The first four events will be logged on the AWS account and region where IAM IC is configured. The last two are logged in the AWS account where the user signs in, in the default region for that user (i.e. the one that's selected when the user signs in).
CloudTrail Events for IAM users logging in
IAM is much simpler. The only event is ConsoleLogin, and this is how you can identify what happened:
If the user signed in successfully, the event contains
"responseElements": { "ConsoleLogin": "Success" }If sign in failed, the event contains
"responseElements": { "ConsoleLogin": "Failure" }and can contain"errorMessage": "Failed authentication"If the user used MFA (regardless of success or failure), the event contains
"additionalEventData": { "MFAUsed": "Yes" }If the user is the root user, the event contains
"userIdentity": { "type": "Root" }
All of these events are always logged in the us-east-1 region.
Identifying when signing in fails due to MFA
The last section was a bit detailed, but it contains all the information we need to identify when a login attempt fails the MFA check!
For IAM IC, we need to find CredentialVerification events which contain either "additionalEventData":{ "CredentialType":"TOTP" } or "additionalEventData":{ "CredentialType":"WEBAUTHN" }, and contain "serviceEventDetails":{ "CredentialChallenge":"Failure" }.
For IAM, we need to find ConsoleLogin events which contain "responseElements": { "ConsoleLogin": "Failure" } and "additionalEventData": { "MFAUsed": "Yes" }.
You can do this manually in the CloudTrail event history, but if we want to automate it and get notifications, we need to send it to CloudWatch Logs.
Sending CloudTrail events to CloudWatch Logs
Sign in to the AWS Console. For IAM IC, sign in to the root account of the Organization
Go to CloudTrail
Click Create trail
For Trail name, enter management-events
If you're configuring this for an organization, check Enable for all accounts in my organization
Select Create new S3 bucket and under Trail log bucket and folder enter a name, or leave it as default
Under AWS KMS alias, enter a name for a KMS key to encrypt the logs
Under CloudWatch Logs, check Enabled
Under Log group name, enter a name for the CloudWatch Logs log group
Under Role name, enter a name for the IAM Role that'll let CloudTrail put logs to CloudWatch Logs
Click Next
Leave these options as default and click Next again
Click Create trail
Wait until the Status column changes to Logging (green)
Configuring Alerts and Notifications for CloudWatch Logs
First, let's create a filter to view the failed logins in CloudWatch Logs. Follow the steps, and on Step 6 choose the right filter depending on whether you're using IAM or IAM IC.
Filtering failed login attempts due to MFA
Open the CloudWatch console
In the panel on the left, under Logs, click Log groups.
Click on the name of the log group that you created for the trail
Click the Metrics filters tab, and click the button Create metric filter
-
Under Filter pattern, enter the pattern that corresponds to IAM or IAM IC, depending on which you're using:
- For IAM:
{ ($.eventName = ConsoleLogin) && ($.additionalEventData.MFAUsed = "Yes") && ($.responseElements.ConsoleLogin = "Failure") } - For IAM IC:
{ ($.eventName = CredentialVerification) && (($.additionalEventData.CredentialType = "TOTP") || ($.additionalEventData.CredentialType = "WEBAUTHN")) && ($.serviceEventDetails.CredentialChallenge = "Failure")}
- For IAM:
Click Next
For Filter name, enter SignInFailedMFA
Under Metric namespace, enter CloudTrailMetrics
For Metric name, enter SigninMFAFailureCount
For Metric value, enter 1.
Click Next
Click Create metric filter
Configuring alerts for failed login attempts due to MFA
On the Metric filters tab, find the metric filter you just created, select it and click Create alarm
Under Whenever SigninMFAFailureCount is..., select Greater/Equal
Under than…, enter 1
Click Next
Under Send a notification to the following SNS topic, select Create new topic
Under Create a new topic…, enter SignInFailedMFAAlarm
Under Email endpoints that will receive the notification…, enter your email address
Click the Create topic button that's right below where you just entered your email address
Click Next
Under Alarm name, enter SignInFailedMFAAlarm
Click Next
Click Create alarm
Open your email inbox, open the email titled AWS Notification - Subscription Confirmation, and click Confirm subscription
Testing Alerts When Signing in to AWS Fails Due to Incorrect MFA Code
To test it, you'll need to attempt to sign in, and enter the incorrect MFA code. There's going to be a delay of a few minutes between when the event actually occurs and when you receive the notification. CloudTrail usually takes an average of 5 minutes to export the event logs to CloudWatch Logs, and the alarm could take a couple more minutes to fire.
You can also monitor the alarm in CloudWatch Alarms:
---
Stop copying cloud solutions, start understanding them. Join over 3600 devs, tech leads, and experts learning how to architect cloud solutions, not pass exams, with the Simple AWS newsletter.
* Real scenarios and solutions
* The why behind the solutions
* Best practices to improve them
If you'd like to know more about me, you can find me on LinkedIn or at www.guilleojeda.com
Top comments (1)
Ich muss es mir nochmal nachher anhören ...aber trotzdem vielen dank schon mal ;-)