A Very Expensive Lesson
I received a huge surprise this month, a forecast for a July month end bill of $586 linked to my AWS root account. As best practice I never use my AWS root account and always build projects with the AWS IAM account login.
The AWS Management Console provides a summary of the AWS services usage, forecasted costs, in particular indicating that Amazon Sagemaker had an end of month $578.57 forecast with cost increasing by over 300% in the current month. I have not used Amazon Sagemaker recently 🙄 and this raised concerns for my unexpected charges.
Amazon Cost Explorer
By navigating to Amazon Cost Explorer, AWS Cost Management provides a summary view of costs, commentary on the right hand side and also a graph outlining daily usage had increased by 270% in the region US-East-1 (N.Virginia).
I navigated each of the panes to investigate further information regarding the costs incurred such as Region, Instance Type and Linked Account.
Instance Type
I checked under Instance Type to see if there were any active EC2 instances. There were no active instances after 11 July 2022.
Usage Type
In this pane, I could identify that after 11 July 2022, someone had created an Amazon Sagemaker Canvas session from 11-17 July (indicated in purple)
Linked Account
I used the AWS root account login details to check any active sessions of Amazon Sagemaker Canvas.
Reduce Costs - Clean up Active AWS resources
The previous week on the 11 July, I cleaned up my AWS account for active resources such as:
a) Deleting EC2 instances in the regions Sydney, Ohio and N.Virginia
b) Shutting down any active Cloud9 sessions
c) Deleting any active Cloud Formation stacks
d) Deleting any AWS Sagemaker processing jobs
e) Emptying S3 buckets for completed AWS Glue jobs
Reduce Costs - Create an AWS Cost Anomaly Detection Report
As an extra measure I created a Cost Anomaly Report that could be emailed to me to identify any suspicious activity to my AWS account over a threshold of $15. You may create a Cost Anomaly Detection Report from this link
AWS Cost Anomaly Detection Report uses machine learning to detect anomalies and the root causes for cloud spend.
You may also download as a csv file the Cost Anomaly Detection Report to view the details of the surprise charges:
Reduce Costs - Delete if not in use - Amazon Sagemaker Canvas
Navigate to Amazon Sagemaker Canvas and click on Canvas and then click Getting Started. I located active sessions of an unauthorized user 'michael-c' who has been generating workloads $40 per day from 13-18 July on my account. I followed the instructions to remediate and prevent unauthorized user from this link
I deleted the app from Amazon Sagemaker Canvas and then deleted the user.
From the diagram you may see an unauthorized user 'michael-c'.
AWS CloudWatch
I further explored AWS CloudWatch to confirm the usage levels of Amazon Sagemaker Canvas used by user 'michael-c' in US East - region (N.Virginia) to understand how many days I was being charged from the Canvas instance.
AWS CloudWatch was able to monitor the metrics of an AWS Service and provide further evidence for my investigation of the surprise charges.
Block Public Access to all Amazon S3 buckets
I wanted to review my Amazon S3 buckets to see if I could further reduce my monthly bill. This is what I discovered:
a) User 'michael-c' had created an S3 bucket with my AWS account number and included a prefix with region US- East-1 (N.Virginia)
I emptied the files within the bucket.
b) This S3 bucket was Public and not private. I updated the settings to Block Public S3 buckets.
Create MFA on Root Account and Admin IAM User Account
To further secure my linked AWS Root account I also implemented Multi-Factor Authentication (MFA) as recommended on the AWS IAM dashboard. Please this blog and follow all the steps to enable MFA on all AWS accounts.
Best practice guidelines to enable MFA include downloading Twilio Authy for an iPhone or Android phone and link it to your mobile device. You may find the steps here.
MFA provides an additional layer of security to identify the user during the login process of your AWS account.
Final Steps - Change passwords
As an extra measure to prevent unauthorized access to my AWS account, I also reset the passwords to the AWS IAM and Root user accounts.
I recommend that you also change your email passwords regularly and enable 2-factor authentication if possible.
I hope you won't have to learn the hard way like me. Until next time, happy learning! 😁
Join us for AWS re:Inforce conference 2022
Next week is AWS re:Inforce conference, 26-27 July 📆
A learning conference on compliance, privacy and identity 🔐🛠️
• Register to watch the keynote & sessions streamed live online 📺
• Link: https://reinforce.awsevents.com
Top comments (2)
Hi wendy,
nice article about identifying and remediating aws costs. I usually also use cloudtrail logs for tracking any suspicious activity within my aws accounts.
Hi Adit, Thanks for the tip. I will also implement Cloudtrail and AWS Budgets :)