The message got my attention:
Client: "My AWS training account was hacked and I got a big bill!"
Me: "How big?"
Client: "7k in 3 hours and it keeps going - help!"
This story is a cautionary tale, it's not the first time I've helped a client recover from bad practices, it's just the most recent. Also a quick search, revealed this situation happens more often than you might realize - hacker news thread
First Steps
This novice (student) user had set up an AWS account 'for training purposes' and had nearly forgotten about it. Our student had set up a budget alert, which triggered due to the account activity, but they 'didn't catch it at first.'
Subsequent email from AWS saying 'unusual activity' and 'please login and confirm' is what brought our student to me. AWS also advised them to change the root user account password, deactivate the account key and to add MFA authentication immediately.
As I see all too often, for convenience our user accessed this 'study' account using a weak password and logging in as the root user.
Resolution
After 10+ messages back and forth with AWS (and over a month) later, working together we were able to properly secure the account and get AWS to remove all of the fraudulent changes. The total amount of fraud accrued over a 24 hours period? 21k.
Avoid Trouble
Because this is just the latest in what I have seen happen many times over the years, I decided to write up a series of best practices around protecting demo/training AWS accounts. Feel free to contribute your suggestions and tips at the end of this article as well.
Safe Account Steps
Here's the high-level list of core account safety steps
- Use a dedicated AWS account
- Monitor the account
- Create IAM users securely
- Secure and do not login with the root account
Next, I'll drill down on how to implement each of these steps:
To implement Step 1 - Dedicated Account...
If possible, create a dedicated AWS account solely for the purpose of demo or training. Do not use this account for any production work. Assign an alias to the login URL (using the IAM console), this masks the AWS account number when non-root IAM users login in.
To implement Step 2 - Monitor the account...
Set up a billing alert for usage on your demo AWS account. I suggest $ 50 USD / day. Send this alert to an email that you check regularly. Respond promptly to any billing alert notifications and/or AWS alert emails.
To implement Step 3 - Create IAM Users...
Create one or more IAM users. If using the console assign a strong password. If using API access, assign keys. Change passwords and/or update keys regularly. Do not check keys into source control. Do not share passwords. Assign MFA to each each IAM user (I use the Duo mobile app for this, but there are many options). Add IAM users to custom groups with appropriate permissions (see groups in next paragraph). Do not assign account admin permissions to the working IAM user accounts.
Create groups and assign only the required permissions, i.e. if you are testing a service, for example Amazon Timestream, then assign create (or admin) permission ONLY to the Timestream service. Add IAM users to custom groups to grant them appropriate permissions. You can use the IAM access advisor to verify that you've granted all needed permissions, example screen shown below.
Step 4 - Secure Root Account...
Assign a strong password to the IAM root account. Update the password regularly. Associate MFA with the root account. Do NOT login with the root account. Follow AWS best practices for securing the root user account - link
Shown below is a screenshot from the AWS IAM console, note that the status of the root account is shown on the first page using check marks (green is good).
Respond to Trouble
If your billing alert fires or if AWS contacts you, respond immediately. One of the reasons our student had such trouble, was that they took 24 hours to respond. This placed an additional burden of proof on them as they hadn't deleted the running services (in this case an EKS cluster in EVERY AWS region) and associated triggering lambdas, ECS images, IAM roles and initial probe EC2 instance. This instance launched the clusters using a Terraform template.
To be fair, the student was a novice cloud user, and didn't actually know HOW to locate all of the running services in order to delete all of the instances. However, had our student stopped the attack at the single EC2 instance launch point, it would've saved both the student and AWS a whole bunch of hassle.
Bottom Line
Fraudulent cloud activity happens all too often. I liken the result of it to having left a stove's gas burner on indefinitely - if you are going to use this method of cooking, make sure you understand what you need to do to keep everyone safe!
As mentioned earlier, have any of you had this experience? Anything to add? Share in comments.
Top comments (3)
Good tips for people doing a professional usage, ready to pay bills, set controls and alerts, and secure account. For students, I have only one recommendation: never open an AWS account. This is sad, but the only safe way. It would be easy for AWS to allow to set limits. And provide a real free tier where paid service need explicit enable. But as long as Amazon prefers making money on student's learning errors, I'll never recommend it. Exception is free labs with no credit card (for example I use AWS Academy with students - limited labs but no risks)
fair point and useful tip - thanks for sharing!
Buy a Yubikey and enable MFA.
Rotate your aws cli credentials frequently (or better, use sessions) aws.amazon.com/premiumsupport/know...