We may agree that we lie to ourselves once in a while. I remember thinking of:
I'll never put my code on a public repository since it's a freelance project maintained only by myself.
The truth is, anything can happen in life.
Month to month I use S3 from AWS to store photos and documents from several apps I got on production and also use Route 53 to route the domain with the app itself. My monthly bill is about 1 USD. Yep, that's almost nothing, but this month (april last year) I had to pay 14,267.86 USD (well at least that's what my bill says)
On april 13th (a sleepy Monday), it was like 10:00 in the morning when I got a happy email from Amazon Web Services giving me the welcome to EC2 services. By the time I received that email I neither know the existence of that service. That made me wonder if I had received that email by mistake or if my account … had been hacked.
Minutes later I wrote a message to AWS support asking about that email and they answered me very quickly and called me like five times in less than 48 hours. They told me that my account may had been compromised, and gave me some list of things to do in order to strengthen my account security and to avoid further EC2 service usage. I deleted all access keys I was using, added multi-factor authentication and changed my password. (I'm not endorsing or promoting AWS but I have to say that the guys from AWS support were extremely kind and useful in this case)
It all started in a boring weekend (at least this is what I believe). I was on the chat with a friend telling him about a side-project I had been working on. And suddenly I decided to put my code on github to show him off all the stuff. It was up, in the web, for like 10 minutes max, after I switched it to be hosted on gitlab to make it private. Two days later I received that email from AWS I told you before.
What happened?
A file containing my AWS credentials hadn't have been ignored in git, so when I pushed my local repository it all, even my credentials, went online in github (for couple of minutes, but they were there).
Suggestion
Please, store your credentials secretly ALWAYS. You can use environment variables for storing access keys and credentials that may compromise your accounts or bills. And never think of a local repository only, when everyone knows that nowadays the internet is all over around.
How this happened?
Since you can list all public repositories on github, I imagine of a job/task/process/program running constantly and cloning each project and looking for .yml files and keywords like "KEY" or "ACCESS_KEY" or something like that. This is only my thought of how could this happen. If it happen like so, it makes me sad of how people can be malicious and with no concerns of consequences of their acts.
TL;DR
Never leave your api keys on public repositories (always check before uploading)
Take a look at your email frequently and don't take a single email as a mistake
Be a hacker so you can track and catch the guys who stole your keys (well that's only a dream)
Hope you have a good day!
(If you're still wondering, no, I didn't have to pay for what I didn't use)
Oldest comments (83)
Wow........... A true horror story........
Sure it was ☠️
Did AWS Support cancel the charge or refund you, or something? I've heard they're pretty understanding and often do stuff like that if you weren't responsible or it was accidental and there was no gain?
Yeah they were so reasonable.. they issued a refund request for me so I didn't had to pay anything I didn't use. They called it "unauthorized usage". And the communication with the support team was very gently and agile
Sounds like they really understand what happened - how awesome. The only other CS that I could think of that is that lit 🔥 would rather be Spotify (@SpotifyCares) or Slack (@SlackHQ).
Netflix is pretty chill on payments issue as well. At least for me, they allowed me to watch and pay later when my credit card got compromised! Not related to the subject, but I felt that they should have their name mentionned here :)
for them 10(20,30?) bucks of delayed payment is less than nothing - recommendation from happy customer - priceless )
Nice story :)
AWS knows before the hacker that your keys are compromised. They are running a similar script/program to check whether your keys are publicly exposed.
So you know that for a fact? Or are you guessing?
I've found Slack (@SlackHQ) support to be awesome too.
I have seen instances like this before where some guys would have exposed their repo publicly with the keys and immediately started receiving emails from AWS that their keys are compromised. I am guessing the fact they would have done it the same way the hackers are doing it.
Hey! I am a student and I have fallen in a similar problem and got a bill of 1.5k dollars and need urgent help. Can you please tell what did you write in your support message. Please help me.
Nice experience with AWS. On the other hand GCP ;)
Appreciate the effort in sharing this with the community. I am preparing for an AWS cert right now and will be using AWS soon. Your post was helpful!
I remember story few years back when someone's blog or app went viral unexpectedly, so it auto-scaled up infrastructure to $5,000 worth of AWS bills. Downside of zero downtime :)
5000k or 5k?
Ah yeah, it's 5,000 not 5,000K indeed. Good spot :)
The worst nightmare of a cloud user coming true.
But my keys are in the .ssh home folder, protected, why would you put them in configs? What kind of awful deployment tools requires that?
Also, the first thing I do (and AWS recommends) is to setup billing alerts, at least you know that something bad is going on.
Seconding the advice to set up billing alerts, and I would make sure it goes to a variety of email addresses in case one of them is compromised.
also setup a Budget w/ 2fa
I had a bill for almost £5000, absolutely terrifying!
I experienced a similar story while I was doing a group assignment in college. They charged me 800€ for absolutely nothing. Hopefully I contacted the support centre and hopefully they retired all those charges
Same. I was only owing 300€ but it was scary
they were pretty fast in resolving the issue, hats off to AWS customer support
So did you finally had to pay anything, like some penalty?
Not even a dime, they considered it as "unauthorized usage" and refunded me all the extra💰
Though they filled me up with documentation on how to strengthen my account and correctly use and storage of secret keys.
That's really great support
Maybe this will help next time:
github.com/awslabs/git-secrets