DEV Community

Juan Manuel Ramallo
Juan Manuel Ramallo

Posted on

I was billed for 14k USD on Amazon Web ServicesÂ đŸ˜±


We may agree that we lie to ourselves once in a while. I remember thinking of:

I'll never put my code on a public repository since it's a freelance project maintained only by myself.

The truth is, anything can happen in life.


Month to month I use S3 from AWS to store photos and documents from several apps I got on production and also use Route 53 to route the domain with the app itself. My monthly bill is about 1 USD. Yep, that's almost nothing, but this month (april last year) I had to pay 14,267.86 USD (well at least that's what my bill says)

On april 13th (a sleepy Monday), it was like 10:00 in the morning when I got a happy email from Amazon Web Services giving me the welcome to EC2 services. By the time I received that email I neither know the existence of that service. That made me wonder if I had received that email by mistake or if my account 
 had been hacked.

Billing dashboard

Minutes later I wrote a message to AWS support asking about that email and they answered me very quickly and called me like five times in less than 48 hours. They told me that my account may had been compromised, and gave me some list of things to do in order to strengthen my account security and to avoid further EC2 service usage. I deleted all access keys I was using, added multi-factor authentication and changed my password. (I'm not endorsing or promoting AWS but I have to say that the guys from AWS support were extremely kind and useful in this case)

It all started in a boring weekend (at least this is what I believe). I was on the chat with a friend telling him about a side-project I had been working on. And suddenly I decided to put my code on github to show him off all the stuff. It was up, in the web, for like 10 minutes max, after I switched it to be hosted on gitlab to make it private. Two days later I received that email from AWS I told you before.


What happened?
A file containing my AWS credentials hadn't have been ignored in git, so when I pushed my local repository it all, even my credentials, went online in github (for couple of minutes, but they were there).

Suggestion
Please, store your credentials secretly ALWAYS. You can use environment variables for storing access keys and credentials that may compromise your accounts or bills. And never think of a local repository only, when everyone knows that nowadays the internet is all over around.

How this happened?
Since you can list all public repositories on github, I imagine of a job/task/process/program running constantly and cloning each project and looking for .yml files and keywords like "KEY" or "ACCESS_KEY" or something like that. This is only my thought of how could this happen. If it happen like so, it makes me sad of how people can be malicious and with no concerns of consequences of their acts.


TL;DR
Never leave your api keys on public repositories (always check before uploading)
Take a look at your email frequently and don't take a single email as a mistake
Be a hacker so you can track and catch the guys who stole your keys (well that's only a dream)

Hope you have a good day!
(If you're still wondering, no, I didn't have to pay for what I didn't use)

Top comments (90)

Collapse
 
yechielk profile image
Yechiel Kalmenson • Edited

Wow! Scary story!

To GitHub's credit, I know that if they detect that one of their own keys were uploaded to GitHub they'll notify you immediately and disable that key.

I know that because... err a friend... made that mistake once 😳

Collapse
 
ghost profile image
Ghost

Did AWS Support cancel the charge or refund you, or something? I've heard they're pretty understanding and often do stuff like that if you weren't responsible or it was accidental and there was no gain?

Collapse
 
juanmanuelramallo profile image
Juan Manuel Ramallo

Yeah they were so reasonable.. they issued a refund request for me so I didn't had to pay anything I didn't use. They called it "unauthorized usage". And the communication with the support team was very gently and agile

Collapse
 
ghost profile image
Ghost

Sounds like they really understand what happened - how awesome. The only other CS that I could think of that is that lit đŸ”„ would rather be Spotify (@SpotifyCares) or Slack (@SlackHQ).

Thread Thread
 
tarzan212 profile image
tarzan212

Netflix is pretty chill on payments issue as well. At least for me, they allowed me to watch and pay later when my credit card got compromised! Not related to the subject, but I felt that they should have their name mentionned here :)

Thread Thread
 
okolbay profile image
andrew

for them 10(20,30?) bucks of delayed payment is less than nothing - recommendation from happy customer - priceless )

Thread Thread
 
iambalajirk profile image
balaji radhakrishnan

Nice story :)
AWS knows before the hacker that your keys are compromised. They are running a similar script/program to check whether your keys are publicly exposed.

Thread Thread
 
xtofl profile image
xtofl

So you know that for a fact? Or are you guessing?

Thread Thread
 
danielcrabtree profile image
Daniel Crabtree

I've found Slack (@SlackHQ) support to be awesome too.

Thread Thread
 
iambalajirk profile image
balaji radhakrishnan

I have seen instances like this before where some guys would have exposed their repo publicly with the keys and immediately started receiving emails from AWS that their keys are compromised. I am guessing the fact they would have done it the same way the hackers are doing it.

Collapse
 
lalitakashyap profile image
Lalita Kashyap

Hey! I am a student and I have fallen in a similar problem and got a bill of 1.5k dollars and need urgent help. Can you please tell what did you write in your support message. Please help me.

Collapse
 
ben profile image
Ben Halpern

Wow........... A true horror story........

Collapse
 
juanmanuelramallo profile image
Juan Manuel Ramallo

Sure it was ☠

Collapse
 
ejeker profile image
Eric Jeker

Ahah, I just got billed an insane amount because I forgot to remove one table after a DynamoDB tutorial I followed. Just contacted the support. I hope I won't have to pay this.

I thought I was on a On-Demand pricing but I had reserved WCU. Still don't really understand how they came up to that amount.

Collapse
 
djviolin profile image
IstvĂĄn Lantos • Edited

For safety measure, create a Budget under Billing in your AWS web console with 2 factor authentication! This will set the maximum how much they can charge you monthly (or quarterly, yearly). More info about Budgets and it's types (be advised that some budget types only trigger notifications, not service halt): docs.aws.amazon.com/awsaccountbill...

This should be your first thing when you register on AWS.

Collapse
 
mkschueler profile image
Mike Schueler

budgets don't put a hardcap on your spend/usage.. it just sends out alerts based on thresholds you set.

you mention 'some budgets only trigger notification, not service halt' I'm not aware of any budget that halts service, please enlighten me

Collapse
 
johand profile image
Johan

That's why I also tend to ignore files in .gitignore_global so if I forget to do it from the .gitignore of the project I can avoid things like this.

Collapse
 
erebos-manannan profile image
Erebos ManannĂĄn

I just don't understand why it's such a big deal to actually look at what is being committed before doing it

Too many people do some sort of git add . && git commit -m "progress" && git push blind commits and it baffles me how they feel comfortable doing it.

Collapse
 
xngwng profile image
Xing Wang • Edited

few more ideas:

Never put your api keys in repos. period. (public or private).

Put them in environment variables but keep them completely separate from the source code repo. like development.env, production.env.

Always follow the practice of least privilege. Even most engineers at a company should only need
development.env variables if they are just writing code and not doing production support.

production.env should be guarded with only very few people that must have access to production system.

Collapse
 
kayis profile image
K

I had the impression KMS is the way to go here.

Put the encrypted keys in the repo and decrypt them before usage.

Collapse
 
mbuechmann profile image
Malte BĂŒchmann • Edited

I do not think that this is a good idea. You would put encrypted keys and the decryption algorithm in the repo. It is still possible to get to the keys.

Thread Thread
 
Sloan, the sloth mascot
Comment deleted
 
mbuechmann profile image
Malte BĂŒchmann • Edited

First of all: The named function hash_hmac does not encrypt. It creates a hash, which cannot be used to restore the original value. It is one way.

If you would use a proper encryption the logic is still faulty.

You have a secret, that cannot be added plainly to the repository. You add some decryption logic, encrypt the original secret and add it to the repository. The original secret is now safe. But now you have another secret (the private key needed for decryption) that cannot be added plainly to the repository.

You still have the same situation plus some extra decryption code, which has to be maintained. Also your build process has to handle the encryption.

Collapse
 
bgadrian profile image
Adrian B.G.

The worst nightmare of a cloud user coming true.

But my keys are in the .ssh home folder, protected, why would you put them in configs? What kind of awful deployment tools requires that?

Also, the first thing I do (and AWS recommends) is to setup billing alerts, at least you know that something bad is going on.

Collapse
 
databasesponge profile image
MetaDave đŸ‡ȘđŸ‡ș

Seconding the advice to set up billing alerts, and I would make sure it goes to a variety of email addresses in case one of them is compromised.

Collapse
 
numerxofficial profile image
Deak Kevin

also setup a Budget w/ 2fa

Collapse
 
empty2k12 profile image
Gero

Your account was not hacked. You sent out the password (yes, access keys are passwords) to the world. You should consider yourself lucky AWS refunded you the money, as technically you are responsible for those charges.

Collapse
 
juanmanuelramallo profile image
Juan Manuel Ramallo

lucky me I didn't had to pay for my mistake 😅 lesson learned anyways 😇

Collapse
 
theodesp profile image
Theofanis Despoudis

Maybe this will help next time:
github.com/awslabs/git-secrets

Collapse
 
joelebeau profile image
Joe LeBeau

Just in case no one else mentioned it, if you mistakenly commit a secret to a git repo, you need to make sure to get it out of the git history as well. help.github.com/articles/removing-... If you are curious if a git repo has had secrets accidentally committed to it in the past you can use tools like TruffleHog to search for them github.com/dxa4481/truffleHog

Collapse
 
kjir profile image
Stéphane Bisinger

You need to consider the secret compromised, and you hate to revoke that key. Cleaning up the history is then a lesser concern...

Collapse
 
paveltrufi profile image
Pavel Razgovorov

I experienced a similar story while I was doing a group assignment in college. They charged me 800€ for absolutely nothing. Hopefully I contacted the support centre and hopefully they retired all those charges

Collapse
 
miguel456 profile image
Miguel Nogueira

Same. I was only owing 300€ but it was scary
they were pretty fast in resolving the issue, hats off to AWS customer support

Collapse
 
miguel456 profile image
Miguel Nogueira

Once I was owing AWS 300$... For something I didn't use and had accidentally forgotten.
It was a forgotten Certificate Manager cert authority. Fortunately my account wasn't compromised!

And there I was, literally crying over 300$, when someone came to me and said, hey, call AWS support

And so I did, the story went smoothly, they were quick and really understanding and finally pardoned my debt... Lol!
I was really grateful that day, a guy from AWS support literally saved my ass. I'm glad you didn't have to pay the 14k and that all went well. I wouldn't know what do do in that case!

I wonder what the hackers were doing with EC2 to owe 14k right away... It must've been a gold pot for them hahahah

Collapse
 
kjir profile image
Stéphane Bisinger

Did the same mistake, pushed just before dinner, while eating I realized I pushed my secret key to a public repo and when I was finished my account was already compromised. I had to manually remove all the garbage, but in those 15-20 minutes I already got a $ 150 bill.
Lesson learned: security as the first thing, even for toy projects. Always.

Collapse
 
perttisoomann profile image
Pert Soomann • Edited

I remember story few years back when someone's blog or app went viral unexpectedly, so it auto-scaled up infrastructure to $5,000 worth of AWS bills. Downside of zero downtime :)

Collapse
 
defman profile image
Sergey Kislyakov

5000k or 5k?

Collapse
 
perttisoomann profile image
Pert Soomann

Ah yeah, it's 5,000 not 5,000K indeed. Good spot :)