I am sure you or the company you work for never want to see such a message on the website of the company. Unfortunately, for some companies, this worst nightmare has come to reality, and few out of them were almost on the verge of losing business until they recovered or negotiated with hackers.
One sad story
Ashley Madison, which got hacked on 15 July 2015 by a group called "The Impact Team" and threatened to expose users' identities, if its parent company, Avid Life Media, did not shut down Ashley Madison and its sister site, Established Men. Few users committed suicide as their highly personal data was made public on torrent. The story does not end here. It is 2020 now, and Ashley Madison users are still being blackmailed.
After being attacked depending on what a company does, consequences may vary, but one side effect is always there.
Any guesses?
Yes, You are correct. The company loses its customers' trust, which does not come in one day.
Enough of this sad story. Let us end it here and think about how you can prevent this from happening with your product. These days most of the companies are offering SAAS and are on Cloud because being on Cloud gives much flexibility. However, just like all good things come with some side effects, the Cloud is no exception. It has a side effect of inadvertently exposing Cloud resources to the public.
We already know how much damage a publically exposed resources can cause to a company, that is why every company has a security team that proactively keeps scanning for unauthorized access or resources that one accidentally let open to the world and bombards with a lot of JIRAs for explanations.
In my opinion, security is just perception, and no product is 100% secure. It is either hard or easy to hack. Now it depends upon how hard we can make our product to be hacked.
Coming to the main point that motivated me to write this blog, my very first blog on medium.com was AWS EBS. One day my manager paged me to find out all the EC2 instances with unencrypted EBS volume and to encrypt them as soon as possible without affecting any production service and without making any changes in EC2 private IP. While working on this task out of curiosity, I ended up Googling hacks that had happened because of unencrypted EBS, and results were scary. The case I highlighted above was one of them where people at AshleyMadison made two mistakes. First, they hardcode secrets in source code, and second, all this critical information was unencrypted. If EBS were encrypted, hackers would have had a hard time getting this data even if the snapshot of EBS was publicly accessible until a hacker gets access to its KMS keys.
Let me tell you one unencrypted publicly accessible snapshot can make you live your worst nightmare. Public Snapshot! Now, if you might think I am an Idiot, who does that? Right?
Wait. Do not judge too early read this report written on Ben Morris's findings, which he presented at DEF CON 27 in August 2019. Several major private companies and even federal agencies unknowingly exposing their sensitive data like admin passwords, application keys, and VPN configuration, which can be exploited to tunnel to their corporate network.
This nightmare is real
This nightmare is real. Most of the time, we are just one misconfiguration away from a potential hack. If you are still not convinced, read this interesting comment from "Hacker News" which enlighten a case when a user needs to share an EBS snapshot between two AWS accounts.
If someone exposes snapshot publically for just a couple of minutes, there are bots planted by Hackers in every region looking for such exposed snapshots and copy as soon as they see it. Hackers can create EBS volume out of it and attach to an EC2 instance to view its data and if there are any secrets or hardcoded API keys present in this then.
Now, consider a scenario where you have a use-case that requires you to share a snapshot with other accounts. While implementing this use-case, you realize that EBS volume contains confidential information and you deleted this sensitive information from EBS before creating a snapshot. SMART!
What if I tell you that others can still see the file you have deleted before creating a snapshot. They can still see your sensitive information. Scary! Isn't it?
Here is proof of what I have just said
I created an EC2 instance with Ubuntu AMI ami-0caae0b310f01ff33, which had an EBS volume of 8 GB. For the demonstration, I created a file production.json with some fake credentials under the /home/ubuntu/test_directory directory.
To prove the point, I deleted production.json and created a snapshot of this EBS volume. Let's say I accidentally shared it with unintended person or made it public.
To simulate hackers, I created an EBS volume from the snapshot snap-0f4a66cec80757, which I created after deleting production.json and attached this volume to another EC2 instance.
On the new EC2 instance, I ran lsblk command to see the device name of the new volume, which is nvme1n1 in this case, and mounted it to /data_from_snapshot directory. Once it is mounted /data_from_snapshot will appear in the output of df command.
Now we are almost set to steal victim's confidential data. All we need is a utility like extundelete or testdisk based on the underlying filesystem, which can recover deleted data.
I used extundelete as my test EC2 has ext4 filesystem. extundelete puts all recovered files in a directory called RECOVERED_FILES with a similar directory structure as of the original volume. Therefore RECOVERED_FILES directory is all that I need to examine to see deleted files.
In this example, we already know we deleted file production.json containing secrets under /home/ubuntu/test_directory directory, so we should check a similar directory structure inside the RECOVERED_FILES directory.
Voilà ! Now I can see the password and other secrets of the file production.json, which was not even present when the snapshot was created. This means information can leak from not just currently available files on EBS but also from old deleted files. Therefore never share a snapshot if your EBS ever had any confidential data.
Why can Hacker see it?
When we delete a file, it only gets unlinked. Delete command only breaks the link between name and inode and marks the inode as unused so that it can be used again. It does not wipe out data from filesystem blocks. If data blocks are not overwritten by new data, then it is possible to recover data based on the underlying filesystem. There are lots of details on how the operating system and filesystem works behind the scene when we delete a file and what all things it checks. I will discuss these things in detail in my future blogs.
What can we do to avoid such scenarios?
AWS has provided a straightforward solution, which is nothing but encrypted EBS volume, So even if you accidentally share an encrypted EBS snapshot with unintended user or AWS accounts, only users with appropriate permission on KMS key used for encryption can see what is inside it. Data is encrypted before it leaves the EC2 instance, which ensures security of data-at-rest as well as data-in-transit between an instance and its attached EBS storage.
I will share how you can automatically find and encrypt unencrypted-volumes without impacting your production services in my future blogs.
Impact on performance
As per AWS, encrypted EBS volume has the same IOPS performance as unencrypted EBS volume. However, encryption does add some overhead on I/O requests since data get encrypted before it leaves the EC2 instance. To minimize overhead on I/O latency, the EBS encryption feature is available on only a few instances type.
Final Note
Always use encrypted EBS volumes as encrypted volumes automatically create encrypted snapshots and never share a snapshot with sensitive data with anyone or whom you do not trust until you have any unavoidable use-case. Even if you have use-case, always think twice because you are just one mistake away from being hacked.
Additional Links —
inshorts.com
Top comments (0)