DEV Community: Cloudanix Inc

My Experience Contributing To The Cartography Open Source Project

Kedar Ghule — Tue, 04 May 2021 10:08:12 +0000

Open-source contributions are a great way to help the tech community. They are also one of the best ways out there to learn and build your skills. Open-source software are publicly available source codes of a product that you can use and modify for free. Chromium, Mozilla Firefox, the Linux Operating system are some of the most popular open-source software.

Recently, we at Cloudanix made open source contributions to the Cartography project by Lyft. We contributed to the pre-existing AWS module by adding support for KMS and API gateway. Since the Cartography project did not have support for Microsoft Azure, we started from scratch to add that capability to Cartography. We added support for Azure services like CosmosDB, SQL, Storage, and Virtual Machines.

After 6 pull requests and dozens of review comments, here is what we thought the benefits of contributing to open source are and what we learnt.

Benefits Of Contributing to Open Source

Improves Coding Skills: From a personal point of view, I can most definitely say that my Python coding skills have improved drastically after working on the Cartography project. I have got the habit of writing docstrings for every function. I have learned to write valuable comments in the code wherever necessary, adhering to the PEP8 style and unit testing.
Exposed to new trends and treasures in technology: Personally, before contributing to Cartography, I had no idea tools like that existed. Furthermore, I had never worked on anything related to AWS or Azure. After contributing to Cartography, I have started getting quite comfortable with these tools and more, like Terraform.
Networking and Peer Recognition: Working on an open-source project provides recognition. Contributing to the same project often makes you recognizable in the community. This is very important in terms of your career and your learning curve.
In-depth knowledge of the project: You can make contributions like opening issues, adding fixes, adding features, and updating documentation. This will give you an in-depth knowledge of the project you are working on. So, suppose in the future, people ask questions about some functionality of that project on StackOverflow. In that case, you can be their go-to person.
Improving software on a user and business level: I will talk thoroughly about this in the "Why We Chose The Above Modules to Contribute?" section.
Valuable experience in collaboration and teamwork: Contributing to open source is a good crash course to collaboration in technology. You are working on a code written by someone else and trying to contribute to it. In this process, you will end up communicating with the original contributors of that project. Knowing what their expectations are, understanding the project's roadmaps, fixing open issues, and then communicating over Slack/Discord is an excellent step towards embedding a sense of teamwork in yourself.

How Did We Approach Contributing to Lyft's Cartography?

So before we contributed to Lyft, we knew our contribution would be adding AWS support for KMS and API Gateway and adding Azure support for CosmosDB, SQL, Storage, and Virtual Machines. However, before we began writing the code for this, there were some steps that we took.

Learning about Source Control: The first step to any open source contribution is to learn about version control systems. The majority of the open-source projects use a version control system. Version control systems are tools that help with merging new code into the project or the main repository. The Cartography repository is hosted on GitHub. So, in this case, we had to get a good command over GitHub. This included knowing operations like -
1. Forking the project,
2. Making commits,
3. Mastering markdown,
4. Committing your code changes, and
5. Opening pull requests.
Assessing My Skills: Before moving ahead with the contribution, we read the Lyft Cartography project's code. This exercise was done to explore the skills we would need directly related to writing the code. The project needs you to have a strong command over the Python programming language as the entire codebase is in Python. Since we would be adding modules related to AWS and Azure cloud services, we needed a basic understanding of these cloud services. Furthermore, we had to get familiar with Python's AWS SDK documentation, known as Boto3, and Azure documentation for CosmosDB, SQL, Storage, and Virtual Machines.
Understanding the existing code: Next, it was time for us to explore the existing codes. We used the AWS intel module codes as references to build the KMS and API Gateway modules. We preferred not to stray too far from the already existing code style and workflow so that all the AWS module codes are streamlined in a particular manner. This approach helped when we added the Azure module and support for Azure services like CosmosDB, SQL, Storage, and Virtual Machines.
Writing the documentation: Contributing to open source doesn't just involve adding code. In our case, we were adding support for entire modules of the cloud services to the project. Hence we had to be comfortable with markdown and writing good and efficient documentation.

Why Did We Choose The Above Modules to Contribute?

Initially, we came across the Cartography project as users. While Cartography supported most of the services we needed, we needed Cartography's functionalities for AWS KMS and API Gateway and resources of Microsoft Azure's CosmosDB, SQL, Storage, and Virtual Machines. This is one of the key benefits of open-source. If we want to add new functionality as per our need to an already existing project, we can do so. Since Cartography is an open-source project, we get to contribute features that meet our business needs. At the same time, Cartography gets those features that other users can use. We assessed our business needs and concluded that we would like feature support for the services I mentioned earlier. Hence, we went ahead and contributed that to the Cartography project!

What Did I Learn?

For me, the PRs I opened for Cartography were my first open-source contributions. The code reviews helped me tremendously to be a better developer.

Writing Clean and Readable Code: My college assignments would always end up being spaghetti. It would not be very readable and barely clean. One of the things I had to adhere to while writing code for Cartography was the PEP8 style. Removing unnecessary spaces, adding trailing commas, not having more than 120 characters in one line of code, and so on. And while I started adhering to it, I could also see the benefits of it. It actually made my code readable - even for me. I found it easier to navigate through the code and be more efficient at writing and debugging the code in general. Another practice followed for the Cartography codes was writing type hints for every function. This is again one of the methods to make your code understandable.
Writing Efficient Code: In the Cartography code, I would gather the resources in one list and then iterate over them and ingest them into neo4j. Little did I know this was not the efficient way to do it. I got a code review by Alex Chantavy saying that I should re-write those parts of the code by using UNWIND for the list of resources inside the Cypher queries instead of looping through them. He further explained to me the reason behind this - "UNWIND makes it a lot faster and helps with potential memory issues." Another feedback from Ramon Petgrave was using list comprehension rather than a list(map(...)). I had used list(map(...)) for the entirety of the code. And this is a big thing - for small codes, you don't care much about efficiency, but for big projects like Cartography, the code's efficiency is vital. I was not very well aware that list comprehension is a faster alternative to list(map(...)). This is one of the benefits of open source as well - you get constructive feedback on your style of code from the best in the industry, and you learn new things at the same time.
Handling Exceptions: Handling exceptions was something I was familiar with and a practice that I did follow while writing my Cartography codes. However, for bigger projects, I had to be specific about what the exception was. Handling exceptions in big projects like Cartography is very important. Ramon Petgrave's review of my Azure code made me realize the importance of being specific during exception handling.
Testing: Lastly, writing tests for the code I wrote was a big task. I am not so familiar with testing. In college, we barely practiced test-driven development. However, I realized it is a necessity to do that in the real world. Writing integration tests for the code I wrote was a good way to assure myself that I am not breaking anything in the already existing codebase.

Conclusion

Lastly, I learned a lot while contributing to the Cartography open source project. Open source contribution really does give you a feeling of satisfaction, and I highly recommend you to contribute to open-source projects.

A huge thank you to the entire team at Cloudanix for providing us this opportunity to contribute to open-source. Thanks are in order to my mentor - Purusottom Mupunu, who contributed to this project with me by laying an initial foundation and then guiding me with my PRs and the thousands of questions I would bug him with. We also want to extend our sincere gratitude to Alex Chantavy and Ramon Petgrave. Their code reviews helped us become better developers than we were before we contributed to Cartography.

Note: This post was originally written by me and was published on Medium on the Cloudanix page.

What is GDPR Compliance?

Kedar Ghule — Thu, 08 Apr 2021 06:07:13 +0000

On the 25th of May 2018, there was a significant reform in Europe's digital privacy laws. After much preparation, the GDPR was enforced. The reform has since enabled Europe to be 'fit' for this digital age. The GDPR was adopted in 2016, and a two-year window was given to businesses to be GDPR compliant. The law replaces outdated data protection laws from the 1990s. So, you must be having a question. What is GDPR?

In this blog post, I am covering the below topics:

What is GDPR?
Why does GDPR exist?
What kind of data does the GDPR protect?
Key Definitions
Seven Principles of GDPR
Rights of an Individual according to the GDPR
Which companies does the GDPR affect?
What are the penalties?
How can Cloudanix help?

What is GDPR?

The GDPR can be considered as the world's most robust law on data protection. The General Data Protection Regulation (GDPR) 2016/679 is legislation that specifies the regulation on data privacy and protection in the European Union (EU) and the European Economic Area (EEA). The provisions of the GDPR are consistent across all 28 EU member states, which means every organization around the world which has a business in the European Union or handles data of EU residents should be GDPR compliant.

The European Union says that the GDPR was designed to "harmonize" data privacy laws across all its member states and provide greater protection and rights to individuals. GDPR was also created to alter how businesses and other organizations can handle the information of their users. The law also slaps large fines and can cause significant reputational damage for those found in breach of the rules.

Why does GDPR exist?

The short answer to that question is the rising public concern over privacy. In general, Europe has had a history of strict rules on how companies can use EU residents' data. Before the digital age set in, the Data Protection Directive came into effect in the EU in 1995. The GDPR replaces this and builds on the outdated law to deal with technology's progress in recent decades. With the progress in technology, more data was produced, and there was a rising skepticism among people on how it was used and who had access to it. This skepticism only keeps rising after every high-profile data breach.

According to the RSA Data Privacy & Security Report, for which RSA surveyed 7,500 consumers in France, Germany, Italy, the UK, and the US, 80% of consumers said lost banking, and financial data is a top concern. Furthermore, and specifically, lost security information (e.g., passwords) and identity information (e.g., passports or driving license) was cited as a significant concern among 76% of the respondents. What is even more important is that 62% of people mentioned that they would blame the company for a breach and not the hacker, while 72% of respondents from the US said that they would boycott a company in such a case. These statistics are enough for the GDPR to exist.

What kind of data does the GDPR protect?

At the heart of GDPR is personal data privacy and protection. To be very specific, this is the information that allows a living person to be directly or indirectly identified from any available data. It can be obvious as a person's name or location data or something less apparent like an IP address, RFID tags, and cookie identifiers. This can be considered as personal data.

There are also provisions in the GDPR for a few special categories of sensitive personal data given greater protection. This personal data includes information about racial and ethnic origin, political opinions, religious beliefs, biometric data, health information, sexual orientation data, and trade unions membership.

Personal data is crucial under GDPR because it covers individuals, organizations, and companies that are either 'controllers' or 'processors.' I will cover these terms next.

Key Definitions

The GDPR applies to 'controllers' and 'processors.' Their roles and responsibilities are specified in Chapter 4 of the GDPR, from Articles 24 to 43.

Controller: A controller is responsible for determining the purposes and means of processing personal data. They are the primary decision-makers who exercise overall control and are responsible for processing personal data. You are a controller if you do any or all of the following:
- Collect or process personal data.
- Decide what personal data should be collected and how it should be processed.
- Decide which individuals to collect personal data about.
- Obtain a commercial gain or other benefits from the processing, except for any payment for another controller's services.
- Process the personal data because of a contract between you and the data subject.
- Data subjects are your employees.
- Have complete independence as to how the personal data is processed.
Processor: A processor is responsible for processing personal data on behalf of a controller. Processors act either on behalf of or on the instructions of the relevant controller. You are a processor if you do any or all of the following:
- Are following instructions from someone else regarding the processing of personal data.
- Were given the personal data by a customer or a third party or told what data to collect.
- Do not decide to collect personal data or what personal data should be collected from individuals.
- Do not decide the lawful basis for the use of that data.
- Do not decide what purpose or purposes the data will be used for.
- Do not decide whether to disclose the data or to whom.
- Do not know how long you will retain the data.
- Are not interested in the result of the data.
- Make some decisions on how data is processed but implement these decisions under a contract with someone else.

A processor has some specific legal obligations as per the GDPR. For example, processors should maintain records of personal data and processing activities. They will have legal liability if you are responsible for a breach.

Controllers have stricter obligations under GDPR than processors.

Seven Principles of GDPR

Lawfulness, fairness, and transparency: Lawfulness includes identifying an appropriate lawful basis (or bases) for data processing. In case a business is processing special category data or criminal offense data, there are specific terms and conditions. Furthermore, nothing unlawful should be done with personal data. Fairness includes because how the processing may affect the individuals concerned. The business should handle people's data in ways they would reasonably expect or should be able to explain any unexpected processing. They should also not mislead people when they collect their data. Transparency ensures that the business is open and honest and complies with the transparency obligations of the right to be informed
Purpose limitation: Businesses should identify their purposes for processing and document them. They should also include details of their purposes in their privacy information for individuals.
Data minimization: Businesses must ensure the personal data they are processing is: adequate – sufficient to properly fulfill their purpose; relevant – has a rational link to that purpose; and limited to what is necessary – they do not hold more than they need for that purpose.
Accuracy: The personal data should be accurate and up to date.
Storage limitation: Personal data should not be kept for longer than needed. Businesses should think about and be able to justify how long they will keep personal data. This will depend on their purposes for holding the data. Businesses should also have a policy setting the standard retention periods wherever possible to comply with documentation requirements. Furthermore, they should periodically review the data they hold and erase or anonymize it when they no longer need it. Lastly, they can keep personal data for longer if they keep it for public interest archiving, scientific or historical research, or statistical purposes.
Integrity and Confidentiality: Businesses must ensure that the data they have is protected and appropriate security measures are in place.
Accountability: The accountability principle requires businesses to take up responsibility for their personal data and comply with other principles. They must have appropriate measures and records in place to be able to demonstrate their compliance.

Rights of an individual according to the GDPR

Which companies does the GDPR affect?

GDPR affects any company that stores, processes, or handles in any capacity, personal information about EU citizens. If your company falls in any of the below categories, you must be GDPR compliant.

Have a presence in an EU country.
Do not have a presence in the EU, but process personal data of European residents.
Have more than 250 employees.
Have less than 250 employees but data-processing impacts data subjects' rights and freedoms and includes sensitive personal data.

In short, every company in the world should be GDPR compliant.

What are the penalties?

In July 2019, British Airways was fined €204,6 million by the ICO for violating article 31 of the GDPR. In September 2018, an incident occurred when the British Airways website diverted users' traffic to a hacker website, which resulted in hackers stealing the personal data of more than 500,000 customers. Other examples include Marriott International in July 2019, who were fined €110 million, and even Google, who were fined €50 million in January 2019.

There are two levels of GDPR fines:

the lower level is up to €10 million, or 2% of the worldwide annual revenue from the previous year, whichever is higher, and
the upper level is twice that size or €20 million and 4% of the worldwide annual revenue.

How can Cloudanix help?

We, at Cloudanix have a variety of recipes for your cloud audit and remediation which implements rules so that you adhere to the various compliance standards, one of them being GDPR. To know more about how we help you to be GDPR compliant, check this link.

References

https://ico.org.uk/for-organisations/guide-to-data-protection/guide-to-the-general-data-protection-regulation-gdpr/key-definitions/controllers-and-processors/

Top 13 AWS EC2 Misconfigurations To Avoid in 2021

Kedar Ghule — Thu, 01 Apr 2021 07:12:05 +0000

In this blog post, we will take a look at the top 13 AWS EC2 misconfigurations that you should avoid. Let us brush up our knowledge on what AWS EC2 is first.

We will be covering the below topics:

What is AWS EC2?
The 13 Common AWS EC2 Misconfigurations
How Can Cloudanix Help?

What is AWS EC2?

Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides a secure and resizable compute capacity in the cloud. In other words, Amazon allows users to rent virtual computers (called EC2 instances) on which they can run their applications. This eliminates your need to invest in hardware resources as you can rely on these virtual servers for your storage and configure your security. The aim of AWS EC2 is to make web-scale distributed computing simpler for developers. The "pay-as-you-go" model of AWS EC2 makes it even more popular.

The 13 Common AWS EC2 Misconfigurations

With every good service and for it to be really secure, you need to take specific measures from your side too. Below are the top 13 common AWS EC2 Misconfigurations that will help you achieve security, efficiency, cost optimization, and adherence to various compliance standards.

Public Snapshots
Non-public EC2 AMI
Encrypted AMI
Not using default VPC
AMI Age
Scheduled Events
EC2 Instance Not In Public Subnet
Unrestricted Netbios Access
Unrestricted Outbound Access
EC2 Reserved Instance Payment Pending
EC2 IAM Roles
Unrestricted CIFS Access
EC2 Reserved Instance Payment Failed

Let us take a look at them one by one in detail.

Public Snapshots

The very first EC2 misconfigurations you should be avoiding is making your snapshots public. Ensure that your EC2 instance snapshots are not publicly accessible. Having public snapshots can expose your personal and sensitive information, thereby violating compliance standards like GDPR, NIST, and PCI DSS. Violating these compliance standards can result in hefty fines and litigation.

Non-public EC2 AMI

The next most common EC2 misconfiguration is publicly sharing your Amazon Machine Images (AMIs) with other AWS accounts. AWS AMIs should not be shared publicly with the other AWS accounts to prevent exposing sensitive data. It is a requirement for NIST, ARPA, MAS compliance standards.

Encrypted AMI

Talking about AMIs, having non-encrypted AMIs is another misconfiguration. When dealing with sensitive data that is crucial to your business, encrypting AMIs is necessary to protect your data from attackers. Amazon Machine Images (AMIs) should be encrypted to fulfill compliance requirements for data-at-rest encryption. Compliance standards required for this are NIST, PCI, ARPA, MAS, HIPAA, GDPR.

Not using default VPC

Using a default VPC is a misconfiguration you should avoid. It is recommended not to using the default VPC. Compliance with this policy is also required for APRA, MAS, NIST.

AMI Age

Your AMI should not be old than a given number of days. This value can be as per your convenience; however, we recommend not having an AMI older than 180 days. Using up-to-date AMIs ensures that your EC2 instances deployed are secure and reliable. Overcoming this misconfiguration is one step towards achieving NIST, APRA, and MAS compliance.

Scheduled Events

Not taking any action on the AWS EC2 scheduled events result in unexpected downtimes that further results in low availability and reliability. You should take the necessary steps on EC2 instances that are scheduled for retirement and/or maintenance.

EC2 Instance Not In Public Subnet

Backend instances should not be running in public subnets. This will help you maintain security. Backend instances are EC2 instances that should run in a private subnet (i.e., behind a NAT gateway). Backend instances do not require direct access to the public internet, such as databases, API, or caching servers.

Unrestricted Netbios Access

No AWS EC2 security group should allow unrestricted inbound access to TCP port 139 and UDP ports 137 and 138 (NetBIOS). Failure to resolve this misconfiguration can result in devastating consequences like Denial of Service (DoS) attacks, man-in-the-middle (MITM) attacks, and data breaches. Furthermore, having this misconfiguration violates a host of compliance standards like - PCI, APRA, MAS, NIST, SOC2.

Unrestricted Outbound Access

EC2 security groups should not allow unrestricted outbound/egress access. The entity implements logical access security software, infrastructure, and architectures over protected information assets to protect them from security events to meet the entity's objectives.

EC2 Reserved Instance Payment Pending

Ensuring that none of your AWS EC2 Reserved Instance purchases have a pending status helps you implement cost optimization. Furthermore, it also helps you comply with APRA, MAS, and AWAF.

EC2 IAM Roles

IAM Roles/Instance profiles should be used instead of IAM Access Keys to appropriately grant access permissions to any application that performs AWS API requests running on your EC2 instances.

Unrestricted CIFS Access

No AWS EC2 security group should allow unrestricted inbound access to TCP port 445 and (CIFS). Failure to resolve this misconfiguration can result in devastating consequences like Denial of Service (DoS) attacks, man-in-the-middle (MITM) attacks, and data breaches. Furthermore, having this misconfiguration violates a host of compliance standards like - PCI, APRA, MAS, NIST.

EC2 Reserved Instance Payment Failed

Ensuring that none of your AWS EC2 Reserved Instance purchases have failed helps you implement cost optimization. Furthermore, it also helps you comply with APRA, MAS, and AWAF.

How Can Cloudanix Help?

EC2 Misconfigurations issues are not new. It is the largest issue faced by organizations for years. It is essential to understand what they are and why acting on them immediately is necessary. Cloudanix provides you with an EC2 recipe that helps audits your AWS account for these misconfigurations and more! We also help you remediate these misconfigurations in an automated way! You can sign up for a free trial here today!