DEV Community: Usman Khan Niazi

Moving data from MongoDB to PostgreSQL using AWS Glue: A Guide

Usman Khan Niazi — Fri, 13 Jan 2023 11:17:16 +0000

If you're working with a large amount of data, you may find that you need to move it from MongoDB to PostgreSQL. This can be a challenging task, but with the help of AWS Glue, it can be made simpler. In this article, we will discuss the steps that you need to take to move data from MongoDB to PostgreSQL using AWS Glue.

Step 1: Connectivity

The first step in moving data from MongoDB to PostgreSQL is to ensure that your Glue job has the necessary connectivity to both the MongoDB and PostgreSQL databases. This may involve setting up VPC peering, security groups, or other networking configurations. It is important to ensure that your Glue job has the necessary permissions and credentials to access both MongoDB and PostgreSQL.

Step 2: Data Structure

The next step is to make sure that the data structure in MongoDB is compatible with the schema in PostgreSQL. This may involve transforming the data as it's being loaded or using a Glue job to create the necessary tables in PostgreSQL. This is important because MongoDB is a document-based database, which allows for flexible and unstructured data, while PostgreSQL is a relational database which is more structured.

Step 3: Data Types

Ensure that the data types used in MongoDB match the data types used in PostgreSQL. This may involve converting data types as part of the ETL process. This is important because MongoDB and PostgreSQL use different data types and if they are not matched properly it could cause errors while loading the data.

Step 4: Security

Security is a crucial aspect of data migration. Make sure that you have the necessary permissions and credentials to access both MongoDB and PostgreSQL. You should also consider encryption of the data in transit and at rest.

Step 5: Performance

When working with a large amount of data, performance is a key consideration. Keep an eye on the performance of the Glue job. You may need to adjust the number of Glue workers or the amount of memory allocated to the job to ensure that it runs efficiently.

Step 6: Monitoring and Error Handling

AWS Glue allows you to monitor the ETL jobs, track the progress and handle errors. It is important to monitor the job and handle any errors that may occur during the data migration process.

Step 7: Data Migration

There are different ways to handle data migration from MongoDB to PostgreSQL, you can either use Glue Job or other data migration tools such as Apache Nifi. The choice of the tool depends on the amount of data, data structure, and the complexity of the migration process.

Conclusion

In conclusion, moving data from MongoDB to PostgreSQL using AWS Glue is a complex task, but with the right steps, it can be made simpler. By following the steps outlined in this article, you can ensure that your data migration process is smooth and successful.

Debunking the Cloud

Usman Khan Niazi — Sun, 23 Jan 2022 11:13:07 +0000

The CS Industry has been a victim of buzzwords for the past couple of years if not decades. From AI, Crypto, Data Science, Blockchain and god knows what else, the industry has been constantly bombarding with these terms. Organizations have used these buzzwords to market themselves tremendously. We’ll take a look at one of these buzzwords to see whether they even have some value or are they just buzzwords

What is Cloud?

One of such buzzword nowadays is “The Cloud”. The Cloud or rather Cloud Computing to be precise, is not something very complicated. In layman’s terms, It can be considered as accessing a set of on-demand computing, storage and networking resources over the internet without the hassle of physically managing them. It refers to the hassle-free scalable and manageable infrastructure that enables organizations to the core business needs rather than focusing on managing these technical things. A formal definition would be:

Cloud computing is the on-demand availability of computer system resources, especially data storage and computing power, without direct active management by the user. Large clouds often have functions distributed over multiple locations, each location being a data center

What does the cloud look like?

The cloud is not a physical cloud (。_。). It is computer physically located somewhere in a datacenter and it is accessed over the internet through a Dashboard or UI. It is really

Types of Cloud

We have 3 major types of cloud.

Public Cloud
Hybrid Cloud
Private Cloud

We’ll discuss them more briefly in another article as they are a topic of their own.

Cloud Providers

Different organizations are providing cloud services. The list is growing so fast but I’ll list down ones with the biggest market share:

Amazon Web Services (AWS)
Microsoft Azure
Google Cloud Platform (GCP)

We’ll be discussing about AWS as it has the biggest market share and it was one of the first to implement the cloud.

Amazon Web Services (AWS)

Amazon Web Services or AWS for short is one of the biggest cloud providers in the world. It is a subsidiary of Amazon. It was one of the first one to introduce the world to the concept of cloud computing. For years, They had no competitors.

The internet runs on AWS

Yes you heard it right. Most of the internet run off of AWS. it is said that somewhere between 3% to 50% of the internet services are hosted on AWS. That’s the reason whenever AWS goes down, it takes majority of the internet with it. However, This is just one example of the failure as AWS supports fault tolerant systems.

What can you do with AWS?

So you might be wondering, what can you do with AWS? AWS can accommodate a gigantic amount of Use-Case’s. Some of them are listed below:

Hosting a Web Store
Deploying your Web Applications
Deploying your AI Applications
Business Analytics
Migration of your existing solutions to AWS

Conclusion

The cloud is not just a buzzword. It is a immensely useful piece of technology which is helping organizations scale vertically and horizontally via cloud platforms. Cloud is taking over the industry at a huge rate. It will bring a lot of opportunities with it.

This was a high level overview of the cloud computing world and AWS. We will look more into the technical details of each of the cloud provider in the upcoming articles.

How to enable chrome extensions in selenium python

Usman Khan Niazi — Mon, 21 Jun 2021 10:24:05 +0000

In this tutorial , I'm going to show you how you can load chrome extensions when you run selenium

Install selenium by typing pip install selenium in the command prompt or terminal
write the following code
Here pathtocrx.crx is the path to where your chrome extension is stored and pathtochromewebdriver.exe is the path to your downloaded chrome web driver. Change both of them to your own paths.

That's all that you need to do. Now you have the mentioned chrome extensions enables and added by default

How to convert a custom Chrome extension to a .crx extension / file type

Usman Khan Niazi — Mon, 21 Jun 2021 10:00:01 +0000

Ever ran into an issue where you need to convert a chrome extension that you wrote or pulled from somewhere on GitHub?

Don’t you worry. I’ve got you covered.

You can convert it with just a few simple steps

Steps:

Open Chrome
type chrome://extensions/ in the search bar
Toggle the Developer Mode in the upper right corner

Developer Mode in the Upper right corner

After Enabling it, this will appear

Click on Pack Extension

Now this should appear. Leave the private key empty (it will be default) and click on browse Extension root dir.
Select the root directory of your extension and click pack extension

That's all you need to do. Your Crx file is now ready and generated. It will be present in the parent directory of your extension root.