Simplify Data Prep with AWS Glue DataBrew

pradeep-misra — Mon, 01 Feb 2021 02:59:17 +0000

Business users always format, transform and summarize data to derive insights. Even with data being ETL to Data Warehousing systems; users had to apply additional transformations in BI tools.

As modern enterprises are increasingly adopting Lake House architectures, there is an even greater need to simplify this process for data prep on large data sets that can scale.

AWS Glue DataBrew

AWS Glue DataBrew is a new visual data preparation tool that makes it easy for data analysts and data scientists to clean and normalize data to prepare it for analytics and machine learning.

It allows you to:

Profile
Clean and Transform
Track Lineage
Orchestrate and Automate

In the remaining sections, I will demonstrate these capabilities using a simple customer segmentation dataset - Link

Data Profiling

First thing we do on a dataset is to profile - either with code or here with a few clicks in Data Brew.

To begin, select the dataset and "Run data profile" button.

This would take us to "Create job" page where we create a "Profile Job" and this will generate summary statistics on our data.

Once Profile job is complete, it provides a profile overview, detailed column statistics and lineage.

Projects

Next, we will transform the data by creating a project on the dataset. We can create Project from the Datasets view by specifying Project Name and Recipe details. For recipe we can either create a new one or use from existing list.

While creating recipes, transforms are done with sample data. For this Project, we will select 5000 random rows.

Project Landing Page

Default view of this page shows us sample rows, distribution and recipe steps.It also shows the available list of transformations that can be done on the dataset.

Schema View-

Transformations steps can be added and together makes a recipe.

DataBrew provides from over 250 built-in transformations to visualize, clean, and normalize your data with an interactive, point-and-click visual interface.

For this dataset, we will create a recipe with two transformations:

Filter Customers with age greater than equal to 30
Encode Gender to 'M' or 'F'

1. Filter Customers with age greater than 30

From Filter category > By Condition > Greater than or equal to:

Select the source column and the filter value

Preview shows the result of the step and helps verify the outcome before applying it

2. Encode Gender to 'M' or 'F'

From the clean category > select Replace value or Pattern

Select source column and the value to be replaced
Note - We need two steps here, one for each value to be replaced.

Preview changes and apply

Then, transform the next value

As we are done with all the steps, we are good to publish our recipe:

Jobs

We had earlier created a Profile Job. We will now create a recipe job that will transform the entire dataset.

We will specify the Dataset, Recipe and Output details.

Once job is created, we get the transformed dataset, job run and history details.

Data Lineage

Lineage shows the various components and their relationships to the output.

Summary

Data Brew provides a powerful abstraction for data preparation. As explained, it fastens and simplifies the Data Prep process allowing Users to focus on the insights and decisions that drives their business.
.

DEV Community: pradeep-misra