Building a Serverless Netflix Data Analytics Platform with AWS Glue, Athena and QuickSight.

Quick Data & Cloud Engineering project

Hey Guys, Today we'll be building a serverless data analytics platform using AWS services.

Our goal is pretty straight forward, We want to take a Netflix dataset in CSV format, store it in Amazon S3, catalogue and transform it using AWS Glue, query the processed data using Amazon Athena, and finally create interactive dashboards using Amazon QuickSight, now called Quick, i think.

The idea is to simulate how companies handle analytics workloads without managing servers or traditional database infrastructure and to get a personal feel of how these services work.

Again, First, we store our raw Netflix dataset inside Amazon S3. Then we use AWS Glue to discover the structure of the data and create metadata using the Glue Data Catalog.

After that, we create an ETL pipeline that cleans the dataset and converts it from CSV into Parquet format. Parquet is much better suited for analytics workloads because it is compressed and column-based and it takes less time to query than CSV as we'll see.

Once the data is processed, Amazon Athena allows us to query it using SQL. Finally, Amazon QuickSight connects to Athena and turns those queries into dashboards and visual insights.

Simple enough.

Table of Contents

• Introduction
• Step-by-Step Setup
  • STEP 1) Setting Up the S3 Data Lake
  • STEP 2) Creating the AWS Glue Data Catalog and Crawler
  • STEP 3) Building the AWS Glue ETL Pipeline
  • STEP 4) Building Analytics with Athena and QuickSight
  • STEP 5) Next Steps (Automate with S3 Event Notification and EventBridge)

1. Setting Up the S3 Data Lake

The first step is creating the storage layer.

Amazon S3 will act as our data lake.

We separate our data into different stages on the S3 Bucket

Raw data
Processed data
Athena query results

This makes the workflow easier to understand and maintain.

Our bucket structure looks like

netflix-analytics-data-lake/

├── raw/
│   └── netflix_titles.csv
│
├── processed/
│
└── athena-results/

The raw folder contains the original csv file.

After creating your S3 Bucket

Block all public access
Enable Bucket versioning &
Server-side encryption

The dataset is uploaded into raw/netflix_titles.csv

1. Creating the AWS Glue Data Catalog and Crawler

Create a database, netflix_analytics

This database will contain our tables.

Create an IAM Role for Glue

The role requires permissions to:

Read from S3
Write metadata
Create Glue tables

You should ideally use the least privilege required for the task but for now we'll use Full Access.

AmazonS3FullAccess

Creating the Glue Crawler

A crawler automatically discovers the structure of our dataset.

Navigate to:

AWS Glue and create a Crawler and name it netflix-raw-data-crawler

The Data source would be the S3 Bucket you created earlier, netflix_analytics

Table prefix, raw_

Run the crawler.

Glue scans the CSV file and creates a table.

Next, you Test the Glue Table with Athena

Before building the ETL(Extract, Transform, Load) pipeline, we verify that Athena can read the table.

Open Athena and select the database and run,

SELECT *
FROM raw_netflix_titles
LIMIT 10;

If everything works, we should see Netflix records.

At this point, we have successfully connected:

S3 to Glue and Glue to Athena

1. Building the AWS Glue ETL Pipeline

Now we move from transforming the raw data into processed data.
The Goal in this step is to convert, the CSV file into Parquet format/file.

Creating the Glue Visual ETL Job

On the AWS Glue page, navigate to Visual ETL on the sidebar,

Create a new job. netflix-csv-to-parquet

The default IAM role it suggests is missing a couple roles, s3:GetObject
s3:PutObject
s3:ListBucket add these and you won't run into the error i ran into.

AWSGlueServiceRole-netflix-analytics

Here we want to add a source, our S3 bucket take the table, apply a mapping to it, drop empty fields, and add a destination to drop the finished parquet file

Adding the Source

Applying Schema Mapping

This mapping handles our schema changes.

DropNullFields

Optional but this removes fields where every value is empty.

Configuration:

Enable:
Empty String
"null" String

Writing Parquet Output

s3://your-bucket/processed/netflix_titles/

Run the Glue job and the output should look like this after a few minutes

processed/netflix_titles/part-00000-snappy.parquet

A Small IAM Challenge

During the first run, the Glue job might fail.

The error:

not authorized to perform: s3:PutObject

Easily fixable by adding s3:GetObject
s3:PutObject
s3:ListBucket to your gluerole
Fixed and it works perfectly.

4. Building Analytics with Athena and QuickSight

Now we have clean Parquet data.

The Next step is Querying the Processed Data with Athena

This Query tells us the Top Producing countries.

SELECT
country,
COUNT(*) AS total_titles
FROM processed_netflix_titles
GROUP BY country
ORDER BY total_titles DESC
LIMIT 10;

If you run a query with the raw csv vs the parquet file, you'll see the parquet is faster which is the entire point of the transformation.

Now we create the dashboard layer.

Create a Quick Account,

Add a New Dataset, Select Athena, Choose the database and processed table

Import into SPICE and you can now you can create your dashboard

You can create visualisations here that answer specific questions.

Visual 1: Content Distribution

Question:

What type of content dominates Netflix?

Next Steps

The current pipeline is a batch analytics workflow.

The next step would be automating this workflow with EventBridge and S3 Event Notifications. I'll do that in a seperate article. See you soon.