Investigating the TMDB movie dataset, part 2

nkpremices — Thu, 12 Mar 2020 18:23:26 +0000

This blog post is the second part of a whole series. I would recommend you read the first part if you want to understand this one.

In this blog post, I am going to talk about data cleaning. we are going to use the results that we got in the first part and build from there.

Data Cleaning

Step 1. Remove some columns with a lot of null values.

df.drop(['imdb_id', 'homepage', 'tagline', 'overview', 'budget_adj', 'revenue_adj'], axis=1, inplace=True) df.head(1)

Step 2. Remove duplicated data

df.drop_duplicates(inplace=True)

Step 3. Remove all null values in the columns that have null values

df.dropna(subset = ['cast', 'director', 'genres'], how='any', inplace=True)

Let's check if there are still null values

df.isnull().sum()

Step 4. Replace zero values with null values in the budget and revenue column.

df['budget'] = df['budget'].replace(0, np.NaN) df['revenue'] = df['revenue'].replace(0, np.NaN) df.info()

Step 5. Drop the runtime column.

df.query('runtime != 0', inplace=True) df.query('runtime == 0')

df.info()

df.describe()

From the table above, we can see that replacing the zeros by null values in the budget and revenue distribution made them look better. We can also see that the minimum makes now more sense

This is the end of the second part. If you got some good time reading, stay tuned. I will post the third part soon.

thank you for reading.

Investigating the TMDB movie dataset

nkpremices — Wed, 11 Mar 2020 15:50:13 +0000

Lately, I've been going through the Data analyst nanodegree program of Udacity. I worked on some projects there and I will be writing blog posts about them in the coming weeks.

Note: This blog post is the first part of a whole series of blogposts where I describe a whole dataset analysis. The aim is to showcase how simple the data analysis can be.

Introduction

About the dataset

The dataset is called TMDB movie data. Downloaded from this page, its original version was removed by Kaggle and replaced with a similar set of movies and data fields from The Movie Database (TMDb). It contains more than 5000 movies and their rating and basic information, including user ratings and revenue data.

A successful movie is evaluated by its popularity, vote average score(Ratings) and revenue. There are some keys that can affect the success of a movie. For example, the Budget, Cast, Director, Tagline Keywords, Runtime, Genres, Production Companies, Release Date, Vote Average, etc.

Looking at how the data is in the dataset, various questions can be asked. For example -

How was the popularity of a movie over the years?
Considering the five recent years, how is the distribution of revenue in different score rating levels ?
How is the distribution of revenue in different popularity levels ?
What kinds of properties are associated with movies that have high popularity?
What kind of properties are associated with movies that have high voting score?
How many movies are released year by year ?
What are the keywords trends by generation ?

In this series of blog posts, we are going to answer the questions above using the TMDB Movie data, Numpy, Pandas, and Matplotlib.

for this blogpost, we will focuss on general comments about the data

Firt of all, let's import the needed packages

import pandas as pd import numpy as np import matplotlib.pyplot as plt import seaborn as sns from collections import Counter %matplotlib inline

Data Wrangling

General Properties

Let's load the info of the dataset

df = pd.read_csv('tmdb-movies.csv') df.info()

Judging form the info above, the dataset has 10866 entries and 21 colums. The types used are int, float and string. Form the total number of entries and the number of entries per column, a lot of columns have null values. Let's check the exact number of null records per column.

list(df.isnull().sum().items())

Looking at the result above, we see that the colums that have null values are cast, homepage, director, tagline, keywodds, overview, genres, production companies. We also see that homepage, tagline, keywords and production_companies have a lot of null records. I decided to get rid of tagline and keywords since they have a lot of null values.

Let's try to get more descriptive information from the dataset

df.describe()

If we look at the popularity column, we can find some outliers. since it has no upperbond, it is better to just retain the original data. We can see that there is a lot of zero values in the budget, revenue and runtime columns. The first guess might be that these movies were not released but if we look at the release_year column we can notice that the minimum value (1996) is avalid year and that there were no null values. Therefore those movies were released. May be the zeroes mean the abscence of data. However, in order to decide on that let's check closely those records

First for the budget

df_budget_zero = df.query('budget == 0') df_budget_zero.head(3)

Then for the revenue

df_revenue_zero = df.query('revenue == 0') df_revenue_zero.head(3)

After checking for Mr Afonso poyart on the film Solace#Production) on wikkipedia, I noteiced that the film was actually a success. WHich means that there was a sucessful release wich also means that there was a budget. Therefore, the zero values were missing data. I would decide based on that to drop the records since this might affect the statistics od the result of my analysis.

Subsequently, lets check the number of null values to decide if the zeros should just be set as ull or completely droped out.

First for the budget zero values

df_budget_0count = df.groupby('budget').count()['id'] df_budget_0count.head(2)

As suggested by the results, there are a lot of zero values than non zero values. Dropping them out would corrupt the results. I better set them as null instead.

Then for the revenue zero values

df_revenue_0count = df.groupby('revenue').count()['id'] df_revenue_0count.head(2)

Same situation. Set to null

Finally for the runtime

The number of zeroes is negligible, they can be droupped out

Summary

Remove some columns with a lot of null values and unnecessary ones for answering the questions : homepage, tagline, imdb_id, overview, budget_adj, revenue_adj.
Remove duplicated data
Remove all null values in the columns that have null values
Replace zero values with null values in the budget and revenue column.
Drop the lines with runtime == 0.

The first part ends here. If you had some good time reading this, kindly check the second part which is about data cleaning.

Thank you for reading

DEV Community: nkpremices