DEV Community: Subh Chaturvedi

25 Years of Indian Headlines: A Data Opportunity

Subh Chaturvedi — Tue, 21 Jun 2022 20:11:34 +0000

Would you be interested in knowing which politician has made the most headlines? Do you want to see which celebrity has best held the attention of journalists over the years? Are you intrigued about the difference in crime news grouped by states?

If your answer is yes to any of those questions, then we share the same curiosity.

Looking at archives maintained by Newspapers, I wondered who would go through over twenty-five thousand pages of news headlines sorted by date and make any heads or tails of the big picture around it. How could someone analyse trends or gains insights from an almost endless list of headlines dating decades ago by just reading them on a website.

But I could not stop myself from doing something about this problem I had at hand. I needed to devise a way to reliably, efficiently and easily comb through the data these archives made available. To accomplish this goal, I set out with this project.

In this writeup I will go over how I created a fundamental tool or method for anyone to easily analyse, understand and research the Indian Headlines data of the past 25 years.

PART 1: The Data

To start with my project, I thought I could easily find a dataset which contained all the data I needed, properly formatted in a csv file and would only leave me to write scripts to analyse that data. If only life was that easy.

Unfortunately, I could not find any dataset suitable to my needs and thus had to dedicate a significant portion of work-time of this project on procuring and cleaning the data myself.

Ideally, we would need a data file with an exhaustive, credible and unbiased collection of headlines from a large enough timeline.

Sourcing the Data

In the absence of pre-prepared datasets, I had to find a credible source to collect headlines data stretching for a significant duration of time. So what better place than the websites of major journalistic organisations like The Times, The Hindu, Deccan Herald or the Indian Express.

But these organisations do not publicly provide an easy way to procure the data we need. So the most obvious way was to read the data off their own websites. Fortunately almost all of these organisations provide a public archive of their journalism which is better option than sitting and recording all the headlines myself for the next decade.

But the archives that these organisations provide do not allow you to easily export the data they make available themselves in a format which can be utilized by us. So the best way to obtain the data we need is to utilize Web Scraping.

Web Scraping

Web Scraping is essentially the process of automatically extract the data we need from websites by utilizing tools which can parse and read data from the source code of these websites (Mostly HTML).

For this project, I use the following python modules/packages which are extremely suited for the task:

BeautifulSoup: A python library that makes it easy to scrape content from web pages. It sits atop an HTML or XML parser, providing Pythonic idioms for iterating, searching, and modifying the parse tree.
Requests: The requests module allows you to send HTTP requests using Python. The HTTP request returns a Response Object with all the response data(the web page code).

❗ I have not covered the Ethics of using Web Scraping Techniques in the scope of this project. Please do your research before putting into practice any code/instructions present here. ❗


A screenshot of the Indian Express Archives page I scraped for the purpose of this project.

I decided to use the Indian Express News Archives because it generates its content statically and the URL to change pages is easily iterable through code. Certain other public archives generate their web pages dynamically through JavaScript, while some others make it very difficult to traverse the content for the scraping script.

Now lets look at the code for the scraping script I wrote:

Code Implementation

This is the basic code needed to obtain the source HTML code from the Indian Express pages. The pageNumber is the variable iterated through a for loop in the range of the pages we need to scrape.

url = "https://indianexpress.com/section/news-archive/page/"+str(pageNumber)+"/"

page_request = requests.get(url)
data = page_request.content
soup = BeautifulSoup(data,"html.parser")

The final obtained source code is stored in the soup variable.

Now, the actual headlines in the code are actually stored inside <h2> tags with the class name as title. So all we need to do is use the BeautifulSoup library and filter the obtained code to read off the <a> tags from the filtered title headline tags. Then we further use get_text function to get the headline string.

for divtag in soup.find_all('h2', {'class': 'title'}):
        for atag in divtag.find_all('a'):

            #HEADLINES
            hl=atag.get_text()
            print("Headline: ",hl) # This prints the scraped headline.

It is much more convenient for us to store the scraped data in a text file, rather than print it on the terminal. A solution to write all of this data is easily implemented with the following code.

file1 = open(r"<ABSOLUTE PATH>\headlines.txt","a",encoding="utf-8")

file1.write(hl.lower()) # stored in lower for easier analysis
file1.write("\n")

file1.close()

When this code is properly looped, it can be used to scrape and obtain the headline data from the complete archive rather than just one page. In this case the pages extend to Page 25073 and thus a the above code can simply be put into a loop iterating over all the pages.

When I ran the final script for the first time, I realised that scraping over 25 thousand pages with this script would take me over multiple days running my the script 24x7, while any interruption to the process could erase all the progress.

Thus for the sake of accessibility, ease-of-use and practicality of the process I implemented the code in a fashion where it could be paused at any moment and would save and remember its progress. While this addressed the problem of interruptions and being able to pause the script execution, it was still projected to take me a very long time to just be able to scrape this data. To decrease the time it would take for the complete scraping of the archive, I divided the 25 thousand pages into 20 equal groups. This allowed my workstation to simultaneously run twenty instances of the same script running parallel workloads and reducing the time needed over twentyfold.

PART 2: The Framework

All of the above described process finally netted me a list of over 6.5 lac headlines in a text file in the following format:

covid-19: up govt extends closure of all schools, colleges till january 23
renotify 27% seats reserved for obcs in local body polls, sc tells maharashtra, sec
talks fail over action against karnal ex-sdm, siege continues
jagan promotes andhra as favoured investment destination, defends scrapping ppas
murder-accused husband of mla visits house, bjp slams congress
over 25,000 cops, cctvs, drones to secure gujarat rath yatra on july 4
in bhutan, teachers, medical staff will now be highest paid civil servants
‘cyclone man’ mrutyunjay mohapatra appointed imd chief
pakistan hotel attack: four killed as armed militants storm 5-star hotel in gwadar port city, says police
indian man convicted in dubai for hacking 15 client websites
[continued]

These headlines encompass a time period of 25 years (Thus the title) from 1997-2022.

Keyword Analysis

It is obviously in a much more accessible format now, but how do we run any kind of analysis on this data file?
The simple answer is Keyword Analysis. In the context of this project, it is the method of classifying, categorizing, labelling, and analysing these headlines by the presence (or lack of) certain words in them.

For example, if a certain headline has the word "Mahatma Gandhi", "Bapu", or "Gandhi Ji", it is very likely to be a headline reporting something about Gandhi. We will be using this form of deduction in the scope of this project to analyse the dataset that we have formed.

Therefore to exercise this idea, we need to implement certain tools to be on our disposal. In the scope of this project, these tools are basically methods/functions which find(/aggregate) the headlines depending on the presence of keywords. Multiple different need based cases can be imagined, but overall they can all be divided into the following fundamental implementations:

Finding the headlines where any of the keyword from a group is found.
Finding the headlines where all of the keywords from a group are found.
Finding the headlines where any of the keywords from each of the multiple groups are found.

Any other needed use-case can be made to fit/work with these fundamental functions. For example, if we need to find the headlines where all of the keywords from each of the multiple groups in found, we can just merge the groups and use the second function for them. On the other hand if we need to find the headlines where the keywords do not occur, we can just eliminate the headlines where those keywords do occur using the first function.

Code Implementation

The above mentioned three fundamental functions are implemented as follows:

Finding the headlines where any of the keyword from a group is found. Here, the code iterates through all the headlines in the headLinesList object and if any of the keyword provided in the list is found then the counter maintained is bumped and finally returned.

# FUNCTION TO COUNT HEADLINES WHERE ANY OF THE WORD/S OCCURS
#   - wordsToSearch is a list of the form ["word1","word2","word3",...]
def countOccurancesAny(wordsToSearch):
    count = 0

    for headline in headLinesList:
        for eachWord in wordsToSearch:
            if headline.find(eachWord.lower()) != -1:
                count += 1
                break

    return count

Finding the headlines where all of the keywords from a group are found. The premise here is almost the same as the 1st function, but instead of searching for one word and then bumping the counter, this function has to wait and make sure all the elements of the list are found in the headline. For optimisation purposes, the loop skips cycles if any of the element is not found.

# FUNCTION TO COUNT HEADLINES WHERE ALL OF THE WORD/S OCCURS
#   - wordsToSearch is a list of the form ["word1","word2","word3",...]
def countOccurancesAll(wordsToSearch):
    mainCount = 0
    wordCount = 0

    totalWords = len(wordsToSearch)

    for headline in headLinesList:
        wordCount = 0
        for eachWord in wordsToSearch:
            if headline.find(eachWord.lower()) == -1:
                break
            else:
                wordCount += 1
        if wordCount == totalWords:
            mainCount += 1

    return mainCount

Finding the headlines where any of the keywords from each of the multiple groups are found. This function basically nests the first function one more time, finding out if a keyword from the groups passed off as 2D lists are present. There is also optimization similar to function 2 present here.

# FUNCTION TO COUNT HEADLINES WHERE ANY OF THE WORD/S FROM MULTIPLE GROUPS OCCURS
#   - wordsToSearch is a list of lists, each of the form ["word1","word2","word3",...]
def countOccurancesGroupedAny(wordsToSearch):
    count = 0
    countMain = 0

    for headline in headLinesList:
        countMain = 0

        for eachGroup in wordsToSearch:
            optimiseCheck1 = 0

            for eachWord in eachGroup:
                if headline.find(eachWord.lower()) != -1:
                    optimiseCheck1 = 1
                    countMain += 1
                    break
            if optimiseCheck1 == 0:
                break

        if countMain == len(wordsToSearch):
            count += 1

    return count

Well, now that we have looked at the tools and platforms I developed for this project, lets look at how these can be used to analyse and interpret data.

PART 3: The Analysis

An analysis was not so much the focus of this project than actually developing a framework allowing for easy analysis of this dataset. But I have still done some analysis and visualizations to demonstrate the usability of the above described code.

I will be using the word "Popular" to denote how much the entity has been in news. It has unreliable correlation with actual popularity.

Most Popular Political Party

First, we create lists with the keywords that will be used to identify appropriate headlines. These lists contain keywords that I think are enough to identify if a headline is aimed at these parties.Then the data is obtained and arranged as a pandas dataframe and then printed sorted by headlines count.

bjp = ["bjp","Bharatiya Janata Party"]
inc = ["INC","Congress"]
aap = ["AAP","Aam Aadmi Party"]
cpi = ["CPI","Communist Party of India"]
bsp = ["BSP","Bahujan Samaj Party"]
dmk = ["DMK","Dravida Munnetra Kazhagam"]
shivsena = ["sena","shiv sena"]

data = {'Party': ['BJP', 'INC', 'AAP', 'CPI', 'BSP', 'DMK', 'SHIV SENA'],
        'Reported': [hd.countOccurancesAny(bjp),hd.countOccurancesAny(inc),hd.countOccurancesAny(aap),hd.countOccurancesAny(cpi),hd.countOccurancesAny(bsp),hd.countOccurancesAny(dmk),hd.countOccurancesAny(shivsena)]}
df = pd.DataFrame(data)
print(df.sort_values("Reported",ascending=False))

The result is visualised in the following graph:

As is clearly discernible, INC is the most reported on by part followed very closely by BJP.

Most Reported Politician

The code to crunch the numbers is very similar to the one we saw above:

modi = ["Modi","Narendra Modi","Narendra Damodardas Modi"]
amitshah = ["amit shah","shah"]
rajnath = ["rajnath singh"]
jaishankar = ["jaishankar"]
mamata = ["mamata","didi"]
kejriwal = ["aravind","kejriwal"]
yogi = ["adityanath","yogi","ajay bisht"]
rahul = ["rahul"]
stalin = ["stalin"]
akhilesh = ["akhilesh"]
owaisi = ["owaisi"]
gehlot = ["gehlot"]
biswa = ["himanta biswa","biswa"]
scindia = ["scindia"]
sibal = ["sibal"]
manmohan = ["manmohan singh"]
mayavati = ["mayavati"]
mulayam = ["mulayam"]
naveen = ["patnaik"]
nitish = ["nitish"]
sonia = ["sonia"]
uddhav = ["uddhav"]

data = {
    'Political' : ['Narendra Modi','Amit Shah','Rajnath Singh','S Jaishankar',
    'Mamata Bannerjee','Aravind Kejrival','Yogi Adityanath','Rahul Gandhi','MK Stalin',
    'Akhilesh Yadav','Asaduddin Owaisi','Ashok Gehlot','Himanta Biswa Sarma', 'Jotiraditya Scindia',
    'Kapil Sibal','Manmohan Singh','Mayavati','Mulayam Singh Yadav','Naveen Patnaik','Nitish Kumar','Sonia Gandhi','Uddhav Thackeray'],
    'Reported' : [hd.countOccurancesAny(modi),hd.countOccurancesAny(amitshah),hd.countOccurancesAny(rajnath),hd.countOccurancesAny(jaishankar),
    hd.countOccurancesAny(mamata),
    hd.countOccurancesAny(kejriwal),hd.countOccurancesAny(yogi),hd.countOccurancesAny(rahul),hd.countOccurancesAny(stalin),
    hd.countOccurancesAny(akhilesh),hd.countOccurancesAny(owaisi),hd.countOccurancesAny(gehlot),hd.countOccurancesAny(biswa),hd.countOccurancesAny(scindia),
    hd.countOccurancesAny(sibal),hd.countOccurancesAny(manmohan),hd.countOccurancesAny(mayavati),hd.countOccurancesAny(mulayam),hd.countOccurancesAny(naveen),
    hd.countOccurancesAny(nitish),hd.countOccurancesAny(sonia),hd.countOccurancesAny(uddhav)]
}
df = pd.DataFrame(data)
print(df.sort_values("Reported",ascending=False))

For the visualization aspect of this I thought of using a Circle Packing Graph. I explored my options with external visualization options but none of them offered what I was looking for.
After a lot of research I stumbled on a way to generate what I wanted through python code and the Circlify module.

So, I wrote the code to generate the graph I was looking for. But it turned out to not be as polished or presentable as I wanted it to be. But I am still including the code snippet I wrote:

# compute circle positions:
circles = circlify.circlify(
    df['Reported'].tolist(), 
    show_enclosure=False, 
    target_enclosure=circlify.Circle(x=0, y=0, r=1)
)

# Create just a figure and only one subplot
fig, ax = plt.subplots(figsize=(10,10))

# Remove axes
ax.axis('off')

# Find axis boundaries
lim = max(
    max(
        abs(circle.x) + circle.r,
        abs(circle.y) + circle.r,
    )
    for circle in circles
)
plt.xlim(-lim, lim)
plt.ylim(-lim, lim)

# list of labels
labels = df['Reported']

# print circles
for circle, label in zip(circles, labels):
    x, y, r = circle
    ax.add_patch(plt.Circle((x, y), r, alpha=0.2, linewidth=2))
    plt.annotate(
        label, 
        (x,y ) ,
        va='center',
        ha='center'
    )

plt.show()

Instead of the Circle Packing Graph, I decided to create the following visualization which presents the data generated by my above code.

Crime Statistics By States

Before diving right into the the crime news statistics, lets first write individual code for calculating State statistics, and crime statistics.


# Crime Statistics

robbery = ['robbery','thief','thieves','chori']
sexual = ['rape','rapist','sexual assault']
dowry = ['dowry','dahej']
drugs = ['drug']
traffick = ['traffick']
cyber = ['hack','cyber crime','phish']
murder = ['murder']

data = {
    'Crime':['Robbery','Sexual','Dowry',"Drugs",'Trafficking','Cyber','Murder'],
    'Reported':[hd.countOccurancesAny(robbery),hd.countOccurancesAny(sexual),hd.countOccurancesAny(dowry),hd.countOccurancesAny(drugs)
    ,hd.countOccurancesAny(traffick),hd.countOccurancesAny(cyber),hd.countOccurancesAny(murder)]
}
df = pd.DataFrame(data)
print(df.sort_values("Reported",ascending=False))

#            Crime  Reported
#   6       Murder      3271
#   1       Sexual      2533
#   5        Cyber      1538
#   3        Drugs      1403
#   0      Robbery       384
#   2        Dowry       217
#   4  Trafficking       170

# States Stats

UttarPradesh = ['Uttar Pradesh', ' UP ',"Agra" ,"Aligarh" ,"Ambedkar Nagar" ,"Amethi" ,"Amroha" ,"Auraiya" ,"Ayodhya" ,"Azamgarh" ,"Baghpat" ,"Bahraich" ,"Ballia" ,"Balrampur" ,"Barabanki" ,"Bareilly" ,"Bhadohi" ,"Bijnor" ,"Budaun" ,"Bulandshahr" ,"Chandauli" ,"Chitrakoot" ,"Deoria" ,"Etah" ,"Etawah" ,"Farrukhabad" ,"Fatehpur" ,"Firozabad" ,"Gautam Buddha Nagar" ,"Ghaziabad" ,"Ghazipur" ,"Gorakhpur" ,"Hamirpur" ,"Hardoi" ,"Hathras" ,"Jalaun" ,"Jaunpur" ,"Jhansi" ,"Kannauj" ,"Kanpur Dehat" ,"Kanpur Nagar" ,"Kasganj" ,"Kaushambi" ,"Kushinagar" ,"Lalitpur" ,"Lucknow" ,"Maharajganj" ,"Mahoba" ,"Mainpuri" ,"Mathura" ,"Meerut" ,"Mirzapur" ,"Moradabad" ,"Muzaffarnagar" ,"Pilibhit" ,"Pratapgarh" ,"Prayagraj" ,"Raebareli" ,"Rampur" ,"Saharanpur" ,"Sambhal" ,"Sant Kabir Nagar" ,"Shahjahanpur" ,"Shamli" ,"Shravasti" ,"Siddharthnagar" ,"Sitapur" ,"Sonbhadra" ,"Sultanpur" ,"Unnao" ,"Varanasi" ]
AndamanNicobar = ["Andaman","Nicobar"]
AndhraPradesh = ["Andhra","Anantapur" ,"Chittoor" ,"East Godavari" ,"Alluri Sitarama Raju" ,"Anakapalli" ,"Annamaya" ,"Bapatla" ,"Eluru" ,"Guntur" ,"Kadapa" ,"Kakinada" ,"Konaseema" ,"Krishna" ,"Kurnool" ,"Manyam" ,"N T Rama Rao" ,"Nandyal" ,"Nellore" ,"Palnadu" ,"Prakasam" ,"Sri Balaji" ,"Sri Satya Sai" ,"Srikakulam" ,"Visakhapatnam" ,"Vizianagaram" ,"West Godavari"]
ArunachalPradesh = ['Arunachal',"Anjaw" ,"Changlang" ,"Dibang Valley" ,"East Kameng" ,"East Siang" ,"Kamle" ,"Kra Daadi" ,"Kurung Kumey" ,"Lepa Rada" ,"Lohit" ,"Longding" ,"Lower Dibang Valley" ,"Lower Siang" ,"Lower Subansiri" ,"Namsai" ,"Pakke Kessang" ,"Papum Pare" ,"Shi Yomi" ,"Siang" ,"Tawang" ,"Tirap" ,"Upper Siang" ,"Upper Subansiri" ,"West Kameng" ,"West Siang" ]
Assam = ['assam',"Bajali" ,"Baksa" ,"Barpeta" ,"Biswanath" ,"Bongaigaon" ,"Cachar" ,"Charaideo" ,"Chirang" ,"Darrang" ,"Dhemaji" ,"Dhubri" ,"Dibrugarh" ,"Dima Hasao" ,"Goalpara" ,"Golaghat" ,"Hailakandi" ,"Hojai" ,"Jorhat" ,"Kamrup" ,"Kamrup Metropolitan" ,"Karbi Anglong" ,"Karimganj" ,"Kokrajhar" ,"Lakhimpur" ,"Majuli" ,"Morigaon" ,"Nagaon" ,"Nalbari" ,"Sivasagar" ,"Sonitpur" ,"South Salmara-Mankachar" ,"Tinsukia" ,"Udalguri" ,"West Karbi Anglong" ]    
Bihar = ['bihar', "Araria" ,"Arwal" ,"Aurangabad" ,"Banka" ,"Begusarai" ,"Bhagalpur" ,"Bhojpur" ,"Buxar" ,"Darbhanga" ,"East Champaran" ,"Gaya" ,"Gopalganj" ,"Jamui" ,"Jehanabad" ,"Kaimur" ,"Katihar" ,"Khagaria" ,"Kishanganj" ,"Lakhisarai" ,"Madhepura" ,"Madhubani" ,"Munger" ,"Muzaffarpur" ,"Nalanda" ,"Nawada" ,"Patna" ,"Purnia" ,"Rohtas" ,"Saharsa" ,"Samastipur" ,"Saran" ,"Sheikhpura" ,"Sheohar" ,"Sitamarhi" ,"Siwan" ,"Supaul" ,"Vaishali" ,"West Champaran"]
Chandigarh = ['Chandigarh']
Chhattisgarh = ['Chhattisgarh', "Balod" ,"Baloda Bazar" ,"Balrampur" ,"Bastar" ,"Bemetara" ,"Bijapur" ,"Bilaspur" ,"Dantewada" ,"Dhamtari" ,"Durg" ,"Gariaband" ,"Gaurela Pendra Marwahi" ,"Janjgir Champa" ,"Jashpur" ,"Kabirdham" ,"Kanker" ,"Kondagaon" ,"Korba" ,"Koriya" ,"Mahasamund" ,"Manendragarh" ,"Mohla Manpur" ,"Mungeli" ,"Narayanpur" ,"Raigarh" ,"Raipur" ,"Rajnandgaon" ,"Sakti" ,"Sarangarh Bilaigarh" ,"Sukma" ,"Surajpur"]
Dadra = ['Dadra', 'Daman', 'Diu']
Delhi = ['Delhi']
Goa = ['Goa']
Gujrat = ['Gujrat', "Ahmedabad" ,"Amreli" ,"Anand" ,"Aravalli" ,"Banaskantha" ,"Bharuch" ,"Bhavnagar" ,"Botad" ,"Chhota Udaipur" ,"Dahod" ,"Dang" ,"Devbhoomi Dwarka" ,"Gandhinagar" ,"Gir Somnath" ,"Jamnagar" ,"Junagadh" ,"Kheda" ,"Kutch" ,"Mahisagar" ,"Mehsana" ,"Morbi" ,"Narmada" ,"Navsari" ,"Panchmahal" ,"Patan" ,"Porbandar" ,"Rajkot" ,"Sabarkantha" ,"Surat" ,"Surendranagar" ,"Tapi" ,"Vadodara" ,"Valsad"]
Haryana = ['Haryana', "Ambala" ,"Bhiwani" ,"Charkhi Dadri" ,"Faridabad" ,"Fatehabad" ,"Gurugram" ,"Hisar" ,"Jhajjar" ,"Jind" ,"Kaithal" ,"Karnal" ,"Kurukshetra" ,"Mahendragarh" ,"Mewat" ,"Palwal" ,"Panchkula" ,"Panipat" ,"Rewari" ,"Rohtak" ,"Sirsa" ,"Sonipat" ,"Yamunanagar"]
HimachalPradesh = ['Himachal',"Bilaspur" ,"Chamba" ,"Hamirpur" ,"Kangra" ,"Kinnaur" ,"Kullu" ,"Lahaul Spiti" ,"Mandi" ,"Shimla" ,"Sirmaur" ,"Solan" ,"Una" ]
JammuKashmir = ["Jammu", "Kashmir", "J&K", "JK", "Anantnag" ,"Bandipora" ,"Baramulla" ,"Budgam" ,"Doda" ,"Ganderbal" ,"Jammu" ,"Kathua" ,"Kishtwar" ,"Kulgam" ,"Kupwara" ,"Poonch" ,"Pulwama" ,"Rajouri" ,"Ramban" ,"Reasi" ,"Samba" ,"Shopian" ,"Srinagar" ,"Udhampur"]
Jharkand = ["Jharkand", "Bokaro" ,"Chatra" ,"Deoghar" ,"Dhanbad" ,"Dumka" ,"East Singhbhum" ,"Garhwa" ,"Giridih" ,"Godda" ,"Gumla" ,"Hazaribagh" ,"Jamtara" ,"Khunti" ,"Koderma" ,"Latehar" ,"Lohardaga" ,"Pakur" ,"Palamu" ,"Ramgarh" ,"Ranchi" ,"Sahebganj" ,"Seraikela Kharsawan" ,"Simdega" ,"West Singhbhum"]
Karnataka = ['Karnataka',"Bagalkot" ,"Bangalore Rural" ,"Bangalore Urban" ,"Belgaum" ,"Bellary" ,"Bidar" ,"Chamarajanagar" ,"Chikkaballapur" ,"Chikkamagaluru" ,"Chitradurga" ,"Dakshina Kannada" ,"Davanagere" ,"Dharwad" ,"Gadag" ,"Gulbarga" ,"Hassan" ,"Haveri" ,"Kodagu" ,"Kolar" ,"Koppal" ,"Mandya" ,"Mysore" ,"Raichur" ,"Ramanagara" ,"Shimoga" ,"Tumkur" ,"Udupi" ,"Uttara Kannada" ,"Vijayanagara" ,"Vijayapura" ,"Yadgir"]
Kerala = ['Kerala', "Alappuzha" ,"Ernakulam" ,"Idukki" ,"Kannur" ,"Kasaragod" ,"Kollam" ,"Kottayam" ,"Kozhikode" ,"Malappuram" ,"Palakkad" ,"Pathanamthitta" ,"Thiruvananthapuram" ,"Thrissur" ,"Wayanad"]
Ladakh = ['Ladakh','leh','kargil']
Lakshadweep = ['Lakshadweep']
MadhyaPradesh = ["Madhya", "Agar Malwa" ,"Alirajpur" ,"Anuppur" ,"Ashoknagar" ,"Balaghat" ,"Barwani" ,"Betul" ,"Bhind" ,"Bhopal" ,"Burhanpur" ,"Chachaura" ,"Chhatarpur" ,"Chhindwara" ,"Damoh" ,"Datia" ,"Dewas" ,"Dhar" ,"Dindori" ,"Guna" ,"Gwalior" ,"Harda" ,"Hoshangabad" ,"Indore" ,"Jabalpur" ,"Jhabua" ,"Katni" ,"Khandwa" ,"Khargone" ,"Maihar" ,"Mandla" ,"Mandsaur" ,"Morena" ,"Nagda" ,"Narsinghpur" ,"Neemuch" ,"Niwari" ,"Panna" ,"Raisen" ,"Rajgarh" ,"Ratlam" ,"Rewa" ,"Sagar" ,"Satna" ,"Sehore" ,"Seoni" ,"Shahdol" ,"Shajapur" ,"Sheopur" ,"Shivpuri" ,"Sidhi" ,"Singrauli" ,"Tikamgarh" ,"Ujjain" ,"Umaria" ,"Vidisha"]
Maharashtra = ["Maharashtra", "Bombay", "Ahmednagar" ,"Akola" ,"Amravati" ,"Aurangabad" ,"Beed" ,"Bhandara" ,"Buldhana" ,"Chandrapur" ,"Dhule" ,"Gadchiroli" ,"Gondia" ,"Hingoli" ,"Jalgaon" ,"Jalna" ,"Kolhapur" ,"Latur" ,"Mumbai" ,"Mumbai Suburban" ,"Nagpur" ,"Nanded" ,"Nandurbar" ,"Nashik" ,"Osmanabad" ,"Palghar" ,"Parbhani" ,"Pune" ,"Raigad" ,"Ratnagiri" ,"Sangli" ,"Satara" ,"Sindhudurg" ,"Solapur" ,"Thane" ,"Wardha" ,"Washim" ,"Yavatmal"]
Manipur = ['Manipur', "Bishnupur" ,"Chandel" ,"Churachandpur" ,"Imphal East" ,"Imphal West" ,"Jiribam" ,"Kakching" ,"Kamjong" ,"Kangpokpi" ,"Noney" ,"Pherzawl" ,"Senapati" ,"Tamenglong" ,"Tengnoupal" ,"Thoubal" ,"Ukhrul"]
Meghalaya = ['Megh', "East Garo Hills" ,"East Jaintia Hills" ,"East Khasi Hills" ,"Mairang" ,"North Garo Hills" ,"Ri Bhoi" ,"South Garo Hills" ,"South West Garo Hills" ,"South West Khasi Hills" ,"West Garo Hills" ,"West Jaintia Hills" ,"West Khasi Hills"]
Mizoram = ['Mizoram', "Aizawl" ,"Champhai" ,"Hnahthial" ,"Khawzawl" ,"Kolasib" ,"Lawngtlai" ,"Lunglei" ,"Mamit" ,"Saiha" ,"Saitual" ,"Serchhip" ]
Nagaland = ['Nagaland', "Chumukedima" ,"Dimapur" ,"Kiphire" ,"Kohima" ,"Longleng" ,"Mokokchung" ,"Niuland" ,"Noklak" ,"Peren" ,"Phek" ,"Tseminyu" ,"Tuensang" ,"Wokha" ,"Zunheboto"]
Odisha = ['Odisha', "Angul" ,"Balangir" ,"Balasore" ,"Bargarh" ,"Bhadrak" ,"Boudh" ,"Cuttack" ,"Debagarh" ,"Dhenkanal" ,"Gajapati" ,"Ganjam" ,"Jagatsinghpur" ,"Jajpur" ,"Jharsuguda" ,"Kalahandi" ,"Kandhamal" ,"Kendrapara" ,"Kendujhar" ,"Khordha" ,"Koraput" ,"Malkangiri" ,"Mayurbhanj" ,"Nabarangpur" ,"Nayagarh" ,"Nuapada" ,"Puri" ,"Rayagada" ,"Sambalpur" ,"Subarnapur" ,"Sundergarh"]
Puducherry = ['Puducherry', "Karaikal" ,"Mahe" ,"Puducherry" ,"Yanam" ]
Punjab = ['Punjab', "Amritsar" ,"Barnala" ,"Bathinda" ,"Faridkot" ,"Fatehgarh Sahib" ,"Fazilka" ,"Firozpur" ,"Gurdaspur" ,"Hoshiarpur" ,"Jalandhar" ,"Kapurthala" ,"Ludhiana" ,"Malerkotla" ,"Mansa" ,"Moga" ,"Mohali" ,"Muktsar" ,"Pathankot" ,"Patiala" ,"Rupnagar" ,"Sangrur" ,"Shaheed Bhagat Singh Nagar" ,"Tarn Taran"]
Rajasthan = ['Rajasthan', "Ajmer" ,"Alwar" ,"Banswara" ,"Baran" ,"Barmer" ,"Bharatpur" ,"Bhilwara" ,"Bikaner" ,"Bundi" ,"Chittorgarh" ,"Churu" ,"Dausa" ,"Dholpur" ,"Dungarpur" ,"Hanumangarh" ,"Jaipur" ,"Jaisalmer" ,"Jalore" ,"Jhalawar" ,"Jhunjhunu" ,"Jodhpur" ,"Karauli" ,"Kota" ,"Nagaur" ,"Pali" ,"Pratapgarh" ,"Rajsamand" ,"Sawai Madhopur" ,"Sikar" ,"Sirohi" ,"Sri Ganganagar" ,"Tonk" ,"Udaipur"]
Sikkim = ['Sikkim','Soreng','Pakyong']
TamilNadu = ["Tamil Nadu", "Ariyalur" ,"Chengalpattu" ,"Chennai" ,"Coimbatore" ,"Cuddalore" ,"Dharmapuri" ,"Dindigul" ,"Erode" ,"Kallakurichi" ,"Kanchipuram" ,"Kanyakumari" ,"Karur" ,"Krishnagiri" ,"Madurai" ,"Mayiladuthurai" ,"Nagapattinam" ,"Namakkal" ,"Nilgiris" ,"Perambalur" ,"Pudukkottai" ,"Ramanathapuram" ,"Ranipet" ,"Salem" ,"Sivaganga" ,"Tenkasi" ,"Thanjavur" ,"Theni" ,"Thoothukudi" ,"Tiruchirappalli" ,"Tirunelveli" ,"Tirupattur" ,"Tiruppur" ,"Tiruvallur" ,"Tiruvannamalai" ,"Tiruvarur" ,"Vellore" ,"Viluppuram" ,"Virudhunagar"]
Telangana = ['Telangana',"Adilabad" ,"Bhadradri Kothagudem" ,"Hanamkonda" ,"Hyderabad" ,"Jagtial" ,"Jangaon" ,"Jayashankar" ,"Jogulamba" ,"Kamareddy" ,"Karimnagar" ,"Khammam" ,"Komaram Bheem" ,"Mahabubabad" ,"Mahbubnagar" ,"Mancherial" ,"Medak" ,"Medchal" ,"Mulugu" ,"Nagarkurnool" ,"Nalgonda" ,"Narayanpet" ,"Nirmal" ,"Nizamabad" ,"Peddapalli" ,"Rajanna Sircilla" ,"Ranga Reddy" ,"Sangareddy" ,"Siddipet" ,"Suryapet" ,"Vikarabad" ,"Wanaparthy" ,"Warangal" ,"Yadadri Bhuvanagiri"]
Tripura = ['Tripura',"Dhalai" ,"Gomati" ,"Khowai" ,"North Tripura" ,"Sepahijala" ,"South Tripura" ,"Unakoti" ,"West Tripura" ]
Uttarakhand = ['Uttarakhand',"Almora" ,"Bageshwar" ,"Chamoli" ,"Champawat" ,"Dehradun" ,"Haridwar" ,"Nainital" ,"Pauri" ,"Pithoragarh" ,"Rudraprayag" ,"Tehri" ,"Udham Singh Nagar" ,"Uttarkashi"]
WestBengal = ['Bengal', "Alipurduar" ,"Bankura" ,"Birbhum" ,"Cooch Behar" ,"Dakshin Dinajpur" ,"Darjeeling" ,"Hooghly" ,"Howrah" ,"Jalpaiguri" ,"Jhargram" ,"Kalimpong" ,"Kolkata" ,"Malda" ,"Murshidabad" ,"Nadia" ,"North 24 Parganas" ,"Paschim Bardhaman" ,"Paschim Medinipur" ,"Purba Bardhaman" ,"Purba Medinipur" ,"Purulia" ,"South 24 Parganas" ,"Uttar Dinajpur"]

data ={
    "States":['Andaman Nicobar', 'Andhra Pradesh', 'Arunachal Pradesh',
        'Assam', 'Bihar', 'Chandigarh',
        'Chhattisgarh', 'Dadra Nagar',
        'Delhi', 'Goa', 'Gujarat', 'Haryana', 'Himachal Pradesh',
        'Jammu Kashmir', 'Jharkhand', 'Karnataka', 'Kerala',
            'Ladakh', 'Lakshadweep', 'Madhya Pradesh', 'Maharashtra',
            'Manipur', 'Meghalaya', 'Mizoram', 'Nagaland', 'Odisha',
            'Puducherry', 'Punjab', 'Rajasthan', 'Sikkim', 'Tamil Nadu'
            , 'Telangana', 'Tripura', 'Uttar Pradesh', 
            'Uttarakhand', 'West Bengal'],
    "Reported":[hd.countOccurancesAny(AndamanNicobar),hd.countOccurancesAny(AndhraPradesh),hd.countOccurancesAny(ArunachalPradesh),
    hd.countOccurancesAny(Assam),hd.countOccurancesAny(Bihar),hd.countOccurancesAny(Chandigarh),
    hd.countOccurancesAny(Chhattisgarh),hd.countOccurancesAny(Dadra),hd.countOccurancesAny(Delhi),
    hd.countOccurancesAny(Goa),hd.countOccurancesAny(Gujrat),hd.countOccurancesAny(Haryana),
    hd.countOccurancesAny(HimachalPradesh),hd.countOccurancesAny(JammuKashmir),hd.countOccurancesAny(Jharkand),
    hd.countOccurancesAny(Karnataka),hd.countOccurancesAny(Kerala),hd.countOccurancesAny(Ladakh),
    hd.countOccurancesAny(Lakshadweep),hd.countOccurancesAny(MadhyaPradesh),hd.countOccurancesAny(Maharashtra),
    hd.countOccurancesAny(Manipur),hd.countOccurancesAny(Meghalaya),
    hd.countOccurancesAny(Mizoram),hd.countOccurancesAny(Nagaland),hd.countOccurancesAny(Odisha),
    hd.countOccurancesAny(Puducherry),hd.countOccurancesAny(Punjab),hd.countOccurancesAny(Rajasthan),
    hd.countOccurancesAny(Sikkim),hd.countOccurancesAny(TamilNadu),hd.countOccurancesAny(Telangana),
    hd.countOccurancesAny(Tripura),hd.countOccurancesAny(UttarPradesh),hd.countOccurancesAny(Uttarakhand),hd.countOccurancesAny(WestBengal)]
}

df = pd.DataFrame(data)
print(df.sort_values("Reported",ascending=False))
print(df)

#                 States  Reported
#   33      Uttar Pradesh     19712
#   20        Maharashtra     11281
#   8               Delhi      6031
#   13      Jammu Kashmir      5658
#   10            Gujarat      4998
#   12   Himachal Pradesh      4335
#   4               Bihar      3712
#   27             Punjab      3627
#   35        West Bengal      3310
#   19     Madhya Pradesh      2543
#   28          Rajasthan      2293
#   9                 Goa      2170
#   11            Haryana      2123
#   1      Andhra Pradesh      2030
#   16             Kerala      2025
#   6        Chhattisgarh      1860
#   5          Chandigarh      1860
#   15          Karnataka      1812
#   30         Tamil Nadu      1799
#   3               Assam      1637
#   25             Odisha      1484
#   31          Telangana      1285
#   17             Ladakh      1158
#   7         Dadra Nagar       587
#   21            Manipur       575
#   34        Uttarakhand       563
#   22          Meghalaya       409
#   2   Arunachal Pradesh       394
#   14          Jharkhand       340
#   26         Puducherry       307
#   32            Tripura       199
#   24           Nagaland       185
#   29             Sikkim       160
#   23            Mizoram       144
#   0     Andaman Nicobar       102
#   18        Lakshadweep        29

Now we will proceed to combine the above code to produce statistics for state wise crime reports grouped by type of crime.

I used the following code to derive values for the analysis we are performing, replacing crimeMerged with the combination of crimes I am analysing.

crimeMerged =  murder

data ={
    "States":['Andaman Nicobar', 'Andhra Pradesh', 'Arunachal Pradesh',
        'Assam', 'Bihar', 'Chandigarh',
        'Chhattisgarh', 'Dadra Nagar',
        'Delhi', 'Goa', 'Gujarat', 'Haryana', 'Himachal Pradesh',
        'Jammu Kashmir', 'Jharkhand', 'Karnataka', 'Kerala',
            'Ladakh', 'Lakshadweep', 'Madhya Pradesh', 'Maharashtra',
            'Manipur', 'Meghalaya', 'Mizoram', 'Nagaland', 'Odisha',
            'Puducherry', 'Punjab', 'Rajasthan', 'Sikkim', 'Tamil Nadu'
            , 'Telangana', 'Tripura', 'Uttar Pradesh', 
            'Uttarakhand', 'West Bengal'],
    "Reported":[hd.countOccurancesGroupedAny([AndamanNicobar,crimeMerged]),hd.countOccurancesGroupedAny([AndhraPradesh,crimeMerged]),hd.countOccurancesGroupedAny([ArunachalPradesh,crimeMerged]),
    hd.countOccurancesGroupedAny([Assam,crimeMerged]),hd.countOccurancesGroupedAny([Bihar,crimeMerged]),hd.countOccurancesGroupedAny([Chandigarh,crimeMerged]),
    hd.countOccurancesGroupedAny([Chhattisgarh,crimeMerged]),hd.countOccurancesGroupedAny([Dadra,crimeMerged]),hd.countOccurancesGroupedAny([Delhi,crimeMerged]),
    hd.countOccurancesGroupedAny([Goa,crimeMerged]),hd.countOccurancesGroupedAny([Gujrat,crimeMerged]),hd.countOccurancesGroupedAny([Haryana,crimeMerged]),
    hd.countOccurancesGroupedAny([HimachalPradesh,crimeMerged]),hd.countOccurancesGroupedAny([JammuKashmir,crimeMerged]),hd.countOccurancesGroupedAny([Jharkand,crimeMerged]),
    hd.countOccurancesGroupedAny([Karnataka,crimeMerged]),hd.countOccurancesGroupedAny([Kerala,crimeMerged]),hd.countOccurancesGroupedAny([Ladakh,crimeMerged]),
    hd.countOccurancesGroupedAny([Lakshadweep,crimeMerged]),hd.countOccurancesGroupedAny([MadhyaPradesh,crimeMerged]),hd.countOccurancesGroupedAny([Maharashtra,crimeMerged]),
    hd.countOccurancesGroupedAny([Manipur,crimeMerged]),hd.countOccurancesGroupedAny([Meghalaya,crimeMerged]),
    hd.countOccurancesGroupedAny([Mizoram,crimeMerged]),hd.countOccurancesGroupedAny([Nagaland,crimeMerged]),hd.countOccurancesGroupedAny([Odisha,crimeMerged]),
    hd.countOccurancesGroupedAny([Puducherry,crimeMerged]),hd.countOccurancesGroupedAny([Punjab,crimeMerged]),hd.countOccurancesGroupedAny([Rajasthan,crimeMerged]),
    hd.countOccurancesGroupedAny([Sikkim,crimeMerged]),hd.countOccurancesGroupedAny([TamilNadu,crimeMerged]),hd.countOccurancesGroupedAny([Telangana,crimeMerged]),
    hd.countOccurancesGroupedAny([Tripura,crimeMerged]),hd.countOccurancesGroupedAny([UttarPradesh,crimeMerged]),hd.countOccurancesGroupedAny([Uttarakhand,crimeMerged]),hd.countOccurancesGroupedAny([WestBengal,crimeMerged])]
}

df = pd.DataFrame(data)
print(df.sort_values("Reported",ascending=False))

The visualization in forms of a heat map for the 3 heavies (in terms of the range of dataset) is as follows:


Here darker colours represent heavier intensity of the labelled crime.

This is a similar heatmap which encompasses a general image of crime intensity over the Indian states:


Map of India: Warmer colours represent heavier intensity.

The crime news statistics actually correspond to the real crime stats of the states as per government data. The only exception are the southern states. The southern states show much lower numbers in journalistic data compared to actual govt data. This raises an interesting question on what the reason can be? Is it journalistic bias? Unfortunately I am not qualified to answer that question but I do encourage you to find the answer to this question.

The Three Khans of Bollywood

I still remember when a famous saying used to be "Bollywood is ruled by the Three Khans". Well to take a lighter (away from crime and politics) look at the data, lets see which khan ends up the most popular amongst the three.

Here is the code which was run to find the most popular amongst the three.

srk = ["srk","shah rukh"]
salman = ["salman","sallu"]
amir = ["aamir",]

data = {"Actor":["SRK","Salman","Amir"],
"Reported":[hd.countOccurancesAny(srk),hd.countOccurancesAny(salman),hd.countOccurancesAny(amir)]
}

df = pd.DataFrame(data)
print(df.sort_values("Reported",ascending=False))

#     Actor  Reported
# 1  Salman       335
# 0     SRK       304
# 2    Amir       130

As it turns out its the "Bhai" of the industry. To be very honest it makes sense with how crazy his fans are.

PART 4: Epilogue

The aim of this endeavour was to make available a tool to be able to perform analysis like the ones shown above. And in that I have succeeded.

The scripts for the Web Scraper and the Headline Analysis Functions is available on the project repository.

It was extremely fun and informative working on this project. I learned and gained experience with the following:

Web Scraping (BeautifulSoup, requests)
Keyword Analysis
Data Visualization
Data Analysis (pandas, matplotlib)
Open Source and Version Control

Thankyou very much for reading through the whole write-up. I really hope you found it interesting. Please leave a comment sharing any feedback.

Hindi TV Serials: A Data Analysis

Subh Chaturvedi — Thu, 19 May 2022 17:35:46 +0000


Credit: ZeeMELT and Kyoorius

Introduction

TV Serials and family dramas have a special place in every Indian’s heart. Nothing can ever replace the iconic “Dhum Ta Terenana” score that amplifies the tension in the air or the “Saas Bahu” dramatic tropes introduced into the Indian Entertainment Industry by these TV Serials.

From classics like “Saas Bhi Kabhi Bahu Thi” and “Sasural Simar Ka” to modern entries like “Shark Tank”, this industry and this culture is ever-evolving and uniquely creative.

Its only fitting then, that when I found a dataset about Hindi TV Serials, I immediately decided to do this analysis and draw some interesting insights from it.

The Dataset

Let us start with looking at the dataset I am going to be using for this analysis project. This dataset titled “Hindi TV Serials” contains almost 800 unique values with the name of the serial, its cast, its IMDB rating and an overview.

It contains all the TV Serials aired on the following channels from 1988 to the present day (May 2022):

Sab TV
Sony TV
Colors TV
StarPlus
Zee TV

Technically the dataset is distributed as a CSV file (181.76kB) and has 736 unique values spread of the following columns:

Name
Ratings
genres
overview
Year
Cast

Example Values from the Dataset

Name	Ratings	genres	overview	Year	Cast
Kyunki Saas Bhi Kabhi Bahu Thi	1.6	"Comedy, Drama, Family"	A mother-in-law's struggle to put up with her three bahu's. The three bahu's have grown up sons. The bahu's sons start to get involved with having girlfriends and the bahu's try and break their relationships up.	2000–2008	"Smriti Malhotra-Irani ,Ronit Roy ,Amar Upadhyay ,Sudha Shivpuri"
Kahaani Ghar Ghar Kii	2.1	Drama	"The show explored the worlds of its protagonists Parvati Aggarwal and Om Aggarwal, who live in a joint family where by Parvati is an ideal daughter-in-law of Aggarwal family and Om the ideal son."	2000–2008	"Sakshi Tanwar ,Kiran Karmarkar ,Mita Vashisht ,Ali Asgar"

I will be analyzing the relationships and the insights that each of the column provides when properly cleaned and arranged.

Setting up the Environment

I start with importing the necessary modules for this project:

pandas
numpy
matplotlib

Then the dataset is imported into the environment through the read.csv method.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

dfmain = pd.read_csv("Hindi TV Serials.csv")

The IMDB ratings

The IMDB ratings are going to be very important throughout this analysis as a way to judge the quality and popularity of a TV Show whenever applicable.

But before we dive-in into how other parameters relate and affect the IMDB rating of a show, let us independently look at these ratings.

Top 5 shows by IMDB ratings

We use the sort_values() function to get an output of the top shows according their IMDB ratings.

print(dfmain.sort_values(["Ratings"], ascending=False))

Output:

                                    Name  Ratings  ...       Year                                               Cast
407                  Mitegi Laxman Rekha      9.7  ...       2018  Aayesha Vindhara ,Ankita Goraya ,Rajeev Saxena...
242                    Shobha Somnath Ki      9.4  ...  2011–2012  Ashnoor Kaur ,Tarun Khanna ,Joy Rattan Singh M...
79                        Love U Zindagi      9.4  ...       2011                                                NaN
586                      Wagle Ki Duniya      9.2  ...     2021–   Sumeet Raghavan ,Pariva Pranati ,Sheehan Kapah...
742  Jagannath Aur Purvi Ki Dosti Anokhi      9.2  ...     2022–   Rajendra Gupta ,Sushmita Mukherjee ,Ismeet Koh...
..                                   ...      ...  ...        ...                                                ...
(remaining output omitted due to irrelevancy)

As is clearly discernible, the top 5 shows according to their ratings are:

Mitegi Laxman Rekha (9.7)
Shobha Somnath Ki (9.4)
Love U Zindagi (9.4)
Wagle Ki Duniya (9.2)
Jagannath Aur Purvi ki Dosti Anokhi (9.2)

Well I am not sure I agree with these results but well if you say so IMDB, if you say so...

The Cast and The Artists

Analyzing the cast column can provide some interesting statistics to look at, but there is a serious problem that limits us from using it to any useful extent.

The problem is the format in which these values are stored in the dataset.

For example take the value for the "Cast" column in the row for Shobha Somnath Ki:
|Cast|
|---|
|Ashnoor Kaur ,Tarun Khanna ,Joy Rattan Singh Mathur ,Sandeep Arora|

This value is troublesome as it is stored as a single <str> type object and thus it is not possible to calculate or discern any data for individual cast members.

Cleaning Data: Solving the Cast Problem

Thankfully, as elaborated by Max Hilsdorf in his Medium blog, the string object present in the cell can be converted into a list object, and subsequently into a one dimensional data type that can allow functions like value_counts() and groupby() to function.

But his solution does not apply to our problem without extensive modifications as the values we wish to convert to a list do not have any pre-existent list based syntax. Therefore we need to convert each cell in the Cast Column into a value based on list syntax i.e. ["a","b","c",...].

We can implement this by writing a function the takes input in the format that we have and then adding the square brackets and the quotation marks and returning it in the format that we need. This is my implementation of such a function:

def clean_artist_list(list_):
    if type(list_) is str:
        list_ = "[" + list_ + "]"
        list_ = list_.replace(',', '","')
        list_ = list_.replace('[', '["')
        list_ = list_.replace(']', '"]')
        list_ = list_.replace(' "', '"')
        return list_
    else:
        return "[]"

This function also takes care to properly handle and replace any disruptive data. I mainly encountered some FLOAT datatypes which threw errors as they could not be treated like strings.

After applying this function and the python eval() function, we have the required list datatypes.

dfmain["Cast"] = dfmain["Cast"].apply(clean_artist_list)
dfmain["Cast"] = dfmain["Cast"].apply(eval)

Before proceeding we also need to create the function needed to convert these 2D lists to 1D. For that we will use:

def to_1D(series):
    return pd.Series([x for _list in series for x in _list])

Top Rated Artist

Now that we can use the Cast data properly, lets find out which artist has the best average IMDB ratings for the shows they worked in.

df_cast_imdb = dfmain.groupby(to_1D(dfmain["Cast"])).mean()
print(df_cast_imdb.sort_values(["Ratings"],ascending=False))

Output:

                Ratings
Tusharr Khanna      9.2
Sahil Mehta         9.2
Vrajesh Hirjee      9.2
Gautami Kapoor      9.1
Vaidehi Amrute      9.1
...                 ...
(remaining output omitted due to irrelevancy)

The artists with the best mean IMDB rating for his shows is Tushar Khanna. He has worked in "Pyaar Tune Kia Kya", "Piyaa Albela" and "Bekaboo".

This however does not necessarily reflect any superiority in acting or talent, but it may show (at least to people who believe in it) some signs of luck an artist brings to a set.

Most Experienced Artist

Now moving to a more concrete relation. We will be finding out which actor has worked in the most TV shows.
It should be noted that the values of this dataset only list the leading cast members in the cast section and thus artist with minor roles are not properly recognized in this analysis.

print(to_1D(dfmain["Cast"]).value_counts())

Output:

Ronit Roy            9
Jennifer Winget      8
Seema Kapoor         7
Sangeeta Ghosh       7
Shahab Khan          7
                    ..
(remaining output omitted due to irrelevancy)

Ronit Roy having worked in 9 shows, comes out to be the most experienced artist in this dataset. No wonder I see him in every other serious father type role.

Genre

Its either comedy (the family kind) or drama (also the family kind) with Indian TV Serials. But don't take my word for it, let us see for ourselves the genre dynamics of Indian TV.

Cleaning Data: Genre

Genres also face the same problem as we faced above with artists. There is a small edit made to handle redundancies due to whitespace characters.

def clean_genre_list(list_):
    if type(list_) is str:
        list_ = "[" + list_ + "]"
        list_ = list_.replace(',', '","')
        list_ = list_.replace('[', '["')
        list_ = list_.replace(']', '"]')
        list_ = list_.replace(' "', '"')
        list_ = list_.replace(" ","")
        return list_
    else:
        return "[]"

It is then used similarly as the Cast solution.

dfmain["genres"] = dfmain["genres"].apply(clean_genre_list)
dfmain["genres"] = dfmain["genres"].apply(eval)

Most Acclaimed Genre

First lets look at which genre claims the best mean IMDB ratings and garners the best critic response.

df_genre_imdb = dfmain.groupby(to_1D(dfmain["genres"])).mean()
print(df_genre_imdb.sort_values(["Ratings"],ascending=False))

Output:

              Ratings
War          6.900000
Horror       6.684211
Adventure    6.680000
Biography    6.650000
Sport        6.500000
Family       6.443478
Crime        6.271429
History      6.162500
Action       5.966667
Comedy       5.961644
(remaining output omitted due to irrelevancy)

Humans do love war, huh.

Bigger Genre

Next lets look at which genre the creators love the most and thus create the most shows based around.

df_genre_count = to_1D(dfmain["genres"]).value_counts()
print(df_genre_count)

df_genre_count.plot(kind = 'bar')
plt.show()

Instead of the text output, a visual representation of the output would be more suitable here, thus we generate a bar graph using the Series.plot() function.

Output:

So THAT is why Indian households end up being so dramatic...

Release Year

Shows like "Sarabhai vs Sarabhai" were definitely much ahead of their time. But lets look at how time affected the rest of the Indian TV.

Cleaning Data: Years

To make use of the data in the Years column, we need to convert it into forms that are not haphazard and unusable like it originally is.

I created two new columns based on the Years column:

First Year: This column tracks the year in which the show started airing.
Years Run: This column tracks how long a show ran.

These columns were created with the following code:

def findstart(list_):
    if type(list_) is str:
        list_ = list_[:4]
        return list_
    else:
        return ""

def duration(list_):
    if type(list_) is str:
        if len(list_) == 9 and list_[0]!="I":
            l1 = int(list_[:4])
            l2 = int(list_[5:])
            return l2-l1
        else:
            return 0
    else:
        return 0

dfmain["First Year"] = dfmain["Year"].apply(findstart)
dfmain["Years Run"] = dfmain["Year"].apply(duration)

The code was made to handle edge cases like wrong datatype and the weird "I XX" values in the Year column.

Busiest Year

Which year was the busiest for the creators? We can use the following code to visualize the frequency of productions across years.

df_year_count = dfmain["First Year"].value_counts().sort_index()
df_year_count = df_year_count.iloc[:-4] #removing the weird I values

df_year_count.plot(kind = 'bar')
plt.show()

Output:

2017 brought us shows like "Naagin 2", "Yeh Rishta Kya Kehlata Hai" and "Yeh Hein Mohabbatein". In total it records the production of 59 shows compared to the runner up 2018 with 46 shows.

Longest Running Show

Indian shows like "Sasural Simar Ka" and "Kyunki Saas Bhi Kabhi Bahu Thi" are infamous for running long enough to be part of a late teenager's life since birth. So its obvious to find out which show actually has the longest runtime.

print(dfmain.sort_values(["Years Run"], ascending=False))

Output:

                                          Name  Ratings  ... First Year Years Run
720                                     C.I.D.      6.8  ...       1998        20
255                                 Hum Paanch      8.2  ...       1995        11
536                                   Yes Boss      8.4  ...       1999        10
0               Kyunki Saas Bhi Kabhi Bahu Thi      1.6  ...       2000         8
1                        Kahaani Ghar Ghar Kii      2.1  ...       2000         8
..                                         ...      ...  ...        ...       ...
(remaining output omitted due to irrelevancy)

"C.I.D." is no-doubt part of every Indian's life. With iconic characters like ACP Pradyuman, Abhijit, and Daya, and a premise revolving around crime in India, its not a surprise that it had a runtime of 20 years.

Analyzing the Overviews

Here comes the part I was most excited for. The written descriptions and overviews of these shows could surely provide me some very interesting insights that could have been the highlights of this project.

Unfortunately after cleaning the data and writing the code to analyze it, it was shocking to see how useless the ordeal was. The data did was not sufficient and quality enough to let me draw any real conclusions from it.

But I will still show the method I used to clean and try analyzing the data.

Cleaning Data: Description

Similar to the approach I took for the problems with other columns, I decided to convert the string based values to a list with every word being an element of the list. Also additionally the words were all turned to lowercase and any special characters were removed so as to make sure that redundancy was minimized.

def clean_ovw_list(list_):

    if type(list_) is str:
        list_ = "[" + list_ + "]"

        #removing all the special characters
        list_ = list_.replace(',', '')
        list_ = list_.replace('.', '')
        list_ = list_.replace('"', '')
        list_ = list_.replace('(', '')
        list_ = list_.replace(')', '')
        list_ = list_.replace('-', '')
        list_ = list_.replace('»', '')

        list_ = list_.replace(' ', '","')
        list_ = list_.replace('[', '["')
        list_ = list_.replace(']', '"]')
        list_ = list_.replace(' "', '"')

        #converting to lower case
        list_ = list_.lower()
        return list_
    else:
        return "[]"

The function was applied:

dfmain["overview"] = dfmain["overview"].apply(clean_ovw_list)
dfmain["overview"] = dfmain["overview"].apply(eval)

Now we have data that we can supposedly work on.

Usage of words over time

I planned to analyze multiple words like "love", "hate", "mother", "mother-in-law", "brother", etc. and their usage over time in the descriptions of TV Serials and even plot graphs showing interesting relations between the trends of different words.

This code gives the count of the words used grouped by years:

df_ovwcount = dfmain.groupby(['First Year',to_1D(dfmain["overview"])]).count().reset_index()

The following code could be used to plot the variance of occurance of words overtime, and also to show contrast in different words.

#Selecting and plotting the first word
df_selectedword = df_ovwcount[df_ovwcount["level_1"].isin(["First Word"])]
plt.plot(df_selectedword["First Year"],df_selectedword["overview"])

#Selecting and plotting the second word
df_selectedword = df_ovwcount[df_ovwcount["level_1"].isin(["Second Word"])]
plt.plot(df_selectedword["First Year"],df_selectedword["overview"])

plt.xticks(rotation=90)
plt.show()

A visualization generated through this code (provided better data) could have looked like this:

This data could have led to a lot of other interesting analysis too, but unfortunately it was not possible.

Most Used Word

We can still draw some simple insights from this data. Let us find out the 50 most used words in the descriptions for Indian TV Serials.

df_ovw_count_simple = to_1D(dfmain["overview"]).value_counts()
print(df_ovw_count_simple.head(50))

Output:

            1843
a            856
the          848
and          647
of           588
to           394
is           338
her          314
in           302
who          201
with         191
story        185
their        158
his          140
on           129
family       128
love         125
an           125
plot         119
add          118
see          117
full         117
summary      114
for          113
from         111
life         107
she          105
by           103
girl          84
as            79
that          79
two           76
are           73
show          72
they          71
but           71
when          66
young         57
about         57
around        56
this          53
lives         52
it            51
has           49
he            49
married       47
series        47
one           44
other         42
revolves      41

Some significant meaningful words come out to be "family", "love" and "life"... That is some Fast & Furious philosophy it seems.

Conclusion

Indian TV is definitely an interesting place to observe and analyze. This project aimed at looking at some of the angles of the vast possibilities that are present with proper datasets.

But the tip of the iceberg that we touched also gave us some interesting results:

Top 5 Indian TV Shows by IMDB Rating.
Artists with the best mean IMDB Rating.
Artists with the most experience.
Genre with the best mean IMDB Rating.
Genre with the most available content.
The release frequency of shows over the years.
The longest running shows.
Usage of certain words in the overviews of TV shows over time.
Most used words in TV Show descriptions.

This project also helped me cement my skills in data analysis, especially learning how to analyze a varied dataset in multi-faceted fashion.

I also gained experience cleaning data and how to treat list like values in cells and treat elements individually.

Thankyou to everyone who actually stuck with reading till here, it was very fun for me to work on this project.

Roadmap Series: Competitive Programming

Subh Chaturvedi — Sun, 08 May 2022 16:34:07 +0000

Competitive Programming is a sport. Consider a programming contest as a contest. Write a piece of code that should satisfy the problem presented and submit, you may get a WA (Wrong Answer). Make variations to code and eventually you will get your precious AC (Accepted/Correct Answer).

Here is how you can master Competitive Programming:

Choosing the right language:

The first step to take is to choose the right language to start your competitive journey with. You may try new languages and even find some other language suiting you better. The commonly used and recommended languages are:

Python: It is one of the easiest to learn language and has very simple syntax to implement. It has wide usage in domains like Machine Learning, Artificial Intelligence and Data Science if you decide to explore that domain.
Java: It is a very highly recommended language by coding communities. It is one of the most preferred languages for competitive programming and easy one.
JavaScript: It is the most popular scripting language and a staple of web-development. Many popular websites like Facebook, Twitter, YouTube, and Gmail use JavaScript to create interactive web pages and flexibly show content to users.
C: It is based on POP (Procedural Oriented Programming) where it divides programs into small functions.
C++: It is the most popular and recommendable programming language. C++ template includes data structures like Lists, Tree, Stacks, Graphs, Arrays etc. It follows OOPS (Object Oriented\ Programming) which help you to solve real time coding challenges.
Go: It is a low level language, open-source language preferential by Google. Useful to build web servers, data pipelines etc.
Ruby: It is an object-oriented language, user friendly, easy to learn. Dynamic Programming Language and also based on other programming languages like Perl, Lisp, Ada, Eiffel etc. It is used for Web Applications, Database Solutions and Data Science etc.

And many more. Once you learn any language, start practicing it. You can practice these languages from any website (I recommend Hackerrank). You can find these websites from search engines. Practice as many questions as you can and once you get confident about it you can go further.

Learning DSA:

After getting a grip in Programming language, now you can start
learning **Data Structure and Algorithms **with its application. To solve any problem or to crack any competitive programming, proficiency of syntax alone is not sufficient.
You should have knowledge about Data structures concepts and algorithms and its types with its implementation to solve real life problems.

Data Structure: Organising data and converting it to a well
structured one. Here is the list of Topics of Data Structure:

Arrays
Linked Lists
Stacks
Queue
Hash Tables
Tree
Heaps
Graphs

Whereas For algorithms, You should know following concepts with its application:

Searching Algorithm

Linear Search
Binary Search
Jump Search
Interpolation Search
Exponential Search
Ternary Search

Sorting Algorithm

Selection Sort
Bubble Sort
Quick Sort
Radix Sort
Merge Sort
Cycle Sort
Heap Sort
Bucket Sort

And a variety of others.

Practice Practice Practice:

Once you understand the techniques of solving the problem then below are some of the good sources of practicing competitive programming

LeetCode: Leetcode has a wide variety of problems that can be easy, medium, and Hard. Its advised to start solving Low and Medium problems and then gradually move to Hard Problems.
CodeChef: CodeChef doesn't need any introduction. This platform contains advanced competitive problems. It's better you don't directly jump to Codechef. Instead start with Leetcode and once you can solve their hard problems easily then gradually move to Codechef.