Scrapping Google Playstore

Purpose Of Project :

To analyze how each fin tech apps is being used and widely accepted by Nigerians, as well as understanding the impact fin tech companies has made on Nigerians finances and Nigeri's Tech Sector.

Project Process:

• Web scraping data from playstore
• Performing Exporatory Data Analysis On data
• Visualizing data on Power BI and Tableau

Project Tools:

• Notebook
• Power BI
• Tableau

Importing Necessary Libraries

import time
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager

Accessing PlayStore Using Selenium

driver = webdriver.Chrome(ChromeDriverManager().install())
driver.get('https://play.google.com/store/search?q=fintech%20in%20nigeria&c=apps')
time.sleep(10)

Getting All The Apps In The Required Page

SCROLL_PAUSE_TIME = 5

# Get scroll height
last_height = driver.execute_script("return document.body.scrollHeight")
time.sleep(SCROLL_PAUSE_TIME)

while True:
    # Scroll down to bottom
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

    # Wait to load page
    time.sleep(SCROLL_PAUSE_TIME)

    # Calculate new scroll height and compare with last scroll height
    new_height = driver.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        break
    last_height = new_height

Scraping App Links

links_fintech = []
elems = driver.find_elements_by_xpath("//a[@href]")
for elem in elems:
    if "details?id" in elem.get_attribute("href"):
        links_fintech.append((elem.get_attribute("href")))

links_fintech = list(dict.fromkeys(links_fintech))

Scraping The Necessary Informations From Each App

list_all_elements = []
for iteration in links_fintech:
    try:
        driver.get(iteration)
        print(iteration)
        time.sleep(3)

        header1 = driver.find_element_by_tag_name("h1")

        downloads = driver.find_elements_by_class_name("htlgb")
        list_downloads = []
        for x in range (len(downloads)):
            if x % 2 == 0:
                list_downloads.append(downloads[x].text)


        titles = driver.find_elements_by_class_name("AHFaub")
        comments = driver.find_element_by_class_name("EymY4b")

        list_elements = [iteration,header1.text, list_downloads.append(downloads[x].text), comments.text.split()[0]]
        for x in range (len(titles)):
            if titles[x].text == "Download":
                list_elements.append(list_others[x])
            if titles[x].text == "Developer":
                for y in list_others[x].split("\n"):
                    if "@" in y:
                        list_elements.append(y)
                        break

        list_all_elements.append(list_elements)
    except Exception as e:
        print(e)

https://play.google.com/store/apps/details?id=ng.gov.cbn.speed.wallet.temp
https://play.google.com/store/apps/details?id=com.oxloan.loan.money.credit.nigeria
https://play.google.com/store/apps/details?id=com.cowrywise.android
https://play.google.com/store/apps/details?id=com.fastmoney.loan.credit.cash.nigeria
https://play.google.com/store/apps/details?id=com.wemabank.alat.prod
https://play.google.com/store/apps/details?id=com.ikonik.lamp_pay......

Creating A CSV File For Scraped DataFrame

import pandas as pd

df = pd.DataFrame(list_all_elements,columns=['URL', 'Name', 'downloads', 'install']) 
df_1 = df.to_csv('fintech_playstore.csv', header = True, index=False, encoding="utf-8")

df_1

https://www.danielherediamejias.com/scraping-google-play-store-with-python-and-selenium/

https://play.google.com/store/search?q=fintechs%20in%20nigeria&c=apps