Scrape Phone Plans

Tayyab Javed — Sun, 29 Dec 2024 23:22:24 +0000

This is a submission for the Bright Data Web Scraping Challenge: Scrape Data from Complex, Interactive Websites

What I Built

I developed a script to extract Mobile Phone plans from Carphone warehouse, i grabbed data such as Phone, their Colors and capacities available, I stored this data in s3 and displayed it using my Laravel project.

I used Bright Data's Browser to avoid any blocks, as telecommunication websites typically block after a certain time.

To extract the data, i had to make use of clicking to open the next page of results, interacting with the page, i found Bright Data's speed and stability very good.

Demo

View Demo

How I Used Bright Data

I used Bright Data scraping browser tool, which was really helpful to enable my own code to be lightweight, since i packaged the script within docker, i did not have to include my own browser, in addition it is nice knowing i have captcha handling and proxies built in.

I encountered a challenge with being limited to navigating only once per session. I resolved this by requesting a new session for each page navigation. While the solution was straightforward, it did require some restructuring of the code.

Microsomes / BrandComplianceMVP

Brand Compliance App

BrightData Challenge - Monthly Phone Plans Aggregator

This project is part of the BrightData Web Scraper Challenge, aiming to aggregate monthly phone plans from Carphone Warehouse, including plans from Vodafone and ID Mobile. The goal is to efficiently scrape detailed data and present it in a user-friendly format 🚀 Challenge Prompt

Use a Web Scraper API to tackle common business challenges like aggregating product prices, monitoring competitors, or collecting reviews across platforms. Use Bright Data’s Web Scraper API for efficient and scalable data collection

🌟 Project Overview Objective

Scrape and aggregate Carphone Warehouse's monthly phone plans.
Collect details such as:
    Phone models
    Storage capacities
    Available colors
    Associated monthly plans

Solution Highlights

Scrape the entire selection of phones and plans available on Carphone Warehouse.
Schedule scraping jobs to run daily or weekly to save costs and keep data fresh.
Use Bright Data’s browser-based scraper to bypass blocking mechanisms.

🛠️ Technologies Used…

View on GitHub

Summary
I thank Bright Data for the motivation to finally explore Bright Data, it was a pleasure and i hope to continue using it in the near future!