This is a submission for the Bright Data Web Scraping Challenge: Scrape Data from Complex, Interactive Websites
What I Built
I developed a script to extract Mobile Phone plans from Carphone warehouse, i grabbed data such as Phone, their Colors and capacities available, I stored this data in s3 and displayed it using my Laravel project.
I used Bright Data's Browser to avoid any blocks, as telecommunication websites typically block after a certain time.
To extract the data, i had to make use of clicking to open the next page of results, interacting with the page, i found Bright Data's speed and stability very good.
Demo
How I Used Bright Data
I used Bright Data scraping browser tool, which was really helpful to enable my own code to be lightweight, since i packaged the script within docker, i did not have to include my own browser, in addition it is nice knowing i have captcha handling and proxies built in.
I encountered a challenge with being limited to navigating only once per session. I resolved this by requesting a new session for each page navigation. While the solution was straightforward, it did require some restructuring of the code.
Microsomes
/
BrandComplianceMVP
Brand Compliance App
BrightData Challenge - Monthly Phone Plans Aggregator
This project is part of the BrightData Web Scraper Challenge, aiming to aggregate monthly phone plans from Carphone Warehouse, including plans from Vodafone and ID Mobile. The goal is to efficiently scrape detailed data and present it in a user-friendly format 🚀 Challenge Prompt
Use a Web Scraper API to tackle common business challenges like aggregating product prices, monitoring competitors, or collecting reviews across platforms. Use Bright Data’s Web Scraper API for efficient and scalable data collection
🌟 Project Overview Objective
Scrape and aggregate Carphone Warehouse's monthly phone plans.
Collect details such as:
Phone models
Storage capacities
Available colors
Associated monthly plans
Solution Highlights
Scrape the entire selection of phones and plans available on Carphone Warehouse.
Schedule scraping jobs to run daily or weekly to save costs and keep data fresh.
Use Bright Data’s browser-based scraper to bypass blocking mechanisms.
🛠️ Technologies Used…
Summary
I thank Bright Data for the motivation to finally explore Bright Data, it was a pleasure and i hope to continue using it in the near future!
Top comments (0)