DEV Community

Tayyab Javed
Tayyab Javed

Posted on

3 2 3 3 2

Scrape Phone Plans

This is a submission for the Bright Data Web Scraping Challenge: Scrape Data from Complex, Interactive Websites

What I Built

I developed a script to extract Mobile Phone plans from Carphone warehouse, i grabbed data such as Phone, their Colors and capacities available, I stored this data in s3 and displayed it using my Laravel project.

I used Bright Data's Browser to avoid any blocks, as telecommunication websites typically block after a certain time.

To extract the data, i had to make use of clicking to open the next page of results, interacting with the page, i found Bright Data's speed and stability very good.

Demo

View Demo

Home page

Mobile plans

How I Used Bright Data

I used Bright Data scraping browser tool, which was really helpful to enable my own code to be lightweight, since i packaged the script within docker, i did not have to include my own browser, in addition it is nice knowing i have captcha handling and proxies built in.

I encountered a challenge with being limited to navigating only once per session. I resolved this by requesting a new session for each page navigation. While the solution was straightforward, it did require some restructuring of the code.

GitHub logo Microsomes / BrandComplianceMVP

Brand Compliance App

BrightData Challenge - Monthly Phone Plans Aggregator

This project is part of the BrightData Web Scraper Challenge, aiming to aggregate monthly phone plans from Carphone Warehouse, including plans from Vodafone and ID Mobile. The goal is to efficiently scrape detailed data and present it in a user-friendly format 🚀 Challenge Prompt

Use a Web Scraper API to tackle common business challenges like aggregating product prices, monitoring competitors, or collecting reviews across platforms. Use Bright Data’s Web Scraper API for efficient and scalable data collection

🌟 Project Overview Objective

Scrape and aggregate Carphone Warehouse's monthly phone plans.
Collect details such as:
    Phone models
    Storage capacities
    Available colors
    Associated monthly plans

Solution Highlights

Scrape the entire selection of phones and plans available on Carphone Warehouse.
Schedule scraping jobs to run daily or weekly to save costs and keep data fresh.
Use Bright Data’s browser-based scraper to bypass blocking mechanisms.

🛠️ Technologies Used…

Summary
I thank Bright Data for the motivation to finally explore Bright Data, it was a pleasure and i hope to continue using it in the near future!

Billboard image

The fastest way to detect downtimes

Join Vercel, CrowdStrike, and thousands of other teams that trust Checkly to streamline monitoring.

Get started now

Top comments (0)

Billboard image

Try REST API Generation for Snowflake

DevOps for Private APIs. Automate the building, securing, and documenting of internal/private REST APIs with built-in enterprise security on bare-metal, VMs, or containers.

  • Auto-generated live APIs mapped from Snowflake database schema
  • Interactive Swagger API documentation
  • Scripting engine to customize your API
  • Built-in role-based access control

Learn more

👋 Kindness is contagious

Discover a treasure trove of wisdom within this insightful piece, highly respected in the nurturing DEV Community enviroment. Developers, whether novice or expert, are encouraged to participate and add to our shared knowledge basin.

A simple "thank you" can illuminate someone's day. Express your appreciation in the comments section!

On DEV, sharing ideas smoothens our journey and strengthens our community ties. Learn something useful? Offering a quick thanks to the author is deeply appreciated.

Okay