State of the Art Automated Web Scraper using Bright Data

#devchallenge #brightdatachallenge #ai #webdev

This is a submission for the Bright Data Web Scraping Challenge: Scrape Data from Complex, Interactive Websites, Build a Web Scraper API to Solve Business Problems and Most Creative Use of Web Data for AI Models (All 3 prompts).

Prerequisite:

Please read the following blog post to get started on the Bright Data for the AI use-cases Bright Data to Tackle AI Data Challenges
Depending upon the operating system you are using, Please make sure to install .NET 8
Signup or Login to the Google AI Studio. It's required for building the fine-tuned instruction set.
Make sure to sign up on BrightData. Also complete the steps for the initial setup for Proxies & Scraping Infrastructure and Web Scraping API. Please make a note on the WSS Browser Credential, Webscraper Api Token

What I Built

I have built an Automated Web Scrapper, a .NET C# Library with a console solution for solving various problems of automated web scraping with the custom fine-tuned AI model building including data aggregation or collection via the 3rd party providers supported by the Bright Data.

Source Code - https://github.com/ranjancse26/AutomatedWebscraper

The Automated Web Scrapper project mainly focuses on the following aspects.

Proxies and Scraping - Headless Web Scraping with the help of Bright Data.
Web Scraper API using Bright Data Web Scrapper APIs for Data Collection and Management related aspects.
Web Data using Bright Data. Considering an example of Glassdoor.
Google Gemini 2.0 (gemini-2.0-flash-exp) LLM integration for getting the structured information for the fine-tuning of the AI models.

Headless Web Scraper Class Diagram

Web Scraper Class Diagram

Service Layer Class Diagram

Demo

Source Code - https://github.com/ranjancse26/AutomatedWebscraper

Please make sure to follow the pre-requisite and update the appSettings.json

Getting started with the Console Demo.

Here's the screenshot of the Bright Data - Booking.com provider-based response by utilizing the Proxies and Scraping mechanism of Headless Web Scraping based solution. One can easily build or extend the library for other providers too.

Booking.com Web Scraping using the Bright Data Web Scraping API.
Note - Depending on the data you are requesting, you may have to wait for few seconds to minutes to complete the Web Scraping API call.

Here's the Glassdoor Request/Response using the Bright Data Web Scraper API.

Glassdoor Debug Information

Here's the screenshot of the Glassdoor Finetune instruction set preparation for fine tuning the Google Gemma model. Similar techniques could be utilized for fine tuning of various LLM models.

Bright Data - Web Data - Glassdoor company finetuned instruction set creation.

What problem does it solve?

Below you will be discussed with the real-world problem that is being solved by the Automated Webscraper Solution.

Using the Headless Browser or the Bright Data Web Scraper API based approach, one could literally accomplish the highly reliable and guaranteed method of automated web scraping in an easy manner. Building a highly scalable, scheduled or automated on-demand or real time vs batch by batch offline method of web scraping of Bright Data's solution could be utilized for performing the web scraping for solving various data collection or aggregation aspects.
Using the Bright Data's Web Data solution, one could simply perform the easy web scraping and download the data for further building the custom fine-tuned AI model.

How I Used Bright Data

The following section explains on how the Bright Data is being leveraged for accomplishing the Automated Web Scraping solution using various techniques.

Proxies and Scraping - Using the Headless Web Scraping with the help of Bright Data, the Booking.com implementation has been accomplished. The Booking.com is being considered as an example. However, one could leverage and build the headless browser-based web scraping solution for any of the Bright Data supported providers.
Web Scraper API - Using Bright Data Web Scrapper APIs for Data Collection and Management related aspects. The Glassdoor has been considered as an example to demonstrate on the Web Scraper API based implementation for fetching the Glassdoor company information by a specific URL.
Web Data - Using Bright Data's Web Data. Considering an example of Glassdoor and by utilizing the Easy Web Scraping technique for initiating and downloading the Glassdoor company information and then later used the downloaded JSON information for building the Google Gemma fine-tuned instruction set by utilizing the Google's latest gemini-2.0-flash-exp LLM model. Google Gemini 2.0 (gemini-2.0-flash-exp) LLM integration is done for building the structured information for the fine-tuning of the AI models.

This submission does qualify for the below prompts