The Tech News Scraper

This is a submission for the Bright Data Web Scraping Challenge: Scrape Data from Complex, Interactive Websites

What I Built

This project scrapes data from websites that offer the latest technological news and updates. It uses JavaScript and Node.js, with Puppeteer and the Bright Data Scraping Browser to handle dynamic content. It scrapes data from two major websites:

1. Artificial Intelligence News
2. The Hacker News

Demo

You can view the source code and instructions for running the project on GitHub.

How I Used Bright Data

I leveraged Bright Data’s Scraping Browser to handle JavaScript-heavy and interactive websites that require dynamic content loading. The project scrapes real-time data, including titles, descriptions, URLs, images, and published dates. Bright Data's browser provided a smooth solution to maintain the scraping process without additional overhead.

Challenge Prompt: Bright Data Web Scraping Challenge

Installation

1. Clone the repository

git clone https://github.com/chethanyadav456/Scraping_Master.git

1. Install dependencies

npm install

1. Run the project

node master.js

1. Create a .env file and add:

MONGO_URI=
BROWSER_WS=

License

This project is licensed under the MIT License - see the LICENSE file for details