The latest articles on DEV Community by Chethan Yadav (@chethanyadav). https://dev.to/chethanyadav https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F1844261%2F7c291da0-77fa-4f62-b22d-52f904f11991.jpg https://dev.to/chethanyadav en Chethan Yadav Sun, 29 Dec 2024 01:36:26 +0000 https://dev.to/chethanyadav/the-tech-news-scraper-14kc https://dev.to/chethanyadav/the-tech-news-scraper-14kc <p><em>This is a submission for the <a href="https://dev.to/challenges/brightdata">Bright Data Web Scraping Challenge</a>: Scrape Data from Complex, Interactive Websites</em></p> <h2> What I Built </h2> <p>This project scrapes data from websites that offer the latest technological news and updates. It uses JavaScript and Node.js, with Puppeteer and the Bright Data Scraping Browser to handle dynamic content. It scrapes data from two major websites: </p> <ol> <li><a href="https://www.artificialintelligence-news.com/artificial-intelligence-news/" rel="noopener noreferrer">Artificial Intelligence News</a></li> <li><a href="https://thehackernews.com/" rel="noopener noreferrer">The Hacker News</a></li> </ol> <h2> Demo </h2> <p>You can view the source code and instructions for running the project on <a href="https://github.com/chethanyadav456/Scraping_Master" rel="noopener noreferrer">GitHub</a>.</p> <p><a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F9ayn0cfn0p9udxvjff81.png" class="article-body-image-wrapper"><img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F9ayn0cfn0p9udxvjff81.png" alt="Articles display webpage" width="800" height="482"></a></p> <h2> How I Used Bright Data </h2> <p>I leveraged Bright Data’s Scraping Browser to handle JavaScript-heavy and interactive websites that require dynamic content loading. The project scrapes real-time data, including titles, descriptions, URLs, images, and published dates. Bright Data's browser provided a smooth solution to maintain the scraping process without additional overhead.</p> <p><strong>Challenge Prompt:</strong> <a href="https://dev.to/devteam/join-us-for-the-bright-data-web-scraping-challenge-3000-in-prizes-3mg2?">Bright Data Web Scraping Challenge</a></p> <h2> Installation </h2> <ol> <li>Clone the repository </li> </ol> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code>git clone https://github.com/chethanyadav456/Scraping_Master.git </code></pre> </div> <ol> <li>Install dependencies </li> </ol> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code>npm <span class="nb">install</span> </code></pre> </div> <ol> <li>Run the project </li> </ol> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code>node master.js </code></pre> </div> <ol> <li>Create a .env file and add: </li> </ol> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>MONGO_URI= BROWSER_WS= </code></pre> </div> <h2> License </h2> <p>This project is licensed under the MIT License - see the LICENSE file for details</p> brightdatachallenge devchallenge webdev api