In this tutorial, we will explore how to scrape user profiles on GeeksforGeeks using Node.js. We'll utilize the Express framework, Axios for making HTTP requests, Cheerio for parsing HTML, and Mongoose for working with MongoDB.
Project Setup
To get started, make sure you have Node.js and npm (Node Package Manager) installed on your machine. Then, follow these steps:
- Create a new directory for your project and navigate to it in your terminal.
- Initialize a new Node.js project by running the following command:
npm init -y
- Install the necessary dependencies:
npm install express axios cheerio cors dotenv mongoose
Setting up the Express Server
Let's start by setting up an Express server that will handle the scraping and API requests.
Create a new file called server.js and add the following code:
const express = require('express');
const axios = require('axios');
const cheerio = require('cheerio');
const cors = require('cors');
require("dotenv").config();
const mongoose = require("mongoose");
const app = express();
app.use(express.json());
app.use(cors());
app.get('/profile/:username', async (req, res) => {
});
const port = process.env.PORT || 3000;
app.listen(port, () => {
console.log(`Server running on port ${port}`);
});
Scraping GeeksforGeeks User Profiles
To scrape user profiles on GeeksforGeeks, we'll use Axios to make HTTP requests and Cheerio to parse the HTML response.
Within the /profile/:username route, add the following code:
app.get('/profile/:username', async (req, res) => {
const { username } = req.params;
try {
const url = `https://auth.geeksforgeeks.org/user/${username}`;
const response = await axios.get(url);
const $ = cheerio.load(response.data);
const name = $('.profile_name').text().trim();
const institution = $('.basic_details_data a').text().trim();
const longestStrek = $('.streakCnt.tooltipped').text().trim();
const rank = $('.rankNum').text().trim();
const coddingScore = $('.score_card_value').text().trim();
const overallScore = coddingScore.slice(0, 3);
const solvedProblemsCount = coddingScore.slice(3).replace(/_/g, '');
const totalSubmissions = $('.heatmap_header .numProblems').text().trim();
const profile = {
name,
username,
institution,
rank,
overallScore,
solvedProblemsCount,
longestStrek,
totalSubmissions
};
res.json(profile);
} catch (error) {
console.log(error);
res.status(500).json({ error: 'Failed to fetch profile data' });
}
});
In the above code, we make a GET request to the GeeksforGeeks user profile URL, load the HTML response using Cheerio, and then scrape the required profile information.
Building the Frontend
Now that we have the backend set up to scrape and store profiles, let's build a simple frontend to display the profile data.
Create a new file called index.html and add the following code:
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>GeeksforGeeks User Profiles</title>
</head>
<body>
<h1>GeeksforGeeks User Profiles</h1>
<input type="text" id="username" placeholder="Enter a username">
<button onclick="fetchProfile()">Fetch Profile</button>
<div id="profile"></div>
<script src="script.js"></script>
</body>
</html>
In the above HTML, we have a simple form to enter a username and a button to fetch the profile data.
Create a new file called script.js and add the following code:
async function fetchProfile() {
const username = document.getElementById('username').value;
const response = await fetch(`http://localhost:3000/profile/${username}`);
const data = await response.json();
const profileDiv = document.getElementById('profile');
profileDiv.innerHTML = JSON.stringify(data, null, 2);
}
This JavaScript code fetches the profile data from the server and displays it in the profile div.
Running the Project
To run the project, start the Express server by running the following command:
node server.js
Open index.html in a web browser, enter a GeeksforGeeks username, and click the "Fetch Profile" button. The profile data will be displayed on the page.
That's it! You have successfully created a Node.js project that scrapes user profiles on GeeksforGeeks. You can now deploy it to a hosting platform and share it with others.
Happy coding!
Top comments (1)
Thank you for sharing your experience with frontend scraping. I completely agree with you.
While frontend scraping can be a convenient approach, it often leads to slower loading times and performance issues. Moreover, when dealing with websites that require authentication or user credentials for scraping, exposing sensitive information on the client-side can pose significant security risks. That's why server-side scraping is a preferred solution, providing better efficiency, security, and a smoother user experience.