Web Scraper API to Solve Business Problems

muthuri KE — Sun, 29 Dec 2024 07:36:02 +0000

Submission for the Bright Data Web Scraping Challenge: Build a Web Scraper API to Solve Business Problems

What I Built

This project is a Product Prices Aggregator API built using NestJS and GraphQL. It solves the business problem of tracking product prices, monitoring competitors, and fetching product details efficiently. The API leverages:

Bright Data: To fetch product data from multiple domains.
Redis: For caching results, ensuring faster subsequent queries.
OpenSearch: For indexing product data, enabling efficient search capabilities.
Kafka: For real-time notifications to users when prices drop below a set threshold.
JWT-based Authentication: For secure and role-based access.

The application is ideal for businesses seeking to track market trends, aggregate competitor pricing, and notify users of valuable deals.

Demo

The application is available as a Docker image:

docker pull ghcr.io/muthuri-dev/product-price-aggregator-api:0.1

Run the application and query using a GraphQL client. For example:

Search Products by Title

query {
  searchProductsByTitle(title: "smart watch") {
    title
    price
    url
  }
}

How I Used Bright Data

Bright Data was the backbone of the data fetching mechanism. The application integrates with the Bright Data API to:

Trigger data collection for specific keywords and domains.
Fetch product data snapshots.

Workflow:

Trigger Bright Data API to collect product data.
Fetch the snapshot data via the API.
Cache the fetched data in Redis.
Index the data in OpenSearch for efficient queries.

GitHub Repository

https://github.com/muthuri-dev/product-prices-aggregator-api.git

Thank you for reviewing my submission for the Bright Data Web Scraping Challenge!