DEV Community

muthuri KE
muthuri KE

Posted on

3 5 4 3 4

Web Scraper API to Solve Business Problems

Submission for the Bright Data Web Scraping Challenge: Build a Web Scraper API to Solve Business Problems

What I Built

This project is a Product Prices Aggregator API built using NestJS and GraphQL. It solves the business problem of tracking product prices, monitoring competitors, and fetching product details efficiently. The API leverages:

  • Bright Data: To fetch product data from multiple domains.
  • Redis: For caching results, ensuring faster subsequent queries.
  • OpenSearch: For indexing product data, enabling efficient search capabilities.
  • Kafka: For real-time notifications to users when prices drop below a set threshold.
  • JWT-based Authentication: For secure and role-based access.

The application is ideal for businesses seeking to track market trends, aggregate competitor pricing, and notify users of valuable deals.

Demo

The application is available as a Docker image:

docker pull ghcr.io/muthuri-dev/product-price-aggregator-api:0.1
Enter fullscreen mode Exit fullscreen mode

Run the application and query using a GraphQL client. For example:

Search Products by Title

query {
  searchProductsByTitle(title: "smart watch") {
    title
    price
    url
  }
}
Enter fullscreen mode Exit fullscreen mode

How I Used Bright Data

Bright Data was the backbone of the data fetching mechanism. The application integrates with the Bright Data API to:

  • Trigger data collection for specific keywords and domains.
  • Fetch product data snapshots.

Workflow:

  1. Trigger Bright Data API to collect product data.
  2. Fetch the snapshot data via the API.
  3. Cache the fetched data in Redis.
  4. Index the data in OpenSearch for efficient queries.

GitHub Repository

https://github.com/muthuri-dev/product-prices-aggregator-api.git
Enter fullscreen mode Exit fullscreen mode

Thank you for reviewing my submission for the Bright Data Web Scraping Challenge!

Do your career a big favor. Join DEV. (The website you're on right now)

It takes one minute, it's free, and is worth it for your career.

Get started

Community matters

Top comments (0)

A Workflow Copilot. Tailored to You.

Pieces.app image

Our desktop app, with its intelligent copilot, streamlines coding by generating snippets, extracting code from screenshots, and accelerating problem-solving.

Read the docs

👋 Kindness is contagious

Dive into an ocean of knowledge with this thought-provoking post, revered deeply within the supportive DEV Community. Developers of all levels are welcome to join and enhance our collective intelligence.

Saying a simple "thank you" can brighten someone's day. Share your gratitude in the comments below!

On DEV, sharing ideas eases our path and fortifies our community connections. Found this helpful? Sending a quick thanks to the author can be profoundly valued.

Okay