DEV Community

agenthustler
agenthustler

Posted on • Edited on

ProductHunt Scraping for Competitive Intelligence

#python #webdev #tutorial #webscraping

ProductHunt is where new products launch. Tracking launches, upvotes, and maker activity gives you an early signal on market trends, competitor moves, and emerging opportunities. Here's how to build a competitive intelligence pipeline from ProductHunt data.

What Intelligence Can You Extract?

  • New product launches in your category
  • Upvote velocity (how fast a product gains traction)
  • Maker activity and serial entrepreneurs
  • Comment sentiment and user feedback
  • Technology trends and emerging categories

Scraping ProductHunt Launches 

# Implementation is proprietary (that IS the moat).
# Skip the build — use our ready-made Apify actor:
# see the CTA below for the link (fpr=yw6md3).
Enter fullscreen mode Exit fullscreen mode

Building a Competitive Dashboard 

import pandas as pd

class CompetitiveIntelligence:
    def __init__(self, scraper):
        self.scraper = scraper
        self.data = []

    def track_category(self, keywords, days=30):
        """Track launches matching keywords over time."""
        from datetime import timedelta

        for i in range(days):
            date = (datetime.now() - timedelta(days=i)).strftime('%Y/%m/%d')
            launches = self.scraper.get_daily_launches(date)

            for product in launches:
                # Check if product matches our keywords
                text = f"{product['name']} {product['tagline']}".lower()
                if any(kw.lower() in text for kw in keywords):
                    product['matched_keywords'] = [
                        kw for kw in keywords if kw.lower() in text
                    ]
                    self.data.append(product)

            time.sleep(2)

        df = pd.DataFrame(self.data)
        print(f"Found {len(df)} matching products in {days} days")
        return df

    def analyze_trends(self, df):
        """Identify trends from tracked launches."""
        if df.empty:
            return {}

        # Most common themes
        all_keywords = []
        for kws in df['matched_keywords']:
            all_keywords.extend(kws)

        from collections import Counter
        keyword_freq = Counter(all_keywords)

        # Average votes by keyword
        df_exploded = df.explode('matched_keywords')
        avg_votes = df_exploded.groupby('matched_keywords')['votes'].mean()

        print("Category trends:")
        for kw, count in keyword_freq.most_common(10):
            avg = avg_votes.get(kw, 0)
            print(f"  {kw}: {count} launches, avg {avg:.0f} upvotes")

        return {'frequency': keyword_freq, 'avg_votes': avg_votes}

# Example: Track AI tool launches
scraper = ProductHuntScraper()
ci = CompetitiveIntelligence(scraper)
ai_launches = ci.track_category(
    ['AI', 'GPT', 'LLM', 'automation', 'agent'],
    days=14
)
ci.analyze_trends(ai_launches)
Enter fullscreen mode Exit fullscreen mode

Launch Velocity Tracking 

# Implementation is proprietary (that IS the moat).
# Skip the build — use our ready-made Apify actor:
# see the CTA below for the link (fpr=yw6md3).
Enter fullscreen mode Exit fullscreen mode

Scaling ProductHunt Intelligence

For continuous, automated competitive monitoring, the ProductHunt Scraper on Apify runs on a schedule and delivers structured data to your analytics pipeline without manual intervention.

For proxy management during web data collection, ScraperAPI handles the infrastructure so you can focus on analysis.

Conclusion

ProductHunt data is a leading indicator for tech market trends. By tracking launches, upvote velocity, and category activity, you can spot competitors early, validate product ideas, and understand where the market is heading. Start with the Python scraper above, build category tracking, and scale to automated monitoring for continuous intelligence.

Top comments (0)