DEV Community

Timosha
Timosha

Posted on

I Built an AI System That Collects 1,800+ Business Signals Per Day

#ai #startup #datascience #business

I built a signal intelligence system that scrapes Reddit, Hacker News, Google Trends, patent filings, job boards, and more -- collecting 1,800+ signals per day about business opportunities.

What It Does

The system runs 32 automated cron jobs across 14+ sources. Each signal gets scored on:

  • Revenue potential (dollar signals mentioned in posts)
  • Pain level (complaints, frustrations, switching behavior)
  • Competition (how many existing solutions)
  • Demand evidence ("I'd pay for...", "why hasn't someone built...")

How It Works 

Reddit (40 subreddits)  ─┐
Hacker News             ─┤
Google Trends           ─┤
Federal Register        ─┤──→ Signal DB ──→ Scoring ──→ Daily Digest
Patent Filings          ─┤
Job Boards              ─┤
Google News             ─┤
GitHub Trending         ─┘
Enter fullscreen mode Exit fullscreen mode

Every morning, I get a digest with the top signals ranked by score. The system detects:

  • Breakout topics (sudden interest spikes)
  • Cross-source convergence (same topic appearing in multiple places)
  • Explicit demand (people literally saying they'd pay for something)

What I've Found So Far

After collecting 9,500+ signals:

  1. Raptive dropped their entry threshold from 100K to 25K pageviews -- most bloggers still think it's 100K
  2. Building permit data is the most underrated real estate indicator -- permits show development months before ground breaks
  3. The "boring business" SaaS market is massively underserved -- HVAC, plumbing, and cleaning companies still use spreadsheets

The Stack

  • Python collectors with PRAW, requests, pytrends
  • SQLite for signal storage
  • Cron for scheduling (32 jobs)
  • Raspberry Pi for residential IP scraping (avoids rate limits)
  • Custom scoring algorithm with enrichment pipeline

What I'm Building With This

I've launched 10 micro-experiments based on signals from this system -- each one is a landing page testing demand for a specific tool:

  • PermitPulse -- Building permit intelligence
  • RegRadar -- Regulatory change monitoring
  • FilingDiff -- SEC filing change detection
  • GovConRadar -- Government contract alerts
  • And 6 more...

Each experiment collects early access signups. After 7 days, I kill the losers and build MVPs for the winners.

Key Insight

Most people build first, then look for customers. This system flips it: find the demand first, then build exactly what people are asking for.

The signal-to-noise ratio is about 5% (105 actionable signals out of 1,853 collected yesterday). But those 105 signals are gold -- real people expressing real pain with real willingness to pay.

What signals do you use to validate business ideas? I'd love to hear other approaches.

Top comments (0)