DEV Community

Warren S. Allen
Warren S. Allen

Posted on

Building an Automated Competitive Intelligence Pipeline (Without Enterprise Pricing)

Last quarter, one of our competitors quietly shipped a feature that directly overlapped with what we'd been building for two months. I found out in a customer call. The customer asked why we were "behind" on something the competitor had already launched. I had no idea it existed.

That was the moment I decided our competitor monitoring process — which was really just people sometimes pasting links in a Slack channel — needed to be replaced with something that actually worked.

We tried the obvious fixes first. A shared Google Sheet with competitor URLs. A weekly review meeting. Assigned owners for each competitor. Each attempt lasted about two weeks before people stopped updating it. Turns out, "remember to check 10 websites every Monday" doesn't scale when everyone has actual work to do.

I built a small system to watch competitor blogs and changelogs, filter the noise, and post the handful of real updates into Slack. Here's what broke, what finally held up, and what I'd do differently if I started over.

What I needed

Two things mattered more than anything else:

  1. Push to Slack — the team lives there. Anything that requires opening another app gets ignored within a week.
  2. Filter the noise — I care about feature launches and pricing changes, not "Meet Our New VP of Sales" posts.

Everything else was nice to have: webhook support for piping data into internal dashboards, covering at least 8-10 competitor blogs, and keeping costs reasonable. We're a small team — I wasn't going to open a procurement process for this.

Attempt 1: Building it ourselves

My first instinct was to rope in one of our engineers and build it. Most competitor blogs have RSS feeds, even if they don't advertise them. We wrote a Python script that polled feeds via cron, deduped with SQLite, and sent new entries to the OpenAI API with a filter prompt:

FILTER_PROMPT = """
You are a competitive intelligence filter. Given a blog post
title and content, classify it as RELEVANT or SKIP.

RELEVANT: new feature launches, pricing changes, major
integrations, funding announcements, product pivots.

SKIP: hiring posts, team culture stories, generic thought
leadership, event recaps, customer spotlights (unless they
reveal product details).

Respond with JSON: {"decision": "RELEVANT"|"SKIP", "summary": "..."}
"""
Enter fullscreen mode Exit fullscreen mode

Relevant items got posted to a Slack channel via incoming webhook.

It worked at first, and I was feeling pretty good about it. Then the quiet failures started.

One competitor redesigned their blog and the RSS endpoint moved. Nothing errored, nothing alerted — the feed just stopped updating. I only noticed weeks later when someone asked about a release I'd missed.

Costs crept up too. Sending every post through GPT-4 was fine in low-volume weeks, but a few noisy weeks made the API bill feel silly. Dropping to a cheaper model helped with cost but the classifier got noticeably worse at separating real launches from marketing fluff.

The final straw was reliability. When the server had an issue and cron stopped running, the system didn't fail loud. It failed by doing nothing — which looks exactly like a slow news week. Nobody noticed for days. At that point I realized we'd built a tool that needed its own monitoring.

Total cost was maybe $15/month between API calls and the server, plus a few hours a month of our engineer's time keeping it alive. The system wasn't bad. But we were spending more time maintaining the monitoring tool than acting on what it found.

What I tried buying instead of building

Feedly + Zapier: Feedly's free tier handles RSS aggregation well. Their AI assistant Leo (on Pro+ at $8.93/month) can filter articles. The problem is getting information out. Direct Slack integration was gated behind their Enterprise plan — the quote I got was well above what a small team would pay. Piping through Zapier works but you lose Leo's filtering (the Zapier integration exports the raw feed). And Zapier's free tier runs out fast when you have a dozen active feeds. I ended up paying roughly $38/month for Feedly Pro + Zapier Starter, and the filtering was worse than our DIY script. Same story with IFTTT — basic integration, no access to the AI layer.

The core issue: AI filtering happened inside Feedly, but the automation integrations couldn't access it. So I was paying for filtering I couldn't use in my pipeline.

Google Alerts: Free, so I set it up as a baseline. It caught maybe a fifth of what other tools found. Email only, keyword matching only, no RSS sources or social media. Fine as a supplement — I still have it running — but not a primary tool.

Brand24, Crayon, Klue: These are serious tools for serious budgets. Brand24 starts at $199/month and is built more for tracking mentions of a brand than tracking what a competitor ships. Crayon and Klue are enterprise competitive intelligence platforms — think dedicated CI teams at large companies. Didn't fit a small team budget, and honestly most of what they offer was overkill for what I needed.

Inoreader: RSS reader with a rules engine. Decent, but the rules are keyword-based — no AI filtering without a paid add-on. Still felt like I'd be hacking a reading tool into a monitoring tool.

None of these hit the overlap of: RSS monitoring + AI filtering + Slack delivery + a price I didn't need to get approved.

What I ended up using

I found SignalHub while searching for something like "RSS AI filter Slack" — not creative, but it surfaced what I was looking for.

Here's the setup that's been running for a few months:

Tracker 1: Competitor Product Updates
8 competitor blogs and changelog RSS feeds. Filter rule (natural language): "New feature launches, pricing changes, new integrations, product deprecations, or platform migrations. Skip hiring posts, company culture articles, and generic thought leadership." Notifications go to #competitor-updates in Slack.

Tracker 2: Industry News
5 industry publications and a couple of relevant subreddit feeds. Filter: "Articles about our product category, market trends, or funding rounds in our space. Skip opinion pieces with no new data." Goes to #industry-news.

Tracker 3: Changelog Deep Watch
3 closest competitors' changelog pages. No filtering — these are low-volume and I want to see everything. Goes to my Slack DMs + a webhook to our internal dashboard.

The webhook part was what got me to try it seriously. I piped Tracker 3 into a simple endpoint that logs competitor releases to our internal tool:

app.post('/api/competitor-releases', (req, res) => {
  const { title, summary, source, url, matched_at } = req.body;

  db.insert('competitor_releases', {
    title,
    summary,
    source_url: url,
    detected_at: matched_at,
    status: 'new'
  });

  res.sendStatus(200);
});
Enter fullscreen mode Exit fullscreen mode

Pricing

The free plan gives you 1 tracker with 5 sources and 1 notification channel — enough to test whether the concept works for you. I'm on Starter at $4.99/month for 5 trackers. All notification channels (Slack, Discord, Telegram, webhooks, email, etc.) are available on every plan, including free. After the Feedly experience where Slack integration was locked behind an enterprise tier, this was a relief.

Where it falls short

It's not perfect. A few things I've noticed:

  • Filter tuning takes iteration. My first filter rules were too broad and I got noisy results for about a week before I tightened them. Write specific rules from the start — "important updates" doesn't cut it.
  • Some sources don't have RSS. A couple of competitors publish updates only on Twitter or in their Discord community. SignalHub monitors RSS and web sources, so for those I still do occasional manual checks.
  • Occasional misses on borderline content. The AI filtering is good but not perfect. Maybe once a month something slips through that should have been caught, or something gets flagged that's not actually relevant. It's roughly on par with what our DIY GPT script did, just without the maintenance burden.
  • No retroactive search. It monitors going forward. If you want to look back at what a competitor did last quarter, you'd need to check their blog archive yourself.

For our use case these tradeoffs are fine. The stuff it catches reliably — feature launches, pricing changes, new integrations — is exactly what we care about most.

Things I'd do differently

A couple of lessons from getting this wrong before getting it right:

Start with three competitors, not ten. When we first set up the DIY version, I added every competitor I could think of plus a bunch of "adjacent" companies. Most of the noise came from sources that weren't actually relevant. I spent the first week tuning filters instead of reading updates. Now I focus on the 3-4 competitors our customers actually compare us to, and add others only when someone on the team specifically asks.

Separate your Slack channels. I originally dumped everything into one #competitive-intel channel. It became noisy fast, and people started muting it — which defeated the entire purpose. Splitting into #competitor-updates (product changes only) and #industry-news (broader market stuff) made a big difference. Different urgency, different audience on the team.

Keep Google Alerts running as a supplement. It's free, takes 30 seconds to set up per competitor, and occasionally catches something from a source you didn't think to monitor. It probably covers 20% of what the primary system catches, but that 20% sometimes includes random blog posts or news articles that wouldn't show up in an RSS-based pipeline.


The before/after for us was pretty clear. We went from 40-ish minutes of manual checking every Monday (that we'd skip half the time) to a handful of Slack notifications per week, each with an AI summary. The surprising part wasn't the time saved — it was how much more consistent the team's awareness became once updates just showed up automatically.

Appendix: Finding competitor RSS feeds

Most blogs expose RSS feeds even if they don't link to them. Common URL patterns:

# WordPress (~40% of the web)
/feed/  ·  /feed/rss/  ·  /blog/feed/

# Ghost
/rss/

# Substack
https://example.substack.com/feed

# GitHub releases
https://github.com/{org}/{repo}/releases.atom

# Medium publications
https://medium.com/feed/@{publication}

# Changelog pages
/changelog/rss  ·  /changelog.xml
Enter fullscreen mode Exit fullscreen mode

Quick check script:

for path in /feed /feed/rss /rss /blog/feed /changelog.xml /changelog/rss; do
  curl -s -o /dev/null -w "%{http_code} %{url}" "https://competitor.com${path}"
done

# Or look for RSS link tags in the HTML
curl -s https://competitor.com/blog | grep -i "application/rss\|application/atom"
Enter fullscreen mode Exit fullscreen mode

Top comments (3)

Collapse
 
matthewhou profile image
Matthew Hou

The opening story hits hard — finding out you're behind from a customer instead of your own monitoring. That's the failure mode of every manual tracking system.

The progression you described (shared sheet → weekly meeting → assigned owners → all dead in two weeks) is almost universal. I've seen the exact same pattern. The fundamental issue is that monitoring is a continuous task being assigned to humans who have discrete tasks competing for their attention.

What's interesting is this is one of the few areas where AI automation genuinely saves attention, not just time. The difference matters — saving time means doing the same work faster, saving attention means doing work that wasn't getting done at all. Competitive monitoring falls squarely in the second category for most teams.

Collapse
 
harsh2644 profile image
Harsh

The shared Google Sheet that dies after two weeks is basically a law of nature at this point. Love that you actually built something sustainable instead of just trying 'better process' for the 5th time.

Collapse
 
klement_gunndu profile image
klement Gunndu

The SQLite dedup plus OpenAI filter prompt approach for RSS monitoring is clean — particularly the insight about the two-week shelf life of manual processes before people stop updating them.