DEV Community

Ava Torres
Ava Torres

Posted on

How to Build a Competitor Intelligence Pipeline Using Free Government Data

Most competitive intelligence work is kind of embarrassing when you look at it closely. Someone on the sales team manually Googles competitors every quarter, pastes stuff into a slide deck, and calls it a CI report. The deck is out of date by the time anyone reads it.

There's a better way, and it starts with a data source most people ignore entirely: government filings.

Your competitors -- especially if they're publicly traded, mid-market, or doing any government contracting -- are publishing detailed information about their business on a regular basis. Revenue trends. Major customer concentration. Executive compensation. Litigation. Contract wins. All public. All free. All just sitting there waiting to be extracted.

Here's how I built a pipeline that actually keeps up.

Start with SEC EDGAR for Public Competitors

If any of your competitors are public companies, EDGAR is the single most useful source you have. The SEC requires quarterly (10-Q) and annual (10-K) filings that include:

  • Revenue and growth rates (actual numbers, not PR-spun narratives)
  • Gross margin and operating margin
  • Customer concentration risk -- if one customer is >10% of revenue, they have to disclose it
  • Pending litigation and regulatory actions
  • Risk factors, which is basically executives writing down what they're actually worried about
  • Management's Discussion & Analysis, which tells you how they're thinking about the business

The 10-K's risk factors section alone is worth more than most competitor analysis decks I've seen. Companies spend serious legal time writing that section. It's candid in ways their marketing absolutely isn't.

I use SEC EDGAR company filings to monitor competitors automatically. Set up a scheduled run every quarter when filing season hits, get the latest 10-K and 10-Q links for your watchlist, feed them into your analysis workflow. I've been routing these into a Notion database that our sales team actually reads -- took me a few hours to wire up, runs itself now.

One specific play: pull the "Customers" section of competitor 10-Ks to see who they're naming as reference customers. That's an account list you didn't have to build yourself.

Business Entity Search for Private Competitors

Most of your actual competitors are probably private -- which means no EDGAR filings. But they're still registered entities, and state SOS records are public.

What you can get from business entity registrations:

  • When the company was actually founded (versus what they claim on their website)
  • Registered agent address, which often reveals where they're really operating from
  • Any name changes or DBA registrations -- useful for tracking rebrands
  • Good standing status -- a competitor that's let their registration lapse is in trouble
  • Officers and directors in states that publish them (varies)

US business entity search covers the major states and gives you structured output. I run this quarterly for my competitor watchlist. It takes about 30 seconds and has caught a couple of interesting signals -- one competitor let their Delaware registration lapse for three months, which tracks with the layoff rumors we were hearing at the time.

This isn't glamorous data but it's consistent. Public companies can manage their PR. They can't quietly let their SOS registration lapse without leaving a trace.

Federal Contract History

If your competitors sell to the government at all, their contract history is public on SAM.gov. This is absurdly underused for competitive intelligence.

What you can see:

  • Total contract award amounts by year
  • Which federal agencies they're working with
  • Contract types (T&M vs. fixed price tells you about margin pressure)
  • Subcontracting roles vs. prime contractor roles
  • Performance period and option years

A competitor winning a $50M multi-year federal contract is something you should know about the week it's awarded, not six months later when their sales team is bragging about it. The SAM.gov federal contracts search pulls this data on demand -- schedule it to run weekly for any competitor you care about.

The other use case: prospecting. If your competitor just won a big federal contract in a vertical you want to enter, the contracting agency is now a warm prospect -- they've already validated the need and budget.

Domain Intelligence for Digital Competitors

For startups and digital-first competitors, WHOIS data can surface things they haven't announced yet. Domain registrations, nameserver changes, and registration dates all tell a story.

A competitor quietly registering a domain like [theircompany]-enterprise.com is probably building an enterprise tier. New domains with their brand in adjacent categories are a roadmap. Domain age on their main site vs. their "founded in 2019" claim sometimes doesn't add up.

WHOIS domain lookup returns structured WHOIS data you can monitor over time. I batch this across competitor domains and track changes -- particularly nameserver changes (usually means they're switching infrastructure) and new registrations that share a registrant email.

Business Directory Presence as a Weak Signal

This one's more qualitative, but I check YellowPages listings for competitors in markets they're claiming to be expanding into. A company that says they're launching in the Southeast but has no local presence in YP, no local phone numbers, and no local office? That's a thin launch -- probably more press release than substance.

YellowPages scraper is useful for this and also for finding competitors' salespeople (listed as individual agents or reps in some categories). If you know who their top field reps are, you know who's building their territory.

Wiring It Into an Actual Workflow

The pipeline I use:

  • Weekly: SAM.gov contract awards check for top 5 competitors, WHOIS monitoring for domain changes
  • Monthly: Business entity status check for private competitors, new dev.to/company blog RSS monitoring
  • Quarterly: Full EDGAR pull when 10-Q/10-K season hits, manual review of risk factors and MD&A sections

Everything feeds into a shared Notion database with a "status changed" flag for anything that looked different from last time. I built a lightweight diff check in n8n that compares the latest run output to the previous one and only surfaces actual changes.

The whole thing takes maybe 2 hours to set up per competitor and then runs itself. That's a much better use of engineering time than manually compiling slide decks.


The scrapers I use for this are all on my Apify profile. All pay-per-run, no subscription required -- which works well for the kind of periodic monitoring cadence I described above. Start with the EDGAR and SAM.gov ones if your competitors have any federal presence; those return the most signal per run.

Top comments (0)