The Problem: Manual Competitive Research Doesn’t Scale
Picture this: Your product team wants to understand what OpenAI just launched. Your sales team needs to know how Anthropic positions Claude against competitors. Your executives want weekly updates on the AI market landscape.
Right now, someone on your team is:
- Opening 20+ browser tabs to Google different queries
- Copy-pasting snippets into a Google Doc
- Trying to remember which article said what
- Formatting everything into a deck for the Monday meeting
- Starting over next week when the questions change
- Each competitive analysis takes 30–45 minutes. Multiply that by every competitor, every week, and you’ve got a full-time job that’s still slow, inconsistent, and impossible to scale.
There’s a better way.
The Solution: Automate Google Searches with SERP APIs
Here’s what changes when you use Bright Data’s SERP (Search Engine Results Page) API (aff): Instead of manually Googling and clicking through results, you programmatically query search engines and get back structured JSON data. No browser. No clicking. No copy-paste.
In this tutorial, you’ll build a production-ready competitive intelligence agent that:
- Takes company domains as input (openai.com, anthropic.com)
- Runs targeted Google searches via Bright Data SERP API
- Extracts and organizes intelligence automatically
- Generates professional PDF reports with sources
Time to build: 2 hours
Time saved per report: 40+ minutes
Code: ~350 lines of clean Python
Let’s build it.
Why SERP APIs Matter
Before we dive into code, let’s talk about why SERP APIs are the right tool for this job.
The search engine problem: Google actively blocks automated scrapers. You’d need to manage proxies, handle CAPTCHAs, deal with IP bans, and maintain brittle HTML parsers that break when Google changes their layout.
What SERP APIs solve:
- Reliability: Proxy rotation, CAPTCHA solving, and rate limiting handled automatically
- Global coverage: Get search results from any country (gl=US, gl=UK, gl=JP) instantly
- Structured data: Clean JSON responses with titles, URLs, and descriptions already parsed
- Legal compliance: Operates within terms of service — no gray area
- Real impact: With traditional scraping, you’d spend weeks building infrastructure before writing any intelligence logic. With SERP APIs, you get to the value in hours.
Getting Started: Clone and Setup
The complete codebase lives on GitHub. For installation and setup, follow the README:
bash
git clone https://github.com/sanbhaumik/bright-data-serp-apis.git
cd competitive-intel-agent
pip install -r requirements.txt
Important: You’ll need [Bright Data SERP API](https://get.brightdata.com/2039fnr15xfy) credentials. Sign up, create a SERP API zone, and add your credentials to a .env file:
bash
cp .env.example .env
Edit .env with your API_KEY and ZONE
The README walks through setup in detail. This blog focuses on how the code actually works.
Architecture: Ask Better Questions, Get Better Intelligence
The secret to useful competitive intel isn’t running more searches — it’s asking the right questions. Our agent runs 4 strategic searches per company:
Query 1: Market Positioning
“anthropic.com vs competitors”
How do they differentiate?
What's the competitive landscape?
Query 2: Customer Intelligence
“anthropic.com customers case study”
Who uses their products?
What problems are they solving?
Query 3: Strategic Moves
“anthropic.com funding OR acquisition OR partnership”
What deals are they making?
Who's investing?
Query 4: Product Strategy
“anthropic.com product launch OR new feature”
What are they building?
Where's the roadmap heading?
Why this works: These queries map to the questions your executives actually ask.
“How do we compare?”
“Who are their customers?”
“What are they building?”
You get answers, not data dumps.
How the Code Works: 3 Core Components
This section details SerpClient, a Python wrapper for the Bright Data SERP API. It explains how the API solves common scraping problems (proxies, CAPTCHAs, parsing) by providing reliable, structured JSON data for all search queries.
1. SerpClient: Your Data Gateway
serp_client.py wraps the Bright Data SERP API in ~50 lines. Here’s why SERP APIs matter:
The scraping problem: Google actively blocks automated scrapers.
You’d need to:
- Manage proxy rotation (which is an expensive operation)
- Solve CAPTCHAs (honestly, at scale, it is complex)
- Maintain brittle HTML parsers that break constantly
- Handle rate limits and IP bans (frustating, always)
- The SERP API solution: All that infrastructure is handled for you.
You get:
- Reliable access through automatic proxy rotation
- Clean JSON responses (no HTML parsing)
- Global geo-targeting (gl=US, gl=UK, gl=JP)
- Legal compliance within terms of service
The SerpClient class handles authentication and request formatting:
python
def query(self, keyword, gl='us', hl='en'):
payload = {
"zone": self.zone,
"url": f"https://www.google.com/search?q={quote_plus(keyword)}&gl={gl.upper()}&hl={hl}",
"format": "json"
}
response = requests.post(
"https://api.brightdata.com/request",
json=payload,
headers={"Authorization": f"Bearer {self.api_key}"}
)
return response.json()
The get_multiple_results() method adds smart filtering — keeping only results with substantial descriptions (50–500 characters) to filter out low-quality content.
2. CompetitiveIntelAgent: Where Intelligence Happens
enrichment_agent.py (~100 lines) orchestrates the 4 strategic queries and structures the results:
python
def research_company(self, domain, company_name=None):
# Run 4 targeted queries
positioning = self.serp.get_multiple_results(
f'{domain} vs competitors', count=3
)
customers = self.serp.get_multiple_results(
f'{domain} customers case study', count=3
)
strategic_moves = self.serp.get_multiple_results(
f'{domain} funding OR acquisition OR partnership', count=3
)
product_news = self.serp.get_multiple_results(
f'{domain} product launch OR new feature', count=3
)
return {
"company_name": company_name,
"domain": domain,
"positioning": positioning,
"customers": customers,
"strategic_moves": strategic_moves,
"product_strategy": product_news
}
What makes this powerful: Each insight includes the source URL. When your executive asks “where did this come from?”, you have verification built in.
That’s the difference between “interesting research” and “intelligence we can act on.”
3. PDF Generator + Main Orchestration
pdf_generator.py (~200 lines) transforms raw intelligence into professional reports with:
- Title page with metadata
- Organized sections per competitor
- Clickable source URLs for verification
- Clean formatting for executive consumption
main.py ties everything together and provides flexible output formats:
python
reports = research_competitors(
["openai.com", "anthropic.com"],
output_format='text', # or 'json' or 'pdf'
generate_pdf=True
)
ROI: Why This Matters
Manual competitive research takes 45 minutes per competitor. This agent does it in 15 seconds for about twenty cents in API calls.
But if you ask me, speed isn’t the real win — it’s consistency. Manual research quality depends on who’s doing it and when. This agent asks the same strategic questions every time and delivers the same professional format. Your sales team gets reliable intel whether it’s Tuesday morning or Friday afternoon.
Scale to ten competitors: manual research takes a full workday. The agent finishes in under a few minutes. Honestly, it’s the difference between having competitive intelligence and not having it at all.
Production Extensions
Now, of course this architecture grows with your needs. Teams I’ve worked with run weekly automated scans of their top twenty competitors, pushing updates directly into Slack every Monday. Others integrate it into their sales workflow — before every major pitch, the agent researches the prospect’s competitors and drops a briefing into Salesforce.
More sophisticated implementations track trends over time, comparing this week’s intelligence against last month’s to spot momentum shifts. Is a competitor launching features faster? You catch it automatically instead of three months too late.
Because you control the queries, you tune for your industry — healthcare teams track FDA approvals, fintech monitors regulatory news, SaaS watches integration announcements. Same codebase, different strategic questions.
There are so many opportunities here. Tell me in comments what you are thinking of.
Key Takeaway
You can build production-ready competitive intelligence tools in an afternoon that would take weeks with traditional approaches. I have done this, you can too.
This tutorial demonstrates:
- Simple integration — 50 lines for the SERP client
- Real business value — Automated competitive research in 15 seconds
- Clean architecture — Easy to extend and customize
- Professional output — PDF reports executives trust
The code is modular, the approach is scalable, and you can deploy this today.
Next steps: Get your Bright Data SERP API credentials, clone the repo, follow the README, and run your first analysis. Fifteen minutes to set up.
When that PDF generates with fresh intelligence and source attribution, you’ll immediately think of three other use cases. Prove the value, then extend where it matters.
Connect with me on LinkedIn where I share insights on AI engineering, data architecture, and building production AI systems. I write about what actually works in the field — no fluff, just practical implementation strategies.
Top comments (0)