DEV Community: Meir

Building an AI Browser Agent That Actually Works on Ticket Sites

Meir — Tue, 31 Mar 2026 09:34:39 +0000

Last month I tried to buy concert tickets. Ticketmaster, StubHub, SeatGeek, VividSeats - I had eight tabs open, each with different prices, different sections, and zero way to compare them without a spreadsheet. I thought: what if an AI agent could just do this for me?

So I built one.Ticket Hunter is an autonomous AI agent that takes a natural-language query like "Taylor Swift Eras Tour NYC," searches Google for ticket platforms, opens real Chromium browsers on up to three sites simultaneously, visually navigates each one - clicking buttons, scrolling, filling forms - and returns structured, price-ranked ticket results. All streamed live to your screen.

Here's exactly how it works, what powers it, and how you can build something similar.

What You'll See in This Article

How Bright Data's Browser API gives AI agents real browsers with built-in anti-bot bypass
How Yutori's n1 model turns screenshots into browser actions — no DOM parsing needed
The full 5-stage LangGraph pipeline that orchestrates everything
Real code from the project with architecture decisions explained
Gotchas I hit and how I solved them

The Tech Stack

Layer	Technology
Framework	Next.js 16 + React 19 + TypeScript
Agent orchestration	LangGraph (from LangChain)
Web search	Bright Data SERP API
Browser infrastructure	Bright Data Browser API
Visual reasoning	Yutori n1 - pixels-to-actions LLM
Structured extraction	OpenRouter (Gemini)
Browser automation	Playwright Core over CDP
Rate limiting	Upstash Redis

The Two Technologies That Make This Possible

Bright Data Browser API: Real Browsers in the Cloud

Here's the core problem with building a browsing agent: ticket platforms don't want bots. Ticketmaster, StubHub, and their peers use aggressive anti-bot measures — CAPTCHAs, browser fingerprinting, IP reputation scoring, JavaScript challenges. Run a standard Puppeteer script against them and you'll get blocked within seconds.

Bright Data's Browser API solves this at the infrastructure level. It provides fully managed Chromium instances in the cloud that you connect to via Chrome DevTools Protocol (CDP). Behind the scenes, Bright Data handles:

Residential proxy rotation across 150M+ IPs in 195 countries
Automatic CAPTCHA solving — no extra code on your side
Browser fingerprint management — real user-agent strings, canvas fingerprints, WebGL hashes
Intelligent session recovery — if a request fails, it retries with a different IP and fingerprint

The connection is a single WebSocket URL:

// Connect Playwright to Bright Data's cloud browser
const browser = await chromium.connectOverCDP(
  "wss://brd-customer-<id>-zone-<zone>:<password>@brd.superproxy.io:9222"
);
const page = await browser.newPage();
await page.goto("https://www.ticketmaster.com/event/...");

From your code's perspective, it's just a Playwright browser. But it's running on Bright Data's infrastructure with all the unblocking built in. The Browser API handles over 100M+ AI agent interactions per day across their customer base — this is battle-tested infrastructure, not a hobby project.

Why this matters for AI agents specifically: When your AI agent decides to click a button or navigate to a page, you can't afford a CAPTCHA popup breaking the flow 15 steps in. Bright Data solves this transparently at the network layer so the agent logic stays clean.

Yutori n1: The "Pixels-to-Actions" Model

Most browser automation works by parsing the DOM — find elements by CSS selectors, extract text from HTML nodes. This breaks constantly. Sites change their markup, use shadow DOM, render content dynamically, or generate randomized class names.

Yutori n1 takes a radically different approach: it looks at screenshots. Built by three former Meta FAIR researchers (Devi Parikh, Abhishek Das, and Dhruv Batra), n1 is a vision-language model specifically trained for web navigation. You give it:

A screenshot of the current page
A text description of the task
History of previous actions

And it returns the next action: click at coordinates (423, 567), type "Taylor Swift", scroll down, etc.

The numbers are compelling:

Benchmark	n1	Claude Opus	GPT-5.4	Gemini 2.5 Pro
Navi-Bench v1 (100 real tasks)	91%	88%	78%	53%
Online-Mind2Web (300 tasks, 136 sites)	78.7%	61.0%	—	69.0%

n1 also runs at 3.6 seconds per step — 2-3x faster than alternatives — and costs $0.75/M input tokens, roughly 8x cheaper than competing vision models. The API follows the OpenAI Chat Completions format, so integration is straightforward:

import OpenAI from "openai";

const n1Client = new OpenAI({
  apiKey: process.env.YUTORI_API_KEY,
  baseURL: "https://api.yutori.com/v1",
});

const response = await n1Client.chat.completions.create({
  model: "n1-latest",
  messages: [
    {
      role: "system",
      content: "You are a web browsing agent specialized in finding ticket information..."
    },
    {
      role: "user",
      content: [
        { type: "text", text: "Find available tickets for Taylor Swift Eras Tour" },
        { type: "image_url", image_url: { url: screenshotDataUrl } }
      ]
    }
  ],
  tools: browserTools, // click, type, scroll, etc.
});

n1 responds with tool calls — left_click at (x, y), type "Taylor Swift", scroll down — which Playwright executes on the Bright Data browser. Then you screenshot again, send the new state back, and repeat.

The 5-Stage Pipeline

The agent runs as a LangGraph state graph with five stages:

                    ┌─────────────────┐
                    │   User Query    │
                    │ "Taylor Swift   │
                    │  Eras Tour NYC" │
                    └────────┬────────┘
                             │
                    ┌────────▼────────┐
                    │  Stage 1: SERP  │
                    │  Search via     │
                    │  Bright Data    │
                    └────────┬────────┘
                             │
              ┌──────────────┼──────────────┐
              │              │              │
     ┌────────▼───┐  ┌──────▼─────┐  ┌────▼────────┐
     │  Stage 2:  │  │  Stage 2:  │  │  Stage 2:   │
     │  Browser 1 │  │  Browser 2 │  │  Browser 3  │
     │ Ticketmstr │  │  StubHub   │  │  SeatGeek   │
     └────────┬───┘  └──────┬─────┘  └────┬────────┘
              │              │              │
     ┌────────▼───┐  ┌──────▼─────┐  ┌────▼────────┐
     │  Stage 3:  │  │  Stage 3:  │  │  Stage 3:   │
     │ n1 Browse  │  │ n1 Browse  │  │  n1 Browse  │
     │ (15 steps) │  │ (15 steps) │  │  (15 steps) │
     └────────┬───┘  └──────┬─────┘  └────┬────────┘
              │              │              │
     ┌────────▼───┐  ┌──────▼─────┐  ┌────▼────────┐
     │  Stage 4:  │  │  Stage 4:  │  │  Stage 4:   │
     │  Extract   │  │  Extract   │  │  Extract    │
     │  Tickets   │  │  Tickets   │  │  Tickets    │
     └────────┬───┘  └──────┬─────┘  └────┬────────┘
              │              │              │
              └──────────────┼──────────────┘
                             │
                    ┌────────▼────────┐
                    │  Stage 5: Merge │
                    │  Deduplicate &  │
                    │  Rank by Price  │
                    └─────────────────┘

Let me walk through each stage.

Stage 1: SERP Search — Finding the Right Platforms

The agent doesn't hardcode URLs. It searches Google via Bright Data's SERP API:

import { BrightDataClient } from "@brightdata/sdk";

const bdclient = new BrightDataClient({ token: process.env.BRIGHT_DATA_API_TOKEN });
const results = await bdclient.scrape({
  url: `https://www.google.com/search?q=${encodeURIComponent(query + " tickets buy")}&brd_json=1`,
  zone: "serp",
  format: "raw",
});

The raw SERP JSON gets parsed through a scoring algorithm that:

Detects known ticket platforms (Ticketmaster, StubHub, SeatGeek, VividSeats, AXS, TickPick, Gametime, Eventbrite, Viagogo, TicketNetwork, Tickets.com)
Scores each URL by relevance — event-specific URLs score higher than homepage results
Checks for query token matches in titles and snippets
Returns the top 3 platform URLs

This means the agent adapts to any event. Search "Lakers vs Celtics" and it finds basketball ticket pages. Search "Hamilton Broadway" and it finds theater listings. No hardcoded routes.

Stage 2: Browser Connection — Opening Real Browsers

For each of the top 3 URLs, the agent opens a Bright Data Browser API session via CDP:

const browser = await chromium.connectOverCDP(cdpUrl);
const context = browser.contexts()[0] ?? await browser.newContext();
const page = context.pages()[0] ?? await context.newPage();
await page.setViewportSize({ width: 1280, height: 800 });

Before navigating, the agent injects a script that prevents popups (ticket sites love target="_blank" links that would break the single-tab agent flow):

// Override target="_blank" to keep navigation in same tab
await page.evaluate(() => {
  const observer = new MutationObserver((mutations) => {
    for (const m of mutations) {
      m.addedNodes.forEach((node) => {
        if (node instanceof HTMLAnchorElement && node.target === "_blank") {
          node.target = "_self";
        }
      });
    }
  });
  observer.observe(document.body, { childList: true, subtree: true });
});

Navigation includes retry logic with exponential backoff. If the page returns a 403, 429, or shows CAPTCHA indicators, the agent retries — and Bright Data's infrastructure handles the actual unblocking behind the scenes.

Stage 3: n1 Agentic Browse Loop — The Heart of the Agent

This is where n1 shines. For each platform, the agent runs a loop of up to 15 steps:

for (let step = 0; step < maxSteps; step++) {
  // 1. Take screenshot
  const screenshot = await page.screenshot({ type: "png" });
  const dataUrl = `data:image/png;base64,${screenshot.toString("base64")}`;

  // 2. Send to n1 with task context + screenshot
  const response = await n1Client.chat.completions.create({
    model: "n1-latest",
    messages: conversationHistory,
    tools: BROWSER_TOOLS,
  });

  // 3. If no tool calls, n1 is done — extract final answer
  if (!response.choices[0].message.tool_calls?.length) {
    finalAnswer = response.choices[0].message.content;
    break;
  }

  // 4. Execute each action
  for (const toolCall of response.choices[0].message.tool_calls) {
    await executeAction(page, toolCall); // click, type, scroll, etc.
  }

  // 5. Stream screenshot to the UI
  emitEvent({ type: "screenshot", data: dataUrl, source: platform });
}

n1 supports 14 action types: left_click, double_click, triple_click, right_click, type, key_press, scroll, hover, drag, goto_url, go_back, refresh, and wait. Coordinates come in a normalized 0-1000 space and get scaled to the 1280x800 viewport.

What makes this powerful: n1 sees the page. It doesn't care if Ticketmaster renames its CSS classes. It doesn't break when StubHub redesigns their layout. It looks at the screenshot and figures out what to click, just like you would.

Stage 4: Structured Extraction — From Text to Data

When n1 finishes browsing, it returns a natural-language summary of what it found. To get structured data, the agent sends this to OpenRouter's Gemini Flash with a strict JSON schema:

const response = await openrouterClient.chat.completions.create({
  model: "google/gemini-3-flash-preview",
  messages: [
    { role: "system", content: "Extract ticket listings as structured JSON..." },
    { role: "user", content: n1FinalAnswer }
  ],
  response_format: {
    type: "json_schema",
    json_schema: {
      name: "ticket_results",
      strict: true,
      schema: {
        type: "object",
        properties: {
          tickets: {
            type: "array",
            items: {
              type: "object",
              properties: {
                eventName: { type: "string" },
                eventDate: { type: "string" },
                venue: { type: "string" },
                section: { type: "string" },
                price: { type: "string" },
                platform: { type: "string" },
                url: { type: "string" },
                // ... 7 more fields
              }
            }
          }
        }
      }
    }
  }
});

If extraction fails (it happens — n1 might report "no tickets found" for a sold-out event), the agent falls back to displaying n1's raw text answer.

Stage 5: Merge & Rank — The Final Output

All tickets from all platforms get merged, deduplicated (by a composite key of platform + section + row + seats + price + URL), and sorted by price ascending. The user sees a clean grid of ticket cards.

The Real-Time Streaming Architecture

One of the things I'm most proud of is the live streaming. While the agent works, users see:

Live browser screenshots updating in real-time across up to 3 panels
An activity log with color-coded events (SERP search started, browser connected, n1 clicking "View Tickets", extraction complete...)
Agent statistics (SERP queries made, pages browsed, AI actions taken, total steps)

This uses Server-Sent Events (SSE) from the Next.js API route:

// API route: POST /api/search
const stream = new ReadableStream({
  async start(controller) {
    const emit = (event: AgentEvent) => {
      controller.enqueue(`data: ${JSON.stringify(event)}\n\n`);
    };

    // Run the agent with the emitter
    await runTicketHunterAgent(initialState, emit);

    emit({ type: "done" });
    controller.close();
  }
});

return new Response(stream, {
  headers: { "Content-Type": "text/event-stream" }
});

On the client side, a custom useTicketSearch hook consumes the SSE stream and manages React state for browser sessions, status entries, and tickets.

The emitter function propagates through the async call stack using Node.js AsyncLocalStorage — this way, deeply nested functions (like the n1 browse loop) can emit events without passing callbacks through every function signature.

What Can Go Wrong (and What I Learned)

Popup chaos

Ticket sites open everything in new tabs. An AI agent navigating with left_click suddenly has its page go blank because the click opened a new window. I solved this with DOM MutationObservers that rewrite target="_blank" to target="_self" in real-time, plus logic to detect when Playwright spawns a popup and switch to it gracefully.

Coordinate translation bugs

n1 returns coordinates in a 0-1000 normalized space. Early on I had off-by-one scaling bugs that caused the agent to click 10 pixels to the right of every button. Small enough to not notice on large buttons, devastating on dropdown menus. The fix: explicit clamping after scaling.

const scaledX = Math.round((x / 1000) * 1280);
const scaledY = Math.round((y / 1000) * 800);
const clampedX = Math.max(0, Math.min(scaledX, 1279));
const clampedY = Math.max(0, Math.min(scaledY, 799));

Hard blocks vs. soft blocks

Not all "blocked" states are the same. A 403 response is a hard block — retry with a new session. But a Cloudflare "checking your browser" interstitial is a soft block that Bright Data's infrastructure will resolve if you wait a few seconds. I had to build detection for 11 hard-block patterns and 9 soft-block patterns, with different retry strategies for each.

The 15-step limit

I set the n1 browse loop to max 15 steps per platform. Too few and the agent can't navigate complex flows (Ticketmaster's event → date selection → section → seat picker is 6+ clicks minimum). Too many and you burn tokens and time. 15 was the sweet spot — enough for most flows, with a forced final-answer call if the limit is reached.

Extraction failures

Sometimes n1 finds tickets but describes them in a way that the structured extraction LLM can't parse cleanly. The fallback — showing n1's raw text answer — saves the user experience. Better to show "Section 102, Row A, $185 each" as plain text than nothing at all.

The Result

A single search takes 30-90 seconds and typically returns 10-30 structured ticket listings across 3 platforms, ranked by price. The user watches the agent work in real-time — clicking through Ticketmaster's seat map, scrolling StubHub's listings, navigating SeatGeek's filters — which is genuinely fun to watch.

The full pipeline:

1 SERP query → top 3 platform URLs
3 parallel browser sessions on Bright Data's infrastructure
Up to 45 n1 vision steps (15 per platform)
3 structured extractions via Gemini Flash
1 merged, deduplicated, price-ranked result set

Running It Yourself

The project is open source:

git clone https://github.com/brightdata/ticket-hunter-agent
cd ticket-hunter-agent
npm install

You'll need API keys for:

Bright Data — Browser API zone + SERP API zone (sign up here)
Yutori — n1 API key (docs.yutori.com)
OpenRouter — for Gemini Flash extraction (openrouter.ai)
Upstash Redis (optional) — for rate limiting

Create a .env.local:

BRD_CDP_URL=wss://brd-customer-<id>-zone-<zone>:<password>@brd.superproxy.io:9222
BRIGHT_DATA_API_TOKEN=your-token
YUTORI_API_KEY=your-key
OPENROUTER_API_KEY=your-key
UPSTASH_REDIS_REST_URL=your-url    # optional
UPSTASH_REDIS_REST_TOKEN=your-token # optional

Then:

npm run dev

Open http://localhost:3000 and search for any event.

Key Takeaways

Vision-based browsing > DOM parsing for sites that fight automation. Yutori n1 doesn't care about class name changes or shadow DOM — it sees pixels.
Infrastructure matters. Bright Data's Browser API handles the hardest part (anti-bot bypass, CAPTCHAs, proxy rotation) so you can focus on agent logic instead of unblocking.
LangGraph's Send API makes parallel branching clean — fan out to 3 browser sessions and merge results back without callback spaghetti.
Always build fallbacks. Structured extraction fails sometimes. Have a graceful degradation path.
Stream everything. Users are far more patient when they can watch the agent work.

FAQ

Q: Does this actually buy tickets?
A: No — Ticket Hunter finds and compares available tickets across platforms. It doesn't complete purchases. The result cards link directly to the platform where you can buy.

Q: How much does it cost to run one search?
A: Roughly $0.02-0.05 in Bright Data bandwidth (3 browser sessions), $0.01-0.03 in n1 tokens (up to 45 steps with screenshots), and <$0.01 for the SERP query and extraction. Total: under $0.10 per search.

Q: Can I add more ticket platforms?
A: Yes — the SERP search automatically discovers platforms. Add the domain to the platform detection map in bright-data.ts and the agent will start browsing it.

Q: Why not just scrape the HTML directly?
A: Ticket platforms use heavy JavaScript rendering, dynamic pricing, interactive seat maps, and anti-scraping measures. A vision-based agent that navigates like a human handles all of these naturally.

Q: What if Bright Data's Browser API is down?
A: Bright Data reports 99.99% uptime across their infrastructure. In 3 months of development, I've had zero Browser API outages.

Have questions about building browsing agents? Drop them in the comments — I'll answer all of them. If you build something cool with Bright Data's Browser API or Yutori n1, I'd love to see it.

GitHub Repo | Bright Data Browser API Docs | Yutori n1 Docs

How I Built an AI Agent That Maps Any Company's Supply Chain in Real-Time

Meir — Mon, 02 Mar 2026 16:33:28 +0000

Supply chain visibility is one of those problems that sounds straightforward until you actually try to solve it. Every company has suppliers. Every supplier has their own suppliers. Customers have distributors who have sub-distributors. And almost none of it is in a clean, queryable database.

I wanted to see if an AI agent could discover this web of relationships automatically - given only a company name - by reading the same public evidence a human analyst would: news articles, supplier directories, annual reports, procurement documents, partner pages, and ISO certification databases.

The result is Supply Chain Sankey: an agent that autonomously discovers upstream suppliers and downstream distributors for any company, backs every relationship with source URLs and confidence scores, and renders the whole thing as an interactive Sankey diagram.

Here's exactly how it works and what I learned building it.

The Problem With Supply Chain Discovery

If you want to know who supplies Apple, you could:

Read Apple's annual report (partial, high-level)
Scrape SEC filings (incomplete, lags by months)
Buy expensive commercial intelligence data
Hire analysts to manually research it

None of these scale, and most are expensive. But the information is publicly available — it's just scattered across thousands of web pages, PDFs, press releases, and partner directories.

This is the gap AI agents can fill. An agent that can search, scrape, and reason over public web content can replicate what an analyst does, but at machine speed and scale.

Architecture Overview

Before diving into the details, here's the full data flow:

Four distinct systems doing the heavy lifting:

Bright Data — web search and scraping (bypasses bot detection)
AWS Bedrock AgentCore — serverless agent hosting with auto-scaling
LangGraph — orchestrating the multi-step agent pipeline
Amazon Nova 2 Lite — the LLM doing query planning, evidence analysis, and classification

Try it live: demos.brightdata.com/supplychain-sankey

Step 1: Query Planning

The agent starts with just a company name. The first LLM call generates 6-7 targeted search queries designed to surface supply chain signals:

PLANNER_SYSTEM_PROMPT = """
Generate {max_queries} targeted search queries to discover
{direction} supply chain relationships for {company}.

Queries should surface:
- Supplier directories and partner pages
- Procurement documents and RFQ filings
- ISO/IATF/AS9100 certificates (often list suppliers by name)
- Annual reports with named suppliers/customers
- Trade publications with named relationships
- filetype:pdf site:example.com for procurement docs
"""

For a query like "Apple Inc. upstream suppliers", the planner might generate:

"Apple Inc" supplier manufacturing partner site:apple.com
"Apple supply chain" TSMC Samsung filetype:pdf
"Apple Inc" component procurement 2024
site:linkedin.com "Apple supplier" manufacturer
"Apple supplier" ISO certificate directory

The key insight: you want diverse query types. Procurement PDFs contain different information than press releases, which differ from supplier directories.

Step 2: Web Search via Bright Data

Each query goes to Bright Data's Web Unlocker API. This is where Bright Data does something standard HTTP clients can't: it handles anti-bot systems, CAPTCHAs, JavaScript rendering, and geo-targeting automatically.

def bright_data_search(query: str, num_results: int = 5) -> list[dict]:
    response = requests.get(
        "https://api.brightdata.com/request",
        headers={"Authorization": f"Bearer {BRIGHT_DATA_API_KEY}"},
        params={
            "url": f"https://www.google.com/search?q={quote(query)}&num={num_results}",
            "zone": BRIGHT_DATA_ZONE,
            "data_format": "raw",
        },
    )
    # Parse and normalize [{url, title, snippet}]
    return parse_search_results(response.json())

The 6-7 queries run in parallel using ThreadPoolExecutor with 6 workers — so a full discovery run takes roughly the same time as a single sequential search.

One thing I learned: bounding search results matters. Too few (< 2) and you miss relationships. Too many (> 8) floods the URL ranking step with noise. The sweet spot is 5 results per query.

Step 3: URL Ranking

After parallel search, you have ~40-50 URLs. Scraping all of them would be slow and wasteful. An LLM ranking step selects the top 10:

URL_RANKER_SYSTEM_PROMPT = """
Rank these URLs by their likely value for supply chain discovery.

PREFER:
- Supplier/partner directories
- Procurement documents and RFQ filings
- ISO/IATF certification databases (often list supplier names)
- Annual reports with named counterparties
- PDFs (strong signal — often formal business documents)

DEPRIORITIZE:
- Generic company about pages
- Repair manuals and user guides
- News without named relationships
- Login walls
- Social media profiles
"""

This step is cheap (small prompt, short response) but saves significant time in the expensive scraping step.

Step 4: Scraping with Bright Data

The selected URLs get scraped in parallel. Bright Data handles two content types differently:

Web pages → fetched as cleaned Markdown:

response = requests.get(
    "https://api.brightdata.com/request",
    headers={"Authorization": f"Bearer {BRIGHT_DATA_API_KEY}"},
    params={
        "url": target_url,
        "zone": BRIGHT_DATA_ZONE,
        "data_format": "markdown",  # Bright Data converts HTML → Markdown
    },
)

PDFs → fetched as binary, then extracted with pypdf:

if url.endswith(".pdf") or "filetype=pdf" in url:
    binary_response = requests.get(
        "https://api.brightdata.com/request",
        params={"url": url, "zone": BRIGHT_DATA_ZONE, "format": "raw"},
    )
    pdf_reader = pypdf.PdfReader(io.BytesIO(binary_response.content))
    text = "\n".join(page.extract_text() for page in pdf_reader.pages)

PDFs are often the most valuable source — formal supplier lists, procurement specifications, ISO audit reports. The binary + pypdf approach handles them cleanly.

The data_format="markdown" trick is worth highlighting: instead of receiving raw HTML and parsing it, Bright Data strips boilerplate and returns clean content. This alone reduces the LLM context needed in the next step.

Step 5: LLM Evidence Reflection

Each scraped page gets sent to the LLM for compression. A raw supplier directory page might be 15,000 characters; what we actually need is 500 characters of structured evidence.

REFLECTION_SYSTEM_PROMPT = """
Extract ONLY supply-chain-relevant content from this page.

Return JSON:
{
  "content": "compressed evidence text (max 900 chars)",
  "highlights": ["key relationship 1", "key relationship 2"],
  "signal_score": 0.0-1.0,
  "named_entities": ["Company A", "Company B"]
}

INCLUDE: supplier/customer/partner relationships, ISO/IATF certificates,
procurement language, shipping/manufacturing evidence, distributor networks.

EXCLUDE: marketing copy, product descriptions, pricing, contact info,
job listings, generic company information.
"""

The signal_score field is critical. Pages scoring ≤ 0.05 get dropped before the expensive downstream LLM calls. This acts as a noise filter — a repair manual might mention a company's name but has zero supply chain signal.

Step 6: Counterparty Parsing and Edge Construction

High-signal evidence gets parsed to extract actual relationships:

# LLM returns structured counterparty data
{
  "counterparties": [
    {
      "name": "Taiwan Semiconductor Manufacturing Company",
      "relationship_type": "supplier",
      "confidence": 0.92,
      "direction": "upstream",
      "evidence_snippet": "TSMC manufactures Apple's A-series chips..."
    }
  ]
}

Edge construction is then deterministic — no LLM needed:

for counterparty in parsed.counterparties:
    edge = {
        "source": counterparty.name if upstream else company,
        "target": company if upstream else counterparty.name,
        "direction": counterparty.direction,
        "relationship_type": counterparty.relationship_type,
        "confidence": counterparty.confidence,
        "tier": -1 if upstream else +1,
        "evidence_urls": source_urls,
    }
    proposed_edges.append(edge)

Deduplication merges edges with the same (source, target, direction, relationship_type) pair, taking the highest confidence score.

Step 7: Entity Filtering

One problem with LLM-extracted entity names: generic terms slip through. An evidence snippet saying "Apple works with authorized suppliers" might produce a node named "authorized suppliers" — which isn't a real company.

An entity filter LLM call handles this:

ENTITY_FILTER_SYSTEM_PROMPT = """
Classify each candidate entity name as a real organization or a generic placeholder.

KEEP: "Ingram Micro (China) Limited", "Foxconn Technology Group", "TSMC"
DROP: "Suppliers", "Partners", "Manufacturers", "Authorized Dealers", "OEM vendors"

Return JSON: {"keep": [...], "drop": [...]}
"""

This step significantly reduces noise in the final graph.

Running on AWS Bedrock AgentCore

The entire LangGraph pipeline runs inside AWS Bedrock AgentCore — a managed runtime for AI agents that handles containerization, scaling, health checks, and observability.

The config is minimal:

# .bedrock_agentcore.yaml
agentId: supplychain_sankey-dKHoaWDft4
region: us-east-1
runtime:
  language: python
  version: "3.11"
  architecture: arm64
network:
  mode: PUBLIC
observability:
  enabled: true

The entrypoint wraps the LangGraph call:

from bedrock_agentcore import BedrockAgentCoreApp

app = BedrockAgentCoreApp()

@app.entrypoint
def handler(payload, context):
    company = payload["company"]
    direction = payload.get("direction", "both")
    mode = payload.get("mode", "discover")

    result = run_supply_chain_discovery(
        company=company,
        direction=direction,
        mode=mode,
        tier_offset=payload.get("tier_offset", 0),
        existing_entities=payload.get("existing_entities", []),
    )

    return result

AgentCore handles the health check endpoint (/ping returns HEALTHY_BUSY while processing), CloudWatch logs, and deployment via CodeBuild → ECR. What would otherwise require a full ECS service or Kubernetes setup is reduced to a agentcore deploy command.

The Lambda proxy sits in front, routing requests from the Next.js frontend to the AgentCore runtime:

# lambda_invoker/handler.py
response = bedrock_agentcore_client.invoke_agent_runtime(
    agentRuntimeArn=AGENT_ARN,
    payload=json.dumps(request_body),
)
# Stream chunks back to the frontend

The Visualization: D3 Sankey

The final output is a Sankey JSON structure that D3 renders as a flow diagram:

{
  "nodes": [
    {"id": "Apple Inc.", "tier": 0},
    {"id": "TSMC", "tier": -1},
    {"id": "Ingram Micro", "tier": 1}
  ],
  "links": [
    {
      "source": "TSMC",
      "target": "Apple Inc.",
      "tier": -1,
      "direction": "upstream",
      "relationship_type": "supplier",
      "confidence": 0.92,
      "status": "confirmed",
      "evidence_urls": ["https://..."]
    }
  ]
}

Node colors encode position in the supply chain:

Blue (tier 0): the root company
Purple (tier < 0): upstream suppliers
Cyan (tier > 0): downstream distributors

Link colors encode confidence:

Green: confirmed relationships
Amber: unconfirmed (plausible but not definitively verified)
Red: rejected

Clicking a node opens an evidence drawer showing the source URLs. Clicking expand buttons triggers another AgentCore invocation to discover the supplier's suppliers — recursively building a deeper map.

What Can Go Wrong

Generic entity leakage: Even with the entity filter, terms like "key partners" occasionally survive. A second-pass filter with more specific examples in the prompt helps.

PDF extraction failures: Some PDFs are image-only (scanned documents). pypdf extracts nothing. The fallback — fetching the PDF URL as markdown — sometimes returns something useful from the surrounding HTML, but often doesn't. Worth logging these separately.

LLM hallucination in entity names: The evidence reflection step occasionally generates entity names that combine two separate companies from the same page. Adding "only extract names that appear verbatim in the source" to the prompt reduced this significantly.

Results

Running the agent on Apple Inc. produces a Sankey diagram with ~14-18 nodes and ~16-20 links in roughly 45-90 seconds, depending on scraping latency. The relationships discovered include:

TSMC, Samsung Electro-Mechanics, Murata, Foxconn (upstream suppliers)
Ingram Micro, TD Synnex, Authorized Resellers (downstream distributors)
Each backed by actual source URLs with specific evidence snippets

Expanding a supplier node (e.g., TSMC) then discovers TSMC's upstream relationships — recursively building a multi-tier map.

Key Takeaways

Bright Data Web Unlocker removes the anti-bot problem entirely — pages that block standard scrapers are fetched cleanly, including JavaScript-rendered content and PDFs
Signal scoring before LLM calls is essential for cost control — filter obvious noise early
AWS Bedrock AgentCore makes deploying a LangGraph agent to production straightforward — no container orchestration overhead
Amazon Nova 2 Lite runs reliably at temperature=0.0 for structured JSON extraction tasks
Entity filtering is a non-optional step — without it, generic role names pollute the graph

FAQ

Can this run against any company?
Yes — any publicly traded or well-documented private company with web presence. Less-documented companies return fewer nodes.

How accurate are the relationships?
Confidence scores reflect evidence quality. "Confirmed" edges (green) come from multiple independent sources. "Unconfirmed" (amber) means one plausible source.

What does Bright Data cost for this workload?
A single discovery run makes ~50 search requests and ~10 scrape requests. At typical Web Unlocker pricing this is a few cents per run.

Can I expand to 3rd-tier suppliers?
Yes — the expand button triggers a new AgentCore invocation with tier_offset set so the new nodes appear at the correct depth in the existing graph.

Does this replace commercial supply chain intelligence tools?
No. Commercial tools have curated databases, historical tracking, and coverage guarantees. This agent is best for discovery and exploration — finding relationships you didn't know to look for.

Running It Yourself

The full repo is open source. Deploy order:

AgentCore: agentcore deploy (handles ECR + runtime provisioning)
Lambda proxy: sam deploy in infra/
Frontend: npm run dev with LAMBDA_URL set

Required env vars: BRIGHT_DATA_API_KEY, BRIGHT_DATA_ZONE.

GitHub repo → supplychain-sankey

If you build something similar or extend this to other domains (M&A networks, academic citation graphs, ecosystem mapping), drop a comment — I'd like to see what variations emerge.

Building PeopleHub: An AI-Powered LinkedIn Intelligence Platform with LangGraph and Bright Data

Meir — Fri, 28 Nov 2025 08:38:37 +0000

I recently open-sourced PeopleHub, an AI-powered people search engine that combines natural language query parsing, LinkedIn profile scraping, and automated research report generation. In this post, I'll walk through the technical architecture, key design decisions, and implementation details.

What is PeopleHub?
Architecture Overview
Tech Stack
AI Query Parser: Natural Language → Structured Search
Bright Data Integration: The Data Pipeline
Multi-Tier Caching Strategy
LangGraph Research Engine: Agentic Workflows
Database Design with Prisma
Performance Optimizations
Lessons Learned

What is PeopleHub?

PeopleHub solves a common problem: finding and researching professionals is either slow (manual LinkedIn searching) or expensive (premium tools charging $50+ per profile).

Key Features:

🗣️ Natural language search - Just type "10 AI engineers in Israel"
⚡ Smart caching - 70-90% cost reduction through intelligent caching
🔬 AI research reports - Automated due diligence with web scraping
💾 Multi-tier persistence - PostgreSQL + Redis for optimal performance
🤖 LangGraph workflows - Agentic multi-step research automation

Use Cases:

Recruiting and talent acquisition
Due diligence on executives/entrepreneurs
Competitive intelligence
Academic research on professional networks
Sales prospecting

Architecture Overview

┌─────────────────────────────────────────────────┐
│          Frontend (Next.js 15 + React)          │
│    SearchBar → Results → Research Reports       │
└────────────────────┬────────────────────────────┘
                     │
        ┌────────────┼────────────┐
        │            │            │
   ┌────▼────┐  ┌───▼──┐  ┌─────▼──────┐
   │ Search  │  │Image │  │ Research   │
   │   API   │  │Proxy │  │    API     │
   └────┬────┘  └──────┘  └─────┬──────┘
        │                        │
        │     ┌──────────────────┴───────┐
        │     │  Prisma ORM + PostgreSQL │
        │     │  Redis Cache (Optional)  │
        │     └──────────┬─────────────┬─┘
        │                │             │
   ┌────▼────────────┐   │    ┌───────▼────────┐
   │  AI Query       │   │    │  LangGraph     │
   │  Parser         │   │    │  Research      │
   │ (Gemini 2.0)    │   │    │  Workflows     │
   └─────────────────┘   │    └───────┬────────┘
                         │            │
                    ┌────▼────────────▼──┐
                    │   Bright Data APIs │
                    │ • Google Search    │
                    │ • LinkedIn Scraper │
                    │ • Web Unblocker    │
                    │ • MCP Server       │
                    └────────────────────┘

Tech Stack

Backend

Framework: Next.js 15.5.4 with App Router (API Routes)
Runtime: Node.js 18+
Language: TypeScript 5 (strict mode)
ORM: Prisma 6.5.0
Database: PostgreSQL (Supabase)
Cache: Redis with ioredis 5.8.2 (optional, hot cache)

AI/LLM

Query Parsing: Google Gemini 2.0 Flash (gemini-2.0-flash-exp)
AI SDK: Vercel AI SDK 5.0.60 (@ai-sdk/google 2.0.17)
Research Workflows: LangChain + LangGraph 1.0.1
Schema Validation: Zod 3.25.76

External APIs

Bright Data: Google Search API, LinkedIn Scraper API, Web Scraper
Custom MCP Client: Model Context Protocol SDK 1.19.1 for advanced tool access

Frontend

UI: React 19.1.0 with Next.js
State: Zustand 5.0.2 + TanStack Query 5.62.18
Styling: Tailwind CSS 4 with custom animation utilities

AI Query Parser

The Problem

Users shouldn't need to learn complex query syntax. They should be able to search naturally:

✅ "10 AI engineers in Israel"
✅ "Software engineers at Google"
✅ "Elon Musk"
✅ "Product managers in San Francisco with startup experience"

The Solution: Structured Output with Gemini 2.0 Flash

I use Vercel's AI SDK with Gemini 2.0 Flash to convert natural language into structured search parameters using Zod schemas:

// src/lib/search/parser.ts
import { google } from '@ai-sdk/google';
import { generateObject } from 'ai';
import { z } from 'zod';

const SearchQuerySchema = z.object({
  count: z.number().min(1).max(50)
    .describe('Number of profiles to find'),
  role: z.string().nullable()
    .describe('Job title or role'),
  location: z.string().optional().nullable()
    .describe('Location or company name'),
  countryCode: z.string().length(2).optional().nullable()
    .describe('2-letter ISO country code'),
  keywords: z.array(z.string())
    .describe('Additional keywords or qualifications'),
  googleQuery: z.string()
    .describe('Optimized Google search query for LinkedIn'),
});

export async function parseSearchQuery(
  query: string
): Promise<ParsedSearchQuery> {
  const { object } = await generateObject({
    model: google('gemini-2.0-flash-exp'),
    schema: SearchQuerySchema,
    prompt: `Parse this search query: "${query}"

    Handle two types:
    1. Job/role search: "5 AI Engineers in Israel"
       → Extract count, role, location, keywords
    2. Individual search: "Elon Musk"
       → Set count=1, role=null, name in keywords

    Generate optimized Google query using:
    site:linkedin.com/in "Role" "Location" keywords`,
  });

  return object;
}

Example Output

Input: "5 AI Engineers in Israel"

Output:

{
  "count": 5,
  "role": "AI Engineer",
  "location": "Israel",
  "countryCode": "IL",
  "keywords": [],
  "googleQuery": "site:linkedin.com/in \"AI Engineer\" \"Israel\""
}

Why Gemini 2.0 Flash?

Fast: ~200-500ms response time
Structured Output: Native Zod schema support
Flexible: Handles both job searches and individual lookups
Cost-effective: $0.00001875 per 1K input tokens

Bright Data Integration

Bright Data is the backbone of PeopleHub's data acquisition. I use three of their APIs:

1. Google Search API

Purpose: Find LinkedIn profile URLs matching search criteria

Implementation:

// src/lib/brightdata/search.ts
const BRIGHTDATA_API_URL = 'https://api.brightdata.com/request';

export async function searchGoogle(
  query: string,
  page: number = 0,
  countryCode?: string | null,
): Promise<BrightDataGoogleSearchResponse> {
  const searchUrl = buildGoogleSearchUrl(query, page, countryCode);

  const response = await fetch(BRIGHTDATA_API_URL, {
    method: 'POST',
    headers: {
      Authorization: `Bearer ${process.env.BRIGHTDATA_API_TOKEN}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      url: searchUrl,
      zone: 'unblocker',
      format: 'raw',
    }),
  });

  return response.json();
}

function buildGoogleSearchUrl(
  query: string,
  page: number,
  countryCode?: string | null
): string {
  const start = page * 10;
  let url = `https://www.google.com/search?q=${encodeURIComponent(query)}&start=${start}&brd_json=1`;

  // Geo-targeting support
  if (countryCode) {
    url += `&gl=${countryCode.toUpperCase()}`;
  }

  return url;
}

Key Features:

JSON Response Format: brd_json=1 parameter returns structured data
Geo-Targeting: gl parameter filters results by country
Site-Specific Queries: site:linkedin.com/in narrows to LinkedIn profiles
Organic Results: Returns titles, links, snippets without ads

2. LinkedIn Scraper API

Purpose: Extract comprehensive LinkedIn profile data

Dataset ID: gd_l1viktl72bvl7bjuj0

Implementation:

// src/lib/brightdata/linkedin.ts
const BRIGHTDATA_API_URL = 'https://api.brightdata.com/datasets/v3';
const LINKEDIN_DATASET_ID = 'gd_l1viktl72bvl7bjuj0';
const MAX_POLLING_ATTEMPTS = 600; // 10 minutes
const POLLING_INTERVAL_MS = 1000; // 1 second

async function triggerLinkedInScrape(
  urls: string[]
): Promise<string> {
  const triggerUrl = `${BRIGHTDATA_API_URL}/trigger`;
  const params = new URLSearchParams({
    dataset_id: LINKEDIN_DATASET_ID,
    include_errors: 'true',
  });

  const payload = urls.map(url => ({ url }));

  const response = await fetch(`${triggerUrl}?${params}`, {
    method: 'POST',
    headers: getApiHeaders(),
    body: JSON.stringify(payload),
  });

  const data = await response.json();
  return data.snapshot_id;
}

async function pollForSnapshot(
  snapshotId: string
): Promise<BrightDataLinkedInResponse[]> {
  let attempts = 0;

  while (attempts < MAX_POLLING_ATTEMPTS) {
    const snapshotUrl = `${BRIGHTDATA_API_URL}/snapshot/${snapshotId}`;
    const params = new URLSearchParams({ format: 'json' });

    const response = await fetch(`${snapshotUrl}?${params}`, {
      method: 'GET',
      headers: getApiHeaders(),
    });

    const data = await response.json();

    // Check if still processing
    if (!Array.isArray(data) && data.status === 'running') {
      attempts++;
      await new Promise(resolve =>
        setTimeout(resolve, POLLING_INTERVAL_MS)
      );
      continue;
    }

    // Data is ready
    return data;
  }

  throw new Error('Timeout waiting for LinkedIn data');
}

export async function fetchLinkedInProfiles(
  linkedinUrls: string[]
): Promise<ProfileData[]> {
  // Trigger async scraping job
  const snapshotId = await triggerLinkedInScrape(linkedinUrls);

  // Poll for results (max 10 minutes)
  const profiles = await pollForSnapshot(snapshotId);

  // Transform to database format
  return profiles.map(profile =>
    transformBrightDataProfile(profile)
  );
}

What Gets Scraped:

Basic info: name, headline, about section
Work experience with company logos and descriptions
Education history
Languages spoken
Connection and follower counts
Profile pictures and banner images
Current company details

Why This Approach?

Async by Design: Trigger job, get snapshot ID, poll for completion
Batch Operations: Scrape multiple profiles in one request
Retry Logic: Handles transient failures gracefully
Timeout Protection: Max 10 minutes prevents infinite loops

3. MCP (Model Context Protocol) Integration

Purpose: Advanced tooling for the research engine

// src/lib/brightdata/client.ts
import { Client } from '@modelcontextprotocol/sdk/client/index.js';
import { StreamableHTTPClientTransport } from '@modelcontextprotocol/sdk/client/streaming.js';

let mcpClient: Client | null = null;

export async function getBrightDataMCPClient(): Promise<Client> {
  // Singleton pattern prevents repeated connections
  if (mcpClient) return mcpClient;

  const apiToken = process.env.BRIGHTDATA_API_TOKEN;
  const transport = new StreamableHTTPClientTransport(
    new URL(`https://mcp.brightdata.com/mcp?api_token=${apiToken}`)
  );

  mcpClient = new Client(
    {
      name: 'peoplehub-client',
      version: '1.0.0',
    },
    {
      capabilities: {},
    }
  );

  await mcpClient.connect(transport);
  return mcpClient;
}

Use Cases:

Web scraping for research reports
Advanced search capabilities
Tool discovery and execution

Multi-Tier Caching Strategy

Caching is critical for cost reduction. PeopleHub uses a two-tier approach:

Tier 1: Redis (Hot Cache)

Purpose: Fast search result caching

// src/lib/redis/search-cache.ts
export async function getCachedSearchResults(
  query: string,
): Promise<CachedSearchResults | null> {
  const key = getCacheKey(CachePrefix.SEARCH_RESULTS, query);
  const cached = await getCache<CachedSearchResults>(key);
  return cached;
}

export async function cacheSearchResults(
  query: string,
  parsedQuery: ParsedSearchQuery,
  results: ProfileSummary[],
): Promise<boolean> {
  const key = getCacheKey(CachePrefix.SEARCH_RESULTS, query);
  const payload: CachedSearchResults = {
    query,
    parsedQuery,
    results,
    count: results.length,
    timestamp: Date.now(),
  };

  return setCache(key, payload, CacheTTL.SEARCH_RESULTS);
}

Benefits:

Sub-millisecond lookups: In-memory data structure
Reduced database load: Offloads hot queries
TTL-based expiration: Configurable freshness window

Tier 2: PostgreSQL (Persistent Cache)

Purpose: Long-term profile storage with intelligent freshness

// src/lib/cache/index.ts
const CACHE_FRESHNESS_DAYS = 180;

export async function getCachedProfile(
  linkedinUrl: string
): Promise<ProfileData | null> {
  // Extract LinkedIn ID to handle regional URL variants
  const linkedinId = extractLinkedInId(linkedinUrl);

  const profile = await prisma.person.findUnique({
    where: { linkedinId },
  });

  if (!profile) return null;

  // Check freshness (< 180 days old)
  const daysSinceUpdate = Math.floor(
    (Date.now() - profile.updatedAt.getTime()) / (1000 * 60 * 60 * 24)
  );

  if (daysSinceUpdate >= CACHE_FRESHNESS_DAYS) {
    return null; // Stale, needs refresh
  }

  return transformToProfileData(profile);
}

export async function saveProfile(
  data: ProfileData
): Promise<ProfileData> {
  await prisma.person.upsert({
    where: { linkedinId: data.linkedinId },
    update: {
      ...data,
      searchCount: { increment: 1 }, // Track popularity
      lastViewed: new Date(),
    },
    create: {
      ...data,
    },
  });

  return data;
}

Batch Optimization:

export async function getCachedProfiles(
  linkedinUrls: string[]
): Promise<Record<string, ProfileData>> {
  const linkedinIds = linkedinUrls
    .map(extractLinkedInId)
    .filter(Boolean);

  // Single SQL query for all profiles
  const profiles = await prisma.person.findMany({
    where: {
      linkedinId: { in: linkedinIds },
    },
  });

  // Filter by freshness
  const result: Record<string, ProfileData> = {};
  const now = Date.now();

  for (const profile of profiles) {
    const daysSinceUpdate = Math.floor(
      (now - profile.updatedAt.getTime()) / (1000 * 60 * 60 * 24)
    );

    if (daysSinceUpdate < CACHE_FRESHNESS_DAYS) {
      result[profile.linkedinId] = transformToProfileData(profile);
    }
  }

  return result;
}

Performance Impact:

First search: ~120 seconds (LinkedIn scraping bottleneck)
Cached search: ~2.5 seconds (database lookup)
Batch lookup: 10-50ms for 100 profiles
Cost reduction: 70-90% (90% cache hit rate)

LangGraph Research Engine

This is PeopleHub's killer feature: automated due diligence reports using LangChain's LangGraph for multi-step agentic workflows.

What is LangGraph?

LangGraph is a framework for building stateful, multi-actor applications with LLMs. It uses a directed graph to define:

Nodes: Individual steps (fetch data, search web, summarize)
Edges: Transitions between steps
State: Shared data across the workflow

Research Workflow Graph

START
  ↓
Initialize Research
  ↓
  ├─→ Fetch LinkedIn Profile ──→ Aggregate Data
  │                                      ↓
  └─→ Execute Google Search          Write Report
         ↓                                ↓
      Scrape URLs (parallel)            END
         ↓
      Summarize Content (parallel)
         ↓
      Aggregate Data

Implementation

State Definition:

// src/lib/research/types.ts
import { Annotation } from '@langchain/langgraph';

export const ResearchStateAnnotation = Annotation.Root({
  personName: Annotation<string>,
  linkedinUrl: Annotation<string>,
  linkedinData: Annotation<ProfileData | undefined>,
  searchQuery: Annotation<string | undefined>,
  searchResults: Annotation<SearchResult[]>,
  scrapedContents: Annotation<ScrapedContent[]>,
  webSummaries: Annotation<WebSummary[]>,
  finalReport: Annotation<string | undefined>,
  status: Annotation<string>,
  errors: Annotation<string[]>,
});

Graph Builder:

// src/lib/research/graph.ts
import { StateGraph, START, END, Send } from '@langchain/langgraph';

export function createResearchGraph() {
  const graph = new StateGraph(ResearchStateAnnotation);

  // Add nodes
  graph.addNode('start', startNode);
  graph.addNode('fetchLinkedIn', fetchLinkedInNode);
  graph.addNode('executeSearch', executeSearchNode);
  graph.addNode('scrapeWebPage', scrapeWebPageNode);
  graph.addNode('summarizeContent', summarizeContentNode);
  graph.addNode('aggregateData', aggregateDataNode);
  graph.addNode('writeReport', writeReportNode);

  // Define edges
  graph
    .addEdge(START, 'start')
    .addEdge('start', 'fetchLinkedIn')
    .addEdge('start', 'executeSearch')
    .addEdge('fetchLinkedIn', 'aggregateData')
    .addConditionalEdges('executeSearch', routeToScraping)
    .addConditionalEdges('scrapeWebPage', routeToSummarization)
    .addEdge('summarizeContent', 'aggregateData')
    .addEdge('aggregateData', 'writeReport')
    .addEdge('writeReport', END);

  return graph.compile({ checkpointer: new MemorySaver() });
}

Parallel Scraping with Send API:

// Route to scraping - creates parallel tasks
export function routeToScraping(
  state: ResearchGraphState
): Send[] {
  if (!state.searchResults?.length) return [];

  // Fan-out: Create one scraping task per URL
  return state.searchResults.map((result) =>
    new Send('scrapeWebPage', {
      ...state,
      url: result.url,
      metadata: {
        source: result.source,
        rank: result.rank,
        title: result.title,
      },
    })
  );
}

// Route to summarization - creates parallel tasks
export function routeToSummarization(
  state: ResearchGraphState
): Send[] {
  if (!state.scrapedContents?.length) return [];

  // Fan-out: Create one summarization task per scraped page
  return state.scrapedContents.map((content) =>
    new Send('summarizeContent', {
      ...state,
      scrapedContent: content,
    })
  );
}

Node Implementations:

// Fetch LinkedIn profile
export const fetchLinkedInNode: ResearchNodeHandler = async (state) => {
  const profile = await fetchLinkedInProfile(state.linkedinUrl);
  return {
    linkedinData: profile,
    status: 'LinkedIn profile fetched',
  };
};

// Execute Google search
export const executeSearchNode: ResearchNodeHandler = async (state) => {
  const results = await searchGoogleForPerson(
    state.personName,
    state.linkedinUrl,
    { maxResults: 15 }
  );

  return {
    searchResults: results,
    status: 'Web search completed',
  };
};

// Summarize scraped content
export const summarizeContentNode: ResearchNodeHandler = async (state) => {
  const { scrapedContent } = state;

  const summary = await summarizeWebContent(
    scrapedContent.url,
    scrapedContent.content,
    state.personName
  );

  return {
    webSummaries: [summary],
    status: `Summarized ${scrapedContent.url}`,
  };
};

// Generate final report
export const writeReportNode: ResearchNodeHandler = async (state) => {
  const bundle = {
    personName: state.personName,
    linkedinUrl: state.linkedinUrl,
    linkedinData: state.linkedinData,
    webSummaries: state.webSummaries,
  };

  const result = await generateResearchReport(bundle);

  return {
    finalReport: result.report,
    status: 'Report ready',
  };
};

Running the Graph:

// src/lib/research/runner.ts
export async function runResearchGraph(
  researchId: string,
  linkedinUrl: string,
  personName: string
): Promise<void> {
  // Update status to 'processing'
  await updateResearchStatus(researchId, 'processing');

  // Create and compile graph
  const compiledGraph = createResearchGraph();

  // Execute with checkpointing
  const result = await compiledGraph.invoke(
    { personName, linkedinUrl },
    { configurable: { thread_id: researchId } }
  );

  // Save report to database
  if (result.finalReport) {
    await saveResearchReport(
      researchId,
      result.finalReport,
      result.webSummaries.map(s => ({ url: s.url, summary: s.summary })),
      { /* metadata */ }
    );
  }
}

Why LangGraph?

Advantages:

Stateful Workflows: Shared state across steps
Parallel Execution: Send API for fan-out/fan-in patterns
Checkpointing: Resume from failures with MemorySaver
Type Safety: TypeScript-first with full type inference
Debuggability: Visual graph representation with Mermaid

Example Research Report:

# Research Report: John Doe

## Professional Background
John Doe is a Senior Software Engineer at Google with 8 years of experience...

## Recent Projects and Achievements
- Led the development of Project X, a distributed system processing 1M+ requests/day
- Published research paper on machine learning optimization at ICML 2024
- Speaker at Google I/O 2024 on Kubernetes best practices

## Technical Expertise
Primary skills: Python, Go, Kubernetes, TensorFlow, distributed systems
Notable contributions to open-source projects...

## Industry Reputation
Recognized as a thought leader in cloud-native architecture...

## Sources
1. [Google Engineering Blog - Project X Launch](https://example.com)
2. [ICML 2024 Paper: ML Optimization Techniques](https://example.com)
3. [Google I/O 2024 Talk](https://example.com)

Database Design with Prisma

Schema Overview

// prisma/schema.prisma
datasource db {
  provider = "postgresql"
  url      = env("DATABASE_URL")
}

model Person {
  id                String   @id @default(cuid())
  linkedinUrl       String   @unique
  linkedinId        String   @unique
  linkedinNumId     String?

  // Basic Info
  firstName         String
  lastName          String
  fullName          String   @db.Text
  headline          String?  @db.Text
  about             String?  @db.Text

  // Location
  location          String?
  city              String?
  countryCode       String?

  // Profile Media
  profilePicUrl     String?
  bannerImage       String?
  defaultAvatar     Boolean  @default(false)

  // Current Company
  currentCompany    String?
  currentCompanyId  String?

  // Rich Data (JSON)
  experience        Json?
  education         Json?
  languages         Json?

  // Social Stats
  connections       Int?
  followers         Int?

  // Metadata
  searchCount       Int      @default(0)
  lastViewed        DateTime @default(now())
  createdAt         DateTime @default(now())
  updatedAt         DateTime @updatedAt

  // Relations
  researches        Research[]

  @@index([fullName])
  @@index([firstName, lastName])
  @@index([lastViewed])
  @@index([linkedinId])
  @@index([currentCompany])
  @@index([location])
  @@index([updatedAt])
  @@map("people")
}

model Search {
  id          String   @id @default(cuid())
  query       String   @db.Text
  results     Json     // Array of person IDs
  resultCount Int
  createdAt   DateTime @default(now())

  @@index([query])
  @@map("searches")
}

model Research {
  id              String   @id @default(cuid())
  personId        String?
  person          Person?  @relation(fields: [personId], references: [id])
  linkedinUrl     String
  personName      String
  report          String   @db.Text
  sources         Json     // Array of { url, summary }
  metadata        Json?    // Graph execution metadata
  status          String   // 'pending' | 'processing' | 'completed' | 'failed'
  errorMessage    String?  @db.Text
  createdAt       DateTime @default(now())
  updatedAt       DateTime @updatedAt

  @@index([personId])
  @@index([linkedinUrl])
  @@index([status])
  @@index([createdAt])
  @@map("researches")
}

Key Design Decisions

1. JSON Fields for Flexibility

// Experience is stored as JSON for flexibility
experience: [
  {
    title: "Senior Software Engineer",
    company: "Google",
    companyId: "1441",
    companyLogo: "https://...",
    location: "Mountain View, CA",
    startDate: "2020-01-01",
    endDate: null,
    description: "Led development of...",
    isCurrent: true
  },
  // ... more experiences
]

Why JSON?

Flexible schema (experiences vary widely)
No need for separate tables and joins
Easier to store Bright Data's nested structures
PostgreSQL JSON operators for querying

2. Strategic Indexes

-- Full-text search on names
@@index([fullName])
@@index([firstName, lastName])

-- Filter by recency
@@index([lastViewed])
@@index([updatedAt])

-- Search by location/company
@@index([location])
@@index([currentCompany])

-- Unique constraints prevent duplicates
linkedinId String @unique
linkedinUrl String @unique

3. Metadata Tracking

searchCount: Int @default(0)  // Popularity metric
lastViewed: DateTime          // Cache invalidation
updatedAt: DateTime           // Freshness check

Benefits:

Identify popular profiles (high searchCount)
Prioritize cache refreshes for frequently viewed profiles
Track data staleness for intelligent revalidation

Performance Optimizations

1. Batch Database Queries

Problem: N+1 query problem when checking cache for multiple URLs

Solution: Single query for all profiles

// Instead of N queries
for (const url of urls) {
  await getCachedProfile(url); // ❌ N queries
}

// Do 1 query
const cached = await getCachedProfiles(urls); // ✅ 1 query

Impact: 100 profiles: ~5 seconds → ~50ms

2. Parallel API Calls

Problem: Sequential Bright Data calls block each other

Solution: Promise.all for independent operations

// Sequential: ~6 seconds total
const parsedQuery = await parseSearchQuery(query);
const summaries = await searchLinkedInProfiles(parsedQuery);
await cacheSearchResults(query, summaries);

// Parallel: ~3 seconds total
const [parsedQuery, summaries] = await Promise.all([
  parseSearchQuery(query),
  searchLinkedInProfiles(parsedQuery), // ❌ Depends on parsedQuery
]);

Caveat: Only parallelize truly independent operations!

3. Connection Pooling

Problem: Serverless functions create new DB connections per request

Solution: Singleton Prisma client

// src/lib/prisma.ts
import { PrismaClient } from '@prisma/client';

const globalForPrisma = global as unknown as { prisma: PrismaClient };

export const prisma = globalForPrisma.prisma ||
  new PrismaClient({
    log: process.env.NODE_ENV === 'development'
      ? ['query', 'error', 'warn']
      : ['error'],
  });

if (process.env.NODE_ENV !== 'production') {
  globalForPrisma.prisma = prisma;
}

4. Image Proxy for CORS

Problem: Direct LinkedIn image URLs blocked by CORS and ad blockers

Solution: Proxy images through our API

// src/app/api/proxy-image/route.ts
export async function GET(request: NextRequest) {
  const imageUrl = request.nextUrl.searchParams.get('url');

  const response = await fetch(imageUrl, {
    headers: {
      'User-Agent': 'Mozilla/5.0...',
    },
  });

  const imageBuffer = await response.arrayBuffer();

  return new Response(imageBuffer, {
    headers: {
      'Content-Type': response.headers.get('Content-Type'),
      'Cache-Control': 'public, max-age=86400', // 24 hours
    },
  });
}

5. Redis for Hot Cache

Problem: PostgreSQL queries still take 10-50ms for popular searches

Solution: Redis in-memory cache for search results

// Cache TTL: 30 minutes for hot searches
export const CacheTTL = {
  SEARCH_RESULTS: 30 * 60, // 30 minutes
  PROFILE_SUMMARY: 60 * 60, // 1 hour
};

// Check Redis first, fallback to PostgreSQL
const cached = await getCachedSearchResults(query);
if (cached) return cached; // ~2ms

// Cache miss, fetch from DB/API
const results = await searchLinkedInProfiles(query);
await cacheSearchResults(query, results);

Lessons Learned

1. Structured Output > Regex Parsing

Before: Manual regex parsing of queries

const match = query.match(/(\d+)\s+(.+)\s+in\s+(.+)/);
if (!match) throw new Error('Invalid query');
const [, count, role, location] = match;

After: Gemini 2.0 Flash with Zod schemas

const { object } = await generateObject({
  model: google('gemini-2.0-flash-exp'),
  schema: SearchQuerySchema,
  prompt: `Parse: "${query}"`,
});

Why Better:

Handles ambiguous queries ("software engineers at Google" - is Google a location or company?)
Extracts implicit information (maps "Israel" → "IL" country code)
No brittle regex maintenance

2. Async by Default for External APIs

Bright Data's LinkedIn scraper takes 60-120 seconds. Never block the user:

// ❌ Bad: User waits 2 minutes
const profiles = await fetchLinkedInProfiles(urls);
return profiles;

// ✅ Good: Return immediately, process in background
const researchId = await createResearch(linkedinUrl, personName);
runResearchGraph(researchId, linkedinUrl, personName); // Fire and forget
return { researchId, status: 'pending' };

Users can check status with polling or webhooks.

3. LangGraph for Complex Workflows

Before: Imperative spaghetti code

async function research(person) {
  const linkedin = await fetchLinkedIn(person);
  const query = await generateQuery(linkedin);
  const results = await search(query);
  const scraped = [];
  for (const url of results) {
    scraped.push(await scrape(url));
  }
  const summaries = [];
  for (const content of scraped) {
    summaries.push(await summarize(content));
  }
  return await writeReport(linkedin, summaries);
}

After: Declarative graph with parallel execution

const graph = new StateGraph(ResearchStateAnnotation)
  .addNode('fetchLinkedIn', fetchLinkedInNode)
  .addNode('search', searchNode)
  .addNode('scrape', scrapeNode) // Parallel
  .addNode('summarize', summarizeNode) // Parallel
  .addNode('writeReport', writeReportNode)
  .addConditionalEdges('search', routeToScraping)
  .compile();

const result = await graph.invoke({ person });

Benefits:

Parallel scraping (15 URLs in 10 seconds, not 150 seconds)
Checkpointing (resume from failures)
Easier to test individual nodes

4. Cache Invalidation is Hard

Challenge: When to refresh cached profiles?

Solution: Multi-factor scoring

function shouldRefresh(profile: Person): boolean {
  const daysSinceUpdate = getDaysSince(profile.updatedAt);
  const isPopular = profile.searchCount > 10;
  const isRecent = getDaysSince(profile.lastViewed) < 7;

  // Refresh if:
  // - Stale (> 180 days)
  // - Popular AND moderately old (> 30 days)
  // - Recently viewed AND old (> 60 days)
  return (
    daysSinceUpdate > 180 ||
    (isPopular && daysSinceUpdate > 30) ||
    (isRecent && daysSinceUpdate > 60)
  );
}

5. Type Safety Everywhere

TypeScript + Zod + Prisma = Compile-time safety across the stack

// 1. Zod validates API input
const SearchQuerySchema = z.object({
  count: z.number().min(1).max(50),
  role: z.string().nullable(),
});

// 2. Prisma generates types from schema
const profile: Person = await prisma.person.findUnique(...);

// 3. TypeScript enforces contracts
function transformProfile(data: ProfileData): Person {
  // Compiler catches type mismatches
}

Result: Caught 50+ bugs at compile-time instead of runtime.

Conclusion

Building PeopleHub taught me:

LLMs for structured extraction - Gemini 2.0 Flash + Zod is perfect for parsing
Bright Data scales - Their APIs handle thousands of LinkedIn profiles without breaking
LangGraph for workflows - State machines beat imperative code for complex flows
Cache aggressively - 90% cache hit rate = 10x cost reduction
Async everything - Never block users on slow external APIs

The entire codebase is open-source on GitHub. Check it out, try it yourself, and let me know what you think!

Tech Stack Summary:

Next.js 15.5.4 + React 19.1.0
TypeScript 5 + Zod 3.25.76
Prisma 6.5.0 + PostgreSQL + Redis
Google Gemini 2.0 Flash + Vercel AI SDK 5.0.60
LangChain + LangGraph 1.0.1
Bright Data APIs (Google Search, LinkedIn Scraper, MCP)

Questions?

Drop a comment below or reach out! Would love to hear your thoughts on:

Alternative approaches to query parsing
Other use cases for LangGraph
Optimization ideas for the research engine
Your experience with Bright Data APIs

Happy coding! 🚀

Building a Multi-Agent Competitive Intelligence Platform with Bright Data and Strands

Meir — Wed, 17 Sep 2025 19:09:59 +0000

How we built an AI-powered competitive intelligence system that transforms weeks of manual research into minutes of automated analysis using Strands agents and Bright Data's enterprise web scraping capabilities.

Introduction

In today's fast-paced business environment, competitive intelligence can make or break strategic decisions. Traditional approaches involve manual research, scattered data sources, and weeks of analysis. What if we could automate this entire process using AI agents that work together seamlessly?

This article walks through building a Multi-Agent Competitive Intelligence Platform that combines:

🤖 Strands Agents for autonomous AI workflows
🌐 Bright Data for enterprise-grade web scraping
🧠 Google Gemini 2.0 for advanced analysis
⚡ FastAPI with real-time streaming
🎨 React + TypeScript frontend with live updates

The Challenge

Traditional competitive intelligence faces several challenges:

Manual Data Collection: Hours spent browsing websites, collecting pricing info, and researching leadership teams
Fragmented Analysis: Data scattered across multiple sources without unified insights
Outdated Reports: By the time analysis is complete, information may already be stale
Limited Scalability: Human analysts can only research a few competitors at a time

Our Solution: Multi-Agent Architecture

We designed a three-agent workflow that mirrors how human analysts work:

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   📊 Researcher │───▶│   🔍 Analyst    │───▶│   📝 Writer     │
│     Agent       │    │     Agent       │    │     Agent       │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │                       │
         ▼                       ▼                       ▼
   Data Collection      Strategic Analysis       Report Generation
   - Web scraping       - SWOT analysis         - Executive summary
   - Market research     - Threat assessment     - Recommendations
   - Company intel       - Competitive position  - Action items

Technical Implementation

1. Setting Up the Environment

First, let's set up our dependencies:

# requirements.txt
fastapi>=0.104.0
uvicorn>=0.24.0
strands>=0.1.0
strands-tools>=0.1.0
litellm>=1.0.0
pydantic>=2.0.0

# Environment variables
export GEMINI_API_KEY="your_gemini_api_key"
export BRIGHTDATA_API_KEY="your_brightdata_api_key"

2. Creating Specialized Agents

Each agent has a specific role and prompt optimized for its task:

from strands import Agent
from strands.models.litellm import LiteLLMModel
from strands_tools import bright_data

def get_gemini_model():
    """Configure Gemini model using LiteLLM"""
    gemini_api_key = os.getenv("GEMINI_API_KEY")

    return LiteLLMModel(
        client_args={"api_key": gemini_api_key},
        model_id="gemini/gemini-2.0-flash",
        params={"max_tokens": 4000, "temperature": 0.3}
    )

# Researcher Agent - Data Collection
RESEARCHER_PROMPT = """You are a Researcher Agent specialized in competitive intelligence data gathering.

Your role:
1. **Data Collection**: Use bright_data tool to gather comprehensive competitor information
2. **Source Discovery**: Find recent news, funding announcements, product launches
3. **Website Analysis**: Scrape pricing pages, product descriptions, company information
4. **LinkedIn Intelligence**: Get structured data about leadership and team size

Focus on recent developments, pricing strategy, leadership team, and market position.
Keep findings under 800 words and include source URLs."""

# Analyst Agent - Strategic Analysis  
ANALYST_PROMPT = """You are an Analyst Agent specialized in competitive intelligence analysis.

Your role:
1. **Strategic Analysis**: Analyze competitive positioning and market strategy
2. **SWOT Assessment**: Identify strengths, weaknesses, opportunities, threats
3. **Business Model Analysis**: Understand revenue streams and value propositions
4. **Competitive Threats**: Assess level of competitive threat and market overlap

Rate competitive threat level from 1-5 and provide actionable insights.
Keep analysis under 600 words."""

# Writer Agent - Report Generation
WRITER_PROMPT = """You are a Writer Agent specialized in competitive intelligence reporting.

Structure reports with:
- Executive Summary (key findings and threat level)
- Business Model Analysis
- Competitive Positioning  
- Strategic Recommendations
- Action Items

Keep reports under 700 words, professional tone, actionable for decision-makers."""

3. Multi-Agent Workflow Implementation

class MultiAgentCompetitiveIntelligence:
    def __init__(self, stream_callback=None):
        gemini_model = get_gemini_model()

        # Create specialized agents
        self.researcher_agent = Agent(
            model=gemini_model,
            system_prompt=RESEARCHER_PROMPT,
            tools=[bright_data],
            callback_handler=StreamingCallbackHandler(stream_callback)
        )

        self.analyst_agent = Agent(
            model=gemini_model,
            system_prompt=ANALYST_PROMPT,
            callback_handler=StreamingCallbackHandler(stream_callback)
        )

        self.writer_agent = Agent(
            model=gemini_model,
            system_prompt=WRITER_PROMPT,
            callback_handler=StreamingCallbackHandler(stream_callback)
        )

    def run_competitive_intelligence_workflow(self, competitor_name, competitor_website=None):
        """Execute the three-stage workflow"""

        # Stage 1: Research
        research_query = f"""Research competitive intelligence for "{competitor_name}".

        Gather comprehensive information about:
        1. Recent company news, funding, and market developments
        2. Pricing strategy and product positioning
        3. Company leadership and team structure
        4. Market position and customer feedback
        5. Financial information and growth metrics
        """

        research_findings = self.researcher_agent(research_query)

        # Stage 2: Analysis
        analysis_query = f"""Analyze these findings for "{competitor_name}":
        {research_findings}

        Provide strategic analysis including:
        1. SWOT assessment
        2. Business model analysis
        3. Competitive positioning
        4. Threat level assessment (1-5)
        5. Strategic insights for competitive response
        """

        strategic_analysis = self.analyst_agent(analysis_query)

        # Stage 3: Report Writing
        report_query = f"""Create comprehensive competitive intelligence report:

        RESEARCH FINDINGS: {research_findings}
        STRATEGIC ANALYSIS: {strategic_analysis}

        Generate professional report with executive summary, 
        business model analysis, and strategic recommendations.
        """

        final_report = self.writer_agent(report_query)

        return {
            "competitor": competitor_name,
            "research_findings": str(research_findings),
            "strategic_analysis": str(strategic_analysis), 
            "final_report": str(final_report),
            "status": "success"
        }

4. Real-Time Streaming with FastAPI

The platform provides real-time updates during analysis:

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import asyncio
import json

@app.post("/analyze/stream")
async def analyze_competitor_stream(request: AnalysisRequest):
    """Streaming endpoint with real-time updates"""

    async def generate_stream():
        events_queue = asyncio.Queue()

        def stream_callback(event):
            """Capture tool calls and status updates"""
            asyncio.create_task(events_queue.put(event))

        # Run analysis with streaming
        async def run_analysis():
            intelligence_system = MultiAgentCompetitiveIntelligence(stream_callback)
            result = intelligence_system.run_competitive_intelligence_workflow(
                competitor_name=request.competitor_name,
                competitor_website=request.competitor_website
            )

            # Send final result
            final_event = {
                "timestamp": datetime.now().isoformat(),
                "type": "complete", 
                "data": result
            }
            await events_queue.put(final_event)
            await events_queue.put(None)  # End signal

        # Start analysis
        analysis_task = asyncio.create_task(run_analysis())

        # Stream events
        while True:
            event = await events_queue.get()
            if event is None:
                break

            yield f"data: {json.dumps(event)}\n\n"

        await analysis_task

    return StreamingResponse(
        generate_stream(),
        media_type="text/event-stream",
        headers={"Cache-Control": "no-cache"}
    )

5. Frontend Integration with React

The React frontend provides live visualization of the analysis process:

// CompetitiveIntelligenceForm.tsx
const [analysisState, setAnalysisState] = useState<'idle' | 'running' | 'complete'>('idle');
const [streamingEvents, setStreamingEvents] = useState<StreamEvent[]>([]);

const startStreamingAnalysis = async (data: FormData) => {
  setAnalysisState('running');

  const response = await fetch(`${API_BASE_URL}/analyze/stream`, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ ...data, stream: true })
  });

  const reader = response.body?.getReader();
  const decoder = new TextDecoder();

  while (true) {
    const { value, done } = await reader!.read();
    if (done) break;

    const chunk = decoder.decode(value);
    const lines = chunk.split('\n');

    for (const line of lines) {
      if (line.startsWith('data: ')) {
        const event = JSON.parse(line.slice(6));
        setStreamingEvents(prev => [...prev, event]);

        if (event.type === 'complete') {
          setAnalysisState('complete');
          setAnalysisResult(event.data);
        }
      }
    }
  }
};

Key Features and Benefits

1. Bright Data Integration

Enterprise-grade web scraping with automatic proxy rotation
CAPTCHA solving and anti-bot detection bypass
Structured data extraction from complex websites
Real-time market intelligence gathering

2. Strands Agents Framework

Autonomous agent workflows with tool integration
Built-in error handling and retry mechanisms
Flexible agent communication and data passing
Easy integration with various AI models

3. Real-Time Streaming

Server-Sent Events for live progress updates
Tool call monitoring and visualization
Interactive progress tracking in the frontend
Session management for long-running analyses

4. Scalable Architecture

Microservices design with FastAPI
Async/await for concurrent processing
Docker containerization ready
Production-ready error handling

Performance and Results

Our platform delivers impressive results:

Speed: Analysis that took weeks now completes in 5-10 minutes
Accuracy: 95%+ data accuracy with Bright Data's enterprise infrastructure
Coverage: Comprehensive analysis across pricing, leadership, market position, and strategy
Scalability: Can analyze multiple competitors simultaneously

Example Output

Here's what the platform generates for a competitor analysis:

{
  "competitor": "Slack",
  "research_findings": "Slack Technologies, valued at $27.7B, serves 18M+ daily active users across 750K+ organizations. Recent developments include AI-powered features, enterprise security enhancements, and strategic partnerships...",
  "strategic_analysis": "SWOT Analysis - Strengths: Market leadership in team communication, strong enterprise adoption. Weaknesses: High pricing for SMBs, feature complexity. Threat Level: 4/5 - Direct competitor with overlapping market...",
  "final_report": "Executive Summary: Slack represents a significant competitive threat with strong market position and continuous innovation. Recommended response: Focus on pricing differentiation and SMB market penetration..."
}

Challenges and Solutions

1. Data Quality and Reliability

Challenge: Ensuring consistent, accurate data from web scraping
Solution: Bright Data's enterprise infrastructure with built-in validation and retry mechanisms

2. Agent Coordination

Challenge: Managing data flow between specialized agents
Solution: Strands framework's built-in agent communication with structured data passing

3. Real-Time Streaming

Challenge: Providing live updates without overwhelming the frontend
Solution: Efficient Server-Sent Events with event queuing and heartbeat mechanisms

4. Scalability

Challenge: Handling multiple concurrent analyses
Solution: Async FastAPI with proper session management and resource cleanup

Future Enhancements

We're planning several exciting improvements:

Multi-Language Support: Expand analysis to global markets
Custom Agent Types: Allow users to define specialized analysis agents
Historical Tracking: Monitor competitor changes over time
Integration APIs: Connect with CRM and business intelligence tools
Advanced Visualizations: Interactive charts and competitive landscape maps

Conclusion

Building a multi-agent competitive intelligence platform with Bright Data and Strands demonstrates the power of combining enterprise-grade web scraping with autonomous AI workflows. The result is a system that:

Automates weeks of manual research into minutes of AI analysis
Scales to analyze multiple competitors simultaneously
Provides real-time insights with live streaming updates
Delivers executive-ready reports with actionable recommendations

The integration between Bright Data's reliable data collection and Strands' flexible agent framework creates a robust foundation for intelligent competitive analysis.

Getting Started

Want to build your own competitive intelligence platform? Here's how to get started:

Clone the repository: git clone https://github.com/brightdata/competitive-intelligence
Set up environment: Configure your Gemini and Bright Data API keys
Install dependencies: pip install -r requirements.txt
Run the demo: python ci_agent.py for CLI or python app.py for API
Explore the frontend: cd ci-agent-ui && npm run dev

The future of competitive intelligence is here – and it's powered by AI agents working together to deliver insights faster than ever before.

Tags: #AI #WebScraping #CompetitiveIntelligence #BrightData #StrandsAgents #Python #React #FastAPI #Automation

GitHub: https://github.com/brightdata/competitive-intelligence
Demo: https://youtu.be/t0kZibf0hLE

Building Event Hunter AI: A Multi-Agent System for Intelligent Event Discovery

Meir — Mon, 25 Aug 2025 13:42:03 +0000

Ever spent hours searching for the perfect tech conference or hackathon? What if an AI agent could do that work for you, delivering personalized event recommendations with detailed analysis in minutes? Here's how I built Event Hunter AI - a sophisticated multi-agent system that transforms how developers find industry events.

The Problem: Event Discovery is Broken 🤯

As developers, we've all been there. You want to attend a conference, find a hackathon, or discover networking events in your field, but:

Google searches return outdated results from 2019
Event listing sites are cluttered and hard to filter
CFP deadlines are buried in dense text
Sponsorship opportunities are unclear or missing
You waste hours manually checking dozens of websites

What if we could solve this with AI? Not just any AI - but a multi-agent system that mimics how a human research assistant would tackle this task.

The Solution: Event Hunter AI ⚡

Event Hunter AI is a sophisticated system that combines:

🔍 Multi-agent architecture with specialized search and analysis agents
🌐 Web scraping capabilities via BrightData's Model Context Protocol (MCP)
🧠 GPT-4.1-mini for intelligent processing and decision-making
⚡ Real-time streaming via WebSocket for live updates
🎨 Modern React frontend with form-based query building

Watch the Demo

Why Multi-Agent Architecture? 🤖

Instead of building a monolithic AI system, I chose a multi-agent approach inspired by how human teams work:

1. Event Search Agent 🕵️‍♂️

Role: Find events across the web
Tools: Search engine access via MCP
Output: List of potential events with URLs

2. Event Details Agent 📋

Role: Extract comprehensive information from event pages
Tools: Web scraping via MCP
Output: Structured event data (dates, CFP status, sponsorship)

3. Main Orchestrator 🎯

Role: Coordinate the other agents and compile final results
Intelligence: Understands user intent and manages workflow

This separation of concerns makes the system more reliable, maintainable, and allows each agent to excel at its specific task.

Tech Stack Deep Dive 🛠️

Backend: Python + FastAPI

The heart of the system uses cutting-edge technologies:

# Multi-agent setup with DeepAgents framework
from deepagents import create_deep_agent
from langchain.chat_models import init_chat_model
from langchain_mcp_adapters.client import MultiServerMCPClient

# Initialize MCP tools from BrightData
mcp_client = MultiServerMCPClient({
    "brightdata": {
        "url": f"https://mcp.brightdata.com/sse?token={brightdata_token}",
        "transport": "sse",
    }
})

# Create specialized agents
event_search_agent = {
    "name": "event-search-agent",
    "description": "Finds industry events using search engines",
    "prompt": event_search_prompt,
}

agent = create_deep_agent(
    tools=mcp_tools,
    model=model,
    instructions=event_hunter_instructions,
    subagents=[event_search_agent, event_details_agent],
)

Key Backend Features:

FastAPI for modern Python web framework
WebSocket streaming for real-time updates
DeepAgents for multi-agent orchestration
BrightData MCP for web scraping at scale
Azure OpenAI integration

Frontend: React + TypeScript

A polished user experience with modern web technologies:

// Real-time WebSocket connection
const connectWebSocket = () => {
  wsRef.current = new WebSocket('ws://localhost:8000/ws/query')

  wsRef.current.onmessage = (event) => {
    const data = JSON.parse(event.data)

    if (data.type === 'stream') {
      setResult(data.content) // Live updates!
    }
  }
}

// Form-based query building
const query = `Find ${formData.vertical} events in ${formData.location} 
${dateRangeText}.${formData.companies ? ` Focus on events where these 
companies might be involved: ${formData.companies}.` : ''}`

Key Frontend Features:

React 19 with TypeScript for type safety
Tailwind CSS for modern styling
Framer Motion for smooth animations
WebSocket integration for live streaming
Responsive design that works on all devices

The Magic: How It Actually Works 🎪

Here's the step-by-step process when you submit a query:

Step 1: Query Processing 📝

User Input: "Find AI hackathons in San Francisco for Q1 2025"
↓
System parses: Industry=AI, Location=San Francisco, Time=Q1 2025

Step 2: Search Agent Activation 🔍

# Search agent gets to work
search_queries = [
    "AI hackathon 2025 San Francisco",
    "artificial intelligence hackathon Bay Area",
    "machine learning competition San Francisco 2025"
]
# Uses BrightData MCP to search Google, Bing, Yandex

Step 3: Details Agent Analysis 📊

# For each event URL found:
event_data = {
    "name": "AI DevFest 2025",
    "date": "March 15-17, 2025", 
    "location": "San Francisco, CA",
    "cfp_status": "Open until Feb 1st",
    "sponsorship": "Available - contact organizers"
}

Step 4: Real-time Streaming ⚡

The frontend receives live updates as each agent completes its work:

"🔍 Searching for AI hackathons..."
"📋 Analyzing event details..."
"✅ Found 12 relevant events!"

Key Features That Make It Special ✨

1. Intelligent Date Awareness 📅

The system knows today's date and filters out past events automatically:

current_date = datetime.now().strftime("%B %d, %Y")
event_search_prompt = f"""
IMPORTANT: Today's date is {current_date}. 
When searching for events, always consider this current date 
and look for upcoming events, not past ones.
"""

2. Comprehensive Event Analysis 🔬

For each event discovered, you get:

Event Name and official website
Exact dates and times
Location details (city, venue, online/hybrid)
CFP status - are submissions still open?
Sponsorship opportunities - can your company get involved?

3. Form-Based Query Building 📋

Instead of trying to craft the perfect prompt, users fill out a simple form:

Location: San Francisco, Europe, Online
Date Range: Calendar picker for precise timeframes
Industry: Dropdown with 12+ verticals
Companies: Optional company focus
Additional Requirements: Free-text for special needs

4. Real-Time Processing Feedback 📡

Users see exactly what's happening:

🔍 Searching for blockchain conferences in Europe...
📋 Found 15 potential events, analyzing details...
✅ Extracted information for 12 high-quality events!

Technical Challenges & Solutions 🧩

Challenge 1: Rate Limiting & Reliability

Problem: Web scraping can be unreliable and rate-limited.
Solution: BrightData MCP provides enterprise-grade scraping with automatic retry logic and proxy rotation.

Challenge 2: Information Extraction Accuracy

Problem: Event websites have inconsistent layouts and data formats.
Solution: Specialized prompt engineering for the Details Agent with clear extraction templates:

event_details_prompt = """
Extract the following information for each event:
- Event Name
- Date(s) 
- Location (city, venue if available)
- Main statement/description
- Whether CFP (Call for Papers) is open
- Whether sponsorship opportunities are available

Format as structured markdown for consistency.
"""

Challenge 3: Real-Time User Experience

Problem: AI processing takes time, users need feedback.
Solution: WebSocket streaming with filtered content to show only meaningful updates.

Challenge 4: Scalability

Problem: Each query requires multiple web requests and AI processing.
Solution: Async Python with FastAPI handles multiple concurrent requests efficiently.

Results & Performance 📈

After building and testing Event Hunter AI:

Speed: Average query completion in 60-90 seconds
Accuracy: 85%+ relevance for discovered events

Coverage: Finds 10-15 high-quality events per query
User Experience: Real-time updates keep users engaged

Sample Output Quality

For a query like "Find fintech conferences in London for 2025", the system returns:

**Event Name**: FinTech World 2025
**Link**: https://fintechworld.com
**Date**: April 22-24, 2025
**Location**: ExCeL London, UK
**Main Statement**: Europe's largest fintech conference bringing together 5,000+ professionals
**Open CFP**: Yes - submissions close March 1st, 2025
**Open Sponsorship**: Yes - platinum packages available

Lessons Learned 🎓

1. Multi-Agent Architecture is Powerful

Breaking down the problem into specialized agents made the system more reliable and easier to debug. Each agent has a clear responsibility and can be optimized independently.

2. Real-Time Feedback is Crucial

Users will wait 90 seconds for good results, but only if they see progress. The streaming WebSocket updates transformed the user experience.

3. Form-Based Input > Prompt Engineering

Instead of expecting users to craft perfect prompts, a structured form with dropdowns and date pickers made the system accessible to everyone.

4. Modern Tooling Matters

BrightData MCP: Made web scraping reliable and scalable
DeepAgents: Simplified multi-agent orchestration
FastAPI: Handled WebSocket streaming beautifully
React + TypeScript: Enabled rapid frontend development

Future Enhancements 🚀

The current system is just the beginning. Here's what's on the roadmap:

1. Personalization Engine 🎯

User profiles with preferred industries and locations
ML-powered recommendations based on past searches
Calendar integration to avoid scheduling conflicts

2. Advanced Filtering 💎

Price range filtering (free vs. paid events)
Event size preferences (intimate vs. large conferences)
Format preferences (online, hybrid, in-person)

3. Notification System 📢

Email alerts for new events matching user criteria
CFP deadline reminders
Early bird pricing notifications

4. Community Features 👥

Event reviews from past attendees
Networking opportunities - see who else is going
Group discounts for multiple registrations

Getting Started 🏃‍♂️

Want to try Event Hunter AI or contribute? Here's how:

Prerequisites

BrightData account for MCP web scraping
Azure OpenAI access for GPT-4.1-mini
Python 3.8+ and Node.js 18+

Quick Setup

# Clone and setup backend
git clone https://github.com/brightdata/event-hunter
cd event-hunter
pip install -r requirements.txt
cp .env.example .env  # Add your API keys

# Setup frontend  
cd event-hunter
npm install
npm run dev

# Start the system
python main.py  # Backend on :8000
# Frontend auto-opens on :5173

Environment Configuration

# .env file
BRIGHTDATA_API_TOKEN=your_token_here
OPENAI_API_KEY=your_azure_openai_key  
OPENAI_BASE_URL=https://your-resource.openai.azure.com/openai/v1/

Impact & Use Cases 💡

Event Hunter AI isn't just a cool tech demo - it solves real problems:

For Individual Developers 👨‍💻

Save 5+ hours per event search
Discover niche events you'd never find manually
Track CFP deadlines automatically
Find sponsorship opportunities for side projects

For Companies 🏢

Identify conference speaking opportunities
Find sponsorship targets aligned with their audience
Track competitor presence at industry events
Plan marketing calendars around key events

For Event Organizers 📅

Competitive analysis of similar events
Market research for new event concepts
Partnership identification with complementary events

The Bigger Picture 🌍

Event Hunter AI represents a new class of AI applications:

1. Multi-Modal Intelligence 🧠

Combining web search, content analysis, and structured data extraction into a cohesive workflow that mimics human research patterns.

2. Real-Time AI Experiences ⚡

Moving beyond static chatbots to dynamic, streaming AI that provides progress updates and builds trust through transparency.

3. Domain-Specific AI Agents 🎯

Rather than general-purpose AI, specialized agents that deeply understand specific problem domains (in this case, event discovery).

4. Human-AI Collaboration 🤝

The system augments human capabilities rather than replacing them - users still make the final decisions about which events to attend.

Conclusion: The Future of AI-Powered Discovery 🔮

Building Event Hunter AI taught me that the future of AI isn't about replacing human judgment - it's about augmenting human capabilities with intelligent automation.

The multi-agent architecture proved that complex problems require specialized solutions. By breaking down event discovery into search and analysis phases, each agent could excel at its specific task while working together toward a common goal.

Most importantly, this project shows how modern AI tooling - from BrightData's MCP to DeepAgents to real-time streaming - makes it possible for individual developers to build sophisticated systems that were previously only feasible for large teams.

What's Next?

The techniques used in Event Hunter AI can be applied to many other discovery problems:

Job search with salary analysis and company culture insights
Product research with competitive analysis and pricing trends
Academic paper discovery with relevance scoring and citation analysis
Travel planning with weather, events, and local recommendations

Try It Yourself! 🎮

Event Hunter AI is open source and ready to use. Whether you're looking for your next conference or want to learn about multi-agent AI systems, give it a try:

🔗 GitHub Repository: https://github.com/brightdata/event-hunter
🌟 Live Demo: Coming soon!
📖 Documentation: Detailed setup guide in the README

Have questions about the implementation or want to contribute? Drop a comment below or open an issue on GitHub!

What kind of AI-powered discovery tools would you build? Share your ideas in the comments! 👇

Building a Free AI-Powered Data Enrichment Tool with Bright Data MCP, Gemini, and Kimi K2

Meir — Thu, 14 Aug 2025 14:56:52 +0000

How I built a complete spreadsheet enrichment solution using free tiers of cutting-edge APIs

TL;DR

I built a full-stack data enrichment tool that transforms basic spreadsheet data into rich, contextual information using Bright Data's MCP (Model Context Protocol) server, Google Gemini, and Kimi K2 through OpenRouter. The best part? It's completely free to use within generous limits:

🌐 Bright Data MCP: 5,000 requests/month (renewed monthly)
🤖 Google Gemini: Generous free tier for AI processing
🧠 Kimi K2 via OpenRouter: You can use their free version
🚀 Result: Professional-grade data enrichment without cost barriers

Demo Video

The Problem

Data enrichment is expensive. Whether you're a startup researching leads, a researcher gathering information, or a developer building demos, traditional data enrichment services can quickly become cost-prohibitive. I wanted to build something that democratizes access to real-time web intelligence.

Architecture Overview

The solution consists of three main components:

┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   React Frontend │    │  FastAPI Backend │    │  AI + Web APIs  │
│                 │    │                  │    │                 │
│ • Spreadsheet   │◄──►│ • LangGraph      │◄──►│ • Bright Data   │
│ • CSV Upload    │    │ • Enrichment     │    │ • Gemini AI     │
│ • Real-time UI  │    │ • Batch Processing│    │ • Kimi K2       │
└─────────────────┘    └──────────────────┘    └─────────────────┘

Key Technologies

Bright Data MCP Server - The Game Changer

The Bright Data MCP server is what makes this project special. Unlike traditional web scraping or API approaches, MCP provides a standardized way for AI models to interact with external data sources.

Why MCP is revolutionary:

🔌 Plug-and-play integration with AI agents
🌍 Real-time web access without rate limiting headaches
🛡️ Built-in compliance and ethical data collection
📊 Structured data extraction ready for AI consumption

LangGraph for Agent Orchestration

I used LangGraph to create a sophisticated enrichment pipeline that can handle both standard and deep research modes:

class EnrichmentPipeline:
    def __init__(self, llm_provider: LLMProvider, deep_research: bool = False):
        self.llm = llm_provider
        self.bright_data_client = None
        self.tools = None
        self.deep_research = deep_research

    def build_graph(self):
        """Build and compile the enrichment graph"""
        graph = StateGraph(EnrichmentContext)

        if self.deep_research:
            graph.add_node("search", self.deep_search_bright_data)
        else:
            graph.add_node("search", self.search_bright_data)

        graph.add_node("extract", self.extract_minimal_answer)
        graph.add_edge(START, "search")
        graph.add_edge("search", "extract")
        graph.add_edge("extract", END)

        return graph.compile()

Implementation Deep Dive

1. Setting Up Bright Data MCP

The MCP integration is surprisingly straightforward:

async def initialize_bright_data(self):
    """Initialize Bright Data MCP client and tools"""
    if self.bright_data_client is None:
        bright_data_config = {
            "mcpServers": {
                "Bright Data": {
                    "command": "npx",
                    "args": ["@brightdata/mcp"],
                    "env": {
                        "API_TOKEN": os.getenv("BRIGHT_DATA_API_TOKEN"),
                        "WEB_UNLOCKER_ZONE": os.getenv("WEB_UNLOCKER_ZONE", "unblocker"),
                        "BROWSER_ZONE": os.getenv("BROWSER_ZONE", "scraping_browser")
                    }
                }
            }
        }

        self.bright_data_client = MCPClient.from_dict(bright_data_config)
        adapter = LangChainAdapter()
        self.tools = await adapter.create_tools(self.bright_data_client)

2. Dual-Mode Intelligence

The system supports two enrichment modes:

Standard Mode (Gemini + Bright Data)

Fast, efficient enrichment for basic use cases:

async def search_bright_data(self, state: EnrichmentContext):
    """Run standard Bright Data search using MCP"""
    try:
        await self.initialize_bright_data()

        query = f"{state.column_name} of {state.target_value}"
        search_tool = next((tool for tool in self.tools if "search" in tool.name.lower()), None)

        if search_tool:
            search_result = await search_tool.ainvoke({"query": query})
            # Process and return structured results

    except Exception as e:
        logger.error(f"Error in search_bright_data: {str(e)}")
        raise

Deep Research Mode (Kimi K2 + ReAct Agent)

For complex research requiring multi-source verification:

async def deep_search_bright_data(self, state: EnrichmentContext):
    """Run deep research using ReAct Agent with Kimi K2"""
    try:
        # Initialize Kimi K2 through OpenRouter
        kimi_llm = ChatOpenAI(
            openai_api_key=os.getenv("OPENROUTER_API_KEY"),
            openai_api_base="https://openrouter.ai/api/v1",
            model_name="moonshotai/kimi-k2"
        )

        # Create ReAct agent with Bright Data tools
        agent = create_react_agent(
            model=kimi_llm,
            tools=self.tools,
            prompt=system_prompt
        )

        result = await agent.ainvoke({
            "messages": [("human", query)]
        })

    except Exception as e:
        logger.error(f"Error in deep_search: {str(e)}")
        raise

3. Batch Processing for Performance

One key optimization was implementing batch processing to handle entire columns efficiently:

@app.post("/api/enrich/batch", response_model=BatchEnrichmentResponse)
async def enrich_batch(request: BatchEnrichmentRequest):
    """Enrich multiple rows in parallel."""
    try:
        # Process each row
        tasks = []
        for row in request.rows:
            if row.strip():
                task = enrich_cell_with_graph(
                    column_name=request.column_name,
                    target_value=row,
                    context_values=request.context_values,
                    llm_provider=gemini_provider,
                    deep_research=request.deep_research
                )
                tasks.append(task)

        # Execute all enrichments in parallel
        enriched_values = await asyncio.gather(*tasks, return_exceptions=True)

        return BatchEnrichmentResponse(
            enriched_values=final_values, 
            status="success", 
            sources=all_sources
        )

    except Exception as e:
        # Error handling...

4. React Frontend with Real-time Updates

The frontend provides a smooth, spreadsheet-like experience with real-time loading states:

const enrichColumn = async (colIndex: number) => {
  // Set loading state for all cells
  const newRows = data.rows.map((row) => {
    const newRow = [...row];
    if (newRow.some((cell) => cell.value?.length)) {
      newRow[colIndex] = { ...newRow[colIndex], loading: true };
    }
    return newRow;
  });
  setData({ ...data, rows: newRows });

  try {
    const response = await fetch(`${API_URL}/api/enrich/batch`, {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({
        column_name: data.headers[colIndex],
        rows: targetValues,
        context_values: contextValues,
        deep_research: deepResearch,
      }),
    });

    const result = await response.json();

    // Update all cells with enriched data
    const enrichedRows = data.rows.map((row, rowIndex) => {
      const newRow = [...row];
      newRow[colIndex] = {
        value: result.enriched_values[rowIndex],
        sources: result.sources[rowIndex],
        enriched: result.enriched_values[rowIndex] !== "",
        loading: false,
      };
      return newRow;
    });

    setData({ ...data, rows: enrichedRows });
  } catch (error) {
    // Error handling...
  }
};

The Free Tier Advantage

Cost Breakdown (Monthly)

Bright Data MCP: 5,000 requests - $0
Google Gemini: ~15M tokens - $0 (within free tier)
OpenRouter Kimi K2: ~500K tokens - $0 (within free tier)
Hosting: Vercel/Railway free tiers - $0

Total: $0/month for moderate usage

Real-world Usage

With 5,000 Bright Data requests per month, you can realistically enrich:

📊 1,000 companies with 5 data points each
🔍 500 research subjects with 10 data points each
📈 250 comprehensive profiles with 20 data points each

Challenges and Solutions

1. Rate Limiting Coordination

Challenge: Managing rate limits across multiple free services.
Solution: Implemented exponential backoff and request queuing:

async def rate_limited_request(self, func, *args, **kwargs):
    """Execute request with exponential backoff"""
    for attempt in range(3):
        try:
            return await func(*args, **kwargs)
        except Exception as e:
            if "rate limit" in str(e).lower():
                wait_time = (2 ** attempt) * 1
                await asyncio.sleep(wait_time)
            else:
                raise
    raise Exception("Max retries exceeded")

2. Data Quality Consistency

Challenge: Ensuring consistent, structured output from different AI models.
Solution: Robust prompt engineering and output validation:

async def extract_minimal_answer(self, state: EnrichmentContext) -> Dict:
    """Use LLM to extract a minimal answer from Bright Data's results."""
    prompt = f"""
    Extract the {state.column_name} of {state.target_value} from this search result:

    {content}

    Rules:
    1. Provide ONLY the direct answer - no explanations
    2. Be concise
    3. If information is not found, respond "Information not found"
    4. Do not provide citations or references
    Direct Answer:
    """

    answer = await self.llm.generate(prompt)
    return {"answer": answer}

3. CSV Parsing Robustness

Challenge: Handling malformed CSV files with quoted content.
Solution: Custom CSV parser with proper quote handling:

const parseCSVLine = (line: string): string[] => {
  const result: string[] = [];
  let current = '';
  let inQuotes = false;

  for (let i = 0; i < line.length; i++) {
    const char = line[i];
    const nextChar = line[i + 1];

    if (char === '"') {
      if (inQuotes && nextChar === '"') {
        // Handle escaped quotes
        current += '"';
        i++; // Skip the next quote
      } else {
        // Toggle quote state
        inQuotes = !inQuotes;
      }
    } else if (char === ',' && !inQuotes) {
      result.push(current);
      current = '';
    } else {
      current += char;
    }
  }

  result.push(current);
  return result;
};

Deployment and Production Considerations

Environment Setup

# Backend dependencies
pip install fastapi uvicorn langgraph langchain-google-genai langchain-openai

# Environment variables
BRIGHT_DATA_API_TOKEN=your_token
GOOGLE_API_KEY=your_gemini_key
OPENROUTER_API_KEY=your_openrouter_key
WEB_UNLOCKER_ZONE=unblocker
BROWSER_ZONE=scraping_browser

Docker Deployment

FROM python:3.11-slim

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .
EXPOSE 8000

CMD ["python", "app.py"]

Future Enhancements

🔄 Real-time Collaboration: WebSocket integration for multi-user editing
📊 Advanced Analytics: Built-in data visualization and insights
🔌 Custom Connectors: Support for additional data sources via MCP
🤖 Smart Suggestions: AI-powered column header and enrichment suggestions
📈 Usage Analytics: Track enrichment patterns and optimize workflows

Key Takeaways

MCP is a game-changer for AI-web integration - it abstracts away the complexity of web scraping while providing reliable, structured data access.
Free tiers can be powerful when combined strategically. The combination of Bright Data MCP, Gemini, and Kimi K2 provides enterprise-grade capabilities at zero cost.
LangGraph simplifies complex workflows - building sophisticated agent pipelines becomes manageable with proper state management.
User experience matters - Real-time updates, smooth animations, and intuitive interfaces make technical tools accessible to non-technical users.
Batch processing is essential for performance - handling multiple enrichments in parallel dramatically improves user experience.

Get Started

The complete source code is available on GitHub, with detailed setup instructions for both development and production deployment. Whether you're building a research tool, lead generation system, or data analysis platform, this foundation provides a solid starting point for any data enrichment needs.

The future of data enrichment is democratized, intelligent, and surprisingly affordable. With tools like Bright Data's MCP server, we're moving toward a world where powerful data intelligence is accessible to everyone - not just enterprises with large budgets.

Want to see this in action? Check out the live demo or explore the GitHub repository to get started with your own data enrichment projects.

Tags: #brightdata #mcp #ai #datascience #webdev #python #react #typescript #langchain #openai #gemini

Testing Grok-4: Why Even the 'Smartest AI' Needs Real-Time Web Access

Meir — Thu, 10 Jul 2025 15:43:04 +0000

Testing Grok-4: Why Even the "Smartest AI" Needs Real-Time Web Access

Yesterday, Elon Musk and xAI unveiled Grok-4, making audacious claims: "the smartest AI in the world" and "better than PhD level in every subject, no exceptions." The benchmarks seem to back it up—Grok-4 has achieved groundbreaking results that put it at the forefront of AI capabilities.

But I decided to put it to a real-world test that revealed something crucial about the current state of AI.

🚀 Grok-4: The Numbers Don't Lie

The benchmark results are genuinely impressive:

ARC-AGI-2 Performance: 16.2% (nearly twice the score of the next best commercial AI model)
Humanity's Last Exam: 25.4% without tools, 44.4% with tools
GPQA (Graduate-level physics): 87-88%

In the Artificial Analysis Intelligence Index, Grok-4 now leads the field, surpassing OpenAI, Google, Anthropic, and Deepseek. This is the first time an xAI model has taken the top spot.

The ARC-AGI benchmark is particularly significant because it tests an AI's ability to recognize patterns, apply abstract concepts to new situations, and reason with minimal examples—capabilities that can't be achieved through simple pattern matching or memorization.

🧪 The Real-World Test: A Critical Question

But here's where things get interesting. I decided to test Grok-4 with a question that reveals a fundamental limitation of even the most advanced AI models—access to real-time, comprehensive web data.

The Question:

What's the title of the scientific paper published in the EMNLP conference between 2018-2023 where the first author did their undergrad at Dartmouth College and the fourth author did their undergrad at University of Pennsylvania?

This question is actually part of OpenAI's BrowseComp benchmark—a challenging dataset of 1,266 problems designed to test AI agents' ability to locate hard-to-find information on the web. BrowseComp questions are deliberately crafted to be "hard to find but easy to verify," requiring agents to persistently navigate through multiple websites, cross-reference information, and synthesize data from various sources.

Perfect for testing the difference between raw intelligence and connected intelligence.

🛠️ The Technical Setup

I tested Grok-4 in two scenarios using Python and LangChain:

Without Web Access (Baseline Test)

# Simple chat completion without system prompt or tools
llm = ChatOpenAI(
    openai_api_key=os.getenv("OPENROUTER_API_KEY"),
    openai_api_base="https://openrouter.ai/api/v1",
    model_name="x-ai/grok-4"
)
# Direct query without any web access capabilities

With BrightData MCP (Web-Enabled Test)

import asyncio
from langchain_openai import ChatOpenAI
from langgraph.prebuilt import create_react_agent
from mcp_use.client import MCPClient
from mcp_use.adapters.langchain_adapter import LangChainAdapter
from dotenv import load_dotenv
import os

load_dotenv()

async def main():
    # Bright Data MCP configuration
    bright_data_config = {
        "mcpServers": {
            "Bright Data": {
                "command": "npx",
                "args": ["@brightdata/mcp"],
                "env": {
                    "API_TOKEN": os.getenv("BRIGHT_DATA_API_TOKEN"),
                    "WEB_UNLOCKER_ZONE": os.getenv("WEB_UNLOCKER_ZONE", "unblocker"),
                    "BROWSER_ZONE": os.getenv("BROWSER_ZONE", "scraping_browser")
                }
            }
        }
    }

    # Initialize MCP client and tools
    client = MCPClient.from_dict(bright_data_config)
    adapter = LangChainAdapter()
    tools = await adapter.create_tools(client)

    # Initialize LLM
    llm = ChatOpenAI(
        openai_api_key=os.getenv("OPENROUTER_API_KEY"),
        openai_api_base="https://openrouter.ai/api/v1",
        model_name="x-ai/grok-4"
    )

    # System prompt for the ReAct agent
    system_prompt = """
    You are a web search agent with comprehensive scraping capabilities. Your tools include:
    - **search_engine**: Get search results from Google/Bing/Yandex
    - **scrape_as_markdown/html**: Extract content from any webpage with bot detection bypass
    - **Structured extractors**: Fast, reliable data from major platforms
    - **Browser automation**: Navigate, click, type, screenshot for complex interactions

    Guidelines:
    - Use structured web_data_* tools for supported platforms when possible
    - Use general scraping for other sites
    - Handle errors gracefully and respect rate limits
    - Think step by step about what information you need and which tools to use
    - Be thorough in your research and provide comprehensive answers
    """

    # Create the ReAct agent
    agent = create_react_agent(
        model=llm,
        tools=tools,
        prompt=system_prompt
    )

    # Test the agent
    search_result = await agent.ainvoke({
        "messages": [("human", "What's the title of the scientific paper " +
                     "published in the EMNLP conference between 2018-2023 " +
                     "where the first author did their undergrad at Dartmouth " +
                     "College and the fourth author did their undergrad at " +
                     "University of Pennsylvania?")]
    })

    print("\nSearch Results:")
    print(search_result["messages"][-1].content)

if __name__ == "__main__":
    asyncio.run(main())

📊 The Results: A Tale of Two AIs

✅ With BrightData MCP: Perfect Accuracy

Using BrightData's Model Context Protocol (MCP), which empowers AI models with real-time, reliable access to public web data, the result was flawless:

Correct Answer: The title of the scientific paper is "Frequency Effects on Syntactic Rule Learning in Transformers". Published in EMNLP 2021, with Kevin Yang (first author, Dartmouth undergrad) and Dan Jurafsky (fourth author, University of Pennsylvania undergrad).

❌ Without BrightData MCP: Confident but Wrong

Without real-time web access, Grok-4 provided a detailed, confident response about "ConvoSumm: Conversation Summarization Benchmark and Improved Abstractive Summarization with Argument Mining" by Alexander R. Fabbri et al. The response included publication details, author verification, and even a link—but it was completely incorrect.

🔍 The Broader Implications

This test reveals something crucial about the current state of AI: raw intelligence without real-time data access has fundamental limitations. Even with Grok-4's impressive "intelligence Big Bang" moment and its breakthrough performance on abstract reasoning tasks, it still struggles with the kind of complex, multi-hop web research that BrowseComp was designed to measure.

The significance becomes even clearer when we consider that OpenAI's own testing showed:

GPT-4o with browsing: only 1.9% accuracy on BrowseComp
OpenAI's specialized Deep Research model: 51.5% accuracy

This demonstrates that effective web research requires more than just intelligence—it requires specialized tools and persistent, strategic browsing capabilities.

🏗️ The Technical Achievement Behind Grok-4

Don't mistake this limitation for a criticism of Grok-4's capabilities. The technical achievements are remarkable:

100 times more training compute than Grok-2
Multi-agent setup where multiple agents work on problems simultaneously "like a study group"
Multimodal capabilities, coding options, and voice features

Elon Musk's 28-month-old AI venture has officially claimed the top spot in independent AI benchmarks, marking the first time the company has led the AI frontier against established players.

🌐 The Future of Connected AI

What this test demonstrates is that the future of AI isn't just about smarter models—it's about smarter, connected models. The combination of Grok-4's reasoning capabilities with real-time web access through tools like BrightData MCP represents a new paradigm:

Benchmark intelligence for abstract reasoning and problem-solving
Real-time data access for current information and verification
Seamless integration through protocols like MCP

MCP acts as the "USB-C port for LLMs: plug in a scraper today, a database tomorrow, or an internal microservice next week—no bespoke integrations required."

💡 Key Takeaways for Developers

Don't rely on LLMs alone for research-intensive tasks
Implement real-time web access through tools like MCP
Test your AI applications with challenging, real-world scenarios
Consider the BrowseComp benchmark for evaluating web research capabilities
Invest in connected AI infrastructure for competitive advantage

🎯 Conclusion: Intelligence + Access = True AI Power

Grok-4 represents a significant breakthrough in AI capabilities, with benchmark results that suggest we're watching models do "things we thought only minds could do". But my test reveals that even the most advanced AI models need real-time web access to reach their full potential.

The combination of Grok-4's reasoning power with BrightData MCP's web access capabilities isn't just additive—it's transformative. It's the difference between having a brilliant research assistant who works from memory versus one who has access to the world's largest library.

The takeaway: Grok-4 is indeed remarkable, but the real magic happens when breakthrough intelligence meets breakthrough connectivity. That's where the future of AI truly lies.

Want to try BrightData MCP yourself? New users get free credit for testing, and you can integrate it with Claude, Cursor, or other MCP-compatible tools in minutes.

🔗 Resources

What do you think about the future of connected AI? Have you experimented with web-enabled AI agents? Share your experiences in the comments! 💬

Building a Unified Search Agent with LangGraph and MCP: From Intent Classification to Production Deployment

Meir — Thu, 26 Jun 2025 16:09:20 +0000

Building a Unified Search Agent with LangGraph and MCP: From Intent Classification to Production Deployment

Ever wished you could have a single AI agent that intelligently decides whether to use Google search, scrape specific websites, or combine multiple data sources based on your query? That's exactly what I built with the Unified Search Agent - a sophisticated multi-modal search system that uses LLM-powered intent classification to route queries to the most appropriate search strategy.

🎯 The Problem

Traditional search solutions often force you to choose upfront: Do you want web search results or scraped data? Should you check multiple sources or stick to one? The Unified Search Agent eliminates this decision fatigue by automatically:

Classifying your intent using Gemini 2.0 Flash
Routing intelligently between Google search and web scraping
Combining results from multiple sources when needed
Scoring and ranking final results for maximum relevance

🏗️ Architecture Overview

The agent follows a sophisticated graph-based workflow built with LangGraph:

START → Intent Classifier → [Google Search | Web Scraping | Both] → Final Processing → END

Intent Classification Categories

The system classifies queries into four categories:

general_search: News, facts, definitions, explanations
product_search: Shopping, prices, reviews, recommendations
web_scraping: Data extraction from specific websites
comparison: Comparing multiple items or services

Smart Routing Logic

URLs detected → Direct to Web Scraping
General search → Google Search only
Product search → Google Search → Web Scraping
Web scraping → Web Scraping only
Comparison → Both methods in parallel

🛠️ Technology Stack

LangGraph: Orchestration and state management
Gemini 2.0 Flash: Intent classification and result processing
Bright Data MCP: Google search and web scraping
Pydantic: Structured data validation
FastMCP: MCP server wrapper for easy integration

🚀 Key Features

1. Intelligent Intent Classification

class IntentClassification(BaseModel):
    intent: str = Field(description="The classified intent")
    confidence: float = Field(description="Confidence score", ge=0.0, le=1.0)
    reasoning: str = Field(description="Explanation for classification")

The system uses structured output to ensure consistent, reliable classification with confidence scoring.

2. Multi-Source Data Integration

The agent seamlessly combines results from:

Google Search via Bright Data's search engine
Web Scraping using Bright Data's Web Unlocker
Parallel processing for comparison queries

3. Advanced Result Scoring

Every result gets scored on two dimensions:

Relevance Score (0-1): How well it answers the query
Quality Score (0-1): Source authority and content quality
Final Score: (relevance * 0.7) + (quality * 0.3)

4. Production-Ready Error Handling

def intent_classifier_node(state: Dict[str, Any]) -> Dict[str, Any]:
    try:
        # Classification logic
        result = structured_llm.invoke(full_prompt)
        # Update state
    except Exception as e:
        # Graceful fallback
        state["intent"] = "general_search"
        state["intent_confidence"] = 0.5
        state["error"] = f"Classification error: {str(e)}"
    return state

Every node includes comprehensive error handling with meaningful fallbacks.

💡 Usage Examples

Basic Search

{
  "query": "latest developments in quantum computing",
  "max_results": 5
}

Product Research

{
  "query": "best noise-canceling headphones under $200",
  "max_results": 8
}

Targeted Web Scraping

{
  "query": "extract pricing from https://example-store.com/laptops",
  "max_results": 10
}

Comparison Analysis

{
  "query": "MacBook Pro M3 vs Dell XPS 15 specifications comparison",
  "max_results": 6
}

🌐 MCP Server Integration

One of the coolest features is that the entire agent is wrapped as an MCP (Model Context Protocol) server:

from fastmcp import FastMCP
from src.agent.graph import create_unified_search_graph

graph = create_unified_search_graph()
mcp = FastMCP("Unified-Search-Agent")

@mcp.tool()
async def search(search_term: str) -> str:
    """Run unified search across multiple platforms"""
    result = await graph.ainvoke({"query": search_term})
    return result.get("final_results", "No results found.")

This means you can easily integrate the search agent into:

Claude Desktop via MCP
Other AI applications that support MCP
Custom workflows that need intelligent search

☁️ LangGraph Cloud Deployment

The agent is designed for seamless deployment on LangGraph Cloud:

{
  "dependencies": ["."],
  "graphs": {
    "agent": "./src/agent/graph.py:graph"
  },
  "env": ".env",
  "image_distro": "wolfi"
}

Simply configure your environment variables and deploy:

# Required API keys
GOOGLE_API_KEY=your_gemini_api_key
BRIGHT_DATA_API_TOKEN=your_bright_data_token

# Optional - for tracing
LANGSMITH_API_KEY=your_langsmith_key

🎨 LangGraph Studio Integration

The agent integrates beautifully with LangGraph Studio for visual debugging:

You can:

Visualize the entire workflow in real-time
Debug specific nodes by editing past states
Hot reload changes during development
Monitor performance with integrated tracing

📊 Example Output Structure

The agent returns comprehensively structured results:

{
  "final_results": [
    {
      "title": "Quantum Computing Breakthrough 2024",
      "url": "https://example.com/quantum-news",
      "snippet": "Scientists achieve new milestone...",
      "source": "google_search",
      "relevance_score": 0.95,
      "quality_score": 0.88,
      "final_score": 0.92,
      "metadata": {
        "search_engine": "google",
        "via": "bright_data_mcp"
      }
    }
  ],
  "query_summary": "Found 5 recent articles about quantum computing advances",
  "total_processed": 12,
  "intent": "general_search",
  "intent_confidence": 0.95
}

🔄 Extending the Agent

The modular architecture makes it easy to extend:

Add New Search Sources

workflow.add_node("scholarly_search", scholarly_search_node)
workflow.add_conditional_edges("intent_classifier", route_with_scholarly)

Customize Intent Categories

# Add to intent_classifier_node
intents = ["general_search", "academic_research", "product_search", "web_scraping"]

Modify Scoring Algorithms

# Update in final_processing_node
final_score = (relevance * 0.6) + (quality * 0.3) + (freshness * 0.1)

🚦 Getting Started

Clone the repository:

git clone https://github.com/MeirKaD/unified-search
cd unified-search-agent

Install dependencies:

pip install -e . "langgraph-cli[inmem]"

Set up environment:

cp .env.example .env
# Add your API keys to .env

Start development server:

langgraph dev

Open LangGraph Studio and start experimenting!

🎯 Real-World Applications

This architecture pattern is perfect for:

Research Assistants: Automatically choosing between academic databases and web search
E-commerce Tools: Combining product catalogs with review sites
Market Research: Gathering data from multiple competitive intelligence sources
Content Curation: Intelligently sourcing content from various platforms
Due Diligence: Comprehensive information gathering for business decisions

🔮 Future Enhancements

The agent architecture supports easy addition of:

Vector database integration for semantic search
Real-time data streams for live information
Specialized extractors for platforms like LinkedIn, Twitter, Reddit
Multi-language support with automatic translation
Custom scoring models trained on your specific use cases

💭 Lessons Learned

Building this agent taught me several key insights:

Intent classification is crucial - Getting this right determines the entire user experience
Structured outputs are game-changers - Pydantic models ensure reliability at scale
Error handling is not optional - Every node needs graceful degradation
Visual debugging saves hours - LangGraph Studio made development 10x faster
MCP integration opens new possibilities - Wrapping as MCP makes the agent universally useful

🎉 Conclusion

The Unified Search Agent represents a new paradigm in AI-powered search - one where intelligence lies not just in finding information, but in knowing how to find it. By combining LangGraph's orchestration capabilities with advanced intent classification and multi-modal search strategies, we've created something that adapts to user needs rather than forcing users to adapt to tool limitations.

The fact that it deploys seamlessly to LangGraph Cloud and integrates via MCP makes it a practical solution for real-world applications. Whether you're building a research assistant, enhancing customer support, or creating the next generation of search experiences, this architecture provides a solid foundation.

Try it out and let me know what you build!

🔗 Links

What search challenges are you trying to solve? Drop a comment and let's discuss how this architecture could help! 👇

Building FactFlux: A Multi-Agent System for Social Media Fact-Checking

Meir — Thu, 19 Jun 2025 14:38:32 +0000

Building FactFlux: A Multi-Agent System for Social Media Fact-Checking

In an era where misinformation spreads faster than wildfire across social platforms, I built FactFlux - an intelligent multi-agent system that automatically fact-checks social media posts using the power of AI agents working in coordination.

🚨 The Problem

Social media has become the primary source of news for millions, but it's also a breeding ground for misinformation. Manual fact-checking is slow and doesn't scale. We need automated solutions that can:

Extract content from multiple social platforms
Identify verifiable claims within posts
Cross-reference against authoritative sources
Provide transparent, evidence-based verdicts

🎯 The Solution: Multi-Agent Architecture

FactFlux uses a team of specialized AI agents working together, each with a specific role in the fact-checking pipeline. Here's how it works:

🎥 Demo

Bright Data's MCP server

🏗️ Agent Team Structure

1. Content Extractor Agent

def create_content_extractor_agent_sync(mcp_tools):
    return Agent(
        name="Content Extractor",
        role="Extract comprehensive data from social media posts",
        model=Gemini(id="gemini-2.0-flash"),
        tools=[mcp_tools],
        instructions=[
            "You MUST use the available tools to extract data from social media posts",
            "For TikTok URLs: Use the web_data_tiktok_posts tool to get structured data",
            "For other platforms or if structured data fails: Use scrape_as_markdown tool",
            "Extract: post text, media URLs, user info, engagement metrics, timestamps",
        ]
    )

Responsibilities:

Extracts post content from 6+ platforms (TikTok, Instagram, Twitter/X, Facebook, YouTube, LinkedIn)
Uses Bright Data tools for reliable scraping
Captures text, media, metadata, and engagement metrics

2. Claim Identifier Agent

def create_claim_identifier_agent():
    return Agent(
        name="Claim Identifier",
        role="Identify factual claims that can be verified",
        model=Gemini(id="gemini-2.0-flash"),
        tools=[ReasoningTools(add_instructions=True)],
        instructions=[
            "Parse extracted content to find specific, verifiable factual claims",
            "Ignore opinions, jokes, satire, and subjective statements",
            "Extract key facts: statistics, events, quotes, dates, locations",
            "Focus on claims that could potentially mislead people if false",
        ]
    )

Responsibilities:

Separates factual claims from opinions and satire
Prioritizes verifiable statements
Rates claim significance and potential for harm

3. Cross-Reference Agent

def create_cross_reference_agent(mcp_tools):
    return Agent(
        name="Cross-Reference Verifier",
        role="Verify claims against multiple authoritative sources",
        model=Gemini(id="gemini-2.0-flash"),
        tools=[mcp_tools, ReasoningTools(add_instructions=True)],
        instructions=[
            "For each claim, search multiple authoritative sources automatically",
            "Check news sites, fact-checkers, government sources, academic sources",
            "For media content, perform reverse image/video searches",
            "Document all sources and their credibility levels",
        ]
    )

Responsibilities:

Searches authoritative sources (news sites, fact-checkers, government sources)
Performs reverse media searches for images/videos
Documents source credibility and consensus patterns

4. Verdict Agent

def create_verdict_agent():
    return Agent(
        name="Verdict Synthesizer",
        role="Analyze all evidence and deliver final fact-check verdict",
        model=Gemini(id="gemini-2.0-flash"),
        tools=[ThinkingTools(add_instructions=True), ReasoningTools(add_instructions=True)],
        instructions=[
            "Structure your response with these sections:",
            "## Post Summary - Describe what the post contained",
            "## Claims Identified - List each claim found",
            "## Verification Results - Detail findings for each claim with sources",
            "## Final Verdict - Clear verdict with confidence score (0-100%)",
        ]
    )

Responsibilities:

Synthesizes all evidence from previous agents
Provides structured analysis with clear sections
Delivers final verdict: TRUE/FALSE/MISLEADING/INSUFFICIENT_EVIDENCE
Includes confidence scoring and detailed reasoning

🔄 The Workflow Process

URL Input → Content Extraction → Claim Identification → Cross-Reference → Final Verdict

Step-by-Step Breakdown:

Input: User provides a social media URL
Extraction: Content Extractor Agent scrapes the post using Bright Data tools
Analysis: Claim Identifier Agent parses content for verifiable facts
Verification: Cross-Reference Agent searches authoritative sources
Synthesis: Verdict Agent combines all evidence into a final assessment

🛠️ Technical Implementation

Core Technologies Used:

Agno Framework: For multi-agent orchestration
Google Gemini 2.0: As the language model powering each agent
Bright Data MCP (Model Context Protocol): For tool integration

Key Features:

Multi-Platform Support

# Supports major social platforms
platforms = [
    "TikTok", "Instagram", "Twitter/X", 
    "Facebook", "YouTube", "LinkedIn"
]

Intelligent Tool Selection

The system automatically chooses the optimal scraping method based on the platform:

Structured data extraction for TikTok
Markdown scraping for other platforms
Fallback mechanisms for reliability

Comprehensive Error Handling

# Built-in resilience for:
error_scenarios = [
    "Invalid URLs",
    "Network failures", 
    "API rate limits",
    "Malformed social media posts",
    "Missing content"
]

🚀 Getting Started

Prerequisites

Python 3.8+
Google Gemini API key
Bright Data API key

Installation

# Clone the repository
git clone https://github.com/MeirKaD/FactFlux.git
cd FactFlux

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Configure environment variables
cp .env.example .env
# Edit .env with your API keys

Configuration

Create a .env file:

GOOGLE_API_KEY=your_google_gemini_api_key_here
BRIGHT_DATA_API_KEY=your_bright_data_api_key_here

Usage

# Run the playground interface
python playground_fact_check.py

The system launches a web interface at http://localhost:7777 where you can:

Input social media URLs
Watch agents work in real-time
See the complete fact-checking process
Review structured reports with sources

📊 Sample Output

When you submit a social media URL, FactFlux produces a comprehensive report:

## Post Summary
TikTok video claiming "95% of ocean plastic comes from 10 rivers in Asia"

## Claims Identified
1. 95% of ocean plastic originates from rivers
2. These rivers are specifically located in Asia
3. Only 10 rivers are responsible

## Verification Results
- Claim 1: PARTIALLY TRUE - Studies show 80-90% from rivers (sources: Nature, Science journals)
- Claim 2: MISLEADING - 8 rivers in Asia, 2 in Africa (source: Environmental Science & Technology)
- Claim 3: TRUE - Confirmed by multiple peer-reviewed studies

## Citations
[1] Lebreton et al. (2017) Nature Communications - doi:10.1038/ncomms15611
[2] Schmidt et al. (2017) Environmental Science & Technology - doi:10.1021/acs.est.7b02368

## Final Verdict: MISLEADING (Confidence: 85%)
The core claim about river pollution is scientifically supported, but the specific statistics are inaccurate.

💡 Why Multi-Agent Architecture?

Specialization Benefits:

Each agent focuses on what it does best
Reduces complexity and improves accuracy
Easier to debug and improve individual components

Scalability:

Can add new agents for specific platforms or verification types
Parallel processing capabilities
Modular design allows independent updates

Transparency:

Shows the complete reasoning chain
Users can see how each agent contributed
Builds trust through explainable AI

🔮 Future Enhancements

Planned Features:

Real-time monitoring of trending posts
Browser extension for instant fact-checking
API endpoints for integration with other platforms
Machine learning to improve claim detection accuracy
Collaborative filtering with human fact-checkers

Technical Improvements:

Support for more social platforms (Reddit, Snapchat, Discord)
Enhanced media analysis (deepfake detection)
Multi-language support
Performance optimizations for high-volume processing

🤝 Contributing

FactFlux is open-source and welcomes contributions:

# Contributing workflow
1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

Areas for contribution:

New platform integrations
Improved claim detection algorithms
Additional verification sources
UI/UX improvements
Documentation and tutorials

🔍 Technical Deep Dive

Agent Coordination

fact_check_team = Team(
    name="Social Media Fact Check Team",
    mode="coordinate",
    model=Gemini(id="gemini-2.0-flash"),
    members=[content_extractor, claim_identifier, cross_reference, verdict_agent],
    show_members_responses=True,
    enable_agentic_context=True,
)

The team uses coordinate mode, meaning agents work sequentially, passing context between each step. This ensures:

Each agent has access to previous findings
No information is lost in the pipeline
The final verdict considers all evidence

Tool Integration

server_params = StdioServerParameters(
    command="npx",
    args=["@brightdata/mcp"],
    env={
        "API_TOKEN": os.getenv("BRIGHT_DATA_API_KEY"),
        "WEB_UNLOCKER_ZONE": "unblocker",
        "BROWSER_ZONE": "scraping_browser"
    }
)

Bright Data integration provides:

Reliable scraping across geo-blocked content
Anti-bot detection mechanisms
High success rates for social media extraction

📈 Impact and Use Cases

Educational Institutions

Teaching media literacy
Research on misinformation patterns
Academic fact-checking projects

News Organizations

Automated preliminary fact-checking
Source verification assistance
Social media monitoring

Individual Users

Personal fact-checking tool
Browser extension for real-time verification
Educational resource for critical thinking

🔒 Ethical Considerations

Responsible AI Principles:

Transparency: All sources and reasoning are shown
Accuracy: Conservative approach when evidence is weak
Privacy: No personal data storage or tracking
Bias Mitigation: Multiple source cross-referencing

Limitations:

Cannot replace human judgment entirely
Dependent on source availability and quality
May struggle with highly contextual or cultural claims
Requires ongoing updates as misinformation tactics evolve

🎯 Conclusion

FactFlux represents a significant step forward in automated fact-checking technology. By leveraging multi-agent architecture, we've created a system that's both powerful and transparent, capable of handling the complexity of modern misinformation while remaining explainable to users.

The project demonstrates how AI agents can work together to solve complex problems that require multiple specialized skills. As misinformation continues to evolve, tools like FactFlux will play a crucial role in maintaining information integrity across social platforms.

Try FactFlux today: GitHub Repository

🏷️ Tags

#ai #factchecking #misinformation #python #multiagent #socialmedia #automation #gemini #brightdata #opensource

Have you built similar AI agent systems? What challenges did you face with multi-agent coordination? Share your experiences in the comments below!

Building an AI-Powered Product Price Insurance Agent with LangGraph & Streamlit

Meir — Mon, 16 Jun 2025 14:28:18 +0000

Building an AI-Powered Product Price Insurance Agent with LangGraph & Streamlit

Ever needed to quickly find the current market value of damaged electronics for insurance claims? Or compare prices across Amazon, Walmart, and Best Buy for bulk purchasing decisions?

I recently built an AI-powered Product Price Insurance Agent that solves exactly this problem. In this post, I'll walk you through the complete architecture, implementation, and lessons learned.

🎯 The Problem

When filing insurance claims for damaged property, you need accurate replacement values—not outdated prices from months ago. Traditional approaches involve:

Manual searching across multiple retailers
Copy-pasting prices into spreadsheets
Inconsistent data formats
No confidence scoring
Time-consuming manual analysis

What if we could automate this entire process with AI?

🏗️ Solution Architecture

I designed a simple yet powerful 3-stage workflow using LangGraph as the orchestration framework.

Why LangGraph?

LangGraph excels at building stateful, multi-step AI workflows. Unlike simple LLM chains, it provides:

State management between nodes
Error handling at each stage
Conditional routing based on results
Easy testing of individual components

🛠️ Tech Stack

🧠 LangGraph: Workflow orchestration
🌐 Bright Data: Web scraping with MCP integration
🤖 Google Gemini: LLM with structured outputs
💬 Streamlit: Interactive chat interface
🐍 Python: Core implementation language

📋 Implementation Deep Dive

State Definition

First, I defined a simple state structure using TypedDict:

from typing import TypedDict, List, Optional, Dict

class InsuranceState(TypedDict):
    """Simple state for product price insurance workflow."""
    # Input
    product_query: str

    # Node outputs
    search_results: Dict[str, str]  # platform -> URL
    price_data: List[Dict[str, any]]  # extracted prices
    final_report: str

    # Optional fields
    error: Optional[str]
    confidence_score: Optional[float]

Node 1: Product Search

The first node searches for product URLs across major platforms using MCP (Model Context Protocol) with Bright Data:

from pydantic import BaseModel, Field
from typing import Optional

class ProductURLs(BaseModel):
    """Structured output for extracted product URLs."""
    amazon: Optional[str] = Field(None, description="Amazon product page URL")
    walmart: Optional[str] = Field(None, description="Walmart product page URL") 
    bestbuy: Optional[str] = Field(None, description="Best Buy product page URL")

async def search_products(state: InsuranceState) -> InsuranceState:
    product_query = state["product_query"]

    try:
        # Configure MCP client for Bright Data
        browserai_config = {
            "mcpServers": {
                "BrightData": {
                    "command": "npx",
                    "args": ["@brightdata/mcp"],
                    "env": {
                        "API_TOKEN": os.getenv("BRIGHT_DATA_API_TOKEN"),
                        "WEB_UNLOCKER_ZONE": os.getenv("WEB_UNLOCKER_ZONE", "unblocker")
                    }
                }
            }
        }

        client = MCPClient.from_dict(browserai_config)
        adapter = LangChainAdapter()
        tools = await adapter.create_tools(client)

        # Create ReAct agent with LLM + tools
        agent = create_react_agent(
            model=llm,
            tools=tools,
            prompt="Find direct product pages on Amazon, Walmart, and Best Buy..."
        )

        result = await agent.ainvoke({
            "messages": [{"role": "user", "content": f"Find product pages for: {product_query}"}]
        })

        # Use structured output instead of regex parsing
        structured_llm = llm.with_structured_output(ProductURLs)
        url_response = await structured_llm.ainvoke(
            f"Extract product URLs from these search results:\n\n{result['messages'][-1].content}"
        )

        # Filter out None values
        urls = {k: v for k, v in url_response.dict().items() if v is not None}

        return {**state, "search_results": urls}

    except Exception as e:
        return {**state, "search_results": {}, "error": f"Search failed: {str(e)}"}

Node 2: Price Extraction

The second node extracts price data from the found URLs:

class ExtractedPrice(BaseModel):
    """Structured output for extracted price information."""
    price: Optional[float] = Field(None, description="Product price as number")
    title: str = Field("", description="Product name/title")
    availability: str = Field("Unknown", description="Availability status")

async def extract_prices(state: InsuranceState) -> InsuranceState:
    search_results = state.get("search_results", {})

    if not search_results:
        return {**state, "price_data": [], "error": "No URLs found"}

    price_data = []

    # Configure MCP client and create extraction agent
    # ... (similar setup as search node)

    for platform, url in search_results.items():
        try:
            # Use platform-specific extractors
            extraction_prompt = f"Extract price, title, and availability from: {url}"
            result = await agent.ainvoke({"messages": [{"role": "user", "content": extraction_prompt}]})

            # Use structured output for reliable parsing
            structured_llm = llm.with_structured_output(ExtractedPrice)
            structured_data = await structured_llm.ainvoke(
                f"Extract price data for {platform}:\n\n{result['messages'][-1].content}"
            )

            price_data.append({
                "platform": platform,
                "price": structured_data.price,
                "title": structured_data.title,
                "url": url,
                "availability": structured_data.availability
            })

        except Exception as e:
            price_data.append({
                "platform": platform,
                "price": None,
                "title": "",
                "url": url,
                "availability": "Error extracting",
                "error": str(e)
            })

    return {**state, "price_data": price_data}

Node 3: Report Generation

The final node calculates statistics and generates a formatted report:

import statistics

async def generate_report(state: InsuranceState) -> InsuranceState:
    price_data = state.get("price_data", [])
    product_query = state["product_query"]

    # Extract valid prices
    valid_prices = [item["price"] for item in price_data if item.get("price") is not None]

    if not valid_prices:
        return {
            **state,
            "final_report": f"❌ No prices found for {product_query}",
            "confidence_score": 0.0
        }

    # Calculate statistics
    median_price = statistics.median(valid_prices)
    average_price = statistics.mean(valid_prices)
    min_price = min(valid_prices)
    max_price = max(valid_prices)

    # Calculate confidence score
    platforms_found = len(valid_prices)
    confidence_score = min(10.0, (platforms_found / 3.0) * 8.0 + 2.0)

    # Use LLM to format the report
    report_prompt = ChatPromptTemplate.from_messages([
        ("system", "Create a clean, professional price analysis report with emojis and clear sections..."),
        ("user", "Create report for: {product}\nMedian: ${median:.2f}\nAverage: ${average:.2f}...")
    ])

    report_chain = report_prompt | llm
    report_response = await report_chain.ainvoke({
        "product": product_query,
        "median": median_price,
        "average": average_price,
        # ... other parameters
    })

    return {
        **state,
        "final_report": report_response.content,
        "confidence_score": confidence_score
    }

Graph Assembly

Putting it all together with LangGraph:

from langgraph.graph import StateGraph

def create_insurance_graph():
    """Create and compile the insurance price analysis graph."""

    workflow = StateGraph(InsuranceState)

    # Add nodes
    workflow.add_node("search_products", search_products)
    workflow.add_node("extract_prices", extract_prices)
    workflow.add_node("generate_report", generate_report)

    # Add edges (linear flow)
    workflow.add_edge("search_products", "extract_prices")
    workflow.add_edge("extract_prices", "generate_report")

    # Set entry and finish points
    workflow.set_entry_point("search_products")
    workflow.set_finish_point("generate_report")

    return workflow.compile()

💬 Streamlit Interface

For the user interface, I built a chat-style app with real-time progress updates:

import streamlit as st
import asyncio

def main():
    st.set_page_config(page_title="🛡️ Product Price Insurance", layout="wide")

    st.markdown("# 🛡️ Product Price Insurance System")
    st.markdown("Find and compare prices across major retailers with AI-powered analysis")

    # Chat interface
    if prompt := st.chat_input("Enter a product name (e.g., 'iPhone 16 256GB')"):
        with st.chat_message("user"):
            st.write(prompt)

        with st.chat_message("assistant"):
            # Progress tracking
            progress_bar = st.progress(0)
            status_text = st.empty()
            results_container = st.container()

            def progress_callback(stage, data, progress_value):
                progress_bar.progress(progress_value)
                if stage == "search_complete":
                    with results_container:
                        display_urls_step(data)
                elif stage == "extract_complete":
                    with results_container:
                        display_prices_step(data)

            # Run analysis with progress updates
            result = asyncio.run(run_analysis_with_progress(prompt, progress_callback))

            # Display final results
            if result.get("final_report"):
                display_final_report(result["final_report"], result.get("confidence_score", 0))

if __name__ == "__main__":
    main()

🧪 Testing Strategy

I built a testing suite to validate each node individually:

async def test_search_products():
    """Test the search_products node in isolation."""

    initial_state = {
        "product_query": "iPhone 16 256GB",
        "search_results": {},
        "price_data": [],
        "final_report": "",
        "error": None,
        "confidence_score": None
    }

    result = await search_products(initial_state)

    assert len(result.get('search_results', {})) > 0
    assert result.get('error') is None
    print(f"✅ Found {len(result['search_results'])} URLs")

📊 Results & Performance

The agent successfully:

Finds product URLs across 3 major platforms in ~3-5 seconds
Extracts accurate prices with 85%+ success rate
Generates professional reports with confidence scoring
Handles errors gracefully with meaningful fallbacks

Sample Output

📱 Product Price Analysis: iPhone 16 256GB

💰 Price Summary:
• Average Price: $899.99
• Median Price: $899.00  
• Price Range: $849.00 - $949.00

🏪 Platform Breakdown:
• Amazon: $899.00 - iPhone 16 256GB Unlocked
• Walmart: $849.00 - Apple iPhone 16 256GB  
• Best Buy: $949.00 - iPhone 16 256GB

🎯 Confidence Score: 8.5/10
✅ 3/3 platforms successfully searched

🚀 Key Learnings

1. Structured Outputs > Regex Parsing

Using with_structured_output() instead of regex made the system dramatically more reliable:

# ❌ Fragile regex approach
price_match = re.search(r"Price:?\s*(\d+\.?\d*)", content)

# ✅ Reliable structured output
structured_llm = llm.with_structured_output(ExtractedPrice)
result = await structured_llm.ainvoke(content)

2. MCP Integration is Powerful

The Model Context Protocol with Bright Data provided:

Built-in rate limiting
Platform-specific extractors
Bot detection bypass
Reliable infrastructure

3. LangGraph Simplifies Complex Workflows

The declarative approach made the system:

Easy to test (each node independently)
Simple to debug (clear state transitions)
Straightforward to extend (just add nodes/edges)

4. Real-time Progress Matters

Users need to see what's happening during the 10-15 second analysis:

Step-by-step updates keep users engaged
Intermediate results build confidence
Error transparency improves trust

🎯 Real-World Applications

This architecture works for many use cases:

📋 Insurance Claims: Accurate replacement valuations
💼 Procurement: Bulk purchasing decisions
📊 Market Research: Competitor price monitoring
🛒 Consumer Shopping: Smart buying decisions

🔮 Future Enhancements

📈 Price History Tracking: Store historical data for trends
🎯 More Platforms: Target, eBay, Newegg integration
📱 Mobile App: React Native or Flutter interface
🔔 Price Alerts: Notify when prices drop
📊 Analytics Dashboard: Business intelligence features

💡 Conclusion

Building this AI agent taught me that modern AI tooling makes complex workflows surprisingly approachable. The combination of:

LangGraph for orchestration
Structured outputs for reliability
Bright Data's MCP for web data
Streamlit for rapid UI development

...creates a powerful foundation for real-world AI applications.

The key is starting simple (3 nodes, linear flow) and focusing on reliability over complexity. You can always add sophistication later.

🛠️ Try It Yourself

Want to build something similar? Here's the tech stack:

pip install langgraph streamlit langchain-google-genai mcp-use python-dotenv

The complete code is structured as:

price-shield-agent/
├── state.py          # State definition
├── nodes.py          # Node implementations  
├── graph.py          # Graph builder
├── app.py            # Streamlit interface
└── test_nodes.py     # Testing suite

Environment variables needed:

BRIGHT_DATA_API_TOKEN
GOOGLE_API_KEY

What would you build with this architecture? Drop a comment below—I'd love to hear your ideas! 🚀

Leave a start on Bright Data's MCP repo : https://github.com/brightdata/brightdata-mcp

Follow me for more AI engineering content and real-world implementations.

EasyDocs: Building an AI-Powered API Documentation Agent with LangGraph and Bright Data

Meir — Thu, 12 Jun 2025 10:15:14 +0000

How I built a production-ready system that transforms natural language queries into copy-paste API documentation

The Problem Every Developer Faces

We've all been there. You're deep in a coding session, need to integrate with an API, and suddenly you're drowning in scattered documentation, outdated Stack Overflow posts, and incomplete code examples. What if you could just ask in plain English: "How do I send a POST request to Stripe's payment API?" and get back clean, production-ready code with authentication, error handling, and everything you need?

That's exactly what I built with EasyDocs – an AI-powered documentation agent that combines LangGraph's workflow orchestration with Bright Data's enterprise web scraping to deliver instant, reliable API documentation.

The system uses a 4-node LangGraph workflow where each step has a specific responsibility:

Analyze Query: Uses Google Gemini to identify platform and operation type
Generate Plan: Creates actionable browser automation steps
Execute Browser: Leverages Bright Data's MCP for robust web scraping
Generate Response: Formats everything into developer-friendly documentation

Demo

The Code: LangGraph State Management

First, let's look at the state structure that flows through our workflow:

from typing import TypedDict, List, Optional

class DemoState(TypedDict):
    query: str
    platform: str
    action_plan: List[str]
    extracted_content: str
    final_response: str
    error: Optional[str]
    operation_type: str
    confidence: float
    estimated_duration: int
    complexity_level: str
    current_step: int
    confidence_level: Optional[int]
    explanation: Optional[str]

This typed state ensures data consistency across all workflow nodes and makes debugging much easier.

Node Implementation: Smart Query Analysis

The first node uses structured output with Pydantic models:

from pydantic import BaseModel, Field
from langchain_google_genai import ChatGoogleGenerativeAI

class QueryAnalysis(BaseModel):
    platform: str = Field(
        description="The platform/service mentioned (e.g., 'stripe', 'openai')"
    )
    operation_type: str = Field(
        description="API operation type (e.g., 'GET', 'POST', 'authentication')"
    )
    confidence: Optional[float] = Field(
        default=None,
        description="Confidence score from 0.0 to 1.0"
    )

llm = ChatGoogleGenerativeAI(model="gemini-2.0-flash", temperature=0)
structured_llm = llm.with_structured_output(QueryAnalysis)

async def analyze_query(state: DemoState) -> DemoState:
    query = state["query"]

    analysis_prompt = f"""
    Analyze this query to extract:
    1. The platform/service they're asking about
    2. The type of API operation they want
    3. Your confidence in this analysis

    Query: {query}

    Look for: Bright Data, Stripe, OpenAI, Twilio, etc.
    Operations: GET, POST, authentication, webhook, etc.
    """

    try:
        analysis = await structured_llm.ainvoke(analysis_prompt)
        return {
            "platform": analysis.platform.lower().replace(" ", "_"),
            "operation_type": analysis.operation_type,
            "confidence": analysis.confidence or 0.8
        }
    except Exception as e:
        # Fallback logic with keyword matching
        return {"platform": "unknown", "operation_type": "general", "confidence": 0.0}

The Game Changer: Bright Data Integration

Here's where it gets interesting. Instead of basic web scraping that breaks with bot detection, I integrated Bright Data's MCP (Model Context Protocol) for enterprise-grade data extraction:

from mcp_use.client import MCPClient
from mcp_use.adapters.langchain_adapter import LangChainAdapter
from langgraph.prebuilt import create_react_agent

async def execute_browser(state: DemoState) -> DemoState:
    action_plan = state["action_plan"]
    query = state["query"]
    platform = state["platform"]

    # Bright Data MCP configuration
    browserai_config = {
        "mcpServers": {
            "BrightData": {
                "command": "npx",
                "args": ["@brightdata/mcp"],
                "env": {
                    "API_TOKEN": os.getenv("BRIGHT_DATA_API_TOKEN"),
                    "WEB_UNLOCKER_ZONE": os.getenv("WEB_UNLOCKER_ZONE"),
                    "BROWSER_ZONE": os.getenv("BROWSER_ZONE")
                }
            }
        }
    }

    client = MCPClient.from_dict(browserai_config)
    adapter = LangChainAdapter()
    tools = await adapter.create_tools(client)

    # Create agent with Bright Data tools
    agent = create_react_agent(
        model=llm,
        tools=tools,
        prompt="""You are a web scraping expert with access to:
        - search_engine: Google/Bing/Yandex results
        - scrape_as_markdown: Bot-detection bypass
        - Structured extractors: Platform-specific scrapers
        - Browser automation: Navigate, click, type, screenshot
        """
    )

    result = await agent.ainvoke({
        "messages": [{
            "role": "user",
            "content": f"Extract API docs for: {query}\nPlatform: {platform}\nPlan: {action_plan}"
        }]
    })

    return process_extraction_result(result)

Deployment Flexibility: Two Interfaces, One Engine

Option 1: Streamlit Web Interface

Perfect for teams and interactive exploration:

import streamlit as st
import asyncio

def main():
    st.title("🤖 API Documentation Agent")

    query = st.text_area("Enter your API documentation query:")

    if st.button("Generate Documentation"):
        with st.spinner("Processing..."):
            # Real-time progress tracking
            for update in run_agent(query):
                if update["status"] == "completed":
                    st.markdown(update["results"]["final_response"])

Option 2: MCP Server for IDE Integration

This is where it gets really cool. The entire agent becomes a tool in your coding environment:

from fastmcp import FastMCP

graph = create_demo_graph()
mcp = FastMCP("API-Documentation-Agent")

@mcp.tool()
async def generate_api_docs(question: str) -> str:
    """Generate API documentation from natural language query"""

    initial_state = {
        "query": question,
        "platform": "",
        "action_plan": [],
        # ... full state initialization
    }

    result = await graph.ainvoke(initial_state)
    return result.get("final_response", "No documentation generated")

if __name__ == "__main__":
    mcp.run(transport="stdio")

Claude Desktop Integration

Add this to your claude_desktop_config.json:

{
  "mcpServers": {
    "EasyDocs": {
      "command": "/path/to/venv/bin/python",
      "args": ["/path/to/mcp_wrapper.py"]
    }
  }
}

Now you can use it directly in Claude Desktop:

@EasyDocs generate documentation for creating a Stripe payment intent

Real Output Example

Here's what you get when asking: "How to send a POST request to Bright Data's Web Scraper API"

Documentation Quality: 🟢 High Confidence (9/10)

Bright Data Web Scraper API Documentation

Quick Start

curl --request POST \
  --url https://api.brightdata.com/datasets/v3/trigger?dataset_id=YOUR_DATASET_ID \
  --header 'Authorization: Bearer YOUR_API_KEY' \
  --header 'Content-Type: application/json' \
  --data '[{"url": "https://example.com"}]'

Authentication

Type: Bearer Token
Header: Authorization: Bearer YOUR_API_KEY
Get API Key: Bright Data dashboard → Settings → API tokens

Request Parameters

dataset_id (required): Your dataset identifier
url (required): Target URL to scrape
type: Set to "discover_new" for discovery phase
limit_per_input: Limit results per input

Response Format

{
"snapshot_id": "snap_12345",
"status": "running",
"dataset_id": "gd_l1vikfnt1wgvvqz95w"
}

Error Handling

400: Invalid parameters or malformed request
401: Invalid or missing API key
429: Rate limit exceeded - implement exponential backoff

Technical Deep Dive: Why This Architecture Works

1. Structured State Management

Using TypedDict ensures type safety while maintaining the flexibility needed for dynamic workflows. Each node knows exactly what data it receives and what it should return.

2. Fault Tolerance

Every node has fallback logic. If the LLM fails, we fall back to keyword matching. If Bright Data is unavailable, we provide basic documentation templates.

3. Modular Design

Each LangGraph node is independently testable:

async def test_analyze_query():
    state = {"query": "How to authenticate with Stripe API"}
    result = await analyze_query(state)
    assert result["platform"] == "stripe"
    assert "auth" in result["operation_type"].lower()

4. Real-time Feedback

The Streamlit interface uses graph.astream() for step-by-step progress:

async for event in graph.astream(initial_state):
    for node_name, state_update in event.items():
        # Update UI with current progress
        display_progress(node_name, state_update)

Performance and Reliability

Bright Data Advantages

Bot Detection Bypass: No more CAPTCHA or IP blocking
Global Infrastructure: Consistent performance worldwide
Structured Extractors: Platform-specific scrapers for major APIs
Rate Limit Handling: Built-in retry logic and request management

Lessons Learned

1. State Design is Critical

I initially used a simple dictionary but quickly realized that TypedDict catches errors early and makes the codebase much more maintainable.

2. Fallback Everything

LLMs fail, APIs go down, networks are unreliable. Every component needs graceful degradation.

3. MCP is a Game Changer

Being able to use your AI agent directly in your coding environment eliminates context switching and makes it actually useful for daily development.

4. Bright Data's Reliability Matters

I tried basic web scraping first. Documentation sites have sophisticated bot detection, and you need enterprise infrastructure to consistently extract data.

Getting Started

# Clone and setup
git clone https://github.com/MeirKaD/EasyDocs
cd EasyDocs
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

# Configure environment
cp .env.example .env
# Add your API keys

# Run web interface
streamlit run app.py

# Or start MCP server
python mcp_wrapper.py

Key Takeaways for Developers

LangGraph makes complex AI workflows manageable - the state-based approach is much cleaner than chaining LLM calls
Structured outputs with Pydantic eliminate debugging nightmares - always use typed models for LLM responses
MCP bridges the gap between AI and daily development - tools that live in your editor get used
Enterprise infrastructure matters - Bright Data's reliability makes this production-ready
Multiple interfaces serve different use cases - web UI for exploration, MCP for integration

Connect and Contribute

This project is open source and I'm actively looking for contributors! Whether you want to add new platform support, improve the UI, or enhance the extraction logic, there's plenty to work on.

Found this useful? Let's connect and build the future of developer tooling together!

References

Project Github repo: https://github.com/MeirKaD/EasyDocs

Bright Data's MCP: https://github.com/brightdata/brightdata-mcp

What API documentation challenges are you facing? Drop a comment and let's discuss how AI agents might solve them!

DEV Community: Meir

Building an AI Browser Agent That Actually Works on Ticket Sites

What You'll See in This Article

The Tech Stack

The Two Technologies That Make This Possible

Bright Data Browser API: Real Browsers in the Cloud

Yutori n1: The "Pixels-to-Actions" Model

The 5-Stage Pipeline

Stage 1: SERP Search — Finding the Right Platforms

Stage 2: Browser Connection — Opening Real Browsers

Stage 3: n1 Agentic Browse Loop — The Heart of the Agent

Stage 4: Structured Extraction — From Text to Data

Stage 5: Merge & Rank — The Final Output

The Real-Time Streaming Architecture

What Can Go Wrong (and What I Learned)

Popup chaos

Coordinate translation bugs

Hard blocks vs. soft blocks

The 15-step limit

Extraction failures

The Result

Running It Yourself

Key Takeaways

FAQ

How I Built an AI Agent That Maps Any Company's Supply Chain in Real-Time

The Problem With Supply Chain Discovery

Architecture Overview

Step 1: Query Planning

Step 2: Web Search via Bright Data

Step 3: URL Ranking

Step 4: Scraping with Bright Data

Step 5: LLM Evidence Reflection

Step 6: Counterparty Parsing and Edge Construction

Step 7: Entity Filtering

Running on AWS Bedrock AgentCore

The Visualization: D3 Sankey

What Can Go Wrong

Results

Key Takeaways

FAQ

Running It Yourself

Building PeopleHub: An AI-Powered LinkedIn Intelligence Platform with LangGraph and Bright Data

Table of Contents

What is PeopleHub?

Architecture Overview

Tech Stack

Backend

AI/LLM

External APIs

Frontend

AI Query Parser

The Problem

The Solution: Structured Output with Gemini 2.0 Flash

Example Output

Bright Data Integration

1. Google Search API

2. LinkedIn Scraper API

3. MCP (Model Context Protocol) Integration

Multi-Tier Caching Strategy

Tier 1: Redis (Hot Cache)

Tier 2: PostgreSQL (Persistent Cache)

LangGraph Research Engine

What is LangGraph?

Research Workflow Graph

Implementation

Why LangGraph?

Database Design with Prisma

Schema Overview

Key Design Decisions

Performance Optimizations

1. Batch Database Queries

2. Parallel API Calls

3. Connection Pooling

4. Image Proxy for CORS

5. Redis for Hot Cache

Lessons Learned

1. Structured Output > Regex Parsing

2. Async by Default for External APIs

3. LangGraph for Complex Workflows

4. Cache Invalidation is Hard