DEV Community: Benny

What We Learned Building a DIY Web Scraper published

Benny — Thu, 05 Nov 2026 12:53:54 +0000

This article was originally published on the Zenrows blog.

We built a Python web scraper from scratch with Requests, BeautifulSoup, and Playwright, then tested it against three protected sites and a documented rate limit. This post walks through the build with every script and output. You'll need Python 3.10 or later and a terminal. The self-hosted part needs no API key, and a free Zenrows key comes in at the end for comparison.

We came away with four lessons.

Browser headers, retries, and random delays are enough for static pages.
Playwright handles pages that fill in their content with JavaScript.
On protected sites, Requests got 403s, and Playwright got a timeout, a challenge page, and an empty shell.
A per-IP quota, like GitHub's 60 requests an hour, stops the scraper, however carefully you set its headers.

Zenrows Fetch with mode=auto returned all three protected pages. The full code is on GitHub.

Setup

python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install requests beautifulsoup4 lxml playwright brotli
python3 -m playwright install chromium

All the scraper code goes into one file, scraper.py, built up step by step. Each test runs as its own stage.

1. Send Browser-Like Headers

import random
import requests

USER_AGENTS = [
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36",
]


def build_headers() -> dict[str, str]:
    return {
        "User-Agent": random.choice(USER_AGENTS),
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        "Accept-Language": "en-US,en;q=0.9",
        "Accept-Encoding": "gzip, deflate, br",
        "Connection": "keep-alive",
    }

By default, Requests identifies itself as a script. This version rotates three browser User-Agent strings and sends Accept, Accept-Language, Accept-Encoding, and Connection headers that match a Chrome request. Browser-like headers may help with simple filters, but they don't reproduce a full browser or reliably bypass bot detection.

2. Retry With Backoff

# continuing scraper.py
import time

RETRY_STATUS = {429, 500, 502, 503, 504}


def fetch(session: requests.Session, url: str, max_retries: int = 3) -> str | None:
    for attempt in range(max_retries):
        try:
            response = session.get(url, headers=build_headers(), timeout=15)

            if response.status_code == 200:
                return response.text

            if response.status_code not in RETRY_STATUS:
                print(f"{url} returned {response.status_code}, not retrying")
                return None

            time.sleep(2 ** attempt)

        except requests.RequestException as error:
            print(f"{url} failed with {type(error).__name__}")
            time.sleep(2 ** attempt)

    return None

fetch() retries on 429, 500, 502, 503, and 504 with exponential backoff, and the 15-second timeout keeps an unresponsive server from stalling the run. Any other status returns immediately. A 404 means the page doesn't exist, so another attempt wouldn't change the result.

3. Add Polite Delays

# continuing scraper.py

def polite_delay(base: float = 1.5, jitter: float = 1.0) -> None:
    time.sleep(base + random.uniform(0, jitter))

A burst of requests from one IP address can trigger site-specific rate limits or bot detection. Each pause is 1.5 seconds plus up to 1 second of random jitter, because a person reading a page doesn't click at fixed intervals.

4. Parse Products Into a Dataclass

Each product on the demo store is an li.product element holding a .product-name, a .product-price, and a product link. BeautifulSoup with the lxml parser handles malformed HTML better than the built-in parser.

# continuing scraper.py

from bs4 import BeautifulSoup
from dataclasses import dataclass, asdict


@dataclass
class Product:
    name: str
    price: str
    url: str


def parse_products(html: str) -> list[Product]:
    soup = BeautifulSoup(html, "lxml")
    products = []

    for card in soup.select("li.product"):
        name = card.select_one(".product-name")
        price = card.select_one(".product-price")
        link = card.select_one("a.woocommerce-LoopProduct-link")

        if not (name and price and link):
            continue

        products.append(Product(
            name=name.get_text(strip=True),
            price=price.get_text(strip=True),
            url=link["href"],
        ))

    return products

5. Scrape Three Pages to CSV

# continuing scraper.py

import csv

BASE_URL = "https://www.scrapingcourse.com/ecommerce/page/{}/"


def scrape(pages: int = 3) -> list[Product]:
    all_products = []

    with requests.Session() as session:
        for page in range(1, pages + 1):
            html = fetch(session, BASE_URL.format(page))

            if html is None:
                continue

            products = parse_products(html)
            print(f"page {page}: {len(products)} products")
            all_products.extend(products)
            polite_delay()

    return all_products


def run_static() -> None:
    products = scrape()

    with open("products.csv", "w", newline="", encoding="utf-8") as file:
        writer = csv.DictWriter(file, fieldnames=["name", "price", "url"])
        writer.writeheader()
        writer.writerows(asdict(p) for p in products)

    print(f"saved {len(products)} products")

# keep this block at the bottom of scraper.py
import sys

if __name__ == "__main__":
    # run the stage named on the command line, for example: python3 scraper.py static
    stage = sys.argv[1] if len(sys.argv) > 1 else "static"
    globals()[f"run_{stage}"]()

The main block runs the stage named on the command line. Run the static stage.

python3 scraper.py static

page 1: 16 products
page 2: 16 products
page 3: 16 products
saved 48 products

It scraped 48 products from the demo store across three pages into products.csv.

6. Handle JavaScript-Rendered Pages

A static fetch of a client-rendered page returns empty fields.

# continuing scraper.py, above the main block

def run_dynamic() -> None:
    url = "https://www.scrapingcourse.com/javascript-rendering"

    with requests.Session() as session:
        html = fetch(session, url)

    if html is None:
        raise RuntimeError(f"Could not fetch {url}")

    soup = BeautifulSoup(html, "lxml")

    for card in soup.select(".product-item")[:3]:
        name = card.select_one("[data-testid=product-name]")
        price = card.select_one("[data-testid=product-price]")
        link = card.select_one("a.product-link")
        print(repr(name.get_text(strip=True)), repr(price.get_text(strip=True)), repr(link["href"]))

python3 scraper.py dynamic

'' '' ''
'' '' ''
'' '' ''

The parser finds the cards, but every field is empty. Add this line to the end of run_dynamic(), outside the loop, to print one card's markup.

print(soup.select(".product-item")[0].prettify()[:400])

<div class="product-item flex flex-col items-center rounded-lg" data-content="product-item" data-testid="product-item" itemscope="" itemtype="http://schema.org/Product">
 <a class="product-link" data-content="product-link" data-testid="product-link" href="">
  <img alt="" class="rounded-lg border shadow-md object-cover" data-content="product-image" data-testid="product-image" decoding="async" fetc

The server sends the page structure with empty href, src, name, and price values, and JavaScript fills them in after the page loads.

Playwright renders the page in headless Chromium and waits for a selector before reading it, so parsing doesn't happen too early. The finally block closes the browser so a failure doesn't leave Chromium processes running.

# continuing scraper.py, above the main block

from playwright.sync_api import sync_playwright


def fetch_rendered(url: str, wait_for: str, timeout: int = 30000) -> str | None:
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page(user_agent=random.choice(USER_AGENTS))

        try:
            page.goto(url, timeout=timeout)
            page.wait_for_selector(wait_for, timeout=timeout)
            return page.content()
        except Exception as error:
            print(f"{url} failed to render: {type(error).__name__}")
            return None
        finally:
            browser.close()

# continuing scraper.py, above the main block

def parse_rendered(html: str) -> list[Product]:
    soup = BeautifulSoup(html, "lxml")
    products = []

    for card in soup.select(".product-item"):
        name = card.select_one("[data-testid=product-name]")
        price = card.select_one("[data-testid=product-price]")
        link = card.select_one("a.product-link")

        if not (name and price and link):
            continue

        products.append(Product(
            name=name.get_text(strip=True),
            price=price.get_text(strip=True),
            url=link["href"],
        ))

    return products


def run_rendered() -> None:
    url = "https://www.scrapingcourse.com/javascript-rendering"
    html = fetch_rendered(url, wait_for="[data-testid=product-name]:not(:empty)")

    if html:
        for product in parse_rendered(html)[:3]:
            print(product)

python3 scraper.py rendered

Product(name='Chaz Kangeroo Hoodie', price='$52', url='https://scrapingcourse.com/ecommerce/product/chaz-kangeroo-hoodie')
Product(name='Teton Pullover Hoodie', price='$70', url='https://scrapingcourse.com/ecommerce/product/teton-pullover-hoodie')
Product(name='Bruno Compete Hoodie', price='$63', url='https://scrapingcourse.com/ecommerce/product/bruno-compete-hoodie')

7. Test Protected Pages With Requests

# continuing scraper.py, above the main block

TARGETS = [
    "https://dexscreener.com/solana",
    "https://www.sofascore.com/football",
    "https://www.g2.com/categories/crm",
]


def run_protected() -> None:
    with requests.Session() as session:
        for url in TARGETS:
            try:
                r = session.get(url, headers=build_headers(), timeout=15)
            except requests.RequestException as error:
                print(f"\n{url}\nfailed: {type(error).__name__}")
                continue

            print(f"\n{url}")
            print("status:", r.status_code, "| bytes:", len(r.content))
            print("server:", r.headers.get("server"))
            print("cf-mitigated:", r.headers.get("cf-mitigated"))
            print("x-datadome:", r.headers.get("x-datadome"))
            print("cookies:", list(r.cookies.keys()))
            print("title:", BeautifulSoup(r.text, "lxml").title)

python3 scraper.py protected

https://dexscreener.com/solana
status: 403 | bytes: 5796
server: cloudflare
cf-mitigated: challenge
x-datadome: None
cookies: ['__cf_bm']
title: <title>Just a moment...</title>

https://www.sofascore.com/football
status: 403 | bytes: 48
server: Varnish
cf-mitigated: None
x-datadome: None
cookies: []
title: None

https://www.g2.com/categories/crm
status: 403 | bytes: 1707
server: cloudflare
cf-mitigated: None
x-datadome: protected
cookies: ['datadome', '__cf_bm']
title: <title>g2.com</title>

All three returned 403, and each was blocked differently. Dexscreener's cf-mitigated: challenge header and "Just a moment..." title mark Cloudflare's JavaScript challenge page. Sofascore blocked the request at its Varnish edge with 48 bytes, no cookies, and no HTML. G2's x-datadome: protected header and datadome cookie show DataDome, and the response is a placeholder page with nothing to parse. Responses may differ by IP, session, time, and target configuration.

8. Hit a Documented Rate Limit

# continuing scraper.py, above the main block

RATE_LIMIT_URL = "https://api.github.com/users/Python"


def run_ratelimit() -> None:
    with requests.Session() as session:
        for i in range(1, 71):
            r = session.get(RATE_LIMIT_URL, headers=build_headers(), timeout=15)
            remaining = r.headers.get("x-ratelimit-remaining")

            if i == 1 or i % 10 == 0 or r.status_code != 200:
                print(f"request {i}: {r.status_code} | remaining: {remaining}")

            if r.status_code != 200:
                print("limit:", r.headers.get("x-ratelimit-limit"))
                print("reset:", r.headers.get("x-ratelimit-reset"))
                break

python3 scraper.py ratelimit

request 1: 200 | remaining: 58
request 10: 200 | remaining: 49
request 20: 200 | remaining: 39
request 30: 200 | remaining: 29
request 40: 200 | remaining: 19
request 50: 200 | remaining: 9
request 60: 403 | remaining: 0
limit: 60
reset: 1792572851

GitHub's API documents a limit of 60 requests an hour for unauthenticated clients, which makes it a clean test even though it's an API. The x-ratelimit-remaining header counted down from the first request, and the 60th request returned 403. The reset value is a Unix timestamp for when the window reopens. The loop leaves out the delay from step 3, and adding it back wouldn't change the result, because this limit is a per-IP quota.

9. Test Protected Pages With Playwright

# continuing scraper.py, above the main block

def run_browser() -> None:
    for url in TARGETS:
        html = fetch_rendered(url, wait_for="body")

        if html is None:
            print(f"\n{url}\nno content returned")
            continue

        soup = BeautifulSoup(html, "lxml")
        print(f"\n{url}")
        print("bytes:", len(html))
        print("title:", soup.title)

python3 scraper.py browser

https://dexscreener.com/solana failed to render: TimeoutError

https://dexscreener.com/solana
no content returned

https://www.sofascore.com/football
bytes: 5563
title: <title>Sofascore</title>

https://www.g2.com/categories/crm
bytes: 2609
title: <title>g2.com</title>

Dexscreener timed out, Sofascore returned a "Verifying you are human" page, and G2 returned a 2,609-byte empty shell.

10. Fetch the Same Pages Through Zenrows

Create a free API key on the signup page. The free plan includes 5,000 credits a month with no credit card. Install the two dependencies, add your key to .env, and add .env to .gitignore.

pip3 install requests python-dotenv

ZENROWS_API_KEY=your_zenrows_api_key

Create zenrows_fetch.py.

import os

import requests
from bs4 import BeautifulSoup
from dotenv import load_dotenv

load_dotenv()

TARGETS = [
    "https://dexscreener.com/solana",
    "https://www.sofascore.com/football",
    "https://www.g2.com/categories/crm",
]


def fetch_zenrows(url: str) -> str | None:
    response = requests.get(
        "https://api.zenrows.com/v1/",
        params={
            "apikey": os.getenv("ZENROWS_API_KEY"),
            "url": url,
            "mode": "auto",
        },
        timeout=180,
    )

    if response.status_code == 200:
        return response.text

    print(f"{url} returned {response.status_code}: {response.text[:200]}")
    return None


for url in TARGETS:
    html = fetch_zenrows(url)

    if html:
        soup = BeautifulSoup(html, "lxml")
        print(f"\n{url}")
        print("bytes:", len(html))
        print("title:", soup.title)

python3 zenrows_fetch.py

https://dexscreener.com/solana
bytes: 2132120
title: <title data-rh="true">Solana DEX Screener</title>

https://www.sofascore.com/football
bytes: 183264
title: <title data-next-head="">Football Live Scores, Fixtures &amp; Odds | Sofascore | Championship League, Premier League</title>

https://www.g2.com/categories/crm
bytes: 827211
title: <title>Best CRM Software: User Reviews from October 2026</title>

All three requests returned 200 with the full page.

Results

Target	Protection	Requests	Playwright	Zenrows Fetch (`mode=auto`)
Dexscreener	Cloudflare challenge	403, "Just a moment..." page	Timed out	200, full page (2,132,120 bytes)
Sofascore	Varnish edge block, then a JavaScript challenge	403, 48 bytes	"Verifying you are human" page	200, full page (183,264 bytes)
G2	DataDome	403, placeholder page	2,609-byte empty shell	200, full page (827,211 bytes)

mode=auto starts with the lowest-cost configuration that works and escalates to a heavier one only when a page needs it. You're billed once, for the configuration that succeeds. It also handles IP-based restrictions like the per-IP quota from the GitHub test. The parser stays the same, and only the fetch layer changes.

Wrapping Up

A self-hosted scraper works well for open pages with bounded volume. Protected targets need stealth plugins, residential proxies, CAPTCHA solvers, or anti-detect browsers, and each one is another component to maintain whenever a protection changes. At thousands of pages, the one-browser-per-call pattern also needs bounded concurrency, browser lifecycle management, crash recovery, and capacity testing. Keep self-hosting when target access is open and bounded, and use a managed layer when protected access and ongoing maintenance are work you don't want to own.

What's Next

Try Zenrows Fetch on your own targets with 5,000 free credits a month, no card required.
Read about Adaptive Stealth Mode.
Send a whole URL list to Batch, which handles the queue, retries, and concurrency.
See what each approach costs to operate in web scraping costs.
Check the web scraping API benchmark, where Zenrows had the best overall success rate across four protected targets.

We Gave a Local LLM Live Web Access. Here’s How

Benny — Thu, 05 Nov 2026 06:47:47 +0000

This article was originally published on the Zenrows blog.

We gave a local LLM live web access, and this post shows how. You'll connect a qwen3 model in Ollama to Zenrows MCP, first with no code through the ollmcp bridge and then with a short Python agent that is its own MCP client. You'll need Ollama, a tool-capable model, Node.js, and Python 3.11 or later. A Zenrows API key is optional.

Ollama publishes web_search and web_fetch as hosted APIs that you call with an Ollama API key, and it has no MCP client of its own. In our test runs, the local qwen3 agent read a Cloudflare-fronted Indeed search and cleared an anti-bot challenge page through Zenrows MCP. A plain requests.get() to the same challenge page returned a 403. The code is on GitHub.

How It Works

You ask a question, Ollama picks a tool, Zenrows makes the requests, and the target site returns the pages. The model never contacts the site. The agent calls the extract tool, Zenrows fetches the page through Fetch with protected access, JavaScript rendering, and proxy rotation in one API call, and the model gets the page as structured JSON fields.

Zenrows MCP has two transports. The remote server runs over Streamable HTTP and authenticates with your API key as a Bearer token. The local server runs as a STDIO subprocess through the @zenrows/mcp npm package with your key in an environment variable. This tutorial uses the local one, since it runs next to Ollama on your machine.

These tools matter for this setup.

scrape returns a page as Markdown by default, with HTML, PDF, and screenshot output also available.
extract returns structured JSON, which matters for local models with small context windows.
browser_* tools open a cloud browser session for tasks that span more than one request.
crawl_* tools collect the URLs behind one start page.
batch_* tools submit a list of URLs as one background job.

1. Install Ollama and a Tool-Capable Model

brew install ollama
brew services start ollama

ollama pull qwen3
ollama show qwen3:latest

Ollama registers the model as qwen3:latest. The ollama show output lists tools under Capabilities, which confirms the model can call the Zenrows tools.

  Model
    architecture        qwen3
    parameters          8.2B
    context length      40960
    embedding length    4096
    quantization        Q4_K_M

  Capabilities
    completion
    tools
    thinking

2. Set Up the Project

python3 -m venv .venv
source .venv/bin/activate

pip install python-dotenv requests

Without a key, the local server creates a Free account with 5,000 credits a month on its first tool call. To use your own account, create one and save the key in .env.

ZENROWS_API_KEY=your_key_here

3. Install the ollmcp Bridge

ollmcp is an open-source Python bridge that connects MCP servers to a local Ollama model over STDIO, SSE, and Streamable HTTP. It also installs the ollama and mcp packages the agent imports later. This tutorial was tested with ollmcp 0.35.0 on Python 3.14.

pip install ollmcp

4. Configure the Server and Chat With No Code

Create zenrows-mcp.json.

{
  "mcpServers": {
    "zenrows": {
      "command": "sh",
      "args": ["-c", "set -a && . ./.env && set +a && exec npx -y @zenrows/mcp"]
    }
  }
}

The sh -c command sources .env into the environment before starting npx, so the JSON holds no secret and can be committed. Add .env to .gitignore, and run ollmcp from this directory because the .env path is relative.

ollmcp --servers-json zenrows-mcp.json --model qwen3:latest

On October 21, 2026, the bridge connected and listed the Zenrows tools (output trimmed).

Found server in config: zenrows
Connecting to server: zenrows
Successfully connected to zenrows with 62 tool(s) and 3 prompt(s) and 1 resource(s)/template(s)
╭──────────────────────────── 🔧 Available Tools ────────────────────────────╮
│ ✓ zenrows.scrape                         ✓ zenrows.extract                 │
│ ✓ zenrows.batch_create                   ✓ zenrows.batch_status            │
...
│ ✓ zenrows.browser_switch_tab             ✓ zenrows.browser_batch           │
╰─────────────────────────── 62/62 tools enabled ────────────────────────────╯
╭───────────────────────────────────────────────╮
│ Current model: qwen3:latest (ollama)          │
│ Capabilities: 🔧 Tools │ 💭 Thinking (medium) │
╰───────────────────────────────────────────────╯

Every enabled tool's schema goes into the model's context, so type /tools and keep scrape and extract switched on for a small local model. /human-in-the-loop asks you to approve each tool call before it runs.

5. Build the Agent in Python

The script starts the Zenrows MCP server with npx, so it doesn't need ollmcp or zenrows-mcp.json to run. Create agent.py. The pinned versions are in the repo's requirements.txt.

from dotenv import load_dotenv
load_dotenv()

import asyncio
import json
import os
import ollama
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client

SERVER = StdioServerParameters(
    command="npx",
    args=["-y", "@zenrows/mcp"],
    env=dict(os.environ),
)
MODEL = "qwen3:latest"
# give the model enough context to read the whole tool result
OPTIONS = {"num_ctx": 16384}
QUESTION = "Extract the page heading from https://www.scrapingcourse.com/antibot-challenge."
ALLOWED_ARGS = {"url"}

async def main():
    async with stdio_client(SERVER) as (read, write):
        async with ClientSession(read, write) as session:
            await session.initialize()

            listed = await session.list_tools()
            tools = [
                {"type": "function", "function": {
                    "name": t.name,
                    "description": t.description,
                    "parameters": t.input_schema,
                }}
                for t in listed.tools if t.name == "extract"
            ]

            messages = [{"role": "user", "content": QUESTION}]
            reply = ollama.chat(model=MODEL, messages=messages, tools=tools, options=OPTIONS)
            messages.append(reply["message"])

            for call in reply["message"].get("tool_calls") or []:
                args = {k: v for k, v in call.function.arguments.items() if k in ALLOWED_ARGS}
                print("calling", call.function.name, args)
                result = await session.call_tool(call.function.name, args)
                text = result.content[0].text
                print("result:", text[:300])
                # pass the model only the parsed fields from the response
                data = json.loads(text).get("data")
                messages.append({"role": "tool", "content": json.dumps(data)})

            print(ollama.chat(model=MODEL, messages=messages, tools=tools, options=OPTIONS)["message"]["content"])

asyncio.run(main())

Three details matter on a local model.

OPTIONS sets the context window. Ollama uses 4,096 tokens by default, and the Indeed prompt later came to about 6,900 tokens. Without num_ctx, Ollama truncated that prompt and the model answered from part of the listings.
ALLOWED_ARGS filters the model's arguments and blocks anything the model invents, so only the URL reaches the server. Zenrows then turns on JavaScript rendering and premium proxies only when the page needs them.
The extract response wraps the parsed fields in a data object next to a few status flags. The script hands the model only that data object, which keeps the tool message small.

The test target is the Scrapingcourse antibot challenge page, which reports its own verdict.

python3 agent.py

Using existing API key from env (secrets dir: /Users/user/.zenrows)
Zenrows MCP server running on stdio (secrets dir: /Users/user/.zenrows)
calling extract {'url': 'https://www.scrapingcourse.com/antibot-challenge'}
result: {"ok":true,"mode":"auto","empty":false,"data":{"challenge_result":{"message":"You bypassed the Antibot challenge! :D","status":"bypassed"},"page":{"challenge_type":"antibot","title":"Antibot Challenge"}}}
The page heading (h1) from the URL is:

**Antibot Challenge**

This is the title extracted from the webpage, which also happens to be the name of the antibot challenge page.

For comparison, without_zenrows.py sends a plain request to the same page.

import requests

r = requests.get("https://www.scrapingcourse.com/antibot-challenge")
print(r.status_code, len(r.text))

python3 without_zenrows.py

403 5517

The request was blocked with a 403.

6. Read a Live Cloudflare-Protected Site

Confirm what sits in front of Indeed.

curl -sI https://www.indeed.com | grep -iE "^(server|date):"

date: Wed, 21 Oct 2026 08:15:02 GMT
server: cloudflare

Create cloudflare_site.py with the same code as agent.py and change the QUESTION line.

QUESTION = "List the company names and job titles from https://www.indeed.com/jobs?q=data+analyst&l=Austin%2C+TX."

python3 cloudflare_site.py

Using existing API key from env (secrets dir: /Users/user/.zenrows)
Zenrows MCP server running on stdio (secrets dir: /Users/user/.zenrows)
calling extract {'url': 'https://www.indeed.com/jobs?q=data+analyst&l=Austin%2C+TX'}
result: {"ok":true,"mode":"auto","empty":false,"data":{"job_listings":[{"benefits":null,"company_logo_url":null,"company_name":"Paywall Bypass","detail_url":"/rc/clk?jk=1d9d5ad6081ee5e0&bb=0Jmy1L_Rym9Qr_CYZLvqO-iSA0EMSX-Z6TVgVMeYzV9VdW2jJoxc32S2GChQtdu5Qm9Zo3b5B6zKmWui0wMrKaTa7wukcUUmE83DPPtvU-K-2B_k_liBF0N
Here are the company names and job titles from the Indeed search results in Austin, TX:

1. **Paywall Bypass** - Data Analyst  
2. **Visa U.S.A. Inc.** - Data & Insights Analyst  
3. **Texas Department of Transportation** - Data Analyst IV  
4. **Austin Community College** - Analyst, Institutional Research (Hourly)  
5. **Austin Community College** - Analyst, Human Resources Data  
6. **University of Texas at Austin** - Data Analyst, College of Liberal Arts  
7. **Steck Systems** - Data Analyst  
8. **Apple** - Business Insights Analyst, Apple Store Online Operations  
9. **Google** - Business Data Analyst II, Google Global Infrastructure, Strategy and Operations  
10. **Google** - Business Data Analyst II, Google Global Infrastructure, Strategy and Operations  
11. **University of Texas at Austin** - Senior Data Analyst  
12. **The University of St. Augustine for Health...** - Institutional Research Analyst  
13. **Ericsson** - Data Analyst Strategy and Operations  
14. **Texas Department of Transportation** - Business Analyst II  
15. **SHI International Corporation** - Associate Analytics Engineer - Data & Visualization  
16. **Health & Human Services Comm** - Data Analyst V  

Let me know if you'd like additional details like locations or salary information!

The extraction returned ok: true with 16 job listings and no challenge response, and the model listed every company and job title on the page.

Choosing an MCP Server

Here is how three MCP servers compared, as documented on October 21, 2026.

	Zenrows MCP	Firecrawl MCP	Playwright MCP
How it runs	Cloud or local	Cloud or local	On your browser
Auth	OAuth sign-in or API key as a Bearer token, or `ZENROWS_API_KEY` environment variable	OAuth sign-in or API key as a Bearer token, or `FIRECRAWL_API_KEY` environment variable	None and runs locally
Protected access	One integration combines fast scraping and full browser automation, with anti-bot bypass on every tool	Their MCP documentation doesn't mention anti-bot handling or proxy rotation	Runs from your own IP with no proxy rotation
Transport	Hosted Streamable HTTP, or local STDIO via the command in the Zenrows MCP documentation	Hosted Streamable HTTP at `mcp.firecrawl.dev`, or local via `npx -y firecrawl-mcp`	Local STDIO via `npx @playwright/mcp`

Wrapping Up

One MCP configuration gave a local Ollama agent access to live pages, including Cloudflare-protected ones, while Zenrows handled access, rendering, and proxies for every request. For a task spanning several pages, ask in plain English and the agent can use the crawl and batch tools.

What's Next

Create a free Zenrows account with 5,000 credits every month.
Use the same server in Claude, Cursor, and other cloud clients.
Run the same requests from scripts and CI jobs with the Zenrows CLI.
Let an agent with no account pay per request over x402 or MPP at agents.zenrows.com.

I built the assistant I clicked on as a kid. It haunts me now!

Benny — Wed, 28 Oct 2026 16:23:46 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine

What I Built

I was five or six the first time a piece of software looked back at me.

It lived on the family PC in our dining room. I was supposed to be learning CorelDRAW, but when I got bored, I played around.

Its name was Clippy.

Clippy sat on its little mat, moving its eyes around while I worked. I clicked it on purpose just to make it dance. My mum liked it too, right up until she had serious work to do. Then she'd push it aside.

That sums up the original. It was charming, and it knew nothing. It could tell you "it looks like you're writing a letter", but it couldn't actually see what you were doing.

So I built Paperclip: a doodled bedsheet ghost that lives in the corner of my Mac and actually watches.

Every 5 seconds, it checks which window I'm in.
When the window changes, a vision model running on my laptop looks at the screen and works out what I'm doing.
If I'm stuck on the same error for 20 minutes, it offers help. Now and then, it suggests something about whatever's on screen.
After midnight, it gets possessed: red ink, limp arms, one pupil drifting on its own, and lines like Where are you going to?.
Hover over it, and it runs from your cursor. In possessed mode, it stops running and follows you.
You can't kill it. Force-quit it, and it comes back, remembering what it was watching.

The name is the joke. The ghost is my own drawing, not a paperclip. It's the ghost of the old assistant, back from the dead with eyes this time.

Demo

The video below shows all of Paperclip's effects

Code

Bennykillua / paperclip-ghost

A ghost that haunts your Mac. Watches your screen with a local vision model (Gemma via Ollama), comes back when you kill it (Temporal), and nothing leaves your machine.

Paperclip

A desktop ghost for macOS that watches what you're doing and butts in. It runs a local vision model through Ollama, so nothing about your screen leaves your machine.

What you need

A Mac
Node.js 20 or newer
Homebrew (setup offers to install the rest with it)

Setup installs the Temporal CLI and Ollama if they're missing, with your permission.

Install and use

Four steps. Setup does the rest.

git clone https://github.com/Bennykillua/paperclip-ghost.git
cd paperclip-ghost
npm run setup              # checks the tools, asks for a model, builds, starts the agents
npm run ghost:test -- all  # see every effect once

Settings live in .env. .env.example lists them with notes. Setup writes VISION_MODEL for you, and Sentry stays off until you add SENTRY_DSN.

The command-line tools are listed in Brewfile. brew bundle installs them.

Then give macOS the two permissions in Permissions and run npm run ghost:start…

View on GitHub

How I Built It

The stack: Gemma 4 (E2B) through Ollama for vision, Temporal to keep the ghost alive, Sentry to trace each glance, and Electron for the window. It's all TypeScript, and it all runs locally.

The fastest model wasn't the right one

I tested four open vision models on my M5 Pro (24 GB). My first test was accidentally rigged. I sent the same screenshot three times, so the later runs did little, and each model produced a different amount of text. I threw that test out and reran it fairly: a fresh screenshot each run and the same output limit for every model.

Model	Avg time	Tokens/s
moondream	0.50s	172
gemma4:e2b	2.69s	77
qwen2.5vl:3b	4.73s	57

Moondream won on speed and lost on everything else. Its description of my screen was "a code editor window with a black background". A ghost can't haunt you with that. Gemma read actual file names and what I was doing, so Gemma won.

The biggest speed win came from asking for less. Instead of a paragraph, the model now returns a small JSON object: {app, activity, detail, notable}. Output dropped from about 131 tokens to about 54. Across 187 real calls, the vision step now takes 1.09s at the median and 1.26s at p95.

One glance takes about 1.5 seconds and costs $0

Step	Typical time
Read the active window	190–325ms
Screenshot	170ms
Resize to 1280px	~50ms
Gemma looks at it	~1,090ms

Even so, it doesn't look every 5 seconds. Reading the window title is cheap and exact, so it does that on every tick. It only calls the model when the window changes, or every 45 seconds otherwise.

A ghost you can't kill

Each step of a glance (read the window, take a screenshot, call the model, decide, speak) is a Temporal activity with its own timeout and retries. The last 30 minutes of what the ghost has seen live in the workflow's state, so they survive a crash. Every hour the workflow starts afresh and carries its memory over, so its history doesn't grow forever. Then launchd keeps all three parts running: the Temporal server, the worker, and the ghost window.

const { describeScreen } = proxyActivities<typeof activities>({
  startToCloseTimeout: '30s',
  retry: { maximumAttempts: 4, initialInterval: '1s', backoffCoefficient: 2 },
});

I tested it:

kill -9 in the middle of a vision call. launchd restarted the worker. Temporal retried the same call as attempt 2, which finished in 567ms. Memory went from 68 to 73 observations, with the oldest one unchanged.
Restarting the Temporal server. 238 observations before, 238 after.
Pointing it at a dead Ollama. Every call made exactly 4 attempts, then gave up cleanly.

After a reinstall, the worker started before the Temporal server was ready and crashed. launchd brought it back, and it carried on from the same memory. Nobody was watching. That's the point.

Watching the watcher with Sentry

Each glance is one invoke_agent span in Sentry, with every step nested under it. The vision call is a gen_ai.chat span with the model name, input and output tokens (on average 499 in and 54 out), and a cost of 0.

The scariest thing I found all week wasn't the ghost. Sentry's SDK records AI inputs and outputs by default. For most apps, that's handy for debugging. For this one, it would have sent a description of my screen to a server every few seconds, which is exactly what I built Paperclip not to do. So it's switched off, and everything else gets stripped before sending:

Sentry.init({
  dsn,
  defaultIntegrations: false,
  dataCollection: { genAI: { inputs: false, outputs: false } },
  maxBreadcrumbs: 0,
  beforeSend: (event) => scrub(event),
  beforeSendTransaction: (event) => scrub(event),
});

Sentry sees how long the ghost took to look, and nothing about what it saw.

Why Does Open Innovation Matter?

Honestly, an AI that watches my screen makes me uneasy. It also excites me. I'm only comfortable running it because it never leaves my laptop.

The closed version of this project streams your screen live to someone else's server. The open version is:

Private: the screenshots, window titles, and model replies stay on my Mac. Even the telemetry is timings only.
Free to run: each glance costs $0, however many times it looks.
Swappable: I tried four models in one evening by changing one line. If a better small vision model comes out next month, Paperclip gets better that day.
Offline: no internet needed. I even bundled the font so the window never calls out.
Unkillable: Clippy disappeared because someone else decided it should. This one lives in a repo anyone can fork.

The Spirit of It

The first time it spoke to me on its own, it was a little scary, and I liked it. That's roughly how I felt about Clippy at six.

When I showed it to a friend, they laughed and said it reminded them of Clippy, and that the jokes were funny. That was the whole goal.

My mum used to push Clippy out of the way when she had real work to do. You can push this one away too. It just comes back.

Prize Categories

Best Use of Gemma: Gemma 4 (E2B) is the eyes, picked over three other open models on speed and accuracy.
Best Use of Temporal: Every glance is a durable workflow that survives kill -9, server restarts, and a dead model server.
Best Use of Sentry Agent Tracing: invoke_agent and gen_ai.chat spans with tokens, latency, and $0 cost, and no content.

I built this solo

I'm not getting up: a voice remote for YouTube that runs fully offline (and understands Pidgin)

Benny — Sat, 24 Oct 2026 22:25:12 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

What I Built

Everyone loves a good alone time. It could be that moment where you're lying down, the show is playing, someone says something important, and you miss it. The laptop is across the room. The remote doesn't exist because it's a laptop.

So I built Couch Commander. I say "Hey Jarvis", a little pixel sloth pops up in the corner of the screen, and I tell it what I want, the way I'd actually say it:

"abeg go back small" → rewinds 15 seconds
"I no hear wetin she talk" → rewinds 10 seconds
"make am louder" → volume up
"skip this one" → next video

It controls YouTube in Chrome on my Mac. No remote, no keyboard, no getting up. Everything runs on my laptop: no internet, no account, no API key.

Demo

Both runs in the video are real recordings at normal speed, not edited.

Code

Bennykillua / couch-commander

Control YouTube from the couch with your voice. Wake word + Whisper + Gemma 4, running fully offline on a Mac. Understands Pidgin too. 🦥

Couch Commander 🦥

Control YouTube from the couch with your voice. No remote, no getting up.

Say "Hey Jarvis", wait for the chime, then say what you want, in English or Pidgin:

"abeg go back small" → rewinds 15s
"I no hear wetin she talk" → rewinds 10s
"make am louder" → volume up
"skip this one" → next video

A little pixel sloth pops up in the corner to show it heard you.

Why it runs fully on my Mac

An always-on mic in your living room only makes sense if nothing leaves the room. Every piece runs locally:

Step	Open-source piece
Wake word	openWakeWord ("hey jarvis")
Knowing when you stopped talking	webrtcvad
Speech to text	Whisper (mlx-whisper, small)
Understanding the command	Gemma 4 E2B via Ollama
Controlling YouTube	AppleScript → Chrome

No account, no API key, no cloud.

How well the brain works

I tested 30 phrases…

View on GitHub

How I Built It

Five open-source pieces are chained together. Each one does one job:

Step	What does it	Why
Wake word	openWakeWord ("hey jarvis")	So it only listens when I call it, not to the whole movie
End of speech	webrtcvad	Stops recording when I stop talking
Speech to text	Whisper (mlx-whisper, small)	Fast on Apple Silicon
Understanding	Gemma 4 E2B via Ollama	Turns my sentence into one command
Action	AppleScript → Chrome	Controls the YouTube player directly

The brain is the interesting part. Gemma gets my sentence and has to return strict JSON, one command from a fixed list or none:

{"command": "rewind", "value": 15}

I set the temperature to 0, used Ollama's JSON mode, and added Pidgin examples to the prompt. If the answer isn't valid, it falls back to none. That rule matters more than anything else in this project. Pausing your movie for no reason is worse than doing nothing.

The numbers

I tested 30 phrases on two local models. 22 of them were phrases the model had never seen as examples, including Pidgin I made up on the spot ("carry am go front small", "I wan hear am well well").

Model	Unseen phrases	Overall	Avg. time
gemma4:e2b	20/22 (91%)	28/30	~190 ms
gemma3:4b	18/22 (82%)	26/30	~330 ms

Two things surprised me:

Thinking mode was killing the speed. My first run of Gemma 4 averaged 1,489 ms per command. Turning off thinking ("think": false) brought it to ~190 ms: 8x faster, with no drop in accuracy. For a remote, nobody needs the model to reason. They need it to answer.

How a model fails matters more than how often. Gemma 4's two misses both returned none, so it just didn't act. Gemma 3's misses were actions: it paused the video when I said "tell me a joke", and when I said "make we go back two minutes" it went forward two minutes. That's the difference between a tool you keep and one you uninstall.

Things that broke

I started with the Apple TV app. AppleScript keystrokes didn't reach the player, so I switched to YouTube in Chrome, where I can run JavaScript directly on the <video> element. Much more reliable. You need to turn on View → Developer → Allow JavaScript from Apple Events in Chrome.
Whisper kept transcribing my commands as "" or "if you do". The audio was raw 16-bit numbers, and Whisper wanted floats between -1 and 1. One line fixed it.
Most of the delay wasn't AI at all. It was a fixed 4-second recording window. Wake word took 0.13s, Whisper 0.3s, Gemma 0.25s, and I was waiting 4 seconds for a timer. Voice activity detection fixed that.

Why Does Open Innovation Matter?

Because this thing has a microphone that's always on, in my living room.

With a closed API, every "Hey Jarvis" would send audio from my house to someone's server. I wouldn't use that, and I wouldn't ask anyone else to. Here, the audio never leaves my Mac. You can unplug the router, and it still works.

Open models also let me measure and swap. I ran the same 30 phrases through two models, compared accuracy and speed, and switched off a feature (thinking) that was slowing me down. That's how I found the 8x. With a closed API, I'd get whatever latency it gives me.

And it costs nothing to run. There's no per-request bill for telling my laptop to rewind.

The Lazy Payoff

I used it on my own couch while building it this week. The honest version: it works best from a few meters away. The laptop mic struggles if I'm far across a noisy room, and Pidgin is harder for the small Whisper model than plain English. My next step is using AirPods as the mic, since they're right by my mouth.

But the core loop works: "Hey Jarvis", sloth pops up, "abeg go back small", it goes back. I didn't move.

Prize Categories

Best Use of Gemma: Gemma 4 E2B is the brain that turns English and Pidgin into commands, running locally through Ollama. I benchmarked it against Gemma 3 4B, and tuning it (thinking off) made it 8x faster.

Built with openWakeWord, webrtcvad, Whisper, Gemma 4, and Ollama. MIT licensed.

The sloth is original pixel art for this project. 🦥

Leadfinder: evidence-backed WhatsApp pitches for Lagos freelancers, on Gemma for $0.32 per 1,000 leads

Benny — Sun, 18 Oct 2026 19:50:12 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

Cold pitches in Lagos fail because they say nothing

Freelancers in Lagos who sell to local businesses (video editors, web designers, social media managers) find most of their clients one WhatsApp message at a time. Most of those messages look the same:

"Hi, I edit videos, let me know if you need anything."

They get ignored.

A pitch that gets a reply needs a reason something specific and true about that business, that the owner can check in ten seconds. Finding that reason by hand means opening the Maps listing, the website, the reviews, and thinking.

Do that for 20 businesses and your evening is gone.

Leadfinder does the looking

Leadfinder is one search box:

Type "event centers in Lekki" and it searches Google Maps.
Type a website or business name, and it audits that one business.

Every business gets a score from 1 to 5, a list of evidence (problems, things that passed, things it couldn't check), and a top reason to pitch. You save the good ones to a board and move them through a series of tasks till it is won or lost.

Open a saved lead, and Gemma reads the website and the business's lowest-rated Google reviews, then writes a 50–125 word pitch. The pitch opens with one problem the owner can verify, says why it matters to their customers, and ends with one question. You can then use the Open in WhatsApp to send a message.

You can also move leads to Pitched sets and set a follow-up reminder for three days later.

Who pays? Freelancers who sell services to local businesses and are tired of sending messages nobody answers.

Demo

Live app: https://leadfinder-9aep.onrender.com

You can create an account and test or..

Just use this email and password. Click to reveal more..

Email: TestJudge@gmail.com
Password: 1234567890

Here is a demo video.

Code

Bennykillua / leadfinder

Find local businesses, see what's weak about them online, and send evidence-based WhatsApp pitches. Gemma does the thinking.

leadfinder

A lead finder for freelancers in Lagos who sell to local businesses: video editors, web designers, social media managers. Search for businesses, see what's weak about each one online, save the good ones to a board, and work them from first pitch to won, one WhatsApp message at a time.

Gemma (open-weight) does the thinking: it reads each business's website, judges how good a lead it is for your offer, and writes a short pitch where every claim comes from evidence the business owner can check.

What it does

Find: one text box. Type "salons in Yaba" to search Google Maps (via SerpApi), or a website / business name to audit it.
Audit: code checks the site (down, no HTTPS, not mobile-friendly, stale content, no contact path, no video, weak Google reviews, PageSpeed scores), then Gemma reads the page text and spots what rules can't.
Board: Saved → Pitched…

View on GitHub

How I Built It

Code finds the facts, but Gemma reads them

I didn't want a model guessing what's wrong with a business. So the work is split.

Code checks facts. The crawler (SerpApi) reads the homepage and contact page (10-second timeout, robots.txt respected), and deterministic checks turn into evidence: no website, site down, no HTTPS, no mobile viewport, Google PageSpeed scores, stale content, no contact path, a WhatsApp button pointing at a placeholder number, no video, weak Maps presence.

SerpApi is a real-time API to access Google search results. We handle proxies, solve captchas, and parse all rich structured data for you.

Gemma does the three jobs code can't:

Read the site, not just check it. It gets about 1,800 characters of page text, the business's worst reviews and the check results, and returns up to three findings the checks missed.
Judge fit for this user. "No video anywhere" is the lead for a video editor and noise for a web designer. The score and top reason are for the person using the app.
Write the pitch from the evidence gathered.

Every answer gets checked before you see it

Small models make things up, so nothing Gemma writes goes straight to the screen. Every reply is validated with zod, then checked by rules in code:

Every pitch must cite evidence IDs, and its opening must say what that evidence says.
Numbers about the business (ratings, review counts, scores, years) must appear in the cited evidence.
Quotes from reviews must be exact.
No hard-sell words, no lists of services, one question at the end, and no compliment unless a passed check or a good review backs it.

A rejected answer gets one retry with the reason fed back. If the retry fails too, the parts that pass are kept and the rest is dropped. Every call is logged with tokens, latency, cost and whether it passed on the first try.

The bugs that taught me the most

Gemma got stuck repeating itself. Ollama kept aborting with "token repeat limit reached". The cause: newer Ollama versions turn repeat_penalty off by default. Setting every option explicitly fixed it.
My validator was too strict. It rejected "1 quick question" because "1" wasn't in the evidence. Now it only checks numbers about the business.
The model copied my example pitch about couples booking a venue, word for word, for an apartment. Examples in prompts are dangerous with small models.
My first Render deploy failed because the database connection opened at import time, and Render doesn't mount the disk during the build. Opening it on first query fixed it.

Stack

Next.js, TypeScript, Tailwind, shadcn/ui, Motion, SQLite with Drizzle, zod. Gemma runs through Ollama on my laptop and OpenRouter in production; one setting switches between them. SerpApi for Google Maps search and reviews. Render for hosting.

Lastly, the type-and-it-becomes-a-card search box is inspired by Shapeshift (MIT).

What isn't done yet

The audit only crawls two pages, and checks like booking forms, prices, and link previews aren't built. I also tested a second mode, "find people who are already looking for a video editor", using SerpApi's Google Jobs and LinkedIn results. 10 Google Jobs results for Lagos gave about 2 strong freelance fits, which was very promising.

Why Does Open Innovation Matter?

1. Because the AI has to cost almost nothing

My users are Lagos freelancers. They won't pay much, and every search means auditing 10 businesses, plus a pitch (often rewritten) for every lead they save. If the AI cost scales with every rewrite, the business doesn't work.

So I measured it. Same 8 real leads, same prompts, same checks:

Model	Deep read: passed / first try	Pitch: passed / first try	Per 1,000 leads
gemma-3-4b (OpenRouter)	50% / 0%	63% / 38%	$0.21
gemma-3-12b (OpenRouter)	75% / 25%	67% / 67%	$0.23
gemma-3-27b (OpenRouter)	100% / 13%	100% / 88%	$0.32
qwen3-32b (OpenRouter)	100% / 50%	100% / 100%	$0.27
gemma3:12b (Ollama, my laptop)	100% / 63%	100% / 100%	$0

On almost the same token volume, Claude Sonnet 5.5 would cost about $8.34 per 1,000 leads and GPT-5.4 about $11.05 (published prices, checked 16 October 2026; approximate, since tokenizers differ). Gemma 27B does it for $0.32, roughly 26 to 35 times cheaper. At that price, the AI isn't the expensive part of my product anymore. The SerpApi searches are.

2. Because I could switch models with one setting

That table took one command. The model name is an environment variable, so comparing five setups cost me under a cent. I picked gemma-3-27b for production because it passed every check on every lead.

3. Because I'm not stuck with one host

The most interesting row is the last one. The same Gemma 12B that OpenRouter rate-limited and timed out on passed 8 out of 8 on my own laptop, with the best first-try rate in the table. Open weights meant that when a host failed me, I could just run the model myself.

4. Where closed models might win

To be fair: small open models needed a strict validator and a retry to be trustworthy. A big closed model might pass on the first try more often. I didn't test that. My bet is that a validator I control, plus a model 26 times cheaper, beats paying more and hoping. And to be clear on privacy: in production, business data passes through OpenRouter to a model host, so I'm not claiming it never leaves my server. The data is public business information anyway.

My Agent Session

https://dev.to/agent_sessions/catching-a-stale-build-instruction-before-it-wiped-out-real-storage-cz3suy

My First User

I sent it with a friend, Chinemerem. She works as a marketing lead. According to her, the experience is smooth, and she was able to find a list of companies and even discover new places around me. Her fav features were the board management and how she can more easily send and manage lead interactions.

She, however, felt sales would appreciate this better.

Another friend also tried it out.

So create an account and give it a try

Prize Categories

Best Use of Gemma: Gemma reads every site, judges fit, and writes every pitch, locally through Ollama and in production through OpenRouter.
Best Use of Render: The app, login, and SQLite database run on a Render web service with a persistent disk.
Best Use of SerpApi: Google Maps search and name lookups, plus Google Maps Reviews for the lowest-rated reviews Gemma reads.

Who built it?

I built it myself

Going outside was never my problem. Saying hello was.

Benny — Sun, 11 Oct 2026 12:14:05 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

What I Built

I'm an introvert by nature but an extrovert by society. In a meeting or anything official, I'm fine, mainly because there's a reason to talk. What I struggle with is talking to people when I don't have to, or what we call small talk, especially with someone I don't necessarily need to talk to.

So when the theme was "Touch Grass", I didn't want another app that tells you to go for a walk. Going outside isn't the hard part for introverts. Talking to the people out there is. Many social apps help you talk to people, but almost all of them keep you talking through a screen. This app is the opposite. It gets you talking to the world around you: the people you already pass every day and never really speak to. It’s about realizing that every random passerby is living a life just as vivid, complex, and emotionally rich as your own. That’s the idea behind the app.

Sonder is a weekly game for that. It makes it easier to connect with the people around you, without the pressure of walking up to a stranger and figuring out what to say. Because sometimes, all it takes to turn a stranger into a friend is a little nudge. Every week you get a few quests on sticky notes set in places you actually go. It comes with two difficulty levels (easy, fine, and scary)

You do the quest out in the world, come back, and write what you found out. Gemma replies like a friend, and your notes build into a wall of people who aren't strangers anymore. It's for anyone who walks past the same faces every day and never says more than "good morning".

How it works

Tell it your normal week. Pick the places you actually go (market, office, gym, bus stop, church or mosque, beach and more), or add your own.
Pick your challenge. Easy, Fine or Scary (see below).
Get your week. Gemma writes a mix of quests for your places. Each has a one-line opener you can actually say.
Go and do them. Write down what you found out. Gemma replies to what you learned, not to the quest.
Watch your wall grow. The People tab keeps every note, week by week.

Settings let you change places, level, and quests per week at any time. New quests swaps the ones you haven't started and keeps the ones you've done.

Three levels

Level	What a quest looks like	Example
Easy	One quick question, under a minute	Ask someone at the bus stop which bus goes somewhere you know
Fine	A light chat with one simple follow-up	Ask a regular at the gym to show you one exercise they swear by
Scary	A real conversation	Ask an older trader at the market what has changed most over the years

Every week mixes quest types, so no two weeks feel the same. There are 14 types, from Learn a name and Ask the way up to Origin story and Return visit (going back to someone you met before).

A real quest, start to finish

Gemma generates the quests:

Quests are displayed.

Give feedback on the quest. What did you learn?

See all the strangers you have interacted with:

"Evening, because he tried morning for a week and was sleepy at work." That's one person at my gym who isn't a stranger anymore.

Not feeling it? Skip with a reason. "Didn't feel safe" switches you to Play it safe automatically.

Or change the settings.

Demo

Sonder runs on my laptop, so there's no public link. It also stores notes about real people you've met, so it runs on your own laptop instead of a server. To try it, clone the repo and follow the three steps in the README. It takes about five minutes if you have Ollama installed.

The video below shows how it works on my MacBook.

I took it outside

My first quest was an Easy one: ask your barber which style everyone is asking for right now. So I went for a haircut and asked Tobi, my barber. I've been going to him for a while, and I'd never really asked him anything beyond "the usual". His answer: everyone wants to dye their hair. 😂

That turned into a whole conversation. He reminded me that I'd talked about dyeing my hair when I came in a month ago, and dragged me for never coming back to do it.

I wrote it down in Sonder, and Gemma replied:

"Seriously? Everyone's going for a dye job? That's wild, what color were people asking for specifically?"

Which is fair. I didn't ask. That's the follow-up for my next haircut. Sonder has a gallery of all your conversations.

My friend also tried it out and did a quest on his way home in the keke (tricycle).

Code

Bennykillua / sonder

A weekly game that gets you talking to the people you walk past every day. Small quests on sticky notes, written by Gemma running locally. Your notes never leave your laptop.

Sonder

Going outside was never the hard part. Saying hello was.

sonder (n.): the realisation that every stranger you pass has a life as vivid and complex as your own.

Sonder is a weekly game that gets you talking to the people you walk past every day. You get a few small quests on sticky notes, set in the places you actually go. You do them out in the world, write down what you found out, and Gemma replies like a friend would. Over time, strangers turn into names.

It runs entirely on your own machine, with Gemma running locally through Ollama.

The problem

I'm an introvert. In a meeting or anything official, I'm fine, because there's a reason to talk. Talking to people when I don't have to is the hard part.

Most "touch grass" apps tell you to go outside. Going outside was never my problem. Sonder is…

View on GitHub

MIT-licensed and built during the challenge. It's one Python file for the backend. The README explains how to run it locally with Ollama in three commands.

How I Built It

The stack is small on purpose:

Gemma, running locally through Ollama
FastAPI and SQLite on my laptop
One HTML page for the whole interface, opened on my phone. I made it whimsical and delightful

The interesting part wasn't the stack. It was getting a small model to write quests worth doing.

What Gemma does

Writes your week. It turns your places and your challenge level into a set of quests, each with an opener you can actually say and, for Fine and Scary, a follow-up question.
Replies to what you found out. When you finish a quest and write a note, Gemma responds to the person you met, like a friend would over text.

How the app is built around a small local model

Small models are fast and private, but they copy examples and drift off-brief. So the app gives Gemma structure instead of freedom:

Each week is planned before Gemma writes anything. The app splits the week into slots, and each slot gets a quest type (14 types, from "Learn a name" to "Origin story") and one of your places. Gemma writes one quest per slot, so the week stays varied and tied to where you actually go.
Gemma returns structured JSON. Ollama's structured output means every quest arrives with the same fields: task, opener, follow-up, and place.
Every quest is checked in code before you see it. It's dropped if it's a label instead of a task, small talk, a copy of its own brief, too long to say out loud, set somewhere you didn't pick, or a repeat.
Safety rules are enforced in code, not just in the prompt:
- never asks about marriage, tribe, politics, money, or age, and never asks for phone numbers or photos
- no faith questions at all
- hospital quests are staff-only
- no alcohol at bars or drink shops
- a "Play it safe" mode keeps everything in shops and queues
83 backup quests fill any gaps, so a bad run from the model never leaves you with an empty week.
Weeks are written in the background. The phone checks back every few seconds, so a slow model never freezes the app.

Why Does Open Innovation Matter?

Because of what Sonder stores. It's a diary of every person you talked to, what they told you, and which conversations you skipped because they felt awkward or unsafe. I wouldn't send that to a server I don't control, and I don't think anyone else should either.

That's also why there's no hosted version: the moment I host it, your notes live on my server.

With an open model running locally:

Nothing leaves my laptop. Gemma, the quests, and the notes all live on my machine.
It costs nothing to run. That matters in Nigeria, where paying per request adds up fast.
I could choose my own speed and quality trade-off. I tried gemma3:1b, which wrote a week in about 2 seconds but gave plainer quests. I went back to gemma3:4b for better ones, and kept it loaded in memory so it's ready when you open the app. With a closed API, that choice wouldn't be mine.

The safety filters are in my code, and anyone can read them in the repo.

My Agent Session

I don't have a DevRelay session to share. I didn't build Sonder with Claude Code. I used Claude in chat as a pair programmer: it drafted code, helped me plan the app, and helped me fix bugs. I ran everything, tested it on my phone and with various friends, debugged with Claude Code, and decided what stayed.

Prize Categories

Best Use of Gemma

Gemma does two jobs in Sonder. Both run locally through Ollama, using gemma3:4b on my laptop (a MacBook):

The quest writer. Each week, the app gives Gemma one brief per quest: a quest type, one of your places, and your challenge level. Gemma writes the task, a one-line opener you can actually say and, on Fine and Scary, a follow-up question. It returns them as structured JSON, and code checks every quest before you see it.
The friend who replies. When you finish a quest and write what you found out, Gemma responds to the person you met, not the quest, like a friend over text.

Running Gemma locally means your notes about real people never leave your machine, and Sonder costs nothing per request.

Built solo.

If you run Sonder yourself, tell me what features you'd like to see or how I can improve it.

Also, if you have some doubts about giving it a try, think of this viral internet quote: "It’s okay to embarrass yourself a little in the pursuit of human connection," which stemmed from a quote by Bianca Sparacino

In a world that seeks connection, we oddly avoid eye contact, we time our text responses to protect ourselves from seeming too eager or too interested, and we hold our feelings back because we don’t want to seem overly emotional or unreasonable. We silence our instincts, and at the end of the day instead of feeling good about ourselves, we feel alone, we feel misunderstood. Remember - it is okay to be emotional, to seek help, to confidently tell someone you enjoy being around that you are infatuated with them. There is nothing wrong with vulnerability, with being human, for that is what creates depth within our relationships, and that is what ultimately unities us.

Build an AI lead generation agent with the OpenAI Agents SDK and Zenrows

Benny — Thu, 08 Oct 2026 09:56:17 +0000

This article was originally published on the Zenrows blog.

In this tutorial, you'll build a lead generation agent that reads a bot-protected business directory, pulls every company listed on the page, enriches each one from its own website, and scores it against your ideal customer profile (ICP). You'll need Python 3.10 or above, an OpenAI API key, and a Zenrows API key.

The OpenAI Agents SDK runs the agent loop, and its @function_tool decorator turns plain Python functions into tools. Zenrows Fetch handles the step that usually breaks, which is getting the page at all. In the test run, one directory page produced 78 companies, and the agent scored each one in under 20 seconds.

The complete code is on GitHub if you'd like to clone it and follow along.

Prerequisites

You need Python 3.10 or above.
You need an OpenAI API key, which the agent, the extraction tool, and the scoring tool all use. Get one from your OpenAI developer dashboard.
You need a Zenrows API key to retrieve the full contents of web pages. Create one on the signup page.

Install the dependencies.

python3 -m pip install openai-agents openai requests pydantic python-dotenv

The SDK runs the agent, requests makes the Zenrows calls, pydantic defines the tool schemas, and python-dotenv loads your keys. Create a .env file in your project root with your OPENAI_API_KEY and ZENROWS_API_KEY, and add .env to .gitignore so the credentials never reach version control.

How the agent is structured

The work splits into four tools, and each one does a single job.

Tool	Input	Output	Why it's separate
`fetch_page`	a URL	the page as Markdown	it's the only job that touches the network, so retries and errors live in one place
`extract_leads`	Markdown plus the source URL	`Lead` records	it runs in chunks because a full directory page exceeds one reliable extraction call
`enrich_lead`	a company and its domain	enrichment signals from the company's site	it makes one fetch per nav link, so it drives the cost and is worth isolating
`score_lead`	enriched lead plus the ICP	a score from 0 to 100 with reasoning	it's pure reasoning with no network calls, so you can rerun it without re-fetching

Splitting the work keeps retries local. When enrichment fails on one lead, the agent retries that tool and keeps the rest of the run intact. Each job's output also stays small enough to hand to the next one.

The agent only sees two composed tools. discover_leads runs fetch_page and extract_leads, and qualify_lead runs enrich_lead and score_lead. If the agent passed an enriched lead between two separate tools, it would have to reproduce every page excerpt as a tool argument, and that overflows the context window on a large directory.

1. Build the Zenrows Fetch tool

Every code block from here goes into one file called tools.py. Append each block to the same file.

import os
import re
import requests
from urllib.parse import urlparse, parse_qs, unquote

from agents import function_tool
from dotenv import load_dotenv
from openai import OpenAI
from pydantic import BaseModel

load_dotenv()

ZENROWS_API_KEY = os.getenv("ZENROWS_API_KEY")
ZENROWS_ENDPOINT = "https://api.zenrows.com/v1/"

if not ZENROWS_API_KEY:
    raise RuntimeError("set ZENROWS_API_KEY in your .env file")

client = OpenAI()

# ===========================================================================
# job 1: fetch a page, protected or not
# ===========================================================================

def _fetch_page(url: str) -> str:
    """fetch logic, importable and testable without the agent."""
    params = {
        "url": url,
        "apikey": ZENROWS_API_KEY,
        "mode": "auto",              # start cheap, escalate only when the target needs it
        # markdown strips nav and boilerplate before the model reads it
        "response_type": "markdown"
    }

    try:
        response = requests.get(ZENROWS_ENDPOINT, params=params, timeout=90)
        response.raise_for_status()
    # return failures as text instead of raising, so one bad url does not end the run.
    # each message tells the model whether the failure is worth retrying
    except requests.exceptions.Timeout:
        return "FETCH_ERROR: timed out after 90s. retry once, then move on."
    except requests.exceptions.HTTPError as exc:
        status = exc.response.status_code
        if status in (401, 403):
            return f"FETCH_ERROR: {status}. api key rejected, do not retry."
        if status == 429:
            return "FETCH_ERROR: 429 concurrency limit. wait, then retry."
        if status in (400, 404):
            return f"FETCH_ERROR: {status}. this url is not retrievable, do not retry."
        return f"FETCH_ERROR: {status}. retry once, then report the failure."
    except requests.exceptions.RequestException as exc:
        return f"FETCH_ERROR: {exc}. do not retry."

    content = response.text.strip()
    if not content:
        return "FETCH_ERROR: empty response body. do not retry."

    return content

# the decorator replaces the function with a FunctionTool object, which is not
# callable. keeping the logic in _fetch_page above leaves it testable
@function_tool
def fetch_page(url: str) -> str:
    """Fetch any web page as markdown, including pages behind anti-bot protection.

    Args:
        url: full url of the page to fetch
    """
    return _fetch_page(url)

# ===========================================================================
# job 2: turn a directory page into leads
# ===========================================================================

# matches the tracking links directories wrap around outbound urls
REDIRECT_LINK = re.compile(r"https?://[a-z0-9.-]*/redirect\?[^\s\)\"]+")

# a full directory page is too long for one reliable extraction call
CHUNK_SIZE = 40000

# slices cut mid-listing, and the extraction prompt is told to skip partial
# entries, so a listing straddling a boundary is dropped by both slices.
# overlapping the window carries each boundary listing whole into one of them;
# _key dedupes the entries the overlap sees twice
CHUNK_OVERLAP = 2000

_fetch_page holds the logic, and fetch_page wraps it with @function_tool so the agent can call it. The decorator replaces the function with a FunctionTool object that you can't call directly, so keeping the logic in a plain function leaves it testable.

Failures come back as text strings, so one bad URL doesn't end the run. Each message also tells the model whether a retry is worth it.

mode=auto lets Zenrows pick the access configuration each target needs. Simple pages resolve on the lightest path, and only protected targets escalate. That matters because protected sites rarely block you with an error. A Cloudflare-protected page can return a 200 status code with a challenge page in the body, and your agent will treat that challenge as a valid page with zero leads on it.

response_type=markdown strips navigation and boilerplate, which gives the model cleaner input for extraction. If your directory is an API-like endpoint, use response_type=json and skip the conversion. The three constants at the bottom of the block belong to the extraction step in the next section.

2. Extract leads from the directory page

Append the extraction code to tools.py.

# strict tool schemas reject bare dicts, so every tool input and output is a model
class Lead(BaseModel):
    company: str
    name: str
    website: str
    source_url: str

class LeadList(BaseModel):
    leads: list[Lead]

def _unwrap_redirects(markdown: str) -> str:
    """rewrite directory tracking links to the destination domain."""
    def replace(match):
        url = match.group(0)
        # the real destination sits url-encoded in the u parameter
        target = parse_qs(urlparse(url).query).get("u", [""])[0]
        if not target:
            return url
        parsed = urlparse(unquote(target))
        return f"{parsed.scheme}://{parsed.netloc}"

    return REDIRECT_LINK.sub(replace, markdown)

def _reject_source_domain(leads: list[dict], source_url: str) -> list[dict]:
    """blank any website that points back at the directory itself."""
    directory = urlparse(source_url).netloc.replace("www.", "")
    for lead in leads:
        host = urlparse(lead["website"]).netloc.replace("www.", "")
        if host == directory:
            lead["website"] = ""
    return leads

def _key(lead: dict) -> str:
    """stable identity for dedupe across chunks."""
    # the model returns the same domain with and without www, so normalise before comparing
    site = lead["website"].lower()
    for prefix in ("https://", "http://", "www."):
        site = site.replace(prefix, "")
    site = site.rstrip("/")
    # fall back to company name so leads without a domain are not collapsed into one
    return site or lead["company"].strip().lower()

def _extract_chunk(chunk: str, source_url: str) -> list[dict]:
    """run one extraction call over a slice of the page."""
    response = client.responses.parse(
        model="gpt-4o-mini",
        max_output_tokens=8000,
        input=[
            {
                "role": "system",
                "content": (
                    "extract every company listed in this page fragment. "
                    "website is the company's own domain, taken from that listing's "
                    "visit website link. "
                    "never use a link from the directory's own domain as the website. "
                    "name is a person's name and is usually absent from a directory "
                    "listing, so leave it empty unless a person is actually named. "
                    "use an empty string for any field the fragment does not state, "
                    "and never invent a value. "
                    "the fragment may start or end mid listing, so skip any partial entry."
                )
            },
            {
                "role": "user",
                "content": f"source_url: {source_url}\n\n{chunk}"
            }
        ],
        text_format=LeadList
    )
    return [lead.model_dump() for lead in response.output_parsed.leads]

def _extract_leads(content: str, source_url: str) -> list[dict]:
    """parse markdown into lead dicts, chunking long pages."""
    # a fetch failure is a string, not a page. parsing it would return an empty
    # list that looks identical to a genuinely empty directory
    if content.startswith("FETCH_ERROR"):
        return []

    # resolve tracking links before the model reads them
    content = _unwrap_redirects(content)

    leads, seen = [], set()
    for start in range(0, len(content), CHUNK_SIZE - CHUNK_OVERLAP):
        chunk = content[start:start + CHUNK_SIZE]
        for lead in _extract_chunk(chunk, source_url):
            key = _key(lead)
            if key in seen:
                continue
            seen.add(key)
            leads.append(lead)

    return _reject_source_domain(leads, source_url)

@function_tool
def extract_leads(content: str, source_url: str) -> list[Lead]:
    """Parse a fetched page into a list of leads.

    Args:
        content: markdown returned by fetch_page
        source_url: url the content came from, recorded on every lead
    """
    return [Lead(**lead) for lead in _extract_leads(content, source_url)]

# fetch and extract are one job from the agent's point of view. exposing them
# separately means the whole page passes through the model's context to get
# from one tool to the next, which overflows the window on a large directory
@function_tool
def discover_leads(source_url: str) -> list[Lead]:
    """Fetch a directory page and return the companies listed on it.

    Args:
        source_url: url of the directory or listing page
    """
    markdown = _fetch_page(source_url)
    if markdown.startswith("FETCH_ERROR"):
        return []
    return [Lead(**lead) for lead in _extract_leads(markdown, source_url)]

# ===========================================================================
# job 3: enrich each lead from its own website
# ===========================================================================

Strict tool schemas reject bare dicts, so every tool input and output is a Pydantic model. _unwrap_redirects rewrites the directory's tracking links to each company's own domain before the model reads them, and _reject_source_domain blanks any website that points back at the directory.

A full directory page is too long for one reliable extraction call, so _extract_leads splits it into 40,000-character chunks with a 2,000-character overlap. The overlap carries any listing that straddles a boundary into one chunk whole, and _key removes the duplicates the overlap creates.

discover_leads combines fetching and extraction in one tool because the raw output of fetch_page can exceed the context length by itself.

Test both jobs without the agent. Save this script as test_fetch_extract.py.

# test_fetch_extract.py
import json
import os
from tools import _fetch_page, _unwrap_redirects, _extract_leads

SOURCE_URL = "https://clutch.co/it-services"
FIXTURE = "fixtures/directory_page.md"
LEADS_CACHE = "fixtures/leads.json"

# ---- job 1: fetch ---------------------------------------------------------
# cache the page so extraction can be tuned without paying for a fetch each time
if os.path.exists(FIXTURE):
    with open(FIXTURE) as f:
        markdown = f.read()
    print(f"using cached page, {len(markdown)} chars")
else:
    markdown = _fetch_page(SOURCE_URL)
    if markdown.startswith("FETCH_ERROR"):
        raise SystemExit(markdown)
    os.makedirs("fixtures", exist_ok=True)
    with open(FIXTURE, "w") as f:
        f.write(markdown)
    print(f"fetched and cached, {len(markdown)} chars")

unwrapped = _unwrap_redirects(markdown)
print(f"{len(unwrapped)} chars after unwrapping redirects, "
      f"{len(markdown) - len(unwrapped)} saved\n")

# ---- job 2: extract -------------------------------------------------------
leads = _extract_leads(markdown, SOURCE_URL)

with open(LEADS_CACHE, "w") as f:
    json.dump(leads, f, indent=2)

print(f"{len(leads)} leads extracted\n")
for lead in leads[:5]:
    print(f"{lead['company']:<32} {lead['website'] or '(none)'}")

missing_site = sum(1 for lead in leads if not lead["website"])
print(f"\n{missing_site} of {len(leads)} leads have no website")

Run it from the project root.

python3 test_fetch_extract.py

Here's the output from a local run against an IT services directory.

using cached page, 543637 chars
457647 chars after unwrapping redirects, 85990 saved

78 leads extracted

Infracore                        https://infracore.net
Miles IT                         https://www.milesit.com
Geniusee                         https://geniusee.com
Peeklogic, LLC                   https://www.peeklogic.com
TechQuarter LLC                  https://www.techquarter.io

0 of 78 leads have no website

The page came back at 543,637 characters, and unwrapping the redirects saved 85,990 of them. Counts vary between runs because directory pages change and the model can read the same page differently.

3. Enrich and score each lead

Append the enrichment code to tools.py.

# matches any markdown link, used to read a company's own navigation
MARKDOWN_LINK = re.compile(r"\[([^\]]+)\]\((https?://[^\s\)]+)\)")

# guessing paths costs a fetch per miss, so read the site's nav instead and
# follow only the links it actually has
SIGNAL_KEYWORDS = {
    "hiring": ["career", "job", "join", "hiring", "work with us", "we are hiring"],
    "services": ["service", "what we do", "solutions", "expertise", "capabilities"],
    "portfolio": ["portfolio", "case stud", "our work", "projects", "clients"],
    "about": ["about", "who we are", "our story", "team"],
    "contact": ["contact", "get in touch", "book a call", "let's talk", "talk to us"],
}

# dict[str, X] is rejected by strict schemas too, which is why signals is a list
class Signal(BaseModel):
    name: str
    found: bool
    url: str
    excerpt: str

class EnrichedLead(BaseModel):
    company: str
    website: str
    signals: list[Signal]

def _discover_links(markdown: str, website: str) -> dict:
    """find real urls for each signal by reading the homepage nav."""
    host = urlparse(website).netloc.replace("www.", "")
    homepage_path = urlparse(website).path.rstrip("/") or "/"
    found = {}

    for text, url in MARKDOWN_LINK.findall(markdown):
        parsed = urlparse(url)

        # only follow links on the company's own domain
        if parsed.netloc.replace("www.", "") != host:
            continue

        # an anchor on the homepage is content already in the homepage excerpt
        if parsed.fragment and (parsed.path.rstrip("/") or "/") == homepage_path:
            continue

        # drop the fragment, it never changes what the server returns
        clean_url = f"{parsed.scheme}://{parsed.netloc}{parsed.path}"

        haystack = f"{text} {url}".lower()
        for signal, keywords in SIGNAL_KEYWORDS.items():
            if signal not in found and any(k in haystack for k in keywords):
                found[signal] = clean_url

    return found

def _enrich_lead(company: str, website: str) -> EnrichedLead:
    """collect signals from the company's own site."""
    # a lead with no domain still flows through to scoring, judged on what little is known
    if not website:
        return EnrichedLead(company=company, website="", signals=[])

    homepage = _fetch_page(website)
    if homepage.startswith("FETCH_ERROR"):
        return EnrichedLead(company=company, website=website, signals=[])

    # the homepage is the most informative single page, and on a one-page site
    # it holds everything the nav links point at
    signals = [Signal(name="homepage", found=True, url=website, excerpt=homepage[:6000])]

    for name, url in _discover_links(homepage, website).items():
        content = _fetch_page(url)
        ok = not content.startswith("FETCH_ERROR")
        signals.append(Signal(
            name=name,
            found=ok,
            url=url,
            # keep excerpts small, scoring reads all of them in one call
            excerpt=content[:2000] if ok else ""
        ))

    return EnrichedLead(company=company, website=website, signals=signals)

@function_tool
def enrich_lead(company: str, website: str) -> EnrichedLead:
    """Collect signals from a company's own website.

    Args:
        company: company name from the directory listing
        website: company's own domain
    """
    return _enrich_lead(company, website)

# ===========================================================================
# job 4: score each lead against the icp
# ===========================================================================

An extracted lead carries only a company name and a domain, which is too little to judge fit. enrich_lead reads the company's homepage, finds the navigation links the site exposes, and fetches only those pages. Guessing paths like /careers would cost a fetch for every miss. The signals that matter live on the company's own site in its hiring pages, services pages, case studies, and contact pages.

Markdown works well when the model does the reading. If you'd prefer structured fields, Zenrows Extract returns JSON without a selector map.

Next, append the scoring code.

class Score(BaseModel):
    score: int
    reasoning: str

class ScoredLead(BaseModel):
    company: str
    website: str
    score: int
    reasoning: str

def _score_lead(lead: EnrichedLead, icp: str) -> ScoredLead:
    """score an enriched lead against the icp description."""
    # no fetching happens here. every page was already retrieved in job 3
    summary = "\n\n".join(
        f"{s.name} page: {'found at ' + s.url if s.found else 'not found'}\n{s.excerpt}"
        for s in lead.signals
    )

    response = client.responses.parse(
        model="gpt-4o-mini",
        max_output_tokens=500,
        input=[
            {
                "role": "system",
                "content": (
                    "score how well this company matches the ideal customer profile. "
                    "return an integer from 0 to 100 and one sentence of reasoning. "
                    "base the score only on evidence in the signals provided. "
                    "a missing page is weak evidence, not disqualifying. "
                    "if the signals are too thin to judge, score below 30 and say so."
                )
            },
            {
                "role": "user",
                "content": (
                    f"ideal customer profile:\n{icp}\n\n"
                    f"company: {lead.company}\n"
                    f"website: {lead.website}\n\n"
                    f"signals:\n{summary}"
                )
            }
        ],
        text_format=Score
    )

    return ScoredLead(
        company=lead.company,
        website=lead.website,
        score=response.output_parsed.score,
        reasoning=response.output_parsed.reasoning
    )

@function_tool
def score_lead(lead: EnrichedLead, icp: str) -> ScoredLead:
    """Score an enriched lead against an ICP description.

    Args:
        lead: enriched lead returned by enrich_lead
        icp: plain-language description of the ideal customer
    """
    return _score_lead(lead, icp)

# an EnrichedLead carries page excerpts the model has no reason to read. passing
# it between two tools means the agent has to reproduce all of it as an argument,
# and it summarises instead, so scoring reads a summary rather than the pages
@function_tool
def qualify_lead(company: str, website: str, icp: str) -> ScoredLead:
    """Enrich a lead from its own website and score it against an ICP.

    Args:
        company: company name from the directory listing
        website: company's own domain
        icp: plain-language description of the ideal customer
    """
    enriched = _enrich_lead(company, website)
    return _score_lead(enriched, icp)

score_lead reads every signal alongside the ICP and returns a score from 0 to 100 with one sentence of reasoning. The prompt ties the score to evidence, so a lead with only a domain scores lower than one with a careers page and named case studies. No fetching happens in this step, which means you can rerun scoring without paying for the pages again.

qualify_lead pairs enrichment and scoring for the same reason discover_leads pairs fetching and extraction. The bulky intermediate data stays inside one Python call.

Test both halves before you wire the agent, so any later failure belongs to the agent loop. Save this script as test_enrich_score.py.

# test_enrich_score.py

import json
import os
import time
from tools import _fetch_page, _unwrap_redirects, _extract_leads, _enrich_lead, _score_lead

SOURCE_URL = "https://clutch.co/it-services"
FIXTURE = "fixtures/directory_page.md"
LEADS_CACHE = "fixtures/leads.json"

ICP = (
    "IT services agencies that build custom software for B2B clients, "
    "publish detailed case studies with named clients, and offer ai and cloud services"
)

# ---- job 1: fetch ---------------------------------------------------------
if os.path.exists(FIXTURE):
    with open(FIXTURE) as f:
        markdown = f.read()
    print(f"using cached page, {len(markdown)} chars")
else:
    markdown = _fetch_page(SOURCE_URL)
    if markdown.startswith("FETCH_ERROR"):
        raise SystemExit(markdown)
    os.makedirs("fixtures", exist_ok=True)
    with open(FIXTURE, "w") as f:
        f.write(markdown)
    print(f"fetched and cached, {len(markdown)} chars")

unwrapped = _unwrap_redirects(markdown)
print(f"{len(unwrapped)} chars after unwrapping redirects, "
      f"{len(markdown) - len(unwrapped)} saved\n")

# ---- job 2: extract -------------------------------------------------------
if os.path.exists(LEADS_CACHE):
    with open(LEADS_CACHE) as f:
        leads = json.load(f)
    print(f"using {len(leads)} cached leads\n")
else:
    leads = _extract_leads(markdown, SOURCE_URL)
    with open(LEADS_CACHE, "w") as f:
        json.dump(leads, f, indent=2)
    print(f"{len(leads)} leads extracted\n")

for lead in leads[:5]:
    print(f"{lead['company']:<32} {lead['website'] or '(none)'}")

missing_site = sum(1 for lead in leads if not lead["website"])
print(f"\n{missing_site} of {len(leads)} leads have no website\n")
print("-" * 60 + "\n")

# ---- jobs 3 and 4: enrich and score ---------------------------------------
# enrichment is one fetch per nav link found, so start with a few leads
sample = [lead for lead in leads if lead["website"]][:3]

for lead in sample:
    start = time.time()
    enriched = _enrich_lead(lead["company"], lead["website"])
    elapsed = time.time() - start

    found = [s.name for s in enriched.signals if s.found]
    print(f"{enriched.company}  ({elapsed:.1f}s)")
    print(f"  found: {', '.join(found) if found else 'nothing'}")

    scored = _score_lead(enriched, ICP)
    print(f"  score: {scored.score}")
    print(f"  {scored.reasoning}\n")

Run it.

python3 test_enrich_score.py

Here are the first two leads from the output.

Infracore  (12.8s)
  found: homepage, contact, services, portfolio, about, hiring
  score: 80
  Infracore fits well within the ideal customer profile as they offer IT services, including cloud solutions and have case studies showcasing customized work for various industries, but they may lack detailed named client presentations.

Miles IT  (12.1s)
  found: homepage, contact, about, services, portfolio, hiring
  score: 75
  Miles IT provides software development and AI services, aligning well with the ideal customer profile, but lacks detailed case studies with named clients which slightly diminishes the score.

Each lead took around 12 seconds to enrich. The page list reflects what each site exposes in its navigation, so a lead with no careers page has no hiring signal.

4. Wire the agent and run it

Save the agent as agent.py.

import asyncio

from agents import Agent, Runner, set_tracing_disabled

from tools import discover_leads, qualify_lead

set_tracing_disabled(True)

SOURCE_URL = "https://clutch.co/it-services"

ICP = (
    "IT services agencies that build custom software for B2B clients, "
    "publish detailed case studies with named clients, and offer ai and cloud services"
)

INSTRUCTIONS = """You find and qualify sales leads from web directories.

Follow this sequence exactly:

1. call discover_leads on the source url the user gives you
2. for each lead that has a website, call qualify_lead with its company, website,
   and the icp description from the user message
3. return the scored leads as a json array sorted by score descending

rules:
- process only the first 10 leads that have a website, then stop and return them
- skip leads with no website, they cannot be scored fairly
- if a tool returns a string starting with FETCH_ERROR, follow the instruction in
  that message. do not retry a call the message tells you not to retry
- never invent a lead, a website, or a score. every value comes from a tool result
"""

agent = Agent(
    name="lead generation agent",
    instructions=INSTRUCTIONS,
    model="gpt-4o-mini",  # the reasoning here is scoring, not planning
    tools=[discover_leads, qualify_lead],
)

async def main():
    result = await Runner.run(
        agent,
        input=f"source url: {SOURCE_URL}\n\nideal customer profile:\n{ICP}",
        max_turns=50,  # one qualify call per lead, plus discovery and the reply
    )

    # the trace shows which tools ran and in what order
    for item in result.new_items:
        if item.type == "tool_call_item":
            print(item.raw_item.name)

    print()
    print(result.final_output)

if __name__ == "__main__":
    asyncio.run(main())

The agent gets the two composed tools and a fixed sequence in its instructions. A small model is enough here because the sequence is already decided and the heavy reasoning happens inside score_lead. The instructions also cap each run at 10 leads, skip leads with no website, and forbid invented values.

Run the agent.

python3 agent.py

The agent prints the ranked leads as JSON. Here are the first two.

[
    {
        "company": "Geniusee",
        "website": "https://geniusee.com",
        "score": 85,
        "reasoning": "Geniusee offers a range of IT services including custom software development, AI solutions, and cloud services, which align closely with the ideal customer profile; however, the absence of detailed case studies with named clients slightly limits the fit."
    },
    {
        "company": "Sigli",
        "website": "https://www.sigli.com",
        "score": 85,
        "reasoning": "Sigli aligns closely with the ideal customer profile by offering custom software development for B2B clients, specializing in AI and cloud services, and publishing case studies, although the specificity of named clients could not be verified."
    }
]

Every value comes from a tool result, so the tool-call trace is the first thing to check when a run misbehaves.

5. Scale across pages with Batch

Save this script as batch.py.

# batch.py
# Scale discovery across several directory pages with Zenrows Batch.
#
# One Fetch call retrieves one page. Batch takes the whole list as a single
# managed job and handles the queue, concurrency and retries, so a directory
# with many pages costs one submission instead of one call per page.

import time

import requests
from agents import function_tool

from tools import ZENROWS_API_KEY, Lead, _extract_leads

BATCH_ENDPOINT = "https://async.api.zenrows.com/v1/jobs"

# batch authenticates by header, unlike fetch which takes apikey as a query param
BATCH_HEADERS = {"X-API-Key": ZENROWS_API_KEY, "Content-Type": "application/json"}

# a job accepts up to 100,000 urls
MAX_URLS_PER_JOB = 100_000

def _submit_batch(urls: list[str]) -> str:
    """submit a url list as one job, return the job id."""
    if len(urls) > MAX_URLS_PER_JOB:
        raise ValueError(f"a job accepts at most {MAX_URLS_PER_JOB} urls")

    payload = {
        "type": "regular",
        "status": "closed",  # run once, accept no further tasks
        "zenrows_params": {"mode": "auto", "response_type": "markdown"},
        "tasks": [{"url": url} for url in urls],
    }
    response = requests.post(
        BATCH_ENDPOINT, headers=BATCH_HEADERS, json=payload, timeout=30
    )
    response.raise_for_status()
    return response.json()["job_id"]

def _collect_batch(job_id: str, max_attempts: int = 60) -> list[tuple[str, str]]:
    """wait for the job to reach a terminal state, then pull each task's markdown."""
    # terminal run states are completed, stopped and deleted. "failed" is a task
    # status, not a run status, so polling for it never returns
    for _ in range(max_attempts):
        time.sleep(5)
        status = requests.get(
            f"{BATCH_ENDPOINT}/{job_id}", headers=BATCH_HEADERS, timeout=30
        )
        status.raise_for_status()
        run = status.json()["latest_run"]
        if run["status"] in ("completed", "stopped", "deleted"):
            break
    else:
        raise TimeoutError(f"job {job_id} did not reach a terminal state in time")

    response = requests.get(
        f"{BATCH_ENDPOINT}/{job_id}/results", headers=BATCH_HEADERS, timeout=30
    )
    response.raise_for_status()

    pages = []
    for task in response.json()["results"]:
        if task["status"] != "successful":
            continue
        # result_url is presigned and valid for 2 hours; re-list the results for a
        # fresh link rather than storing this one
        markdown = requests.get(task["result_url"], timeout=60).text
        pages.append((task["url"], markdown))

    return pages

@function_tool
def discover_leads_batch(source_urls: list[str]) -> list[Lead]:
    """Fetch several directory pages as one job and return every company listed.

    Args:
        source_urls: directory or listing page urls to fetch
    """
    job_id = _submit_batch(source_urls)

    leads = []
    for url, markdown in _collect_batch(job_id):
        leads.extend(Lead(**lead) for lead in _extract_leads(markdown, url))

    return leads

if __name__ == "__main__":
    # two pages of the same directory, submitted as one job
    urls = [
        "https://clutch.co/it-services",
        "https://clutch.co/it-services?page=2",
    ]
    job_id = _submit_batch(urls)
    print(f"submitted {job_id}")

    for url, markdown in _collect_batch(job_id):
        print(f"{url}: {len(markdown)} chars")

One directory page rarely produces enough leads to matter, and a directory usually has many pages. Running each page as its own Fetch call means you manage concurrency and retries yourself. Zenrows Batch takes the whole URL list as one managed job and handles the queue, concurrency, retries, and result delivery. A single job accepts up to 100,000 URLs.

Batch authenticates with an X-API-Key header, while Fetch takes the key as a query parameter. Each successful task returns a presigned result_url that stays valid for two hours.

You can also reach Batch through the Python SDK, the zenrows batch CLI, and the Zenrows MCP server, and none of them need the polling loop. This example uses raw REST so every step stays visible. For the full implementation guide, read running large-scale scraping jobs with Batch.

Wrapping up

You now have an AI lead generation agent that reads a directory with Zenrows Fetch, pulls out every company listed, visits each company's domain, and scores the results against a plain-language ICP. The run in this tutorial pulled 78 companies off a single page, and agent.py scores the first 10 of them per run at under 20 seconds each.

That's automated lead generation with no vendor list underneath it. The agent reads the source, and the source is whichever directory your market uses.

What's next

Raise or remove the 10-lead cap in INSTRUCTIONS when you want a full pass over the directory.
Put a human review checkpoint between scoring and outreach, then push qualifying leads into a CRM such as HubSpot or Salesforce, an email sequencing tool, or a CSV export.
Wrap agent.py in an HTTP endpoint and trigger it as a webhook step from n8n, Zapier, or Power Automate, passing the source URL and the ICP in the request body.
Skip the wrapper entirely with the Zenrows MCP server, which exposes the same fetch capability to any MCP-compatible agent. This walkthrough of the MCP server covers the setup.

FAQ

How much does it cost to run this agent?

A standard Zenrows request costs one credit. mode=auto selects the configuration each target needs, and a page that requires the full configuration costs up to 25 credits. These weights stay fixed across all plans.

In this tutorial, the directory page resolved at the standard rate. Enrichment drives the cost because it fetches the homepage plus each nav link it finds, which came to five or six fetches per lead. A 10-lead run costs roughly 60 credits. You only pay for successful requests, so a blocked page costs nothing. The Zenrows pricing documentation has the details.

Can I use a different LLM?

You can swap the model behind the agent wrapper because the Zenrows tools are plain Python functions. extract_leads and score_lead depend on OpenAI's structured output API, so you'll need to replace those two calls if you switch providers.

Does this agent work on LinkedIn?

You shouldn't point it at LinkedIn. LinkedIn's user agreement prohibits automated access, and enforcement can lead to account restrictions. Point the agent at sources that permit automated access, such as public directories, business registries, and review platforms.

How does this compare to using Apollo or Clay?

Indexed datasets return the same companies when everyone queries with the same filters, and those filters are limited to the fields the dataset captures. This agent reads the source directly and scores each company on signals from its own site, so two teams with the same ICP can still surface different leads.

Build a web-aware TypeScript agent with Mastra and Zenrows

Benny — Wed, 07 Oct 2026 12:54:06 +0000

This article was originally published on the Zenrows blog.

In this tutorial, you'll build a Mastra agent with a custom Zenrows Fetch tool that reads protected and JavaScript-rendered pages, then add a workflow that fetches a list of URLs in parallel. You'll need Node.js 22.13 or later, an OpenAI or Anthropic API key, and a Zenrows API key.

The finished agent compares live Walmart product listings and returns either Markdown to reason over or structured JSON with fields like price and rating. The complete code is on GitHub.

Prerequisites

You need Node.js 22.13 or later.
You need an API key for one model provider. This tutorial uses OpenAI by default and shows the Anthropic alternative. Get an OpenAI key from your OpenAI developer dashboard or an Anthropic key from your Anthropic developer dashboard.
You need a Zenrows API key to retrieve the full contents of web pages. Create one from your Zenrows dashboard.

Why the built-in fetch tool falls short

Mastra's built-in webFetchTool makes plain HTTP requests, doesn't run JavaScript, and returns raw HTML. That's enough for simple pages. On browser-rendered or protected pages, such as a product listing protected by Cloudflare, it breaks quietly.

Cloudflare returns its challenge page with a normal HTTP status code, so the tool raises no error. The agent treats the challenge as the result and reports a missing or wrong price.

The fix belongs in the transport layer, which controls HTTP requests, headers, proxies, and whether JavaScript runs. Zenrows Fetch with mode=auto handles DataDome, Akamai, and Cloudflare in a single call. Wrapping that call in Mastra's createTool keeps typed input and output schemas around the fetch.

1. Set up the Mastra project

npm create mastra@latest

The CLI scaffolds the project structure, config files, and a starter agent, and installs @mastra/core and its dependencies. Follow the prompts to name the project and pick your LLM provider. Then add Zod if the scaffold didn't include it.

npm install zod

Create a .env file in the project root with your keys. You only need the key for the provider you plan to use.

ZENROWS_API_KEY=your_zenrows_key
OPENAI_API_KEY=your_openai_key
ANTHROPIC_API_KEY=your_anthropic_key

Add .env to .gitignore so your credentials stay out of version control. Mastra loads .env automatically, but the standalone test scripts in this tutorial run outside Mastra's loader, so install dotenv for them.

npm install dotenv --save

Your project structure should look similar to this.

src/
└── mastra/
    ├── agents/
    │   └── agent.ts
    ├── tools/
    │   └── schedule-tools.ts
    └── index.ts

Start the dev server.

npm run dev

Mastra Studio opens at http://localhost:4111. It reads the registered agents from your Mastra instance and gives you a chat UI to test them.

2. Build the Zenrows Fetch tool

Create src/mastra/tools/zenrows-fetch.ts next to the template's example tools.

import { createTool } from '@mastra/core/tools';
import { z } from 'zod';

export const zenrowsFetchTool = createTool({
  id: 'zenrows-fetch',

  description:
  'Fetches web pages using Zenrows. Use this tool when you need information from a specific URL. Set extractJson to true when you need structured data such as product details, pricing, availability, ratings, or other fields. Leave extractJson false when you need the page content as Markdown.',

  inputSchema: z.object({
    url: z.string().url(),
    extractJson: z.boolean().optional(),
  }),

  outputSchema: z.object({
    content: z.union([z.string(), z.record(z.string(), z.unknown())]),
    statusCode: z.number(),
    url: z.string(),
  }),

  execute: async ({ url, extractJson }) => {
    const zenrowsUrl = new URL('https://api.zenrows.com/v1/');

    zenrowsUrl.searchParams.set('url', url);
    zenrowsUrl.searchParams.set(
      'apikey',
      process.env.ZENROWS_API_KEY!,
    );
    zenrowsUrl.searchParams.set('mode', 'auto');

    if (extractJson) {
      zenrowsUrl.searchParams.set('extract', 'auto');
    } else {
      zenrowsUrl.searchParams.set('response_type', 'markdown');
    }

    const response = await fetch(zenrowsUrl);

    if (!response.ok) {
      const error = await response.text();

      throw new Error(
        `Zenrows request failed with status ${response.status}: ${error}`,
      );
    }

    if (extractJson) {
      const body = await response.json();

      return {
        content: body.parsed,
        statusCode: response.status,
        url,
      };
    }

    const content = await response.text();

    return {
      content,
      statusCode: response.status,
      url,
    };
  },
});

A createTool definition needs five parts. The id names the tool, and the description is the only signal the model uses to decide whether to call it, so keep it brief and exact. The inputSchema validates inputs, the outputSchema types the response, and execute calls Zenrows Fetch with mode=auto.

z.string().url() rejects malformed URLs before they reach the API. extractJson is optional and defaults to undefined, which falls through to the Markdown path.

Test the tool on its own with src/mastra/tools/test-tool.ts. It doesn't need the dev server or the agent.

import 'dotenv/config';
import { zenrowsFetchTool } from './zenrows-fetch';

if (!zenrowsFetchTool.execute) {
  throw new Error('Zenrows tool does not have an execute function');
}

const result = await zenrowsFetchTool.execute(
  {
    url: 'https://www.walmart.com/search?q=bike',
    extractJson: false,
  },
  {} as any
);

if (!result || 'statusCode' in result === false) {
  throw new Error('Zenrows tool returned an unexpected result');
}

console.log(result.statusCode);
console.log(
  typeof result.content === 'string'
    ? result.content.slice(0, 300)
    : result.content
);

Run it.

npx tsx src/mastra/tools/test-tool.ts

The output prints the status code, then the first lines of the page as Markdown.

200
[](https://www.walmart.com)

[Skip to Main Content](https://www.walmart.com#maincontent)

[](https://www.walmart.com/all-departments)[](https://www.walmart.com/)

[![Walmart](https://i5.walmartimages.com/dfw/63fd9f59-14e2/9d304ce6-96de-4331-b8ec-c5191226d378/v1/spark-icon.svg)](https://www.walmart.com/)

If you'd prefer to skip maintaining a custom tool, the Zenrows MCP server is an alternative integration. This walkthrough of the Zenrows MCP server and its implementation covers the setup.

3. Add structured extraction

// part of the previous script
zenrowsUrl.searchParams.set('mode', 'auto');

if (extractJson) {
  // server side parsing, returns fields instead of page text
  zenrowsUrl.searchParams.set('extract', 'auto');
} else {
  zenrowsUrl.searchParams.set('response_type', 'markdown');
}

// parses as JSON
if (extractJson) {
  const body = await response.json();

  return {
    content: body.parsed,
    statusCode: response.status,
    url,
  };
}

const content = await response.text();

return {
  content,
  statusCode: response.status,
  url,
};

These two excerpts from the tool handle the JSON path. When extractJson is true, the request sends extract=auto, and Zenrows Extract parses the page server side into named fields. The response branch then reads the body as JSON and returns body.parsed. When extractJson is false, the tool requests Markdown and reads the body as text.

Markdown suits pages the agent needs to read and reason over. Product listings work better as fields. Test the JSON path with src/mastra/tools/test-tool-extend-extract.ts.

import 'dotenv/config';
import { zenrowsFetchTool } from './zenrows-fetch';

if (!zenrowsFetchTool.execute) {
  throw new Error('Zenrows tool does not have an execute function');
}

const result = await zenrowsFetchTool.execute(
  {
    url: 'https://www.walmart.com/search?q=bike',
    extractJson: true,
  },
  {} as any,
);

if (!result || 'statusCode' in result === false) {
  throw new Error('Zenrows tool returned an unexpected result');
}

console.log(result.statusCode);
console.log(
  typeof result.content === 'string'
    ? result.content.slice(0, 300)
    : result.content
);

npx tsx src/mastra/tools/test-tool-extend-extract.ts

Here's the parsed response, truncated to two product records.

{
  "products": [
    {
      "badge": null,
      "current_price": 149.99,
      "fulfillment_label": null,
      "is_sponsored": true,
      "item_id": "10479873441",
      "low_stock": null,
      "options_from_price": 149.99,
      "original_price": 207.99,
      "rating": 3.9,
      "review_count": 130,
      "title": "Ktaxon 20\" Mountain Bike, 7 Speed Bike with Disc Brakes, White",
      "walmart_plus_savings": null
    },
    {
      "badge": null,
      "current_price": 227,
      "fulfillment_label": null,
      "is_sponsored": true,
      "item_id": "475992717",
      "low_stock": null,
      "options_from_price": null,
      "original_price": null,
      "rating": 4.4,
      "review_count": 739,
      "title": "Mongoose Rebel X1 BMX Bike, 20-in. Wheels, Kids Ages 7-14 Years, Gray Child Bicycle",
      "walmart_plus_savings": null
    }
  ],
  "search_query": {
    "query": "bike",
    "total_results": null
  }
}

4. Wire the tool into a Mastra agent

Create the agent at src/mastra/agents/market-research-agent.ts.

import { Agent } from '@mastra/core/agent';
import { zenrowsFetchTool } from '../tools/zenrows-fetch';

export const marketResearchAgent = new Agent({
  id: 'market-research-agent',
  name: 'Market Research Agent',

  description:
    'An agent that researches and compares products using live web data.',

  instructions: `
You are a product market research agent.

Your job is to research and compare products using the zenrows_fetch tool.

When the user provides one or more product URLs:

1. Fetch each URL using zenrows_fetch.
2. When product information such as name, price, availability, or rating is needed, use structured extraction.
3. Extract the relevant product information from each result.
4. Never invent or assume information that was not returned by the tool.
5. If information is missing, clearly state that it is unavailable.
6. Preserve the currency returned by the source.
7. Do not directly compare prices when the products use different currencies unless a reliable currency conversion is available.
8. When comparing products, present the results clearly in a table when appropriate.
9. After presenting the data, provide useful observations based only on the retrieved information.

Do not use web search when the user asks you to use zenrows_fetch only.
`,

  model: 'openai/gpt-5.6-terra',

  tools: {
    zenrows_fetch: zenrowsFetchTool,
  },
});

The instructions need to be specific. A vague prompt gives you a confident answer with a price from training data and no way to tell which fields came from the page. These instructions forbid invented values, require the agent to flag missing data, and tell it when to use structured extraction.

You can switch to any supported model by changing the model line. Make sure the matching key is in your .env file.

// changing model. Replace it in the previous script
// openai
model: 'openai/gpt-5.6-terra',

// anthropic
model: 'anthropic/claude-sonnet-4-5',

import { Mastra } from '@mastra/core/mastra';
import { LibSQLStore } from '@mastra/libsql';
import { DuckDBStore } from '@mastra/duckdb';
import { MastraCompositeStore } from '@mastra/core/storage';
import {
  MastraStorageExporter,
  MastraPlatformExporter,
  Observability,
  SensitiveDataFilter,
} from '@mastra/observability';
import { agent } from './agents/agent';
import { marketResearchAgent } from './agents/market-research-agent';
import { startScheduleTool, stopScheduleTool } from './tools/schedule-tools';

export const mastra = new Mastra({
  bundler: {
    externals: ['@duckdb/node-bindings'],
  },
  agents: {
    agent,
    marketResearchAgent,
  },
  tools: { startScheduleTool, stopScheduleTool },
  storage: new MastraCompositeStore({
    id: 'composite-storage',
    default: new LibSQLStore({
      id: 'mastra-storage',
      url: process.env.TURSO_DATABASE_URL || 'file:./mastra.db',
      authToken: process.env.TURSO_AUTH_TOKEN || undefined,
    }),
    domains: {
      observability: await new DuckDBStore().getStore('observability'),
    },
  }),
  observability: new Observability({
    configs: {
      default: {
        serviceName: 'mastra',
        exporters: [new MastraStorageExporter(), new MastraPlatformExporter()],
        spanOutputProcessors: [new SensitiveDataFilter()],
      },
    },
  }),
});

Test the agent with src/mastra/tools/test-agent.ts.

import 'dotenv/config';
import { mastra } from '..';

const agent = mastra.getAgent('marketResearchAgent');

const result = await agent.generate(
  'Compare the bikes on https://www.walmart.com/search?q=bike. Give me the five cheapest with their ratings, and flag any that are sponsored.',
);

console.log(JSON.stringify(result.toolCalls, null, 2));
console.log(result.text);

npx tsx src/mastra/tools/test-agent.ts

The script prints the tool call first. The agent chose structured extraction on its own.

[
  {
    "type": "tool-call",
    "runId": "018d9545-eddd-4fde-8cd1-1884316f4212",
    "from": "AGENT",
    "payload": {
      "toolCallId": "call_3H76SowFUi3SRknDFYXrJu7W",
      "toolName": "zenrows_fetch",
      "args": {
        "url": "https://www.walmart.com/search?q=bike",
        "extractJson": true
      },
      "providerMetadata": {
        "openai": {
          "itemId": "fc_06708f1f9910751e006a84956972bc81a084b7ebc5cc25f70d"
        }
      }
    }
  }
]

Then it prints the agent's answer.

The five lowest-priced bikes in the retrieved Walmart search results are:

| Rank | Bike | Price* | Rating | Reviews | Sponsored |
|---:|---|---:|---:|---:|---|
| 1 | Dynacraft 16 Inch Suspect Boys BMX Bike for Child 5-7 Years | 108 | 4.2/5 | 772 | No sponsorship flag returned |
| 2 | 18" Kent Bicycle Abyss Boy's Freestyle BMX Child Bicycle, Blue | 128 | 4.2/5 | 2,325 | No sponsorship flag returned |
| 3 | 20" Kent Tempest BMX Bicycle, Fits Riders 4'2"-5', Black/Aqua, Child, Unisex | 138 | 4.3/5 | 1,496 | No sponsorship flag returned |
| 4 | Ktaxon 24" Women's 7-Speed Cruiser Bike with Basket & Rack, Green | 189.99 | 4.2/5 | 78 | **Sponsored** |
| 5 | Mongoose Rebel X1 BMX Bike, 20-in. Wheels, Kids Ages 7-14, Gray | 227 | 4.4/5 | 740 | **Sponsored** |

The agent ranked the five cheapest bikes from the live page. Where is_sponsored came back null, it said so and didn't guess. This run happened at a different time from the earlier tool test, so some values differ.

5. Test the agent in Mastra Studio

npm run dev

Open http://localhost:4111, select the market research agent, and send it this prompt.

Compare the bikes on Walmart. Give me the five cheapest with their ratings, and flag any that are sponsored.

Studio shows the zenrows_fetch call the agent chose and the ranked table it built from the returned fields.

To add memory, follow the steps in the Mastra memory documentation.

6. Add a workflow for bulk URLs

Create src/mastra/workflow/bulk-fetch.ts.

import { createStep, createWorkflow } from '@mastra/core/workflows';
import { z } from 'zod';
import { zenrowsFetchTool } from '../tools/zenrows-fetch';

const prepareUrlsStep = createStep({
  id: 'prepare-urls',

  inputSchema: z.object({
    urls: z.array(z.string().url()),
  }),

  outputSchema: z.array(
    z.object({
      url: z.string().url(),
    }),
  ),

  execute: async ({ inputData }) => {
    return inputData.urls.map(url => ({
      url,
    }));
  },
});

const fetchProductStep = createStep({
  id: 'fetch-product',

  inputSchema: z.object({
    url: z.string().url(),
  }),

  outputSchema: z.object({
    url: z.string(),
    content: z.record(z.string(), z.unknown()),
    statusCode: z.number(),
  }),

  execute: async ({ inputData }) => {
    // createTool types execute as optional, so narrow it before calling
    if (!zenrowsFetchTool.execute) {
      throw new Error('Zenrows tool does not have an execute function');
    }

    const result = await zenrowsFetchTool.execute(
      {
        url: inputData.url,
        extractJson: true,
      },
      {} as any,
    );

    if (!result || 'statusCode' in result === false) {
      throw new Error('Zenrows tool returned an unexpected result');
    }

    return {
      url: inputData.url,
      // the tool returns a union, this workflow always extracts
      content: result.content as Record<string, unknown>,
      statusCode: result.statusCode,
    };
  },
});

export const productResearchWorkflow = createWorkflow({
  id: 'product-research-workflow',

  inputSchema: z.object({
    urls: z.array(z.string().url()),
  }),

  outputSchema: z.array(
    z.object({
      url: z.string(),
      content: z.record(z.string(), z.unknown()),
      statusCode: z.number(),
    }),
  ),
})
  .then(prepareUrlsStep)
  // concurrency controls how many urls run at once
  .foreach(fetchProductStep, { concurrency: 5 })
  .commit();

Mastra workflows are deterministic, so they suit a known list of URLs. prepare-urls turns the input list into one task per URL, and foreach runs fetch-product on five URLs at a time with structured extraction on.

createTool types execute as optional and as taking two arguments. The step narrows it before the call and passes both arguments, because calling it with the input alone doesn't compile.

// add to the imports
import { productResearchWorkflow } from './workflow/bulk-fetch';

// add to the Mastra constructor, alongside agents
workflows: {
  productResearchWorkflow,
},

Test it with src/mastra/workflow/test-workflow.ts.

import 'dotenv/config';
import { mastra } from '..';

const workflow = mastra.getWorkflow('productResearchWorkflow');

const run = await workflow.createRun();

const result = await run.start({
  inputData: {
    urls: [
      'https://www.walmart.com/search?q=tanktop',
      'https://www.walmart.com/search?q=shirts',
      'https://www.walmart.com/browse/home/shop-water-bottles/4044_623679_639999_7751805_4269055_2976621',
    ],
  },
});

console.log(JSON.stringify(result, null, 2));

npx tsx src/mastra/workflow/test-workflow.ts

The truncated output shows two of the three results. Every URL returned status 200, and the results come back in the same order as the input.

{
  "result": [
    {
      "url": "https://www.walmart.com/search?q=tanktop",
      "content": {
        "products": [
          {
            "badge": "rollback",
            "current_price": 11.98,
            "original_price": 29.98,
            "fulfillment_label": "Shipping, arrives today",
            "is_sponsored": true,
            "item_id": "342153793",
            "low_stock": true,
            "rating": 4.5,
            "review_count": 15482,
            "title": null
          }
          // 8 more products
        ],
        "search_query": { "query": "tanktop", "total_results": null }
      },
      "statusCode": 200
    },
    {
      "url": "https://www.walmart.com/browse/home/shop-water-bottles/...",
      "content": {
        "products": [
          {
            "badge": null,
            "current_price": 15.95,
            "original_price": 17.98,
            "is_sponsored": null,
            "item_id": "1604723",
            "rating": 4.7,
            "review_count": 19927,
            "title": "Brita Standard Replacement Water Filter (3 Pack)"
          }
          // 11 more products
        ],
        "search_query": null
      },
      "statusCode": 200
    }
  ]
}

You can also run the workflow from Mastra Studio.

Wrapping up

You now have a Mastra agent backed by Zenrows that fetches data from protected sites and returns structured results. The Mastra documentation covers more advanced patterns, and the Zenrows MCP server gives you a different integration path.

What's next

Add memory so the agent remembers earlier comparisons.
Move large URL lists to Zenrows Batch so the agent stops firing every request at once. The Batch documentation walks through your first job.
See the same fetch pattern in Python with this guide to an AI lead generation agent with the OpenAI Agents SDK.

FAQ

Why does Mastra's built-in webFetchTool fail on some sites?

It sends HTTP requests without a browser and without anti-bot handling. Cloudflare-protected targets return a challenge page or block page and raise no exception, so the agent reasons over the challenge copy or empty markup and returns an incomplete answer.

Does the Zenrows tool work with both OpenAI and Anthropic models?

It does. Only the agent's model line changes when you switch providers. The tool definition stays the same because Mastra passes your Zod inputSchema to the AI SDK in whatever tool-calling format the provider expects.

How do I handle rate limits when the agent calls the tool frequently?

Handle 429 responses by limiting concurrency, queueing requests, and retrying with exponential backoff and jitter. For large workloads, send the URLs through Zenrows Batch so the agent doesn't fire every request at once.

Can I connect Zenrows through MCP?

You can. Zenrows has an official MCP server, and Mastra's MCPClient works with it like any other MCP server. It supports Streamable HTTP and local transport and gives agents Fetch and browser tools without a wrapper. You still need an API key.

Web data for LLM fine-tuning: a practical 2026 guide

Benny — Tue, 08 Sep 2026 14:01:42 +0000

This article was originally published on the Zenrows blog. Read the original here: https://www.zenrows.com/blog/web-data-llm-fine-tuning

This guide shows you how to collect a real-data seed for LLM fine-tuning from protected web sources using Zenrows Fetch, format it as instruction-output pairs, validate it, and measure whether the fine-tune held. You need Python 3.9+, a Zenrows API key, and requests and python-dotenv installed.

All code is on GitHub.

Before you start

Python 3.9 or later
Zenrows account for your API key
pypdf if you are collecting from PDFs

python3 -m pip install requests python-dotenv pypdf

# .env
ZENROWS_API_KEY=your_zenrows_api_key_here

Why a real-data seed matters

In a fine-tuning set of a few hundred rows, one mislabeled extraction is a measurable percentage of your entire training signal. Synthetic-only datasets amplify teacher model biases through each retrain cycle. The real-data seed anchors the generator and breaks the drift loop.

Authoritative sources — documentation, regulatory filings, expert-reviewed publications — are almost always behind access controls. Standard crawlers get flagged on the TLS handshake. Zenrows Fetch handles rendering and access in a single call.

Step 1: Collect clean Markdown from protected sources

mode=auto is Adaptive Stealth Mode. Zenrows picks the retrieval strategy per target — JavaScript rendering, proxy escalation, or direct fetch — without configuration on your end.

import os
import requests
from dotenv import load_dotenv

load_dotenv()

apikey = os.getenv("ZENROWS_API_KEY")
url = "https://www.scrapingcourse.com/ecommerce/"

params = {
    "url": url,
    "apikey": apikey,
    "mode": "auto",
    "response_type": "markdown",  # clean Markdown, not raw HTML
}

response = requests.get("https://api.zenrows.com/v1/", params=params)
print(response.text)

Step 2: Handle PDFs and non-HTML sources

Use response_type=pdf for PDFs. js_render=true is required for PDF responses.

params = {
    "url": url,
    "apikey": apikey,
    "js_render": "true",    # required for PDF responses
    "response_type": "pdf", # returns raw bytes, not text
}

response = requests.get("https://api.zenrows.com/v1/", params=params)

with open("source-document.pdf", "wb") as f:
    f.write(response.content)

Extract text and chunk by structure, not by character count.

from pypdf import PdfReader
import io

reader = PdfReader(io.BytesIO(response.content))

def chunk_by_page(reader, max_chars=6000):
    chunks = []
    for page in reader.pages:
        text = page.extract_text()
        if not text or not text.strip():
            continue  # skip blank or image-only pages
        if len(text) <= max_chars:
            chunks.append(text)
        else:
            for i in range(0, len(text), max_chars):
                chunks.append(text[i:i + max_chars])
    return chunks

chunks = chunk_by_page(reader)
print(f"Produced {len(chunks)} chunks from {len(reader.pages)} pages")

Step 3: Track provenance

Every row in the seed needs a source, retrieval date, and license.

from datetime import date

source = {
    "url": pdf_url,
    "retrieved": date.today().isoformat(),
    "license": license_url,  # check the source's terms before collecting at scale
}

Step 4: Run batch collection

Match concurrency to your plan limit. time.sleep(0.5) spaces out requests per worker.

import time
from concurrent.futures import ThreadPoolExecutor

MAX_CONCURRENT = 50  # check your Zenrows dashboard for your plan's limit

def fetch(url):
    params = {
        "url": url,
        "apikey": apikey,
        "response_type": "markdown",
        "mode": "auto",
        "wait": 3000,
    }
    response = requests.get("https://api.zenrows.com/v1/", params=params)
    time.sleep(0.5)  # pace requests within each worker

    if response.status_code != 200:
        return {"url": url, "status": response.status_code, "content": None}
    return {"url": url, "status": response.status_code, "content": response.text}

urls = [
    "https://playboard.co",
    "https://www.samsung.com",
    "https://www.kickstarter.com",
]

with ThreadPoolExecutor(max_workers=MAX_CONCURRENT) as executor:
    results = list(executor.map(fetch, urls))

Step 5: Format as instruction-output pairs

The Markdown is the input. The extracted JSON is the output. Write one JSON object per line in JSONL format.

import json

def to_training_example(instruction, markdown_content, extracted_json):
    return {
        "messages": [
            {"role": "system", "content": instruction},
            {"role": "user", "content": markdown_content},
            {"role": "assistant", "content": json.dumps(extracted_json)},
        ]
    }

instruction = "Extract the product name, price, and availability as JSON."

with open("training_data.jsonl", "w") as f:
    for markdown_content, extracted_json in seed_examples:
        example = to_training_example(instruction, markdown_content, extracted_json)
        f.write(json.dumps(example) + "\n")

Step 6: Validate before training

Every row must clear three checks before it enters the training set — length, schema, and duplicate hash.

import hashlib
from jsonschema import validate, ValidationError

schema = {
    "type": "object",
    "properties": {
        "name": {"type": "string", "minLength": 1},
        "price": {"type": "string", "minLength": 1},
    },
    "required": ["name", "price"],
}

seen = set()

def keep(markdown, extracted):
    if len(markdown.strip()) < 200:
        return False  # page rendered to almost nothing
    try:
        validate(instance=extracted, schema=schema)
    except ValidationError:
        return False  # wrong types or missing fields
    h = hashlib.sha256(" ".join(markdown.lower().split()).encode()).hexdigest()
    if h in seen:
        return False  # duplicate or near-duplicate
    seen.add(h)
    return True

clean_examples = [(url, md, js) for url, md, js in seed_examples if keep(md, js)]
print(f"Collected {len(seed_examples)} pages, {len(clean_examples)} passed validation")

Step 7: Hold out real data and measure

Set aside 10 to 20 percent before any synthetic expansion and keep it out of training permanently.

import random

random.seed(42)
random.shuffle(seed_examples)

split = int(len(seed_examples) * 0.85)
train_examples = seed_examples[:split]
holdout_examples = seed_examples[split:]  # never touches the training set

After fine-tuning, run the model against the holdout and measure task accuracy. Record the score after each retrain. A declining score is an early sign of drift.

What's next

How to extract web data for AI training — broader context on training data collection
Zenrows Batch — managed concurrency and retries for large URL lists
Zenrows MCP server — web access for agents without managing the collection layer

Build a web-aware agent with OpenAI Agents SDK and Zenrows

Benny — Fri, 04 Sep 2026 07:27:21 +0000

This article was originally published on the Zenrows blog. Read the original here: https://www.zenrows.com/blog/web-aware-agent-openai-agents-sdk-zenrows

This guide shows you how to register Zenrows Fetch as a @function_tool next to WebSearchTool in a single OpenAI Agents SDK agent, so the model routes between discovery and live-page retrieval on its own each turn. You need Python 3.9+, an OpenAI API key, and a Zenrows API key.

All the code is on GitHub.

Before you start

Python 3.9 or later
An OpenAI API key from your OpenAI developer dashboard
A Zenrows API key from your Zenrows dashboard

Install dependencies:

python3 -m pip install openai-agents python-dotenv

Add both keys to a .env file and add it to .gitignore:

OPENAI_API_KEY=your_openai_api_key_here
ZENROWS_API_KEY=your_zenrows_api_key_here

Load them in your script:

from dotenv import load_dotenv

load_dotenv()

Set up a baseline agent with the built-in search

WebSearchTool is a hosted tool that runs on OpenAI's infrastructure. Enabling it takes one line.

import asyncio
from dotenv import load_dotenv
from agents import Agent, Runner, WebSearchTool

load_dotenv()

agent = Agent(
    name="Product Researcher",
    instructions=(
        "You research products and web pages. "
        "Report exactly what you find, and state plainly when "
        "specific information is missing from your results."
    ),
    tools=[WebSearchTool()],
)

async def main():
    result = await Runner.run(
        agent,
        "Which laptops did Apple release most recently, and where can I buy them?",
    )
    print(result.final_output)

asyncio.run(main())

This works for broad discovery queries. The agent returns relevant pages with source links. The gap appears on live or protected targets.

Where the built-in search stops

Issue 1: indexed snippets, not live pages

Built-in search returns OpenAI's cached copy of a page, not the current version. For documentation, that's usually fine. For prices, stock levels, or anything that changes frequently, the cached copy lags.

Issue 2: JavaScript-rendered content

Many pages ship an HTML shell and fill in prices, inventory, and reviews via JavaScript at runtime. The search index stores the shell. The data you want was never captured.

import asyncio
from dotenv import load_dotenv
from agents import Agent, Runner, WebSearchTool

load_dotenv()

agent = Agent(
    name="Product Researcher",
    instructions=(
        "You retrieve product details from web pages. "
        "Report exactly what you find, and say so plainly if content is missing."
    ),
    tools=[WebSearchTool()],
)

async def main():
    result = await Runner.run(
        agent,
        "What is the current price and stock status of "
        "https://www.amazon.com/dp/B0GR1JKMBV/ref=fs_a_mbt2_us1?th=1",
    )
    print(result.final_output)

asyncio.run(main())

The agent finds the page. It returns the product title and a clear statement that the price wasn't accessible:

I visited the Amazon product page for ASIN B0GR1JKMBV...
The page displays: "To see product details, add this item to your cart."
Therefore I could not retrieve the current price or availability.

Add Zenrows Fetch as a @function_tool

import os
import requests
from agents import function_tool

@function_tool
def fetch_page_content(url: str) -> str:
    """Fetch the full, current content of a specific web page as Markdown.

    Use this when you need the complete content of a known URL rather than
    a search result summary, such as live prices, stock levels, or data
    that loads via JavaScript after the page opens.

    Args:
        url: The full URL of the page to retrieve.
    """
    response = requests.get(
        "https://api.zenrows.com/v1/",
        params={
            "url": url,
            "apikey": os.getenv("ZENROWS_API_KEY"),
            # mode=auto lets Zenrows pick the right settings per site
            "mode": "auto",
            "response_type": "markdown",
        },
        timeout=90,
    )

    # return the error as a string so the agent can react instead of crashing
    if response.status_code != 200:
        return f"Zenrows returned {response.status_code} for {url}"

    return response.text

Four things to note:

@function_tool builds the schema from the function signature automatically. No JSON spec to write.
mode=auto activates Adaptive Stealth Mode: JavaScript rendering and premium proxy escalation only when the target actually needs them.
response_type=markdown strips markup so the model reads clean text.
The docstring is what the model reads every turn to decide whether to call this tool. The phrase "rather than a search result summary" is what routes it away from built-in search when you have a specific URL.

Run both tools in a single agent loop

import asyncio
import os
import requests
from dotenv import load_dotenv
from agents import Agent, Runner, WebSearchTool, function_tool

load_dotenv()

@function_tool
def fetch_page_content(url: str) -> str:
    """Fetch the full, current content of a specific web page as Markdown.

    Use this when you need the complete content of a known URL rather than
    a search result summary, such as live prices, stock levels, or data
    that loads via JavaScript after the page opens.

    Args:
        url: The full URL of the page to retrieve.
    """
    response = requests.get(
        "https://api.zenrows.com/v1/",
        params={
            "url": url,
            "apikey": os.getenv("ZENROWS_API_KEY"),
            "mode": "auto",
            "response_type": "markdown",
        },
        timeout=90,
    )

    if response.status_code != 200:
        return f"Zenrows returned {response.status_code} for {url}"

    return response.text

# the model routes between both tools from their descriptions alone
agent = Agent(
    name="Web-Aware Researcher",
    instructions=(
        "You research products on the web. "
        "Use web search to find the product page URL. "
        "Then call fetch_page_content on that URL and report the price "
        "and stock status from the fetched page content. "
        "State which tool each figure came from."
    ),
    tools=[WebSearchTool(), fetch_page_content],
)

def print_trace(items):
    for item in items:
        raw = getattr(item, "raw_item", None)
        name = getattr(raw, "name", None) or getattr(raw, "type", "")
        print(f"[{item.type}] {name}")

        if item.type == "tool_call_item" and name == "fetch_page_content":
            print("  args:", getattr(raw, "arguments", ""))

        if item.type == "tool_call_output_item":
            out = str(getattr(item, "output", ""))
            print(f"  output: {len(out)} chars")
            print(f"  body: {out[:500]}")

async def main():
    result = await Runner.run(
        agent,
        "Find the PriceOye page for the iPhone 17 Pro "
        "and report its current price and stock status.",
    )

    print_trace(result.new_items)
    print("\n---\n")
    print(result.final_output)

asyncio.run(main())

The trace shows the two-step routing. Turn one: web_search_call for discovery. Turn two: fetch_page_content with the URL it found.

[tool_call_item] web_search_call
[message_output_item] message
[tool_call_item] fetch_page_content
  args: {"url":"https://priceoye.pk/mobiles/apple/apple-iphone-17-pro"}
[tool_call_output_item]

---

Here are the details for the iPhone 17 Pro from its PriceOye product page:

Current Price: Rs 471,999
Stock Status: Only 1 left in stock

Price and stock status are directly extracted from the fetched page content
(functions.fetch_page_content tool).

No conditionals. No orchestration code. The routing lives entirely in the tool descriptions.

When to use each tool

Use WebSearchTool for discovery: finding pages, open-ended research, answering questions from indexed public content.

Use Zenrows as the retrieval layer when you have a specific URL and need the live contents — protected targets, JavaScript-rendered pages, anything where freshness matters.

For structured field extraction instead of full-page Markdown, use Zenrows Extract. For high-volume or scheduled retrieval across many URLs, use Zenrows Batch.

The same pattern carries over to multi-agent setups: building a web research multi-agent system with AG2 and Zenrows. And to smolagents, under a @tool decorator: Zenrows smolagents guide.

Conclusion

You now have an agent with two retrieval paths. WebSearchTool handles discovery. Zenrows reads the full live page once the URL is known, including JavaScript-rendered and anti-bot-defended targets. The model routes between them from the tool descriptions alone.

What's next

Zenrows Batch for high-volume or scheduled retrieval
Zenrows MCP server if you prefer ready-made tools over a custom wrapper
OpenAI Agents SDK integration guide for the full reference

To read

Benny — Thu, 03 Sep 2026 16:30:00 +0000

Prosper Otemuyiwa for Valyu AI

Aug 26

Investment Research APIs in 2026: 7 DeepResearch Workflows for Deal Teams

#ai #api #finance #python

12 min read

[Boost]

Benny — Thu, 03 Sep 2026 16:27:11 +0000

Prosper Otemuyiwa for Valyu AI

Aug 26

Investment Research APIs in 2026: 7 DeepResearch Workflows for Deal Teams

#ai #api #finance #python

12 min read