DEV Community: Benny

How to Build a Price Monitoring Agent with Pydantic AI and ZenRows

Benny — Wed, 22 Jul 2026 20:46:40 +0000

Most price monitoring systems look simple on paper: point a scraper at a product page, ask an LLM to extract the price, and store the result in a database. The problem arises when the same workflow runs continuously against heavily protected targets such as Amazon and Walmart. At scale, reliability becomes the real challenge. The why comes down to two failures that happen quietly: retrieval and extraction. Retrieval fails when a protected site returns a blank JavaScript shell, incomplete data, and a block page instead of the product data. Extraction fails when the LLM returns a field in a data type or shape that your downstream database write action does not expect. Both failures pass for normal output, so the pipeline keeps running and drops records downstream without raising an error.

This tutorial shows you how you can build a price monitoring agent using ZenRows and Pydantic AI. ZenRows retrieves the page and returns product details with a 99.93% success rate against protected websites, while Pydantic AI extracts and validates LLM output against a defined schema. The tutorial walks through the two-step pipeline from a single product page to a multi-site price monitoring run across Amazon and Walmart, two heavily protected e-commerce sites.

The two failures in a price monitoring pipeline

Two failure modes break the price monitoring pipeline: retrieval and extraction.

1. Retrieval returns a block page

You send a request to the protected site, and the response looks successful at the HTTP level. But the page still contains no usable product data or encounters an error 1020. The code below shows this failed retrieval directly.

import requests

# Demo page that behaves like a protected target
PROTECTED_URL = "https://www.scrapingcourse.com/antibot-challenge"

resp = requests.get(PROTECTED_URL, timeout=30)

print(f"status code: {resp.status_code}")
print(f"bytes returned: {len(resp.text)}")
print("contains 'price'?", "price" in resp.text.lower())
print(resp.text[:200])

The request returns a 403, so the pipeline does not receive the product page required for price monitoring.

benny@Mac price_monitoring % python3 test.py    
status code: 403
bytes returned: 5507
contains 'price'? False
<!DOCTYPE html><html lang="en-US"><head><title>Just a moment...</title><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta http-equiv="X-UA-Compatible" content="IE=Edge"><meta nam
benny@Mac price_monitoring %

When this happens, your pipeline would assume that your script has received data and move on. But without any prices, you can't monitor the pages for any product.

2. Extraction returns inconsistent information

Your first run can return data successfully. Across runs, the same field can drift into different formats. For example, one run may return a price with currency symbols attached. Another may return the value as text, and the third as a number. The example below demonstrates how an LLM behaves when this type of data drift occurs.

# Simulated raw LLM extraction responses from two runs

raw_llm_responses = [
    {
        "product": "Echo Dot (5th Gen)",
        "price": 29.99
    },
    {
        "product": "Echo Dot (5th Gen)",
        "price": "$29.99"
    }
]

def write_to_db(price: float) -> None:
    if not isinstance(price, float):
        raise TypeError(
            f"expected float, got {type(price).__name__}"
        )


for i, response in enumerate(raw_llm_responses, start=1):
    price = response["price"]

    print(
        f"run {i}: price={price!r}, "
        f"type={type(price).__name__}"
    )


for i, response in enumerate(raw_llm_responses, start=1):
    try:
        write_to_db(response["price"])
        print(f"run {i}: write OK")

    except TypeError as e:
        print(f"run {i}: write FAILED -> {e}")

Below is the output of the extraction script. It returns valid-looking data, but the output schema can drift between runs. Without validation, downstream writes fail when the data type does not match the expected format.

benny@Mac price_monitoring % python3 test_llm.py
run 1: price=29.99, type=float
run 2: price='$29.99', type=str
run 1: write OK
run 2: write FAILED -> expected float, got str
benny@Mac price_monitoring %

Such inconsistencies can break validation logic and downstream workflows, causing pipelines to fail unpredictably during automated or scheduled executions. Next, let's take a look at how ZenRows handles the retrieval problem. However, before we start, let's see everything you need to follow along.

Prerequisites for this tutorial

Use Python 3.10 or newer for this tutorial. If you have an older Python version, you need to create a dedicated virtual environment with a current Python installation to avoid dependency conflicts.

Install the required packages using python -m pip install "pydantic-ai-slim[anthropic]" requests python-dotenv. This tutorial was tested locally with Pydantic AI 0.8.1 and Anthropic SDK 0.111.0.
ZenRows API key. Create an account at zenrows.com and copy your key from the dashboard. This key authenticates every scrape the researcher runs.
Anthropic API key. The extraction agent uses Claude to read the scraped markdown and return a validated record. Generate a key in the Anthropic Console.
Create a .env file and save your API keys there.

Setting up ZenRows for e-commerce retrieval

One way to address retrieval failures is to use a scraping API to retrieve data from protected pages and pass the page content to the extraction step.

ZenRows web scraping API handles the retrieval layer for protected pages. It renders the pages, uses proxies, and returns markdown in a single request. You can then pass this output to your extraction agent for the next stage of your pipeline, which you will see in the next section of this piece. The code below shows this flow with js_render and premium_proxy.

import os
import requests
from dotenv import load_dotenv

load_dotenv()

ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]

# Target page (NO markdown formatting)
url = "https://www.scrapingcourse.com/ecommerce/"

params = {
    "url": url,
    "apikey": ZENROWS_API_KEY,
    "js_render": "true",
    "premium_proxy": "true",
    "proxy_country": "us",
    "response_type": "markdown",
}

response = requests.get(
    "https://api.zenrows.com/v1/",
    params=params,
    timeout=60
)

print(f"status code: {response.status_code}")
print(f"bytes returned: {len(response.text)}")
print("\n".join(response.text.splitlines()[:30]))

ZenRows manages anti-bot bypass via "js_render": "true" which ensures ZenRows loads the page like a real user browser, and "premium_proxy": "true" which enables residential proxies. For this use case, set proxy_country=us to route requests through a US IP address. That keeps our price comparisons consistent across runs. The response type is markdown, which is easier for LLMs to parse and read. Your response after running the code should be similar to the output below.

benny@Mac price_monitoring % python3 zenrows_store.py
status code: 200
bytes returned: 6777
[Skip to navigation](http://www.scrapingcourse.com#site-navigation) [Skip to content](http://www.scrapingcourse.com#content)

[Ecommerce Test Site to Learn Web Scraping](https://www.scrapingcourse.com/ecommerce/)

ScrapingCourse.com

Search for: Search

Menu

- [Shop](https://www.scrapingcourse.com/ecommerce/)

<!--THE END-->

- [Home](https://www.scrapingcourse.com/ecommerce/)
- [Cart](https://www.scrapingcourse.com/ecommerce/cart/)
- [Checkout](https://www.scrapingcourse.com/ecommerce/checkout/)
- [My account](https://www.scrapingcourse.com/ecommerce/my-account/)

<!--THE END-->

- [$0.00 0 items](https://www.scrapingcourse.com/ecommerce/cart/ "View your shopping cart")
- No products in the cart.

# Shop

Default sorting Sort by popularity Sort by latest Sort by price: low to high Sort by price: high to low

Showing 1-16 of 188 results

benny@Mac price_monitoring % ;

Now, let's define your price schema to ensure your LLM reads the extracted content correctly and maps each product field into a predictable format for downstream processing.

Defining the price schema and building the extraction agent

Now that ZenRows has retrieved the data from the website, Pydantic AI turns the markdown into a validated price record. You define the schema first, then run the agent against it. For a price monitoring agent, Pydantic protects your pipeline from messy extraction output.

You will start by defining your schema, which acts as a contract between the extraction layer and the rest of your pipeline. In the code below, that is PriceRecord. After that, you build the agent against the schema. So if the field has the wrong type, Pydantic AI reprompts the model with the validation error and returns the right output.

from datetime import datetime, timezone
from pydantic import BaseModel, Field
from dotenv import load_dotenv

load_dotenv()

class PriceRecord(BaseModel):
    name: str
    price: float
    currency: str
    marketplace: str
    availability: str
    scraped_at: datetime = Field(default_factory=lambda: datetime.now(timezone.utc))

from pydantic_ai import Agent

price_agent = Agent(
    "anthropic:claude-sonnet-4-6",
    output_type=PriceRecord,
    instructions=(
        "Extract the product price record from the Markdown. "
        "price is a number with no currency symbol. "
        "currency is the ISO code, for example USD. "
        "marketplace is the site name, for example Amazon."
    ),
)

Now, let's run it against our ZenRows output.

import os
import requests
from datetime import datetime, timezone
from typing import Optional

from dotenv import load_dotenv
from pydantic import BaseModel, Field
from pydantic_ai import Agent


load_dotenv()

ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]

PRODUCT_URL = "https://www.scrapingcourse.com/ecommerce/"


# --- Step 1 retrieve with ZenRows ---

def fetch_markdown(url: str) -> Optional[str]:
    params = {
        "url": url,
        "apikey": ZENROWS_API_KEY,
        "js_render": "true",
        "premium_proxy": "true",
        "proxy_country": "us",
        "response_type": "markdown",

    }

    resp = requests.get(
        "https://api.zenrows.com/v1/",
        params=params,
        timeout=60,
    )

    if resp.status_code != 200:
        print(f"retrieval blocked: {resp.status_code}")
        return None

    return resp.text


# --- Step 2 schema + agent ---

class PriceRecord(BaseModel):
    name: str
    price: float
    currency: str
    marketplace: str
    availability: str
    scraped_at: datetime = Field(
        default_factory=lambda: datetime.now(timezone.utc)
    )


price_agent = Agent(
    "anthropic:claude-sonnet-4-6",
    output_type=PriceRecord,
    instructions=(
        "Extract the product price record from the Markdown. "
        "price is a number with no currency symbol. "
        "currency is the ISO code, for example USD. "
        "marketplace is the site name, for example ScrapingCourse."
    ),
)


# --- Step 3 run pipeline ---

markdown = fetch_markdown(PRODUCT_URL)

if markdown is None:
    raise Exception("No markdown returned from ZenRows")


print(f"markdown length: {len(markdown)}")


result = price_agent.run_sync(markdown)

record = result.output


print("\n--- extracted record ---")
print(record.model_dump())


print("\n--- type checks ---")
print(
    "price is float? ",
    isinstance(record.price, float),
    "->",
    repr(record.price)
)

print(
    "name is str?    ",
    isinstance(record.name, str)
)

print(
    "scraped_at set? ",
    record.scraped_at.isoformat()
)

From the output below, you can see that the ZenRows returned page content instead of a block or challenge response, hence the markdown length of 6750. The LLM also converted the messy markdown into your expected fields and shows the schema validation.

venv) benny@Mac price_monitoring % python zenrows_pydantic.py
markdown length: 6750

--- extracted record ---
{'name': 'Abominable Hoodie', 'price': 69.0, 'currency': 'USD', 'marketplace': 'ScrapingCourse', 'availability': 'In Stock', 'scraped_at': datetime.datetime(2026, 6, 22, 23, 5, 8, 154880, tzinfo=datetime.timezone.utc)}

--- type checks ---
price is float?  True -> 69.0
name is str?     True
scraped_at set?  2026-06-22T23:05:08.154880+00:00
(venv) benny@Mac price_monitoring %

Running the pipeline across multiple marketplaces

The script below uses the same retrieve-then-extract framework as in the previous two sections. However, now it is a loop over multiple URLs, collecting a validated PriceRecord from each site.

To demonstrate multi-marketplace price monitoring and ZenRows' capability, this tutorial runs the workflow against Amazon and Walmart. Amazon and Walmart are among the most aggressively protected e-commerce sites on the web, making them a meaningful test of retrieval reliability. Successfully extracting product data from these targets demonstrates ZenRows' ability to handle real-world protected websites, not just scraper-friendly demo pages.

import os
import requests
from datetime import datetime, timezone
from typing import Optional

from dotenv import load_dotenv
from pydantic import BaseModel, Field
from pydantic_ai import Agent, UnexpectedModelBehavior


load_dotenv()

ZENROWS_API_KEY = os.getenv("ZENROWS_API_KEY", "")

MIN_MARKDOWN_LENGTH = 200  # a real product page dwarfs any block or error blob


# The same product across three marketplaces, one product page per URL,
# all on US domains so the prices share a currency.
TARGETS = [
    ("Amazon",  "https://www.amazon.com/Apple-iPhone-Version-256GB-Titanium/dp/B0DHJDPYYR"),
    ("Walmart", "https://www.walmart.com/ip/Restored-Apple-iPhone-16-Pro-Carrier-Unlocked-256GB-Black-Titanium-Refurbished/13323059232"),
]


# --- Step 1: retrieve with ZenRows ---

def fetch_markdown(url: str) -> Optional[str]:
    params = {
        "url": url,
        "apikey": ZENROWS_API_KEY,
        "js_render": "true",
        "premium_proxy": "true",
        "proxy_country": "us",
        "response_type": "markdown",

    }
    resp = requests.get(
        "https://api.zenrows.com/v1/",
        params=params,
        timeout=60,
    )

    # ZenRows returns a non-200 when it cannot deliver the page.
    if resp.status_code != 200:
        return None

    markdown = resp.text.strip()

    # Too short to be a product page. A block or error blob lands here.
    if len(markdown) < MIN_MARKDOWN_LENGTH:
        return None

    # An error came back as JSON instead of Markdown.
    if markdown.startswith("{") and '"code"' in markdown:
        return None

    return markdown


# --- Step 2: schema + agent ---

class PriceRecord(BaseModel):
    name: str
    price: float
    currency: str
    marketplace: str
    availability: str
    scraped_at: datetime = Field(
        default_factory=lambda: datetime.now(timezone.utc)
    )


price_agent = Agent(
    "anthropic:claude-sonnet-4-6",
    output_type=PriceRecord,
    instructions=(
        "Extract the product price record from the Markdown. "
        "price is a number with no currency symbol. "
        "currency is the ISO code, for example USD."
    ),
)


# --- Step 3: run the pipeline across every marketplace ---

def run_monitor() -> tuple[list[PriceRecord], int]:
    results: list[PriceRecord] = []
    failed = 0

    for marketplace, url in TARGETS:
        print(f"{marketplace}: {url}")

        # Retrieval failure: ZenRows returns a blocked response. Log and skip.
        markdown = fetch_markdown(url)
        if markdown is None:
            print("  retrieval blocked, skipping")
            failed += 1
            continue

        # Extraction failure: the model never returns a valid record and
        # Pydantic AI exhausts its retries. Log and skip.
        try:
            result = price_agent.run_sync(
                f"Marketplace: {marketplace}\n\n{markdown}"
            )
        except UnexpectedModelBehavior as e:
            print(f"  extraction failed after retries, skipping ({e})")
            failed += 1
            continue

        record = result.output
        record.marketplace = marketplace   # use the known label, not the LLM's casing
        results.append(record)
        print(f"  ok -> {record.name} {record.price} {record.currency}")

    return results, failed


if __name__ == "__main__":
    records, failed_count = run_monitor()

    print(f"\nvalidated records: {len(records)}")
    print(f"failed URLs: {failed_count}")

    for record in records:
        print(record.model_dump())

    # The decision: cheapest marketplace first.
    print()
    for record in sorted(records, key=lambda r: r.price):
        print(f"{record.marketplace:<10} {record.price} {record.currency}")

From our output below, you will notice that both sites ran and the product's information was retrieved from Amazon and Walmart. This tells us how reliable ZenRows is for retrieval and scraping.

venv) benny@Mac price_monitoring % python zenrows_multi_copy.py
/Users/benny/Desktop/price_monitoring/venv/lib/python3.9/site-packages/urllib3/__init__.py:35: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020
  warnings.warn(
Amazon: https://www.amazon.com/Apple-iPhone-Version-256GB-Titanium/dp/B0DHJDPYYR
  ok -> Apple iPhone 16 Pro, 256GB, Black Titanium - Unlocked (Renewed) 745.93 USD
Walmart: https://www.walmart.com/ip/Restored-Apple-iPhone-16-Pro-Carrier-Unlocked-256GB-Black-Titanium-Refurbished/13323059232
  ok -> Restored Apple iPhone 16 Pro - Carrier Unlocked - 256GB Black Titanium (Refurbished) 697.87 USD

validated records: 2
failed URLs: 0
{'name': 'Apple iPhone 16 Pro, 256GB, Black Titanium - Unlocked (Renewed)', 'price': 745.93, 'currency': 'USD', 'marketplace': 'Amazon', 'availability': 'In Stock', 'scraped_at': datetime.datetime(2026, 6, 22, 23, 27, 51, 184428, tzinfo=datetime.timezone.utc)}
{'name': 'Restored Apple iPhone 16 Pro - Carrier Unlocked - 256GB Black Titanium (Refurbished)', 'price': 697.87, 'currency': 'USD', 'marketplace': 'Walmart', 'availability': 'In Stock', 'scraped_at': datetime.datetime(2026, 6, 22, 23, 28, 6, 863848, tzinfo=datetime.timezone.utc)}

Walmart    697.87 USD
Amazon     745.93 USD
(venv) benny@Mac price_monitoring %

You can find the code snippets in this article in this GitHub repository.

Wrapping up

With ZenRows handling retrieval and Pydantic AI validating each extraction, this pipeline remains reliable even when working with protected targets like Amazon and Walmart, which traditional scrapers often fail to scrape or return incomplete data for. ZenRows' strength lies in its ability to reliably access protected sites, with a 99.93% success rate against sites that block everything else. The result is a list of typed product records across three marketplaces in a consistent shape, ready to insert into a database or feed an alerting layer without any official integration.

Try ZenRows for free to build more reliable price-monitoring workflows!

How to Build a Competitor Intelligence Agent with CrewAI and ZenRows

Benny — Thu, 16 Jul 2026 21:45:31 +0000

A competitor intelligence agent enables real-time pricing visibility, automated positioning tracking, and structured competitor briefs for brands without manual research. At the center of these workflows are a researcher agent responsible for gathering data and an analysis agent responsible for generating a summary and a downstream report.

However, one of the things that makes the researcher agent's output trustworthy is its retrieval layer, since a poor retrieval-layer output can make the analysis agent's recommendation questionable. If the researcher agent searches the webpage and retrieves a challenge page, an empty response, or blocked content, every downstream conclusion becomes less trustworthy. With a 99.93% success rate on protected websites, ZenRows provides the reliable retrieval layer that makes these workflows practical in production.

This tutorial shows why a CrewAI researcher agent can fail on protected competitor pages. It starts with a custom ZenRows-based tool setup, then later shows the MCP server as an alternative approach.

Prerequisites

This tutorial works best with Python 3.10 or newer. If you are on an older Python version, create a dedicated virtual environment with a current Python installation to avoid dependency conflicts.

Python 3.10 to 3.13. CrewAI requires this range.
ZenRows API key. Create an account at zenrows.com and copy your key from the dashboard. This key authenticates every scrape the researcher agent runs for data extraction.
Anthropic API key. The crew uses Claude to drive both agents. Generate a key in the Anthropic Console.
Install dependencies and the required packages using pip install "crewai[anthropic]" crewai-tools zenrows python-dotenv.
Create a .env file and save your API keys there.

Why the built-in ScrapeWebsiteTool fails on competitor pages

The problem, as established earlier, starts before the analysis. A CrewAI workflow depends primarily on the information collected, because a competitor intelligence agent relies on a chain of steps. The researcher AI agent collects information, the analysis agent interprets it, and the final output is built from what the researcher agent retrieved. This means the scraping layer becomes the foundation of the entire workflow. If the researcher agent receives incomplete content, every other agent working with it will work with the wrong data.

CrewAI is an open-source framework that enables faster prototyping of AI-driven data extraction pipelines by orchestrating autonomous multi-agent systems. One of its features is CrewAI's built-in ScrapeWebsiteTool, which, though designed to extract and read website URL content, makes standard HTTP requests without any JavaScript rendering. This is a problem because many competitor pages use Cloudflare, DataDome, or similar anti-bot systems that can block standard HTML requests.

Browsers load the page structure first, then JavaScript fetches and inserts pricing tables and other dynamic content. A standard request using just CrewAI's built-in ScrapeWebsiteTool would receive the page shell but not the data your researcher agent needs. Likewise, it would return a challenge page or a Cloudflare 1020 error instead of the competitor page if that specified website has an anti-bot protection layer. The agent, however, doesn't validate the output. It just takes the incomplete output as the actual content.

Let's set up the ScrapeWebsiteTool for making HTTP requests and target a protected site. Run it yourself and see the challenge it faces.

from crewai_tools import ScrapeWebsiteTool

# Cloudflare-protected site
tool = ScrapeWebsiteTool(
    website_url="https://www.yelp.com"
)

text = tool.run()
print(text)

The output below shows that ScrapeWebsiteTool returned an exception and could not retrieve the relevant results from Yelp because it was behind Cloudflare's protection. The analyst agent will work with bad data. This is one example of the limitation of CrewAI's ScrapeWebsiteTool, not proof that it always returns an empty page.

((venv) ) benny@Mac competitor_intelligence % python scrapertool.py
The following text is scraped website content:

yelp.com Please enable JS and disable any ad blocker
((venv) ) benny@Mac competitor_intelligence %

ZenRows solves the retrieval-layer issue by successfully fetching pages from protected sites. ZenRows is an AI web data infrastructure that combines JavaScript rendering and premium proxy rotation to retrieve data from protected sites. Other recommendations, according to CrewAI's docs, are Scrapfly and Firecrawl. ZenRows is the right fit for this workflow because it is purpose-built for retrieving data from heavily protected targets and maintains a 99.93% success rate across supported domains. Below is a script using ZenRows on the same target (Yelp) to retrieve data.

import os
import requests
from dotenv import load_dotenv

load_dotenv()

ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]

# Same protected competitor URL used with ScrapeWebsiteTool
url = "https://www.yelp.com"

params = {
    "url": url,
    "apikey": ZENROWS_API_KEY,
    "js_render": "true",
    "premium_proxy": "true"
}

response = requests.get(
    "https://api.zenrows.com/v1/",
    params=params,
    timeout=180
)

print("status code:", response.status_code)
print("bytes returned:", len(response.text))
print(response.text[:1000])

ZenRows was able to reach the site and successfully retrieved the actual page HTML.

((venv) ) benny@Mac competitor_intelligence % python zenrowss.py

status code: 200
bytes returned: 501630

<html lang="en-US" prefix="og: http://ogp.me/ns#" style="margin: 0; padding: 0; border: 0; font-size: 100%; font: inherit; vertical-align: baseline;" data-cookbook-theme="light" class="">
<head>
<script>document.documentElement.className=document.documentElement.className.replace(/no-js/,"js");</script>
<meta http-equiv="Content-Language" content="en-US">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<link rel="mask-icon" sizes="any" href="https://s3-media0.fl.yelpcdn.com/assets/srv0/yelp_large_assets/b2bb2fb0ec9c/assets/img/logos/yelp_burst.svg" content="#FF1A1A">
<link rel="icon" href="https://s3-media0.fl.yelpcdn.com/assets/srv0/yelp_large_assets/594b78d06612/assets/img/logos/favicon.ico">

<script>
window.ga = window.ga || function() {
    (ga.q = ga.q || []).push(arguments)
};
ga.l = +new Date;
window.ygaPageStartTime = new Date().getTime();
</script>

<script>
window.yelp = window.yelp || {};
window.yelp.cookieTypePreferencesHeader = '["ANALYTICS"...';
</script>

...

((venv) ) benny@Mac competitor_intelligence %

The difference is visible in the response itself. CrewAI's built-in ScrapeWebsiteTool completed the request but returned a Cloudflare challenge page instead of the competitor's website. From CrewAI's perspective, the task did not fail because content was returned. The analyst CrewAI agent will then analyze the data and draw a false conclusion if you use only ScrapeWebsiteTool. In the next section, you will learn how to build a ZenRows tool for reliable web scraping and to avoid that failure.

Building the ZenRows custom tool for CrewAI web scraping

CrewAI allows you to create a custom tool by turning a Python function into something an agent can call. The function receives the input, performs a calculation, and returns the result to the agent. To add a custom web scraping tool to CrewAI, decorate a Python function with @tool. This will return a scraped page as a string and pass the function into our agent's tools list. You can learn more about this from the official CrewAI documentation.

from crewai.tools import tool

@tool("Scrape competitor page")
def scrape_page(url: str) -> str:
    """Describe what the tool does and when the agent should call it."""
    # call your scraper here
    return "page content as a string"

Now that the tool wrapper exists, you will use it with ZenRows and let the researcher call it instead of making normal HTTP requests. This ensures the agent gets the right extracted data. In the code below, js_render runs the page in a headless browser so JavaScript can load, premium_proxy routes the request through residential proxies, and response_type="markdown" returns clean markdown format for the agent. Using js_render and premium_proxy can increase your credit cost per request. Read more about how to navigate resources from the official ZenRows documentation. The tool created is then added to the researcher agent's tool list.

import os

from crewai import Agent
from crewai.tools import tool
from zenrows import ZenRowsClient

# retries=2 retries failed requests (429 and 5xx) with exponential back-off
client = ZenRowsClient(os.environ["ZENROWS_API_KEY"], retries=2)


@tool("Scrape competitor page as Markdown")
def scrape_page_markdown(url: str) -> str:
    """Fetch the full content of a single competitor web page and return it as
    Markdown. Pass one URL as a string. Use this to read pricing pages, product
    pages, and positioning copy from sites that block standard scrapers."""
    response = client.get(
        url,
        params={
            "js_render": "true",
            "premium_proxy": "true",
            "response_type": "markdown",
        },
    )
    response.raise_for_status()
    return response.text

researcher = Agent(
    role="Competitor Pricing Researcher",
    goal=(
        "Scrape each competitor pricing page and extract product names, "
        "prices, and positioning claims"
    ),
    backstory=(
        "You read competitor pricing pages and report exactly what is on the "
        "page. You never invent numbers."
    ),
    tools=[scrape_page_markdown],
    verbose=True,
)

The researcher can now open the competitor page on the web and return the content as Markdown. In the next section, you will extend the tool by adding an analyst agent before running the whole crew.

Building the competitor intelligence crew

To build a competitor intelligence crew, use two agents: a researcher and an analyst. The researcher uses ZenRows to scrape the competitor pricing page, and the analyst turns the researcher's output into a structured brief. Crew.kickoff() will run them in sequence and return the comparison to you. Let's break everything down.

1 . The researcher agent scrapes target pages and pulls product information. The ZenRows tool appears in the tools list to ensure a page's protection layer doesn't limit web scraping tasks.

from crewai import Agent

from zenrows_tools import scrape_page_markdown

researcher = Agent(
    role="Competitor Pricing Researcher",
    goal=(
        "Scrape each competitor pricing page and extract every plan name, "
        "its price, and any positioning claim on the page"
    ),
    backstory=(
        "You read competitor pricing pages and report exactly what is on the "
        "page. You never invent numbers or fill gaps with assumptions."
    ),
    tools=[scrape_page_markdown],
    verbose=True,
)

2 . The Analyst agent compares and writes the detailed comparison report. It doesn't use the ZenRows tool because it doesn't need to access any webpage. It only works with whatever the researcher gives it. This also ensures you spend fewer tokens and that the analyst uses only the output from your data collection.

analyst = Agent(
    role="Competitive Intelligence Analyst",
    goal="Turn the researcher's scraped data into a structured competitive brief",
    backstory=(
        "You compare pricing and positioning across competitors and write a "
        "brief a product team can act on. You analyze; you do not scrape."
    ),
    verbose=True,
)

Each agent has a task, and the analyst's task uses the research task as context so CrewAI can pass the scraped output forward. The crew holds both the agent and the task and runs everything sequentially. CrewAI uses an LLM to drive agents and print a report from the scraped pages. The code below shows how that sequence works in practice.

import os
from dotenv import load_dotenv

load_dotenv()

from crewai import Agent, Task, Crew, Process, LLM
from crewai.tools import tool
from zenrows import ZenRowsClient

# -----------------------------
# ZenRows Client
# -----------------------------
zenrows = ZenRowsClient(os.environ["ZENROWS_API_KEY"], retries=2)

# -----------------------------
# Scraping Tool (wait added so client-side content renders)
# -----------------------------
@tool("Scrape page as Markdown")
def scrape_page_markdown(url: str) -> str:
    """Scrape a page and return it as Markdown. Return only visible scraped content."""
    response = zenrows.get(
        url,
        params={
            "js_render": True,
            "premium_proxy": True,
            "response_type": "markdown",
            "wait": 3000,  # wait 3s for client-side content to load. requires js_render
        },
    )
    response.raise_for_status()
    return response.text

# -----------------------------
# LLM
# -----------------------------
llm = LLM(model="anthropic/claude-sonnet-4-6", temperature=0.1)

# -----------------------------
# TARGETS (product pages, not category pages)
# Swap these for two comparable products. Confirm they resolve first.
# -----------------------------
TARGETS = [
    "https://www.ikea.com/us/en/p/markus-office-chair-vissle-dark-gray-90289172/",
    "https://www.amazon.com/dp/B00FS3VJAO",  # comparable office chair
]

# -----------------------------
# RESEARCHER (STRICT SCRAPER)
# -----------------------------
researcher = Agent(
    role="Retail Product Data Extractor",
    goal=(
        "Extract ONLY visible product name and price from retail product pages. "
        "No inference, no assumptions."
    ),
    backstory=(
        "You are a deterministic extraction system. "
        "If data is not visible in scraped content, it does not exist."
    ),
    tools=[scrape_page_markdown],
    llm=llm,
    verbose=True,
)

# -----------------------------
# ANALYST (STRICT GROUNDED)
# -----------------------------
analyst = Agent(
    role="Retail Pricing Analyst",
    goal=(
        "Compare product pricing across two retailers using ONLY scraped data. "
        "No external knowledge or assumptions."
    ),
    backstory=(
        "You ONLY use provided scraped outputs. "
        "Missing fields must be marked as 'NOT FOUND IN SCRAPE'."
    ),
    llm=llm,
    verbose=True,
)

# -----------------------------
# TASK 1: SCRAPING
# -----------------------------
research_task = Task(
    description=(
        "Scrape each product page below and extract the product name and its "
        "current price. Use only what is visible on the page. If a value is not "
        "present, write NOT FOUND IN SCRAPE.\n\n"
        + "\n".join(TARGETS)
    ),
    expected_output="Product name and current price for each URL, from scraped content only.",
    agent=researcher,
)

# -----------------------------
# TASK 2: ANALYSIS
# -----------------------------
analysis_task = Task(
    description=(
        "Compare the two products using only the researcher's scraped data. "
        "Do not add outside knowledge. Mark anything missing as NOT FOUND IN SCRAPE.\n\n"
        "Produce:\n"
        "- a table of product name and price per retailer\n"
        "- which product is cheaper\n"
        "- one or two insights drawn from the data"
    ),
    expected_output="A short pricing comparison: a table and a brief takeaway.",
    agent=analyst,
    context=[research_task],
)

# -----------------------------
# CREW
# -----------------------------
crew = Crew(
    agents=[researcher, analyst],
    tasks=[research_task, analysis_task],
    process=Process.sequential,
    verbose=True,
)

# -----------------------------
# RUN
# -----------------------------
if __name__ == "__main__":
    result = crew.kickoff()
    print("\n\n===== FINAL OUTPUT =====\n")
    print(result)

Here is part of the output of the code below (it was a long report). The report is generated entirely from the scraped content returned by ZenRows. A standard scraper would not work reliably here because the retrieval step would likely return a challenge page, incomplete HTML, or a JavaScript shell instead of the underlying content, leaving nothing for the analyst agent to work with.

From the output below, you will notice that the researcher agent remains grounded in the scraped data. For example, it does not attempt to convert currencies or fill in missing information that is not present in the source. Once you have gotten to this point, you have two options. The first is to pass the full page content to downstream agents and let them perform their own analysis, as we did in the script above. The second option is to extract structured fields immediately after retrieval and store them in a database or comparison table. This is often the better choice for use cases where you care about specific attributes rather than the full page content. ZenRows AutoParse is designed for this workflow, returning structured data that can be inserted directly into downstream systems without requiring a second extraction step.

The code is available in this GitHub repository.

╭────────────────────────────────────────────────────────────────── Crew Completion ──────────────────────────────────────────────────────────────────╮
│                                                                                                                                              │
│  Crew Execution Completed                                                                                                                   │
│  Name: crew                                                                                                                                 │
│  ID: 6972714b-a13f-4f66-8ffe-47c908b5ed01                                                                                                    │
│                                                                                                                                              │
│  Final Output: ## Pricing Comparison: Office Chairs                                                                                         │
│                                                                                                                                              │
│  ### Product & Price Table                                                                                                                  │
│                                                                                                                                              │
│  | Retailer | Product Name | Scraped Price |                                                                                                │
│  |----------|--------------|---------------|                                                                                                │
│  | **IKEA** | MARKUS Office Chair | $299.99 USD |                                                                                          │
│  | **Amazon** | BestOffice Ergonomic Office Chair (Mid-Back, Black) | BOB 290.15 |                                                       │
│                                                                                                                                              │
│  ---                                                                                                                                         │
│                                                                                                                                              │
│  ### Which Product Is Cheaper?                                                                                                              │
│                                                                                                                                              │
│  **Cannot be determined.** The Amazon listing returned a price denominated in **BOB (Bolivian Boliviano)**, while the IKEA listing         │
│  returned a price in **USD**. These two currencies cannot be compared without a conversion rate, and applying one would require             │
│  external knowledge outside the scraped data. No valid apples-to-apples price comparison is possible from the data as returned.             │
│                                                                                                                                              │
│  ---                                                                                                                                         │
│                                                                                                                                              │
│  ### Insights from the Scraped Data                                                                                                         │
│                                                                                                                                              │
│  1. **Currency anomaly on Amazon:** The scraper returned `BOB290.15` for the Amazon listing — BOB is the currency code for the             │
│     Bolivian Boliviano. This is flagged in the source data as a likely geo-detection or display error. The true USD price for this          │
│     listing is **NOT FOUND IN SCRAPE** and cannot be assumed.                                                                               │
│                                                                                                                                              │
│  2. **Product detail disparity:** IKEA's scraped data returned only a short product name (`MARKUS`), while Amazon's scrape returned         │
│     a lengthy, keyword-stuffed title. This reflects a structural difference in how each retailer formats product data—not a reflection      │
│     of product quality or completeness.                                                                                                     │
│                                                                                                                                              │
│  ---                                                                                                                                         │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

Returning structured data with autoparse

Instead of returning a full markdown document, you can pass autoparse=True so that ZenRows returns structured output. Autoparsing reads the page's structured markup and hands you a JSON string for the analyst to use. Keep in mind that Autoparse is experimental, domain-limited, and only works on pages with a parser.

To autoparse, modify the tool using the code below.

@tool("Scrape competitor page as structured JSON")
def scrape_page_structured(url: str) -> str:
    """Fetch a single competitor product or pricing page and return auto-parsed
    structured data as a JSON string. Pass one URL as a string. Use this when
    you need discrete fields such as product name and price rather than prose."""
    response = zenrows.get(
        url,
        params={
            "js_render": True,
            "premium_proxy": True,
            "autoparse": True,  # replaces response_type; autoparse returns JSON
        },
    )
    response.raise_for_status()
    return response.text

To switch the tool to structured output, replace "response_type": "markdown" with "autoparse": True. Autoparse cannot be combined with response_type, so it goes in its place, not alongside it.

You can keep js_render and premium_proxy on here, because these pricing pages build their content client-side and sit behind anti-bot protection, so without them, autoparse parses an empty shell. Autoparse itself adds no cost; you only pay for js_render and premium_proxy, so this request's cost stays the same as the Markdown version.

Use the ZenRows MCP server instead of a custom tool

MCP is the right choice when the crew is part of a larger multi-agent system and tool maintenance overhead matters; the custom @tool is right when you need tight control over parameters or when Node.js is not available. The MCP exposes the same scraping capability through an agent tool, but shifts the same integration from code to server configuration. CrewAI can mount any MCP server as an agent tool. CrewAI's MCPServerAdapter starts the server, exposes its tools, and stops the server when the block exits. Everything that uses those tools, the agents, the tasks, and kickoff(), has to live inside the with block, because the tools are only connected while the server is running. The server is open source; here is the GitHub repository. You can read more about it via the Official ZenRows MCP documentation.

The MCP server runs on Node.js, so to use it, you'll need to install Node 20.6 or later. ZenRows also offers a hosted server that you can connect to via URL if you want to avoid the Node dependency, which can simplify deployment.

pip install "crewai-tools[mcp]"

Since ZenRows hosted is the recommended way for any app calling an LLM API directly, let's explore that route. The MCP server is free.

import os
from typing import Any

from dotenv import load_dotenv

load_dotenv()

from crewai import Agent, Task, Crew, Process, LLM
from crewai.tools import BaseTool
from crewai_tools import MCPServerAdapter

# Claude drives every agent. ANTHROPIC_API_KEY is read from the environment.
llm = LLM(model="anthropic/claude-sonnet-4-6", temperature=0.1)

# Product pages to compare. Confirm they resolve first.
TARGETS = [
    "https://www.ikea.com/us/en/p/markus-office-chair-vissle-dark-gray-90289172/",
    "https://www.amazon.com/dp/B00FS3VJAO",  # comparable office chair
]

# Hosted ZenRows MCP server. No Node, no npx, just your API key.
SERVER_PARAMS = {
    "url": "https://mcp.zenrows.com/mcp",
    "transport": "streamable-http",
    "headers": {"Authorization": f"Bearer {os.environ['ZENROWS_API_KEY']}"},
}


class NullStrippedTool(BaseTool):
    """Wraps an MCP tool and removes None arguments before the call.

    The MCP adapter fills unset optional fields with null, which the ZenRows
    server rejects. Stripping None here keeps only the values that were set.
    """

    inner: Any = None

    def _run(self, *args: Any, **kwargs: Any) -> Any:
        clean = {k: v for k, v in kwargs.items() if v is not None}
        return self.inner._run(*args, **clean)


def wrap(tools):
    return [
        NullStrippedTool(
            name=t.name,
            description=t.description,
            args_schema=t.args_schema,
            inner=t,
        )
        for t in tools
    ]


def build_and_run():
    # The MCP tools are only live inside this block.
    with MCPServerAdapter(SERVER_PARAMS, "scrape") as mcp_tools:
        scrape_tools = wrap(mcp_tools)

        researcher = Agent(
            role="Retail Product Data Extractor",
            goal=(
                "Extract ONLY the visible product name and current price from "
                "each product page. No inference, no assumptions."
            ),
            backstory=(
                "You are a deterministic extraction system. If a value is not "
                "visible in the scraped content, it does not exist."
            ),
            tools=scrape_tools,
            llm=llm,
            verbose=True,
        )

        analyst = Agent(
            role="Retail Pricing Analyst",
            goal=(
                "Compare product pricing across two retailers using ONLY "
                "scraped data. No external knowledge or assumptions."
            ),
            backstory=(
                "You ONLY use provided scraped outputs. Missing fields must be "
                "marked as 'NOT FOUND IN SCRAPE'."
            ),
            llm=llm,
            verbose=True,
        )

        research_task = Task(
            description=(
                "Scrape each product page below and extract the product name "
                "and its current price. Use only what is visible on the page. "
                "If a value is not present, write NOT FOUND IN SCRAPE.\n\n"
                "When you call the scrape tool, enable JavaScript rendering and "
                "premium proxy, and wait for the page content to load, since "
                "these pages render product data client-side.\n\n"
                + "\n".join(TARGETS)
            ),
            expected_output=(
                "Product name and current price for each URL, from scraped "
                "content only."
            ),
            agent=researcher,
        )

        analysis_task = Task(
            description=(
                "Compare the two products using only the researcher's scraped "
                "data. Do not add outside knowledge. Mark anything missing as "
                "NOT FOUND IN SCRAPE.\n\n"
                "Produce:\n"
                "- a table of product name and price per retailer\n"
                "- which product is cheaper\n"
                "- one or two insights drawn from the data"
            ),
            expected_output="A short pricing comparison: a table and a brief takeaway.",
            agent=analyst,
            context=[research_task],
        )

        crew = Crew(
            agents=[researcher, analyst],
            tasks=[research_task, analysis_task],
            process=Process.sequential,
            verbose=True,
        )

        result = crew.kickoff()
        print("\n\n===== FINAL OUTPUT =====\n")
        print(result)


if __name__ == "__main__":
    build_and_run()

You can find every code snippet in this tutorial in this GitHub repository.

Wrapping up

You now have a production-ready competitor intelligence workflow built with CrewAI and ZenRows, rather than a scrapers that fail when competitors place their pricing pages behind Cloudflare, DataDome, or other anti-bot systems.

The current system works by having a researcher agent retrieve data directly from competitors' websites using ZenRows, which has a documented 99.93% success rate, while the analyst agent uses that data to generate a report. Whether you choose the custom @tool for total control via Integration or the ZenRows MCP server for easier tool management, the architecture remains the same: reliable retrieval, grounded analysis, and actionable intelligence generated from live web data.

Try ZenRows for free and get your comprehensive competitor analysis today!

A CTO's 5-phase roadmap to AI-native internal tools (and why most pilots stall)

Benny — Wed, 10 Dec 2025 07:20:00 +0000

Executives keep declaring that AI is the future. Reports show that adoption is rising. Yet only a tiny fraction of enterprises ever reach real impact. The numbers look impressive until you ask why so many AI pilots stall before they scale.

In our conversation with Shubham Gupta, CTO at ToolJet, he argued that the real barrier is not model quality or governance. It is the quiet weight of operational debt that sits inside every enterprise. Most critical workflows still run on spreadsheets, disconnected files, and email threads. Gupta put it bluntly that "Our biggest competitor is Excel."

Once you see the problem through that lens, a sharper question appears. If internal operations remain stuck in manual systems, what does it take for an enterprise to adopt AI native internal tools at scale?

This article brings together industry research and Gupta’s firsthand experience to answer that question. It explores the obstacles slowing internal AI adoption and the role of AI native internal tools in removing those obstacles. It then lays out the five-phase roadmap Gupta believes leaders can use to achieve real progress rather than more pilots.

What the industry currently gets wrong about internal AI adoption

IBM reported that enterprise-wide AI adoption stalls due to poor model customization, complex infrastructure, and regulatory requirements. Gupta argues that the real reasons for internal AI adoption failure are legacy systems, fragmented processes, and deprioritization of internal tool development.

For example, team approvals for new product launches still move through email threads. Employees cannot act on data buried in unstructured processes, and the workflows that AI should automate still exist in spreadsheets and email chains.

The industry sees AI sophistication as the main hurdle, but Gupta insists that the true limitation to internal AI adoption is the lack of software.

"Companies with large operational requirements need software for employees to do their work more efficiently," said Gupta. "Imagine there are about 5,000 employees, in the absence of proper internal software, chaos will ensue," he added.

Relegation of internal tools is another adoption obstacle that industry reports often overlook, and Gupta says most enterprises have yet to overcome. With engineers working on customer-facing assets, internal tool development often takes a back seat.

"Engineering teams are always juggling between delivering consumer-facing products and prioritizing these internal tools, which ultimately causes a lot of operational inefficiency," Gupta shared.

To effectively adopt AI internally, organizations must update their manual practices and increase development time for internal software using low-code platforms like ToolJet.

Why internal tools are the natural first home for AI

Analysts often point out that AI succeeds fastest in environments where work follows a repeatable pattern. Gupta sees the same reality inside enterprises.

“Internal tools are repetitive and do not require a lot of contextual thinking, which makes AI well placed to build and manage them."

Employee sentiments also support this direction. A survey by Talker Research revealed that 62% of employees want to use AI in their roles to sort data in spreadsheets. Routine tasks such as data sorting, CRUD operations, approval chains, and inventory tracking are examples of workflows that follow a fixed pattern. They are the logical starting point to introduce AI-powered internal tools because they generate compounding value quickly and provide a deterministic process that enables AI algorithms to function reliably.

The industry view and Gupta’s experience point toward the same starting point for AI adoption. Enterprises waste momentum when they try to automate strategic or ambiguous work first. Gupta suggests focusing on the internal tools that already run on predictable workflows because these are the environments where AI produces reliable gains and compounds value quickly.

The determinism challenge most AI tool builders ignore

Determinism in AI-native internal tools

Enterprises fear AI's inherently non-deterministic nature. Executives want consistent and auditable outcomes from the AI tools they adopt within their organizations because AI unpredictability triggers mistrust, poses compliance issues, and can harm decision-making.

For example, a single hallucinated permission can expose an entire customer dataset. Analysts focus on establishing evals and guardrails, but the industry still lacks a shared standard for how they should work.

Gupta says the solution is an architecture with three interconnected layers, as illustrated in the image above: a multi-agent structure, an eval system, and a human-in-the-loop process. Determinism increases when enterprises break automation into smaller, single-responsibility agents.

"If you spread every AI action into multiple single responsibility prompts, the agent's output quality improves," he said.

The second layer, an eval system, serves as a feedback loop that ensures behavioral consistency, security, and reliability.

"It puts guardrails in place, so when AI responses fall below a certain standard, organizations can re-loop their agents," Gupta shared.

The final layer, human checkpoints, is what locks the architecture into place.

"No AI action on data should be performed automatically. A human should always authorize and test those actions in a development staging environment before they go to a production environment," Gupta highlighted.

This approach to achieving predictable AI outcomes should serve as a framework for enterprises that want to build trustworthy AI-powered internal tools. But tackling non-determinism is only one part of mitigating the vulnerabilities that can affect an organization's trust in AI tools.

Enterprise architecture is the real battlefield

The hardest part of AI adoption isn't the model, but the architecture that surrounds it and the expectations enterprise teams refuse to compromise.

Enterprises want:

Data control for proprietary data and isolated tools that span different departments
Governance to ensure a tool works transparently and meets regulatory requirements
Predictable Software Development Life Cycle (SDLC) for engineering teams that are resistant to disruptive workflows
Integration into existing systems to accommodate tailored workflows

Analysts believe AI tools will replace traditional workflows within enterprises. Rather than outright replacement, Gupta contends that internal AI tooling must respect an organization's existing architecture, protect that foundation, and extend the system when it requires further customization.

In practice, the right internal AI tools offer self-hosting options, provide flexible SSO, and support extensibility through code, plugins, or third-party integrations.

"You can self-host ToolJet internally inside your VPN or deploy it in your existing cloud services to make sure that none of your sensitive data escapes or reaches our servers. The data stays in your own cloud or database, so you can completely air-gap ToolJet inside your network," Gupta noted.

Opinionated platforms lock organizations into vendor-controlled environments, which risks data exposure, while enterprise-friendly AI tools give organizations full control over where their data resides.

Internal tools often span departments, and each team might build its own application while still requiring cross-departmental coordination. Enterprises want unified access control, and a strong Single Sign-On (SSO) provides secure access to the tools.

"Whatever SSO a company supports is supported by ToolJet. Whether they prefer multiple SSOs for multiple departmental tools or a single SSO to rule them all, that also works with ToolJet," said Gupta.

Opinionated platforms typically require organizations to conform to their specific authentication protocols, while adaptive AI tools integrate with multiple identity providers.

Respecting engineering workflows is another priority for internal development teams that have established processes. Gupta explained,

"Engineering teams are very picky with tools because they have a certain way of working and they don't enjoy people messing with it."

Opinionated platforms dictate a new SDLC model, but flexible AI platforms like ToolJet allow engineers to follow their own SDLC while the platform adopts it. Enterprise AI tooling succeeds when platforms adapt to an organization’s constraints, rather than fighting them.

Understanding the real obstacles to internal AI adoption is an important first step, but enterprises also need an action plan that actually drives adoption.

The roadmap to adopting AI native internal tools

Leadership wants results with AI, but often without a clear strategy or a full picture of how to achieve it. Gupta provided a phased approach for business leaders to know where to begin the adoption process and how to scale it across their organization.

Roadmap to adopting AI native internal tools

Phase 1: Audit where manual work still lives
Industry reports often advise executives to conduct an AI readiness assessment that evaluates whether their organization is ready to integrate and adopt AI. But this assessment is a broad and ongoing process.

Instead, Gupta recommends that executives revisit the manual, high-friction, and data-heavy parts of their existing workflows, then redesign them so AI becomes a natural part of the lifecycle.

"Check how many people are handling a bunch of Excel sheets overflowing with customer records or transactional data in your company," he said.

Engage with teams across product, engineering, operations, or sales departments to identify bottlenecks. For example, customer success teams may be losing track of approval requests buried in emails.

Gupta suggested this is how to spot inefficiencies and opportunities for automation. Transforming this manual process into an internal software will ensure that every request has timestamps and owners attached. Teams still make the higher-level decisions, but with an AI-powered internal tool, they start benefitting from automated routine tasks and smoother feedback.

Phase 2: Ship the first AI-generated internal app fast
Analysts say business leaders should define Key Performance Indicators (KPIs) before introducing an internal AI tool. While KPIs outline business objectives, they can create an illusion of clarity. Instead, Gupta says executives should choose one painful workflow and build a solution fast using platforms like ToolJet, which cut time-to-app through natural-language app creation.

The real success metric to track is "shipping a working internal app in one to two weeks, depending on the use case. Then, iterate based on feedback," he said. When employees experience optimized workflows and regain time to focus on higher-value work, internal tool adoption gains momentum.

Phase 3: Create a repeatable build system
Industry experts debate which build system is superior between AI, low-code, and pro-code development. But Gupta says executives must approach their decision as a development spectrum.

"Some enterprises need AI-powered development with a prompt-specific approach for rapid prototyping. Others want low-code solutions, where non-technical teams can drag-and-drop and customize pre-built components in a visual builder. Engineering teams may want to customize features even further and add custom connectors for specific functionalities using pro-code development," he said.

A practical approach is to reduce development backlogs and tool build time using AI and low-code platforms, then leverage the flexibility and precision of pro-code for custom workflows and complex features.

Phase 4: Harden with governance and access control
Reports show that the timeline for implementing and scaling AI governance in an enterprise can take 6 to 18 months. Gupta suggests a different approach: build trust into the tool from day one through determinism and access control. Start with Role-Based Access Control (RBAC) for tools that handle significant business processes, manage sensitive data, and report critical information. IT teams should limit users' permissions to only what is necessary to perform their roles.

"If a specific team doesn't need to have access to a tool, they shouldn't have access to it because the principle of least privilege and regulatory adherence is very critical to large enterprises," said Gupta.

Also, implement SSO early with strong encryption to protect authentication data. Above all, maintain human ownership throughout the entire lifecycle of developing and scaling AI internal tools. Human employees should validate decisions and provide ethical oversight, while AI apps or agents handle the operational load and execution.

Phase 5: Scale through internal evangelism
Industry reports often attribute poor internal tool adoption to governance. But Gupta challenges this view, arguing that scaling is a value-proof problem. When teams can see a tool's value quickly, adoption spreads naturally, and scaling becomes feasible.

Gupta encouraged executives to "regularly share wins and updates on a successful internal tool to build trust, and create internal programs like Hackathons which reward employees who actively build tools that contribute to enterprise-wide integration."

A McKinsey survey reinforces Gupta's suggestions, stating that "employees believe access to AI tools in betas or pilots, along with incentives such as financial rewards and recognition, will accelerate AI adoption." A successful pilot within one team will encourage a culture of experimentation, where employees feel empowered to explore AI solutions across organizational processes.

Gupta shared that the ToolJet team uses its own platform internally across different business functions, achieving productivity gains and reducing development costs.

"We used ToolJet to build an internal CRM that handles licensing keys creation, payments, refunds, and recurring invoices for our customers. This used to take a lot of our manual effort, and if we had to build this product from scratch, it would have taken us another two months. But with ToolJet, we were able to launch the product faster, and our dev team spends more time on actual customer-facing products."

He also shared that the team built their release notes engine on ToolJet in one week.

"We evaluated external software, but it was costing us upwards of $1,000 a month. So we decided to build internally using ToolJet, and it took us a week to spin it off. And now we are completely relying on our internal software to manage and publish release notes."

Enterprises that build internal AI tools early for high-friction tasks will gain faster data analysis, reduced manual error, clearer communication, improved IT support, and productive employees who can focus on more strategic tasks.

What it means to build an AI native internal tools strategy

Organizations that want to be ready for an AI-driven future must start with their internal systems. The roadmap presented in this article provides executives and team leads with concrete steps towards building an AI-native internal tooling network that puts them at an advantage to win at AI.

It doesn't require a radical company-wide overhaul. The key is building targeted internal tools that automate mechanical, multi-step work to compound wins quickly. Then, teams can iteratively expand those tools into well-scoped tasks while investing in guardrails.

The leaders who successfully implement this roadmap will move from pilots to full-scale impact.

Vibe Coding Isn’t Dumb — You’re Just Overthinking It (Especially With Tools Like ToolJet & Lovable Around)

Benny — Sat, 29 Nov 2025 20:42:34 +0000

There’s a growing sentiment among developers today about “vibe coding.” But most of the pushback comes from folks who still believe every project needs enterprise-grade architecture from day one. The reality is simple: if your goal is to ship, not philosophize, vibe coding with AI is one of the fastest ways to get ideas into the world.

A recent post broke down why and how AI works best when you remove friction. By friction, I mean mainstream stacks, short PRDs, version control, and clean context resets.

But here’s the part most people miss. Low-code and AI-native builders are literally designed for this workflow. Tools like ToolJet and Lovable take those same vibe-coding principles and bake them directly into the development experience.

Where ToolJet Shines

If you’re building something with real internal logic, workflows, or data operations, ToolJet thrives. It’s not just a “drag-and-drop UI builder.” Its AI agents understand business rules, connect to databases instantly, generate API integrations, and help scaffold real features without you wrestling with boilerplate.

It make the “use a mainstream stack + provide a clear PRD + ship step by step” workflow feel natural. You give it structure, and it gives you production-ready apps that don’t collapse under real usage.

This is vibe coding for serious apps. The kind you internal team can use enterprise wide.

Where Lovable Shines

Lovable is the opposite energy in the best way. It’s the perfect expression of pure vibe coding:

type an idea → get an app → iterate with natural language.

It’s fast, playful, and incredible for prototypes, validation, hackathon-style creativity, and “I just want to see this working in five minutes” moments.

When the screenshot says:

“Pick a mainstream stack so AI gets it right”

Lovable basically automates that decision for you. It handles the stack, scaffolding, routing, structure — so you stay focused on the idea.

Vibes > setup.

The funny thing is: vibe coding doesn’t mean you avoid complexity — it means you stage it.

So maybe start in Lovable to get the first version fast.

Move to ToolJet when your prototype becomes a real internal product.

Or stay in Lovable if your whole goal was speed, not enterprise depth.

Both tools turn the guidelines from the original post into practical, repeatable workflows.

The Takeaway

Vibe coding isn’t “lazy dev.”
It’s a modern mindset. ship fast, reduce friction, give AI constraints, and build only what matters.

ToolJet and Lovable amplify that mindset. Smartest developers aren’t the ones writing everything from scratch. They’re the ones who know how to combine AI, low-code, and clarity to ship 10× faster without losing their sanity.

I tested ToolJet’s Agent and here is what I found!

Benny — Sat, 22 Nov 2025 15:19:00 +0000

I spent last weekend playing with ToolJet’s new AI agents and… low-code might actually be dead in the “holy crap, this is what we’ve been waiting for” way.

Let me explain.

I ran a nightmare prompt that would've normally triggered a week of meetings and 20 Jira tickets:

“Build an internal tool for our sales team to manage customer contracts: upload PDFs, extract key dates with OCR, route approvals with RBAC, send reminders, and log everything.”

No schema. No PRD. No wireframes. Just the prompt/sentence above.

And here’s what the ToolJet agent team gave me in minutes:

A full PostgreSQL schema with normalized tables, foreign keys, indexes, and a scalable structure
A real UI — file uploads, searchable tables, and a Kanban approval board
RBAC baked in with viewer/editor/admin roles without config hell
and Audit logs for every action, exportable to CSV, and that comes with Slack/Zapier notifications for approvals + reminders

Honestly? It felt like working with a full-stack team without getting burned out!

And that got me thinking.

These agents aren’t “GPT wrapped in a UI.” They behave like specialized teammates. There is an agent for DB design, UI scaffolding, API wiring, logic building, debugging, etc. They talk to each other, hand off tasks, and refine.

That’s the part that feels like the real shift.

Want receipts? Here are the demos that convinced me:

My honest take after testing everything from Retool to Bubble to AI coders. Most tools break when you throw enterprise complexity at them because enterprise tools need permissions, environments, audit trails, and integrations.

ToolJet’s agent system didn’t blink. It felt scalable, deliberate, and surprisingly “developer-aware.”

So if you’re drowning in CRUD requests, or you’ve shipped one too many “temporary” internal tools that outlived your will to maintain them, give ToolJet AI a shot.

Your weekend (and sanity) will thank you.

Building a Competitive 'Deal Hunter' Agent Using n8n + BrightData Real-Time Web Data

Benny — Sun, 31 Aug 2025 13:15:59 +0000

This is a submission for the AI Agents Challenge powered by n8n and Bright Data and my first n8n workflow.

What I Built

Searching for deals manually can be time-consuming and frustrating. Most of the time, this costs us as we often miss out on the best prices and limited-time offers. This AI agent solves this problem by automating the deal-hunting process and providing a single, intelligent interface for finding the best discounts.

How It Works
This AI agent, built with n8n and powered by Bright Data, is designed for efficient web scraping. The agent then uses the Bright Data n8n Verified Node as a tool to perform real-time web scrapes for the requested product based on the user's chat message. The output (the scraped data) is then presented to the user in a clear, friendly, and conversational summary.

Demo

Agent link: http://localhost:5678/workflow/2hIkozBxX4QEcxyZ

Demo Video: https://youtu.be/TGljGiiHpRo

n8n Workflow

https://gist.github.com/Bennykillua/107287dc7a31dda58c6b5a7c0bf6ab88

Technical Implementation

Our trigger is a chat.
The agent is configured as a highly enthusiastic “Deal Hunter” AI. Its system prompt ensures it only responds when the user requests a deal, maintaining an energetic and competitive tone. This keeps the agent focused on just finding and suggesting the best possible product deals.
For the model, I make use of OpenAI’s GPT-4.1-mini model. This model strikes a balance between cost efficiency and high-quality natural language understanding.
The memory, I made use of a simple memory buffer that stores the last 10 exchanges. Hence, it remembers what items the user is interested in during the session and avoids repeating questions unnecessarily.

Bright Data Verified Node

The node (brightDataTool), which I used as a tool for my AI agent, is set to filter a marketplace dataset. It dynamically receives parameters from the AI agent (Dataset, Field_Name, Field_Value). These parameters are auto-generated based on the user’s request (e.g., “laptop under $500” → dataset = electronics, field = price, value = 500).

The node returns a filtered snapshot of matching deals from Bright Data’s real-time datasets.

Journey: Process, Challenges, and Learnings

I started learning about n8n just 10 days ago, though I have heard of the tool. I learned through YouTube tutorials, driven by my desire to complete this task. My background is in data analysis and business process automation, utilizing Microsoft Power Platform, which has provided me with a strong foundation. However, working with an AI-powered agent in n8n was a very new experience for me, and to be honest, it feels quite different compared to the more structured workflows in Power Platform.

Exploring n8n has been exciting because it combines low-code workflow automation with AI-driven logic and third-party data integrations, making it very powerful and fun to build with. I’m still learning the tool, but I already see how flexible it is for creating intelligent automations.

Challenges

Since this was my first deep dive into AI agents, I had to adapt to how they handle prompts, memory, and external tool usage, which was a learning curve. There are features I would have liked to implement but couldn't cause of the deadline:

Sending emails automatically after finding deals.
Returning richer data through the Bright Data node (more descriptive results and structured insights).

These were limited by both the tight deadline and my current skill level with n8n.

What I Learned

Building with n8n is very approachable even for someone coming from a Power Platform background. AI agents introduce a new layer of flexibility and intelligence that traditional automation tools lack. Lastly, the Bright Data node integration demonstrated to me how external data can enrich AI workflows, bridging the gap between automation and actionable, real-world insights.

To be honest, this has been a great first step into AI-driven automation with n8n, and I’m looking forward to building more advanced workflows. The community has also been pretty helpful.

Learn how to create a Coze plugins from Scratch

Benny — Fri, 25 Oct 2024 22:02:17 +0000

This tutorial will teach you how to create a plugin from scratch in Coze.

Link: https://hackernoon.com/how-to-create-plugins-from-scratch-in-coze-coingecko

My Year in Review: Hey there!

Benny — Sat, 31 Dec 2022 07:53:02 +0000

Honestly, I wasn't planning on writing a year in review. This is my first time writing one, and I am also trying to figure out how to write one. But 2022 was interesting, to say the least. I got into tech in 2021, and 2022 was me exploring and finding my feet within the industry. I also contributed to various open-source projects, made new friends, and experienced fantastic career opportunities.

I will try to make this personal, but then again, I'm not expressive, so I'll just list some highlights. So here we go in no particular order.

2022 highlights

I finally got the hang of the data analysis landscape and started crushing it at UrBizEdge. I learned a couple of new tools and more about old tools I frequently used. I also hit the one-year milestone. Thanks, Michael Olafusi.
Teaching people about data tools and Microsoft technologies; taught over 400 people.
I got accepted as a technical writer into more publications, though I did not write for some. On the bright side, I, however, got to write for a bunch of new B2B, B2C, and SaaS organizations like Linux foundation, Bird eats bugs, Loggly, Opslevel, Microsoft 365 blog, and a lot more via HitSubscribe. What made this notable was how broad the topics always were.
More about technical writing: I also got accepted into freeCodeCamp, was able to write a post, and made the top contributor list.
I got the contract technical role with Sematext four months ago to create glossaries about various DevOps and cloud terminologies. It is a pretty exciting project because, besides writing, the team and I also work on the information architecture and showcase helpful product demos to help everyone get started.
I got some amazing feedback from the various short tutorial videos I recorded about data analysis, data science, and several data technologies, languages, and tools like Microsoft Excel, SQL, Python, Power BI, Power Query, and the Power Platform.
In 2022, Hashnode did an Open source themed writeathon. I learned about open source and gave it a go this year. I got the chance to contribute docToolChain, a doc-as-code generation tool (here is a quickstart guide), Numpy, and a few smaller projects like Oppia (free education for all) and Forecast work.
For more small wins in open source, the Spotify team added an article I wrote about Spotify's Chartify as an official resource. I was also able to make a tiny doc fix to Microsoft Power BI's documentation. This is a tool I use a lot, so saying this felt awesome would be an understatement.
Participated in Hackoberfest 2022 and wrote an article via opensource.com to onboard curious folks. 🙈 I also got a tree planted in my name. Thanks Hacktoberfest and DigitalOcean!
My friends (Blessing, Timonwa, and Light) and I Introduced more than 70 girls to technical writing and open source by mentoring at the Empower Her X Non-Tech in Tech Bootcamp. The conversations, questions, and feedback were also promising and intriguing. I love teaching as well, so this was fun and fulfilling. A few got gigs from the articles they wrote during the Bootcamp, and others made their first open-source contributions.
Created a technical writing resource repository with a friend, Blessing of CSS tricks. The repository resources and discussions are supposed to act as a roadmap for anyone looking to get started in technical writing. Since it is open source, you can contribute to it.
I attended a few tech events like DataFestAfrica and OSCAFEST22 (the first physical tech event I have attended and volunteered at).
Besides data analysis and technical writing, I spoke at, co-organized, and managed meetups that hosted free virtual learning sessions every Sunday. Attendees learned more about data analysis, Microsoft Excel, Power Platform technologies, Microsoft Azure, Power BI, SQL, and financial modeling in the community sessions. I co-organized 114 meetup sessions with 70 speakers and the speaker's lineup this year was also impressive. We had Ruth Pozuelo, Bob Umlas, David Abu, Travis Oliphant (the creator of Numpy and SciPy), and many more. Can I include myself as one of the impressive speakers this year?
For community growth! Each meetup community I managed (Microsoft Excel and Data Analysis Learning Community, Power BI Learning Community, and Financial Modeling & Valuation Learning Community) also grew in the number of members and engagements by over 51%.
Talking about meetups, conferences, and advocacy, I got invited to speak at a couple of meetups like Microsoft Power BI UK, Microsoft Student Ambassadors Kenya, and Resagratia data digest. Oh! And the most significant speaking conference opportunity this year was being a speaker at Write the Docs Prague 2022. The experience was great! From the past speaker's lineup, we (Mustapha and I) were also the first speakers ever from Nigeria. Here is a quick recap of the conference, our slide, sketch-note, and the video session.
Self-confidence and social anxiety. I battled a lot with this, and although I have gotten better at it, there is still a lot to be done.
The biggest win of 2022 is that I am still here, got the chance to meet a couple of my mutuals, and made new friends. GRATEFUL is putting it lightly 🤲🏿 🤲🏿

These years also had a lot of downs. From the rejections emails, a couple of which pretty hurt ‘cause I was in the 2nd, 3rd, or 4th interview stage, to the period I struggled with low productivity and the letdowns. But, generally, this year's vibe was rough and more of “getting the hang”. Next year will have a more upward trajectory.

For 2023…

I'd like to breathe and do better in a lot of things:

Get better at my career and work.
Be a lot more visible career-wise.
Do a physical talk and speak at more conferences. Most of my talks last year have been mostly online.
Connect more and be a lot more intentional.
More consistent with my daily 1-hour activity.
Take a trip… or more.
More community work.
Most importantly, have a better personal relationship with God.

Thanks to everyone who has been a part of my 2022 journey. Thanks for your support and guidance. I wish you a merry Christmas and a happy new year. Have a blessed 2023, everyone!

Cover image is by PhotoGerard

Object-oriented Programming in Python 🐍

Benny — Tue, 06 Apr 2021 12:33:23 +0000

Object-oriented Programming (OOP) is a computer programming model that helps one organizes their software design around data, class, and object rather than functions and logic.
Just like you have Knowledge and can perform various methods, Objects have data and can perform various methods.

Class is the blueprint used to create a user-defined data structure. It holds the common attribute and behaviors of the object. For example, the class “Ride” will hold objects “car”, “plane” and “bike”.

Object is an instance of a class. It has both a property (variables) and behavior (methods). The python object “Car” will have properties; “make”, “model”, “four_wheels” and behaviors; “start”, “stop”.

Method is the behavioral factor of the object. Modifying a method in the class “Ride” would affect all instances of it; “car”, “plane”, and “bike”.

Instance is simply any object made from the class. An instance of the “Ride” class would contain real unique data. A class is like a form while an instance is a filled variation of that form.

It is good practice to identify all the objects you want to manipulate and how they relate among themselves. This is data modeling.

Principles of OOP

1. Encapsulation

State, variables, and methods should be private unless they are being declared public.

An object “car” of a class “Ride” can manage itself via methods “start” and other class “House” cannot touch it expect
allowed to. We can only use the methods to communicate with our object. If we want another class “House” to change the state of the “car” object, we have to create public methods “over_ride_and_start” to modifies or invoke the start
variable. This connection is encapsulation.

2. Abstraction

Hide all implementation of the class from anything outside the class.

Our object should hide details and only expose relevant information to another object. Thus public method should be created so object can be called. The “Al voice recognition” may start our “car” object (in the “Ride” class) and open the “Lights” object (in the “Home” class) without having access to the details of our car. This makes maintenance of
an extremely large database less daunting and lowers the chances of bugs.

3. Inheritance

Child class should have a mechanism that enables it to inherit behavior and properties from the parent class.

Reusability is the advantage here. Child class “car”, “boat” objects should acquire properties (“has_an_engine”, “can_be_used_for_transport”) from the parent class “Ride”. However, they should have their own unique properties.

4. Polymorphism

Interfacing with objects and receive different forms or results.

This allows us to implement a class exactly as intended for the parent class. This prevents errors in types and each
sub-class keeps its own methods. Our class “Ride” has a speed method. Our child class; “Object”, “boat” and “plane” can utilize this method. However, they would each have their own iteration of the speed method.

OOP in Python 🐍

Let's Begin! 🚀

Here is a Link to the notebook on my Github. You could download it and follow along.

Class is defined with the class keyword followed by the name of the class.

class Car:
    pass #This is also where subsequent code will eventually go.

The pass acts as a place holder and thus prevents python from returning an error. At the moment, our car class is without attributes or methods. Let's instantiate our object class by calling the class.

car()

Every instance of our class is stored in a new memory address. which means every instance of the Car class represents two different objects. Wanna try?

a = Car()
b = Car()
print (a == b)

Our class has a docstring attribute called doc which holds the documentation of the object.

class Car:
    #this is the docstring and it gives a brief about the class
    """ A four-wheeled road vehicle that is powered by an engine car has four wheels"""

Car.__doc__

Let’s make our class interesting by giving it attributes. There are two types; Class attributes and Instance attributes.
Every attribute in our class would be defined in a method called .init() with self as the first parameter. These are called instance attributes. However, class attributes are attributes that have the same value for all instances of that class. Hence it would be assigned outside the .init() method. All instances of car class (model and colour) are four-wheeled.

class Car:
    #Class attributes
    wheels = "four"
    def __init__(self, model, colour):
         self.model = model
         self.colour = colour

Let’s give our instance attributes parameters values. 🧨

Toyota = Car("Toyota", "yellow")
kia = Car ("Kia", "black")
Ford = Car("ford", "blue")
Honda = Car("Honda", "brown")

We have four Car instances with each having unique values. Let’s access the data in these attributes.

#instance attributes
Toyota.model
print(Toyota.model)

kia.colour
print(kia.colour)

#class attributes
Toyota.wheels
print(Toyota.wheels)

kia.wheels
print (kia.wheels)

Like I mentioned all instances in a class have .wheels, .model and .colour. However, the values of the instances attributes (.model and colour) would differ; encapsulation and polymorphism. We can also update and modify the values in our instance without changing the values.

#modifying our attributes
Toyota.colour = "red" #previously yellow
print (Toyota.colour)

kia.wheels = "two" #previously four
print (kia.wheels)

It worked!! 😎 So objects are mutable.

Let’s look at instance methods. Just like .init(), the first parameter is also always self and is defined in our class. Let’s create two instance methods; speed and details.

class Car:
    wheels = "four"
    def __init__(self, model, colour):
         self.model = model
         self.colour = colour

    # Instance method
    def details(self):
        return f"this {self.model} is {self.colour}"

    # Another instance method
    def speed(self, speed):
        return f"This {self.model} is {speed} at top speed"

Let's apply these new methods.

Toyota = Car("Toyota","yellow")
print (Toyota.details())

#Toyota.speed(280)
print (Toyota.speed(280))

Let’s look at inheritance. Let’s say we want to add a new class “engine”. We can’t always specific the engine for each car. Let’s create a child class for it

class Hybrid_engine(Car):
    pass

class Gas_engine(Car):
    pass

Now let’s instantiate these engines for the cars.

Toyota = Gas_engine("Toyota", "yellow")
kia = Gas_engine("kia", "black")
Honda = Gas_engine("Honda", "brown")
Ford = Hybrid_engine("Ford", "blue")

Let’s try some basic functions.

print(type(Toyota))
print(isinstance(Toyota, Car))
print(type(Ford))
print (isinstance (Ford, Gas_engine))
#Toyota, Kia, Honda and Ford are all instance of Car 
#but Ford wouldn’t be in the Hypdrid engine child class instance.

Thus, all objects created from the child class is an instance of the parent (car) but might not be in the same child class (Gas_engine and Hybrid_engine). Let’s give it the parent’s functionality specific to it (Inheritance)

class Hybrid_engine(Car):
    def speed(self, speed="210"):
        return f"{self.model} Speed; {speed}"

Ford = Hybrid_engine("Ford", "blue")
print (Ford.speed())

A change in the parent class wouldn’t affect the child class.

class Car:
    wheels = "four"
    def __init__(self, model, colour):
         self.model = model
         self.colour = colour

    # Instance method
    def details(self):
        return f"this {self.model} is {self.colour}"

    # Another instance method
    def speed(self, speed):
        return f"This {self.model} is {speed} at top speed"

Toyota = Gas_engine("Toyota","yellow")
print (Toyota.speed(367))

#however ford wouldn’t get the change since we define his child class

Ford = Hybrid_engine ("Ford", "blue")
print (Ford.speed(467))

However, if we didn’t want this to happen. We would need to use the .super built-in function for the child class. 🙃

class Hybrid_engine (Car):
    def speed(self, speed="210"):
        return super().speed(speed)

Ford = Hybrid_engine ("Ford", "blue")
print (Ford.speed())

Summary 🎉

In this article, we discussed Object-oriented Programming and the principles which govern it.

Then we looked at how to define a class, how to instantiate an object from a class, and giving this class some sought of documentation using .doc. Let's not forget we looked at an instance and class attributes and defining properties and behaviors of an object. We also discussed some function such as type(), isinstance() and super(). 🚀

I hope you found this notebook helpful. Remember to share on your favorite social media so other folks can find it, too. Thanks 👍