DEV Community: Rachel-Lee Nabors

IE6, AI, and the future of browsing the Web

Rachel-Lee Nabors — Thu, 05 Jun 2025 14:33:38 +0000

Chrome is the next IE6.

The signs are there. A few years back, Google cut funding to longstanding community contractors working on standards with the W3C. Community members I spoke with at CSS Day 2023 in Amsterdam found the move confusing, given that Google had supported this vital work for so many years. And maybe it was important—to the Web. But what Google was saying with its actions was, “This work is not important for Google right now.”

And with good reason. For the first time ever, Google searches are dropping. And despite rhetoric that “users won’t pay to use the Web” they are paying to use AI services competing with the Web like Perplexity. This happened once before when mobile apps hit the scene, and Google rose to defend the Web (and its advertising income) by launching Chrome, AMP, and a host of other initiatives to keep the web competitive with walled garden app stores.

But this time is different. AI just broke the Web that Google built.

Chrome’s IE6 era

While Google’s Chrome Platform team has been shipping new standards for web developers, Chrome, the browser itself, hasn’t made any significant improvements to its user experience in years. (By comparison, Firefox has shipped more UI updates in the same 5 years, including ones like sidebar navigation for tabs.) Perhaps Google sees no need to “fix” what is already working for ~70% of the Internet.

A similar situation occurred with Microsoft’s Internet Explorer 6 (IE6). After finally achieving a 90% browser market share, Microsoft lost interest and diverted its resources to sell more Windows and Office licenses.

The next largest browser is Safari, with around 15% of the browser market share, thanks to its iron grip on iOS. If you see Google/lobbyists pushing for Apple to allow other browser engines onto its devices in courts, it is likely more about gobbling up that high-value slice of market share than giving users the power of choice. Chrome is, and always has been, a vehicle for ad revenue and data collection for Google’s money-making duo: Search and Ads. In protecting these interests, Google has gone so far as to backpedal on deprecating third-party cookies, which underpin data collection and ad targeting.

Although Chrome receives the bulk of browser use, and thus Google has little motivation to innovate and compete, the writing on the wall is that Chrome’s days are numbered. User behavior suggests people are finding LLMs more convenient for finding answers than Googling, where they must leap over hurdles of ads, dive several pages into the search results, and then pogo in and out of websites to find answers to some of their most banal questions. And Google knows this. On May 7th, Eddy Cue, Apple’s senior VP of services, shared that searches on Safari dipped for the first time last month, which he attributed to people using AI like Perplexity. Google I/O touted a vision of an Agentic and AI-driven future brought to you by Gemini.

Google is shifting its attention

We can watch externally and see how Alphabet is reorganizing Google to accommodate this upcoming modality. We can see them testing AI-driven search results in Google. They reorganized their devices and Chrome teams a few weeks back, cutting industry-favorite Chrome team members. What do these actions say?

Google is saying, “I’m not interested in continuing to pour the same resources into Chrome anymore. I’m not interested in stewarding the Web as we know it.”

Consider also that the FTC is currently deciding if Google and Chrome should be split up (which could see Chrome bought by Perplexity or even Yahoo). Google executives would have to be blind not to see where the ball is going, that ChatGPT and its ilk threaten its core business model of serving ads to people searching for answers. This may explain why, at Google I/O this year, new AI features were shipping at the OS level, not the browser level. You could interpret all of this as Google pulling back resources from a product that is about to be outmoded, sold off, or, at the very least, doesn’t require further innovation to remain the top browser.

The Web’s post-ads era

If you build websites, you might be offended—“How dare Google stop investing in the free and open web?”

My friend, Google was always investing in keeping the Web competitive with apps so its ad revenue wouldn’t decline, to the point where it has been creating a “shadow Internet” in the form of AMP. It has never been altruistic, though it has wisely employed altruists.* And arguably, there’s nothing inherently “evil” about supporting an ecosystem of content with ads. The downside, however, is that the APIs, protocols, and standards Google fostered grew the Web only in one direction: ad-based revenue. The Agentic Web is poised to disrupt the ads-supported ecosystem because LLMs can deliver facts directly, and AI Agents can pierce through ads and data-collecting tech to harvest content on behalf of users, no human eyes required.

The question Google—and you—should be asking is, “How do we adapt to a world where people have a magical answer box embedded in their machine and don’t need to visit websites much anymore?”

This series of posts seeks to address the new challenges facing the Web, its users, and its creators with the introduction of “magical answer boxes,” generated content, and agentic systems. I shall endeavor to share neither optimistic nor pessimistic views but only to drive at the truth of what is happening and what we can do to bring about a thriving ecosystem of creators and consumers. I seek to address the following:

Money Economic models, payments, and protocols for a post-ads economy
Identity How we distinguish between ourselves, each other, and our “extended selves,” both personally and programatically.
Reach Optimization for reaching audiences using agents and LLMs
Security Methods to defend servers and information from unwanted agents and bots
Evolution: The creation of new browsers and “browser alternatives”

The Web is always a work in progress, and now it is changing more rapidly than ever before. Let’s navigate this shift together.

—RL Nabors

Homework/further reading

From Penny Press to Protocols: The Structural Shift AI Forces on the Internet picks up where Stratechery left off, drawing parallels with the telegraph’s disruption of the Penny Press and proposing new protocols for content consumption.
Everything Is Chrome is a deep dive into Chrome’s history, positing that Google has been trying to control the Web through Chrome’s marketplace power.

Originally posted as a thread on BlueSky, enough people asked for a link to the article version that it seemed like the perfect launch post for this promised series. Thank you to everyone who pre-signed up for this first missive and for encouraging me to share my thoughts and findings on how AI is reshaping our relationship with the Web and each other.

Special thanks to the following folks for their proofreading and feedback: Jack Herrington, Paul Asjes, Vale, Bob Monsour.

About the author

I’ve had the privilege of contributing to projects at Microsoft Edge, Meta’s React team, and AWS Amplify, in addition to working with startups focused on AI, auth, and design systems. As an Invited Expert at the W3C, I helped shape the Web Animation Standard and collaborated with Mozilla on development tools and documentation initiatives. Through these experiences, I’ve learned that while I may have been doing this work for quite some time, the real magic happens in the community. I believe deeply that we—developers, creators, and users—are the true architects of the Web. While standards bodies and large companies influence its direction, the Web ultimately belongs to all of us. Every small contribution we make helps pave the way forward.

The opinions I share in these posts are my own and do not reflect those of my clients or employers, past or present. They reflect only publicly available information.

Thanks for reading The Agentic Web! Sign up to receive new posts in your email.

Scrape raw HTML with AgentQL's REST API

Rachel-Lee Nabors — Mon, 31 Mar 2025 10:26:52 +0000

Sometimes you need to extract data from HTML but you don't have a URL to pass to AgentQL's REST API. Fret not: now our REST API endpoint supports querying directly from raw HTML!

With this functionality, you can scrape data from pages even if you're working behind a firewall, fetching pages with a custom crawler, or integrating with internal tools. Pass the HTML as a string and your AgentQL query, and AgentQL will return structured data in JSON.

You asked for it: scraping web pages without a URL

You asked if it was possible to scrape data without Playwright. You told us you were already fetching HTML using custom crawlers. We heard you! This new capability is perfect for querying data from:

Private and internal network pages
Previously crawled pages and HTML dumps
Archived HTML files and snapshots

It can even be used to scrape difficult-to-reach and heavily anti-botted pages. You can navigate to the page using a stealth crawler or your own browser, save the page's HTML or copy it as a string, and follow the steps below!

How to extract data from an HTML string

You can pass HTML directly in your API request like so:

curl -L 'https://api.agentql.com/v1/query-data' \
-H 'Content-Type: application/json' \
-H 'X-API-Key: <YOUR-API-KEY>' \
-d '{
  "html": "<!DOCTYPE html><html><body><h1>Main Page</h1></body></html>",
  "query": "{ heading }"
}'

AgentQL will process the HTML and return structured JSON:

{
  "heading": "Main Page"
}

Got a large, unwieldy chunk of HTML? Or a local file(s) you want to send without the copy-pasting all the HTML every time? Most HTML is going to run into JSON formatting errors if you pass it through raw, anyway. Try this out:

curl -L 'https://api.agentql.com/v1/query-data' \
-H 'Content-Type: application/json' \
-H 'X-API-Key: <YOUR-API-KEY>' \
-d "$(jq -n \
 --arg html "$(cat blog.html)" \
 '{query: "{ heading }", html: $html}'
)"

This combines reading the file with cat alongside jq's power to properly format HTML for a JSON context (escaping double quotes, etc).

Get started extracting data with HTML and AgentQL

This feature is available now—no opt-in or special flag required. Learn more in our guide to getting data from HTML with AgentQL or the REST API Reference

If you have any questions, join our Discord, and we will help you out. We love hearing from you! Find us on X, or Bluesky, too!

—The TinyFish team building AgentQL

AgentQL Enters the Agentic World with Langchain and LlamaIndex

Rachel-Lee Nabors — Sun, 16 Mar 2025 05:05:40 +0000

AgentQL now integrates with Langchain and LlamaIndex, connecting AI agents and retrieval-augmented generation (RAG) models to the web. AgentQL extracts real-time, structured data from web pages and allows agents to interact with links, forms, and other web UI.

AgentQL’s tools enable Langchain and LlamaIndex agents to navigate the web, extract data, and take action. For structured retrieval, AgentQL functions as a document loader/web reader, making live data available for AI-powered search and decision-making. Whether you are building autonomous AI agents or dynamic retrieval systems, these integrations turn the entire web into a data source for AI applications.

With this launch, AgentQL fully steps into the agentic world—where AI systems can not only retrieve information but also act on it.

AgentQL Tools for Langchain Agents and Workflows

Langchain is a framework for building AI-powered agents and workflow automation. You can use AgentQL as a tool inside Langchain agents to:

Extract structured data from web pages in real time
Click, navigate, and interact with websites using a Playwright-powered browser
Use the web as a dynamic knowledge source for AI-powered research

Read the full integration guide and check out our examples:

AgentQL as a WebReader for LlamaIndex and RAG Pipelines

LlamaIndex is a framework for retrieval-augmented generation (RAG)—helping LLMs query structured data. You can use AgentQL as a WebReader inside LlamaIndex to:

Index live web pages and search them as structured documents
Retrieve fresh market data, research papers, and news in real time
Power AI search with always-updated information

Read the full integration guide!

Start Building Today

AgentQL now integrates seamlessly with Langchain agents and LlamaIndex retrieval workflows. AI applications can reach beyond foundation models by fetching, interacting with, and processing live web pages and their contents.

Langchain guide: docs.agentql.com/integrations/langchain
LlamaIndex guide: docs.agentql.com/integrations/llamaindex

We can't wait to see what you build! Give us a shout out on Discord, X, or Bluesky!

—The TinyFish team building AgentQL

AgentStack + AgentQL: Give your Agents access to the Internet

Rachel-Lee Nabors — Thu, 13 Mar 2025 20:33:20 +0000

AgentQL now integrates with AgentStack, enabling developers to rapidly scaffold AI agents that can extract and interact with live web data.

AgentStack is a developer tool designed to quickly set up agent projects, offering a suite of CLI utilities for efficient code generation. With the AgentQL integration, you can enhance your agents with real-time web data extraction, all within a streamlined development process.

What Can You Do With AgentQL + AgentStack?

Sentiment Analysis on Live Data: Build agents that fetch the latest product reviews and analyze customer sentiment in real-time.
Automated Research Assistants: Develop agents that gather and summarize current information from the web, keeping you updated effortlessly.
Market Monitoring Tools: Create agents that track competitor websites for pricing and product updates, delivering insights directly to your dashboard.

How It Works

Make an agent with AgentStack and equip it with AgentQL so it can access the web:

Set up AgentStack using this installation guide to set up your development environment.
Add AgentQL to your Agent's stack by following our integration documentation.
Generate Agents and Tasks: Utilize AgentStack's CLI commands to create agents and define tasks that leverage AgentQL's capabilities.

Get started with these examples

Explore these examples to see the integration in action:

Sentiment Analyzer: This agent collects and analyzes sentiment from live web data. View on GitHub
Research Assistant: This agent is designed to perform automated web research and provide concise summaries. View on GitHub
Market Monitor: This agent tracks prices and gets you the best deals. View on GitHub

Try It Today

The AgentQL and AgentStack integration is live now. Visit the integration guide to get started.

We can't wait to see what you build! Give us a shout out on Discord, X, or Bluesky!

—The TinyFish team building AgentQL

AgentQL MCP Server: Structured Web Data for Claude, Cursor, Windsurf, and more

Rachel-Lee Nabors — Thu, 13 Mar 2025 04:32:50 +0000

AgentQL now supports Model Context Protocol (MCP), making it easier than ever to integrate live, structured web data into tools that support MCP—like Claude, Cursor, and Windsurf.

MCP is an open standard that allows AI assistants to use external tools. With the AgentQL MCP Server, your AI models can now extract live data from web pages—transforming unstructured HTML into structured, usable data.

Every web page: an endpoint

Before today, most AI models were limited to static knowledge or aging web scrapes, and much content on the web is provided without an API to access. Now, with AgentQL + MCP, your assistant can:

Extract data from unstructured HTML websites
- Extract just the content from websites without ads or UI
- Pull live YouTube search results, filtering out ads
Business Intelligence from recent news and markets
- Get pricing data from different marketplaces
- Collect financial report information from public web pages
Research with fresh data from the web
- Include recent news from outlet sites in the context of your conversation
- Retrieve data from online articles

All in real-time. No scraping scripts. No extra APIs. No vector storage.

Set Up AgentQL MCP Server

To get started, install the AgentQL MCP package:

npm install -g agentql-mcp

Try out AgentQL MCP Server today

Try running some of the following commands:

Get the social links from agentql.com
Return the top five video titles and their links from https://www.youtube.com/results?search_query=agentql in a list in markdown
Return the top international news articles from ground.news into a CSV

And, in our opinion AgentQL’s “killer usecase”—using Claude as a cookbook:

Get this recipe for me: https://www.justonecookbook.com/pressure-cooker-japanese-curry/

Now your AI models can retrieve structured web data in real-time—without needing a prebuilt API or manual scraping.

Open to collaborations

The AgentQL MCP Server is opensource, and we value the community’s feedback and contributions. We work hard to make tools that let your agents reach beyond their foundation model’s knowledge and leverage the whole world wide web. If you’ve got ideas or challenges, we want to hear them and solve them with you.

We can't wait to see what you build! Give us a shout out on Discord, X, or Bluesky!

—The TinyFish team building AgentQL

Dify + AgentQL: Build AI Apps with Live Web Data, No Code Needed

Rachel-Lee Nabors — Tue, 11 Mar 2025 22:38:17 +0000

AgentQL now integrates seamlessly with Dify, making it easier than ever to build AI applications that access and process real-time web data. Dify provides a user-friendly, low-code platform for designing and deploying AI applications—no complex backend setup required. Now, with AgentQL’s Extract Web Data tool, your AI apps can retrieve live information from any webpage in real time.

What Can You Build with AgentQL + Dify?

Check out live demos:

Price Deal Finder Compare product prices across websites in real time
Research Assistant Pull fresh data from the web to answer research questions
News Aggregator Extract and structure news or job postings from any website

How to Use AgentQL in Dify

Using AgentQL in an Agent

Create a new app in Dify and select "Agent."
Under Tools, click + Add.
Search for "AgentQL."
Select Extract Web Data under AgentQL.
Prompt the agent to extract web data using either:
- An AgentQL query (learn more)
- A natural language prompt

Using AgentQL in a Workflow

Create a new app in Dify and choose "Chatflow" or "Workflow."
Click Add Block.
Under Tools, search for "AgentQL".
Select Extract Web Data under AgentQL. Configure input variables in the tool’s UI, then run the pipeline to extract web page data.

For detailed setup instructions, check out the Dify integration guide.

Try It Today

The AgentQL + Dify integration is live now. Build AI-powered applications that interact with real-time web data—without touching a single API. Visit the integration guide to start building your AI applications with live web data.

We can't wait to see what you build! Give us a shout out on Discord, X, or Bluesky!

—The TinyFish team building AgentQL

Zapier + AgentQL: No-Code Web Data for Smarter Workflows

Rachel-Lee Nabors — Mon, 10 Mar 2025 22:08:42 +0000

Our first Integration Week drop is here: AgentQL is now available in Zapier! It's never been easier to bring fresh web data into your no-code workflow automations.

If you use Google Sheets, Notion, Slack, Airtable, or any of the 8,000+ apps Zapier connects to, you can now pull in live, structured data from any webpage—no coding required. Check out our templates in Zapier and see what you can build today.

What Can You Do With AgentQL + Zapier?

This integration brings the power of AgentQL to the automation-first world, helping teams save time and eliminate manual web research and data entry. AgentQL extracts data from the web into a format you can plug into other Zaps. You could do any of the following and more:

Aggregate news headlines to Google sheets or Excel spreadsheets
Monitor competitor prices and update spreadsheets automatically
Pull news headlines and send them to Slack for your team
Track product availability and get notified when stock changes
Extract job postings and add them to an Airtable for easy tracking

How It Works

No scraping scripts. No complex setup. Just instant access to web data in your favorite tools in a few steps:

Create a Zap in Zapier.
Select AgentQL's "Extract Data" action.
Use your AgentQL API key (from the dev portal) to connect AgentQL to Zapier.
Enter the URL & query to extract structured data—learn more about how to write an AgentQL query in this guide or try the "suggest a query" feature in the Playground.
Send that data anywhere—email, Google Sheets, Notion, Slack, Discord, and more.

Try AgentQL and Zapier in your workflows today

The integration is live now. Head to our Zapier integration page to get started. Try out our templates for aggregating posts from sites into Google sheets or Excel spreadsheets. You can even create your own browser extension to grab data as you browse!

We can't wait to see what you build! Give us a shout out on Discord, X, or Bluesky!

—The TinyFish team building AgentQL

Something is coming.

Rachel-Lee Nabors — Sat, 08 Mar 2025 04:32:40 +0000

Next week, we’re launching five new ways to bring AgentQL into the tools you already love to use.

We’ve been watching you—how you build, where you automate, and what you need to make your life easier.

So we built you something special.

Something that makes it easier than ever to integrate live web data into your workflows.

Five days, five integrations (and a few surprises).

First reveal drops Monday.

Are you ready?

Follow along next week as we reveal each drop on X, Bluesky, and LinkedIn.
Join the conversation in our Discord and tell us what you think.
Be the first to try it—every launch will include examples and documentation to get started fast.

Stay tuned.

—The TinyFish team building AgentQL

Automated web application testing with AI and Playwright

Rachel-Lee Nabors — Fri, 28 Feb 2025 22:23:46 +0000

Co-authored with our friends at Heal.dev

End-to-end tests are one of the most valuable testing strategies in any codebase. They are the ultimate integration tests that ensure the application's critical paths are thoroughly tested. With them, testing teams ensure the high quality releases and reliable user experiences while reducing the need for human testers to perform repetitive tasks and manual testing.

However, these are often overlooked when building web applications, with good reasons! The most common justifications for not using end-to-end testing are:

Cost of implementations: Even if modern testing processes are easier to use than ever, they still use traditional testing methods. Users still need to spend significant time and manual effort choosing selectors on a page, and variable testing time can lead to slow iteration cycles.
Cost of maintenance: End-to-end tests are often coupled with the implementation of the application's user interface, meaning they require a maintenance when modifications are made that impact appearance and user behavior, like UI updates or new features and capabilities.
Flakiness: Because a website's performance is impacted by a wide range of parameters, from CPU scheduling to network stability, end-to-end tests tend to be flaky. Web applications behave slightly differently from one run to another, leading to a significant loss of time when executing such tests.

This tutorial explores AI-powered testing to build more stable tests, leading to greater testing efficiency, shorter testing cycles, and faster release cycles. To do so, you will use:

Playwright is a browser automation tool and testing library by Microsoft. Its robust locator API significantly reduces flakiness when interacting with an element.
AgentQL has AI-powered features that turn any web page into an API you can interact with using Natural Language queries. Instead of targeting elements on the page imperatively using CSS selectors like .submit_button12lsi or XPath , you describe what they are semantically, like "submit_button". These self-healing tests dramatically reduce maintenance as they no longer break when the page's layout or content changes.
Node's built-in test runner
Node's native support for TypeScript (still experimental)

This means the package.json will be particularly small:

{
  "name": "agentql-tutorial",
  "type": "module",
  "version": "1.0.0",
  "main": "index.js",
  "scripts": {
    "test": "node --env-file=.env --test --experimental-strip-types test/**.ts"
  },
  "author": "",
  "license": "MIT",
  "dependencies": {
    "@playwright/test": "^1.49.0",
    "agentql": "^1.6.0",
    "playwright": "^1.49.0"
  }
}

Using Node.js test runner with Playwright and TypeScript

Setting up tests with Playwright and Node.js

Start with a test for the TODOMVC application hosted by the Playwright team that will:

Add elements to the To-do List
Confirm that the elements are present

The whole test file is available here, or you can follow along below

Setting up the test context

You need to provide a Playwright Page for each test to run your tests. That way, if you create multiple tests, they will stay independent from each other:

import {
  after,
  afterEach,
  before,
  beforeEach,
  describe,
  test
} from 'node:test';

import { chromium } from 'playwright';
import type { Page } from 'playwright';
import { expect } from '@playwright/test';

import type { TestContext, SuiteContext } from 'node:test';

const TEST_TO_PAGE = new WeakMap<TestContext | SuiteContext, Page>();
let browser = null;
before(async () => {
  browser = await chromium.launch();
});

after(async () => {
  await browser.close();
});

beforeEach(async (testContext) => {
  const page = await browser.newPage();
  TEST_TO_PAGE.set(testContext, page);
});

afterEach(async (testContext) => {
  await TEST_TO_PAGE.get(testContext).close();
});

Notice that two imports are made with import type instead of just import. This ensures Node.js knows what imports to remove when transforming code from TypeScript to JavaScript. For instance, Page is not exported by the playwright module, but its type is! So, such import needs to be removed before running the code; otherwise, it will crash with an import error.

Now take a look at the hooks (before, after, beforeEach, afterEach) in the code:

let browser = null;
before(async () => {
  browser = await chromium.launch();
});

after(async () => {
  await browser.close();
});

The before hook is executed once before each test, and the after hook is executed once after all the tests. This is the perfect place to create and destroy resources needed for the test session!

Start the browser instance you will use for testing: close it in the after hook to ensure it is permanently closed. If closing the browser depends on tests in the suite, it might not happen, as the tests might crash and prevent the browser.close() method from being called.

Additional hooks run before and after each test. In the beforeEach, create a Page for each test. In the Node.js test runner, each test has a single TestContext object. You can use this object to link resources related to the test here:

const TEST_TO_PAGE = new WeakMap<TestContext | SuiteContext, Page>();
beforeEach(async (testContext) => {
  const page = await browser.newPage();
  TEST_TO_PAGE.set(testContext, page);
});

afterEach(async (testContext) => {
  await TEST_TO_PAGE.get(testContext).close();
});

This creates a new page before each test and links it with the current test using a WeakMap.

Note: you could also have used the Playwright test runner directly, as it would have handled these fixtures!

Write the test

test('should allow to add a TODO item', async (testContext) => {
  const page = TEST_TO_PAGE.get(testContext);
  await page.goto('https://demo.playwright.dev/todomvc');
  const newTodo = page.getByPlaceholder('What needs to be done?');

  // Create 1st todo.
  await newTodo.fill('Use AgentQL');
  await newTodo.press('Enter');

  // Make sure the list only has one todo item.
  await expect(page.getByTestId('todo-title')).toHaveText('Use AgentQL');

  // Create 2nd todo.
  await newTodo.fill('Use Heal.dev');
  await newTodo.press('Enter');

  // Make sure the list now has two todo items.
  await expect(page.getByTestId('todo-title')).toHaveText(['Use AgentQL', 'Use Heal.dev']);
});

The first two lines of the tests are here to:

Get the Page object
Navigate to the URL

Then, use a locator to find the input for new To-do List items:

const newTodo = page.getByPlaceholder('What needs to be done?');

This locator represents the component on the page. Playwright will handle all interactions with it for you!

await newTodo.fill('Use AgentQL');
await newTodo.press('Enter');

This tells Playwright to type Use AgentQL in the input field and to press ENTER to submit the task.

Now check that the task has been properly added to the list:

await expect(page.getByTestId('todo-title')).toHaveText('Use AgentQL');

This asks the Playwright assertion library to check that within the elements on the page with the id todo-title, one has the text "Use AgentQL," meaning that the item has been added to the list.

Add another item, "Use Heal.dev," and check if it is present in the list, too:

await newTodo.fill('Use Heal.dev');
await newTodo.press('Enter');
await expect(page.getByTestId('todo-title')).toHaveText(['Use AgentQL', 'Use Heal.dev']);

Running the tests

Run the test with the following in your terminal:

node --experimental-strip-types test/without-agentql.ts

You don't need the --test flag when running a single test file. Node.js will run these standalone files as tests automatically!

Running these tests with --experimental-strip-types allows you to run .ts files without node-ts or any translation!

Self-healing tests with AgentQL's smart locators

You just wrote a test for the To-do List application! But what happens if the placeholder of the task input changes? Or if the testId used for the elements in the list is modified? The test will break!

Of course, you can always maintain the test as part of your development cycle and manually change the locators, but what if you could use AI to handle that?

Using AI to identify the target elements is a great way to stabilize these tests. AgentQL uses machine learning models that enable you to test using the meaning of elements on the page rather than their implementation details.

Adding AgentQL's AI-powered testing tools to the test

You can follow along or find the complete file here.

First, import AgentQL into your test:

import type { PageExt } from 'agentql';
import * as AgentQL from 'agentql';

And add another beforeEach hook to wrap Page with AgentQL:

beforeEach(async (testContext) => {
  const page = TEST_TO_PAGE.get(testContext);

  const pageExt = await AgentQL.wrap(page);

  TEST_TO_PAGE.set(testContext, pageExt);
});

This adds the following methods to the page:

getByPrompt returns an HTML element from the page.
queryData returns data from the page.

These powerful methods can invoke AI to locate elements and data on the page by focusing on what the elements expose as features or data instead of their implementation.

getByPrompt('Entry to add todo items') replaces getByPlaceholder('What needs to be done?') in the test:

test('should allow to add a TODO item', async (testContext) => {
  const page = TEST_TO_PAGE.get(testContext);
  await page.goto('https://demo.playwright.dev/todomvc');
  const newTodo = await page.getByPrompt('Entry to add todo items');

  // Create 1st todo.
  await newTodo.fill('Use AgentQL');
  await newTodo.press('Enter');
  const todoItemsQuery = `{
    todo_items[]
  }`;

  // Make sure the list only has one todo item.

  const { todo_items: itemsBefore } = await page.queryData(todoItemsQuery);

  Assert.deepStrictEqual(itemsBefore, ['Use AgentQL']);

  // Create 2nd todo.

  await newTodo.fill('Use Heal.dev');
  await newTodo.press('Enter');

  // Make sure the list only has one todo item.
  const { todo_items: itemsAfter } = await page.queryData(todoItemsQuery);
  Assert.deepStrictEqual(itemsAfter, ['Use AgentQL', 'Use Heal.dev']);
});

This means that as long as there is an input to add a task to the list, AgentQL will provide a locator for it, even if its placeholder changes.

This also works for extracting data from the page:

const todoItemsQuery = `{
  todo_items[]
}`;

const { todo_items: itemsBefore } = await page.queryData(todoItemsQuery);

This AgentQL query returns the elements on the page that match the description todo_items without impacting how the page is implemented.

Both methods are self-healing and resilient to page changes, making your test more robust!

Running the test

node --test --experimental-strip-types --env-file=.env test/**.ts

Note: Use the --env-file flag to target the file containing AGENTQL_API_KEY=<MY_API_KEY>

What’s next?

This example mitigates some of the top challenges of end-to-end testing: the cost of maintenance and flakiness. But there are still things your testing team can do to keep these tests running over time for more complex applications.

Update relevant tests as part of your feature development process

Using AgentQL's smart locators that target elements by their on-page context rather than hardcoded selectors makes tests more stable over time. But this only works as long as user interactions on the page stay the same. For example, the next version of this To-do List could require users to add a time estimate to the entry. You would still need to update the relevant tests to address new user stories like this.

Build a testing infrastructure

Hosting end-to-end tests locally is difficult. Browsers require a certain amount of CPU and memory (and GPUs on some websites). Also, scheduling browser-based tests can be challenging. You must ensure the network state is stable to load the target website and assets and even to connect to AgentQL servers!

Integrate complex testing workflows

Many testing building blocks are painful to build right. For example, testing sending and receiving emails, designing clever ways to test randomness, and writing assertions. How do you automate “check that a modal is open” in a stable way? That remains tricky!

The Future of End-to-End Testing is AI-Powered Testing Tools

Next generation reliable testing tools will embrace self-healing testing, relieving testing efforts by reducing maintenance overhead and ensuring tests remain stable as applications evolve. Heal.dev is an test automation platform that leverages AgentQL to make this a reality by:

Writing reliable end-to-end tests
Maintaining your tests with AI-driven insights
Managing the entire test infrastructure for you
Adding implicit assertions to catch defects before they reach users

By combining AI-powered tools like AgentQL with modern testing platforms, we’re entering an era where stability, maintainability, and efficiency are no longer trade-offs. With the right approach, you can build tests that:

Adapt to UI changes without constant updates
Reduce flakiness by targeting functionality rather than fragile selectors
Scale effortlessly with AI-assisted assertions and infrastructure management

But stability is just the beginning.

True test automation isn’t just about catching bugs—it’s about enabling teams to move faster confidently. By integrating AI-driven solutions, developers can shift their focus from fixing flaky tests to shipping high-quality software at scale.

If unreliable tests are slowing you down, it’s time to stop fighting your test suite—and start healing it. Check out Heal.dev and experience the future of test automation today.

Introducing Scheduled Scraping Workflows

Rachel-Lee Nabors — Mon, 02 Dec 2024 17:00:00 +0000

AgentQL now supports scheduled scraping jobs! 🎉

You told us scraping the same sites over and over was a hassle—especially when paired with manual scheduling tools. We heard you loud and clear. We created an experimental scheduling feature to give you a simpler, more intuitive way to automate recurring jobs.

We built this experimental feature for developers who regularly scrape the same websites for updated information. Instead of juggling custom scripts and cron jobs, you can now set up and automate these tasks directly from the AgentQL Dev Portal. And here’s the best part: you can download the scraped data as JSON, so it’s ready to drop into your workflows.

If your work involves tracking updates on dynamic sites—like news headlines, product prices, or public datasets—this feature is your new best friend. Think of it as your always-on assistant for regular data collection.

Get started scheduling

Head over to the new scheduling page on our Dev Portal and use the Add New Workflow button.

Set the URL(s) for the pages that you'd like to extract data from and the AgentQL query you'd like to run, and specify when you'd like it to run.

The full Step-by-Step Guide to scheduling scraping jobs is available in our documentation.

Experimental limitations

Because this feature is still experimental, we're limiting the number of scraping jobs you can schedule to:

2 workflows per user
5 urls per workflow
10 runs per workflow

If you need more scraping jobs, please reach out to us!

We’re excited to see how you use Scheduled Scraping to streamline your workflows. This is just the start—try it out, and let us know how we can make it even better on our community Discord. Your input will shape what comes next!

Happy scheduling! 🐟

—The Tiny Fish Team Building AgentQL

AgentQL Launch Week Recap—make the web AI-ready

Rachel-Lee Nabors — Mon, 18 Nov 2024 21:01:12 +0000

Last week, we rolled out five exciting updates to make AgentQL faster, stealthier, and easier to use than ever before with a JavaScript SDK, REST API endpoint, and query generator. Whether you’re building AI agents, scraping data, or launching websites, these new features were built to help you unlock data and interact with the web like never before.

One week, one step closer to making the web AI-ready

Here’s a quick recap of everything we launched during AgentQL Launch Week—with links to explore each feature in detail.

REST API—AgentQL Everywhere

This endpoint gives you the power to retrieve data from any public web page via HTTP request with only a URL and an AgentQL query, unlocking hundreds of workflows. We showed off the AgentQL REST API with a Zapier integration—try it for yourself with this Zapier template (just add your free API key).

Learn more

JavaScript SDK—AgentQL for Fullstack Developers

Our Python SDK has been popular with the data engineering crowd for awhile now, but from collaborations with companies like Heal.dev and feedback from our community, we learned that the AgentQL’s powerful parsing and automation capabilities would be welcome in Node.js environments. Fullstack teams can now integrate AgentQL into their testing, scraping, and automation workflows effortlessly.

JavaScript SDK Launch Post
Get started scraping and automating with the AgentQL JavaScript SDK in five minutes
Install the JavaScript SDK
Check out the API Reference
Get up and running quickly with our collection of JavaScript Examples on GitHub

Query Generation—AI-Powered Query Creation

With Query Generation, the AgentQL Playground can write queries for you. Just describe what you’re looking for in natural language, and AgentQL generates a fully functional query. Our founder Keith Zhai had much to share on X about how this feature unlocks not just engineers but researchers, journalists, and analysts.

The implications for business and journalism are profound. For years, the bottleneck wasn’t a lack of data—it was a lack of accessible tools to make sense of it.

You can learn more in the original Query Generation Launch Post or try it out on the Playground.

Fast Mode—Fast by Default

Speed met precision with Fast Mode, now the default for all AgentQL queries. This lighter-weight mode delivers snappy results without sacrificing quality, making it perfect for high-volume scraping, monitoring, and quick prototyping.

Learn more in the Fast Mode Launch Post or check out this handy guide for how to switch to Fast Mode for faster execution times. (If you need extra precision for complex or data-intensive dynamic websites, you can still enable Standard Mode to prioritize accuracy over speed!)

Stealth Mode—Enhanced Bot Detection Evasion

We upgraded Stealth Mode to minimize bot detection when scraping or automating actions on third-party websites. Check out the launch post, this handy guide to Avoiding Bot Detection with Stealth Mode, or implement it today with our Stealth Mode Example Script (now in JavaScript, too!)

What’s Next? You tell us.

We can’t wait to see how you’ll use these new features, and we’re already thinking about what’s next for AgentQL. Your feedback is critical to shaping our roadmap, so if you haven’t already, we’d love for you to take a moment to fill out our Community Survey.

Thank you for joining us for Launch Week! We hope these new features empower you to build, automate, and explore with AgentQL. If you have questions, ideas, or want to chat about all things AgentQL, join us on our community Discord. Let’s build the future together.

—The Tiny Fish Team Building AgentQL

Get data from any page: AgentQL’s Rest API Endpoint—Launch week day 5

Rachel-Lee Nabors — Fri, 15 Nov 2024 14:36:11 +0000

We’ve made it to the final day of Launch Week, and we’re wrapping up with our biggest product offering yet: the AgentQL REST API!

This simple API turns any page into an endpoint for content. With only a URL and a query string, you can query web pages and retrieve the same precise, structured data you have come to expect from our SDKs and Playground—even from dynamic content or complex websites. The possibilities are endless, from scheduling scraping jobs to building data pipelines, or even empowering AI agents to interact with the web in all new ways.

Like the demo? Get the Zapier template and add your free API key!

What is the AgentQL REST API?

The AgentQL REST API allows you to send queries via HTTP to retrieve structured data from any publicly available website with a single request. This flexibility makes it a powerful tool for developers, automation experts, and AI agent providers working with dynamic data applications.

Use AgentQL with any programming language or tool that supports HTTP. By integrating with tools like Zapier or embedding it in backend systems, AgentQL’s REST API opens up new possibilities for automating large-scale data acquisition and training data collection. It’s an ideal solution for users who need the precision of AgentQL’s smart locators without the SDK setup—just plug in your query and get your data!

Unlock the Internet’s Data

There’s so much information available on the web that’s been dammed up in HTML soup. We get it. Not every organization can build an API to share their information. That’s why AgentQL was designed to do the heavy lifting for both publishers and consumers!

This REST API is for anyone looking to add fast, precise, structured data collection to their workflows. It’s handy for:

AI Agents: The REST API is a perfect match for autonomous AI agents that need to pull in real-world data dynamically. With AgentQL’s REST endpoint, agents can retrieve and interact with live data, making them more adaptable and versatile.
Web Scraping & Monitoring Applications: If your use case involves tracking product prices, following market trends, monitoring real estate listings, or scraping job boards, the REST API lets you automate it from one endpoint.
Developers & Data Engineers: Integrate AgentQL into your backend or data processing pipelines for efficient web scraping, analytics, and content retrieval.
Automation Experts & No-Code Users: Set up automated data flows in platforms like Zapier without needing extensive coding knowledge.

How to Get Started with the AgentQL REST API

Get your API key: Every request requires an X-API-Key header. Generate your free API key here—no credit card is required!
Define Your Request: Pass the URL you want to fetch data from and an AgentQL query.

{
  "url": "https://scrapeme.live/shop",
  "query": "{ products[] { product_name product_price } }"
}

Set your headers: Before making the API request, include the necessary headers for authentication and content type. These headers authorize the request and specify the data format being sent.

X-API-Key: use your AgentQL API key for authentication.
Content-Type: use application/json to indicate that the request body is in JSON format, allowing the server to interpret the data correctly.

Send the Request: Use your preferred HTTP client (like curl, Postman, or an HTTP library in Python or your preferred language) to make a POST request to the AgentQL REST API endpoint.

curl -X POST "https://api.agentql.com/v1/query-data" \
  -H "X-API-Key: $AGENTQL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://scrapeme.live/?s=fish&post_type=product",
    "query": "{ products[] { product_name product_price } }"
  }'

(Make sure to replace $AGENTQL_API_KEY with your actual API key!)

Get Results: The API response will return data as structured JSON, ready to be used directly in applications, AI agents, or analytics workflows.

{
  "data": {
    "products": [
      {
        "product_name": "Qwilfish",
        "product_price": "£77.00"
      },
      {
        "product_name": "Huntail",
        "product_price": "£52.00"
      },
      ...
    ]
  },
  "metadata": {
    "request_id": "ecab9d2c-0212-4b70-a5bc-0c821fb30ae3"
  }
}

Use Cases for the REST API

The REST API unlocks a wide array of possibilities for developers and agent builders:

Automated scraping activities: Add fetching data from any web page to your automation tasks, whether it's dynamic content or JavaScript-based content—AgentQL can handle it. Extract products, product details, and more.
Large-scale data acquisition: For training data collection, nothing beats structured, real-time data, fresh from the Internet.
Automated Data Pipelines: Feed up-to-date data into your backend processing or data analytics systems. This is perfect for e-commerce monitoring, news aggregation, or lead generation.
Agentic AI Integrations: AI agents can now pull dynamic data on-demand, making decisions based on up-to-date information. Whether your agent monitors trends, analyzes competitor data, or interacts with external content, the REST API equips it with live data capabilities.
Zapier Integrations for No-Code Automation: Use the REST API with Zapier to trigger data extraction workflows and automate repetitive tasks without writing code.

Ready to get real-time data from the web?

Get started making some AgentQL REST API calls today!

API Reference docs
Guide to scraping data from live web pages with the API
Zapier template (just add your free API key!)

That’s a wrap for Launch Week!

This Launch Week we shared a new JavaScript SDK, Fast Mode, Stealth Mode, Query Generation from Natural Language and now this. We hope the REST API becomes an invaluable part of your toolkit, whether you’re a developer, automation expert, or building the next generation of agentic AI. We'd love to see what you build and are here to help on our community Discord.

Thank you for being part of AgentQL Launch Week! We're working hard to make sure AgentQL is your favorite tool for automation and data retrieval. Please let us know what you think—take our survey!

Happy coding and automating!

—The Tiny Fish Team Building AgentQL