DEV Community: AgentQL

Scrape raw HTML with AgentQL's REST API

Rachel-Lee Nabors — Mon, 31 Mar 2025 10:26:52 +0000

Sometimes you need to extract data from HTML but you don't have a URL to pass to AgentQL's REST API. Fret not: now our REST API endpoint supports querying directly from raw HTML!

With this functionality, you can scrape data from pages even if you're working behind a firewall, fetching pages with a custom crawler, or integrating with internal tools. Pass the HTML as a string and your AgentQL query, and AgentQL will return structured data in JSON.

You asked for it: scraping web pages without a URL

You asked if it was possible to scrape data without Playwright. You told us you were already fetching HTML using custom crawlers. We heard you! This new capability is perfect for querying data from:

Private and internal network pages
Previously crawled pages and HTML dumps
Archived HTML files and snapshots

It can even be used to scrape difficult-to-reach and heavily anti-botted pages. You can navigate to the page using a stealth crawler or your own browser, save the page's HTML or copy it as a string, and follow the steps below!

How to extract data from an HTML string

You can pass HTML directly in your API request like so:

curl -L 'https://api.agentql.com/v1/query-data' \
-H 'Content-Type: application/json' \
-H 'X-API-Key: <YOUR-API-KEY>' \
-d '{
  "html": "<!DOCTYPE html><html><body><h1>Main Page</h1></body></html>",
  "query": "{ heading }"
}'

AgentQL will process the HTML and return structured JSON:

{
  "heading": "Main Page"
}

Got a large, unwieldy chunk of HTML? Or a local file(s) you want to send without the copy-pasting all the HTML every time? Most HTML is going to run into JSON formatting errors if you pass it through raw, anyway. Try this out:

curl -L 'https://api.agentql.com/v1/query-data' \
-H 'Content-Type: application/json' \
-H 'X-API-Key: <YOUR-API-KEY>' \
-d "$(jq -n \
 --arg html "$(cat blog.html)" \
 '{query: "{ heading }", html: $html}'
)"

This combines reading the file with cat alongside jq's power to properly format HTML for a JSON context (escaping double quotes, etc).

Get started extracting data with HTML and AgentQL

This feature is available now—no opt-in or special flag required. Learn more in our guide to getting data from HTML with AgentQL or the REST API Reference

If you have any questions, join our Discord, and we will help you out. We love hearing from you! Find us on X, or Bluesky, too!

—The TinyFish team building AgentQL

AgentQL Enters the Agentic World with Langchain and LlamaIndex

Rachel-Lee Nabors — Sun, 16 Mar 2025 05:05:40 +0000

AgentQL now integrates with Langchain and LlamaIndex, connecting AI agents and retrieval-augmented generation (RAG) models to the web. AgentQL extracts real-time, structured data from web pages and allows agents to interact with links, forms, and other web UI.

AgentQL’s tools enable Langchain and LlamaIndex agents to navigate the web, extract data, and take action. For structured retrieval, AgentQL functions as a document loader/web reader, making live data available for AI-powered search and decision-making. Whether you are building autonomous AI agents or dynamic retrieval systems, these integrations turn the entire web into a data source for AI applications.

With this launch, AgentQL fully steps into the agentic world—where AI systems can not only retrieve information but also act on it.

AgentQL Tools for Langchain Agents and Workflows

Langchain is a framework for building AI-powered agents and workflow automation. You can use AgentQL as a tool inside Langchain agents to:

Extract structured data from web pages in real time
Click, navigate, and interact with websites using a Playwright-powered browser
Use the web as a dynamic knowledge source for AI-powered research

Read the full integration guide and check out our examples:

AgentQL as a WebReader for LlamaIndex and RAG Pipelines

LlamaIndex is a framework for retrieval-augmented generation (RAG)—helping LLMs query structured data. You can use AgentQL as a WebReader inside LlamaIndex to:

Index live web pages and search them as structured documents
Retrieve fresh market data, research papers, and news in real time
Power AI search with always-updated information

Read the full integration guide!

Start Building Today

AgentQL now integrates seamlessly with Langchain agents and LlamaIndex retrieval workflows. AI applications can reach beyond foundation models by fetching, interacting with, and processing live web pages and their contents.

Langchain guide: docs.agentql.com/integrations/langchain
LlamaIndex guide: docs.agentql.com/integrations/llamaindex

We can't wait to see what you build! Give us a shout out on Discord, X, or Bluesky!

—The TinyFish team building AgentQL

AgentStack + AgentQL: Give your Agents access to the Internet

Rachel-Lee Nabors — Thu, 13 Mar 2025 20:33:20 +0000

AgentQL now integrates with AgentStack, enabling developers to rapidly scaffold AI agents that can extract and interact with live web data.

AgentStack is a developer tool designed to quickly set up agent projects, offering a suite of CLI utilities for efficient code generation. With the AgentQL integration, you can enhance your agents with real-time web data extraction, all within a streamlined development process.

What Can You Do With AgentQL + AgentStack?

Sentiment Analysis on Live Data: Build agents that fetch the latest product reviews and analyze customer sentiment in real-time.
Automated Research Assistants: Develop agents that gather and summarize current information from the web, keeping you updated effortlessly.
Market Monitoring Tools: Create agents that track competitor websites for pricing and product updates, delivering insights directly to your dashboard.

How It Works

Make an agent with AgentStack and equip it with AgentQL so it can access the web:

Set up AgentStack using this installation guide to set up your development environment.
Add AgentQL to your Agent's stack by following our integration documentation.
Generate Agents and Tasks: Utilize AgentStack's CLI commands to create agents and define tasks that leverage AgentQL's capabilities.

Get started with these examples

Explore these examples to see the integration in action:

Sentiment Analyzer: This agent collects and analyzes sentiment from live web data. View on GitHub
Research Assistant: This agent is designed to perform automated web research and provide concise summaries. View on GitHub
Market Monitor: This agent tracks prices and gets you the best deals. View on GitHub

Try It Today

The AgentQL and AgentStack integration is live now. Visit the integration guide to get started.

We can't wait to see what you build! Give us a shout out on Discord, X, or Bluesky!

—The TinyFish team building AgentQL

AgentQL MCP Server: Structured Web Data for Claude, Cursor, Windsurf, and more

Rachel-Lee Nabors — Thu, 13 Mar 2025 04:32:50 +0000

AgentQL now supports Model Context Protocol (MCP), making it easier than ever to integrate live, structured web data into tools that support MCP—like Claude, Cursor, and Windsurf.

MCP is an open standard that allows AI assistants to use external tools. With the AgentQL MCP Server, your AI models can now extract live data from web pages—transforming unstructured HTML into structured, usable data.

Every web page: an endpoint

Before today, most AI models were limited to static knowledge or aging web scrapes, and much content on the web is provided without an API to access. Now, with AgentQL + MCP, your assistant can:

Extract data from unstructured HTML websites
- Extract just the content from websites without ads or UI
- Pull live YouTube search results, filtering out ads
Business Intelligence from recent news and markets
- Get pricing data from different marketplaces
- Collect financial report information from public web pages
Research with fresh data from the web
- Include recent news from outlet sites in the context of your conversation
- Retrieve data from online articles

All in real-time. No scraping scripts. No extra APIs. No vector storage.

Set Up AgentQL MCP Server

To get started, install the AgentQL MCP package:

npm install -g agentql-mcp

Try out AgentQL MCP Server today

Try running some of the following commands:

Get the social links from agentql.com
Return the top five video titles and their links from https://www.youtube.com/results?search_query=agentql in a list in markdown
Return the top international news articles from ground.news into a CSV

And, in our opinion AgentQL’s “killer usecase”—using Claude as a cookbook:

Get this recipe for me: https://www.justonecookbook.com/pressure-cooker-japanese-curry/

Now your AI models can retrieve structured web data in real-time—without needing a prebuilt API or manual scraping.

Open to collaborations

The AgentQL MCP Server is opensource, and we value the community’s feedback and contributions. We work hard to make tools that let your agents reach beyond their foundation model’s knowledge and leverage the whole world wide web. If you’ve got ideas or challenges, we want to hear them and solve them with you.

We can't wait to see what you build! Give us a shout out on Discord, X, or Bluesky!

—The TinyFish team building AgentQL

Dify + AgentQL: Build AI Apps with Live Web Data, No Code Needed

Rachel-Lee Nabors — Tue, 11 Mar 2025 22:38:17 +0000

AgentQL now integrates seamlessly with Dify, making it easier than ever to build AI applications that access and process real-time web data. Dify provides a user-friendly, low-code platform for designing and deploying AI applications—no complex backend setup required. Now, with AgentQL’s Extract Web Data tool, your AI apps can retrieve live information from any webpage in real time.

What Can You Build with AgentQL + Dify?

Check out live demos:

Price Deal Finder Compare product prices across websites in real time
Research Assistant Pull fresh data from the web to answer research questions
News Aggregator Extract and structure news or job postings from any website

How to Use AgentQL in Dify

Using AgentQL in an Agent

Create a new app in Dify and select "Agent."
Under Tools, click + Add.
Search for "AgentQL."
Select Extract Web Data under AgentQL.
Prompt the agent to extract web data using either:
- An AgentQL query (learn more)
- A natural language prompt

Using AgentQL in a Workflow

Create a new app in Dify and choose "Chatflow" or "Workflow."
Click Add Block.
Under Tools, search for "AgentQL".
Select Extract Web Data under AgentQL. Configure input variables in the tool’s UI, then run the pipeline to extract web page data.

For detailed setup instructions, check out the Dify integration guide.

Try It Today

The AgentQL + Dify integration is live now. Build AI-powered applications that interact with real-time web data—without touching a single API. Visit the integration guide to start building your AI applications with live web data.

We can't wait to see what you build! Give us a shout out on Discord, X, or Bluesky!

—The TinyFish team building AgentQL

Zapier + AgentQL: No-Code Web Data for Smarter Workflows

Rachel-Lee Nabors — Mon, 10 Mar 2025 22:08:42 +0000

Our first Integration Week drop is here: AgentQL is now available in Zapier! It's never been easier to bring fresh web data into your no-code workflow automations.

If you use Google Sheets, Notion, Slack, Airtable, or any of the 8,000+ apps Zapier connects to, you can now pull in live, structured data from any webpage—no coding required. Check out our templates in Zapier and see what you can build today.

What Can You Do With AgentQL + Zapier?

This integration brings the power of AgentQL to the automation-first world, helping teams save time and eliminate manual web research and data entry. AgentQL extracts data from the web into a format you can plug into other Zaps. You could do any of the following and more:

Aggregate news headlines to Google sheets or Excel spreadsheets
Monitor competitor prices and update spreadsheets automatically
Pull news headlines and send them to Slack for your team
Track product availability and get notified when stock changes
Extract job postings and add them to an Airtable for easy tracking

How It Works

No scraping scripts. No complex setup. Just instant access to web data in your favorite tools in a few steps:

Create a Zap in Zapier.
Select AgentQL's "Extract Data" action.
Use your AgentQL API key (from the dev portal) to connect AgentQL to Zapier.
Enter the URL & query to extract structured data—learn more about how to write an AgentQL query in this guide or try the "suggest a query" feature in the Playground.
Send that data anywhere—email, Google Sheets, Notion, Slack, Discord, and more.

Try AgentQL and Zapier in your workflows today

The integration is live now. Head to our Zapier integration page to get started. Try out our templates for aggregating posts from sites into Google sheets or Excel spreadsheets. You can even create your own browser extension to grab data as you browse!

We can't wait to see what you build! Give us a shout out on Discord, X, or Bluesky!

—The TinyFish team building AgentQL

Something is coming.

Rachel-Lee Nabors — Sat, 08 Mar 2025 04:32:40 +0000

Next week, we’re launching five new ways to bring AgentQL into the tools you already love to use.

We’ve been watching you—how you build, where you automate, and what you need to make your life easier.

So we built you something special.

Something that makes it easier than ever to integrate live web data into your workflows.

Five days, five integrations (and a few surprises).

First reveal drops Monday.

Are you ready?

Follow along next week as we reveal each drop on X, Bluesky, and LinkedIn.
Join the conversation in our Discord and tell us what you think.
Be the first to try it—every launch will include examples and documentation to get started fast.

Stay tuned.

—The TinyFish team building AgentQL

Introducing Scheduled Scraping Workflows

Rachel-Lee Nabors — Mon, 02 Dec 2024 17:00:00 +0000

AgentQL now supports scheduled scraping jobs! 🎉

You told us scraping the same sites over and over was a hassle—especially when paired with manual scheduling tools. We heard you loud and clear. We created an experimental scheduling feature to give you a simpler, more intuitive way to automate recurring jobs.

We built this experimental feature for developers who regularly scrape the same websites for updated information. Instead of juggling custom scripts and cron jobs, you can now set up and automate these tasks directly from the AgentQL Dev Portal. And here’s the best part: you can download the scraped data as JSON, so it’s ready to drop into your workflows.

If your work involves tracking updates on dynamic sites—like news headlines, product prices, or public datasets—this feature is your new best friend. Think of it as your always-on assistant for regular data collection.

Get started scheduling

Head over to the new scheduling page on our Dev Portal and use the Add New Workflow button.

Set the URL(s) for the pages that you'd like to extract data from and the AgentQL query you'd like to run, and specify when you'd like it to run.

The full Step-by-Step Guide to scheduling scraping jobs is available in our documentation.

Experimental limitations

Because this feature is still experimental, we're limiting the number of scraping jobs you can schedule to:

2 workflows per user
5 urls per workflow
10 runs per workflow

If you need more scraping jobs, please reach out to us!

We’re excited to see how you use Scheduled Scraping to streamline your workflows. This is just the start—try it out, and let us know how we can make it even better on our community Discord. Your input will shape what comes next!

Happy scheduling! 🐟

—The Tiny Fish Team Building AgentQL

Get data from any page: AgentQL’s Rest API Endpoint—Launch week day 5

Rachel-Lee Nabors — Fri, 15 Nov 2024 14:36:11 +0000

We’ve made it to the final day of Launch Week, and we’re wrapping up with our biggest product offering yet: the AgentQL REST API!

This simple API turns any page into an endpoint for content. With only a URL and a query string, you can query web pages and retrieve the same precise, structured data you have come to expect from our SDKs and Playground—even from dynamic content or complex websites. The possibilities are endless, from scheduling scraping jobs to building data pipelines, or even empowering AI agents to interact with the web in all new ways.

Like the demo? Get the Zapier template and add your free API key!

What is the AgentQL REST API?

The AgentQL REST API allows you to send queries via HTTP to retrieve structured data from any publicly available website with a single request. This flexibility makes it a powerful tool for developers, automation experts, and AI agent providers working with dynamic data applications.

Use AgentQL with any programming language or tool that supports HTTP. By integrating with tools like Zapier or embedding it in backend systems, AgentQL’s REST API opens up new possibilities for automating large-scale data acquisition and training data collection. It’s an ideal solution for users who need the precision of AgentQL’s smart locators without the SDK setup—just plug in your query and get your data!

Unlock the Internet’s Data

There’s so much information available on the web that’s been dammed up in HTML soup. We get it. Not every organization can build an API to share their information. That’s why AgentQL was designed to do the heavy lifting for both publishers and consumers!

This REST API is for anyone looking to add fast, precise, structured data collection to their workflows. It’s handy for:

AI Agents: The REST API is a perfect match for autonomous AI agents that need to pull in real-world data dynamically. With AgentQL’s REST endpoint, agents can retrieve and interact with live data, making them more adaptable and versatile.
Web Scraping & Monitoring Applications: If your use case involves tracking product prices, following market trends, monitoring real estate listings, or scraping job boards, the REST API lets you automate it from one endpoint.
Developers & Data Engineers: Integrate AgentQL into your backend or data processing pipelines for efficient web scraping, analytics, and content retrieval.
Automation Experts & No-Code Users: Set up automated data flows in platforms like Zapier without needing extensive coding knowledge.

How to Get Started with the AgentQL REST API

Get your API key: Every request requires an X-API-Key header. Generate your free API key here—no credit card is required!
Define Your Request: Pass the URL you want to fetch data from and an AgentQL query.

{
  "url": "https://scrapeme.live/shop",
  "query": "{ products[] { product_name product_price } }"
}

Set your headers: Before making the API request, include the necessary headers for authentication and content type. These headers authorize the request and specify the data format being sent.

X-API-Key: use your AgentQL API key for authentication.
Content-Type: use application/json to indicate that the request body is in JSON format, allowing the server to interpret the data correctly.

Send the Request: Use your preferred HTTP client (like curl, Postman, or an HTTP library in Python or your preferred language) to make a POST request to the AgentQL REST API endpoint.

curl -X POST "https://api.agentql.com/v1/query-data" \
  -H "X-API-Key: $AGENTQL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://scrapeme.live/?s=fish&post_type=product",
    "query": "{ products[] { product_name product_price } }"
  }'

(Make sure to replace $AGENTQL_API_KEY with your actual API key!)

Get Results: The API response will return data as structured JSON, ready to be used directly in applications, AI agents, or analytics workflows.

{
  "data": {
    "products": [
      {
        "product_name": "Qwilfish",
        "product_price": "£77.00"
      },
      {
        "product_name": "Huntail",
        "product_price": "£52.00"
      },
      ...
    ]
  },
  "metadata": {
    "request_id": "ecab9d2c-0212-4b70-a5bc-0c821fb30ae3"
  }
}

Use Cases for the REST API

The REST API unlocks a wide array of possibilities for developers and agent builders:

Automated scraping activities: Add fetching data from any web page to your automation tasks, whether it's dynamic content or JavaScript-based content—AgentQL can handle it. Extract products, product details, and more.
Large-scale data acquisition: For training data collection, nothing beats structured, real-time data, fresh from the Internet.
Automated Data Pipelines: Feed up-to-date data into your backend processing or data analytics systems. This is perfect for e-commerce monitoring, news aggregation, or lead generation.
Agentic AI Integrations: AI agents can now pull dynamic data on-demand, making decisions based on up-to-date information. Whether your agent monitors trends, analyzes competitor data, or interacts with external content, the REST API equips it with live data capabilities.
Zapier Integrations for No-Code Automation: Use the REST API with Zapier to trigger data extraction workflows and automate repetitive tasks without writing code.

Ready to get real-time data from the web?

Get started making some AgentQL REST API calls today!

API Reference docs
Guide to scraping data from live web pages with the API
Zapier template (just add your free API key!)

That’s a wrap for Launch Week!

This Launch Week we shared a new JavaScript SDK, Fast Mode, Stealth Mode, Query Generation from Natural Language and now this. We hope the REST API becomes an invaluable part of your toolkit, whether you’re a developer, automation expert, or building the next generation of agentic AI. We'd love to see what you build and are here to help on our community Discord.

Thank you for being part of AgentQL Launch Week! We're working hard to make sure AgentQL is your favorite tool for automation and data retrieval. Please let us know what you think—take our survey!

Happy coding and automating!

—The Tiny Fish Team Building AgentQL

Natural Language Query Generation for Faster Results—Launch week day 4

Rachel-Lee Nabors — Thu, 14 Nov 2024 17:23:00 +0000

Proponents of the free and open web have often argued that “information wants to be free.” But that phrase is actually a misquote of Steward Brand from the first Hackers Conference in 1984. What he actually said is much closer to where we are today:

[I]nformation sort of wants to be expensive because it is so valuable—the right information in the right place just changes your life. On the other hand, information almost wants to be free because the costs of getting it out is getting lower and lower all of the time.

This is truer today than ever. We were promised that access to data on the web would be simple only to find it dammed behind walls of text, blockaded by a lack of APIs and the knowledge to wield them, or locked behind platforms and sold to the highest bidder.

We want to democratize access to public data so everyone can get what they need as naturally as asking a friend for help. After observing how developers interact with AgentQL’s Playground, we realized there was a way to make it even easier for people to query data from a page. What if you could describe the content you’re looking for, and AgentQL generated the query for you?

Today, we’re excited to unveil Query Generation in the AgentQL Playground, a feature that turns your natural language input into fully functional AgentQL queries. Now, instead of puzzling out syntax or sifting through dense data structures, you only need to describe the content you want to extract from a single page, and AgentQL will handle the rest.

Try it yourself in the AgentQL Playground.

Query generation is perfect for:

No-code users: No code? No problem! AgentQL’s Playground can fetch data from any public-facing web page!
Developers new to AgentQL: If you’re evaluating AgentQL and want to get started without diving into query syntax, Query Generation offers a hands-on introduction without a steep learning curve.
Busy Developers: Even if you’re familiar with AgentQL, Query Generation can speed up prototyping and exploration, especially when you’re working with dense or unfamiliar page structures.
Anyone Handling Complex Documents: If the data you need is buried in a complex or dynamically generated document, Query Generation provides a straightforward way to retrieve it.

Why Use Query Generation?

By using natural language to generate queries, you can:

Use Your Words: Explain in your own words what you’re looking for and AgentQL will create a query to do just that.
Focus on Results, Not Syntax: Free up time for analyzing and using your data rather than constructing queries.
Save Time: No need to learn or recall query syntax; just describe what you need, and AgentQL will create the query for you.
Reduce Friction: Avoid the trial and error that often comes with building complex queries, especially on pages with dense or nested information.

How to Get Started with Query Generation

Getting started is as simple as heading to the Playground:

Open the AgentQL Playground: If you’re new to AgentQL, this is your space to explore and experiment.
Click “Suggest a Query”: You’ll find a prompt in the Playground where you can describe what you’re looking for.
Describe Your Desired Output: Type in what you want to retrieve—whether it’s product names, prices, reviews, or something else entirely.
Generate the Query: AgentQL will translate your input into a query and display it for you to use or refine further.

Once AgentQL generates the query, you can run it immediately or make any necessary adjustments. It’s a fast, intuitive way to find your target data without digging into syntax.

Try it Yourself: Head to the AgentQL Playground to start experimenting with natural language query creation.

Ready to Simplify Query Building?

Whether you’re new to AgentQL or a seasoned pro, Query Generation makes data extraction faster and easier than ever. We’re piloting this feature in the Playground, so give it a try and let us know what you think!

Your feedback is what made this launch week possible, and we want more! Join our Discord or take our survey so we can keep making AgentQL serve your needs better!

Stay tuned for tomorrow’s feature—it's a big surprise!

Happy querying!

—The Tiny Fish Team Building AgentQL

Stealth Mode—Enhanced Bot Detection Evasion—Launch week day 3

Rachel-Lee Nabors — Wed, 13 Nov 2024 11:11:23 +0000

One of the biggest challenges with scraping or automating actions on external websites is getting past bot detection. It’s a constant hurdle for developers who rely on web data but find their scripts blocked or throttled by anti-bot systems due to automated activity detection.

That’s why we invested in Playwright Stealth (a port of puppeteer-extra-plugin-stealth) when its original project was no longer maintained. We call this powerful tool Stealth Mode.

AgentQL in Stealth Mode

Stealth Mode is AgentQL’s solution for minimizing the chance of detection when running scripts on third-party sites. Built with bot-detection evasion in mind, Stealth Mode leverages Playwright to simulate human-like browsing behaviors, reducing tell-tale signs of automation that can trigger anti-scraping mechanisms. It works by masking automation indicators, mimicking natural browsing, and adjusting browser settings to evade detection—making your interactions smoother, safer, and less likely to get blocked.

Modern websites often deploy sophisticated bot detection systems that analyze browser behavior, properties, and interactions to distinguish human users from bots. Without Stealth Mode, several signals can reveal the use of automation, such as:

navigator.webdriver property: Indicates whether a browser is controlled by automation.
Headless mode detection: Certain differences in how headless browsers behave compared to regular browsers.
Missing or inconsistent browser APIs: Bots often miss certain APIs or provide inconsistent values (for example, WebGL, and media codecs).

These detection methods can result in websites blocking automated sessions, presenting CAPTCHA challenges, or even banning IP addresses. AgentQL Stealth Mode helps to bypass such measures by minimizing the traces of automation and simulating real users in a real browser environment.

More than just bot detection: Why imitate regular users?

Stealth Mode is ideal for anyone using AgentQL to scrape or automate interactions with sites they don’t directly control. There are a few good reasons to imitate real humans:

Data Scrapers working with popular sites: If you’re extracting public data from sites with lots of bot traffic, Stealth Mode reduces the chance of interruptions, allowing your scripts to run smoothly.
More human-like interactions: Whether you’re testing in a live environment or automating workflows for dynamic web pages, Stealth Mode lets AgentQL behave more like a real person, getting more consistent results.
Developers focused on reliability: Stealth Mode is a must-have for developers who need uninterrupted script execution, particularly when running frequent, high-volume, or long-duration tasks on external URLs.

As websites become increasingly sophisticated in detecting automation and bot activity, tools that simulate human behavior are essential for reliable, uninterrupted access. With Stealth Mode, you can ensure that your scripts operate effectively on third-party sites, allowing you to focus on building and gathering content on websites rather than troubleshooting detection issues.

Stealth Mode is particularly beneficial for:

Ensuring Stability: Your scripts are less likely to be flagged and disrupted, so you can collect data or perform actions reliably.
Expanding Automation Possibilities: With fewer detection risks, you’re free to scale your operations without constant modifications.
Reducing Manual Intervention: Fewer interruptions mean less time tweaking or restarting scripts due to detection.

How to Enable and Use Stealth Mode

You can enable Stealth Mode in AgentQL by calling the enable_stealth_mode function on an AgentQL page object:

from playwright.async_api import async_playwright
import agentql
async with async_playwright() as Playwright, await playwright.chromium.launch(
    headless=False,
) as browser:
    page = await agentql.wrap_async(browser.new_page())

        # enables stealth mode
    await page.enable_stealth_mode()
    await page.goto("https://bot.sannysoft.com/")
    await page.wait_for_timeout(30000)

Visit our documentation to learn more about using Stealth Mode with headless browsers.

Run the sample script

We’ve prepared an in-depth working example script to show how to use Stealth Mode and modified browser attributes alongside other anti-bot best practices to overcome anti-bot measures by:

Randomizing various HTTP request headers the browser sends to the server. Randomizing User-Agent, Accept-Language, Referer, etc, helps make consecutive requests look more like they are coming from normal users.
Use real request headers. Use browser fingerprinting—set real request headers on a headless browser to imitate a real human.
Randomizing browser window size. Real users use different sizes of windows. Some websites track the window size; if it’s the same for all requests, it’s a sign of a bot.
Randomizing timezone and geolocation. Some websites track the timezone and geolocation, and if it’s the same for all requests, it’s a sign of a bot.
(Optional) Using a proxy server. You would need to secure proxy services from an external proxy provider (NetNut, BrightData, or similar) to configure things like host, username, and password separately.

Best Practices for Avoiding Bot Detection

When using Stealth Mode, keep these practices in mind to maximize effectiveness:

Adjust interaction timing: Extend the period of time between actions and use random delays to mimic natural browsing patterns and reduce bot detection triggers.
Use rotating User Agents: Cycling through different user-agent strings can further reduce detection risks.
Reduce request frequency: A real human won’t flood a server with thousands of requests. Avoid overloading sites with excessive requests all at once—this improves reliability and reduces detection likelihood.

Ready to Start Scraping & Automating in Stealth?

AgentQL’s Stealth Mode is an essential tool for reliably executing scripts and collecting data without triggering anti-bot detection. Visit our Stealth Mode reference for a full feature breakdown, check out our guide to using Stealth Mode, explore our example scripts to see Stealth Mode in action.

Your feedback is what made this launch week possible, and we want more! Join our Discord or take our survey so we can keep making AgentQL serve your needs better!

We are keen to take on maintaining the original Playwright Stealth but are having difficulty reaching the maintainer. If you or someone you know can put us in touch, please reach out!

Stay tuned for tomorrow’s feature: Query Generation—an AI-powered way to create complex queries faster than ever.

Happy coding!

—The Tiny Fish Team Building AgentQL

Fast Mode—AgentQL is “fast by default”—Launch week day 2

Rachel-Lee Nabors — Tue, 12 Nov 2024 19:02:30 +0000

We built AgentQL to tackle the complexities of parsing HTML and automating tasks with precision and accuracy without needing XPath, CSS or DOM selectors, or regular expressions. But we also know that sometimes speed is everything. We created a Fast Mode that uses a lighter model for just such situations. In production, we noticed that it was meeting our high quality bar for the majority of your tasks—with faster execution times.

So, today, we’re excited to announce that Fast Mode is now the default for all AgentQL users. We’ve added this optimized, lighter-weight mode to our core functionality because it keeps quality high while providing a massive improvement to response times, giving you quicker results right when you need them.

What is Fast Mode?

Fast Mode is AgentQL’s way of delivering quality results with reduced processing time. It’s built on a lighter-weight model that’s quick on the draw yet maintains the accuracy that AgentQL is known for when handling complex websites. So whether you’re scraping, testing, automating, or running continuous data flows, Fast Mode ensures your workflows stay responsive and agile.

Who Benefits from Fast Mode?

Fast Mode is for AgentQL users who value quick feedback cycles without concurrent requests and want to cut down on waiting times without compromising on the quality they’ve come to expect. Here are some ideal scenarios:

Data Scrapers Handling High Volumes: AgentQL allows for faster web scraping of complex and dynamic websites to pull insights more quickly, letting you scale up without hitting bottlenecks.
Simple Scrapers Suffering from Slow Web Scraping: Extracting data takes even less time with Fast Mode, making even an unoptimized web scraper perform just that much better.
Automation Workflows: If you’re testing UIs, verifying data, or automating interactions, Fast Mode lets you run checks in seconds, keeping your development pipeline flowing.
Prototyping and Experimentation: Perfect for testing new ideas and building proofs of concept, Fast Mode gives you results almost instantly, so you can focus on iterating and refining.

How Do You Use Fast Mode?

You’re already using it! Fast Mode is now the default for every script execution, so no setup is required. Just start running queries, and AgentQL will optimize for speed automatically, no need to worry about unoptimized scripts.

If you need extra precision for complex or data-intensive dynamic websites, you can still enable Standard Mode to prioritize accuracy over speed. But for most scenarios, Fast Mode is a more than suitable approach. If you have already configured Standard Mode, here’s a handy guide for how to switch to Fast Mode for faster execution times.

Ready to Go Fast? You Already Are!

We hope Fast Mode enhances your development experience by making your scraping projects and workflows quicker and smoother. Your feedback is what made this launch week possible, and we want more! Join our Discord or take our survey so we can keep making AgentQL serve your needs better!

Stay tuned—tomorrow’s launch is Stealth Mode for enhanced bot detection evasion. You’re going to love it!

Happy coding!

—The Tiny Fish Team Building AgentQL