DEV Community: 许映洲

I Spent 3 Hours Debugging a CSS Selector. Then I Found a Better Way.

许映洲 — Fri, 29 May 2026 12:28:56 +0000

I Spent 3 Hours Debugging a CSS Selector. Then I Found a Better Way.

Last Tuesday I needed to scrape product prices from an e-commerce site. "Simple," I thought. Just query the price element and done.

Three hours later I was still staring at my screen, cursing whoever decided to use dynamically-generated class names.

What went wrong

The page used something like this:

<span class="price sc-dkzDqf jhKMRz eBvMAx">¥199</span>

Those classes? They change every deploy. My querySelector('.price') worked for about 2 days, then the frontend team pushed an update and everything broke. No warning, no deprecation, just... gone.

I tried being smarter. Used [class*="price"] — worked until they added a "price-match" badge that had nothing to do with actual prices. Then I tried span.price — but sometimes the price was in a <div>.

At 11 PM I realized: I was spending more time maintaining selectors than actually getting data.

The shift in thinking

Here's what I eventually figured out: instead of hard-coding selectors, what if I could just describe what I wanted in plain English and let the tool figure out the DOM?

Something like:

# Instead of writing this:
page.querySelector('.price.sc-dkzDqf')

# What if I could just do this:
extract "the product price" --from https://example.com/product/123

That's basically what I've been using for the past month. The tool figures out the selector, and when the page changes, it re-discovers it.

Real numbers from a real project

I tracked my time for 2 weeks before and after:

Before: ~45 min per site to write scraper, then 10-15 min/week fixing broken selectors
After: ~5 min per site to set up, 0 min/week maintenance (selectors auto-recover)

Over a month, across 8 sites, that's roughly 12 hours saved. Not life-changing, but enough to actually go home on time.

The catch

It's not perfect. Complex interactions (multi-step forms, SPAs with heavy JS) still need manual scripting. And if a page changes its entire structure, no amount of auto-discovery will save you.

But for 80% of my scraping tasks — get text, get prices, get links — it's been surprisingly reliable.

If you're spending more time fixing selectors than building features, maybe it's time to try a different approach. The xbrowser CLI is what I've been using — it's open source, works with your existing Chrome, and doesn't require writing 50 lines of Puppeteer every time.

I Replaced 30 Minutes of Daily Browser Chores with One Cron Job

许映洲 — Thu, 28 May 2026 16:05:27 +0000

Every morning, I'd sit down at my desk, grab a coffee, and open my browser.

Then the ritual began.

Log into Google Search Console. Check yesterday's indexing stats. Switch to Ahrefs. Look up a handful of keyword rankings. Open Bing Webmaster Tools. Manually submit the three articles I published last night. Jump over to a backlink checker. See if those reciprocal links are still alive.

Done? Not quite. I still had to open Medium and publish yesterday's draft. Then Dev.to. Then schedule a Reddit post in the relevant subreddit.

By the time I finished all of this, my coffee was cold and a third of my morning was gone.

What Do All These Tasks Have in Common?

If you look closely, every single one of these operations follows the same pattern:

Open a browser
Navigate to a URL
Log in (and re-log in when the session expires)
Find the right input field or button
Fill in content or click something
Wait for the result
Close the tab

None of these steps are hard. But every single one requires your personal involvement. Your hand bounces between mouse and keyboard. Your eyes dart across a dozen tabs.

Here's the thing — you're a paid engineer, not a human Selenium.

What if these operations could be turned into command-line tasks?

# Submit sitemap
seo ping --sitemap https://mysite.com/sitemap.xml

# Check Google rankings
google search "my keyword" --json | jq '.[0].position'

# Publish an article
devto publish --file article.md --tags "javascript,webdev"

# Check backlinks
curl-check --url https://partner.com --find "mysite.com"

One command. Press Enter. Done.

From Manual to Automated: A Three-Step Process

Step 1: Abstract Operations into Commands

Any browser operation can be decomposed into: open page → locate element → perform action → get result.

Take "submit sitemap to Google." Manually, you'd:

Open https://www.google.com/ping?sitemap=
Append your sitemap URL
Press Enter
See the success message

As a CLI command, this is one line:

xbrowser navigate "https://www.google.com/ping?sitemap=https://mysite.com/sitemap.xml"

Or take "check if a new article is indexed." Manually:

Open Google
Search site:mysite.com/my-new-article
Check if there are results

CLI version:

xbrowser google search "site:mysite.com/my-new-article" --json

You get structured JSON back. You can filter with jq, search with grep, pipe it into other commands.

Step 2: Combine Commands into a Script

Individual commands solve individual tasks. But your daily work is a sequence of tasks — the kind you repeat every morning.

So write a script:

#!/bin/bash
# daily-seo-check.sh

SITE="https://mysite.com"
KEYWORDS=("javascript tutorial" "node.js guide" "react best practices")
LOG_FILE="/var/log/seo-$(date +%Y%m%d).log"

echo "=== SEO Daily Report $(date) ===" >> "$LOG_FILE"

# Submit sitemap
echo "[1/4] Submitting sitemap..." >> "$LOG_FILE"
xbrowser seo ping --sitemap "$SITE/sitemap.xml" >> "$LOG_FILE" 2>&1

# Check rankings
echo "[2/4] Checking rankings..." >> "$LOG_FILE"
for kw in "${KEYWORDS[@]}"; do
  position=$(xbrowser google search "$kw" --json | jq -r '.[0].position // "Not found"')
  echo "  $kw: Position $position" >> "$LOG_FILE"
done

# Check index count
echo "[3/4] Checking index count..." >> "$LOG_FILE"
count=$(xbrowser google search "site:$SITE" --json | jq 'length')
echo "  Indexed pages: $count" >> "$LOG_FILE"

# Check backlinks
echo "[4/4] Checking backlinks..." >> "$LOG_FILE"
while IFS=',' read -r url anchor; do
  found=$(xbrowser crawl "$url" --find "$anchor" | jq -r '.found')
  echo "  $url: $found" >> "$LOG_FILE"
done < backlinks.csv

echo "Done. Report saved to $LOG_FILE"

This script does everything you used to do manually every morning — submit sitemap, check rankings, count indexed pages, verify backlinks.

Run it once:

chmod +x daily-seo-check.sh
./daily-seo-check.sh

Twenty seconds later, you have your daily data. No browser. No clicking around. No waiting for pages to load.

Step 3: Hand It Off to Cron

The script works. Running it manually takes 20 seconds. But you'll still forget sometimes.

So let the machine do it:

# Edit crontab
crontab -e

Add these lines:

# Daily SEO check at 9 AM
0 9 * * * /home/user/scripts/daily-seo-check.sh

# Hourly ranking check, append to log
0 * * * * xbrowser google search "javascript tutorial" --json >> /var/log/rankings-hourly.log

# Submit sitemap at 3 AM daily (off-peak)
0 3 * * * xbrowser seo ping --sitemap https://mysite.com/sitemap.xml

# Weekly backlink audit, every Monday at 8 AM
0 8 * * 1 /home/user/scripts/check-backlinks.sh

Save and exit. From now on, these tasks run automatically.

The first thing you do each morning isn't opening a browser — it's checking the log:

cat /var/log/seo-$(date +%Y%m%d).log

Output looks something like this:

=== SEO Daily Report 2025-01-15 09:00:01 ===
[1/4] Submitting sitemap... OK
[2/4] Checking rankings...
  javascript tutorial: Position 7
  node.js guide: Position 12
  react best practices: Position 3
[3/4] Checking index count...
  Indexed pages: 847
[4/4] Checking backlinks...
  https://partner1.com: found
  https://partner2.com: found
  https://partner3.com: NOT FOUND ⚠️
Done.

Thirty seconds to review. Infinitely faster than doing it by hand.

Not Just SEO: Any Repetitive Browser Task Can Be Automated

Maybe you're not an SEO engineer. But you definitely have browser tasks you repeat daily.

Developers: Check GitHub Actions build status every morning? See if CI passed? Verify npm package publication?

# Hourly CI monitor
0 * * * * xbrowser github actions --repo myorg/myrepo --status failed >> /var/log/ci-monitor.log

Content creators: Need to cross-post to Medium, Dev.to, and Hashnode every day?

# One command, multiple platforms
xbrowser devto publish --file article.md --tags "js,webdev"
xbrowser medium publish --file article.md --tags "javascript,web-development"

E-commerce operators: Check competitor pricing, store ratings, inventory alerts daily?

# Daily competitor price check at 8 AM
0 8 * * * xbrowser crawl "https://competitor.com/product/123" --extract '.price' >> /var/log/price-monitor.log

Social media managers: Schedule posts, check engagement metrics, monitor brand mentions?

# Hourly brand mention check
0 * * * * xbrowser twitter search "mybrand" --json | jq '.[].text' >> /var/log/brand-mentions.log

The specific task doesn't matter. What matters is that the pattern is always the same.

The Power of Pipes: CLI's Real Advantage

You might be thinking: "I could do all this with a Python script."

You're right. But CLI has something Python scripts can't match: pipe composition.

# Check rankings → filter top 10 → send to Slack
xbrowser google search "javascript tutorial" --json \
  | jq '[.[] | select(.position <= 10)]' \
  | curl -X POST "$SLACK_WEBHOOK" -d @-

# Scrape competitor price → compare threshold → email alert
xbrowser crawl "$COMPETITOR_URL" --extract '.price' \
  | awk -v threshold=99 '{if ($1 < threshold) print "ALERT: Competitor price dropped to " $1}' \
  | mail -s "Price Alert" me@example.com

# Daily SEO report → generate Markdown → convert to PDF → email
cat /var/log/seo-$(date +%Y%m%d).log \
  | pandoc -f markdown -o /tmp/seo-report.pdf \
  | mail -s "Daily SEO Report" team@example.com -A /tmp/seo-report.pdf

Each command does one thing well. Pipes chain them together. This is the Unix philosophy at work — and it's why CLI automation is more elegant than writing monolithic scripts.

You don't need to write a new script for every new requirement. You just recombine existing commands with pipes.

Common Concerns

"Won't automated actions get my account banned?"

Not if you control the frequency. Cron's minimum granularity is one minute. Set it to run once per hour, and nobody will mistake you for a bot. It's exactly like doing it manually — except you don't have to sit at your computer.

"What if the operation fails?"

Scripts can handle errors just fine. Cron can send emails on failure:

# Send email notification on failure
0 9 * * * /home/user/scripts/daily-seo.sh || echo "SEO check failed" | mail -s "Alert" me@example.com

Or get more granular with set -e and trap inside the script:

#!/bin/bash
set -euo pipefail

cleanup() {
  echo "Script failed at line $1" | mail -s "SEO Script Error" me@example.com
}
trap 'cleanup $LINENO' ERR

"I don't have a server. Can I run cron locally?"

macOS has launchd (more powerful than cron). Linux has systemd timers. Windows has Task Scheduler. And cron itself works everywhere.

If you have a spare VPS (the $5/month kind), even better — upload your scripts, configure cron, and forget about it.

Let's Do the Math

Conservatively, here's how much time an SEO engineer spends on daily repetitive browser tasks:

Task	Manual Time	CLI Time
Submit sitemap	3 min	2 sec
Check 5 keyword rankings	10 min	10 sec
Check index count	2 min	2 sec
Check 10 backlinks	15 min	5 sec
Publish to 3 platforms	10 min	15 sec
Check competitor pricing	5 min	3 sec
Total	45 min	37 sec

45 minutes saved per day. That's 22.5 hours per month — nearly three full workdays.

You could spend that time writing code, doing analysis, or honestly, just taking a break. All of those are better than clicking around in a browser.

What You Can Do Today

If you want to start right now, I'd suggest beginning with the smallest possible automation:

Identify the browser task you repeat most often today
Find a CLI command that does the same thing
Verify the result is correct
Add it to cron

Don't try to automate everything at once. Automate one thing. Save some time. Then automate the next thing.

Iterative. No rush.

I've been using xbrowser for this — a browser automation CLI that wraps Puppeteer's capabilities into a command-line interface. It supports Google search, web scraping, SEO ping, and other common operations, and works well with cron for scheduled automation. If you have similar daily repetitive tasks, give it a try.

Playwright Is a Test Framework, Not an Automation Tool — Here's the Difference

许映洲 — Thu, 28 May 2026 15:57:15 +0000

A friend asked me last week: "I want to search Google from the command line and get the results. How hard can it be?"

"Easy," I said. "Just use Playwright."

He came back the next day: "I spent all afternoon reading Playwright docs and I still haven't gotten it working. I'm at 30 lines of code and I don't even have the search results yet."

I looked at his code:

const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch({ headless: true });
  const context = await browser.newContext();
  const page = await context.newPage();

  await page.goto('https://www.google.com');
  await page.fill('input[name="q"]', 'node.js best practices');
  await page.press('input[name="q"]', 'Enter');
  await page.waitForSelector('#search');

  const results = await page.$$eval('.g', (els) =>
    els.map((el) => ({
      title: el.querySelector('h3')?.textContent,
      url: el.querySelector('a')?.href,
    }))
  );

  console.log(results);
  await browser.close();
})();

23 lines. Just to search Google.

And that code is fragile — Google's search result page structure changes frequently, and those CSS selectors could break at any moment. When they do, you have to open the browser, debug the DOM, find new selectors, update the code, and run it again.

"Isn't there a simpler way?"

There is. But before we get to that, we need to clear up a fundamental misunderstanding.

Test Framework ≠ Automation Tool

This is a distinction most people miss. Playwright, Selenium, Cypress — they are all test frameworks, not automation tools.

A test framework's core design goals are:

Assertion-driven: Verify that page behavior matches expectations
Report generation: Produce HTML/JSON test reports
Parallel execution: Run tests across multiple browsers and devices simultaneously
CI integration: Work within GitHub Actions, Jenkins, and other pipelines
Debugging tools: Trace viewers, screenshot diffing, video playback

These features matter when you're writing tests. But if you just want to "search Google and get results"?

You don't need assertions. You don't need test reports. You don't need parallel execution. You need: open page → extract data → move on.

An automation tool's core design goals are completely different:

CLI-driven: One command, one operation
One-shot execution: No need for persistent test suites
Pipe-friendly: Output can be processed by other commands
Fast feedback: Seconds, not minutes
Script-friendly: Embed directly in bash scripts or cron jobs

In other words: test frameworks care about "is it correct?" Automation tools care about "is it fast?"

Three Approaches, One Task

Let's compare three approaches using the same task: Google search for "node.js tutorial", extract the top 10 results.

Approach 1: Playwright

// search.js
const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch({ headless: true });
  const page = await browser.newPage();

  await page.goto('https://www.google.com/search?q=node.js+tutorial');
  await page.waitForSelector('#search');

  const results = await page.$$eval('.g', (elements) =>
    elements.slice(0, 10).map((el) => ({
      title: el.querySelector('h3')?.textContent || '',
      url: el.querySelector('a')?.href || '',
      snippet: el.querySelector('.VwiC3b')?.textContent || '',
    }))
  );

  console.log(JSON.stringify(results, null, 2));
  await browser.close();
})();

To run:

npm install playwright    # Install dependencies, download browser (~300MB)
node search.js            # Run

Cost: 300MB browser download + 23 lines of code + ongoing selector maintenance.

Approach 2: Selenium

# search.py
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import json

driver = webdriver.Chrome()
driver.get("https://www.google.com/search?q=node.js+tutorial")

WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.ID, "search"))
)

results = []
elements = driver.find_elements(By.CSS_SELECTOR, ".g")[:10]

for el in elements:
    title = el.find_element(By.CSS_SELECTOR, "h3").text if el.find_elements(By.CSS_SELECTOR, "h3") else ""
    url = el.find_element(By.CSS_SELECTOR, "a").get_attribute("href") if el.find_elements(By.CSS_SELECTOR, "a") else ""
    snippet = el.find_element(By.CSS_SELECTOR, ".VwiC3b").text if el.find_elements(By.CSS_SELECTOR, ".VwiC3b") else ""
    results.append({"title": title, "url": url, "snippet": snippet})

print(json.dumps(results, indent=2))
driver.quit()

To run:

pip install selenium          # Install dependency
# Also need to download ChromeDriver separately
python search.py              # Run

Cost: Install dependencies + download ChromeDriver + 27 lines of code + ongoing selector maintenance.

Approach 3: CLI Tool

xbrowser google search "node.js tutorial" --json --limit 10

To run:

npx xbrowser google search "node.js tutorial" --json --limit 10

Cost: One command.

Output:

[
  {
    "title": "Node.js Tutorial - W3Schools",
    "url": "https://www.w3schools.com/nodejs/",
    "snippet": "Learn Node.js with our comprehensive tutorial..."
  },
  {
    "title": "The Node.js Handbook - FreeCodeCamp",
    "url": "https://www.freecodecamp.org/news/the-nodejs-handbook/",
    "snippet": "This handbook is a getting-started guide to Node.js..."
  }
]

No browser to install. No code to write. No selectors to maintain.

The Comparison Table

Dimension	Playwright	Selenium	CLI Tool
Purpose	Test framework	Test framework	Automation tool
Install size	~300MB	~150MB + Driver	~50MB
Code per operation	20-50 lines	20-50 lines	1 line
Learning curve	High (Page Objects, Locators, Contexts)	High (WebDriver protocol, wait strategies)	Low (command-line args)
Selector maintenance	Manual	Manual	Built-in
Output format	Raw (format yourself)	Raw (format yourself)	Native JSON
Pipe support	Requires extra handling	Requires extra handling	Native
Best for	Regression tests, E2E tests	Compatibility tests, cross-browser	Daily automation, data extraction
Target audience	QA engineers	QA engineers	Developers, ops, content creators
CI integration	Built-in	Built-in	Via shell scripts
Assertion capability	Powerful	Powerful	None (use jq/awk)

They're Not Replacements — They're Complements

I'm not here to knock Playwright or Selenium. They're top-tier tools in their domains.

If you're doing regression testing — verifying 50 pages worth of core functionality before every release — Playwright is the right choice. You need assertions. You need test reports. You need the Trace Viewer to debug failing cases.

If you're doing compatibility testing — making sure your product works on Chrome, Firefox, and Safari — Selenium's cross-browser capabilities are unmatched.

But if you're doing any of these:

Checking Google rankings on a schedule
Scraping web data in bulk
Submitting sitemaps automatically
Monitoring competitor price changes
Cross-posting content to multiple platforms
Quickly searching and getting structured results from the command line

You don't need a test framework. You need an automation tool.

Using a test framework for automation is like using a cannon to kill a mosquito. It works, but the cost is way too high.

When to Use What

Here's a simple heuristic:

You need to verify "is it correct?" → Test framework
You need to quickly "get something done" → Automation tool

More specifically:

Use Playwright / Selenium when:

E2E testing: Ensure signup flows and payment flows don't break
Regression testing: Run full suite before every release
Visual regression: Screenshot comparison to detect UI changes
Performance testing: Measure page load times and interaction latency
CI/CD integration: Run automatically in your pipeline

Use CLI automation tools when:

Data extraction: Pull structured data from web pages
SEO operations: Check rankings, submit links, verify indexing
Content management: Bulk publishing, scheduled publishing
Monitoring and alerting: Periodic page status checks, price change detection
Daily chores: Search, screenshot, form filling, downloading

With shell scripts + CLI tools, you can even do things test frameworks aren't designed for:

# Monitor competitor price every hour, auto-alert on Slack when it drops
watch -n 3600 'xbrowser crawl "$COMPETITOR_URL" --extract ".price" \
  | xargs -I{} bash -c "[[ {} < 99 ]] && echo Price dropped to {} | slacksend"'

# Daily ranking report at 9 AM, generate Markdown
0 9 * * * for kw in "js tutorial" "node guide" "react tips"; do \
  echo "### $kw"; xbrowser google search "$kw" --json --limit 3 \
  | jq -r ".[] | \"- [\(.title)](\(.url))\""; done > /tmp/rank-report.md

Playwright and Selenium can't do this — not because it's technically impossible, but because it's not what they were designed for. They weren't built for "quickly execute one-off tasks."

My Recommendation

If you're a developer whose daily work involves browser operations but not testing:

Use CLI tools for 80% of your daily browser tasks
Introduce Playwright only when you genuinely need testing
Don't use Playwright for things CLI should handle — you'll just create maintenance burden

If you're a QA engineer whose primary job is testing:

Playwright or Selenium should be your primary tools
But for occasional non-testing browser operations, CLI tools save a lot of time
They complement each other. No conflict.

If you're an ops / content creator / SEO engineer whose daily work is repetitive browser operations:

CLI automation tools are your best friend
Combined with cron and shell scripts, you can automate your entire daily workflow
No need to learn a test framework. The command line is enough.

If you're looking for a CLI browser automation tool, xbrowser is worth checking out. It wraps Playwright's browser control capabilities into a command-line interface — Google search, web scraping, SEO ping, all done in one line, with native JSON output and pipe support. Built for daily automation workflows and cron-friendly scheduled tasks.

My Web Scraper Died at 3 AM Because of reCAPTCHA

许映洲 — Thu, 28 May 2026 15:46:31 +0000

3:17 AM. My phone buzzed.

It was an alert from the monitoring system — the scheduled web scraper had crashed.

I grabbed my laptop, VPN'd into the server, and pulled the logs:

Error: ElementClickInterceptedException: element click intercepted
  by iframe element: <iframe src="https://www.google.com/recaptcha/...">

The page was stuck on a reCAPTCHA challenge.

Honestly, this wasn't the first time. Last month it was hCaptcha. The month before that, Cloudflare Turnstile. Same script every time: the target site upgrades their bot detection, my scraper gets caught off guard, data collection stops, and the downstream pipeline breaks.

I stared at that CAPTCHA widget on the screen and thought: how am I supposed to deal with this?

Approach 1: Go Stealth — Hide Your Automation Traces

The first thing that comes to mind: "don't let them know you're a bot."

The Puppeteer community has a popular solution: puppeteer-extra-plugin-stealth. It patches various fingerprints that Headless Chrome exposes — navigator.webdriver, Chrome DevTools Protocol artifacts, missing plugin lists, and so on.

Here's what the code looks like:

import puppeteer from 'puppeteer';
import StealthPlugin from 'puppeteer-extra-plugin-stealth';

puppeteer.use(StealthPlugin());

const browser = await puppeteer.launch({ headless: true });
const page = await browser.newPage();
await page.goto('https://example.com/protected-page');

// Pretend to be human
await page.evaluate(() => {
  Object.defineProperty(navigator, 'webdriver', { get: () => false });
});

Sounds great, right? The problem is — it doesn't guarantee 100% effectiveness.

Cloudflare's bot detection team is continuously upgrading their systems. The maintainers of stealth plugins are essentially playing a cat-and-mouse game. You patch something today, they come up with a new detection method tomorrow.

More critically, stealth only reduces the probability of triggering a CAPTCHA — it doesn't eliminate it. For a scheduled task that needs to run 24/7, "it probably won't trigger" isn't good enough.

Verdict: Useful, but not a silver bullet. Good for reducing trigger frequency, but not suitable as your only line of defense.

Approach 2: Pay to Bypass — CAPTCHA Solving Services

If machines can't solve it, get a "smarter machine" to do it.

Services like 2Captcha, Anti-Captcha, and CapSolver work on a simple principle: you send them a screenshot of the CAPTCHA, they either use AI to solve it or dispatch it to a real human to click through, and they send you back the result.

import { solve } from '2captcha-ts';

async function bypassCaptcha(page, siteKey) {
  const result = await solve({
    sitekey: siteKey,
    pageurl: page.url(),
    method: 'recaptcha'
  });

  await page.evaluate((token) => {
    document.querySelector('#g-recaptcha-response').value = token;
  }, result.data);

  await page.click('#submit-button');
}

Each solve costs about $0.001 to $0.003. Doesn't sound like much. But let's do the math:

One scraping job hits 1,000 pages
30% of them trigger CAPTCHAs
That's 300 × $0.003 = $0.90/day = ~$27/month

A few dozen dollars a month is still within budget. But the real problems aren't about money:

Unstable success rates: reCAPTCHA v3 scores users based on behavioral signals. The token returned by the solving service might not score high enough to pass verification.
Privacy concerns: You're sending the target site's URL and site key to a third party.
High latency: From submission to result, it can take 10 seconds on the fast end, or a minute or two on the slow end — seriously dragging down scraping speed.
Ethical gray area: CAPTCHAs exist to distinguish humans from machines. Paying real humans to solve them for your bot... well, you see the issue.

Verdict: Functional, but cost, stability, and compliance are all questionable. Suitable for small-scale, non-critical use cases.

Approach 3: Guerrilla Warfare — Rotate Through Proxy Pools

Another approach: since too many requests from one IP triggers CAPTCHAs, just keep switching IPs.

import puppeteer from 'puppeteer';

const proxyList = [
  'http://user:pass@proxy1:8080',
  'http://user:pass@proxy2:8080',
  'http://user:pass@proxy3:8080',
];

async function crawlWithProxy(url) {
  const proxy = proxyList[Math.floor(Math.random() * proxyList.length)];
  const browser = await puppeteer.launch({
    args: [`--proxy-server=${proxy}`]
  });
  const page = await browser.newPage();
  await page.goto(url);
  // ... scraping logic
}

Anyone who's used proxy pools knows the drill:

Free proxies: Don't bother. If you can even connect, consider yourself lucky. Speed and reliability are essentially zero.
Paid proxies: Residential proxies have the best quality but are expensive — dozens of dollars per GB. Data center proxies are cheaper but easier to detect.
Inconsistent quality: Some IP ranges are already blacklisted by major websites. Buying them is basically throwing money away.

A friend of mine does e-commerce data collection. He spends over $2,000/month on proxies alone. He once told me, with complete sincerity: "My proxy bill is ten times my server bill."

Verdict: Can reduce trigger frequency, but expensive, and effectiveness depends entirely on proxy quality. Suitable for teams with budgets.

Approach 4: Call for Help — Pause on CAPTCHA, Let a Human Handle It

The three approaches above share a common flaw: they all try to "beat" the CAPTCHA.

But think about it differently — CAPTCHAs exist for a reason. They're designed to tell humans and machines apart. So why not let a human handle it?

This is the core idea behind "human-in-the-loop" automation:

The automation script runs normally
When a CAPTCHA is detected, pause immediately
Notify a human (send a message, push a notification, open a preview link)
The human manually completes the CAPTCHA in the browser
The automation script resumes

CAPTCHA Detection Logic

The first step is knowing when a CAPTCHA appears on the page. The detection logic is actually straightforward:

const CAPTCHA_SELECTORS = [
  'iframe[src*="recaptcha"]',
  'iframe[src*="hcaptcha"]',
  'iframe[src*="challenges.cloudflare.com"]',
  'iframe[src*="turnstile"]',
  '.g-recaptcha',
  '.h-captcha',
  '#captcha',
  'div[data-sitekey]',
];

async function detectCaptcha(page) {
  for (const selector of CAPTCHA_SELECTORS) {
    try {
      const element = await page.$(selector);
      if (element) {
        const box = await element.boundingBox();
        return {
          detected: true,
          type: guessCaptchaType(selector),
          selector,
          position: box,
        };
      }
    } catch {
      // selector didn't match, skip
    }
  }
  return { detected: false };
}

function guessCaptchaType(selector) {
  if (selector.includes('recaptcha')) return 'reCAPTCHA';
  if (selector.includes('hcaptcha')) return 'hCaptcha';
  if (selector.includes('cloudflare') || selector.includes('turnstile')) return 'Cloudflare Turnstile';
  return 'unknown';
}

This covers the major CAPTCHA providers: Google reCAPTCHA, hCaptcha, and Cloudflare Turnstile. It uses iframe src attributes and class names for feature matching — simple but effective.

What Happens After Detection?

Once a CAPTCHA is detected, the key is to let a real person see the current page and manually interact with it:

async function handleCaptcha(page) {
  const captcha = await detectCaptcha(page);
  if (!captcha.detected) return false;

  console.log(`[CAPTCHA] ${captcha.type} detected. Pausing for human intervention...`);

  // Generate a live preview URL
  const previewUrl = `http://localhost:9222/devtools/inspector.html?ws=localhost:9222`;

  // Send notification (Slack, Telegram, Discord, etc.)
  await sendNotification({
    title: 'CAPTCHA detected — manual intervention needed',
    message: `Type: ${captcha.type}\nPreview: ${previewUrl}`,
    urgent: true,
  });

  // Poll until CAPTCHA disappears (human has solved it)
  while (true) {
    await sleep(2000);
    const check = await detectCaptcha(page);
    if (!check.detected) {
      console.log('[CAPTCHA] CAPTCHA resolved. Resuming...');
      break;
    }
  }

  return true;
}

The Complete Workflow

Embed the detection logic into your normal scraping flow:

async function smartCrawl(urls) {
  const browser = await puppeteer.launch({ headless: false }); // Note: non-headless mode
  const page = await browser.newPage();

  for (const url of urls) {
    await page.goto(url, { waitUntil: 'networkidle2' });

    // Check for CAPTCHA after every page load
    const blocked = await handleCaptcha(page);
    if (blocked) {
      // CAPTCHA handled, reload the page
      await page.reload({ waitUntil: 'networkidle2' });
    }

    // Normal scraping logic
    const data = await page.evaluate(() => {
      return {
        title: document.querySelector('h1')?.textContent,
        price: document.querySelector('.price')?.textContent,
      };
    });

    await saveData(data);
    await sleep(randomInt(1000, 3000)); // Random delay to mimic human behavior
  }
}

Why This Approach Works

100% success rate: A real human solves the CAPTCHA. There's no such thing as "recognition failure."
Zero extra cost: No proxy pools, no CAPTCHA solving services.
Compliant: A real person is operating the browser. Nothing is being "bypassed."
Resilient: No matter how the target site upgrades their anti-bot system, the final step is always handled by a human.

Of course, the caveat is: this approach requires that your use case doesn't need to be fully unattended. For most small-to-medium scraping tasks, the occasional need for human intervention is perfectly acceptable.

Back to That 3 AM Alert

Honestly, I don't really worry about CAPTCHAs anymore.

My approach is simple: stealth plugin as the first line of defense to reduce trigger frequency, and when a CAPTCHA does appear, the script pauses and sends me a notification. I wake up, tap through it, and the scraper continues.

It's cheaper than paying for CAPTCHA solving services, less hassle than maintaining proxy pools, and way more reliable than trying to outsmart the CAPTCHA.

CAPTCHAs exist for a reason. They protect websites from being overwhelmed by malicious bots. That's a valid design goal. Instead of trying to "beat" them, make your automation smart enough to know when to ask for help.

After all, the best code isn't code that can solve every problem — it's code that knows when to get a human involved.

If you want to try this "human-in-the-loop" approach, check out xbrowser — it has built-in CAPTCHA detection and live preview. When a CAPTCHA is detected, it automatically pauses and generates a preview link so you can take over with one click.

I Just Wanted to Scrape One Page. Why Did I Write 50 Lines of Puppeteer?

许映洲 — Thu, 28 May 2026 15:37:25 +0000

Last Friday at 4:30 PM, my product manager walked over: "Hey, can you grab the titles from the Hacker News homepage and send me an Excel file?"

I thought: That's it? Five minutes tops.

Two hours later, I was still debugging CSS selectors.

How Things Spiraled Out of Control

Step 1: Initialize the Project

mkdir hacker-news-scraper && cd hacker-news-scraper
npm init -y
npm install puppeteer

Hit enter, waited three minutes. Puppeteer needs to download a full Chromium browser — over 200 MB. I stared at the progress bar and started questioning my life choices.

Step 2: Write the Code

"It's just a document.querySelectorAll, right?" That's what I thought. Then I opened my editor:

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({
    headless: true,
    args: ['--no-sandbox', '--disable-setuid-sandbox']
  });
  const page = await browser.newPage();

  try {
    await page.goto('https://news.ycombinator.com', {
      waitUntil: 'networkidle2',
      timeout: 30000
    });

    await page.waitForSelector('.titleline > a', {
      timeout: 10000
    });

    const titles = await page.evaluate(() => {
      const items = document.querySelectorAll('.titleline > a');
      return Array.from(items).map(el => ({
        title: el.textContent,
        url: el.href
      }));
    });

    console.log(JSON.stringify(titles, null, 2));
  } catch (err) {
    console.error('Scraping failed:', err.message);
  } finally {
    await browser.close();
  }
})();

I counted: 27 lines. And this is the minimal version — no User-Agent spoofing, no retry logic, no proxy support, no concurrency control. Add all of that and you're well past 50 lines.

Step 3: Run It

node index.js

Error: Navigation timeout of 30000 ms exceeded.

Switched to domcontentloaded, got past that. But then waitForSelector timed out — because .titleline was a relatively new class name. Hacker News had silently changed it from .storylink at some point, and nobody sent me the memo.

Step 4: Debug

Set headless: false, watched the browser open. Oh right, the selector did change. Fixed it, ran it again, finally got results.

Step 5: Wrap Up

Formatted the data as CSV, sent it to the PM. Then deleted the project directory — because I knew the next time someone wanted to scrape a different website, none of this code would be reusable.

Total time: two hours. For 30 titles.

Why Is "Simple" Browser Scraping So Complicated?

Let's think about this calmly. Where does the complexity come from?

The Framework Is Overkill

Puppeteer and Playwright are, at their core, browser testing frameworks. They're designed for developers writing complex E2E test suites — simulating user logins, filling out forms, verifying page states. Scraping webpage titles? That's maybe 1% of what they can do, but you pay the price for the other 99%.

Installing Puppeteer literally installs an entire browser on your machine. It's like wanting to open a can of soup and having to assemble an entire kitchen first.

Starting from Scratch Every Time

I wrote a scraper for Hacker News. Can I reuse it for Reddit? Nope. Different selectors, different loading strategies, different anti-bot measures. Every website is a brand new adventure.

There's no "I scraped this site before" memory, no universal selector strategy, no ability to automatically adapt when pages change. Every single time, you start from zero.

The async/await Marathon

Look at any Puppeteer script — it's a sea of await:

await browser.launch()
await browser.newPage()
await page.goto()
await page.waitForSelector()
await page.evaluate()
await browser.close()

Every single operation is asynchronous. Every one needs await. I'm not saying async is bad — browser operations genuinely need to be async. But for an "open page, grab data" task, the cognitive overhead is excessive.

Error Handling Explosion

Timeouts, missing elements, network errors, page redirects, SSL errors… every step can fail, every step needs a try-catch. A robust scraping script often has more error handling code than actual business logic.

try {
  await page.goto(url, { timeout: 30000 });
} catch (e) {
  if (e.name === 'TimeoutError') {
    // Retry with a different waitUntil strategy?
  } else {
    // Actually broken?
  }
}

try {
  await page.waitForSelector(sel, { timeout: 10000 });
} catch (e) {
  // Selector changed? Page not loaded? Blocked by anti-bot?
}

You think you're scraping data, but you're actually writing an error-handling framework.

Not Reusable

Switch to a different website and everything changes — selectors, loading strategies, anti-bot mechanisms. The only reusable part from your last script is the puppeteer.launch() boilerplate. Everything else gets rewritten.

It's like having to reinvent the knife every time you want to cook a meal.

What If Browser Operations Were as Simple as curl?

curl is beautifully simple:

curl https://api.github.com/users/octocat | jq '.login'

One line, you get your data. But curl has a fatal flaw: it doesn't execute JavaScript.

It's 2026. A huge number of websites are client-side rendered. When you curl them, you get an empty HTML shell and a bunch of <script> tags. The actual data only appears after a browser executes the JavaScript.

So what we need is a curl that can execute JavaScript.

Not a testing framework. Not a browser automation library. Just a command-line tool. You give it a command, it gives you data. Done.

What Can One Line Do?

Let's go back to the Hacker News titles scenario:

xbrowser scrape https://news.ycombinator.com

That's it. The page content in Markdown format goes straight to your terminal.

Only want the titles? Add a selector:

xbrowser goto https://news.ycombinator.com , text --selector ".titleline"

Want JSON output?

xbrowser goto https://news.ycombinator.com , text --selector ".titleline" --json

No npm init. No async/await. No try-catch. One command, results come out.

Search Engine Results

PM says: "Check where our company ranks on Google for 'AI agent'."

The traditional approach? Fire up Puppeteer, simulate a search, parse the SERP page, handle Google's dynamic loading… another 50 lines right there.

Now:

xbrowser search "AI agent" --engine google --limit 10 --full

Returns titles, URLs, and summaries. Supports Google, Bing, Baidu, DuckDuckGo — multiple engines out of the box.

Screenshots

"Take a screenshot of this page."

xbrowser goto https://news.ycombinator.com , screenshot --full-page

Full-page screenshot. No need to worry about browser window size, lazy-loaded images, or viewport settings.

Fill and Submit Forms

"Test the signup flow."

xbrowser goto https://example.com/signup , fill "#email" "test@example.com" , fill "#password" "123456" , click "#submit" , screenshot

Comma-separated command chain, one line. As natural as writing a shell pipeline.

Monitor Page Changes

"Notify me when this price drops below 500."

while true; do
  xbrowser text --selector ".price" | grep -q "^4[0-9][0-9]$" && notify-send "Price dropped!"
  sleep 3600
done

Integrates naturally with cron, shell scripts, CI/CD pipelines. Because it's a command-line tool, not an API library.

It's Not Just About "Simple"

You might be thinking: Isn't this just Puppeteer wrapped in a CLI?

Not quite. There's a fundamentally different philosophy behind this.

Waterfall vs. Faucet

Puppeteer and Playwright are like a waterfall — powerful, but you have to stand underneath to collect water, and you'll get drenched in the process. You have to manage async operations, handle lifecycles, write boilerplate.

A CLI tool should be like a faucet — turn it on, water comes out. Turn it off, it stops. Simple, direct, on-demand.

Framework vs. Tool

A framework demands you think its way. You must understand its conceptual model: Browser → Page → Frame → Element, each step is async, each step can fail.

A tool should think your way. What do you want? "Open this page" — goto. "Get this text" — text. "Take a screenshot" — screenshot. Simple as that.

Programming Interface vs. Command Interface

The flexibility of a programming interface (API) is irreplaceable — complex automation scenarios genuinely need fine-grained control. But for 80% of "open a page, grab some data" use cases, a command interface (CLI) is 10x more efficient.

Think of it like Git: you can use libgit2 to write a program that manipulates your repository, but most of the time you just run git commit -m "xxx" and call it a day.

When to Use What?

To be clear: I'm not saying Puppeteer or Playwright are bad. They're incredibly powerful in their domain. The problem is using them for the wrong jobs.

Scenario	Recommended Tool
Scrape one page's data	CLI
Extract search engine results	CLI
Quick screenshot	CLI
Integrate with shell scripts	CLI
Complex E2E test suites	Playwright
Fine-grained browser control	Puppeteer
Performance testing	Lighthouse / k6
Large-scale crawling systems	Scrapy / Custom

Tools should fit the scenario, not the other way around. Using a sledgehammer to drive a nail isn't the hammer's fault — it's yours.

Back to That Friday Afternoon

If I'd had this tool back then, my Friday would have gone like this:

xbrowser scrape https://news.ycombinator.com > hn.md

Three seconds. Then I'd toss the Markdown file to the PM and get back to my actual work.

Not because the technology is revolutionary, but because the tool matches the scale of the problem.

Scraping one page's titles should never require a full project setup.

I built xbrowser to solve exactly this — a tool that turns browser operations into command-line commands. If you're also tired of writing full projects for one-off scraping tasks, give it a try.

My AI Agent Burned 26K Tokens Doing the Same Browser Task 10 Times

许映洲 — Thu, 28 May 2026 15:04:23 +0000

Last week I ran an experiment: I asked my AI agent to search Juejin for "Rust tutorial" articles — once a day, for 10 days straight.

The result? It consumed roughly 26,000 tokens on browser operations alone. And every single day, the execution flow was identical: find the search box, type the keyword, wait for results, extract text. The AI treated each session like its first visit, fumbling from scratch every time.

I did the math. At GPT-4o pricing, that's about /bin/zsh.40–0.60 wasted. Not a fortune — but the real pain is that every token was spent on pure, unadulterated repetition.

Why Do AI Agents Burn So Many Tokens on Browser Tasks?

Let's look at a typical scenario. I ask my AI agent to search CSDN for an article title:

First attempt (exploration phase):

[Agent] Screenshots the page to analyze layout...... ~800 tokens
[Agent] Tries to locate search box → guesses #search-box... ~600 tokens
[Agent] Selector doesn't exist, inspects DOM to find the real one... ~500 tokens
[Agent] Found it: input[placeholder="搜CSDN"]
[Agent] Types search term, waits for page load... ~400 tokens
[Agent] Extracts search result list text... ~300 tokens
Total: ~2,600 tokens

Second attempt (exact same task):

[Agent] Screenshots the page to analyze layout...... ~800 tokens (again)
[Agent] Locates search box... ~600 tokens (guessing again)
[Agent] Guesses wrong once... ~500 tokens (retrying again)
[Agent] Types, waits, extracts... ~700 tokens
Total: ~2,600 tokens

10 iterations = 26,000 tokens, with nearly identical execution paths every time.

I broke down the problem into four root causes:

1. No Memory Between Sessions

AI agents have zero cross-session memory. Yesterday it figured out that input[placeholder="探索"] is Juejin's search box. Today? Gone. Forgotten. That same exploration process repeats, day after day.

2. Selector Guessing Is a Probabilistic Game

When AI analyzes page structure — usually via screenshots or DOM snapshots — finding the right CSS selector is essentially guesswork. Lucky guess? Great. Wrong guess? That's a few hundred tokens down the drain per retry. From my measurements, on complex pages the AI averages 2–3 attempts before landing on the correct selector.

3. Context Bloat from Page Data

A typical webpage's HTML runs tens of kilobytes. A single screenshot is hundreds of kilobytes. Every browser operation shoves this data into the AI's context window. Just transmitting page structure costs 800–1,500 tokens per operation.

4. Zero Reusability

The results of the first exploration — selectors, workflows, error-handling experience — vanish entirely. Every session is Groundhog Day. Every time, the AI pays the same tuition.

The Core Insight: Turn Exploration Results into Reusable Commands

Once I understood the problem, the solution felt almost embarrassingly obvious:

Let the AI explore once. Record what it learns (selectors, flows). Invoke the recorded command every time after.

Concrete example. After the first Juejin search, I recorded this:

Juejin Search:
- Search box selector: input[placeholder="探索"]
- Result list: .content-main .entry
- Flow: click search box → type keyword → press Enter → wait for load → extract text

Then I wrapped it into a CLI command:

juejin search "Rust 教程" --json

From that point on, the AI just constructs this command string. No more page analysis.

Token cost comparison:

Approach	Per-operation tokens	10x total	Repetition rate
AI operates browser directly	~2,600	~26,000	100%
Calls pre-built command	~50	~500	0%

From 26,000 down to 500. A 98% reduction. That 50 tokens is just the cost of the AI assembling the command string.

In Practice: From 3,000+ Tokens Down to 50

Let me walk through a real scenario — "AI agent monitors the price of a JD.com product daily."

Before: AI manually operates the browser (~3,000 tokens/time)

The AI generates code like this every single time:

// Code the AI generates fresh every session
const page = await browser.newPage();
await page.goto('https://item.jd.com/100012043978.html', {
  waitUntil: 'domcontentloaded'
});

// AI guesses selectors... attempt 1
let price = await page.('.p-price span', el => el.textContent).catch(() => null);
if (!price) {
  // attempt 2
  price = await page.('.price J-p-100012043978', el => el.textContent).catch(() => null);
}
if (!price) {
  // attempt 3, inspect DOM...
  const html = await page.content();
  // parse HTML structure, re-locate...
  price = await page.('[class*="price"] span', el => el.textContent);
}

// handle anti-scraping, wait for dynamic loading, catch errors...

This code looks reasonable, right? The problem is the AI regenerates this entire flow from scratch every time. And in practice, AI-generated code is even more bloated — it adds excessive try-catch blocks, verbose comments, debug logging, because it's "thinking out loud" as it writes.

After: Wrapped as a command (~50 tokens/time)

I packaged common browser operations into plugins. Now the AI agent just calls a CLI command:

# The AI only needs to generate this one line
xbrowser jd price --url "https://item.jd.com/100012043978.html" --json

Returns:

{
  "price": "2999.00",
  "originalPrice": "3299.00",
  "discount": "-9.1%",
  "inStock": true,
  "timestamp": "2025-05-28T10:30:00Z"
}

50 tokens. Done. The AI doesn't need to know JD's price selector. It doesn't need anti-scraping logic. It doesn't need to wait for dynamic content. The plugin handles all the dirty work internally.

Another example: searching content platforms

# Before: AI operates browser to search Juejin articles, ~2,600 tokens
# After:
xbrowser juejin search "Rust async programming" --limit 5 --json

Returns:

{
  "results": [
    {
      "title": "Rust 异步编程完全指南",
      "author": "xxx",
      "url": "https://juejin.cn/post/7xxx",
      "likes": 328,
      "date": "2025-05-25"
    }
  ]
}

Clean. Structured. Predictable. The AI gets JSON it can directly reason over — no more scraping text out of HTML.

The Deeper Principle: Amortize Exploration Cost into Reusable Assets

After using this approach for a month, I've arrived at a simple formula:

Total Token Cost = First Exploration Cost + Repeat Count × Per-operation Cost

Without encapsulation:  Total = 2,600 + N × 2,600
With encapsulation:     Total = 2,600 + N × 50

When N = 10:  saves 98%
When N = 50:  saves 99.6%

That initial 2,600 tokens is a necessary investment — the AI has to figure out the page structure at least once. The key insight: you only pay this tax once.

Here's an analogy: AI agents doing browser tasks is like hiring someone new to order lunch from the same restaurant every day, but they re-read the menu, ask the waiter for recommendations, and agonize over the choice every single time. Encapsulating commands is like letting them memorize their regular order. Next time, they just say the dish name.

Plugin Architecture: One Plugin Per Website

The natural extension of this approach is encapsulating by website. Each plugin packages operations for a specific site, and the AI agent calls them on demand:

# Search engines
xbrowser baidu search "AI Agent browser automation" --json
xbrowser google search "playwright token cost" --json
xbrowser bing search "CDP protocol guide" --json

# Content platforms
xbrowser juejin search "Rust 教程" --limit 10 --json
xbrowser csdn search "TypeScript generics" --json
xbrowser zhihu search "how to learn system design" --json

# E-commerce
xbrowser jd price --url "https://item.jd.com/xxx" --json

Each plugin is an npm package containing:

Selector definitions — which CSS selectors this website uses
Operation flows — what to do first, what next, how to handle errors
Data extraction — how to transform raw HTML into structured JSON

The AI doesn't need to know any of this. It just needs to know "there's a command for that."

Scaling the Approach: Beyond Token Savings

Once you start thinking this way, the benefits compound beyond just token efficiency.

Reliability

An AI guessing selectors will fail sometimes. A plugin with hardcoded selectors doesn't guess — it knows. When a website changes its layout, you update one plugin file instead of hoping the AI figures it out on the next attempt.

Speed

Constructing a CLI command takes the AI one inference step. Operating a browser manually takes 5–10 inference steps (screenshot → analyze → guess → retry → act). That's not just token savings — it's latency savings. Your agent responds faster.

Composability

Once operations are CLI commands, you can compose them:

# Search multiple platforms and compare results
xbrowser baidu search "Rust async" --json > baidu.json
xbrowser google search "Rust async" --json > google.json
diff <(jq '.results[].title' baidu.json) <(jq '.results[].title' google.json)

Try doing that with raw browser automation. You'd need the AI to orchestrate multiple browser sessions, manage context windows, and merge results. With commands, it's just shell scripting.

Auditability

When something goes wrong, you can inspect the command and its output directly. No need to replay the AI's entire chain-of-thought to figure out where the selector guess went wrong.

Some Broader Reflections

Here's what I think is the deeper issue.

The biggest cost of AI agents isn't the API bill — it's the waste of repetitive labor. A single browser operation costing 3,000 tokens isn't expensive. Repeating it 100 times is 300,000 tokens. And worse, these repetitions have zero learning value — the AI explores the same page every time but doesn't get any smarter.

Converting one-time exploration cost into reusable assets is the right engineering direction for AI agents. It's the same logic behind every abstraction we already use: we don't manually configure environments every deployment, so we have Docker. We don't manually compile every build, so we have CI/CD. AI agent browser operations need the same "encapsulation" mindset.

Imagine if every commonly-used website had a pre-built command library — GitHub operations, Jira queries, Slack message reads, Confluence page fetches. How much would AI agent token efficiency improve? How much more reliable would agents become? (Pre-built commands don't guess wrong selectors.)

That's the leap from "AI agents that sort of work" to "AI agents that actually work reliably."

If you're building browser automation for AI agents, I'd suggest trying this encapsulation approach. I've been using xbrowser — it's an open-source CLI tool purpose-built for this pattern. Plugin-based, per-website commands, and it's on GitHub if you want to take a look.

I Replaced 50-Line Puppeteer Scripts with Single CLI Commands

许映洲 — Thu, 28 May 2026 00:48:33 +0000

I Replaced 50-Line Puppeteer Scripts with Single CLI Commands â€” Here's How

Last month I spent 3 hours debugging a Puppeteer script. The task? Go to Hacker News, click the top story, scrape the content, and save it. That's it. Three actions. Three hours â€” because the selector changed, the page loaded slower than my timeout, and the async/await chain threw an error I couldn't reproduce locally.

That's when I decided to build xbrowser â€” a CLI tool that treats browser automation as commands, not code. One line to search Google. One line to scrape any page. One line to chain a complete multi-step workflow. No scripts. No async management. No boilerplate. Think of it as the web scraping CLI that sits between Playwright and curl â€” purpose-built for developers who need to interact with the web, not test it.

# What took me 50 lines of Puppeteer
xbrowser "goto https://news.ycombinator.com , click '.titleline > a' , text"

That's it. One command. Readable. Replayable. Done.

Why I Couldn't Keep Using Puppeteer and Playwright

Don't get me wrong â€” Playwright is incredible for testing. Selenium pioneered the space. But I'm not testing web apps. I'm using the web: scraping competitors, checking SEO rankings, automating my social media posting, monitoring price changes. And every headless browser tutorial I found was about testing, not about building real web scraping pipelines or automation workflows.

For those tasks, the testing tools feel like using a sledgehammer to hang a picture frame:

The setup tax is real. Every new task means npm init, install dependencies, download a browser, write boilerplate. I just want to scrape one page â€” why do I need a project?

Scripts don't compose. My "scrape Hacker News" script doesn't help me "scrape Reddit." The selectors are different, the structure is different, but the core operation is the same: go to URL, extract content.

AI agents can't use them. I build AI tools, and giving an LLM a 50-line async script to manage is a recipe for hallucinated selectors and broken promises. Agents need simple, declarative commands.

I wanted curl for interactive browser tasks. So I built it.

Install Once, Automate Everything

npm i -g @dyyz1993/xbrowser

That's the entire setup. No WebDriver. No config files. xbrowser ships with its own managed Chromium that includes CDP fingerprint protection â€” sites can't easily detect automation.

From there, you have 35+ composable commands at your fingertips.

Feature 1: Search the Web From Your Terminal

No API keys. No OAuth. No rate limits. Just search.

# Google
xbrowser search "best headless browser 2026" --engine google --num 10

# Bing
xbrowser search "best headless browser 2026" --engine bing --num 10

# Baidu (for Chinese-language results)
xbrowser search "æ— å¤´æµè§ˆå™¨è‡ªåŠ¨åŒ–" --engine baidu --num 10

Each command returns structured JSON: titles, URLs, snippets. Pipe to jq, save to file, or feed directly into your AI agent's context.

Real use case: I track how my open-source project ranks across search engines. Every Monday I run:

xbrowser search "xbrowser browser automation" --engine google --num 30 \
  | jq '.results[] | select(.url | contains("xbrowser.dev")) | .position'

Takes 5 seconds. No script. No API key. Just results.

Feature 2: Scrape Without Writing a Scraper

The scrape command handles JavaScript rendering, lazy-loaded content, and complex layouts â€” and outputs clean Markdown by default. It's the web scraping tool I always wished Playwright had built in:

# Any page â†’ clean Markdown
xbrowser scrape https://example.com/blog/my-article

# Crawl an entire site (respects robots.txt)
xbrowser crawl https://example.com --depth 3 --max-pages 100

# Generate a complete URL map
xbrowser map https://example.com

I use scrape daily for:

Content research: Scrape competitor articles â†’ feed to LLM for analysis
SEO auditing: Map all URLs on a site, check for orphaned pages
Documentation: Scrape API docs and convert to Markdown for offline reading
Web crawler workflows: Chain scrape with crawl for bulk data extraction

The crawl command follows internal links, respects robots.txt, deduplicates URLs, and handles SPA hash routes. It's an ethical, complete web crawler in one command.

Feature 3: Chain Commands â€” The Real Magic

This is the feature that makes people say "wait, you can do that?"

Instead of writing scripts, you chain operations with 6 operators (&&, ,, +, ->, ;, ||):

# Go to a page, click the top link, extract text
xbrowser "goto https://news.ycombinator.com , click '.titleline > a:first-of-type' , text"

# Complete workflow: navigate â†’ fill form â†’ submit â†’ extract
xbrowser "goto https://app.example.com/login \
  + fill '#email' 'user@example.com' \
  + fill '#password' 'secret' \
  + click '#login' \
  -> wait '#dashboard' \
  , screenshot --output dashboard.png"

The syntax reads like natural language. && means "then and only then" (stop on error). , means "do all of these." -> means "pipe to next." || means "fallback if failed."

For AI agent developers, this is transformative. Instead of generating 50-line scripts, your agent constructs a single chain string:

User: "Go to Hacker News, click the top story, and summarize it"

Agent builds:
xbrowser "goto https://news.ycombinator.com , click '.titleline > a:first-of-type' , text"

No async. No error handling boilerplate. No debugging. Just intent â†’ command â†’ result.

Feature 4: Record Once, Replay Forever

Some workflows are too complex for a one-liner. That's where recording comes in:

# Start recording (opens visible browser)
xbrowser record start --url https://example.com

# Do your thing â€” click around, fill forms, navigate
# xbrowser captures every action

# Stop and save
xbrowser record stop --output my-workflow.yaml

# Replay headlessly anytime
xbrowser replay my-workflow.yaml --headless

# Export to Python, JavaScript, or Bash
xbrowser convert my-workflow.yaml --lang python --output workflow.py

I use this for:

Daily standup reports: Record the Jira â†’ Confluence navigation once, replay every morning
Price monitoring: Record a competitor's pricing page, replay daily, diff the output
Onboarding: Record a complex internal tool setup, give new hires the replay script

The convert command is particularly powerful â€” it auto-generates working Puppeteer/Playwright/Selenium scripts from your recorded actions. Record in the browser, ship as code.

Feature 5: 68 Plugins for Every Platform

xbrowser ships with 68 built-in plugins that encapsulate site-specific knowledge:

Category	Plugins
Search Engines	Google, Bing, Baidu
AI Assistants	DeepSeek, ChatGPT, Claude, Doubao, QianWen, YuanBao
Social Media	Twitter/X, Reddit, Quora, Weibo, Zhihu, XiaoHongShu, Douyin
Developer	GitHub, Dev.to, Medium, Hashnode, CSDN, Juejin
Image Platforms	Unsplash, Pexels, Pinterest, Getty, Shutterstock, and 15 more
E-Commerce	Taobao, JD, 1688
SEO	Backlink analysis, site audit, keyword tracking

Each plugin provides high-level commands tailored to that platform:

# GitHub: Get any user's profile
xbrowser github get-profile --username torvalds

# Unsplash: Search and download images
xbrowser unsplash search "mountain sunset" --download first

# Doubao: Generate AI images
xbrowser doubao image --prompt "cyberpunk city at sunset" --cdp 9221

# SEO: Audit any page
xbrowser seo audit https://your-website.com

No selectors to write. No DOM to inspect. The plugin handles the complexity.

When to Use xbrowser vs. Playwright vs. Selenium

I'll be direct:

	xbrowser	Playwright	Selenium
Best for	Web tasks & automation	App testing	Legacy testing
Setup	`npm i -g` (1 step)	npm + browser download	npm + WebDriver
Learning curve	CLI commands	JavaScript API	Language bindings
Search/Scrape	Built-in helpers	Write it yourself	Write it yourself
Plugins	68 built-in	None	None
Anti-detection	Built-in CDP protection	Third-party plugins	External tools
AI agent friendly	âœ… CLI commands	âŒ Scripts	âŒ Scripts
Test framework	Not a test tool	â Best in class	Good

Use Playwright for testing your web app. Use xbrowser for everything else.

Real-World Workflows I Run Daily

Here's what my actual automation looks like:

Morning competitive check:

xbrowser search "xbrowser alternatives" --engine google --num 20 --json > /tmp/competitors.json
xbrowser search "browser automation CLI" --engine bing --num 20 --json >> /tmp/competitors.json

Weekly SEO audit:

xbrowser map https://xbrowser.dev > /tmp/sitemap.txt
xbrowser seo audit https://xbrowser.dev --json > /tmp/seo-report.json

Content research for blog posts:

xbrowser scrape https://competitor-blog.com/latest-post | llm summarize

AI search intelligence (query 14 AI engines at once):

xbrowser ai-search-engines "how to do browser automation in 2026"

Each of these would be a 30-80 line Puppeteer script. With xbrowser, they're single commands I can alias, schedule with cron, or embed in AI agent workflows.

Get Started in 30 Seconds

npm i -g @dyyz1993/xbrowser
xbrowser search "hello world" --engine google

That's it. Two commands and you're automating the web.

Docs & examples: xbrowser.dev
Source code: github.com/dyyz1993/xbrowser (MIT license)
68 plugins: Search, scrape, crawl, record, replay, and automate 68+ platforms

If you're tired of writing 50-line scripts for tasks that should take one command â€” or if you're building AI agents that need to browse the web â€” give it a try.