Our AI agent pipeline kept stalling on CAPTCHA-heavy scraping until we rebuilt it around queues and short browser sessions

I learned this one the hard way.

We had a scraping workflow that looked solid in staging:

• Playwright handled JavaScript-heavy pages
• a few retries cleaned up transient failures
• extracted data flowed into downstream AI steps

Then real traffic hit.

A target site tightened anti-bot rules. CAPTCHA pages started showing up more often. A few browser workers got stuck for minutes at a time. The queue behind them kept growing. And suddenly the scraping issue wasn’t just a scraping issue anymore.

It broke the rest of the pipeline:

• n8n stopped getting fresh records
• a classifier waited on inputs that never arrived
• enrichment jobs reprocessed stale items
• Make workflows fired on partial data
• downstream LLM calls kept happening anyway

That last part was the killer. Browser failures are visible. Wasted AI calls from unstable orchestration are sneakier.

The fix was not “get better at bypassing CAPTCHAs.”

The fix was rebuilding the scraper around:

• a request queue
• short-lived browser sessions
• aggressive retries
• solver fallback as a last resort

Once we did that, the whole agent pipeline got boring again. Which is exactly what you want.

The mistake: we started with the browser

A lot of scraping systems get built browser-first.

Open Playwright. Hit a page. If it works, keep going. If it fails, retry. If a CAPTCHA appears, bolt on 2Captcha or Browserless and hope.

That works for demos.

It’s the wrong mental model for production.

The production model is queue-first.

If you’re feeding scraped data into n8n, Make, Zapier, OpenClaw, or custom workers, scraping is usually the first domino. If it stalls, everything after it gets weird:

• extraction prompts run on incomplete batches
• dedupe sees inconsistent state
• lead scoring uses stale records
• support triage fires with missing context
• retries multiply LLM usage

A queue-first design gives you control over failure.

Instead of one long-lived worker trying to survive everything, each URL becomes a work item with state:

• pending
• in progress
• blocked
• retrying
• escalated
• failed
• complete

That sounds boring. It is also the difference between a crawler that survives traffic spikes and one that turns into a browser graveyard.

What changed for us

We moved to a pipeline that looked more like this:

1. Every URL goes into a queue with a unique key
2. Cheap fetch path tries first
3. Browser lane only handles pages that actually need JavaScript
4. Blocked pages get requeued with retry metadata
5. Sessions retire fast
6. CAPTCHA solver is a fallback lane, not the main path
7. Clean output gets handed to downstream AI steps

That architecture solved two problems at once:

• scraping became much more reliable
• downstream AI agents stopped burning cycles on unstable inputs

Queue-first beats browser-first

If you’re using Crawlee or Apify, this pattern is already baked into the tooling.

A practical rule:

• queue URLs first
• do not let browser workers own crawl state

Why this matters:

• jobs can crash without losing the crawl
• duplicate URLs can be dropped early
• blocked requests can be retried cleanly
• hard pages can be escalated without freezing the rest of the system

A simplified queue worker might look like this:

async function processJob(job) {
  try {
    const result = await fetchCheap(job.url);
    if (result.ok) return complete(job, result.data);

    if (result.requiresBrowser) {
      return enqueueBrowserLane(job);
    }

    return retry(job, result.reason);
  } catch (err) {
    return retry(job, err.message);
  }
}

Then the browser lane becomes just another worker tier, not the center of the universe.

Use the cheapest path first

One of the biggest mistakes in scraping stacks is using a full browser for everything.

Don’t.

If a page can be fetched with plain HTTP, use plain HTTP.

If you only need a browser for a small subset of pages, isolate that subset.

A simple decision tree works well:

• try HTTP first
• if the page needs JS rendering, use Playwright
• if blocked, retry with a fresh session/proxy
• if still blocked after N attempts, send to solver

That one change cuts both latency and cost.

For less hostile targets, a lightweight crawler can go surprisingly far:

import { CheerioCrawler } from 'crawlee';

const crawler = new CheerioCrawler({
  maxConcurrency: 100,
  maxRequestsPerMinute: 250,
  async requestHandler({ $, request }) {
    const title = $('title').text();
    console.log(request.url, title);
  },
});

await crawler.run(['https://example.com']);

If you don’t need a browser, don’t pay the browser tax.

Short browser sessions changed everything

The most useful thing we did was stop trying to preserve sessions forever.

That feels wrong at first.

A lot of teams treat a browser session like something precious:

• keep cookies warm
• keep the page open
• keep the worker alive
• try to recover from every failure in place

That’s exactly what made our pipeline fragile.

Instead, we moved to short, bounded sessions.

With Playwright, isolated contexts make this easy:

import { chromium } from 'playwright';

const browser = await chromium.launch({ headless: true });

async function runShortSession(url) {
  const context = await browser.newContext();
  const page = await context.newPage();

  try {
    await page.goto(url, { waitUntil: 'domcontentloaded', timeout: 30000 });
    const html = await page.content();
    return { ok: true, html };
  } catch (err) {
    return { ok: false, error: err.message };
  } finally {
    await context.close();
  }
}

Each task gets a fresh context.

If it gets blocked, times out, or lands on a CAPTCHA, the session is disposable. Close it. Requeue the job. Move on.

That one design decision did more for reliability than any stealth plugin we tried.

Retire sessions aggressively

If you’re using Crawlee, lean into SessionPool.

The point is not to save every session. The point is to throw away bad ones quickly.

import { BasicCrawler, ProxyConfiguration } from 'crawlee';

const proxyConfiguration = new ProxyConfiguration({});

const crawler = new BasicCrawler({
  useSessionPool: true,
  sessionPoolOptions: {
    maxPoolSize: 100,
  },
  maxRequestRetries: 10,
  async requestHandler({ request, session }) {
    const proxyUrl = await proxyConfiguration.newUrl(session.id);
    console.log(`Processing ${request.url} with ${proxyUrl}`);

    // If blocked, retire the session and let retry logic handle the request.
    // session.retire();
  },
  async failedRequestHandler({ request }) {
    console.log(`Failed after retries: ${request.url}`);
  },
});

The production mindset is:

• requests are disposable
• sessions are temporary
• blocked identities should be replaced, not rescued

That’s a much healthier model for anti-bot environments.

CAPTCHAs are usually a routing problem

This was the biggest mindset shift for me.

A CAPTCHA looks like the problem because it’s visible.

Usually it’s just the point where your upstream choices finally get billed.

If you’re seeing lots of CAPTCHA pages, the real issue is often one or more of these:

• weak proxy strategy
• unrealistic headers/fingerprints
• overusing browsers
• poor retry semantics
• sessions living too long
• no clean fallback lane

A weak scraper treats a block as fatal.

A stronger scraper treats it as routing information.

That means:

• mark the attempt as blocked
• retire the session
• rotate identity
• requeue the request
• escalate only after repeated failure

The fallback lane should stay small

The wrong design is sending every challenge straight to 2Captcha.

That turns your fallback into your main lane.

The better design is explicit escalation:

HTTP fetch
  -> browser render
    -> retry with fresh session
      -> retry with rotated proxy
        -> solver fallback

A tiny pseudocode version:

async function handleUrl(job) {
  let result = await tryHttp(job.url);
  if (result.ok) return result;

  result = await tryBrowser(job.url, job.sessionId);
  if (result.ok) return result;

  for (let i = 0; i < 3; i++) {
    result = await tryBrowser(job.url, newSessionId(), rotatedProxy());
    if (result.ok) return result;
  }

  return enqueueSolver(job);
}

That keeps solver usage rare, which is exactly where you want it.

A concrete Playwright + solver example

If you do need a solver, keep it isolated.

Here’s the shape of a Playwright + 2Captcha flow in Python:

from playwright.sync_api import sync_playwright
from twocaptcha import TwoCaptcha

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    solver = TwoCaptcha("YOUR_API_KEY")

    page.goto("https://patrickhlauke.github.io/recaptcha/")

    captcha_frame = page.wait_for_selector("iframe[src*='recaptcha']")
    src = captcha_frame.get_attribute("src")
    site_key = src.split("k=")[-1].split("&")[0]

    token = solver.recaptcha(
        sitekey=site_key,
        url=page.url,
    )

    page.evaluate(
        """(token) => {
            document.getElementById('g-recaptcha-response').value = token;
        }""",
        token["code"]
    )

    browser.close()

Useful? Sure.

But this should live in a fallback worker, not your default path.

Browser pricing quietly forces good architecture

One reason this pattern matters: hosted browser pricing punishes sloppy session design.

If you’re using Browserless or similar services, cost usually maps to things like:

• session duration
• reconnects
• concurrent browsers
• CAPTCHA solve operations
• proxy bandwidth

That pricing model is basically telling you what good architecture looks like:

• keep browser jobs short
• avoid giant sticky sessions
• use browser automation only when needed
• isolate expensive fallback steps

Service	What the pricing model pushes you toward
Browserless	Short sessions, strict queueing, limited reconnects, browser use only where needed
ScrapingBee	Budgeting by request type, careful use of JS rendering and premium proxies
2Captcha	Rare fallback usage, not blanket solving

If traffic spikes and your browser lane does everything, your costs get ugly fast.

The hidden cost was downstream AI retries

This is the part I think a lot of teams miss.

The browser bill is visible.

The bigger problem is what unstable scraping does to the rest of the automation stack.

A blocked page can cause:

• duplicate enrichment
• repeated extraction prompts
• reclassification of stale records
• support triage on incomplete context
• agent loops firing on unsettled state

If you’re paying per token, that gets painful quickly.

This is exactly why predictable AI pricing changes the way these systems can be built.

Once the scrape layer is queue-first and failure-tolerant, you can let the AI layer keep running without worrying that every retry branch is quietly inflating your bill.

That’s the practical appeal of Standard Compute for this kind of workflow.

It’s a drop-in OpenAI API replacement, so existing SDKs and HTTP clients still work, but the cost model is flat monthly instead of per-token. For agent pipelines that scrape, classify, enrich, dedupe, and route 24/7, that matters more than people expect.

When your workflow retries a lot, branches a lot, or runs nonstop, predictable cost is not a nice-to-have. It changes what you can safely automate.

A minimal queue worker setup

If you’re building this yourself, even a simple queue architecture goes a long way.

Example shape:

workers/
  fetch-worker.js
  browser-worker.js
  solver-worker.js
  ai-worker.js
queue/
  enqueue.js
  retry.js
  dead-letter.js

And a very simple worker contract:

{
  "id": "job_123",
  "url": "https://target.example/page",
  "attempt": 2,
  "lane": "browser",
  "lastError": "captcha_detected",
  "nextAction": "rotate_session"
}

That gives you enough structure to do sane things like:

• retry only the failed stage
• send hard pages to a solver queue
• dead-letter impossible cases
• keep downstream AI workers processing clean successes

Practical rules that actually helped

Here are the rules I wish someone had drilled into me earlier.

1. Treat every request as disposable

If one request gets blocked, the crawl should continue.

2. Treat every session as temporary

Do not build around heroic long-lived browser sessions.

3. Keep browser tasks small

Open page, do the work, extract what you need, close context.

4. Use retries more aggressively than feels natural

A blocked request is not necessarily a failed request.

5. Keep solver usage behind a queue boundary

It should be a fallback lane, not a detour that hijacks the worker.

6. Don’t spend browser budget where HTTP will do

A lot of pages don’t need Playwright.

7. Protect the AI layer from scraper instability

Don’t let partial or stale scrape state trigger expensive downstream nonsense.

The architecture I’d start with now

If I were rebuilding this from scratch today, I’d do something like:

• Crawlee or Apify Request Queue for crawl state
• CheerioCrawler for cheap pages
• PlaywrightCrawler only where JS is required
• SessionPool with fast retirement on blocked sessions
• proxy rotation tied to session identity
• solver fallback in a separate queue
• downstream AI workers consuming only clean, committed outputs
• Standard Compute for the LLM layer so retries and nonstop automations don’t create token anxiety

That last part matters more once the rest of the system gets reliable.

When scraping is flaky, you’re mostly fighting fires.

When scraping becomes stable, you finally see the real workload: extraction, classification, enrichment, triage, and agent decisions running continuously. That’s when per-token billing starts getting annoying, especially in n8n, Make, Zapier, OpenClaw, or custom agent stacks that branch and retry all day.

Final takeaway

The breakthrough for us was not “better CAPTCHA bypass.”

It was this:

• queue first
• keep sessions short
• retire blocked identities fast
• use solvers rarely
• shield downstream AI workflows from scraper chaos

Once we did that, the scraping layer stopped acting like a single point of failure.

And once the AI layer sat on predictable flat-rate compute instead of per-token billing, the whole pipeline became much easier to run 24/7 without babysitting both reliability and cost.

That combination is what actually made the automation usable.

Not stealth plugins. Not hero sessions. Not wishful thinking.

Just queues, boundaries, and accepting that failure is part of the design.