DEV Community

Arson
Arson

Posted on

Stop Wasting API Calls: Detect What's Blocking Your Agent Before You Solve

#ai #python #webdev #automation

Most AI agents handle web blocks the same way: hit a wall, throw a CAPTCHA solver at it, hope for the best.

The problem? Half the time it's not a CAPTCHA.

I've been building GateSolve, a CAPTCHA solving API for AI agents. And the most common failure pattern I see is agents submitting solve requests for pages that aren't actually blocked by a CAPTCHA.

They're blocked by:

  • JS challenges (Cloudflare interstitials that auto-resolve in 5-10 seconds)
  • Auth walls (login pages that need credentials, not CAPTCHA tokens)
  • Silent blocks (403 responses with no CAPTCHA at all)
  • Rate limits (429 responses that just need a backoff)

Solving a CAPTCHA on an auth-walled page does nothing. You burn credits and get nowhere.

The Fix: Classify Before You Solve

I built a free endpoint that classifies any URL's access block:

curl "https://gatesolve.dev/api/detect?url=https://example.com"
Enter fullscreen mode Exit fullscreen mode

Response:

{
  "url": "https://example.com",
  "classification": "public-ok",
  "httpStatus": 200,
  "solvable": false,
  "confidence": 0.9,
  "details": "Page accessible, no blocks detected",
  "recommendation": "No action needed. URL is publicly accessible."
}
Enter fullscreen mode Exit fullscreen mode

The classifications:

Classification What it means Solvable?
public-ok Page is accessible No action needed
captcha CAPTCHA widget detected Yes - use GateSolve
js-challenge Cloudflare interstitial Wait or use stealth browser
auth-wall Login/credentials required No - authenticate instead
blocked-silent 403 with no pattern No - change IP/UA
rate-limited 429 Too Many Requests No - back off

Real Example: CAPTCHA Detected

When a real CAPTCHA is found:

{
  "url": "https://some-protected-site.com",
  "classification": "captcha",
  "captchaType": "cloudflare-turnstile",
  "httpStatus": 403,
  "solvable": true,
  "confidence": 0.9,
  "recommendation": "Submit to GateSolve /api/solve with type=\"cloudflare-turnstile\". Extract siteKey from the page source."
}
Enter fullscreen mode Exit fullscreen mode

Now you know what you're dealing with before spending a solve credit.

For Browser Automation: Client-Side Detection

If you're using Puppeteer or Playwright, our plugins now include detectBlock() — a DOM-level check that classifies the page after goto():

import { detectBlock, solveOnPage } from '@gatesolve/puppeteer-plugin';

const page = await browser.newPage();
await page.goto('https://target-site.com');

const block = await detectBlock(page);

if (block.classification === 'captcha' && block.solvable) {
  // Actually a CAPTCHA - solve it
  await solveOnPage(page, { apiKey: 'gs_...' });
} else if (block.classification === 'js-challenge') {
  // Just wait for Cloudflare to resolve
  await page.waitForTimeout(10000);
} else if (block.classification === 'auth-wall') {
  // Need credentials, not a CAPTCHA solve
  console.log('Login required');
}
Enter fullscreen mode Exit fullscreen mode

This prevents the most common automation failure: your agent assumes the page loaded, tries to find elements on a Cloudflare interstitial, fails, retries, and spirals into a loop.

Install 

# Puppeteer
npm install @gatesolve/puppeteer-plugin

# Playwright
npm install @gatesolve/playwright-plugin

# Python SDK
pip install gatesolve
Enter fullscreen mode Exit fullscreen mode

The /api/detect endpoint requires no API key and no signup. Just hit it with a URL.

Links:

Building agent infrastructure is weird. The biggest wins come from helping agents NOT do things — like not wasting a CAPTCHA solve on a login page.

Top comments (0)