DEV Community

luisgustvo
luisgustvo

Posted on

Bypassing Captcha in CrewAI Workflows with CapSolver

#webdev #scraping #ai #captcha

TL;DR: Autonomous AI agents built with CrewAI often hit a wall when encountering CAPTCHAs. Integrating CapSolver provides a robust, automated solution to solve these challenges, ensuring your web-based workflows run without interruption.

Introduction

In the world of AI-driven automation, frameworks like CrewAI are revolutionizing how we build multi-agent systems for complex tasks such as web scraping, data aggregation, and automated browsing. However, a common and persistent hurdle remains: the CAPTCHA. These challenges are designed to block automated scripts, bringing even the most sophisticated CrewAI workflow to a grinding halt.

CapSolver offers a powerful and reliable way to overcome this obstacle. By seamlessly integrating CapSolver's AI-powered solving capabilities, your CrewAI agents can handle various CAPTCHA types automatically, maintaining the flow of your automation and ensuring compliance with website protections.

Understanding CrewAI

CrewAI is a cutting-edge, fast-paced Python framework for orchestrating collaborative AI agent systems. It is built from the ground up—independent of other agent frameworks—to provide both high-level usability and deep customization for developers.

Core Features of CrewAI

Feature Description
Multi-Agent Collaboration Enables teams of AI agents to work together autonomously, featuring natural decision-making and dynamic task delegation.
Event-Driven Workflows (Flows) Offers precise execution control, state consistency, and conditional branching for complex business logic.
Standalone Design Zero external framework dependencies, optimized for speed and minimal resource consumption.
Production-Ready Engineered to meet enterprise standards for reliability, scalability, and performance.

Introducing CapSolver

CapSolver is a premier CAPTCHA solving service that utilizes advanced AI to bypass a wide array of CAPTCHA challenges. With support for numerous types and lightning-fast response times, CapSolver is the essential tool for maintaining uninterrupted automation.

CapSolver's Extensive CAPTCHA Support

CapSolver is capable of solving virtually any CAPTCHA encountered in the wild, including:

  • reCAPTCHA v2 (Image & Invisible)
  • reCAPTCHA v3 (Score-based verification)
  • Cloudflare Turnstile
  • Cloudflare Challenge (5s)
  • AWS WAF
  • GeeTest and many other advanced protection mechanisms.

The Necessity of Integration

When your CrewAI agents are tasked with interacting with the web, CAPTCHA challenges are inevitable. Integrating CapSolver transforms these roadblocks into minor, automated steps:

  1. Uninterrupted Agent Workflows: Agents can execute their tasks from start to finish without manual intervention.
  2. Scalable Automation: Easily handle concurrent CAPTCHA challenges across multiple agents and large-scale operations.
  3. Cost-Efficiency: You only pay for successfully solved CAPTCHAs, ensuring a high return on investment.
  4. High Accuracy: Benefit from industry-leading success rates across all supported CAPTCHA types.

Setup and Installation

To begin, install the necessary Python packages in your environment:

pip install crewai 'crewai[tools]' requests
Enter fullscreen mode Exit fullscreen mode

Creating a Custom CapSolver Tool for CrewAI

CrewAI's flexibility allows for the creation of custom tools that agents can invoke when needed. We will wrap the CapSolver API into a BaseTool that any agent can use to solve a CAPTCHA.

Basic CapSolver Tool Implementation 

import requests
import time
from crewai.tools import BaseTool
from typing import Type
from pydantic import BaseModel, Field

# IMPORTANT: Manage your API Key securely, e.g., via environment variables
CAPSOLVER_API_KEY = "YOUR_CAPSOLVER_API_KEY"

class CaptchaSolverInput(BaseModel):
    """Input schema for the CaptchaSolver tool."""
    website_url: str = Field(..., description="The URL of the website with the CAPTCHA")
    website_key: str = Field(..., description="The site key of the CAPTCHA")
    captcha_type: str = Field(default="ReCaptchaV2TaskProxyLess", description="Type of CAPTCHA to solve")

class CaptchaSolverTool(BaseTool):
    name: str = "captcha_solver"
    description: str = "Solves CAPTCHA challenges using the CapSolver API. Supports reCAPTCHA v2/v3, Turnstile, and more."
    args_schema: Type[BaseModel] = CaptchaSolverInput

    def _run(self, website_url: str, website_key: str, captcha_type: str = "ReCaptchaV2TaskProxyLess") -> str:
        create_task_url = "https://api.capsolver.com/createTask"

        task_payload = {
            "clientKey": CAPSOLVER_API_KEY,
            "task": {
                "type": captcha_type,
                "websiteURL": website_url,
                "websiteKey": website_key
            }
        }

        response = requests.post(create_task_url, json=task_payload)
        result = response.json()

        if result.get("errorId") != 0:
            return f"Error creating task: {result.get('errorDescription')}"

        task_id = result.get("taskId")

        # Poll for the result
        get_result_url = "https://api.capsolver.com/getTaskResult"
        for _ in range(60):  # Max 60 attempts (120 seconds)
            time.sleep(2)

            result_payload = {
                "clientKey": CAPSOLVER_API_KEY,
                "taskId": task_id
            }

            res = requests.post(get_result_url, json=result_payload).json()

            if res.get("status") == "ready":
                solution = res.get("solution", {})
                # Returns the appropriate token (gRecaptchaResponse or token)
                return solution.get("gRecaptchaResponse") or solution.get("token")
            elif res.get("status") == "failed":
                return f"Task failed: {res.get('errorDescription')}"

        return "Timeout waiting for CAPTCHA solution"
Enter fullscreen mode Exit fullscreen mode

Handling Specific CAPTCHA Submission Methods

Once a token is retrieved from CapSolver, the agent must know how to submit it to the target website.

1. reCAPTCHA v2/v3 Token Injection

For reCAPTCHA, the token must be injected into the hidden g-recaptcha-response textarea before form submission.

from selenium import webdriver
from selenium.webdriver.common.by import By

def submit_recaptcha_token(driver, token: str):
    """Inject reCAPTCHA token and submit the form."""
    # Find the hidden textarea
    recaptcha_response = driver.find_element(By.ID, "g-recaptcha-response")

    # Make it visible (optional, for debugging) and set the token
    driver.execute_script("arguments[0].style.display = 'block';", recaptcha_response)
    recaptcha_response.clear()
    recaptcha_response.send_keys(token)

    # Submit the form
    form = driver.find_element(By.TAG_NAME, "form")
    form.submit()
Enter fullscreen mode Exit fullscreen mode

2. Cloudflare Challenge (5s) via Cookies

The Cloudflare 5-second challenge does not return a token but a set of cookies and a User-Agent. These must be used in subsequent HTTP requests to access the protected page.

import requests

def access_cloudflare_protected_page(url: str, cf_solution: dict):
    """
    Uses the Cloudflare Challenge solution (cookies and user_agent) 
    to access the protected page via a requests session.
    """
    session = requests.Session()

    # Set the cookies from the CapSolver solution
    for cookie in cf_solution["cookies"]:
        session.cookies.set(cookie["name"], cookie["value"])

    # Set the User-Agent that was used to solve the challenge
    headers = {
        "User-Agent": cf_solution["user_agent"]
    }

    # Access the protected page
    response = session.get(url, headers=headers)
    return response.text
Enter fullscreen mode Exit fullscreen mode

Advanced Best Practices

To ensure your CrewAI-CapSolver integration is robust and cost-effective, consider these best practices:

  • Error Handling with Exponential Backoff: Implement a retry mechanism with exponential backoff for API calls that fail due to transient errors. This prevents overloading the API and increases the chance of success.
  • Balance Management: Regularly check your CapSolver balance using the getBalance API to prevent task failures due to insufficient funds.
  • Token Caching: For tasks that repeatedly access the same page within a short timeframe, implement a simple cache for the solved CAPTCHA tokens (e.g., valid for 1-2 minutes) to save costs and reduce latency.

Conclusion

Integrating CapSolver with CrewAI is the key to unlocking the full potential of autonomous AI agents for web-based tasks. By combining CrewAI's powerful multi-agent orchestration with CapSolver's industry-leading CAPTCHA solving capabilities, developers can build robust, scalable, and truly autonomous automation solutions that navigate the modern web with ease.

Ready to get started? Sign up for CapSolver today and use bonus code CREWAI for an extra 6% bonus on every recharge!

Frequently Asked Questions (FAQ)

Q: Is CrewAI free to use?
A: Yes, CrewAI is an open-source framework released under the MIT license. While the framework itself is free, you will incur costs for the underlying LLM APIs (like OpenAI) and CAPTCHA solving services like CapSolver.

Q: How do I find the CAPTCHA site key?
A: The site key is typically located in the page's HTML source. Look for the data-sitekey attribute, often found within the div element that contains the CAPTCHA widget.

Q: Can I use CapSolver with other Python frameworks?
A: Absolutely. CapSolver provides a standard REST API that can be integrated with any Python framework, including Scrapy, Selenium, Playwright, and others.

Q: How much does CapSolver cost?
A: CapSolver offers competitive, volume-based pricing. Visit capsolver.com for the most current pricing details. Don't forget to use code CREWAI for a 6% bonus on your first recharge.

Top comments (0)