Zil Norvilis

Posted on Feb 15

The Captcha Wall: Solving Challenges Programmatically with Ruby

#ruby #scraping #automation #webdev

The Ultimate Boss Battle

You’ve optimized your headers. You’re rotating residential proxies. Your TLS fingerprinting is perfectly masked. But then, it happens. You navigate to the target page, and instead of data, you see the familiar, taunting grid of traffic lights and crosswalks.

The CAPTCHA.

For a long time, the advice for scrapers was: "If you hit a CAPTCHA, give up." But in a professional context, that isn't always an option. While you cannot "solve" a modern reCAPTCHA or hCaptcha using pure Ruby logic (those days are long gone), you can orchestrate the solution.

Here is how to programmatically bypass the wall using Ruby, Ferrum, and a solving service.

1. The Strategy: Token Injection

Modern CAPTCHAs (like reCAPTCHA v2/v3 or hCaptcha) don't actually care if you clicked the images. They care about a g-recaptcha-response token.

When a human solves a CAPTCHA, a hidden text area in the HTML is populated with a long string of gibberish (the token). When the form is submitted, the server validates that token.

Our goal is to:

Extract the Site Key (public ID) from the target page.
Send that key to a Solving Service (like 2Captcha, Anti-Captcha, or CapMonster).
Wait for a human or AI on their end to solve it.
Receive the token and inject it into the hidden field of our browser.
Submit the form.

2. Step 1: Finding the Site Key

Every CAPTCHA is tied to a sitekey. You can find this in the HTML of the target page.

# Using Ferrum to find the sitekey
sitekey = browser.at_css("div.g-recaptcha")["data-sitekey"]
# OR for hCaptcha
sitekey = browser.at_css("div.h-captcha")["data-sitekey"]

3. Step 2: Orchestrating the Solution

We’ll use a solving service API. Most follow a two-step process: Post the task -> Poll for result.

require 'http'

API_KEY = "your_2captcha_api_key"
page_url = "https://target-site.com/login"

# 1. Request the solution
response = HTTP.post("http://2captcha.com/in.php", params: {
  key: API_KEY,
  method: "userrecaptcha",
  googlekey: sitekey,
  pageurl: page_url,
  json: 1
})

request_id = JSON.parse(response.body)["request"]

# 2. Poll for the result (CAPCHAs take 20-40 seconds to solve)
token = nil
loop do
  sleep 5
  check = HTTP.get("http://2captcha.com/res.php", params: {
    key: API_KEY,
    action: "get",
    id: request_id,
    json: 1
  })

  result = JSON.parse(check.body)
  if result["status"] == 1
    token = result["request"]
    break
  elsif result["request"] != "CAPCHA_NOT_READY"
    raise "Error solving CAPTCHA: #{result["request"]}"
  end
end

4. Step 3: The Magic Injection

Now that we have the token, we need to put it into the page. Browsers hide these text areas with display: none, so we use JavaScript via Ferrum to fill it.

# Inject the token into the hidden textarea
browser.execute("document.getElementById('g-recaptcha-response').innerHTML = '#{token}';")

# Sometimes you also need to trigger the callback function if the site uses one
browser.execute("onSuccess('#{token}');") # If a callback is defined

# Finally, submit the form
browser.at_css("button#login-btn").click

5. Stealth Considerations

Even if you solve the CAPTCHA, the site might still block you if it detects your browser is "automated" immediately after.

Wait for Idle: After injecting the token, wait a random amount of time (1–3 seconds) before clicking submit.
Move the Mouse: Use Ferrum to move the mouse cursor to the button before clicking. Some advanced protections track the mouse path to see if it "teleported" to the button.
Check for v3: reCAPTCHA v3 doesn't show a wall; it gives you a score (0.0 to 1.0). If you get a low score, you’re blocked silently. Solving v3 requires sending the action parameter to your solving service to mimic a high-score user.

6. The Ethical Boundary

Bypassing CAPTCHAs is the "heavy artillery" of scraping. It costs money (per-captcha fees) and increases the load on the target server.

Look for an API first: Many sites that use CAPTCHAs have an official, paid API. Usually, that is cheaper and more stable than paying for CAPTCHA solves and proxy rotations.
Don't Abuse: Use this only for data that is publicly available but protected by aggressive anti-bot layers.

Summary

The "Captcha Wall" is no longer impenetrable. By combining a headless browser like Ferrum with an external API, your Ruby scrapers can navigate through even the most protected enterprise sites.

You aren't just a writer of code; you are an architect of automation.

Have you successfully bypassed a difficult CAPTCHA recently? Which service did you find most reliable? Let's talk in the comments!

DEV Community