DEV Community: Valentina Skakun

Building a Production-Ready Job Board Scraper with Python

Valentina Skakun — Mon, 02 Mar 2026 14:33:08 +0000

For a simpler approach using CSS selectors and BeautifulSoup, check out our beginner's guide to job scraping. This article covers advanced techniques for production-scale job data extraction.

Effective job scraping pipelines prioritize hidden API endpoints over brittle DOM parsing. Most tech companies rely on Applicant Tracking Systems like Greenhouse or Lever that expose structured JSON data. Directly consuming these endpoints reduces ban rates and ensures type-safe data extraction.

We reserve resource-intensive headless browsers for complex Single Page Applications like Workday or protected aggregators. The following reference implementation demonstrates this logic by automatically detecting hidden APIs for major hiring platforms.

The Universal AI Job Scraper
Job Data Pipeline Architecture
- The Platform Specific Strategy
- Identifying the Underlying ATS
Direct ATS Integration Patterns
- Greenhouse JSON Endpoints
- Lever Hidden API Endpoints
- Handling Ashby and BambooHR
Handling Complex Enterprise ATS
- Playwright Network Interception
- Handling Workday CSRF and Sessions
- Simulating Human Inputs for Bot Defense
Bypassing Aggregator Defenses
- Google SERP Discovery Strategy
- Unwinding Redirect Chains
- LinkedIn and Indeed DOM Strategies
Job Data Normalization
- Standardizing Salary Ranges via Regex
- Extracting Tech Stacks with FlashText
- Detecting Expired Listings
Scaling and Infrastructure
- Residential Proxy Rotation Logic
- TLS Fingerprint Management
Conclusion

The Universal AI Job Scraper

If you need a resilient solution that parses job postings from any domain (Indeed, Greenhouse, Company Careers) without writing custom selectors, use AI extraction. The following script uses HasData's scraping API to handle residential proxy rotation, JavaScript rendering, and Large Language Model (LLM) to normalize unstructured HTML into a strict JSON schema.

import requests
import json
from typing import Dict, Any

API_KEY = "YOUR_HASDATA_API_KEY"

def scrape_job_posting(url: str, api_key: str) -> Dict[str, Any]:
    """
    Scrapes any job posting URL using AI-powered extraction.
    Bypasses anti-bot protections and normalizes data to a strict schema.
    """
    payload = {
        "url": url,
        "proxyType": "residential", # Essential for bypassing Cloudflare on Indeed/Glassdoor
        "proxyCountry": "US",
        "jsRendering": True,        # Renders React/Angular based ATS pages (Workday)
        "aiExtractRules": {
            "jobs": {
                "type": "list",
                "output": {
                    "title": {"type": "string", "description": "Job title"},
                    "company": {"type": "string", "description": "Company name"},
                    "location": {"type": "string", "description": "Job location (city, state, remote)"},
                    "salary_range": {"type": "string", "description": "Salary range or hourly rate if mentioned"},
                    "job_type": {"type": "string", "description": "Full-time, Part-time, Contract, etc."},
                    "requirements": {"type": "string", "description": "List of technical requirements and skills"},
                    "posted_date": {"type": "string", "description": "Date posted in ISO format"},
                    "application_url": {"type": "string", "description": "Direct URL to apply"}
                }
            }
        }
    }

    try:
        response = requests.post(
            "https://api.hasdata.com/scrape/web",
            headers={
                "x-api-key": api_key,
                "Content-Type": "application/json"
            },
            json=payload,
            timeout=60
        )
        response.raise_for_status()

        # Return the AI extracted content directly
        return response.json().get('scrapingResult', {}).get('extract', {})

    except requests.exceptions.RequestException as e:
        print(f"Extraction failed: {e}")
        return {}

# Usage Example
data = scrape_job_posting("https://www.indeed.com/q-usa-jobs.html?vjk=a81504eef3366054", API_KEY)
print(json.dumps(data, indent=2))

Extraction example:

{
  "jobs": [
    {
      "title": "Voice recording (Spanish Accent, Latin America)",
      "company": "Mercor",
      "location": "New York, NY",
      "salary_range": "$25 - $35 an hour",
      "job_type": "Full-time",
      "requirements": "We're collecting ~3 hours of phone audio...",
      "application_url": "/rc/clk?jk=ccaa0338f396988e..."
    },
    {
      "title": "WAREHOUSE ATTENDANT (PART TIME)",
      "company": "Restaurant Associates",
      "location": "Chicago, IL 60607",
      "salary_range": "$32.43 an hour",
      "job_type": "Part-time",
      "requirements": "Previous food service experience is preferred.",
      "application_url": "/rc/clk?jk=32c48f4daa02f665..."
    }
  ]
}

The jsRendering parameter ensures that Single Page Applications are fully loaded before the AI attempts extraction, while the residential proxy setting handles the IP reputation checks common on aggregators.

Job Data Pipeline Architecture

Efficient scraping at scale requires a routing layer. Treating every URL as a generic HTML page wastes resources and increases ban rates. You must identify the underlying Applicant Tracking System (ATS) to select the optimal extraction strategy.

The Platform Specific Strategy

High-volume scraping requires cost efficiency. Hitting a public JSON endpoint costs a fraction of a cent in compute time while rendering a full browser session for Workday costs significantly more in RAM and proxy bandwidth.

Your pipeline should implement a waterfall logic flow:

Check for Known APIs. If the URL matches a known ATS like Greenhouse, route to a lightweight requests handler.

Check for Enterprise SPAs. If the target is Workday or Taleo, route to a Playwright instance with session management.

Fallback to Universal Extraction. For unknown domains or protected aggregators like Indeed, route to the AI-powered extraction API demonstrated in the previous section.

This decision tree ensures you only pay the "rendering tax" when absolutely necessary.

Identifying the Underlying ATS

Before initiating a scrape, detect the ATS provider to select the correct parser. Most career pages function as a white-label frontend for a major provider. Analyze the URL structure or window objects to identify the backend.

from typing import Optional

def detect_ats_provider(url: str) -> str:
    """
    Routes job URLs to specific scrapers based on domain signatures.
    """
    ats_patterns = {
        'greenhouse': ['greenhouse.io', 'boards.greenhouse.io'],
        'lever': ['lever.co', 'jobs.lever.co'],
        'workday': ['myworkdayjobs.com', 'workday.com'],
        'ashby': ['ashbyhq.com', 'jobs.ashbyhq.com'],
        'bamboohr': ['bamboohr.com'],
        'icims': ['icims.com'],
        'smartrecruiters': ['smartrecruiters.com'],
        'jobvite': ['jobvite.com'],
        'taleo': ['taleo.net'],
        'linkedin': ['linkedin.com/jobs'],
        'indeed': ['indeed.com']
    }

    url_lower = url.lower()
    for ats, patterns in ats_patterns.items():
        if any(pattern in url_lower for pattern in patterns):
            return ats

    return 'unknown'

Use a lightweight check to route traffic before spawning a heavy browser instance.

Direct ATS Integration Patterns

Once your routing layer identifies the provider, use these specific extraction patterns. Platforms like Greenhouse and Lever expose public JSON endpoints used to power their frontend UIs. For more complex platforms like Ashby, we use a hybrid approach.

Greenhouse JSON Endpoints

Greenhouse is the most developer-friendly ATS. Every job board has a corresponding API endpoint that returns structured data without authentication. You only need the board token, which is usually the company name found in the URL.

import requests
from typing import List, Dict

def fetch_greenhouse_jobs(board_token: str) -> List[Dict]:
    """
    Fetches raw job data directly from Greenhouse's public API.
    No proxies or rendering required.
    """
    url = f"https://boards-api.greenhouse.io/v1/boards/{board_token}/jobs?content=true"

    try:
        response = requests.get(url, timeout=10)
        response.raise_for_status()
        return response.json().get('jobs', [])
    except requests.RequestException as e:
        print(f"Greenhouse API Error: {e}")
        return []

# Usage
# jobs = fetch_greenhouse_jobs("hasdata")

This method returns clean JSON with fields like absolute_url, updated_at, and metadata. Unlike HTML parsing, this endpoint remains stable even if the company completely redesigns their career page CSS.

Lever Hidden API Endpoints

Lever follows a similar pattern but uses a different endpoint structure. While they do not document this publicly for scrapers, the API is open to allow for client-side rendering of job lists.

def fetch_lever_jobs(company_name: str) -> List[Dict]:
    """
    Targeting Lever's internal posting API.
    """
    url = f"https://api.lever.co/v0/postings/{company_name}?mode=json"

    try:
        response = requests.get(url, timeout=10)
        response.raise_for_status()
        return response.json()
    except requests.RequestException as e:
        print(f"Lever API Error: {e}")
        return []

The response includes rich data often hidden from the HTML view, including internal team categorizations, commitment levels (Full-time/Contract), and hiring manager details if exposed.

Handling Ashby and BambooHR

Newer platforms like Ashby and legacy systems like BambooHR are less consistent. Ashby heavily relies on React hydration, making direct API calls difficult due to complex query parameters. For these targets, we revert to the AI extraction method to handle the JavaScript rendering and DOM parsing.

def scrape_ashby_or_bamboo(url: str, api_key: str) -> Dict:
    """
    Fallback method for JS-heavy ATS platforms using HasData AI.
    """
    payload = {
        "url": url,
        "proxyType": "residential",
        "jsRendering": True, # Essential for Ashby's React hydration
        "aiExtractRules": {
            "jobs": {
                "type": "list",
                "output": {
                    "title": {"type": "string", "description": "Job title"},
                    "company": {"type": "string", "description": "Company name"},
                    "location": {"type": "string", "description": "Job location (city, state, remote)"},
                    "salary_range": {"type": "string", "description": "Salary range or hourly rate if mentioned"},
                    "job_type": {"type": "string", "description": "Full-time, Part-time, Contract, etc."},
                    "requirements": {"type": "string", "description": "List of technical requirements and skills"},
                    "posted_date": {"type": "string", "description": "Date posted in ISO format"}
                }
            }
        }
    }

    response = requests.post(
        "https://api.hasdata.com/scrape/web",
        headers={"x-api-key": api_key, "Content-Type": "application/json"},
        json=payload
    )

    return response.json().get("aiResponse", {}).get("jobs", [])

Using this hybrid approach allows you to scrape 80% of your targets (Greenhouse/Lever) with zero infrastructure cost, reserving your paid scraping credits for the 20% of difficult targets (Ashby/BambooHR).

Handling Complex Enterprise ATS

Enterprise platforms like Workday, Taleo, and iCIMS represent the hardest tier of job scraping. These Single Page Applications (SPAs) rely on stateful sessions, dynamic CSRF tokens, and heavy JavaScript execution. Sending a simple GET request will return a 200 OK status with a loading spinner but no data. To scrape these, you must move from HTTP clients to browser orchestration.

Playwright Network Interception

Parsing the DOM of a Workday site is inefficient because the HTML classes are often obfuscated and nested within multiple div layers. However, the data is loaded via internal JSON APIs after the page renders. Instead of scraping the visual elements, configure Playwright to intercept the background network traffic.

from playwright.sync_api import sync_playwright

def scrape_workday_via_interception(url: str):
    """
    Loads Workday in a headless browser but extracts data
    by intercepting the internal XHR response.
    """
    jobs_data = []

    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page()

        # Define the interception logic before navigation
        def handle_response(response):
            # Workday typically serves job data from endpoints containing 'wday/cxs'
            if "wday/cxs" in response.url and response.request.resource_type in ["fetch", "xhr"]:
                try:
                    data = response.json()
                    # Append valid job listings to our list
                    if "jobPostings" in data:
                        jobs_data.extend(data["jobPostings"])
                except Exception:
                    pass

        # Attach the event listener
        page.on("response", handle_response)

        # Trigger the page load which fires the XHR requests
        page.goto(url, wait_until="networkidle")

        browser.close()
        return jobs_data

This approach allows you to extract clean, structured JSON directly from the server response, bypassing the need to maintain complex CSS selectors for the HTML interface.

Handling Workday CSRF and Sessions

Workday is aggressive about session management. It generates a wday_vps_cookie and a corresponding CSRF token upon the first page load. If you attempt to use Python requests without these, the server rejects the connection.

If you need to scale scraping without keeping a browser open, use a hybrid approach:

Launch Playwright to load the initial page
Extract the Cookie header and X-CSRF-Token from the network logs
Inject these credentials into a high-performance HTTP client (like requests or httpx) to iterate through pagination pages

This method reduces the RAM overhead by 80% compared to clicking "Next" in a browser instance for every page.

Simulating Human Inputs for Bot Defense

Enterprise firewalls often fingerprint TLS handshakes and analyze interaction patterns. To avoid detection when using headless browsers, you must actively mask automation signals. This starts by scrubbing the navigator.webdriver property to ensure it remains undefined.

Beyond static properties, you must simulate organic behavior by replacing fixed time.sleep() calls with randomized intervals and implementing non-linear mouse movements, as sophisticated platforms like Taleo analyze cursor entropy to identify bots.

import random
import time

def humanize_interaction(page):
    """
    Injects random mouse movements and variable delays to mimic human behavior.
    """
    # 1. Randomize mouse movement
    for _ in range(3):
        x = random.randint(100, 800)
        y = random.randint(100, 600)
        page.mouse.move(x, y)
        time.sleep(random.uniform(0.1, 0.3))

    # 2. Variable scroll to trigger lazy loading
    page.mouse.wheel(0, random.randint(300, 700))
    time.sleep(random.uniform(0.5, 1.5))

Implementing these heuristics is mandatory when scraping enterprise portals to prevent your IP from being flagged as a bot.

Bypassing Aggregator Defenses

Aggregators like LinkedIn, Indeed, and Glassdoor employ the most sophisticated anti-bot systems, including TLS fingerprinting and behavioral analysis. Scraping their search result pages directly is resource-intensive and leads to rapid IP exhaustion. A more scalable architecture involves delegating the "Discovery" phase to Google and reserving direct scraping only for the final extraction.

Google SERP Discovery Strategy

Instead of fighting Indeed's pagination and rate limits, treat Google as your discovery layer. Google has already indexed the job postings. You can extract direct URLs to the job listings by scraping Google SERP using advanced search operators.

Use the site: operator to filter for direct listing pages:

LinkedIn: site:linkedin.com/jobs/view "python developer"
Indeed: site:indeed.com/viewjob "data engineer"
Greenhouse: site:boards.greenhouse.io "backend"

The following script uses the HasData SERP API to fetch targeted job URLs. This bypasses the need to manage captchas or IP rotation for Google searches.

import requests

def discover_jobs_via_serp(query: str, api_key: str):
    """
    Uses Google SERP to find direct job URLs, bypassing aggregator search restrictions.
    Query example: 'site:linkedin.com/jobs/view "python developer" "remote"'
    """
    url = "https://api.hasdata.com/scrape/google/serp"

    params = {
        "q": query,
        "location": "United States",
        "deviceType": "desktop",
    }

    headers = {
        'x-api-key': api_key,
        'Content-Type': "application/json"
    }

    try:
        response = requests.get(url, params=params, headers=headers, timeout=30)
        response.raise_for_status()

        data = response.json()

        # Extract organic links targeting job postings
        job_links = []
        if "organicResults" in data:
            for item in data["organicResults"]:
                if "linkedin.com/jobs/view" in item.get("link", ""):
                    job_links.append(item["link"])

        return job_links

    except requests.RequestException as e:
        print(f"SERP Discovery Failed: {e}")
        return []

# Usage
# links = discover_jobs_via_serp('site:linkedin.com/jobs/view "data engineer"', "YOUR_HASDATA_API_KEY")

This method yields clean URLs pointing directly to the job description pages, which have lighter protection than the search interfaces.

Unwinding Redirect Chains

Aggregators often mask the true source of a job with an "Apply" link that passes through a tracking redirect (e.g., indeed.com/rc/clk?jk=...). To build a high-quality dataset, you should resolve these redirects to find the canonical URL on the employer's ATS.

Once you resolve the final URL, pass it back to the Universal ATS Scraper defined in the first section. This ensures you scrape the data from the source of truth (Greenhouse/Lever) rather than the potentially stale aggregator listing.

LinkedIn and Indeed DOM Strategies

Sometimes you cannot bypass the aggregator. If you need to scrape specific metadata that only exists on the aggregator (such as "Easy Apply" status or "Number of Applicants"), you must interact with their DOM directly.

This requires advanced browser fingerprint management and residential proxies. We have created specialized guides and tools for these specific platforms. For production workloads, we recommend using the dedicated scrape/web endpoint with proxyType: residential enabled, as datacenter IPs are blocked instantly by these platforms.

Job Data Normalization

Raw job data is noisy and unstructured. Salaries appear as mixed strings like "$120k-150k" or "$60/hr" and tech stacks are buried in free text. To build a queryable dataset, you must normalize these fields into standardized integers and canonical entities.

Standardizing Salary Ranges via Regex

Salary normalization requires handling three variables: currency detection, period conversion (hourly to yearly), and abbreviation expansion ('k' notation). The following function unifies these into a single annual salary range.

import re
from typing import Dict, Any

def normalize_salary(text: str) -> Dict[str, Any]:
    """
    Parses salary strings into standardized annual integer ranges.
    Converts hourly rates to yearly (based on 2080 hours).
    """
    if not text:
        return {'min': None, 'max': None, 'currency': 'USD'}

    # 1. Define patterns for "100k-150k", "$50/hr", "€60,000"
    patterns = [
        r'(\d{1,3}(?:,\d{3})*)\s*k?\s*[-–to]\s*(\d{1,3}(?:,\d{3})*)\s*k?',
        r'\$\s*(\d{1,3}(?:,\d{3})*)\s*[-–to]\s*\$?\s*(\d{1,3}(?:,\d{3})*)',
    ]

    # 2. Detect Logic
    currency = 'GBP' if '£' in text else 'EUR' if '€' in text else 'USD'
    is_hourly = any(x in text.lower() for x in ['hour', 'hr', '/hr'])
    multiplier = 2080 if is_hourly else 1000 if 'k' in text.lower() else 1

    text_clean = re.sub(r'[$,£€]', '', text)

    for pat in patterns:
        match = re.search(pat, text_clean, re.IGNORECASE)
        if match:
            min_val = float(match.group(1).replace(',', '')) * multiplier
            max_val = float(match.group(2).replace(',', '')) * multiplier
            return {
                'min': int(min_val),
                'max': int(max_val),
                'currency': currency,
                'period': 'yearly'
            }

    return {'min': None, 'max': None, 'currency': currency}

This logic ensures that a wage of "$50/hr" and a salary of "$104,000" are treated as equivalent values in your database for accurate filtering and analysis.

Extracting Tech Stacks with FlashText

Regex is inefficient for extracting thousands of keywords from long descriptions. The Aho-Corasick algorithm (implemented via the flashtext library) performs replacement and extraction in linear time. It maps multiple variations of a term to a single canonical ID.

from flashtext import KeywordProcessor
from typing import List

def extract_tech_stack(description: str) -> List[str]:
    """
    Extracts canonical tech tags using O(N) FlashText algorithm.
    Maps variations like 'React.js' -> 'React'.
    """
    processor = KeywordProcessor(case_sensitive=False)

    # Map variations to canonical names
    tech_map = {
        'Python': ['python', 'py', 'python3'],
        'JavaScript': ['javascript', 'js', 'node.js'],
        'React': ['react', 'reactjs', 'react.js'],
        'AWS': ['aws', 'amazon web services', 'ec2'],
        'Docker': ['docker', 'containerization'],
        'Kubernetes': ['kubernetes', 'k8s'],
        'PostgreSQL': ['postgresql', 'postgres', 'psql'],
        'Go': ['golang', 'go lang']
    }

    processor.add_keywords_from_dict(tech_map)

    found_keywords = processor.extract_keywords(description)
    return sorted(list(set(found_keywords)))

Using FlashText reduces processing time by orders of magnitude compared to looping through regex patterns for every keyword. This is critical when processing millions of job descriptions.

Detecting Expired Listings

Job postings have a short lifecycle. Aggregators often display cached pages for jobs that are closed on the source ATS. Scraping "zombie jobs" pollutes your database and wastes resources.

Implement a "Death Check" logic during the scrape:

Status Code. If the ATS returns 404 or 410, the job is dead.

Redirects. Many ATS platforms (Greenhouse, Lever) redirect expired job URLs to their main career page. If response.url does not match the requested URL, mark as expired.

Schema Validation. Check the validThrough property in the JSON-LD schema if present.

import requests

def check_job_health(url: str) -> str:
    """
    Determines if a job is ACTIVE, EXPIRED, or REDIRECTED_TO_INDEX.
    """
    try:
        # Prevent auto-redirects to analyze the chain
        resp = requests.get(url, timeout=10, allow_redirects=True)

        # 1. Check Hard 404
        if resp.status_code in [404, 410]:
            return "EXPIRED_HTTP_CODE"

        # 2. Check Soft 404 (Redirect to Career Home)
        # If the final URL path is significantly shorter than the original
        if len(resp.url) < len(url) - 10 and "jobs" not in resp.url.split("/")[-1]:
            return "EXPIRED_REDIRECT_TO_INDEX"

        # 3. Check JSON-LD Validity
        # (Simplified logic: In production parse the full HTML)
        if '"validThrough"' in resp.text:
            # Extract date string and compare to datetime.now()
            pass

        return "ACTIVE"

    except requests.RequestException:
        return "UNKNOWN"

Tracking these signals allows you to automatically purge stale listings from your database and maintain high data quality for your users.

Scaling and Infrastructure

Scaling a job scraper involves more than increasing concurrent threads. Aggregators and Enterprise ATS platforms employ sophisticated defenses that detect non-human traffic patterns. To scrape at scale you must manage IP reputation and cryptographic signatures.

Residential Proxy Rotation Logic

Datacenter IPs (AWS, GCP, Azure) are flagged by default on platforms like Indeed and LinkedIn. Using them results in immediate CAPTCHAs or 403 errors. You must use residential proxies which route traffic through legitimate ISP connections.

The following function demonstrates how to use rotating residential proxies for "hard targets" with HasData API.

import requests

def make_resilient_request(url: str, api_key: str):
    """
    Configures proxy rotation based on target state requirements.
    """
    payload = {
        "url": url,
        "proxyType": "residential", # Mandatory for Indeed/LinkedIn
        "proxyCountry": "US",
    }

    return requests.post(
        "https://api.hasdata.com/scrape/web",
        headers={
                "x-api-key": api_key,
                "Content-Type": "application/json"
            },
        json=payload
    )

Using residential proxies increases the cost per request but is often the only way to access data from protected domains. Route internal ATS API calls (Greenhouse/Lever) through cheaper datacenter proxies and reserve residential bandwidth for the aggregators.

TLS Fingerprint Management

Standard Python libraries like requests or urllib leak their identity during the TLS handshake. They use OpenSSL cipher suites that differ significantly from Chrome or Firefox. Security providers like Cloudflare and Akamai analyze this JA3 Fingerprint and block connections before they even send HTTP headers.

If you see 403 Forbidden errors despite using high-quality residential proxies, your TLS fingerprint is likely the cause. You have two engineering solutions:

Client-Side Impersonation. Use libraries like curl_cffi or tls-client in Python which emulate browser-specific TLS Hello packets.

Server-Side Offloading. Use a scraping API that manages the handshake. The scrape/web endpoint demonstrated throughout this guide automatically rotates TLS fingerprints.

By addressing both IP reputation and TLS signatures you ensure your pipeline remains resilient against modern bot mitigation strategies.

Conclusion

Building a production-grade job scraper requires architectural discipline rather than simple script writing. We have demonstrated that stability comes from decoupling discovery from extraction and prioritizing hidden APIs over visual DOM elements. Your pipeline must route traffic intelligently. Send Greenhouse links to lightweight JSON parsers and reserve heavy residential proxies for hostile aggregators like Indeed.

Data quality is equally critical. You must implement strict normalization layers for salaries and tech stacks to turn raw text into queryable insights. Without these hygiene checks, your dataset will degrade due to expired listings and inconsistent formatting.

Start by implementing the Universal ATS Scraper function provided in the first section of this guide. It offers the highest return on investment by covering the majority of tech listings with zero maintenance. As you scale to aggressive aggregators, integrate the residential proxy rotation and TLS management logic to ensure long-term resilience.

We Benchmarked CAPTCHA Solving Services to Find the Best One

Valentina Skakun — Mon, 16 Feb 2026 14:03:40 +0000

We tested 5 CAPTCHA solving APIs to measure actual performance. Here's how to run the same benchmarks yourself, with complete working code.

Why This Matters

CAPTCHA latency directly impacts automation throughput. A service that takes 20 seconds versus 6 seconds to solve Cloudflare Turnstile means 140 seconds of idle time per 10 requests. Success rates matter even more, a 70% rate means 30% of your budget goes to retries.

We measured both metrics across 5 services:

2Captcha
CapMonster Cloud
Anti-Captcha
DeathByCaptcha
SolveCaptcha

Quick Setup

Install dependencies:

pip install 2captcha-python anticaptchaofficial capmonstercloudclient deathbycaptcha requests playwright
playwright install

All examples below run 50 iterations and save timing data to text files. Adjust the range for faster testing:

for i in range(1, 50):  # Change to 10 instead of 50 for quick tests

Example 1: Image CAPTCHA (2Captcha)

The simplest benchmark. Upload an image (save it to images subfolder in the project folder), get text back, measure time and accuracy.

import time
from twocaptcha import TwoCaptcha
from twocaptcha.api import ApiException

API_KEY = "YOUR-API-KEY"
IMAGE_FILE = "images/captcha.jpg"
CORRECT_ANSWER = "hwjrc"

solver = TwoCaptcha(API_KEY)
correct_count = 0

with open("image_results.txt", "w", encoding="utf-8") as file:
    for i in range(1, 50):
        start_time = time.perf_counter()

        try:
            result = solver.normal(IMAGE_FILE, numeric=0)
            end_time = time.perf_counter()
            elapsed = end_time - start_time

            answer = result["code"].strip()

            if answer.lower() == CORRECT_ANSWER.lower():
                correct_count += 1
                file.write(f"{i}. OK ({answer}) — {elapsed:.2f} sec\n")
            else:
                file.write(f"{i}. WRONG ({answer}) — {elapsed:.2f} sec\n")

            file.flush()

        except ApiException as e:
            end_time = time.perf_counter()
            elapsed = end_time - start_time
            file.write(f"{i}. API Error: {e} — {elapsed:.2f} sec\n")
            file.flush()
            continue

        time.sleep(2)  # Rate limiting

accuracy = (correct_count / 50) * 100
print(f"Accuracy: {accuracy:.2f}%")

Output example:

1. OK (hwjrc) — 2.33 sec
2. OK (hwjrc) — 2.41 sec
3. WRONG (hwjrz) — 2.18 sec
...
Accuracy: 58.00%

This pattern works for any CAPTCHA solving service with minimal changes. Just swap the SDK and adjust the API call.

Example 2: Cloudflare Turnstile with Browser Automation (Anti-Captcha)

More complex. We use Playwright to load the page, solve the CAPTCHA via API, inject the token, and submit.

import time
from anticaptchaofficial.turnstileproxyless import turnstileProxyless
from playwright.sync_api import sync_playwright

API_KEY = "YOUR-API-KEY"
SITE_KEY = "0x4AAAAAAAGlwMzq_9z6S9Mh"
URL = "https://clifford.io/demo/cloudflare-turnstile"

solver = turnstileProxyless()
solver.set_verbose(0)
solver.set_key(API_KEY)

with sync_playwright() as p:
    browser = p.chromium.launch(headless=False)
    page = browser.new_page()

    with open("turnstile_results.txt", "w", encoding="utf-8") as file:
        for i in range(1, 50):
            page.goto(URL)
            start_time = time.perf_counter()

            try:
                solver.set_website_url(URL)
                solver.set_website_key(SITE_KEY)

                # Solve CAPTCHA via API
                token = solver.solve_and_return_solution()

                if token != 0:
                    end_time = time.perf_counter()
                    elapsed = end_time - start_time
                    file.write(f"{i}. Success — {elapsed:.2f} sec\n")
                    file.flush()

                    # Inject token into page
                    page.evaluate(
                        """
                        (token) => {
                            const input = document.querySelector('input[name="cf-turnstile-response"]');
                            if (input) input.value = token;
                        }
                        """,
                        token,
                    )

                    page.click("button[type='submit']")
                    page.wait_for_timeout(2000)
                else:
                    end_time = time.perf_counter()
                    elapsed = end_time - start_time
                    error_message = solver.get_error_description()
                    file.write(f"{i}. Solve error: {error_message} — {elapsed:.2f} sec\n")
                    file.flush()

            except Exception as e:
                end_time = time.perf_counter()
                elapsed = end_time - start_time
                file.write(f"{i}. Error: {str(e)} — {elapsed:.2f} sec\n")
                file.flush()

    browser.close()

Key steps:

Navigate to target page
Extract site key (hardcoded here, but can be scraped)
Send to CAPTCHA service API
Wait for token (polling happens inside SDK)
Inject token into DOM
Submit form
Log timing

This pattern applies to reCAPTCHA v2, Invisible, and other challenge types that require browser validation.

Example 3: Building Your Own Integration (SolveCaptcha)

Some services don't provide SDKs. Here's how to build a custom integration with manual polling:

import time
import requests
from playwright.sync_api import sync_playwright

API_KEY = "YOUR-API-KEY"
SITE_KEY = "6Le-wvkSAAAAAPBMRTvw0Q4Muexq9bi0DJwx_mJ-"
URL = "https://www.google.com/recaptcha/api2/demo"

def solve_captcha():
    # Step 1: Submit CAPTCHA task
    submit_url = "https://api.solvecaptcha.com/in.php"
    payload = {
        "key": API_KEY,
        "method": "userrecaptcha",
        "googlekey": SITE_KEY,
        "pageurl": URL,
        "json": 1
    }

    start = time.perf_counter()
    r = requests.post(submit_url, data=payload).json()

    if r.get("status") != 1:
        raise RuntimeError(f"Submit error: {r}")

    captcha_id = r["request"]

    # Step 2: Poll for solution
    result_url = "https://api.solvecaptcha.com/res.php"
    while True:
        time.sleep(5)  # Wait before polling

        res = requests.get(result_url, params={
            "key": API_KEY,
            "action": "get",
            "id": captcha_id,
            "json": 1
        }).json()

        if res.get("status") == 1:
            end = time.perf_counter()
            return res["request"], end - start

        if res.get("request") != "CAPCHA_NOT_READY":
            raise RuntimeError(f"Solve error: {res}")

# Use with Playwright
with sync_playwright() as p:
    browser = p.chromium.launch(headless=False)
    page = browser.new_page()

    page.goto(URL)
    token, elapsed = solve_captcha()

    print(f"Solved in {elapsed:.2f} seconds")

    # Inject token
    page.evaluate(f"""
        document.getElementById('g-recaptcha-response').value = '{token}';
    """)

    browser.close()

This is more work than using an SDK, but gives you full control over:

Polling intervals
Timeout handling
Error retry logic
Cost tracking

Performance Results

Here's what we measured across 50 attempts per CAPTCHA type:

Service	reCAPTCHA v2	Turnstile	Image
CapMonster	32.23s	6.24s	3.28s
2Captcha	50.71s	16.96s	2.33s
Anti-Captcha	92.86s	20.33s	13.82s
SolveCaptcha	84.80s	17.24s	8.46s
DeathByCaptcha	34.54s	13.07s	2.13s

Key findings:

CapMonster is 3x faster on Turnstile (6.24s vs 20.33s for Anti-Captcha)
2Captcha and DeathByCaptcha have the best Image CAPTCHA accuracies (58% and 64% vs 14% for CapMonster)
Success rates vary significantly - Anti-Captcha only achieves 86% on reCAPTCHA v2

Practical Tips

1. Python 3.13 Compatibility

DeathByCaptcha breaks on Python 3.13 due to removed imghdr module:

# Error you'll see:
ModuleNotFoundError: No module named 'imghdr'

Fix: Use Python 3.11 or manually patch:

py -3.11 your_script.py

2. Proper Error Handling

Always wrap API calls in try/except and log failures:

try:
    result = solver.solve()
    elapsed = time.perf_counter() - start
    file.write(f"{i}. Success — {elapsed:.2f}s\n")
except ApiException as e:
    file.write(f"{i}. API Error: {e}\n")
except Exception as e:
    file.write(f"{i}. Error: {e}\n")

This prevents one failed solve from crashing your entire benchmark.

3. Rate Limiting

Add delays between requests to avoid hitting API rate limits:

time.sleep(2)  # 2 seconds between solves

Most services allow 60-120 requests per minute. Check your provider's limits.

4. Audio CAPTCHA Support

Only 2Captcha handles Audio CAPTCHAs:

from twocaptcha import TwoCaptcha

solver = TwoCaptcha(API_KEY)
result = solver.audio("audio_file.mp3", "en")
answer = result["code"]

Success rate: 27%. Not great, but better than 0% from all other services.

5. Cost Optimization

Track cost per successful solve:

total_cost = attempts * price_per_solve
successful_solves = sum(1 for r in results if r["success"])
cost_per_success = total_cost / successful_solves

A cheaper service with 70% success rate may cost more than an expensive one with 100% success rate.

Decision Matrix

Choose based on your priority:

Speed? CapMonster (6.24s Turnstile, 32.23s reCAPTCHA v2)

Reliability? 2Captcha (100% success on major types, only Audio support)

Cost? SolveCaptcha ($0.55/1K reCAPTCHA v2, but no SDK)

Image accuracy? DeathByCaptcha (64% success rate)

Complete Benchmark

The full analysis includes:

Performance tables for all 6 CAPTCHA types
Cost per mille (CPM) calculations
Integration complexity breakdown
All test scripts on GitHub

Read the full benchmark with detailed results

Run It Yourself

Clone the benchmark repository. Install dependencies for your chosen service. Add your API key and run:

cd 2captcha
python image.py          # Test image recognition
python turnstile.py      # Test Cloudflare Turnstile
python recaptcha2.py     # Test reCAPTCHA v2
python geetest.py        # Test GeeTest v4

Results save to .txt files.

Notes

Testing environment: Python 3.13.7, Playwright 1.40+, official SDKs where available.
Full benchmark data: Complete performance tables, cost breakdowns, and all test scripts available in our full analysis.
GitHub repository: All scripts are open source and ready to run, captcha-solver-benchmark
Join our web scraping community to discuss automation strategies, CAPTCHA solving, proxy management, and more:
HasData Discord

Have you benchmarked CAPTCHA services? What metrics matter most for your use case? Drop your experience in the comments 👇

Automating SEO with Python and Scraping API

Valentina Skakun — Fri, 06 Feb 2026 12:59:27 +0000

Software as a Service costs for SEO can hit hundreds of dollars every month. Most of these platforms just wrap search data with a basic UI. You can build the same logic using Python and a scraping API. This approach gives you total control over the data. You skip the marketing and just get the insights.

Build Instead of Buy
Recursive Keyword Discovery
Semantic Intent Mapping
SERP Similarity Heatmaps
AI Overview Visibility Tracking
Conclusion

Code Snippets and Resources

Full scripts and detailed data outputs would make this post too long to read. I am sharing the core logic and functional snippets of 4 main scripts here to show the engineering behind the tools. If you want all scripts and deep dives into the outputs you can find the full guide on the HasData Blog.

Build Instead of Buy

Search engines require JavaScript rendering and complex proxy rotation. Writing a DIY scraper is a maintenance nightmare. A better way is just using a dedicated scraping API to handle the browser headers and CAPTCHAs. This lets you focus on the data logic. You get structured JSON instead of HTML. You pay for the data you use rather than a flat monthly fee for features you might not even need.

Recursive Keyword Discovery

Most keyword tools rely on old databases. You can find real time intent by scraping Google Autosuggest recursively. The logic uses a thread pool to append characters to a seed keyword. This forces the API to reveal long tail queries.

import requests
import xml.etree.ElementTree as ET
import string
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from urllib.parse import quote
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry

# --- CONFIGURATION ---
API_KEY = "HASDATA_API_KEY"
BASE_KEYWORD = "coffee"
MAX_DEPTH = 2  # Depth of recursion (e.g., 1 = 'coffee a', 2 = 'coffee ab')

# Set this according to your HasData plan's concurrency limit.
# For paid plans, this is 15–1500. For the trial plan, keep it at 1.
MAX_WORKERS = 1 

# --- SESSION SETUP (PERFORMANCE OPTIMIZATION) ---
# We use a global Session object to enable TCP connection reuse (Keep-Alive).
# This significantly reduces the overhead of establishing new SSL handshakes for every request.
session = requests.Session()

# Configure the HTTP Adapter with a connection pool.
# pool_maxsize must be greater than or equal to MAX_WORKERS to prevent blocking.
adapter = HTTPAdapter(
    pool_connections=1, 
    pool_maxsize=MAX_WORKERS + 5,  # Adding a small buffer
    max_retries=Retry(total=3, backoff_factor=1, status_forcelist=[500, 502, 503, 504])
)

# Mount the adapter to both HTTP and HTTPS protocols
session.mount("https://", adapter)
session.mount("http://", adapter)

def fetch_suggestions(query):
    """
    Sends a request to Google Suggest via HasData API using a rotating proxy.
    """
    # 1. URL Encode the query to handle spaces and special characters correctly
    # 'coffee ad' becomes 'coffee%20ad'
    encoded_query = quote(query)
    target_url = f"https://suggestqueries.google.com/complete/search?output=toolbar&hl=en&q={encoded_query}"

    # HasData requires the API key in the headers
    headers = {
        "x-api-key": API_KEY,
        "Content-Type": "application/json"
    }

    # payload configuration
    payload = {
        "url": target_url,
        "jsRendering": False,      # JS is not needed for the XML endpoint, saves credits/time
        "outputFormat": ["html"]   # HasData will return the raw response body
    }

    try:
        # IMPORTANT: Use session.post() instead of requests.post() to leverage the connection pool
        response = session.post(
            "https://api.hasdata.com/scrape/web",
            headers=headers,
            json=payload,
            timeout=30
        )

        if response.status_code != 200:
            # Log errors for debugging
            print(f"Error {response.status_code} for '{query}': {response.text}")
            return []

        # 2. Parse the Response
        # Google Suggest returns XML. HasData passes this raw content in the response body.
        xml_content = response.content
        root = ET.fromstring(xml_content)

        suggestions = []
        for child in root:
            if len(child) > 0 and 'data' in child[0].attrib:
                suggestions.append(child[0].attrib['data'])

        return suggestions

    except ET.ParseError:
        return [] # Ignore parsing errors (e.g., malformed XML)
    except Exception as e:
        print(f"Exception for '{query}': {e}")
        return []

def generate_search_terms_suffix(base, current_suffix, depth, max_depth):
    """
    Recursively generates search terms by appending characters to a suffix.
    Example: 
    Depth 1: coffee a ... coffee z
    Depth 2: coffee aa ... coffee zz
    """
    terms = []
    for char in string.ascii_lowercase:
        new_suffix = current_suffix + char
        term = f"{base} {new_suffix}"
        terms.append(term)

        # Recursive call to go deeper if max_depth is not reached
        if depth < max_depth:
            terms.extend(generate_search_terms_suffix(base, new_suffix, depth + 1, max_depth))

    return terms

def run_harvest():
    print(f"Generating queries for Depth {MAX_DEPTH}...")
    queries = generate_search_terms_suffix(BASE_KEYWORD, "", 1, MAX_DEPTH)
    print(f"Total queries to process: {len(queries)}")

    results = set()
    start_time_global = time.time()

    # ThreadPoolExecutor manages concurrent execution
    with ThreadPoolExecutor(max_workers=MAX_WORKERS) as executor:
        future_to_query = {executor.submit(fetch_suggestions, q): q for q in queries}

        count = 0
        total = len(queries)

        print(f"\nStarting harvest with {MAX_WORKERS} workers...")

        for future in as_completed(future_to_query):
            query = future_to_query[future]
            count += 1

            # Progress logging every 20 requests to monitor speed
            if count % 20 == 0:
                elapsed = time.time() - start_time_global
                rps = count / elapsed if elapsed > 0 else 0
                print(f"Progress: {count}/{total} | Speed: {rps:.2f} req/s")

            try:
                suggestions = future.result()
                if suggestions:
                    for s in suggestions:
                        results.add(s)
            except Exception:
                pass # Skip errors during mass scraping

    total_time = time.time() - start_time_global
    print(f"\nFinished in {total_time:.2f} seconds. Average Speed: {len(queries)/total_time:.2f} req/s")
    return results

if __name__ == "__main__":
    unique_keywords = run_harvest()

    filename = "long_tail_keywords_hasdata.csv"

    # Save results to CSV
    with open(filename, "w", encoding="utf-8") as f:
        f.write("Keyword\n")
        f.write("\n".join(unique_keywords))

    print(f"Done. Collected {len(unique_keywords)} unique keywords.")
    print(f"Saved to {filename}")

This method collects thousands of unique keywords in seconds. You are getting the data directly from the source.

Semantic Intent Mapping

Search intent tells you if a user wants to buy something or just learn. You can automate this by analyzing the top ten URLs for any query. Using regex to scan for paths like /shop/ or /blog/ provides a breakdown of the SERP composition.

import requests
import pandas as pd
from urllib.parse import urlparse

# Configuration
API_KEY = "HASDATA_API_KEY"
KEYWORD = "instant coffee"

def get_serp_links(query):
    url = "https://api.hasdata.com/scrape/google/serp"
    params = {
        "q": query,
        "gl": "us",
        "hl": "en",
        "deviceType": "desktop"
    }
    headers = {"x-api-key": API_KEY}

    try:
        response = requests.get(url, params=params, headers=headers, timeout=20)
        if response.status_code == 200:
            data = response.json()
            # Extract organic links
            return [result['link'] for result in data.get('organicResults', [])]
        return []
    except Exception as e:
        print(f"Error: {e}")
        return []

def classify_url(url):
    """
    Classifies a URL based on domain reputation, path keywords, and structure.
    """
    try:
        parsed = urlparse(url)
        domain = parsed.netloc.lower()
        path = parsed.path.lower()

        # 1. Specialized Platforms (Domain-based)
        if "wikipedia.org" in domain:
            return "Encyclopedic (Wikipedia)"

        if "youtube.com" in domain or "youtu.be" in domain:
            return "Video Content (YouTube)"

        if any(x in domain for x in ['reddit.com', 'quora.com', 'stackoverflow.com']):
            return "UGC (Forum/Discussion)"

        # 2. General Forum Detection (Path-based)
        if any(x in path for x in ['/forum', '/threads', '/community', '/board']):
            return "UGC (Forum/Discussion)"


        # 3. Homepage Detection
        if path == "" or path == "/":
            return "Homepage (Brand)"

        # 4. Transactional Keywords
        if any(x in path for x in ['/product', '/shop', '/item', '/collections', '/buy', '/pricing', '/store']) or "amazon.com" in domain:
            return "Transactional (Product)"

        # 5. Informational Keywords
        if any(x in path for x in ['/blog', '/guide', '/news', '/article', '/how-to', '/tips', '/wiki']):
            return "Informational (Blog)"

        # 6. Fallback (General Page)
        return "General Page"
    except:
        return "Unknown"

if __name__ == "__main__":
    print(f"Analyzing SERP for: '{KEYWORD}'...\n")
    links = get_serp_links(KEYWORD)

    if links:
        # Create a DataFrame for clean visualization
        df = pd.DataFrame(links, columns=['URL'])
        df['Type'] = df['URL'].apply(classify_url)

        # Calculate percentages
        breakdown = df['Type'].value_counts(normalize=True) * 100

        # Display the Data
        print("--- SERP Composition ---")
        print(df[['Type', 'URL']].to_string(index=True))

        print("\n--- Strategic Verdict ---")
        # Handle cases where multiple types might have the same top percentage
        if not breakdown.empty:
            dominant_type = breakdown.idxmax()
            percent = breakdown.max()

            print(f"Dominant Type: {dominant_type} ({percent:.1f}%)")

            if "Informational" in dominant_type:
                print("Action: Create a long-form Guide or Blog Post.")
            elif "Transactional" in dominant_type:
                print("Action: Create a Product Page or Collection Page.")
            elif "Homepage" in dominant_type:
                print("Action: High difficulty. Requires strong Brand Authority.")
            elif "Video" in dominant_type:
                print("Action: Text alone won't rank. Produce a high-quality Video.")
            elif "UGC" in dominant_type:
                print("Action: Engage in community discussions or create 'Real Review' style content.")
            elif "Encyclopedic" in dominant_type:
                print("Action: Definitional intent. Very hard to outrank Wikipedia directly.")
            else:
                print("Action: Mixed SERP. Manual review recommended.")
        else:
            print("Action: No valid classification data available.")
    else:
        print("No results found.")

If sixty percent of the results are products you know you need a landing page. This removes the guesswork from content strategy.

SERP Similarity Heatmaps

Targeting the same topic with two pages causes cannibalization. You can solve this by comparing the Jaccard Index of search results. If two keywords share many of the same URLs they belong on the same page.

import requests
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
from collections import Counter
import time

# --- CONFIGURATION ---
API_KEY = "HASDATA_API_KEY"
KEYWORDS = [
    "health benefits of coffee",
    "benefits of coffee for men",
    "benefits of coffee for women",
    "health benefits of black coffee",
    "health benefits of mushroom coffee",
    "health benefits of decaf coffee",
]
# ---------------------

def get_top_urls_set(query):
    """
    Requests the API and returns a SET of organic URLs.
    Using a set is convenient for mathematical intersection operations.
    """
    url = "https://api.hasdata.com/scrape/google/serp"
    params = {
        "q": query,
        "gl": "us",
        "hl": "en",
        "deviceType": "desktop"
      }
    headers = {"x-api-key": API_KEY}

    print(f"Scanning SERP for: '{query}'...")
    try:
        response = requests.get(url, params=params, headers=headers, timeout=25)
        if response.status_code == 200:
            data = response.json()
            organic = data.get('organicResults', [])
            # Extract links only. Taking the first 10 if more are returned.
            links = [result['link'] for result in organic[:10] if 'link' in result]
            return set(links)
        else:
            print(f"Error {response.status_code} for query '{query}'")
            return set()
    except Exception as e:
        print(f"Exception for query '{query}': {e}")
        return set()

def calculate_jaccard(set1, set2):
    """Calculates Jaccard Index between two sets."""
    intersection = len(set1.intersection(set2))
    union = len(set1.union(set2))
    if union == 0:
        return 0.0
    return intersection / union

def visualize_similarity_heatmap(similarity_df):
    """Builds and displays the heatmap."""
    plt.figure(figsize=(10, 8))
    sns.set_theme(context='notebook', style='whitegrid', font_scale=1.1)

    # Create heatmap
    # annot=True shows numbers in cells
    # fmt=".0%" formats numbers as percentages
    # cmap="YlGnBu" - color scheme (Yellow to Blue)
    ax = sns.heatmap(
        similarity_df, 
        annot=True, 
        fmt=".0%", 
        cmap="YlGnBu", 
        vmin=0, 
        vmax=1,
        cbar_kws={'label': 'Jaccard Similarity Score'}
    )

    plt.title(f"SERP Similarity Heatmap", fontsize=16, pad=20)
    plt.xticks(rotation=45, ha='right')
    plt.yticks(rotation=0)
    plt.tight_layout()
    print("Visualizing heatmap...")
    plt.show()

def show_top_common_urls(results_data):
    """Counts and prints the most frequently occurring URLs."""
    all_urls_flat = []
    # Collect all URLs from all sets into one list
    for url_set in results_data.values():
        all_urls_flat.extend(list(url_set))

    # Count frequency
    url_counts = Counter(all_urls_flat).most_common(15)

    print(f"\n### Top Recurring URLs (across {len(KEYWORDS)} queries)")
    print(f"{'Frequency':<10} | URL")
    print("-" * 80)
    for url, count in url_counts:
        # Show only if it appears more than once, unless we have very few queries
        if count > 1 or len(KEYWORDS) <= 2: 
             print(f"{count:<10} | {url}")      

# --- MAIN LOGIC ---
if __name__ == "__main__":
    # 1. Data Collection
    results_data = {}
    print("--- Start SERP Collection ---\n")
    for keyword in KEYWORDS:
        results_data[keyword] = get_top_urls_set(keyword)
        # Short pause between requests
        time.sleep(1) 
    print("\n--- Collection Finished ---")

    # 2. Calculation of Similarity Matrix (Jaccard)
    n = len(KEYWORDS)
    similarity_matrix = pd.DataFrame(
        index=KEYWORDS, 
        columns=KEYWORDS, 
        dtype=float
    )

    # Double loop to compare every query against every other query
    for i in range(n):
        for j in range(n):
            kw1 = KEYWORDS[i]
            kw2 = KEYWORDS[j]
            score = calculate_jaccard(results_data[kw1], results_data[kw2])
            similarity_matrix.iloc[i, j] = score

    # 3. Output Results to Console (Text Table)
    print("\n### Similarity Matrix (Jaccard Index)")
    pd.set_option('display.max_columns', None)
    pd.set_option('display.width', 1000)
    # Format output as percentages
    print(similarity_matrix.style.format("{:.1%}").to_string())
    print("\n" + "="*50 + "\n")

    # 4. Common URL Analysis
    show_top_common_urls(results_data)

    # 5. Visualization (Heatmap)
    visualize_similarity_heatmap(similarity_matrix)

Using Seaborn to plot these scores as a heatmap makes the clusters obvious.

You see exactly which topics need their own articles and which should be merged.

AI Overview Visibility Tracking

Standard rank trackers often miss AI Overviews. These summaries push organic results down the page. You can monitor this by parsing the aiOverview object in your API response. This script checks if your domain appears in the AI citations.

import requests
import pandas as pd
import time
from urllib.parse import urlparse

# --- CONFIGURATION ---
API_KEY = "HASDATA_API_KEY"
TARGET_DOMAIN = "webmd.com" # The domain we are monitoring for citations

KEYWORDS = [
    "health benefits of coffee",
    "benefits of coffee for men",
    "benefits of coffee for women",
    "health benefits of black coffee",
    "health benefits of mushroom coffee",
    "health benefits of decaf coffee"
]

def normalize_domain(url):
    """
    Extracts the base domain from a URL to ensure accurate matching.
    e.g., 'https://www.sub.example.com/page' -> 'sub.example.com'
    """
    try:
        parsed = urlparse(url)
        # Returns netloc (e.g., www.example.com). We strip 'www.' for broader matching.
        return parsed.netloc.lower().replace('www.', '')
    except Exception:
        return ""

def fetch_serp_data(query):
    """
    Executes a SERP API request targeting Google's AI Overview.
    Note: AI Overviews are volatile; successful triggering depends on
    location (US) and device type (Desktop/Mobile).
    """
    endpoint = "https://api.hasdata.com/scrape/google/serp"
    params = {
        "q": query,
        "gl": "us",       # Geo-location: US is critical for SGE consistency
        "hl": "en",       # Language: English
        "deviceType": "desktop"
    }
    headers = {"x-api-key": API_KEY}

    try:
        response = requests.get(endpoint, params=params, headers=headers, timeout=30)
        if response.status_code == 200:
            return response.json()
        else:
            print(f"[API Error] Status: {response.status_code} for query '{query}'")
            return None
    except Exception as e:
        print(f"[Network Error] {e}")
        return None

def extract_ai_citations(serp_data):
    """
    Parses the JSON response to locate the 'aiOverview' block.
    Extracts structured references (citations) if present.
    """
    # HasData typically returns SGE data in the 'aiOverview' key
    ai_overview = serp_data.get('aiOverview')

    if not ai_overview:
        return None, [] # AI Overview not triggered for this query

    # Extract the 'references' array which contains the source links
    references = ai_overview.get('references', [])

    # Map into a cleaner format for analysis
    citations = []
    for ref in references:
        citations.append({
            'index': ref.get('index'),
            'title': ref.get('title'),
            'url': ref.get('link'),
            'source_name': ref.get('source')
        })

    return ai_overview, citations

def run_monitor():
    results = []
    print(f"--- Starting AI Overview Monitor for: {TARGET_DOMAIN} ---")
    print(f"Processing {len(KEYWORDS)} keywords...\n")

    for query in KEYWORDS:
        print(f"Analyzing: '{query}'...")

        data = fetch_serp_data(query)

        if not data:
            continue

        # 1. Check for AI Overview existence
        ai_block, citations = extract_ai_citations(data)

        is_triggered = ai_block is not None
        is_cited = False
        citation_rank = None
        found_url = None

        # 2. Analyze Citations if AI Overview exists
        if is_triggered:
            target_clean = normalize_domain(f"https://{TARGET_DOMAIN}")

            for cit in citations:
                cited_domain = normalize_domain(cit['url'])

                # Check for substring match (handles subdomains and exact matches)
                if target_clean in cited_domain:
                    is_cited = True
                    citation_rank = cit['index'] # 0-based index in the citation carousel
                    found_url = cit['url']
                    break # Stop after finding the first occurrence

        # 3. Aggregate Data
        results.append({
            "Keyword": query,
            "AI Triggered": is_triggered,
            "Is Cited": is_cited,
            "Citation Index": citation_rank if is_cited else "-",
            "Cited URL": found_url if is_cited else "-"
        })

        # Respect API rate limits
        time.sleep(1)

    # --- REPORTING ---
    df = pd.DataFrame(results)

    print("\n" + "="*60)
    print("AI OVERVIEW VISIBILITY REPORT")
    print("="*60)

    # formatting booleans for readability
    df['AI Triggered'] = df['AI Triggered'].map({True: '✅ Yes', False: '❌ No'})
    df['Is Cited'] = df['Is Cited'].map({True: '✅ YES', False: '❌ No'})

    # Handle display if dataframe is empty
    if not df.empty:
        # Use to_markdown if available, otherwise string
        try:
            print(df.to_markdown(index=False))
        except ImportError:
            print(df.to_string(index=False))

        # Summary Metrics
        total = len(df)
        triggered = len(df[df['AI Triggered'] == '✅ Yes'])
        cited = len(df[df['Is Cited'] == '✅ YES'])

        print("\n--- SUMMARY METRICS ---")
        print(f"AI Coverage: {triggered}/{total} keywords ({(triggered/total)*100:.1f}%)")
        if triggered > 0:
            print(f"Share of Voice: {cited}/{triggered} AI Overviews ({(cited/triggered)*100:.1f}%)")
        else:
            print("Share of Voice: N/A (No AI Overviews generated)")

    else:
        print("No data collected.")

if __name__ == "__main__":
    run_monitor()

Tracking this allows you to see if you are losing traffic to AI even when your organic rankings stay high.

Conclusion

The repository at HasData Python for SEO contains the full implementation of these features.

Use These Python Patterns for Price Scraping

Valentina Skakun — Thu, 22 Jan 2026 09:41:11 +0000

I see too many scraping scripts using CSS selectors to find prices. Then frontend developers change class names. Your scraper breaks. You fix it. They change it again.

There is a better way. Here are three patterns for building resilient price monitors in Python.

Intercepting Internal APIs

Most modern sites hydrate their DOM using JSON. You can find this data by filtering for XHR requests in DevTools.

Instead of fighting WAFs with requests use Playwright to intercept this traffic passively.

The Selector Priority Waterfall

Sometimes you must parse HTML. Do not rely on a single selector. Build a hierarchy of reliability.

JSON-LD (Structured Data)
Meta Tags (itemprop="price")
Data Attributes (data-price)
CSS Classes


# Check for machine-readable data first
json_ld = soup.find("script", {"type": "application/ld+json"})
if json_ld:
    data = json.loads(json_ld.string)
    return data["offers"]["price"]

Strict Data Normalization

Never store raw strings. Never use floats. Use the decimal library and handle locale differences. 1.000 is one thousand in the US but one in Germany.

from decimal import Decimal, InvalidOperation
import re


def normalize_price(raw_text, locale_hint="AUTO"):
    """
    Converts localized price strings to precise Decimal objects.

    Args:
        raw_text: Dirty strings like "€ 1.234,56", "$1,234.56", or "£1 234.56"
        locale_hint: "US", "EU", or "AUTO" for heuristic detection

    Returns:
        Decimal: Safe for financial calculations (never float)
    """
    if not raw_text:
        return None


    # Step 1: Remove artifacts
    # We strip everything except digits, commas, and dots
    # This handles space separators (e.g., "1 200.00" becomes "1200.00")
    cleaned = re.sub(r'[^\d.,]', '', raw_text)

    if not cleaned:
        raise ValueError(f"No numeric data found in: {raw_text}")

    # Step 2: Detect Format
    if locale_hint == "AUTO":
        # If both separators exist, the right-most one is the decimal
        if ',' in cleaned and '.' in cleaned:
            last_comma = cleaned.rfind(',')
            last_period = cleaned.rfind('.')
            locale_hint = "EU" if last_comma > last_period else "US"

        # If only comma exists, check context
        elif ',' in cleaned:
            # Ambiguous Case: "1,234"
            # Logic: If exactly 2 digits follow the comma, assume EU (cents)
            # Otherwise assume US thousands separator
            parts = cleaned.split(',')
            locale_hint = "EU" if len(parts[-1]) == 2 else "US"

        else:
            # Default to US if no comma is present
            locale_hint = "US"

    # Step 3: Normalize to Python Standard (US)
    if locale_hint == "EU":
        # Convert "1.234,56" -> "1234.56"
        normalized = cleaned.replace('.', '').replace(',', '.')
    else:
        # Convert "1,234.56" -> "1234.56"
        normalized = cleaned.replace(',', '')

    try:
        return Decimal(normalized)
    except InvalidOperation:
        raise ValueError(f"Normalization failed: {raw_text} -> {normalized}")


# Unit Tests for Validation
if __name__ == "__main__":
    test_cases = [
        ("$1,234.56", "US", Decimal("1234.56")),
        ("€ 1.234,56", "EU", Decimal("1234.56")),
        ("Price: 1,200", "US", Decimal("1200")),    # US Integer
        ("1,20 €", "AUTO", Decimal("1.20")),        # EU Decimal
    ]


    for raw, locale, expected in test_cases:
        result = normalize_price(raw, locale)
        print(f"Input: {raw:15} | Mode: {locale:4} | Result: {result}")
        assert result == expected

Notes

If you scrape at scale you will hit blocking issues. WAFs analyze your TLS fingerprint. Standard Python scripts get blocked even with good proxies. You must ensure your scraper mimics the TLS signature of a real browser.

We published a deep dive on how to implement this architecture. It includes code for AI parsing and multi-region monitoring.

Check out the full article on our blog: Read the Full Article

Scraping ZoomInfo with One Universal Script Using SeleniumBase

Valentina Skakun — Wed, 10 Dec 2025 14:47:04 +0000

Learn how to scrape ZoomInfo data from search results, person profiles, and company pages using a single SeleniumBase script. Follow clear steps with explanations and see what data you can get without login.

Before You Start
Prerequisites and Setup
Available Data Points
Universal ZoomInfo Scraping Script
Anti-Scraping Measures
Notes

Before You Start

ZoomInfo can be scraped in a simple and stable way. The site stores key data inside application/json blocks.

You can extract search results, profiles, and company details without complex CSS selectors.

Prerequisites and Setup

ZoomInfo shows a press-and-hold captcha if it suspects automated actions.

Simple request libraries or headless Selenium/Playwright won’t work. You need tools like SeleniumBase, Playwright Stealth, or Patchright that patch fingerprints, headers, and hide headless mode.

We'll use SeleniumBase in UC mode (built on Undetectable Browser). Install it with:

pip install seleniumbase

Available Data Points

Here’s what you can get from each page type:

Page Type	Key Data Points	Notes
Search Pages	name, title, profile link, company link	First 5 pages only. Emails/phone/images usually missing. Some fields empty or null
Person Profiles	full name, title, photo, bio, masked email/phone, work address, social links, work & education history, employer info, colleagues, similar profiles, web mentions, AI signals	Most data complete. Contact info partially hidden
Company Pages	legal name, size, employees, tech stack, finances, competitors, executives, address, social links, news, acquisitions, org charts, email patterns, hiring trends, intent signals, awards, comparables	Some contact info, historic financials, email samples, and intent data may be partial or masked

This gives a developer-focused overview in a compact format. You can quickly decide which data points you need for your project.

Universal Scraping Script

This one script works for search pages, person profiles, and company pages. The steps:

Set the page URL.
Launch the browser in undetectable mode.
Loop if multiple pages (search results only).
Extract JSON from <script type="application/json">.
Remove unnecessary keys (__nghData__, cta_config).
Save the data in JSON.

from seleniumbase import SB
from selenium.webdriver.common.by import By
import time, json

# Base URL for the page (search, person, or company)
base_url = "https://www.zoominfo.com/people-search/<filters>"  # or person/company URL
pages = 5  # for search pages; set 1 for single profile/company
all_data = []

with SB(uc=True, test=True) as sb:
    for page in range(1, pages + 1):
        url = f"{base_url}?pageNum={page}" if pages > 1 else base_url
        sb.uc_open_with_reconnect(url, 4)
        time.sleep(1)  # Wait for JSON scripts to render
        try:
            scripts = sb.find_elements('script[type="application/json"]', by=By.CSS_SELECTOR)
            for el in scripts:
                content = el.get_attribute("innerHTML")
                data = json.loads(content)
                data.pop("__nghData__", None)
                data.pop("cta_config", None)
                all_data.append(data)
        except Exception as e:
            print(f"Page {page} error:", e)

# Save the data
with open("zoominfo_data.json", "w", encoding="utf-8") as f:
    json.dump(all_data, f, ensure_ascii=False, indent=2)

This JSON contains almost all available data:

Search pages: names, titles, profile links, company links. Emails, phones, and images mostly hidden.
Person pages: full personal info, masked contacts, work/education, colleagues, AI signals.
Company pages: legal name, employees, tech stack, finances, executives, news, acquisitions, hiring trends, awards, etc. Some fields partially masked.

Anti-Scraping Measures

ZoomInfo has strong anti-bot protections. Check status codes before parsing data:

Status Code	Meaning	Recovery
200	Success	Continue
429	Rate limited	Wait 30-60s, retry
403	Forbidden (IP blocked)	Switch IP/proxy, retry next day
503	Service unavailable	Retry after 5 min
Empty 200	Honeypot	Switch IP
Redirect /error	Detected scraper	Add delay, rotate proxy

Avoid errors, captchas, and bans by masking bots, rotating proxies, adding pauses, and handling errors with longer delays.

Notes

This post covers the universal ZoomInfo scraping script and key data points.

For a complete walkthrough with step-by-step explanations, visuals, and tips, check out the full article on our blog: Read the Full Article

All examples are in Github Repo.

Analyzing News Sentiment with VADER

Valentina Skakun — Thu, 04 Dec 2025 14:37:35 +0000

In this tutorial, we'll walk through how to analyze the sentiment of news headlines using VADER — a fast and effective sentiment analysis model designed for social media text, which works well for news headlines too.

We will:

Collect news data using HasData's Google News API.
Use VADER to analyze the sentiment of each article's headline and snippet.
Classify articles into positive, neutral, and negative categories.
Save the results in a structured JSON format.

Introduction
Setup
Fetching Google News Data
Sentiment Analysis with VADER
Saving and Visualizing Results
Full Code
Next Steps
Further Reading

Introduction

Understanding the sentiment of news headlines is essential for various applications like trend analysis, brand monitoring, and political sentiment tracking.

To perform sentiment analysis, we’ll use VADER (Valence Aware Dictionary and sEntiment Reasoner), which is particularly effective for short texts such as headlines and tweets. The key advantage of VADER is that it’s a lightweight, rule-based model that works well for social media and news data, making it an ideal choice for our case.

Setup

You will need the following tools and libraries:

Python 3.x
requests for API calls
nltk for sentiment analysis
matplotlib for optional visualizations
JSON for saving results

pip install requests matplotlib nltk

You also need to download the VADER lexicon from NLTK:

import nltk
nltk.download('vader_lexicon')

Make sure to get your HasData API key from your HasData dashboard.

Fetching Google News Data

We’ll start by collecting news data from Google News using HasData's API. The goal is to fetch the headline and snippet from each article.

import requests
import json

API_KEY = "HASDATA-API-KEY"

params_raw = {
    "q": "",
    "gl": "us",
    "hl": "en",
    "topicToken": "CAAqJggKIiBDQkFTRWdvSUwyMHZNREpxYW5RU0FtVnVHZ0pWVXlnQVAB",  # Example: Entertainment
}

params = {k: v for k, v in params_raw.items() if v}
news_url = "https://api.hasdata.com/scrape/google/news"
news_headers = {"Content-Type": "application/json", "x-api-key": API_KEY}

resp = requests.get(news_url, params=params, headers=news_headers)
resp.raise_for_status()
data = resp.json()

In this example, we're fetching articles from the Entertainment section. You can change the topicToken to explore other categories like Business, Technology, etc.

Sentiment Analysis with VADER

Now that we have the data, let’s analyze the sentiment of each article’s headline and snippet. VADER returns a score between -1 and +1, where:

Positive sentiment: +0.05 or higher
Negative sentiment: -0.05 or lower
Neutral sentiment: between -0.05 and +0.05

We'll classify the news articles based on these scores.

from nltk.sentiment.vader import SentimentIntensityAnalyzer

sid = SentimentIntensityAnalyzer()

grouped_news = {"positive": [], "neutral": [], "negative": []}

for item in data.get("newsResults", []):
    highlight = item.get("highlight", {})
    text_to_analyze = highlight.get("title", "") + " " + highlight.get("snippet", "")

    score = sid.polarity_scores(text_to_analyze)
    news_item = {
        "title": highlight.get("title"),
        "link": highlight.get("link"),
        "snippet": highlight.get("snippet"),
        "source_name": highlight.get("source", {}).get("name"),
        "date": highlight.get("date"),
        "thumbnail": highlight.get("thumbnail")
    }

    if score['compound'] >= 0.05:
        grouped_news["positive"].append(news_item)
    elif score['compound'] <= -0.05:
        grouped_news["negative"].append(news_item)
    else:
        grouped_news["neutral"].append(news_item)

In this code:

Sentiment score: We analyze both the title and snippet of each news article using VADER's polarity_scores.
Classification: Based on the compound score, we classify the article as positive, neutral, or negative.
Group data: All articles are saved into one of these three categories.

Saving and Visualizing Results

Once the sentiment analysis is complete, we’ll save the results in a structured JSON format.

with open("news_sentiment.json", "w", encoding="utf-8") as f:
    json.dump(grouped_news, f, ensure_ascii=False, indent=2)

Optionally, you can also visualize the distribution of positive, neutral, and negative articles using matplotlib.

import matplotlib.pyplot as plt

sentiments = ['positive', 'neutral', 'negative']
counts = [len(grouped_news['positive']), len(grouped_news['neutral']), len(grouped_news['negative'])]

plt.figure(figsize=(8, 6))
plt.bar(sentiments, counts, color=['green', 'gray', 'red'])
plt.title("News Sentiment Distribution")
plt.xlabel("Sentiment")
plt.ylabel("Number of Articles")
plt.show()

This chart will give you a quick overview of how the news is distributed across positive, neutral, and negative sentiments.

Full Code

Here's the full code that combines everything, from fetching news data to performing sentiment analysis and visualizing the results. You can easily copy and run it:

import requests
import json
from nltk.sentiment.vader import SentimentIntensityAnalyzer
import matplotlib.pyplot as plt
import nltk
import re

# Download NLTK's VADER lexicon
nltk.download('vader_lexicon')

API_KEY = "HASDATA-API-KEY"  # Replace with your HasData API Key

# Set parameters for the API request
params_raw = {
    "q": "",
    "gl": "us",
    "hl": "en",
    "topicToken": "CAAqJggKIiBDQkFTRWdvSUwyMHZNREpxYW5RU0FtVnVHZ0pWVXlnQVAB",  # Example: Entertainment
}

params = {k: v for k, v in params_raw.items() if v}
news_url = "https://api.hasdata.com/scrape/google/news"
news_headers = {"Content-Type": "application/json", "x-api-key": API_KEY}

# Fetch news data from Google News
resp = requests.get(news_url, params=params, headers=news_headers)
resp.raise_for_status()
data = resp.json()

# Initialize VADER SentimentIntensityAnalyzer
sid = SentimentIntensityAnalyzer()

# Create a dictionary to store the categorized news
grouped_news = {"positive": [], "neutral": [], "negative": []}

# Process each article and classify based on sentiment
for item in data.get("newsResults", []):
    highlight = item.get("highlight", {})
    text_to_analyze = highlight.get("title", "") + " " + highlight.get("snippet", "")

    score = sid.polarity_scores(text_to_analyze)
    news_item = {
        "title": highlight.get("title"),
        "link": highlight.get("link"),
        "snippet": highlight.get("snippet"),
        "source_name": highlight.get("source", {}).get("name"),
        "date": highlight.get("date"),
        "thumbnail": highlight.get("thumbnail")
    }

    # Classify based on sentiment score
    if score['compound'] >= 0.05:
        grouped_news["positive"].append(news_item)
    elif score['compound'] <= -0.05:
        grouped_news["negative"].append(news_item)
    else:
        grouped_news["neutral"].append(news_item)

# Save the results in a JSON file
with open("news_sentiment.json", "w", encoding="utf-8") as f:
    json.dump(grouped_news, f, ensure_ascii=False, indent=2)

# Visualize sentiment distribution
sentiments = ['positive', 'neutral', 'negative']
counts = [len(grouped_news['positive']), len(grouped_news['neutral']), len(grouped_news['negative'])]

plt.figure(figsize=(8, 6))
plt.bar(sentiments, counts, color=['green', 'gray', 'red'])
plt.title("News Sentiment Distribution")
plt.xlabel("Sentiment")
plt.ylabel("Number of Articles")
plt.show()

Next Steps

Improve sentiment accuracy by using a custom sentiment model tailored to news data.
Combine this with topic modeling to analyze sentiment by specific topics.
Use this analysis for real-time news tracking in specific domains (e.g., political, business).

Building a Topic Frequency Chart from Google News Headlines

Valentina Skakun — Tue, 25 Nov 2025 15:18:29 +0000

This tutorial shows how to collect Google News data using HasData’s Google News API and visualize the most common topics or keywords in news headlines. We’ll process the data, remove stop words, and create a simple frequency chart with Python.

Introduction
Setup
Fetching Google News Data
Processing Headlines
Creating a Frequency Chart
Full Code
Next Steps
Further Reading

Introduction

News data is rich, but raw headlines can be messy. Common words like “the”, “of”, and “in” dominate the text, making it hard to extract meaningful insights. In this guide, we’ll:

Fetch news headlines via HasData’s Google News API.
Extract the highlight.title field from each article.
Count the frequency of meaningful words.
Visualize the top keywords using matplotlib.

This approach is useful for tracking trending topics, analyzing industry coverage, or quickly summarizing news from a specific domain.

Setup

You will need:

Python 3
requests
matplotlib
nltk
Standard library modules: json, collections.Counter, re

pip install requests matplotlib nltk

You also need to download the stopwords data from NLTK. You can do this by running the following command in Python:

import nltk
nltk.download('stopwords')

Make sure you have a HasData API key. You can get one for free from your HasData dashboard.

Fetching Google News Data

We’ll use the API to fetch headlines from a specific topic. You can change the topicToken to fetch different sections like Technology, Business, or Sports.

import requests
import json

API_KEY = "HASDATA-API-KEY"

params_raw = {
    "q": "",
    "gl": "us",
    "hl": "en",
    "topicToken": "CAAqJggKIiBDQkFTRWdvSUwyMHZNREpxYW5RU0FtVnVHZ0pWVXlnQVAB",  # Example: Entertainment
}

params = {k: v for k, v in params_raw.items() if v}
news_url = "https://api.hasdata.com/scrape/google/news"
news_headers = {"Content-Type": "application/json", "x-api-key": API_KEY}

resp = requests.get(news_url, params=params, headers=news_headers)
resp.raise_for_status()
data = resp.json()

Processing Headlines

We’ll now extract the titles and filter out common stop words using NLTK's built-in list of stopwords.

from collections import Counter
import re
from nltk.corpus import stopwords

# Load stopwords from NLTK
stop_words = set(stopwords.words('english'))

titles = [item.get("highlight", {}).get("title", "") for item in data.get("newsResults", [])]

words = []
for title in titles:
    for word in re.findall(r'\w+', title.lower()):
        if word not in stop_words and len(word) > 2:
            words.append(word)

counter = Counter(words)
most_common = counter.most_common(20)

Now we have a list of words that appear most frequently in the headlines, excluding common stop words.

Creating a Frequency Chart

Finally, we visualize the results using matplotlib.

import matplotlib.pyplot as plt

if not most_common:
    print("No meaningful words.")
else:
    labels, counts = zip(*most_common)
    plt.figure(figsize=(12,6))
    plt.bar(labels, counts, color='skyblue')
    plt.xticks(rotation=45, ha='right')
    plt.title("Top 20 meaningful words in news headlines")
    plt.ylabel("Frequency")
    plt.tight_layout()
    plt.show()

You should see a clear bar chart showing the most common topics from the headlines.

Full Code

import requests
import json
from collections import Counter
import matplotlib.pyplot as plt
import re
import nltk
from nltk.corpus import stopwords

# Download stopwords if not already downloaded
nltk.download('stopwords')

API_KEY = "HASDATA-API-KEY"

# Parameters for Google News API request
params_raw = {
    "q": "",
    "gl": "us",
    "hl": "en",
    "topicToken": "CAAqJggKIiBDQkFTRWdvSUwyMHZNREpxYW5RU0FtVnVHZ0pWVXlnQVAB"
}

params = {k: v for k, v in params_raw.items() if v}

news_url = "https://api.hasdata.com/scrape/google/news"
news_headers = {"Content-Type": "application/json", "x-api-key": API_KEY}

# Fetch news data
resp = requests.get(news_url, params=params, headers=news_headers)
resp.raise_for_status()
data = resp.json()

# Extract titles
titles = [item.get("highlight", {}).get("title", "") for item in data.get("newsResults", [])]

# Load stopwords from NLTK
stop_words = set(stopwords.words('english'))

# Process words from titles
words = []
for title in titles:
    for word in re.findall(r'\w+', title.lower()):
        if word not in stop_words and len(word) > 2:
            words.append(word)

# Count words
counter = Counter(words)
most_common = counter.most_common(20)

# Plot results
if not most_common:
    print("No meaningful words.")
else:
    labels, counts = zip(*most_common)
    plt.figure(figsize=(12,6))
    plt.bar(labels, counts)
    plt.xticks(rotation=45, ha='right')
    plt.title("Top 20 meaningful words in news headlines")
    plt.ylabel("Frequency")
    plt.tight_layout()
    plt.show()

Next Steps

Expand the stop words list to filter more common words.
Analyze key topics using bigrams or trigrams for richer insights.
Combine multiple topic sections to see trends across industries.
Automate periodic fetching to track trends over time.

Downloading TikTok Videos with Python (Part 2)

Valentina Skakun — Thu, 13 Nov 2025 09:23:55 +0000

This is Part 2 of the series.

Part 1 covered extracting profile data and video lists (via HasData + TikTok endpoints). Here, we’ll use those results to download videos.

Requirements
Basic Downloader
Handling Expired Links
Notes

Requirements

You’ll need:

Python 3.8+
requests
HasData API key (if you want to refresh links)

pip install requests

Basic downloader

Simple, memory-safe way to download each video file in chunks.

import json
import requests
import os
import time

DOWNLOAD_FOLDER = "tiktok_videos"
os.makedirs(DOWNLOAD_FOLDER, exist_ok=True)

HEADERS = {
    "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36",
    "accept": "*/*",
    "referer": "https://www.tiktok.com",
}

def download_video(url, filepath):
    try:
        with requests.get(url, headers=HEADERS, stream=True, timeout=30) as r:
            r.raise_for_status()
            with open(filepath, "wb") as f:
                for chunk in r.iter_content(chunk_size=8192):
                    if chunk:
                        f.write(chunk)
        print(f"[OK] {filepath}")
        return True
    except requests.HTTPError as e:
        print(f"[HTTP {e.response.status_code}] {filepath}")
        return False
    except Exception as e:
        print(f"[ERROR] {filepath}: {e}")
        return False


# Load video list from Part 1
with open("videos_clean_full.json", "r", encoding="utf-8") as f:
    videos = json.load(f)

for video in videos:
    vid = video.get("id")
    url = video.get("downloadAddr")
    if not url:
        print(f"[WARN] No link for {vid}")
        continue

    path = os.path.join(DOWNLOAD_FOLDER, f"{vid}.mp4")
    if os.path.exists(path):
        print(f"[SKIP] {vid}")
        continue

    download_video(url, path)
    time.sleep(0.5)

This handles streaming, writes to disk in chunks, and skips existing files.

Handling expired links

TikTok download links expire fast. If you reuse old links, you’ll get 403 Forbidden. When that happens, request a fresh link using HasData.

from bs4 import BeautifulSoup
import json

def refresh_download_url(video_page_url, api_key, country="DE"):
    api_url = "https://api.hasdata.com/scrape/web"
    payload = {
        "url": video_page_url,
        "proxyType": "residential",
        "proxyCountry": country,
        "blockResources": False,
        "wait": 1000,
        "jsRendering": True
    }
    headers = {
        "Content-Type": "application/json",
        "x-api-key": api_key
    }

    try:
        r = requests.post(api_url, headers=headers, json=payload, timeout=30)
        r.raise_for_status()
        data = r.json()
        html = data.get("content")
        if not html:
            return None

        soup = BeautifulSoup(html, "html.parser")
        script = soup.find("script", id="__UNIVERSAL_DATA_FOR_REHYDRATION__")
        if not script or not script.string:
            return None

        raw = json.loads(script.string)
        items = raw.get("__DEFAULT_SCOPE__", {}).get("webapp.video-detail", {}).get("itemModule", {})
        for v in items.values():
            d = v.get("video", {}).get("downloadAddr")
            if d:
                return d
    except Exception as e:
        print(f"[ERROR] refresh: {e}")
    return None

Retry failed downloads and refresh URLs on 403.

MAX_RETRIES = 3
HASDATA_API_KEY = "YOUR-API-KEY"

for v in videos:
    vid = v.get("id")
    url = v.get("downloadAddr")
    page = f"https://www.tiktok.com/@funny_funny66066/video/{vid}"
    path = os.path.join(DOWNLOAD_FOLDER, f"{vid}.mp4")

    attempts = 0
    while attempts < MAX_RETRIES:
        ok = download_video(url, path)
        if ok:
            break
        attempts += 1
        print(f"[RETRY] {vid} ({attempts}) — refreshing URL")
        url = refresh_download_url(page, HASDATA_API_KEY) or url
        time.sleep(1)

    if not ok:
        print(f"[FAIL] {vid}")

Notes

More useful resources:

Scraping TikTok Profiles and Videos with Python (Part 1)

Valentina Skakun — Thu, 06 Nov 2025 09:38:38 +0000

TikTok is a JavaScript-heavy app, and most of the profile data never appears in the static HTML. Even worse, TikTok limits you to about 100 requests per hour per IP, which makes any large-scale scraping almost impossible.

That’s why it’s better to use a web scraping API like HasData.

It handles proxy rotation, bypasses rate limits, and renders JavaScript automatically, so you get the entire page content every time.

Requirements
Get the Page Using HasData API
Extract Profile Data
Get Videos from the Profile
Bypass Rate Limits and IP Bans
What’s Next

Requirements

You’ll need:

Python 3.8+
requests
beautifulsoup4

A free or paid HasData account to get your API key

Install the dependencies:

pip install requests beautifulsoup4

Get the Page Using HasData API

Instead of making direct requests to TikTok (which trigger rate limits), we’ll send them through HasData.

Each request is routed through a pool of residential proxies, so TikTok sees them as real users from different devices.

import requests
import json
from bs4 import BeautifulSoup

url = "https://www.tiktok.com/@funny_funny66066"

# Get your API key at https://hasdata.com/sign-up
api_key = "YOUR-API-KEY"
api_url = "https://api.hasdata.com/scrape/web"

payload = json.dumps({
    "url": url,
    "proxyType": "residential",
    "proxyCountry": "DE",
    "blockResources": False,
    "wait": 1000,
    "jsRendering": True
})

headers = {
    "Content-Type": "application/json",
    "x-api-key": api_key
}

# Send the request to HasData API
response = requests.post(api_url, headers=headers, data=payload)

# Parse the response as JSON
data = response.json()

# Extract content from the response
content_html = data.get("content")

# Parse HTML
soup = BeautifulSoup(content_html, "html.parser")

At this point, you have a fully rendered TikTok profile page, fetched through a proxy.
No 403 errors, no empty HTML, and no 100-requests-per-hour limits.

Extract Profile Data

TikTok hides most user data inside a <script> block called
__UNIVERSAL_DATA_FOR_REHYDRATION__.
It contains JSON with the entire page content: username, followers, bio, and more.

script = soup.find("script", id="__UNIVERSAL_DATA_FOR_REHYDRATION__")
if not script:
    raise SystemExit("Data script not found. TikTok structure may have changed.")

rehydration_data = json.loads(script.string)

# Extract user details
user_detail = rehydration_data.get("__DEFAULT_SCOPE__", {}).get("webapp.user-detail")

with open("tiktok_user_detail.json", "w", encoding="utf-8") as f:
    json.dump(user_detail, f, ensure_ascii=False, indent=4)

print("Saved user detail data to tiktok_user_detail.json")

This gives you structured JSON data for the user profile, ready to analyze or reuse.

Get Videos from the Profile

Once you have the secUid from the JSON above, you can query TikTok’s internal endpoint /api/post/item_list/ to get the video list.

api_base = "https://www.tiktok.com/api/post/item_list/"
secuid = user_detail["userInfo"]["user"]["secUid"]

params = {
    "aid": "1988",
    "count": "35",
    "cursor": 0,
    "device_platform": "web_pc",
    "secUid": secuid
}

videos_response = requests.get(api_base, params=params)
videos_json = videos_response.json()

with open("tiktok_videos.json", "w", encoding="utf-8") as f:
    json.dump(videos_json, f, ensure_ascii=False, indent=4)

This request will return structured data for each video: description, stats, and direct media URLs.

Bypass Rate Limits and IP Bans

Without proxies, TikTok blocks requests at about 100 per hour. HasData solves this by:

Rotating residential IPs automatically
Emulating real browser behavior
Handling headers and cookies for you

What’s Next

In Part 2, we’ll download the videos you’ve extracted and handle common errors like 403 Access Denied when a download link expires.

SERP Similarity Tool with HasData & Streamlit

Valentina Skakun — Tue, 28 Oct 2025 17:44:16 +0000

This is a submission for the 2025 Hacktoberfest Writing Challenge: Maintainer Spotlight

I needed a quick way to check how similar Google SERPs are for different keywords. Couldn’t find a simple, free tool that worked well, so I built one.

It uses HasData’s SERP API and runs on Streamlit. HasData gives you 1000 free credits when you sign up, and their API scales smoothly if you need more. The app is live, and the code is open on GitHub.

Run on Streamlit | View on GitHub

The Problem

SEO tools that actually give you a similarity breakdown of SERPs are either limited or behind a paywall. Keyword Insights worked, but it capped at five keywords and now costs money. That’s frustrating when you need quick checks for multiple keywords.

I wanted something fast and simple.

The Solution

The tool compares multiple keywords and shows how much their SERPs overlap.

Powered by HasData SERP API
Built with Streamlit
Works with any number of keywords
Free to start with HasData’s trial credits

Just enter your API key, add keywords (comma or new line), and get a SERP similarity breakdown in seconds.

How to Use

Step 1: Get a HasData API Key
Sign up at HasData and get your 1000 free credits.

Step 2: Run the App
Open the Streamlit link or clone the GitHub repo. No setup needed, just run it.

Step 3: Enter Keywords
Paste your keywords separated by commas or new lines.

Step 4: Check Results
The tool shows a SERP similarity breakdown with clear, structured data.

Under the Hood

Python 3.11+
Streamlit for UI
HasData SERP API for search data
Simple to extend with filters, charts, or exports
Deployed via Streamlit Cloud

Clone the repo, set your API key, install requirements pip install -r requirements.txt, and run streamlit run streamlit_app.py, or just use the hosted version.

Conclusion

Practical, minimal, and free to start. It does one job well: showing how similar SERPs are between keywords.

Scrape Data with Selenium in Python

Valentina Skakun — Tue, 28 Oct 2025 09:30:00 +0000

This is the third and the last post in series about scraping with Selenium in Python. In this one, we’ll focus on extracting data: locating elements, reading text, handling the Shadow DOM, and exporting your results.

Step 1: Locate Elements Using the By API
Step 2: Handle Shadow DOM and Nested Elements
Step 3: Extract Text and Attributes
Step 4: Parse Tables and Lists
Step 5: Export Data to CSV or JSON

Step 1: Locate Elements Using the By API

The old find_element_by_* methods are gone in old Selenium. Now you should use the By class.

from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Chrome()
driver.get("https://example.com")

# Examples of locators
title = driver.find_element(By.TAG_NAME, "h1")
links = driver.find_elements(By.CSS_SELECTOR, "a")

print(title.text)
print(f"Found {len(links)} links")

driver.quit()

Keep your selectors structured, By.CSS_SELECTOR is usually all you need.

Step 2: Handle Shadow DOM and Nested Elements

Some modern pages hide data inside shadow roots (common in web components). Selenium doesn’t access them directly, but you can reach them with a bit of JavaScript.

shadow_host = driver.find_element(By.CSS_SELECTOR, "custom-element")
shadow_root = driver.execute_script("return arguments[0].shadowRoot", shadow_host)
inner_button = shadow_root.find_element(By.CSS_SELECTOR, "button")
inner_button.click()

You won’t need this often, but it’s good to know when a normal locator suddenly stops working.

Step 3: Extract Text and Attributes

.text gives you the visible content of an element. For hidden or internal data, use .get_attribute().

item = driver.find_element(By.CSS_SELECTOR, ".product")
name = item.text
price = item.get_attribute("data-price")

print(name, price)

Step 4: Parse Tables and Lists

You can easily scrape structured data like tables or lists using simple loops.

rows = driver.find_elements(By.CSS_SELECTOR, "table tr")

for row in rows:
    cells = [cell.text for cell in row.find_elements(By.TAG_NAME, "td")]
    print(cells)

Or, for a list of items:

items = driver.find_elements(By.CSS_SELECTOR, ".item")
data = [i.text for i in items]
print(data)

This keeps your scraper fast and easy to debug.

Step 5: Export Data to CSV or JSON

Once you’ve collected data, save it in a structured format.

import csv, json

data = [
    {"name": "Alice", "age": "101"},
    {"name": "Bob", "age": "11"}
]

# CSV
with open("output.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=data[0].keys())
    writer.writeheader()
    writer.writerows(data)

# JSON
with open("output.json", "w", encoding="utf-8") as f:
    json.dump(data, f, indent=2)

Simple and reusable, you can plug this into any scraping script.

Final Notes

It was the last post, but here are some useful resources:

If you want any examples I might have missed, leave a comment and I’ll add them.

Selenium Navigation and Page Interaction

Valentina Skakun — Mon, 20 Oct 2025 21:00:00 +0000

This is the second post in series about scraping with Selenium in Python.
In this one, you’ll learn how to move between pages, handle tabs and pop-ups, fill forms, scroll, and even run JavaScript.

Step 1: Navigate Between Pages
Step 2: Work with Tabs and Pop-ups
Step 3: Perform Clicks, Form Fills, and Keyboard Actions
Step 4: Scroll and Load More Content
Step 5: Execute JavaScript

Step 1: Navigate Between Pages

You can control the browser flow with a few simple commands.

driver.get("https://example.com")     # Open a page
driver.refresh()                      # Reload it
driver.back()                         # Go to the previous page
driver.forward()                      # Move forward again

That’s all you need to move around.

Step 2: Work with Tabs and Pop-ups

Modern websites love to open new tabs and alerts. You can switch between them:

# Get all open window handles
windows = driver.window_handles  

# Switch to the second tab
driver.switch_to.window(windows[1])

# Close it and go back to the first one
driver.close()
driver.switch_to.window(windows[0])

To handle alerts:

alert = driver.switch_to.alert
print(alert.text)
alert.accept()  # or alert.dismiss()

Step 3: Perform Clicks, Form Fills, and Keyboard Actions

The basics of interaction: clicks, inputs, and keyboard actions.

from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys

driver.find_element(By.ID, "search").send_keys("Selenium" + Keys.ENTER)
driver.find_element(By.CSS_SELECTOR, ".submit-btn").click()

Keep your locators simple, CSS selectors are usually enough.

Step 4: Scroll and Load More Content

Dynamic pages often load content as you scroll. Here’s a quick way to simulate user scrolling:

import time

last_height = driver.execute_script("return document.body.scrollHeight")

while True:
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    time.sleep(2)  # Wait for new content to load
    new_height = driver.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        break
    last_height = new_height

This pattern works great for infinite scroll pages.

Step 5: Execute JavaScript

In Selenium 4.36+, Chrome and Edge use the new BiDi (Bidirectional) protocol.
You can still run custom JavaScript like this:

driver.execute_script("console.log('Running JS from Python');")
result = driver.execute_script("return document.title;")

BiDi also lets you capture logs and network data.

Final Notes

In the next article, we’ll cover locating elements, reading text, handling the Shadow DOM, and exporting your results.
Meanwhile, here are some useful resources:

The Complete Guide to Web Scraping with Selenium in Python
Join our Discord
Selenium Scraping Examples in Python and NodeJS (GitHub) If you want any examples I might have missed, leave a comment and I’ll add them.

DEV Community: Valentina Skakun

Building a Production-Ready Job Board Scraper with Python

Table of Contents

The Universal AI Job Scraper

Job Data Pipeline Architecture

The Platform Specific Strategy

Identifying the Underlying ATS

Direct ATS Integration Patterns

Greenhouse JSON Endpoints

Lever Hidden API Endpoints

Handling Ashby and BambooHR

Handling Complex Enterprise ATS

Playwright Network Interception

Handling Workday CSRF and Sessions

Simulating Human Inputs for Bot Defense

Bypassing Aggregator Defenses

Google SERP Discovery Strategy

Unwinding Redirect Chains

LinkedIn and Indeed DOM Strategies

Job Data Normalization

Standardizing Salary Ranges via Regex

Extracting Tech Stacks with FlashText

Detecting Expired Listings

Scaling and Infrastructure

Residential Proxy Rotation Logic

TLS Fingerprint Management

Conclusion

We Benchmarked CAPTCHA Solving Services to Find the Best One

Why This Matters

Quick Setup

Example 1: Image CAPTCHA (2Captcha)

Example 2: Cloudflare Turnstile with Browser Automation (Anti-Captcha)

Example 3: Building Your Own Integration (SolveCaptcha)

Performance Results

Practical Tips

1. Python 3.13 Compatibility

2. Proper Error Handling

3. Rate Limiting

4. Audio CAPTCHA Support

5. Cost Optimization

Decision Matrix

Complete Benchmark

Run It Yourself

Notes

Automating SEO with Python and Scraping API

Table of Contents

Code Snippets and Resources

Build Instead of Buy

Recursive Keyword Discovery

Semantic Intent Mapping

SERP Similarity Heatmaps

AI Overview Visibility Tracking

Conclusion

Use These Python Patterns for Price Scraping

Intercepting Internal APIs

The Selector Priority Waterfall

Strict Data Normalization

Notes

Scraping ZoomInfo with One Universal Script Using SeleniumBase

Table of Contents

Before You Start

Prerequisites and Setup

Available Data Points

Universal Scraping Script

Anti-Scraping Measures

Notes

Analyzing News Sentiment with VADER

Table of Contents

Introduction

Setup

Fetching Google News Data

Sentiment Analysis with VADER

Saving and Visualizing Results

Full Code

Next Steps

Further Reading

Building a Topic Frequency Chart from Google News Headlines

Table of Contents

Introduction

Setup