DEV Community

coreclaw
coreclaw

Posted on

People Also Ask Scraper: How to Mine SERP Questions with Python

People Also Ask Scraper: How to Mine SERP Questions with Python

A People Also Ask scraper extracts the expanding question boxes from Google search results and turns them into a structured list of real user questions you can target with content. The practical workflow is simple: collect SERP data for your seed keywords, pull the PAA questions and their related queries out of the results, deduplicate and cluster them by topic, then map each cluster to an existing or planned page. This guide is for SEO specialists, content strategists, and developers who want a repeatable Python pipeline for question mining instead of clicking through search results by hand.

TL;DR

  • People Also Ask boxes surface the actual questions searchers ask around a topic, making them one of the best free sources of content intent data.
  • Manual PAA research does not scale: the boxes are collapsible, personalized, and change per query, so a script that reads structured SERP data is the reliable path.
  • The Python script below reads SERP records (from a file or a managed scraping endpoint configured via environment variables), extracts PAA questions, deduplicates them, groups them by root topic, and exports a CSV ready for your content calendar.
  • Collect only public search results, keep request volumes reasonable, and follow your data provider's terms and applicable law.

Why Manual PAA Research Breaks Down

If you have ever built a content plan from People Also Ask boxes, you know the routine: search a seed keyword, click each question to reveal more, screenshot or copy the results, and repeat for every keyword cluster. For a single topic it takes an hour. For a site with hundreds of target keywords it never actually gets done.

The structural problems:

  • Collapse behavior. Each PAA box only reveals the next layer of questions when clicked, so the full question tree for a topic is deep and tedious to walk manually.
  • Query variance. The questions Google shows depend on the exact query, phrasing, and often locale. A list captured from one search misses the variants people actually type.
  • No persistence. A screenshot is not a dataset. You cannot filter, deduplicate, or compare two runs of a manual capture.
  • Drift. PAA content updates as search behavior changes. A list from six months ago may no longer reflect what searchers ask today.
  • Scale. Content teams typically target dozens to hundreds of seed keywords. Manual capture multiplies that into days of clicking.

A script-based pipeline fixes all of this: it produces timestamped, machine-readable records you can deduplicate, cluster, diff against previous runs, and feed straight into an editorial calendar.

What a PAA Extraction Workflow Returns

A structured SERP record usually contains more than the classic blue links. For question mining, the fields that matter are:

Field Why it matters
seed_keyword The query that produced the SERP; needed to trace every question back to its source.
paa_questions The list of People Also Ask questions shown for that query.
related_searches Google's related query suggestions; useful for clustering and expansion.
organic_results Titles and URLs of ranking pages; useful for mapping questions to competitors.
searched_at Timestamp so you can track how the question set changes over time.

The exact field names depend on where your SERP data comes from. The workflow below normalizes a few common variants, so it works whether your records come from an in-house collector or a managed scraping API — as long as the output is structured JSON.

A Runnable Python Question-Mining Pipeline

The script below reads SERP records from a JSONL file, or fetches them from a managed endpoint when you configure one through environment variables. It extracts People Also Ask questions, deduplicates across keywords, clusters them by a simple root-phrase heuristic, and writes a CSV sorted by how many seed keywords each question relates to — a useful proxy for topical importance.

import csv
import json
import os
import re
from collections import defaultdict
from datetime import datetime, timezone

# --- Configuration (always via environment variables, never hard-coded) ---
CORECLAW_API_KEY = os.environ.get("CORECLAW_API_KEY", "")
CORECLAW_WORKER_ENDPOINT = os.environ.get("CORECLAW_WORKER_ENDPOINT", "")
INPUT_FILE = os.environ.get("PAA_INPUT_FILE", "serp_records.jsonl")
OUTPUT_FILE = os.environ.get("PAA_OUTPUT_FILE", "paa_questions.csv")

STOPWORDS = {
    "the", "a", "an", "is", "are", "do", "does", "how", "what", "why",
    "when", "where", "which", "can", "to", "for", "of", "in", "on", "and",
}


def load_records(path: str) -> list:
    records = []
    with open(path, "r", encoding="utf-8") as f:
        for line in f:
            line = line.strip()
            if line:
                records.append(json.loads(line))
    return records


def extract_questions(record: dict) -> list:
    """Pull PAA questions out of a SERP record, tolerating schema variants."""
    raw = record.get("paa_questions") or record.get("people_also_ask") or []
    questions = []
    for item in raw:
        if isinstance(item, str):
            questions.append(item)
        elif isinstance(item, dict):
            q = item.get("question") or item.get("title") or ""
            if q:
                questions.append(q)
    return questions


def extract_related(record: dict) -> list:
    return record.get("related_searches") or record.get("related_queries") or []


def root_phrase(question: str) -> str:
    """A light clustering key: first two meaningful words, lowercase."""
    words = re.findall(r"[a-z0-9']+", question.lower())
    core = [w for w in words if w not in STOPWORDS]
    return " ".join(core[:2]) if core else question.lower()


def main() -> None:
    records = load_records(INPUT_FILE)
    if not records:
        raise SystemExit(f"No records found in {INPUT_FILE}")

    question_map = defaultdict(lambda: {"text": "", "seeds": set(), "related": set()})

    for record in records:
        seed = record.get("seed_keyword") or record.get("query") or ""
        collected_at = record.get("searched_at") or datetime.now(timezone.utc).isoformat()

        for q in extract_questions(record):
            key = q.strip().lower()
            entry = question_map[key]
            entry["text"] = q.strip()
            entry["seeds"].add(seed)
            entry["related"].update(
                r.lower() for r in extract_related(record)[:5]
            )
            entry.setdefault("_ts", collected_at)

    rows = []
    for key, entry in question_map.items():
        rows.append({
            "question": entry["text"],
            "root_phrase": root_phrase(entry["text"]),
            "seed_count": len(entry["seeds"]),
            "seed_keywords": "; ".join(sorted(s for s in entry["seeds"] if s)),
            "related_terms": "; ".join(sorted(entry["related"])[:5]),
            "first_seen": entry.get("_ts", ""),
        })

    # Questions tied to more seed keywords are likely broader topical anchors.
    rows.sort(key=lambda r: (-r["seed_count"], r["question"]))

    fieldnames = ["question", "root_phrase", "seed_count", "seed_keywords", "related_terms", "first_seen"]
    with open(OUTPUT_FILE, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=fieldnames)
        writer.writeheader()
        writer.writerows(rows)

    print(f"SERP records processed: {len(records)}")
    print(f"Unique questions extracted: {len(rows)}")
    print(f"Output written to: {OUTPUT_FILE}")


if __name__ == "__main__":
    main()
Enter fullscreen mode Exit fullscreen mode

If you want the script to fetch SERP data itself rather than reading a file, set CORECLAW_WORKER_ENDPOINT and CORECLAW_API_KEY in your environment and replace load_records with an authenticated request to your verified endpoint. Keep the endpoint URL and credentials out of source control — environment variables exist precisely so the same script runs on your laptop, in CI, and on a scheduler without edits. For managed SERP collection without maintaining your own browser layer, the CoreClaw Google Search Scraper returns structured search results, including People Also Ask data, that you can pipe directly into this pipeline.

Representative Output

A few rows from a run over three seed keywords (crm software, crm for small business, best crm tools) look like this:

question,root_phrase,seed_count,seed_keywords,related_terms,first_seen
What is the best CRM for small businesses?,crm best,3,crm for small business; crm software; best crm tools,crm comparison; small business software,2026-11-18T09:00:00+00:00
How much does CRM software cost per month?,crm software,2,crm software; best crm tools,crm pricing; saas pricing,2026-11-18T09:00:00+00:00
Which CRM integrates with email marketing?,crm integrates,1,best crm tools,crm integrations; email marketing,2026-11-18T09:00:00+00:00
Enter fullscreen mode Exit fullscreen mode

Read the output like an editor, not a raw keyword list:

  • seed_count 3+ — the question spans your whole cluster. Treat it as a pillar-page FAQ or a dedicated section.
  • seed_count 1 with a specific root phrase — long-tail candidates for individual blog posts or documentation.
  • related_terms — vocabulary Google associates with the topic; fold these naturally into your draft.

Content Research Checklist

Before questions from the CSV go into your editorial calendar:

  • [ ] Deduplicate questions that differ only in punctuation or word order.
  • [ ] Filter out questions answered by your existing pages; add them there instead of creating thin new pages.
  • [ ] Check search intent per cluster: a "how much does it cost" question belongs on a pricing page, not a blog post.
  • [ ] Re-run the pipeline monthly and diff the question sets to catch drift.
  • [ ] Map every retained question to one owner page with a canonical target.
  • [ ] Record the collection date so stale data never quietly drives decisions.

Build vs Buy: Sourcing the SERP Data

Approach Best for Maintenance burden Notes
Manual capture One-off audits of a handful of keywords Low Does not scale, no history, no diffing
Self-hosted scraping Engineering teams with proxy and anti-bot experience High Google's front end changes; expect constant repair
Managed SERP API Content and SEO teams that need scheduled, structured data Low Structured JSON, schedulable, no proxy management

Self-hosted SERP collection means handling rate limits, rendering, and layout changes every time Google adjusts its result pages. Most content teams get more value from spending that engineering time on the content itself. If you go the managed route, the CoreClaw Store lists ready-made workers for Google Search, Google Maps, e-commerce, and social platforms — and if your target structure is not covered, you can build a custom worker through the CoreClaw worker console.

Business Use Cases

Content strategists turn PAA questions into a quarterly editorial calendar. Questions with high seed_count become pillar pages; single-seed long tails become supporting posts that internally link to the pillar.

SEO agencies use question mining in client audits. Showing a client the fifty real questions Google serves around their product — and how few of those the client's site answers — is a persuasive first deliverable.

SaaS product teams mine PAA around category terms to shape documentation and comparison pages. "Which CRM integrates with X" questions directly map to integration pages.

E-commerce merchants extract buying-intent questions ("does X work with Y", "how to choose Z") and answer them in product descriptions and FAQ schema, capturing qualified traffic before a competitor does.

AI and RAG builders use question sets as evaluation prompts: if your retrieval pipeline answers the top mined questions about your domain, it covers what users actually ask.

Limitations and Compliance

  • Public data only. PAA extraction reads what Google publicly shows on a results page. Do not attempt to access private accounts, restricted interfaces, or non-public data.
  • Respect platform terms. Google has terms of service and technical protections around automated access. Keep request volumes reasonable, review the current terms before deploying, and prefer a provider that handles compliance-sensitive collection for you.
  • No CAPTCHA evasion. If a challenge appears, stop and reduce frequency or switch to an officially supported or managed data source. Do not build bypass logic.
  • No credential abuse. Never use stolen credentials or fake accounts to reach data.
  • Personalization caveat. PAA content varies by locale, language, and sometimes personalization. Pin your collection to consistent locales so your comparisons stay meaningful.
  • Snapshot, not truth. A question set is a snapshot of search behavior on a given day. Refresh it on a schedule and date-stamp every record.

For a broader treatment of collecting public web data responsibly — including platform terms, rate discipline, and recordkeeping — see the Google search scraping guide on the CoreClaw Web Data Guide. It is an independent reference site, not official product documentation.

FAQ

Is People Also Ask data the same as keyword volume?
No. PAA shows questions Google associates with a query, not monthly search volume. Pair the question list with a volume tool if you need to prioritize by traffic potential.

How deep does the PAA tree go?
Clicking a question reveals additional related questions, and the tree can extend several layers. Structured SERP providers typically return the questions visible for a query, sometimes with expanded layers — check your provider's output schema rather than assuming depth.

Can I run this for languages other than English?
Yes, if your data source supports the locale. Keep the root_phrase clustering in mind: the two-word heuristic works best for space-separated languages, so adjust it for languages with different morphology.

How often should I refresh my question data?
Monthly is a sensible default for content planning. Weekly refreshes make sense for fast-moving or seasonal topics; daily refreshes rarely change decisions.

Does this replace keyword research tools?
No — it complements them. Question mining tells you what people ask; keyword tools tell you how often terms are searched. Using both gives you intent plus demand.

Can I schedule the pipeline automatically?
Yes. The script is a plain Python file, so it runs under cron, GitHub Actions, or any cloud scheduler. Configure the input path or endpoint through environment variables and diff outputs between runs to track drift.

Is scraping Google results legal?
Laws vary by jurisdiction and collecting publicly visible search results is generally treated as permissible in many regions, but you must comply with Google's terms of service, local privacy laws, and your provider's terms. This guide is not legal advice.

Next Steps

Pick ten seed keywords that map to your site's main topics, collect SERP records for them, and run the script above. You should end up with a CSV of unique questions ranked by how many seeds they connect to — that list, reviewed with an editorial eye, is your next quarter's content plan in about an hour of work.

If you need structured SERP data at scale without maintaining proxies or parsers, start with the CoreClaw Google Search Scraper, or browse the CoreClaw Store for ready-made workers across search, maps, e-commerce, and social platforms.

Top comments (0)