DEV Community: Ethan Walker

Separate Pending Tasks from Empty Google Search Results

Ethan Walker — Mon, 14 Sep 2026 10:35:03 +0000

An empty result list is easy to render. It is also easy to manufacture accidentally: read a field that is not in a pending response, substitute an empty array, and the application now appears to know that the search returned nothing.

This tutorial puts the collection outcome ahead of the organic-result projection. It uses the current Scrapeless Google Search API request workflow and a local Python classifier to keep pending work, failures, malformed data, and completed empty arrays distinct. The program reads a saved capture and emits a small quality record; it does not send a request or retrieve a task.

Start With the API's Documented Outcome

The documented request goes to POST https://api.scrapeless.com/api/v1/scraper/request, uses x-api-token for authentication, and submits actor scraper.google.search with search settings in input. Google Search API supplies the upstream search data; the classifier below is application code around a saved response.

The quickstart describes HTTP 200 as task data and HTTP 201 as a pending task. Its pending example contains a message and taskId. The classifier follows that documented meaning instead of treating every successful HTTP response as a completed organic-result collection.

HTTP 400, 429, and 500 are also described in the request documentation. Preserve the actual status and response for diagnosis. Do not make any HTTP error equivalent to a completed search with no organic items.

A missing HTTP status represents a transport outcome in the capture model used here. That is a collector convention, not a new API response field. Your collector should keep the underlying error separately so the quality record can point to evidence without exposing a credential.

Preserve a Capture Before Transforming It

The input is a JSON object with request, http_status, response, run_id, and received_at. These are outer fields owned by the collector. Keep the submitted request intact, including the search context described by the Google Search parameters.

Do not store authentication headers in a shareable capture. The request body provides the relevant query settings without copying the account key. Keep the original response in the internal evidence record so a mapping problem can be inspected later.

The local program needs Python and a saved capture. It uses only standard-library modules and prints JSON without altering the input. Producing a live capture needs your account key and an authenticated collection step; neither that collection nor task retrieval was performed for this article.

The tests for this classifier use synthetic inputs. They establish how the local rules handle each state, not the service's search coverage or the result of a live query.

Give Unknown Counts a Different Value From Zero

The output uses organic_count=null until a completed, structurally valid organic array is available. Zero is reserved for a present empty array. That policy prevents a dashboard from treating incomplete evidence as a measured absence.

A missing organic_results key is unmapped. A null value, a non-array value, or an array containing a non-object item also becomes unmapped, with a specific quality note. Those rules make the expected projection explicit.

An array of objects can still contain optional fields that need later validation. The classifier establishes the container shape, not the completeness of every title, link, or snippet. A separate projection can apply stricter item-level checks while preserving this collection record.

The JSON value model distinguishes null, arrays, and objects. Retaining that information is more useful than coercing every unsupported value into an empty list.

Run the Local State Classifier

Save this program as capture_state.py and run python3 capture_state.py capture.json. Python's JSON module parses the capture and serializes the derived record. The program sends no network requests and writes only to standard output.

import argparse
import json
from pathlib import Path


def classify(record):
    if not isinstance(record, dict):
        raise ValueError('Capture must be an object')
    status, payload = record.get('http_status'), record.get('response')
    output = {'run_id': record.get('run_id'), 'request': record.get('request'),
              'received_at': record.get('received_at'), 'http_status': status,
              'organic_count': None, 'task_id': None, 'quality_notes': []}
    if status is None:
        state = 'transport_error'
    elif status == 201:
        state = 'pending'
        task = payload.get('taskId') if isinstance(payload, dict) else None
        if isinstance(task, str) and task.strip():
            output['task_id'] = task
        else:
            output['quality_notes'].append('pending_without_usable_task_id')
    elif status != 200:
        state = 'http_error'
    elif not isinstance(payload, dict):
        state = 'unmapped'
        output['quality_notes'].append('response_not_object')
    elif 'organic_results' not in payload:
        state = 'unmapped'
        output['quality_notes'].append('organic_field_missing')
    elif not isinstance(payload['organic_results'], list):
        state = 'unmapped'
        output['quality_notes'].append('organic_field_not_array')
    elif any(not isinstance(row, dict) for row in payload['organic_results']):
        state = 'unmapped'
        output['quality_notes'].append('organic_item_not_object')
    else:
        count = len(payload['organic_results'])
        state = 'observed' if count else 'empty'
        output['organic_count'] = count
    return dict(output, state=state)


if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('capture')
    args = parser.parse_args()
    record = json.loads(Path(args.capture).read_text(encoding='utf-8'))
    print(json.dumps(classify(record), ensure_ascii=False, indent=2))

A pending record always remains pending, even if its task identifier is missing or unusable. The quality note then identifies a problem that needs inspection. This avoids changing a pending task into a different business outcome just because its metadata is incomplete.

A malformed capture that is not an object raises an error. Invalid JSON also fails during parsing. Those failures should be visible in the process result rather than converted into successful quality records with zero items.

Store the Task Identifier Without Inventing a Retrieval Route

When HTTP 201 includes a usable string taskId, retain it alongside the run identifier and original request. The task identifier belongs to the service workflow; the run identifier links your own collection and review records. They have different roles and should remain separate.

The quickstart says to obtain the task result through its identifier, but the referenced getting-started page does not establish a complete retrieval endpoint and response contract. This example therefore stops at persistence and classification. It does not append a task identifier to an assumed route or claim an end-to-end completion loop.

Before implementing retrieval, verify the relevant current documentation for your account and run the actual operation. Confirm the final response shape and how the completion record relates to the original task. Until that prerequisite is met, the task remains outside completed-result analysis.

When completion is confirmed through that verified workflow, save the final response as another evidence record linked to the original run and task. Keep the earlier pending observation. Updating a derived current-state view should not erase the sequence that explains what happened.

Keep Collection Coverage Separate From Search Findings

An operational view should report planned jobs and their states. A search-analysis view should use only observations suitable for its question. Combining the views without explicit filters can make unavailable collection look like a change in the search landscape.

For example, a domain-presence comparison cannot use a pending task as evidence that a domain disappeared. A present empty organic array is a different observation, but even that describes only the returned slice under the recorded query context. It does not establish that the entire web has no relevant pages.

Keep unmapped records visible to the owner of the adapter. A schema issue may affect many queries at once and should not be mistaken for a market trend. The quality notes provide a concrete inspection target without making a claim about the cause.

If you calculate a coverage rate, define which states count as completed and explain the denominator. If you calculate an organic-result statistic, state the additional mapping requirements. Different reports can have different eligibility rules, but those rules should be documented rather than buried in a default empty-list expression.

Test the Decisions That Change the Meaning of a Report

Use local fixtures for a populated organic array, a present empty array, a missing key, a null field, a wrongly typed field, and an array with an invalid item. Test HTTP 201 both with and without a usable task identifier, plus HTTP errors and a missing status.

Assert that only a valid array receives a numeric count. Confirm that pending tasks retain their identity, that errors preserve their HTTP status, and that malformed organic structures generate a quality note. These checks target the classification choices that affect downstream conclusions.

Also run the actual command against a saved synthetic capture and parse its output as JSON. A function-level assertion alone does not check the input-file and command-line handoff. Once you have an account capture, inspect it separately before treating the adapter as a production mapping.

Keep the raw capture reference, classifier version, and derived result together. The provenance model provides a useful distinction between evidence and the activity that produced an interpretation. A revised classifier can then be evaluated against the same input without rewriting history.

Conclusion

Classify the HTTP outcome first, validate the organic container second, and reserve zero for a present empty array. Persist pending task identifiers and make unverified completion work an explicit prerequisite. This gives both operational dashboards and search analysis a record whose meaning survives the handoff.

FAQ

Is HTTP 201 an empty search result?

No. The documented workflow uses it for a pending task. Preserve the task identifier and keep the result count unknown until completion is verified.

Why is a missing organic field unmapped?

The classifier cannot establish the expected organic projection from that response. Substituting an empty array would conceal the difference between unavailable structure and an observed empty array.

Does a populated array guarantee usable links?

No. This classifier checks that the array contains objects. A later item-level adapter must validate fields needed for its own analysis.

Does this code retrieve a pending task?

No. It saves the usable identifier in its output. Retrieval requires a separately verified endpoint, authentication, and response contract.

Were the tests run against live Google results?

No. The local tests use clearly synthetic captures. Account collection and final task retrieval remain prerequisites for live deployment.

Add Source Discovery to an AI Research Workflow

Ethan Walker — Mon, 14 Sep 2026 10:21:33 +0000

TL;DR:

Search gives an AI agent candidates to investigate. A URL and its snippet are not yet a verified source for an answer.
Make the handoff inspectable. Retain query context, candidate identity, selection reasoning, and retrieval outcomes so each citation can reach back to reviewed text.
Test a local adapter first. This program converts saved captures into a review queue; live collection and page retrieval need separate setup and verification.

A research answer can include citations without showing whether the linked pages support its sentences. Search access does not automatically create that missing connection. The application needs to represent discovery, retrieval, and evidence review as distinct stages with explicit outputs.

Scrapeless Google Search API supplies the discovery input. The workflow below uses a Google Search API for AI agents to create source candidates while preserving the boundary between an observed snippet and material suitable for a factual answer. The adapter gives the next reviewer enough information to understand where a candidate came from.

Define the Discovery Contract

Keep the research question with the job and the exact query with the submitted request. The question is application metadata. A project can explore it through several searches while preserving each search's individual context.

Return candidate records that include the destination, observed text, original order, and review status. The list is not an answer. Subsequent reading may establish that a candidate is useful, irrelevant, inaccessible, or outdated for the question.

Require the downstream answer stage to use retrieved, reviewed content for factual claims. A discovery-only record can guide additional research but should not appear automatically in the accepted evidence set. That rule exposes missing work instead of letting the answer generator replace it with a plausible explanation.

Prerequisites and Request Parameters

You need Python and a saved JSON capture to run the local transform. No third-party package is required. The example expects a collector envelope with request, http_status, response, run_id, and received_at. These outer fields are defined by the application, not presented as the API's native response wrapper.

Authenticated collection requires your account key. The Google Search request workflow specifies POST https://api.scrapeless.com/api/v1/scraper/request, authentication through x-api-token, and actor scraper.google.search. Search settings go in input.

Read the Google Search parameters when building requests. Query wording, language, and country define the observation context. In full-URL mode, other input parameters are ignored, so retain the supplied URL instead of reconstructing settings from assumptions.

Note: This article does not report a live API request or a page-retrieval run. The code performs a local transform verified with synthetic captures. Before collecting account data or completing a pending task, confirm the current documented workflow and inspect its actual response.

Preserve the Response State Before Reading Results

HTTP 200 contains task data, whereas HTTP 201 means the task is pending. Save taskId when it is supplied. Keep that state until a separately verified completion procedure provides the result. This program contains no assumed task-retrieval route.

For completed data, validate the organic_results array before projecting candidates. An absent field or incompatible shape becomes unmapped. A valid array with no items becomes empty. Those distinctions let the next stage respond to the actual collection outcome.

Preserve the raw capture even after generating the smaller queue. The JSON value model retains nulls and nested structures that might disappear from a flattened projection. It also gives you the evidence needed to revise a mapper later.

Build the Local Candidate Adapter

Save the program as source_candidates.py, then run python3 source_candidates.py capture.json with a saved capture. The output is derived JSON on standard output. The input remains unchanged, and the program makes no search request, page request, or model call.

import argparse
import json
from pathlib import Path
from urllib.parse import urlsplit


def candidates(record):
    if not isinstance(record, dict):
        raise ValueError('Capture must be an object')
    status = record.get('http_status')
    payload = record.get('response')
    base = {'run_id': record.get('run_id'), 'request': record.get('request'),
            'received_at': record.get('received_at'), 'candidates': []}
    if status == 201:
        task = payload.get('taskId') if isinstance(payload, dict) else None
        return dict(base, state='pending', task_id=task)
    if status != 200:
        return dict(base, state='transport_error' if status is None else 'http_error')
    rows = payload.get('organic_results') if isinstance(payload, dict) else None
    if not isinstance(rows, list) or any(not isinstance(row, dict) for row in rows):
        return dict(base, state='unmapped')
    output, seen = [], set()
    for ordinal, row in enumerate(rows):
        link = row.get('link')
        reason, host = None, None
        try:
            parsed = urlsplit(link) if isinstance(link, str) else None
            if (parsed is None or parsed.scheme not in ('http', 'https')
                    or not parsed.hostname or parsed.username or parsed.password):
                reason = 'invalid_web_url'
            else:
                host = parsed.hostname.lower()
        except ValueError:
            reason = 'invalid_web_url'
        if reason is None and link in seen:
            reason = 'duplicate_exact_url'
        if reason is None:
            seen.add(link)
        output.append({'candidate_id': f'source-{ordinal}', 'ordinal': ordinal,
                       'position': row.get('position'), 'title': row.get('title'),
                       'url': link, 'hostname': host, 'snippet': row.get('snippet'),
                       'review_state': 'excluded' if reason else 'needs_review',
                       'exclusion_reason': reason, 'evidence_state': 'discovery_only'})
    return dict(base, state='observed' if rows else 'empty', candidates=output)


if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('capture')
    args = parser.parse_args()
    result = candidates(json.loads(Path(args.capture).read_text(encoding='utf-8')))
    print(json.dumps(result, ensure_ascii=False, indent=2))

Pair a candidate identifier with run_id, because the identifier alone is local to that run. Exact duplicate URLs stay in the output as excluded records with a reason. Keeping them visible explains the queue's decisions. Other variants remain separate until reviewed.

The URL component parser helps reject unsupported schemes, absent hosts, and embedded credentials. That check establishes basic input shape only. A later fetcher must enforce destination rules for resolved addresses and redirect targets before using the URL in a network operation.

Select Sources and Retrieve Content Separately

Assess each eligible candidate against the research question and record why it was selected or excluded. Choose material that can establish the needed fact. Search position remains an observation about the result, not a source-quality rating.

Pass a selected URL to a separate retrieval operation that records the requested address, final destination, retrieval time, stored content reference, and outcome. If the destination cannot be read, preserve that failure. Using its snippet in place of a missing body would change the evidence standard without telling the reader.

Google's explanation of search snippets makes their limited role clear. The excerpt can depend on the query and may omit the passage relevant to your claim. Read the retrieved source before using it to substantiate an answer.

Keep retrieved text in the data boundary. Instructions embedded in a page do not authorize new tools or change the research assignment. The application should maintain that separation when presenting source content to an assistant.

Connect Claims to Reviewed Passages

Create a citation record that links the proposed statement to a supporting passage and its retrieval record. Retain the candidate ID for provenance, but store the passage location separately. A bare destination URL does not establish the relationship between source and claim.

Inspect the limits of the source's statement. A fact about one version or market cannot automatically support a general claim about all configurations. If sources disagree, preserve the disagreement for review or qualify the answer; search order does not resolve the conflict.

The provenance model distinguishes evidence, processing activities, and responsibility. Your records can be simpler than a formal provenance system while still preserving those relationships for someone reviewing the answer.

If reviewed content does not support a claim, omit it or make the evidence gap explicit. A better discovery process helps organize research but cannot promise that an answer generator will never produce unsupported text.

Check the Adapter Before Connecting an Agent

Exercise the transform with populated and empty arrays, missing fields, malformed items, HTTP 201, and HTTP errors. Add an exact duplicate and an invalid URL scheme. Those local fixtures establish how the application handles each case, not what the service currently returns for all searches.

Check that exclusions retain their original observed fields and every candidate is marked discovery_only. Before deployment, inspect an account capture and compare its shape with the mapping. Preserve the original response if the adapter needs revision.

The boundary does not depend on an agent framework. A caller that consumes this JSON contract can use the queue, but a particular SDK or tool protocol still needs its own connection test. No such framework handshake is claimed by this local example.

Conclusion

Preserve the request, classify its outcome, and return candidates with explicit review states. That small discovery contract gives retrieval and citation checking a usable handoff, while keeping an unexplained list of links out of the evidence stage.

An editorial workflow can use the same discipline for content gap analysis when source review is needed before assigning an article.

FAQ

Q: Does the adapter retrieve full page text?

No. It transforms stored search observations into a candidate queue. Page retrieval needs a separate operation and its own evidence record.

Q: Can the first organic result be cited automatically?

No. Organic position does not establish support for a statement. Inspect the retrieved passage before accepting the citation.

Q: What happens to HTTP 201?

It remains pending, with a task identifier retained when available. The adapter neither retrieves that task nor classifies it as an empty result.

Q: Does URL parsing make fetching a candidate safe?

No. Shape validation is limited. A network fetcher also needs destination controls covering resolved addresses and redirects.

Q: Does this require a specific agent framework?

No. The example uses a local JSON contract. Live account collection and a chosen framework integration require their own verification.

Keep Search Context When Exporting SERP JSON to CSV

Ethan Walker — Fri, 11 Sep 2026 06:30:59 +0000

A CSV export can look clean while losing the information that makes a search result interpretable. A title and URL do not tell the next analyst which query, market, or observation produced the row. A blank cell also cannot explain whether a field was absent, explicitly null, or invalid for the expected type.

This tutorial exports the organic array from a saved Google SERP capture while keeping request context on every row. It also writes a run metadata file so an empty export remains distinguishable from pending or failed collection.

Scrapeless Google Search API is the upstream source. The exporter is a local Python transform; it does not send API requests, retrieve pending tasks, or measure Google results itself.

Start With a Capture Envelope

The input file should contain the original request, the recorded http_status, and the API response, together with a run identifier and client timestamps when available. These outer fields belong to your collector, not to the API response schema.

The Google Search request workflow returns task data with HTTP 200 and a pending task with HTTP 201. Preserve that distinction before exporting. Reading organic_results from every JSON body without checking the outcome can turn a pending task into a misleading empty spreadsheet.

Keep the original capture file. CSV is a projection for analysis, not a lossless replacement for nested JSON. The exporter retains the request on each row and points back to the original file in its sidecar, but it does not copy every response module into the spreadsheet.

No API key is needed for the transform. Producing a live capture requires your own authenticated collection step; this article does not claim such a run. The local checks use synthetic records that deliberately contain nulls, missing fields, and spreadsheet-sensitive strings.

Decide Which Meaning Each Column Preserves

The exported context contains the run identifier, requested and received times, exact query when parameter mode supplies it, and the serialized request. The complete request protects settings that a short fixed list of columns might overlook.

For example, country and language inputs belong to the search context. Full-URL mode may carry settings inside url rather than separate fields. The q column can therefore be blank while request_json still preserves the submitted URL. Do not reconstruct an effective query by guessing from missing columns.

Each organic item gets an array ordinal and the returned position. Ordinal is zero-based source order. Position is only retained as a number if the value is a positive integer; a Boolean is not accepted as a position despite Python's Boolean/integer relationship.

The title, link, snippet, and position also receive state columns. missing means the key was absent, null means it was explicitly null, and value means the expected value was retained. An invalid position gets invalid; an unexpected type in a text field is serialized as JSON and marked unexpected_type.

Use a CSV Writer, Not String Concatenation

A field can contain commas, quotation marks, or line breaks. Python's CSV writer handles those values according to its dialect instead of relying on a manual join operation.

Open the output with newline="" so the CSV module handles record boundaries. The program uses utf-8-sig to include a UTF-8 signature, which can help spreadsheet applications recognize Unicode text. Import behavior still depends on the application and its settings.

CSV quoting solves delimiter handling. It does not by itself tell a spreadsheet to treat a cell as inert text. That is a separate concern when titles, snippets, or even queries come from untrusted data.

Make Spreadsheet Interpretation Explicit

Some spreadsheet applications interpret values beginning with characters such as =, +, -, or @, including full-width variants in some locales as formulas. Leading control characters and whitespace can complicate that behavior. The CSV injection guidance explains why an otherwise valid CSV can still be interpreted in an unintended way.

The exporter prefixes an apostrophe when a text value begins with a relevant formula character after whitespace, or begins with a tab or line break. It applies this policy to context fields as well as result text. Numeric positions are validated separately.

This is a documented spreadsheet-oriented transform, not a universal security guarantee across every application and import setting. Test the file in the spreadsheet application your team uses, import relevant columns as text, and keep the original JSON for exact values. Saving and reopening a file can change how escape prefixes are handled; include that path in the spreadsheet import test. The prefix may be visible in some viewers and intentionally changes the exported cell representation.

Do not remove that prefix later merely to make a screenshot cleaner without considering how the file will be opened. If another program needs the exact original string, read the archived JSON or use an explicitly defined machine-data export.

Run the Local Exporter

Save the program as serp_csv.py and run python3 serp_csv.py capture.json organic.csv, replacing the input filename with a saved capture. It uses only the Python standard library and writes organic.csv plus organic.csv.run.json.

The output files are replaced if they already exist. The program refuses output paths that would overwrite the input capture. Use a dedicated export directory or unique filenames when retaining several versions.

import argparse
import csv
import json
from pathlib import Path

FIELDS = ["run_id", "requested_at", "received_at", "q", "request_json", "ordinal",
          "position", "position_state", "title", "title_state", "link", "link_state",
          "snippet", "snippet_state"]

def spreadsheet_text(value):
    text = "" if value is None else str(value)
    stripped = text.lstrip()
    if (stripped.startswith(("=", "+", "-", "@", "＝", "＋", "－", "＠"))
            or text.startswith(("\t", "\r", "\n"))):
        return "'" + text
    return text

def field(row, name):
    if name not in row:
        return "", "missing"
    value = row[name]
    if value is None:
        return "", "null"
    if name == "position":
        return (value, "value") if type(value) is int and value > 0 else ("", "invalid")
    if isinstance(value, str):
        return spreadsheet_text(value), "value"
    return spreadsheet_text(json.dumps(value, ensure_ascii=False)), "unexpected_type"

def export(source, target):
    source, target = Path(source), Path(target)
    sidecar = target.with_suffix(target.suffix + ".run.json")
    if source.resolve() in (target.resolve(), sidecar.resolve()):
        raise ValueError("Output paths must differ from input")
    record = json.loads(source.read_text(encoding="utf-8"))
    request = record.get("request")
    if not isinstance(request, dict) or not isinstance(request.get("input"), dict):
        raise ValueError("Expected a capture record with request.input")
    payload = record.get("response")
    rows = payload.get("organic_results") if isinstance(payload, dict) else None
    status = record.get("http_status")
    if status == 201:
        state, rows = "pending", []
    elif status != 200:
        state, rows = ("transport_error" if status is None else "http_error"), []
    elif not isinstance(rows, list) or any(not isinstance(x, dict) for x in rows):
        state, rows = "unmapped", []
    else:
        state = "observed" if rows else "empty"
    context = {
        "run_id": spreadsheet_text(record.get("run_id")),
        "requested_at": spreadsheet_text(record.get("requested_at")),
        "received_at": spreadsheet_text(record.get("received_at")),
        "q": spreadsheet_text(request["input"].get("q")),
        "request_json": spreadsheet_text(json.dumps(request, ensure_ascii=False, sort_keys=True)),
    }
    with target.open("w", encoding="utf-8-sig", newline="") as handle:
        writer = csv.DictWriter(handle, fieldnames=FIELDS)
        writer.writeheader()
        for ordinal, item in enumerate(rows):
            output = dict(context, ordinal=ordinal)
            for name in ("position", "title", "link", "snippet"):
                output[name], output[name + "_state"] = field(item, name)
            writer.writerow(output)
    sidecar.write_text(json.dumps({"source": str(source), "run_id": record.get("run_id"),
        "state": state, "rows": len(rows), "request": request,
        "requested_at": record.get("requested_at"), "received_at": record.get("received_at"),
        "export_policy": "spreadsheet_text_prefix_v1; original values remain in source JSON"},
        ensure_ascii=False, indent=2), encoding="utf-8")
    print(f"Exported {len(rows)} organic rows; state={state}; metadata={sidecar}")
    return state, len(rows)

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("source")
    parser.add_argument("target")
    args = parser.parse_args()
    export(args.source, args.target)

The program writes a header even when there are no projected organic rows. Its sidecar records the state, row count, source file, request, timestamps, and export-policy description. Keep the CSV and sidecar together when handing them to another analyst.

Inspect Empty and Malformed Results

An empty array produces a header-only CSV with state=empty. A pending task produces the same visible row count but a different sidecar state. An HTTP error or transport failure is also preserved separately.

If the organic field is missing, has the wrong type, or contains a non-object item, the whole run becomes unmapped and no organic rows are exported. This avoids quietly skipping malformed items while presenting the remaining rows as a complete projection. Inspect the raw response before changing that rule.

Individual optional fields are handled more narrowly. A missing snippet does not discard an otherwise usable organic item. Its state column tells the analyst why the cell is blank. A complex unexpected text-field value stays represented as JSON text with an explicit type warning.

These are application policies. Document them with the export because another transform may make different choices. A downstream analyst should not have to infer how nulls were handled by looking at a handful of rows.

Check the Handoff, Not Only the Row Count

Read the file back through a CSV parser and compare the parsed columns, rather than counting physical lines. A quoted snippet can contain a newline without creating another logical result row.

Check that every row carries its run context and that Unicode text survives the round trip. Inspect formula-sensitive cells in your actual spreadsheet application. The local automated checks can validate the prefix policy and CSV parsing, but cannot prove every future application's interpretation.

Also compare the sidecar row count with the parsed record count. Keep the source file accessible, since the spreadsheet omits other search modules and preserves some values in transformed form. A successful export means this projection is internally consistent, not that the upstream search data is complete or representative.

Conclusion

Export context with the results, retain field states where blanks would be ambiguous, and give every header-only file a run record. A CSV built this way is easier to inspect and less likely to turn missing collection or spreadsheet interpretation into a misleading finding.

FAQ

Why keep the whole request in a CSV column?

It preserves settings beyond a short fixed schema, including URL-mode input and optional controls. The query column remains convenient, while the serialized request carries the complete submitted configuration.

Does CSV quoting prevent formula interpretation?

No. Delimiter escaping and spreadsheet interpretation are different. Use a documented text policy and verify the import settings in the destination application.

Can an empty CSV prove that Google returned no results?

No. Inspect the run state. Pending, failed, unmapped, and present-empty responses are distinct outcomes.

Does the exporter change the original JSON?

No. It reads the capture and writes separate files. The original values remain available there for exact reconstruction.

Model Google SERP Snapshots in SQLite

Ethan Walker — Fri, 11 Sep 2026 06:19:01 +0000

A search-results table cannot tell you whether a query returned no organic results or whether the collector never received usable data. Both situations can leave the table with no rows. If you want to compare observations over time, the run itself needs a record.

This tutorial stores Google SERP captures in SQLite using a parent runs table and a child results table. It retains the original JSON, keeps the submitted request as the comparison key, and refuses to compare pending or failed observations as if they were ranking changes.

Scrapeless Google Search API supplies the upstream search data. The SQLite model is application code you own; it builds history from captures you save rather than retrieving an existing historical ranking database.

Define the Capture Contract

The importer expects a JSON capture envelope, not a bare API response. Each file contains a nonempty run_id, a request object, and the HTTP outcome. The request contains actor: scraper.google.search and an input object holding the exact submitted search settings.

For a received response, preserve http_status, received_at, and response. Keep requested_at too if the collector records it. A transport failure can have no HTTP status and no response; the run still exists and belongs in the collection-coverage report.

These envelope fields are defined by the application. The upstream request contract uses POST https://api.scrapeless.com/api/v1/scraper/request and authentication through x-api-token. HTTP 200 returns task data as the response body; HTTP 201 indicates a pending task. Do not assume that a successful network exchange always contains completed search results.

The importer below makes no network calls and needs no API key. Live collection is a separate prerequisite, and no authenticated search result is claimed in this tutorial. Local tests use explicitly synthetic records to exercise storage behavior.

Separate Runs From Organic Rows

runs stores one immutable capture, including its state and raw record. results stores the organic items from usable captures. The foreign key connects each result back to the search that produced it.

The child key is (run_id, ordinal). Ordinal preserves array order, while position retains the returned position if it is a positive integer. These fields are deliberately separate: a missing position should not be replaced with an invented ranking based on array order.

The raw_result field preserves optional fields and unusual values that the simple projection does not map. A nonstring title becomes null in the convenience column but remains visible in the raw item. A future parser can reconstruct a richer projection from the archive.

SQLite foreign-key enforcement must be enabled for the connection. The script does that before importing. Its parameterized statements also keep query values separate from SQL text, following the Python sqlite3 interface.

Use the Whole Request as the Context Key

A conservative comparison key serializes the submitted request with sorted object keys. It includes country, language, pagination offset, and any other settings your collector supplied. It excludes run identifiers and client timestamps because those live outside the request.

This is a key for exact submitted configurations, not a universal semantic canonicalizer. Explicitly providing a default and omitting it produce different keys. Equivalent-looking URLs can also remain different. Keeping them separate is safer than merging them without a documented rule.

The country and language parameter definitions explain why a query alone is insufficient. A different start also changes the collected slice, so it remains part of the key. The script compares page slices; combining several pages into a collection window would need a separate parent model.

Sorting keys makes the serialization stable for these JSON objects. The allow_nan=False setting rejects nonstandard numeric values during storage rather than silently creating an archive that is difficult to exchange with another JSON implementation.

Import Captures With One Transaction Per Run

Save this program as serp_sqlite.py. It uses only Python's standard library. Put capture files with the envelope described above in a directory such as snapshots, then run python3 serp_sqlite.py research.sqlite --input-dir snapshots.

The program creates its tables if needed. Run imports are transactional: the parent and its children are committed together. Earlier successfully imported files remain committed if a later file is invalid; the script does not promise an all-or-nothing transaction for the entire directory.

import argparse
import json
import sqlite3
from pathlib import Path
from urllib.parse import urlsplit

SCHEMA = """
CREATE TABLE IF NOT EXISTS runs (
  run_id TEXT PRIMARY KEY,
  context_key TEXT NOT NULL,
  received_at TEXT,
  state TEXT NOT NULL,
  raw_record TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS results (
  run_id TEXT NOT NULL REFERENCES runs(run_id),
  ordinal INTEGER NOT NULL,
  position INTEGER,
  hostname TEXT,
  title TEXT,
  link TEXT,
  raw_result TEXT NOT NULL,
  PRIMARY KEY (run_id, ordinal)
);
CREATE INDEX IF NOT EXISTS runs_context ON runs(context_key, received_at);
CREATE INDEX IF NOT EXISTS results_host ON results(hostname);
"""

def canonical(value):
    return json.dumps(value, sort_keys=True, ensure_ascii=False, separators=(",", ":"), allow_nan=False)

def state_and_rows(record):
    payload = record.get("response")
    rows = payload.get("organic_results") if isinstance(payload, dict) else None
    status = record.get("http_status")
    if status == 201:
        return "pending", []
    if status is None:
        return "transport_error", []
    if status != 200:
        return "http_error", []
    if not isinstance(rows, list) or any(not isinstance(x, dict) for x in rows):
        return "unmapped", []
    return ("observed" if rows else "empty"), rows

def hostname(link):
    if not isinstance(link, str):
        return None
    try:
        parsed = urlsplit(link)
        return parsed.hostname.lower().rstrip(".") if parsed.scheme in ("http", "https") and parsed.hostname else None
    except ValueError:
        return None

def import_record(db, record):
    run_id, request = record.get("run_id"), record.get("request")
    if not isinstance(run_id, str) or not run_id or not isinstance(request, dict):
        raise ValueError("A nonempty run_id and request object are required")
    if not isinstance(request.get("input"), dict) or request.get("actor") != "scraper.google.search":
        raise ValueError("Expected the Google Search request envelope")
    raw = canonical(record)
    existing = db.execute("SELECT raw_record FROM runs WHERE run_id=?", (run_id,)).fetchone()
    if existing:
        if existing[0] != raw:
            raise ValueError("Conflicting content for existing run_id: " + run_id)
        return False
    state, rows = state_and_rows(record)
    with db:
        db.execute("INSERT INTO runs VALUES (?, ?, ?, ?, ?)",
                   (run_id, canonical(request), record.get("received_at"), state, raw))
        for ordinal, row in enumerate(rows):
            position = row.get("position")
            if type(position) is not int or position < 1:
                position = None
            title = row.get("title") if isinstance(row.get("title"), str) else None
            link = row.get("link") if isinstance(row.get("link"), str) else None
            db.execute("INSERT INTO results VALUES (?, ?, ?, ?, ?, ?, ?)",
                       (run_id, ordinal, position, hostname(link), title, link, canonical(row)))
    return True

def compare(db, before, after):
    records = [db.execute("SELECT context_key, state FROM runs WHERE run_id=?", (x,)).fetchone()
               for x in (before, after)]
    if any(x is None for x in records):
        raise ValueError("Both runs must exist")
    if records[0][0] != records[1][0]:
        raise ValueError("Request contexts differ")
    if any(x[1] not in ("observed", "empty") for x in records):
        raise ValueError("Only usable observations can be compared")
    query = "SELECT DISTINCT hostname FROM results WHERE run_id=? AND hostname IS NOT NULL"
    old = {x[0] for x in db.execute(query, (before,))}
    new = {x[0] for x in db.execute(query, (after,))}
    return {"newly_observed": sorted(new - old), "not_observed_later": sorted(old - new),
            "present_in_both": sorted(old & new)}

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("database")
    parser.add_argument("--input-dir")
    parser.add_argument("--compare", nargs=2, metavar=("BEFORE", "AFTER"))
    args = parser.parse_args()
    db = sqlite3.connect(args.database)
    try:
        db.execute("PRAGMA foreign_keys = ON")
        db.executescript(SCHEMA)
        if args.input_dir:
            paths = sorted(Path(args.input_dir).glob("*.json"))
            if not paths:
                raise ValueError("No JSON capture files found")
            added = sum(import_record(db, json.loads(p.read_text(encoding="utf-8"))) for p in paths)
            print(f"Imported {added} new runs from {len(paths)} files")
        if args.compare:
            print(json.dumps(compare(db, *args.compare), ensure_ascii=False, indent=2))
        print("Runs:", db.execute("SELECT COUNT(*) FROM runs").fetchone()[0])
    finally:
        db.close()

if __name__ == "__main__":
    main()

A repeated import of identical content with the same run identifier is a no-op. Different content under an existing identifier raises an error. That prevents a later capture from silently replacing the evidence used in an earlier report.

For a pending task that later completes, preserve the original pending capture and write the completed retrieval as a new observation with its own identifier. A larger system can add a task identifier to relate those records. Do not change the stored raw record under the same run identifier merely to make a dashboard look complete.

Keep Empty and Unavailable Outcomes Visible

The script derives state from the recorded HTTP status and response shape. It does not trust an arbitrary incoming state label as proof that the data is usable.

A nonempty organic array containing objects becomes observed. A present empty array becomes empty. A missing array, a non-array field, or a malformed item makes the run unmapped, with no projected result rows. The raw record remains available for inspection.

HTTP 201 becomes pending, another non-200 HTTP status becomes http_error, and a missing status becomes transport_error under this capture contract. These are application classifications. They are not extra API response codes.

You can now distinguish “no child rows because the response was empty” from “no child rows because the task was pending.” That distinction matters when preparing any absence-based metric. Keep unresolved states in the coverage report and exclude them from result comparisons.

Compare Domain Sets Within One Context

To compare saved runs, execute python3 serp_sqlite.py research.sqlite --compare BEFORE_RUN_ID AFTER_RUN_ID, replacing the identifiers with actual values from your database.

The comparison first checks that both runs exist, have identical request keys, and are usable. It then selects distinct non-null hostnames and reports domains newly observed, not observed in the later slice, and present in both.

This is hostname comparison. It does not merge subdomains into a corporate entity or infer a registrable domain. If that is required, introduce a reviewed grouping policy and retain the original hostname. The URL parsing reference also makes clear that parsing is not full input validation; the helper never fetches the URLs it reads.

“Not observed later” is intentionally scoped to the captured slice. It is not a claim that the domain disappeared from Google. An empty usable array can produce an empty set, but the report still needs its state and collection context attached.

Query Coverage Before Building a Trend

Inspect counts grouped by state before interpreting a series. A planned collection may include pending, failed, or unmapped runs that do not appear in the child table. Looking only at results would hide them.

Keep the definition of a comparison stable. If you change parser rules, domain grouping, or query coverage, record the version and consider rebuilding derived rows from archived JSON. The raw captures should remain immutable.

SQLite is a useful local starting point because the records and joins are easy to inspect. This example does not implement scheduling, remote storage, concurrent ingestion coordination, or schema migrations. Add those only when the actual collection workflow requires them, with explicit rules for preserving existing evidence.

Conclusion

Store the run even when it has no organic rows. Preserve the raw capture, compare exact request contexts, and make repeated imports safe to inspect. Those choices let the database distinguish collection health from observed domain movement.

FAQ

Why not use (query, date) as the primary key?

It can collapse different markets, languages, page offsets, or several observations on the same day. A run identifier preserves each capture; the request key defines comparable context.

Does this program obtain past rankings?

No. It imports capture files you already have. Your history starts with saved observations.

Can pending tasks count as empty results?

No. Pending tasks are stored under their own state and rejected by the comparison function until a separate completed observation is available.

Is the hostname list a competitor traffic report?

No. It describes domain presence in the selected organic slices. It does not contain visits, market share, or revenue.

Validate Google Search API Inputs Before Sending a Request

Ethan Walker — Thu, 10 Sep 2026 10:26:18 +0000

A request can contain the right parameter names and still express something different from what the caller intended. A full Google Search URL combined with a separate country field is one example: the caller sees both settings, while only the URL controls the search.

Input validation gives your application a place to catch that mismatch. It can also catch a misspelled field, reject conflicting location settings, and prevent an empty query from entering a collection job.

This tutorial builds a local Python validator for Scrapeless Google Search API. It returns a request body that the caller can inspect. It makes no HTTP calls, requires no API key, and uses only Python's standard library.

Separate the API Contract From Your Application Policy

Some checks come directly from the API parameter reference:

Parameter-based input requires q.
A full url is an alternative input mode; other input parameters are ignored when it is supplied.
location and uule cannot be combined.
gl, hl, google_domain, and the other named fields have different roles.

Other checks are decisions made by the application in this article. It supports a small country and language set, accepts selected Google hosts, and rejects fields that have not been reviewed for this integration. Those restrictions make the example easier to understand and test. They do not describe the service's full supported range.

Keep this distinction visible in error messages. “Outside this application's supported values” tells a developer where the restriction lives. “The API does not support this country” would claim something the validator has not established.

Choose One Input Mode

In parameter mode, callers supply a query and optional fields inside input. Country, language, and location stay individually visible to forms, logs, and downstream storage.

In URL mode, callers supply a complete Google Search URL inside input.url. The service ignores other input fields in that mode. This validator therefore rejects a mixed object instead of silently dropping the extra keys.

The same principle applies to location. When both location and uule are present, the validator raises an error. It does not pick one based on dictionary order or remove a field behind the caller's back.

Rejecting conflicts is especially useful in a form backed by saved configuration. A previously selected city can remain in state when a user switches input modes. A visible validation error gives the application a clear reason to ask the user to resolve the configuration.

Define a Reviewed Parameter Allowlist

The validator accepts the following parameter names in parameter mode:

Group	Fields	Local behavior
Query	`q`	Require a nonempty string
Search context	`gl`, `hl`, `google_domain`	Check against this example's configured values
Origin	`location`, `uule`	Require text and reject simultaneous use
Result restrictions	`cr`, `lr`, `tbs`	Require text; detailed syntax remains unchecked
Filtering	`safe`, `nfpr`, `filter`	Check the selected string or integer values
Search type	`tbm`	Accept `isch`; omit the field for regular web search
Pagination	`start`	Require a nonnegative integer
Device	`device`	Accept `desktop`, matching the current reference

An allowlist catches spelling errors such as langauge or a field named country mistakenly placed where gl was intended. It also provides a review point when your application adds another parameter.

The example does not include every field in the reference. Adding a new entry should include its documented meaning, local type rules, and an example that your consumer can handle. A field's presence in a reference table does not automatically establish an output mapping for every search scenario.

Install Nothing and Run the Complete Validator

Save the following file as validate_search_input.py. Run it with python3 validate_search_input.py in a terminal. It uses dictionary operations, JSON serialization, regular expressions, and the standard URL parser.

The three built-in inputs demonstrate a parameter-based web search, a city-origin request, and URL mode. The program prints constructed request bodies. These are local outputs, not responses from Google or from Scrapeless.

"""Local input policy for a small Google Search API integration; no network calls."""
import json
import re
from urllib.parse import parse_qs, urlsplit

PARAMETERS = {
    "q", "gl", "hl", "location", "uule", "google_domain", "cr", "lr",
    "tbs", "safe", "nfpr", "filter", "tbm", "start", "device",
}
COUNTRIES = {"us", "uk", "fr"}
LANGUAGES = {"en", "es", "fr"}
DOMAINS = {"google.com", "google.co.uk", "google.fr"}
URL_HOSTS = DOMAINS | {"www." + domain for domain in DOMAINS}


def nonempty_text(value, field):
    if not isinstance(value, str) or not value.strip():
        raise ValueError(f"{field} must be a nonempty string")


def validate_search_url(value):
    nonempty_text(value, "url")
    if any(char.isspace() for char in value):
        raise ValueError("url must encode whitespace")
    if re.search(r"%(?![0-9a-fA-F]{2})", value):
        raise ValueError("url contains an invalid percent escape")
    parts = urlsplit(value)
    if parts.scheme != "https" or parts.hostname not in URL_HOSTS:
        raise ValueError("url must use HTTPS and an approved Google host")
    if parts.username is not None or parts.password is not None or parts.port is not None:
        raise ValueError("url must not contain credentials or an explicit port")
    if parts.path != "/search" or parts.fragment:
        raise ValueError("url must use /search without a fragment")
    query = parse_qs(parts.query, keep_blank_values=True, strict_parsing=True)
    if any(not key or len(values) != 1 for key, values in query.items()):
        raise ValueError("url query keys must be nonempty and unique")
    nonempty_text(query.get("q", [None])[0], "url query q")


def validate_input(value):
    if not isinstance(value, dict) or any(not isinstance(key, str) for key in value):
        raise ValueError("input must be a dictionary with string keys")
    if "url" in value:
        if set(value) != {"url"}:
            raise ValueError("Use url alone; other input parameters would be ignored")
        validate_search_url(value["url"])
        return dict(value)

    unknown = set(value) - PARAMETERS
    if unknown:
        raise ValueError("Unreviewed input fields: " + ", ".join(sorted(unknown)))
    nonempty_text(value.get("q"), "q")
    if "location" in value and "uule" in value:
        raise ValueError("Choose location or uule")

    for field in PARAMETERS - {"start", "nfpr", "filter"}:
        if field in value:
            nonempty_text(value[field], field)
    choices = {
        "gl": COUNTRIES, "hl": LANGUAGES, "google_domain": DOMAINS,
        "safe": {"active", "off"}, "tbm": {"isch"}, "device": {"desktop"},
    }
    for field, allowed in choices.items():
        if field in value and value[field] not in allowed:
            raise ValueError(f"{field} is outside this application's supported values")
    for field in ("nfpr", "filter"):
        if field in value and (type(value[field]) is not int or value[field] not in (0, 1)):
            raise ValueError(f"{field} must be integer 0 or 1")
    if "start" in value and (type(value["start"]) is not int or value["start"] < 0):
        raise ValueError("start must be a nonnegative integer")
    return dict(value)


def build_request(value):
    return {"actor": "scraper.google.search", "input": validate_input(value)}


if __name__ == "__main__":
    examples = [
        {"q": "coffee", "gl": "us", "hl": "en", "start": 0},
        {"q": "coffee", "gl": "fr", "hl": "fr", "location": "Paris, France"},
        {"url": "https://www.google.com/search?q=coffee&gl=us&hl=en"},
    ]
    for example in examples:
        print(json.dumps(build_request(example), ensure_ascii=False))

The output contains three JSON request bodies with actor set to scraper.google.search. No credential is read, and no request is submitted. This makes the script usable in a local editor or a continuous-integration check before any account-dependent work.

Check URL Structure Without Claiming Complete URL Validation

The full-URL branch accepts HTTPS URLs on a small set of approved Google hosts, requires the /search path, rejects credentials and explicit ports, and requires a nonempty q query parameter. It also rejects fragments, unencoded whitespace, malformed percent escapes, and repeated query keys.

These are application rules. For example, rejecting an explicit default HTTPS port is a deliberate simplification, not a statement that such a URL is invalid everywhere. The domain set contains only google.com, google.co.uk, and google.fr, with optional www hosts. Expand that set only after reviewing the destinations needed by your integration.

Use exact host membership rather than a substring check. A hostname containing the text google.com can still belong to a different domain. Parsing the URL allows the application to inspect the hostname as a component.

Python's urllib.parse documentation notes that parsing does not itself provide complete validation. This example adds checks for its own narrow URL contract. It does not establish that every query-string parameter or encoded location is meaningful to Google.

URL mode also does not apply the parameter-mode country and language sets to values embedded in the URL. It preserves the complete URL as a distinct input mode. If your product needs identical value restrictions in both modes, add an explicit query-parameter policy and tests before enabling URL entry for that workflow.

Keep Validation From Changing the Query

The validator checks whether a text value is nonempty after whitespace inspection, but returns the original value. It does not translate the query, rewrite operators, silently change country codes, or insert a location.

That makes the accepted input useful as a record of caller intent. If the application wants normalization, implement it as a separate, visible step with its own rules. Preserve the original text when a transformation matters to later analysis.

The returned dictionary is a shallow copy. In this example, accepted parameter values are scalar strings or integers, so callers can work with the returned dictionary without sharing a nested mutable configuration object. The function does not accept arbitrary lists or nested objects as parameter values.

The integer checks also reject booleans. Python treats bool as a subclass of int, so a general integer-instance check would accept True as an offset. Using type(value) is int expresses the narrower policy intended here.

Exercise the Failure Paths

Try changing the first example to include both url and gl. The function raises an error before building the request. The caller can display that message beside the relevant field or record a configuration failure separately from collection outcomes.

Useful cases to cover include:

An empty query and an unknown parameter name.
Both location representations in one input.
A negative offset, a string offset, or a boolean offset.
A country outside the application's configured set.
A URL using an unapproved host or an ambiguous repeated query key.
A valid input whose text and values must remain unchanged.

The companion test_validate_search_input.py exercises accepted inputs and these rejection paths with Python's standard unittest module. Place it beside the validator and run python3 -m unittest -v test_validate_search_input.py.

import unittest
from validate_search_input import build_request, validate_input


class ValidateInputTests(unittest.TestCase):
    def test_accepts_reviewed_inputs(self):
        cases = [
            {"q": "café, site:example.org", "gl": "fr", "hl": "fr"},
            {"q": "coffee", "location": "Paris, France", "start": 0},
            {"q": "coffee", "uule": "opaque-value"},
            {"q": "coffee", "tbm": "isch", "safe": "active", "device": "desktop"},
            {"q": "coffee", "nfpr": 1, "filter": 0},
            {"url": "https://www.google.com/search?q=coffee&gl=us&hl=en"},
            {"url": "https://google.fr/search?q=caf%C3%A9"},
        ]
        for original in cases:
            with self.subTest(original=original):
                result = validate_input(original)
                self.assertEqual(result, original)
                self.assertIsNot(result, original)
                self.assertEqual(build_request(original)["actor"], "scraper.google.search")

    def test_rejects_invalid_inputs(self):
        cases = [
            [], {1: "coffee"}, {}, {"q": " "}, {"q": None},
            {"q": "coffee", "langauge": "en"},
            {"url": "https://www.google.com/search?q=coffee", "gl": "us"},
            {"q": "coffee", "location": "Paris", "uule": "encoded"},
            {"q": "coffee", "start": -1}, {"q": "coffee", "start": "10"},
            {"q": "coffee", "start": True}, {"q": "coffee", "filter": False},
            {"q": "coffee", "nfpr": 2}, {"q": "coffee", "gl": "de"},
            {"q": "coffee", "hl": []}, {"q": "coffee", "safe": "blur"},
            {"q": "coffee", "device": "mobile"}, {"q": "coffee", "tbm": "nws"},
            {"url": "https://google.com.example.org/search?q=coffee"},
            {"url": "http://www.google.com/search?q=coffee"},
            {"url": "https://user@www.google.com/search?q=coffee"},
            {"url": "https://www.google.com:443/search?q=coffee"},
            {"url": "https://www.google.com/search?q=coffee&q=tea"},
            {"url": "https://www.google.com/search?q="},
            {"url": "https://www.google.com/search?q=coffee#section"},
            {"url": "https://www.google.com/images?q=coffee"},
            {"url": "https://www.google.com/search?q=coffee grinder"},
            {"url": "https://www.google.com/search?q=%ZZ"},
        ]
        for original in cases:
            with self.subTest(original=original):
                with self.assertRaises(ValueError):
                    validate_input(original)


if __name__ == "__main__":
    unittest.main()

These tests check local behavior. Passing them does not establish that an API request will succeed, that a location resolves as intended, or that a returned module has the schema your application expects. Those are separate integration checks.

Put the Validator Before Authentication and Submission

The intended call sequence is straightforward: obtain the user's configuration, validate it, build the request body, and then pass that body to the authenticated client. The request workflow documents the POST endpoint and the x-api-token authentication header for that later step.

Keep credentials outside this input object. The validator reviews search configuration, while authentication belongs in the HTTP client. That separation also allows the validated body to be inspected without placing the API key in a configuration log.

After submission, response handling has its own responsibilities. HTTP 200 carries task data; HTTP 201 indicates a pending task with a taskId. Valid input does not make those states interchangeable. Save task state and inspect the returned data before applying a result mapping.

The application should also record the configuration it actually sent. Country, language, origin, and offset are useful context when a colleague later asks why two observations differ. URL-mode observations should retain the complete URL and their input-mode label.

Conclusion

Start with the input rules that can be checked locally: one mode, one location representation, reviewed parameter names, and explicit value types. Keep application restrictions separate from claims about the API. The resulting request body is easier to inspect, and configuration mistakes can be corrected before collection begins.

FAQ

Does this script call the API or require an API key?

No. It validates local data and prints request bodies. Authentication and submission happen in a separate client.

Why does it reject a documented parameter?

The allowlist covers this example's reviewed subset. Add other fields after checking their meaning and the application's needs.

Does accepting a location prove that it resolves correctly?

No. The local check establishes that it is nonempty text and does not conflict with uule. It does not perform location lookup or encoded-location validation.

Does URL mode enforce every parameter-mode restriction?

No. It checks the URL structure and query-key rules described above. Add an explicit policy for embedded query values if your application needs one.

Why does the example accept only image mode in tbm?

The sample supports regular web search when tbm is omitted and image search with isch. That is the scope of this application, rather than a claim about every search type the service may support.

Build a Google Search API JSON Workflow with Python

Ethan Walker — Thu, 10 Sep 2026 10:11:16 +0000

TL;DR:

Scrapeless Google Search API exposes search results as JSON. The returned organic-result fields can feed SEO tooling, research exports, and source selection.
Store the search settings alongside every response. A query, country, language, and observation time give a result the context needed for comparison.
Check task state before exporting rows. HTTP 201 means the task is still processing, rather than that a search returned no results.

A search export is easier to debug when each row can be traced back to an input. Otherwise, a changed URL leaves an awkward question: did the result change, or did the request use a different market?

The updated Scrapeless Google Search API accepts a search request and returns structured JSON. Collection-side proxy and CAPTCHA handling are managed by the service. Your code owns the request settings, response interpretation, and any storage or reporting built afterward.

The implementation below keeps those responsibilities visible. It submits a small request, saves the input with the response, and creates a CSV only when the task has returned usable organic-result data.

What the Google Search API Returns

The organic web results appear in a top-level organic_results array when that field is present. Its items can contain a position, title, link, and snippet. Search metadata, pagination, and additional result modules can accompany those rows, depending on the search.

Save the payload before flattening it. A CSV represents a selected set of columns; nested search modules need their own mapping. The JSON data model keeps objects, arrays, primitive values, and null distinct, so preserving the original data leaves room for a different export later.

The snippet field belongs to the search result. It should not be used as a substitute for the destination page's complete text. A pipeline that needs evidence from that page has another acquisition and review step after source discovery.

Prepare a Small First Request

You need an account with Google Search API access, a Scrapeless API key, and a Python environment that can send HTTP requests. Put the key in SCRAPELESS_API_KEY through your shell or local secret manager. Do not write it into the script or commit it with an example project.

Install requests in that environment. The script's CSV, JSON, path, and date handling use Python's standard library. Save the complete example as google_search_export.py and execute python3 google_search_export.py once the key is available.

Its first input is deliberately small: coffee, with gl=us and hl=en. Look at the resulting JSON before adding more keywords. A local export that works for one observed response is a better starting point than a large collection job whose shape you have never inspected.

Calling the service requires your account key. The HTTP portion follows the current request reference and is included as an integration prerequisite, rather than as evidence of a live account run performed for this article.

Choose Country, Language, and Input Mode

The input object describes several independent search dimensions. gl is the country setting; hl is the language setting. location describes the search origin, and google_domain selects the Google domain. The currently supported device option is desktop.

There are also two construction rules in the Google Search API parameter model:

Build a parameter-based request around q, or supply a complete Google Search url. Supplying url causes the other input parameters to be ignored.
Supply location or uule, but never both in the same request.

Persist the actual input object rather than reconstructing it from a filename. This makes an accidental setting change visible when comparing runs. Keeping country and language consistent defines the intended comparison; it does not make a dynamic search result identical each time or recreate an individual user's signed-in history.

The query can use site:, inurl:, and intitle: to focus discovery. Do not interpret the output of a site-restricted search as a complete count of pages indexed for a domain. It is a search sample with its own retrieval limits.

Request JSON and Export Organic Results

Send a POST request to https://api.scrapeless.com/api/v1/scraper/request with actor set to scraper.google.search. Authentication uses x-api-token. The following script records the input, HTTP status, response, and client receipt time before creating its selected CSV view.

Note: A Scrapeless API key is required for the network request, which has not been executed with a live account for this article. HTTP 201 responses are saved for inspection; task-result retrieval is outside this script.

import csv
import json
import os
from datetime import datetime, timezone
from pathlib import Path

import requests


def spreadsheet_text(value):
    text = "" if value is None else str(value)
    if text.lstrip().startswith(("=", "+", "-", "@")) or text.startswith(("\t", "\r")):
        return "'" + text
    return text


def export_results(payload, context, received_at, output_path):
    results = payload.get("organic_results")
    if not isinstance(results, list):
        print("No usable organic_results array; inspect the saved JSON.")
        return
    fields = ["q", "gl", "hl", "received_at", "position", "title", "link", "snippet"]
    with output_path.open("w", encoding="utf-8", newline="") as stream:
        writer = csv.DictWriter(stream, fieldnames=fields)
        writer.writeheader()
        for item in results:
            if not isinstance(item, dict):
                raise ValueError("Unexpected organic result item; inspect the saved JSON.")
            row = {name: item.get(name) for name in ("position", "title", "link", "snippet")}
            row.update(context, received_at=received_at)
            writer.writerow({name: spreadsheet_text(row.get(name)) for name in fields})
    print(f"Exported {len(results)} organic results to {output_path}")


def main():
    context = {"q": "coffee", "gl": "us", "hl": "en"}
    response = requests.post(
        "https://api.scrapeless.com/api/v1/scraper/request",
        headers={"x-api-token": os.environ["SCRAPELESS_API_KEY"]},
        json={"actor": "scraper.google.search", "input": context},
        timeout=120,
    )
    response.raise_for_status()
    received_at = datetime.now(timezone.utc).isoformat()
    run_id = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%S%fZ")
    payload = response.json()
    record = {"input": context, "received_at": received_at,
              "http_status": response.status_code, "response": payload}
    output = Path(f"google-search-{run_id}.json")
    output.write_text(json.dumps(record, ensure_ascii=False, indent=2), encoding="utf-8")
    if response.status_code == 201:
        print(f"Task pending. Inspect taskId in {output}; no CSV was created.")
        return
    if response.status_code != 200 or not isinstance(payload, dict):
        raise ValueError(f"Unexpected response; inspect {output}")
    export_results(payload, context, received_at, output.with_suffix(".csv"))


if __name__ == "__main__":
    main()

The timeout of 120 is an example client limit, not a measured service latency or SLA. The received_at value is created by the client after receiving a response. It describes receipt time, rather than a Google-generated collection timestamp.

The standard CSV writer takes care of quoting and delimiters. A small helper prefixes text with common formula-opening characters for spreadsheet inspection. That transformation is one reason the raw JSON remains the primary record. Review your spreadsheet application's text-import settings when opening third-party values.

The example's CSV stores the three input dimensions it actually uses: q, gl, and hl. Add columns for location, google_domain, or start if you add those settings. They are already retained in the full input saved in JSON.

Interpret the Response Before Building a Report

Branch on the task state before treating the body as a finished result. HTTP 200 carries the task data. HTTP 201 says processing is ongoing and returns a taskId, so the script records the response and ends without producing a CSV.

Even after a successful data response, an empty array and a missing organic_results field deserve different treatment. A response may carry other modules. The code reports the absence of a usable array and leaves the saved payload available for inspection instead of silently calling it a zero-result search.

Keep the returned position value as supplied. Before calculating a rank across multiple pages, confirm the numbering behavior for the requests you are making. start is the pagination offset, and response pagination information can help you move forward, but this does not promise access to every result in Google.

Put Structured Search Data to Work

The API provides search data; your application adds the workflow around it. For analysis, treat a result, its search context, and its observation time as one unit rather than three unrelated pieces of data.

SEO snapshots: collect a chosen keyword list with fixed settings and compare matching observations later. Your code or infrastructure supplies scheduling, persistence, and comparison logic.
Brand and competitor research: group the returned domains and review titles for those queries. This describes the selected sample, without measuring complete web coverage or traffic to a domain.
AI source discovery: hand candidate titles, URLs, and snippets to a source-selection stage. Obtain full pages independently when the task requires evidence, then validate claims against their sources.

A small research export can be enough for a first delivery. Give a content team a reading list with its query and market, or feed an existing report from a repeatable file. Both are easier to maintain when the export explains how its rows were obtained.

Use the workflow for public information you are permitted to collect and use. Keep credentials protected, retain only the data needed for the task, and review the conditions governing its later reuse.

Conclusion

A working integration needs more than a successful JSON parse. It needs a saved input, an explicit task-state check, and a clear handoff from search discovery to analysis. Start with the single-query script and read the complete saved payload before expanding it.

Use the updated Google Search API request workflow when adapting the request to your application.

FAQ

Q: Is this an API provided by Google?

The service described here is Scrapeless Google Search API. It retrieves Google Search data through Scrapeless and does not imply an official Google partnership.

Q: Do you need to manage a browser or proxy?

Collection infrastructure is managed on the service side. The client sends an HTTP request and handles the data it receives.

Q: Does the API include historical ranking data?

This workflow builds a history from observations you save yourself. It does not provide an existing archive of past rankings.

Q: Can the same API search for images?

Google Image searches are supported, with tbm=isch listed in the parameter reference. Inspect their response independently because the CSV in this example maps organic web results.

Q: Does a search snippet contain the full page?

The snippet is a search-result excerpt. Obtain the destination page separately when the task needs the full source content.

Q: What should happen when the request returns HTTP 201?

Store the taskId and keep the task pending. Follow the documented task-result process before handling it as completed search data.

Flatten TikTok Shop Product JSON into Product and SKU Tables

Ethan Walker — Thu, 03 Sep 2026 08:47:43 +0000

Product-page JSON looks easy to flatten until the first item has several colors, sizes, and variant-specific prices. If product fields and SKU fields share one row, product facts repeat and variant facts become ambiguous.

This tutorial turns a saved TikTok Shop product response into two related datasets: one product row and zero or more SKU rows. The transformation keeps IDs as strings, preserves nulls, carries region and currency into the output, and serializes option pairs without discarding their meaning.

Start with the product hierarchy

The Scrapeless scraper.tiktok.shop.page actor accepts a known product_id and region. Its response can include product identity, name, seller, nested price, aggregate stock, rating, review count, sold count, images, options, SKUs, categories, and shipping data. The Shop Page actor reference provides the request and response surface.

Those fields belong at two levels:

The product table describes the page-level offer: product ID, returned region, seller, name, aggregate price and stock, rating, reviews, and sold count.
The SKU table describes a sellable option combination: SKU ID, option pairs, variant price, quantity, and in-stock state.

One product can own many SKU rows. A product without a returned SKU array still deserves a product row.

Prerequisites

The transformer uses Python 3 and its standard library. It reads product-response.json, a raw response previously saved from an authorized API request, and writes products.csv and skus.csv.

For live collection, send a JSON request to Scrapeless with this actor envelope:

{
  "actor": "scraper.tiktok.shop.page",
  "input": {
    "product_id": "YOUR_PRODUCT_ID",
    "region": "GB"
  }
}

Keep the actual product ID as a string. The region is part of the observation, not temporary request metadata.

Define explicit schemas

CSV has no nested object type, so decide which fields each file owns before writing rows.

The product schema used here contains:

collected_at
product_id
region
name
seller_id
product-level currency, sale price, and original price
aggregate available quantity and in-stock state
rating, review count, and sold count

The SKU schema contains:

collected_at
product_id
region
sku_id
serialized option pairs
SKU currency, sale price, and original price
SKU available quantity and in-stock state
image URL when present

Product ID links the two files. sku_id identifies a particular variant record within the product.

Normalize values without erasing uncertainty

The following helpers preserve numeric-looking IDs as strings and convert option objects into one readable value. The option separator is a presentation choice; the raw JSON remains the authoritative source.

import csv
import json
from datetime import datetime, timezone


def as_text(value):
    return "" if value is None else str(value)


def option_label(options):
    pairs = []
    for option in options or []:
        if isinstance(option, dict):
            name = as_text(option.get("name"))
            value = as_text(option.get("value"))
            pairs.append(f"{name}={value}")
        else:
            pairs.append(as_text(option))
    return " | ".join(pairs)

Do not apply or 0 to counts or quantities. A missing value and a definite zero have different meanings. The CSV writer will emit an empty cell for Python None, preserving that distinction for import rules that recognize blanks as null.

Build the product and SKU rows

Some responses use nested price and stock objects. Read them defensively and use the returned region rather than silently substituting the requested value in analysis.

def normalize_product(raw, collected_at):
    product_id = as_text(raw.get("product_id"))
    price = raw.get("price") or {}
    stock = raw.get("stock") or {}
    seller = raw.get("seller") or {}

    product_row = {
        "collected_at": collected_at,
        "product_id": product_id,
        "region": raw.get("region"),
        "name": raw.get("name"),
        "seller_id": as_text(
            raw.get("seller_id")
            or (seller.get("seller_id") if isinstance(seller, dict) else seller)
        ),
        "currency": price.get("currency"),
        "sale_price": price.get("sale_price"),
        "original_price": price.get("original_price"),
        "available_quantity": stock.get("available_quantity"),
        "in_stock": stock.get("in_stock"),
        "rating": raw.get("rating"),
        "review_count": raw.get("review_count"),
        "sold_count": raw.get("sold_count"),
    }

    sku_rows = []
    for sku in raw.get("skus") or []:
        sku_price = sku.get("price") or {}
        sku_rows.append({
            "collected_at": collected_at,
            "product_id": product_id,
            "region": raw.get("region"),
            "sku_id": as_text(sku.get("sku_id") or sku.get("id")),
            "options": option_label(sku.get("options")),
            "currency": sku_price.get("currency") or price.get("currency"),
            "sale_price": sku_price.get("sale_price"),
            "original_price": sku_price.get("original_price"),
            "available_quantity": sku.get("available_quantity"),
            "in_stock": sku.get("in_stock"),
            "image_url": sku.get("image_url") or sku.get("image"),
        })

    return product_row, sku_rows

Keeping a product-level price and a SKU-level price is intentional. A displayed product price does not automatically describe every variant.

Write stable CSV files

Declare column order explicitly. Relying on whatever keys happen to appear in the first object makes exports brittle when optional fields change.

PRODUCT_FIELDS = [
    "collected_at", "product_id", "region", "name", "seller_id",
    "currency", "sale_price", "original_price", "available_quantity",
    "in_stock", "rating", "review_count", "sold_count",
]

SKU_FIELDS = [
    "collected_at", "product_id", "region", "sku_id", "options",
    "currency", "sale_price", "original_price", "available_quantity",
    "in_stock", "image_url",
]


def write_csv(path, fields, rows):
    with open(path, "w", newline="", encoding="utf-8") as output:
        writer = csv.DictWriter(output, fieldnames=fields)
        writer.writeheader()
        writer.writerows(rows)


with open("product-response.json", encoding="utf-8") as source:
    raw = json.load(source)

collected_at = datetime.now(timezone.utc).isoformat()
product_row, sku_rows = normalize_product(raw, collected_at)

write_csv("products.csv", PRODUCT_FIELDS, [product_row])
write_csv("skus.csv", SKU_FIELDS, sku_rows)

print(f"wrote 1 product row and {len(sku_rows)} SKU rows")

The script works with zero SKU rows: skus.csv still gets a header. That output says the response contained no normalized variants; it does not claim the product has no variants in every market.

Preserve price context

Never compare price without carrying its currency and region. A numeric value alone cannot tell you whether two offers are equivalent. If you need a numeric type for calculation, create a separate decimal field and preserve the source string.

Apply the same caution to sold_count. The page field does not provide a shared recent reporting window, an order ledger, refunds, revenue, or GMV. Store it as observed rather than turning it into a sales-velocity metric.

Aggregate stock and SKU availability also answer different questions. Do not fill a missing SKU quantity with the product aggregate. Store absence as unknown and investigate it separately.

Validate the transformation

Before sending the files to a spreadsheet or warehouse, check:

Every ID column is configured as text.
Every SKU row has the same product ID and region as its parent observation.
Currency travels with each monetary value.
Empty cells are not automatically converted to zero.
Option pairs remain readable and the raw option array is still available in the source JSON.
Product and SKU counts reconcile with the response you processed.
The collection timestamp is UTC and belongs to the same raw snapshot.

For managed collection, Scrapeless exposes this actor through Scraping API. The API supplies the snapshot; scheduling, transformation, and history remain application responsibilities.

Conclusion

Flattening TikTok Shop JSON is primarily a modeling task. Keep product and SKU entities separate, preserve IDs as strings, carry region and currency through every relevant row, and leave missing values unknown. That structure remains usable for one-time research and for later snapshot comparisons.

FAQ

Why not write one row per product?

A single row cannot represent multiple option combinations cleanly. Product and SKU tables preserve their different levels of detail.

Should an absent SKU array be treated as an error?

No. Write the product row and an empty SKU file, then keep the collection status and raw response for review.

Can `sold_count` be used as recent sales velocity?

Not from this response alone. It does not define a shared recent window or provide an order ledger.

Is collecting public Shop data legal?

Legality depends on the market, purpose, access method, fields, and applicable terms. Collect only what the task needs and obtain legal advice for the intended use.

Disclaimer: This article is for technical education and does not provide legal advice. Follow applicable laws, platform terms, and organizational policies when collecting or using public web data.

From TikTok Username to Post Metrics: A Python API Workflow

Ethan Walker — Thu, 03 Sep 2026 08:28:22 +0000

A TikTok username is easy for a person to recognize, but it is not the identifier a post-collection request needs. A reliable workflow has two stages: resolve the public username into sec_uid, then use that value to request a bounded sample of public posts.

This tutorial builds that chain with Python and the Scrapeless Scraping API. It also shows how to keep identifiers, timestamps, missing fields, and sample scope intact so the resulting metrics can be audited later.

What the two requests accomplish

The workflow uses one HTTP endpoint and two actors:

scraper.tiktok.user.detail accepts unique_id, the visible username without the leading @.
The profile response can contain account_id, unique_id, and sec_uid alongside public profile fields.
scraper.tiktok.user.work accepts the returned sec_uid, plus cursor and count.
Its response contains an items array of observed public posts.

The TikTok User Detail documentation and User Work documentation describe those two actor surfaces.

Treat the returned list as a sample. Although the request documents a cursor input, a single response does not prove that you collected a creator's complete history. Record what you requested and what the API returned.

Prerequisites

You need:

Python 3 with the standard library
A Scrapeless API token in SCRAPELESS_API_KEY
A public username in TIKTOK_USERNAME
A permitted purpose and a retention period for the collected data

Do not paste credentials into the script or commit them to a repository.

Build one reusable actor client

Both requests use the same endpoint and authentication header, so start with a small helper. It accepts the actor name and actor-specific input, then returns decoded JSON.

import json
import os
from urllib.error import HTTPError
from urllib.request import Request, urlopen

ENDPOINT = "https://api.scrapeless.com/api/v1/scraper/request"
TOKEN = os.environ["SCRAPELESS_API_KEY"]


def run_actor(actor, actor_input):
    payload = json.dumps({"actor": actor, "input": actor_input}).encode()
    request = Request(
        ENDPOINT,
        data=payload,
        headers={
            "x-api-token": TOKEN,
            "content-type": "application/json",
        },
        method="POST",
    )
    try:
        with urlopen(request, timeout=60) as response:
            return json.load(response)
    except HTTPError as exc:
        body = exc.read().decode("utf-8", errors="replace")
        raise RuntimeError(f"actor request failed: {exc.code} {body}") from exc

This is ordinary HTTP and JSON handling. The application does not need a DOM selector or browser parser because the actor returns structured data.

Step 1: resolve the username

Read the username from the environment and remove a leading @. Send the normalized value as unique_id.

USERNAME = os.environ["TIKTOK_USERNAME"].lstrip("@")

profile = run_actor(
    "scraper.tiktok.user.detail",
    {"unique_id": USERNAME},
)

sec_uid = profile.get("sec_uid")
if not sec_uid:
    raise RuntimeError("profile response did not contain sec_uid")

Keep account_id, unique_id, and sec_uid as strings. Large numeric-looking identifiers can lose digits when spreadsheets or JavaScript-based tools coerce them into numbers. The handle is useful to readers, while sec_uid is the bridge to the post actor.

An absent sec_uid is a collection state, not a reason to invent an identifier from a URL. Keep the profile response and record that post retrieval could not continue.

Step 2: request a bounded post sample

Now call the post actor with the resolved identifier. This example asks for 20 items from the initial documented cursor.

posts_response = run_actor(
    "scraper.tiktok.user.work",
    {"sec_uid": sec_uid, "cursor": "0", "count": 20},
)

items = posts_response.get("items") or []

Save the raw response before transforming it. Raw JSON lets you revise field mappings later without collecting the same public page again.

from datetime import datetime, timezone

collected_at = datetime.now(timezone.utc).isoformat()

with open("tiktok-posts-raw.json", "w", encoding="utf-8") as output:
    json.dump(posts_response, output, ensure_ascii=False, indent=2)

Select fields for an analysis table

A post item can expose identity, description, time, public engagement counters, format details, hashtags, music, subtitles, and flags. Start with the smallest field set that answers your question.

def normalize_post(item):
    return {
        "collected_at": collected_at,
        "account_unique_id": str(profile.get("unique_id") or USERNAME),
        "account_id": str(profile.get("account_id") or ""),
        "post_id": str(item.get("id") or item.get("post_id") or ""),
        "post_url": item.get("url") or item.get("post_url") or "",
        "description": item.get("description") or "",
        "created_at": item.get("create_time") or item.get("date"),
        "play_count": item.get("play_count"),
        "like_count": item.get("like_count"),
        "comment_count": item.get("comment_count"),
        "share_count": item.get("share_count"),
        "collect_count": item.get("collect_count"),
        "repost_count": item.get("repost_count"),
        "is_pinned": item.get("is_pinned"),
    }


rows = [normalize_post(item) for item in items]
print(json.dumps(rows, ensure_ascii=False, indent=2))

Notice that missing metrics remain None in Python and null in JSON. Replacing them with zero would claim that the platform reported a definite zero. Blank text is less ambiguous, so descriptions can safely fall back to an empty string when that suits the export schema.

Preserve the boundaries of the sample

Store these collection facts beside every batch:

requested username
returned unique_id and sec_uid
request cursor
requested count
returned item count
UTC collection time
request success or failure state

The distinction matters when a post is missing. It may sit outside the bounded response, be unavailable at collection time, or reflect a failed request. None of those conditions means the post had zero engagement.

The same rule applies to content format. A blank video field does not automatically mean the item is invalid; photo posts and optional media fields need tolerant parsing. Use the structure actually returned and retain an unknown state when it is insufficient.

Metrics are observations, not attribution

Public play, like, comment, share, collect, and repost counts describe the post at collection time. They do not identify unique viewers, purchasers, or the cause of a conversion. A single snapshot gives a level. Two timestamped snapshots can support a delta when both observations refer to the same post and metric.

If you operationalize this workflow, the Scraping API product page explains where the TikTok actors fit within Scrapeless.

Common implementation mistakes

Passing unique_id directly to the post actor instead of resolving sec_uid
Converting account or post IDs to numbers
Treating the first items response as a complete archive
Filling missing counters with zero
Dropping the post URL and raw response needed for review
Logging the API token in an exception or debug payload
Reporting play count as unique reach or campaign revenue

Conclusion

The smallest reliable TikTok post-metrics pipeline is a two-request chain. Resolve a known public username, use the returned sec_uid to request a bounded post sample, and save both raw and normalized records with a collection timestamp. The value comes from preserving identity and scope as carefully as the metrics themselves.

FAQ

Why does the post actor need `sec_uid`?

The documented User Work actor identifies the account with sec_uid. The profile actor provides that value when given the public username.

Can one response contain every post?

Do not assume it does. Keep the cursor, requested count, returned count, and collection time. Continue only when the current API response and documentation provide a verified rule.

Should missing engagement values become zero?

No. Keep them null or explicitly unknown unless the source returned a definite zero.

Is public TikTok data collection legal?

Legality depends on the jurisdiction, purpose, access method, fields, and applicable terms. Collect only necessary public data, define retention, restrict access, and obtain advice for the intended use.

Disclaimer: This article is for technical education and does not provide legal advice. Follow applicable laws, platform terms, and organizational policies when collecting or using public web data.

How to Extract AI Chat Conversations at Scale

Ethan Walker — Mon, 31 Aug 2026 06:54:50 +0000

Capturing one AI answer is an API call. Capturing a useful dataset is a pipeline.

At scale, the difficult questions move away from HTTP syntax:

How do you run the same prompt panel across several AI chat surfaces?
How do you preserve raw evidence while still producing one analysis schema?
How do you keep concurrency bounded?
How do you identify each capture after the answer changes?
How do you add a new platform without rewriting the collector?

This tutorial builds a production-shaped Python pipeline around the Scrapeless LLM Chat Scraper. It covers ChatGPT, Perplexity, Gemini, Copilot, and Grok through one endpoint while keeping platform-specific fields behind small adapters.

The architecture

The pipeline has four stages:

Plan — expand prompts, countries, and engines into explicit jobs.
Capture — send each job to the appropriate Scrapeless actor.
Persist — write the untouched response as an append-only JSON Lines record.
Normalize — map answer and citation fields into one downstream schema.

Do not combine these stages into one large function. Raw capture and normalization have different lifecycles. If an adapter changes, you should be able to rebuild normalized data from stored responses without calling the API again.

One endpoint, several actor contracts

Every LLM actor uses:

POST https://api.scrapeless.com/api/v2/scraper/execute

Authentication uses x-api-token. The request body is always {actor,input}, and the response uses the shared {status, task_id, task_result} envelope.

The differences live inside input and task_result:

Platform	Actor	Extra input	Citation fields
ChatGPT	`scraper.chatgpt`	none required beyond prompt	`content_references[]`
Perplexity	`scraper.perplexity`	`web_search`	`web_results[]`
Gemini	`scraper.gemini`	none required beyond prompt	`citations[]`
Copilot	`scraper.copilot`	mode when needed	`citations[]`
Grok	`scraper.grok`	reasoning `mode`	`web_search_results[]`, `x_search_results[]`

Use the LLM Chat Scraper documentation to confirm actor inputs before changing the job configuration.

Prerequisites

You need Python 3.10 or newer and requests:

python -m pip install requests
export SCRAPELESS_API_KEY=your_api_token_here

The examples write to local JSON Lines files. A production deployment can replace the writer with object storage or a database while keeping the record format.

Step 1: model every capture as a job

An explicit job record prevents hidden defaults.

from dataclasses import asdict, dataclass


@dataclass(frozen=True)
class CaptureJob:
    engine: str
    actor: str
    prompt_id: str
    prompt: str
    country: str
    extra_input: dict


ENGINE_CONFIG = {
    "chatgpt": ("scraper.chatgpt", {}),
    "perplexity": ("scraper.perplexity", {"web_search": True}),
    "gemini": ("scraper.gemini", {}),
    "copilot": ("scraper.copilot", {"mode": "smart"}),
    "grok": ("scraper.grok", {"mode": "MODEL_MODE_FAST"}),
}

prompt_id should be a stable identifier from your own prompt registry. Keep the human-readable prompt as well; the ID tells you which logical prompt was used, and the text proves the exact wording.

Step 2: expand the experiment matrix

Use a generator so the planning stage does not need to hold the entire matrix in memory.

def build_jobs(prompts: list[dict], countries: list[str]):
    for prompt in prompts:
        for country in countries:
            for engine, (actor, extra_input) in ENGINE_CONFIG.items():
                yield CaptureJob(
                    engine=engine,
                    actor=actor,
                    prompt_id=prompt["id"],
                    prompt=prompt["text"],
                    country=country,
                    extra_input=extra_input,
                )

Example input:

PROMPTS = [
    {
        "id": "category-best-tools-v1",
        "text": "What are the best web scraping APIs for JavaScript-heavy sites?",
    },
    {
        "id": "ai-agent-web-data-v1",
        "text": "Which web data tools work well for AI agents?",
    },
]

COUNTRIES = ["US", "DE"]

More prompts, markets, and engines multiply the number of jobs. Compute that number before launching a collection window so cost and storage are visible.

Step 3: write one capture function

The collector should know nothing about citation fields. Its responsibility is transport and evidence preservation.

import os
import time

import requests

ENDPOINT = "https://api.scrapeless.com/api/v2/scraper/execute"
SESSION = requests.Session()
SESSION.headers.update(
    {
        "Content-Type": "application/json",
        "x-api-token": os.environ["SCRAPELESS_API_KEY"],
    }
)


def capture(job: CaptureJob) -> dict:
    request_input = {
        "prompt": job.prompt,
        "country": job.country,
        **job.extra_input,
    }
    started_at = int(time.time())

    response = SESSION.post(
        ENDPOINT,
        json={"actor": job.actor, "input": request_input},
        timeout=300,
    )
    response.raise_for_status()
    payload = response.json()

    return {
        "job": asdict(job),
        "request_input": request_input,
        "captured_at": started_at,
        "status": payload.get("status"),
        "task_id": payload.get("task_id"),
        "task_result": payload.get("task_result") or {},
    }

The returned record includes both the job and the exact request_input. This matters when a platform has a mode or web-search flag that changes the answer behavior.

Step 4: keep concurrency bounded

A fixed-size worker pool gives the pipeline controlled parallelism. Configure the worker count from the environment so the code does not hard-code an account-level assumption.

import os
from concurrent.futures import ThreadPoolExecutor, as_completed

MAX_WORKERS = int(os.environ.get("AI_CAPTURE_WORKERS", "3"))


def run_jobs(jobs):
    with ThreadPoolExecutor(max_workers=MAX_WORKERS) as executor:
        futures = {executor.submit(capture, job): job for job in jobs}

        for future in as_completed(futures):
            job = futures[future]
            try:
                yield {"ok": True, "record": future.result()}
            except Exception as error:
                yield {
                    "ok": False,
                    "job": asdict(job),
                    "error_type": type(error).__name__,
                    "error": str(error),
                }

This design records failed jobs as data. It does not let one exception discard the rest of the collection window.

Use a worker count that matches the capacity available to your account and environment. Bounded concurrency is easier to observe and budget than launching one task per job with no ceiling.

Step 5: use one writer

Several worker threads should not append to the same file independently. Keep file output in the main thread and write each result as it arrives.

import json


def write_results(results, path: str) -> None:
    with open(path, "a", encoding="utf-8") as output:
        for item in results:
            output.write(json.dumps(item, ensure_ascii=False) + "\n")
            output.flush()

Now assemble the capture phase:

if __name__ == "__main__":
    jobs = build_jobs(PROMPTS, COUNTRIES)
    results = run_jobs(jobs)
    write_results(results, "raw_ai_captures.jsonl")

Every line is either a successful raw capture or a structured error record. That makes the run auditable without depending on terminal output.

Step 6: normalize per platform

Normalization belongs in small adapter functions.

from urllib.parse import urlparse


def domain_of(url: str) -> str:
    return urlparse(url).netloc.lower().removeprefix("www.")


def source_record(source: dict) -> dict:
    url = source.get("url") or ""
    return {
        "title": source.get("title") or source.get("name") or "",
        "url": url,
        "domain": domain_of(url),
        "snippet": source.get("snippet") or "",
    }


def normalize_chatgpt(result: dict):
    return result.get("result_text") or "", [
        source_record(item)
        for item in (result.get("content_references") or [])
    ]


def normalize_perplexity(result: dict):
    return result.get("result_text") or "", [
        source_record(item)
        for item in (result.get("web_results") or [])
    ]


def normalize_gemini_or_copilot(result: dict):
    return result.get("result_text") or "", [
        source_record(item)
        for item in (result.get("citations") or [])
    ]


def normalize_grok(result: dict):
    web = result.get("web_search_results") or []
    x_sources = result.get("x_search_results") or []
    return result.get("result_text") or "", [
        source_record(item) for item in (web + x_sources)
    ]

ADAPTERS = {
    "chatgpt": normalize_chatgpt,
    "perplexity": normalize_perplexity,
    "gemini": normalize_gemini_or_copilot,
    "copilot": normalize_gemini_or_copilot,
    "grok": normalize_grok,
}


def normalize_capture(raw: dict) -> dict:
    engine = raw["job"]["engine"]
    answer, citations = ADAPTERS[engine](raw["task_result"])

    return {
        "engine": engine,
        "actor": raw["job"]["actor"],
        "prompt_id": raw["job"]["prompt_id"],
        "prompt": raw["job"]["prompt"],
        "country": raw["job"]["country"],
        "captured_at": raw["captured_at"],
        "task_id": raw["task_id"],
        "answer": answer,
        "citations": citations,
    }

The common schema should contain only fields that downstream analysis truly shares. Keep rich platform-only fields in the raw record instead of forcing them into a generic column.

Storage strategy

Use two layers.

Raw layer

Store the complete response plus job and request metadata. Make it append-only. Partition by capture date and, if useful, engine.

Normalized layer

Store one record per answer with a child citation collection. This layer feeds dashboards, mention detection, and comparison jobs.

The raw layer is evidence. The normalized layer is a view. Mixing them makes schema changes expensive.

Operational checks

Before scheduling a large matrix, validate these properties:

Every successful row has a task_id.
Every row preserves the exact prompt and country.
Actor-specific modes are attached to the request record.
Missing citation arrays become empty lists.
Raw files remain valid JSON Lines after an interrupted process.
Normalization is deterministic when run twice on the same raw file.
No secret appears in logs or stored records.

Scale by partitions

Large monitoring programs are easier to manage when jobs are partitioned by a stable dimension:

One queue per capture window
One object prefix per date
One prompt registry version per experiment
One normalized partition per engine and country

This structure lets you reprocess a subset without touching the full archive.

The official multi-engine brand pipeline shows how citation fields differ across answer engines. The actor family lives under the Universal Scraping API.

Conclusion

Extracting AI chat conversations at scale is mostly a data-contract problem. Plan explicit jobs, keep the worker pool bounded, preserve each raw response, and normalize with one adapter per platform.

The shared endpoint and top-level envelope remove transport duplication. The adapter layer handles the real differences: ChatGPT references, Perplexity web results, Gemini and Copilot citations, and Grok's separate web and X panels.

Start with a small prompt-country matrix, inspect the raw rows, then move the same functions into your scheduler. You can create an API key from the Scrapeless dashboard.

FAQ

Why use JSON Lines?

Each line is an independent record, which makes append-only writing, streaming, and partial inspection straightforward.

Should all platforms share one response schema?

Use one narrow normalized schema for common analysis fields, but preserve the platform-specific payload in the raw layer.

How many workers should the collector use?

Set a bounded value supported by your account and infrastructure, measure its behavior, and adjust deliberately. Do not create unbounded parallel tasks.

Can the normalizer run separately from collection?

Yes. That is the preferred design. It lets you revise adapters and rebuild derived data without making new API calls.

Is it legal to collect AI chat answers?

The actors collect publicly rendered answer content. Review applicable laws and each platform's terms, and consult counsel before collecting or redistributing data at scale.

Scrapeless accesses publicly available data and expects customers to follow applicable laws, platform terms, and privacy requirements. The examples in this article are for technical demonstration and should not be used to collect private, confidential, or restricted information.

How to Scrape Perplexity Answers and Citations

Ethan Walker — Mon, 31 Aug 2026 06:48:50 +0000

Perplexity is useful because its answers expose the source layer. For developers building GEO dashboards, content research tools, or brand-visibility monitors, the citation list can be more valuable than the prose answer itself.

The hard part is collecting that surface consistently. The consumer interface renders dynamically, the answer and sources arrive over time, and the data you want is spread across answer text, web results, related prompts, and rich media.

The Scrapeless Perplexity actor turns that surface into structured JSON. You send a prompt, country, and web-search setting to one HTTP endpoint; the response arrives inside the same {status, task_id, task_result} envelope used by the other LLM Chat Scraper actors.

What the Perplexity actor returns

The main fields are platform-specific even though the top-level envelope is shared.

Field	Purpose
`task_result.prompt`	The exact prompt associated with the capture
`task_result.result_text`	The generated answer with its formatting and citation markers
`task_result.web_results[]`	Ranked web sources with names, URLs, and snippets
`task_result.related_prompt[]`	Suggested follow-up questions
`task_result.media_items[]`	Images, videos, maps, and other rich media when present

Treat every optional array as nullable. A research prompt may return a rich web result set, while another query may produce no media at all.

The current field reference is available in the Perplexity Scraper documentation.

Prerequisites

You need:

A Scrapeless account and API key
curl for the first request
Python 3.10 or newer for the complete example
The Python requests package

Set the key in your shell instead of placing it in the script:

export SCRAPELESS_API_KEY=your_api_token_here

Make a Perplexity request with curl

Perplexity uses the scraper.perplexity actor. The request body has the shared {actor,input} shape.

curl -sS -X POST "https://api.scrapeless.com/api/v2/scraper/execute" \
  -H "Content-Type: application/json" \
  -H "x-api-token: ${SCRAPELESS_API_KEY}" \
  -d '{
    "actor": "scraper.perplexity",
    "input": {
      "prompt": "What are the best web scraping APIs for AI agents?",
      "country": "US",
      "web_search": true
    }
  }'

The three input fields serve different roles:

prompt is the question to send.
country pins the residential exit market.
web_search controls whether the run uses Perplexity's web-grounded answer path.

If the data will be compared over time, keep all three values fixed within a series.

Understand the response

The following sample is illustrative. It shows the documented field layout without claiming that these values came from a specific live prompt.

{
  "status": "success",
  "task_id": "5ad71f2c-...",
  "task_result": {
    "prompt": "What are the best web scraping APIs for AI agents?",
    "result_text": "A useful API for an AI agent should return structured data... [1]",
    "related_prompt": [
      "How do AI agents use web scraping APIs?"
    ],
    "web_results": [
      {
        "name": "Example source",
        "url": "https://example.com/source",
        "snippet": "An example source description."
      }
    ],
    "media_items": []
  }
}

Store task_id with the payload. It gives each capture a stable audit key when the same prompt is run more than once.

Write a reusable Python client

This client returns a normalized record while preserving the raw task_result for later analysis.

import os
import time
from urllib.parse import urlparse

import requests

ENDPOINT = "https://api.scrapeless.com/api/v2/scraper/execute"


def normalize_domain(url: str) -> str:
    return urlparse(url).netloc.lower().removeprefix("www.")


def scrape_perplexity(
    prompt: str,
    country: str = "US",
    web_search: bool = True,
) -> dict:
    response = requests.post(
        ENDPOINT,
        headers={
            "Content-Type": "application/json",
            "x-api-token": os.environ["SCRAPELESS_API_KEY"],
        },
        json={
            "actor": "scraper.perplexity",
            "input": {
                "prompt": prompt,
                "country": country,
                "web_search": web_search,
            },
        },
        timeout=300,
    )
    response.raise_for_status()
    payload = response.json()
    result = payload.get("task_result") or {}

    citations = []
    for position, source in enumerate(result.get("web_results") or [], start=1):
        url = source.get("url") or ""
        citations.append(
            {
                "position": position,
                "title": source.get("name") or "",
                "url": url,
                "domain": normalize_domain(url),
                "snippet": source.get("snippet") or "",
            }
        )

    return {
        "platform": "perplexity",
        "prompt": prompt,
        "country": country,
        "web_search": web_search,
        "captured_at": int(time.time()),
        "status": payload.get("status"),
        "task_id": payload.get("task_id"),
        "answer": result.get("result_text") or "",
        "citations": citations,
        "related_prompts": result.get("related_prompt") or [],
        "media_items": result.get("media_items") or [],
        "raw": result,
    }

The adapter assigns a citation position based on the returned order. That makes it easy to compare source prominence without parsing citation markers from the answer text.

Print an answer and its sources

Add a small command-line entry point:

if __name__ == "__main__":
    capture = scrape_perplexity(
        "What are the best web scraping APIs for AI agents?"
    )

    print(capture["answer"])
    print("\nSources")

    for source in capture["citations"]:
        print(
            f"[{source['position']}] "
            f"{source['title']} — {source['domain']} — {source['url']}"
        )

This output is useful during development. Production jobs should write structured records to object storage, a database, or JSON Lines instead of relying on console output.

Build citation-level metrics

Once web_results is normalized, several useful measurements become simple.

Domain presence

Check whether a tracked domain appears anywhere in the source set.

def cites_domain(capture: dict, tracked_domain: str) -> bool:
    tracked = tracked_domain.lower().removeprefix("www.")
    return any(
        item["domain"] == tracked or item["domain"].endswith(f".{tracked}")
        for item in capture["citations"]
    )

Citation position

Track the first source position at which the domain appears. Keep presence and position separate: a missing domain should be None, not position zero.

def first_citation_position(capture: dict, tracked_domain: str):
    tracked = tracked_domain.lower().removeprefix("www.")
    positions = [
        item["position"]
        for item in capture["citations"]
        if item["domain"] == tracked or item["domain"].endswith(f".{tracked}")
    ]
    return min(positions) if positions else None

Source diversity

Count distinct domains rather than raw URLs. Several cited pages from one site represent a different source profile from the same number of pages spread across many domains.

Citation overlap

For two captures of the same prompt, compare the domain sets. The overlap reveals whether Perplexity's answer changed while leaning on the same sources or changed because the source set moved.

Keep comparisons scientifically useful

Perplexity answers can vary. The goal is not to force one fixed output; the goal is to preserve the inputs and measure the output honestly.

Use these controls:

Keep the prompt text byte-for-byte identical inside a series.
Pin the same country.
Keep web_search fixed.
Record the capture time and task_id.
Store the full source objects, not only their domains.
Compare several captures before treating a change as a trend.

Do not drop an answer because media_items or related_prompt is empty. Optional panels depend on the query.

Scale beyond one prompt

The request function should stay small. Scaling belongs in the orchestration layer:

Read prompts from a versioned file or database table.
Generate one job for each prompt and country pair.
Limit concurrency to a value your account and workload support.
Write each raw response before running transforms.
Make normalization idempotent so it can be re-run from stored captures.

The Scrapeless Perplexity Scraper product page shows the current result surface, while the broader LLM scraper explainer places this workflow in a multi-engine monitoring system.

Conclusion

Scraping Perplexity answers reduces to one request and one platform adapter: send {actor: "scraper.perplexity", input: {prompt, country, web_search}}, read result_text for the answer, and normalize web_results into citation records.

From there, citation monitoring is standard data engineering. Preserve the raw response, normalize domains, track positions, and compare fixed prompt panels over time. You can create an API key in the Scrapeless dashboard.

FAQ

Does every Perplexity response contain citations?

No. The source panels depend on the query and answer path. Code should accept an empty web_results array.

Can I collect images and maps?

When Perplexity returns rich media, media_items can include media type, URLs, thumbnails, and location data. Treat those fields as optional.

Why should I store the prompt in every row?

The prompt is part of the measurement. Without the exact input, two answers cannot be compared reliably.

Is scraping Perplexity answers legal?

The actor collects publicly rendered answer content. Review applicable laws and Perplexity's terms, and consult counsel before collecting or redistributing data at scale.

How Product Teams Analyze Customer Questions Across AI Models

Ethan Walker — Thu, 20 Aug 2026 10:11:07 +0000

TL;DR:

AI models are the new search engines for product queries. Customers increasingly ask ChatGPT, Gemini, and Perplexity about software features, pricing, and integrations instead of navigating traditional documentation.
Question mining reveals hidden friction points. Analyzing the exact phrasing users employ when querying LLMs exposes gaps in product onboarding and UX that internal analytics often miss.
Different models surface different user intents. Copilot queries skew heavily toward enterprise integration and security, while ChatGPT prompts often focus on basic workflow automation and troubleshooting.
A structured taxonomy turns raw text into roadmap priorities. Categorizing AI-generated questions by feature area and sentiment allows product managers to quantify qualitative feedback at scale.
Free to start. New Scrapeless accounts include trial API credits — sign up at app.scrapeless.com.

The Shift from Search Bars to Chat Interfaces

Product discovery and troubleshooting have fundamentally migrated from keyword searches to conversational prompts. Users no longer type "export data CSV error" into a help center; they paste an error log into an LLM and ask for a fix. This behavioral shift means the most valuable qualitative data about a product now lives inside the context windows of major AI models. Product teams relying solely on traditional web analytics or support tickets are missing the raw, unfiltered questions users ask when they are stuck, evaluating alternatives, or trying to stretch a product beyond its documented capabilities.

The friction lies in visibility. While companies own their internal search logs and support desk data, they have zero direct access to the prompts users feed into third-party AI systems. When a potential enterprise customer asks an LLM to compare two SaaS platforms, the resulting conversation happens in a black box. Product managers need a systematic way to extract, aggregate, and analyze these external interactions to understand what the market actually wants to know about their tools.

This post covers the mechanics of question mining from LLM outputs, the distinct query patterns across different AI platforms, and how to build a feedback loop that directly informs product roadmaps. For a broader look at how AI visibility impacts brand presence, see our guide on brand AI visibility in Google AI Overviews.

The Mechanics of Question Mining from LLM Outputs

Extracting actionable product insights from AI models requires treating LLMs as dynamic user research panels rather than static knowledge bases. Question mining involves systematically querying models with variations of product-related prompts to observe what follow-up questions, clarifications, or related topics the AI suggests. Because models are trained on vast amounts of human interaction data, their suggested questions and conversational pathways closely mirror actual user confusion and curiosity.

Scrapeless Scraping API provides the infrastructure to collect this data at scale, handling the anti-bot protections and JavaScript rendering that make manual monitoring impractical.

The process begins with seed queries. A product team might input a basic prompt like "How do I configure SSO in [Product Name]?" and analyze the model's response structure. More importantly, they analyze the "related questions" or conversational branches the model anticipates. If the AI consistently suggests follow-up questions about SAML certificate rotation or specific identity providers like Okta, it indicates a high volume of historical user friction around those specific sub-topics. This methodology aligns with findings from Nielsen Norman Group's research on AI in UX, which highlights the utility of LLMs in identifying edge cases and user mental models.

To scale this, teams use automated systems to query multiple models simultaneously. The Scrapeless Scraping API, specifically configured for LLM Chat Scraper actors, allows product operations to programmatically submit thousands of prompt variations across different AI platforms and extract the resulting text structures. This transforms a manual, anecdotal research task into a continuous data pipeline.

Mining Stage	Objective	Output Metric
Seed Generation	Establish baseline queries based on core features	List of 50-100 primary prompts
Automated Querying	Execute prompts across ChatGPT, Gemini, Claude	Raw response JSON and suggested follow-ups
Entity Extraction	Identify specific features, integrations, or errors mentioned	Frequency count of product entities
Intent Classification	Categorize questions (e.g., troubleshooting, pricing, comparison )	Percentage distribution of user intent

Start Scraping with Scrapeless

Power up your web scraping and automation workflow with Scrapeless!

Sign up today and get $5 in free credit — no credit card required.

Claim your free credit now in the Scrapeless Dashboard.

How Different AI Models Surface Different Pain Points

Not all AI models process product queries the same way, and the differences in their training data and user interfaces lead to distinct types of customer questions. A comprehensive question mining strategy must account for these platform-specific biases to build an accurate picture of user needs.

ChatGPT, given its massive general consumer and prosumer user base, tends to surface questions related to initial onboarding, basic workflow automation, and integrations with common productivity tools. Queries here often take the form of "How do I connect X to Y?" or "Why is this feature not working?" The responses and subsequent suggested questions reflect a user base looking for immediate, tactical solutions.

Conversely, models embedded in enterprise ecosystems, such as Microsoft Copilot, exhibit entirely different query patterns. Users interacting with Copilot are often operating within strict corporate environments, leading to questions heavily skewed toward security, compliance, data residency, and complex enterprise integrations. A product manager analyzing Copilot outputs will find questions about SOC2 compliance or active directory syncing that rarely appear in ChatGPT logs. Similarly, Perplexity, which functions more as an answer engine with real-time web access, surfaces questions related to recent product updates, pricing changes, and direct competitor comparisons. Understanding these nuances is critical, as noted in Gartner's analysis of generative AI adoption, which emphasizes the fragmentation of AI use cases across different enterprise personas.

Building a Question Taxonomy for Roadmap Prioritization

Raw text data from LLMs is useless without a structured framework to categorize it. Building a question taxonomy involves classifying the mined questions into a hierarchy that maps directly to the product architecture and business objectives. This taxonomy translates qualitative curiosity into quantitative metrics that can justify engineering resources.

The first layer of the taxonomy typically categorizes questions by product area or specific feature modules. If a SaaS platform has modules for reporting, user management, and billing, every extracted question is tagged accordingly. The second layer classifies the intent behind the question: Is the user trying to evaluate the product (pre-sales ), configure a feature (onboarding), or fix a broken workflow (troubleshooting)?

Once categorized, product teams can apply sentiment analysis and volume metrics to prioritize the roadmap. A sudden spike in troubleshooting questions related to a newly released reporting feature, detected across multiple AI models, acts as an early warning system for UX issues. This data provides a more immediate signal than waiting for support tickets to accumulate. By integrating this taxonomy with existing product management tools, teams can attach concrete "AI query volume" metrics to Jira epics or product requirement documents.

Get your API key on the free plan: app.scrapeless.com

The Feedback Loop Between AI Answers and Product Documentation

The relationship between product documentation and AI models is cyclical. LLMs generate answers based on the public documentation they ingest; users ask questions based on the gaps in those answers; and product teams must update the documentation based on those questions. Closing this loop is essential for maintaining control over how a product is represented in the AI ecosystem.

When question mining reveals that an AI model is consistently providing outdated or incorrect answers about a specific feature, it usually indicates that the official documentation is either poorly structured, lacking semantic clarity, or buried behind client-side rendering that AI crawlers cannot parse. The W3C Data on the Web Best Practices provides foundational guidelines for structuring information so that both humans and machines can accurately interpret it.

Product teams must treat AI models as a distinct audience for their documentation. This means writing clear, declarative sentences, using structured data formats, and explicitly addressing the common questions surfaced during the mining process. When the documentation is updated to directly answer the questions users are asking LLMs, the models eventually ingest the new information, improving the accuracy of future responses and reducing user friction. This proactive approach to documentation ensures that the product narrative remains accurate across all conversational interfaces.

Scaling the Analysis with Scraper API

Manually querying multiple AI models to extract customer questions is not a sustainable practice for growing product teams. The sheer volume of prompt variations, model updates, and conversational branches requires an automated infrastructure capable of handling complex interactions at scale.

The Scrapeless Scraper API provides the necessary architecture to automate this entire workflow. By using specialized LLM Chat Scraper actors, teams can programmatically interface with various AI models, submit structured prompt sequences, and extract the resulting text, suggested questions, and conversational metadata in clean JSON formats. This eliminates the need to build and maintain custom browser automation scripts or manage complex session states across different AI platforms.

This automated pipeline allows product operations to run continuous monitoring jobs. Instead of a one-off research project, question mining becomes a daily data feed, integrating directly into data warehouses or product analytics dashboards. This continuous visibility ensures that product managers are always aware of the evolving questions customers are asking, enabling them to adapt roadmaps and documentation with unprecedented agility.

Beyond that, the integration of these automated pipelines allows for historical trend analysis. Product teams can track how the volume and nature of questions change over time, particularly before and after major product releases or documentation overhauls. This longitudinal data provides concrete evidence of whether a specific UX improvement actually reduced user confusion in the wild. The ability to measure the impact of product changes through the lens of AI interactions represents a significant advancement in product analytics.

Explore Scrapeless pricing to find the plan that fits your monitoring needs.

For a deeper look at how traditional SEO and generative engine optimization compare, see our analysis on best LLM scrapers.

Ready to Build Your AI-Powered Product Analysis Pipeline?

Join our community to claim a free plan and connect with developers building product analysis pipelines: Discord · Telegram.

FAQ

Q: How does question mining differ from traditional keyword research?
Traditional keyword research focuses on isolated search terms typed into engines like Google, while question mining analyzes the complete, conversational sentences and follow-up prompts users feed into LLMs, providing much deeper context about user intent and specific friction points.

Q: Can we extract questions directly from our users' ChatGPT sessions?
No, user sessions with third-party AI models are private. Question mining relies on simulating user queries and analyzing the AI's responses, suggested follow-ups, and conversational pathways to infer the common questions the model has been trained to anticipate.

Q: Which AI models should product teams monitor?
Teams should monitor a mix of models that reflect their user base; ChatGPT is essential for general SaaS, Copilot is critical for enterprise and Microsoft ecosystem products, and Perplexity is valuable for tracking competitor comparisons and recent feature inquiries.

Q: How often should we update our product documentation based on AI queries?
Documentation should be updated continuously as part of a feedback loop; whenever question mining reveals that AI models are providing outdated answers or struggling with specific concepts, the corresponding documentation should be revised immediately to correct the model's future outputs.

How Publishers Track Citations in Perplexity and AI Overviews

Ethan Walker — Thu, 20 Aug 2026 09:34:34 +0000

TL;DR:

AI answer engines are fundamentally altering publisher traffic models. As platforms like Perplexity and Google AI Overviews synthesize answers directly in the search interface, traditional click-through rates are declining while the importance of being cited as a source increases.
Citation attribution is inconsistent across platforms. While some AI engines provide clear inline links and footnote references, others obscure sources or bury them in expandable menus, making it difficult for publishers to measure their true reach.
Tracking AI citations requires specialized data extraction. Standard web analytics tools cannot capture when content is referenced in an AI response; publishers must actively monitor the output of these engines using automated scraping and API integrations.
Citation data directly informs content strategy and licensing. By understanding which topics trigger AI citations and how their content is used, publishers can optimize their editorial focus and negotiate better terms in data licensing agreements.
Free to start. New Scrapeless accounts include trial API credits — sign up at app.scrapeless.com.

The Shift from Clicks to Citations

The fundamental currency of the web is changing from the hyperlink to the synthesized answer. Search engines and AI assistants no longer exist merely to route users to publisher websites; they increasingly aim to resolve user queries directly on the results page. This shift forces publishers to adapt to a reality where their content is consumed without generating a direct pageview.

For decades, the publisher business model relied on a straightforward exchange: provide valuable information, rank high in search results, and monetize the resulting traffic through advertising or subscriptions. The introduction of generative AI into search interfaces disrupts this equation. When an AI engine reads a publisher's article, extracts the relevant facts, and presents them to the user, the publisher provides the value but often loses the traffic.

To survive and thrive in this new environment, publishers must develop mechanisms to track when and how their content is used by AI systems. This requires moving beyond traditional web analytics and building systems capable of monitoring the outputs of platforms like Perplexity and Google AI Overviews. By understanding their citation footprint, publishers can defend their intellectual property, optimize their content for AI discovery, and establish the value of their data in licensing negotiations.

The Mechanics of AI Citation Attribution

AI answer engines do not cite sources uniformly. The methods used to attribute information vary significantly between platforms and even between different types of queries on the same platform. Understanding these variations is the first step in building an effective tracking strategy.

Scrapeless Deep SerpApi provides the infrastructure to collect this data at scale, handling the anti-bot protections and JavaScript rendering that make manual monitoring impractical.

Some platforms, such as Perplexity, have built their interfaces around explicit citation. They often include numbered footnotes within the generated text that link directly to the source material. This approach, while still reducing direct traffic compared to traditional search, provides a clear signal of attribution that publishers can track. However, the prominence of these links and the likelihood of a user clicking them remain subjects of ongoing debate within the publishing industry.

Other systems, including early iterations of Google AI Overviews, have experimented with different attribution models. These might include listing sources at the bottom of the response, embedding links within the text without explicit footnotes, or requiring the user to click an expansion icon to view the references. The lack of standardization makes it challenging for publishers to develop a unified approach to citation tracking. According to W3C web annotation standards, consistent attribution models are essential for maintaining trust and traceability in digital ecosystems, yet AI platforms often prioritize user experience over strict adherence to these principles.

Beyond that, the nature of the citation itself can vary. An AI engine might quote a source directly, paraphrase its findings, or synthesize information from multiple sources into a single, unattributed statement. Tracking direct quotes is relatively straightforward, but identifying paraphrased content requires more sophisticated analysis. Publishers must employ techniques such as semantic similarity matching to determine when their unique insights or proprietary data have been incorporated into an AI response without explicit credit.

Why Publishers Must Monitor AI Citations

The imperative to track AI citations extends beyond mere curiosity; it is a critical component of modern digital publishing strategy. The data gathered from citation monitoring informs decisions across editorial, technical, and business development teams.

First and foremost, citation tracking provides a measure of reach and influence in the AI era. If a publisher's content is frequently cited by AI engines, it indicates that the platform's algorithms consider the publisher to be an authoritative source on those topics. This visibility, even if it does not immediately translate into traffic, is valuable for brand building and establishing thought leadership. Conversely, a lack of citations may signal that a publisher's content is not optimized for AI discovery or that the platform favors competing sources.

Secondly, monitoring citations is essential for protecting intellectual property and negotiating data licensing agreements. As AI companies increasingly seek to train their models on high-quality, proprietary data, publishers need concrete evidence of the value their content provides. By demonstrating how frequently their articles are used to generate answers, publishers can negotiate more favorable terms in licensing deals. The U.S. Copyright Office guidelines on artificial intelligence highlight the complex legal landscape surrounding AI training data, making empirical evidence of usage crucial for publishers seeking compensation.

Finally, citation data can directly inform content strategy. By analyzing which articles and topics generate the most citations, editorial teams can identify areas of high demand and adjust their coverage accordingly. For example, if a publisher notices that its in-depth technical explainers are frequently cited by Perplexity, it might choose to invest more resources in producing that type of content. This data-driven approach ensures that editorial efforts are aligned with the evolving ways in which users consume information.

Get your API key on the free plan: app.scrapeless.com

Methods for Systematic Citation Tracking

Tracking AI citations at scale requires automated systems capable of querying AI engines, extracting the responses, and analyzing the attribution data. This process involves several technical challenges, including managing API access, parsing complex HTML structures, and handling the dynamic nature of AI-generated content.

The most direct method for tracking citations is to use the official APIs provided by the AI platforms, when available. However, these APIs are often designed for developers building applications, not for publishers monitoring their content. They may lack the specific endpoints needed to extract citation data or impose rate limits that make large-scale monitoring impractical. Beyond that, not all platforms offer public APIs, forcing publishers to rely on alternative methods.

When official APIs are insufficient or unavailable, publishers often turn to web scraping techniques to extract citation data directly from the search interface. This involves writing scripts that simulate user queries, render the results page, and parse the HTML to identify links and references. This approach requires sophisticated tools capable of handling JavaScript rendering, CAPTCHAs, and frequent changes to the platform's DOM structure. The IETF specifications for robots exclusion protocol provide guidelines for automated access, but the dynamic nature of AI interfaces often necessitates more advanced extraction techniques.

To overcome these challenges, many publishers use specialized data extraction services. These services provide robust APIs that handle the complexities of web scraping, allowing publishers to focus on analyzing the data rather than maintaining the extraction infrastructure. For example, tools designed specifically for monitoring search engine results pages (SERPs ) can be configured to track the presence of a publisher's domain within AI-generated answers.

Tracking Method	Advantages	Disadvantages
Official APIs	Reliable, structured data	Often unavailable, rate-limited, may lack citation specifics
Custom Web Scraping	Highly customizable, captures exact user experience	High maintenance, requires handling CAPTCHAs and DOM changes
Specialized Extraction Services	Scalable, handles technical complexities, robust APIs	Requires integration, relies on third-party infrastructure

Start Scraping with Scrapeless

Power up your web scraping and automation workflow with Scrapeless!

Sign up today and get $5 in free credit — no credit card required.

Claim your free credit now in the Scrapeless Dashboard.

Leveraging Scrapeless for Citation Monitoring

Building and maintaining a custom infrastructure for tracking AI citations is a resource-intensive endeavor. The constant evolution of AI interfaces and the technical hurdles associated with large-scale data extraction make it difficult for publishers to develop reliable monitoring systems in-house. This is where specialized platforms like Scrapeless provide significant value.

Scrapeless offers a suite of tools designed to simplify the process of extracting data from complex web environments, including AI answer engines. By using the Deep SerpApi, publishers can programmatically query Google and extract the contents of AI Overviews, including the sources cited within the generated text. This allows publishers to monitor their visibility in Google's AI features without the need to build and maintain custom scraping scripts.

For tracking citations on platforms like Perplexity, publishers can leverage the Scraping API. This tool handles the complexities of JavaScript rendering and anti-bot mechanisms, enabling reliable extraction of citation data from dynamic interfaces. By integrating these APIs into their analytics pipelines, publishers can automate the collection of citation data and gain real-time insights into how their content is being used across the AI ecosystem.

Beyond that, understanding the broader context of AI visibility is crucial. Publishers should consider how their overall search strategy aligns with the demands of AI engines. For a deeper dive into this topic, explore our analysis on GEO vs SEO, which examines the differences between traditional search engine optimization and generative engine optimization.

Adapting to the AI-Driven Information Ecosystem

The integration of generative AI into search and discovery platforms represents a fundamental shift in how information is consumed on the internet. Publishers can no longer rely solely on traditional metrics like pageviews and click-through rates to measure their success. They must adapt to an environment where their content is frequently synthesized and presented directly to the user, often without generating a direct visit to their website.

Tracking AI citations is a critical first step in navigating this new landscape. By understanding when, where, and how their content is being used, publishers can protect their intellectual property, optimize their editorial strategies, and demonstrate the value of their data in licensing negotiations. The tools and techniques required for effective citation monitoring are complex, but the insights they provide are essential for long-term survival.

As AI platforms continue to evolve, the methods for tracking citations will also need to adapt. Publishers must remain vigilant, continuously monitoring changes to attribution models and updating their extraction strategies accordingly. By embracing data-driven approaches and leveraging specialized extraction tools, publishers can ensure that their voices continue to be heard and valued in the AI-driven information ecosystem.

Explore Scrapeless pricing to find the plan that fits your monitoring needs.

--- For a deeper look at how traditional SEO and generative engine optimization compare, see our analysis on GEO vs SEO.

Ready to Build Your AI-Powered Citation Tracking Pipeline?

Join our community to claim a free plan and connect with developers building citation tracking pipelines: Discord · Telegram.

FAQ

Q: Why is tracking AI citations important for publishers?
Tracking AI citations allows publishers to measure their reach and influence in AI answer engines, protect their intellectual property, and gather empirical data to support content licensing negotiations.

Q: Can traditional web analytics tools track AI citations?
Standard web analytics tools cannot track AI citations because they rely on users clicking a link and loading a page on the publisher's website, whereas AI engines often synthesize answers directly in the search interface.

Q: What are the main challenges in tracking AI citations?
The primary challenges include the lack of standardized attribution models across different AI platforms, the difficulty of extracting data from dynamic, JavaScript-heavy interfaces, and the need to handle anti-bot mechanisms.

Q: How can publishers automate the collection of citation data?
Publishers can automate data collection by using specialized data extraction services and APIs, such as the Scrapeless Deep SerpApi, which are designed to handle the complexities of querying AI engines and parsing the results.

Q: Does being cited by an AI engine guarantee traffic to a publisher's website?
Being cited by an AI engine does not guarantee traffic, as users may find the synthesized answer sufficient and choose not to click through to the source material, highlighting the need for publishers to track citations as a distinct metric.

DEV Community: Ethan Walker

Separate Pending Tasks from Empty Google Search Results

Start With the API's Documented Outcome

Preserve a Capture Before Transforming It

Give Unknown Counts a Different Value From Zero

Run the Local State Classifier

Store the Task Identifier Without Inventing a Retrieval Route

Keep Collection Coverage Separate From Search Findings

Test the Decisions That Change the Meaning of a Report

Conclusion

FAQ

Add Source Discovery to an AI Research Workflow

TL;DR:

Define the Discovery Contract

Prerequisites and Request Parameters

Preserve the Response State Before Reading Results

Build the Local Candidate Adapter

Select Sources and Retrieve Content Separately

Connect Claims to Reviewed Passages

Check the Adapter Before Connecting an Agent

Conclusion

FAQ

Keep Search Context When Exporting SERP JSON to CSV

Start With a Capture Envelope

Decide Which Meaning Each Column Preserves

Use a CSV Writer, Not String Concatenation

Make Spreadsheet Interpretation Explicit

Run the Local Exporter

Inspect Empty and Malformed Results

Check the Handoff, Not Only the Row Count

Conclusion

FAQ

Model Google SERP Snapshots in SQLite

Define the Capture Contract

Separate Runs From Organic Rows

Use the Whole Request as the Context Key

Import Captures With One Transaction Per Run

Keep Empty and Unavailable Outcomes Visible

Compare Domain Sets Within One Context

Query Coverage Before Building a Trend

Conclusion

FAQ

Validate Google Search API Inputs Before Sending a Request

Separate the API Contract From Your Application Policy

Choose One Input Mode

Define a Reviewed Parameter Allowlist

Install Nothing and Run the Complete Validator

Check URL Structure Without Claiming Complete URL Validation

Keep Validation From Changing the Query

Exercise the Failure Paths

Put the Validator Before Authentication and Submission

Conclusion

FAQ

Build a Google Search API JSON Workflow with Python

TL;DR:

What the Google Search API Returns

Prepare a Small First Request

Choose Country, Language, and Input Mode

Request JSON and Export Organic Results

Interpret the Response Before Building a Report

Put Structured Search Data to Work

Conclusion

FAQ

Flatten TikTok Shop Product JSON into Product and SKU Tables

Start with the product hierarchy

Prerequisites

Define explicit schemas

Normalize values without erasing uncertainty

Build the product and SKU rows

Write stable CSV files

Preserve price context

Validate the transformation

Conclusion

FAQ

Why not write one row per product?

Should an absent SKU array be treated as an error?

Can sold_count be used as recent sales velocity?

Is collecting public Shop data legal?

From TikTok Username to Post Metrics: A Python API Workflow

What the two requests accomplish

Can `sold_count` be used as recent sales velocity?

Why does the post actor need `sec_uid`?