DEV Community: Elowen

Batch SERP Requests with Retry, Status Tracking, and Audit Logs

Elowen — Wed, 02 Sep 2026 01:52:21 +0000

A batch of 100 search requests rarely succeeds all at once. Network jitter, rate limits, and transient errors produce a mix of successful and failed responses. Treating failures as exceptions to abort the entire batch loses data and hides patterns.

Start with a request record that captures each attempt:

{
  "request_id": "batch-20260902-001",
  "query": "salesforce revops trends",
  "engine": "google",
  "location": "United States",
  "device": "desktop",
  "attempt": 1,
  "status": "pending",
  "queued_at": "2026-09-02T08:00:00Z"
}

The values above are illustrative. A production system should persist each request before execution and update status after each attempt.

Define four statuses:

pending: request queued, not yet sent
in_progress: request sent, waiting for response
success: response received and parsed
failed: error after max retries

A retry policy with exponential backoff handles transient errors. This pseudocode illustrates the pattern:

max_retries = 3
base_delay = 1  # second

for attempt in range(1, max_retries + 1):
    update_status(request_id, "in_progress", attempt)
    try:
        response = serp_request(query, location, device)
        save_response(request_id, response)
        update_status(request_id, "success", attempt)
        break
    except TransientError as e:
        if attempt == max_retries:
            update_status(request_id, "failed", attempt, error=str(e))
        else:
            wait(base_delay * (2 ** (attempt - 1)))

The code above is a generic illustration. Replace TransientError with the actual exception types from your HTTP client.

For the batch as a whole, maintain a summary:

{
  "batch_id": "batch-20260902",
  "total": 100,
  "success": 94,
  "failed": 6,
  "failure_reasons": {
    "timeout": 4,
    "rate_limit": 2
  },
  "started_at": "2026-09-02T08:00:00Z",
  "completed_at": "2026-09-02T08:15:00Z"
}

The summary enables audit, retry of only failed items, and a reliable report. Without it, a batch of 94 successes and 6 failures looks like a binary pass-or-fail. With it, you can investigate the 6 failures, retry them with adjusted parameters, and explain the data quality to stakeholders.

The TalorData API returns a consistent response structure for both successful and failed requests. The failure details appear in the response metadata. Capture both so your pipeline can differentiate between an empty search result and a request that never completed.

SERP API Vendor Selection in 2026 – A CTO‘s Decision Framework

Elowen — Tue, 01 Sep 2026 09:20:02 +0000

SERP APIs look simple on the surface: send a query, get results. But beneath that simplicity lies a complex vendor selection problem that CTOs and engineering leaders routinely underestimate.

By the time you factor in pricing models, data quality, latency characteristics, vendor lock-in risks, and the true cost of integration, the "obvious" choice often turns out to be anything but.

This framework breaks down exactly what to evaluate when selecting a SERP API vendor in 2026 — and why the cheapest headline price is almost never the actual cost.

Why SERP API Selection Deserves a Framework

Most teams choose a SERP API the same way: they Google "best SERP API," skim a few comparison posts, pick the one with the lowest price, and move on.

This approach fails because every provider quotes a different unit:

SerpApi quotes monthly plan cost
DataForSEO quotes per-request (asynchronous)
Bright Data quotes per-result (one search = 10 results = 10 billable units)
ScraperAPI quotes per-credit (1 Google search = 25 credits)
Talordata quotes per-successful-response

Until you convert everything to the same unit — cost per 1,000 successful searches at your actual volume — you cannot compare them.

The Three Pricing Models Every CTO Must Understand

1. Per-Request Markup

The API adds a markup to the base cost of each request. Hidden cost: margins get squeezed at scale.

2. Monthly Subscription + Overage

A fixed fee plus usage overage. Hidden cost: you pay for unused capacity. If your usage is inconsistent — seasonal campaigns, project-based workloads, quarterly audits — you're paying for credits you never use. At 60% average utilisation on a $75/month SerpApi plan, your effective cost jumps from $15/1K to $25/1K.

3. Pay-Per-Success

You pay only for successful responses. Failed requests (timeouts, empty results, network issues) are completely free.

Why this matters: Pay-per-success aligns vendor incentives with your outcomes. The vendor is financially motivated to return valid data — not just to process your request and bill you regardless of result.

The Hidden Costs Most Vendors Don't Mention

Hidden Cost	Example	Impact
Credit expiry	SerpApi credits reset monthly	60% utilisation = $25/1K effective, not $15/1K
Credit multipliers	ScraperAPI: 1 Google search = 25 credits	$49/month plan = 4,000 actual searches = $12.25/1K
Per-result billing	Bright Data bills per result, not per search	1 search × 10 results = 10 billable units
Asynchronous overhead	DataForSEO requires polling + retry logic	Engineering time + complexity = hidden cost
Plan commitment	Monthly subscriptions with minimum spend	Paying for capacity you don't use

What "Data Quality" Actually Means in Production

Price is easy to compare. Data quality is not — but it matters more.

A 1,009-query benchmark comparing six major SERP API providers (Talordata, Serpapi, Bright Data, Oxylabs, bd_fast, Serper) revealed significant differences that don't appear on any pricing page:

Overall Ranking

Rank	Platform	Overall Score	Majority Alignment	SERP Feature Coverage
1	Talordata	79.19	77.20%	36.20%
2	Serpapi	78.06	75.01%	36.70%
3	Bright Data	77.80	78.43%	28.60%
4	Oxylabs	69.83	80.72%	5.40%

The top three platforms are very close: Talordata scores 79.19, Serpapi 78.06, and Bright Data 77.80, with only 1.39 points separating them.

Result Completeness

Platform	Total Organic Rows	Avg Results/Query	Zero-Result Rate
Talordata	8,931	8.85	2.08%
Serpapi	8,908	8.83	2.28%
Bright Data	8,867	8.79	2.38%
Serper	6,942	6.88	12.59%

Talordata returns the largest number of organic rows: 8,931. Serper returns only 6,942 — 1,989 fewer rows.

Why this matters for your AI pipeline: If your SERP API returns 22.3% fewer organic results per query, your RAG system has less context, your rank tracking is noisier, and your AI agents miss relevant pages.

Core Field Completeness

All six platforms perform strongly on core organic fields — title, link, snippet, position. The differentiator is in SERP feature coverage: Talordata covers 36.20% of SERP features vs Bright Data's 28.60% and Oxylabs' 5.40%.

Latency: The User Experience Tax

If a user is waiting for results, asynchronous APIs are wrong architecture.

Talordata delivers P90 response times under 1 second, backed by a 99.9% request success rate and automatically rotating proxy pools.

Compare this to asynchronous providers like DataForSEO: you submit a task, wait 30 seconds, poll for results, handle retry logic, and manage timeouts. That's not just latency — it's engineering complexity that adds to your total cost of ownership.

A Practical Decision Framework

When evaluating SERP API vendors, assess each provider against these five criteria:

1. Pricing Transparency

Can you calculate your monthly bill in under 60 seconds?
Are there hidden multipliers, credit expiry, or minimum commitments?
Does the vendor charge for failed requests?

Talordata: 1:1 billing, one request = one credit. No multipliers. Credits that expire are automatically restored when you buy any new plan — prior spend isn't wasted. Overage requests are rejected rather than billed — no surprise charges.

2. Data Completeness

How many organic results does the API return per query on average?
What's the zero-result rate?
Does it parse AI Overviews, knowledge panels, and other SERP features?

Talordata: Largest organic row count (8,931), 2.08% zero-result rate, AI Overview support.

3. Engine Coverage

Does the API support only Google, or multiple engines?
Do you need Bing, Yandek, or regional engines for your use case?

Talordata: Google, Bing, Yandex, and DuckDuckGo through a single endpoint.

4. Integration Overhead

Is the API synchronous or asynchronous?
Does it require complex state management?
Are there official SDKs for your stack?

Talordata: Synchronous, sub-second responses, official Python SDK, and LangChain integration.

5. Vendor Lock-in Risk

Can you migrate your code to another provider easily?
Does the API use standard parameter patterns?

Talordata: Interface design is compatible with SerpApi specifications — same parameters, making migration straightforward.

The Bottom Line

Pricing isn't just about being cheaper — it communicates what a vendor is willing to stand behind. If developers can predict their bill in five seconds, that's a competitive advantage on its own.

When evaluating SERP API vendors in 2026:

Don't compare headline prices — convert everything to cost per 1,000 successful searches at your actual volume
Don't ignore data quality — a 2% zero-result rate vs 12% means thousands of failed queries per month
Don't underestimate latency — async APIs add complexity and degrade user experience
Do the math on credit expiry — if you're only using 60% of your plan, your effective cost is 40% higher than advertised

The vendor with the lowest headline price is rarely the vendor with the lowest total cost of ownership.

Freshness, Completeness, and Consistency: Three Quality Checks for SERP Data

Elowen — Mon, 31 Aug 2026 01:53:30 +0000

A SERP dataset can look tidy while still leading a team to the wrong conclusion. Three checks make the failure easier to name: freshness, completeness, and consistency. Each check answers a different question, so each needs a different test.

1. Freshness: how current is the observation?

Freshness concerns the time gap between the search event and the moment the data is used. A common failure is a daily rank monitor reading yesterday's cached response after a campaign or landing page changed. The rows are valid records; they describe an earlier state.

Practical checks

Store captured_at in UTC for every response.
Compare the newest capture with the reporting window.
Set a maximum age for each use case, such as 15 minutes for an alert or 24 hours for a weekly trend.
Flag records that exceed the limit instead of silently mixing them with current observations.

Freshness checklist

[ ] Does every record have a capture time?
[ ] Is the time zone unambiguous?
[ ] Is the age threshold written down?
[ ] Are stale rows visible in the output?

2. Completeness: what expected pieces are missing?

Completeness asks whether the dataset contains the observations and fields needed for its stated purpose. A location report may include ten queries for one city while the other cities have eight. A parser may keep organic results while dropping a present People Also Ask section. Both cases create gaps that can look like real search differences.

Practical checks

Define the expected query, location, device, and capture-time combinations before collection.
Count received records against that expected set.
Check required fields and explicitly track optional sections.
Report missing combinations and fields as gaps with reasons when known.

Completeness checklist

[ ] Is there an expected observation list?
[ ] Do counts match by query, location, and device?
[ ] Are required fields present?
[ ] Are absent SERP sections distinguishable from empty sections?

3. Consistency: do comparable records follow the same rules?

Consistency concerns comparability across records and runs. A rank comparison becomes unstable when one batch uses a different device, parser version, or position convention. The values may each be internally valid, yet the calculation combines different definitions.

Practical checks

Keep request settings beside the parsed result, including device, location, language, and domain.
Version the normalization code and record the version used.
Apply one position convention, such as one-based organic rank, across the dataset.
Re-run a small fixture through each parser release and compare the normalized output.

Consistency checklist

[ ] Are comparison dimensions identical?
[ ] Is the position definition documented?
[ ] Is the parser or schema version recorded?
[ ] Do fixture results stay stable across releases?

Use the checks together

Run the checks in sequence: freshness tells you when the observation was true, completeness tells you what is present, and consistency tells you whether records can be compared. A quality report can then label each issue clearly instead of collapsing every gap into a generic error count.

These checks work with any SERP collection method. The important design choice is to make the expected inputs, timestamps, and interpretation rules visible alongside the result.

How to Estimate SERP API Requests Before You Build a Monitor

Elowen — Fri, 28 Aug 2026 02:08:51 +0000

Before building a SERP monitor, estimate how many requests the schedule will create. The core calculation is simple: queries × markets × devices × runs.

This guide turns that calculation into a small planning routine. The numbers are hypothetical examples for sizing a project; they are not a report from a live system.

Start with the request dimensions

List every dimension that creates a distinct SERP request:

Queries: the phrases or query variants you track.
Markets: countries, cities, or search locations.
Devices: desktop, mobile, or other supported devices.
Runs: how often the monitor executes during the planning period.

If one query runs in three markets on two devices, one scheduled pass creates six requests. Add the run frequency only after the per-pass count is clear.

Work through a hypothetical estimate

Suppose a monitor has 40 queries, 3 markets, 2 devices, and 4 weekly runs. The estimate is:

40 queries × 3 markets × 2 devices × 4 runs per week = 960 requests per week

This arithmetic is a generic sizing example. Replace each input with the scope your team has approved.

Keep the calculation in code

A small function makes the assumptions visible in a pull request or planning document:

function estimateWeeklyRequests({ queries, markets, devices, runsPerWeek }) {
  const values = [queries, markets, devices, runsPerWeek];
  if (values.some((value) => !Number.isInteger(value) || value < 1)) {
    throw new Error('All request dimensions must be positive integers');
  }
  return queries * markets * devices * runsPerWeek;
}

const estimate = estimateWeeklyRequests({
  queries: 40,
  markets: 3,
  devices: 2,
  runsPerWeek: 4
});

console.log(estimate); // 960 (hypothetical estimate)

The function calculates volume only. Authentication, response handling, and analysis belong in separate parts of the monitor.

Turn volume into batches

Use the estimate to choose a schedule that people can operate:

Planning case	Queries	Markets	Devices	Weekly runs	Estimated requests
Small weekly report	25	1	1	1	25
Regional monitor	40	3	2	4	960
Multi-market daily check	80	5	2	7	5,600

These rows are hypothetical planning examples. They show how each dimension changes the total; they say nothing about provider limits or actual spend.

After estimating, group work by a stable unit such as market and device. Give each batch an owner, a run window, and an expected request count. This request total provides the volume input for a separate budget estimate. Actual budget depends on the provider's pricing model, plan, and any extra charges.

A change to any input should trigger a fresh request estimate before the schedule changes.

The planning checklist

Write the query count and the reason each query exists.
List markets and devices as explicit values.
Choose the run frequency for the decision the monitor supports.
Calculate the total for one planning period.
Divide the work into batches with an owner and a window.

This five-minute estimate gives a monitor a visible operating shape before implementation begins.

SERP API or Web Scraping? A Practical Decision Guide for Search Data Projects

Elowen — Thu, 27 Aug 2026 02:43:49 +0000

SERP API and web scraping can both bring search data into a project. The useful choice depends on the job: collecting a few search-query result sets has different constraints from a monitoring service or a customer-facing application.

This guide compares three common scenarios. The examples are generic decision examples, so they describe trade-offs rather than measured results.

Scenario 1: One-time information extraction

Suppose a researcher needs the results for a few search queries in a single report. A SERP API can return structured result data for those requests. Scraping a search results page can also work for a permitted, one-time task when the layout is known and the output will be used once.

The scraper still needs selector code, HTML cleanup, rendering decisions, and a check against the relevant terms and robots guidance. Search result pages can change between requests, so the report workflow needs a quick check that the extracted titles, URLs, and result types are complete.

An API is useful here when the report needs consistent fields, several markets, or a second pass later. The extra setup can pay off when the request will be repeated or compared.

Scenario 2: Ongoing rank monitoring

Now consider a hypothetical weekly check for a set of queries. A scraper must keep up with layout changes, consent screens, JavaScript rendering, rate limits, and anti-bot controls. Each change can turn into an investigation before the monitoring job produces a usable row.

An API gives the job explicit request parameters and a response shape designed for data use. The team can focus maintenance on query definitions, scheduling, and alert rules. Stability still depends on the provider contract and the way the team handles changes.

For recurring monitoring, compare the expected maintenance hours with the cost of an API request. Include the time spent detecting silent parser drift, since a job that returns incomplete data can look healthy.

Scenario 3: An application that needs stable structured data

Imagine a product that stores search results for an internal dashboard. The application needs predictable fields, validation, and a clear way to handle a missing value. An API usually fits this shape because the request and response can become part of the application contract.

Scraping can support the same product, yet it requires a parser, fixtures, layout-change tests, rendering support, and an escalation path when a target page changes. The product team also needs a documented permission and data-licensing review for every source it accesses.

A compact decision table

Project need	API fit	Scraping burden	Decision signal
One permitted, one-time extraction	Useful when structured fields or repeatability matter	Low initial effort, limited reuse	Choose the smallest compliant tool
Scheduled rank monitoring	Strong fit for repeatable requests	Ongoing parser and access maintenance	Count maintenance hours over the schedule
Stable data inside an application	Strong fit for a documented contract	Parser, tests, rendering, and source reviews	Prioritize predictable fields and ownership

Check four constraints before choosing

Maintenance: Who will update selectors, request parameters, and tests when the search experience changes?
Stability: Can the job detect incomplete or unexpected output before it reaches a report or product feature?
Structure: Does the next step need typed values, consistent positions, or a stable JSON contract?
Compliance: Do the source terms, robots instructions, permissions, and data licenses allow the planned access and reuse?

Write the answers beside the project decision. A small one-off script can be the right fit for a permitted task. A recurring or productized workflow often benefits from an API because its operating contract is easier to make explicit.

TalorData can serve as the collection layer when a project needs SERP API requests. A newly registered account receives 500 responses immediately after registration

Build a SERP API Query Builder That Keeps Search Settings Explicit

Elowen — Wed, 26 Aug 2026 09:33:46 +0000

When a teammate asks for “mobile results in Canada” or “news results in English,” the request should become a visible configuration before it becomes an HTTP call. Hidden defaults make search output hard to explain and harder to reuse.

This generic example builds a small JavaScript query builder. It keeps engine, q, device, location, language, and the json: 2 response setting in one object, validates the contract, and returns request options that another workflow can inspect.

Start with a readable query contract

const queryConfig = {
  engine: 'google',
  q: 'technical SEO audit',
  device: 'desktop',
  location: 'United States',
  google_domain: 'google.com',
  gl: 'us',
  hl: 'en',
  json: 2,
  num: 10
};

This is a generic configuration example. The values describe an intended request; they are not presented as a measured result.

The object answers the questions that usually stay implicit: which search engine, which phrase, which device, which market, which language, and which response format.

Validate before building a request

function validateQuery(config) {
  const required = ['engine', 'q', 'device', 'hl', 'json'];
  const missing = required.filter((key) => !config[key]);
  if (missing.length) {
    throw new Error(`Missing query settings: ${missing.join(', ')}`);
  }

  const allowedDevices = new Set(['desktop', 'mobile', 'tablet']);
  if (!allowedDevices.has(config.device)) {
    throw new Error(`Unsupported device: ${config.device}`);
  }

  if (config.json !== 2) {
    throw new Error('json must be the numeric value 2');
  }

  if (!Number.isInteger(config.num) || config.num < 1 || config.num > 100) {
    throw new Error('num must be an integer between 1 and 100');
  }
}

Validation turns an incomplete business request into a clear configuration error before the network layer is involved. Add project-specific checks for supported locations, engines, or result types as your API contract grows.

Build form data from the same object

TalorData accepts the query as form-encoded parameters. The builder below maps only fields that have values, so a caller can see exactly which settings will be sent.

function buildSerpRequest(config, token) {
  validateQuery(config);

  const body = new URLSearchParams();
  const fields = [
    'engine', 'q', 'device', 'location', 'google_domain',
    'gl', 'hl', 'num', 'json'
  ];

  for (const field of fields) {
    const value = config[field];
    if (value !== undefined && value !== '') body.set(field, String(value));
  }

  return {
    method: 'POST',
    url: 'https://serpapi.talordata.net/serp/v1/request',
    headers: {
      Authorization: `Bearer ${token}`,
      'Content-Type': 'application/x-www-form-urlencoded'
    },
    body
  };
}

Call it with a secret loaded from your runtime environment. Keep the token outside the configuration object so logs can safely print the query settings without exposing credentials.

const request = buildSerpRequest(queryConfig, process.env.TALORDATA_TOKEN);
console.log(Object.fromEntries(request.body));

The console output shows the intended form fields. It does not require an API call, which makes this layer easy to unit test with generic configurations.

Use named presets for repeated tasks

Teams often repeat the same market and device combinations. A preset keeps those decisions reviewable while allowing the query text to change.

const presets = {
  usDesktop: {
    device: 'desktop', location: 'United States', gl: 'us', hl: 'en'
  },
  ukMobile: {
    device: 'mobile', location: 'United Kingdom', gl: 'uk', hl: 'en'
  }
};

function makeConfig(preset, overrides) {
  return {
    engine: 'google',
    json: 2,
    num: 10,
    ...preset,
    ...overrides
  };
}

const config = makeConfig(presets.usDesktop, {
  q: 'technical SEO audit'
});

Presets reduce copy-and-paste drift. The final object still contains every setting, so a saved request can be understood without opening the preset definition.

Make the configuration the hand-off artifact

Store the resolved configuration with the job definition or pass it between workflow steps as data. Useful fields include engine, q, device, location, google_domain, gl, hl, num, and json (set to 2).

When a result looks surprising, the first check becomes concrete: inspect the resolved request object. That is faster than reconstructing defaults from several code paths.

The builder remains small because it owns one job: turn an explicit search configuration into a request. Authentication, scheduling, response parsing, and downstream analysis can stay in their own modules.

TalorData can serve as the collection layer for this pattern. New accounts receive 500 responses immediately after registration

How to Verify Public Profiles Without Hiding Uncertain Matches

Elowen — Tue, 25 Aug 2026 07:51:22 +0000

You have a spreadsheet of public figures and need to find a credible profile for each person. A simple script may accept the first matching name, which can attach the wrong role or social account to someone who shares that name.

This walkthrough classifies each person as matched, review_needed, or not_found and keeps the supporting results. The labels show which rows are ready, need review, or returned no usable candidate.

For a public example, I searched Tim Berners-Lee public profile in US English desktop results on August 25, 2026. Wikipedia ranked first, Internet Hall of Fame second, MIT seventh, and X eighth.

The institutional and independent pages supported the name and public role. The X result matched the name and handle, while a dated institutional page remained the stronger source for a current role.

Turn search results into three decisions

Each status has a narrow meaning:

matched: the name matches and credible sources support the expected role.
review_needed: a candidate exists, while role or account evidence remains incomplete.
not_found: the search returned no usable candidate. This state says nothing about whether the person exists.

The code below applies those rules to three anonymous inputs. These inputs demonstrate the batch logic; the Tim Berners-Lee results above came from the public query.

function normalize(text = '') {
  return text.toLowerCase().replace(/[^a-z0-9]+/g, ' ').trim();
}

function hostname(link) {
  try {
    return new URL(link).hostname.replace(/^www\./, '');
  } catch {
    return '';
  }
}

function verifyTarget(target) {
  const expectedName = normalize(target.name);
  const checked = target.results.map((result) => {
    const text = normalize(`${result.title} ${result.description || ''}`);
    const nameMatch = text.includes(expectedName);
    const roleMatch = target.expected_role_terms.some((term) =>
      text.includes(normalize(term))
    );
    const domain = hostname(result.link);
    const trustedSource = target.trusted_domains.some((trusted) =>
      domain === trusted || domain.endsWith(`.${trusted}`)
    );

    return { ...result, nameMatch, roleMatch, trustedSource };
  });

  const candidates = checked.filter((result) => result.nameMatch);

  if (candidates.length === 0) {
    return { target: target.id, status: 'not_found', evidence: [] };
  }

  const strong = candidates.filter(
    (result) => result.roleMatch && result.trustedSource
  );

  return {
    target: target.id,
    status: strong.length > 0 ? 'matched' : 'review_needed',
    evidence: candidates.map(({
      position, title, source, link, nameMatch, roleMatch, trustedSource
    }) => ({
      position, title, source, link, nameMatch, roleMatch, trustedSource
    }))
  };
}

const targets = [
  {
    id: 'Person A',
    name: 'Person A',
    expected_role_terms: ['research director'],
    trusted_domains: ['institution-a.example'],
    results: [{
      position: 1,
      title: 'Person A - Research Director',
      description: 'Biography and current research role.',
      source: 'Institution A',
      link: 'https://institution-a.example/people/person-a'
    }]
  },
  {
    id: 'Person B',
    name: 'Person B',
    expected_role_terms: ['engineering lead'],
    trusted_domains: ['company-b.example'],
    results: []
  },
  {
    id: 'Person C',
    name: 'Person C',
    expected_role_terms: ['chief scientist'],
    trusted_domains: ['institution-c.example'],
    results: [{
      position: 2,
      title: 'Person C - Conference Speaker',
      description: 'Session biography for an industry event.',
      source: 'Conference A',
      link: 'https://conference-a.example/speakers/person-c'
    }]
  }
];

console.log(targets.map(verifyTarget));

Running the code produces all three statuses. Person A has a name, role term, and trusted institutional domain. Person B has no candidate. Person C has a matching name, while its event page lacks the expected role and trusted domain.

This is a first screening step. Open every review_needed source and check the person, role, and publication date. Name and keyword checks can still confuse namesakes, so the evidence stays attached to every decision.

For larger lists, save the query context, result position, title, source, link, access time, and decision status. A later reviewer can then see how the conclusion was reached.

TalorData can supply the Google results for this workflow. New accounts receive 500 responses immediately after registration

When One SERP Parser Meets Two Google Response Formats

Elowen — Sat, 22 Aug 2026 02:46:39 +0000

The same SERP parser can work on one Google response and fail on another. The difference often starts with the JSON shape.

I compared two response formats. Google Search returned 9 results in organic, with the rank stored in position. Google News returned 13 results in news, with the rank stored in rank.

The failure is easy to reproduce

A parser that reads response.organic and item.position handles the Search response. A News response contains news instead. Calling .map() on the missing response.organic value raises an error, so the News results never reach the next processing step.

// Works for the regular Google Search response shape
const rows = response.organic.map((item) => ({
  rank: item.position,
  title: item.title,
  url: item.link
}));

The parser needs a small conversion function for each known shape. The rest of the pipeline can then consume one output format.

function parseOrganic(response) {
  return response.organic.map((item) => ({
    type: 'organic',
    rank: Number(item.position),
    title: item.title || null,
    url: item.link || null,
    source: item.source || null,
    captured_at: response.search_metadata?.created_at || null
  }));
}

function parseNews(response) {
  return response.news.map((item) => ({
    type: 'news',
    rank: Number(item.rank),
    title: item.title || null,
    url: item.link || null,
    source: item.source || null,
    captured_at: response.search_metadata?.created_at || null
  }));
}

function parseSearchResponse(response) {
  if (Array.isArray(response.organic)) return parseOrganic(response);
  if (Array.isArray(response.news)) return parseNews(response);
  throw new Error('Unsupported response shape');
}

The output fields are type, rank, title, url, source, and captured_at. Downstream code can use these names for either response type.

Clean image fields before storage

The News response also contains image and thumbnail data. Rank analysis has no use for those fields, so I remove fields named image, thumbnail, or favicon, along with values that start with data:image/.

function removeEmbeddedImages(value) {
  if (Array.isArray(value)) return value.map(removeEmbeddedImages);
  if (!value || typeof value !== 'object') return value;

  return Object.fromEntries(Object.entries(value)
    .filter(([key, child]) => {
      const namedImage = ['image', 'thumbnail', 'favicon'].includes(key);
      const embedded = typeof child === 'string' && child.startsWith('data:image/');
      return !namedImage && !embedded;
    })
    .map(([key, child]) => [key, removeEmbeddedImages(child)]));
}

Result of the cleanup

The two responses produced 22 normalized records: 9 from Google Search and 13 from Google News. The cleanup removed 41 image-related fields.

This output is ready for a later CSV, database, or analysis integration. Each additional search type needs its own response check and conversion function.

How Overlapping Google News Pages Can Trigger Duplicate Alerts

Elowen — Wed, 19 Aug 2026 05:47:57 +0000

A monitoring pipeline can store a clean final table and still process the same story twice. This happens when each row reaches the alert check before paginated results are deduplicated.

I tested this failure mode offline with two real Google News responses for the query coffee. The start=0 request returned 13 results. The start=10 request returned 10. Concatenating both arrays produced 23 rows.

The start parameter is an offset. It does not guarantee that the previous response contains exactly 10 results. In this sample, the start=0 request returned 13 results, so the two response ranges overlapped.

One URL appeared twice

The 23 rows contained 22 unique URLs. This Axios story appeared in both responses:

https://www.axios.com/local/charlotte/2026/08/17/wnba-player-downtown-concord-coffee-shop

It had an in-response rank of 13 in the start=0 request and 3 in the start=10 request. A program that checks every row would send that story into the alert decision twice.

The offline simulation ran 23 alert checks before deduplication. The duplicated Axios URL appeared in two checks. Deduplicating first reduced the input to 22 checks.

No live monitoring system or notification service was connected during this test. These numbers describe an offline simulation of the processing order.

Normalize the URL before comparing rows

The same page can arrive with a trailing slash, a fragment, or UTM tracking parameters. Cleaning those parts creates a more stable deduplication key.

function normalizeUrl(raw) {
  const url = new URL(raw);
  url.hash = '';

  ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content']
    .forEach((key) => url.searchParams.delete(key));

  if (url.pathname !== '/') {
    url.pathname = url.pathname.replace(/\/+$/, '');
  }

  return url.toString();
}

This rule keeps other query parameters. Some sites use them to identify different pages, so removing every parameter can merge separate results.

Deduplicate before the alert check

The reliable order is: concatenate the responses, normalize each URL, deduplicate the rows, then run the alert logic.

const seen = new Set();
const uniqueRows = [];

for (const row of rows) {
  const key = normalizeUrl(row.url);
  if (seen.has(key)) continue;

  seen.add(key);
  uniqueRows.push({ ...row, normalized_url: key });
}

for (const row of uniqueRows) {
  console.log('Run alert check once for:', row.normalized_url);
  // Call your own alert decision function here.
}

In this offline simulation, the input dropped from 23 rows to 22 unique URLs. The Axios story entered the alert check once after deduplication.

Keep the start value and the original in-response rank with each row. They show which requests produced an overlap. Use the normalized URL as the deduplication key for this workflow.

When testing a paginated monitor, inspect the rows received by the alert function as well as the final stored table. Display-layer deduplication cannot undo an earlier duplicate check.

Turn SERP Results into a Competitor Market Map by Domain Category

Elowen — Tue, 11 Aug 2026 03:20:31 +0000

A competitor list tells you who you already know. A SERP market map can show which types of sites repeatedly control discovery across a topic: product companies, publishers, directories, communities, marketplaces, or review sites.

The useful output is not another ranking export. It is a table that answers:

Which domains appear across multiple related queries?
Which domain categories occupy the most search visibility?
Which sites rank broadly, and which only own one narrow intent?
Where is the market crowded with direct vendors, and where do intermediaries dominate?

This walkthrough builds that output from several SERPs, rather than treating one keyword as the whole market.

1. Define a query set around buyer tasks

Start with queries that represent different ways someone explores the market. For a search data category, that might look like this:

QUERIES = [
    "serp api",
    "google search api",
    "serp data for seo tools",
    "search api for ai agents",
    "competitor rank tracking api",
]

The query set matters more than the size of the script. If every query has the same intent, the map will exaggerate one part of the market. Include category discovery, use-case, comparison, and implementation queries when they are relevant.

2. Collect structured organic results

The function below sends each query to TalorData SERP API and keeps the fields needed for analysis.

from urllib.parse import urlparse
import requests

ENDPOINT = "https://serpapi.talordata.net/serp/v1/request"


def normalize_domain(url):
    host = urlparse(url).netloc.lower()
    return host.removeprefix("www.")


def fetch_results(query):
    response = requests.post(
        ENDPOINT,
        headers={
            "Authorization": "Bearer <TALORDATA_TOKEN>",
            "Content-Type": "application/x-www-form-urlencoded",
        },
        data={
            "engine": "google",
            "q": query,
            "gl": "us",
            "hl": "en",
            "device": "desktop",
            "json": "2",
        },
        timeout=60,
    )
    response.raise_for_status()

    rows = []
    for result in response.json().get("organic", []):
        url = result.get("link") or result.get("url") or ""
        if not url:
            continue
        rows.append({
            "query": query,
            "position": result.get("position"),
            "domain": normalize_domain(url),
            "title": result.get("title", ""),
            "url": url,
        })
    return rows

Keep the country, language, device, and collection date consistent. Otherwise, changes in search context can look like differences in market structure.

3. Classify domains by market role

Domain classification needs judgment. A transparent lookup table is a good starting point because a reviewer can correct it.

CATEGORY_BY_DOMAIN = {
    "example-vendor.com": "direct_vendor",
    "example-directory.com": "directory",
    "example-media.com": "publisher",
    "example-community.com": "community",
}


def classify_domain(domain):
    return CATEGORY_BY_DOMAIN.get(domain, "unclassified")

Do not silently force every unknown domain into a convenient category. Keep unclassified visible, review the most frequent unknowns, and update the lookup table. That makes the map auditable instead of pretending the categories are objective facts.

4. Aggregate visibility across queries

A simple position-weighted score gives higher-ranked results more influence while still rewarding repeated appearances.

from collections import defaultdict


def position_weight(position):
    if not isinstance(position, int) or position < 1:
        return 0
    return 1 / position


all_rows = []
for query in QUERIES:
    all_rows.extend(fetch_results(query))

domain_stats = defaultdict(lambda: {
    "appearances": 0,
    "queries": set(),
    "visibility_score": 0.0,
})

for row in all_rows:
    stats = domain_stats[row["domain"]]
    stats["appearances"] += 1
    stats["queries"].add(row["query"])
    stats["visibility_score"] += position_weight(row["position"])

market_map = []
for domain, stats in domain_stats.items():
    market_map.append({
        "domain": domain,
        "category": classify_domain(domain),
        "appearances": stats["appearances"],
        "query_coverage": len(stats["queries"]),
        "visibility_score": round(stats["visibility_score"], 3),
    })

market_map.sort(
    key=lambda row: (row["query_coverage"], row["visibility_score"]),
    reverse=True,
)

This is not a universal visibility metric. It is a practical comparison score for one controlled query set. Keep the formula documented so future snapshots remain comparable.

5. Produce two review tables

The domain table shows recurring sites:

domain,category,appearances,query_coverage,visibility_score
example-vendor.com,direct_vendor,4,3,1.583
example-media.com,publisher,3,3,0.867
example-directory.com,directory,2,2,0.750

Then aggregate the same rows by category:

category_stats = defaultdict(lambda: {
    "domains": set(),
    "appearances": 0,
    "visibility_score": 0.0,
})

for row in market_map:
    stats = category_stats[row["category"]]
    stats["domains"].add(row["domain"])
    stats["appearances"] += row["appearances"]
    stats["visibility_score"] += row["visibility_score"]

category_map = [
    {
        "category": category,
        "domain_count": len(stats["domains"]),
        "appearances": stats["appearances"],
        "visibility_score": round(stats["visibility_score"], 3),
    }
    for category, stats in category_stats.items()
]

The category table reveals whether discovery is controlled mainly by vendors or by intermediaries such as publishers and directories. That distinction can change your content and partnership strategy.

6. Add the limits to the report

A SERP market map is a search snapshot, not a complete company database. Record the query set, collection date, location, language, device, category definitions, and scoring formula beside the output. Also keep the original URLs so someone can inspect why a domain was included.

For a first pass, TalorData SERP API provides structured Google results that can feed this workflow. New accounts include 500 responses, which can support a focused multi-query map before you expand the query set.

The main shift is simple: stop reading the SERP as ten isolated rankings. Aggregate related searches, classify the domains by role, and turn repeated visibility into a market structure you can review.

Evaluate a SERP API with a Practical Accuracy, Latency, and Cost Checklist

Elowen — Mon, 10 Aug 2026 01:45:36 +0000

Choosing a SERP API is usually framed as a feature comparison: supported engines, locations, pricing, response format, and maybe JavaScript rendering. Those things matter, but they are not enough if the API will run inside a production SEO workflow, AI agent, market research pipeline, or competitor monitoring system.

A more useful evaluation looks at three things together:

accuracy: does the returned SERP match the search context you asked for?
latency: is the response time predictable enough for your workflow?
cost: how many useful outputs do you get per request, retry, and stored snapshot?

This post outlines a practical checklist and a small test harness you can adapt before committing to a SERP data provider.

Start with fixed test cases

Do not evaluate an API with random one-off queries. Build a small test matrix that reflects your real use case.

TEST_CASES = [
    {
        "name": "commercial_us_desktop",
        "q": "serp api pricing",
        "gl": "us",
        "hl": "en",
        "device": "desktop",
        "location": "United States",
    },
    {
        "name": "local_mobile",
        "q": "emergency plumber near me",
        "gl": "us",
        "hl": "en",
        "device": "mobile",
        "location": "Austin, Texas, United States",
    },
    {
        "name": "informational_ai_search",
        "q": "ai search visibility monitoring",
        "gl": "us",
        "hl": "en",
        "device": "desktop",
        "location": "United States",
    },
]

A good test matrix should include the types of searches your workflow actually needs: local, commercial, informational, mobile, desktop, and any country or language settings that matter.

Make the request measurable

For a SERP API test, store the request parameters, response time, and basic response shape for every run. Here is a minimal example using TalorData SERP API.

import os
import time
import requests

ENDPOINT = "https://serpapi.talordata.net/serp/v1/request"
TOKEN = os.environ["TALORDATA_TOKEN"]


def run_serp_request(case):
    payload = {
        "engine": "google",
        "q": case["q"],
        "gl": case["gl"],
        "hl": case["hl"],
        "device": case["device"],
        "location": case["location"],
        "json": "2",
    }

    started = time.perf_counter()
    response = requests.post(
        ENDPOINT,
        headers={
            "Authorization": "Bearer <TALORDATA_TOKEN>",
            "Content-Type": "application/x-www-form-urlencoded",
        },
        data=payload,
        timeout=60,
    )
    elapsed_ms = round((time.perf_counter() - started) * 1000)

    data = response.json()

    return {
        "case_name": case["name"],
        "q": case["q"],
        "gl": case["gl"],
        "hl": case["hl"],
        "device": case["device"],
        "location": case["location"],
        "status_code": response.status_code,
        "elapsed_ms": elapsed_ms,
        "organic_count": len(data.get("organic", [])),
        "has_paa": bool(data.get("people_also_ask")),
        "has_knowledge": bool(data.get("knowledge")),
        "request_params": data.get("request_params", {}),
        "search_metadata": data.get("search_metadata", {}),
    }

In a real script, read the token from an environment variable or secret manager. Keep <TALORDATA_TOKEN> as a placeholder in shared examples.

Accuracy checklist

Accuracy is not just whether the API returns JSON. Check whether the result matches the search context.

Use a checklist like this:

Accuracy checks
- Does the response include organic results for normal queries?
- Are location, country, language, and device parameters stored or echoed back?
- Do top domains look plausible for the query intent?
- Does a local query produce local-looking results?
- Are expected SERP features present when relevant, such as people_also_ask or knowledge?
- Are result URLs, titles, positions, and snippets stable enough to review?
- Can the same request be repeated later with the same parameters?

For accuracy testing, do not rely only on a single query. One strange SERP does not prove the provider is wrong, and one clean SERP does not prove the provider is right. Look for patterns across the test matrix.

Latency checklist

Latency should be measured in the workflow where the API will be used. A dashboard, batch report, and real-time agent do not have the same tolerance.

Latency checks
- Median response time across the test matrix
- Slowest response time in the test run
- Difference between local and broad queries
- Difference between desktop and mobile requests
- Timeout behavior in your client code
- Whether retries are needed for your workflow
- Whether the workflow can run asynchronously

For batch SEO reporting, a slower request may be acceptable if the output is reliable. For an agent that answers a user in real time, predictable response time may matter more.

Cost checklist

The lowest request price is not always the lowest workflow cost. Cost should include failed attempts, retries, duplicate checks, and the number of requests required to produce one useful output.

Cost checks
- Requests needed per report
- Requests needed per monitored keyword-location pair
- Retry rate during normal runs
- Duplicate calls caused by missing caching or poor scheduling
- Storage cost for keeping snapshots
- Human review time saved by structured output
- Cost per useful row, not only cost per request

For example, a local SEO monitor may need keywords x locations x devices requests per snapshot. A content gap workflow may need requests across many related queries. A RAG or agent workflow may only need a few searches, but latency and relevance may be more important.

Build an evaluation table

After running the test matrix, reduce the output into a review table.

case_name
query
location
device
status_code
elapsed_ms
organic_count
has_paa
has_knowledge
accuracy_note
latency_note
cost_note
pass_for_workflow

A row might look like this:

case_name: local_mobile
query: emergency plumber near me
location: Austin, Texas, United States
device: mobile
status_code: 200
elapsed_ms: 1840
organic_count: 9
has_paa: true
accuracy_note: local service and directory results visible
latency_note: acceptable for batch monitoring
cost_note: one request per keyword-location-device snapshot
pass_for_workflow: yes for weekly local SEO report

This format is easier to discuss than a generic vendor comparison. It ties the API evaluation to the workflow you actually need to run.

A practical conclusion

A SERP API should be evaluated by the quality of the workflow it enables. If the data is structured, the request context is preserved, the latency fits the use case, and the cost model still makes sense after retries and snapshots, then the API is much easier to trust in production.

If you want to test this kind of checklist with structured Google SERP data, TalorData SERP API can be used with parameters such as q, location, gl, hl, device, and json=2. New accounts can use 500 included responses to run a small evaluation matrix before expanding the test.

Build a People Also Ask Research Workflow with SERP API Data

Elowen — Sat, 08 Aug 2026 03:41:12 +0000

People Also Ask data is easy to screenshot and hard to reuse. A few questions in a browser tab can help with brainstorming, but they do not become a reliable research asset until they are collected, normalized, deduplicated, and grouped by intent.

This walkthrough shows a small workflow for turning People Also Ask results into a content research table. The output is a list of user questions with source query, location, intent group, and review notes.

The output we want

The final table should answer these questions:

Which user questions appear across several related searches?
Which questions are informational, commercial, comparative, or troubleshooting-oriented?
Which questions already map to existing content?
Which questions deserve a new section, FAQ, article, or product page update?

A minimal output can look like this:

question
normalized_question
source_query
location
intent_group
content_action
notes

The important part is that the question is not just copied into a document. It is attached to enough context for later review.

Request SERP data with PAA included

TalorData SERP API can return structured Google SERP data, including people_also_ask. A request can be made with the same SERP endpoint used for other Google result checks.

curl -X POST 'https://serpapi.talordata.net/serp/v1/request' \
  -H 'Authorization: Bearer <TALORDATA_TOKEN>' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d 'engine=google' \
  -d 'q=serp api pricing' \
  -d 'gl=us' \
  -d 'hl=en' \
  -d 'device=desktop' \
  -d 'json=2'

For research, run this across a cluster of related queries instead of a single keyword. A single SERP can be noisy, but repeated patterns across multiple SERPs are usually more useful.

Start with a query set

QUERIES = [
    "serp api pricing",
    "google search api for seo",
    "serp api for ai agents",
    "search results api comparison",
]

CONTEXT = {
    "gl": "us",
    "hl": "en",
    "device": "desktop",
    "location": "United States",
}

Keep this query set focused. If the set mixes unrelated topics, the deduped PAA list will become difficult to interpret.

Extract and normalize questions

Different response shapes can be handled defensively. The goal is to collect question-like text from the people_also_ask section without assuming every item has the same shape.

import re


def normalize_question(text: str) -> str:
    text = text.strip().lower()
    text = re.sub(r"\s+", " ", text)
    text = re.sub(r"[?!.]+$", "", text)
    return text


def extract_paa_questions(response_json, source_query, context):
    rows = []

    for item in response_json.get("people_also_ask", []):
        question = item.get("question") or item.get("title") or ""
        question = question.strip()
        if not question:
            continue

        rows.append({
            "question": question,
            "normalized_question": normalize_question(question),
            "source_query": source_query,
            "location": context.get("location", ""),
            "gl": context.get("gl", ""),
            "hl": context.get("hl", ""),
            "device": context.get("device", ""),
        })

    return rows

This keeps the extraction step small. Do not assign intent yet. First collect the evidence.

Deduplicate across queries

A question may appear with small wording differences. Start with exact normalization, then review close duplicates manually or with a separate similarity step.

def dedupe_questions(rows):
    grouped = {}

    for row in rows:
        key = row["normalized_question"]
        if key not in grouped:
            grouped[key] = {
                "question": row["question"],
                "normalized_question": key,
                "source_queries": set(),
                "locations": set(),
            }

        grouped[key]["source_queries"].add(row["source_query"])
        grouped[key]["locations"].add(row["location"])

    output = []
    for item in grouped.values():
        output.append({
            "question": item["question"],
            "normalized_question": item["normalized_question"],
            "source_queries": ", ".join(sorted(item["source_queries"])),
            "locations": ", ".join(sorted(item["locations"])),
            "query_count": len(item["source_queries"]),
        })

    return sorted(output, key=lambda row: row["query_count"], reverse=True)

query_count is useful because repeated questions are usually better candidates for content review than one-off questions.

Add intent groups

A lightweight intent classifier is enough for the first pass. It does not need to be perfect; it needs to make the review faster.

def classify_intent(question: str) -> str:
    q = question.lower()

    if any(term in q for term in ["pricing", "cost", "free", "paid"]):
        return "commercial"
    if any(term in q for term in ["vs", "compare", "alternative", "best"]):
        return "comparison"
    if any(term in q for term in ["how", "what is", "why", "guide"]):
        return "informational"
    if any(term in q for term in ["not working", "error", "fix", "limit"]):
        return "troubleshooting"

    return "mixed_or_review"

This is deliberately simple. The output should be reviewed by a human before it becomes a content plan.

Map questions to content actions

The final step is to decide what to do with each question.

def suggest_action(intent_group, query_count):
    if query_count >= 3 and intent_group == "commercial":
        return "review product or pricing page"
    if query_count >= 2 and intent_group == "comparison":
        return "consider comparison section or article"
    if intent_group == "informational":
        return "map to FAQ or educational article"
    if intent_group == "troubleshooting":
        return "map to docs or support content"
    return "manual review"

The goal is not to automate editorial judgment. The goal is to make the research table structured enough that editorial judgment can happen faster.

Example final row

Question: How do I get Google search results from an API?
Source queries: google search api for seo, serp api for ai agents
Query count: 2
Intent group: informational
Content action: map to FAQ or educational article
Note: Review whether existing docs explain authentication, parameters, and output fields clearly.

That row is much more useful than a raw PAA export. It tells the team why the question matters and where it might belong.

If you are building a PAA research workflow, TalorData SERP API can provide structured SERP data that includes people_also_ask. New accounts can use the included 500 responses to test a small query cluster before expanding the research workflow.