DEV Community

Elowen
Elowen

Posted on

When One SERP Parser Meets Two Google Response Formats

The same SERP parser can work on one Google response and fail on another. The difference often starts with the JSON shape.

I compared two response formats. Google Search returned 9 results in organic, with the rank stored in position. Google News returned 13 results in news, with the rank stored in rank.

The failure is easy to reproduce

A parser that reads response.organic and item.position handles the Search response. A News response contains news instead. Calling .map() on the missing response.organic value raises an error, so the News results never reach the next processing step.

// Works for the regular Google Search response shape
const rows = response.organic.map((item) => ({
  rank: item.position,
  title: item.title,
  url: item.link
}));
Enter fullscreen mode Exit fullscreen mode

The parser needs a small conversion function for each known shape. The rest of the pipeline can then consume one output format.

function parseOrganic(response) {
  return response.organic.map((item) => ({
    type: 'organic',
    rank: Number(item.position),
    title: item.title || null,
    url: item.link || null,
    source: item.source || null,
    captured_at: response.search_metadata?.created_at || null
  }));
}

function parseNews(response) {
  return response.news.map((item) => ({
    type: 'news',
    rank: Number(item.rank),
    title: item.title || null,
    url: item.link || null,
    source: item.source || null,
    captured_at: response.search_metadata?.created_at || null
  }));
}

function parseSearchResponse(response) {
  if (Array.isArray(response.organic)) return parseOrganic(response);
  if (Array.isArray(response.news)) return parseNews(response);
  throw new Error('Unsupported response shape');
}
Enter fullscreen mode Exit fullscreen mode

The output fields are type, rank, title, url, source, and captured_at. Downstream code can use these names for either response type.

Clean image fields before storage

The News response also contains image and thumbnail data. Rank analysis has no use for those fields, so I remove fields named image, thumbnail, or favicon, along with values that start with data:image/.

function removeEmbeddedImages(value) {
  if (Array.isArray(value)) return value.map(removeEmbeddedImages);
  if (!value || typeof value !== 'object') return value;

  return Object.fromEntries(Object.entries(value)
    .filter(([key, child]) => {
      const namedImage = ['image', 'thumbnail', 'favicon'].includes(key);
      const embedded = typeof child === 'string' && child.startsWith('data:image/');
      return !namedImage && !embedded;
    })
    .map(([key, child]) => [key, removeEmbeddedImages(child)]));
}
Enter fullscreen mode Exit fullscreen mode

Result of the cleanup

The two responses produced 22 normalized records: 9 from Google Search and 13 from Google News. The cleanup removed 41 image-related fields.

This output is ready for a later CSV, database, or analysis integration. Each additional search type needs its own response check and conversion function.

Top comments (0)