DEV Community

Elowen
Elowen

Posted on

Batch SERP Requests with Retry, Status Tracking, and Audit Logs

A batch of 100 search requests rarely succeeds all at once. Network jitter, rate limits, and transient errors produce a mix of successful and failed responses. Treating failures as exceptions to abort the entire batch loses data and hides patterns.

Start with a request record that captures each attempt:

{
  "request_id": "batch-20260902-001",
  "query": "salesforce revops trends",
  "engine": "google",
  "location": "United States",
  "device": "desktop",
  "attempt": 1,
  "status": "pending",
  "queued_at": "2026-09-02T08:00:00Z"
}
Enter fullscreen mode Exit fullscreen mode

The values above are illustrative. A production system should persist each request before execution and update status after each attempt.

Define four statuses:

  • pending: request queued, not yet sent
  • in_progress: request sent, waiting for response
  • success: response received and parsed
  • failed: error after max retries

A retry policy with exponential backoff handles transient errors. This pseudocode illustrates the pattern:

max_retries = 3
base_delay = 1  # second

for attempt in range(1, max_retries + 1):
    update_status(request_id, "in_progress", attempt)
    try:
        response = serp_request(query, location, device)
        save_response(request_id, response)
        update_status(request_id, "success", attempt)
        break
    except TransientError as e:
        if attempt == max_retries:
            update_status(request_id, "failed", attempt, error=str(e))
        else:
            wait(base_delay * (2 ** (attempt - 1)))
Enter fullscreen mode Exit fullscreen mode

The code above is a generic illustration. Replace TransientError with the actual exception types from your HTTP client.

For the batch as a whole, maintain a summary:

{
  "batch_id": "batch-20260902",
  "total": 100,
  "success": 94,
  "failed": 6,
  "failure_reasons": {
    "timeout": 4,
    "rate_limit": 2
  },
  "started_at": "2026-09-02T08:00:00Z",
  "completed_at": "2026-09-02T08:15:00Z"
}
Enter fullscreen mode Exit fullscreen mode

The summary enables audit, retry of only failed items, and a reliable report. Without it, a batch of 94 successes and 6 failures looks like a binary pass-or-fail. With it, you can investigate the 6 failures, retry them with adjusted parameters, and explain the data quality to stakeholders.

The TalorData API returns a consistent response structure for both successful and failed requests. The failure details appear in the response metadata. Capture both so your pipeline can differentiate between an empty search result and a request that never completed.

Top comments (0)