TL;DR:
- Search gives an AI agent candidates to investigate. A URL and its snippet are not yet a verified source for an answer.
- Make the handoff inspectable. Retain query context, candidate identity, selection reasoning, and retrieval outcomes so each citation can reach back to reviewed text.
- Test a local adapter first. This program converts saved captures into a review queue; live collection and page retrieval need separate setup and verification.
A research answer can include citations without showing whether the linked pages support its sentences. Search access does not automatically create that missing connection. The application needs to represent discovery, retrieval, and evidence review as distinct stages with explicit outputs.
Scrapeless Google Search API supplies the discovery input. The workflow below uses a Google Search API for AI agents to create source candidates while preserving the boundary between an observed snippet and material suitable for a factual answer. The adapter gives the next reviewer enough information to understand where a candidate came from.
Define the Discovery Contract
Keep the research question with the job and the exact query with the submitted request. The question is application metadata. A project can explore it through several searches while preserving each search's individual context.
Return candidate records that include the destination, observed text, original order, and review status. The list is not an answer. Subsequent reading may establish that a candidate is useful, irrelevant, inaccessible, or outdated for the question.
Require the downstream answer stage to use retrieved, reviewed content for factual claims. A discovery-only record can guide additional research but should not appear automatically in the accepted evidence set. That rule exposes missing work instead of letting the answer generator replace it with a plausible explanation.
Prerequisites and Request Parameters
You need Python and a saved JSON capture to run the local transform. No third-party package is required. The example expects a collector envelope with request, http_status, response, run_id, and received_at. These outer fields are defined by the application, not presented as the API's native response wrapper.
Authenticated collection requires your account key. The Google Search request workflow specifies POST https://api.scrapeless.com/api/v1/scraper/request, authentication through x-api-token, and actor scraper.google.search. Search settings go in input.
Read the Google Search parameters when building requests. Query wording, language, and country define the observation context. In full-URL mode, other input parameters are ignored, so retain the supplied URL instead of reconstructing settings from assumptions.
Note: This article does not report a live API request or a page-retrieval run. The code performs a local transform verified with synthetic captures. Before collecting account data or completing a pending task, confirm the current documented workflow and inspect its actual response.
Preserve the Response State Before Reading Results
HTTP 200 contains task data, whereas HTTP 201 means the task is pending. Save taskId when it is supplied. Keep that state until a separately verified completion procedure provides the result. This program contains no assumed task-retrieval route.
For completed data, validate the organic_results array before projecting candidates. An absent field or incompatible shape becomes unmapped. A valid array with no items becomes empty. Those distinctions let the next stage respond to the actual collection outcome.
Preserve the raw capture even after generating the smaller queue. The JSON value model retains nulls and nested structures that might disappear from a flattened projection. It also gives you the evidence needed to revise a mapper later.
Build the Local Candidate Adapter
Save the program as source_candidates.py, then run python3 source_candidates.py capture.json with a saved capture. The output is derived JSON on standard output. The input remains unchanged, and the program makes no search request, page request, or model call.
import argparse
import json
from pathlib import Path
from urllib.parse import urlsplit
def candidates(record):
if not isinstance(record, dict):
raise ValueError('Capture must be an object')
status = record.get('http_status')
payload = record.get('response')
base = {'run_id': record.get('run_id'), 'request': record.get('request'),
'received_at': record.get('received_at'), 'candidates': []}
if status == 201:
task = payload.get('taskId') if isinstance(payload, dict) else None
return dict(base, state='pending', task_id=task)
if status != 200:
return dict(base, state='transport_error' if status is None else 'http_error')
rows = payload.get('organic_results') if isinstance(payload, dict) else None
if not isinstance(rows, list) or any(not isinstance(row, dict) for row in rows):
return dict(base, state='unmapped')
output, seen = [], set()
for ordinal, row in enumerate(rows):
link = row.get('link')
reason, host = None, None
try:
parsed = urlsplit(link) if isinstance(link, str) else None
if (parsed is None or parsed.scheme not in ('http', 'https')
or not parsed.hostname or parsed.username or parsed.password):
reason = 'invalid_web_url'
else:
host = parsed.hostname.lower()
except ValueError:
reason = 'invalid_web_url'
if reason is None and link in seen:
reason = 'duplicate_exact_url'
if reason is None:
seen.add(link)
output.append({'candidate_id': f'source-{ordinal}', 'ordinal': ordinal,
'position': row.get('position'), 'title': row.get('title'),
'url': link, 'hostname': host, 'snippet': row.get('snippet'),
'review_state': 'excluded' if reason else 'needs_review',
'exclusion_reason': reason, 'evidence_state': 'discovery_only'})
return dict(base, state='observed' if rows else 'empty', candidates=output)
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('capture')
args = parser.parse_args()
result = candidates(json.loads(Path(args.capture).read_text(encoding='utf-8')))
print(json.dumps(result, ensure_ascii=False, indent=2))
Pair a candidate identifier with run_id, because the identifier alone is local to that run. Exact duplicate URLs stay in the output as excluded records with a reason. Keeping them visible explains the queue's decisions. Other variants remain separate until reviewed.
The URL component parser helps reject unsupported schemes, absent hosts, and embedded credentials. That check establishes basic input shape only. A later fetcher must enforce destination rules for resolved addresses and redirect targets before using the URL in a network operation.
Select Sources and Retrieve Content Separately
Assess each eligible candidate against the research question and record why it was selected or excluded. Choose material that can establish the needed fact. Search position remains an observation about the result, not a source-quality rating.
Pass a selected URL to a separate retrieval operation that records the requested address, final destination, retrieval time, stored content reference, and outcome. If the destination cannot be read, preserve that failure. Using its snippet in place of a missing body would change the evidence standard without telling the reader.
Google's explanation of search snippets makes their limited role clear. The excerpt can depend on the query and may omit the passage relevant to your claim. Read the retrieved source before using it to substantiate an answer.
Keep retrieved text in the data boundary. Instructions embedded in a page do not authorize new tools or change the research assignment. The application should maintain that separation when presenting source content to an assistant.
Connect Claims to Reviewed Passages
Create a citation record that links the proposed statement to a supporting passage and its retrieval record. Retain the candidate ID for provenance, but store the passage location separately. A bare destination URL does not establish the relationship between source and claim.
Inspect the limits of the source's statement. A fact about one version or market cannot automatically support a general claim about all configurations. If sources disagree, preserve the disagreement for review or qualify the answer; search order does not resolve the conflict.
The provenance model distinguishes evidence, processing activities, and responsibility. Your records can be simpler than a formal provenance system while still preserving those relationships for someone reviewing the answer.
If reviewed content does not support a claim, omit it or make the evidence gap explicit. A better discovery process helps organize research but cannot promise that an answer generator will never produce unsupported text.
Check the Adapter Before Connecting an Agent
Exercise the transform with populated and empty arrays, missing fields, malformed items, HTTP 201, and HTTP errors. Add an exact duplicate and an invalid URL scheme. Those local fixtures establish how the application handles each case, not what the service currently returns for all searches.
Check that exclusions retain their original observed fields and every candidate is marked discovery_only. Before deployment, inspect an account capture and compare its shape with the mapping. Preserve the original response if the adapter needs revision.
The boundary does not depend on an agent framework. A caller that consumes this JSON contract can use the queue, but a particular SDK or tool protocol still needs its own connection test. No such framework handshake is claimed by this local example.
Conclusion
Preserve the request, classify its outcome, and return candidates with explicit review states. That small discovery contract gives retrieval and citation checking a usable handoff, while keeping an unexplained list of links out of the evidence stage.
An editorial workflow can use the same discipline for content gap analysis when source review is needed before assigning an article.
FAQ
Q: Does the adapter retrieve full page text?
No. It transforms stored search observations into a candidate queue. Page retrieval needs a separate operation and its own evidence record.
Q: Can the first organic result be cited automatically?
No. Organic position does not establish support for a statement. Inspect the retrieved passage before accepting the citation.
Q: What happens to HTTP 201?
It remains pending, with a task identifier retained when available. The adapter neither retrieves that task nor classifies it as an empty result.
Q: Does URL parsing make fetching a candidate safe?
No. Shape validation is limited. A network fetcher also needs destination controls covering resolved addresses and redirects.
Q: Does this require a specific agent framework?
No. The example uses a local JSON contract. Live account collection and a chosen framework integration require their own verification.
Top comments (0)