Flatten an answer card into the surrounding snippets and your retriever loses who said what. The text still looks useful. That's the annoying part.
Lab note 064: when your ingestion job meets knowledge_graph.answerText beside aiAnswer.text, should those answers become retrieval documents?
We'd keep them, with a catch: retrieving an answer doesn't retrieve the publisher material attached to it. A source list gives the block attribution, but it doesn't establish which source supports each sentence.
What should an answer card represent?
An answer card should become an attributed document about the answer shown on the search page.
That's a narrower promise than “this document contains evidence from these publishers.” Your retrieval prompt needs to know the difference before the model turns proximity into certainty.
Answer bodies are useful for discovery, comparison, and labeling. Organic snippets belong alongside them as publisher previews, with their different origins preserved.
Neither should quietly inherit the other's authority.
The embedding unit is the answer body or the organic snippet. The observation unit includes its page, query, collection time, and source attachments.
Same words, different jobs.
Start with page containers
We use the vexaloom-labs SERP Actor with outputFormat: "pages" because the page container keeps the answer blocks beside their surrounding results. Blocks appear only when the served page contains them, so a successful collection need not yield an answer card.
Example setup: this input requests a small page window and keeps blocks while excluding ads and related searches.
import hashlib
import json
import os
import unicodedata
from collections import Counter
import tiktoken
from apify_client import ApifyClient
run_input = {
"mode": "search",
"queries": ["retrieval augmented generation"],
"domain": "yandex.com",
"language": "en",
"regionId": 213,
"maxPages": 1,
"maxItems": 100,
"includeAds": False,
"includeBlocks": True,
"includeRelatedSearches": False,
"outputFormat": "pages",
}
client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor(
"vexaloom-labs/yandex-serp-and-knowledge-graph-scraper"
).call(run_input=run_input)
if run["status"] != "SUCCEEDED":
raise RuntimeError(f"Collection ended with {run['status']}")
The page adapters use knowledge_graph.answerText and aiAnswer.text separately. Their names are part of the boundary we're preserving.
A knowledge graph answer isn't automatically an AI-generated answer just because it sits in a card. We retain the distinction instead of guessing how its text was produced.
Keep the page as the parent
There isn't a conversational thread here. The useful parent is the SERP page observation, with each answer or snippet attached through its container field and, for organic results, its list index.
That gives us this relationship:
Page observation
├─ knowledge_graph answer → attached sources
├─ standalone AI answer → attached sources
└─ organic snippet → publisher result link
Text hash → shared embedding
Observation → its own parent and provenance
These are our corpus fields, rather than additional fields returned by the Actor:
| Corpus field | What it preserves |
|---|---|
document_kind |
Whether retrieval found a knowledge graph answer, AI answer, or organic preview |
content_hash |
The normalized text identity used for embedding reuse |
provenance.parent |
Query, page, page URL, language, observation time, and search metadata |
provenance.block |
The answer's label and source attachments, or the organic result's identity |
citation_scope |
That an attached source list belongs to the block |
We keep source objects intact. Their presence doesn't give us permission to manufacture sentence-level citation mappings.
The adapters below preserve that hierarchy and count unexpected shapes instead of treating them as valid text.
audit = Counter()
def digest(value):
return hashlib.sha256(value.encode("utf-8")).hexdigest()
def page_units(page):
parent = {
key: page.get(key)
for key in ("query", "page", "url", "lang", "scraped_at")
}
parent["search_metadata"] = page.get("search_metadata")
parent_id = digest(json.dumps(
parent, ensure_ascii=False, sort_keys=True
))
adapters = (
(
"knowledge_graph",
"answerText",
"knowledge_graph_answer",
("label", "title", "source_url", "sources", "attributes", "seeAlso"),
),
(
"aiAnswer",
"text",
"ai_answer",
("label", "sources"),
),
)
for field, text_key, kind, metadata_keys in adapters:
block = page.get(field)
if block is None:
continue
if not isinstance(block, dict):
audit["invalid_answer_container"] += 1
continue
body = block.get(text_key)
if body is None:
continue
if not isinstance(body, str):
audit["invalid_answer_text"] += 1
continue
yield kind, body, {
"parent_id": parent_id,
"parent": parent,
"block_field": field,
"citation_scope": "block",
"block": {
key: block[key]
for key in metadata_keys
if key in block
},
}
organic = page.get("organic_results", [])
if not isinstance(organic, list):
audit["invalid_organic_container"] += 1
return
for index, result in enumerate(organic):
if not isinstance(result, dict):
audit["invalid_organic_result"] += 1
continue
snippet = result.get("snippet")
if not isinstance(snippet, str):
continue
yield "organic_snippet", snippet, {
"parent_id": parent_id,
"parent": parent,
"block_field": "organic_results",
"block_index": index,
"block": {
key: result[key]
for key in (
"title", "link", "host",
"position", "organicPosition"
)
if key in result
},
}
The organic title stays in metadata rather than being silently prepended to the snippet. If your labeling task needs title-plus-snippet text, make that a named transformation and change the hash accordingly.
Clean text without erasing the speaker
Our cleaning is deliberately boring: normalize Unicode and collapse whitespace.
We preserve case, punctuation, and words. Aggressive normalization can erase distinctions your retrieval or labeling task needs, particularly in technical names.
Queries, source URLs, and labels stay out of the embedding text. Those belong in provenance; otherwise unrelated answers can look similar because they share collection scaffolding.
This cell sets an explicit token budget and the normalization policy.
encoding = tiktoken.get_encoding("cl100k_base")
token_budget = 2048
def normalize_text(value):
return " ".join(unicodedata.normalize("NFC", value).split())
Choose the tokenizer and budget for your embedding model before using this in production. The choice above is a corpus policy, not an Actor limit.
Long answers go to a review file instead of being truncated. That preserves the full block for a later chunking pass, where every child must inherit its parent identity, document kind, and block-level attachments.
A shorter chunk doesn't earn a more precise citation.
Deduplicate embeddings, retain observations
The Actor removes duplicate URLs and block items during collection. That serves a different purpose from deduplicating normalized corpus text.
Our hash lets exact duplicate text share an embedding while every emitted observation retains its own provenance. An identical answer attached to different sources must not have those attachments overwritten by whichever row arrived last.
The writer creates a retrieval corpus, a unique-text embedding queue, rejected units, and an audit record. It doesn't call an embedding service.
seen_hashes = set()
with (
open("corpus.jsonl", "w", encoding="utf-8") as corpus_file,
open("embedding_texts.jsonl", "w", encoding="utf-8") as embedding_file,
open("review.jsonl", "w", encoding="utf-8") as review_file,
open("audit.jsonl", "w", encoding="utf-8") as audit_file,
):
for page in client.dataset(run["defaultDatasetId"]).iterate_items():
for kind, raw_text, provenance in page_units(page):
text = normalize_text(raw_text)
token_count = len(encoding.encode(text, disallowed_special=()))
content_hash = digest(text)
if not text or token_count > token_budget:
reason = "empty_text" if not text else "over_token_budget"
audit[reason] += 1
review_file.write(json.dumps({
"reason": reason,
"text": raw_text,
"document_kind": kind,
"provenance": provenance,
"token_count": token_count,
}, ensure_ascii=False) + "\n")
continue
corpus_file.write(json.dumps({
"text": text,
"document_kind": kind,
"content_hash": content_hash,
"token_count": token_count,
"provenance": provenance,
}, ensure_ascii=False) + "\n")
audit["accepted_observations"] += 1
if content_hash not in seen_hashes:
seen_hashes.add(content_hash)
embedding_file.write(json.dumps({
"content_hash": content_hash,
"text": text,
"token_count": token_count,
}, ensure_ascii=False) + "\n")
audit["unique_embedding_texts"] += 1
audit_file.write(json.dumps(dict(audit), sort_keys=True) + "\n")
Keep the embedding index connected to the observation store by content_hash. Retrieval must return attributed observations alongside the vector match, not just the shared text.
The iterator avoids loading the dataset at once, although the hash set still grows with unique text. Move that membership check into persistent storage when its memory cost becomes inconvenient.
Where the convenient shortcuts fail
Flattening the page fails because it destroys the distinction between a generated answer, a knowledge graph answer, and a publisher preview. A shared parent doesn't imply a shared author.
Hash deduplication also doesn't solve evaluation leakage. Paraphrased queries can retrieve overlapping answers, and a summary may repeat wording from a publisher snippet without matching it exactly.
Before assigning train and test splits, group related queries and connect observations that share exact text. Add source overlap and near-duplicate checks where your evaluation requires them. These files intentionally contain no split assignment.
Even that grouping needs judgment: a broad source domain is usually too coarse a boundary, while a single query string is often too narrow.
Privacy needs its own pass before embeddings leave your boundary. Queries can contain personal details; snippets and source attachments can expose names, addresses, or identifying URL parameters.
This code preserves those fields for attribution. It does not anonymize them. Keep the raw observation store restricted, minimize the metadata you export, and apply your redaction policy before computing production hashes and embeddings.
A hash of sensitive text isn't a privacy policy wearing a smaller hat.
Retrieve the answer with its role attached
We'd admit answer cards into the corpus as attributed answer documents. We wouldn't promote their attached source lists into verified evidence.
The Yandex SERP & Knowledge Graph Scraper supplies the separate surfaces; the corpus decides what those surfaces mean during retrieval.
For an answering system, returning a card can trigger source acquisition rather than finish it. Fetching and evaluating the underlying material is a separate ingestion step.
For a labeling task, retain document_kind so generated answers don't quietly become examples of publisher prose. That distinction is useful even when the text happens to match.
Reproduce the corpus build
Install apify-client and tiktoken, then set APIFY_TOKEN in your environment. Execute the Python cells in order.
Inspect the audit and review files before indexing accepted text. An absent answer block is allowed; malformed containers and oversized answers deserve separate attention.
Join vector matches back to corpus observations, then apply your privacy and evaluation-split policies before downstream use.
Example input, exactly as used above:
{
"mode": "search",
"queries": ["retrieval augmented generation"],
"domain": "yandex.com",
"language": "en",
"regionId": 213,
"maxPages": 1,
"maxItems": 100,
"includeAds": false,
"includeBlocks": true,
"includeRelatedSearches": false,
"outputFormat": "pages"
}
Top comments (0)