Your Question-Level Split Can Leak the Same Answer Twice

Different question URLs can carry the same answer straight across your evaluation boundary.

You split threads, rerun retrieval evaluation, and get scores comfortable enough to make you uncomfortable. The partition code checks question URLs. The answer evidence has other travel arrangements.

Lab note 009: can we preserve question context without letting the same answer cross that boundary? The catch comes early: the grouping below protects relationships visible in the collected rows. It can't prove that missing provenance, paraphrases, or uncollected copies are absent.

Why can a question-level split leak answers?

Quora sometimes displays answers written for merged or equivalent questions. Those answer rows carry answer_is_related: true and source_question_url / source_question_title, alongside the fetched question's context.

A fetched question URL identifies where an answer was collected, not necessarily where it was originally written.

The failure mode looks like this:

Fetched question A ── answer X, originally written for B
Fetched question B ── answer X

Split by fetched URL:
train: A → X             evaluation: B → X

Group shared evidence:
{A, B, X} → one partition

Deduplicating inside each partition doesn't fix this. Each side can contain a locally unique copy of the same answer.

We need answer identity for corpus records and a broader relationship graph for partition ownership. Those are different jobs.

Collect a snapshot with room for provenance

The Actor supplies the answer rows; the corpus design belongs to the ingestion pipeline.

This Example input follows question links and expands related questions while skipping profile enrichment and HTML retention:

import os
import json
import hashlib
import unicodedata
from collections import Counter, defaultdict
from urllib.parse import urlsplit, urlunsplit

from apify_client import ApifyClient

run_input = {
    "mode": "urls",
    "directUrls": [
        "https://www.quora.com/What-is-machine-learning"
    ],
    "resultTypes": ["answer"],
    "includeAnswers": True,
    "maxAnswersPerQuestion": 50,
    "includeRelatedQuestions": True,
    "maxRelatedQuestions": 10,
    "includeAuthorProfiles": False,
    "includeAnswerHtml": False,
    "maxItems": 100
}

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor(
    "vexaloom-labs/quora-keyword-search-and-author-discovery-scraper"
).call(run_input=run_input)

if run["status"] != "SUCCEEDED":
    raise RuntimeError(f"Collection ended with {run['status']}")

items = client.dataset(run["defaultDatasetId"]).iterate_items()

Related-question discovery and related-answer provenance serve different purposes. Following another question finds more pages; an answer's source-question fields explain where that answer originated.

Neither mechanism guarantees that this seed exposes the overlap we're guarding against. The code handles overlap when present.

Make the answer the text unit

We'd embed the answer body and retain question contexts as metadata. That keeps a fetched page's wording from silently replacing the answer's original prompt.

For labeling, the same record can carry both contexts without treating them as interchangeable.

Corpus element	What we retain	Why
Text unit	Cleaned answer_text	Keeps answer identity independent of its display location
Answer identity	answer_id, with answer_url as fallback	Joins observations of the same answer
Fetched context	Question ID, URL, and title	Preserves where collection encountered it
Source context	Source-question URL and title, plus the related flag	Preserves the original question when supplied
Speaker role	A corpus label meaning “answer author”	Avoids inventing expertise or authority

If an answer later needs chunking, every child chunk must inherit its answer identity and partition group. Splitting after chunking gives the same leakage problem smaller shoes.

We also keep parent_question_url out of reply reconstruction. Discovery ancestry isn't a conversational reply chain, and comment bodies are unavailable.

Separate identity from partition membership

We use independent grouping structures. Answer aliases determine which observations become a corpus record. Partition relationships connect those answers to question contexts and matching text.

Otherwise, connecting answers through a shared question would accidentally merge distinct answers into a single document. A tidy graph can still do the wrong job.

This cell defines grouping, conservative URL cleanup, and text normalization:

class Groups:
    def __init__(self):
        self.parent = {}

    def find(self, key):
        self.parent.setdefault(key, key)
        trail = []
        while self.parent[key] != key:
            trail.append(key)
            key = self.parent[key]
        for previous in trail:
            self.parent[previous] = key
        return key

    def join(self, *keys):
        if keys:
            root = self.find(keys[0])
            for key in keys[1:]:
                self.parent[self.find(key)] = root

def normalize_url(value):
    if not isinstance(value, str):
        return None
    try:
        parts = urlsplit(value.strip())
    except ValueError:
        return None
    if parts.scheme.lower() not in {"http", "https"} or not parts.netloc:
        return None
    return urlunsplit((
        parts.scheme.lower(), parts.netloc.lower(),
        parts.path, parts.query, ""
    ))

def clean_text(value):
    if not isinstance(value, str):
        return ""
    value = unicodedata.normalize("NFC", value)
    return "\n".join(
        " ".join(line.split()) for line in value.splitlines()
    ).strip()

def fingerprint(value):
    return hashlib.sha256(value.encode("utf-8")).hexdigest()

identities = Groups()
boundaries = Groups()
staged = []
rejected = Counter()

URL cleanup removes fragments and normalizes host casing. It deliberately preserves paths and query strings. We'd rather miss an alias than merge unrelated question URLs through an enthusiastic cleanup rule.

Text cleanup preserves line boundaries and casing. Code fragments and acronyms deserve better than automatic lowercasing.

Connect provenance before filtering text

A row can carry useful relationship evidence even when its text won't enter the corpus. We connect that evidence first.

The identity graph also joins an ID to its permalink when both occur. That lets a URL-only observation join a later observation carrying the ID.

This cell stages usable answers while retaining every observed thread association:

association_fields = (
    "question_id", "question_url", "question_title",
    "source_question_url", "source_question_title",
    "answer_is_related"
)

for row in items:
    fetched = normalize_url(row.get("question_url"))
    source = normalize_url(row.get("source_question_url"))
    questions = [
        ("question_url", url)
        for url in (fetched, source) if url
    ]
    boundaries.join(*questions)

    aliases = []
    answer_id = row.get("answer_id")
    if isinstance(answer_id, (str, int)) and not isinstance(answer_id, bool):
        answer_id = str(answer_id).strip()
        if answer_id:
            aliases.append(("answer_id", answer_id))

    answer_url = normalize_url(row.get("answer_url"))
    if answer_url:
        aliases.append(("answer_url", answer_url))

    identities.join(*aliases)
    boundaries.join(*aliases, *questions)

    text = clean_text(row.get("answer_text"))
    if text:
        boundaries.join(
            *aliases, *questions, ("text_hash", fingerprint(text))
        )

    if not aliases:
        rejected["missing_answer_identity"] += 1
        continue
    if not text:
        rejected["missing_answer_text"] += 1
        continue
    if not fetched:
        rejected["missing_fetched_question"] += 1
        continue
    if row.get("answer_is_related") is True and not source:
        rejected["missing_related_source"] += 1
        continue

    association = {
        field: row[field]
        for field in association_fields if field in row
    }
    staged.append((aliases[0], text, association))

Matching normalized text joins partition groups even when answer identities differ. We keep those identities separate in the corpus because identical wording alone doesn't establish shared authorship.

That's a conservative trade-off: copied boilerplate can connect otherwise unrelated threads. For evaluation, we prefer that inconvenience to quietly rewarding memorized text.

The rejected counters describe exclusions from this corpus, including non-answer rows. They aren't collection success rates.

Write JSONL without overwriting context

Repeated observations can contain different text versions. Here we choose the longest normalized variant, with a lexical tie-breaker, and retain hashes of all observed variants.

That's a reproducible selection policy, not a claim that the longest version is newest or best. A temporal benchmark needs an explicit version policy instead.

This cell builds partition assignments, checks answer and text ownership, and writes the corpus:

batches = defaultdict(list)
for alias, text, association in staged:
    batches[identities.find(alias)].append((text, association))

aliases_by_root = defaultdict(list)
for alias in identities.parent:
    aliases_by_root[identities.find(alias)].append(alias)

component_members = defaultdict(list)
for node in boundaries.parent:
    component_members[boundaries.find(node)].append(node)

group_ids = {
    root: fingerprint(json.dumps(sorted(members), ensure_ascii=False))
    for root, members in component_members.items()
}

corpus = []
ownership = {}

for root, batch in batches.items():
    answer_keys = sorted(aliases_by_root[root])
    if sum(kind == "answer_id" for kind, _ in answer_keys) > 1:
        rejected["conflicting_answer_ids"] += len(batch)
        continue

    variants = sorted({text for text, _ in batch})
    selected_text = min(variants, key=lambda text: (-len(text), text))
    variant_hashes = sorted({fingerprint(text) for text in variants})

    observations = [
        json.loads(encoded)
        for encoded in sorted({
            json.dumps(association, sort_keys=True, ensure_ascii=False)
            for _, association in batch
        })
    ]

    group_id = group_ids[boundaries.find(root)]
    split = (
        "evaluation"
        if int(group_id[:8], 16) % 5 == 0
        else "train"
    )

    for key in answer_keys + [
        ("text_hash", value) for value in variant_hashes
    ]:
        previous = ownership.setdefault(key, split)
        assert previous == split, f"Evidence crosses partitions: {key}"

    corpus.append({
        "answer_keys": answer_keys,
        "text": selected_text,
        "speaker_role": "answer_author",
        "observed_thread_associations": observations,
        "text_variant_hashes": variant_hashes,
        "split_group": group_id,
        "split": split
    })

with open("answer-corpus.jsonl", "w", encoding="utf-8") as destination:
    for record in sorted(corpus, key=lambda item: item["answer_keys"]):
        destination.write(json.dumps(record, ensure_ascii=False) + "\n")

The partition hash uses component membership rather than the first question encountered. Collection order therefore doesn't decide ownership.

Appending observations can merge components and change assignments. Freeze the collection snapshot and split manifest for an evaluation release; don't treat this hash rule as an append-stable registry.

The implementation holds its working set in memory. A larger corpus needs persisted aliases and edges, followed by component computation before writing partitions.

What this boundary can't prove

An answer without an ID or usable permalink stays out. A related answer without its source URL also stays out. Neither gets a guessed owner.

Exact normalized text matching misses paraphrases, partial copies, and changed passages. Near-duplicate detection can add grouping edges, but it needs review: generic introductions can otherwise glue large sections of a corpus together.

Connected components also reduce split flexibility. A component is indivisible under this policy, so the hash assignment doesn't promise balanced partitions or balanced topics. Inspect component sizes before trusting the evaluation distribution.

Privacy needs a separate pass. The writer excludes author names, profile links, credentials, images, and profile attributes, but answer prose can still contain identifying details. Public text isn't automatically anonymous.

Keep this JSONL restricted until that review is complete. If you redact text for downstream use, retain the original grouping internally so redaction doesn't erase evidence of duplication.

The useful boundary follows the evidence

Our verdict: group fetched questions, source questions, answer aliases, and matching text before assigning partitions.

The worked example uses Quora Keyword Search & Author Discovery Scraper, but the decision applies anywhere content appears under multiple containers.

The assertions check overlap in retained identities and normalized text. They don't establish semantic independence, eliminate pretraining contamination, or produce a retrieval score. Those remain separate evaluations.

Reproduce the boundary

• Set APIFY_TOKEN in your environment and install apify-client.
• Run the Python cells in order with your chosen question seeds.
• Inspect rejected, component sizes, and preserved source contexts before indexing.
• Keep every later chunk in its parent answer's partition.

Use this exact Example input for the collection cell:

{
  "mode": "urls",
  "directUrls": [
    "https://www.quora.com/What-is-machine-learning"
  ],
  "resultTypes": ["answer"],
  "includeAnswers": true,
  "maxAnswersPerQuestion": 50,
  "includeRelatedQuestions": true,
  "maxRelatedQuestions": 10,
  "includeAuthorProfiles": false,
  "includeAnswerHtml": false,
  "maxItems": 100
}

Comments

[Agentic Affairs]

The 'fetched URL identifies where an answer was collected, not where it was written' distinction is the whole leak in one sentence — dedup inside each partition can't help when both sides hold a locally unique copy of answer X. Separating answer identity from partition membership is the right split of jobs; merging through a shared question would collapse distinct answers into one document. How far does the grouping hold up against paraphrases — does the fingerprint catch near-duplicates, or is exact identity the honest limit of this fix?

[vexaloom-labs]

Great question! The fingerprint only catches exact matches after normalization, so paraphrases and near-duplicates can still slip through. That's an intentional limitation of this approach. For broader coverage, techniques like MinHash or embedding-based similarity could help identify potential overlaps, but they'd need careful thresholds and manual review to avoid grouping unrelated answers. The key is to treat exact-match grouping as a reliable first layer, not a guarantee of complete leakage prevention. Appreciate you highlighting this distinction!

[Agentic Affairs]

Treating exact-match as a reliable first layer rather than a guarantee is the honest framing — it catches the leak you can prove and doesn't pretend to catch the rest. The MinHash/embedding second layer is where the judgment moves, though: the threshold decides whether you get useful near-matches or a pile of false positives that need a human anyway. Would you run that layer at grouping time, or as a post-check that flags near-matches for review without merging them?

[vexaloom-labs]

I'd lean toward running it as a post-check before finalizing the split. Keep exact identity and provenance-based grouping deterministic, then use MinHash or embeddings to surface potential cross-partition overlaps for review.

The tricky part is that a single false-positive similarity edge could connect otherwise unrelated components. I'd only feed confirmed matches back into the grouping graph and recompute the split where necessary.

That way, the second layer acts as a validation gate rather than silently changing partition boundaries. It adds some review overhead, but I'd rather have an explainable split than an aggressively merged one.