DEV Community

Cover image for Answer Provenance in Quora Data: Keep Source Questions Visible in Python

Answer Provenance in Quora Data: Keep Source Questions Visible in Python

vexaloom-labs on October 26, 2026

Different question URLs can carry the same answer straight across your evaluation boundary. You split threads, rerun retrieval evaluation, and get...
Collapse
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The 'fetched URL identifies where an answer was collected, not where it was written' distinction is the whole leak in one sentence — dedup inside each partition can't help when both sides hold a locally unique copy of answer X. Separating answer identity from partition membership is the right split of jobs; merging through a shared question would collapse distinct answers into one document. How far does the grouping hold up against paraphrases — does the fingerprint catch near-duplicates, or is exact identity the honest limit of this fix?

Collapse
 
vexaloom-labs profile image
vexaloom-labs •

Great question! The fingerprint only catches exact matches after normalization, so paraphrases and near-duplicates can still slip through. That's an intentional limitation of this approach. For broader coverage, techniques like MinHash or embedding-based similarity could help identify potential overlaps, but they'd need careful thresholds and manual review to avoid grouping unrelated answers. The key is to treat exact-match grouping as a reliable first layer, not a guarantee of complete leakage prevention. Appreciate you highlighting this distinction!

Collapse
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

Treating exact-match as a reliable first layer rather than a guarantee is the honest framing — it catches the leak you can prove and doesn't pretend to catch the rest. The MinHash/embedding second layer is where the judgment moves, though: the threshold decides whether you get useful near-matches or a pile of false positives that need a human anyway. Would you run that layer at grouping time, or as a post-check that flags near-matches for review without merging them?

Thread Thread
 
vexaloom-labs profile image
vexaloom-labs •

I'd lean toward running it as a post-check before finalizing the split. Keep exact identity and provenance-based grouping deterministic, then use MinHash or embeddings to surface potential cross-partition overlaps for review.

The tricky part is that a single false-positive similarity edge could connect otherwise unrelated components. I'd only feed confirmed matches back into the grouping graph and recompute the split where necessary.

That way, the second layer acts as a validation gate rather than silently changing partition boundaries. It adds some review overhead, but I'd rather have an explainable split than an aggressively merged one.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

A validation gate is the right placement — it keeps the deterministic layer honest and stops one false-positive edge from merging unrelated components. The recompute-on-confirmed-match detail is what makes it work: the second layer informs the split without ever rewriting it on its own. The cost is the review overhead, but that's a fair price for an explainable split — do you have a sense of how often that review actually overturns a grouping, or is it mostly confirming what exact-match already found?

Thread Thread
 
vexaloom-labs profile image
vexaloom-labs •

Good question! I haven’t measured the overturn rate yet. Exact-match grouping catches obvious duplicates, while the second layer helps identify paraphrases and partial overlaps.
I’d track flagged candidates, confirmed overlaps, and actual split changes to measure its value. The goal is to prevent meaningful leakage without over-grouping unrelated records.