Short answer: keep identity, observations, and evidence as separate records, then make semantic retrieval propose matches that a deterministic policy can verify. For real-estate listing discovery, that means a changed rent or unit number creates a new observation without erasing the citation that justified the match. The same shape works for an e-commerce catalog where the job is detecting semantically near-duplicate product records.
I would make the relational record the source of truth and treat lexical and vector indexes as disposable projections. A single mutable “listing text” field is attractive on day one; six months later it makes a stale price look current, and it gives an auditor no way to tell which wording produced a match. Grounding is a data-model decision, not a prompt trick.
How do schema design and retrieval architecture keep real-estate discovery grounded?
Start with three identities: the business entity, the observed revision, and the evidence span. A listing or product has a stable entity_id. Each feed observation gets an immutable revision_id, an observation time, and a content hash. Typed facts such as price, area, currency, brand, or SKU live on the revision, while a source span stores the URL, retrieval time, and offsets for the exact text used as evidence.
| Record | Example fields | Operational purpose |
|---|---|---|
| entity | entity_id, canonical_key, lifecycle_state | Stable identity and merge target |
| revision | revision_id, entity_id, observed_at, body_hash | Time-bounded content and replay |
| fact | revision_id, name, value, unit, confidence | Exact filters without parsing prose |
| source_span | revision_id, url, retrieved_at, start, end | Citation and audit trail |
| vector_ref | revision_id, model_id, vector_key | Reproducible semantic candidates |
Canonical URLs are useful hints, not identity keys. Syndicated property pages and mirrored product feeds routinely give the same entity several URLs. Normalize addresses or product identifiers, but preserve apartment or package variants; dropping a unit number can merge two homes, while dropping pack size can merge two otherwise identical products.
The retrieval-augmented generation (RAG) pattern is useful here because retrieved evidence conditions the generated response, but the model should receive the revision and source span alongside the text. The original RAG paper describes this separation between a retriever and a generator; the paper is listed in References. A citation that contains only a URL is weak: the page can change before someone checks it.
Build a candidate pipeline before you decide a similarity threshold
Near-duplicate detection is a ranking problem followed by an identity decision. First apply hard constraints that are cheap and explainable: locale, currency, active date window, SKU or address tokens, and feed permissions. Then retrieve a wider semantic candidate set, and finally rerank using features such as token overlap, normalized identifiers, image hashes where permitted, and revision freshness.
Do not let one score silently become truth. A shared address with a new rent is usually a new revision of one listing; two descriptions with similar marketing copy but different addresses may be a syndication family. Keep both source records and record the reason for grouping. Borderline pairs should enter a review queue rather than being auto-merged.
Here is the boundary I use in Go. It returns evidence objects, so a caller cannot accidentally discard provenance while swapping an index implementation.
package retrieval
import (
"context"
"time"
)
type Evidence struct {
EntityID string
RevisionID string
Field string
Value string
Quote string
SourceURL string
Retrieved time.Time
}
type Query struct {
Text string
Filters map[string]string
MaxResults int
}
type Retriever interface {
Search(context.Context, Query) ([]Evidence, error)
}
The interface is deliberately boring. A self-hosted index, a managed search service, or a test fixture can implement it, while the policy layer still checks revision state and citation completeness. That separation is a practical portability test: replace the index and the evidence contract should remain unchanged.
Where do real systems lose identity, freshness, and citations?
Temporal leakage is the expensive one. If a vector key uses entity_id instead of revision_id, a re-index can pair today’s availability with yesterday’s description. Imagine a property feed that changes a two-bedroom unit to “leased” at 09:00, then republishes an older description at 09:07: an index keyed only by the listing ID can retrieve the old text while the filter reads the new status. The generated answer sounds coherent because both fragments mention the same address, yet neither revision was true at the same time. Identity collapse follows when normalization strips unit, region, or package qualifiers. Citation drift appears when an answer stores a source URL but not the quoted span and retrieval timestamp.
Parser coercion causes quieter damage. 1,200 sqft and 1.200 m2 are different values, and a missing currency is not an invitation to infer one. Store units explicitly; reject or quarantine records that cannot be parsed. Confidence describes evidence quality, not permission to invent a field.
One sentence is enough: stale evidence is a correctness bug.
Keep the evidence.
I use labeled pairs to tune thresholds: confirmed duplicate, same entity with a revision, related but distinct, and unrelated. Report false-merge and false-split rates separately. I am not sure a threshold learned in one city or catalog transfers to another language, feed mix, or embedding model, so every threshold change should carry a dataset version and model identifier.
Verify the contract with SLOs, fixtures, and rollback
Set separate SLOs for ingestion freshness, retrieval quality, and citation coverage. As a starting point, an ingestion objective might require 99% of accepted revisions to become searchable within 15 minutes; that is a proposed target, not a universal standard. Track observation-to-index lag, vector lag, conflict counts, and the share of results whose revision is no longer current.
Capacity planning must use bursts. A feed averaging 50,000 updates per day is under one revision per second, yet a partner can deliver many minutes of traffic in one batch. Size the queue for a short 10x burst, then validate the assumption with production histograms rather than trusting the average.
Fixtures should include address variants, changed prices, copied descriptions, deleted records, missing units, and conflicting amenities. Tests should prove that a revision change creates a new vector reference, that a merge preserves every source span, and that generation refuses an unsupported claim. A 422 from a validation boundary is more useful than silently accepting malformed facts.
Rollback is a data operation: pause the offending feed, mark affected revisions inactive, restore the last trusted index snapshot, and replay from a known cursor. Retain snapshots for audit and legal retention; deleting the old evidence makes a later explanation impossible. I keep the cursor, normalization release, and model identifier together in the deployment record, because restoring vectors without restoring the parser can recreate the same false merge. The extra bookkeeping feels tedious during a quiet launch, but it is the difference between a reversible index rebuild and a forensic exercise after a catalog correction.
Choose boundaries that fit the team and the workload
| Boundary | Good fit | Trade-off |
|---|---|---|
| Relational truth plus replaceable indexes | Teams needing strong citations and rebuilds | Synchronization and queue operations are yours |
| Search platform owns schema and vectors | Fast prototype with modest audit needs | Export, filtering, or revision semantics may be limited |
| Fully transactional search path | Sub-second updates with strict read-after-write | Higher coupling and operational complexity |
For a small platform team, the first boundary is usually the least surprising: the database owns identity and evidence, while indexes can be rebuilt from revisions. It is not suitable when global inventory must be read-after-write in a few milliseconds and asynchronous indexing is unacceptable; a tightly integrated transactional design may be the better choice. Stick with lexical filters when users ask only for exact addresses, SKUs, or prices and you do not have labeled semantic judgments yet.
The result is a retrieval architecture that makes a duplicate decision explainable. Search can improve, embedding models can change, and feeds can be replayed without rewriting the evidence trail that supports each result.
Top comments (0)