Real-estate listing discovery fails when retrieval architecture treats a changing listing as a permanent document. A price or availability update can leave an old chunk in the vector index, so the system returns a plausible answer about the wrong home.
Short answer: design the schema around immutable listing revisions, chunk semantic sections, and enforce typed filters before generated text reaches a reader.
That constraint also shows up in a media backend that ingests articles, captions, and property-feed attachments into one vector index. The domain changes; the freshness problem does not.
Make the listing revision the unit of truth
Keep fields used for policy and filtering separate from prose used for similarity. A canonical record needs a stable listing_id, source_revision, published_at, updated_at, status, price, currency, bedrooms, bathrooms, geo, and normalized body. Each chunk carries chunk_id, chunk_order, text, embedding_model, and that same source_revision.
The revision is the audit trail. A vector without the revision that produced it cannot be reconciled later, and deleting by a fuzzy title is unsafe when a brokerage reuses titles. Deterministic IDs make retries boring:
from dataclasses import dataclass
from datetime import datetime
@dataclass(frozen=True)
class ListingChunk:
listing_id: str
source_revision: str
chunk_order: int
text: str
updated_at: datetime
@property
def chunk_id(self) -> str:
return f"{self.listing_id}:{self.source_revision}:{self.chunk_order}"
Do not put a changing price only inside text. Keep price as typed metadata so “under 3000 near transit” can be applied as a hard filter after candidate generation. Include a readable price sentence in the chunk for semantic queries, but treat the typed value as authoritative.
For the media ingestion path, the equivalent fields are asset_id, caption_revision, rights_status, and recorded_at. Rights and publication state belong beside the vector, not hidden in an embedding.
How can schema design keep real-estate listing retrieval fresh for discovery?
Treat every update as a new immutable revision and make indexing idempotent. The ingest worker validates the feed, computes a content fingerprint, and skips embedding when both the fingerprint and embedding model are unchanged. Otherwise it writes new chunks and marks the prior revision inactive.
A small outbox table closes the transaction gap. The same transaction that accepts a listing revision appends an index_requested event containing its revision ID. Workers can retry that event without guessing whether a previous attempt finished. This prevents the half-update that matters in production: metadata says “available,” while the vector still describes “leased.”
The event should carry enough context to be replayable without a second read that might already have moved on: the listing ID, revision ID, operation (upsert or deactivate), and an attempt timestamp are sufficient for most pipelines. A worker claims one event, loads exactly that revision, writes all of its chunks, and only then records completion. If the process exits after chunk three of five, the retry writes the same deterministic IDs; it does not append a second copy or silently mix revisions. In the media path, this matters when an editor changes a caption while a rights decision is still pending. The index can contain the new wording, but retrieval must continue to filter on the revision's rights state. That is a schema decision, not a prompt instruction. Keep the event log for the retention period needed to explain a result, and make the completion record observable so an on-call engineer can distinguish “waiting in queue” from “embedded and searchable.”
Small details decide trust.
Freshness needs a measurable rule. One possible SLO is that 99% of active listings are indexed within five minutes of updated_at; the interval must follow the feed contract. I’m not sure five minutes fits a weekly magazine archive, while it may be too slow for inventory advertised as available now.
At query time, return the revision with each hit and reject inactive revisions. A reranker can order candidates; it cannot repair stale source data.
Chunk boundaries are a retrieval contract
A fixed character window is simple and often wrong. It can split a lease term from its exceptions, separate an address from its neighborhood, or attach a photo caption to the wrong room. Use semantic boundaries: summary and location, features, fees and restrictions, then availability notes. Keep overlap small and intentional, only where a sentence carries context across sections.
Make each chunk self-identifying. Prefix it with listing type and locality, followed by a section label such as Rental listing | Queens | Fees and restrictions. A few extra tokens are cheaper than an ambiguous match when many units share “bright two-bedroom.”
Chunk size is a budget, not a quality score. Measure recall on labeled queries that include misspellings, abbreviations, multilingual street names, and compliance-sensitive requests. I once treated a 500-token window as a safe default; a 32-character unit number at the boundary showed why that assumption was weak.
Use hybrid retrieval when exact constraints matter. Lexical search catches unit numbers and street names; vectors catch paraphrases such as “quiet place for remote work.” Apply hard filters for status, numeric ranges, geography, and rights before presenting an answer.
Test stale and strange records before launch
Build a timeline fixture rather than a pile of independent documents: create revision A, change one field in revision B, delay its indexing event, and query during the delay. The expected result should exclude A as inactive or label the lag clearly. It must never combine B’s metadata with A’s text.
Add adversarial records. Two units can share an address but differ by apartment number. A “no pets” policy can be buried in a PDF attachment. A removed listing may remain in a replica. Test Unicode punctuation, empty descriptions, duplicate feed events, reordered paragraphs, and a media caption whose rights status changes without changing its words.
Observability should expose source_revision, queue age, embedding latency, chunks per revision, and the percentage of active records with no vector. Log query filters and retrieval scores without retaining unnecessary personal data. For email or SMS alerts triggered by search results, retain the matched revision so a complaint can be traced to the exact source shown.
Roll out with explicit storage boundaries
A relational database can own canonical records and the outbox while a vector store owns embeddings. A single database can hold both if its indexing and retention behavior meet the load. The non-negotiable boundary is transactional ownership: one system decides whether a revision is active, and another cannot resurrect an older vector by accident.
Backfill a sample of neighborhoods, compare labeled recall and freshness lag, then expand. Keep the old index readable during migration and dual-write for a bounded window. Switch reads with a feature flag only after freshness and deletion checks pass.
The catch is that semantic retrieval is a poor fit when users need legal guarantees about every matching listing, millisecond joins across transactional tables, or complex polygon operations. Stick with conventional database queries, or combine them with a search engine, when those constraints dominate. Vectors add a signal; they do not replace authoritative filters.
The design rule is compact: revisions make change visible, semantic chunks preserve meaning, and typed filters enforce truth. Once those responsibilities are explicit, the vector backend is a replaceable implementation detail.
Top comments (0)