DEV Community

JorisRhodes8286
JorisRhodes8286

Posted on

Real-Estate Retrieval with Postgres: Incremental Indexing for Listing Discovery in 2026

Short answer: keep the transactional listing record authoritative, then update a separate retrieval index from a durable change stream. Re-embedding every listing on every search is the least complex way to create a bill and a freshness problem at the same time.

For real-estate discovery, incremental indexing is mostly a data-lifecycle decision. The search result has to reflect a price edit, a withdrawal, and a new amenity without making the application wait for a large batch job. RAG can add useful context around a listing, but it cannot repair an index that still contains a property marked as available after it was withdrawn.

Start with the bill and the retention policy

The bill is made of events, not searches alone. A listing creates an initial embedding, edits create re-embedding work, and deletes create tombstone or removal work. Storage keeps vectors, text, metadata, and often an audit trail. The dominant term is usually the number of changed chunks multiplied by embedding work, so the first useful measurement is a change histogram, not a dashboard of query latency.

I model the monthly work as a small ledger:

Work item Driver Retention question
Initial indexing new or imported listings Do we need old versions?
Update indexing changed searchable fields Which fields actually changed?
Removal withdrawn or expired listings How quickly must it disappear?
Retrieval storage chunks, vectors, metadata When can stale versions be deleted?
Evaluation judged queries and snapshots Which test data must remain reproducible?

That last column is where teams quietly spend money. Keeping every historical vector makes rollback easy, but it also lets a stale description win a nearest-neighbor search if the filter is wrong. I retain a compact event record and the current searchable projection; I keep old vectors only for a deliberately bounded evaluation window. The catch is operational: a late event can require replaying that projection, and a short retention window makes an old incident harder to reconstruct.

I started out assuming that a five-minute batch would be “fresh enough.” Then a listing's status changed during a batch window and the result page continued to surface it. The fix was not a faster model. It was an explicit freshness contract: availability changes are applied synchronously to filters, while semantic text changes are queued for embedding.

One sentence is enough here.

How should incremental indexing shape retrieval architecture for listing discovery?

Use two paths with one source of truth. The write path stores the normalized listing and emits a change event containing an immutable listing identifier, a revision, the changed fields, and the event timestamp. A worker consumes that event, builds the searchable projection, chunks only the affected text, and writes vectors tagged with the same revision. A query joins semantic candidates with current transactional filters before it returns anything.

The revision is more important than the queue technology. If events arrive out of order, the indexer must compare revisions and ignore an older event that would overwrite a newer projection. If a worker retries after a timeout, the vector write should be idempotent for (listing_id, revision, chunk_id). These keys make a replay boring, which is exactly what you want at 02:00.

Here is a deliberately vendor-neutral worker sketch. The interfaces stand in for a queue, an embedding service, and a vector store; the ordering and tombstone behavior are the parts worth preserving.

from dataclasses import dataclass
from typing import Iterable


@dataclass(frozen=True)
class ListingChange:
    listing_id: str
    revision: int
    changed_fields: tuple[str, ...]
    status: str


def index_change(change: ListingChange, store, embed, chunker) -> None:
    current = store.current_revision(change.listing_id)
    if current is not None and change.revision <= current:
        return

    if change.status != "active":
        store.tombstone_listing(change.listing_id, change.revision)
        store.mark_revision(change.listing_id, change.revision)
        return

    listing = store.read_listing(change.listing_id)
    text = store.searchable_text(listing)
    chunks: Iterable[str] = chunker(text, changed_fields=change.changed_fields)

    for chunk_id, chunk in enumerate(chunks):
        vector = embed(chunk)
        store.upsert_vector(
            listing_id=change.listing_id,
            revision=change.revision,
            chunk_id=chunk_id,
            text=chunk,
            vector=vector,
        )

    store.remove_older_revisions(change.listing_id, change.revision)
    store.mark_revision(change.listing_id, change.revision)
Enter fullscreen mode Exit fullscreen mode

The tombstone branch is intentional. Deleting only from the primary database leaves a race in which a vector candidate is returned between filtering and cleanup. A tombstone lets the retrieval layer reject the candidate immediately, while a later compaction removes its storage. For a withdrawn property, that is a correctness feature, not an optimization.

Chunking and freshness are coupled

Chunk boundaries decide how much work an edit creates. A long description split into fixed windows may cause a small change near the beginning to rewrite many chunks. Field-aware chunks are easier to reason about: keep address and price metadata structured, keep the description and amenities as separate text units, and attach the listing revision to every unit. Do not embed fields that should be exact filters, such as price ranges or availability dates; apply those as predicates.

Freshness should be measured by event age, not by worker uptime. Track the time from source revision to visible revision, the age of the oldest unprocessed event, and the count of active listings whose vector revision is behind the transactional revision. Alert on those measures separately. A queue can be empty while a failed producer is emitting nothing, and a queue can be busy while the visible index is still within its contract.

The retrieval pipeline then has a clear order:

  1. Parse the query into exact constraints and semantic intent.
  2. Apply safe constraints such as status, geography, and price to candidate retrieval.
  3. Fetch semantic candidates from the current revision set.
  4. Re-rank with structured signals and return the revision used.

That returned revision is useful to support staff. When someone asks why a property appeared, you can identify the source snapshot and the chunk that produced the match instead of guessing from a mutable page.

Failure modes worth testing before launch

Test the ugly transitions, not just a clean insert. Send an update, a withdrawal, and a reactivation in rapid succession. Deliver those events out of order. Replay the same event. Kill a worker after it writes one chunk but before it records the revision. The expected result is one active revision, no stale candidate, and a retry that converges without multiplying vectors.

A useful test fixture has one listing with a changed price, one with changed amenities, and one that is withdrawn while its description is still queued. The assertions should inspect both the transactional filter and the semantic index. A green embedding call does not prove that a withdrawn listing is gone.

Keep observability tied to identifiers: listing ID, source revision, index revision, event ID, and model version. Log hashes of text rather than raw contact details. Real-estate feeds can contain phone numbers and applicant information, so retention and access controls are part of retrieval architecture, not a later compliance patch.

There is a boundary to this design. If the product needs legal or financial audit reconstruction for every historical listing state, aggressive vector retention is not suitable; keep an immutable archive and accept the storage and deletion burden. If the feed is tiny and changes are rare, a scheduled rebuild may be simpler than operating a change stream. Stick with a batch index when that contract is explicit and users do not depend on minute-level availability.

I'm not sure a universal freshness target exists. Your mileage will vary with feed quality, moderation queues, and how dangerous a stale result is. Write the target down, measure it from source events, and choose the simplest architecture that can meet it.

References

Further reading

Top comments (0)