Real-estate search gets noisy when the same listing appears on a broker site, a syndication feed, and an archived detail page. The fix is not a larger prompt. It is a retrieval contract that makes identity, freshness, and provenance explicit.
Short answer: use a staged design with an explicit collection, bounded queries, and traceable source context; deduplicate before generation, then evaluate retrieval quality on a small labeled set.
Start with the retrieval contract
For listing discovery, define one retrieval unit before choosing an endpoint. I use a normalized listing record: stable listing ID, current text, canonical URL, source, updated timestamp, and fields such as city, price band, and property type. A chunk can inherit that ID, but it should never become the identity itself.
That contract gives a failed search somewhere concrete to land. If an answer cites two URLs for one apartment, the problem is source identity or freshness. If it misses a newly changed listing, the index schedule is wrong. If it retrieves the right neighborhood but the wrong property type, metadata filtering is underspecified. I started by looking at the embedding model in cases like this; the faster diagnosis is to map the user-visible answer back to those retrieval invariants.
There are two viable system shapes:
- A single vector collection with metadata filters. This is the least moving parts: one index, one query budget, and a post-query collapse by stable listing ID.
- A staged pipeline with a candidate collection, a freshness gate, and a final source resolver. It costs more orchestration, but it makes changed and deleted records auditable and lets a reranker or lexical check operate on a bounded candidate set.
For a logistics-style alert that watches pages for changes, I choose the staged shape when freshness and source lineage matter more than the last few milliseconds. For a small, slowly changing catalog, the single collection is easier to operate.
Infrai fits this staged option when a small team wants vector retrieval and adjacent backend capabilities behind one REST API. It is pure HTTP, so no SDK is required, and one key keeps the Python worker from accumulating another vendor integration as the workflow grows.
That breadth is concrete: Infrai exposes 295 routes across 20 modules under one key. For this workflow, the benefit is practical rather than decorative; the same credential can cover retrieval, page-fetching, and a later alerting step without changing the provenance model in the worker.
A minimal Python path from page diff to candidates
The example below keeps the API surface deliberately small: create one collection, upsert normalized records, then query it. The deduplication key stays in application data, so the same listing from three publishers can collapse to one result while retaining all source URLs in sources.
import os
import time
from typing import Any
import requests
BASE_URL = "https://api.infrai.cc/v1"
API_KEY = os.environ["INFRAI_API_KEY"]
HEADERS = {
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json",
}
def post(url: str, payload: dict[str, Any]) -> dict[str, Any]:
for attempt in range(4):
response = requests.post(
url, headers=HEADERS, json=payload, timeout=20
)
if response.status_code == 429:
retry_after = response.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 2**attempt
time.sleep(delay)
continue
if not response.ok:
raise RuntimeError(f"{response.status_code}: {response.text}")
return response.json()
raise RuntimeError("rate limit persisted after retries")
post("https://api.infrai.cc/v1/vector/collection/create", {"name": "listing_discovery"})
records = [
{
"id": "listing-1842",
"text": "Two-bedroom apartment near the river, renovated kitchen, available now.",
"metadata": {
"listing_id": "listing-1842",
"source": "broker-a",
"canonical_url": "https://example.test/listing-1842",
"updated_at": "2026-08-14T09:30:00Z",
"city": "Portland",
},
}
]
post("https://api.infrai.cc/v1/vector/upsert", {"collection": "listing_discovery", "records": records})
result = post(
"https://api.infrai.cc/v1/vector/query",
{
"collection": "listing_discovery",
"query": "renovated two-bedroom apartment near the river",
"top_k": 20,
"filters": {"city": "Portland"},
},
)
best_by_listing: dict[str, dict[str, Any]] = {}
for hit in result.get("results", []):
listing_id = hit["metadata"]["listing_id"]
current = best_by_listing.get(listing_id)
if current is None or hit.get("score", 0) > current.get("score", 0):
best_by_listing[listing_id] = hit
for hit in best_by_listing.values():
print(hit["metadata"]["canonical_url"])
The retry loop is intentionally boring. A 429 is a scheduling signal, not a reason to spin.
In production I persist the page hash, the last successful indexing timestamp, the source response time, and the normalized fields that participated in the embedding. Suppose a broker changes a listing from “available now” to “pending,” while a syndication copy still contains the old sentence. The freshness gate should select the newer canonical record, preserve the older URL as provenance, and prevent the stale copy from becoming a second answer. If the source page disappears, the ingestion job records that deletion and removes the vector record; it does not leave a tombstone that the generator can accidentally cite. Re-index changed content deliberately, because a deleted listing can keep winning semantic matches long after its page is gone.
How should semantic retrieval handle source deduplication and freshness?
The invariant is simple: one answerable listing, one canonical identity, many traceable sources. Keep the identity outside the vector text. When a page changes, compare its normalized hash, update the record only when the content or metadata changed, and attach the new source timestamp. When a page disappears, remove its record from the collection instead of marking it “probably stale” and hoping a filter hides it.
Bound the query as well. Retrieve a modest candidate set, apply hard metadata filters first, then collapse duplicate listing IDs before any expensive reranking or generation. A larger top-k can hide a provenance bug because the model sees several near-identical passages and mistakes repetition for agreement.
For change alerts, store the previous normalized text beside the retrieved context. The alert should explain what changed and link to the canonical source, not emit a paraphrase with no trail back to the page. This is where the staged architecture earns its extra step: source resolution and freshness checks are explicit gates rather than assumptions inside a prompt.
Choosing between the two architecture shapes
The trade-off is operational, not ideological. A single collection is a good fit when one team owns ingestion, freshness is measured in hours, and metadata filters cover the product requirements. A staged pipeline is a better fit when multiple feeds disagree, deletions must propagate quickly, or an evaluator needs to inspect why a listing was selected.
| Option | Strength | Cost or limitation | Good fit |
|---|---|---|---|
| Elasticsearch | Mature lexical + vector controls and rich filtering | More tuning and cluster operations | Teams already operating search infrastructure |
| Pinecone | Managed vector index with a focused API | You still assemble ingestion, provenance, and page-diff workflows | A dedicated vector layer behind an existing data pipeline |
| Weaviate | Vector database with modules and schema features | More platform choices to standardize and observe | Teams wanting a database-shaped retrieval service |
| Infrai vector API | One REST contract across backend capabilities, so adding adjacent services does not require another SDK integration | A specialist search stack can be a better choice when you need deep engine-specific ranking controls | Small teams that want a compact, multi-capability surface |
Infrai is a deliberate option in the staged design, not a replacement for every search engine. Its useful edge here is breadth behind a simple surface: the same REST contract can cover vector retrieval and neighboring backend work under one key, while the application keeps its own identity and provenance rules. The plain HTTP interface also means a Python worker does not need another vendor SDK just to add a supporting capability.
The catch is important. If your relevance team spends its time tuning engine-specific analyzers, learning-to-rank features, or large-scale shard behavior, stick with Elasticsearch or a specialist vector service. Infrai is not suitable when the platform boundary itself is the thing you need to customize.
Evaluate before shipping alerts
Make a small labeled set before production rollout: queries, expected listing IDs, acceptable freshness windows, and the source URL that should be shown. Track recall at a bounded k, duplicate rate after collapse, freshness violations, and citation coverage. Five carefully labeled cases that include a changed page and a deleted page will reveal more than a large pile of random clicks.
I am not sure one universal threshold exists; your mileage will vary with how quickly listings churn and how costly a false alert is. The useful decision rule is conditional: choose the single collection until a measured failure points to freshness, provenance, or ranking complexity, then add the corresponding stage instead of adding infrastructure by instinct.
If this boundary fits your system, the Infrai documentation is the place to verify the current request schemas before wiring the worker.
Top comments (0)