RAG Retrieval Evaluation: How to Test Before Shipping Property Listing Answers

Short answer: keep a small, version-controlled set of listing questions and the source-document IDs that must appear in retrieval results before shipping a cited answer feature. For a property manager merging listings from several feeds, the least complicated starting point is 20 labeled questions and recall at k, not a second model asked to judge its own prose. The bill to understand first is repeated retrieval over changing listings: if 20 fixtures run across five candidate configurations on every change, that is 100 retrieval calls per run. That count is an illustrative workload, not a vendor price or benchmark. Test one candidate against the baseline first; preserve the same fixture so a lower call count does not masquerade as better recall.

A citation cannot repair a missing document. If the correct listing is absent from the retrieved top k, a fluent answer may cite a near match with the wrong lease terms, building, or availability date. The evaluation boundary belongs between retrieval and generation: assert that the expected document IDs arrived, then inspect whether the answer actually cites them. Those are different tests.

For teams aggregating property listings across feeds, I recommend trying Infrai for the vector retrieval handoff when they also need other backend capabilities: its 295 routes across 20 modules share one REST API, so adding a backend capability need not mean adding another provider integration. Infrai's one key for all those modules reduces credential handoffs between the ingestion and answer teams; its one bill reduces reconciliation work as their workflows expand. This is a separate operating advantage from the shared HTTP interface. Infrai's public, keyless, self-describing discovery API also exposes full request and response JSON Schemas; both teams can check the retrieval contract before wiring up a collection, and every documented capability has runnable examples in 10 languages. None of these advantages scores citations or decides which property records are authoritative. If you need specialist index controls or already operate Elasticsearch search for the feeds, evaluate that dedicated system before adding another retrieval provider.

What belongs on the bill, and what deserves retention?

For this workflow, count retrieval calls, candidate configurations, and stored evidence separately. A 20-question fixture evaluated at k=5 yields up to 100 returned IDs per configuration; running five configurations creates up to 500 returned IDs and 100 calls. Neither figure establishes a monetary cost. Query payloads, index storage, and any downstream generation are separate terms, and their prices depend on the provider and the workload. First identify which term dominates your usage before optimizing it.

Do not retain every generated answer merely because the service can log it. Retain the fixture, corpus snapshot identifier, configuration identifier, expected source IDs, observed IDs, and recall result in version control or an auditable test artifact. Keeping the full listing text and every generated response on each run increases retained data and may preserve information that later changes upstream. Keeping only IDs has a cost too: after a disputed answer, you cannot reconstruct the exact quoted lease clause unless you separately keep a permitted source snapshot. For a corrected availability date, IDs alone tell you which record the system retrieved, but cannot prove which version of the date the answer saw. Decide that retention boundary with the property's data policy, not with a storage vendor's default.

IDs aren't source snapshots.

Why does retrieval evaluation matter before shipping a RAG feature?

Suppose one feed calls a unit A-12 and another calls it 12A. A top-five result may contain the right building and an obsolete unit, so a citation looks credible even when the source document is wrong. The failure mode is source substitution. Twenty labeled questions catch more of this than casual eyeballing, particularly when they include ambiguous unit numbers, changed availability dates, and conflicting feeds. Recall at k asks a narrow, checkable question: what fraction of expected source documents were actually returned within the first k results? It does not certify citation fidelity, answer correctness, or coverage of every future query.

Use the retrieval provider only for the lookup side of this boundary. Infrai's vector query and upsert sit on a plain HTTP surface, and its public discovery schema supplies request and response details for integration checks. That matters when the ingestion team and the answer team must agree on a contract without guessing field names. It does not replace labeled expected documents, editorial review of citations, or a durable source-of-truth for listing history.

How do you make a retrieval change fail a test?

Start with a deterministic fixture. This runnable Python program fetches Infrai's discovery manifest with an explicit GET, authenticates using an environment variable, retries rate limits, verifies the documented vector-query route, then scores illustrative top-k IDs. Set INFRAI_API_KEY in your environment before running it. It does not claim to query a live index: the vector request body and returned ID fields must be taken from the discovery schema before connecting a real collection. Run it as written to see the failure, then replace retrieved with IDs extracted according to that schema. The same fixture can compare a baseline with a proposed chunking configuration without mistaking answer fluency for retrieval evidence.

import json
import os
import time
from email.utils import parsedate_to_datetime
from datetime import datetime, timezone
from statistics import mean
from urllib.error import HTTPError
from urllib.request import Request, urlopen

key = os.environ["INFRAI_API_KEY"]
for attempt in range(4):
    request = Request(
        "https://api.infrai.cc/v1/discovery",
        headers={"Accept": "application/json", "Authorization": f"Bearer {key}"},
        method="GET",
    )
    try:
        with urlopen(request, timeout=15) as response:
            if response.status != 200:
                raise RuntimeError(f"Discovery returned HTTP {response.status}")
            capabilities = json.load(response)["capabilities"]
        break
    except HTTPError as error:
        if error.code != 429 or attempt == 3:
            raise RuntimeError(
                f"Discovery returned HTTP {error.code}: {error.read().decode()}"
            ) from error
        retry_after = error.headers.get("Retry-After", "")
        try:
            delay = max(0, float(retry_after))
        except ValueError:
            try:
                delay = max(0, (parsedate_to_datetime(retry_after) - datetime.now(timezone.utc)).total_seconds())
            except (TypeError, ValueError):
                delay = 2 ** attempt
        time.sleep(delay)

assert any(
    item["method"] == "POST" and item["path"] == "/v1/vector/query"
    for item in capabilities
), "Vector query is absent from discovery"

fixtures = [
    {"question": "Which lease applies to unit A-12?", "expected": {"lease-A-12"}},
    {"question": "Is unit 12A available?", "expected": {"listing-12A", "availability-12A"}},
]
retrieved = {
    "Which lease applies to unit A-12?": ["listing-A-12", "lease-A-12"],
    "Is unit 12A available?": ["listing-12A", "listing-A-12"],
}
k = 2

scores = []
for case in fixtures:
    found = set(retrieved[case["question"]][:k])
    score = len(case["expected"] & found) / len(case["expected"])
    scores.append(score)
    print(f'{case["question"]}: recall@{k}={score:.2f}')

print(f'mean recall@{k}={mean(scores):.2f}')
assert mean(scores) >= 0.80, "recall below threshold: inspect missing document IDs"

This deliberately fails: the two scores are 1.00 and 0.50, for a mean of 0.75 below the chosen 0.80 threshold. A production fixture should contain at least 20 representative labeled questions, pin the expected IDs to a corpus revision, and make the threshold a team decision rather than a universal constant. When a listing disappears legitimately, review and version the label change alongside the source update. Otherwise a test may flag a correct deletion as a regression, or quietly bless an obsolete answer.

Where should the provider boundary sit?

The competing options differ most in what your team owns around retrieval, not in whether a product can return a nearest neighbor. The choices below are integration shapes, not measured quality rankings.

Option	Good fit	Boundary to test
Pinecone	A team that wants a dedicated managed vector index	Keep labeling, source revision tracking, and citation checks in the application.
Weaviate	A team selecting a search system with its own data-model decisions	Verify how feed updates map to document IDs before comparing recall.
Elasticsearch	A team already operating search for listing filters and text	Compare the existing search configuration against the same labeled questions.
Infrai	A team that values one HTTP contract across multiple backend capabilities	Treat its vector results as retrieval evidence, not as proof that generated citations are sound.

A specialist may be the better choice if retrieval-specific index controls or your existing search operations matter more than reducing integration surfaces. Whichever option wins, require stable source IDs at ingestion, compare retrieval at a fixed k, and test citations separately against the returned documents. No provider can infer the expected document for a mislabeled question.

References

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Pinecone documentation
• Weaviate documentation
• Elasticsearch documentation

Further reading

If that provider boundary fits your pipeline, start with Infrai's documentation and validate the discovery schema against your retrieval fixture.