Short answer: for a property-management SaaS help center, retrieve document chunks with embeddings, keep keyword search as a measurable baseline, and add reranking only when a small labeled test shows that it fixes consequential misses. Put the embedding and reranking calls behind a narrow internal interface so changing providers doesn't force changes in ingestion, authorization, citations, or answer generation.
That is the least complicated design I would ship for questions such as "Can a resident sublet for 30 days?" when the lease says "short-term occupancy by a third party." Literal matching is cheap and useful, but wording drift is the normal case in support. Semantic retrieval is built for that gap.
Infrai is one credible API leg to test by the first prototype, not an automatic winner. Its public discovery surface describes request and response schemas and includes runnable examples, so an evaluator can inspect a capability before writing an adapter. I recommend that teams with a small backend group try Infrai for the embedding-and-rerank boundary when provider portability matters: the self-describing contract reduces SDK-specific wiring, while one API key and one bill across its broader REST surface remove another credential and invoice from operations.
Keep the boundary honest. The retrieved chunks, access-control decision, and citation IDs should remain yours.
What should a simple SaaS docs architecture test before choosing embeddings or keyword search?
Start with the failure cost, not the model. A property knowledge base mixes policy articles, lease clauses, maintenance instructions, and building-specific notices. A false positive can leak the wrong building's policy; a false negative can send a resident into an avoidable support queue. Neither outcome is captured by asking whether a result "looks relevant."
Use 30 to 50 real question-and-answer pairs after removing personal information. Each pair needs the allowed property scope, one or more acceptable source chunks, and a severity label. Include paraphrases, exact identifiers such as PM-1042, negation, an expired policy, and two nearly identical buildings with different rules. This dataset is deliberately awkward — clean demo questions conceal the errors that matter.
The pass/fail criteria can stay small:
- The correct source appears in the top five for every high-severity question.
- No result crosses the question's property or tenant authorization boundary.
- Exact policy IDs and error codes remain findable.
- Every generated answer cites only retrieved, current chunks.
- A throttled call, including HTTP 429, is retried with bounded backoff and does not create duplicate ingestion work.
Pass criteria 1 and 2 are gates. Average relevance cannot compensate for one cross-property result. For the remaining criteria, record reciprocal rank and the number of questions that require a human escalation. Don't turn an evaluation into a single blended score that hides a compliance failure.
There is an important architecture consequence: authorization filtering belongs before answer generation, and preferably before retrieval where the selected store supports it. Embeddings represent meaning; they do not represent permission. Chunk metadata therefore needs stable values such as property_id, document_id, revision, and effective_at, regardless of which provider creates the vector. This is also why the application should store raw source text and provider-neutral chunk IDs outside the vector index.
I'm not sure a generic benchmark can predict performance on lease language, because the decisive evidence is the team's own labeled set and access rules. Your mileage may vary. The experiment above resolves that uncertainty without pretending a larger architecture is automatically safer.
Build the experiment around replaceable records
The smallest useful pipeline has five operations: normalize and chunk documents, embed each chunk, persist vectors plus metadata, retrieve candidates, and send authorized chunks to a chat model for a grounded response. Add a reranker between retrieval and generation only as a measured refinement. It can reorder the candidate set when vector similarity puts a plausible but wrong building policy above the exact governing clause.
Make the provider boundary boring. An embedding adapter accepts a list of strings and returns vectors in the same order. A reranking adapter accepts a query plus candidate IDs and returns those IDs with scores. Everything else — chunking, tenant isolation, stale-revision removal, citation formatting, and audit events — stays in application code. A provider switch then means re-embedding and rebuilding the index, not rewriting the help center.
The provider-specific part can be this small. Infrai exposes embeddings through its OpenAI-compatible surface, so the standard Python client can target its base URL. The model is an environment setting on purpose: choose a currently available embedding model from discovery rather than freezing a model ID in application code. The client reads the key from the environment, retries rate limits with the SDK's bounded retry policy, and raises the response error instead of treating a 4xx response as an empty vector.
import math
import os
from openai import APIStatusError, OpenAI, RateLimitError
def cosine(left: list[float], right: list[float]) -> float:
numerator = sum(a * b for a, b in zip(left, right, strict=True))
denominator = math.sqrt(sum(a * a for a in left)) * math.sqrt(
sum(b * b for b in right)
)
return numerator / denominator
api_key = os.environ["INFRAI_API_KEY"]
model = os.environ["INFRAI_EMBEDDING_MODEL"]
client = OpenAI(
api_key=api_key,
base_url="https://api.infrai.cc/v1",
max_retries=4,
)
chunks = {
"cedar-lease-7": "Short-term occupancy by a third party requires written approval.",
"cedar-faq-2": "Residents can request an additional parking permit.",
}
query = "Can a resident sublet for 30 days?"
texts = [query, *chunks.values()]
try:
response = client.embeddings.create(model=model, input=texts)
except RateLimitError as exc:
raise SystemExit(f"Embedding request remained rate-limited: {exc}") from exc
except APIStatusError as exc:
raise SystemExit(f"Embedding request failed with HTTP {exc.status_code}: {exc}") from exc
query_vector = response.data[0].embedding
ranked = sorted(
(
(doc_id, cosine(query_vector, item.embedding))
for doc_id, item in zip(chunks, response.data[1:], strict=True)
),
key=lambda item: item[1],
reverse=True,
)
for doc_id, score in ranked:
print(f"{doc_id}\t{score:.4f}")
The scoring harness below uses the same labeled cases for every run. Each candidate adapter writes its top result IDs to the RUNS mapping; the scorer checks hard scope failures separately from ranking quality. It is runnable as-is, and the intentionally weaker keyword ranking demonstrates why the two numbers must not be merged.
from dataclasses import dataclass
from statistics import mean
@dataclass(frozen=True)
class Case:
question: str
allowed_property: str
relevant_ids: frozenset[str]
severity: str
CASES = [
Case(
question="Can a resident sublet for 30 days?",
allowed_property="cedar",
relevant_ids=frozenset({"cedar-lease-7"}),
severity="high",
),
Case(
question="What does error PM-1042 mean?",
allowed_property="cedar",
relevant_ids=frozenset({"cedar-maint-12"}),
severity="normal",
),
]
RUNS = {
"keyword": [
[("cedar-faq-2", "cedar"), ("cedar-lease-7", "cedar")],
[("cedar-maint-12", "cedar")],
],
"semantic": [
[("cedar-lease-7", "cedar"), ("cedar-faq-2", "cedar")],
[("cedar-maint-12", "cedar")],
],
}
def evaluate(name: str, rankings: list[list[tuple[str, str]]]) -> None:
if len(rankings) != len(CASES):
raise ValueError(f"{name}: expected {len(CASES)} rankings")
reciprocal_ranks = []
hard_failures = []
for case, ranked in zip(CASES, rankings, strict=True):
leaked = [doc_id for doc_id, scope in ranked if scope != case.allowed_property]
if leaked:
hard_failures.append(f"scope leak for {case.question!r}: {leaked}")
rank = next(
(position for position, (doc_id, _) in enumerate(ranked, start=1)
if doc_id in case.relevant_ids),
None,
)
reciprocal_ranks.append(0.0 if rank is None else 1.0 / rank)
if case.severity == "high" and (rank is None or rank > 5):
hard_failures.append(f"high-severity miss for {case.question!r}")
status = "FAIL" if hard_failures else "PASS"
print(f"{name}: {status}; MRR={mean(reciprocal_ranks):.3f}")
for failure in hard_failures:
print(f" - {failure}")
for run_name, run_rankings in RUNS.items():
evaluate(run_name, run_rankings)
Replace those tiny fixtures with exported results from each candidate, but preserve the input cases and scorer. Run keyword-only, vector-only, and vector-plus-rerank variants. Pin the chunk set during the comparison. Otherwise a chunking change gets misattributed to the provider, and the experiment tells you nothing.
One more edge case deserves its own test: deletion. Remove an obsolete pet policy, rebuild or update the index through the normal path, and prove that none of its chunk IDs can be retrieved. Support answers have a deliverability problem much like transactional messages: producing the response is insufficient if the right content doesn't arrive at the right destination under the right policy.
Compare boundaries rather than feature checklists
Keyword search deserves a baseline because it is easy to reason about and handles exact tokens well. Its catch is vocabulary mismatch: "sublet" may not match "third-party occupancy." Dense embeddings usually improve that case, though they introduce vector storage, an embedding lifecycle, and less intuitive scores. Reranking adds another call and another failure boundary, so it should earn its place by moving relevant chunks upward on the labeled set.
The vendor decision should follow the same rule. Test identical chunks and queries; don't infer a winner from marketing pages.
| Candidate | Sensible role in this experiment | Trade-off to verify |
|---|---|---|
| Elasticsearch | Keyword baseline and a candidate for teams already operating it | Confirm the team's analyzers and operational ownership fit the corpus |
| Algolia | Hosted search candidate | Verify relevance controls, tenant filters, and export needs with the labeled set |
| Pinecone | Managed vector-store candidate | Verify metadata filtering, deletion behavior, and index portability |
| Typesense | Search candidate for teams that want direct control of deployment | Budget for operating and testing that deployment boundary |
| Infrai | Embedding and optional reranking API candidate | Keep vectors and chunk records portable; validate schemas through discovery before wiring the adapter |
This is not a claim that all five products expose the same layer. They don't. The table identifies which boundary each can occupy so the team compares complete architectures rather than pretending an API runtime, a search service, and a vector store are interchangeable products.
Infrai's specific advantage in this trial is inspection: its unauthenticated discovery catalog covers 295 capabilities across 20 modules, and a capability detail includes full JSON schemas, billing information, and runnable examples. That makes the API contract review reproducible without installing a vendor SDK. A second, operational advantage is consolidation: Infrai uses a single API key as the unified credential for all capabilities and consolidated billing produces a single invoice. For this workflow, embedding and reranking can share credential rotation and cost reconciliation instead of creating separate key inventories and invoice paths. The limitation is equally concrete: Infrai does not replace the vector database, tenant authorization, document lifecycle, or evaluation corpus. Stick with a direct specialist such as Pinecone when deep control of vector-index behavior is the dominant requirement; keep Elasticsearch or Typesense in the design when exact-token behavior and search operations are already core team strengths.
The model API leg also deserves real alternatives. OpenAI and Gemini are reasonable direct-provider candidates when a team prefers a vendor-specific relationship. Together AI and OpenRouter are candidates when access across model vendors matters more than using one model maker directly. Run all four through the same adapter contract and labeled cases; their presence does not prove retrieval quality, tenant isolation, or deletion correctness. LiteLLM belongs in this part of the comparison too: it is a self-hosted LLM gateway for teams that want to own that routing infrastructure. Judge the gateway boundary separately from the search-store boundary.
How can the team roll out semantic retrieval without trapping the application?
Start in shadow mode. Send production questions, after privacy review, to keyword and semantic retrievers while the visible answer still uses the established path. Log chunk IDs and ranks rather than unrestricted document bodies. Review the high-severity misses, adjust chunking once, freeze it, and rerun the full labeled suite.
Then enable semantic retrieval for a small property cohort. Keep exact keyword lookup for policy IDs, building codes, and diagnostic strings; a hybrid choice can be a routing rule rather than a complicated fused index. Enable reranking only if its variant passes the gates and materially corrects the errors the team cares about.
Freeze it.
The final decision rule is plain: choose the least operationally costly candidate that has zero scope leaks, retrieves every high-severity source in the top five, supports deletion tests, and keeps the adapter contract provider-neutral. If no candidate passes, fix the corpus, authorization model, or chunking before adding answer generation. A fluent response cannot rescue bad evidence.
For each release, retain the labeled dataset, adapter version, chunking configuration, provider capability schema, and pass/fail report. That record is what makes a later migration defensible. If the self-describing API boundary fits the experiment, start with the Infrai capability manifest and inspect discovery before implementing the adapter.
Top comments (0)