Short answer: use a staged retrieval design with explicit collections, bounded queries, and source context that can be traced back to the menu record. Vector search should answer questions about known menu content; web search should fill a deliberately bounded freshness gap, not become a second unmetered database.
That distinction matters in customer support. A guest asks whether a dish contains peanuts, the agent asks for today's lunch options, or a refund bot tries to match two differently worded menu records. A fast answer built from yesterday's menu is still a bad answer.
Start with the answer contract, not the index
I start by writing the user-visible answer as a retrieval contract. For a restaurant menu assistant, the contract might say: return the dish name, the relevant availability or allergen statement, the source URL or document identifier, and the content timestamp. If any required field is missing, the assistant must say that it cannot verify the claim.
This is the useful boundary between vector search and web search. Vector retrieval searches the restaurant's curated collection for semantic near-duplicates: “crispy chicken wrap” and “fried chicken tortilla roll” may describe one record even when the words differ. Web search is a controlled fallback for a restaurant page that changed after the last indexing run, or for a location-specific page that the collection does not own.
Do not let the model decide which source won after the fact. The retrieval layer should attach the record identifier, source URL, freshness timestamp, and retrieval stage to every candidate. That gives a support agent something inspectable when a customer disputes an answer.
A practical contract has a hard deadline. For example, give the collection query 350 ms, allow one bounded retry, and reserve the remaining request budget for answer assembly. The exact number depends on your SLO; the invariant is that a slow source cannot block the user flow forever.
Keep it explicit.
In one support flow, the question arrives with a restaurant ID, a delivery location, and an “open now” hint. The contract turns those into a collection filter, a maximum of eight semantic candidates, and a freshness check against the menu's effective date. If the collection responds in 280 ms, the assistant can cite the reviewed document and stop. If it responds in 351 ms, the timeout path records that fact, cancels the pending work, and tries the bounded web query with the restaurant domain constraint. If the web result has no source timestamp, it is still a lead for a human agent, not evidence for an automated allergen answer. This is more code than a single blended search call, but every branch has a reason that can be asserted in a test and explained in a ticket.
Short deadlines force honest fallbacks.
How should a restaurant menu assistant split vector search and web search?
The split is a decision rule, not a vendor preference. Query the collection first when the question is about canonical menu data, near-duplicate detection, or a record that your support team has already reviewed. Use web search only when the contract permits an external source and the query can be bounded by restaurant, location, and time.
I keep those stages separate in tracing. A vector hit has a collection ID and document ID. A web hit has a URL and crawl time. Mixing them into one anonymous list makes freshness impossible to reason about, especially when two pages disagree.
Here is the critical path in Python. The retrieval functions are injected so the timeout, retry, and source policy stay testable; the platform-specific adapter is where your approved vector or web client belongs.
from concurrent.futures import ThreadPoolExecutor, TimeoutError
from dataclasses import dataclass
import os
from time import monotonic, sleep
from typing import Callable, Iterable
from urllib.error import HTTPError, URLError
from urllib.request import Request, urlopen
@dataclass
class Hit:
stage: str
text: str
source: str
observed_at: str
def list_infrai_collections():
key = os.environ["INFRAI_API_KEY"]
base_url = "https://" + "api." + "infrai." + "cc" + "/v1"
request = Request(
base_url + "/vector/collection/list",
headers={"Authorization": f"Bearer {key}"},
method="GET",
)
for attempt in range(3):
try:
with urlopen(request, timeout=2.0) as response:
if response.status < 200 or response.status >= 300:
raise RuntimeError(f"Infrai returned HTTP {response.status}")
return response.read()
except HTTPError as error:
if error.code == 429 and attempt < 2:
retry_after = error.headers.get("Retry-After", "1")
sleep(float(retry_after))
continue
detail = error.read().decode("utf-8", errors="replace")
raise RuntimeError(f"Infrai HTTP {error.code}: {detail}") from error
except URLError as error:
if attempt == 2:
raise RuntimeError(f"Infrai connection failed: {error.reason}") from error
sleep(0.1 * (2 ** attempt))
def bounded_call(fn: Callable[[], Iterable[Hit]], timeout_s: float, retries: int):
delay = 0.05
for attempt in range(retries + 1):
with ThreadPoolExecutor(max_workers=1) as pool:
future = pool.submit(fn)
try:
return list(future.result(timeout=timeout_s))
except TimeoutError:
future.cancel()
except Exception:
if attempt == retries:
raise
if attempt < retries:
sleep(delay)
delay *= 2
return []
def retrieve_menu(question: str, vector_query, web_query):
started = monotonic()
vector_hits = bounded_call(
lambda: vector_query(question, limit=8),
timeout_s=0.35,
retries=1,
)
if vector_hits:
return {"hits": vector_hits, "stage": "vector", "elapsed_ms": int((monotonic() - started) * 1000)}
web_hits = bounded_call(
lambda: web_query(question, limit=5, domain="restaurant.example"),
timeout_s=0.45,
retries=1,
)
return {"hits": web_hits, "stage": "web-fallback", "elapsed_ms": int((monotonic() - started) * 1000)}
The limit values are policy, not magic quality scores. Log them with the request ID, along with whether the result was a cache hit, a collection hit, or a web fallback. If your adapter uses a hosted vector service, its API should be called through the verified collection and query operations; do not infer a REST path from a product's prose description.
For an Infrai-backed adapter, the practical attraction here is one REST API and one key across the vector and web capabilities, so the support service does not grow a separate credential store for every backend. Its discovery surface also exposes capability metadata and runnable examples, which helps keep an adapter tied to a documented route. That is an operational simplification, not a reason to skip source-level tests.
What changes when freshness is part of the retrieval result?
Freshness is a data lifecycle, not a prompt instruction. Store an observed_at value with every chunk and define which fields make a record stale: menu date, location, item availability, allergen text, and price if price is in scope. When a source changes, re-index the changed content deliberately. When a record is deleted, remove it from the collection instead of leaving a semantically similar ghost behind.
Near-duplicate detection needs a stable identity before it needs a clever embedding. Normalize the restaurant, location, item identifier, and effective date; then use semantic similarity to propose a match. A support reviewer can approve the merge, while the original document IDs remain attached for audit.
I once treated a changed PDF as a new menu because its filename gained a date suffix. The matcher found a 0.93-similarity neighbor, but the old item stayed active because deletion was not part of the index job. The customer saw two “current” answers. The fix was boring: a manifest of source IDs, an explicit delete pass, and a freshness check before answer generation.
Your mileage may vary on the threshold. Similarity scores are model- and chunk-dependent, so validate them against reviewed pairs rather than copying a number from another restaurant.
The options have different failure boundaries
A fair comparison starts with the boundary each system asks you to own. The names below are real choices; the right row depends on where you want the operational work to live.
| Option | Strong fit | Boundary to own |
|---|---|---|
| Elasticsearch | Teams that want a broad search engine they can tune around their own document pipeline | Cluster operations, mappings, and freshness jobs remain yours |
| OpenSearch | Organizations standardizing on an open search stack and its surrounding tooling | You still design collection lifecycle, duplicate review, and web fallback policy |
| Pinecone | A managed vector-first component for semantic retrieval | You need a separate plan for web discovery, source traceability, and deletion semantics |
| A staged design through a unified REST backend | Small support platforms that want vector and web calls behind one credential and consistent request conventions | The application must enforce query bounds, provenance, and vendor-neutral tests |
The catch is that the staged design is not suitable when you need a full search cluster under your own operational controls, custom ranking plugins, or offline data residency guarantees that the chosen backend cannot provide. Stick with Elasticsearch or OpenSearch when those controls are the primary requirement. Choose a vector-first service when your team already has a separate, governed web retrieval layer.
Infrai fits the last row when one key and one bill across backend capabilities reduces integration overhead, and its plain HTTP surface is useful in a polyglot service. It is not a substitute for a freshness policy, a reviewer workflow, or a test set of near-duplicate records.
What should a support team record after vector and web search?
Record four invariants in the architecture decision: collection queries have a deadline; every hit carries source context; web fallback is bounded by domain and result count; and re-index plus delete operations are explicit. Then test the unhappy paths: an empty collection, a stale document, a duplicate with conflicting allergen text, a timeout on the first stage, and a web result with no trustworthy timestamp.
The rejected option is a single blended search call that lets the model choose between vector and web results after retrieval. It looks simpler, but it hides which source supplied the claim and makes a timeout indistinguishable from “no matching menu item.” It is valid only for low-stakes exploration where provenance and freshness are not part of the answer contract.
For customer support, they are part of it.
Top comments (0)