Short answer: give the itinerary planner a deadline, then divide that deadline between query parsing, candidate retrieval, reranking, and citation assembly. A grounded answer that arrives after the user has changed the dates is a failed answer, even if its citations are correct.
The Decision Record: A Deadline, Not a Throughput Contest
For a travel itinerary planner, the useful unit is the end-to-end interaction: the user submits a city, dates, constraints, and a preference, and receives a plan with evidence. Set a product deadline first. A practical starting point is 900 ms at the service boundary for a warm request; treat that as a design target, not a universal benchmark. Your own traces should replace it as traffic and corpus size change.
The invariants are stricter than the stopwatch. Every recommendation must retain the source passage that supports it, filters such as date and opening hours must be applied before generation, and a timeout must produce a visibly partial result rather than an uncited guess. I have seen teams tune the vector index while their citation join quietly consumed the remaining budget. The slow part was not retrieval.
Here is an architecture record I use for the first implementation:
| Stage | Initial budget | Failure boundary | Output |
|---|---|---|---|
| Parse and normalize | 60 ms | Reject malformed dates; keep the prior query | typed query |
| Filtered candidate search | 280 ms | Return fewer candidates with provenance | document IDs + spans |
| Rerank and deduplicate | 220 ms | Skip rerank, preserve first-pass order | evidence set |
| Citation and answer assembly | 260 ms | Return evidence-only response | answer + citations |
| Reserve and network jitter | 80 ms | Stop fan-out and close the request | timing metadata |
These numbers add to 900 ms, but they are not promises to the browser. Measure p95 and p99 separately; averages hide the airport Wi-Fi case. A queue with a bounded deadline is easier to reason about than four independent calls that each believe they have a full second.
Deadlines move.
When the product adds a map preview or a second destination, the old split becomes fiction. Keep the budget in configuration, emit a timer for each stage, and review the largest tail after every corpus or model change. I don't treat a faster median as a win if p99 now crosses the interaction deadline. The planner's contract is about the slowest acceptable request, not the demo laptop.
How Should a Retrieval Architecture Spend Latency Budgets for a Travel Itinerary Planner?
Start by making the query structured. “Three quiet days in Kyoto near rail, no museums on Monday” should become locations, dates, hard constraints, and soft preferences. The parser can run locally and cheaply, while the search request carries only fields that affect filtering. This prevents a reranker from spending time rescoring documents that could have been excluded with a metadata predicate.
Use two retrieval lanes when the corpus warrants it: lexical search for exact place names, dates, and transit lines; semantic search for descriptions such as “quiet neighborhood.” Merge by document identity, then cap the candidate set before reranking. The cap is a control surface. Raising it may improve recall, but it also increases serialization, model, and citation work.
The critical path should carry one deadline through every function. Python makes the contract explicit:
from time import monotonic
def retrieve_plan(query, deadline_ms=900):
deadline = monotonic() + deadline_ms / 1000
parsed = parse_query(query)
candidates = search_candidates(parsed, until=deadline)
if monotonic() >= deadline:
return evidence_only(candidates, reason="search deadline")
ranked = rerank(candidates, until=deadline)
if monotonic() >= deadline:
return evidence_only(ranked, reason="rerank deadline")
return assemble_with_citations(parsed, ranked, until=deadline)
Each returned passage should include a stable source ID, a span or chunk ID, and the metadata used for filtering. Do not reconstruct citations from the generated prose. Store the evidence set beside the answer so an audit can replay the decision.
Failure Boundaries That Matter More Than Top-k
Timeouts are normal control flow. A partial itinerary can list verified stops and mark unscheduled items; it must not silently invent a connection. Likewise, an empty result should explain which constraint removed the candidates, because “no places found” is not actionable for a traveler. Make that state machine explicit: complete, evidence_only, and no_match are different responses, and each deserves a distinct metric. A retry may be useful for a transient dependency, but it must consume the same request deadline; resetting the clock on every hop turns a graceful degradation path into a slow-motion outage from the user's point of view. Log the selected branch, candidate count, and evidence freshness so an on-call engineer can tell a sparse destination corpus from a saturated service without replaying private traveler data.
Watch for fan-out. Weather, attraction hours, and transit feeds often arrive from separate systems, so run them concurrently behind one deadline and accept the subset whose timestamps are still valid. Cache immutable facts such as venue coordinates, but attach freshness to mutable facts such as opening hours. Your mileage may vary: a regional transit feed may have a different update contract than a booking feed, and the cache policy should follow that contract.
Evaluation needs two scoreboards. Retrieval tests measure recall of the right attraction or route under hard filters. Grounding tests verify that each sentence is supported by a retained span and that the citation points to the same entity and date. A fast answer with stale hours should fail the second scoreboard.
The Rejected Option and Its Valid Use Case
I would reject a single, large semantic search followed by an unconstrained generation call for the interactive planner. It makes the latency tail opaque, allows date constraints to arrive too late, and leaves no clean place to return evidence when generation times out.
That design is still valid for an overnight research job: compile a broad destination brief, tolerate minutes rather than milliseconds, and run a human review step before publication. The catch is that the batch workflow has a different deadline and audit path; moving it into the chat request changes both.
Top comments (0)