A travel itinerary planner that aggregates carrier, lodging, and transfer listings should start with a 200 ms retrieval budget, then spend most of it on the one operation that must inspect the live candidate set. Keep the prompt assembly path bounded, fetch static descriptive chunks separately from volatile availability, and reject stale offers before generation. This is a design target, not a benchmark: the right number is the smallest budget that survives a replay of the planner's real traffic while leaving time for ranking and generation.
Short answer: split descriptive retrieval from live listing validation, measure each stage at the tail, and treat freshness as a hard eligibility rule rather than a ranking hint.
The bill has to be understood before the architecture. For an aggregator, it is made of source fetches, normalization, chunk creation, embedding work, retained vector and document bytes, retrieval requests, and live validation calls. A useful first estimate is:
def monthly_retrieval_cost(
source_refreshes,
refresh_cost,
embedded_chunks,
embedding_cost,
retained_gb,
storage_cost,
queries,
query_cost,
):
return {
"refresh": source_refreshes * refresh_cost,
"embedding": embedded_chunks * embedding_cost,
"retention": retained_gb * storage_cost,
"query": queries * query_cost,
}
Use actual invoice units in that function. Don't insert public list prices and assume they represent a production bill; retries, replicas, refresh frequency, and retained generations can move the dominant term. If the model says embedding is largest, the meaningful change is not shaving milliseconds from a cache lookup. It is stopping unchanged descriptions from being embedded again.
That decision has a cost during recovery. Retaining only the active normalized document, its source fingerprint, and one prior index generation reduces the material kept online, but it also limits how far an operator can roll back without refetching and rebuilding. For feeds with weak historical access, longer raw-source retention may be the safer choice even when storage becomes the dominant line item. Durability beats a tidy bill.
How should a travel itinerary planner set retrieval architecture latency budgets?
Start at the user-facing deadline and work backward. For a 200 ms retrieval target, the following allocation is a planning hypothesis that the team must replace with replay results; it is deliberately explicit so that an unowned queue cannot consume the margin.
| Stage | Budget | Exit condition |
|---|---|---|
| Request parsing and policy filters | 10 ms | Dates, geography, party size, and source permissions are valid |
| Descriptive candidate retrieval | 55 ms | A bounded candidate set is returned |
| Live listing lookup | 75 ms | Price, availability, and effective timestamp are attached |
| Ranking and deduplication | 35 ms | Duplicate offers collapse under a stable listing identity |
| Prompt assembly | 10 ms | Only eligible evidence is serialized |
| Contingency | 15 ms | Reserved for scheduler and network variance |
The numbers sum to 200 ms. They do not prove that 200 ms is attainable. Instrument queue time and service time independently, record percentiles by source and itinerary shape, and run the budget against traces that include popular city pairs, sparse rural routes, long date ranges, and multi-leg requests. An average hides the exact cases that make a planner feel unreliable.
Parallelism helps only across independent work. Descriptive retrieval can run beside source-policy resolution, but live validation depends on candidate identity and cannot honestly begin before candidates exist. Launching everything at once may make a trace look busy while increasing discarded work. Set one deadline on the whole retrieval operation, pass the remaining time to each child call, and cancel work that can no longer contribute.
Keep it bounded.
When the deadline expires, return fewer validated candidates rather than mixing validated and unknown offers. A generated itinerary can omit a hotel; it should not present an expired room as bookable. This distinction also gives the product layer a clean degraded state: “fewer current options” is operationally different from “unverified options,” even if both produce a shorter result.
Chunking and freshness belong to different clocks
Carrier descriptions, station accessibility notes, baggage rules, and neighborhood summaries change on a different clock from seats, rooms, prices, and departure disruptions. Putting both classes into one chunk creates a nasty coupling: every volatile update forces the stable prose through normalization and embedding again, while a slow refresh can leave an otherwise relevant chunk carrying an obsolete offer.
Use two records joined by a stable source identity. The descriptive record is chunked for semantic retrieval and includes provenance. The volatile record is keyed for direct lookup and carries the source timestamp, ingestion timestamp, and the planner's eligibility deadline. Retrieval finds descriptive candidates; validation then joins current volatile state before ranking. The generator receives neither an expired record nor a record whose freshness cannot be established.
from dataclasses import dataclass
from datetime import datetime, timezone
@dataclass(frozen=True)
class LiveListing:
listing_id: str
observed_at: datetime
valid_until: datetime
available: bool
def eligible(listing: LiveListing, now: datetime) -> bool:
if now.tzinfo is None or now.utcoffset() is None:
raise ValueError("now must be timezone-aware")
return listing.available and listing.observed_at <= now < listing.valid_until
request_time = datetime.now(timezone.utc)
Chunk boundaries should follow claims that can be refreshed and cited together, not an arbitrary character count. A baggage rule and a terminal description may share a page yet need separate chunks because their ownership and revision cadence differ. On the other hand, splitting every sentence creates more embedding work, more near-duplicates, and a larger ranking problem. There isn't a universal chunk size; settle it with a labeled query set that includes exact identifiers, broad destination intent, constraint-heavy itineraries, and ambiguous place names.
The retrieval-augmented generation paper describes generation conditioned on retrieved non-parametric memory alongside parametric memory. For this planner, that retrieved evidence is still not a transaction guarantee. Search relevance answers “does this listing fit the request?” Live validation answers “may the planner present it now?” Combining those questions into one similarity score erases a boundary the system needs.
Failure modes need explicit ownership
Freshness failures rarely arrive with a convenient label. A source can publish an older snapshot after a newer one, two feeds can use different identifiers for the same journey, a refresh can finish after its successor, or a timezone conversion can move a departure across the requested date. Name these cases in tests and metrics. A generic “retrieval failed” counter won't tell an operator whether to tune chunking, repair identity mapping, or isolate a lagging source.
Consider a deterministic replay with three updates for the same overnight rail listing. Update A is observed at 09:00 and says the listing is available; update B is observed at 09:02 and closes it; update C contains the older 09:01 state but arrives last because its source partition was delayed. The ingestion timestamps therefore run A, B, C, while the source observations run A, C, B. If the active record follows arrival order, C reopens a listing that B already closed. The test should submit that exact sequence, retain B as active, increment an out-of-order rejection counter for C, and show that prompt assembly never sees C. Then run a second request whose deadline expires after descriptive retrieval but before live validation: the result must omit the listing, not reuse A, C, or a cached availability flag. This one compact exercise crosses ordering, freshness, deadline propagation, metrics, and prompt eligibility; a test that mocks each function separately can pass while the assembled planner still gets the decision wrong.
Arrival order lies.
Use monotonic acceptance per source partition: an update may replace active state only when its source version or observed timestamp is newer under that source's documented ordering rule. Keep ingestion time separate, because arrival order is not event order. If a source offers no trustworthy ordering field, I'm not sure a strict freshness claim is defensible; resolving that requires a source contract or periodic authoritative snapshots, not a clever ranking feature.
Test the architecture with deterministic replays. Freeze request time, replay out-of-order updates, inject duplicate identities, and vary stage delays until the total deadline is exhausted. Assert that expired and unvalidated listings never enter prompt assembly. Also assert the less dramatic behavior: unchanged descriptive content should retain its fingerprint and skip re-embedding, while a live-state change should not rewrite the descriptive chunk.
Observability should follow the same boundaries. Record candidate count before and after validation, age of the newest accepted source observation, stale-rejection count, deduplication ratio, timeout stage, and unused budget. Avoid logging complete traveler prompts or raw supplier payloads merely because they make debugging easier; structured identifiers and reason codes usually answer the operational question with less retained sensitive data.
The deployment rule is evidence before expansion
Deploy one source partition and one itinerary class behind a shadow comparison, then evaluate relevance labels, freshness rejection, deadline completion, and rebuild behavior before widening coverage. The catch is that this split architecture is not suitable when the upstream system already provides a transactional, queryable inventory view with latency inside the product deadline; in that case, stick with direct constrained lookup and use retrieval only for descriptive material.
It is also a poor fit when the team cannot maintain stable cross-source identities. Semantic similarity can propose duplicates, but it should not silently decide that two bookable offers are the same. Until identity rules exist, preserve source-scoped results and accept a less polished itinerary rather than merge inventory that may have different conditions.
The final acceptance gate is plain: every presented offer was current at request time under a documented rule, every retrieval stage stayed within its assigned deadline during representative replay, and the index can be rebuilt from retained inputs. If one of those claims lacks evidence, expand neither traffic nor retention shortcuts.
Top comments (0)