DEV Community: Arritola92

The Fare Observation Schema I'd Build Before a Flight Scraper

Arritola92 — Mon, 12 Oct 2026 08:45:03 +0000

I would design the record before choosing a travel API or public-page collector. A flight price becomes comparable only when its itinerary and fare assumptions are preserved.

My default output would be a time-stamped observation. A transaction-ready offer would have a separate state and confirmation path.

The record carries the query

This illustrative example uses fictional itinerary details:

{
  "observation_id": "example-observation",
  "source": "example-public-search",
  "observed_at": "2026-10-01T12:00:00Z",
  "query": {
    "origin": "AAA",
    "destination": "BBB",
    "departure_date": "2026-11-01",
    "return_date": "2026-11-08",
    "adults": 1,
    "market": "example-market",
    "currency": "USD"
  },
  "itinerary_key": "example-reviewed-itinerary",
  "cabin": "economy",
  "baggage_description": "Not specified in this sample",
  "price": {
    "amount": "500.00",
    "currency": "USD",
    "basis": "displayed_trip_total",
    "taxes_included": null
  },
  "status": "observed",
  "revalidated_at": null
}

I represent monetary values using a decimal-safe storage type, not a binary floating-point value. An unknown tax or baggage condition should remain unknown.

The example airports and amount are placeholders. They are not a collected fare or a booking offer.

Identity requires more than flight number

My itinerary key would reflect legs, dates and times with timezone context, airports, and relevant carrier information. The comparison key would additionally account for passenger assumptions, cabin, and the fare bundle required by the use case.

I preserve operating and marketing carriers separately when the source exposes them. Similarly, I retain the provider's native offer ID without assuming it is stable across repeated searches.

Different offers for the same flight can have different conditions. I don't merge them merely because the schedule matches.

I use an explicit lifecycle

For page observations, observed means the collector captured a displayed result. It does not mean the amount is available for purchase.

For an authorized booking integration, I add states such as revalidated, expired, and unavailable, based on the provider's actual responses. The Duffel offer model is one reference for understanding that different contract.

I would never set revalidated simply because I fetched the same search page again. Confirmation needs the appropriate authoritative flow.

My fixture matrix exercises price mapping

I would include a direct trip, a connection, a return journey, different passenger counts, baggage variations, multiple currencies where relevant, and no results.

For rendered pages, I also test late-loaded cards, market redirects, a consent screen, and a layout containing fares beside ancillary fees. The parser must attach the correct amount to the correct itinerary.

A no-results response and a failed extraction get different states. Otherwise a parser regression can look like disappearing inventory.

I compare providers under controlled conditions

I would run equivalent route queries close in time, preserve all relevant parameters, and compare only records meeting the same eligibility rules. My report would include missing comparable results, not merely the rows each source returned.

I would distinguish observed fare, revalidated offer, and final checkout amount in downstream interfaces. A monitoring dataset may only have the first, which is still useful when labeled accurately.

For research collection, I would exclude traveler names, account-only data, payment information, and booking records. Those belong to a separate authorized transaction system.

I accept the first collector when it can explain each amount and its missingness on this fixture matrix. Only after that would I increase routes, schedules, or source coverage.

My Data Contract for a Gemini Answer-Observation Pipeline

Arritola92 — Mon, 12 Oct 2026 08:44:16 +0000

I make the observed surface a required field. A generated Gemini API response and a captured consumer-product answer should never become indistinguishable after ingestion.

The pipeline would have two stages: answer observation and cited-source verification. Each produces evidence with its own timestamp.

The observation preserves the experiment

Here is an illustrative record, not output from a provider:

{
  "observation_id": "example-observation",
  "surface": "gemini_api",
  "provider": "example-provider",
  "prompt_id": "example-category-question",
  "prompt_version": "example-v1",
  "prompt_text": "Illustrative research prompt",
  "language": "en",
  "region": null,
  "account_context": "not_applicable_to_this_example",
  "model_id": "example-model",
  "observed_at": "2026-10-01T12:00:00Z",
  "answer_text": "Illustrative answer",
  "citations": [],
  "status": "complete_without_citations"
}

For a consumer-surface observer, I record the available context and explicitly mark unavailable conditions. I don't populate a model ID based on a guess.

The Gemini Search grounding documentation is my reference for an official grounded-generation path. It should not be used as evidence that API output reproduces the consumer interface.

Citation extraction needs its own acceptance test

I preserve citation URL, visible label, position or supported relationship, and any provider reference ID. If the provider does not expose attachment to a passage, I don't invent that mapping.

My fixtures would include an answer with no citations, one citation, several citations, repeated domains, redirecting URLs, and a brand mentioned only in a cited source.

The last case checks whether answer-text mentions and source mentions remain separate. A flattened text field can silently turn source material into an apparent generated recommendation.

I compare normalized URLs while retaining originals. Tracking parameters may be removable, but path or query differences can identify distinct documents or versions.

I repeat a versioned prompt panel

A single prompt result is one observation. I create a small panel of relevant factual, category, comparison, and navigational prompts, then repeat part of it across scheduled windows.

I hold controllable conditions constant and record what cannot be controlled. Prompt wording, provider configuration, parser version, and observation time belong in the experiment history.

If a provider changes extraction behavior, I would distinguish that event from a change in the answer source. Saving permitted raw outputs and a small reviewed panel helps diagnose the difference.

Source support is downstream work

After answer collection, I would fetch authorized public citation URLs and store final URL, fetch time, content hash, and relevant passage evidence. A separate result would say whether the passage supports the specific claim under review.

I use states such as supported, not_supported_by_reviewed_passage, ambiguous, and source_unavailable. An unavailable page is not proof of a false citation, and topical relevance is not enough to set supported.

The source fetch timestamp must remain distinct from the answer timestamp. A later edit can change what a reviewer sees.

I keep incomplete outcomes visible

Timeout, provider error, partial answer, no answer, and complete answer without citations should have separate states. An empty citations array by itself cannot tell me which condition occurred.

My metrics would include answer fidelity on the reviewed sample, citation precision, record completeness, failure distribution, and natural variation across repeated prompts. I avoid calling all of these one success rate.

I also avoid confidential prompts and another user's chats or uploads. The collection scope should match the research question.

My first milestone would be a small prompt panel whose observations and citations can be manually traced. Automation becomes useful when that trace survives every export and transformation, rather than disappearing behind a visibility score.