DEV Community

Becky_dev
Becky_dev

Posted on

Contract Testing for AI Hotel Search: How to Keep Travel Agents Grounded in Reality

AI travel agents are getting much better at searching and recommending hotels.

But there is a problem that is easy to miss:

Sometimes the AI is not wrong. The data it received was already wrong.

A supplier changes a field type.

A cancellation policy changes its format.

A price that used to mean “per night” suddenly means “for the entire stay.”

A hotel-level attribute gets attached to a specific room.

A cached rate is already outdated.

The AI sees valid-looking data and produces a confident answer.

That is why hotel search systems need more than schema validation.

They need Contract Testing.

1. The Real Problem Is Not Always the AI

Imagine a hotel API returns:

{
  "price": 250,
  "currency": "USD",
  "cancellation": "Free cancellation"
}
Enter fullscreen mode Exit fullscreen mode

Everything looks fine.

The JSON is valid.

The types are correct.

The request succeeded.

But what does $250 actually mean?

  • $250 per night?
  • $250 for the entire stay?
  • $250 before tax?
  • $250 including tax?
  • $250 for two adults?
  • $250 for one room?

And what does “Free cancellation” mean?

  • Free cancellation until 24 hours before check-in?
  • Until 6 PM the day before?
  • Until midnight?
  • Or does it apply only to one specific rate?

The schema cannot tell us.

This is the fundamental problem with AI travel infrastructure:

Valid data is not necessarily meaningful data.

Contract testing should therefore verify not only whether a response is structurally correct, but whether its meaning is still correct.

2. Four Layers of Contract Testing

For AI hotel search, I think about contract testing in four layers:

1. Structural

Does the response follow the expected schema?

For example:

class Price(BaseModel):
    amount: float
    currency: str
    basis: Literal["per_night", "per_stay"]
Enter fullscreen mode Exit fullscreen mode

This catches obvious problems such as missing fields, invalid types, or unexpected values.

But structural validation is only the first layer.

2. Semantic

Does the data actually mean what the contract says it means?

For example:

{
  "amount": 250,
  "currency": "USD",
  "basis": "per_night"
}
Enter fullscreen mode Exit fullscreen mode

If the supplier suddenly starts returning the total stay price while keeping "basis": "per_night", the schema still passes.

The contract test should fail.

3. Behavioral

Can the AI safely use the result for the intended action?

A hotel result may be good enough for:

Discovery

but not good enough for:

Recommendation

and a recommendation may still not be ready for:

Booking

This distinction matters because AI agents increasingly move from answering questions to taking actions.

4. Operational

What happens when the real world gets messy?

Hotels APIs have:

  • slow suppliers
  • partial responses
  • duplicate requests
  • changing prices
  • expired rates
  • uncertain booking results

These are not theoretical edge cases.

They are normal conditions in travel systems.

3. Scope Is Just as Important as Schema

One of the most common sources of AI mistakes is scope confusion.

Consider these attributes:

Hotel
 └── Room
      └── Rate
Enter fullscreen mode Exit fullscreen mode

“Hotel has a swimming pool” is a hotel-level attribute.

“Room has a balcony” may be room-level.

“Free cancellation until September 20” is usually rate-level.

If the data pipeline accidentally moves a rate-level cancellation policy to the hotel level, the AI may interpret it as applying to every room.

The JSON can still be perfectly valid.

That is why contracts should define not only what a field is, but also where it applies.

4. Multi-Supplier Aggregation Makes This Harder

AI hotel search often combines multiple suppliers.

This creates another problem:

Different suppliers describe the same concept differently.

One supplier might return:

free_cancel_before = 2026-09-20T18:00
Enter fullscreen mode Exit fullscreen mode

Another might return:

cancellation = "Free cancellation"
Enter fullscreen mode Exit fullscreen mode

Another might return a nested policy with multiple deadlines.

If everything is normalized too aggressively, important information can disappear.

A safer approach is to keep both:

Canonical value

and

Original source information

For example:

{
  "cancellation_deadline": "2026-09-20T18:00:00Z",
  "source": {
    "supplier": "supplier_a",
    "original_value": "Free cancellation until 18:00"
  }
}
Enter fullscreen mode Exit fullscreen mode

This gives the AI a normalized value while keeping enough information for debugging and verification.

For travel infrastructure, traceability is part of correctness.

5. Freshness Determines Whether an Answer Is Actionable

Hotel data changes constantly.

A price returned five minutes ago may already be different.

So instead of treating every successful API response as equally trustworthy, systems should distinguish between different readiness levels:

Discovery
    ↓
Recommendation
    ↓
Verification
    ↓
Booking
Enter fullscreen mode Exit fullscreen mode

A cached result might be perfectly acceptable for discovery.

But before booking, the system may need to verify:

  • current price
  • room availability
  • cancellation policy
  • occupancy
  • taxes and fees
  • booking conditions

This is especially important for AI agents.

An agent should know the difference between:

“I found a hotel that matches your request.”

and:

“This room is currently available at this price and can be booked.”

Those are two different levels of certainty.

6. MCP Tools Are Contracts Too

With MCP, the contract is not only the API response.

The tool itself becomes part of the agent's runtime environment.

For example, a hotel MCP may expose tools such as:

searchHotels
getHotelDetail
getHotelSearchTags
Enter fullscreen mode Exit fullscreen mode

The agent needs to understand:

  • what each tool does
  • what parameters it accepts
  • what the returned fields mean
  • whether the tool is read-only
  • whether it can actually perform a booking
  • whether the result is current enough for the requested action

This is why MCP testing should include tools/list, tool descriptions, schemas, and side-effect expectations.

A tool that says “search hotels” should not accidentally be interpreted as “book hotels.”

The difference may look small to a human.

For an autonomous agent, it is critical.

7. Test the Agent's Behavior, Not Its Exact Words

Traditional testing often asks:

Did the model generate the expected answer?

That is difficult to maintain.

Models change.

Prompt wording changes.

Different models may express the same conclusion differently.

For AI travel systems, it is more useful to test structured behavior.

For example:

User asks for:
Tokyo hotel under $200 with free cancellation

Expected behavior:
1. Search hotels
2. Filter by price
3. Verify cancellation conditions
4. Return hotels that satisfy all constraints
Enter fullscreen mode Exit fullscreen mode

The exact wording does not matter.

What matters is whether the agent used the right data and took the right actions.

This makes tests more robust across model upgrades.

8. The Most Dangerous State: Unknown

Booking systems have a particularly important edge case:

unknown transaction state.

Imagine the agent sends a booking request.

The supplier times out.

Did the booking fail?

Maybe.

Did it succeed?

Maybe.

The worst thing the agent can do is automatically retry without checking.

Because the first request might actually have succeeded.

The correct state is:

UNKNOWN
Enter fullscreen mode Exit fullscreen mode

The system should then reconcile the booking status before attempting another action.

This principle is bigger than hotel booking.

Whenever an AI agent performs an external side effect, it needs a safe strategy for uncertain outcomes.

9. Golden Fixtures and Production Drift

A good contract-testing system should also maintain realistic test data.

Golden fixtures can represent situations such as:

  • normal hotel results
  • sold-out rooms
  • multiple cancellation deadlines
  • price changes
  • partial supplier responses
  • stale cached data
  • uncertain booking outcomes

Then every major supplier or schema change can be tested against those fixtures.

But tests alone are not enough.

Production systems should also monitor for contract drift.

For example:

Supplier A:
Cancellation field changed
↓
Semantic test failure
↓
Deployment blocked
Enter fullscreen mode Exit fullscreen mode

Or:

Production:
Unexpected price basis detected
↓
Alert
↓
Supplier investigation
Enter fullscreen mode Exit fullscreen mode

This turns contract testing from a CI/CD task into an ongoing reliability system.

10. Where RollingGo Fits

For developers building AI travel applications, the same principles apply to hotel MCP infrastructure.

A hotel MCP should make the boundary between search, hotel information, and actual transaction capability clear.

For example, a developer may connect a hotel MCP and expose tools such as:

{
  "server": "hotel-search",
  "tools": [
    "searchHotels",
    "getHotelDetail",
    "getHotelSearchTags"
  ]
}
Enter fullscreen mode Exit fullscreen mode

The important question is not simply:

“Does the MCP return hotel data?”

It is:

“Can an AI agent understand exactly what this data means and what it is allowed to do with it?”

That distinction becomes increasingly important as travel agents move from search to recommendation, verification, and eventually booking.

Conclusion

AI agents are becoming better at travel.

But better reasoning cannot compensate for ambiguous or drifting data.

A hotel API can return valid JSON and still cause an AI agent to make a completely wrong recommendation.

That is why contract testing for AI travel systems needs to go beyond:

“Does the schema pass?”

It should also ask:

  • Does the data still mean what we expect?
  • Is the value attached to the correct scope?
  • Is the information fresh enough for the action?
  • Can the agent safely use it?
  • What happens when the supplier behaves unexpectedly?
  • What happens when a transaction ends in an unknown state?

The goal is not to make AI sound more confident.

It is to make sure the AI has something reliable to be confident about.

Because in travel infrastructure, the most dangerous bug is often not an obvious error.

It is correct-looking data with the wrong meaning.

Top comments (0)