Unifying WhatsApp, Voice, and Email into One Agent Conversation State

The demo version of a multi-channel AI agent is easy: guest messages on WhatsApp, agent replies on WhatsApp, done. Production is harder, because guests don't stay on one channel. Someone messages WhatsApp about a booking, calls to change the date an hour later, then emails a follow-up question the next morning — and expects the agent to remember all of it as one continuous interaction, not three unrelated ones.

We've spent a good chunk of the last few months building exactly this for hospitality AI agents — voice, WhatsApp, SMS, and email all needing to resolve to the same underlying guest conversation. Here's what the architecture actually looks like, and where it broke before it worked.

The naive version, and why it falls apart

The obvious first design is: one conversation thread per channel, keyed by channel-specific identifier (phone number for voice/SMS, WhatsApp ID, email address). Each channel's handler reads and writes to its own thread, and an LLM call at the top handles whatever context it's given.

This works exactly until a guest switches channels mid-interaction, which in our data happens constantly — people call when typing is inconvenient, switch to WhatsApp when they want a written confirmation, and fall back to email for anything that needs an attachment. With per-channel threads, channel-switching guests get an agent with amnesia: it has no idea the WhatsApp conversation and the phone call an hour later are the same person mid-booking.

Conversation state as an identity-first model, not a channel-first one

The fix is inverting the primary key. Instead of conversation:channel:identifier, the source of truth becomes guest_profile_id, with channel-specific identifiers as resolved pointers into that profile rather than the root of the data model.

guest_profile
├── verified_identifiers: [phone, email, whatsapp_id]
├── active_intent: { type: "booking_modification", state: {...} }
├── Unifying WhatsApp, Voice, and Email into One Agent Conversation State: [ {channel, timestamp, content, direction}, ... ]
└── channel_threads: { voice: thread_id, whatsapp: thread_id, email: thread_id }

Every inbound message, regardless of channel, resolves to a guest_profile_id first, then appends to a single unified conversation_log that every channel handler reads from when constructing context for the LLM. The channel is metadata on each turn, not the organizing structure of the data itself.

The hard part isn't the schema, it's identity resolution

Once you accept identity as the root of the model, the actual engineering problem becomes: how do you reliably decide that this phone call and that WhatsApp message are the same person?

Hard identifiers are the easy case — a guest who's given a confirmed email or a phone number verified via OTP can be matched with high confidence. The genuinely hard case is the first contact on a new channel: a guest who's messaged WhatsApp before now calls from a different number (already common with dual-SIM phones and business trips), or emails from an address that doesn't match anything on file.

We settled on a tiered matching strategy rather than trying to get this right with a single heuristic:

Verified hard match — confirmed identifier already linked to a profile. Auto-merge, no confirmation needed.

Strong soft match — name plus a recent, specific detail only that guest would know (an active booking reference, a very recent interaction) mentioned unprompted. Merge with a light confirmation: "I see you messaged us about a booking for Saturday — is that the same one?"

Weak match — name similarity alone, or a guess based on timing. Never auto-merge. Treat as a new profile and let a human or a later interaction resolve it.

Getting tier 3 wrong is worse than getting tier 2 wrong. A false-positive merge that mixes two different guests' booking history is a trust-breaking failure in a way that a missed merge (treating the same guest as new twice) just isn't. We tuned hard toward precision over recall here, deliberately.

Context windowing across channels needs its own logic

Even with unified identity, naively dumping the entire cross-channel conversation_log into the LLM's context on every turn gets expensive and noisy fast — a guest with a six-month history and forty prior interactions doesn't need all forty replayed to answer "what time is checkout."

What worked:

Recency-weighted inclusion: full detail for the current session, summarized detail for anything older than a configurable window (we use 48 hours for active bookings, longer for loyalty/repeat-guest context).

Intent-scoped context: if the guest has an active_intent (mid-booking, mid-complaint), that intent's full state gets priority inclusion regardless of channel or recency, since it's what the current turn is actually about.

Channel-aware summarization: a voice call transcript and a WhatsApp thread compress differently — voice tends to be more redundant/conversational and compresses well; WhatsApp messages are already terse and lose more meaning per token cut.
Handoff between channels needs to be explicit, not inferred

One failure mode we didn't anticipate: an agent that's technically aware of cross-channel history can still produce a confusing experience if it doesn't say it's aware. A guest who called about a booking and then emails a follow-up, only to get a reply that doesn't reference the call at all, reasonably assumes the system doesn't actually know what's going on — even if it technically does.

The fix was making cross-channel continuity visible in the response itself: "Following up on your call earlier about Saturday's booking — yes, that's confirmed for 4 guests." That single line does more for guest trust than any amount of backend correctness that stays invisible.

What this actually buys you

None of this is about making the LLM smarter. It's a data modeling and identity resolution problem wearing an AI costume — the same class of problem as customer 360 systems that predate LLMs by a decade, just with a conversational interface on top and tighter latency requirements on voice.

The payoff is real, though: guests increasingly expect an agent to "just know" what happened on a different channel, the same way they'd expect a good human receptionist to remember a conversation from an hour ago. Getting the identity model right is what makes that expectation possible to meet, and getting it wrong is the single fastest way to make a multi-channel AI agent feel broken.

We build this multi-channel identity and conversation layer for hospitality AI agents at HuemanAI — unifying WhatsApp, voice, SMS, and email into one guest profile. If you're solving a similar cross-channel identity problem, I'd enjoy comparing notes in the comments.