Making non-atomic actions atomic using intents

Context

At a previous job, we built a service we called our operations platform. Its aim was to unify the SaaS tooling used by our ops teams with domain data from our core business, so we could directly see how the ops efforts contributed to our growth. A key feature here was an appointment booking system, where users could book consultations with our sales team. For a start, we chose to store appointments in our database, but rely on Calendly for the actual scheduling. I'm not sure this was the right choice, but the key reason was that our sales team depended on several of Calendly's features, and we needed to move fast.

With this approach, the sales team continued to work in Calendly, and we embedded the Calendly UI in our app for users to schedule appointments. We would then receive webhooks from Calendly and create the appointment in our backend. Our backend served as the source of truth for our app.

As one might expect, we encountered several issues in this approach. This post is about one of those.

Constraints

One of the features our sales team relied on was Calendly's pooling feature (round robin). They could create pools consisting of several agents, and a customer who booked an appointment would be assigned to one of the agents in the pool. However, if the customer later rescheduled, the customer would keep the same agent they were assigned to. The business wanted to change this, so that reschedulings would go back into the pool.

Calendly didn't provide any option for this. The only solution was to do it in two steps: cancel the existing appointment on Calendly and book a new one, thereby picking a new agent from the pool. But this was untenable: we needed the process to be seamless for our users.

Solution

To solve this, I came up with a high-level API called PendingCancellations. A PendingCancellation represents an intent to cancel an appointment. This intent may be acted upon (thereby cancelling the appointment) or discarded. We would still do rebooking in multiple steps, but with a PendingCancellation, it would look like this (at a high level):

create PendingCancellation (intent to cancel old appointment)
 -> book new appointment
  -> execute PendingCancellation (actually cancel old appointment)

For the implementation, we wanted to present this as a single step to our customers. We also need to keep in mind this detail: appointments are actually created on Calendly (which calls them "events"), then synced to our database via webhooks.

So the full implementation looked like this:

• User clicks "Reschedule" in our app
• The app tells our backend to create a PendingCancellation, linked to the user and appointment in our database.
• Then the app redirects to Calendly for booking a new appointment (not rescheduling)
• User completes the booking
• Backend receives Calendly's event.created webhook.
• We process the webhook. If there is a PendingCancellation, we go ahead and cancel the old appointment and create the new one.

Failure modes

This was a fine API. But, of course, I also had to think about failure modes:

What happens if the user changes their mind and exits without completing the new booking?

Nothing. The old appointment remains active. All unprocessed PendingCancellations are ignored and cleaned up after a few hours. This is the beauty of the intent system: if it isn't explicitly acted upon, it has no effect.

What happens if the user enters this flow (clicks "Reschedule") multiple times?

We only allow for one PendingCancellation per appointment. If there is an existing PendingCancellation, we simply update its timestamp.

What about transactional guarantees (consistency)? Since this is not an atomic operation, what happens if one action fails (cancelling the old one or booking the new one)?

This was tricky. Normally, we could wrap the whole operation on our backend in a database transaction (cancel old + create new). This way, if one failed, there would be no side effects, and we could safely retry the webhook.

However, since the appointments are first created on Calendly before being synced to our database, we don't have control over the "create" part. Regardless of what happens in our database transaction, the new appointment already exists on Calendly, and it might cause confusion for a user if they saw no change in their app.

So I chose to do something different: when we receive the webhook, first create the new appointment before cancelling the old one. Also don't bother wrapping it in a transaction. That way, if the cancellation failed, we would end up showing the user both old and new appointments. This is undesirable, of course, but, to my mind, it was better to see 2 appointments than 0 (if cancel was executed first).

In hindsight, I'm not sure this was the better decision. Seeing 2 appointments after rebooking might be worse, since it might tempt the user to try cancelling the old one. It also meant we had to make the webhook processing idempotent so we could safely retry the webhook without creating a third appointment. 😄

Dealing with limitations

We still had a few more challenges to solve. A key one was linking the new appointment from Calendly to the old one in our database. Essentially, when we receive Calendly's event.created webhook, how do we know if this is a rebooking? Calendly's UI doesn't allow us to attach any metadata to the event (such as old_appointment_id=abc), so we have no way of differentiating between our rebooked appointments and truly new appointments.

The only thing we had was the user ID (sort of—but that's a story for another time). We also knew two things:

• a user can only have one appointment of a given type at any time
• the new and old appointments must have the same type

So I worked with that: when we receive a new event booking webhook, fetch all PendingCancellations for that user. From there, we use heuristics: find the most recent PendingCancellation matching the new appointment's type, and pick that. This worked reliably.

Conclusion

I'm really proud of the PendingCancellation API. It's a nice, lightweight solution that fits neatly into our constraints and is generally reliable. It of course comes with tradeoffs, but is generally pleasant to work with.

Two sources of inspiration I had were Stripe's PaymentIntents and Android Intents. They're not exactly similar, but the name "Intent" conveys a desire to perform an action, without any actual commitment. It is entirely possible for the desired action to expire or be invalidated before it can be completed, and the intent should handle this gracefully. I was also inspired by the concept of change requests, something we already used quite effectively in our codebase.