Asynchronous transaction in distributed system

Synchronous and Asynchronous Transactions

Synchronous transactions occur in real time, where the system waits for a response before proceeding. For example, making a payment and waiting for confirmation. Asynchronous transactions, on the other hand, happen without waiting for an immediate response. The system moves on, and the result is handled later. For example, sending an email and processing its delivery later.

Why Asynchronous Transactions?

2PC (Two-Phase Commit) and 2PL (Two-Phase Locking) ensure isolated and consistent transactions but can block progress if participants are slow or unavailable. Long-running transactions and cross-organizational participation make 2PC unsuitable due to its blocking nature. You can learn about these in my previous blogs.

Imagine transferring money between two banks using a cashier’s check. The sender’s bank first deducts the amount from the source account. Then, the check is sent to the recipient’s bank, where it is deposited into the destination account. For this method to work, the check must be delivered without being lost or deposited multiple times. Since the two banks don’t have to wait for each other during the process, this is an example of a non-blocking, asynchronous atomic transaction. However, during the transfer, the accounts are temporarily out of sync. This means asynchronous transactions are atomic but not isolated.

This concept can be extended to persistent messages sent over a network, ensuring each message is delivered and processed only once. We will see how asynchronous transactions can be implemented based on this principle using the outbox pattern design.

Outbox Pattern

Modern applications often replicate data across different storage systems for specific purposes. Consider Instagram, where user data and posts are stored in a relational database, but features like searching for users or hashtags are powered by a service like Elasticsearch. When a user updates their profile, adds a post, or modifies a hashtag, both the relational database and the search index must be updated. If Instagram's backend updates the database but crashes before updating the search index, the data would become inconsistent. To handle this, updates must be organized into a reliable transaction.

Using 2PC might seem like a solution, but services like Elasticsearch do not natively support it. Additionally, blocking Instagram's backend while waiting for the search service to respond isn’t ideal. Instead, eventual consistency is a practical approach.

Here’s how it can work: when a user updates their profile or posts a new image, the backend appends a persistent message to an outbox table in the relational database as part of the local transaction. This ensures the message is only added if the transaction succeeds. For example, when a user uploads a new photo with hashtags, the photo’s metadata and hashtags are stored in the database, and a message about updating the search index is added to the outbox.

A relay process monitors the outbox table. It picks up new messages and forwards them to the search service. Once the update is successfully processed, the relay removes the message from the outbox. If the relay fails before deleting the message, it might resend it. To prevent duplicates in the search index, an idempotency key is included with each message.

Instead of directly sending updates to Elasticsearch, the relay could send them to a message broker like Kafka or AWS SNS, ensuring reliable delivery in the same order messages were created. This approach mirrors state machine replication, where Instagram's state—like user data or post metadata—is synchronized through a log of operations maintained in the outbox table.

I am building a productivity tool for software leads/managers where I am implementing this outbox table and relay in the CLI client. I will share the complete design in another article.

Conclusion

Asynchronous transactions, implemented using patterns like the outbox, offer a reliable, scalable solution for ensuring eventual consistency in modern distributed systems.

Here are links to my previous posts, which I publish every Sunday on distributed systems:

• Building Resilient Applications: Insights into Scalability and Distributed Systems
• Understanding Server Connections in Distributed Systems
• How are your connections with web secure and integral?
• Understanding System Models in Distributed Systems
• Ping & Heartbeat in distributed systems
• How Real-Time Editing Works: Understanding Event Ordering in Distributed Systems
• High Availability for Social Media Platforms: Leader-Follower Architecture and Leader Election
• ACID Properties in Single Transactions Explained
• How is Concurrency Control Maintained to Ensure Isolation?
• Ensuring Atomicity in Modern Databases
• HTTP Caching in Distributed Systems

Feel free to check them out and share your thoughts!