Ensuring Message Order in Distributed Systems: Addressing Pub/Sub Ordering Key Limitations with Subscriber-side Sorting

Executive Summary

In event-driven and distributed architectures, preserving the order of messages is critical for maintaining state consistency, processing accuracy, and system reliability. Google Cloud Pub/Sub offers ordering keys to ensure that messages with the same key are delivered in the order they were published. However, this feature does not always ensure in-order delivery at the subscriber level, which is particularly critical for platforms where sequence integrity is vital (e.g., financial systems, event sequencing). This paper outlines the causes behind unordered message reception of Google Pub/Sub despite the use of ordering keys and outlines a practical solution involving subscriber-side message sorting to enforce the expected message sequence.

Overview

Google Cloud Pub/Sub is a widely adopted messaging service designed for scalable, asynchronous communication. In a use case where the sequence of events is essential, Pub/Sub provides support for ordering keys, a mechanism intended to preserve the order of messages that share the same key.

While ordering keys are conceptually straightforward, their behaviour in real-world, distributed systems can be unpredictable for message delivery. Applications often report receiving messages out of order at the subscriber end.

Problem Statement and Root Causes

Problem Statement

Despite enabling ordering keys in Google Cloud Pub/Sub, intended to ensure that messages with the same key are delivered in the exact sequence they were published, the real-world implementations frequently show that subscribers still receive messages in a non-deterministic, out-of-order fashion.

Root Causes

-Subscriber Concurrency and Load Balancing
Even with a single ordering key, if multiple threads are consuming messages, race conditions can cause out-of-order processing.
-Message Redelivery After Failures
When a message delivery is not acknowledged on time due to network issues, the Pub/Sub may redeliver it, potentially after newer messages have already been processed.
-Autoscaling and Stateless Subscriptions
Auto-scaled or stateless subscriber deployments cannot guarantee that messages from the same ordering key are handled by the same worker.
-Latency in Distributed Systems
Cross-region deployments and transient network issues can induce delivery delays that disrupt the message sequence.

Use Cases of Google Pub/Sub

1. Music Playlist: Real-time data streaming and analytics, enabling features like personalized recommendations and playlist updates

2. App Notification: Leverages for real-time notifications and event processing, ensuring seamless communication between users

3. Booking: Event-driven architecture, handling booking updates, notifications, and real-time data synchronization

4. Retail and E-commerce: Inventory management, order processing, and real-time customer notifications

5. Financial Services: Transaction processing, and real-time alerts

6. Gaming Platforms: Multiplayer gaming for real-time game state updates and player interactions

Solution: External Sorting at the Subscriber Level

To mitigate these challenges, we have implemented external ordering mechanisms at the subscriber level. This strategy involves sorting using Ordering key and Session ID for each session which is effective without the prior knowledge of the quantity of data with each session.

Key Components of External Sorting

Generating a Custom Ordering Key on the Publisher’s End
Custom ordering keys are generated at the publisher’s end for each message which will be beneficial to arrange data in specific order at subscriber’s end.

Generating Session ID for each session
To effectively bifurcate messages of different sessions which also help on the subscriber’s end to sort messages without prior knowledge of the number of messages sent by the publisher.

Benefits of Subscriber-side Sorting

Trade-offs and Considerations

While subscriber-side sorting restores order, it also introduces:

Latency: Small delays are introduced for message buffering.

Memory Overhead: Additional memory is required to buffer and sort messages.

Complexity: Requires careful handling of duplicate messages.

Conclusion

Google Cloud Pub/Sub’s ordering keys provide a valuable foundation for sequencing messages, but they do not guarantee ordered delivery in all conditions. For systems where strict message order is non-negotiable, subscriber-side message sorting is a practical and proven solution. By enriching messages with ordering key and session id and implementing intelligent buffering at the consumer end, developers can ensure consistency, correctness, and reliability in distributed processing pipelines.

Appendix: Environment Setup, Test Data and Sample Output

Environment Setup

Prerequisites

1. Create a Google Cloud account
2. Generate a private key
3. Set up topics in Pub/Sub
4. Create subscriptions for the topics

Steps to follow post the creation of the account and private key

1. Download and install the Google Cloud SDK using the following link: Google Cloud SDK Installation Guide.
2. Open the Command Prompt (CMD) and run the command: gcloud init.
3. Sign in by selecting your account and completing the login process.

Prerequisites before running the publisher and subscriber code:

1. Set up a virtual environment.
2. Install the required library by executing: pip install google-cloud-pubsub.
3. Place the private key file (key.json) in the repository directory.

Test Data

Below is the test data showcasing the results before and after implementing the solution. The comparison highlights the key metrics and their respective changes over time.

Data Sent by Publisher

1st Session: 3349

2nd Session: 9942

Data received at the subscriber end with no implementation of proposed solution

Explanation: All the messages are received on subscriber end in an unordered manner of messages as well as session id.

Data sorted by implementing proposed solution at the subscriber end

1st Session: 3349

2nd Session: 9942

Sample Output

Two workflows with different message counts will be published. Each message signifies a key event, and the sequence of these events plays a vital role. Both workflows are being published by the same source.

Below are the different communication types:

One-to-one Communication:

• 1. In one-to-one communication, a message is sent from a single publisher to a single subscriber.
• 2. This is more like a direct messaging system, where the message is intended for a specific recipient.

Publisher End

Message published for the first session with an ordering key.

Message published for the second session with an ordering key.

Output of Subscriber

One-to-Many Communication

• A single publisher sends messages to multiple subscribers.
• Subscribers receive messages based on their subscription to specific topics.
• Efficient broadcasting of information to multiple endpoints without requiring direct connections between the publisher and subscribers

Publisher End
Message published for the first session with an ordering key.

Message published for the second session with an ordering key.

Output of Subscriber

Message received for the first subscriber.

Message received by the second subscriber

Authors:

1. Jigar Solanki
Jigar Solanki serves as an Engineer at eInfochips, where he significantly contributes to QA Automation, Development and DevOps initiatives. With over 4 years of dedicated experience at eInfochips, Jigar has developed strong proficiency in delivering innovative, high-quality engineering solutions. He holds a Bachelor of Technology degree in Computer Science and Engineering from Parul University, Vadodara.

2. Arvind Mukati
Arvind Mukati serves as a Senior Engineer (Level 1) at eInfochips, where he plays a pivotal role in QA automation and DevOps initiatives. With over 4 years of dedicated experience at eInfochips, Arvind has cultivated a strong proficiency in delivering innovative and high-quality solutions. He holds a Master of Computer Applications degree from the Acropolis Institute of Technology and Research, Indore.

3. Mansi Thakkar
Mansi Thakkar serves as an Engineer at eInfochips, actively contributing to QA automation, Development and DevOps initiatives. With over 3 years of experience at eInfochips, Mansi has honed her expertise in delivering high-quality solutions. She holds a Master of Computer Applications (MCA) degree from LJ University, Ahmedabad.