What is Publish/Subscribe (Pub/Sub)?

Pub sub is another one of communication design patterns solving one specific problem - decoupling system.

Lets visualise the problem first...

In traditional communication models like Request Response, services often directly communicate with each other, leading to a tightly coupled system where each service is aware of the others. This tight coupling can make the system hard to scale, maintain, and update because changes in one component can ripple through to others, necessitating widespread modifications.

In a tightly coupled system for uploading a video to a platform like YouTube, the process might involve direct requests and responses between multiple services.

1. Client Application - The user uploads a video through the client application.
2. Video Upload Service - The client sends the video to the Video Upload Service.
3. Video Processing Service - Once the upload is complete, the Video Upload Service makes a direct request to the Video Processing Service to encode the video.
4. Thumbnail Generation Service - After encoding, the Video Processing Service requests the Thumbnail Generation Service to create a thumbnail.
5. Metadata Service - Once the thumbnail is ready, the Video Processing Service updates the video metadata by directly contacting the Metadata Service.
6. Notification Service - Finally, the Metadata Service requests the Notification Service to inform the user that their video is ready.

The response might propagate back through each service in the sequence, probably solved in multiple ways, but if we decide to add new service using this model, for example add 3.5 service for generating subtitles (making thing up here) then Video Processing Service will now end up with two dependencies.

As the system gets more complicated everything becomes more connected/dependent on each other and we end up with higher coupling.

Definitions

Definitions of entities that take part in pub/sub communication pattern.

• Who is publisher?

An entity that sends messages to a Pub/Sub system, targeting a specific topic for those messages -> send messages to the topic.

• Who is subscriber?

An entity that receives messages from a Pub/Sub system by expressing interest in a specific topic -> subscribe to the topic.

• What is a topic?

A virtual channel in a Pub/Sub system used to categorize messages. Publishers send messages to topics, and subscribers receive messages from topics they are subscribed to.

• What is a message?

A unit of data sent by a publisher to a topic within a Pub/Sub system, intended for consumption by subscribers.

• What is a message broker/event bus?

An intermediary platform that facilitates message transfer between publishers and subscribers, managing topics, message queuing, delivery, and sometimes message transformation.

How does Pub/Sub decouples the communication?

Pub-sub addresses this by introducing a message broker or event bus that acts as an intermediary between publishers and subscribers. Publishers broadcast messages without needing to know who the subscribers are. Similarly, subscribers listen for messages of interest without knowing who the publishers are.

This decoupling allows for greater flexibility, scalability, and ease of integration. Components can be added, removed, or updated independently, as long as they adhere to the shared messaging format. This is particularly beneficial in distributed systems where components might span multiple services or networks.

Video Upload Example as Pub/Sub

1. Client Application (Publisher) - The user uploads a video through the client application, which publishes a message to the "VideoUploaded" topic on the message broker, including the video's data or reference.
2. Message Broker (Event Bus) - Acts as an intermediary to manage the communication between publishers and subscribers. It receives messages published to topics and routes them to the appropriate subscribers.
3. Video Processing Service (Subscriber and Publisher) - Subscribed to the "VideoUploaded" topic. Upon receiving a message that a video has been uploaded, it processes the video (e.g., encoding) and then publishes a message to the "VideoProcessed" topic upon completion.
4. Thumbnail Generation Service (Subscriber and Publisher) - Subscribed to the "VideoProcessed" topic. It generates a thumbnail for the video once it receives a message from this topic and can publish a message to a "ThumbnailGenerated" topic if other services need to react to this event.
5. Metadata Service (Subscriber) - Also subscribed to the "VideoProcessed" topic (or "ThumbnailGenerated" if it needs the thumbnail to complete its task), it updates the video's metadata accordingly.
6. Notification Service (Subscriber) - This service might be subscribed to the "VideoProcessed", "ThumbnailGenerated", or another topic indicating the video is fully ready and available for viewers. It sends a notification to the user once it receives the relevant message.

This new pub/sub pattern allowed us to decouple the communication, by adding an intermediary -> message broker. Thanks to it no publisher and subscribers don't really need to know about each other, the whole app won't stop working if one services starts failing.

Its also worth to note that not every service needs to be a publisher or subscriber at the same time. Everything depends on implementation details and may vary depending on requirements.

---

Hope you liked this intro to Pub/Sub! Cheers!