Using Pub/Sub for event-driven Go backends

Pub/Sub is a powerful building block in a backend application. It's often used for building event-driven backend applications.

Pub/Sub can be used to achieve great benefits:

• Improving app reliability by reducing the blast radius of faulty components and bottlenecks.
• Increasing the speed of response to the use.
• Helping reduce cognitive overhead for developers by inverting the dependencies between services.

The difference of using Pub/Sub: A practical example

For those not familiar with Pub/Sub, let's take a look at an example API in a user registration service.

The behavior we want to implement is that upon registration, we send a welcome email to the user and create a record of the signup in our analytics system. Now let's see how we could implement this only using APIs, compared to how a Pub/Sub implementation might look.

Using an API only approach

Using API calls between services, we might design a system which looks like this when the user registers:

1. The user service starts a database transaction and records the user in its database.
2. The user service makes a call to the email service to send a welcome email.
3. The email service then calls an email provider to actually send the email.
4. Upon success, the email service replies to the user service that the request was processed.
5. The user service then calls the analytics service to record the signup.
6. The analytics service the writes to the data warehouse to record the information.
7. The analytics service then replies to the user service that the request was processed.
8. The user service commits the database transaction.
9. The user service then can reply to the user to say the registration was successful.

Notice how we have to wait for everything to complete before we can reply to the user to tell then we've registered them.
This means that if our email provider takes 3 seconds to send the email, we've now taken 3 seconds to respond to the user,
when in reality once the user was written to the database, we could have responded to the user instantly at that point to
confirm the registration.

Another downside to this approach is if our data warehouse is currently broken and reporting errors, our system will also
report errors whenever anybody tries to signup! Given analytics is purely internal and doesn't impact users, why should the analytics system being down impact user signup?

Using Pub/Sub

A more ideal solution would be if we could decouple the behaviour of emailing the user and recording our analytics, such that the user service only has to record the user in its own database and let the user know they are registered - without worrying about the downstream impacts.

Thankfully, this is exactly what Pub/Sub topics allow us to do.

In this example, when a user registers, we:

1. The user service starts a database transaction and records the user in its database.
2. Publish a signup event to the signups topic.
3. Commit the transaction and reply to the user to say the registration was successful.

At this point the user is free to continue interacting with the application and we've isolated the registration behaviour
from the rest of the application.

In parallel, the email and analytics services will receive the signup event from the signups topic and will then perform their respective tasks. If either service returns an error, the event will automatically be backed off and retried
until the service is able to process the event successfully, or reaches the maximum number of attempts and is placed into the deadletter queue (DLQ).

Notice how in this version, the processing time of the two other services did not impact the end user and in fact the user service is not even aware of the email and analytics services. This means that new systems which need to know about new users signing up can be added to the application, without the need to change the user service or impacting its performance.⚡️

Build your own Go backend with Pub/Sub

Normally it can be tricky to build using Pub/Sub. It means spending time provisioning resources in the cloud, and for local dev you need a whole other setup.

At Encore, we've made it easier by making Pub/Sub is a native component in Encore's Open Source Infrastructure SDK.

This means you can declare Pub/Sub topics and subscriptions directly in your Go code using as little as 3 lines of code.

Encore's CLI takes care of running your app locally with local infrastructure, and the Cloud Platform takes care of provisioning and deploying your app to your cloud in AWS/GCP.

Let's take a look at how you use it.

Creating a Pub/Sub Topic with Encore

When you create a Pub/Sub topic with Encore, you declare it as package level variable. Regardless of where you create a topic, it can be published to from any service, and subscribed to from any service.

package user

import "encore.dev/pubsub"

type SignupEvent struct{ UserID int }

var Signups = pubsub.NewTopic[*SignupEvent]("signups", pubsub.TopicConfig{
    DeliveryGuarantee: pubsub.AtLeastOnce,
})

Notice how there is no cloud-specific configuration, which means Encore can automatically set up the Pub/Sub infrastructure locally and in AWS and GCP.

With Encore, the only thing you define in application code is what matters to the behavior of the application — the infrastructure semantics.

This means you can define things like exactly-once delivery and at-least-once delivery. Learn more in the docs.

Publishing events

To publish an Event, call Publish on the topic passing in the event object (which is the type specified in the pubsub.NewTopic[Type] constructor).

For example:

messageID, err := Signups.Publish(ctx, &SignupEvent{UserID: id})
if err != nil {
    return err
}

// If we get here the event has been successfully published,
// and all registered subscribers will receive the event.

// The messageID variable contains the unique id of the message,
// which is also provided to the subscribers when processing the event.

By defining the Signups topic variable as an exported variable you can also publish to the topic from other services in the same way.

Creating a Pub/Sub subscription with Encore

To Subscribe to events, you create a Subscription as a package level variable by calling the pubsub.NewSubscription function.

Each subscription needs:

• the topic to subscribe to
• a name which is unique for the topic
• a configuration object with at least a Handler function to process the events
• a configuration object

Here's an example of how you create a subscription to a topic:

package email

import (
    "encore.dev/pubsub"
    "user"
)

var _ = pubsub.NewSubscription(
    user.Signups, "send-welcome-email",
    pubsub.SubscriptionConfig[*SignupEvent]{
        Handler: SendWelcomeEmail,
    },
)
func SendWelcomeEmail(ctx context.Context, event *SignupEvent) error {
    // send email...
    return nil
}

Subscriptions can be in the same service as the topic is declared, or in any other service of your application.

Each subscription to a single topic receives the events independently of any other subscriptions to the same topic. This means that if one subscription is running very slowly, it will grow a backlog of unprocessed events. However, any other subscriptions will still be processing events in real-time as they are published. Learn more in the docs.

Infrastructure tracking in the cloud

When you deploy your app, you simply git push and Encore provisions the infrastructure defined by your application.

Encore's Cloud Dashboard gives you an overview of the state of the provisioned infrastructure in all environments. And you even get a cost analytics dashboard that breaks down your cloud bill per logical resource like Pub/Sub supscription.

Try it yourself with a tutorial

If you're curious about how you build event-driven Go backends with Encore. You can try it yourself with this tutorial:
Building an event-driven uptime monitor

(If you haven't used Encore before, start by installing the CLI.)

Good luck, have fun! 🚀

PS. You're welcome to join Encore's Go developer community on Discord to chat with other backend developers and get help with your project.

Comments

[Rod Lecocq]

What about the double write pattern problem here?
Does you Encore client take care of using sth like the outbox pattern? Because here if the message sending fails you will rollback your database tx and send an error to the user, but the message sending works but your db commit fails you get inconsistency

[Marcus Kohlberg]

Hey @rod2j - great point! There is indeed a package for implementing the outbox pattern. See the docs here: encore.dev/docs/primitives/pubsub/...

[Olliekayden]

Leveraging Pub/Sub in event-driven Go backends facilitates seamless tngofullform communication between services, promoting scalability and decoupling. With publishers broadcasting events and subscribers reacting accordingly, this approach fosters a more responsive and resilient architecture. Integrating Pub/Sub enables asynchronous processing, enhancing the efficiency and adaptability of the backend system.