DEV Community: Maciej Toporowicz

Transactional Outbox Pattern in a monolithic application

Maciej Toporowicz — Thu, 16 Apr 2020 09:36:10 +0000

Typically I can see the Transactional Outbox Pattern mentioned in the context of microservices and, to be honest, it's a bit surprising to me - to my mind it is as relevant in the context of monolithic applications.

In this article, I'll describe the pattern in detail and then provide an implementation example with Kotlin, Spring and PostgreSQL.

What is it for?

The transactional outbox pattern describes how to achieve reliable communication. The use case is: you'd like to atomically make an update to the database and invoke another action, typically on an external system. This action could be:

calling an external API, e.g. by making an HTTP request;

Example scenario: a user creates an account in our application, so we'd like to save the account in the database and send a welcome e-mail using AWS SES.
publishing a message to a queue;

Example scenario: an order is placed in our shopping app, so we'd like to save it to the database and enqueue an OrderPlaced event in AWS SQS.
in Domain Driven Design - updating another aggregate. This is an action invoked on the same system as the original database transaction. It would mean you'd like to atomically run two separate database transactions;

Example scenario: an order is placed in our shopping app, so we'd like to save the order and an invoice for it - order and invoice being two separate aggregates.

Side note

To my mind, the DDD example translates nicely into "calling an external API" or "publishing a message to a queue" when talking about microservices. Microservices may often represent specific bounded contexts and it is very likely that there would be a need to make an update in multiple contexts atomically. The transactional outbox pattern may prove to be very useful to make the communication between them reliable

Let's first take a look at the problem that this pattern lets us solve.

The problem with transactional consistency

Let's take a look at the "welcome email" example. When a user creates an account in our app, we'd like to reliably save it to the database and sent a welcome email using AWS SES. We could have the following piece of code in our application:

databaseTransaction {
    persist(account) // call to the database
}
sendWelcomeEmailFor(account) // call to SES

Now, what if the call to SES fails? We'll be left with an account in our database, but the user will never get the welcome email.

One might say that with SES having an uptime of at least 99.9% we can be virtually sure these two instructions will run just fine. But the failure may also be caused by us. We could e.g. configure our SES client with wrong credentials.

What if we can't or simply don't want to lose our messages?

We can't do the following:

databaseTransaction {
    persist(account) // call to the database
    sendWelcomeEmailFor(account) // call to SES
}

In case of a network issue we may not be able to reach SES. We'll rollback the databaseTransaction, so the account will not be saved. At the same time it means our users won't be able to create an account if SES is unreachable, which may not be perfect from the business perspective.

Another possible network issue may cause the response from SES not come back to us even though the call succeeds behind the scenes. We'll rollback the databaseTransaction, but the welcome email will be sent even though the related account is not in our database. This kind of inconsistency may not be desired either.

Another possibility is that sending the email succeeds, but the commit of databaseTransaction fails for some reason. We'll end up with the same inconsistency as with the "lost response" scenario.

Why not just use two-phase-commit?

Apart from the possibility that we may not be able (or not easily able) to use 2PC with the tools we chose, we need to take into account that 2PC is a synchronous protocol. It means we need to wait until all the participants of the transaction completed their actions and are ready to commit.

In our "welcome email" example, imagine a user needs to wait until the email is sent to receive a successful response. Or worse - as already described - the user is unable to create the account because of SES being down. Of course it depends on what the business needs, but I suspect that in general it'd be fine from the business perspective to let the user create an account even though the email service is down, and send the email as soon as it is up and running again.

I recommend the following article on this subject: Starbucks Does Not Use Two-Phase Commit.

To sum up - 2PC makes the communication between services synchronous, effectively reducing the availability of our service because we're depending on the availability of all the participants of the transaction. We won't be able to complete a transaction if a particular participant is down. What's more, our performance may suffer due to a participant taking a long time to finish its work.

The pattern

The idea is simple. In our database we create an additional table called e.g. outbox_message. We can then change our code to the following:

databaseTransaction {
    persist(account) // 'insert into account ...'
    persist(accountCreatedEvent) // 'insert into outbox_message ...'
}

We should be now able to respond to the user that the account has been successfully created. We still need to send the welcome email though.

Now, there are two strategies for processing the messages from the outbox_message table:

polling the 'outbox_message' table:

It means setting up e.g. a CRON job or a timer in your application to read and process the messages.
transaction log tailing:

It means monitoring the database's transaction log file to see what gets inserted into the outbox_message table and reacting to the inserts.

I'll describe only the polling mechanism a.k.a. 'the polling publisher'. Two reasons for it:

to my mind it is much simpler to implement;
I have not had a chance to implement the second one yet.

If you feel like reading up on it, you can refer to Transaction Log Tailing @ microservices.io - I believe it is a good start.

Polling publisher

In our application, somewhere in the background, we need to read the message from the outbox_message table, send the welcome email and then mark the message as processed so that it does not get processed again. We'll basically implement a simple queue in the database.

We may set up a CRON job or a timer in our application to do the polling.

each n milliseconds {
    databaseTransaction {
        message = readLastOutboxMessage()
        process(message) // send email using AWS SES
        markAsProcessed(message)
    }
}

At first it may look as if we were back to the original problem because we're trying to make a call to an external service in the middle of a database transaction. It is, however, not true. If we analyze what happens when any of the steps fail, we'll see that the message will be delivered eventually. There are, however, two specific scenarios:

sending the email succeeds behind the scenes but we don't receive the response (e.g. due to network issue);
commit of the transaction fails;

In both scenarios, the email will be sent, but we won't mark the message as processed. It means the message will be processed one more time and, as a result, the email will be sent more than once. The transactional outbox pattern guarantees at-least-once delivery. We either have to be alright with the external service being invoked more than once or the service needs to handle duplicate messages. The latter is, however, a topic for a separate article.

Implementation example

I'll create a simple web application to simulate the "welcome email" scenario.

I'll use Kotlin with Spring to model the application and PostgreSQL as the database. I believe the code samples can be easily translated to other languages and frameworks.

I chose PostgreSQL since I'm using it for a side project where I've already implemented this pattern and it has proved to be working just fine, so the code listed below is battle-tested. Besides, I'll use a SKIP LOCKED feature which is specific to PostgreSQL ≥9.5. It is used to lock rows for editing in order to build a concurrent queue. A similar feature should be available at least in Oracle Database and SQL Server¹.

You can see the complete working implementation @ https://github.com/maciejtoporowicz/transactional-outbox-example

We'll need two tables in our database:

account { id, name, email }
outbox_message { id, type, json_content }

View source code @ GitHub

When an account is created, in our business logic we need to create two objects: one for representing the account and another one for representing events related to the account creation. We can then ask our accountRepo to save both objects at once.

class AccountModule(private val accountRepo: AccountRepo) {
    fun createNewAccount(name: String, email: String) {
        val (account, relatedEvents) = Account.newAccount(name, email)

        accountRepo.save(account, relatedEvents)
    }
}

View source code @ GitHub

Here's the implementation of the AccountRepo::save method with some context:

class AccountRepoAdapter(
        private val accountRepo: SpringAccountRepo,
        private val outboxMessageRepo: SpringOutboxMessageRepo,
        private val jsonMapper: JsonMapper) : AccountRepo {

    @Transactional
    override fun save(account: Account, relatedEvents: List<AccountEvent>) {
        accountRepo.save(asEntity(account))
        outboxMessageRepo.saveAll(
                relatedEvents.stream()
                        .map {
                            OutboxMessage(
                                    eventType = it.javaClass.canonicalName,
                                    jsonContent = jsonMapper.toJson(it)
                            )
                        }
                        .toList()
        )

View source code @ GitHub

This is an adapter class between the business logic contained in AccountModule and the infrastructure-related classes - SpringAccountRepo and SpringOutboxMessageRepo which do the actual inserts to the database.

As you can see, I'm using the canonical name of the class as the OutboxMessage#type.

With the messages in the database, we can proceed to the polling mechanism. We've got a OutboxMessagePollerTimer class with a tick() method running at constant intervals:

@Scheduled(fixedRateString = "\${outboxMessages.polling.interval}")
fun tick() {
    log.info("Polling for outbox messages")
    taskExecutor.execute(outboxMessagePollerFactory.newOutgoingEventPoller())
}

View source code @ GitHub

The polling interval should be fine tuned to match the requirements and capabilities of the application.

Each time tick() is invoked, a new task is inserted into a thread pool to poll the outbox_message table and handle pending outbox messages. Here's the implementation of the task:

(...)

transaction.begin()

val message = deleteFirstMessage(entityManager)

if (message == null) {
    transaction.rollback()
    return
}

try {
    val messageContent = extractMessageContentFrom(message)
    applicationEventPublisher.publishEvent(messageContent)
    transaction.commit()
}

(...)

View source code @ GitHub

The published event gets picked up by a WelcomeEmailTrigger:

class WelcomeEmailTrigger {
    private val log = LoggerFactory.getLogger(WelcomeEmailTrigger::class.java)

    @EventListener
    fun onAccountCreated(event: AccountCreatedEvent) {
        log.info("Received event: $event")
    }
}

View source code @ GitHub

applicationEventPublisher.publishEvent(...) is synchronous, so it'll return only if all event handlers finish processing the published event. We should be safe to commit the transaction afterwards.

Key takeways

Transactioncal consistency reduces the availability of your system.
Do you need transactional consistency?
Eventual consistency does not mean that your system will be slow. It may often be a matter of milliseconds for it to become consistent.

Sources

https://www.2ndquadrant.com/en/blog/what-is-select-skip-locked-for-in-postgresql-9-5/ ↩

Fake the system clock for a single application with libfaketime

Maciej Toporowicz — Tue, 17 Mar 2020 20:41:52 +0000

A bit of a back story

My use case was quite specific - I wanted to conduct some end-to-end tests of a Java app running in a Docker container based on CentOS. The piece of code that I wanted to test was relying on date comparison:

if (happenedYesterday(event)) {
    foo();
} else {
    bar();
}

Given that I was not able to modify the test data, the easiest thing was to somehow make the application think it's yesterday, make it create theevent, then restore the original date and make the application invoke the above piece of code.

The setup

Since I would like to show how I got from an idea to a working solution, I need to provide a way to reproduce all the mistakes that I've made as part of this exercise - I need an environment which is as close to the original as possible. To achieve it and also to keep the examples as simple as possible, I'll use fabric8/java-centos-openjdk8-jdk Docker image.

Let's start a container from an unmodified image and open up its shell:

[me@pc ~]$ docker run --name centos -d -it fabric8/java-centos-openjdk8-jdk /bin/bash
...
[me@pc ~]$ docker exec -u 0 -it centos /bin/bash
[root@centos /]#

NOTE: -u 0 argument makes the command log into the container as root (0 is root's user id)

In all the examples in this post, I'll use pc as in [me@pc ~]$ to indicate commands invoked on my local machine and centos as in [root@centos /]# for commands invoked inside the container.

Setting the system date in Docker - naive approach

In my naive approach, I thought this step would be as simple as running one of these commands:

[root@centos /]# date -s "15 Oct 2019 19:05"
date: cannot set date: Operation not permitted
Tue Oct 15 19:05:00 UTC 2019
[root@centos /]# hwclock --set --date "15 Oct 2019 19:05"
hwclock: Cannot access the Hardware Clock via any known method.
hwclock: Use the --debug option to see the details of our search for an access method.

Unfortunately, it wasn't.

I tried to find some workarounds, but as far as I understand, Docker reuses the clock of the host machine, so overriding the date in the container is either not doable or not easily doable¹²³. As I'm just a casual user of Docker, I didn't want to dig deeper. However, when looking for the workarounds, I stumbled upon a different way to change the time - libfaketime.

libfaketime

libfaketime is a library which is able to 'override' system calls that applications use to retrieve current date or time. It is then able to provide a fake value for these calls. What's more, you don't have to change a line of your existing code or add it to your app's dependency list - it'll be transparent. Since I'm not a Linux guru, using this library at first felt like it was a wrapper for my java application, though it's not how it works.

The installation is straightforward - you grab the source code of the library and run make install in the root directory of the checked out sources. It'll result in a bunch of files getting created in /usr/local/lib/faketime. To automate this process, I created the following Dockerfile:

FROM fabric8/java-centos-openjdk8-jdk

USER root

RUN yum -y groupinstall 'Development Tools' && \
    yum -y install make unzip wget && \
    mkdir faketime && \
    cd faketime && \
    wget https://github.com/wolfcw/libfaketime/archive/master.zip && \
    unzip master.zip && \
    cd libfaketime-master && \
    make install

ENTRYPOINT /bin/bash

Now, it should be possible to start it all up and test the library:

[me@pc ~]$ docker build -t centos .
...
[me@pc ~]$ docker run --name centos -d -it centos /bin/bash
...
[me@pc ~]$ docker exec -u 0 -it centos /bin/bash
[root@centos /]# date
Wed Feb 19 17:16:34 UTC 2020
[root@centos /]# LD_PRELOAD=/usr/local/lib/faketime/libfaketime.so.1 FAKETIME="-15d" date
Tue Feb  4 17:16:49 UTC 2020

With the path to libfaketime provided in the LD_PRELOAD variable, the FAKETIME variable set to -15 days, the invocation of date rendered a date 15 days in the past. To me, the most interesting bit is the LD_PRELOAD part.

LD_PRELOAD variable and the preloading mechanism

The LD_PRELOAD variable allows to specify paths to libraries which are to be loaded before any other libraries are loaded. What's important, all the symbols (e.g. functions) contained in the preloaded libraries take precedence over the symbols from libraries loaded afterwards⁴⁵⁶.

It means that if a program uses a function foo() from library A and library A is linked at runtime, it is possible to provide a path to library B containing a different implementation of foo() in the LD_PRELOAD variable. As a result, when the program references foo() in its source code, the implementation from library B will be invoked.

libfaketime replaces the symbols related to interactions with the system clock using the preloading mechanism.

Given all the knowledge gathered so far, it's time to fake the date in a Java app.

Setting a fake date for a Java app

First of all - to test the library, I created a simple application that prints the current date and time each second, forever. Here's the code:

import java.time.LocalDateTime;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;

public class Main {
  private static final int NO_DELAY = 0;

  public static void main(String[] args) {
    Runnable printCurrentDateTime = () -> System.out.println(LocalDateTime.now());

    Executors
      .newSingleThreadScheduledExecutor()
      .scheduleAtFixedRate(printCurrentDateTime, NO_DELAY, 1, TimeUnit.SECONDS);
  }
}

Then, to test the library with the application, I ran the CentOS container again with a volume set to the directory where the java class was.

[me@pc ~]$ docker run --name centos -v /home/test/:/app -d -it centos /bin/bash
...
[me@pc ~]$ docker exec -u 0 -it centos /bin/bash
[root@centos /]# cd /app
[root@centos app] javac Main.java
[root@centos app] java Main
2020-02-20T21:53:36.132
2020-02-20T21:53:37.113
2020-02-20T21:53:38.113
^C
[root@centos app] LD_PRELOAD=/usr/local/lib/faketime/libfaketime.so.1 FAKETIME="-15d" FAKETIME_DONT_FAKE_MONOTONIC=1 java Main
2020-02-05T21:55:51.787
2020-02-05T21:55:52.766
2020-02-05T21:55:53.765
^C

It's working as expected, though I added another variable to the command: FAKETIME_DONT_FAKE_MONOTONIC=1. From the README of libfaketime:

Java-/JVM-based applications work but you need to pass in an extra argument
(FAKETIME_DONT_FAKE_MONOTONIC). See usage basics below for details. Without
this argument the java command usually hangs.

Indeed, without it even this tiny program hanged after printing the first date. I must admit I wasted some time debugging it just because I skipped reading the readme.

Cleaning it up

To shorten the commands, LD_PRELOAD and FAKETIME_DONT_FAKE_MONOTONIC values can be specified as the environment values of the Docker image. I omitted FAKETIME because this one is likely to change.

...

USER root

ENV LD_PRELOAD /usr/local/lib/faketime/libfaketime.so.1
ENV FAKETIME_DONT_FAKE_MONOTONIC 1

RUN ...

After these changes, it should be possible to run FAKETIME="-15d" java Main to render the same output as before.

Changing the time dynamically at runtime

It is possible to specify the FAKETIME value in a file instead of a variable. It makes it possible to change the value at any moment. libfaketime will pick it up after ten seconds⁷.

It is possible to do this system-wide or just for a user. I'll describe the latter.

The file needs to be named .faketimerc and it needs to be placed in the home directory. It should contain only the value of the FAKETIME variable, just like this:

-15d

Running java Main in the shell should now render the expected output. While the program running keeps running, in another shell session we can type echo -10d > ~/.faketimerc. The output should change after ten seconds.

Different variants of the FAKETIME value

Specifying a relative offset is not the only way to fake the time. Here are some more variants:

different offset mutlipliers: all the examples used "-16d", but instead of "d" it can also be "m", "h", "y" or nothing for seconds; the offset can be set in the past (-10d) or in the future (+10d) > EXAMPLES: > * -120 is 120 seconds behind > * +2h is 2 hours in the future > * +1y is 1 year in the future
'start at' date: FAKETIME="@2020-12-24 20:30:00", where the clock will start ticking from this date for each new process, but it's possible to configure it to keep the clock ticking instead⁷
absolute date: FAKETIME="2020-12-24 20:30:00" will render a fixed value, as if the time stopped at this point

Summary

These are not all the features that libfaketime supports. I suggest skimming through the list of features in the readme just to get familiar with the possiblities - just in case you ever need to use any of them.

Key takeways

I'll use libfaketime in cases similar to this one
LD_PRELOAD mechanism can be used on Linux machines to replace pieces of code without modifying the original code
I should have read the friendly manual earlier to save time

DEV Community: Maciej Toporowicz

Transactional Outbox Pattern in a monolithic application

What is it for?

Side note

The problem with transactional consistency

Why not just use two-phase-commit?

The pattern

Polling publisher

Implementation example

Key takeways

Further reading

Sources

Fake the system clock for a single application with libfaketime

A bit of a back story

The setup

Setting the system date in Docker - naive approach

libfaketime

LD_PRELOAD variable and the preloading mechanism

Setting a fake date for a Java app

Cleaning it up

Changing the time dynamically at runtime

Different variants of the FAKETIME value

Summary

Key takeways

Further reading

Sources