Knee-deep in Spring Boot, Transactional Event Listeners and CGLIB proxies

A colleague and I spent two hours tracking down a strange bug in our Spring Boot application today. The cause was so interesting that I have to write about it here. Since I obviously can't write about customer projects here, I'm presenting the problem using a sample application instead. So here we go.

Let's say we have a domain-driven application with two aggregates: Order and Invoice. We also have an orchestrator that automatically should create new invoices once an order transitions into the SHIPPED state.

We start with the following application service for creating new orders:

@Service
public class OrderService {

    private final OrderRepository orderRepository;

    public OrderService(OrderRepository orderRepository) {
        this.orderRepository = orderRepository;
    }

    @Transactional
    public Long createOrder() {
        var order = new Order();
        return orderRepository.saveAndFlush(order).getId();
    }

    @Transactional
    public void shipOrder(Long orderId) {
        orderRepository.findById(orderId).ifPresent(order -> {
            order.ship();
            orderRepository.saveAndFlush(order);
        });
    }
}

The Order.ship() method will change the state of the order to SHIPPED and publish an OrderStateChangedEvent.

Then we need another application service for creating new invoices:

@Service
public class InvoiceService {
    private final InvoiceRepository invoiceRepository;

    InvoiceService(InvoiceRepository invoiceRepository) {
        this.invoiceRepository = invoiceRepository;
    }

    @Transactional
    Long createInvoiceForOrder(Order order) {
        var invoice = new Invoice(order);
        return invoiceRepository.saveAndFlush(invoice).getId();
    }
}

Finally, we need an orchestrator that creates the invoice when the order is shipped:

@Component
class InvoiceCreationOrchestrator {

    private final OrderRepository orderRepository;
    private final InvoiceService invoiceService;

    InvoiceCreationOrchestrator(OrderRepository orderRepository, InvoiceService invoiceService) {
        this.orderRepository = orderRepository;
        this.invoiceService = invoiceService;
    }

    @TransactionalEventListener
    public void onOrderStateChangedEvent(OrderStateChangedEvent event) {
        if (event.getNewOrderState().equals(OrderState.SHIPPED)) {
            orderRepository
                .findById(event.getOrderId())
                .ifPresent(invoiceService::createInvoiceForOrder);
        }
    }
}

There! Now we just run the application, create a new order, ship it and... no invoice gets created. There are no exceptions in the log either. So what went wrong?

It turns out the problem is in the @TransactionalEventListener. It is by default configured to run after the transaction has been committed. This is exactly what we want, but there is a caveat in how Spring actually implements this.

Domain events are published using the ordinary application event publisher. Spring will actually catch them using an ordinary @EventListener as well. However, instead of invoking the transactional event listener directly, Spring will register a TransactionSynchronization with the TransactionSynchronizationManager. This will invoke the transactional event listener after the transaction has successfully committed, but before the transaction synchronization manager has cleaned itself up.

Now, our event listener is invoking the createInvoiceForOrder method, which has the @Transactional annotation. The default propagation for @Transactional is REQUIRED. This means that if there already is an active transaction, the method should participate in it; otherwise it should create its own transaction.

Because this method is being invoked inside a TransactionSynchronization, there actually is an "active" transaction but it has already been committed. Thus, the call to saveAndFlush will result in a TransactionRequiredException. This exception is swallowed by TransactionSynchronizationUtils (another Spring class) and logged using the DEBUG level. Thus, the only way to detect this exception is by having DEBUG logging turned on for the org.springframework.transaction.support package.

The solution to this problem is to make sure that the InvoiceService always runs inside its own transaction. So we change the method like this:

@Service
public class InvoiceService {
    @Transactional(propagation = REQUIRES_NEW)
    Long createInvoiceForOrder(Order order) {
      // Rest of the method omitted
    }
}

It is anyway a good practice to configure all your application services to always use REQUIRES_NEW since they are responsible for controlling the transactions.

Now we run the application again and... it still does not work. The application behaves exactly the same. What's wrong now?

It turns out that createInvoiceForOrder is not actually running inside a transaction at all. The transaction is started and committed by the call to saveAndFlush() in the repository, and that method still uses REQUIRED transaction propagation. How come?

The InvoiceService is not implementing any interfaces, so Spring is using a CGLIB proxy to add the transaction interceptors. However, the method createInvoiceForOrder happens to have package visibility and the transaction interceptor is only applied to public methods. So we need to change the method to be public:

@Service
public class InvoiceService {
    @Transactional(propagation = REQUIRES_NEW)
    public Long createInvoiceForOrder(Order order) {
      // Rest of the method omitted
    }
}

Now we run the application once more and it finally works!

Comments

[Dilip Vishwakarma]

Thank you for sharing your experience.
Keep it up.

[ampofila]

Where and how is the OrderStateChangedEvent created? I do not see any class by that name in the example and I do not know if Spring creates such event and emits it without calling send() somewhere.

[Petter Holmström]

It's not explicitly listed in the example, but the event would be created inside the Order aggregate root, where it would be registered. Spring Data would then publish the event when the aggregate root is saved (see dev.to/peholmst/building-aggregate... for more information).