Controlling Hibernate Commit programmatically for batch execution on Quarkus

Problem with many commits during batch execution

If you need to process a huge amount of data on a batch execution, it's important to control the number of commits that are done by Hibernate.

Commit is one of the most expensive database operations.

Using the @Transactional annotation

With Quarkus, you can control the execution of the commit using the @Transactional annotation. When the method annotated with @Transactional finishes successfully, Hibernate sends one commit to the database.

import javax.transaction.Transactional;
//
@Transactional(rollbackOn = Exception.class)
public void processClientOrders(Client client) {
     client.updateOrders();
     client.insertHistory();
     client.updateClient();
}

In the above example, when the processClientOrders() finishes, Hibernate sends one commit to the database.

Difficulty in using @Transactional annotation to control batch execution

On a batch execution, it's a good idea to control the numbers of commits using the quantity of processed items or time between commits.

On my point of view, is a bit difficult to control this using the @Transactional annotation, specially to control the quantity of time between commits.

Executing commit programmatically

The contribution I try to give here is one idea to control the commits using the programmatic way.

Define the commit parameters on you application

This item is not obligatory, but I think is a good idea. Using this approach you can tune your values at deploy time.

Define two parameters on your application.properties:

# Minimum quantity of items processed to execute commit
app.quantity.of.items.to.commit=1000

# Minimum quantity of time (milliseconds) to execute commit
app.quantity.of.time.ms.to.commit=1000

Control the Transaction programmatically

To control the Transaction programmatically you need to inject the UserTransaction and then begin and commit the transaction manually.

@Inject
UserTransaction transaction;

@ConfigProperty(name = "app.quantity.of.items.to.commit")
int quantityItemsCommit;

@ConfigProperty(name = "app.quantity.of.time.ms.to.commit")
int quantityTimeCommit;

public void batchProcess(List<Client> listOfClients)
                 throws SystemException,NotSupportedException, 
                 SecurityException, IllegalStateException,
                 RollbackException, HeuristicMixedException,  
                 HeuristicRollbackException {
  var beforeCommit = Instant.now();
  var quantityOfClientsPendingCommit = 0;
  transaction.begin();

  for (Client client : listOfClients ) {
    processClientOrders(client);
    var now = Instant.now();
    var delay = Duration.between(beforeCommit, now).toMillis();
    if (++quantityOfClientsPendingCommit >= quantityItemsCommit  
        || delay >= quantityTimeCommit) { 
      transaction.commit();
      transaction.begin();
      beforeCommit = Instant.now();
      quantityOfClientsPendingCommit = 0;
    }
  }
  transaction.commit();
}

private void processClientOrders(Client client) {
     client.updateOrders();
     client.insertHistory();
     client.updateClient();
}

The above code executes commit when the quantity of clients processed is greater or equal the limit or the time after the last commit is greater or equal the limit.

Final words

I hope that this article can add something and if you find some mistake that I did or have one idea to improve this concept, please comment.

Credits

Stack Overflow post about calculating time in milliseconds

Red Hat documentation about Managing JTA transactions programmatically