Writing integration tests for your spring-data-elasticsearch-based applications

We all know it by now, testing is an important part of our job as Software Engineers. If that is not currently the case for you or your organization, I couldn't recommend enough, that you start including it in your development workflow, asap.

We should also agree that testing, in particular integration testing, can sometimes be challenging or too complex to put together. Which is probably why, some developers simply skip that essential part.

To make things less complicated and more accessible, libraries like assert4j, mockito, spring-test, testcontainers have been created over the years. The easier it is to write tests, the more people will be willing to write them.

In a recent project I worked on, that uses Elasticsearch via Spring Data, we decided to write integration tests to validate that:

• our data were being correctly indexed in the Elasticsearch Server (ES),
• our search queries were working and performing as expected

For that purpose, we needed a way to easily load fresh data into a testing ES, and then run our search queries on it. We already had a tool to massively inject data from our UAT environment, so it would be nice if we could somehow extract these data back and use them as fixtures to feed our testing ES.

After a few searches online, not finding a good tool for the job, I decided to create and publish my own take on the problem : spring-esdata-loader.

It is an open source Java 8+ testing library to help write integration tests for spring-data elasticsearch-based projects, by allowing you to easily load data into ES using simple annotations, leveraging Spring's entity mappings (i.e classes annotated with @Document, @Field, etc) and via specific JUnit 4's Rules or JUnit Jupiter's Extensions.

The library reads all the metadata it needs from the entity classes (index name, index type, etc) , uses them to create/refresh the related indices and, feeds ES with the data using the ElasticsearchOperations already present in your Spring's test application context.

It offers the following out-of-the-box 📦 :

Features

• Simple API and no configuration required
• Support for JUnit 4 via @LoadEsDataRule, @DeleteEsDataRule
• Support for JUnit Jupiter via
  • @LoadEsDataConfig / @LoadEsDataExtension
  • @DeleteEsDataConfig / @DeleteEsDataExtension
• Built-in support for gzipped data
• Multiple data formats(dump, manual)
• Fast and Memory-efficient loading of data
• Written in Java 8
• Based on Spring (Data, Test)

Installation & Usage

The library is split into 2 independent sub-modules, both are available on JCenter and Maven Central for download:

• spring-esdata-loader-junit4 for testing with JUnit 4
• spring-esdata-loader-junit-jupiter for testing with JUnit Jupiter

So, to get started:

1.Add the appropriate dependency to your gradle or maven project

	JUnit 4	JUnit Jupiter
Gradle	dependencies {     testImplementation 'com.github.spring-esdata-loader:spring-esdata-loader-junit4:VERSION' }	dependencies {     testImplementation 'com.github.spring-esdata-loader:spring-esdata-loader-junit-jupiter:VERSION' }
Maven	<dependency>     <groupId>com.github.spring-esdata-loader</groupId>     <artifactId>spring-esdata-loader-junit4</artifactId>     <version>VERSION</version>     <scope>test</scope> </dependency>	<dependency>     <groupId>com.github.spring-esdata-loader</groupId>     <artifactId>spring-esdata-loader-junit-jupiter</artifactId>     <version>VERSION</version>     <scope>test</scope> </dependency>

2.Write your test class.

You can have a look at :

• JUnit4 Sample - for an example using JUnit 4 API
• JUnit Jupiter Sample - if you prefer JUnit Jupiter API

Supported Data Formats

spring-esdata-loader currently supports 2 formats to load data into Elasticsearch : DUMP and MANUAL.

Dump data format

Here is an example :

{"_index":"author","_type":"Author","_id":"1","_score":1,"_source":{"id":"1","firstName":"firstName1","lastName":"lastName1"}}
{"_index":"author","_type":"Author","_id":"2","_score":1,"_source":{"id":"2","firstName":"firstName2","lastName":"lastName2"}}
{"_index":"author","_type":"Author","_id":"3","_score":1,"_source":{"id":"3","firstName":"firstName3","lastName":"lastName3"}}
{"_index":"author","_type":"Author","_id":"4","_score":1,"_source":{"id":"4","firstName":"firstName4","lastName":"lastName4"}}
{"_index":"author","_type":"Author","_id":"5","_score":1,"_source":{"id":"5","firstName":"firstName5","lastName":"lastName5"}}
{"_index":"author","_type":"Author","_id":"6","_score":1,"_source":{"id":"6","firstName":"firstName6","lastName":"lastName6"}}
{"_index":"author","_type":"Author","_id":"7","_score":1,"_source":{"id":"7","firstName":"firstName7","lastName":"lastName7"}}
{"_index":"author","_type":"Author","_id":"8","_score":1,"_source":{"id":"8","firstName":"firstName8","lastName":"lastName8"}}
{"_index":"author","_type":"Author","_id":"9","_score":1,"_source":{"id":"9","firstName":"firstName9","lastName":"lastName9"}}
{"_index":"author","_type":"Author","_id":"10","_score":1,"_source":{"id":"10","firstName":"firstName10","lastName":"lastName10"}}

You can use a tool like elasticdump (requires NodeJS) to extract existing data
from your Elasticsearch server, and them dump them into a JSON file.

$ npx elasticdump --input=http://localhost:9200/my_index --output=my_index_data.json

The above command will run elasticdump to extract data from an index named my_index on a ES server located at http://localhost:9200 and then save the result into a file named my_index_data.json

If you change the --output part above into --output=$ | gzip my_data.json.gz the data will be automatically gzipped

Manual data format

In this format, you specify your target data directly (no metadata like _index, _source, ...), as an Array of JSON objects.

This is more suitable when you create test data from scratch (as opposed to dumping existing ones from a ES server) because it is easier to tweak later on to accommodate future modifications in tests.

Here is an example :

[
    {"id":"1","firstName":"firstName1","lastName":"lastName1"},
    {"id":"2","firstName":"firstName2","lastName":"lastName2"},
    {"id":"3","firstName":"firstName3","lastName":"lastName3"},
    {"id":"4","firstName":"firstName4","lastName":"lastName4"},
    {"id":"5","firstName":"firstName5","lastName":"lastName5"},
    {"id":"6","firstName":"firstName6","lastName":"lastName6"},
    {"id":"7","firstName":"firstName7","lastName":"lastName7"},
    {"id":"8","firstName":"firstName8","lastName":"lastName8"},
    {"id":"9","firstName":"firstName9","lastName":"lastName9"},
    {"id":"10","firstName":"firstName10","lastName":"lastName10"}
]

More complete examples can be found on the project's github under demo/.

Happy Hacking 📠!