How to mock Elasticsearch with Jest?

Originally published here

Intro

I guess you are probably using Elasticsearch or planning to since you've opened this article 🙂 Another assumption - you definitely found it a great idea to lock your functionality with some integration tests. And it really is!

The current company I’m working for has 90+% of code coverage with both unit and integration tests! I would recommend everyone cover their code base with tests since as one wise man said:

[Without unit tests] You’re not refactoring, you’re just changing shit.
— Hamlet D’Arcy

Setup

Imagine you have a simple RESTful server with some logic that uses Elasticsearch. In the current showcase — CRUD server.

const Hapi = require('@hapi/hapi');
const Qs = require('qs');
const {createHandler} = require("./create/index.js");
const {readAllHandler, readHandler} = require("./read/index.js");
const {updateHandler} = require("./update/index.js");
const {deleteAllHandler, deleteHandler} = require("./delete/index.js");

const init = async () => {

    const server = Hapi.server({
    port: 3000,
    host: 'localhost',
    query: {
        parser: (query) => Qs.parse(query)
    }
    });

    server.route({
    method: 'POST',
    path: '/',
    handler: createHandler
    });

    server.route({
    method: 'GET',
    path: '/{id}',
    handler: readHandler
    });

    server.route({
    method: 'GET',
    path: '/',
    handler: readAllHandler
    });

    server.route({
    method: 'PATCH',
    path: '/{id}',
    handler: updateHandler
    });

    server.route({
    method: 'DELETE',
    path: '/{id}',
    handler: deleteHandler
    });

    server.route({
    method: 'DELETE',
    path: '/',
    handler: deleteAllHandler
    });

    await server.start();

    server.events.on('log', (event, tags) => {
    console.log({event}, {tags})
    if (tags.error) {
        console.log(`Server error: ${event.error ? event.error.message : 'unknown'}`);
    }
    });
    console.log('Server running on %s', server.info.uri);
};

process.on('unhandledRejection', (err) => {

    console.log(err);
    process.exit(1);
});

init();

Now you need to cover the logic of each route with some tests to lock functionality and prevent business logic from being broken.

One obvious but not simple solution is to use Docker and spin up Elastic for tests every time.

However, does it worth it? I mean you really want to have a longer spin-up time for the pipeline environment? Maybe there are already-built solutions for that?

This plugin downloads and caches Elasticsearch binary when jest starts, then the plugin automatically starts Elastic on defined ports and tears it down when tests are done.

How to add tests?

I have an example of with/without jest tests setup in the pull request so you can compare all the changes. But now let’s go through it step by step.

1. Install additional modules

yarn add --dev jest @shelf/jest-elasticsearch @types/jest

2. Add jest-config.js

touch jest.config.js

module.exports = {
    preset: '@shelf/jest-elasticsearch',
    clearMocks: true,
    collectCoverage: true,
    coverageDirectory: "coverage",
    coverageProvider: "v8"
};

Alternatively, you can generate jest config on your own using the CLI tool.

3. Add jest-es-config.js for the plugin

touch jest-es-config.js

const {index} = require('./src/elastic.js');

module.exports = () => {
    return {
    esVersion: '8.4.0',
    clusterName: 'things-cluster',
    nodeName: 'things-node',
    port: 9200,
    indexes: [
        {
        name: index,
        body: {
            settings: {
            number_of_shards: '1',
            number_of_replicas: '1'
            },
            mappings: {
            dynamic: false,
            properties: {
                id: {
                type: 'keyword'
                },
                value: {
                type: 'integer'
                },
                type: {
                type: 'keyword'
                },
                name: {
                type: 'keyword'
                },
            }
            }
        }
        }
    ]
    };
};

4. Extend package.json script to run tests

{
    "scripts": {
    "test": "jest"
    "serve": "node src/index.js"
    }
}

5. Tune elastic client

const dotenv = require('dotenv')
dotenv.config()
const {Client} = require('@elastic/elasticsearch');

module.exports.client = new Client({
    node: process.env.NODE_ENV === 'test' ? 'http://localhost:9200' : process.env.ES_URL
})

module.exports.index = 'things'

Add a condition that will use NODE_ENV to connect to local spined-up elastic whenever we are running tests.

Profit!

Now all the things are ready to write and run tests. All routes are fully covered and stored here:
elastic-jest-example

As an example let’s cover create function business logic.

const {ulid} = require('ulid');
const {client, index} = require("../elastic.js");

module.exports.createHandler = async (request, h) => {
    if (Object.keys(request.payload))
    try {
        const res = await this.create(request.payload)
        return h.response(res).code(200);
    } catch (e) {
        console.log({e})
        return h.response({e}).code(400);
    }
}

// let's cover this function with some tests
module.exports.create = async (entity) => {
    const {
    type,
    value,
    name,
    } = entity;

    const document = {
    id: ulid(),
    type: type.trim().toLowerCase(),
    value: +value.toFixed(0),
    name: name.trim()
    }

    await client.index({
    index,
    document
    });
    return document.id
}

Create a test file and add a couple of statements.

touch  src/create/index.test.js

const {create} = require("./index.js");
const {client, index} = require("../elastic");

describe('#create', () => {

// clear elastic every time before running it the statement.
// It's really important since each test would be idempotent.
    beforeEach(async () => {
    await client.deleteByQuery({
        index,
        query: {
        match_all: {}
        }
    })
    await client.indices.refresh({index})
    })

    it('should insert data', async () => {
    expect.assertions(3);
    const res = await create({type: 'some', value: 100, name: 'jacket'})
    await client.indices.refresh();
    const data = await client.search({
        index,
        query: {
        match: {
            "id": res
        }
        }
    })

    expect(res).toEqual(expect.any(String))
    expect(res).toHaveLength(26);
    expect(data.hits.hits[0]._source).toEqual({
        "id": res,
        "name": "jacket",
        "type": "some",
        "value": 100
        }
    );
    })

    it('should insert and process the inserted fields', async () => {
    const res = await create({type: 'UPPERCASE', value: 25.99, name: ' spaces '})
    await client.indices.refresh();
    const data = await client.search({
        index,
        query: {
        match: {
            "id": res
        }
        }
    })
    expect(data.hits.hits[0]._source).toEqual({
        "id": res,
        "name": "spaces",
        "type": "uppercase",
        "value": 26
        }
    );
    })
});

A basic testing flow for each business logic function can be simply described like this:

insert data-> run tested function -> check outputs -> clear data -> repeat

Data insertion/deletion can be improved by unifying them into helpers and using additional mooching libs.

The elastic teardown is managed by @shelf/jest-elasticsearch lib itself.

One more good convention to follow is to cover each function you are testing with describeblock, so that later you can easily run a specific test with IDE helper without reruning whole suite:

Resources

Now you know how to test your Elasticsearch queries using jest.
jest-elasticsearch

Also, you have a small blueprint repo with ready to use setup!
elastic-jest-example

Hope this article will help you set up and test your elastic project!

Want to connect?

Follow me on Twitter!

Read more:
How to update 63 million records in MongoDB 50% faster?

Optimizing massive MongoDB inserts, load 50 million records faster by 33%!