How to Create a Full Autocomplete Search Application with Elasticsearch, Kibana and NestJS - Part 3

Hi guys, welcome to back to the Part 3 of the Elastic, Kibana and NestJS series. In the part 1 of this series, we installed and configured elasticsearch (check it here if you missed it), in the part 2, we connected elasticsearch with Kibana and ran a few queries (check here for part 2).

In this article, we will be writing the NodeJS code that will connect and query elasticsearch.

Loading data into elasticsearch

To enable us write our code effectively, we need data loaded into our elasticsearch. We will be using a sample dataset from Kaggle (Download it here.

Follow the FOUR steps below to load it up into elasticsearch

1. Open up Kibana (http://localhost:5601)
2. Under Get started by adding integrations click on Upload a file
3. Click on import and enter the name of the index you want to put the data in.
4. Click on import (final page)

If you made it to this point, you have successfully imported data into elasticsearch.

Querying for sample

Goto DevTools (Hamburger on the top left corner of the screen > Scroll down to Management > DevTools)

Run the query below (select it and click on the play button)

GET tmdb_movies/_search

If you see this, we are good to go!

---

Now, Let's dive into coding, shall we 😊?

NestJs

NestJS is a progressive Node. js framework that helps build server-side applications. Nest extends Node. js frameworks like Express or Fastify adding modular organization and a wide range of other libraries to take care of repetitive tasks. It's open-source, uses TypeScript, and is a very versatile Node.

Creating a NestJs Application

Run the command below to install nestcli and create a new NestJs Application (in the article, the name of the app will be nest-elastic).

$ npm i -g @nestjs/cli
$ nest new nest-elastic

You will be asked to select a package manager, you can select npm, yarn or pnpm. I will be selecting yarn (you can choose any other one you want 😉). Your project will be setup and we should be ready to code!

Adding elasticsearch to your app

Run the command below to add elasticsearch to your nest-elastic and other dependencies.

yarn add @elastic/elasticsearch @nestjs/elasticsearch @nestjs/config

In your root folder add your .env file with the following contents

ELASTICSEARCH_NODE=https://localhost:9200
ELASTICSEARCH_USERNAME=elastic
ELASTICSEARCH_PASSWORD=elasticPasswordGoesHere
ELASTICSEARCH_MAX_RETRIES=10
ELASTICSEARCH_REQ_TIMEOUT=50000
ELASTICSEARCH_INDEX=tmdb_movies

Let's create a separate module that handles only search using elasticsearch. A simple shortcut is to use the command below (you are welcome to do it manually, if you want too)

nest g resource search

Update the search.module.ts to have the content below

import { Module } from '@nestjs/common';
import { SearchService } from './search.service';
import { SearchController } from './search.controller';
import { ConfigModule, ConfigService } from '@nestjs/config';
import { ElasticsearchModule } from '@nestjs/elasticsearch';

@Module({
  imports: [
    ConfigModule,
    ElasticsearchModule.registerAsync({
      imports: [ConfigModule],
      useFactory: async (configService: ConfigService) => ({
        node: configService.get('ELASTICSEARCH_NODE'),
        auth: {
          username: configService.get('ELASTICSEARCH_USERNAME'),
          password: configService.get('ELASTICSEARCH_PASSWORD'),
        },
        maxRetries: configService.get('ELASTICSEARCH_MAX_RETRIES'),
        requestTimeout: configService.get('ELASTICSEARCH_REQ_TIMEOUT'),
      }),
      inject: [ConfigService],
    }),
  ],
  controllers: [SearchController],
  providers: [SearchService],
  exports: [SearchService],
})
export class SearchModule {}

Update search.service.ts with the content below:

import { ConfigService } from '@nestjs/config';
import { Injectable } from '@nestjs/common';
import { ElasticsearchService } from '@nestjs/elasticsearch';

type dataResponse = {
  UnitPrice: number;
  Description: string;
  Quantity: number;
  Country: string;
  InvoiceNo: string;
  InvoiceDate: Date;
  CustomerID: number;
  StockCode: string;
};

@Injectable()
export class SearchService {
  constructor(
    private readonly esService: ElasticsearchService,
    private readonly configService: ConfigService,
  ) {}

  async search(search: {key: string}) {
    let results = new Set();
    const response = await this.esService.search({
      index: this.configService.get('ELASTICSEARCH_INDEX'),
      body: {
        size: 50,
        query: {
          match_phrase: search
        },
      },
    });
    const hits = response.hits.hits;
    hits.map((item) => {
      results.add(item._source as dataResponse);
    });

    return { results: Array.from(results), total: response.hits.total };
  }
}

---

Now, let's add movies modules

nest g resource movies

Update movies.controller.ts with the content below:

import { SearchService } from './../search/search.service';
import { Body, Controller, Post } from '@nestjs/common';

@Controller('movies')
export class MoviesController {
  constructor(private readonly searchService: SearchService) {}

  @Post('search')
  async search(@Body() body) {
    return await this.searchService.search(body.data);
  }
}

Then movies.module.ts

import { SearchModule } from './../search/search.module';
import { Module } from '@nestjs/common';
import { MoviesService } from './movies.service';
import { MoviesController } from './movies.controller';

@Module({
  imports: [SearchModule],
  controllers: [MoviesController],
  providers: [MoviesService],
})
export class MoviesModule {}

Finally, update app.module.ts

import { MoviesModule } from './movies/movies.module';
import { ConfigModule } from '@nestjs/config';
import { Module } from '@nestjs/common';
import { AppController } from './app.controller';
import { AppService } from './app.service';
import { SearchModule } from './search/search.module';

@Module({
  imports: [MoviesModule, ConfigModule.forRoot(), SearchModule],
  controllers: [AppController],
  providers: [AppService],
})
export class AppModule {}

Your package.json should look like this

{
  "name": "nest-elastic",
  "version": "0.0.1",
  "description": "",
  "author": "Yusuf Ganiyu",
  "private": true,
  "license": "UNLICENSED",
  "scripts": {
    "prebuild": "rimraf dist",
    "build": "nest build",
    "format": "prettier --write \"src/**/*.ts\" \"test/**/*.ts\"",
    "start": "nest start",
    "start:dev": "nest start --watch",
    "start:debug": "nest start --debug --watch",
    "start:prod": "node dist/main",
    "lint": "eslint \"{src,apps,libs,test}/**/*.ts\" --fix",
    "test": "jest",
    "test:watch": "jest --watch",
    "test:cov": "jest --coverage",
    "test:debug": "node --inspect-brk -r tsconfig-paths/register -r ts-node/register node_modules/.bin/jest --runInBand",
    "test:e2e": "jest --config ./test/jest-e2e.json"
  },
  "dependencies": {
    "@elastic/elasticsearch": "^8.4.0",
    "@nestjs/common": "^9.0.0",
    "@nestjs/config": "^2.2.0",
    "@nestjs/core": "^9.0.0",
    "@nestjs/elasticsearch": "^9.0.0",
    "@nestjs/mapped-types": "*",
    "@nestjs/platform-express": "^9.0.0",
    "reflect-metadata": "^0.1.13",
    "rimraf": "^3.0.2",
    "rxjs": "^7.2.0"
  },
  "devDependencies": {
    "@nestjs/cli": "^9.0.0",
    "@nestjs/schematics": "^9.0.0",
    "@nestjs/testing": "^9.0.0",
    "@types/express": "^4.17.13",
    "@types/jest": "28.1.4",
    "@types/node": "^16.0.0",
    "@types/supertest": "^2.0.11",
    "@typescript-eslint/eslint-plugin": "^5.0.0",
    "@typescript-eslint/parser": "^5.0.0",
    "eslint": "^8.0.1",
    "eslint-config-prettier": "^8.3.0",
    "eslint-plugin-prettier": "^4.0.0",
    "jest": "28.1.2",
    "prettier": "^2.3.2",
    "source-map-support": "^0.5.20",
    "supertest": "^6.1.3",
    "ts-jest": "28.0.5",
    "ts-loader": "^9.2.3",
    "ts-node": "^10.0.0",
    "tsconfig-paths": "4.0.0",
    "typescript": "^4.3.5"
  },
  "jest": {
    "moduleFileExtensions": [
      "js",
      "json",
      "ts"
    ],
    "rootDir": "src",
    "testRegex": ".*\\.spec\\.ts$",
    "transform": {
      "^.+\\.(t|j)s$": "ts-jest"
    },
    "collectCoverageFrom": [
      "**/*.(t|j)s"
    ],
    "coverageDirectory": "../coverage",
    "testEnvironment": "node"
  }
}

Running the app

You can fire up the app in dev environment using yarn start:dev

Testing

You can access the full source code here

Summary

In this article, we are able to import data into elastic using kibana and also connect our NestJs backend app to the leverage the power of elasticsearch.

In the next article, we will be building a simple frontend to query and visualize results from elasticsearch in realtime.

Thanks for reading!