Using RedisJSON and RediSearch operation in Redis

There is a certain point of time in the software development lifecycle, to make a choice for our database. In this article, we shed light upon Redis and discuss why it’s an awesome database of choice. And why you should choose it to implement in your next project.

What is Redis?

Redis stands for REmote DIctionary Server. It is an open-source database. It uses an in-memory data structure to store its data. It is primarily used for cache and message brokers. It definitely is a good option to serve our application as a database.
Redis is a NoSQL database that stores data in Key/Value pairs. Unlike many other commonly used databases Redis uses memory to store its data rather than persistent data storing data on the disk.

Creating a Redis instance

We will use the Redis Enterprise cloud platform where we can get a generous free plan to start with it. But we still can interact with a Redis database with a local setup through the command line. You can create a fresh new account in Redis Enterprise and start implementing it at link here.

During the subscription process please make sure you have added RediSearch and RedisJSON modules during the initial configuration. These two modules in particular make Redis the powerful in-memory database every developer wants it to be: RedisJSON and RediSearch.

After the database is set up you can grab the connection string and password which we will add to our project in a minute.

Scaffolding our project

For our convenience, we will create a blank Next.js project from scratch with the command and name it redis-next-app

npx create-next-app redis-next-app

To interact with the Redis database we need a Redis client to be installed. We will be making use of redis-om as our client library. It is still possible to write a raw Redis command API but it's easier to just use a client library abstracting it.

RedisOM: Redis Object Mapping is a toolbox of high-level abstraction that makes it easier to work with Redis data in a programming environment. It's a fairly new library and simple to use. The Redis OM libraries let us transparently persist our domain objects in Redis and query them using a fluent, language-centric API. It also supports object mapping for Redis in Node.js

We will create a file that will consist of environment variables called

CONNECTION_URL=redis://default:PASSWORD@HOST:PORT

The values here are taken from a free instance or redis database we created with Redis Enterprise. Moreover, I would highly recommend making use of RedisInsight. It is a free application we can download and use to help with visualizing Redis data with models like JSON and time series. You can find it at https://redis.com/redis-enterprise/redis-insight/.

In later point of time if we need to see the data stored inside of Redis instance this tool becomes really handy.

Create a new folder and we will call it utils and inside there we will name it newJob. Inside here we will write our script to connect it to our database.

import { Client } from 'redis-om';

const client = new Client();


async function connect() {
  if (client.isOpen()) return;
  await client.open(process.env.CONNECTION_URL);
  if (!client.isOpen()) {
    // handle issue here  
  }
}

async function disconnect() {
  await client.close();
}

Here we connected to Redis using a client. The Client class has methods to open, close, and execute raw commands against Redis. We created two functions that basically connect to (only if there is no connection) and disconnect from the database. We will make use of this handy function as we go along.

Populating data

To fiddle around with Redis, we will get some data saved inside of our database. For that purpose, we will combine RedisOM and our Redis cloud instance and save Jobs data into our database.

To begin we need to do some object mapping. We will start by creating an entity and schema. For our schema, we will have just a list of hiring jobs.

class Job extends Entity {}
let jobSchema = new Schema(
 Job,
 {
   company: { type: 'string' },
   experience: { type: 'string' },
   website: { type: 'string' },
   title: { type: 'text', textSearch: true },
 },
 {
   dataStructure: 'JSON',
 }
);

Entities are the classes that we work with. The things being created, read, updated, and deleted. Any class that extends Entity is an entity.

Schemas define the fields on our entity, their types, and how they are mapped internally to Redis. By default the entities map to JSON documents using RedisJSON, we can also change it to use Hashes if needed.

Note: We have added the text search true boolean property to the title because we will use this value later for Searching and querying our data.

After we have prepared our entity and schema ready we need to make a repository. A repository provides the means to add CRUD functionalities like creating, reading, writing and removing entities.

export async function postNewJob(param) {
  await connect();
  const repository = client.fetchRepository(jobSchema);
  const newJob = repository.createEntity(param);
  const id = await repository.save(newJob);
  await disconnect();
  return id;
}

This function takes a parameter of values we pass in later through a simple form UI. The entities created by .createEntity are not saved to Redis but we need to get all the properties on the entity and call the .save() method.

We can also create and save in a single call to keep short with the createAndSave() method. At a later point in time, we can access this object entity with .fetch() passing it the ID we want to access it

const jobVacancy = await jobSchema.fetch('**object ID**');

To execute this we will need an API route we can make a request. We will create a new file that will act as a separate route in itself.

import { postNewJob } from '../../utils/newJob';

export default async function handler(req, res) {
  const id = await postNewJob(req.body);
  if (!id) return res.status(500).send('Error posting a job');
  res.status(200).json({ id });
}

Finally, we need to hook up a form to submit our form data. We will make use of ChakraUI which is a react library building accessible interfaces.

import { useRef } from 'react';

export default function Jobform() {
  const formElement = useRef();
 const toast = useToast();
 const handleSubmit = async (e) => {
   e.preventDefault();
   const form = new FormData(e.target);
   try {
     const formData = Object.fromEntries(form.entries());
     const res = await fetch('/api/jobs', {
       method: 'POST',
       headers: {
         'Content-Type': 'application/json',
       },
       body: JSON.stringify(formData),
     });
     if (res.ok) {
       await res.json().then(
         toast({
           title: 'Your new job is added.',
           description: "We've added your job to public to see.",
           status: 'success',
           duration: 6000,
           isClosable: true,
         })
       );
     }
     formElement.current.reset();
   } catch (err) {
     console.log(err);
   }
 };

 return (
   <>
    //all form with fields goes here
   </>
 );
}

Here we created an onSubmit handler which takes in all the data the user has filled in the form fields with the respective schema we defined. This later makes a POST request to the endpoint we created and saves all data in the database with a success toast notification finally clearing the form fields.

Performing query operation

After we have saved a decent number of data we can now perform query operations to our database. To perform the query operation we need to use another Redis module RedisSearch (until now we were just using the RedisJSON module).

Using RedisSearch with RedisOM is a powerful combination. Make sure we do have the RedisSearch module checked to make use of this feature. To start using this feature we first need to make an index. To build an index, just call .createIndex() in our repository.

export async function createIndex() {
  await connect();
  const repository = client.fetchRepository(jobSchema);
  await repository.createIndex();
}

If you change your schema at any time, RedisOM will automatically rebuild the index for you. We will need another endpoint to call this function we exported. For this we will create a new API route and call it createindex.js

import { createIndex } from '../../utils/newJob';

export default async function handler(_req, res) {
  await createIndex();
  res.status(200).send('Index is created');
}

Once we have the index created by calling this endpoint from the browser at

http://localhost:3000/api/createindex

We can start searching and querying our database. A basic search operation would look like

const allJobs = await jobSchema.search().return.all();
console.log(allJobs);

This will return all the jobs we have saved inside our DB till this point. We can implement a string search, in which searches match the entire string. But we want to make use of a Full-text search which can look for words, partial words, and exact phrases within a body of text. For example, if we search for apple, it matches apples, ape, application, and apple too and ignores punctuation.

To see it in action we can create a function that will return all the matches

//Adding full Text search on the job title
export async function getJobs(query) {
  await connect();
  const repository = client.fetchRepository(jobSchema);
  return await repository
    .search()
    .where('title')
    .does.match(query)
    .sortBy('title', 'DESC')
    .return.all();
}

We also need to create an endpoint that calls this function. Where we will pass the request query object to our function.

import { getJobs } from '../../utils/newJob';

export default async function handler(req, res) {
  const jobs = await getJobs(req.query);
  res.status(200).json({ jobs });
}

Time To Live(TTL) implementation

One of the great features of Redis is storing a Hash or key/value pairs for a limited span of time which automatically deletes itself and expires itself. TTL is generally provided in seconds or negative value to signal an error. We can select an entity to expire for a certain point of time with the .expiry() method.

For this to work, we need to provide it with the entity ID and time as a second parameter. This would look something like

const ttlInSeconds = 12 * 60 * 60  // 12 hours
await studioRepository.expire('***entity ID*', ttlInSeconds)

This will expire the entity with the given ID after 12 hours of the time interval. It will come in handy in a scenario when we want some entity to expire itself in a certain period of time.

Until this point, we have explored and implemented two core modules of Redis which are RedisJSON and RedisSearch. You can find the code in the repository on Github here.

Conclusion

This has been an overview of using RedisOM for Node.js and exploring how it can become handy in our projects. ​​We can also get a reference to the README and API docs on GitHub to get a deeper understanding of the library.

This is a relatively new library with a lot of helpful features baked in. It leverages the use of classes to create entities allowing us to create schema and interact with the database easily. It removes the need for more pesky, low-level commands with a fluent interface.

Learn More About Redis

Try Redis Cloud for Free
Watch this video on the benefits of Redis Cloud over other Redis providers
Redis Developer Hub - tools, guides, and tutorials about Redis
RedisInsight Desktop GUI

This post is in collaboration with Redis