DEV Community: Aniket Rathi

A beginner's guide to REDIS cache.

Aniket Rathi — Tue, 15 Feb 2022 18:38:46 +0000

What is caching

Caching is an intermediary stage to provide an abstract storage mechanism to support your backend. It enables efficient data extraction which are optimized to reduce the response time of your server. In simple terms, we store data in a temporary location so that the data can be easily accessed with minimal retrieval cost. It also reduces the bandwidth of data sent over the network thus making your application fast and user-friendly. Once a certain piece of data is obtained after several computations, it is stored in cache and we can directly access it the next time we need it skipping the additional cost of computation.

Factors to decide when to involve cache in your backend

Data chunk being used frequently
Caching makes sense only if you use a computated chunk of data very frequently. If this is not the case, caching wouldn't make any sense as a new set of data always has to be calculated and stored in cache.
Deciding your TTL
TTL is the time in seconds after which your key inside cache will get expired. It is of utmost importance that you have to decide the optimal time after which you want to update/remove the key from cache. The logic to maintain an up-to-date cache plays an important part in both your response time and more importantly not providing stale data in your response.

How does REDIS cache work

Redis stands for REmote DIctionary Server. It has the ability to store and manipulate high-level data types. It is an in-memory database, its data access operations are faster than any other disk-based database, which makes Redis the perfect choice for caching. Its key-value data storage system is another plus because it makes storage and retrieval much simpler. Using Redis, we can store and retrieve data in the cache using the SET and GET methods, respectively (just like Hashmap in java or dictionary in python).

Setting up Redis

We will be discussing about implementing Redis for a typical NodeJS server. To start with, we need to install redis node client. Also make sure that Redis is installed and running in your local. To find how to install and spin up redis do checkout here.

Working with Redis in Node layer

Using Redis is very simple. For any route receiving requests, we need to check if the route has cache enabled or not. If enabled we need to find if the data for requested key exists in the cache. If it exists, then without any database operation, we directly return it from the middleware itself. If not, then we compute that data and before returning it we also store in key-pair format in the Redis cache. The key used to store the data can be any custom string which can be formed using several parameters of your request.

const logger = require('winston-logger')
const CONFIG = require('configs/config')
const { redis: redisClient } = require('redis-client')
const axios = require('axios')

const getData = async (_, args, ctx) => {
  try {
    let { data: { slug, query } } = args

    //creating unique key based on slug
    let cacheName = `MEDIA_PAGE_COLLECTION-${slug}`
    let cacheData = await redisClient.get(cacheName)
    if (cacheData) {
      let data = JSON.parse(cacheData)
      return {
        data
      }
    } else {
      let url = `${CONFIG.contentful.baseUrl}/spaces/${CONFIG.contentful.spaceId}/environments/${CONFIG.contentful.environment}`

      let response = await axios({
        url,
        method: 'POST',
        headers: { 'Authorization': `Bearer ${CONFIG.accessToken}`},
        customQuery: {
          query
        }
      })
      data = response.data

      await redisClient.setex(cacheName, 43200, JSON.stringify(data))

      return {
        data
      }
    }
  } catch (error) {
    logger.error('ERROR WHILE FETCHING data >>>', error)
    return error
  }
}

The above code is an example of how to implement Redis-cache. Firstly we check if data exists or not. If does not exist then we then create a key dynamically store the data against the key. While storing the data, we have provided 3 parameters. First is the key for which the data has to be stored. Second is the TTL for which the data should be stored in the cache and the third parameter is the content. After the TTL, the key-value pair expires.
I have also attached a basic flow chart to demonstrate the basic functionality of how a typical cache works.

To install and use Redis for a node server checkout here

Database Indexing

Aniket Rathi — Fri, 17 Sep 2021 04:43:13 +0000

Indexing is always the most crucial part of a database design where you have to trade-off between write operation speeds and read/update/delete speeds. While deciding the type of indexes, one must always consider several factors like the kind of expected queries, read-to-write ratio, amount of memory in your system, size of your database etc.

While coming up with an indexing strategy, one must always have a deep understanding of the application's queries and what are the most frequently referred fields. Relative frequency of each query justifies whether it even needs an index or not.

Before using the strategy in production, indexes must be designed and tested over with different configurations in the dev-environment itself to inspect which strategy performs the best.
Inspect the current indexes created for your collections to ensure they are supporting your current and planned queries. If an index is no longer used, drop the index.

The following are some indexing strategies that can come handful while designing your database structure :

Indexing for your queries

An index supports a query when the index contains all the fields scanned by the query. Creating indexes that support queries results in greatly increased query performance.

Indexing for sorting related queries

One can support efficient queries, by using the strategies here while specifying the sequential order and sort order of index fields.

Ensuring that indexes fit in RAM

When the index fits in RAM, one can avoid reading the indexes from the disk and get faster processing speeds.

Create Queries that Ensure Selectivity

Selectivity is the ability of a query to narrow results using the index. Selectivity allows databases to use indexes for a larger portion of the work associated with fulfilling the query.

Different types of indexes available -

Single Field
Compound Indexing (Using multiple fields to create single index)
Text Indexing
Partial Indexing (Indexing a field based on a condition)
Sparse Indexing (Ensures that the each row in database contains the index or else the row is skipped)
TTL Indexing (Special indexing that can remove rows after certain amount of time)

A combination of the above mentioned indexes can easily enhance the processing speeds of your queries.

Using Environment Variables

Aniket Rathi — Sat, 14 Aug 2021 05:07:48 +0000

Environment variables are the most important part of your backend when it comes to deployment. They store the configuration of your server and hence should never be exposed. I am going to cite an example about using the environment variables in a node application.
The package.json can be a place to store your environment variables but is not at all a secure option.

dotenv

The .env file is a special type of file used to define environment variables for your node application in a key:value format. But nodejs is unable to parse this file. Here comes dotenv which takes care of these environment variables and helps node parse the .env file.

1. Creating the file
The .env file needs to be created in the root directory of your application. This file can contain your port, jwt secret key etc.

PORT=5000
JWT_SECRET_KEY="SHHHHHHH"

2. Configuring the dotenv
First you need to install dotenv as a dev package.

npm i -D dotenv

You can use your environment variables by importing them from .env file. So far so good. Your starting point(app.js) can consider the change when you pivot from dev to prod. But if you have imported and used your environment variables in other files, this can cause you trouble unless you initialize dotenv in each file. This is a frequent mistake made by beginners. With some tweaks in your scripts used to start the application, this trouble can be fixed easily.
3. Changing scripts
You might already have 2 scripts to run your application in dev and prod:

{
       "scripts": {
        "start": "node app.js",
        "dev": "node app.js"
        // For nodemon users ====
        "dev": "nodemon app.js"
    }
}

We need to change the dev script so that the node knows when to use your .env file

{
       "scripts": {
        "start": "node app.js",
        "dev": "node -r dotenv/config app.js"
        // For nodemon users ====
        "dev": "nodemon -r dotenv/config app.js"
    }
}

And done!
Now you don't require the following lines of code in any file including your app.js/index.js .

const dotenv = require('dotenv')
const myEnv = dotenv.config()

Ensure that you ignore the file in .gitignore

It’s time to get over mongoose for your Node application

Aniket Rathi — Fri, 30 Jul 2021 14:13:47 +0000

MongoDB is an easy-to-use and open-source database system designed to store data in document format. This is quite different than the traditional SQL databases which store data in the form of tables. Hence this makes MongoDB fall in the NoSQL category. These documents in MongoDB consist of a series of key/value pairs (similar to a hashmap in java or dictionary in python). A table of SQL is equivalent to a collection in MongoDB whereas a row is equivalent to a document. Each collection does not have a fixed definition of a document. Since the data to be stored is not structural, this makes MongoDB quite flexible for which we can see that the firms around the globe have been migrating their databases to MongoDB.
With this flexibility comes the problem of unstructured data. Developers familiar with SQL databases find it difficult to switch to such unstructured storage systems. This is where the mongoose comes into the picture. It is a powerful MongoDB ODM(Object Database Modelling). This helps in defining the structure of documents and connecting to MongoDB. This facilitates the developers who worked with SQL databases to migrate to the Mongo community. It also provides abstraction thus making object definitions more readable. So far so good.

Where’s the problem?

When building small applications, one can surely choose mongoose. But when it comes to large-scale applications where a user-data object is not restricted to just his address, email, and username, defining a structure becomes a tedious task. Also, such a type of data modeling will not be recommended by your data analysts.
Mongoose acts more like middleware in your node application that can validate the incoming data and restrict it for bad inputs. But do you really need an extra NPM package to do that for you? This can easily be handled for essential input fields by your services before forwarding it to your DB service. With time the structure of the document will change drastically and making a schema out of it will be a very complicated task. When dealing with information coming from packets, this will differ from one packet to another. Any packet containing important information won’t enter your database if it does not satisfy the mongoose schema hence deviating from a real-world scenario.

What should be done?

Well, if you are into NodeJS, you might be familiar with creating your own middlewares. You can use MongoDB native driver to connect to the database and write your middleware over it. This middleware can authenticate the required fields for you and pass your document in the next() method. This ensures that you don't create inappropriate documents and does not restrict your incoming data to a pre-defined schema. Your system should be able to welcome data of any shape allowing analysts to look out for irregularities among fields too instead of a restricted schema. Also, you might like to skip an extra package when your package.json increases with time.