Building Cache Layer Using Redis and Mongoose

#node #redis #database #mongodb

Introduction.

If you ever built an api you will find that you will be needing to cache some GET requests that repeat alot and a find (if you are using mongoose) or select (sql) queries can be expensive over time. We are going to introduce a solution to this problem in this article.

Solution.

We will be following a very simple strategy here, But before we start you need to be familiar with mongoose and node.js

Strategy

Imagine we are working with a Query to fetch all dev.to blogs and The model will be called Blogs

Blogs Model

const blogSchema = new mongoose.Schema({
    owner : {
        // user in the database 
        type: mongoose.Types.ObjectId,
        required: true,
        ref: "User"
    },
    title: {
        type : String,
        required: true
    },
    tags: {
        type : [mongoose.Types.ObjectId],
    },
    blog: {
        type : String
    }
});

now the request to fetch all the blog

app.use("/api/blogs",(req,res,next)=>{
         const blogs = await Blogs.find({}); 
          res.send(blogs);
});

now after we get the image of what we are working with lets get back to the strategy

send a query to the database to ask for a certain thing
if this query has been fetched before aka exists in cache (redis)?
if yes, Then return the cached result
if no, Cache it in redis and return the result

The trick here is that there is a function in mongoose that is automatically executed after every operation
The function is called exec.

so we need to overwrite this exec function to do the caching logic.

first step to overwrite

const exec = mongoose.Query.prototype.exec;
mongoose.Query.prototype.exec = async function (){
    // our caching logic
    return await exec.apply(this, arguments);
}

now we need to make a something that tells us what gets cached and what doesn't. Which is a chainable function.

making the chainable function

mongoose.Query.prototype.cache = function(time = 60 * 60){
    this.cacheMe = true; 
    // we will talk about cacheTime later;
    this.cacheTime = time;
    return this;
}

So Now if i wrote

Blogs.find({}).cache(); // this is a valid code

Now if you are not familiar with Redis GO GET FAMILIAR WITH IT. there are thousands of videos and tutorials and it won't take that much time.

We need some data structure or types for the cached results. After some thinking, I've found out this is the best structure and I will explain why.

Blogs is the collection name;

let's say you are doing Blogs.find({"title" : "cache" , user : "some id that points to user" })

then Query will be { "title" : "cache" , "user" : "some id ... " , op : "find" // the method of the query } ;

result is the result we got from database;

This structure is called NestedHashes.

Why we are doing Nested Hashes like this

we need to say if Blogs got a new Update or Insert or Delete operation delete the cached result. Because the cached result is old and not updated by any of the new operations.

NOW back to code.

mongoose.Query.prototype.exec = async function(){
    const collectionName = this.mongooseCollection.name;

    if(this.cacheMe){   
      // You can't insert json straight to redis needs to be a string 

        const key = JSON.stringify({...this.getOptions(),
             collectionName : collectionName, op : this.op});
        const cachedResults = await redis.HGET(collectionName,key);

      // getOptions() returns the query and this.op is the method which in our case is "find" 

        if (cachedResults){
          // if you found cached results return it; 
            const result = JSON.parse(cachedResults);
            return result;
        }
     //else 
    // get results from Database then cache it
        const result = await exec.apply(this,arguments); 

        redis.HSET(collectionName, key, JSON.stringify(result) , "EX",this.cacheTime);
       //Blogs - > {op: "find" , ... the original query} -> result we got from database
        return result;
    }

    clearCachedData(collectionName, this.op);
    return exec.apply(this,arguments);
}

Remember the part where I said we need to clear cached data in case of Update, Insert or Delete.

clear the cached data

async function clearCachedData(collectionName, op){
    const allowedCacheOps = ["find","findById","findOne"];
    // if operation is insert or delete or update for any collection that exists and has cached values 
    // delete its childern
    if (!allowedCacheOps.includes(op) && await redis.EXISTS(collectionName)){
        redis.DEL(collectionName);
    }
}

Expected Results

Much faster find queries.

What to Cache

Don't Cache large data Imagine if you have a find query that return 20 MB or even 100 MB worth of data you will be slowing down your whole application.
Don't Cache Requests that don't get a lot of traffic and that is highly dependent on your application.
Don't Cache Important Data like users or transactions.

Final Notes

My redis setup.
cacheTime paramter is option I put a default of 1 hour but you can edit it as you wish, i suggest 1 or 2 days.

Top comments (4)

Fazlul Kabir • Oct 8 '22

Thanks for the article. It was a great starting point for me. But I had to make some improvements to get accurate behaviour. Here's the gist: gist.github.com/zisan34/64f5029449...