DEV Community: Medam Mahesh

Optimizing Pagination: Range-Based Queries with Cursors and Happy Databases 🚀

Medam Mahesh — Sat, 23 Sep 2023 15:38:12 +0000

👋 Introduction:

Pagination is used when you have a lot of records in your database collection. Since you can't just dump all the records on the screen, we give a bunch of the records and tell the client how to get the next bunch !!🔄

For the longest time, I had thought offset-based pagination was the way to do this (Haven't we all seen the page number section before ?).

Using Yahoo Search here cause Google disabled it

It comes naturally that when each page has 10 records and you click on Page 3, you have to skip the 30 records and return (31-40). If you translate the above to a DB query, it is as below.

db.article.aggregate([
    { "$skip": 3*10 },
    { "$limit": 10 }
]);

Turns out MongoDB is not optimized for that sort of query. As the page number keeps increasing (say 2,000), we will keep skipping more records (skipping 2,000*10 = 20,000 records). For each page, MongoDB scans the result set based on your filter and starts skipping the records (after skipping 20,000 records) to get to the records in your final result. The time taken for this process increases with the number of records skipped.📈

On a MongoDB collection containing 100,000 records, the time taken by offset pagination looks as below:

Time taken (in µs) vs Records skipped [Code]

🔍 Range-based Pagination:

If you look at what all the big-wigs are doing, they seem to be using cursors, and they seem to be not facing this performance issue at all. If you look deeply, they are more or less using Range-based pagination.
Slack API Twitter API Facebook API Disqus API

🤔 Cursor, what's that?

A cursor is nothing but an identifier to fetch your next set of results. The client should not care what it means. In offset-based pagination, the offset(or page number) can be called your cursor. Don't confuse it with MongoDB cursors.

An ideal cursor for Range-based pagination has to be:

Unique and
You are OK with sorting on it

Keep in mind that you also have to create an appropriate index on this cursor field for the magic to happen.😜
If you do not have such a field in your collection schema, you can still use range-based pagination (detailed at the end).

Range-based pagination is not some out-of-the-world algorithm that you need to rack your brains out. It is just a small twist on your normal thought process to take advantage of the database indexes. Let's take a deeper look.

👨🏻‍💻 Querying Pattern:

Let's take the cursor as id which is an actual field on your database. Send this to the client so that they can query for the next set of results. You can choose to hide what your cursor means by having an encryption layer before sending it to the client.

Querying first time:

db.client.aggregate([{
  "$limit": 10,
},
{
  "$sort": {
      "id": 1,
  }
}])

Query Nth time
Use the value of id from the last document in the previous result.

db.client.aggregate([
  {
    "$match": {
       "id":{
          "$gt": "<Last value of id>",
       }
    },
  },
  {
    "$sort": {
       "id": 1,
    }
  },
  {
    "$limit": 10,
  }
])

What makes range-based pagination more scalable is that the query pattern can be indexed, thus reducing the result set and skipping the latency-introducing $skip step.

🙈 What if you don't have the ideal cursor? Make the not-so-ideal one, I guess.

You almost always have the ideal cursor (i.e. a primary key) on your collection. For some reason, you can't use it. Maybe because you want to provide options to your client on what they can sort by.
In that case, you can make your cursor a compound field by using the ideal cursor field (say id field) as the secondary sort and primary sort on the client's requested sort (title field).
The query pattern becomes like below:

db.client.aggregate([
  {
     "$match": {
       "$or": [
         {
           "id": { "$gt": "<Last value of id>"},
           "title": { "$eq": "<Last value of title>"}
         },
         {
           "title": { "$gt": "<Last value of title>"}
         }
       ]
     }
  },
  {
     "$sort": {
       "title":1,
       "id":1,
     }
  }
])

Note: The promise of latency reduction here may change on a case-to-case basis when you are making cursor from multiple fields.

📊 Comparison with Offset-based pagination

Offset works best for the first few pages but time-taken increases as the offset gets bigger whereas range-based pagination with indexes keeps the time taken almost constant.

Time taken as we iterate through database [Code]

TLDR: Range-based pagination takes advantage of database indexes to keep the performance from deteriorating even when iterating through the complete collection 🏆.

It may not be for you if:

You need offset-based random access: With range-based pagination, we do not have a concept of pages and total count as we are always fetching next to the cursor. If random access is a strong design requirement for you, then range-based pagination may not be for you.
You have complex sorting and uniqueness requirements: You have a lot of fields you want to sort on and you do not wish to create an index on all of them (or if the indexing does not work as expected 😟).
You do not need far-away records: Your client audience does not really require you to get far-away records. Maybe you provide a search as a feature and thus only the first few pages are enough. Like when have we visited the 2nd page of Google Search Result? While this is no reason to not implement cursor-based pagination, you can say I do not really have to 🫣.

Full Stack Interview Help

Medam Mahesh — Fri, 03 Dec 2021 12:36:53 +0000

Hi all,

I have a Interview with Full Stack which requires NodeJS and ReactJs next weekend. I have worked with them for Freelance projects up until a year ago.

The interview has pair coding round it seems. Now, I want to know the best coding practices to used for let's say a MERN stack application. Any good open source projects for coding practices will be helpful.

Also a good resource for interview questions preparation will be very helpful.

Thanks in advance,

Discount Engine Opensource Alternative

Medam Mahesh — Wed, 17 Feb 2021 15:13:26 +0000

Hi,

I am looking to add a discount engine for Buy 2 Get 1 and similar discounts to my shopping cart calculator. I would like to know about any Opensource options available in this space.

This is a custom website I am building, not with a third party like Wordpress, Woocommerce etc.,.

Thanks in advance,

Making Shopify App installable

Medam Mahesh — Tue, 12 Jan 2021 09:25:58 +0000

Hi,

I want to make my Shopify Website installable. I do not intend to make it a complete PWA. Just having a 'Add To Home Screen' would suffice for me. I do not want to go much into cache and offline behaviour also.
I know how to make this for a self-developed website (i.e. Linking with manifest.json). I tried to edit the theme files and achieve the same result but it didn't work. I am getting the 'start_url' error.

P.S. I do not want to use any third party Shopify apps for this.

10 VS Code Keyboard Shortcuts for Productivity Boost

Medam Mahesh — Tue, 01 Dec 2020 12:35:10 +0000

Each and every one of us uses keyboard shortcuts to some extent like Cut (Ctrl\Cmd + X), Copy (Ctrl\Cmd + C) and Paste (Ctrl\Cmd + V). Please don't stop reading because I mentioned those three.

I promise you this blog will only discuss shortcuts a cut above the basics and only those which are beneficial to the devs.

Why Keyboard Shortcuts ?

Firstly, Keyboard Shortcuts are just efficient. Especially when it comes to coding IDE like VS Code and IntelliJ, keyboard shortcuts are a boon to coders. While it is not that easy to get used to these shortcuts, you will definitely thank yourself for putting the effort once you get used to them.

The Shortcuts

1. Multi-cursor

This is the first keyboard shortcut I mastered. And also the most helpful one especially when on HTML (adding classes used to be such a pain 😭 ).

How to use it : Just Ctrl click at all the places where you want to start writing. You will see multiple cursors popping up. And when you type, all of them start writing.
Keyboard Shortcut : Ctrl/Cmd + Mouse Left Click

2. Global Search

Some of you might already be familiar with how to do this by using mouse as it is a very very important feature when working on big projects with loads of interconnected code. But did you know that you could do that with keyboard shortcut in a snap.

How to use it : Now most of us are already familiar with Ctrl/Cmd + F for searching inside a file. Now taking it to the next level by pressing Ctrl/Cmd + Shift + F, you can do a global search on your entire project.
Keyboard Shortcut : Ctrl/Cmd + Shift + F

3. Moving By Words Not Letters

While we can move around the extremes of the line using Home and End, what if we want to quickly change attributes in HTML / edit arguments in function calls ? 🤔🤔

How to use it : We know that we can move the cursor by letter with Arrow Keys but if you want to move the cursor by words, you have to Ctrl/Cmd + [Left/Right Arrow Keys]
Keyboard Shortcut : Ctrl/Cmd + [Left/Right Arrow Keys]

4. Delete Words

Following the footsteps of previous shortcut, would it not make stuff easier if you could quick delete words as well 🤩🤩

How to use it : Now, this is a two step thing. Firstly, select the word where your cursor is with Ctrl/Cmd + D. Next, click Delete or start writing what you need in place of word selected.
Keyboard Shortcut : Select Word (Ctrl/Cmd + D) and then start typing away

5. Moving Between Open Files

Most of us know how to move between different applications on our machine (i.e. Alt + Tab). In a similar fashion when it comes to big projects, we have to move between some files very often. And using mouse to do this is not very productive.

How to use it : You can switch between open files this using using Ctrl/Cmd + Tab (right) and Ctrl/Cmd + Shift + Tab(left) to make this movement quick.
Keyboard Shortcut :
- Right Open File : Ctrl/Cmd + Tab
- Left Open File : Ctrl/Cmd + Shift + Tab

6. Quick Comment / Uncomment

This is the most often used keyboard shortcut while I code. The best part of it is that it is language agnostic i.e. automatically comments code in the proper syntax. Also commenting and uncommenting is also used as a quick debug technique. Commenting is a good coder's duty and to do it productively, here is a shortcut.

How to use it : Make a selection and Ctrl/Cmd + / for a multi-line comment. Or just press Ctrl/Cmd + / to comment the current line the cursor is on. To uncomment, repeat the process again.
Keyboard Shortcut : Ctrl/Cmd + /

7. Jump To Function

When you have a code project, it is very likely that you are using code from other files a lot. And ability to quickly move to different pieces of code is a major help.

How to use it : Just take the cursor to the function call and press F12 or Fn + F12 based on the keyboard. The file with the function is opened with your cursor at the definition.
Keyboard Shortcut : F12 or Fn + F12

8. Open Project Files (Not `Ctrl/Cmd + O`)

I am not going to discuss the normal Open File and you chose the file from the File Explorer. This is a shortcut to quickly open files from the project without using mouse.

How to use it : Press Ctrl/Cmd + P and start typing the file name. A dropdown with all possible files you might be referring to in the project shows up.
Keyboard Shortcut : Ctrl/Cmd + P

9. Code Completion / Auto Complete / Quick Import

Having a quick suggestion related to what you might be typing is always a big help. Let it be the variable name (or) function definition. You can also quick import from different files using these shortcuts

How to use it : Press Ctrl/Cmd + Space to see possible suggestions and use Ctrl/Cmd + Shift + Space to view the function definition where you are typing to get an understanding of its parameters.
Keyboard Shortcut : Ctrl/Cmd + Space or Ctrl/Cmd + Shift + Space

10. Expand / Collapse Functions

Even though we understand that we should break our code into files for ease of re-use and easy understanding. Sometimes we can not help but see that our files are growing in size. Traversing through large files is a very time-consuming task. Would it not be easier if you could collapse all the functions and only expand those that you need ? And that's what this shortcut has to offer.

How to use it : Press Ctrl/Cmd + K and then Ctrl/Cmd + 0 to collapse all sub regions. And Ctrl/Cmd + K and then Ctrl/Cmd + J to expand all sub regions. To collapse the current (where the cursor is) sub region, press Ctrl/Cmd + Shift + [ and to expand the current sub region, press Ctrl/Cmd + Shift + ]. Here sub region means a block of code.
Keyboard Shortcut :
- Expand All : Ctrl/Cmd + K and then Ctrl/Cmd + J
- Collapse All : Ctrl/Cmd + K and then Ctrl/Cmd + 0
- Collapse Current : Ctrl/Cmd + Shift + [
- Expand Current : Ctrl/Cmd + Shift + ]

Also VS Code Official Cheat Sheet : https://code.visualstudio.com/shortcuts/keyboard-shortcuts-windows.pdf
https://code.visualstudio.com/shortcuts/keyboard-shortcuts-macos.pdf
Or just use another keyboard shortcut, press Ctrl/Cmd + Shift + P to search other keyboard shortcuts

[Featured Image taken from https://code.visualstudio.com/]

How to store user data in Shopify App ?

Medam Mahesh — Sun, 30 Aug 2020 16:23:55 +0000

Hi,

I have recently started looking into shopify app development. I know how to build UI for Shopify App for now.

My App Idea uses an API which provides an api_key.

Now when a Shopify Customer installs my app, he enters this API key on the Frontend Form that I had created and tries to submit.

DOUBT

How and where to store this information ? How to retrieve this information once saved ?

I have been burning the midnight oil for this. So, if any experienced people are present. Please lend me a hand.
I am very much willing to clarify any doubt in my explanation.

Thanks in advance,

How to protect your JS ?

Medam Mahesh — Mon, 13 Jul 2020 16:20:51 +0000

Hi,

I am looking for a way to secure my JS code.

Edit: I do not want my code to be run by others in their local by cloning my API behaviour. Secondly, if Obfuscation is OK, is there any best practices other than going to an online obfuscator

I know there is Obfuscating JS. But, I do not know how safe it is.

It'd be great if someone can answer these for me.

Is Obfuscation safe ?
How to properly Obfuscate your JS ?
Are there any techniques to secure JS apart from Obfuscate ?

How to get started with Spring Boot ?

Medam Mahesh — Sun, 12 Jul 2020 06:36:44 +0000

Hi,

I am a fresher at a company where they work on Java based frameworks like SpringBoot.
I am an experienced developer in MERN stack and Flask based applications. But, I never worked with something related to Java. Are there any go to places to learn SpringBoot ? As I am fairly new to Java as well, I think a more ground up learning will be better for me.
It'd be great if you can refer something from your experience.
Thanks in advance 😊,

How to make a API Call to Heroku from Chrome Extension

Medam Mahesh — Mon, 29 Jun 2020 14:35:19 +0000

Hi Dev,

So, I am stuck here trying to make this API Call to a Heroku project from a Chrome Extension I am developing.

I am familiar with Flask and NodeJs.
I understand this has something to do with headers somewhere.
I tried searching but could not find a point solution.

Help me solve this problem.

Thanks in advance,

Which SaaS is your favourite ?

Medam Mahesh — Mon, 22 Jun 2020 17:37:31 +0000

Hello fellow Dev,

Hope you are having a fun day going through exciting Git Repos, working on your side project and lastly reading fun DEV.to posts.

Even if you do not want to tell me about it, I know you are involved in a SaaS activity :P.

Here is a little fun activity,

`Score that SaaS`

Does it save your ass (Productivity) - 1 points
My day does not end without it (Part of Compulsory Routine) - 10 points
It's my livelihood (Work in/related to SaaS) - 100 points
Add 1000 more for any other use you feel like.

I think every piece of information you (the reader) contributes will go a long way for others. So, I urge you not to hesitate.

Why am I asking this ?

I feel everyone has right to have the knowledge that can help them. I am planning on laying out a series discussing how a particular SaaS can help lift someone's daily burdens or open a new avenue for them.

AwesomeCSS #1 | Apply Image Filters Just Using CSS

Medam Mahesh — Mon, 22 Jun 2020 16:24:01 +0000

This was something awesome that I had stumbled upon last week.

I am not much of a social butterfly but I do know that there is always a lot of buzz about filters going on in the social media. Coincidentally, the technique of this also revolves around a CSS property called filter.

Incase you are interested in watching video :

Interesting Finds

CSS `filter` property

The filter property defines visual effects (like blur and saturation) to an element (often ).
~ W3 Schools Link

CSS `mix-blend-mode` property

The mix-blend-mode property specifies how an element's content should blend with its direct parent background.
~ W3 Schools Link

Checkout the Codepen :

Thanks for reading till the end,

[Please have a look at my YouTube Channel. I post interesting web-related videos on it : https://www.youtube.com/channel/UCvf5cqRQk6aOmFZrgoU6K-Q]

Intro Post

Medam Mahesh — Sun, 21 Jun 2020 15:36:40 +0000

Well, this is a little embarrassing to do because I have been on DEV.to for the past 6 months or so. But, I feel it is necessary to let others in the community know about me and my experience.

The Intro

I am Medam Mahesh.
My coding journey started 4 years ago.
I am good at Frontend and Backend Development, Machine Learning, Artificial Intelligence and owing to my past experience
I love to chat about literally anything and everything new as long as it is related to coding.

My Experience with DEV

It has been very awesome. The community is active, great and sensible to newcomers. You do not see a lot of hate speech going on which is great.

Thanks for being such a great community, DEV.

DEV Community: Medam Mahesh

Optimizing Pagination: Range-Based Queries with Cursors and Happy Databases 🚀

👋 Introduction:

🔍 Range-based Pagination:

🤔 Cursor, what's that?

👨🏻‍💻 Querying Pattern:

🙈 What if you don't have the ideal cursor? Make the not-so-ideal one, I guess.

📊 Comparison with Offset-based pagination

It may not be for you if:

Full Stack Interview Help

Discount Engine Opensource Alternative

Making Shopify App installable

10 VS Code Keyboard Shortcuts for Productivity Boost

Why Keyboard Shortcuts ?

The Shortcuts

1. Multi-cursor

2. Global Search

3. Moving By Words Not Letters

4. Delete Words

5. Moving Between Open Files

6. Quick Comment / Uncomment

7. Jump To Function

8. Open Project Files (Not Ctrl/Cmd + O)

9. Code Completion / Auto Complete / Quick Import

10. Expand / Collapse Functions

How to store user data in Shopify App ?

DOUBT

How to protect your JS ?

How to get started with Spring Boot ?

How to make a API Call to Heroku from Chrome Extension

Which SaaS is your favourite ?

Here is a little fun activity,

Score that SaaS

Why am I asking this ?

AwesomeCSS #1 | Apply Image Filters Just Using CSS

Interesting Finds

CSS filter property

CSS mix-blend-mode property

Intro Post

The Intro

My Experience with DEV

8. Open Project Files (Not `Ctrl/Cmd + O`)

`Score that SaaS`

CSS `filter` property

CSS `mix-blend-mode` property