Server-Side Pagination with Express.js and MongoDB

Introduction

When working with large datasets, various techniques are employed to optimize queries, enhance user experience, and improve performance. One of those techniques is Pagination. Pagination involves dividing large datasets into smaller, manageable subsets.

Pagination can be implemented in two ways:

• Client-Side Pagination: In client-side pagination, the client application queries the entire dataset from the server, and pagination logic is handled in the browser. Client-side pagination is easy to implement for small datasets. Client-side pagination also prevents additional queries to the database.
• Server-Side Pagination: In server-side pagination, the server returns a subset of the data at a time when queried. Server-side pagination reduces data load on the client and improves performance for large datasets by only fetching necessary data.

In this beginner-friendly tutorial, you’ll implement server-side pagination in Express.js and MongoDB. You’ll create a collection for storing movies in MongoDB and an API endpoint that retrieves data from a MongoDB collection. We’ll cover two methods in this tutorial:

1. Basic pagination implemented without the use of an external library.
2. Pagination using the mongoose-paginate-v2 library.

Prerequisites

To follow along with this tutorial, you'll need:

• A code editor like Visual Studio Code or any code editor of your choice.
• An API client such as Postman for testing your API endpoints.
• Node.js installed on your computer.
• Git is installed on your computer.
• A MongoDB instance either locally or on the web using MongoDB Atlas.
• Basic knowledge of Express.js.

Boilerplate Setup

To keep the article concise, I have created a boilerplate code that can be accessed here on GitHub. The boilerplate contains the basic setup for an Express.js application.

To get started with the tutorial:

1. Clone the repository: Firstly, clone the repository from GitHub using the following command:
   

   git clone https://github.com/michaelikoko/Express-Pagination.git
   

2. Navigate to the project directory: The starter directory contains the boilerplate code. Navigate to the starter using the following command:
   

   cd Express-Pagination/starter
   

   The starter directory should have the following folder structure:
   

3. Install dependencies: Install the needed node dependencies, using any package manager you choose. For NPM run the following command:
   

   npm install
   

4. Connect to MongoDB: Ensure you have a MongoDB instance running locally or on MongoDB Atlas. Create a .env file in the current directory, and provide the application port and database, as shown below:
   

   PORT=5090
   MONGODB_URI=mongodb+srv://<username>:<password>@cluster1.menjafx.mongodb.net/?retryWrites=true&w=majority
   

5. Create the Movie Schema: In the file models/Movie.js, define the Movie Schema:
   

   const mongoose = require("mongoose");
   const movieSchema = new mongoose.Schema(
     {
       title: {
         type: String,
         required: true,
         minLength: [2, "Movie title length is too short"],
         maxLength: [100, "Movie title length is too long"],
         trim: true,
       },
       director: {
         type: String,
         required: true,
       },
       genre: {
         type: String,
         required: true,
       },
       releaseYear: {
         type: Date,
       },
     },
     { timestamps: true }
   );
   
   module.exports = mongoose.model("Movie", movieSchema);
   

6. Populate the database: To make testing easier, populate the database with mock data contained in MOCK_DATA.json by running the script in the file populate.js:
   

   npm run populate
   

7. Run the development server: To start the development server, input the following command in the terminal:
   

   npm run server
   

Basic Pagination

In this section, we’ll implement basic pagination in Express.js and MongoDB without the use of any external library. We will create an API endpoint that retrieves a subset of movie records from the database based on the values of the specified query parameters. The routes have already been set up in the boilerplate code, so we’ll focus on the controller function. We will be working on the getMoviesCustom function, in the controllers/movies.js file.

We’ll use Movie.find cursor to fetch movies and the skip and limit methods for pagination.

• skip: The skip method controls where MongoDB starts returning records. The skip method has only one parameter offset, which is the number of records to be skipped.
• limit: The limit method defines the maximum amount of documents to be returned by MongoDB.

Two query parameters are required for basic pagination:

• page: This parameter specifies the page to be returned. The page parameter defaults to 1 if not specified. To extract the page query parameter input the following in the getMoviesCustom function:
  

  const page = parseInt(req.query.page, 10) || 1;
  

• limit: This parameter specifies the number of movies to return per page. The limit parameter defaults to 10. To extract the limit query parameter input the following in the getMoviesCustom function:
  

  const limit = parseInt(req.query.limit, 10) || 10;
  

We need to calculate the value of the offset variable. As explained earlier, the offset variable will be passed as the parameter to the skip method. It is calculated based on the requested page number and the number of documents per page. To do this, add the following line in the getMoviesCustom function:

  const offset = (page - 1) * limit;

Let’s assume we want 10 movies to be returned per page. Therefore, the limit variable will have a value equal to 10. The value for the offset variable will be calculated as follows:

• For the first page, the page parameter has a value of 1. Therefore offset will have a value equal to 0. This means zero documents are skipped at the beginning.
• For the second page, the page parameter has a value of 2. Therefore offset will have a value equal to the value of the limit, which is 10. This means the first 10 documents are skipped.
• For the third page, the page parameter has a value of 3. Therefore offset will have a value equal to the value of 2*limit, which is 20. This means the first 20 documents are skipped.

Similar logic applies to subsequent pages.

To retrieve a subset of movie documents based on the calculated offset values and limit, add the following line to the getMoviesCustom function:

  const movies = await Movie.find().skip(offset).limit(limit).exec();

In our API response, we also want to return the following pieces of information:

• totalItems: The total number of documents in the Movie collection. This is done using the countDocuments method. In the getMoviesCustom function, add the following line;
  

  const totalItems = await Movie.countDocuments({});
  

• totalPages: The total number of pages. This is done by dividing the total of documents (totalItems), by the number of documents per page (limit), and rounding off the nearest whole number. To calculate the total number of pages, in the getMoviesCustom function, add the following line:
  

  const totalPages = Math.ceil(totalItems / limit);
  

Putting it all together, the getMoviesCustom controller function should look like this:

async function getMoviesCustom(req, res) {
  const page = parseInt(req.query.page, 10) || 1;
  const limit = parseInt(req.query.limit, 10) || 10;
  const offset = (page - 1) * limit;

  const movies = await Movie.find().skip(offset).limit(limit).exec();
  const totalItems = await Movie.countDocuments({});
  const totalPages = Math.ceil(totalItems / limit);

  return res.status(200).json({ totalItems, page, totalPages, movies });
}

Using the mongoose-paginate-v2 Library

In this section, we’ll implement pagination using the mongoose-paginate-v2 library. An API route has already been created in the boilerplate code, so we’ll focus on the getMoviesLibrary controller function.

The mongoose-paginate-v2 is a pagination library for Mongoose. According to the docs:

The main usage of the plugin is you can alter the return value keys directly in the query itself so that you don't need any extra code for transformation.

To use mongoose-paginate-v2, we need to add the plugin to the schema and make use of the model paginate method. In the model/Movie.js file, make the following adjustments to the schema:

const mongoose = require("mongoose");
const mongoosePaginate = require("mongoose-paginate-v2");

const movieSchema = new mongoose.Schema(
  ...
);

movieSchema.plugin(mongoosePaginate);
module.exports = mongoose.model("Movie", movieSchema);

In the getMoviesLibrary controller function in controllers/movies.js, input the following code:

function getMoviesLibrary(req, res) {
  const { page, limit } = req.query;
  const options = {
    page: parseInt(page, 10),
    limit: parseInt(limit, 10),
  const movies = Movie.paginate({}, options).then((result) => {
    return res.status(200).json({
      totalItems: result.totalDocs,
      currentPage: result.page,
      totalPages: result.totalPages,
      movies: result.docs,
    });
  });
}  
};

The Model paginate method returns a promise, and has three parameters:

1. query: The query parameter is an object that states the condition for filtering documents. In our case, since we are returning all records, we pass an empty object {}.

2. options: The options parameter is an object that contains various properties that control how the data is paginated. Some of the object properties include:

   • page: The current page number. It automatically defaults to 1 if no value is provided.
   • limit: The number of documents per page. It automatically defaults to 10 if no value is provided. You can find a list of all the options object's properties in the documentation.

3. callback(err, result): The parameter is an optional function that gets executed when the pagination results are returned or there is an error in the query.

The Model paginate method returns an object after the promise is fulfilled. The properties of the returned object provide information about the paginated result. Some of the properties of the object include:

• docs: An array of documents for the specified page that matches the query.
• totalDocs: The total number of documents in the collection that match the query.
• page: The current page number.
• totalPages: The total number of pages based on the total documents and the limit.

Visit the documentation for the full list of the object properties.

The mongoose-paginate-v2 library provides the helper class PaginationParameters. The PaginationParameters class enables passing the entire request query parameter to the paginate method. This abstraction eliminates the need to declare the parameters individually in the controller function. To use the helper class, replace the content of getMoviesLibrary with the following:

function getMoviesLibrary(req, res) {
  Movie.paginate(...new PaginationParameters(req).get()).then((result) => {
    return res.status(200).json({
      totalItems: result.totalDocs,
      currentPage: result.page,
      totalPages: result.totalPages,
      movies: result.docs,
    });
  });
}

Testing the API

In this section, we’ll use Postman (or any other API client) to test the behavior of the endpoints. The two API endpoints created in the tutorial are:

• /api/v1/movies/custom: This endpoint is routed to the getMoviesCustom controller, which contains the logic for our basic implementation of pagination.
• /api/v1/movies/library: This endpoint is routed to the getMoviesLibrary controller, which contains the logic for our implementation of pagination using the mongoose-paginate-v2 library.

Any request made to both endpoints with the same query parameters should give the same results.

Let’s make the following requests:

1. Make a GET request to /api/v1/movies/customand /api/v1/movies/library. The page and limit parameters were not specified, so they should revert to their default values. The page parameter would have a value of 1, the limit parameter would have a value of 10, and a total of 3 pages. The result should look like this:
2. Make a GET request to /api/v1/movies/custom?page=2&limit=4 and /api/v1/movies/library?page=2&limit=4. The limit parameter has a value of 4 and the page parameter has a value of 2. The request should result in the API endpoint returning 4 documents. The response should indicate that the current page is 2 out of a total of 8 pages. (After populating the database with MOCK_DATA.json, there should be a total of 30 records). The result should look like this:

Conclusion

You now have a basic understanding of how to implement server-side pagination using Express.js and MongoDB. You created two endpoints, one for basic pagination and the other for pagination using the mongoose-paginate-v2 library.

You can explore additional query features to enhance your application, such as filtering and sorting. For further reading, check out the following resources:

• mongoose-paginate-v2 Documentation.
• Mongoose Documentation.
• MongoDB Documentation.
• Express.js Documentaion.