DEV Community: Shankar

Scaling a Node.js Application with Distributed Caching: Part 3 - Hazelcast

Shankar — Sat, 01 Jul 2023 02:30:00 +0000

In the second part of this series, we explored how to scale a Node.js application using distributed caching with Memcached. Now, let's take a look at another popular caching solution - Hazelcast.

Introduction to Hazelcast

Hazelcast is an open-source, in-memory data grid based on Java. It provides distributed data structures and computation services for your applications, allowing you to share and partition your data across a cluster of machines.

Why Choose Hazelcast?

Hazelcast offers a set of distributed data structures and services that are highly available and extremely scalable. It's a great choice for applications that require a high volume of reads and writes, as well as those that need to maintain state across multiple servers.

Setting Up Hazelcast

To use Hazelcast in a Node.js application, you need to install the hazelcast-client package. You can do this using the following command:

npm install hazelcast-client

After installing the hazelcast-client package, you can use it in your application like this:

const { Client } = require('hazelcast-client');

// Create a Hazelcast client
const client = await Client.newHazelcastClient();

// Get a distributed map
const map = await client.getMap('my-distributed-map');

// Put and get a value from the map
await map.put('key', 'value');
const value = await map.get('key');
console.log(value); // 'value'

In the above code, we first require the hazelcast-client package and then create a new Hazelcast client. We then get a distributed map and put a key-value pair in the map. After that, we retrieve the value of the key from the map.

Scaling with Hazelcast

When scaling a Node.js application with Hazelcast, you can use multiple Hazelcast instances. The Hazelcast client will automatically distribute the data among the instances.

Conclusion

In this part, we explored how Hazelcast can be a powerful tool for scaling a Node.js application. It allows you to distribute data across multiple machines, which can significantly speed up your application. In the next part of this series, we will compare the performance of Redis, Memcached, and Hazelcast to help you choose the best tool for your needs.

Scaling a Node.js Application with Distributed Caching: Part 2 - Memcached

Shankar — Fri, 30 Jun 2023 14:30:00 +0000

In the first part of this series, we explored how to scale a Node.js application using distributed caching with Redis. Now, let's take a look at another popular caching solution - Memcached.

Introduction to Memcached

Memcached is an open-source, high-performance, distributed memory object caching system. It is an in-memory key-value store for small chunks of arbitrary data (strings, objects) from results of database calls, API calls, or page rendering.

Why Choose Memcached?

Memcached's simple design promotes quick deployment, ease of development, and solves many problems facing large data caches. Its API is available for most popular languages, making it a versatile choice for developers.

Setting Up Memcached

First, you need to install Memcached on your machine. You can do this using the following commands:

# For Ubuntu
sudo apt-get install memcached

# For MacOS
brew install memcached

Integrating Memcached in Node.js

To use Memcached in a Node.js application, you need to install the memcached package. You can do this using the following command:

npm install memcached

After installing the memcached package, you can use it in your application like this:

const Memcached = require('memcached');
const memcached = new Memcached('localhost:11211');

memcached.set('key', 'value', 10, function (err) { /* handle error */ });

memcached.get('key', function (err, data) {
  console.log(data);
});

In the above code, we first require the memcached package and then create a new Memcached instance. We then set a key-value pair in Memcached with a lifetime of 10 seconds. After that, we retrieve the value of the key from Memcached.

Scaling with Memcached

When scaling a Node.js application with Memcached, you can use multiple Memcached servers. The memcached package allows you to pass an array of server locations when creating a new Memcached instance:

const memcached = new Memcached(['localhost:11211', 'localhost:11212']);

In the above code, we create a new Memcached instance with two Memcached servers. The memcached package will automatically distribute the keys among the servers.

Implementing Memcached

Let's dive into the implementation of distributed caching in a Node.js application using Memcached. Like Part1, we'll use an example of caching the results of a database query.

const Memcached = require('memcached');
const memcached = new Memcached('localhost:11211');

// Example route handler in an Express.js application
app.get('/users/:id', async (req, res) => {
  const userId = req.params.id;
  const cacheKey = `user:${userId}`;

  // Check if the data exists in the cache
  memcached.get(cacheKey, async (err, cachedData) => {
    if (cachedData) {
      // Data found in the cache, return it
      const user = JSON.parse(cachedData);
      res.json(user);
    } else {
      // Fetch data from the database
      const user = await fetchUserFromDatabase(userId);

      // Store the data in the cache with an expiration time (e.g., 1 hour)
      memcached.set(cacheKey, JSON.stringify(user), 3600, function (err) { /* handle error */ });

      // Return the data to the client
      res.json(user);
    }
  });
});

// Function to fetch user data from the database
async function fetchUserFromDatabase(userId) {
  // Code to query the database and retrieve user data
  // ...

  // Simulating a delay in fetching data from the database
  await new Promise((resolve) => setTimeout(resolve, 1000));

  // Return the fetched user data
  const user = { id: userId, name: 'John Doe', email: 'john@example.com' };
  return user;
}

Conclusion

In this part, we explored how Memcached can be a powerful tool for scaling a Node.js application. It allows you to cache data in memory, which can significantly speed up your application. In the next part of this series, we will compare the performance of Redis and Memcached to help you choose the best tool for your needs.

Scaling a Node.js Application with Distributed Caching: Part 1 - Redis

Shankar — Fri, 30 Jun 2023 11:47:46 +0000

Introduction:

Scalability is a crucial aspect of modern web applications, and as the user base grows, ensuring optimal performance becomes paramount. Distributed caching is a powerful technique that can significantly enhance the scalability of a Node.js application. In this article, we will explore how to implement distributed caching to scale a Node.js application, improving response times and reducing the load on your backend servers.

Why Distributed Caching?

Distributed caching involves storing frequently accessed data in a cache that is distributed across multiple nodes or servers. This allows for faster retrieval of data, reducing the need to query databases or perform expensive computations. By caching frequently accessed data, we can alleviate the load on our backend servers and achieve better scalability.

Choosing a Distributed Cache:

There are several popular distributed caching systems available, such as Redis, Memcached, and Hazelcast. In this example, we will focus on using Redis, a widely adopted in-memory data store with excellent support for caching.

Setting Up Redis:

To utilize distributed caching in your Node.js application, you need to set up Redis, a popular in-memory data store that excels at caching. Follow these steps to get Redis up and running:

Installation: Start by installing Redis on your server or use a managed Redis service. Redis provides clear installation instructions for various operating systems on their official website. Choose the installation method that suits your environment.
Configuration: Once Redis is installed, it's essential to configure it properly. Redis uses a configuration file (redis.conf) to manage settings such as port number, memory limits, and persistence options. The default configurations are often suitable for development, but for production environments, consider optimizing these settings based on your application's needs.
Starting the Server: After installation and configuration, start the Redis server. If you installed Redis on your local machine, you can typically start it using the command redis-server. If you're using a managed Redis service, follow the instructions provided by your service provider to start the server.
Connecting from Node.js: To connect your Node.js application to Redis, you need a Redis client library. There are several options available, including popular libraries like "ioredis" and "node-redis". Install the library of your choice using npm or yarn, and then import it into your Node.js application.
Establishing the Connection: Create a Redis client instance by instantiating the Redis client from the chosen library. Specify the appropriate configuration options, such as the host and port of your Redis server. For example, with "ioredis":

const Redis = require('ioredis');
const client = new Redis({
  host: 'localhost',
  port: 6379,
});

Testing the Connection: To ensure the connection to Redis is established successfully, you can perform a simple test by issuing a Redis command, such as PING. For example:

client.ping((err, result) => {
  if (err) {
    console.error('Error connecting to Redis:', err);
  } else {
    console.log('Redis connection established successfully');
  }
});

If the connection is successful, you should see the "Redis connection established successfully" message in the console.

Setting up Redis is a crucial step in implementing distributed caching in your Node.js application. By properly installing, configuring, and establishing the connection to Redis, you create the foundation for leveraging the caching capabilities of Redis to enhance the performance and scalability of your application.

Implementing Distributed Caching in Node.js:

Let's dive into the implementation of distributed caching in a Node.js application using Redis. We'll use an example of caching the results of a database query.

// Install the Redis client library using: npm install redis

const redis = require('redis');
const client = redis.createClient();

// Example route handler in an Express.js application
app.get('/users/:id', async (req, res) => {
  const userId = req.params.id;
  const cacheKey = `user:${userId}`;

  // Check if the data exists in the cache
  client.get(cacheKey, async (err, cachedData) => {
    if (cachedData) {
      // Data found in the cache, return it
      const user = JSON.parse(cachedData);
      res.json(user);
    } else {
      // Fetch data from the database
      const user = await fetchUserFromDatabase(userId);

      // Store the data in the cache with an expiration time (e.g., 1 hour)
      client.setex(cacheKey, 3600, JSON.stringify(user));

      // Return the data to the client
      res.json(user);
    }
  });
});

// Function to fetch user data from the database
async function fetchUserFromDatabase(userId) {
  // Code to query the database and retrieve user data
  // ...

  // Simulating a delay in fetching data from the database
  await new Promise((resolve) => setTimeout(resolve, 1000));

  // Return the fetched user data
  const user = { id: userId, name: 'John Doe', email: 'john@example.com' };
  return user;
}

In this example, we have a route handler for retrieving user data from a Node.js application. It first checks if the user data exists in the Redis cache by using the get function. If the data is found in the cache, it is returned to the client immediately. Otherwise, it fetches the data from the database, stores it in the cache using the setex function with an expiration time of 1 hour (3600 seconds), and then returns the data to the client.

Benefits and Considerations:

Distributed caching offers numerous benefits for scaling a Node.js application:

Reduced response times:

By caching frequently accessed data, response times can be significantly improved, resulting in a better user experience.

Lower database load:

Caching helps reduce the number of database queries, offloading the backend servers and improving overall performance.

Scalability:

By reducing the load on the backend servers, distributed caching enables easier horizontal scaling of your application.

However, it's essential to consider cache invalidation strategies, memory management, and the potential trade-offs between caching and real-time data consistency.

Conclusion:

Distributed caching with tools like Redis can greatly enhance the scalability and performance of a Node.js application. By implementing caching strategies and leveraging the power of in-memory data stores, you can reduce response times, lower backend server load, and achieve better scalability as your application grows. Start exploring distributed caching in your Node.js applications today to unlock the benefits of improved performance and scalability.

Hasura x MEWS

Shankar — Sat, 24 Jun 2023 17:21:24 +0000

Hasura gives you instant GraphQL / Rest API on top of SQL databases like Postgres and MySQL. You can also use Hasura Remote Joins to join data from your Postgres database with MEWS API in GraphQL.

Step 1: Setup Hasura Instance

Signup to Hasura Cloud
Create a free tier project

Step 2: Configure & Deploy Custom Resolver

I've created a project in Glitch so you can remix and spin up your own project.
MEWS has a demo environment which you can use to test the integration. Get the environment's credentials here
Now head over to .env file in Glitch and add these
- MEWS_BASE_URL
- MEWS_CLIENT_TOKEN
- MEWS_ACCESS_TOKEN
- MEWS_CLIENT
Click on Share at the top of the Glitch UI and copy the 'Live Site' URL.

Step 3: Add Remote Schema

In Hasura Console, go to the Remote Schemas tab.

Step 4: Start querying reservations

query getReservationsFromMEWS($startUTC: String!, $endUTC: String!) {
  getReservations(startUTC: $startUTC, endUTC: $endUTC) {
    Reservations {
      Id
      CustomerId
      AssignedResourceId
      Customer {
        Id
        FirstName
      }
      Room {
        Id
        State
      }
    }
  }
}

Add variables to the query

{
  "startUTC": "2023-06-01T00:00:00Z",
  "endUTC": "2023-06-02T00:00:00Z"
}

Voila!