DEV Community: Slava

How to collect metrics from node.js applications in PM2 with exporting to Prometheus

Slava — Mon, 12 Feb 2024 21:33:58 +0000

It’s no secret that for stable and reliable working of node.js applications, it’s necessary to monitor their performance and get useful insights from their metrics. It means being able to collect information about app state before issues appear to preventing failures.

In this article, I’d like to present a method for collecting metrics node.js applications from running in PM2 and exporting this data to Prometheus.

When you simply start a node.js application using the node command or through PM2 with the pm2 start app.js command the new application is launched as a single instance. Collecting and exporting any metrics in this case would not be difficult. To do this, you can install the prom-client package and add the metrics you need, for example, the number of requests to our application.



import client from 'prom-client';
import { createServer } from 'http';

const registry = new client.Registry();
const PREFIX = `nodejs_app_`;

export const metricRequestsTotal = new client.Counter({
    name: `${PREFIX}request_counter`,
    help: 'Show total request count',
    registers: [registry],
});

// Start your application
// ...
nodejsapp.get('/*', async (req, res) => {
    metricRequestsTotal.inc();
});

// Start Prom server for export metrics
const promServer = createServer(async (req, res) => {
    res.setHeader('Content-Type', registry.contentType);
    res.end(await registry.metrics());
    return;
});

promServer.listen(9100, () =>
    console.log('Prom server started')
);

In this example, when your application is launched, it also stars metric exporting on an additional port 9100. Of course you can use a separate URL with the main application, but it shouldn’t be public for users.

This approach will work as long as you don’t need to run multiple instances of the application in cluster mode. For collecting metrics in cluster mode, prom-client provides a good example with metric aggregation. But what if you’re running the application through the process manager PM2, where you don’t have access to the running cluster?

In one of my previous articles, I talked about the pm2-prom-module, which allows collecting overall statistics for PM2 applications but couldn’t collect internal statistics from each application. However, starting from version 2.0, such limitation is no longer present, and all statistics from PM2 and within the application are provided through a single module.

o communicate between node.js application and pm2-prom-module, you need to install the npm package pm2-prom-module-client in your application. As a result, the example above will look like this:



import client from 'prom-client';
import { initMetrics } from 'pm2-prom-module-client';

const registry = new client.Registry();
const PREFIX = `nodejs_app_`;

const metricRequestCounter = new client.Counter({
    name: `${PREFIX}request_counter`,
    help: 'Show total request count',
    registers: [registry],
});

// Register your `registry` to send data to pm2-prom-module
initMetrics(registry);

// ...
nodejsapp.get('/*', async (req, res) => {
    // ...
    metricRequestCounter?.inc();
    // ...
});

Now pm2-prom-module will provide PM2 statistics and also internal statistics from your applications.



/// Common metrics from PM2

# HELP pm2_free_memory Show available host free memory
# TYPE pm2_free_memory gauge
pm2_free_memory{serviceName="my-app"} 377147392

# HELP pm2_cpu_count Show available CPUs count
# TYPE pm2_cpu_count gauge
pm2_cpu_count{serviceName="my-app"} 4

# HELP pm2_available_apps Show available apps to monitor
# TYPE pm2_available_apps gauge
pm2_available_apps{serviceName="my-app"} 2

/// Apps metrics
# HELP nodejs_app_request_counter Show total request count
# TYPE nodejs_app_request_counter counter
nodejs_app_request_counter{app="app1",serviceName="my-app"} 10
nodejs_app_request_counter{app="app2",serviceName="my-app"} 17

By default, all statistics are aggregated across all running instances in every app, but if you want detailed statistics for each instance, you can enable this through the module’s configuration parameter:



pm2 set pm2-prom-module:aggregate_app_metrics false

This module, unlike some others, allows collecting metrics separately for each application, and they do not intersect.

Overall, the statistics provided by PM2 are enough for basic application monitoring. However, in our case, for example, we wanted more detailed information about page rendering time or loading of certain data blocks. For such purposes, we used Histogram metric type.



// ...
export const metricRequestTime = new client.Histogram({
    name: 'nodejs_app_page_execute',
    help: 'Time to processing request',
    registers: [registry],
    buckets: [0.1, 0.2, 0.3, 0.5, 0.7, 1, 2, 3, 5],
    labelNames: ['code', 'page'],
});

// ...
const endMetricRequestTime = metricRequestTime.startTimer();
// ...
endMetricRequestTime({ code: 200, page: 'Homepage });

In example above, we use 2 additional parameters: code (HTTP response code) and page (page identifier), which provide detailed statistics. For example, you can build such graphs in Grafana:

Most frequently pages sum(rate(nodejs_app_page_execute_count{serviceName=”$serviceName”,code=”200"}[5m])) by (page)
The number of HTTP (response codes) success or errors. sum(rate(nodejs_app_page_execute_count{serviceName=”$serviceName”}[5m])) by (code)
The page load time specifying, for example, the 99th percentile. histogram_quantile(0.99, sum(rate(nodejs_app_page_execute_bucket{serviceName=”$serviceName”, code=”200"}[1m])) by (page, le))
And many other graphs for each individual page.

By the way, I don’t recommend using the URL of the page as the value for page — it will significantly increase memory usage and the size of the response in Prometheus, and mess in the graphs. It’s better to use page identifier.

So, to summarize, we have a single point for collecting and serving statistics for all applications running in PM2, and a monitoring system built on Prometheus with functional dashboards in Grafana.

If you like pm2-prom-module, I would appreciate if you could star it on Github. Thanks for reading.

Auto scaling Node.js applications with PM2 and pm2-autoscale module

Slava — Mon, 04 Dec 2023 20:51:03 +0000

In this article, i would like to tell you how you can easily automatically scale your Node.js applications to handle increased traffic loads using PM2 and keep your server resources under control.

PM2 Cluster Mode

PM2 is a process manager for Node.js applications that allows users to easily manage their Node.js applications and keep them running smoothly. With cluster mode PM2 allows networked Node.js applications to be scaled across all CPUs available, without any code modifications.

For example, if you have server with 8 CPUs you can run 8 instances of your application and increase the performance and reliability of your application.

To run application in cluster mode you can easily create ecosystem configuration file and start you application.

module.exports = {
    apps: [
        {
            name: 'app',
            script: 'build/app.js',
            instances: '8',
            autorestart: true,
            watch: false,
            max_memory_restart: '512M',
            vizion: false,
            exec_mode: 'cluster',
        },
    ],
};

This configuration allows you run 8 workers on your server. And for most cases it would be enough (especially if you have only one application on your server).

But imagine you want to add one more application to your server. Of course you want split resources between your applications. Now you change configuration file for every app and set 4 workers for every application. It will work until any of your applications can handle increased traffic load. If one of the application uses 100% CPU of every worker you should add more workers. In that case you should connect to your server and use pm2 scale command to scale more workers for loaded app and decrease workers for another app.

pm2 scale app +3 # Add 3 new additional instances for your app

When application load has been decreased, you also should connect to the server and revert your changes back. Not comfortable and easy solution. Plus you have to have solution to monitor you applications to detect when you need scale one of your apps.

Another case, for example, you have server with 48 CPUs and you want to run multiple apps on it. If you run every application with instances=max PM2 will run 48 instances and imagine that every instance uses approximately 100Mb of RAM (~5GB for all instances). So if you have 10 application it means you will use about 50GB of the server memory without any server load. You are using your server resources ineffectively.

Unfortunately PM2 does not have any good and simple solutions how to dynamically increase workers and monitor your applications. And with free version you can see CPU utilization and Memory usage only in terminal.

PM2 Autoscale Module

To solve this issue i wrote plugin pm2-autoscale that helps to optimize application’s performance by automatically adjusting the number of instances running based on the CPU utilization of every application.

To use pm2-autoscale module your should install it with command:

pm2 install pm2-autoscale

Module supports few configuration options

scale_cpu_threshold Maximum value of CPU utilization one of application instances when the module will try to increase application instances. (default to 30)
release_cpu_threshold Average value of all CPUs utilization of the application when the module will decrease application instances (default to 5)
debug Enable debug mode to show logs from the module (default to false)

To modify the module config values you can use the following commands:

pm2 set pm2-autoscale:debug true
pm2 set pm2-autoscale:scale_cpu_threshold 50

Now you can modify your ecosystem configuration to run any application with minimum required instances. For example, if you have 8 CPUs and 2 applications, you can set instances=2 for every application and keep available resources of your server. When module detects that CPU utilisation is higher then scale_cpu_threshold it will start increasing instances to max CPUs-1 only if server has available free memory. When module detects CPU utilization is decreasing it will stop useless instances.

See example how it works. I tested it with Apache Benchmark.

0|pm2-autoscale | App "app" has 1 worker(s). CPU: 0.
0|pm2-autoscale | App "app" has 1 worker(s). CPU: 17.
0|pm2-autoscale | App "app" has 1 worker(s). CPU: 29.
0|pm2-autoscale | INFO: Increase workers
0|pm2-autoscale | App "app" scaled with +1 worker
0|pm2-autoscale | App "app" has 2 worker(s). CPU: 26,34.
0|pm2-autoscale | App "app" has 2 worker(s). CPU: 29,43.
0|pm2-autoscale | App "app" has 2 worker(s). CPU: 28,38.
0|pm2-autoscale | INFO: Increase workers
0|pm2-autoscale | INFO: App "app" is busy
0|pm2-autoscale | App "app" scaled with +1 worker
0|pm2-autoscale | App "app" has 3 worker(s). CPU: 27,35,33.
0|pm2-autoscale | App "app" has 3 worker(s). CPU: 25,31,22.
0|pm2-autoscale | App "app" has 3 worker(s). CPU: 24,29,23.
0|pm2-autoscale | App "app" has 3 worker(s). CPU: 23,25,20.
0|pm2-autoscale | App "app" has 3 worker(s). CPU: 15,18,17.
0|pm2-autoscale | App "app" has 3 worker(s). CPU: 11,13,13.
0|pm2-autoscale | App "app" has 3 worker(s). CPU: 8,9,10.
0|pm2-autoscale | App "app" has 3 worker(s). CPU: 6,7,7.
0|pm2-autoscale | App "app" has 3 worker(s). CPU: 5,5,5.
0|pm2-autoscale | INFO: Decrease workers
0|pm2-autoscale | INFO: App "app" is busy
0|pm2-autoscale | App "app" decresed one worker
0|pm2-autoscale | App "app" has 2 worker(s). CPU: 4,4.
0|pm2-autoscale | App "app" has 2 worker(s). CPU: 3,3.
0|pm2-autoscale | App "app" has 2 worker(s). CPU: 2,2.
0|pm2-autoscale | INFO: Decrease workers
0|pm2-autoscale | INFO: App "app" is busy
0|pm2-autoscale | App "app" decresed one worker

Thus, if the CPU usage of the application is high, the module can increase the number of instances running to handle the load. Similarly, if the CPU usage is low, the module can reduce the number of instances running to save resources.

Source code of the module is available on GitHub https://github.com/VeXell/pm2-autoscale and if you want to add/change something — just create a pull request.

Thank you for reading this article. Hope you found this useful.

PM2 module to monitoring node.js application with export to Prometheus and Grafana

Slava — Wed, 29 Nov 2023 12:18:46 +0000

Monitoring a node.js application is one of the crucial stages in analyzing the quality and proper functioning of the application. It allows you to identify ongoing issues and quickly make decisions to fix them.

PM2 provides internal tools for monitoring all your applications, but in the free version, they are only accessible through the console command pm2 monit. Alternatively, PM2 offers a subscription service called PM2 Plus with additional functionality.

Unfortunately, neither of the mentioned methods are suitable for monitoring because you need to see not only current performance metrics but also changes in values over time (for example, the growth of memory usage within the application or the number of its reloads). Also payment subscription does not integrated into your overall monitoring system and located on separated service.

In most cases, applications use the combination of Prometheus + Grafana, which allows collect data and display it in the form of graphs and also to set up alerts for changes in any metrics.

Unfortunately, PM2 does not provide a convenient and simple way to export data to Prometheus, so it was a motivation to implement my own module that would allow doing this. In addition, this module should be able to export additional metrics which my another module pm2-autoscale, relies.

Finally pm2-prom-module has been released and available for everyone. You can install it just with command pm2 install:



pm2 install pm2-prom-module

After installation monitoring service will be available on port 9988 and you can reach it by url http://localhost:9988/.

Of course you can configure module and change port and service name, for example:



pm2 set pm2-prom-module:port 10801
pm2 set pm2-prom-module:service_name MyApp

Specifying the service_name is very convenient when installing the module across different projects, while using a single dashboard in Grafana. This allows you to create a dropdown list with all the services and quick switch between them.

This module collect all available metrics provided by PM2 and a few additional ones. The complete list looks something like this:

Available free memory
Number of CPUs
Number of running applications
Number of instances for each application
Average memory usage per application
Total memory used for each application
Average CPU load for each application
Current CPU load for each application
Number of restarts for each application
Uptime for each application

As well as all the statistics within PM2:

Used Heap Size
Heap Usage
Heap Size
Event Loop Latency p95
Event Loop Latency
Active handles
Active requests
HTTP req/min
HTTP P95 Latency
HTTP Mean Latency

Sometimes, PM2 returns very peculiar data that requires double-checking, such as Active requests or HTTP req/min (in reality, it seems more like req/sec).

Here is example of module output:



# HELP pm2_free_memory Show available host free memory
# TYPE pm2_free_memory gauge
pm2_free_memory{serviceName="my-app"} 377147392

# HELP pm2_cpu_count Show available CPUs count
# TYPE pm2_cpu_count gauge
pm2_cpu_count{serviceName="my-app"} 4

# HELP pm2_available_apps Show available apps to monitor
# TYPE pm2_available_apps gauge
pm2_available_apps{serviceName="my-app"} 1

# HELP pm2_app_instances Show app instances count
# TYPE pm2_app_instances gauge
pm2_app_instances{app="app",serviceName="my-app"} 2

# HELP pm2_app_average_memory Show average using memory of an app
# TYPE pm2_app_average_memory gauge
pm2_app_average_memory{app="app",serviceName="my-app"} 60813927

# HELP pm2_app_total_memory Show total using memory of an app
# TYPE pm2_app_total_memory gauge
pm2_app_total_memory{app="app",serviceName="my-app"} 121626624

# HELP pm2_event_loop_latency_p95 Event Loop Latency p95. Unit "ms"
# TYPE pm2_event_loop_latency_p95 gauge
pm2_event_loop_latency_p95{app="app",instance="1",serviceName="my-app"} 2.55
pm2_event_loop_latency_p95{app="app",instance="2",serviceName="my-app"} 2.48

Finally you can have Grafana dashboard with detailed statistic of your application. If you like it you can download this dashboard from this link.

In our case, each node.js project is packaged into a Docker container, and inside, we install PM2 and additional modules. All parameters, such as the service name or port, are passed through arguments. As a result, the Docker container looks something like this:



FROM node:18-bullseye-slim 

#### Install PM2
RUN npm i -g pm2@5.2.2

# ...
# Build project
#...

ARG PROJECT_NAME
ARG ENV_METRICS_PORT

#### Install modules for PM2
RUN pm2 install pm2-autoscale && pm2 install pm2-prom-module && pm2 set pm2-prom-module:port $ENV_METRICS_PORT && pm2 set pm2-prom-module:service_name $PROJECT_NAME

CMD ["pm2-runtime", "--json", ".ecosystem.config.js"]

How to quickly generate dynamic images with Node.js and Puppeteer.

Slava — Mon, 27 Nov 2023 15:28:17 +0000

In our time, numerous websites create pages that users share in different social networks or messengers. Thanks to Open Graph tags, links can have a preview image that attracts even more attention, for example with og:image tag. But usually, many websites don’t put much effort into preview images and simply add one image to the majority of pages. If there is no image, parsers try to automatically find the first available suitable image and use it.

But imagine how great it would be to have a personalized image on the page, for example, when you have profiles of your users or events in which they participate.

For example, take a look at the image below that we generate for sharing on social networks. It has a custom font, gradient, and even localization of text on the image into different languages.

In our project, the initial versions of images were generated using PHP, and for a while, this was sufficient because the images were quite simple and only contained the user’s picture. However, as soon as we added the user’s name, problems with text positioning immediately arose. For instance, here’s a small example of creating a simple image in PHP:


<?php

$width = 400;
$height = 200;
$image = imagecreatetruecolor($width, $height);

$backgroundColor = imagecolorallocate($image, 255, 255, 255);
imagefill($image, 0, 0, $backgroundColor);

$textColor = imagecolorallocate($image, 0, 0, 0);

// Load a custom TrueType font
$fontFile = 'path/to/your/font.ttf';

// Set the text to be displayed (considering localization)
$language = isset($_GET['lang']) ? $_GET['lang'] : 'en';
$text = getLocalizedText($language);

$fontSize = 24;

// Set the position for the text to be displayed
$textbox = imagettfbbox($fontSize, 0, $fontFile, $text);
$textX = ($width - ($textbox[2] - $textbox[0])) / 2;
$textY = ($height - ($textbox[5] - $textbox[3])) / 2 + ($textbox[5] - $textbox[3]);

imagettftext($image, $fontSize, 0, $textX, $textY, $textColor, $fontFile, $text);

header("Content-Type: image/png");
imagepng($image);
imagedestroy($image);

function getLocalizedText($language) {
    switch ($language) {
        case 'en':
            return 'Hello, PHP!';
        case 'fr':
            return 'Bonjour, PHP!';
        default:
            return 'Hello, PHP!';
    }
}

As seen from the example, if the text is too long or if you would like change its size, it may extend beyond the boundaries of the image. And if you add gradients, transparency, different fonts, emojis, the code will become complex and challenging to maintain.

The solution I would like to present allows you to reduce development time and simplifies the maintenance of such images. Moreover, it is flexible and easily scalable for other purposes.

So every web developers knows that HTML is the simplest and most convenient markup language. Along with CSS styles, you can create an interface that is flexible and takes into account the positioning of any elements on the page — images, text, tables, lists and etc.

Let’s have a look and on the example below. This is simple HTML page with many css styles like gradient in text, shadows, limiting text rows for long text and other.


<html>
    <head>
        <style>
            html, body {
                margin: 0;
                padding: 0;
            }

            body {
                background-color: #0093E9;
                background-image: linear-gradient(160deg, #0093E9 0%, #80D0C7 100%);

                font-family: Helvetica, sans-serif;
                padding: 5%;
                text-align: center;
            }

            header {
                position: relative;
            }

            header .emoji {
                position: absolute;
                top: -10px;
                left: 0;
                transform: rotate(20deg);
                font-size: 3rem;
            }

            * {
                box-sizing: border-box;
            }

            h1 {
                text-transform: uppercase;
                font-size: 3rem;
                background: -webkit-linear-gradient(45deg, #85FFBD 0%, #FFFB7D 100%);
                -webkit-background-clip: text;
                -webkit-text-fill-color: transparent;
            }

            .wrapper {
                display: flex;
                padding-top: 2rem;
            }

            .avatar {
                display: flex;
                justify-content: center;
                align-items: center;
            }
            .avatar img {
                width: 140px;
                height: 140px;
                border-radius: 100px;
                border: 5px solid rgba(255,255,255, 0.5);
                box-shadow: 0 0 10px rgba(0,0,0,0.2);
                object-fit: cover;
            }

            .content {
                padding: 1rem 2rem;
            }

            .content .text {
                font-size: 1.5rem;
                display: -webkit-box;
                -webkit-line-clamp: 3; /* number of lines to show */
                        line-clamp: 3; 
                -webkit-box-orient: vertical;
                overflow: hidden;
                color: rgba(255,255,255, 0.8);
            }
        </style>
    </head>
    <body>
        <div>
            <header>
                <h1>Hello, Javascript</h1>
                <div class="emoji">🤖</div>
            <header>
            <div class="wrapper">
                <div class="avatar">
                    <img src="https://sun9-57.userapi.com/impg/O3egMIWPZjhcKSThZ2hn7ByaQmET8ySOq5e4ww/O_ngP3qqEd8.jpg?size=1178x1789&quality=95&sign=71fcbf49ffff80fad9f0ef39f598cf69&type=album" alt=""/>
                </div>
                <div class="content">
                    <div class="text"><strong>Lorem Ipsum</strong>  is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.</div>
                </div>
            </div>
        </div>
    </body>
</html>

What we need it is run a web server which can serve this HTML code and run node.js application with Puppeteer library, which launches a headless version of Google Chrome, an then you can take a screenshot of the page and get the desired image.

Below is an example code that allows you to capture the content of a page with specified width, height and scale. It is crucial to handle exceptions and close the browser pages if something goes wrong; otherwise, it may lead to a memory leak.

const browser = await puppeteer.launch({
    headless: true,
    ignoreHTTPSErrors: true,
    executablePath: browserPath,
    args: ['--no-sandbox', '--disable-setuid-sandbox'],
    dumpio: true,
});

const page: Page = await browser.newPage();
await page.setJavaScriptEnabled(false);
await page.setViewport({
    width,
    height,
    deviceScaleFactor,
});

try {
    const result = await page.goto(url, {
        waitUntil: 'load',
    });

    if (result.status() !== 200) {
       page.close();
       throw new PageNotFoundError(`Incorrect status page ${result.status()}`);
    }
} catch (error) {
    page.close();
    throw new PageFetchError(error as string);
}

const data = await page.screenshot({
    type: imageType,
    quality: imageQuality,
    encoding: 'binary',
});

page.close();

And finally as result you can get image like that which you can upload to you S3 storage and serve it to users. This is really simple to support and manage.

In our case we create a small node.js application with Puppeteer that has been packaged into a Docker container and run HTTP server inside to manage external requests. This application allows us to generate images with various format (like png, jpgor webp )for any page of the website.

So, the full logic of service for OG images generation might be:

When your React app is rendering a page your generate special link in html meta tags og:image (with user id, required image size and extension) to nginx proxy.
The proxy server checks if the image is already in the S3 storage or not.
If the image exists, it is served; if not, a request is made to the node.js service, which generates the image, serves it, and asynchronously uploads it to S3.

During the development we faced few minor problems such as:

Chromium issues on a Ubuntu Docker container were resolved by directly downloading Google Chrome.
Font compatibility: one of our fonts broke Apple emoji font. We decided to replace it.

Despite minor technical hurdles, the new node.js service significantly reduced image generation and support time compared to the previous PHP code. The solution allows us quickly update images, just change HTML code, simple to test it in the browser and use all power of CSS.

It’s a great way to achieve flexibility and scalability in generating images dynamically for any of your purposes (not only to generate Open Graph images).

If you have any more questions or if there’s anything else I can help you with, feel free to ask!

Here is example of docker container which we are using with all required libraries to work with images.

FROM ubuntu:20.04

RUN ln -s /usr/share/zoneinfo/Etc/UTC /etc/localtime \
 && apt -y update \
 && apt -y install \
  git \
        openssh-server \
  gconf-service \
  libasound2 \
  libatk1.0-0 \
  libc6 \
  libcairo2 \
  libcups2 \
  libdbus-1-3 \
  libexpat1 \
  libfontconfig1 \
  libgcc1 \
  libgconf-2-4 \
  libgdk-pixbuf2.0-0 \
  libglib2.0-0 \
  libgtk-3-0 \
  libnspr4 \
  libpango-1.0-0 \
  libpangocairo-1.0-0 \
  libstdc++6 \
  libx11-6 \
  libx11-xcb1 \
  libxcb1 \
  libxcomposite1 \
  libxcursor1 \
  libxdamage1 \
  libxext6 \
  libxfixes3 \
  libxi6 \
  libxrandr2 \
  libxrender1 \
  libxss1 \
  libxtst6 \
  ca-certificates \
  fonts-liberation \
  libappindicator1 \
  libnss3 \
  lsb-release \
  xdg-utils \
  wget \
  curl \
  libnss3-dev \
  libgbm-dev \
  libu2f-udev \
  udev \
 && (curl -fsSL https://deb.nodesource.com/setup_14.x | bash -) \
 && apt-get install -y nodejs \
 && wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb \
 && apt install ./google-chrome-stable_current_amd64.deb \
 && rm -rf ./google-chrome-stable_current_amd64.deb \
 && apt-get clean \
 && rm -rf /var/cache/apt/lists

# Add new fonts
COPY ./fonts /root/.fonts
RUN fc-cache -fv

# Build and run your node.js app
...

# Run node
CMD ["node", "app.js"]