DEV Community: Zameer Fouzan

Monitoring PM2 in production

Zameer Fouzan — Fri, 01 Nov 2024 13:30:00 +0000

In large-scale Node.js production environments, monitoring multiple applications can become challenging. The New Relic APM agent for Node.js helps capture logs, traces, and in-depth performance metrics from individual applications. But what about the overall health and resource consumption of all Node.js processes themselves, and critical process-level metrics like CPU and memory usage?

PM2 is a popular process manager for Node.js applications, designed to simplify deployment and ensure reliability. It provides robust features, including automatic application restarts, load balancing, and monitoring capabilities, making it an essential tool for managing production-grade Node.js environments.

PM2's monitoring API provides easy access to detailed telemetry data, like active processes, and resource consumption, along with additional helpful details like Git commit, active branch, Node.js version, and entry script. These insights can be invaluable when troubleshooting and identifying the root cause of performance issues.

What can I monitor from PM2?

PM2 API provides generous information that can be captured. Ideally, all the metrics that can be visualized using the PM2’s monitor API on your host can be captured and exported to New Relic.

In this blog post, we’ll focus on several key metrics that are important for monitoring PM2.

Application identity and host information

Application details, including the host ID, for applications currently running as part of PM2, make it easy to pinpoint which instance is associated with each application. This is especially helpful in environments with multiple services, as you can easily identify specific processes and manage them effectively.

Metrics: appName, hostId

Resource metrics

Metrics from each process, like CPU and memory usage, provide essential insights into the resource consumption and performance of your application. With axm_monitor, you can also track HTTP latency, active requests, and event loop latency to monitor responsiveness in real time.

Metrics: monit.cpu, monit.memory, axm_monitor (for example, HTTP latency, active requests, event loop latency)

Process information

Process-level details such as process ID, status, and uptime help monitor each application's health and lifecycle. Tracking when a process was last updated or started helps you identify issues like frequent restarts or unusually high uptime.

Metrics: pid, pm2_env.status, pm2_env.pm_uptime, pm2_env.created_at, timestamp, pm2_env.update_time

Debug pointers: Logs, errors, restarts, and crashes

Debugging details like log paths, error codes, and restart counts allow quick troubleshooting. Having both the standard output and error logs, alongside information on restarts, helps pinpoint stability issues or potential bugs within applications.

Metrics: pm2_env.error_file, pm2_env.out_file, pm2_env.exit_code, pm2_env.unstable_restarts, pm2_env.restart_time, pm2_env.status

Version control and deployment insights

Deployment details, including Git commit history and Node.js version, make it easy to track deployed versions. PM2 captures branch, revision, and commit data, giving clear visibility into which version is running and simplifying root cause analysis after code changes.

Metrics: pm2_env.node_version, pm2_env.version, pm2_env.versioning.*

How do I capture insights from PM2?

Currently, there’s no direct integration between New Relic agents and PM2, but we can build our own integration using Flex. Flex is an easy and agentless option to build integrations between your data source and New Relic.

Configuration

Create a new configuration file named pm2_monit.yml, then add the following configuration.

This configuration is simply calling the PM2 jlist API and sanitizing the JSON output data using JQ. JQ is a command line utility that's used to process, query, and transform JSON data. Flex has support for JQ built in, which makes sanitizing and transforming the data easier.

As the original output can also include additional, sensitive data, which could be unnecessary, it can be easily dropped with the JQ and remove_keys functions of Flex.

Basic configuration

integrations:
  - name: nri-flex
    timeout: 60s
    interval: 30s
    config:
      name: PM2status
      apis:
        - name: PM2Process
          event_type: PM2Sample
          commands:
        # Linux specific command
            - run: USER=ubuntu; su - $USER bash -c "pm2 jlist"
        # command for windows/mac
        # - run: npx pm2 jlist

JSON transformation with JQ

At this stage, the output from pm2 jlist is a raw JSON. Let’s sanitize and transform the JSON output with JQ to retain only the necessary fields, using the remove_keys and rename_keys functions to streamline the data further:

       # Sanitize and transform the JSON output to required format   
          jq: >- 
              [] | { 
                  pid,
                  name,
                  pm2_env: {
                    script: .pm2_env.script?,
                    out_file: .pm2_env.out_file?,
                    error_file: .pm2_env.error_file?,
                    watch: .pm2_env.watch?,
                    exit_code: .pm2_env.exit_code?,
                    node_version: .pm2_env.node_version?,
                    versioning: .pm2_env.versioning?,
                    version: .pm2_env.version?,
                    unstable_restarts: .pm2_env.unstable_restarts?,
                    restart_time: .pm2_env.restart_time?,
                    created_at: .pm2_env.created_at?,
                    pm_uptime: .pm2_env.pm_uptime?,
                    status: .pm2_env.status?,
                    unique_id: .pm2_env.unique_id?
                  },
                  pm_id,
                  monit
                } | del(.pm2_env.versioning.remotes) 
          remove_keys:
            - pm_id
          rename_keys:
            name: appName
          custom_attributes:
            hostId: localhost

Switching user context

The command block in Flex configuration switches the user context to where PM2 processes are running, which is necessary, as Flex runs under the root (sudo) user by default. If you're running PM2 locally on Windows or Mac, use npx pm2 jlist command block instead of the Linux-specific command.

The complete configuration should look like this:

integrations:
  - name: nri-flex
    timeout: 60s
    interval: 30s
    config:
      name: PM2status
      apis:
        - name: PM2Process
          event_type: PM2Sample
          commands:
            - run: USER=ubuntu; su - $USER bash -c "pm2 jlist"
        # - run: npx pm2 jlist
       # Sanitize and transform the JSON output to required format   
          jq: >- 
              [] | { 
                  pid,
                  name,
                  pm2_env: {
                    script: .pm2_env.script?,
                    out_file: .pm2_env.out_file?,
                    error_file: .pm2_env.error_file?,
                    watch: .pm2_env.watch?,
                    exit_code: .pm2_env.exit_code?,
                    node_version: .pm2_env.node_version?,
                    versioning: .pm2_env.versioning?,
                    version: .pm2_env.version?,
                    unstable_restarts: .pm2_env.unstable_restarts?,
                    restart_time: .pm2_env.restart_time?,
                    created_at: .pm2_env.created_at?,
                    pm_uptime: .pm2_env.pm_uptime?,
                    status: .pm2_env.status?,
                    unique_id: .pm2_env.unique_id?
                  },
                  pm_id,
                  monit
                } | del(.pm2_env.versioning.remotes) 
          remove_keys:
            - pm_id
          rename_keys:
            name: appName
          custom_attributes:
            hostId: localhost

Validation

Once the configuration is ready, we can validate it with Flex's debug mode. If you’re using the standalone binary mode of Flex, use the following command to test your configuration:

sudo ./nri-flex -config_file pm2_monit.yml --pretty --verbose

To test the configuration using New Relic's infrastructure agent Flex integration, execute the following command:

sudo /var/db/newrelic-infra/newrelic-integrations/bin/nri-flex --verbose --pretty --config_file ./pm2_monit.yml

Read more about Flex testing and debugging in this document.

Upon successful execution of your configuration, you should expect an output similar to the one displayed below, without the Flex debug output:

{
    "name": "com.newrelic.nri-flex",
    "protocol_version": "3",
    "integration_version": "1.15.2",
    "data": [
        {
            "metrics": [
                {
                    "appName": "expressApp-otel",
                    "event_type": "PM2Sample",
                    "hostId": "ec2-webserver",
                    "integration_name": "com.newrelic.nri-flex",
                    "integration_version": "1.15.2",
                    "monit.cpu": 0,
                    "monit.memory": 46641152,
                    "pid": 1304,
                    "pm2_env.created_at": 1728589283166,
                    "pm2_env.error_file": "logs/error.log",
                    "pm2_env.exit_code": 1,
                    "pm2_env.node_version": "16.17.0",
                    "pm2_env.out_file": "logs/app.log",
                    "pm2_env.pm_uptime": 1729058602005,
                    "pm2_env.restart_time": 0,
                    "pm2_env.script": "./server.js",
                    "pm2_env.status": "online",
                    "pm2_env.unique_id": "858b1616-bfd7-41c0-8861-de5babcfe106",
                    "pm2_env.unstable_restarts": 0,
                    "pm2_env.version": "1.0.0",
                    "pm2_env.versioning.ahead": "false",
                    "pm2_env.versioning.branch": "master",
                    "pm2_env.versioning.branch_exists_on_remote": "true",
                    "pm2_env.versioning.comment": "update: Added query param to fetch mapped category in response\nfix: fallback envvar value\n",
                    "pm2_env.versioning.next_rev": "\u003cnil\u003e",
                    "pm2_env.versioning.prev_rev": "f195db41aec0592142ef478c424ec1722c7318a4",
                    "pm2_env.versioning.remote": "origin",
                    "pm2_env.versioning.repo_path": "/home/ubuntu/workspace/node-express-app",
                    "pm2_env.versioning.revision": "8d31b501b38229171be428db1dc7e2b412694116",
                    "pm2_env.versioning.type": "git",
                    "pm2_env.versioning.unstaged": "true",
                    "pm2_env.versioning.update_time": "2024-10-16T06:03:22.325Z",
                    "pm2_env.versioning.url": "git@github.com:zmrfzn/node-express-app.git",
                    "pm2_env.watch": "false"
                },
                {
                    "event_type": "flexStatusSample",
                    "flex.Hostname": "ip-172-31-30-74",
                    "flex.IntegrationVersion": "1.15.2",
                    "flex.counter.ConfigsProcessed": 1,
                    "flex.counter.EventCount": 2,
                    "flex.counter.EventDropCount": 0,
                    "flex.counter.PM2Sample": 2,
                    "flex.time.elapsedMs": 263,
                    "flex.time.endMs": 1730119876587,
                    "flex.time.startMs": 1730119876324
                }
            ],
            "inventory": {},
            "events": []
        }
    ]
}

Here’s the simplified data that we’re capturing with Flex from the PM2 processes after transformations with JQ:

       {
            "appName": "expressApp-otel",
            "hostId": "ec2-webserver",
            "integration_name": "com.newrelic.nri-flex",
            "integration_version": "1.15.2",
            "monit.cpu": 0,
            "monit.memory": 80068608,
            "pid": 1310,
            "pm2_env.created_at": 1728589676754,
            "pm2_env.error_file": "logs/error.log",
            "pm2_env.exit_code": 1,
            "pm2_env.node_version": "16.17.0",
            "pm2_env.out_file": "logs/app.log",
            "pm2_env.pm_uptime": 1729058602010,
            "pm2_env.restart_time": 0,
            "pm2_env.script": "./server.js",
            "pm2_env.status": "online",
            "pm2_env.unique_id": "83e7e3d5-17d7-41b0-87fe-5201148c826a",
            "pm2_env.unstable_restarts": 0,
            "pm2_env.version": "1.0.0",
            "pm2_env.versioning.ahead": "false",
            "pm2_env.versioning.branch": "otel",
            "pm2_env.versioning.branch_exists_on_remote": "true",
            "pm2_env.versioning.comment": "chore: update OTEL SDK packages to latest",
            "pm2_env.versioning.next_rev": "<nil>",
            "pm2_env.versioning.prev_rev": "f9fb795ec5ec07d24bd858b55287cbffda44b365",
            "pm2_env.versioning.remote": "origin",
            "pm2_env.versioning.repo_path": "/home/ubuntu/workspace/otel/node-express-app",
            "pm2_env.versioning.revision": "aad10ae5445469718cd2da5041d140e39be8ef78",
            "pm2_env.versioning.type": "git",
            "pm2_env.versioning.unstaged": "true",
            "pm2_env.versioning.update_time": "2024-10-16T06:03:22.319Z",
            "pm2_env.versioning.url": "git@github.com:zmrfzn/node-express-app.git",
            "pm2_env.watch": "false",
            "timestamp": 1729513381684
          }

Verification

Flex sends all processed data via the New Relic events API, which allows efficient handling of various types of event data. In this configuration, we’re naming our event PM2Sample, which helps clearly identify and differentiate it from other events in the system.

All the data associated with this event can be easily queried using New Relic Query Language (NRQL) on this table itself.

FROM PM2Sample SELECT * SINCE 10 MINUTES AGO LIMIT 5

Visualization

Once the data is accessible on the New Relic platform, you can easily query specific metrics relevant to your needs and create tailored visualizations. This allows you to analyze performance trends, monitor process health, and also gather versioning details for the individual applications currently running with PM2.

With customizable visualizations, you can present the data in a way that best suits your objectives and enhances your understanding of complex information. Additionally, these metrics can also be used to set up personalized alerts.

Let’s dive into the PM2Sample. Now we can effortlessly query the average CPU utilization from individual applications:

FROM PM2Sample select average(monit.cpu) as 'CPU Usage %' facet appName

You can also capture time series data, showcasing how memory consumption compares to CPU usage over time for all your applications under PM2, which are outside of your APM metrics.

FROM PM2Sample select average(monit.memory)/1048576 as 'Avg Memory M/b' , average(monit.cpu) as 'Avg CPU%' EXTRAPOLATE TIMESERIES

NRQL provides an easy way to query and visualize metrics, with powerful dashboard capabilities. Below is an example of a custom dashboard setup specifically designed for PM2 monitoring. It captures various metrics, including memory usage per application, CPU vs. memory for all active applications, and the latest app revision details of the active processes with PM2.

Import this dashboard using the JSON from this location: PM2_monit_dashboard. Make sure to replace all the placeholder account Ids (1234567) with your New Relic account ID before importing it.

Conclusion

PM2’s telemetry offers process-level insights, capturing essential metrics like CPU, memory, and application logs. Flex’s customizable integration with New Relic allows you to bring these data points into a single, unified view. This combination not only enhances visibility into application performance, but also simplifies root-cause analysis and proactive monitoring

What Next?

For more information, visit the following:

Instrumenting AWS Lambda functions with OpenTelemetry SDKs

Zameer Fouzan — Sat, 25 Nov 2023 12:30:00 +0000

In this blog post, we'll explore how to use OpenTelemetry to instrument and monitor serverless applications running on AWS Lambda. We'll look at the Open Telemetry SDK, a component that simplifies the configuration and deployment of OpenTelemetry agents for Lambda functions.

What is OpenTelemetry?

OpenTelemetry is an open-source project that provides a unified framework for collecting, processing, and exporting observability data from distributed systems. Observability data includes metrics, traces, and logs that help developers and operators understand the behavior and performance of their applications.

OpenTelemetry supports multiple languages, platforms, and vendors, and offers a vendor-neutral API and SDK for instrumenting applications. It also provides a set of exporters that allow users to send their data to various backends, such as New Relic, Jaeger, Zipkin, AWS X-Ray, and many more.

Why use OpenTelemetry for serverless?

Serverless applications are composed of ephemeral, event-driven functions that run on demand in response to triggers. This makes them scalable, cost-effective, and easy to deploy but also introduces new challenges for observability.

For example, serverless functions have short lifespans and may not have access to persistent storage or network resources. This means traditional methods of collecting and exporting observability data, such as using agents or sidecars, may not work well for serverless functions. Additionally, serverless functions may have complex invocation patterns and dependencies, which make it difficult to trace the end-to-end execution flow and identify bottlenecks or errors.

OpenTelemetry addresses these challenges by providing a lightweight and flexible solution for instrumenting serverless functions. With OpenTelemetry you can:

Add minimal code changes to the functions to enable automatic instrumentation of common libraries and frameworks.
Configure the sampling rate and span attributes to control the amount and quality of trace data collected.
Choose from a variety of exporters to send their data to the backend of their choice.
Leverage the Open Telemetry Lambda Collector to simplify the deployment and configuration of OpenTelemetry agents for Lambda functions (not covered in this blog).

We'll focus on the automatic instrumentation of the Lambda function using the OpenTelemetry libraries and SDKs for Node.js.

Pre-requisites

What do you need to follow this guide successfully?

New Relic ingest license key. (Don’t have an account? Sign up for a free New Relic account).
Basic knowledge of OpenTelemetry

Libraries overview

These are the few key libraries and APIs that are essential in instrumenting the AWS Lambda function with the Node.js runtime.

@opentelemetry/instrumentation-aws-lambda: This module provides automatic instrumentation for the AWS Lambda module, and it's' included in the @opentelemetry/auto-instrumentations-node bundle as a plug-in. This is necessary to instrument our AWS Lambda functions.

@opentelemetry/sdk-trace-node: This module provides automated instrumentation and tracing for Node.js applications. It exposes an API called NodeTracerProvider, which can be used with the registerInstrumentations API from @opentelemetry/instrumentation to load individual instrumentation plugins.

OTLPTraceExporter: This is a common API provided in the @opentelemetry/exporter-trace-otlp-http and @opentelemetry/exporter-trace-otlp-proto packages for exporting the collected telemetry to a collector or a choice of backend. We'll use New Relic's native OpenTelemetry Protocol (OTLP) endpoint to export our telemetry data.

Instrumenting our serverless functions with OpenTelemetry

In this example, we're using the serverless framework to quickly set up the Lambda function along with an API gateway for the entry point. The lambda function is a simple Koa REST API with a few functional endpoints.

Step 1: Install packages

First, install all required packages for Node.js and Node runtime on the Lambda function. Copy the command below and run it in the root of your Node app

npm install --save 
@opentelemetry/auto-instrumentations-node
@opentelemetry/sdk-trace-node
@opentelemetry/exporter-trace-otlp-proto

Step 2: Configure SDK

Next, create a new file with the name otel-wrapper.js in the root or src folder of your Function in the serverless framework project. Now, copy and paste the following code into the newly created file:

const { NodeTracerProvider } = require("@opentelemetry/sdk-trace-node");
const { registerInstrumentations } = require("@opentelemetry/instrumentation");

const {
  getNodeAutoInstrumentations,
} = require("@opentelemetry/auto-instrumentations-node");
const { BatchSpanProcessor } = require("@opentelemetry/sdk-trace-base");

const { Resource } = require("@opentelemetry/resources");
const {
  SemanticResourceAttributes,
} = require("@opentelemetry/semantic-conventions");

const {
  OTLPTraceExporter,
} = require("@opentelemetry/exporter-trace-otlp-proto");

const {
  CompositePropagator,
  W3CBaggagePropagator,
  W3CTraceContextPropagator,
} = require("@opentelemetry/core");

// For troubleshooting, set the log level to DiagLogLevel.DEBUG
const { diag, DiagConsoleLogger, DiagLogLevel } = require("@opentelemetry/api");
// var logger = diag.createComponentLogger(DiagLogLevel.WARN);
diag.setLogger(new DiagConsoleLogger(), DiagLogLevel.WARN);

const COLLECTOR_STRING = `${process.env.OTEL_EXPORTER_OTLP_ENDPOINT}/v1/traces`;

/**
 * The `newRelicExporter` is an instance of OTLPTraceExporter
 * configured to send traces to New Relic's OTLP-compatible backend.
 * Make sure you have added your New Relic Ingest License to NR_LICENSE env-var
 */
const newRelicExporter = new OTLPTraceExporter({
  url: COLLECTOR_STRING,
  headers: {
    "api-key": `${process.env.NR_LICENSE}`,
  },
});

const provider = new NodeTracerProvider({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: process.env.OTEL_SERVICE_NAME,
  }),
});
provider.addSpanProcessor(
  new BatchSpanProcessor(
    newRelicExporter,
    //Optional BatchSpanProcessor Configurations
    {
      // The maximum queue size. After the size is reached spans are dropped.
      maxQueueSize: 1000,
      // The maximum batch size of every export. It must be smaller or equal to maxQueueSize.
      maxExportBatchSize: 500,
      // The interval between two consecutive exports
      scheduledDelayMillis: 500,
      // How long the export can run before it is cancelled
      exportTimeoutMillis: 30000,
    }
  )
);
provider.register({
  propagator: new CompositePropagator({
    propagators: [new W3CBaggagePropagator(), new W3CTraceContextPropagator()],
  }),
});

registerInstrumentations({
  instrumentations: [
    /**
     * AutoInstrumentations META package for Node.js
     * contains bundled libraries for most libraries and frameworks
     */
    getNodeAutoInstrumentations({
      "@opentelemetry/instrumentation-fs": {
        enabled: process.env.ENABLE_FS_INSTRUMENTATION,
        requireParentSpan: true,
      },
      // Enable instrumentation for KOA framework
      "@opentelemetry/instrumentation-koa": {
        enabled: true,
      },
      /**
       * Enable instrumentation for AWS Lambda framework
       * Use the requestHook and responseHook to
       * add additional attributes to the traces
       */
      "@opentelemetry/instrumentation-aws-lambda": {
        enabled: true,
        disableAwsContextPropagation: true,
        requestHook: (span, { event, context }) => {
          span.setAttribute("faas.name", context.functionName);

          if (event.requestContext && event.requestContext.http) {
            span.setAttribute(
              "faas.http.method",
              event.requestContext.http.method
            );
            span.setAttribute(
              "faas.http.target",
              event.requestContext.http.path
            );
          }

          if (event.queryStringParameters)
            span.setAttribute(
              "faas.http.queryParams",
              JSON.stringify(event.queryStringParameters)
            );
        },
        responseHook: (span, { err, res }) => {
          if (err instanceof Error)
            span.setAttribute("faas.error", err.message);
          if (res) {
            span.setAttribute("faas.http.status_code", res.statusCode);
          }
        },
      },
    }),
  ],
});

Highlighting a few key things in the instrumentation in otel-wrapper.js, under the configurations for instrumentation-aws-lambda, the property disableAwsContextPropagation has been set to TRUE. This is required to skip Amazon X-Ray parent extraction, which may conflict with the OpenTelemetry traceparent headers.

Most Node.js OpenTelemetry libraries offer two methods, requestHook and responseHook, for adding custom attributes to spans. These hooks can also be used to add conditional attributes to spans. Take note of the sample responseHook below, which adds an extra attribute to the span called faas.error containing the error message thrown by the code.

responseHook: (span, { err, res }) => {
          if (err instanceof Error)
            span.setAttribute("faas.error", err.message);
          if (res) {
            span.setAttribute("faas.http.status_code", res.statusCode);
          }
        },

Step 3: Configure environment

In the code above in the otel-wrapper.js, there are a couple of environment variables that are being referred to. We need to define these in our function configuration. Since we're using the serverless framework in this example, we need to add them to our serverless.yml configuration file.

Most importantly, to start our function with the OpenTelemetry instrumentations, we need to launch our function a little differently. Notice the NODE_OPTIONS environment variable in the list of all the defined variables. We're setting the argument to use our otel-wrapper.js as the module to be initialized to invoke our function.

functions:
  api:
    handler: index.handler
    environment:
      NODE_OPTIONS: "--require ./otel-wrapper"
      OTEL_SERVICE_NAME: "Node-Lambda-Otel-SDK"
            # External API for Distributed traces
      EXPRESS_OTEL_API_ENDPOINT: "http://3.230.230.121/v3/api"
            # License key stored in SSM parameter.
      NR_LICENSE: ${ssm:/nr_experiments_ingest_key}
      OTEL_EXPORTER_OTLP_ENDPOINT: https://otlp.nr-data.net:4317
            # Disabling FS instrumentation to reduce noise
      ENABLE_FS_INSTRUMENTATION: false

Step 4: Explore data

Once your function is deployed, it's time to explore the telemetry data from the Lambda in the New Relic platform.

Lambda requests metrics from Spans

Attributes from Distributed Traces

Look at the highlighted attributes in the screenshot above: faas.http.method, faas.http.target, and faas.name. These attributes were added by the requestHook method, but not all the attributes are present in this trace; that's because there are conditional attributes depending on the incoming request.

Our Lambda has two different endpoints which are bridged via API gateway, /path and /weather. Let's take a look at the traces from the second endpoint, which is /weather?location=bangalore

The /weather endpoint in the Lambda function calls an external entity that's also instrumented with OpenTelemetry. As a result, the distributed traces capture both the entities and the New Relic platform automatically generates a service map that shows the complete journey of the request.

Lambda distributed traces and service map

In this specific trace, the additional attribute faas.http.queryParams is now visible in the list of attributes. Refer to the screenshot below.

Conditional attributes from requestHook

When exploring the spans in the New Relic UI, you can also check the attributes of the called entity from the distributed traces screen.

Attributes from the called OTEL entity

Conclusion

We've seen how OpenTelemetry can help us instrument and monitor serverless applications running on AWS Lambda. Instrumenting AWS Lambda functions with OpenTelemetry SDKs and exporting traces to New Relic can greatly enhance the observability of serverless applications. By following the steps outlined in this blog post, you can gain valuable insights into the performance, behavior, and dependencies of your Lambda functions. By exporting traces from OpenTelemetry to New Relic, we can take advantage of its powerful features, including service map visualizations, transactions, alerting, and much more.

Next steps?

Check out the source code for the application used in this blog on this GitHub repository: github.com/zmrfzn/lambda-opentelemetry

If you're interested in learning more about OpenTelemetry and serverless observability, you can check out these resources:

OpenTelemetry Functions as a Service

OpenTelemetry AWS Lambda repository

Set up OpenTelemetry for full-stack JavaScript development

How to import Google Cloud logs without an agent

Zameer Fouzan — Fri, 10 Mar 2023 06:30:21 +0000

Push metrics, events, logs, and traces into New Relic without using an installed agent

With New Relic, you can collect, process, explore, query, and create alerts on your log data with scalability for your needs. While there are various options for getting logs into New Relic using an agent or one of the open source plug-ins like FluentBit, in the world of serverless architecture it’s challenging to rely on install-based agents to forward the logs. This is when an agentless approach is really helpful—You can import all your logs to New Relic without installing any agent!

In this blog post, we’ll use all of the Google Cloud Platform native services to set up an agentless log forwarder. By default, Google Cloud services write logs to Google Cloud Logging (formerly Stackdriver). With the logs router, we’ll configure a sink with a destination that is a Cloud Pub/Sub topic. Whenever a log entry is generated and sent to this topic, it triggers a linked Cloud Function that is configured through the push-based subscription on the topic. The Cloud Function accepts all the logs and forwards them to New Relic through the New Relic Log API.

Before you begin

Before you start this tutorial, make sure you have completed these prerequisites:

Enable the Google Cloud APIs for the following services:
- Cloud Pub/Sub
- Cloud Functions
Get your New Relic Insert API Key, which is different from your Ingest License Key.

During the configuration, make sure you do these tasks:

Set up the correct inclusion and exclusion filters on the Google Cloud sink (also known as the logs router).
Create runtime ENV variables to store your New Relic license key for security.

Now you're ready to get started implementing and configuring the log forwarding from Google Cloud to New Relic.

Configure log forwarding from Google Cloud to New Relic

Create a Cloud Function in Google Cloud. Enter basic details like this example:
Select PubSub as the trigger for your Cloud Function, and create a new topic, as shown in this clip.
In the Cloud Function configuration, add a runtime environment variable named API_KEY. This is where you add your Insight Insert Key, which is important so our license is not exposed in our Cloud Function code. Generate this key by selecting the highlighted link on the right panel on the Account Settings > API Keys page.

Note: This Insight Insert Key is not the Insights License or Browser API Key on the Account Settings > API page.

Now, copy this Node.js code sample and paste it into your Google Cloud Functions source.

const https = require("https");

exports.nrLogForwarder = (message, context) => {
  const pubSubMessage = Buffer.from(message.data, "base64").toString();

  /* Setup the payload for New Relic with decoded message from Pub/Sub
      with "message", "logtype" as atrributes
  */
  const logPayload = {
    message: pubSubMessage,
    logtype: "gcpStackdriverLogs",
  };

  // configure the New Relic Log API http options for POST
  const options = {
    hostname: "log-api.newrelic.com",
    port: 443,
    path: "/log/v1",
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "Api-Key":
        process.env
          .API_KEY /* ADD YOUR NR INSIGHTS INSERT LICENSE TO THE RUNTIME ENV VAR */,
    },
  };

  // HTTP Request with the configured options
  const req = https.request(options, (res) => {
    console.log(`statusCode: ${res.statusCode}`);
    const body = [];

    res.on("data", (d) => {
      body.push(d);
    });
    res.on("end", () => {
      const resString = Buffer.concat(body).toString();
      console.log(`res: ${resString}`);
    });
  });

  req.on("error", (error) => {
    console.error(error);
    callback(null, "error!");
  });

  // write the payload to our request
  req.write(JSON.stringify(logPayload));

  req.end();
};

In Google Cloud Functions, this code accepts the message from the Pub/Sub topic and then removes the Pub/Sub formatting. This is necessary because Pub/Sub might add additional information to the message, such as metadata or specific formatting, that’s not relevant in the context of logs. After the messages is decoded, the extracted logs are then forwarded through the New Relic Log API.

Ensure that the entry point is updated with the same exact name as the function name in the code. Then deploy the Google Cloud Function.
Verify that the deployment of your function is successful. Hint: look for green check. This can take up to a few minutes after you selected Deploy in the previous step.
After the function is deployed, switch to Google Cloud Logs Explorer under Logging. This example video shows how to create a sink here to forward the logs to the Pub/Sub topic that we created earlier.
On the next screen, let’s add details to the sink and configure it to forward the logs. When you fill in the sink details for NRLogForwarder, it should look like this:
For the Sink Destination, choose Cloud Pub/Sub Topic and then select the topic we created in step 2.
Optionally, you can add an inclusion filter. By default, the sink is configured to forward all the logs that appear in the Logs Explorer.

You can configure it to forward only the logs for the services that you want to use with New Relic. This is also helpful when you have many services running on your Google Cloud.

For example, you might have a microservices architecture, as in this sample, and you want to capture all the logs generated by these services. These services could be running on a serverless model, such as Cloud Functions or Cloud Run, or Google App Engine. These logs could provide valuable insights into your customers’ journeys on your platform, which can help you debug and identify any bottlenecks.

In this example, I used Winston logger with my Node.js apps and wanted to forward all logs that are from the Google Cloud Winston Logger transport to New Relic.

You’ll need to update your Project ID or project name and copy this snippet:
```
log_name="projects/<Your Project ID or Project Name>/logs/winston_log"
```
Then add in your inclusion filter under Sink Configuration like this example:
Required step: Add an exclusion filter.

If you don’t complete this step, your sink, Cloud Functions, and Pub/Sub will go in an infinite loop of capturing and forwarding the logs, increasing your billing.

Because Cloud Function execution also generates logs, you'll also need to add an exclusion filter under the sink configurations to exclude the logs generated by our Cloud Function, which forwards the log.

Copy this code:
```
resource.labels.function_name="my-nr-logforwarder”
```
Add the code in your exclusion filter, replacing my-nr-logforwarder with your function’s name, like this example:
Now let's go to our New Relic account, verify the implementation, and explore the forwarded logs!

The messages are published as base64 by Pub/Sub, and the code we added in the Cloud Function is responsible for converting it into a readable string format. Using the inherent parsing mechanism on New Relic for the logs, the message is parsed as a JSON. This allows us to gain access to a range of attributes, such as authorizationinfo, emailid, and method names, which we can then use to query the logs. With this data, we can efficiently monitor and track the progress of our services, making sure they are running optimally, and troubleshoot any issues that arise.

You’ve seen that by setting up and using native services from Google Cloud, you can forward all your logs to New Relic. Then you can enjoy the benefits of enhanced log monitoring and explore other capabilities of New Relic.

Read the full blog post at New Relic: How to import Google Cloud logs without an agent

Not an existing New Relic user? Sign up for a free account to get started! 👨‍💻