Exposing Hazelcast Metrics with JMX and Prometheus

Monitoring and managing the performance of your distributed systems is crucial, especially when using a data grid like Hazelcast. One of the effective ways to expose metrics from Hazelcast is by utilizing the JMX (Java Management Extensions) Prometheus exporter. This blog will guide you through the steps to set up the JMX Prometheus exporter to expose Hazelcast metrics for monitoring.

Step 1: Download the JMX Prometheus Exporter

First, you need to download the JMX Prometheus Java agent. This can be easily obtained from the Maven repository. Use the following link to download the JAR file:

• Download jmx_prometheus_javaagent-1.0.1.jar

Make sure to save the JAR file in a convenient location on your server.

Step 2: Create the JMX Exporter Configuration File

Next, you'll need to create a configuration file named jmx_exporter.yml. This file will specify which metrics to collect from the Hazelcast instance. Below is an example configuration:

startDelaySeconds: 0
attrNameSnakeCase: true
lowercaseOutputName: true
lowercaseOutputLabelNames: true
whitelistObjectNames:
  - "com.hazelcast:type=Metrics,*"
rules:
  - pattern: "^com.hazelcast<type=Metrics, instance=(.*), prefix=(.*), tag([0-9]+)=(.*)><>(.+):"
    name: hazelcast_$5
    labels:
      instance: $1
      prefix: $2
      tag$3: $4

  - pattern: "^com.hazelcast<type=Metrics, instance=(.*), prefix=(.*)><>(.+):"
    name: hazelcast_$3
    labels:
      instance: $1
      prefix: $2

Important Note:

When running the JMX exporter as a Java agent, do not configure the jmxUrl or hostPort. This is because you’re not monitoring a remote JVM; you’re monitoring the local Hazelcast instance.

Step 3: Enable Metrics in Hazelcast

Before starting your Hazelcast application, you must enable metrics in its configuration. This can be done by modifying the Java code as follows:

Config config = new Config();
config.setProperty("hazelcast.jmx", "true");
config.setProperty("hazelcast.metrics.enabled", "true");

This code snippet ensures that JMX and metrics are enabled within your Hazelcast configuration.

Step 4: Start the Hazelcast Application with JMX Exporter

Now it's time to start your Hazelcast application with the JMX Prometheus exporter. Use the following command to run your application:

java -javaagent:<path>/jmx_prometheus_javaagent-1.0.1.jar=8080:<path>/jmx_exporter.yml -cp 'common-model-1.9.0-SNAPSHOT.jar:lib/*' io.dxchange.resource.efs.HazelcastServer

Note:

• Replace <path> with the actual path to your JAR and configuration file.
• You can change the 8080 port to any available port if needed.

Step 5: Verify the Metrics

To ensure everything is set up correctly, you can verify the exposed metrics using the following curl command:

curl http://localhost:8080/metrics

This command will return a list of metrics that have been collected from your Hazelcast instance. You should see a response similar to the following:

  
  
  HELP hazelcast_some_metric Description of some metric


  
  
  TYPE hazelcast_some_metric counter


hazelcast_some_metric{instance="your-instance", prefix="your-prefix"} 123

...

Example Output:

Conclusion

By following the steps outlined in this blog, you can effectively expose Hazelcast metrics using the JMX Prometheus exporter. This setup will enable you to monitor the performance of your Hazelcast instances, allowing for better management and optimization of your distributed applications. Happy monitoring!