DEV Community: Gauthier POGAM--LE MONTAGNER

Use Coolify to self-host SigNoz

Gauthier POGAM--LE MONTAGNER — Wed, 10 Sep 2025 08:38:57 +0000

Observability can be a tricky subject requiring a lot of dedication to do it properly. And one thing we don't want is to spend hours deploying the required services before even starting to integrate observability in our ecosystem.

Luckily for us, Coolify now have a template to easily deploy SigNoz, an open source observability platform. This guide will go through the steps to set it up in Coolify.

Adding the service

Update: the template has been merged in Coolify. You can now find it in the list of Coolify' services.

At the moment of writing, SigNoz' template is still in PR. To add it to your Coolify:

Copy the content of signoz.yaml from the PR.
In Coolify, create a new Docker Compose service and select the server to host it.
Paste the content of signoz.yaml in the "Docker Compose file" field.
Rename the service name to SigNoz.

URLs configuration

Once created, you are ready to set the URLs you'll use with SigNoz.

First, update the URL of the signoz container. For example: https://signoz.example.com:8080.

Then, let's configure the Otel Collector. This service collects the traces, metrics and logs from your applications and services and supports many different formats: OTel' GRPC and HTTP formats, Prometheus metrics, and many logs formats (FluentBit/FluentD, syslogs, logs from cloud services, ...).

Each format should be received on a different port, which we will need to expose for the OTel Coollector to be accessible. You have two strategies to do so:

Configuring a domain for each receiver, or...
Directly exposing the ports to the host and the outside world.

Which option you prefers depends on you security needs and how you architecture your domains.

One subdomain per receiver

This solution only requires you to map your subdomain to the Otel Collector service. We will cover the two default receivers, the HTTP and GRPC receivers.

Make sure your subdomains have been registered and point to your server. In my case, I chose https://signoz-grpc.example.com and https://signoz-http.example.com.
Open the "Otel Collector" service settings.
Add your domains with the format https://<subdomain>.example.com:<port in container>, separated by commas.

For example, this is what I have setup:

https://signoz-grpc.example.com:4317,https://signoz-http.example.com:4318

If you want to drain logs from Heroku, you can add a subdomain to receive the logs: ...,https://signoz-heroku.example.com:8081.

Exposing the ports on the host

If you prefer to use a single domain for all receivers, you can edit the Docker Compose to directly expose the ports on the otel-collector container:

services:
  # ...
otel-collector:
  image: signoz/signoz-otel-collector:latest
  container_name: signoz-otel-collector
  # ...
  ports:
  - 4317:4317 # GRPC Collector
  - 4318:4318 # HTTP Collector

  # ...

You can now append the port to your service URL to send data to receiver: https://signoz.example.com:4318

Fixing the health checks

If you tried to run the services, you should have noticed Coolify keeps reporting SigNoz as unhealthy. This is because some containers exit after performing their task, causing Coolify to consider them unhealthy.

To have Coolify properly reporting the health status, you will need to exclude Schema Migrator Sync, Schema Migrator Async and Init Clickhouse from the healthcheck.

Open the settings of the service.
Check Exclude from service status and save the settings.
When you excluded the three services, you can now (re)deploy all SigNoz services for the health check to ignore these three containers.

Enabling SMTP emailing

SigNoz uses emailing for two things: inviting users and to send alerts.

SigNoz emails

To enable SMTP emailing (including inviting new team members), you need to set the following variables from the Environment Variables tab of your Coolify service:

SIGNOZ_EMAILING_ENABLED enables emailing capabilities in SigNoz.
SIGNOZ_EMAILING_SMTP_ADDRESS is the address of the SMTP server to use, in the format host:port.
SIGNOZ_EMAILING_SMTP_FROM is the email address to use in the From field.
SIGNOZ_EMAILING_SMTP_AUTH_USERNAME and SIGNOZ_EMAILING_SMTP_AUTH_PASSWORD are used to authenticate with the SMTP server.

More environment variables are available to use to use authenticate via Identity / Secret or use TLS instead of SmartTLS. Read Passing environment variables not included in the template to learn how to add them.

Alert Manager emails

Email alerts can only be sent if an SMTP server is configured specifically for the alert manager. The global SMTP configuration and the Alert Manager configuration use different environment variables.

Note: SigNoz has a current known issue preventing email alerting configuration from being saved. You can track the progress of this issue in their bug tracker.

To enable email alerts, you need to set the following variables from the Environment Variables tab of your Coolify service:

SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__SMARTHOST is the address of the SMTP server to use, in the format host:port.
SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__FROM is the email address to use in the From field.
SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__AUTH__USERNAME and SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__AUTH__PASSWORD are used to authenticate with the SMTP server.

Passing environment variables not included in the template

You may want to define environment variables that are not included in SigNoz' template (for example new TLS settings for the SMTP / Alert Manager email configuration). Luckily, Coolify makes this process pretty simple:

Edit the Docker Compose in Coolify.
Add the desired variables to the environment of the signoz container with the following format: - <VARIABLE_NAME>=${<VARIABLE_NAME>}. For example: SIGNOZ_EMAILING_SMTP_FROM=${SIGNOZ_EMAILING_SMTP_FROM}.
Go in the Environment Variables tab of you Coolify service and set the value of the newly added variables.

Updating SigNoz

To update SigNoz to a more recent version, you just need to pull the latest version in Coolify.

Open the service in Coolify
Top right of the screen, click on "Advanced" and select "Pull Latest Images and Restart".
Wait for the migration and full restart of SigNoz (can take a few minutes).

Conclusion

SigNoz is now setup and ready to accept logs and traces from your applications and infrastructure.

Their documentation is a good start to learn how to onboard your resources. OpenTelemetry documentation is also a reference, containing the recommended steps to best leverage the OpenTelemetry SDKs and tools for your stack.

Did you find any way to improve the Coolify template, SigNoz setup, or want to suggest a change? Feel free to let me know! I'd be very happy to learn how to make SigNoz even more pleasant to use.

Office365: make a Room Calendar visible for all employees

Gauthier POGAM--LE MONTAGNER — Tue, 01 Apr 2025 12:56:48 +0000

After creating a Room mailbox in Microsoft Admin Center, you may want to make the room' calendar visible for everyone in your organization. However, Microsoft does not expose any settings to do it from the UI.

The solution is to use Powershell to edit the calendar access permission.

# Install the ExchangeOnlineManagement module if you don't have it
Install-Module -Name ExchangeOnlineManagement

# Open a session from Powershell
Connect-ExchangeOnline -UserPrincipalName <YOU_ADMIN_USERNAME> -ShowProgress $true

# Grant everyone access to the mailbox by setting the default permission
Set-MailboxFolderPermission -Identity "<ROOM_MAILBOX_EMAIL>:\Calendar" -User Default -AccessRights Reviewer

# Check the permissions
Get-MailboxFolderPermission -Identity "<ROOM_MAILBOX_EMAIL>:\Calendar"

# Disconnect from the session
Disconnect-ExchangeOnline -Confirm:$false

Now, go in your Outlook client (desktop / mobile / web) and add a coworker calendar. You should now be able to find the Room' calendar and see who booked what.

Maven Resource Filtering: injecting POM' properties in your resources

Gauthier POGAM--LE MONTAGNER — Sat, 17 Sep 2022 16:43:31 +0000

In my previous blog post How to output Log4J2 logs as JSON?, I mentioned that we can extract some properties from our project' pom.xml to add them to our logs. This can be done using Maven' maven-resource-plugin and its filtering capabilities.

I will explain what is resource filtering, how it can be configured, and how to apply this to extract properties from a pom.xml to inject them into a log4j2.xml configuration.

What is resource filtering?

Before compiling the source or your tests, the resources of your application are copied from your project' resource folder (src/main/resources) to Maven' target folder. Maven automatically copies these resources using maven-resource-plugin, one of Maven' standard plugins, called by most goals that require it (such as compile or test-compile).

These resources are any file that your application will use but that should not be compiled, such as properties files, assets, TLS certificates,... Most applications have at least one resource file: a logger configuration file like Log4J2' log4j2.xml file.

When building your project, you might want these files to contain some of the many properties defined in your pom.xml, such as your application' name, version, or one of the properties defined in your pom.xml. By default, however, Maven won't apply any change to your resources. You will have to configure it to injects the properties you want in your resources files. Injecting properties in your resources is what we call filtering the resources of your project.

Enabling resource filtering

To filter your resources, you must tell Maven which files it should filter, and in these files you should define the variables Maven should replace.

Enabling filtering in Maven is pretty simple: you just have to enable it in the configuration of your application resources. This can be done by editing the resource tag in the build section of your application to add the property <filtering>true</filtering> to your resources configuration:

<project>
  ...
  <build>
    ...
    <resources>
      <resource>
        <directory>src/main/resources</directory>
        <filtering>true</filtering>
      </resource>
      ...
    </resources>
    ...
  </build>
  ...
</project>

Once enabled, every resources in src/main/resources will be filtered. Therefore, we can now use filtering to replace variables in our resource files. Let's create a file test.txt and add the following content in it:

Hello, I am project ${project.name}, version ${project.version}. You can identify me by my full name ${project.groupId}:${project.artifactId}.

You can now run the command mvn compile and open target/classes/test.txt. You will find the following content:

Hello, I am project my-test-app, version 1.0.0-SNAPSHOT. You can identify me by my full name com.company.dev:my-test-app.

Filter specific files

By default, filtering applies to every files in your resources that are not binary files (like images). To avoid replacing content that use the same syntax as a property but which should not be replaced, it is recommended to manually select which files are filtered and exclude everything else.

Fortunately, Maven makes it very simple to configure which resources should be filtered by using the <includes> tag:

<project>
  ...
  <build>
    ...
    <resources>
      <!-- Enable filtering for specific resources -->
      <resource>
        <directory>src/main/resources</directory>
        <filtering>true</filtering>
        <includes>
          <include>test.txt</include>
        </includes>
      </resource>
      <!-- Disable filtering for everything else.
           This part is necessary so Maven continues to copy non-filtered resources.
      -->
      <resource>
        <directory>src/main/resources</directory>
        <filtering>false</filtering>
        <excludes>
          <exclude>test.txt</exclude>
        </excludes>
      </resource>
      ...
    </resources>
    ...
  </build>
  ...
</project>

You will note that we added the <resource> tag twice: one with filtering enabled, and a second time without it. If we only added the first tag with the <includes> configured, Maven would only copy (and filter) the resources included in this configuration (in our case, only log4j2.xml) and would ignore every other file.

To ensure all resources of our application are copied including non-filtered ones, we must add a second <resource> tag with filtering disabled and which excludes the filtered files. The list of files to exclude in the second <resource> tag is the same as the included files in the first resource configuration.

With the configuration above, only our test.txt file is filtered and any other file in our resource folder (or its sub-folders) will not be filtered, allowing us to use ${project.name} without worrying that it could be replaced by Maven.

Configuring resources filtering for Log4j2

Applying resource filtering to automatically inject some information now becomes very easy. The first step is to take the code above and enable filtering for log4j2.xml:

<project>
  ...
  <build>
    ...
    <resources>
      <!-- Enable filtering for specific resources -->
      <resource>
        <directory>src/main/resources</directory>
        <filtering>true</filtering>
        <includes>
          <include>log4j2.xml</include>
        </includes>
      </resource>
      <!-- Disable filtering for everything else.
           This part is necessary so Maven continues to copy non-filtered resources.
      -->
      <resource>
        <directory>src/main/resources</directory>
        <filtering>false</filtering>
        <excludes>
          <exclude>log4j2.xml</exclude>
        </excludes>
      </resource>
      ...
    </resources>
    ...
  </build>
  ...
</project>

With our Log4J2 configuration file now filtered, we can easily use the properties defined in our pom.xml to enrich our logs. Let's configure a RollingFile appender that output JSON logs (more info in my post How to output Log4J2 logs as JSON?):

...
<RollingFile name="json-file" fileName="${sys:mule.home}${sys:file.separator}logs${sys:file.separator}${project.artifactId}.log.json"
                 filePattern="${sys:mule.home}${sys:file.separator}logs${sys:file.separator}${project.artifactId}-%i.log.json">
  <JSONLayout compact="true" eventEol="true" properties="true" stacktraceAsString="true" includeTimeMillis="true">
    <!--  Project properties -->
    <KeyValuePair key="appName" value="${project.name}" />
    <KeyValuePair key="version" value="${project.version}" />
  </JSONLayout>
</RollingFile>
...

Our logs will now come with two additional properties: the application name and its version. You can now leverage these additional information, or any other properties extracted from your pom.xml, to enrich your logs and improve their filtering in a tool like Datadog or Splunk.

Conclusion

Resource filtering is a very useful feature of Maven's maven-resource-plugin plugin, allowing you to inject properties from your pom.xml into your resources. With a simple configuration of that plugin, references to properties are injected into your resources when your application is built.

Fish: Start ssh-agent on session opening on MacOS

Gauthier POGAM--LE MONTAGNER — Tue, 23 Aug 2022 12:34:00 +0000

A while ago, I switched to Fish after getting tired of setting up my ZSH after a new system reset (yes, I should backup my config files...). One of the thing I missed however was a proper setup of my ssh agent on session opening. Let's see how to set it up in a few minutes.

What is SSH Agent?

When relying a lot on SSH to log on other machines or to push code using Git (or any other version control system), it starts to be pretty inconvenient to enter your password each time. That's where SSH Agent will help: it is a deamon (process running in the background) that will save your SSH private key and its password in its memory and will enter them for you when opening an SSH session.

While being a very common tool in the Unix world, the version on MacOS has a few additions that allow it to use the system' keychain to store and encrypt your password, increasing the security of your SSH Agent setup.

What is Fish?

Fish is a wonderful Shell with many things ready out-of-the box, making it the most convenient shell I used without having to customise it. If you want to learn more about Fish, there are good articles here on dev.to, and the Fish doc does a good job at presenting it.

Fisher, the plugin manager

Fisher is a simple and convenient plugin manager for Fish. With a single command, you can install new plugins, update them, or remove them, without ever having to change any file. Convenient! Therefore, I recommend to install it to simplify your plugin management.

Like most shells, Fish has quite a few plugin managers. Feel free to pick the one that suits you the best if you already have one. If you use a different plugin manager, you should be able to easily change the command to install the plugin with it.

Enabling SSH Agent at launch

While searching for a way to automatically enable SSH Agent on MacOS, I found that the usual commands you run on most Unix systems are valid but they are not taking advantage of MacOS' keychain. Using the keychain allow decryption of your private' key password when you login while protecting it when you aren't logged thanks to the encryption of the keychain.

The plugin was written and published by nifoc and ensure only a single session of the process is started in your user. I reviewed the code and did not find anything fishy in it, so I decided to use it.

To install it, simply run fisher install nifoc/ssh-agent-macos.fish. This will download the plugin and add its files in the right directories of your fish configuration folder (.config/fish).

To add your SSH key to the agent and save its password in the keychain, run ssh-add --apple-use-keychain. You will be prompted for your password and once entered correctly, it will be saved in your Mac' keychain.

Each time your shell will be opened, the key will be loaded by the agent and made available without having to enter your SSH password every time you use open an SSH session.

Security consideration

As using SSH Agent allows anyone with access to your session to use your SSH key as they want, remember to encrypt the disk of your Mac, add a password to your session (and diable auto-connect), and make sure to lock your session when going away from your computer. We never know what might happen!

How to easily rebuild an app after editing a resource?

Gauthier POGAM--LE MONTAGNER — Fri, 08 Jul 2022 10:48:17 +0000

When editing resource files in Anypoint Studio such as the log configuration, property files, or DataWeave files, you probably noticed that the application is not automatically rebuilt and deployed by the Studio. This is because the Studio only tracks files in src/main/mule or files that are used as resources of a Transform Message component.

This is very inconvenient as starting a new runtime can take a very long time, even with good hardware. Fortunately, there is an easy way to manually trigger the build and redeployment of one or multiple applications that are already running.

Let me show you how to redeploy your app easily. First, open the Project menu and select "Clean...":

This opens a window where I can either clean all projects, triggering a rebuild of all projects, or select one or more projects that I want to rebuild:

In this screenshot, only my Experience API will be rebuilt. If the application was already deployed, the runtime will now automatically redeploy it without restarting the whole runtime.

Tip: if your application does not automatically redeploy after editing a Mule Configuration File, make sure "Build Automatically" is checked in the Project menu (see the first screenshot above).

How to output Log4J2 logs as JSON?

Gauthier POGAM--LE MONTAGNER — Fri, 18 Feb 2022 13:00:30 +0000

Edit: the instructions in this blog post can be applied to any Log4J configuration, not only Mule applications. I mostly talk about Mule apps as I felt this topic is poorly documented while the need for it is very common in Mule dev teams.

Mulesoft platform is very convenient to manage the whole life-cycle of Mule applications. However, reviewing your application' logs can be unpractical as Anypoint Platform' log viewer is limited and not available for all subscription levels. To address this issue or to better integrate with your organisation log management tool, you might want to configure your applications to output logs in the JSON format before collecting them and sending them to your log management platform.

This blog post will guide you through the changes you need to make to Log4J2 to output logs as JSON and where to configure this in your Mule applications. I tried to explain the various components we are manipulating as I was a bit lost myself when I started to configure my logs.

Why choosing JSON as your output format?

If you ever opened a JSON log file, you probably noticed that logs in JSON format are unreadable. Writing logs as text is much easier to read for humans.

However, in a production environment, you don't want to SSH directly on the server to read the logs. Additionally, your company might be using log centralisation tools such as Datadog or Splunk that are shared between teams and centralise logs in a single tool.

For these tools, parsing of text logs can be tricky. You have to write complicated parsing rules using regular expressions to extract data from the text logs that might be hard to maintain. Additionally, every time you want to add a new information to your log, you'd have to edit the pattern in your Log4J2 configuration and updating your parsing rules to extract the new information.

With JSON Layout, every log is structured in a JSON object that contains not only the log message and its status, but also the context information related to it. With each value in a different field, no parsing is needed to extract the various information about or log as they are all available in dedicated fields.

This is an example of a JSON formatted log:

{
  "timeMillis": 1641027600000,
  "thread": "WrapperListener_stop_runner",
  "level": "INFO",
  "loggerName": "org.mule.runtime.core.internal.logging.LogUtil",
  "message": "\n**********************************************************************\n* Application shut down normally on: 01/01/22    *\n* 12:00                                                              *\n* Up for: 0 days, 1 hours, 1 mins, 1.000 sec                       *\n**********************************************************************",
  "endOfBatch": true,
  "loggerFqcn": "org.apache.logging.slf4j.Log4jLogger",
  "contextMap": {},
  "threadId": 123,
  "threadPriority": 5,
  "appName": "my-test-api",
  "version": "1.0.0-SNAPSHOT",
  "apiType": "my-type",
  "service": "my-test-api"
}

Recap about Log4J2

Log4J2 is a logging framework, a Java library that you can extensively configure to output logs in the format and following the layout that suits your needs.

To do so, you need to configure Log4J2 using the format you prefer, such as properties file, XML, YAML, ... For this example, we will use the XML format but all these instructions can be easily adapted to other configuration formats.

A few definitions before we starts:

Appenders are the various channels which you can use to output your logs. For example, you can use the RollingFileAppender to output logs in a file and rename the file when conditions (named Trigger Policies) are met, or the HttpAppender which sends logs via HTTP to a target URL.
Layouts are defining how a log entry will be output. Log4J2 are offering different layouts, such as the Pattern Layout, which define a text-formatted log, or the JSON Layout that we will use.

Log4J2 documentation is very complete and I strongly recommend you to read it to see everything possible with Log4J2.

Configuring Log4J2

As mentioned earlier, Log4J2 configuration can be written in many languages. In Mule applications, the configuration uses the XML format and is located in src/main/resources/log4j2.xml, the default path for Maven projects.

This files contains the various appenders configured for your application (including their layout) as well as the loggers configuration, which define the level of the various log categories of your application. This is an example of Mule' default Log4J2 configuration:

<Configuration>
    <Appenders>
        <Console name="Console" target="SYSTEM_OUT">
            <PatternLayout pattern="%-5p %d [%t] %c: %m%n"/>
        </Console>
    </Appenders>

    <Loggers>

        <!-- CXF is used heavily by Mule for web services -->
        <AsyncLogger name="org.apache.cxf" level="WARN"/>

        <!-- Apache Commons tend to make a lot of noise which can clutter the log-->
        <AsyncLogger name="org.apache" level="WARN"/>

        <!-- Reduce startup noise -->
        <AsyncLogger name="org.springframework.beans.factory" level="WARN"/>

        <!-- Mule classes -->
        <AsyncLogger name="org.mule" level="INFO"/>
        <AsyncLogger name="com.mulesoft" level="INFO"/>

        <AsyncRoot level="INFO">
            <AppenderRef ref="Console"/>
        </AsyncRoot>
    </Loggers>

</Configuration>

The JSON Layout

In the example above, the Console appender is using the Pattern Layout, which outputs logs as text. We will replace this layout by a JSONLayout to output our logs as JSON:

<Appenders>
    <Console name="Console" target="SYSTEM_OUT">
        <JSONLayout />
    </Console>
</Appenders>

If you run your application and check the console, you will notice that your logs are now formatted in JSON. This JSON is probably taking a lot of lines, so we will configure our appender to output a more compact format, and add a few options to improve our log:

<Appenders>
    <Console name="Console" target="SYSTEM_OUT">
        <JSONLayout compact="true" eventEol="true" properties="true" stacktraceAsString="true" includeTimeMillis="true" />
    </Console>
</Appenders>

Let's detail these new options:

compact will write the JSON log without writing any space or new line between keys. This will make it harder to read but will take much less space in the log file.
eventEol will append an end-of-line character (\n) after each record, writing one log per line in your log file.
properties will add the thread context map in the generated JSON. The thread context map includes the correlation ID, the processor path and every variables added to the logging context using the Tracing Module.
stacktraceAsString will write stacktraces as a single string instead of nested objects. This will simplify integration in log management tools and will make it easier to read as each line will end with the new line caracter (\n), allowing a JSON reader to properly format the stacktrace.
includeTimeMillis will output the log' time as a timestamp with millisecond precision instead of an instant object, allowing log management tools to properly set the time of a log entry.

These settings will ensure that our log takes the minimum amount of space when written while adding all the necessary informations in an easy to parse format.

Adding properties

When using the JSON layout, you can specify additional properties that will be added to your log payload. These properties might be used to add informations about the log that are otherwise unknown to Log4J2 such as the name or version of your application, the type of API it implements (experience, process or system), or any information you might find useful.

Properties can be added as children of the JSONLayout tag by using the KeyValuePair tag:

<JSONLayout compact="true" eventEol="true" properties="true" stacktraceAsString="true" includeTimeMillis="true">
    <KeyValuePair key="appName" value="my-app" />
    <KeyValuePair key="version" value="1.0.0" />
    <KeyValuePair key="appType" value="process" />
</JSONLayout>

Associated with Maven' resource filtering, it is even possible to extract these informations from the project' pom.xml. I will write another article on how to use Maven' filtering soon, but here is an example of what we use internally:

<JSONLayout compact="true" eventEol="true" properties="true" stacktraceAsString="true" includeTimeMillis="true">
    <!--  Project properties -->
    <KeyValuePair key="appName" value="${project.name}" />
    <KeyValuePair key="version" value="${project.version}" />
    <KeyValuePair key="appType" value="${app.type}" />
</JSONLayout>

Writing logs to a JSON file

Now that our logs are formatted in JSON and contains every information we need, we will configure Log4J2 so it outputs logs in a RollingFileAppender. To do this, we will replace our Console appender by the RollingFileAppender and configure it to write logs in a file named my-app.log.json, under the logs folder of our runtime. Files will be rolled once the log file' size reach 10MB, and the last 10 files will be kept on the disk:

<Appenders>
    <RollingFile name="json-file" fileName="${sys:mule.home}${sys:file.separator}logs${sys:file.separator}my-app.log.json"
             filePattern="${sys:mule.home}${sys:file.separator}logs${sys:file.separator}my-app-%i.log.json">
        <JSONLayout compact="true" eventEol="true" properties="true" stacktraceAsString="true" includeTimeMillis="true">
            <!--  Project properties -->
            <KeyValuePair key="appName" value="${project.name}" />
            <KeyValuePair key="version" value="${project.version}" />
            <KeyValuePair key="appType" value="${app.type}" />
        </JSONLayout>
        <SizeBasedTriggeringPolicy size="10 MB" />
        <DefaultRolloverStrategy max="10"/>
    </RollingFile>
</Appenders>

With this appender, logs will be written in $MULE_HOME/logs/my-app.log.json. When a new log is produced by your application, it will be added to the tail of the file. When your log file' size exceed 10 MB, it will be renamed as $MULE_HOME/logs/my-app-1.log.json and a new file $MULE_HOME/logs/my-app.log.json will be created. When the new file is full, $MULE_HOME/logs/my-app-1.log.json is renamed to $MULE_HOME/logs/my-app-2.log.json, $MULE_HOME/logs/my-app.log.json to $MULE_HOME/logs/my-app-1.log.json and a new $MULE_HOME/logs/my-app.log.json is created.

The files will continue to roll until 10 of them exists, at which point the oldest file will be deleted. This is very convenient to keep the most recent files on the disk but avoiding filling your disk while keeping the most recent logs available for debug.

How to easily read JSON files?

If you open one of these JSON log files, you will notice that it is very hard to read. You could copy-paste their content into a JSON beautifier, but this is far from convenient and will make you loose a lot of time.

Instead, I recommend the jq command. This command will allow you to format you JSON as well as filter them and choose the fields you want to see. It can be installed on any machine and makes it very easy to prettify your JSON file:

cat my-app.log.json | jq

Conclusion

Now that you logs are written in JSON log files, they become much easier to integrate into your log management platform. Data can easily be parsed by platforms that usually support JSON logs natively, mapping is much easier as each information is in its own field, and you can add new properties to your JSON logs with only a short line of XML.

And to show you the final result of our setup, this is a Log4J2 file that we use and which rely on JSON logging and Mule resource filtering to produce informative and easy to parse logs picked by Datadog agent:

<?xml version="1.0" encoding="utf-8"?>
<Configuration>
    <Appenders>
        <!-- Human readable logs for easy debugging via SSH -->
        <RollingFile name="file" fileName="${sys:mule.home}${sys:file.separator}logs${sys:file.separator}${project.artifactId}.log"
                 filePattern="${sys:mule.home}${sys:file.separator}logs${sys:file.separator}${project.artifactId}-%i.log">
            <PatternLayout pattern="%-5p %d [%t] [%MDC] %c: %m%n" />
            <SizeBasedTriggeringPolicy size="10 MB" />
            <DefaultRolloverStrategy max="10"/>
        </RollingFile>

        <!-- JSON formatted logs consummed by our Datadog agent -->
        <RollingFile name="json-file" fileName="${sys:mule.home}${sys:file.separator}logs${sys:file.separator}${project.artifactId}.log.json"
                 filePattern="${sys:mule.home}${sys:file.separator}logs${sys:file.separator}${project.artifactId}-%i.log.json">
            <JSONLayout compact="true" eventEol="true" properties="true" stacktraceAsString="true" includeTimeMillis="true">
                <!--  Project properties-->
                <KeyValuePair key="appName" value="${project.name}" />
                <KeyValuePair key="version" value="${project.version}" />
                <KeyValuePair key="appType" value="${app.type}" />
                <!--  Datadog log ingestion properties -->
                <KeyValuePair key="service" value="${project.artifactId}" />
            </JSONLayout>
            <SizeBasedTriggeringPolicy size="10 MB" />
            <DefaultRolloverStrategy max="10"/>
        </RollingFile>
    </Appenders>
    <Loggers>
        <!-- Http Logger shows wire traffic on DEBUG. -->
        <!--AsyncLogger name="org.mule.service.http.impl.service.HttpMessageLogger" level="DEBUG" />-->
        <AsyncLogger name="org.mule.service.http" level="WARN"/>
        <AsyncLogger name="org.mule.extension.http" level="WARN"/>

        <!-- Mule logger -->
        <AsyncLogger name="org.mule.runtime.core.internal.processor.LoggerMessageProcessor" level="INFO"/>

        <AsyncRoot level="INFO">
            <AppenderRef ref="file" />
            <AppenderRef ref="json-file" />
        </AsyncRoot>
    </Loggers>
</Configuration>

Mulesoft Clusters not synchronizing: tips to solve common issue

Gauthier POGAM--LE MONTAGNER — Thu, 06 Jan 2022 17:25:58 +0000

To ensure reliability, we use Mule Clusters to synchronize our runtimes and ensure high availability between our applications.

Contrary to Server Groups which only deploy applications on all servers, Clusters also allow the runtimes to synchronize with each other. When clustered, their in-memory caches are synchronized and the schedulers are run only on the primary node.

After upgrading to Mule 4.4, we noticed that our schedulers were run on every runtimes in our cluster, resulting in the same operation being processed multiple times when the scheduler would run. The cluster had the same behaviour as Server Groups: they were not synchronized and schedulers and caches were managed individually on each node.

Understanding the issue: clusters logs

By default, Mule runtimes do not output logs related to clusters. If there is an issue, the runtime will fail silently to create the cluster and you will have no way to know there is a problem as nothing is reported in the Runtime Manager either.

To help you find the root cause, you can enable cluster logs from your runtime Log4J2 configuration (located in $MULE_HOME/conf/log4j2.xml):

<AsyncLogger name="com.mulesoft.mule.runtime.module.cluster" level="DEBUG"/>
<AsyncLogger name="com.hazelcast.internal.cluster" level="DEBUG"/>

By adding these two lines in the Loggers section of your configuration, the cluster logs will be written in $MULE_HOME/logs/mule_ee.log. The setup of the cluster is one of the first operation done at runtime startup; you can find the corresponding log entries just after the runtime boots up.

Link to official documentation.

`$MULE_HOME/.mule/mule-cluster.properties`

If the cluster was not properly setup for some reason, the cluster configuration file might not be created. In this case, you will have an error saying that $MULE_HOME/.mule/mule-cluster.properties could not be found.

This problem is a bit tricky because the file cannot be regenerated without editing the cluster, or created by hand if you manage your cluster using the runtime manager.

First, verify if the file $MULE_HOME/.mule/mule-cluster.properties does exists and if not, make sure the runtime has the rights to write in $MULE_HOME and $MULE_HOME/.mule/.

If the file is missing, we must trigger an update of the cluster configuration, which will rewrite mule-cluster.properties on every nodes of the cluster. To do so, you must change the number of nodes in the cluster. You can do it in two ways:

Install the runtime on a new server and add it to the cluster. This is time consuming if you don't have any server ready and no automation to install the runtime.
Remove a server from the cluster and add it back. This is the preferred solution but it will not avoid downtime.

Let's see how to proceed with the second solution:

First, you need to make sure there is at least 2 nodes up in your cluster with your application deployed.
In the runtime manager, go in the Servers page and click on your cluster's name to open its settings.
In the server list, click on the server you would like to remove (we will add it back in a second).
At the top of the opened page, click on Remove Server From Cluster. The server you removed and the other nodes in the cluster will now restart and the cluster status will be in the status Created if there is only one node in it.

You will now have to wait for a few minutes so every applications restart properly in the cluster. You will notice that no application is deployed on the node your just removed from the cluster.

In the meantime, you can SSH into your cluster' nodes and check whether $MULE_HOME/.mule/mule-cluster.properties has been created properly. If so, you can open it and will find the property mule.cluster.nodes listing all nodes in your cluster.

When every application has been restarted in the cluster, you can add the node back to the cluster. This will trigger a new restart of the applications, but your nodes should now be able to communicate with each other.

Ensuring the nodes can communicate

By default, all nodes should be on the same network and will use ports 5701, 5702 and 5703 to communicate with each others. If you have an error stating that the clusters could not communicate with each other, the two runtimes might not have the rights to call each other.

Make sure that all runtimes in your cluster fulfil the pre-requites and reach to your network team if something is not working properly.

What's next?

This small guide tries to save your some time but does not cover every issues you might encounter with your cluster. If you still have issues with your cluster, contact Mulesoft support: they know their product much more than I do!

DataWeave: accept lambda as parameter of a function

Gauthier POGAM--LE MONTAGNER — Mon, 03 Jan 2022 14:07:33 +0000

When writing DataWeave code, it is common to use functions that take a lambda as parameter, such as map or flatten. Using lambda in DataWeave is well documented and easy to understand for developers that already worked with functional programming in other languages.

However, creating a DataWeave function that accepts a lambda as parameter, while relatively easy, is not well documented in Mulesoft' documentation.

As creating functions taking lambda as parameters can be very useful, I will explain how to pass a lambda as parameter of a function. Common use cases are to accept a callback or apply a transformation or a predicate passed as a parameter.

What is a lambda?

In functional programming, data are immutable and thus cannot be modified. To change a value, we do not modify it directly. Instead, the value is processed and a new value defined.

%dw 2.0
var initialValue = 21
// Error: a variable cannot be modified after its declaration
var newValue = 42
// Success: a new variable is created containing the new value.
var newValue = 42 * 2

Arrays follow the same principle and their values cannot be modified. Instead, we will use functions that applies some behaviour to each element. This behaviour is defined in a lambda, a function without a name that can be saved in a variable or passed as a parameter.

%dw 2.0
var initialValue = [1, 2, 3]

var newValue = initialValue map ((value) -> value * 2)
newValue // [2, 4, 6]

In the example above, (value) -> value * 2 is a lambda that takes the existing value and multiply it by 2. This function is passed as a parameter of map.

DataWeave allows a lambda to you anonymous parameters, and to be passed without being surrounded by variables. This shorten the code and can make it more clear. The following line does the same thing as its equivalent above:

var newValue = initialValue map $ * 2

For more information, refer to Mulesoft's documentation.

Taking lambdas as parameters

To write a function that accepts a lambda as parameter, you need two things:

Define the parameter and type it so it is clear it accepts a lambda.
Call the lambda and give it the required parameters.

To define and type the parameter, you must specify the parameter name and the signature of the function it accepts:

helloNewYear(lambda: (param1: String, param2: Number) -> String) = ...

In this example, we declare a function helloNewYear that accepts one parameter: a lambda. This lambda should be a function accepting a String and a Number as parameter, and returning a String.

Once we have our parameter, we need to call it. It is called in the same way as any other function, taking the parameter name as function name:

helloNewYear(lambda: (param1: String, param2: Number) -> String) = lambda("Hello", 2022)

Now, we can call our function and pass it the lambda we want to execute:

helloNewYear((sentence, year) -> sentence ++ " " ++ year)

This will output "Hello 2022". Your lambda can contain any behaviour you wish to define.

Code Example

%dw 2.0

// Processing function that takes a lambda as parameter
fun updateEvenCharacters(
    str: String,
    // The type of the parameter is the expected signature of the lambda to call
    mapper: (str: String, num: Number) -> String
): String =
    str splitBy ""
        // The lambda is called as a normal function, with the parameter name being the function name
        map (if (isEven($$)) mapper($, $$) else $)
        joinBy ""

output text/plain
---
// Call the processing function and pass a lambda as second parameter 
updateEvenCharacters(
    "Hello World!",
    // The parameter should be the lambda to call.
    (value, index) ->  upper(value) ++ index
)