DEV Community: Sergio Matone

OpenTelemetry - stepping into gno.land's observability tools

Sergio Matone — Thu, 11 Dec 2025 16:05:02 +0000

Disclamer: this post is a showcase originally published in Gnops.io and it's reported here as is

What is OTel

OpenTelemetry (OTel) is a specification that translates into APIs and SDKs for collecting and analyzing telemetry data by instrumenting code of a target application.
Telemetries are intended in the form of three types:

Metrics, a measurement captured at runtime.
Traces, the path of a request through the application.
Logs, a recording of an event.

Code should be instrumented, meaning making it capable of emitting telemetry data. This can be:

code-based, explicitly adopting a specific SDK and adding specific instructions into the codebase
zero-code, when the code cannot be modified, by providing information about what’s happening at the edges of the application

The relevant components in the OpenTelemetry landscapes are:

Data: Defined in the OpenTelemetry Protocol (OTLP) and with vendor-agnostic semantic conventions
OpenTelemetry Collector: a vendor-agnostic proxy that can receive, process, and export telemetry data. Data can be in multiple formats (OTLP, Jaeger, Prometheus) and can be exported to one or more backends
Observability Backends: Specific services able to elaborate observability data using proprietary technologies (e.g. Prometheus, Jaeger, Zipkin)

All this data facilitates the analysis of the performances of a specific instrumented application, allowing the observability.

Why OTel

OpenTelemetry is a free and open source technology and it does not depend on any other system, service or technology, in this way we can maximize the possibilities of integration with other services and technologies and minimize or totally avoid the vendor lock-in.

OTel in Gnoland

First of all, the current version of Gnoland will focus on OpenTelemetry metrics, traces may come next in a short time.
All the gno.land code has been actively instrumented to collect a predefined set of metrics, mainly counters and histograms and some gauges. the code has been explicitly instrumented using the official Go SDK.

The reference file setting up OpenTelemetry in the official Gno repository is at tm2/pkg/telemetry/metrics/metrics.go.
This holds an init method which is in charge of creating the metric provider, which in turn will be collecting the instrumented metrics and exporting them periodically to either an http or a grpc endpoint.

Here the relevant source code:

//Exporter setup
func Init(config config.Config) error {
 ...
  // Use oltp metric exporter with http/https or grpc
  switch u.Scheme {
  case "http", "https":
    exp, err = otlpmetrichttp.New(
      ctx,
      otlpmetrichttp.WithEndpointURL(config.ExporterEndpoint),
    )
    if err != nil {
      return fmt.Errorf("unable to create http metrics exporter, %w", err)
    }
  default:
    exp, err = otlpmetricgrpc.New(
      ctx,
      otlpmetricgrpc.WithEndpoint(config.ExporterEndpoint),
      otlpmetricgrpc.WithInsecure(),
    )
    if err != nil {
      return fmt.Errorf("unable to create grpc metrics exporter, %w", err)
    }
  }

  // creating metric provider
  provider := sdkMetric.NewMeterProvider(
    // Default period is 1m
    sdkMetric.WithReader(sdkMetric.NewPeriodicReader(exp)),
    sdkMetric.WithResource(
      resource.NewWithAttributes(
        semconv.SchemaURL,
        semconv.ServiceNameKey.String(config.ServiceName),
        semconv.ServiceVersionKey.String("1.0.0"),
        semconv.ServiceInstanceIDKey.String(config.ServiceInstanceID),
      ),
    ),
  )

  // instrument OTel SDK
  OTel.SetMeterProvider(provider)
  meter := provider.Meter(config.MeterName)
  //...
}

Throughout the instrumentation part, convention over configuration has been heavily adopted. For each metric a const is defined, the latter is used as key to instrument the metric, which in turn is saved into an exported var, which represents the metric handler.

const numMempoolTxsKey = "num_mempool_txs_hist"

var NumMempoolTxs metric.Int64Histogram

if NumMempoolTxs, err = meter.Int64Histogram(
  numMempoolTxsKey,
  metric.WithDescription("valid mempool transaction count"),
); err != nil {
  return fmt.Errorf("unable to create histogram, %w", err)
}

The instrumented item will be used somewhere else within the codebase to collect the corresponding metric.

// logTelemetry logs the mempool telemetry
func (mem *CListMempool) logTelemetry() {
if !telemetry.MetricsEnabled() {
  return
}

// Log the total number of mempool transactions
metrics.NumMempoolTxs.Record(context.Background(), int64(mem.txs.Len()))
}

Note that if the telemetry is not explicitly enabled, no attempt to access the metric handler will be made.

Gnoland Binary setup

Since relevant parts of code have been already instrumented, what is missing is setting up the relevant section of OpenTelemetry in the configuration file.

The following configuration items are available in Gno setup:

telemetry.enabled, whether telemetry is enabled
telemtery.exporter_endpoint, the endpoint to export metrics to
telemtetry.meter_name, name of the meter, meaning the entity the will create instruments for the application
telemetry.service_instance_id, an instance id to identify the current instance emitting the metrics
telemetry.service_name, a generic name to identify the service running, for example a testnet or a validator set

All the previous name references will be included in each metric emitted by the Gnoland application.

OpenTelemetry in action: Orchestrating multiple services in Docker

In order to see OpenTelemetry in action we need

a set of observability tools
an instrumentable application, Gnoland!

To the first group belong:

OpenTelemetry collector
Prometheus as backend, receiving data exported by the OTel Collector
Grafana as UI to visualize results in a set of visual panels, configured to elaborate data collected by Prometheus

For the application part we will use:

a Gno validator that starts producing blocks
Supernova, the stress-testing Gno networks tool, to generate some traffic as transactions created (package deployment)
an RPC node, used as handy communication tier between the Gno validator and Supernova

The complete docker compose file for the orchestrated services can be found in the misc/telemetry directory on GitHub.

You can launch the services using from the Gno repository folder:

cd misc/telemetry
docker compose up -d

Instrumenting applications

When using the Gnoland binary, instrumenting the application is just one step to the side. As said before it is important to just enable telemetry from config, the telemetry.enabled entry, providing an endpoint to the collector, telemetry.exporter_endpoint.
Then to identify a current set of applications, like a validator set, a generic service name can be added via telemetry.service_name, together with the unique name given the specific instance providing metrics, using telemetry.service_instance_id.
These last two parameters will be included as labels in each metric emitted by the instrumented application.

The validator node will generate secrets, adjust config to enable telemetry and produce labeled metrics and it will also generate a genesis file having the validator itself within the validator set entries.

The RPC service will wait for the validator node to be up and running and share the genesis file to boot itself. It will not only be useful as the endpoint for Supernova, but also to produce other OTel metrics, labeled with a different source service (telemetry.service_instance_id = rpc000).

Periodically Supernova will rerun and by connecting through the RPC service it will increase the amount of data that the observability stack can consume. It is possible to manually stop Supernova to avoid running it infinitely.

Configuring the observability stack

As seen above the key component of the OTel observability environment is the collector. It is in charge of communicating from one side with the instrumented applications and from the other side with a backend which receives exported data.

We can put together what we have seen before, our collector setup will be able to:

gather metrics from a grpc interface exposed by the collector itself and accessed by the application
collect only metrics, no traces or logs
expose data to a Prometheus backend on a given port (in this case we picked port 8090)

Here is the simple and final configuration of the collector

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:

exporters:
  prometheus:
    endpoint: collector:8090

service:
  telemetry:
    logs:
      level: "debug"
  pipelines:
    metrics:
      receivers: [ otlp ]
      processors: [ batch ]
      exporters: [ prometheus ]

Prometheus will receive data from the collector. The only configuration needed is a dedicated scraper connecting to the collector using the endpoint where Prometheus is expecting data to be gathered from. In this specific case we will use a local service name with the designed port, so the final endpoint will be: collector:8090.

scrape_configs:
  - job_name: 'OpenTelemetry'
    static_configs:
      - targets: [ 'collector:8090' ]

Grafana is the last and final tier and it will in turn gather data from Prometheus at a specific port (predefined is 9090), by configuring a specific datasouce. Then this data should be combined in a meaningful way to be shown in several UI panels. Eventually those panels are combined together into a dashboard, the OTel dashboard in this case.

datasources:
  - name: Prometheus
    type: prometheus
    url: http://prometheus:9090
    uid: prometheus
    ...

The metrics exposed can be used and combined in various ways and depending on the metric type, they are provided with additional data that allows meaningful aggregations.

For example a metric of type histogram will expose three metrics named using convention over configuration suffixes:

_count, counting the number of samples
_sum, summing up the value of all samples
a set of multiple buckets _bucket with a label le, which contains a count of all samples whose values are less than or equal to the numeric value contained in the le label.

These data can be combined together using specific expressions. One of them can be useful to understand the trend of values in a given period of time by leveraging the formula rate(hist_sum)/rate(hist_count).

For example, referencing the metric vm_gas_used_hist collecting the gas used by VM executions in Gno, can be aggregated to retrieve the average gas used, with the following formula:

rate(vm_gas_used_hist_sum)/rate(vm_gas_used_hist_count)

Indeed for a gauge metric, like inbound_peers_gauge, that holds the number of inbound peers for a validator, we can use another common expression using the avg_over_time() function to obtain the average inbound peer count.

avg_over_time(inbound_peers_gauge)

Of course it is useful to add additional parameters to the previous expressions to define a time range and pointing to a specific application instance, represented by the node parameter. The final expression looks like that:

rate(vm_gas_used_hist_sum{exported_instance=~"${node}"}[$__rate_interval])/rate(vm_gas_used_hist_count{exported_instance=~"${node}"}[$__rate_interval])

As said node is s reference variable defined in Grafana, which queries Prometheus with the following expression:

label_values(exported_instance)

where exported_instance is a label contained in each metric and defined and added by the instrumented application. In this specific case we can distinguish metrics referring to the validator node and to the RPC node.

Moving resource into a Kubernetes cluster

After creating the setup in a Docker Compose environment, moving toward Kubernetes is the next expected step.
Let's see what we have and what we need to adapt to make it working in a Kubernetes environment.

OpenTelemetry collector: this will become a fresh pair of deployment/service combo in the Kube

apiVersion: apps/v1
kind: Deployment
metadata:
  name: open-telemetry
spec:
  replicas: 1
  selector:
    matchLabels:
      app: otel-run
  template:
    metadata:
      labels:
        app: otel-run
    spec:
      containers:
        - name: otel
          image: otel/OpenTelemetry-collector-contrib:0.111.0
          ports:
            - name: otlp-http
              containerPort: 4317
            - name: otlp-grpc
              containerPort: 4318
            - name: otlp-exporter
              containerPort: 8889
...

---

apiVersion: v1
kind: Service
metadata:
  name: otel-svc
  namespace: monitoring
spec:
  selector:
    app: otel-run
  ports:
    - name: otlp-http
      port: 4317
      targetPort: otlp-http
    - name: otlp-exporter
      port: 8889
      targetPort: otlp-exporter

Prometheus: given a deployed Prometheus stack in Kubernetes (deployment + service) it only needs to be provided with the right scrape config, which should point to the endpoint made available by the OTel collector exposed by the Kubernetes service

  - job_name: 'OpenTelemetry'
    static_configs:
      - targets: [ 'collector:8090' ]

Grafana: what is needed in a common Grafana configuration provided with a Prometheus datasource, is the addition of the dedicated dashboard into the list of available ones. The dashboard is not far from what was configured in the Docker Compose version.
As long as the Prometheus datasource is available, the same dashboard configuration will display the UI panels the same way.
Gnoland binary: what is needed is just a valid configuration enabling OTel and pointing to the right endpoint of the OTel collector.
This configuration should be provided at boot time, in Kubernetes this is possible by leveraging an initContainer.
In particular it is important to point to the right Kubernetes service representing the OTel collector. In order to follow the above configuration the endpoint will look like: otel-svc.svc.cluster.local:4317.

...
initContainers:
- name: init-config
  image: ghcr.io/gnolang/gno/gnoland:{{ .Values.global.binaryVersion }}
  command:
  - sh
  - -c
  - |
    gnoland config set telemetry.enabled true
    gnoland config set telemetry.service_instance_id val-01
    gnoland config set telemetry.exporter_endpoint otel-svc.svc.cluster.local:4317
...

Conclusion

We have seen here why and how Open Telemetry was adopted in Gnoland. Moreover we have been able to introduce OTel into the observability stack,
first in a simple Docker Compose and then in Kubernetes.
Finally OpenTelemetry became part of the observability stack in Gnoland!

Built-in min() and max() methods in Go 1.21

Sergio Matone — Wed, 24 Apr 2024 08:18:59 +0000

I recently realized that in Go 1.21 the built-in methods min() and max() where introduced, which allows you to get the min and max of a set of heterogeneous items. This is now doable thanks to the introduction of Generics in Go 1.18.

So, for example when I have a string, let's say a title, and I want to get the substring which has a length gathered from the minimum value between a given number and the length of the string itself. Once I would have used the math.Min method, which requires two float64 types as parameters, and I would have done the following:

const MAX_TITLE_LENGTH int = 50

// a placeholder title
var title string = "..." // the given title

titleMaxLength := int( 
  math.Min( float64(MAX_TITLE_LENGTH), float64(len(title)) ) 
)

croppedTitle := title[:titleMaxLength]

Whereas now I can achieve the same by simplifying using built-in min() method, without casting over and over the operands:

titleMaxLength := min( MAX_TITLE_LENGTH, len((title)) )

If we check the definition of the method, there is another important thing to notice, the method is not limited to a pair of parameters, but it accepts an unlimited number of (homogeneous) parameters.

package builtin
...
func min[T cmp.Ordered](x T, y ...T) T

Note:
Even though we are already at Go 1.22, I realized a little late that this feature was actually introduced in Go 1.21. What should I say about that? It's better late than never!

Containerizing AOM AV1 and encoding videos with FFmpeg in an Node.js app

Sergio Matone — Mon, 27 Nov 2023 15:50:33 +0000

I have recently had the opportunity to work and study the AV1 Codec by AOM (Alliance for Open Media). It is the state-of-the-art video codec for manipulating videos today, and it is stunning how it is able to compress videos producing files with reduced size and a pretty much untouched quality (I will later do a small comparison with the ancestor AVC/H.264 by MPEG).
And since I am a big, better huge fan of FFmpeg I cannot step back from compiling a fully working FFmpeg binary within a container image. Later I will describe how I achieved that and how it can be used within a Node.js application.

You can check out the source code in GitHub.

A bit of history

The creation of the AOM group is a sort of strong answer to the last video codecs standardized by the MPEG group.
Starting from the end of the previous millennium, the standardization of audio and video formats and codecs was dominated by the MPEG group, with the huge success of the MP3 standard (aka MPEG-1 layer 3) for audio, followed by the other great success of the AVC codec for videos, created jointly with ITU.T (which internally nominated the standard H.264 and used to commonly refer to it, instead of AVC).

The licensing model of codecs by MPEG started to have some critics already with AVC, but it reached its peak with the upcoming HEVC standard (aka H.265) whose licensing model appeared unacceptable for many players of the video & internet industry led by Google.

The result was the creation of the new standardization group Alliance for Open Media (AOM), whose main focus was creating the next optimize royalty-free video codec: AV1.

Containerizing AV1 with FFmpeg in a Docker IM

Evaluating the performance of the AV1 codec would be almost impossible without leveraging FFmpeg, the state-of-the-art and reference software to manipulate, compress and low-level edit of videos, the software is also the reference for big Video streaming platforms like Netflix, Disney+ and YouTube.

// Detect dark theme var iframe = document.getElementById('tweet-1710440696941809868-840'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1710440696941809868&theme=dark" }

And creating a portable solution would be not possible as well without building a Container Image in Docker which allows pin versions of libraries and reuse, update and run them without taking care to the current system configuration.

Dockerfile for FFmpeg + AV1

To achieve a fully-functional FFmpeg application from Docker, let's start from an Ubuntu Image and then build on top of it all the single codecs and libraries.
In this case it is very useful to deal with a Docker Image since versions of libraries and download url counterparts can be accumulated at the beginning of the Dockerfile and eventually overwritten via build arguments.

Most libraries during compilation and build, will write information that FFmpeg compilation step will then retrieve using pkg-config.
The following codecs libraries will be downloaded and compiled statically, each reporting in parenthesis the flag to enable it during FFmpeg building from source step.

libfreetype (--enable-libfreetype)
libx264 (--enable-libx264)
libx265 (--enable-libx264)
libfdk-aac (--enable-libfdk-aac)
liblame-mp3 (--enable-libmp3lame)
libaom (--enable-libaom)
libsvtav1 (--enable-libsvtav1)

For AV1 both libaom and libsvtav1 will be built. However according to FFmpeg documentation about AV1, SVT-AV1 is now the reference for AV1 encoding

SVT-AV1 was adopted by AOMedia as the basis for the future development of AV1 as well as future codec efforts.

After all these steps in the Dockerfile, it is possible to proceed with the download and build of Ffmpeg from source code (I picked a pinned version from a git tag: 5.1), enabling with flags the previously compiled codecs.
As for the other libraries compiled in previous steps, the configure, make, make install will happen in a single RUN step.

RUN git clone --depth 1 --branch n5.1 https://git.ffmpeg.org/ffmpeg.git ffmpeg && cd ffmpeg && \
  PKG_CONFIG_PATH="$BUILD_PREFIX/lib/pkgconfig" ./configure \
  --prefix="$BUILD_PREFIX" \
  --pkg-config-flags="--static" \
  --extra-cflags="-I$BUILD_PREFIX/include" \
  --extra-ldflags="-L$BUILD_PREFIX/lib -L$BUILD_PREFIX/lib64" \
  --extra-libs="-lpthread -lm -lz" \
  --extra-ldexeflags="-static" \
  --bindir="$HOME/bin" \
  --enable-gpl \
  --disable-shared \
  --enable-static \
  --enable-nonfree \
  --disable-libfreetype \
  --enable-libfdk-aac \
  --enable-libmp3lame \
  --enable-libx264 \
  --enable-libx265 \
  --enable-libaom \
  --enable-libsvtav1 \
  --disable-doc \
  --disable-podpages \
  --disable-debug \
  && \
  make -j 10 && \
  make install

Note for SVT-AV1

In order to add the compiled libsvtav1 library to the FFmpeg build, some patches provided by libsvtav1 library itself should be applied to the FFmpeg source code, so an additional RUN step has been added.

RUN git clone --depth 1 --branch n5.1 https://git.ffmpeg.org/ffmpeg.git ffmpeg && \
  git config --global user.email "media@minimalgap.com" && \
  git -C ffmpeg am $BUILD_PREFIX/ffmpeg_plugin/n5.1/*.patch

The output of this image is a layer with a compiled and working FFmpeg (and FFprobe) binary ready to use.

Using FFmpeg binary

Now that we have a working Docker image, we can build on top of that other applications that can easily leverage FFmpeg (or its sibling FFprobe) binary.
Again we will take advantage of Docker in particular of multi-stage build.

I've created a basic Node.js application whose purpose is to create a compressed video encoded with AV1 and place it in a specific folder with
.mp4 extension starting from command line and an input video and specifying a codec among AV1, H.264 and HEVC.

Of course this application is a POC and can be expanded with any other possible option.

The Dockerfile for the Node.js will include the FFmpeg binary via the COPY directive. I have a used a very common pattern that Docker enables using multi stage build: the COPY directive has a peculiarity clearly stated in the official Dockerfile reference doc:

COPY accepts a flag --from=<name> that can be used to set the source location to a previous build stage (created with FROM .. AS <name>) that will be used instead of a build context sent by the user. In case a build stage with a specified name can't be found an image with the same name is attempted to be used instead.

I have also included a docker-compose version where both the images can be built in a single shot.

Gotcha - BUILDKIT

Since BuildKit is enabled by default in the newer versions of Docker Engine, when building a compose file with multiple services it is not guaranteed that the order of appearance in the compose file is respected because of BuildKit itself attempting to optimize the build process.

To overcome this problem it is possible to:

Disable BuildKit

DOCKER_BUILDKIT=0 docker-compose build

Alternatively build ffmpeg image separately and then the remaining services

  docker-compose build ffmpeg
  docker-compose build

AOM AV1 vs MPEG HEVC [aka H.265] vs MPEG AVC [aka H.264]

The application previously created and build is able to encode videos from command line, for sake of simplicity it will allow only two options:

-i for input file path
-codec for video codec to be picked among av1, hevc, h264

Again to keep it simple each audio will be encoded using HE_AAC audio codec.

Whereas for each video codec there is an hardcoded configuration:

for AV1: -c:v libsvtav1 -pix_fmt yuv420p -crf 45 -preset 7 -svtav1-params fast-decode=1:film-grain=7:tune=0
for HEVC: libx265 -pix_fmt yuv420p -profile:v main -preset slower -crf 27
for H.264: libx264 -pix_fmt yuv420p -profile:v high -level 4.1 -preset slower -crf 22 -tune film

The purpose here is just to provide a quick way to compare results of videos processed with each codec both visually and with respect to file size and CPU time to process a video.

Notes on CRF

Constant Rate Factor (CRF) is a rate control mode that allows the encoder to attempt to achieve a certain output quality for the whole file when output file size is of less importance.

for AV1 [libsvtav1]: the valid CRF value range is 0-63, with the default being 50. Lower values correspond to higher quality and greater file size
for HEVC [libx265]: the mode works exactly the same as in libx264, except that maximum value is always 51, even with 10-bit support. The default is 28, and it should visually correspond to libx264 video at CRF 23
for H.264 [libx264]: The range of the CRF scale is 0–51, where 0 is lossless (for 8 bit only, for 10 bit use -qp 0), 23 is the default, and 51 is worst quality possible

Making a deep comparison among the results achieved from the 3 different codecs is out of the scope of this post. It will also be unfair, since all the three codecs have plenty of options and it's not easy to achieve three identical and fair configurations to make a real comparisons.
Moreover the results would be strongly influenced by the input video and its codecs configuration.

I will here highlight the main feedbacks so far:

AV1 is in general the slower to process but it produces the smallest sized output file.
H.264/AVC is the fastest one but it produces the output file with the biggest size.
HEVC is as slow as AV1 but it tends to create bigger sized files (this is a guess because it is strongly influenced by codec configuration)
AV1 apparently produce amazingly high quality video even if compression is at a very high rate

Test Video 1: small sized video

Source Video
Video codec: AVC/H264
Duration: ~ 20 s
File size: 54 MB
Ffprobe Output:

Duration: 00:00:18.88, start: 0.000000, bitrate: 23842 kb/s
Stream #0:0: Video: h264 (High) (avc1 / 0x31637661), yuvj420p(pc, bt709, progressive), 1920x1080, 23673 kb/s, 25 fps, 25 tbr, 25k tbn (default)
Stream #0:1: Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 127 kb/s (default)

HW: Mac Book Pro 2018, CPU Intel i7 2,6 GHz 6-cores, RAM 32 GB

Codec	File Size	CPU Time
AV1	2.5 MB	38 s
HEVC	5.7 MB	464 s
H.264	13.3 MB	24 s

HW: Linux Station, CPU Intel Xeon Gold 6230 2,1 GHz 8-cores, RAM 16 GB

Codec	File Size	CPU Time
AV1	2.5 MB	58 s
HEVC	5.7 MB	328 s
H.264	13.3 MB	40 s

Test Video 2: raw video

Source Video
Video codec: N/A - Raw video
Duration: ~ 4.30 min
File size: 6.3 GB
Ffprobe Output:

Duration: 00:00:04.23, start: 0.000000, bitrate: 12740202 kb/s
Stream #0:0: Video: rawvideo, yuv420p10le(progressive), 4096x2160, SAR 1:1 DAR 256:135, 60 fps, 60 tbr, 60 tbn

HW: Mac Book Pro 2018, CPU Intel i7 2,6 GHz 6-cores, RAM 32 GB

Codec	File Size	CPU Time
AV1	3 MB	136 s
HEVC	4 MB	780 s
H.264	15.4 MB	85 s

HW: Linux Station, CPU Intel Xeon Gold 6230 2,1 GHz 8-cores, RAM 16 GB

Codec	File Size	CPU Time
AV1	3 MB	159 s
HEVC	4 MB	625 s
H.264	15.4 MB	84 s

Recap

As said the purpose of these compression tests is not a deep comparison among codec results and parameters.
From the previous tests I can state that AV1 is giving back strongly compressed videos with a long processing duration compared to H.264. It is also giving similar results to HEVC but with a faster compression time with and comparable or smaller sized videos.
Compared to H.264/AVC, the codec AV1 is taking an inversely proportional CPU time compared to order of magnitude in terms of reduced size.

Check out the source code in GitHub.

References

Handling Rate Limits in GetStream API Calls in Go leveraging Channels and Goroutines

Sergio Matone — Tue, 10 Oct 2023 11:10:59 +0000

TL; DR

This is my own solution to overcome rate limits for GetStream's API Endpoints when employing Go SDK.

GetStream is a great SaaS offering Chat & Feed services and a wide variety of SDKs acting as clients to handle those items and their users as well.
Speaking of chats in GetStream, I faced a severe problem in one of the projects I worked on: Rate Limits in API calls.
Indeed GetStream requires the various clients to perform their requests respecting a specific rate limiting in requesting the endpoints available, such as QueryChannels, QueryUsers, CreateChannel. Limits for the different endpoints are listed here

It is not easy to deal with these boundaries, especially when you create batch tasks, because we tend to imagine and code big loops to apply multiple requests to a set of homogeneous data.
In my case, in the startup I was working for, we had a task to handle a large set of groups, each corresponding to a chat group in GetStream; participants in these groups used to be added or removed by a sort of group's administrator. Since this might not be reflected real-time in the chat groups, there was a nightly task that basically checks for each user whether he should be added to or removed from a chat group.

Probably watching the problem upside down, it would be more clever to check all the existing chat groups instead of every user, but this is not the point here. Let's suppose this is the best implementation we could use.

Sometimes the task was limited to a small bunch of changes, but most of the time many users were repeatedly added and removed from groups. And that made the task failing often because after several requests it started to time out over and over for all the following ones.

In order to find a solution there are a couple things to take into account:

Get Stream is setting in the response the headers that specify for each kind of api (remember that limits are tied to each specific endpoint): which are the defined limits (X-RateLimit-Limit), how many requests are still available (X-RateLimit-Remaining) and when eventually the block for hitting rate limit will be reset (X-RateLimit-Timestamp) in Unix timestamp format.
The task needs a way to distinguish among each of all the (employed) Api endpoints, whether the timeout has been reached, and in case it is, a way to hold all the following requests for a given time.

All of that makes me think about generating a sort of barrier for the Api requests that will arrive after the rate limit for the given endpoint is hit. There should be multiple barriers, one for each endpoint and they should be able to block following requests for a specific amount of time given. I named that timed-barrier.

Being inside a Go environment I guessed that Channels and Goroutines would be a natural means to deal with the issue. I achieved distinguishing among the different Api endpoints creating a Timed-barrier reference struct.

type RateLimiter struct {
  apiName string
  token   chan struct{}
}

The RateLimiter struct represents the Timed-barrier having an Api Endpoint reference and a buffered channel with capacity 1.

☠️ Initially I thought about an unbuffered channel. Since the scenario is not aiming to coordinate two Goroutines against each other but a group of them, it would have created an immediate deadlock.

The buffered channel will be the trick to overcome the rate limits issue. Indeed the first call will occupy the single slot available in the channel and it will release it after calling the API, depending on the return value of current rate limits. If the value of the remaining calls is greater than zero, it will release the channel's slot immediately, otherwise it will wait until the timeout has expired.

But let see this in details by analyzing the RateLimiter's method CallApiAndBlockOnRateLimit:

func (r *RateLimiter) CallApiAndBlockOnRateLimit(logger *log.Logger, apiCall GetStreamApiCaller) error {
  r.token <- struct{}{}
  // Injected api call
  resp, err := apiCall()
  if err != nil {
    <-r.token
    return err
  }
  if resp.RateLimitInfo.Remaining == 0 {
    go func(duration int64) {
      start := time.Now()
      time.Sleep((time.Second * time.Duration(duration-start.Unix())).Abs())
      <-r.token
    }(resp.RateLimitInfo.Reset) // <-- when the current limit will reset (Unix timestamp in seconds)
  } else {
    <-r.token
  }
  return nil
}

The first step is occupying the single slot in the buffered channel. It means that any subsequent call will be blocked waiting for an available slot in the channel.
The Api endpoint is called leveraging a clojure or higher-order function type, such as

type GetStreamApiCaller func() (resp *stream.Response, err error)

that is basically supposed to call a given GetStream endpoint, which would place the response in a specific type, according to the Api called. The role of the function is extracting a general data structure from the specific response type.
That general response type is stream.Response which is defined as follows:

type Response struct {
  RateLimitInfo *RateLimitInfo `json:"ratelimit"`
}

type RateLimitInfo struct {
  // Limit is the maximum number of API calls for a single time window (1 minute).
  Limit int64 `json:"limit"`
  // Remaining is the number of API calls remaining in the current time window (1 minute).
  Remaining int64 `json:"remaining"`
  // Reset is the Unix timestamp of the expiration of the current rate limit time window.
  Reset int64 `json:"reset"`
}

It is clear that this is exactly the key of the method implementation.
The RateLimitInfo contains all the useful information to proceed.
The Remaining field will inform if any call is still available for the given Api endpoint. If it is, the channel is freed immediately, otherwise the channel will remain occupied until a specific time is passed. This is achieved leveraging the Reset field which will hold the slot within a Goroutine, until the given time (duration given as Unix timestamp in seconds) has passed, by just sleeping.
Moreover even if in a group of homogeneous and distinguished calls order may not be important, blocking all the subsequent calls waiting for the slot of the channel to be empty, will guarantee consistency in the order in which they will be called after the channel slot is released.

All you have to do is create a distinct RateLimiter object for each kind of Api Endpoint in GetStream.

Library in action

In a real world scenario, given a GetStream chat client:

  getStreamChatClient, err := stream.NewClient("<API_KEY>", "<API_SECRET>")

And a distinct set of RateLimiter objects, collected in a map having as key type a GetStream Api Endpoint:

type GetStreamApiName string

const (
  CreateChannel GetStreamApiName = "CreateChannel"
  QueryChannel  GetStreamApiName = "QueryChannel"
)

rateLimiterMap := map[GetStreamApiName]RateLimiter{
  CreateChannel: {
    apiName: string(CreateChannel),
    token:   make(chan struct{}, 1),
  },
  QueryChannel: {
    apiName: string(CreateChannel),
    token:   make(chan struct{}, 1),
  },
}

A method to call GetStream QueryChannel Api leveraging RateLimiter can be defined as following:

func queryChannels(filters map[string]any) (queryResp *stream.QueryChannelsResponse, err error) {
  rateLimiter := rateLimiterMap["QueryChannel"]
  if errCall := rateLimiter.CallApiAndBlockOnRateLimit(logger, func() (resp *stream.Response, err error) {
    queryResp, err = getStreamChatClient.QueryChannels(context.Background(),
      &stream.QueryOption{
        Filter: filters,
      },
    )
    if err != nil {
      return nil, err
    }
    return &queryResp.Response, nil
  });
  errCall != nil {
    return nil, errCall
  }
  return queryResp, nil
}

🚀 That's it! I hope you like the way I faced the issue. Feel free to comment or propose your own idea to deal with this.

⚔️ In case anyone finds it useful I packed all of the above together in a Github repo.
You should be able to use it also as module in your code using:

go get -u github.com/sw360cab/getstream-rate-limiter

Migrating AWS SDK from v2 to v3 for S3

Sergio Matone — Tue, 03 Oct 2023 09:07:19 +0000

Disclaimer: Sample of code described here can be found at sw360cab/aws-sdk-migration-v3

Short time ago, I found myself in the very rare situation where you can dedicate some time slots to library upgrades in any of your projects. The purists will blame me for this “rare” adjective and bad practice, but even with all the good intentions, to me it is not so common to reserve time for this activity.
This makes me often feel like a lazy and bad software engineer, but breaking changes usually kills me.
And the following story will make no exception.

As I said I was starting upgrading libraries in a Node.js Git repository related to a project of QUA, the startup I own and run.
I started running the command:

npm outdated

and after a small check I ran:

npm update

I achieved a consistent result. Spoiler: I can state that because I have a solid unit/integration test base.
But then I noticed a warning:

NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
Please migrate your code to use AWS SDK for JavaScript (v3).

After digging into my code and googling a little bit, I found out that I was still using AWS SDK v2 throughout all the code. After a moment to recover from the loathing of this news, I branched and started to imagine how painful this journey would have been.

Note: AWS provides an automatic tool to migrate code from v2 to v3: Migrating your code to SDK for JavaScript V3 - AWS SDK for JavaScript. Of course I preferred the hard and painful way.

It turns out that it wasn't much to migrate. I will go through the changes soon, but the key breaking changes are:

SDK library is now split into dedicated libraries instead of importing the whole SDK and then initializing the required service.
Configuration can be done from ini files or local AWS CLI configuration.

Migration in details

Let's go.
The main service I leverage in my codebase are:

AWS S3
AWS SES

First step was adjust libraries version in package.json by uninstalling current AWS SDK

npm uninstall aws-sdk --save

AWS Credentials setup

npm i @aws-sdk/credential-providers --save

npm i @aws-sdk/client-s3 @aws-sdk/lib-storage --save

npm i @aws-sdk/client-ses --save

Credentials Setup

In the codebase using v2 of SDK I used to setup SDK credentials either using environment variables or via a JSON config file (which was not committed 😇)

export function initAws () {
  // if env variables are not provided
  // fallback to hardcoded static file
  if (process.env.AWS_ACCESS_KEY_ID === undefined
  && process.env.AWS_SECRET_ACCESS_KEY === undefined) {
    try {
      AWS.config.loadFromPath(awsConfigPath)
    } catch(e) {
      console.error(`AWS config file not found at ${awsConfigPath}`, e.message);
      process.exit(1);
    }
  }
};

Unfortunately with v3 loading credentials using a JSON file is not supported anymore, though it is still valid using environment variables.
However the closest way to achieve setup via external configuration files is using the library method fromIni:

import { fromIni } from "@aws-sdk/credential-providers";

This method creates an AwsCredentialIdentityProvider that read

a shared credentials file at ~/.aws/credentials
a shared configuration file at ~/.aws/config

Both files are expected to be INI formatted with section names corresponding to profiles.
I have shared an example of an AWS config file since it does not contain any sensible data.

secrets/aws/config

[default]
region=eu-west-1
output=json

Whereas credentials file will include sensible information, again in ini format

secrets/aws/credentials

[default]
aws_access_key_id=<YOUR_ACCESS_KEY_ID>
aws_secret_access_key=<YOUR_SECRET_ACCESS_KEY>

NOTE: the line [default] is referred to the <default> profile, the ini files can contain more than one profile, all in the format [<profile name>], but the profile to be employed should be reflected in the AWS SDK configuration.

The default reference path can be overridden using environment variables:

AWS_SHARED_CREDENTIALS_FILE for credentials
AWS_CONFIG_FILE for config

const awsCredentialsPath = process.env.AWS_SHARED_CREDENTIALS_FILE || path.join(__dirname,'/../secrets/aws/credentials');
const awsConfigPath = process.env.AWS_CONFIG_FILE || path.join(__dirname,'/../secrets/aws/config');

return fromIni({
  filepath: awsCredentialsPath,
  configFilepath: awsConfigPath,
})

The initialization of a service, like AWS SES, will be changed,

from v2

import AWS from 'aws-sdk';

AWS.config.loadFromPath(awsConfigPath)
const SES = new AWS.SES();

to v3

import { fromIni } from "@aws-sdk/credential-providers";
import { SESClient } from "@aws-sdk/client-ses";

const awsConfigEnv = fromIni({
  filepath: awsCredentialsPath,
  configFilepath: awsConfigPath,
})
const SES = new SESClient({region: "eu-west-1", credentials: awsConfigEnv});

S3

The main AWS service I leverage in my codebase is S3 to send/retrieve data to/from AWS S3 buckets.

Couple remarks:

in v2 to achieve a promise from a client method, a further promise() method should be called. In v3 the client methods tend to return natively Promises.
In the following example I used native Node.js Passthrough stream. It is almost useless when using a file read stream directly, but I left it as a reminder because it is common in most use cases, especially when files will be uploaded through an HTTP POST.

⚠️ These HTTP requests in Node.js are not pure readable streams but usually an object inheriting from the Node.js stream class, implementing methods like write. Passthrough streams allow passing data coming from an HTTP connection directly to the bucket without having to store them in local memory, meanwhile.

S3 Upload

In v2 I used to upload an “object” into a bucket like that:

import AWS from 'aws-sdk';

const uploadToS3 = (filePath, bucket, key) => {
  const s3Stream = new PassThrough();
  // pipe file and s3 stream to upload
  fs.createReadStream(filePath, { highWaterMark: 1024 * 16 }).pipe(s3Stream);

  return S3.upload({
    Bucket: bucket,
    Key: key,
    Body: s3Stream
  }).promise()
  .then(() => `${bucket}/${key}`)
  .catch(e => {
    console.error("unable to upload", e);
  })
};

Now in v3 I have to use the brand new method upload:

import { S3Client } from "@aws-sdk/client-s3";
import { Upload } from "@aws-sdk/lib-storage";

const uploadToS3 = (filePath, bucket, key) => {
  const s3Stream = new PassThrough();
  // pipe file and s3 stream to upload
  fs.createReadStream(filePath, {highWaterMark: 1024 * 16}).pipe(s3Stream);

  const upload = new Upload({
    client: new S3Client({region, credentials: awsConfigEnv}),params: {
      Bucket: bucket,
      Key: key,
      Body: s3Stream,
    }
  });

  return upload.done()
  .then(_ => `${bucket}/${key}`)
  .catch(e => {
    console.error("unable to upload", e);
  })
};

In v3 the upload part is lead by a dedicated SDK library: lib-storage, which provides an Upload class. Its constructor returns an EventEmitter whereas the done() method returns a Promise which successfully resolves when the upload to S3 bucket has finished.

S3 Download

v2:

import AWS from 'aws-sdk';
import { Readable } from 'node:stream';

const basicDownload = function(bucket, key, destPath) {
  const params = { Bucket: bucket, Key: key };
  return S3.getObject(params).promise()
  .then((download_) => {
    Readable.from(download_.Body).pipe(new fs.createWriteStream(destPath));
  }).catch(e => {
    console.error("unable to download", e);
  })
};

Now in v3 :

import { S3Client, GetObjectCommand } from "@aws-sdk/client-s3";

const basicDownload = function(bucket, key, destPath) {
  const params = { Bucket: bucket, Key: key };
  return new S3Client({region, credentials: awsConfigEnv}).send(new GetObjectCommand(params))
  .then(async (download_) => {
    download_.Body.pipe(new fs.createWriteStream(destPath));
  }).catch(e => {
    console.error("unable to download", e);
  })
};

In v2 Readable was needed to create a stream from the buffer returned by AWS SDK. Again a Node.js stream can be useful to pipe directly an HTTP response
In v3 the response from AWS SDK contains directly a Node.js Stream in the Body field.

References

Functional Programming: an accumulator of methods to transform objects in Javascript

Sergio Matone — Tue, 26 Sep 2023 10:09:15 +0000

While I was trying to solve a problem within my code, I had to modify an object by applying a set of transforming methods. I had to do the same with a whole array of strings, all requesting the same set of transformers to be applied.

Since I am a big fan of Functional Programming I wanted to use a map-reduce approach. I came up with a generic method leveraging the native reduce method of Array objects, the idea is to “accumulate” a string which at each iteration becomes both the result of the last transformation method and the input for the next transformation method.

Here is a placeholder:

const tranformers = [s => s.trim(), s => s.toUpperCase()];

const multiTransform = (str) => tranformers.reduce(
  (accumulator, currentValue) => {
    return currentValue(accumulator);
  },  // reduce callback
  str // initial value
);

console.log("Expecting the value 'HELLO WORLD' from `multiTransform` method:", multiTransform(" hello world "));

And in order to show how it can be applied to a set of strings, I will leverage the brand new Node.js Native Test Runner, which is still experimental but can be used from versions of Node.js >= v18.0.0 LTS (or either v16.17.0):

import assert from 'node:assert';
import test from 'node:test';

test('array transformation', (t) => {
  const arr = [" alice ","bob"," paul"];
  assert.deepEqual(["ALICE","BOB","PAUL"], arr.map(multiTransform));
});

---
> ✔ array transformation (1.886ms)

In vanilla Javascript, soon everything can become a mess due to the nature of the language, which is weakly typed. So the idea behind this accumulator pattern is to apply a certain number of transformations to a set of homogeneous objects; Strings are the most straightforward example of course.
Another possibility is that transformation methods include the capability to receive different object types and transform them according to the type of the object itself.

Unreachable static website distributed via AWS CloudFront

Sergio Matone — Wed, 13 Sep 2023 10:07:11 +0000

DISCLAMER: The following was a very specific situation I find myself in when my own website hosted in an AWS S3 bucket and distributed via AWS CloudFront, stopped working. Here is how I solved the issue smoothly.

I own a very simple personal website and I used to host it in an AWS bucket.

👉 This is it, Minimal Gap!

Distributing using my own domain just required to enable Static website hosting in AWS S3 Bucket properties and play a little bit with dns.

Here is a quick summary of how it was done.

S3 Bucket Name: minimalgap.com
DNS setup: cloud CNAME minimalgap.com.s3-website-eu-west-1.amazonaws.com

As a further step when I put it online, I decided to distribute it using a CDN, of course the easiest step was to leverage AWS CloudFront.

To do that I created an AWS distribution, I left most of the options as they are, just changed the following properties:

Origin Settings, for Origin Domain Name, enter the bucket website endpoint. You get this from the Static website hosting section of Properties for the Amazon S3 bucket. This will result in Origin Type being S3 static website.
I optionally generated a TLS certificate via AWS, to hold a certificate related to my own domain (minimalgap.com).

All of the above is well documented here

However, out of nowhere, my CloudFront Distribution stopped working. After a lot of digging I found out that some AWS configurations may have changed internally and my distribution was lacking of “something”, that prevent it to work correctly.

It was an issue with the Origin configuration itself. Indeed:

picking my own distribution in AWS Cloud Front and clicking on its ID,
getting to the view Origins, I found the correct configuration pointing to my S3 bucket,
trying to select and edit the problem was clear: the protocol selection was missing (HTTP, HTTPS, Match viewer).

Once I selected one of the values the distribution was back working as expected without any issue (see image below).
Therefore that something was actually the protocol selection within the Origin configuration!

As I said my idea is that AWS changed a little bit some configuration so distributions which were already setup, need to be manually fixed.
Maybe this will result useful for someone else encountering the same issue.

Go: Using reflection to create a generic lookup method

Sergio Matone — Thu, 17 Mar 2022 12:14:25 +0000

Starting the journey in GoLang

As a newbie GoLang developer I had to deal with a set of new statements to be understood and digested. I also had to face a new way of thinking that shifted me from what I am used to in JS / TS / NodeJS.\
One aspect which was really tough at the beginning was interfaces. In my developer experience, I had already encountered interfaces in Java and I didn't like them too much. This wasn't a good way to start with them in Go.\
Then I realized how powerful and useful they could be while creating a whole Go application or library (and they are probably too in Java, no offense).

The other thing that gave me kind of bad time was the lack of Generics, which luckily has been recently filled in Go 1.18.
The latter was even more annoying when I dealt with a Go project by one of my clients, which is based on Go 1.16 and cannot be upgraded because of some dependency requirements.

Go is sometimes very verbose, and this can be either good or bad, depending on the situation. For example in that codebase I found a package named arrays which was taking care of looking up a specific element within a given array/slice.
Why a package? Because they could not find a way to achieve the lookup in array methods without creating each for any type that is needed. So any type that needs this lookup method, either built-in (string, int, int64) or custom (struct), should have its own method doing pretty much the same as all the others do.

When I see too many repetitions and copy/paste of code I smell a rat and I like to remove all the repetitions as soon as possible. Using code repetitions is not only evil, but when you find a bug in a part of the code they may have been repeated tons of times.\
What I wanted is a generic method that can lookup for a value in a slice only if they both hold the same type.

Hands-on

Creating a generic method was pretty simple as changing the parameters from a given type

func ContainsInt(items []int, value int) bool { ... }

to the most generic one in Go: interface{} also known as any interface.

func containsAny(items interface{}, value interface{}) bool { ... }

But for the body of the method it is another story, even if its original version was pretty trivial:

func ContainsInt(items []int, value int) bool {
    for _, item := range items {
    if item == value {
           return true
       }
    }
    return false
}

The best way to deal with this is the reflect package which holds all the methods to reflect, meant as analyzing at its deepest level, a data structure.

The first problem is that using an any interface type would lose the explicit type check at compile time, which is one of the best things when writing Go code (but I'll tell you more on that later). This is even worse because the expected parameters should be: a slice and a "same-type" single value.

First thing to do is checking that the first parameter holds effectively a slice data structure. This can be achieved with the Kind method of the reflect package.

if itemsValue.Kind() != reflect.Slice { return false }

Once that we have been assured that the first type is effectively a slice, comparing each of its values with the given value is relatively easy. Indeed the DeepEqual method allows to compare two values and as the official documentation states:

Two values of type Interface are deeply equal if they hold deeply equal concrete values.

So for each item in the slice the Interface method is called, which returns the current value as an interface{} and that can be deeply compared with the given value, which is already of type interface{}

for i := 0; i < itemsValue.Len(); i++ {
       if reflect.DeepEqual(itemsValue.Index(i).Interface(), value) {
   return true
       }
    }

Moreover we don't even need type checking, because the previous method will yield implicitly false if the compared values are not of the same type. Again the official documentation:

Values of distinct types are never deeply equal.

Here is the complete method statement, with a couple of debugging logs:

func containsAny(items interface{}, value interface{}) bool {
    itemsValue := reflect.ValueOf(items)
    if itemsValue.Kind() != reflect.Slice {
       return false
    }

    for i := 0; i < itemsValue.Len(); i++ {
       if reflect.DeepEqual(itemsValue.Index(i).Interface(), value) { // value, short for -> reflect.Value(value).Interface()
           fmt.Printf("%s and %s \n", itemsValue.Index(i).Type().String(), reflect.ValueOf(value).Type().String())
           fmt.Printf("%v and %v \n", itemsValue.Index(i), reflect.ValueOf(value))
           return true
       }
    }
    return false
}

Too much generic

As I wrote earlier now that everything is so generic I lost some benefits of type checking in Go. There will be no way, except at runtime, to know if the method has been called using the correct types, since interface{} can hold anything at compile time.

This led me to reuse the previous method declarations with explicit types from the original package but I replaced their body with a simple call to the generic method.

Several benefits come from that:

exposed methods have a specific type declaration that is checked at compile time;
generic method can be unexposed by the package since any call to it will happen within the package;
there is only a source of truth (or of failure) which is the generic method itself;
the package can be extended to an unlimited number of types only by creating a new public method declaration with the new given types, and it will need to call the sole generic method in it.

You can find the full code with a bunch of tests here

Conclusions

After starting learning Go almost one year ago and joining my first real and in-production project, all based on a GoLang codebase, I am satisfied with having faced these kinds of challenges and having achieved the next level skills.\
Probably Go gurus will disagree with the solution I came up with, but I will be happy to read their wise advice. I know that for sure this was more an intellectual exercise, since with Go 1.18 all of that may be overtaken by built-in Generics.

Traefik 2.5 - What a Mesh!

Sergio Matone — Mon, 11 Oct 2021 09:19:48 +0000

Disclaimer: This post is freely inspired by an Official Traefik Labs blog post.
I took some part and adapted to achieve a simpler working example (in my humble opinion).

Few weeks ago Traefik Labs announced the new release of Traefik Proxy, now at version 2.5.
Many features have been shipped:

HTTP/3
Private & Provider Plugins
Load Balancer Healthchecks
Kubernetes 1.22 support
TCP Middleware
Consul Connect integration

The latter is one of the most interesting newcomers. Service Mesh is always an important challenge when dealing with quite large distributed architectures.

Among Traefik Labs products there is already Traefik Mesh, a lightweight Service Mesh, not invasive built on top of Traefik Proxy. These peculiarities are achieved by leaving the "Sidecar Pattern" in favour of a "DaemonSet Pattern", so a per-node instead of a per-pod design pattern.
Using a DaemonSet in this context brings along a shortcoming (that Traefik Labs has never hidden), mTLS communication among services is not available and it should be implemented by application itself, for example using Traefik Mesh at TCP level only.

Consul by HashiCorp is one of the reference solutions when talking about Service Mesh. Indeed it leverages the Sidecar pattern and offers mTLS communication.

What really is Service Mesh?

Let's take a step back on service mesh.
Service Mesh is intended as a network layer in the cluster that allows fast and reliable communication among services and applications within the cluster itself.

Service mesh is based on this concepts:

Discoverability - services need to discover each other, in order to be able to talk together.
Configurability - services need to be configured. There should be an abstraction layer for network (no need to know by applications)
Segmentation - services need fine-grained access control between them, to ensure that only specific sets of services should be able to communicate with each other. In a few words traffic control.

That said, how do Consul components deal with all these concepts?

Consul Catalog registers all of your services into a directory. They will be discoverable throughout all the network [Service Discoverability]
Consul Connect is a proxy layer that routes all service-to-service traffic through an encrypted and authenticated (Mutual TLS) tunnel. [Abstraction layer]
Consul Intentions acts as a service-to-service firewall, and authorization system. Full segmentation can be easily achieved by creating a Deny all rule, and then adding each explicit service-to-service connection rule needed. [Traffic Control]

As stated before Consul leverages the Sidecar Pattern, injecting a sidecar container for each service Pod in the mesh.
This sidecar container is called Envoy Proxy and:

it routes traffic among services pods via other sidecar containers
it enforces the firewall rules in the mesh network using Intentions

The following diagram illustrates how the communication works when to service works:

target service is discovered via Consul Catalog
service access is verified and ruled by Consul Intentions
traffic flows from source service via Envoy Proxy through corresponding Envoy Proxy of target service and finally to the target service itself.

Working with Consul Connect in Kubernetes

Let's see how the previous concepts can be applied to Kubernetes. Service registration to Consul Connect can be achieved by adding a simple annotation in Pod Spec:

spec:
  replicas: ...
  selector:
    matchLabels:
    ...
  template:
    metadata:
      labels:
        ...
      annotations:
        consul.hashicorp.com/connect-inject: "true"

The previous annotation will automatically:

inject Envoy Proxy as sidecar container
register a new service in the Consul Catalog

There is another annotation we should take into account.

annotations:
  consul.hashicorp.com/service-sync: "true"

We will see next why this is a relevant annotation. But at the moment let's remember that this will exclusively register a service into the Consul Catalog, skipping the Envoy Proxy injection.

Traefik Proxy 2.5 - Consul Connect integration

After explaining a little bit of Consul let's see how Traefik Proxy is integrating Consul Connect.

First of all Traefik Proxy will drop the Sidecar Proxy (similarly to Traefik Mesh), instead a Traefik native support to Consul Connect will be in charge.
Eventually, since Traefik Proxy is based on the concept of Providers which enables configuration discovery, the Consul Catalog provider will be the one in charge.

Let's now go back for a while at the previous annotations of Consul Connect.
consul.hashicorp.com/service-sync is exactly what is useful: it does not inject sidecar Envoy Proxy, but it rather registers the service into the Consul Catalog. So in order to allow Traefik Proxy to be seen by Consul without injecting it with a sidecar container, the previous annotation will be required.

Hands-On

Requirements

From here on we will suppose you have:

a configured and working K8s cluster
Helm installed and available

All the commands can be found at the following repository

Consul Connect Configuration

First of all we will leverage Helm to install Consul.

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

Here is a list of possible values for the Consul Helm Chart:

global:
  name: consul
  datacenter: dc1
  image: hashicorp/consul:1.10
  imageEnvoy: envoyproxy/envoy:v1.19-latest
  metrics:
    enabled: true
  tls:
    enabled: true
  #   enableAutoEncrypt: true
  #   verify: true
  #   serverAdditionalDNSSANs:
  #     ## Add the K8s domain name to the consul server certificate
  #     - "consul-server.consul-system.svc.cluster.local"
server:
  # Scale this according to your needs:
  replicas: 1
  affinity: null
ui:
  enabled: true
controller:
  enabled: true  
connectInject:
  # This method will inject the sidecar container into Pods:
  enabled: true
  # But not by default, only do this for Pods that have the explicit annotation:
  #        consul.hashicorp.com/connect-inject: "true"
  default: false
syncCatalog:
  # This method will automatically synchronize Kubernetes services to Consul:
  # (No sidecar is injected by this method):
  enabled: true
  # But not by default, only for Services that have the explicit annotation:
  #        consul.hashicorp.com/service-sync: "true"
  default: false
  # Synchronize from Kubernetes to Consul:
  toConsul: true
  # But not from Consul to K8s:
  toK8S: false

Note: I explicitly disabled the following section which prevented Consul Helm Chart to be fully bootstrapped:

enableAutoEncrypt: true
  verify: true
  serverAdditionalDNSSANs:
    ## Add the K8s domain name to the consul server certificate
    - "consul-server.consul-system.svc.cluster.local"

First create a namespace, then run the Helm chart:

kubectl create namespace consul-system
helm upgrade --install -f consul-values.yaml consul \
  hashicorp/consul --namespace consul-system

You can adjust these values as you like BUT remember in order to leverage Traefik Proxy integration, Envoy Proxy should not be enabled by default. So basically the advice here is disabling by default Envoy Proxy Injection and Service Registration in Helm values.

Traefik Proxy Configuration

Traefik Proxy can also be configured via Helm, but I'd rather configure it explicitly to exploit its simplicity and straightforwardness.

Important: Before proceeding remember that Traefik Proxy needs a copy of the Consul certificate authority TLS certificate. Copy the Secret resource from the consul namespace into the traefik namespace:

kubectl get secret consul-ca-cert -n consul-system -oyaml | \
  sed 's/namespace: consul-system$/namespace: traefik-system/' | \
  kubectl apply -f -

Then remember to mount the secret into the Deployment of Traefik Proxy:

volumes:
  - name: consul-ca-cert
    mountPath: "/certs/consul-ca/"
    type: secret

The Traefik Proxy CLI arguments are those which enable Consul Catalog provider:

  ## Consul config: 
  # Enable Traefik to use Consul Connect:
  - "--providers.consulcatalog.connectAware=true" 
  # Traefik routes should only be created for services with explicit `traefik.enable=true` service-tags:
  - "--providers.consulcatalog.exposedByDefault=false"
  # For routes that are exposed (`traefik.enable=true`) use Consul Connect by default:
  - "--providers.consulcatalog.connectByDefault=true"
  # Rename the service inside Consul: `traefik-system-ingress`
  - "--providers.consulcatalog.servicename=traefik-system-ingress"
  # Connect Traefik to the Consul service:
  - "--providers.consulcatalog.endpoint.address=consul-server.<consul_namespace>.svc.cluster.local:8501"
  - "--providers.consulcatalog.endpoint.scheme=https"
  - "--providers.consulcatalog.endpoint.tls.ca=/certs/consul-ca/tls.crt"

The cli arguments above include the reference to the Consul CA certificate created via a secret and mounted on a volume.
Moreover as stated above, the service will be discoverable by Consul leveraging annotations:

only service-sync is enabled (not the Envoy Proxy injection)
there is a correspondence between the service name in the annotations and in the Traefik CLI arguments

consul.hashicorp.com/service-sync: "true"
consul.hashicorp.com/service-name: "traefik-system-ingress"

Here the full Traefik Proxy configuration (Deployment+Service):

apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik-ingress-controller
  namespace: traefik-consul
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: traefik
  namespace: traefik-consul
  labels:
    app: traefik
spec:
  replicas: 1
  selector:
    matchLabels:
      app: traefik
  template:
    metadata:
      labels:
        app: traefik
    spec:
      serviceAccountName: traefik-ingress-controller
      containers:
        - name: traefik
          image: traefik:v2.5
          args:
            - --api.insecure
            - --accesslog
            - --entrypoints.web.address=:80
            - --entrypoints.websecure.address=:443
            # Consul config: 
            # Enable Traefik to use Consul Connect:
            - "--providers.consulcatalog.connectAware=true" 
            # Traefik routes should only be created for services with explicit `traefik.enable=true` service-tags:
            - "--providers.consulcatalog.exposedByDefault=false"
            # For routes that are exposed (`traefik.enable=true`) use Consul Connect by default:
            - "--providers.consulcatalog.connectByDefault=true"
            # Rename the service inside Consul: `traefik-system-ingress`
            - "--providers.consulcatalog.servicename=traefik-system-ingress"
            # Connect Traefik to the Consul service:
            - "--providers.consulcatalog.endpoint.address=consul-server.consul-system.svc.cluster.local:8501"
            - "--providers.consulcatalog.endpoint.scheme=https"
            - "--providers.consulcatalog.endpoint.tls.ca=/certs/consul-ca/tls.crt"
            - "--providers.consulcatalog.endpoint.tls.insecureskipverify=true"
          ports:
            - name: web
              containerPort: 80
            - name: websecure
              containerPort: 443
            - name: admin
              containerPort: 8080
          resources:
            limits:
              cpu: "2"
              memory: "500Mi"
          volumeMounts:
            - name: consul-ca-cert
              mountPath: "/certs/consul-ca/"
              readOnly: true
      volumes:
      - name: consul-ca-cert
        secret:
          secretName: consul-ca-cert
---
apiVersion: v1
kind: Service
metadata:
  name: traefik
  namespace: traefik-consul
  annotations: 
    # Register the service in Consul as `traefik-system-ingress`:
    consul.hashicorp.com/service-sync: "true"
    consul.hashicorp.com/service-name: "traefik-system-ingress"
spec:
  selector:
    app: traefik
  ports:
  - name: web
    protocol: TCP
    nodePort: 30080
    port: 80
    targetPort: 80
  - name: admin
    protocol: TCP
    nodePort: 30808
    port: 8080
    targetPort: 8080
  - name: webtls
    protocol: TCP
    nodePort: 30443
    port: 443
    targetPort: 443

If everything works, accessing the Consul UI you should find a service named "traefik-system-ingress" registered via Kubernetes.

Tip: in a development environment Port Forwarding is an easy and straightforward way to access Consul UI from your own browser

kubectl port-forward --address 0.0.0.0 --namespace consul-system service/consul-ui 18500:443

Then you can access Consul UI at https://<your host IP>:18500/

Configure a service route by adding Consul tags to Services

Let's now add a service to the Consul service Mesh.
From the Consul side the service only needs the Envoy sidecar proxy (remember the annotation: consul.hashicorp.com/connect-inject: "true").
However the same service may be exposed also on the Traefik Proxy side, and here is where the magic happens, and there are two key components:

the Consul Catalog provider from the Traefik domain
service tags annotations from the Consul domain

When using the Consul Catalog provider, Traefik Proxy routes are added by creating service tags in Consul itself. This can be done via the Consul HTTP API, or from the Kubernetes API, by adding an annotation in the Deployment Pod spec: consul.hashicorp.com/service-tags.

Following the relevant part:

  spec:
    replicas: ...
    selector:
      matchLabels:
      ….
    template:
      metadata:
        labels:
          ...
        annotations:
          consul.hashicorp.com/connect-inject: "true"
          consul.hashicorp.com/service-tags: "\
            traefik.enable=true,\
            traefik.http.routers.whoami.entrypoints=web,\
            traefik.http.routers.whoami.rule=Host(`ks3.minimalgap.com`)"

The value for the service-tags annotation must be all in one line, tags separated with commas, and no spaces. In order to make it more readable, the line is wrapped onto multiple lines with the \ character after the commas.
This is different by employing a pure Kubernetes Provider Configuration in Traefik Proxy, where you would expect these values in the definition of an Ingress component.

After deploying a service annotated as above, if you now access the Traefik Dashboard you will notice that the Whoami service is now marked with Consul Catalog provider.

Note: Remember to set up a valid and correct FQDN in the service-tags annotation for the Host Rule in the router section.

Note on TLS

You may note that the service above is exposed on a simple HTTP endpoint, skipping TLS support.
I struggled a little bit with configuring TLS. As far as I know a Let's Encrypt resolver should be employed.

This requires a further configuration for the Traefik Proxy CLI, e.g.:

## TLS CHALLENGE
- --certificatesresolvers.autoresolver.acme.tlschallenge
## LetsEncrypt definitions
- --certificatesresolvers.autoresolver.acme.email=foo@you.com
- --certificatesresolvers.autoresolver.acme.storage=acme.json
## Please note that this is the staging Let's Encrypt server.
## Once you get things working, you should remove that whole line altogether.
- --certificatesresolvers.autoresolver.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory

and the corresponding reference in the service discovered via Consul, changing the service-tags annotation:

annotations:
  traefik.enable=true,\
  traefik.http.routers.whoami.entrypoints=web,\
  traefik.http.routers.whoami.rule=Host(`ks3.minimalgap.com`),\
  traefik.http.routers.whoami.tls=true,\
  traefik.http.routers.whoami.tls.certresolver=autoresolver"

Since this topic it's outside of the scope of this post, for sake of simplicity I left apart the TLS configuration.
I advise you to carefully read the Traefik Proxy docs to attempt a fully TLS configuration; there is a detailed User Guide section for Kubernetes & Let's Encrypt.
Moreover the whole process would be much simpler if you own a domain that is compliant with the dnsChallenge.

Playing with Consul Intentions

It's now time for fun!
So far we have configured:

Consul Connect (via Helm)
Traefik Proxy (with Consul Catalog provider)
a service (whoami) that is visible by both Consul Connect and Traefik Proxy

Now we can access the service using cUrl. This can be tricky depending from your configuration, you may use:

Port Forwarding the whole Traefik service (Gotcha sudoing, check here)

kubectl  port-forward --address 0.0.0.0 service/traefik 80:80 8080:8080 -n traefik-consul

Attaching a shell to a Pod and executing curl from there (if cUrl is available within the Pod itself)

kubectl exec -ti -n consul-system consul-server-0 -- curl http://ks3.minimalgap.com:80

Creating a K8s IngressRoute in Traefik Proxy

Let's suppose one of the above finally works. The response would be something like:

curl http://ks3.minimalgap.com:80 -vv
*   Trying 34.240.156.240:80...
* Connected to ks3.minimalgap.com (34.240.156.240) port 80 (#0)
> GET / HTTP/1.1
> Host: ks3.minimalgap.com
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
IP: 127.0.0.1
IP: ::1
IP: 10.42.0.140
IP: fe80::d09e:6dff:feee:cb78
RemoteAddr: 127.0.0.1:43100
GET / HTTP/1.1
Host: ks3.minimalgap.com
User-Agent: curl/7.79.1
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 10.42.0.139
X-Forwarded-Host: ks3.minimalgap.com
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Server: traefik-f9bbb4fb8-fzx6v
X-Real-Ip: 10.42.0.139

At this point we can state we achieved a working configuration. It's time to create Intentions!
Consul Intentions are more or less firewall rules for services, they have a Source Service, a Target Service, a Description and a Deny/Allow option.
These rules can easily be visually created from the Consul Dashboard. Suppose that we want to create a Deny All rule:

* (All Services) for the Source Service
* (All Services) for the Destination Service
Should this source connect to the destination? choose Deny
click Save to create the Intention.

Now test the whoami service again, using your real service domain name:

curl http://ks3.minimalgap.com:80 -vv

You should see a response of 502 Bad Gateway.
Note: the response may be still 200 OK for few minutes. This is because the Intentions may take some time to update to the envoy sidecar. I promise after a while the Bad Gateway response should come up!

Alternatively Consul Intentions can be configured via Kubernetes API. This is achieved via K8s CRDs. Let's use this method to configure an Allow Rule from the Traefik Proxy (traefik-system-ingress) to the destination service (whoami).

First define a Service Definition CRD:

apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceDefaults
metadata:
  name: whoami
  namespace: consul-demo
spec:
  protocol: 'http'

The Service Intentions can be defined via YAML in a similar way respect to Consul Dashboard:

apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceIntentions
metadata:
  name: whoami
  namespace: consul-demo
spec:
  destination:
    # Name of the destination service affected by this ServiceIntentions entry.
    name: whoami
  sources:
    # The set of traffic sources affected by this ServiceIntentions entry.
    # When the source is the Traefik Proxy:
    - name: traefik-system-ingress
      # The set of permissions to apply for Traefik Proxy to access whoami:
      # The first permission to match in the list is terminal, and stops further evaluation.
      permissions:
        - action: allow

Accessing the endpoint should stop giving (quite immediately) a Bad Gateway response in favour of a successful response again.

Conclusions

Traefik Labs keeps on doing giant leaps and integrating Consul Connect is another step beyond for Traefik Proxy. This indicates the path of this product is humble but reliable and flexible, with an open-minded philosophy behind that is never scared of comparing and collaborating with other important competitors and actors in the CNCF big landscape picture.

Probably Consul Connect is a product more suitable in advanced use cases where big clusters are involved. But Consul Connect integration is a kind of Enterprise level feature that demonstrates how Traefik Proxy is powerful: it works the same way, from the homemade pet project to a huge diverse enterprise cluster.

Minimal Gap is online

Sergio Matone — Tue, 28 Sep 2021 09:09:53 +0000

I would like to share my brand new website for my freelance activity: MinimalGap

As the brand name says, I decided to go for a minimal landing page to give a taste of who I am and what I can do in the huge world of IT tech.

Feedback is welcome so feel free to comment on this.

What a trip the "Traefik Autumn Edition": true memories from a Traefik Ambassador

Sergio Matone — Wed, 23 Dec 2020 13:39:33 +0000

It was April the 24th 2020, we were in the middle of the lockdown because of the global pandemic, among the tons of emails (most of them Adv) received everyday, there was one coming from Patricia Dugan from Traefik Labs, which says I was entitled as Traefik Ambassador.
What? Let me double check that this is not a spam email which will let me win a couple of iPhones after giving all my personal data and possibly my credit card number. No it's not, she really wanted to communicate that to me.

Wow, I am so proud, I am so happy! Yeeeaaah... But, wait a second, from great powers come great responsibilities. Which are my duties, what should I do? I have been a Traefik Ambassador since half an hour and I still don't know what should I do, that is a shame. Soon they will realize that I do not deserve to be a Traefik Ambassador, what a shame! I am so sad 🤕.
I was already seeing myself speaking at a conference, presented as a Traefik Ambassador. This would never happen, moreover conferences are not feasible too, because of this damned Covid-19. I remember when we used to meet at the CNCF Meetup here in Torino (🇮🇹), with Gianluca, Barbara, Giuseppe and all the others, and after listening to great talks we just used to have friendly chat in front of 🍺 and 🍕. Oh, beloved old days, will these ever happen again? When will we be able to organize a meeting in person again? Oh, but wait a second, let me remove the "in person" part… Organizing meeting 🤔, Traefik Ambassador... mmm, maybe I'm still on track. If I ask both Gianluca and Patricia to join forces to organize something together, I can still deserve my Traefik Ambassador entitlement.

Whether the previous lines happened in my mind in a couple of hours after receiving the news of me becoming a Traefik Ambassador, I can assure that what happened next took more or less a couple days.
I went through proposing that to Gianluca and Patricia, surprisingly they were both very positive about the proposal, so that with < 20 emails we were done in setting up not only a single event but a complete season of events. Thanks to Gianluca we found immediately a nicename: "Traefik Autumn Edition". The rest went so smoothly that I can barely remember anything stopping our flow 💪🏽.

The drawback was that eventually I was in charge for the very first event! 😱

Episodes 01: Introducing Traefik Proxy 2.3. A Journey into Traefik Plugins via Gitops [29/10/2020]

And here we go: first event.

I'm so glad we bring back the #CNCF Turin meetup with an Awesome series about #Traefik. I am kicking off my adventure as #traefikambassador. Thx to @traefik @CNCF @patricia_dugan and @GianArb for making it possible and letting the #Traefik community growing in 🇮🇹.
Stay Tuned!📣 https://t.co/PNlhsIk0Ye
— Sergio Matone (@sw360cab) October 5, 2020

Although very excited, I succeeded in introducing the Traefik Labs Stack. I then explored Traefik Proxy and what is new in version 2.3. Finally I went through an overview of Traefik Plugins. The whole event was topped with extended demos and live coding. I showed basic usage of Traefik Proxy in Kubernetes environment, going through the definition of a Traefik Plugins (I made my own for the event).

Demo gods assisted me almost till the end when I tried reflecting edits in code leveraging GitOperations, when the demo using ArgoCD did not work... But I can prove it works with a video!

and here are the slides:
https://speakerdeck.com/sw360cab/traefik-autumn-edition-2020-s01e01

and the Traefik plugin code:
https://github.com/sw360cab/cncftaeplugin

Full video of the Event is here:

Episodes 02: Kubernetes Ingress Today and Tomorrow [19/11/2020]

Second event was immediately on his way:

Couple weeks ago Episode 01 of #Traefik Autumn Edition 🍂
was on the edge! Now it's time to look forward for Episodes 02,🗓 19th Nov fo: #Kubernetes Ingress Today and Tomorrow. Presented by Kevin Crawley (@notsureifkevin) at #CNCF Turin. Join us:
👉 https://t.co/zT2zz8GqFH https://t.co/3SQGpDAyC2
— Sergio Matone (@sw360cab) November 13, 2020

Kevin Crawley aka @notsureifkevin (yes I am sure it was him!), from Traefik Labs, drove us through the new possibilities given by release 1.18 of Kubernetes. He highlighted many notable changes including adding the new IngressClass resource and modifications made to the Ingress resource. It introduces a new field that specifies how Ingress paths should match and support wildcard domains. Kevin had a great demo using Traefik Proxy, MetalLB, and Kind.
We then had a great excursus of the new Service API specification in Kubernetes. This new API aims to bring vendor neutrality to the Ingress networking landscape. Kevin demonstrated how these recent changes operate using Traefik Proxy.

We finally had a great networking time where we were able to discuss how hard it is to achieve working configurations in Kubernetes when dealing with real world scenarios, even some shortcomings in Traefik Proxy that they are willing to improve soon.

Here are the slides:
https://docs.google.com/presentation/d/1-YpI9StVmaRgIkTq-0Hy0CdfX8OlOcv5xFJoeyevVRU/edit#slide=id.g9e184c83e8_0_5

Full video of the Event is here:

Episodes 03: How to monitor Traefik workloads with Prometheus & Grafana [17/12/2020]

In an heartbeat we were at the Final Episode:

I still need to recover from last Episode at #CNCF Turin that was stunning! Now I need to pull myself together for the Great season Finale of #Traefik Autumn Edition, which will be held by @idomyowntricks. Sign up ASAP 👇#traefikambassador https://t.co/XmfH8hRI3K
— Sergio Matone (@sw360cab) November 24, 2020

Brian Christner (@idomyowntricks) from theByte showed the usage of Prometheus and Grafana leveraging native support in Traefik Proxy. He spent a lot of time showing code and GUIs instead of sticking with tons of slides. He answered a lot of doubts, gave advices and took us real-world scenarios

Besides Docker Captain, Traefik Ambassador and Cloud expert, Brian is hosting online courses. Luckily he gave me a preview of the complete Traefik Proxy course, so I take the chance to suggest everybody taking it. Check this out

Slides of the Event:
https://56k-share.s3.eu-central-1.amazonaws.com/Traefik_Observability.pdf

Full video of the Event is here:

Last Words

It has been a real adventure to organize, present and join the first Traefik Autumn Edition. It would not have been possible without the help of Gianluca Arbezzano (@gianarb) who hosted the whole Season at the Turin CNCF Meetup and Patricia Dugan (@patricia) who is Head of Community at Traefik Labs and who never stopped encouraging and supporting me.
Especially in the period we are living, we started loving online events, streaming, webinars and so on. But actually we have enough of that and I thank all the people who joined and listened for us patiently.

A final thanks goes to CNCF that allows the existence of its Meetup in my city, Torino (🇮🇹), Equinix Metal that gave us bonus VPS for hosting our demos, and Traefik Labs for supporting us and creating a great community around Traefik!

Stay safe, stay strong.

Mongoose Debug messages with a custom logging library or style

Sergio Matone — Thu, 05 Nov 2020 17:39:09 +0000

Mongoose Custom Logger

When Using Mongoose as MongoDb object modeling for NodeJs, one interesting feature is the possibility to log to standard output what Mongoose is actually doing while interfacing with MongoDB. This can be easily achieved by configuring Mongoose library itself:

mongoose.set('debug', true)

Which will print any action that Mongoose is performing (e.g. insert, update, remove, find) in a colored and very readable fashion.

This is basically an inner call of the library to console.log.

However when employing a custom logger library, or a custom logging style, the Mongoose log style becomes soon non-compliant.
Imagine adding a logging library to your project or a logger that logs to file instead of stdout, or even a custom log format for logging to standard output.
In all the former cases Mongoose logs would be lost or not conformant.

Luckily Mongoose is providing an additional parameter to the previous method. From the official doc, the parameter can be either a writable stream or a callback for logging a message using a custom format.

mongoose.set('debug', function(collectionName, methodName, ...methodArgs) {}); // use custom function to log collection methods + arguments

If a writable stream is passed, it will log to that stream, without colorization. If a callback function is passed, it will receive the collection name, the method name, then all arguments passed to the method. For example, if you wanted to replicate the default logging, you could output from the callback Mongoose: ${collectionName}.${methodName}(${methodArgs.join(', ')}).

As stated above the custom callback can be something like this:

function(collectionName, methodName, ...methodArgs) {
  customLogger.log(`${collectionName}.${methodName}(${methodArgs.join(', ')})`)
}

where customLogger is a custom logging library or imported dependency. The result is something like that:

While this is a step forward, this will just print a plain string, throwing away the nice colorization of the original Mongoose method, which is not cool!
Whereas for the Mongoose string, it is just a matter of pure console coloring optimization, in the form of:

customLogger.log(`\x1B[0;36mMongoose:\x1B[0m:`);

The format of the nice part of the collection and operation performed is not straightforward to be printed.
But after digging a while in the lib, I found out that it uses a standard method of the native Node.js library util, in particular util.inspect, which:

returns a string representation of object that is intended for debugging. The output of util.inspect may change at any time and should not be depended upon programmatically. Additional options may be passed that alter the result. util.inspect() will use the constructor's name and/or @@toStringTag to make an identifiable tag for an inspected value.

In the case of the Mongoose lib, the object is slightly manipulated to remove new lines and extra white spaces.
The result is similar to:

util.inspect(m, false, 10, true).replace(/\n/g, '').replace(/\s{2,}/g, ' ');

So putting altogether, the final configuration of the Mongoose logging is something like that:

mongoose.set('debug', (collectionName, methodName, ...methodArgs) => {
  const msgMapper = (m) => {
    return util.inspect(m, false, 10, true)
      .replace(/\n/g, '').replace(/\s{2,}/g, ' ');
  };
  customLogger.log(`\x1B[0;36mMongoose:\x1B[0m: ${collectionName}.${methodName}` + `(${methodArgs.map(msgMapper).join(', ')})`)
});

which outputs:

I am happy that I can finally achieve the same log output created by the Mongoose library with any custom logging library I may employ.