DEV Community: MyUnisoft

Migrating from Nextcloud to Azure S3

Thomas.G — Tue, 20 May 2025 12:14:44 +0000

Hello 👋

Back for a new MyUnisoft technical article, this time with the help of my colleague Nicolas MARTEAU. Today, we will share our journey to completely refactor our document management architecture and how we migrated from Nextcloud to Azure S3 as our storage technology.

We weren’t able to cover every detail—both for security reasons 🛡️ and to protect sensitive data 🔒—but I hope you will enjoy what I could share. 😊

👀 Why moving away from Nextcloud ?

Performance 🤖

Until now, we have managed several tens of millions of documents with Nextcloud. However, stability and performance had become an issue, with regular downtime 🕒 and delays ⏳ of several minutes for a simple document upload at times.

💬 These upload delays sometimes led to misunderstandings among users. For example, in certain integrations, it was not uncommon for users to delete their Accounting entries 🧾 after a few seconds because they thought the attachment was missing.

The complexity and limited functionality of the existing APIs quickly became a significant obstacle 🛑. Simply making a document available in a specific folder could require four or five separate HTTP requests. We needed a more robust storage solution that could scale effectively 📈 and provide consistent, fast response times. 🚀

Infrastructure 🏢

Furthermore, we needed to reduce Nextcloud's impact on our infrastructure. Unlike Azure, Nextcloud doesn't scale well and required too much maintenance from our DevOps team.

😬 Architectural issues

In the past, users accessed documents stored directly on Nextcloud, with some files displayed through the platform’s built-in viewers.

This initial choice was certainly made for simplicity, but it has evolved into a significant architectural challenge as we began exposing storage directly to customers. Changing a storage server without affecting our users has become complex, and it also complicates the management of certain security and observability concerns.

The primary issue is with PDF documents, such as ledgers, which contain hardcoded URLs pointing to specific storage servers. This requires us to maintain these URLs for years to ensure continued access.

As part of our migration to S3, we are addressing these issues by routing all requests through the same service (GED).

This approach enables us to resolve several issues and enhance the product’s functionality:

Requiring authentication for some sensible documents.
Providing full observability over who uploads or downloads specific documents.
Enabling updates to storage capabilities without impacting customers.
Integrating new storage technologies seamlessly and transparently—for instance, through potential future integrations with services like Microsoft OneDrive.

📢 The plan

The first step was to draft an action plan 📝 and thoroughly document the existing setup. After several weeks of work, we established the key steps:

Route all document downloads through the GED service.
Route all document uploads through the GED service.
Migrate all existing documents to our new Azure storage, ensuring zero impact 🚫 on the end user.
Managing the Nextcloud links found in the PDFs already exported by our clients before the migration. Since these links pointed directly to our Nextcloud servers💀, we had to find a reliable solution to route these calls through the GED.

Each stage comes with its own set of challenges, which we’ll examine in detail later in the article.

Our primary concern, however, was to correct previous architectural missteps 🔍.

1️⃣ Download

The first step was to re-abstract downloads and previews, routing them through our backend. This required us to manage both legacy documents 📜 still stored on Nextcloud and new documents that would be hosted on Azure storage.

One challenge we’re facing is that the token generated by Nextcloud lacks any information about the tenant associated with the document. Without this, our backend cannot identify the relevant database cluster and tenant.

To resolve this, we created a new opaque token that embeds the tenant ID:

import crypto from "node:crypto";

const token = `${tenantId}-${crypto.randomBytes(16).toString("hex")}`;
console.log(token); // => 1-f82158a508b8bfbed82b601e2ed60edd

🔮 Previews

Nextcloud offered automatic previews of uploaded files, a feature we relied on extensively, so we needed to re-implement an equivalent ourselves.

We decided not to generate previews at upload, as this would have added significant complexity and cost, along with the challenge of handling asynchronous generation.

For PDFs, we just return an optimized preview of the first page, and for images, we use the Sharp library.

function getImageTransformer(
  query: { x?: string; y?: string; },
  ext: string
): sharp.Sharp {
  const { x, y } = getDimensions(query);
  const transformer = Azure.isAlphaImage(ext)
    ? sharp().png()
    : sharp({ failOn: "none" }).jpeg({ mozjpeg: true, quality: 50 });

  return transformer.resize({
    fit: "inside",
    withoutEnlargement: true,
    height: y,
    width: x
  });
}

📦 Headers and encoding

When returning documents, it’s essential to set the correct HTTP headers and apply proper encoding to values like file names.

const contentType = mime.contentType(
  path.extname(request.body.document)
);
const { body, contentLength } = await getFileFromAzure(request);
// pipe body to reply/response

reply.header(
  "Content-Disposition",
  `attachment; filename="${encodeURIComponent(filename)}"`
);
reply.header("Content-Type", contentType);
reply.header("Content-Length", contentLength);

I frequently see editors forget to re-inject the file name.

Monitoring

Being able to monitor developments and misuse is critical to guaranteeing the stability of our infrastructures.

Built-in file viewer

Since Nextcloud could display multiple documents within a viewer, we chose to re-implement a minimal yet functional viewer to retain this capability.

While our front-ends offer more advanced display modules, this lightweight viewer remains useful in several scenarios:

Replacing or rewriting legacy URLs in PDFs.
External links shared via APIs.
Providing quick access for debugging purposes.

2️⃣ Upload

Successfully prototyping an upload wasn’t as complex as expected… but, as always, the devil is in the details.

For inter-service uploads between Node applications, another Fastify plugin was added to our workspace package, providing methods to interact with the GED API 🔀.

📉 Optimize PDFs and images

Many of the PDFs and images submitted by our users are quite large and can be optimized. For this, we use Ghostscript 👻 to optimize PDFs and the Sharp package for images.

To date, we’ve reduced the size of received PDFs and images by an average of 50%, with no loss in quality ✨.

Compression is performed asynchronously using setImmediate to ensure fast server response times.
A compression value of "null" indicates that the compression ratio is below 5% 🤷‍♀️, making the update to the file on Azure negligible.
Otherwise, the file is updated in the cloud ✔.

Most of these optimizations are carried out via streams, so that the file or image is never completely buffered.

However, we needed to remain vigilant about rising CPU consumption and enhance our infrastructure setup 🏗️ to handle increased workloads effectively.

🖼️ HEIC/HEIF

Apple's proprietary HEIC format 📱 presented a significant challenge, often requiring conversion to JPG or PNG for compatibility.

Given that Python bindings to libheif showed much better performance, we initially opted to create our own N-API Node.js binding for libheif, using low-level libraries for rapid JPG and PNG conversion.

HEIF-converter

For maintenance reasons, we chose to use Sharp by building libvips directly on our machines and installing the necessary tools (libheif, mozjpeg, libpng, etc.).

🔒 Security

When managing file uploads and storage, vigilance is essential 🕵️‍♂️ in several areas:

Monitor for spoofed HTTP headers 🛡️, such as altered content-type headers.
Scan files for viruses and malicious content 🦠.

Otherwise, an attacker could misuse your brand and storage capabilities to distribute malicious content and compromise users 🚨.

Make it a habit to consult the OWASP cheat sheets to ensure maximum protection against errors and oversights: OWASP File Upload Cheat Sheet.

We used clamscan (which relies on ClamAV) to scan the files 👁️, and file-type to accurately identify the file type instead of relying solely on the request headers 🧨.

📊 Monitoring

As we regain control, it’s essential not to overlook usage monitoring through logs and other metrics.

3️⃣ Migrating Nextcloud documents

To gradually phase out our Nextcloud servers, we developped a temporary Node.js API 🦾, responsible for transferring resources from Nextcloud to Azure. This service handled upload concurrency, which we've limited to 64 simultaneous uploads to avoid overloading the server 🔥.

Without detailing every feature of this internal tool, it was designed to support key functionalities such as pausing and resuming the migration process, as well as monitoring the status of each transfers (successes ✅, errors ❌, totals, etc.).

Step 1: Data Extraction

We extracted from the Nextcloud database all the tokens 📄 (used to retrieve document data from the database) and the file paths on the server (to transfer the resources), saving them into .csv or .txt files.

Step 2: Environment Setup

We then set up a NAS server to run the Node.js tool and directly access the file system 🦄, bypassing the Nextcloud API. This approach was chosen to maximize performance and enable efficient stream-based, parallel processing of the document transfers.

Step 3: Create the DB and go 🧨

All that remained was to create the SQLite databases (we chose to generate one database per firm to avoid excessively large files), using the Nextcloud exports that contained tens of millions of rows, and then start the transfers ✅.

Let’s just say we ran into a few surprises along the way 🤫, and the migration ended up taking us several days 🤭.

4️⃣ Legacy URLs in PDF

Some URLs are permanently embedded in PDFs 📄, so we need to consider strategies for rewriting them using the information available.

Since Nextcloud tokens didn’t contain any tenant information, we created a minimal API (microservice) supported by an SQLite database to maintain the relationship between a token and its corresponding tenant ID.

PRAGMA journal_mode = OFF;
PRAGMA synchronous = 0;
PRAGMA locking_mode = EXCLUSIVE;

CREATE TABLE IF NOT EXISTS "tokens" (
  "token" TEXT PRIMARY KEY NOT NULL,
  "schema" INTEGER NOT NULL
) WITHOUT ROWID;

We can manage thousands of tokens within just a few milliseconds ⏱️ using purely synchronous I/O. Additionally, we implemented an LRU cache to ensure that repetitive requests are handled even more quickly.

The final step is to configure HAProxy 🔀 to redirect nextcloud viewer requests to a specific GED endpoint, where the URL is parsed to retrieve tokens and correlate them with their respective tenants, using the project setup described above.

const { link } = request.query;

if (!URL.canParse(link)) {
  // Problem
}
// Other URL validation here

const tokens = link.match(
  /(?<=\/)([1-9]{1,4}-\w{15,32}|\w{15})(?=\W|$)/g
);
// Correlate tokens with our microservice database

reply.redirect(`/ged/document/view?tokens=${correlatedTokens.join("|")}`);

This is only a partial overview of the implementation. We use a combination of the WHATWG URL API and regular expressions to extract tokens, ensuring sufficient security to mitigate any ReDoS attack vectors.

We then redirect the request with all tokens to our built-in viewer.

🔬 What we've learned

This project taught us that errors in URLs saved within PDF documents are hard to forgive. Due to some technical debt and a lack of foresight, we ended up with an unintended /ged/ged prefix. Today we're having a bit of a laugh about it, and if you see this prefix you'll know it wasn't meant to be 😆.

Managing files with proper streaming while handling errors proved far more challenging than anticipated, plaguing us for weeks with ghost files, memory leaks, and other unexpected bugs. At this level of usage, it’s technical excellence or nothing.

❤️ Credits

A migration project of this scale doesn’t happen overnight—it took us well over a year to complete all the steps outlined above. A big thank you 🙏 to everyone involved for their dedication and effort ❤️.

Nicolas 👨‍💻, for leading the project development from A to Z.
The infrastructure team 🏗️ (Vincent, Jean-Charles, and Cyril) for their consistent support throughout the project.
Aymeric, for managing and leading the migration of downloads and uploads for his team services.
Many others 👥 for their reviews and support 📝.

This project was incredibly rewarding 🏆, both for its challenges and the range of architectural issues it addressed 📐.

Thank you, see you soon for another technical adventure 😉😊

👋👋👋

Designing MyUnisoft Next-Gen Accounting APIs

Thomas.G — Thu, 04 Apr 2024 14:20:41 +0000

Hello 👋

I'm excited to present my latest technical article detailing the design and development process behind the next iteration of the MyUnisoft Set of accounting external APIs.

Internally known as MAD for MyUnisoft Accounting Data 😄, this project showcases our team's innovative approach and dedication to enhancing API and Partners experience.

💡 Context about the business requirements

You might be questioning the need for developing new APIs. We embarked on this effort to address a spectrum of issues and fulfill unmet needs, including:

🔄 Introducing a modern and efficient format for exporting and importing accounting folders, along with their parameters (particularly useful for migrations between tenants).
📊 Incorporating accounting data that was previously absent from historical partners APIs, such as Analytics.
🚀 Constructing high-performance, user-friendly APIs for our partners, adhering to modern API best practices.

When I refer to high performance, I'm not solely emphasizing faster response times. It also includes improving the security and strength of our systems by using cache and queues, as well as ensuring better observability.

👻 TRA

Historically, the CEGID TRA format has been predominant in the French market, but it faces a number of issues:

Tailored for CEGID's specific needs, many columns within the format prove irrelevant for MyUnisoft's purposes.
Featuring a column-oriented structure with fixed positions, it becomes resistant to evolution and challenging to implement and maintain 😡.
Parsing and generating data in this format incur significant costs and complexities 😞.

However, it can be difficult for software editors and accounting firms to adapt to unfamiliar formats quickly. That's why we've decided to be compatible with TRA.

The basic idea is to enable faster integration and smoother migration to our format. If this format has worked so far, we can learn a lot from it 😊.

💪 Accounting imputation for Factur-X

A significant disappointment lies in the realization that simply exchanging Factur-X via APIs will prove largely inadequate for creating a satisfactory experience for accounting editors and their customers (CPA), as the format does not permit embedding data for accounting imputation 😞.

To address this issue, we are planning to implement Factur-X natively within MAD. This format will enable the fields to be omitted, provided that the attached file is a valid Factur-X. Developers will only need to fill in the required fields for imputation purposes.

🚫 There will be no necessity for creating non-standard extensions of the XML format.

✒️ Technical specification

From the beginning, my aim was to develop accounting APIs that incorporate the following features:

Versioned interface contracts to establish lifecycles and facilitate iterative changes more smoothly.
Utilization of identical interfaces for both import and export functions, thereby simplifying API interactions.
🧠 Maximizing intelligence and independence to minimize dependency on other APIs. For instance, this includes eliminating the necessity for IDs or internal codes during imports.
Striving for neutrality when crafting the format, ensuring compatibility and flexibility across various systems and use cases.

Lastly, it's imperative to highlight the significance of neutrality in our approach. Too often, technical teams can fall into the trap of designing formats and APIs solely with internal use cases in mind, often under the pressure of immediate business requirements.

However, it's essential to recognize that our target audience primarily comprises developers who experience accounting through the lens of their own unique business contexts. By prioritizing neutrality in our design principles, we aim to simplify the lives of our consumers and empower them with flexible solutions that seamlessly adapt to diverse workflows.

Through MAD, we're committed to fostering an ecosystem that prioritizes developer-centric design, ensuring that our APIs serve as versatile tools capable of meeting a wide range of needs 🌐.

🔍 Gathering information on existing assets

The initial step involved conducting an inventory of existing APIs and comprehensively understanding the data consumed by our partners. We didn't have to do a lot of work, given that we have a nice public documentation available 😎!

We reviewed the APIs of several of our competitors, complemented by our experience integrating with various accounting editors, which guided us in making the right design decisions.

Additionally, we meticulously analyzed our TRA backend to extract the required columns for a preliminary version. This enabled us to identify the essential elements for a version 1, including:

accounting folder
payments
banks
journals
exercices
analytics (axes and their sections)
entries & movements

🔍 Our plan is to expand this list to incorporate more elements over time.

📚 Building interfaces

The second step involved crafting prototypes for our JSON object interfaces, utilizing TypeScript and JSON Schema.

Here is the finalized version for Journal, which underwent several weeks of iteration, concurrent with the creation of initial drafts of SQL.

interface SimplifiedAccount {
  producerId?: string;
  number: string;
  name: string;
}

type TypeJournal = "Achat" | "Vente" | "Banque";

interface Journal {
  producerId: string;
  name: string;
  customerReferenceCode: string;
  type: TypeJournal;
  counterpartAccount: SimplifiedAccount | null;
  additionalProducerProperties: {
    type: TypeJournalInternal;
    locked: boolean;
  };
}

💡 Properties that are too specific to MyUnisoft are stored in additionalProducerProperties

Simultaneously iterating on the SQL has proven essential, allowing us to eliminate keys that significantly impact performance without providing commensurate value.

Here is an another example with Bank

interface Bank {
  producerId: string;
  IBAN: string;
  BIC: string;
  account: SimplifiedAccount | null;
  journal: {
    producerId: string;
    name: string;
    type: "BANQUE";
    customerReferenceCode: string;
  };
  additionalProducerProperties: {
    isDefault: boolean;
  };
}

A significant effort has been invested in creating objects that are both clean and user-friendly.

🌀 (Bonus) Known your abstraction

We frequently fall into the trap of assuming that our abstractions are universally understood across a entire ecosystem. In accounting, for instance, it's typical to categorize transactions into an abstraction known as an "Entry." Each entry comprises a set of balanced movements (or transactions).

However, many software solutions, including Dataviz, often prefer consuming raw movements without additional abstractions. This preference explains why such solutions historically lean towards formats like FEC, which are simpler compared to TRA-heavy formats.

From my experience working with partners over the years, I've learned that many of them employ various abstractions for VAT, accounts, or analytics. Interestingly, the choice of abstraction can sometimes be influenced by the nature of their software.

The way we structure our data can play a vital role in ensuring a positive experience for specific consumers 💡.

💬 Technical implementation

For this project we decided to create an in-house package that would bring together all the necessary parts. This decision primarily aims to accommodate diverse use cases without the need for repetitive API (or SDK) integration.

The second step was to design a testable architecture for the package. For some parts I drew inspiration from open source libs such as undici and their mocking API.

At one point I was thinking that it might be a problem to implement HATEOAS... However, I discovered that with our architecture, dynamically injecting new links to specific resources using transformers is surprisingly straightforward 😅.

📦 Package usage

Here's a simple example

import * as MAD from "@myunisoft/mad";

const exporter = new MAD.Export({
  version: "1.x",
  accountingFolderId: "10",
  postgresDataSource
});

const { data: exercices } = await exporter.exercices();

Underneath we fetch elements from the database and transform them into the correct format, leveraging the SemVer version injected into the constructor.

Furthermore, we've integrated an observability layer to track the execution times of each component. However, this aspect likely requires further refinement 😕.

async exercices(): Promise<ExportResponse<postgre.Exercice[]>> {
  const startTime = performance.now();

  const rawExercices = await this.provider.exercices();
  const sqlTime = performance.now() - startTime;

  const transformed = this.transformer.exercices(rawExercices);

  return {
    data: transformed.data,
    executionTime: {
      sql: sqlTime,
      transformer: transformed.executionTime,
      total: performance.now() - startTime
    }
  };
}

The provider can simply be mocked up if you need to run unit tests (by forcing a global provider or forcing it using the class constructor).

const kGlobalDispatcher = mock.fn();

before(() => {
  MAD.setGlobalProvider(kGlobalDispatcher);
});

beforeEach(() => {
  kGlobalDispatcher.mock.resetCalls();
});

after(() => {
  MAD.resetGlobalProvider();
});

(Bonus) TRA compatibility

Effectively writing or reading a TRA isn't inherently complex, although it can be quite tedious due to the time-consuming process of properly implementing all its sections.

The challenge lies in being as fast as possible ⚡.

After one or two months of work, our JavaScript implementation can now read and write a complete TRA of several million lines in just a few hundred milliseconds 📈💻.

To achieve this, we've minimized the number of operations and iterations for each section. Additionally, each section is equipped with a comprehensive definition containing all column information, such as position, alignment, and length.

const config: TRAConfig = {
  section: "rib",
  JSONToTRA,
  TRAToJSON,
  columns: [
    {
      name: "fixe",
      mandatory: true,
      type: "a",
      length: 3,
      align: "left",
      position: 0
    },
    {
      name: "identifiant",
      mandatory: true,
      type: "a",
      length: 3,
      align: "left",
      position: 3
    }
    // ... other fields
  ]
}

Each section is equipped with functions designed for efficient conversion between TRA and JSON formats.

function JSONToTRA(
  lineObj: IRib,
  opts: TRAParsingOptions
): string {
  return [
    convertJSONValue(config.columns[0], lineObj.fixe, opts),
    convertJSONValue(config.columns[1], lineObj.identifiant, opts),
    convertJSONValue(config.columns[2], lineObj.auxiliaire, opts),
    // other fields..
    os.EOL
  ].join("");
}

We utilized tools like flamegraphs to pinpoint the longest-running elements and effectively eliminate them. Additionally, we optimized our process by implementing caching mechanisms to avoid redundant operations, such as repeatedly parsing the same dates.

Exploring a Rust implementation could be intriguing to further reduce execution time 🚀. Perhaps in the future, if performance becomes a critical concern 👀.

🌏 API

In parallel with the work done on the MAD package, we drew up a plan for creating the APIs 📝.

However, we encountered a challenge: some of the APIs we'll be exposing are potentially risky ⚠️, as they may permit exporting entire folders along with all associated parameters.

To address this, we're designing asynchronous APIs, incorporating status tracking and caching mechanisms integrated with our event architecture. This setup enables us to trigger webhooks as needed, enhancing security and reliability.

The sequence diagram above illustrates the interactions between the various components involved in exporting an entire accounting folder.

For this export, we have two distinct routes:

mad/all?format=json&version=1.0.0 to initialize the export.
mad/all/status?exportId=export_sch-5_acf-1 to fetch the status.

The second route will return a DONE status when processing is complete (Real-time webhook notification can be subscribed to as an option).

{
    "id": "export_sch-1_acf-1",
    "status": "DONE",
    "url": "<url>"
}

📜 Documentation

We've seamlessly extended our documentation, which is readily accessible on Github. Recently, we've enhanced the user experience by incorporating an online Vitepress site, hosted with Github Pages.

It required several weeks of effort, and the pivotal elements include:

📝 A comprehensive description outlining our decisions.
📋 A detailed specification encompassing all interfaces, schemas, etc.
📚 Additional insights, explanations, and guidance tailored for beginners in accounting.
🗺️ Instructions on navigating MyUnisoft for developers unfamiliar with our product.

My primary objective is to ensure a positive developer experience (DX) while maintaining thorough and accessible documentation. You're welcome to explore it yourself here, and we eagerly await your feedback 😊.

📆 What's next?

We're only at the very beginning of the project. We will continue to add new content to our export format and work on new options for our various APIs.

In parallel, we're working on the import (I'll probably have a chance to write a new article when we're well advanced on this).

❤️ Credits

This project is primarily a collaborative effort, made possible by the contributions of key team members:

Benoit GARIAZZO (Product Owner)
Alexandre MALAJ (Specification, MAD package & SQL).
Cédric LIONNET (API)
Pierre DEMAILLY (TRA/JSON compatibility & review)
Frédéric GUIOU (Help with naming and currently leading import)
Me 😁 (Specification, Observability, Documentation)
Léon and Aymeric for their invaluable assistance with existing accounting APIs & formats.

And all other team members who provided valuable feedback and support during the implementation process 🙌.

🙏 That concludes this article. Thank you for reading!

Consuming Loki logs with Grafana API and Node.js

Thomas.G — Wed, 25 Oct 2023 10:28:24 +0000

Hello 👋

In my last article we were discussing how me and my team had set up and built dashboards with Grafana and Loki 😎.

Logs monitoring with Loki, Node.js and Fastify.js

Thomas.G for MyUnisoft ・ Jun 12 '23

#node #javascript #monitoring #grafana

Today we're going to discuss how you can leverage your logs for third-party tools using Grafana's API and Node.js 😍.

💡 The idea that starts it all

Several months ago, I started thinking about how to use our logs to build a personalized CLI experience.

After a bit of searching, I quickly found that Grafana had an API for queryings Loki logs. I immediately went crazy at the thought of what could be done with it 🔥.

🐢🚀 Node.js SDK

Over the weekend, I set to create an open source Node.js SDK.

import { GrafanaLoki } from "@myunisoft/loki";

const api = new GrafanaLoki({
  // Note: if not provided, it will load process.env.GRAFANA_API_TOKEN
  apiToken: "...",
  remoteApiURL: "https://name.loki.com"
});

const logs = await api.queryRange(
  `{app="serviceName", env="production"}`,
  {
    start: "1d",
    limit: 200
  }
);
console.log(logs);

👀 Notice the support of duration for options like start and end.

It already supports Stream and Matrix and several other APIs:

GET /loki/api/v1/labels
GET /loki/api/v1/label/:name/values
GET /loki/api/v1/series

We've also added datasource APIs (which may be required depending on what you'r building).

It also include a LogParser inspired by Loki pattern.

// can be provided as an option to queryRange
const parser = new LogParser<{
  method: string, endpoint: string, statusCode: number
}>("<method:httpMethod> <endpoint> <statusCode:httpStatusCode>");

const logs = await api.queryRange(
  `... LogQL here ...`, { parser }
);
for (const logLine of logs) {
  console.log(logLine.method);
  console.log(logLine.endpoint);
}

It's still far from perfect, don't hesitate to contribute 💪.

OpenAlly / loki

Node.js Loki SDK

Loki

Node.js Grafana API SDK (Loki, Datasources ..)

🚧 Requirements

Node.js version 24 or higher

🚀 Getting Started

This package is available in the Node Package Repository and can be easily installed with npm or yarn

$ npm i @openally/loki
# or
$ yarn add @openally/loki

📚 Usage

import { GrafanaApi } from "@openally/loki";
import { LogQL, StreamSelector } from "@sigyn/logql";

const api = new GrafanaApi({
  authentication: {
    type: "bearer",
    token: process.env.GRAFANA_API_TOKEN!
  },
  remoteApiURL: "https://name.loki.com"
});

const ql = new LogQL(
  new StreamSelector({ app: "serviceName", env: "production" })
);
const logs = await api.Loki.queryRange(
  ql, // or string `{app="serviceName", env="production"}`
  {
    start: "1d",
    limit: 200
  }
);
console.log(logs);

…

View on GitHub

🔎 LogQL builder

We've also built a utility package to construct log queries, inspired by the WHATWG URL/URLSearchParams API.

The package lets you build any LogQL programmatically from scratch

import { LogQL } from "@sigyn/logql";

const logql = new LogQL("foo");
logql.streamSelector.set("env", "prod");
logql.lineEq("bar");
logql.lineNotEq("baz");

// {env=\"prod\"} |= `foo` |= `bar` != `baz`
console.log(logql.toString());

But also able to parse raw string 😇

const logql = new LogQL(
  "{app=\"foo\", env=\"preprod\"} |= `foo` != `bar`"
);

console.log([...logql.streamSelector.entries()]);
console.log(logql.lineFilters.lineContains());
console.log(logql.lineFilters.lineDoesNotContain());

You can use subclasses in your tools at any time, for example to retrieve labels from a LogQL 😮

import { StreamSelector } from "@sigyn/logql";

const logql = '...';
const labels = new StreamSelector(logql).kv();

📜 Credits

I'm not alone and didn't do all the work 👯. Thanks to Pierre Demailly and Sofian Doual for their contribution/support.

🙏 Conclusion

These two tools have enabled us to initiate the development of new tools. One of them is an alerting agent for Loki named Sigyn, to whom I'll be dedicating an article in the near future.

Having APIs really opens up a lot of opportunities and it'll be interesting to see what me and my team are able to produce in the future with them.

Best Regards,
Thomas

Logs monitoring with Loki, Node.js and Fastify.js

Thomas.G — Mon, 12 Jun 2023 12:45:10 +0000

Hello 👋

Over the past few months, I've been spending a lot of time creating dashboards on Grafana using Loki for MyUnisoft (the company I work for).

So I decided to write you an article to explain my adventure from the point of view of a back-end Node.js developer.

👀 Why grafana and loki?

For context, the tool was implemented by a DevOps employee who left the company two years ago. So I didn't really choose the tool myself. I had no previous experience with either of these tools, so I had to experiment and discover as I went along.

Grafana is pretty well-known and mature, so I wasn't worried. On the other side, I quickly liked Loki to manage and search my logs. So I thought, "I've got to learn to make dashboards out of this".

📃 Writing good logs

💬 Several examples of logs are multiline to avoid scrolling and simplify reading.

When you write logs, you must constantly think about the information they will provide and make sure they can be analyzed without too much difficulty 😵. It always takes several iterations to get a good result.

In my experience, you can find two kinds of logs:

The debug logs you know are useless to your dashboards. They'll come in handy in case of bugs or when you're looking for more context about a given request.
All other logs that can provide exploitable data for dashboards (and for a human reader)

Here an example of a debug log we print when our services start (quite useful to customize the --max-old-space-size flag).

const availableHeapSize = prettyBytes(
  v8.getHeapStatistics().total_available_size
);
server.log.info(`Total available heap size: ${availableHeapSize}`);

Another common example is logging a JSON payload (you could exploit the data, of course, but that's often not the end goal).

this.entry.once("payloadReady", (payload) => {
  this.log("payload before publishing:");
  this.log(JSON.stringify(payload, null, 2));
});

😱 Be sure to redact the data (you may divulge confidential information without realizing it).

Most of the time, you won't get anything out of these logs, and they don't require any particular effort. In this article, I'm going to concentrate on the logs that we can exploit for our dashboards.

🐤 Practical example

Here's an example of bad logs we initially set up for a document upload middleware/plugin

Successfully uploaded '...'
//
Failed to upload '...'

There are several problems:

No scope (we can't easily search for all the logs related to our middleware).
Can easily be mixed with other logs (from other middlewares or services).
It can be quite a challenge to parse the state (failed or successful).
Lack information about the request or the user.

A better way of writing it:

[uploader|req-x5d4] document 'name.jpg' uploaded (state: ok)

You can easily extend the log with other information such as extension, size, runtime, etc. (without breaking the Loki pattern or regexp).

(state: ok|ext: .jpg|size: 52.5 kB|upload-time: 0.503 ms)

⚠️ Always use the same unit when logging size (bytes) or time (milliseconds). Setting the right 'unit' on Grafana will do the work of cleaning the value for you.

📜 Format clarification

A good log has to be structured to allow good searches with LogQL. Here's a bad example where it will be really difficult to extract information.

hello-world.jpg 52.5 kB|0.503 ms

I would recommend adding a label before each value (it's also easier for a human to read). Having start, end, and separator characters can also be a great help.

doc 'hello-world.jpg' [size: 52.5 kB|exec: 0.503 ms]

Here the required RegExp to parse all labels with Loki

doc '(?P<doc>\S+)' \[size: (?P<size>\S+) kB|exec: (?P<exec>\S+) ms\]

Loki 2.0 also allows you to retrieve labels with a simplified syntax called pattern.

pattern `doc '<doc>' [size: <size> kB|exec: <exec> ms]`

🔎 Implementation in the framework/code

⚠️ code examples are simplified/truncated

The Fastify framework includes the Pino logger by default (a really great logger with lots of cool features that doesn't compromise on performance). The framework itself allows a lot of really cool stuff, like controlling the level of logs at runtime.

On my team, we have chosen to customize the default request and response logs to include additional information. To achieve this, you need:

setup fastify constructor option disableRequestLogging to true
exploit two hooks (onRequest and onResponse).

server.decorateRequest("standardLog", null);
server.addHook("onRequest", async(request: FastifyRequest) => {
  request.log.info(
    `(${request.id}) receiving request ...`
  );
  request.standardLog = standardLog.bind(request);
});

server.addHook("onResponse", async(request: FastifyRequest) => {
  request.log.info(
    request.standardLog(
      `response returned "${request.method} ${request.raw.url}"
    )
  );
});

The standardLog decorator allows us to display information about the request and the token in use (for authenticated endpoints).

We have to handle several types of tokens (it all depends on who is consuming the API). A classical user or a partner through our partner API).

Each of them includes:

The postgreSQL schema of the customer
The user's or third-party's ID
The accounting folder (which can be null)

function standardLog(
  this: FastifyRequest,
  msg: string
): string {
  if (
    this.server.hasRequestDecorator("tokenInfo") &&
    this.tokenInfo !== null) {
    const { tokenInfo } = this;
    let tokenInfoLog = `${tokenInfo.type}`;

    switch (tokenInfo.type) {
      case "api":
        tokenInfoLog += `|s:${...}|t:${...}|acf:${...}`;
        break;
      case "user":
        tokenInfoLog += `|s:${...}|p:${...}|acf:${...}`;
        break;
      default:
        tokenInfoLog = "";
        break;
    }

    return `(${this.id}|${tokenInfoLog}) ${msg}`;
  }

  return `(${this.id}|none) ${msg}`;
}

Here's what it looks like in the logs

(req-1rti) receiving request "POST /ged/base-docs/docs"
(req-1rti|user|s:538|p:1|acf:23) response returned
"POST /ged/base-docs/docs", statusCode: 201 (460.106ms)

These informations will be critical to populating our dashboards with information about customers and the scope of actions.

🌀 Logs ingestion

My team currently uses promtail with a YML configuration to retrieve service logs.

server:
  http_listen_port: 9080
positions:
  filename: /var/lib/promtail/positions.yml
clients:
  - url: https://xxx.fr/loki/api/v1/push

scrape_configs:
  - job_name: svc_api_dev
    static_configs:
    - labels:
        __path__: /home/xxx/logs/service-dev-*.log
        app: api
        env: dev
        host: xxx.fr
        job: svc_api_dev
      targets:
      - localhost

I'm not going to spend too much time on this chapter, as you can find a lot of tutorials on the Internet on how to set up and configure promtail.

💡 If you don't want to depend on promtail for sending the logs to Loki, you can use the fastify plugin pino-loki

📊 OCR Dashboard

MyUnisoft is a French accounting editor, so Optical Character Recognition is an important feature of our software. Monitoring is essential to adapting to the needs of our customers and responding promptly to incidents.

👀😍 We won the OCR silver medal 2023 awarded by Le Monde du Chiffre.

Here is an example of a dashboard we have built just by using the following log:

(req-2987|user|s:24|p:5710|acf:7398) OCR xxx.jpg
[type: invoice|ext: .jpg|size: 2925.73 kB]

Group by label

We can construct the above graph with the following logQL. Notice the sum by extension.

sum(
  count_over_time(
    {app="ocr",env="$env"}
      |= "OCR"
      |= "|ext:"
      | regexp `\|ext: \.(?P<extension>\S+)\|`
  [$__range])
) by (extension)

RegExp are built using Golang syntax (I personnaly use regex101.com to test them). It is useful here to extract/detect the extension label.

We can use the syntax $env to inject a dashboard variable (here I can view my dashboard in production, staging or dev just by moving a value in a select).

I also use the built-in $__range variable, which loads data over the currently selected range in Grafana.

Unwrap

It took me a long time to really understand unwrap. There is virtually no clear documentation or explanation on the Internet!

Here it will use the size label and extract all values to compute them with min_over_time and min functions. This is useful for extracting numerical values (counter, execution time, etc.).

min(
  min_over_time(
    {app="ocr",env="$env"}
      |= "OCR"
      | regexp `\|size: (?P<size>[.0-9]+) kB\]`
      | unwrap size
      | __error__ = ""
  [$__range])
) by (app)

💬 | __error__ = "" avoid crashing Loki with unexpected Error (could happen if unwrap of size fail for any reasons).

Then, on the right side of the stat graph, we select the corresponding unit.

💡 Tips and tricks

display name

For a long time, I had some really bad raw labels in my charts (with a format similar to JSON).

You can customize it by editing the display name option. You can use variables in this field to retrieve label values directly.

no data

Sometimes the graphs will display "no data" because no logs have been detected in the selected range. This can be problematic in certain graphics (such as gauges).

But maybe you'd rather have zero. No problem, just use the No value option.

filter using a regexp and variable

On some dashboards, you will probably want to filter dynamically according to several criteria. One way of doing this is to use a regexp and a dashboard variable.

This is what I do in one of my dashboards to filter the results for one or more partners.

count(
  sum(
    count_over_time(
      {app="api",env="production"}
        |= "] CALL"
        |= "$endpoint"
        |~ `\]\[$thirdparty\]`
        | regexp `\((?P<schemaId>[0-9]+):(?P<folderId>[0-9]+)\)`
    [$__range])
  ) by (schemaId, folderId)
)

Here's the line that does the job: you need to use backticks syntax to be able to inject a variable in it.

|~ `\]\[$thirdparty\]`

🚀 Data is the key to success

As developers, we don't realize enough how monitoring and the data it generates can be powerful sources of improvement. At MyUnisoft, we work with over a hundred partners.

Understanding how they use our APIs and the various anomalies they experience has become essential for us to continue to grow exponentially.

This allows us to continually improve to provide a better experience for both our accounting customers and our developer partners who maintain the integrations.

Our ability to materialize and exploit this data has become an essential part of my team's expertise. It's really exciting to see what can be achieved with simple logs.

👋 Conclusion

I'm so glad I finally managed to write this article (and I hope it brought you some value). Many thanks to my team and especially to Cédric, without whom I would not have realized all that.

I still struggle with a lot of Grafana features, such as transformations or alerts, but I will continue to dig and improve.

🙏 Thanks for reading me 🙏

Moving MyUnisoft Node.js back to TypeORM

Thomas.G — Tue, 02 Aug 2022 18:53:46 +0000

Hello 👋,

Recently I took the time to reflect on my last two years at MyUnisoft. I finally told myself that I wasn't writing enough about the difficulties we faced with my team 😊.

Today I decided to write an article about our transition to TypeORM. A choice we made over a year ago with my colleague Alexandre MALAJ who joined a few months after me.

We'll see why and how this choice allowed us to enhance the overall DX for my team 🚀. And that in the end it was a lot of trade-offs, and obviously, far from a perfect solution too.

🔍 The problem

At MyUnisoft we work with a PostgreSQL database with static and dynamic schema (each client is isolated in one schema). And uniquely without counting the duplication of the schemas we have about 500 tables.

The Node.js stack was split into several services coupled to the database (or to third-party services for some of them). Developers before us were writing raw queries and there were no unit or functional tests 😬. When I took over as lead it was hell to succeed in testing each service properly. Among the painful things 😱:

strong coupling.
heavy docker configuration
complexity to generate business data for our tests.

We had to find a solution to improve and secure our developments while iterating on production releases 😵.

Decentralizing with events wasn’t a possibility since of existing codes and dependencies (and we had no DevOps at the time).

💡 The solution

We started thinking about creating an internal package that would serve as an abstraction to interact with the database. We don't want to go for microservices 😉, so having a package that centralizes all this seems like a good compromise for us.

Among our main objectives:

Generate a compliant database locally or on Docker.
Easily generate fake data.
Built to allow us to carry out our functional and business tests.
Centralized code review (which also allows us to track changes more easily)

We also had ideas like building a schema in a running database (which could be used for partner API testing and sandboxing).

The question remained whether we should continue writing raw queries or not 😨. I'm not necessarily a big fan of ORMs, but we had a diversity of tables and requirements that made the writing of raw queries complicated at time.

We looked at the different solutions in the ecosystem by checking our constraints with the schemas. After must research, we concluded that TypeORM was viable (other ORM had critical issues).

Far from perfect, but we had to give it a try 💃!

Note: with hindsight, we are now also very interested in Massive.js. This could have been one of our choices.

🐥 Let the story begin

👶 Baby steps

My colleague Alexandre spent several months migrating the database on TypeORM 😮. I helped him by reviewing each table and relations.

We did not opt for a migration script at the time (for us the choice was still too vague).

We have made a gource to illustrate our work:

One of the problems we quickly encountered was that it was not possible to use the ActiveRecord pattern with dynamic schemas 😭. However this is ok for static schema because you can define them with the @Entity decorator.

@Entity({ schema: "sch_interglobal" })
export class JefactureWebhook extends BaseEntity {}

The management of datasources (connection) by schema/client was a bit infernal. We created our abstraction on top of TypeORM to handle all this properly and regarding our schema initialization requirements.

One of our encounters being quite complicated has been to clone a schema when we add a new client on the fly 🐝(that's something we do in our tests, in the authentication service for example).

We were able to achieve this by using the @EventSubscriber decorator on a static table we use to register new customers’ information.

@EventSubscriber()
export default class Sub_GroupeMembre {
  listenTo() {
    return Entities.schInterglobal.GroupeMembre;
  }

  async afterInsert(event: UpdateEvent) {
    const { idGroupeMembre } = event.entity!;

    const queryManager = datasources.get("default")!;
    await queryManager.query(
      `SELECT clone_schema('sch1', 'sch${idGroupeMembre}')`
    );

    const connection = await (new DataSource({})).initialize();
    datasources.set(`sch${idGroupeMembre}`, connection);
  }
}

The tricky part was to build an SQL script to properly clone a schema with all tables, relations, foreign keys etc.. But after many difficulties we still managed to get out of it 😅.

📜 Blueprints

When I started this project I was inspired by Lucid which is the ORM of the Adonis.js framework.

By the way, Lucid was one of our choices, but like many of Harminder's packages, it is sometimes difficult to use them outside of Adonis (which is not a criticism, it is sometimes understandable when the goal is to build a great DX for a framework).

But I was quite a fan of Lucid's factory API so we built an equivalent with TypeORM that we called "Blueprint".

Here is an example of a blueprint:

new Blueprint<IConnectorLogs>(ConnectorLogsEntity, (faker) => {
  return {
    severity: faker.helpers.arrayElement(
      Object.values(connectorLogSeverities)
    ),
    message: faker.lorem.sentence(5),
    public: faker.datatype.boolean(),
    requestId: faker.datatype.uuid(),
    readedAt: null,
    createdAt: faker.date.past(),
    thirdPartyId: String(faker.datatype.number({
      min: 1, max: 10 
    })),
    idSociete: null
  };
});

The callback includes the faker lib as well as internal custom functions to generate accounting data. You can use this blueprint to generate data like this:

const user = await Blueprints.sch.ConnectorLogs
  .merge({ readedAt: new Date() })
  .create();

The API is similar but it appears our objectives and TypeORM forced us to make different choices.

ES6 Proxy usage

You may have noticed but something is weird with this API. Every time you hit Blueprints.sch it triggers an ES6 proxy trap that will return a new instance of a given Blueprint.

It was quite satisfying for me to manage to use a Proxy for a real need and at the same time manage to return the right type with TypeScript.

import * as schBlueprints from "./sch/index";
import { Blueprint, EntityBlueprint } from "../blueprint";

// CONSTANTS
const kProxyHandler = {
  get(obj: any, prop: any) {
    return prop in obj ? obj[prop].build() : null;
  }
};

type EmulateBlueprint<T> = T extends Blueprint<infer E, infer S> ?
  EntityBlueprint<E, S> : never;
type DeepEmulateBlueprint<Blueprints> = {
  [name in keyof Blueprints]: EmulateBlueprint<Blueprints[name]>;
}

export const sch = new Proxy(
  schBlueprints, kProxyHandler
) as DeepEmulateBlueprint<typeof schBlueprints>;

📟 Seeder

We worked from the beginning of the project to build a relatively simple seeding API. The idea was mainly to be able to generate the static data required for our services to work properly.

Here's an example of one simple seed script that generates static data with a blueprint:

export default async function run(options: SeederRunOptions) {
  const { seeder } = options;

  await seeder.lock("sch_global.profil");
  await sch.PersPhysique
    .with("doubleAuthRecoveryCodes", 6)
    .createMany(10);

  seeder.emit("loadedTable", tableName);
}

When we generate a new database locally or in Docker we can see the execution of all the seeds:

🌀 Docker and testcontainers

When Tony Gorez was still working with us at MyUnisoft he was one of the first to work around how we can set up our tests inside a Docker and run them in our GitLab CI.

The execution of our tests was relatively long (time to build the Docker etc). That's when he told us about something a friend had recommended to him: testcontainers for Node.js.

Once set up but what a magical feeling... The execution of our tests was faster by a ratio of 4x. Tony has been a great help and his work has allowed us to build the foundation of the tests for our services.

On my side I worked on an internal abstraction allowing everyone not to lose time on setup:

require("dotenv").config();
const testcontainers = require("@myunisoft/testcontainers");

module.exports = async function globalSetup() {
  await testcontainers.start({
    containers: new Set(["postgres", "redis"]),
    pgInitOptions: {
      seedsOptions: {
        tables: [
          "sch_interglobal/groupeMembre",
          "sch_global/thirdPartyApiCategory"
        ]
      }
    }
  });
};

📦 Difficulties with a package 😱

Not everything in the process goes smoothly 😕. In the beginning, it was really difficult to manage the versioning. We used to use npm link a lot to work with our local projects but it was far from perfect (it was more like hell 😈).

And by the way, you have to be very careful with everything related to NPM peerDependencies (especially with TypeScript). If you use a version of typeorm in the package, you necessarily must use the same one in the service otherwise you will have problems with types that do not match.

"peerDependencies": {
  "@myunisoft/postgre-installer": "^1.12.1"
}

We had the same issue with our internal Fastify plugin. It cost us a few days sometimes the time to understand that we had screwed up well on the subject 🙈.

In the end, after some stabilizations, we could release new versions very quickly.

I'm not necessarily completely satisfied with the DX on this subject at the moment and I'm thinking of improving it with automatic releases using our commits.

Others APIs

I couldn't even cover everything because this project is so large. For example, we have a snapshot API that allows us to save and delete data during our tests...

Speaking of tests, it is always difficult to give you examples without being boring. But there too the work was colossal.

I would like to underline the work of Cédric Lionnet who has always been at the forefront when it came to solidifying our tests.

💸 Hard work pays off

After one year of hard work the project is starting to be actively used by the whole team across all HTTP services 😍. Everyone starts to actively contribute (and a dozen developers on a project is a pretty interesting strike force ⚡).

Sure we had a lot of issues but we managed to solve them one by one 💪 (I'm not even talking about the migration to TypeORM 3.x 😭).

But thanks to our effort, we are finally able to significantly improve the testing within our Node.js services. We can also start to work in localhost whereas before, developers used remote environments.

In two years we have managed to recreate a healthy development environment with good practices and unit and functional testing on almost all our projects.

📢 My take on TypeORM

If I were in the same situation tomorrow I would probably try another way/solution (like Massive.js). For example, TypeORM poor performance will probably be a topic in the future for my team.

As I said at the beginning, I'm not a fan of ORMs and in the context of personal projects, I do without them almost all the time.

However, I must admit that we succeeded with TypeORM and that the result is not too bad either. There is probably no silver bullet 🤷.

🙇 Conclusion

Many engineers would have given up at the beginning thinking that it would not be worth the energy to fight 😰.

It's a bit simple to always want to start from scratch 😝. For me it was a challenge, to face reality which is sometimes hard to accept and forces us to make different choices 😉.

It was also a great team effort with a lot of trusts 👯. We had invested a lot and as a lead I was afraid I had made the wrong choice. But with Alexandre it is always a pleasure to see that today all this pays off.

I'm not quoting everyone but thanks to those who actively helped and worked on the project especially in the early stage.

Thanks for reading and as usual see you soon for a new article 😘

MyUnisoft - l'aventure Node.js

Thomas.G — Mon, 13 Sep 2021 06:46:31 +0000

Bienvenue voyageur(se) 👋

Aujourd'hui je viens vous conter mon aventure chez MyUnisoft en tant que lead technique back-end (API & Node.js). C'est aussi celle de mon équipe qui continue de grandir en embarquant des ingénieurs très talentueux 😍.

Si vous êtes un (expert-)comptable alors je vais vous embarquer dans un récit qui s'éloigne probablement de ce que vous avez l'habitude de lire 📰. Mais pas d'inquiétude je ferais l'effort de vous vulgariser au maximum mon univers.

Qui suis-je ?

Moi c'est Thomas, j'ai 27 ans et je développe depuis l'âge de dix ans 🐤. Je suis un amoureux du code et j'entreprends des projets depuis mon plus jeune âge.

Je suis un expert Node.js et JavaScript. Fort aise sur des sujets comme la sécurité, le monitoring et l'architecture logicielle. Si mon parcours vous intéresse 👀 je vous invite à consulter mon LinkedIn.

Chapitre 1

Découvrons sans attendre le premier chapitre 💃.

Genèse

J'ai rejoint MyUnisoft en aout 2020 pour m'occuper de la maintenance et évolution du back-end Node.js 🐢. À ce moment-là je suis le seul développeur et ma première préoccupation est évidemment de faire mes preuves auprès de Cyril (CTO) et Régis (CEO).

J'ai commencé par travailler sur la mise en place du connecteur Quickbooks pour ensuite très vite m'attaquer à l'évolution de l'API partenaires (qui servira aussi de fondation plus tard pour l'accès cabinet).

L'écriture d'une documentation a été évidemment un des gros points pour garantir une meilleure expérience à nos partenaires (expérience que nous continuerons d'améliorer dans le temps).

Ces premiers chantiers m'ont permis d'avoir une première approche du domaine de la comptabilité en abordant plusieurs notions comme les journaux, le plan comptable, les écritures, etc 😵.

Par ailleurs, je souhaite remercier Leon Souvannavong qui m'a beaucoup aidé sur les sujets métiers depuis mon intégration (ainsi que les autres développeurs de l'équipe back-end comptabilité 💖).

Novembre 2020

Quelques mois passe et nous intégrons un second développeur en alternance 👯. Ayant déjà une forte expérience en mentorat je ne m'inquiète pas sur le fait de réussir à accompagner convenablement un débutant. Nous recrutons donc Nicolas Hallaert qui ne cessera de me surprendre dans sa vitesse d'adaptation et d'apprentissage ⚡.

Lui et moi avons travaillé ensemble sur divers sujets comme MyDataRH, le SSO, ou encore des interfaces génériques que vous retrouverez dans nos diverses interconnexions partenaires.

Mon périmètre s'étend de plus en plus et je monte rapidement en confiance. Dans la même période Oleh Sych rejoint l'équipe Node.js (développeur non francophone).

📌 Le seul développeur que je n'ai pas personnellement choisi. Au début j'avais un peu peur mais j'ai été très rapidement étonné par son niveau technique et sa réactivité à mes remarques.

Nous convenons très rapidement qu'il travaillera sur la mise à jour et migration de code "legacy" (écrit par des développeurs qui ne sont plus là). J'essaye de l'accompagner et de l'intégrer le mieux possible pour que la barrière de la langue ne soit pas un frein pour lui ✔️.

En écrivant ces lignes aujourd'hui je peux témoigner du chemin parcouru avec lui. Nous allons de l'avant sur plusieurs projets (Gestion Électronique des Documents, Discussion, Crédit-bail entre autres).

Janvier 2021

Après avoir démontré mes capacités et acquit la confiance de la direction je prends officiellement le lead de l'équipe Node.js 🎉. C'est un rôle qui me convient bien et j'ai toujours apprécié ce genre de responsabilité.

J'interviens de plus en plus sur des sujets en lien avec l'authentification 🔑 et je prends rapidement la main dessus.

Le reste de mon temps est dédié à la création d'un nouveau connecteur API avec Dext.

Février 2021

Une période chargée puisque nous avons embarqué deux nouveaux développeurs expérimentés dans l'équipe.

Le premier étant mon associé de longue date Alexandre MALAJ avec qui je travaille en binôme depuis maintenant plus d'une décennie 😲.
Le second est Cédric LIONNET qui nous a été recommandé en interne. Il entame une transition vers Node.js après plusieurs années de C++. C'est un ingénieur rigoureux ainsi qu'un amoureux de la qualité de code 💎.

Ces deux intégrations ont été le point de départ de ce qui est aujourd'hui la fondation de l'équipe Node.js.

Alexandre a investi des centaines d'heures sur la création d'une couche ORM (contenant +500 tables et +2,000 relations). Cédric de son côté à grandement contribuer à l'ajout de tests unitaires et abstractions qui sont aujourd'hui activement utilisées au travers de nos services http.

Fort de mon expérience de plus quatre ans en gestion d'équipe remote, nous travaillons rapidement à la mise en place de conventions et d'un modèle de communication efficace.

Il est primordial de construire une bonne entente ainsi que diverses habitudes de communication orale pour pouvoir rapidement acquérir une symbiose des compétences techniques et humaines.

Mars 2021

Je commence à travailler sur l'intégration d'un nouveau connecteur avec EmaSphere.

Quand Nicolas n'est pas en cours il travaille sur l'intégration SSO avec Zendesk (support) et 360 learning (MyAcademy). Sur le côté il travaille sur le Google sheet (les liens dynamiques).

Avec Alexandre nous avons décidé de lancer une initiative DDD (Domain Driven Design) au sein de MyUnisoft.

Amener de la qualité et de la rigueur dans les échanges et la conception du logiciel est pour moi très important. Insuffler une meilleure compréhension du métier aux équipes techniques apporteraient énormément de valeurs à nos clients.

Avril 2021

J'accompagne très activement de plus en plus de partenaires 😎. Le catalogue des connecteurs ne cessent de grandir ce qui me fait vraiment plaisir 😇.

Et encore beaucoup d'autres intégrations sont à venir d'ici fin 2021. Nous travaillons en ce moment même sur une mise à jour conséquente qui aura pour objectif d'apporter un ensemble de fonctionnalités manquantes (paramétrages, logs ...).

Avec l'équipe nous participons à la ludum dare 48 qui consiste à créer un jeu vidéo en 72h. Nous avons créé un jeu web utilisant le moteur Pixi.js (projet ici).

Une expérience très enrichissante qui nous aura permis de mieux nous connaître et de renforcer nos liens.

Mai 2021

L'équipe intègre deux développeurs supplémentaires:

Tan Karasu qui nous rejoint pour un stage de six mois. Développeur en reconversion qui a su me convaincre par son mental et son investissement.
Mark Malaj cousin d'Alexandre. Nous avions déjà eu l'occasion de collaborer ensemble pendant une année, période pendant laquelle je l'ai formé à Node.js. C'est naturellement un plaisir pour moi de pouvoir recollaborer avec lui au sein de MyUnisoft.

Alexandre et Mark travailleront en collaboration avec Jean-Claude FORTIER sur la conception et le développement de la Gestion Interne MyUnisoft. Un chantier qui est donc entre de bonnes mains.

Tan de son côté aura investi énormément de temps sur la création de nouvelle abstractions pour communiquer avec notre base de données Redis. Par ailleurs, nos projets utiliseront l'excellent package ioredis.

Juin 2021

J'ai eu l'occasion de travailler sur l'implémentation et l'intégration du format Factur-X pour nos partenaires (actuellement utilisé en production par EBP).

Une bonne occasion de jouer avec les nouveaux types de TypeScript 4 pour convertir dynamiquement les structures XML en type JSON propre.

J'éprouve une certaine fatigue à cause des différents onboardings. C'est une première pour moi de gérer autant d'intégration en si peu de temps (même si cela reste une excellente expérience).

Il est parfois difficile de jongler entre ma vélocité personnelle qui me permet d'avancer des sujets métier critique et investir du temps en accompagnement de mon équipe (ce qui améliore probablement la vélocité à moyen-long terme).

Aout 2021

L'équipe continue de se structurer 🔨 dans le bon sens et nous avançons positivement sur nos sujets. La période est relativement calme à cause des différents départs en vacances 🌞.

Nous intégrons néanmoins encore deux développeurs expérimentés:

Quentin Lepateley travaillant sur le frontend MyUnisoft depuis un an et demi. Ce n'est donc pas un petit nouveau et il arrive dans l'équipe en étant déjà familier avec les membres de l'équipe.
Tony Gorez nous venant tout droit de Payfit. Je travaille depuis maintenant une bonne année avec lui sur des projets open source comme NodeSecure. C'est vraiment un grand plaisir de pouvoir travailler avec lui au sein de la même équipe!

Quentin travaille activement sur notre migration vers le framework Fastify.js. L'idée est de rapidement mettre en place un monorepo utilisant la fonctionnalité de workspace npm 7 pour héberger les différents plugins utilisés sur nos services.

Tony quant à lui va rapidement venir m'épauler sur les intégrations partenaires. À court terme il travaillera sur la stabilisation du connecteur Quickbooks.

Mon sentiment sur l'équipe

Il reste du chemin à parcourir c'est une certitude. Nous devons apprendre à mieux nous connaître et comprendre qu'elles sont les forces et faiblesses de chacun.

Nous devons définir qu'elles seront nos pratiques et méthodologies tout en prenant évidemment en compte le contexte et les équipes qui nous entourent.

Mais je suis très enthousiaste. Nous avons beaucoup d'appétence pour notre métier et une grande motivation à faire devenir réalité les ambitions de MyUnisoft.

En avant pour un second chapitre ?

Nous continuons de grandir et nombreux sont les challenges devant nous. De belle intégration sont encore à venir et je pense que MyUnisoft constitue l'une des meilleures équipes Node.js francophone 💪.

C'est pour moi une fierté d'être à la tête d'un groupe d'ingénieurs que j'apprécie et respecte 🙇. J'ai vraiment hâte de voir ce que nous allons accomplir dans les prochains mois 🚀.

🙏 Merci à vous de m'avoir lu.

Cet article a été volontairement épuré de beaucoup de détails techniques (mais j'espère tout de même avoir réussi à accrocher un peu de votre attention).

Nous écrirons certainement plus d'articles à l'avenir pour vous parler de nos innovations et avancement technique.

🚀🚀🚀