DEV Community: Thomas.G

NodeSecure hidden capability: mama

Thomas.G — Sat, 10 Jan 2026 07:12:29 +0000

Hello 👋

I'm starting a new short-article series focused on highlighting lesser-known parts of the NodeSecure project. The goal is to help new contributors by giving them a clearer view of the back-end building blocks that power the project.

Chapter 1: mama

Mama stand for ManifestManager. This package was designed to manage and load an npm manifest (a package.json file, for simplicity).

Under the hood, it uses @nodesecure/npm-types to provide precise, up-to-date types (including runtime-related fields). We will dive into that package in another article.

import { ManifestManager } from "@nodesecure/mama";

// synchronous version: ManifestManager.fromPackageJSONSync
const mama = await ManifestManager.fromPackageJSON(
  process.cwd()
);
console.log(mama.document);

This package provides many utilities used across back-end components in the Scanner monorepo.

Here are a few of them:

Integrity

You can easily extract a hash by using the readonly getter integrity.

console.log(mama.integrity);

Scanner uses this to assert that the package.json in the tarball matches the one uploaded to the registry (known as manifest confusion).

Here are the properties we hash:

{
  name,
  version,
  dependencies,
  license: license ?? "NONE",
  scripts
}

When a mismatch is detected, the tool reports it as a global warning, as shown in the CLI UI:

module type

Inspired by the recent node-modules-inspector tool built by Antfu, we re-implemented the same module type detection:

// "dts" | "faux" | "dual" | "esm" | "cjs"
console.log(mama.moduleType);

entry files

mama can recursively extract entry files using the Node.js exports field (or legacy fields like main). The API is lazy and returns an IterableIterator<string>:

console.log([...mama.getEntryFiles()]);

This API is used in the tarball package in combination with JS-X-Ray’s EntryFilesAnalyser.

async scanFiles(): Promise<ScannedFilesResult> {
  const location = this.manifest.location;
  const [
    composition,
    spdx
  ] = await Promise.all([
    getTarballComposition(location),
    conformance.extractLicenses(location)
  ]);

  const code = await new SourceCodeScanner(this.manifest).iterate({
    manifest: [...this.manifest.getEntryFiles()]
      .flatMap(filterJavaScriptFiles()),
    javascript: composition.files
      .flatMap(filterJavaScriptFiles())
  });

  return {
    conformance: spdx,
    composition,
    code
  };
}

author

Parse the NPM author field if present and then return a Contact interface.

interface Contact {
  email?: string;
  url?: string;
  name: string;
}

For example, John Doe <john.doe@gmail.com> produces the following object:

{
  "name": "John Doe",
  "email": "john.doe@gmail.com"
}

Others

The module also provides additional utilities around reading and managing manifests, such as:

Parsing package specs (including scope/org, package name, and semver range)
Detecting local lockfiles

The end

The full module documentation is available here.

Thanks you for reading

Migrating from Nextcloud to Azure S3

Thomas.G — Tue, 20 May 2025 12:14:44 +0000

Hello 👋

Back for a new MyUnisoft technical article, this time with the help of my colleague Nicolas MARTEAU. Today, we will share our journey to completely refactor our document management architecture and how we migrated from Nextcloud to Azure S3 as our storage technology.

We weren’t able to cover every detail—both for security reasons 🛡️ and to protect sensitive data 🔒—but I hope you will enjoy what I could share. 😊

👀 Why moving away from Nextcloud ?

Performance 🤖

Until now, we have managed several tens of millions of documents with Nextcloud. However, stability and performance had become an issue, with regular downtime 🕒 and delays ⏳ of several minutes for a simple document upload at times.

💬 These upload delays sometimes led to misunderstandings among users. For example, in certain integrations, it was not uncommon for users to delete their Accounting entries 🧾 after a few seconds because they thought the attachment was missing.

The complexity and limited functionality of the existing APIs quickly became a significant obstacle 🛑. Simply making a document available in a specific folder could require four or five separate HTTP requests. We needed a more robust storage solution that could scale effectively 📈 and provide consistent, fast response times. 🚀

Infrastructure 🏢

Furthermore, we needed to reduce Nextcloud's impact on our infrastructure. Unlike Azure, Nextcloud doesn't scale well and required too much maintenance from our DevOps team.

😬 Architectural issues

In the past, users accessed documents stored directly on Nextcloud, with some files displayed through the platform’s built-in viewers.

This initial choice was certainly made for simplicity, but it has evolved into a significant architectural challenge as we began exposing storage directly to customers. Changing a storage server without affecting our users has become complex, and it also complicates the management of certain security and observability concerns.

The primary issue is with PDF documents, such as ledgers, which contain hardcoded URLs pointing to specific storage servers. This requires us to maintain these URLs for years to ensure continued access.

As part of our migration to S3, we are addressing these issues by routing all requests through the same service (GED).

This approach enables us to resolve several issues and enhance the product’s functionality:

Requiring authentication for some sensible documents.
Providing full observability over who uploads or downloads specific documents.
Enabling updates to storage capabilities without impacting customers.
Integrating new storage technologies seamlessly and transparently—for instance, through potential future integrations with services like Microsoft OneDrive.

📢 The plan

The first step was to draft an action plan 📝 and thoroughly document the existing setup. After several weeks of work, we established the key steps:

Route all document downloads through the GED service.
Route all document uploads through the GED service.
Migrate all existing documents to our new Azure storage, ensuring zero impact 🚫 on the end user.
Managing the Nextcloud links found in the PDFs already exported by our clients before the migration. Since these links pointed directly to our Nextcloud servers💀, we had to find a reliable solution to route these calls through the GED.

Each stage comes with its own set of challenges, which we’ll examine in detail later in the article.

Our primary concern, however, was to correct previous architectural missteps 🔍.

1️⃣ Download

The first step was to re-abstract downloads and previews, routing them through our backend. This required us to manage both legacy documents 📜 still stored on Nextcloud and new documents that would be hosted on Azure storage.

One challenge we’re facing is that the token generated by Nextcloud lacks any information about the tenant associated with the document. Without this, our backend cannot identify the relevant database cluster and tenant.

To resolve this, we created a new opaque token that embeds the tenant ID:

import crypto from "node:crypto";

const token = `${tenantId}-${crypto.randomBytes(16).toString("hex")}`;
console.log(token); // => 1-f82158a508b8bfbed82b601e2ed60edd

🔮 Previews

Nextcloud offered automatic previews of uploaded files, a feature we relied on extensively, so we needed to re-implement an equivalent ourselves.

We decided not to generate previews at upload, as this would have added significant complexity and cost, along with the challenge of handling asynchronous generation.

For PDFs, we just return an optimized preview of the first page, and for images, we use the Sharp library.

function getImageTransformer(
  query: { x?: string; y?: string; },
  ext: string
): sharp.Sharp {
  const { x, y } = getDimensions(query);
  const transformer = Azure.isAlphaImage(ext)
    ? sharp().png()
    : sharp({ failOn: "none" }).jpeg({ mozjpeg: true, quality: 50 });

  return transformer.resize({
    fit: "inside",
    withoutEnlargement: true,
    height: y,
    width: x
  });
}

📦 Headers and encoding

When returning documents, it’s essential to set the correct HTTP headers and apply proper encoding to values like file names.

const contentType = mime.contentType(
  path.extname(request.body.document)
);
const { body, contentLength } = await getFileFromAzure(request);
// pipe body to reply/response

reply.header(
  "Content-Disposition",
  `attachment; filename="${encodeURIComponent(filename)}"`
);
reply.header("Content-Type", contentType);
reply.header("Content-Length", contentLength);

I frequently see editors forget to re-inject the file name.

Monitoring

Being able to monitor developments and misuse is critical to guaranteeing the stability of our infrastructures.

Built-in file viewer

Since Nextcloud could display multiple documents within a viewer, we chose to re-implement a minimal yet functional viewer to retain this capability.

While our front-ends offer more advanced display modules, this lightweight viewer remains useful in several scenarios:

Replacing or rewriting legacy URLs in PDFs.
External links shared via APIs.
Providing quick access for debugging purposes.

2️⃣ Upload

Successfully prototyping an upload wasn’t as complex as expected… but, as always, the devil is in the details.

For inter-service uploads between Node applications, another Fastify plugin was added to our workspace package, providing methods to interact with the GED API 🔀.

📉 Optimize PDFs and images

Many of the PDFs and images submitted by our users are quite large and can be optimized. For this, we use Ghostscript 👻 to optimize PDFs and the Sharp package for images.

To date, we’ve reduced the size of received PDFs and images by an average of 50%, with no loss in quality ✨.

Compression is performed asynchronously using setImmediate to ensure fast server response times.
A compression value of "null" indicates that the compression ratio is below 5% 🤷‍♀️, making the update to the file on Azure negligible.
Otherwise, the file is updated in the cloud ✔.

Most of these optimizations are carried out via streams, so that the file or image is never completely buffered.

However, we needed to remain vigilant about rising CPU consumption and enhance our infrastructure setup 🏗️ to handle increased workloads effectively.

🖼️ HEIC/HEIF

Apple's proprietary HEIC format 📱 presented a significant challenge, often requiring conversion to JPG or PNG for compatibility.

Given that Python bindings to libheif showed much better performance, we initially opted to create our own N-API Node.js binding for libheif, using low-level libraries for rapid JPG and PNG conversion.

HEIF-converter

For maintenance reasons, we chose to use Sharp by building libvips directly on our machines and installing the necessary tools (libheif, mozjpeg, libpng, etc.).

🔒 Security

When managing file uploads and storage, vigilance is essential 🕵️‍♂️ in several areas:

Monitor for spoofed HTTP headers 🛡️, such as altered content-type headers.
Scan files for viruses and malicious content 🦠.

Otherwise, an attacker could misuse your brand and storage capabilities to distribute malicious content and compromise users 🚨.

Make it a habit to consult the OWASP cheat sheets to ensure maximum protection against errors and oversights: OWASP File Upload Cheat Sheet.

We used clamscan (which relies on ClamAV) to scan the files 👁️, and file-type to accurately identify the file type instead of relying solely on the request headers 🧨.

📊 Monitoring

As we regain control, it’s essential not to overlook usage monitoring through logs and other metrics.

3️⃣ Migrating Nextcloud documents

To gradually phase out our Nextcloud servers, we developped a temporary Node.js API 🦾, responsible for transferring resources from Nextcloud to Azure. This service handled upload concurrency, which we've limited to 64 simultaneous uploads to avoid overloading the server 🔥.

Without detailing every feature of this internal tool, it was designed to support key functionalities such as pausing and resuming the migration process, as well as monitoring the status of each transfers (successes ✅, errors ❌, totals, etc.).

Step 1: Data Extraction

We extracted from the Nextcloud database all the tokens 📄 (used to retrieve document data from the database) and the file paths on the server (to transfer the resources), saving them into .csv or .txt files.

Step 2: Environment Setup

We then set up a NAS server to run the Node.js tool and directly access the file system 🦄, bypassing the Nextcloud API. This approach was chosen to maximize performance and enable efficient stream-based, parallel processing of the document transfers.

Step 3: Create the DB and go 🧨

All that remained was to create the SQLite databases (we chose to generate one database per firm to avoid excessively large files), using the Nextcloud exports that contained tens of millions of rows, and then start the transfers ✅.

Let’s just say we ran into a few surprises along the way 🤫, and the migration ended up taking us several days 🤭.

4️⃣ Legacy URLs in PDF

Some URLs are permanently embedded in PDFs 📄, so we need to consider strategies for rewriting them using the information available.

Since Nextcloud tokens didn’t contain any tenant information, we created a minimal API (microservice) supported by an SQLite database to maintain the relationship between a token and its corresponding tenant ID.

PRAGMA journal_mode = OFF;
PRAGMA synchronous = 0;
PRAGMA locking_mode = EXCLUSIVE;

CREATE TABLE IF NOT EXISTS "tokens" (
  "token" TEXT PRIMARY KEY NOT NULL,
  "schema" INTEGER NOT NULL
) WITHOUT ROWID;

We can manage thousands of tokens within just a few milliseconds ⏱️ using purely synchronous I/O. Additionally, we implemented an LRU cache to ensure that repetitive requests are handled even more quickly.

The final step is to configure HAProxy 🔀 to redirect nextcloud viewer requests to a specific GED endpoint, where the URL is parsed to retrieve tokens and correlate them with their respective tenants, using the project setup described above.

const { link } = request.query;

if (!URL.canParse(link)) {
  // Problem
}
// Other URL validation here

const tokens = link.match(
  /(?<=\/)([1-9]{1,4}-\w{15,32}|\w{15})(?=\W|$)/g
);
// Correlate tokens with our microservice database

reply.redirect(`/ged/document/view?tokens=${correlatedTokens.join("|")}`);

This is only a partial overview of the implementation. We use a combination of the WHATWG URL API and regular expressions to extract tokens, ensuring sufficient security to mitigate any ReDoS attack vectors.

We then redirect the request with all tokens to our built-in viewer.

🔬 What we've learned

This project taught us that errors in URLs saved within PDF documents are hard to forgive. Due to some technical debt and a lack of foresight, we ended up with an unintended /ged/ged prefix. Today we're having a bit of a laugh about it, and if you see this prefix you'll know it wasn't meant to be 😆.

Managing files with proper streaming while handling errors proved far more challenging than anticipated, plaguing us for weeks with ghost files, memory leaks, and other unexpected bugs. At this level of usage, it’s technical excellence or nothing.

❤️ Credits

A migration project of this scale doesn’t happen overnight—it took us well over a year to complete all the steps outlined above. A big thank you 🙏 to everyone involved for their dedication and effort ❤️.

Nicolas 👨‍💻, for leading the project development from A to Z.
The infrastructure team 🏗️ (Vincent, Jean-Charles, and Cyril) for their consistent support throughout the project.
Aymeric, for managing and leading the migration of downloads and uploads for his team services.
Many others 👥 for their reviews and support 📝.

This project was incredibly rewarding 🏆, both for its challenges and the range of architectural issues it addressed 📐.

Thank you, see you soon for another technical adventure 😉😊

👋👋👋

Designing MyUnisoft Next-Gen Accounting APIs

Thomas.G — Thu, 04 Apr 2024 14:20:41 +0000

Hello 👋

I'm excited to present my latest technical article detailing the design and development process behind the next iteration of the MyUnisoft Set of accounting external APIs.

Internally known as MAD for MyUnisoft Accounting Data 😄, this project showcases our team's innovative approach and dedication to enhancing API and Partners experience.

💡 Context about the business requirements

You might be questioning the need for developing new APIs. We embarked on this effort to address a spectrum of issues and fulfill unmet needs, including:

🔄 Introducing a modern and efficient format for exporting and importing accounting folders, along with their parameters (particularly useful for migrations between tenants).
📊 Incorporating accounting data that was previously absent from historical partners APIs, such as Analytics.
🚀 Constructing high-performance, user-friendly APIs for our partners, adhering to modern API best practices.

When I refer to high performance, I'm not solely emphasizing faster response times. It also includes improving the security and strength of our systems by using cache and queues, as well as ensuring better observability.

👻 TRA

Historically, the CEGID TRA format has been predominant in the French market, but it faces a number of issues:

Tailored for CEGID's specific needs, many columns within the format prove irrelevant for MyUnisoft's purposes.
Featuring a column-oriented structure with fixed positions, it becomes resistant to evolution and challenging to implement and maintain 😡.
Parsing and generating data in this format incur significant costs and complexities 😞.

However, it can be difficult for software editors and accounting firms to adapt to unfamiliar formats quickly. That's why we've decided to be compatible with TRA.

The basic idea is to enable faster integration and smoother migration to our format. If this format has worked so far, we can learn a lot from it 😊.

💪 Accounting imputation for Factur-X

A significant disappointment lies in the realization that simply exchanging Factur-X via APIs will prove largely inadequate for creating a satisfactory experience for accounting editors and their customers (CPA), as the format does not permit embedding data for accounting imputation 😞.

To address this issue, we are planning to implement Factur-X natively within MAD. This format will enable the fields to be omitted, provided that the attached file is a valid Factur-X. Developers will only need to fill in the required fields for imputation purposes.

🚫 There will be no necessity for creating non-standard extensions of the XML format.

✒️ Technical specification

From the beginning, my aim was to develop accounting APIs that incorporate the following features:

Versioned interface contracts to establish lifecycles and facilitate iterative changes more smoothly.
Utilization of identical interfaces for both import and export functions, thereby simplifying API interactions.
🧠 Maximizing intelligence and independence to minimize dependency on other APIs. For instance, this includes eliminating the necessity for IDs or internal codes during imports.
Striving for neutrality when crafting the format, ensuring compatibility and flexibility across various systems and use cases.

Lastly, it's imperative to highlight the significance of neutrality in our approach. Too often, technical teams can fall into the trap of designing formats and APIs solely with internal use cases in mind, often under the pressure of immediate business requirements.

However, it's essential to recognize that our target audience primarily comprises developers who experience accounting through the lens of their own unique business contexts. By prioritizing neutrality in our design principles, we aim to simplify the lives of our consumers and empower them with flexible solutions that seamlessly adapt to diverse workflows.

Through MAD, we're committed to fostering an ecosystem that prioritizes developer-centric design, ensuring that our APIs serve as versatile tools capable of meeting a wide range of needs 🌐.

🔍 Gathering information on existing assets

The initial step involved conducting an inventory of existing APIs and comprehensively understanding the data consumed by our partners. We didn't have to do a lot of work, given that we have a nice public documentation available 😎!

We reviewed the APIs of several of our competitors, complemented by our experience integrating with various accounting editors, which guided us in making the right design decisions.

Additionally, we meticulously analyzed our TRA backend to extract the required columns for a preliminary version. This enabled us to identify the essential elements for a version 1, including:

accounting folder
payments
banks
journals
exercices
analytics (axes and their sections)
entries & movements

🔍 Our plan is to expand this list to incorporate more elements over time.

📚 Building interfaces

The second step involved crafting prototypes for our JSON object interfaces, utilizing TypeScript and JSON Schema.

Here is the finalized version for Journal, which underwent several weeks of iteration, concurrent with the creation of initial drafts of SQL.

interface SimplifiedAccount {
  producerId?: string;
  number: string;
  name: string;
}

type TypeJournal = "Achat" | "Vente" | "Banque";

interface Journal {
  producerId: string;
  name: string;
  customerReferenceCode: string;
  type: TypeJournal;
  counterpartAccount: SimplifiedAccount | null;
  additionalProducerProperties: {
    type: TypeJournalInternal;
    locked: boolean;
  };
}

💡 Properties that are too specific to MyUnisoft are stored in additionalProducerProperties

Simultaneously iterating on the SQL has proven essential, allowing us to eliminate keys that significantly impact performance without providing commensurate value.

Here is an another example with Bank

interface Bank {
  producerId: string;
  IBAN: string;
  BIC: string;
  account: SimplifiedAccount | null;
  journal: {
    producerId: string;
    name: string;
    type: "BANQUE";
    customerReferenceCode: string;
  };
  additionalProducerProperties: {
    isDefault: boolean;
  };
}

A significant effort has been invested in creating objects that are both clean and user-friendly.

🌀 (Bonus) Known your abstraction

We frequently fall into the trap of assuming that our abstractions are universally understood across a entire ecosystem. In accounting, for instance, it's typical to categorize transactions into an abstraction known as an "Entry." Each entry comprises a set of balanced movements (or transactions).

However, many software solutions, including Dataviz, often prefer consuming raw movements without additional abstractions. This preference explains why such solutions historically lean towards formats like FEC, which are simpler compared to TRA-heavy formats.

From my experience working with partners over the years, I've learned that many of them employ various abstractions for VAT, accounts, or analytics. Interestingly, the choice of abstraction can sometimes be influenced by the nature of their software.

The way we structure our data can play a vital role in ensuring a positive experience for specific consumers 💡.

💬 Technical implementation

For this project we decided to create an in-house package that would bring together all the necessary parts. This decision primarily aims to accommodate diverse use cases without the need for repetitive API (or SDK) integration.

The second step was to design a testable architecture for the package. For some parts I drew inspiration from open source libs such as undici and their mocking API.

At one point I was thinking that it might be a problem to implement HATEOAS... However, I discovered that with our architecture, dynamically injecting new links to specific resources using transformers is surprisingly straightforward 😅.

📦 Package usage

Here's a simple example

import * as MAD from "@myunisoft/mad";

const exporter = new MAD.Export({
  version: "1.x",
  accountingFolderId: "10",
  postgresDataSource
});

const { data: exercices } = await exporter.exercices();

Underneath we fetch elements from the database and transform them into the correct format, leveraging the SemVer version injected into the constructor.

Furthermore, we've integrated an observability layer to track the execution times of each component. However, this aspect likely requires further refinement 😕.

async exercices(): Promise<ExportResponse<postgre.Exercice[]>> {
  const startTime = performance.now();

  const rawExercices = await this.provider.exercices();
  const sqlTime = performance.now() - startTime;

  const transformed = this.transformer.exercices(rawExercices);

  return {
    data: transformed.data,
    executionTime: {
      sql: sqlTime,
      transformer: transformed.executionTime,
      total: performance.now() - startTime
    }
  };
}

The provider can simply be mocked up if you need to run unit tests (by forcing a global provider or forcing it using the class constructor).

const kGlobalDispatcher = mock.fn();

before(() => {
  MAD.setGlobalProvider(kGlobalDispatcher);
});

beforeEach(() => {
  kGlobalDispatcher.mock.resetCalls();
});

after(() => {
  MAD.resetGlobalProvider();
});

(Bonus) TRA compatibility

Effectively writing or reading a TRA isn't inherently complex, although it can be quite tedious due to the time-consuming process of properly implementing all its sections.

The challenge lies in being as fast as possible ⚡.

After one or two months of work, our JavaScript implementation can now read and write a complete TRA of several million lines in just a few hundred milliseconds 📈💻.

To achieve this, we've minimized the number of operations and iterations for each section. Additionally, each section is equipped with a comprehensive definition containing all column information, such as position, alignment, and length.

const config: TRAConfig = {
  section: "rib",
  JSONToTRA,
  TRAToJSON,
  columns: [
    {
      name: "fixe",
      mandatory: true,
      type: "a",
      length: 3,
      align: "left",
      position: 0
    },
    {
      name: "identifiant",
      mandatory: true,
      type: "a",
      length: 3,
      align: "left",
      position: 3
    }
    // ... other fields
  ]
}

Each section is equipped with functions designed for efficient conversion between TRA and JSON formats.

function JSONToTRA(
  lineObj: IRib,
  opts: TRAParsingOptions
): string {
  return [
    convertJSONValue(config.columns[0], lineObj.fixe, opts),
    convertJSONValue(config.columns[1], lineObj.identifiant, opts),
    convertJSONValue(config.columns[2], lineObj.auxiliaire, opts),
    // other fields..
    os.EOL
  ].join("");
}

We utilized tools like flamegraphs to pinpoint the longest-running elements and effectively eliminate them. Additionally, we optimized our process by implementing caching mechanisms to avoid redundant operations, such as repeatedly parsing the same dates.

Exploring a Rust implementation could be intriguing to further reduce execution time 🚀. Perhaps in the future, if performance becomes a critical concern 👀.

🌏 API

In parallel with the work done on the MAD package, we drew up a plan for creating the APIs 📝.

However, we encountered a challenge: some of the APIs we'll be exposing are potentially risky ⚠️, as they may permit exporting entire folders along with all associated parameters.

To address this, we're designing asynchronous APIs, incorporating status tracking and caching mechanisms integrated with our event architecture. This setup enables us to trigger webhooks as needed, enhancing security and reliability.

The sequence diagram above illustrates the interactions between the various components involved in exporting an entire accounting folder.

For this export, we have two distinct routes:

mad/all?format=json&version=1.0.0 to initialize the export.
mad/all/status?exportId=export_sch-5_acf-1 to fetch the status.

The second route will return a DONE status when processing is complete (Real-time webhook notification can be subscribed to as an option).

{
    "id": "export_sch-1_acf-1",
    "status": "DONE",
    "url": "<url>"
}

📜 Documentation

We've seamlessly extended our documentation, which is readily accessible on Github. Recently, we've enhanced the user experience by incorporating an online Vitepress site, hosted with Github Pages.

It required several weeks of effort, and the pivotal elements include:

📝 A comprehensive description outlining our decisions.
📋 A detailed specification encompassing all interfaces, schemas, etc.
📚 Additional insights, explanations, and guidance tailored for beginners in accounting.
🗺️ Instructions on navigating MyUnisoft for developers unfamiliar with our product.

My primary objective is to ensure a positive developer experience (DX) while maintaining thorough and accessible documentation. You're welcome to explore it yourself here, and we eagerly await your feedback 😊.

📆 What's next?

We're only at the very beginning of the project. We will continue to add new content to our export format and work on new options for our various APIs.

In parallel, we're working on the import (I'll probably have a chance to write a new article when we're well advanced on this).

❤️ Credits

This project is primarily a collaborative effort, made possible by the contributions of key team members:

Benoit GARIAZZO (Product Owner)
Alexandre MALAJ (Specification, MAD package & SQL).
Cédric LIONNET (API)
Pierre DEMAILLY (TRA/JSON compatibility & review)
Frédéric GUIOU (Help with naming and currently leading import)
Me 😁 (Specification, Observability, Documentation)
Léon and Aymeric for their invaluable assistance with existing accounting APIs & formats.

And all other team members who provided valuable feedback and support during the implementation process 🙌.

🙏 That concludes this article. Thank you for reading!

Consuming Loki logs with Grafana API and Node.js

Thomas.G — Wed, 25 Oct 2023 10:28:24 +0000

Hello 👋

In my last article we were discussing how me and my team had set up and built dashboards with Grafana and Loki 😎.

Logs monitoring with Loki, Node.js and Fastify.js

Thomas.G for MyUnisoft ・ Jun 12 '23

#node #javascript #monitoring #grafana

Today we're going to discuss how you can leverage your logs for third-party tools using Grafana's API and Node.js 😍.

💡 The idea that starts it all

Several months ago, I started thinking about how to use our logs to build a personalized CLI experience.

After a bit of searching, I quickly found that Grafana had an API for queryings Loki logs. I immediately went crazy at the thought of what could be done with it 🔥.

🐢🚀 Node.js SDK

Over the weekend, I set to create an open source Node.js SDK.

import { GrafanaLoki } from "@myunisoft/loki";

const api = new GrafanaLoki({
  // Note: if not provided, it will load process.env.GRAFANA_API_TOKEN
  apiToken: "...",
  remoteApiURL: "https://name.loki.com"
});

const logs = await api.queryRange(
  `{app="serviceName", env="production"}`,
  {
    start: "1d",
    limit: 200
  }
);
console.log(logs);

👀 Notice the support of duration for options like start and end.

It already supports Stream and Matrix and several other APIs:

GET /loki/api/v1/labels
GET /loki/api/v1/label/:name/values
GET /loki/api/v1/series

We've also added datasource APIs (which may be required depending on what you'r building).

It also include a LogParser inspired by Loki pattern.

// can be provided as an option to queryRange
const parser = new LogParser<{
  method: string, endpoint: string, statusCode: number
}>("<method:httpMethod> <endpoint> <statusCode:httpStatusCode>");

const logs = await api.queryRange(
  `... LogQL here ...`, { parser }
);
for (const logLine of logs) {
  console.log(logLine.method);
  console.log(logLine.endpoint);
}

It's still far from perfect, don't hesitate to contribute 💪.

OpenAlly / loki

Node.js Loki SDK

Loki

Node.js Grafana API SDK (Loki, Datasources ..)

🚧 Requirements

Node.js version 24 or higher

🚀 Getting Started

This package is available in the Node Package Repository and can be easily installed with npm or yarn

$ npm i @openally/loki
# or
$ yarn add @openally/loki

📚 Usage

import { GrafanaApi } from "@openally/loki";
import { LogQL, StreamSelector } from "@sigyn/logql";

const api = new GrafanaApi({
  authentication: {
    type: "bearer",
    token: process.env.GRAFANA_API_TOKEN!
  },
  remoteApiURL: "https://name.loki.com"
});

const ql = new LogQL(
  new StreamSelector({ app: "serviceName", env: "production" })
);
const logs = await api.Loki.queryRange(
  ql, // or string `{app="serviceName", env="production"}`
  {
    start: "1d",
    limit: 200
  }
);
console.log(logs);

…

View on GitHub

🔎 LogQL builder

We've also built a utility package to construct log queries, inspired by the WHATWG URL/URLSearchParams API.

The package lets you build any LogQL programmatically from scratch

import { LogQL } from "@sigyn/logql";

const logql = new LogQL("foo");
logql.streamSelector.set("env", "prod");
logql.lineEq("bar");
logql.lineNotEq("baz");

// {env=\"prod\"} |= `foo` |= `bar` != `baz`
console.log(logql.toString());

But also able to parse raw string 😇

const logql = new LogQL(
  "{app=\"foo\", env=\"preprod\"} |= `foo` != `bar`"
);

console.log([...logql.streamSelector.entries()]);
console.log(logql.lineFilters.lineContains());
console.log(logql.lineFilters.lineDoesNotContain());

You can use subclasses in your tools at any time, for example to retrieve labels from a LogQL 😮

import { StreamSelector } from "@sigyn/logql";

const logql = '...';
const labels = new StreamSelector(logql).kv();

📜 Credits

I'm not alone and didn't do all the work 👯. Thanks to Pierre Demailly and Sofian Doual for their contribution/support.

🙏 Conclusion

These two tools have enabled us to initiate the development of new tools. One of them is an alerting agent for Loki named Sigyn, to whom I'll be dedicating an article in the near future.

Having APIs really opens up a lot of opportunities and it'll be interesting to see what me and my team are able to produce in the future with them.

Best Regards,
Thomas

Logs monitoring with Loki, Node.js and Fastify.js

Thomas.G — Mon, 12 Jun 2023 12:45:10 +0000

Hello 👋

Over the past few months, I've been spending a lot of time creating dashboards on Grafana using Loki for MyUnisoft (the company I work for).

So I decided to write you an article to explain my adventure from the point of view of a back-end Node.js developer.

👀 Why grafana and loki?

For context, the tool was implemented by a DevOps employee who left the company two years ago. So I didn't really choose the tool myself. I had no previous experience with either of these tools, so I had to experiment and discover as I went along.

Grafana is pretty well-known and mature, so I wasn't worried. On the other side, I quickly liked Loki to manage and search my logs. So I thought, "I've got to learn to make dashboards out of this".

📃 Writing good logs

💬 Several examples of logs are multiline to avoid scrolling and simplify reading.

When you write logs, you must constantly think about the information they will provide and make sure they can be analyzed without too much difficulty 😵. It always takes several iterations to get a good result.

In my experience, you can find two kinds of logs:

The debug logs you know are useless to your dashboards. They'll come in handy in case of bugs or when you're looking for more context about a given request.
All other logs that can provide exploitable data for dashboards (and for a human reader)

Here an example of a debug log we print when our services start (quite useful to customize the --max-old-space-size flag).

const availableHeapSize = prettyBytes(
  v8.getHeapStatistics().total_available_size
);
server.log.info(`Total available heap size: ${availableHeapSize}`);

Another common example is logging a JSON payload (you could exploit the data, of course, but that's often not the end goal).

this.entry.once("payloadReady", (payload) => {
  this.log("payload before publishing:");
  this.log(JSON.stringify(payload, null, 2));
});

😱 Be sure to redact the data (you may divulge confidential information without realizing it).

Most of the time, you won't get anything out of these logs, and they don't require any particular effort. In this article, I'm going to concentrate on the logs that we can exploit for our dashboards.

🐤 Practical example

Here's an example of bad logs we initially set up for a document upload middleware/plugin

Successfully uploaded '...'
//
Failed to upload '...'

There are several problems:

No scope (we can't easily search for all the logs related to our middleware).
Can easily be mixed with other logs (from other middlewares or services).
It can be quite a challenge to parse the state (failed or successful).
Lack information about the request or the user.

A better way of writing it:

[uploader|req-x5d4] document 'name.jpg' uploaded (state: ok)

You can easily extend the log with other information such as extension, size, runtime, etc. (without breaking the Loki pattern or regexp).

(state: ok|ext: .jpg|size: 52.5 kB|upload-time: 0.503 ms)

⚠️ Always use the same unit when logging size (bytes) or time (milliseconds). Setting the right 'unit' on Grafana will do the work of cleaning the value for you.

📜 Format clarification

A good log has to be structured to allow good searches with LogQL. Here's a bad example where it will be really difficult to extract information.

hello-world.jpg 52.5 kB|0.503 ms

I would recommend adding a label before each value (it's also easier for a human to read). Having start, end, and separator characters can also be a great help.

doc 'hello-world.jpg' [size: 52.5 kB|exec: 0.503 ms]

Here the required RegExp to parse all labels with Loki

doc '(?P<doc>\S+)' \[size: (?P<size>\S+) kB|exec: (?P<exec>\S+) ms\]

Loki 2.0 also allows you to retrieve labels with a simplified syntax called pattern.

pattern `doc '<doc>' [size: <size> kB|exec: <exec> ms]`

🔎 Implementation in the framework/code

⚠️ code examples are simplified/truncated

The Fastify framework includes the Pino logger by default (a really great logger with lots of cool features that doesn't compromise on performance). The framework itself allows a lot of really cool stuff, like controlling the level of logs at runtime.

On my team, we have chosen to customize the default request and response logs to include additional information. To achieve this, you need:

setup fastify constructor option disableRequestLogging to true
exploit two hooks (onRequest and onResponse).

server.decorateRequest("standardLog", null);
server.addHook("onRequest", async(request: FastifyRequest) => {
  request.log.info(
    `(${request.id}) receiving request ...`
  );
  request.standardLog = standardLog.bind(request);
});

server.addHook("onResponse", async(request: FastifyRequest) => {
  request.log.info(
    request.standardLog(
      `response returned "${request.method} ${request.raw.url}"
    )
  );
});

The standardLog decorator allows us to display information about the request and the token in use (for authenticated endpoints).

We have to handle several types of tokens (it all depends on who is consuming the API). A classical user or a partner through our partner API).

Each of them includes:

The postgreSQL schema of the customer
The user's or third-party's ID
The accounting folder (which can be null)

function standardLog(
  this: FastifyRequest,
  msg: string
): string {
  if (
    this.server.hasRequestDecorator("tokenInfo") &&
    this.tokenInfo !== null) {
    const { tokenInfo } = this;
    let tokenInfoLog = `${tokenInfo.type}`;

    switch (tokenInfo.type) {
      case "api":
        tokenInfoLog += `|s:${...}|t:${...}|acf:${...}`;
        break;
      case "user":
        tokenInfoLog += `|s:${...}|p:${...}|acf:${...}`;
        break;
      default:
        tokenInfoLog = "";
        break;
    }

    return `(${this.id}|${tokenInfoLog}) ${msg}`;
  }

  return `(${this.id}|none) ${msg}`;
}

Here's what it looks like in the logs

(req-1rti) receiving request "POST /ged/base-docs/docs"
(req-1rti|user|s:538|p:1|acf:23) response returned
"POST /ged/base-docs/docs", statusCode: 201 (460.106ms)

These informations will be critical to populating our dashboards with information about customers and the scope of actions.

🌀 Logs ingestion

My team currently uses promtail with a YML configuration to retrieve service logs.

server:
  http_listen_port: 9080
positions:
  filename: /var/lib/promtail/positions.yml
clients:
  - url: https://xxx.fr/loki/api/v1/push

scrape_configs:
  - job_name: svc_api_dev
    static_configs:
    - labels:
        __path__: /home/xxx/logs/service-dev-*.log
        app: api
        env: dev
        host: xxx.fr
        job: svc_api_dev
      targets:
      - localhost

I'm not going to spend too much time on this chapter, as you can find a lot of tutorials on the Internet on how to set up and configure promtail.

💡 If you don't want to depend on promtail for sending the logs to Loki, you can use the fastify plugin pino-loki

📊 OCR Dashboard

MyUnisoft is a French accounting editor, so Optical Character Recognition is an important feature of our software. Monitoring is essential to adapting to the needs of our customers and responding promptly to incidents.

👀😍 We won the OCR silver medal 2023 awarded by Le Monde du Chiffre.

Here is an example of a dashboard we have built just by using the following log:

(req-2987|user|s:24|p:5710|acf:7398) OCR xxx.jpg
[type: invoice|ext: .jpg|size: 2925.73 kB]

Group by label

We can construct the above graph with the following logQL. Notice the sum by extension.

sum(
  count_over_time(
    {app="ocr",env="$env"}
      |= "OCR"
      |= "|ext:"
      | regexp `\|ext: \.(?P<extension>\S+)\|`
  [$__range])
) by (extension)

RegExp are built using Golang syntax (I personnaly use regex101.com to test them). It is useful here to extract/detect the extension label.

We can use the syntax $env to inject a dashboard variable (here I can view my dashboard in production, staging or dev just by moving a value in a select).

I also use the built-in $__range variable, which loads data over the currently selected range in Grafana.

Unwrap

It took me a long time to really understand unwrap. There is virtually no clear documentation or explanation on the Internet!

Here it will use the size label and extract all values to compute them with min_over_time and min functions. This is useful for extracting numerical values (counter, execution time, etc.).

min(
  min_over_time(
    {app="ocr",env="$env"}
      |= "OCR"
      | regexp `\|size: (?P<size>[.0-9]+) kB\]`
      | unwrap size
      | __error__ = ""
  [$__range])
) by (app)

💬 | __error__ = "" avoid crashing Loki with unexpected Error (could happen if unwrap of size fail for any reasons).

Then, on the right side of the stat graph, we select the corresponding unit.

💡 Tips and tricks

display name

For a long time, I had some really bad raw labels in my charts (with a format similar to JSON).

You can customize it by editing the display name option. You can use variables in this field to retrieve label values directly.

no data

Sometimes the graphs will display "no data" because no logs have been detected in the selected range. This can be problematic in certain graphics (such as gauges).

But maybe you'd rather have zero. No problem, just use the No value option.

filter using a regexp and variable

On some dashboards, you will probably want to filter dynamically according to several criteria. One way of doing this is to use a regexp and a dashboard variable.

This is what I do in one of my dashboards to filter the results for one or more partners.

count(
  sum(
    count_over_time(
      {app="api",env="production"}
        |= "] CALL"
        |= "$endpoint"
        |~ `\]\[$thirdparty\]`
        | regexp `\((?P<schemaId>[0-9]+):(?P<folderId>[0-9]+)\)`
    [$__range])
  ) by (schemaId, folderId)
)

Here's the line that does the job: you need to use backticks syntax to be able to inject a variable in it.

|~ `\]\[$thirdparty\]`

🚀 Data is the key to success

As developers, we don't realize enough how monitoring and the data it generates can be powerful sources of improvement. At MyUnisoft, we work with over a hundred partners.

Understanding how they use our APIs and the various anomalies they experience has become essential for us to continue to grow exponentially.

This allows us to continually improve to provide a better experience for both our accounting customers and our developer partners who maintain the integrations.

Our ability to materialize and exploit this data has become an essential part of my team's expertise. It's really exciting to see what can be achieved with simple logs.

👋 Conclusion

I'm so glad I finally managed to write this article (and I hope it brought you some value). Many thanks to my team and especially to Cédric, without whom I would not have realized all that.

I still struggle with a lot of Grafana features, such as transformations or alerts, but I will continue to dig and improve.

🙏 Thanks for reading me 🙏

Securizing your GitHub org

Thomas.G — Sun, 19 Feb 2023 15:48:01 +0000

Hello 👋

I started open source a bit naively (like everyone I guess 😊).

But the more I progress and the more important/popular some of my projects become 😎. That's great, but at some point you have to deal with a lot of things related to security (like Vulnerability disclosure).

You start to hear and see a lot of scary stories around you 😱. Not to mention all the acronyms where you don't understand anything at first 😵 (VMT, CVE, SAST, SCA, CNA ...).

As I was working on an open source security project, I put pressure on myself to be ready. Also as a member of the Node.js Security WG I thought it was an interesting topic and that I was probably not the only one who was worried about not being up to the task 😖.

So I rolled up my sleeves and tackled the problem 💪. Here is my feedback/journey on how I improved the security of my NodeSecure GitHub organization.

👀 We use Node.js and JavaScript (but most recommendations are valid for other ecosystems).

Security Policy and Vulnerability Disclosure

Adding a root SECURITY.md file explaining how developers and security researchers should report vulnerability is important. You don't want a security threat to be turned into a public issue (This gives you time to analyze and possibly fix before disclosure).

⚠️ If you are a developer, never report a security threat using a public GitHub issue. This is a serious mistake. This could even put your business/team at risk.

I don't want to bullshit you, so let me share with you the OpenSSF guide that helped me set up my first reporting strategy: Guide to implementing a coordinated vulnerability disclosure process for open source projects.

I started from scratch by reading this guide and taking inspiration from their templates 🐤. As a small open source team we don't especially have DNS or mail servers (not even a defined Vulnerability Management Team A.K.A VMT).

I was a bit puzzled to put my personal email as I'm not alone 😟.

I quickly learned that Github added a new feature to report/create private security issue 😍. You can enable it in the Security tab (I think it's also now possible to enable it on every repositories at once).

And this is what it finally looks like:

Use OpenSSF scorecard

The OSSF scorecard initiative is really good to assess your project against security best practices. I am not the first to write about this.

You can easily setup the GitHub action workflow by following those instructions.

Once configured, you will have a set of alerts available in the Security tab.

This will give you an overview of the different subjects to improve (workflows, dependencies etc). Each of these alerts contains a full description of the actions to be taken to fix the problem.

I have personally used these recommendations to dig and train myself. The next chapters will help you improve your score.

📢 By the way NodeSecure CLI has a first-class support of the scorecard.

🔓 Enable branch protection

I am a bad student 😳. Almost all of my projects had no branch protection on the main / master branch 🙈.

To set up the protection, go to Settings > Branches and edit your main branch.

GitHub has quite a few options on the subject 😵. If you don't know what to choose in terms of options, don't check anything (it's ok to begin ✔️).

If you want to be more restrictive, be careful because it could block you (some options are only viable in projects with many contributors/reviewers).

As far as I am concerned I often choose:

Require a pull request before merging
Require conversation resolution before merging
Require status checks to pass before merging

🐲 Workflows Hardening

I fell down when I saw all that it was necessary to know to secure workflows with GitHub actions 😲.

You must pay attention to the permissions granted to your jobs / GITHUB_TOKEN
Ensure your GitHub Actions are pinned to a SHA
Hardening the runner (see the StepSecurity HardenRunner)
Probably a lot of other stuff I haven't had time to see yet 😆

Fortunately there is a great free online tool that help you by doing all the hard work (it will open a pull-request and automatically fix issues).

// Detect dark theme var iframe = document.getElementById('tweet-1617557370728767488-427'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1617557370728767488&theme=dark" }

The tool was created by StepSecurity. I had the opportunity to talk with the CEO and they listen to the maintainers which is really cool.
Thanks to them ❤️!

Configure Dependabot

It is recommended to use Dependabot for updating your dependencies and GitHub actions (yes, it also supports updating workflows in a secure way 😍).

You only need to add a .github/dependabot.yml config file. Personally I recommend a weekly interval (with a lot of projects daily is a bit horrible).

version: 2
updates:
  - package-ecosystem: github-actions
    directory: /
    schedule:
      interval: weekly

  - package-ecosystem: npm
    directory: /
    schedule:
      interval: weekly

The StepSecurity tool we have seen in the previous chapter is also capable of doing it 🚀.

Also, think to enable Dependabot alerts in the Security tab. This will allow the bot to open pull-request to fix known vulnerabilities by looking at your dependencies (referenced in package.json or others).

🔬 Adding CodeQL scanning

To enhance security even more you can add a SAST tool like CodeQL. Like scorecard it will report security scanning alert but for your codebase.

Here an example:

A great way to make sure that newly added code does not contain vulnerabilities that were obvious to detect.

👀 Note that once again StepSecurity can set up the workflow for you.

📜 Enable Security advisories (and others)

Github Security tab as a lot of cool features that help you maintain the security of your project. If you have followed all my previous chapters, most of them should be enabled now.

Make sure to also enable Secret scanning alerts.

For an organization many of these parameters can be forced on all repositories. Go to Settings > Code security and analysis. You will have the options to enable/disable all.

💡 OpenSSF Best Pratices program

Previously known as CII-Best-Practices, this program indicates that the project uses a set of security-focused best development practices for open source software.

So I registered my first project on the website. It was a good surprise because it allowed me to question the quality of my documentation and tests 😬.

Seeing the different levels and questions really helps you think about what you're missing (and possibly learn about the concepts you don't know about yet.. Like SBOM).

I am still working on completing the first step/badge for the CLI project which now has a score of 8.7 out of 10 🎉 on the OpenSSF scorecard.

🎯 Conclusion

That's it for this article. I've covered what I've done/learned in the last couple of months. Here are some really cool additional links 💃:

If you work with NPM, I invite you to read our latest article about package managers:

📦 Everything you need to know: package managers

Antoine Coulon for NodeSecure ・ Nov 18 '22

#npm #node #opensource

Obviously, I probably still have a lot to learn. But I hope this will help other maintainers/developers ❤️.

🙏 Thanks for reading me 🙏

JS-X-Ray 6.0

Thomas.G — Mon, 16 Jan 2023 15:48:18 +0000

Hello 👋

It's been a while since the last article on JS-X-Ray 😲!

JS-X-Ray 3.0

Thomas.G for NodeSecure ・ Feb 28 '21

#node #javascript #security #opensource

In this article I will present you the latest major version 👀. I didn't do an article on version 4 and 5 because they didn't introduce new features (only breaking changes on the API).

📢 What is JS-X-Ray ?

If you are new in town, JS-X-Ray is an open source JavaScript SAST (Static Application Security Testing). The tool analyzes your JavaScript sources for patterns that may affect the security and quality of your project 😎.

Among the notable features:

Retrieving dependencies (CJS & ESM support) and detecting suspicious import/require.
Detecting unsafe RegEx.
Detecting obfuscated source (and provide hints on the tool used).

As well as a lot of other detections.

Major release 4 and 5

These versions introduced changes on warnings (and we improved how we manage them in the codebase). We added new descriptors for each of them:

i18n (for translation in CI or CLI).
experimental
severity (Information, Warning, Critical)

Those information are visible in the NodeSecure CLI interface:

Major release 6

🐬 Ok, let's dive into this major release to discover the surprises 🎉 it has in store for us.

🚀 Introducing VariableTracer

Almost a year of work on this new mechanism / class that brings a whole new dimension to JS-X-Ray.

const tracer = new VariableTracer()
  .enableDefaultTracing()
  .trace("crypto.createHash", {
    followConsecutiveAssignment: true, moduleName: "crypto"
  });

tracer.walk(node);

This class is able to follow all declarations, assignments and patterns (and those even through very obscure patterns).

const aA = Function.prototype.call;
const bB = require;

const crypto = aA.call(bB, bB, "crypto");
const cr = crypto.createHash;
cr("md5"); // weak-crypto warning is throw here

This allows us to implement Probes in a much simpler way (which makes maintenance and testing much easier).

Here an example with the isWeakCrypto probe:

function validateNode(node, { tracer }) {
  const id = getCallExpressionIdentifier(node);
  if (id === null || !tracer.importedModules.has("crypto")) {
    return [false];
  }

  const data = tracer.getDataFromIdentifier(id);

  return [
    data !== null &&
    data.identifierOrMemberExpr === "crypto.createHash"
  ];
}

By default the Tracer follows all ways of requiring dependencies with CJS and also usage of eval or Function.

🚧 Removing unsafe-assign warning

This warning was required at the beginning of the project because it was difficult for me to correctly identify some malicious patterns.

However, with the introduction of the new Tracer, which is very complete and precise, this warning no longer makes sense has it only generates unnecessary noise and false positives.

📜 Better ESM source parsing

We previously had a lot of parsing-error warnings because the NodeSecure scanner failed to detect if the file was using either CJS or ESM.

That new version will automatically retry with ESM enabled if it fails with CJS.

📉 Reducing false positives

To continue the momentum of the previous sections. This version drops a lot of warnings and significantly improves others.

Reducing false positives for encoded-literal warning by introducing new way of detecting safe values.
Improve short-identifiers by also storing ClassDeclaration, MethodDefinition and Function parameters.

We are also introducing a new suspicious-file warning when a file contain more than 10 encoded-literal warnings to avoid having file with hundreds or thousands of warnings.

Of the 500 most popular NPM packages, we previously had 24k warnings with version 5. The latest version brings that number down to approximatively 5k warnings.

🔬 Improving coverage

A lot of work has been done to add unit tests on all the probes of the project. We are near 100% of coverage 💪.

Thanks to the amazing work of our contributors:

👀 What's next ?

Here what I'm working for the next major release:

Adding support of TypeScript sources (probably by allowing a customization of the parser).
A new API that allows to dynamically extend the SAST with new custom probes (and custom warnings).
Introducing new built-in detections and warnings (unsafe URL etc).

I will continue to work to reduce the number of false positives and keep improving obfuscated codes detection.

Please think to drop a star on github ❤️!

NodeSecure / js-x-ray

JavaScript & Node.js open-source SAST scanner. A static analyser for detecting most common malicious patterns 🔬.

JavaScript AST analysis. This package has been created to export the NodeSecure AST analysis to enable better code evolution and allow better access to developers and researchers.

The goal is to quickly identify dangerous code and patterns for developers and security researchers. Interpreting the results of this tool will still require you to have basic knowledge of secure coding.

Goals

The objective of the project is to detect potentially suspicious JavaScript code. The target is code that is added or injected for malicious purposes.

Most of the time hackers will try to hide the behaviour of their code as much as possible to avoid being spotted or easily understood. The work of the library is to understand and analyze these patterns that will allow us to detect malicious code.

Feature Highlight

Retrieve required dependencies and files for Node.js
Detect unsafe regular expressions
Get warnings when the AST analysis detects a…

View on GitHub

That's it for today! Thanks for reading me 😉

Moving MyUnisoft Node.js back to TypeORM

Thomas.G — Tue, 02 Aug 2022 18:53:46 +0000

Hello 👋,

Recently I took the time to reflect on my last two years at MyUnisoft. I finally told myself that I wasn't writing enough about the difficulties we faced with my team 😊.

Today I decided to write an article about our transition to TypeORM. A choice we made over a year ago with my colleague Alexandre MALAJ who joined a few months after me.

We'll see why and how this choice allowed us to enhance the overall DX for my team 🚀. And that in the end it was a lot of trade-offs, and obviously, far from a perfect solution too.

🔍 The problem

At MyUnisoft we work with a PostgreSQL database with static and dynamic schema (each client is isolated in one schema). And uniquely without counting the duplication of the schemas we have about 500 tables.

The Node.js stack was split into several services coupled to the database (or to third-party services for some of them). Developers before us were writing raw queries and there were no unit or functional tests 😬. When I took over as lead it was hell to succeed in testing each service properly. Among the painful things 😱:

strong coupling.
heavy docker configuration
complexity to generate business data for our tests.

We had to find a solution to improve and secure our developments while iterating on production releases 😵.

Decentralizing with events wasn’t a possibility since of existing codes and dependencies (and we had no DevOps at the time).

💡 The solution

We started thinking about creating an internal package that would serve as an abstraction to interact with the database. We don't want to go for microservices 😉, so having a package that centralizes all this seems like a good compromise for us.

Among our main objectives:

Generate a compliant database locally or on Docker.
Easily generate fake data.
Built to allow us to carry out our functional and business tests.
Centralized code review (which also allows us to track changes more easily)

We also had ideas like building a schema in a running database (which could be used for partner API testing and sandboxing).

The question remained whether we should continue writing raw queries or not 😨. I'm not necessarily a big fan of ORMs, but we had a diversity of tables and requirements that made the writing of raw queries complicated at time.

We looked at the different solutions in the ecosystem by checking our constraints with the schemas. After must research, we concluded that TypeORM was viable (other ORM had critical issues).

Far from perfect, but we had to give it a try 💃!

Note: with hindsight, we are now also very interested in Massive.js. This could have been one of our choices.

🐥 Let the story begin

👶 Baby steps

My colleague Alexandre spent several months migrating the database on TypeORM 😮. I helped him by reviewing each table and relations.

We did not opt for a migration script at the time (for us the choice was still too vague).

We have made a gource to illustrate our work:

One of the problems we quickly encountered was that it was not possible to use the ActiveRecord pattern with dynamic schemas 😭. However this is ok for static schema because you can define them with the @Entity decorator.

@Entity({ schema: "sch_interglobal" })
export class JefactureWebhook extends BaseEntity {}

The management of datasources (connection) by schema/client was a bit infernal. We created our abstraction on top of TypeORM to handle all this properly and regarding our schema initialization requirements.

One of our encounters being quite complicated has been to clone a schema when we add a new client on the fly 🐝(that's something we do in our tests, in the authentication service for example).

We were able to achieve this by using the @EventSubscriber decorator on a static table we use to register new customers’ information.

@EventSubscriber()
export default class Sub_GroupeMembre {
  listenTo() {
    return Entities.schInterglobal.GroupeMembre;
  }

  async afterInsert(event: UpdateEvent) {
    const { idGroupeMembre } = event.entity!;

    const queryManager = datasources.get("default")!;
    await queryManager.query(
      `SELECT clone_schema('sch1', 'sch${idGroupeMembre}')`
    );

    const connection = await (new DataSource({})).initialize();
    datasources.set(`sch${idGroupeMembre}`, connection);
  }
}

The tricky part was to build an SQL script to properly clone a schema with all tables, relations, foreign keys etc.. But after many difficulties we still managed to get out of it 😅.

📜 Blueprints

When I started this project I was inspired by Lucid which is the ORM of the Adonis.js framework.

By the way, Lucid was one of our choices, but like many of Harminder's packages, it is sometimes difficult to use them outside of Adonis (which is not a criticism, it is sometimes understandable when the goal is to build a great DX for a framework).

But I was quite a fan of Lucid's factory API so we built an equivalent with TypeORM that we called "Blueprint".

Here is an example of a blueprint:

new Blueprint<IConnectorLogs>(ConnectorLogsEntity, (faker) => {
  return {
    severity: faker.helpers.arrayElement(
      Object.values(connectorLogSeverities)
    ),
    message: faker.lorem.sentence(5),
    public: faker.datatype.boolean(),
    requestId: faker.datatype.uuid(),
    readedAt: null,
    createdAt: faker.date.past(),
    thirdPartyId: String(faker.datatype.number({
      min: 1, max: 10 
    })),
    idSociete: null
  };
});

The callback includes the faker lib as well as internal custom functions to generate accounting data. You can use this blueprint to generate data like this:

const user = await Blueprints.sch.ConnectorLogs
  .merge({ readedAt: new Date() })
  .create();

The API is similar but it appears our objectives and TypeORM forced us to make different choices.

ES6 Proxy usage

You may have noticed but something is weird with this API. Every time you hit Blueprints.sch it triggers an ES6 proxy trap that will return a new instance of a given Blueprint.

It was quite satisfying for me to manage to use a Proxy for a real need and at the same time manage to return the right type with TypeScript.

import * as schBlueprints from "./sch/index";
import { Blueprint, EntityBlueprint } from "../blueprint";

// CONSTANTS
const kProxyHandler = {
  get(obj: any, prop: any) {
    return prop in obj ? obj[prop].build() : null;
  }
};

type EmulateBlueprint<T> = T extends Blueprint<infer E, infer S> ?
  EntityBlueprint<E, S> : never;
type DeepEmulateBlueprint<Blueprints> = {
  [name in keyof Blueprints]: EmulateBlueprint<Blueprints[name]>;
}

export const sch = new Proxy(
  schBlueprints, kProxyHandler
) as DeepEmulateBlueprint<typeof schBlueprints>;

📟 Seeder

We worked from the beginning of the project to build a relatively simple seeding API. The idea was mainly to be able to generate the static data required for our services to work properly.

Here's an example of one simple seed script that generates static data with a blueprint:

export default async function run(options: SeederRunOptions) {
  const { seeder } = options;

  await seeder.lock("sch_global.profil");
  await sch.PersPhysique
    .with("doubleAuthRecoveryCodes", 6)
    .createMany(10);

  seeder.emit("loadedTable", tableName);
}

When we generate a new database locally or in Docker we can see the execution of all the seeds:

🌀 Docker and testcontainers

When Tony Gorez was still working with us at MyUnisoft he was one of the first to work around how we can set up our tests inside a Docker and run them in our GitLab CI.

The execution of our tests was relatively long (time to build the Docker etc). That's when he told us about something a friend had recommended to him: testcontainers for Node.js.

Once set up but what a magical feeling... The execution of our tests was faster by a ratio of 4x. Tony has been a great help and his work has allowed us to build the foundation of the tests for our services.

On my side I worked on an internal abstraction allowing everyone not to lose time on setup:

require("dotenv").config();
const testcontainers = require("@myunisoft/testcontainers");

module.exports = async function globalSetup() {
  await testcontainers.start({
    containers: new Set(["postgres", "redis"]),
    pgInitOptions: {
      seedsOptions: {
        tables: [
          "sch_interglobal/groupeMembre",
          "sch_global/thirdPartyApiCategory"
        ]
      }
    }
  });
};

📦 Difficulties with a package 😱

Not everything in the process goes smoothly 😕. In the beginning, it was really difficult to manage the versioning. We used to use npm link a lot to work with our local projects but it was far from perfect (it was more like hell 😈).

And by the way, you have to be very careful with everything related to NPM peerDependencies (especially with TypeScript). If you use a version of typeorm in the package, you necessarily must use the same one in the service otherwise you will have problems with types that do not match.

"peerDependencies": {
  "@myunisoft/postgre-installer": "^1.12.1"
}

We had the same issue with our internal Fastify plugin. It cost us a few days sometimes the time to understand that we had screwed up well on the subject 🙈.

In the end, after some stabilizations, we could release new versions very quickly.

I'm not necessarily completely satisfied with the DX on this subject at the moment and I'm thinking of improving it with automatic releases using our commits.

Others APIs

I couldn't even cover everything because this project is so large. For example, we have a snapshot API that allows us to save and delete data during our tests...

Speaking of tests, it is always difficult to give you examples without being boring. But there too the work was colossal.

I would like to underline the work of Cédric Lionnet who has always been at the forefront when it came to solidifying our tests.

💸 Hard work pays off

After one year of hard work the project is starting to be actively used by the whole team across all HTTP services 😍. Everyone starts to actively contribute (and a dozen developers on a project is a pretty interesting strike force ⚡).

Sure we had a lot of issues but we managed to solve them one by one 💪 (I'm not even talking about the migration to TypeORM 3.x 😭).

But thanks to our effort, we are finally able to significantly improve the testing within our Node.js services. We can also start to work in localhost whereas before, developers used remote environments.

In two years we have managed to recreate a healthy development environment with good practices and unit and functional testing on almost all our projects.

📢 My take on TypeORM

If I were in the same situation tomorrow I would probably try another way/solution (like Massive.js). For example, TypeORM poor performance will probably be a topic in the future for my team.

As I said at the beginning, I'm not a fan of ORMs and in the context of personal projects, I do without them almost all the time.

However, I must admit that we succeeded with TypeORM and that the result is not too bad either. There is probably no silver bullet 🤷.

🙇 Conclusion

Many engineers would have given up at the beginning thinking that it would not be worth the energy to fight 😰.

It's a bit simple to always want to start from scratch 😝. For me it was a challenge, to face reality which is sometimes hard to accept and forces us to make different choices 😉.

It was also a great team effort with a lot of trusts 👯. We had invested a lot and as a lead I was afraid I had made the wrong choice. But with Alexandre it is always a pleasure to see that today all this pays off.

I'm not quoting everyone but thanks to those who actively helped and worked on the project especially in the early stage.

Thanks for reading and as usual see you soon for a new article 😘

NodeSecure Vuln-era

Thomas.G — Thu, 21 Jul 2022 08:15:29 +0000

😍 Logo and cover by our beloved medhi bouchard ❤️

Hello 👋,

Back for a little article about the rebranding of one of the NodeSecure tools: Vulnera (previously vuln, the vuln-era has begun!).

An opportunity for me to also write about this wonderful project that was born with the redesign of the back-end less than a year ago ⌚. If you don't remember I wrote an article:

Announcing new NodeSecure back-end

Thomas.G for NodeSecure ・ Sep 11 '21

#node #javascript #security

Don't wait and dive in 🌊 with me to discover this tool 💃.

What is Vulnera ? 👀

Vulnera is a package that allows you to programmatically fetch your Node.js project vulnerabilities from multiple sources or strategies:

NPM Audit (Github Advisory Database)
Sonatype OSS Index
deprecated Node.js Security WG Database
Snyk

📢 Feel free to push new sources (we have a guide on how to add/contribute one).

The code was originally designed for vulnerability management within the Scanner. Yet, its API is evolving with the objective of making it a full-fledged project.

import * as vulnera from "@nodesecure/vulnera";

const def = await vulnera.setStrategy(
  vulnera.strategies.NPM_AUDIT
);

const vulnerabilities = await def.getVulnerabilities(process.cwd(), {
  useStandardFormat: true
});
console.log(vulnerabilities);

Standard vulnerability format 👯

We have created a standard format to reconcile the different sources.

export interface StandardVulnerability {
  /** Unique identifier for the vulnerability **/
  id?: string;
  /** Vulnerability origin, either Snyk, NPM or NodeSWG **/
  origin: Origin;
  /** Package associated with the vulnerability **/
  package: string;
  /** Vulnerability title **/
  title: string;
  /** Vulnerability description **/
  description?: string;
  /** Vulnerability link references on origin's website **/
  url?: string;
  /** Vulnerability severity levels given the strategy **/
  severity?: Severity;
  /** Common Vulnerabilities and Exposures dictionary */
  cves?: string[];
  /** Common Vulnerability Scoring System (CVSS) **/
  cvssVector?: string;
  /** CVSS Score **/
  cvssScore?: number;
  /** The range of vulnerable versions */
  vulnerableRanges: string[];
  /** The set of versions that are vulnerable **/
  vulnerableVersions: string[];
  /** The set of versions that are patched **/
  patchedVersions?: string;
  /** Overview of available patches **/
  patches?: Patch[];
}

You can always use the original formats of each source of course 😊. We have implemented and exposed TypeScript interfaces for each of them.

Usage in Scanner 🔬

On the scanner we have all the necessary information because we go through the dependency tree 🎄. At the end of the process, we recover all vulnerabilities by iterating spec by spec within the hydratePayloadDependencies strategy method.

const {
  hydratePayloadDependencies,
  strategy
} = await vulnera.setStrategy(
  userStrategyName // SNYK for example
);
await hydratePayloadDependencies(dependencies, {
  useStandardFormat: true,
  path: location
});

payload.vulnerabilityStrategy = strategy;

The following diagram explains the overall behavior and interactions between the Scanner and Vulnera.

If you want to learn more about the Payload you can check the TypeScript interface here.

What's next ? 🚀

Some sources are more difficult to exploit than others (for NPM we use Arborist which simplifies our lives).

const { vulnerabilities } = (await arborist.audit()).toJSON();

However, we have to think and create mechanics to exploit sources like Sonatype 😨. This is required for API like getVulnerabilities().

Among the major subjects and ideas we are working on:

Create a private database to benchmark the sources between them (see #29).
Merging multiple sources in one (see #25).
Fetch vulnerabilities of a given remote package (with support for private registry like verdaccio). At the moment we only support the analysis of a local manifest or a payload of the scanner.

Credits 🙇

This project owes much to our core collaborator Antoine COULON who invested a lot of energy to improve it 💪.

Fun fact: its first contribution 🐤 on NodeSecure was also on the old version of the code Scanner that managed vulnerabilities.

But I don't forget individual contributions 👏

Mathieu Kahlaoui for adding the getVulnerabilities() API
Oleh Sych for adding Snyk strategy
Medhi for his work on the logo

NodeSecure / vulnera

Programmatically fetch security vulnerabilities with one or many strategies (NPM Audit, Sonatype, Snyk, Node.js DB).

The vuln-era has begun! Programmatically fetch security vulnerabilities with one or many strategies. Originally designed to run and analyze Scanner dependencies it now also runs independently from an npm Manifest.

Requirements

Node.js v22 or higher

Getting Started

This package is available in the Node Package Repository and can be easily installed with npm or yarn.

$ npm i @nodesecure/vulnera
# or
$ yarn add @nodesecure/vulnera

Usage example

import * as vulnera from "@nodesecure/vulnera";

await vulnera.setStrategy(
  vulnera.strategies.GITHUB_ADVISORY
);

const definition = await vulnera.getStrategy();
console.log(definition.strategy);

const vulnerabilities = await definition.getVulnerabilities(process.cwd(), {
  useFormat: "Standard"
});
console.log(vulnerabilities);

Available strategy

The default strategy is NONE which mean no strategy at all (we execute…

View on GitHub

Thanks 🙏 for reading me and see you soon for another article!

NodeSecure CLI v2.0.0

Thomas.G — Wed, 29 Jun 2022 09:48:03 +0000

Hello 👋,

I am writing this article with excitement and after several months of work. With the core team we are thrilled to announce that we are publishing a new version of the UI.🚀.

As you are reading these lines I am probably under the sun ☀️ of Tel Aviv for the NodeTLV conference where I will give a talk about NodeSecure and some other tools.

What an incredible journey 😍. Four years ago I was working on my tool alone 😥... But now more than a dozen developers are contributing to the project and I can only thank all of you for your precious support 🙏.

If you are new, then let me introduce you to the project

🐤 Getting started with NodeSecure

NodeSecure is an organization gathering a lot of individual projects that will allow you to improve the security and quality of your projects 💪. With our tools you can visually discover the dependencies you use on a daily basis and learn more about them 📚.

Our most notable project is:

NodeSecure / cli

JavaScript security CLI that allow you to deeply analyze the dependency tree of a given package or local Node.js project.

🐢 Node-Secure CLI 🚀

a Node.js CLI to deeply analyze the dependency tree of a given NPM package or Node.js local app

📜 Features

Run a static scan on every JavaScript files and sort out warnings (unsafe-regex, unsafe-import etc) and the complete list of required expr and statements (files, node.js module, etc.).
Return complete composition for each packages (extensions, files, tarball size, etc).
Packages metadata from the npm registry API (number of releases, last publish date, maintainers etc).
Search for licenses files in the tarball and return the SPDX expression conformance of each detected licenses.
Link vulnerabilities from the multiple sources like GitHub Advisory, Sonatype or Snyk using Vulnera.
Add flags (emojis) to each packages versions to identify well known patterns and potential security threats easily.
First-class support of open source security initiatives like OpenSSF Scorecard.
Generate security report (PDF).

🚧 Requirements

Node.js v22 or higher

💃 Getting

…

View on GitHub

How can you use it? It's easy, you just have to install globally the CLI with npm:

$ npm i @nodesecure/cli -g

# Analyze a remote package on the NPM Registry.
# Note: also work with a private registry like gitlab or verdaccio
$ nsecure auto fastify

# Analyze a local manifest (or local project).
# -> omit the package name to run it at the cwd.
$ cd /myproject
$ nsecure auto

We have many other projects and many opportunities for you to contribute. Feel free to join us on Discord to discuss.

👀 What's changed in v2.0.0 ?

A lot to be honest 😆. Our initial idea was simply to improve and complete the interface (We went a bit overboard I guess 😅).

One of the things that became problematic was the lack of space in the interface 😨. So we had to completely redesign the UX. I have to thank Medhi Bouchard, who spent dozens of hours designing UI on figma (Without him all this would have been much more difficult to achieve 💪).

Multiple views

This new interface offers several distinct views:

Home (global informations about the project you asked to analyze).
Network (where we are drawing the dependency tree).
Settings (which allows you to customize your experience with the tool)

Note: It is also possible to switch between each view with a keyboard shortcut (which corresponds to the capitalized character).

Home view

The home view is a replacement for the old Global stats button. We have been working to bring more attention to the information.

To summarize the information we find in this view;

Global stats on the project (direct vs indirect, size, downloads)
Licenses and Extensions
Authors
Global warnings (not visible in the screenshot since there is none).
Links to Github and NPM.

We plan to expand this view with even more information and really cool gadgets. We also want to bring more attention and information around the creators and maintainers.

🔧 Settings view

This is the new kid in the town. There is not much to customize yet but that will come with time.

One of the key ideas of NodeSecure is that each developer and maintainer can customize their experience with the tool.

Some of our warnings have a lot of false positives that is real, so you will be able to ignore them if you don't find them relevant.

Eventually the options will allow to make more clear-cut decisions like tagging a maintainer's library (which will be useful during incidents like the one with Faker.js or node-ipc).

🌎 Network view

We have slightly improved the network view and updated the colors for something more pleasant.

In version 1.4.0 of our Vis-network implementation, we have also implemented different theme for parent and child nodes (What you can see in the screenshot below).

Note: We have not abandoned the "Dark" theme. Eventually it will be possible to switch from a light to a dark theme in the settings.

🚀 New left pannel

We wanted to keep the spirit of the old interface where we could retrieve information about a package very quickly. However we want to avoid as much as possible the need to scroll to get the information.

No more popup 💃. All information is now directly accessible in this new panel.

This new design is divided into the following sub-panels:

Overview (Package informations, github stats, etc).
Files and size (with bundlephobia).
Scripts and Dependencies.
Threats and issues in JavaScript source.
Vulnerabilities.
Licenses conformance (SPDX).

There is also much more information than before. For example, I've been wanting to implement vulnerabilities in the interface for two years and it's now done:

Note: I remind you that we support multiple strategy for vulnerabilities like Sonatype or Snyk.

Scripts

This new version allows you to consult the scripts of a package. Really cool combined with the 📦 hasScript flag. Most supply chain attack uses a malicious script ... so it became important for us to be able to consult them in the UI.

Threats in source code

This version implements the latest release of JS-X-Ray which includes new features;

Detecting weak crypto algorithm (md5, sha1 ...).
Warnings now have a level of severity (like vulnerabilities).

There is still a lot of work to be done on the interface, especially to better visualize the faulty code. You will notice that the links to access NPM and Unpkg are now always present in the header.

Licenses conformance

The information is still the same, but the design is a little more enjoyable. We have also added a small tooltip if you want to know more about SPDX.

The title and file name are clickable. The first one will open the license page on the SPDX website and the second one the file itself on unpkg.

Others

We have slightly improved the short descriptions of the flags and they are now clickable (this will open the wiki directly to the relevant flag).

Also in the scripts & dependencies section you will find a show/hide button on the third-party dependencies.

Still the same behavior as in the old version, it will hide in the network all the children of the package.

New documentation/wiki

We have developed a brand new documentation-ui module that allows us to implement a wiki on any of our projects.

In this new version you can open the wiki by clicking on the button with the book icon on the right side of the screen. We now also have documentation on the warnings of our static analyzer JS-X-RAY accessible in the SAST Warnings pannel of the wiki.

👯 Credits

All this work is possible thanks to the different contributors and contributions they made those last few months.

Their simple presence, good mood and spirit were a source of inspiration and motivation for me. Thanks you very much ❤️

Conclusion

As always we move forward and evolve. We continue to work hard to improve security in the JavaScript ecosystem and we look forward to being joined by other developers with the same commitment.

Thanks for reading me and see you soon for another great story!

A technical tale of NodeSecure - Chapter 2

Thomas.G — Mon, 06 Jun 2022 19:24:48 +0000

Hello 👋,

I'm back at writing for a new technical article on NodeSecure. This time I want to focus on the SAST JS-X-Ray 🔬.

I realized very recently that the project on Github was already more than two years old. It's amazing how time flies 😵.

It's been a long time since I wanted to share my experience and feelings about AST analysis. So let's jump in 😉

💃 How it started

When I started the NodeSecure project I had almost no experience 🐤 with AST (Abstract Syntax Tree). My first time was on the SlimIO project to generate codes dynamically with the astring package (and I had also looked at the ESTree specification).

One of my first goals for my tool was to be able to retrieve the dependencies in each JavaScript file contained within an NPM tarball (By this I mean able to retrieve any dependencies imported in CJS or ESM).

I started the subject a bit naively 😏 and very quickly I set myself a challenge to achieve with my AST analyser:

function unhex(r) {
   return Buffer.from(r, "hex").toString();
}

const g = Function("return this")();
const p = g["pro" + "cess"];

const evil = p["mainMod" + "ule"][unhex("72657175697265")];
evil(unhex("68747470")).request

The goal is to be able to output accurate information for the above code. At the time I didn't really know what I was getting into 😂 (But I was passionate about it and I remain excited about it today).

I thank Targos who at the time submitted a lot of code and ideas.

To date the SAST is able to follow this kind of code without any difficulties 😎... But it wasn't always that simple.

🐤 Baby steps

One of the first things I learned was to browse the tree. Even for me today this seems rather obvious, but it wasn't necessarily so at the time 😅.

I discovered the package estree-walker from Rich Harris which was compatible with the EStree spec. Combined with the meriyah package this allows me to convert a JavaScript source into an ESTree compliant AST.

import { readFile } from "node:fs/promises";

import { walk } from "estree-walker";
import * as meriyah from "meriyah";

export async function scanFile(location: string) {
  const strToAnalyze = await readFile(location, "utf-8");

  const { body } = meriyah.parseScript(strToAnalyze, {
    next: true, loc: true, raw: true, module: true
  });

  walk(body, {
    enter(node) {
      // Skip the root of the AST.
      if (Array.isArray(node)) {
        return;
      }

      // DO THE WORK HERE
    }
  });
}

I also quickly became familiar with the tool ASTExplorer which allows you to analyze the tree and properties for a specific code.

As a beginner, you can be quickly scared by the size and complexity of an AST. This tool is super important to better cut out and focus on what is important.

I also had fun re-implementing the ESTree Specification in TypeScript. It helped me a lot to be more confident and comfortable with different concepts that were unknown to me until then.

At the beginning of 2021 I also had the opportunity to do a talk for the French JS community (it's one more opportunity to study).

😫 MemberExpression

JavaScript member expression can be quite complicated to deal with at first. You must be comfortable with recursion and be ready to face a lot of possibilities.

Here is an example of possible code:

const myVar = "test";
foo.bar["hel" + "lo"].test[myVar]();

Computed property, Binary expression, Call expression etc. The order in which the tree is built seemed unintuitive to me at first (and I had a hard time figuring out how to use the object and property properties).

Since i created my own set of AST utilities including getMemberExpressionIdentifier.

🚀 A new package (with its own API)

When NodeSecure was a single project the AST analysis was at most a few hundred lines in two or three JavaScript files. All the logic was coded with if and else conditions directly in the walker 🙈.

To evolve and maintain the project, it became necessary to separate the code and make it a standalone package with its own API 👀.

I wrote an article at the time that I invite you to read. It contains some nice little explanations:

JS-X-Ray 1.0

Thomas.G for NodeSecure ・ Mar 30 '20

#javascript #node #security #ast

The thing to remember here is that you probably shouldn't be afraid to start small and grow into something bigger later. Stay pragmatic.

Easy to write, hard to scale 😭

It's easy to write a little prototype, but it's really hard to make it scale when you have to handle dozens or hundreds of possibilities. It requires a mastery and understanding of the language that is just crazy 😵. This is really what makes creating a SAST a complicated task.

For example, do you know how many possibilities there are to require on Node.js? In CJS alone:

require
process.mainModule.require
require.main.require

I probably forget some 😈 (as a precaution I also trace methods like require.resolve).

But as far as I'm concerned, it's really what I find exciting 😍. I've learned so much in three years. All this also allowed me to approach the language from an angle that I had never experienced or seen 👀.

Probes

On JS-X-Ray I brought the notion of "probe" into the code which will collect information on one or more specific node. The goal is to separate the AST analysis into lots of smaller pieces that are easier to understand, document and test.

Very far from perfection 😞. However, it is much better than before and the team is now helping me to improve all this (by adding documentation and tests).

It was for JS-X-Ray 3.0.0 and at the time i have written the following article (which includes many more details if you are interested).

JS-X-Ray 3.0

Thomas.G for NodeSecure ・ Feb 28 '21

#node #javascript #security #opensource

VariableTracer

This is one of the new killer feature coming to JS-X-Ray soon. A code able to follow the declarations, assignment, destructuration, importating of any identifiers or member expression.

In my experience being able to keep track of assignments has been one of the most complex tasks (and I've struggled with it).

This new implementation/API will offer a new spectrum of tools to develop really cool new features.

const tracer = new VariableTracer().trace("crypto.createHash", {
  followConsecutiveAssignment: true
});

// Use this in the tree walker
tracer.walk(node);

This simple code will allow us, for example, to know each time the method createHash is used. We can use this for information purposes, for example to warn on the usage of a deprecated hash algorithm like md5.

Here an example:

const myModule = require("crypto");

const myMethodName = "createHash";
const callMe = myModule[myMethodName];
callMe("md5");

The goal is not necessarily to track or read malicious code. The idea is to handle enough cases because developers use JavaScript in many ways.

We can imagine and implement a lot of new scenarios without worries 😍.

By default we are tracing:

eval and Function
require, require.resolve, require.main, require.mainModule.require
Global variables (global, globalThis, root, GLOBAL, window).

✨ Conclusion

Unfortunately, I could not cover everything as the subject is so vast. One piece of advice I would give to anyone starting out on a similar topic would be to be much more rigorous about documentation and testing. It can be very easy to get lost and not know why we made a choice X or Y.

Thanks for reading this new technical article. See you soon for a new article (something tells me that it will arrive soon 😏).

How i work as a Maintainer/Mentor

Thomas.G — Tue, 26 Apr 2022 14:36:42 +0000

Hello 👋

I wanted to write a relatively short post about how I like to work and interact as a maintainer and mentor (and probably as a technical lead too).

Over time I realize that many are in a position where they speculate on the right behavior to have with me (when they contribute or ask me for help). That's why I feel I have to write an article so I can build on it and avoid confusion.

What many think 👀

Many people have a very different approach and opinion on this kind of thing. For example, things that come up often in the mind of developers:

Never expect an answer or time for free 💰.
Wait for an approval, work by yourself before taking the time of the maintainer etc.
Speculate what the maintainer want or think.

Some maintainers/mentors are colder than others too. I personally hate these approaches and they don't suit me at time.

To be clear: I am not making any judgment here.

My way 💃

First of all I am a very accessible person 😇 and it is always with pleasure that I answer your questions... No matter what your level is (junior or more experimented).

Those who know me know that I love to spend hours helping (or even discuss more ordinary things). It is frequent that contributors or mentee do not ask me anything by fear of disturbing me. For contributors, it is sometimes the fear of being judged or of not being up to the task.

Some will think that if I'm not the one making the effort to interact... then I'm not interested in them. That's wrong. Unfortunately, I have too many people and things to deal with. Whether as a maintainer, lead or a mentor, I hate micromanaging relations because it take me to much time and energy.

Personally I see it more as "if you don't ask anything it mean you don't want to be involved/don't need me". The more I know the person, the more we have in common, the easier it will be for me to initiate the conversation (this is a two-way relation).

Helping juniors 🐤

I am always willing to take a lot of time to support juniors who have motivation and who prove it by actions.

It is the same in the context of open source. Sometimes this can take time and be difficult for you. Sometimes you will fail but it's really not a big deal and I will never blame you.

Some approach me in the hope of getting a job easily. I only help beginners who I know well and who have proven their motivation and skills.

Do not be afraid to make mistakes

Mistakes will be frequent when making contributions. Wanting to achieve perfection is not realistic so feel free to send something incomplete. I will work with you to complete what is wrong.

That's my job and my responsibility as a maintainer. So don't ever think that I'll be disappointed that you're missing something (or that you might not have understood).

That's all 😊. To sum up I am someone who loves to help and build projects with other developers 👯. I don't particularly chase money and for me gratitude will always be much more profitable in the long run.

💗💗💗