DEV Community: Andrés Correa Casablanca

How to create a React components ESM+CJS library

Andrés Correa Casablanca — Fri, 15 Sep 2023 06:00:00 +0000

Originally published at Coder Spirit

NOTE: This article is focused on the technical aspects of package creation and publishing, and assumes that you already know how to create React components. If you are not familiar with React, I recommend reading the official documentation first.

Introduction

Creating (and publishing) good, small and easy to use libraries is a bit more complicated than just writing nice, modular and reusable code. Each language and ecosystem has its own particularities, in our case we'll focus on the JavaScript and TypeScript ecosystems, putting special emphasis on frontend development.

The following points are usually relevant when creating & publishing a JavaScript/TypeScript library:

There are different module systems (ESM, CommonJS, IIFE, UMD, etc.) that are not compatible with each other. Supporting all of them is not always necessary, but it's a good idea to support at least ESM and CommonJS; doing so is not always trivial.
We might have to consider different environments (Browser(s), Node, Deno, Bun, etc.).
In the case of JavaScript, we might have to target older versions of the language (to ensure compatibility with older browsers or engines).
In the case of TypeScript, we should compile our code before publishing it (to ensure that it can be consumed by pure JavaScript projects).
We might want to make our bundles as small as possible, as well as tree-shakeable.
As the cherry on top, we might also want to provide type definitions to maintain interoperability with TypeScript, and source maps to make debugging easier.

With all this in mind, let's see how we can create a React component library that fulfills all these requirements 😄.

Project Setup

NOTE: Although I've done my best to describe all the relevant details in this article, it might be useful for you as a reader to explore some real code in case you find holes in the explanation that follows.

The Package Manager

For the rest of this article, I'll assume that we are using pnpm as our package manager of choice. If you are using npm or yarn, then you'll have to adapt the commands accordingly. In any case, I recommend using pnpm over the others.

To create a new project, we can run the following commands:

mkdir beautiful-tree # or whatever name you want to give to your project
cd beautiful-tree
pnpm init # This will create a package.json file. We'll edit it later.

Dependencies

We want to install react and react-dom as peer dependencies (and not as "normal" dependencies), if you are not sure why, then I recommend reading the article from the previous link.

pnpm add --save-peer react react-dom

The rest of the dependencies are development dependencies, so we'll install them as such.

We'll use typescript to type-check our code
react and react-dom are necessary for our tests to work, while @types/react and @types/react-dom are necessary for our type checks.
rollup is the module bundler that we'll use. @rollup/plugin-terser is a plugin that we'll use to minify our code. @rollup/plugin-typescript is a plugin that we'll use to transpile our TypeScript code, and rollup-plugin-dts will help us to emit type definitions. tslib is also necessary as a peer dependency for @rollup/plugin-typescript.
We'll use vitest as our test runner, together with @testing-library/react to test our React components, and jsdom to provide a browser-like environment for our tests.
publint will help us validate that our package.json file is correct (specifically the exports and types fields)

pnpm add --save-dev \
     jsdom \
     publint \
     react \
     react-dom \
     rollup \
     @rollup/plugin-terser \
     @rollup/plugin-typescript \
     @testing-library/react \
     @types/react \
     @types/react-dom \
     rollup-plugin-dts \
     tslib \
     typescript \
     vitest

When it comes to "normal" dependencies, we need them to be either dual ESM/CJS packages, or at least ESM packages. In our case, we won't be installing any, but it's important to keep that in mind if we want to meet our requirements.

TypeScript

Although we can run pnpm tsc --init to create a new tsconfig.json file, for this example we'll create it manually, so we can explain each relevant option in detail.

NOTE: From now on, you will find that the code snippets contain comments with insights that would take too much time to explain in the main text. I recommend reading them.

Some of them will be inside JSON files, so you'll have to strip them out before using the code.

{
  "compilerOptions": {
    // To ensure that our code is compatible with "slightly old" browsers,
    // we target an older version of ECMAScript (ES2020 in this case).
    // See https://www.typescriptlang.org/tsconfig#target
    "target": "es2020",

    // We load these 3 libraries in the global scope so TypeScript knows
    // about the DOM and ES2020 features.
    // See https://www.typescriptlang.org/tsconfig#lib
    "lib": ["ES2020", "DOM", "DOM.Iterable"],

    // To have compatibility with ES modules, we can use the values:
    // - ES2015: Very basic support for ES modules
    // - ES2020: Supports dynamic imports and import.meta
    // - ES2022: Supports top-level await
    // See https://www.typescriptlang.org/tsconfig#module
    "module": "ES2020",

    // The 'bundler' resolution strategy is similar to the 'node16' and
    // 'nodenext' strategies (in that it supports package.json "exports" and
    // "imports" fields), but it allows us to not have to specify the file
    // extension when importing files (which is nice, because we'll be bundling
    // everything anyway, so the extensions are not relevant).
    // See https://www.typescriptlang.org/tsconfig#moduleResolution
    "moduleResolution": "Bundler",

    // This tells TypeScript to use the `react-jsx` factory function when
    // transpiling JSX syntax.
    "jsx": "react-jsx",

    // Our code will be placed in the ./src directory
    "baseUrl": "./src",

    // We'll use Rollup to emit code, instead of tsc.
    "noEmit": true,

    // Most bundlers have limitations when dealing with features such as
    // `const enum` (which can affect code generation across different files).
    // Because of this, it is a good idea to ensure that every module is
    // compilable on its own, without relying on other modules.
    // See https://www.typescriptlang.org/tsconfig#isolatedModules
    "isolatedModules": true,

    // I'm surprised that this option is still not enabled by default, because
    // it's basically a bug fix for a mistake they made in their past
    // assumptions on how ES modules work.
    // See https://www.typescriptlang.org/tsconfig#esModuleInterop
    "esModuleInterop": true,

    // Feel free to not use these options if you don't want to, but my
    // suggestion is to always use it, so you can catch more errors at compile
    // time.
    "strict": true,
    "checkJs": true,
  },
  // In a real project, we might need to add some more directories to the
  // "exclude" array, but for this example we'll keep it simple.
  "exclude": [
    "dist/**/*",
    "node_modules/**/*",
  ]
}

Rollup

As we mentioned before, we'll use Rollup as our module bundler, as it combines flexibility with ease of use, and it allows us to generate different types of bundles.

Let's create a rollup.config.mjs file at the root of our repository (adapt the referred files to your needs):

import { defineConfig } from 'rollup'

// We'll use this plugin to generate the .d.ts files
import dts from 'rollup-plugin-dts'

// We'll use this plugin to transpile our TypeScript code
import pluginTs from '@rollup/plugin-typescript'

// We'll use this plugin to minify our code
import terser from '@rollup/plugin-terser'

// Some constants that we'll use later
const input = 'src/main.ts'
const external = ['react', 'react-dom', 'react/jsx-runtime']
const globals = {
  react: 'React',
  'react-dom': 'ReactDOM',
  'react/jsx-runtime': 'jsxRuntime',
}

export default defineConfig([
  {
    input,
    output: [
      // We tell Rollup to generate a CommonJS bundle
      {
        // The .cjs extension is not strictly necessary, but it helps
        file: 'dist/beautiful-tree.cjs',
        format: 'cjs',
        globals, // We tell Rollup how to map the external dependencies
        sourcemap: true, // We want sourcemaps for debugging purposes
      },
      // We tell Rollup to generate an ESM bundle
      {
        // The .mjs extension is not strictly necessary, but it helps
        file: 'dist/beautiful-tree.mjs',
        format: 'es',
        globals, // We tell Rollup how to map the external dependencies
        sourcemap: true, // We want sourcemaps for debugging purposes
      },
      // We tell Rollup to generate an IIFE bundle
      // (for browser environments)
      {
        name: 'BeautifulTree', // The global name given to the "bundle"
        file: 'dist/beautiful-tree.iife.js',
        format: 'iife',
        globals, // We tell Rollup how to map the external dependencies
        sourcemap: true, // We want sourcemaps for debugging purposes
      },
      // We tell Rollup to generate an UMD bundle
      // (as a sort of "universal" default)
      {
        name: 'BeautifulTree', // The global name given to the "bundle"
        file: 'dist/beautiful-tree.umd.js',
        format: 'umd',
        globals, // We tell Rollup how to map the external dependencies
        sourcemap: true, // We want sourcemaps for debugging purposes
      },
    ],
    // We tell Rollup to treat the following dependencies as external
    // (see the "external" constant above)
    external,
    plugins: [pluginTs()],
  },
  // We have another set of output files that uses a slightly different
  // configuration: We want to minify the code, so we use `terser` as a plugin,
  // and add the `min` infix to the output filenames.
  {
    input,
    output: [
      {
        file: 'dist/beautiful-tree.min.cjs',
        format: 'cjs',
        globals,
        sourcemap: true,
      },
      {
        file: 'dist/beautiful-tree.min.mjs',
        format: 'es',
        globals,
        sourcemap: true,
      },
      {
        name: 'BeautifulTree',
        file: 'dist/beautiful-tree.min.iife.js',
        format: 'iife',
        globals,
        sourcemap: true,
      },
      {
        name: 'BeautifulTree',
        file: 'dist/beautiful-tree.min.umd.js',
        format: 'umd',
        globals,
        sourcemap: true,
      },
    ],
    external,
    plugins: [pluginTs(), terser()],
  },
  // We also want to generate type definitions for our library:
  {
    input,
    output: [
      // Both files will be equal, but we need to generate them separately
      // to deal with some edge cases related to how TypeScript loads types.
      // An interesting thread on the topic (sadly, in Xitter):
      // https://twitter.com/AndaristRake/status/1695549037556949344
      { format: 'cjs', file: 'dist/beautiful-tree.d.cts' },
      { format: 'es', file: 'dist/beautiful-tree.d.mts' },
    ],
    external,
    plugins: [dts()],
  },
])

You probably noticed that we are generating .cjs and .mjs files, when in theory using the .js extension should be enough if our package.json exports are correctly configured (if you don't know about that, don't worry, we'll get to it in a few lines). The reason for that is plain and simple defensive programming; I've seen too many weird bugs caused by tooling that does not properly "understand" the exports field in package.json.

Creating a dummy React component

Now that we have both TypeScript and Rollup configured, we can create a dummy React component to test that everything is working as expected.

Let's create a file called BeautifulTree.tsx inside the ./src directory:

export type BeautifulTreeProps = {
  id: string
  width: number
  height: number
}

export function BeautifulTree({ id, width, height }: BeautifulTreeProps): JSX.Element {
  return (
    <svg
      xmlns="http://www.w3.org/2000/svg"
      id={id}
      viewBox={`0 0 ${width} ${height}`}
      style={{
        width: `${width}px`,
        height: `${height}px`,
      }}
      className={'beautiful-tree-react'}
    >
    {/* replace with real code to draw a tree */}
    <circle
      cx={width * 0.5}
      cy={height * 0.5}
      r={Math.min(width, height) * 0.5}
    />
    </svg>
  )
}

and a main.ts file inside the ./src directory:

// It is a good practice to separate type imports/exports from
// value imports/exports
export type { BeautifulTreeProps } from './BeautifulTree'
export { BeautifulTree } from './BeautifulTree'

Building our dummy React component

Now, if we run the command

pnpm rollup --config rollup.config.mjs

we should see that a bunch of files have been generated inside the ./dist directory (type definitions, source maps, and minified/non-minified versions of different bundles).

Visualizing our dummy React component

To visualize our dummy React component, we'll use Storybook. I won't get into too much details, because Storybook itself guides us through the process of creating a new project with an interactive tutorial.

# Install Storybook and initialize its configuration
pnpm dlx storybook@latest init

When it asks us about the builder for the project, choose Vite, which is much faster than the other proposed alternative (at the time being, "Webpack 5").

We can close the Storybook process by typing Ctrl+C in the terminal, and start it again with the command:

pnpm run storybook

Now that we have Storybook installed, we can create a new "story" where we can visualize our component.

Let's create a file called BeautifulTree.stories.tsx inside the ./src/stories directory:

import { BeautifulTree } from '../BeautifulTree'
import type { Meta, StoryObj } from '@storybook/react'

// To learn more about this, check the official Storybook documentation.
const meta = {
  title: 'BeautifulTree',
  component: BeautifulTree,
  parameters: { layout: 'centered' },
  tags: ['autodocs'],
  argTypes: {},
} satisfies Meta<typeof BeautifulTree>

export default meta

type Story = StoryObj<typeof BeautifulTree>

export const MyFirstStory: Story = {
  args: {
    id: 'my-first-story',
    width: 200,
    height: 200,
  },
}

If we now go to the Storybook UI, we should see that we have a new story called "MyFirstStory", and if we click on it we should see our dummy React component.

We could certainly live without Storybook, but it's a nice tool to verify visually that our components are working as expected, while also serves as a sort of interactive documentation.

Vitest

Once we have our component, and we have verified that it works as expected, we can prepare tests based on this knowledge. We can also prepare unit tests for internal details that are not tied to the UI, but those should be simpler to implement and don't need much explanation.

For our tests, we'll use Vitest, which is a lightweight test runner with native support for ESM, TypeScript and JSX, and happens to be much faster than Jest.

Our configuration file is vitest.config.mts:

// eslint-disable-next-line import/no-unresolved
import { defineConfig } from 'vitest/config'

export default defineConfig({
  test: {
    // We'll use the `jsdom` environment to run tests that depend on the
    // browser DOM
    environment: 'jsdom',
  },
})

We'll create an example tests file called BeautifulTree.test.tsx inside the ./src/tests directory:

import { BeautifulTree } from '../BeautifulTree'
import { describe, expect, it } from 'vitest'
import { render } from '@testing-library/react'

describe('BeautifulTree', () => {
  it('renders a boring circle', () => {
    const rendered = render(
      <BeautifulTree
        id="my-boring-test-tree"
        width={200}
        height={200}
      />
    )

    // NOTE: Snapshots are like black boxes, and depending too much on them for
    //       testing can make our tests brittle. Use them sparingly.
    //
    // The first time that we run this test, it will generate a snapshot that
    // will be used to compare against future runs. We trust that the first
    // execution is correct because we saw it in Storybook before.
    expect(rendered).toMatchSnapshot()
  })
})

To run our tests, we can execute the following command:

pnpm vitest run

The `package.json` File

After we have our components ready, we can start thinking about what do we have to write in our package.json file to ensure that our library is compatible with different module systems and environments.

{
  "name": "@beautiful-tree/react",
  "version": "0.1.1",
  "private": false,
  // `main` is the legacy way of defining the CJS entry point of the library.
  // We keep it for backwards compatibility.
  "main": "./dist/beautiful-tree.cjs",
  // `module` is the legacy way of defining the ESM entry point of the library.
  // We keep it for backwards compatibility.
  "module": "./dist/beautiful-tree.mjs",
  // top-level `types` is the legacy way of telling how to load the type
  // definitions. We keep it for backwards compatibility.
  "types": "./dist/beautiful-tree.d.cts",
  // `exports` is the modern way of defining the entry points of the library.
  "exports": {
    // Because we have minified and non-minified versions of our code, we add
    // an extra level of nesting to the `exports` object (usually not needed).
    // This one is for the path "@beautiful-tree/react";`
    ".": {
      // We need an extra level of nesting for our "import" key because it
      // needs to have its own independent "types" key.
      "import": {
                // the `types` entry must be always the first one.
        "types": "./dist/beautiful-tree.d.mts",
        "default": "./dist/beautiful-tree.mjs" // ESM entry point
      },
      "require": {
        "types": "./dist/beautiful-tree.d.cts",
        "default": "./dist/beautiful-tree.cjs" // CJS entry point
      },
      // We don't bother on adding types for the browser and default
      // entries, because we don't use these modules during development.
      "browser": "./dist/beautiful-tree.iife.js", // browser entry point
      "default": "./dist/beautiful-tree.umd.js" // default entry point
    },
    // This one is for the path "@beautiful-tree/react/min";`
    "./min": {
      "import": {
        "types": "./dist/beautiful-tree.d.mts",
        "default": "./dist/beautiful-tree.min.mjs"
      },
      "require": {
        "types": "./dist/beautiful-tree.d.cts",
        "default": "./dist/beautiful-tree.min.cjs"
      },
      "browser": "./dist/beautiful-tree.min.iife.js",
      "default": "./dist/beautiful-tree.min.umd.js"
    }
  },
  // We use the `files` allow-list to ensure that only the generated bundles
  // are published to npm.
  "files": ["dist"],
  "scripts": {
    "build": "rollup --config rollup.config.mjs",
    "test": "vitest run",
    "test:watch": "vitest"
  },
  "devDependencies": {
    // ...
  },
  "peerDependencies": {
    // ...
  }
}

Validating our package

Publint

A very nice tool that we can use to validate that our package.json file declares type definitions and entry points correctly is publint, which also has its own online checker.

pnpm publint --strict # I recommend using the --strict option

AreTheTypesWrong

Another tool that we can use to validate that our package properly exports type definitions is the website Are The Types Wrong?.

We can also use its CLI version, although I don't recommend adding it to our CI pipeline, because it's difficult to exclude some irrelevant problems with enough precision (we would exclude too much, or too little).

# To install it
pnpm add --save-dev @arethetypeswrong/cli

# To run it
pnpm attw --pack

Conclusion

Thank you for reading until the end! 😄 I hope that this article has been useful/interesting to you. If you liked it, it would mean a lot to me if you could share it with your friends and colleagues, leave a comment, or add a reaction 💖.

A better way to use Dotenv

Andrés Correa Casablanca — Mon, 26 Jun 2023 06:00:00 +0000

Preface

In the ever-changing world of NodeJS development, the choice of tools can profoundly impact a developer's journey. Among these tools, the dotenv package has long been a trusted partner, enabling developers to load environment variables from .env files into process.env.

However, despite its versatility and widespread use, the standard practices in utilizing dotenv often leave much to be desired. This article will explore how refining our usage patterns of dotenv can result in simpler and more robust code.

Introduction

Consider the following snippets, showing the most common way of using dotenv:

require('dotenv').config(/* options */) // For CommonJS

// code depending on environment variables

import 'dotenv/config' // For ESM-compatible code

// code depending on environment variables

The ESM case stands out because it doesn't initiate a function call and instead relies on side effects during import. In their documentation, they try to justify it by pointing fingers to the "unintuitive" behavior of ESM imports. I beg to differ, the problem is not with ESM, but with the usage patterns they propose in their documentation.
- Their proposed examples encourage defining module-scoped variables that are later exported, which is not ideal, as it makes it difficult to control the execution flow and to test modules in isolation.
- dotenv allows us to avoid performing implicit global side effects. However, its default behavior promotes these side effects, especially in the "ESM" case.

If we take care of implementing proper encapsulation and avoiding global or module-scoped variable declarations, then we can write instead (for the ESM case):

import { config } from 'dotenv'

config(/* options */)

// code depending on environment variables

This is more verbose, but also more explicit, gives us more control (as we can pass parameters to the config call) and makes it easier to test our code in isolation.

But I went ahead of myself, why do we care about extra flexibility if we are only loading .env files? Well, we might want to:

Load other files instead, for example, something like .env.test, .env.local, or .env.ci.
Decide whether to override existing environment variables or not when loading the .env file.
Load the defined variables not into process.env, but into another object of our choice.

Having said that, I don't think that we should be worrying about these details in our application code. This logic belongs elsewhere.

Removing clutter

You see, one question that comes to my mind is why are we using dotenv for production code in the first place? We are certainly not going to rely on .env files up there in the cloud, are we?

Given that we (hopefully) want to use dotenv for development-only purposes, we can apply these steps:

Move that dependency to the devDependencies section of our package.json file. This has the benefit of reducing the size of our production bundle (by 71.6 kB) and the complexity of our application logic.
Shift the usage of dotenv to the preloading phase of our application, this way we don't need to import it inside our application, and we'll use it only in our development scripts. This has the benefit that we don't need to worry about the order of execution of application code depending on environment variables.

node -r dotenv/config ./src/index.js # Dev environment

node ./src/index.js # Production environment

If we need to parametrize the behavior of dotenv, we can set DOTENV_CONFIG_<OPTION> environment variables, for example:

DOTENV_CONFIG_PATH="./.env.test"  node -r dotenv/config ./src/tests.js

or if we care about cross-platform compatibility (i.e. Windows support), we can use cross-env (which I also recommend to install as a dev dependency):

cross-env DOTENV_CONFIG_PATH="./.env.test"  node -r dotenv/config ./src/tests.js

Another way to look at this is that we are moving the responsibility of loading environment variables from our application to the environment itself. The complexity is not gone, but it is now in a place where it is easier to manage.

Final tweaks

There are many "plugin" packages for dotenv that extend its functionality, one of the most interesting ones is dotenv-expand, which allows us to "expand" environment variables in our .env files.

For example, if we have the following .env file, dotenv-expand will expand the HOST variable into development.example.com:

NODE_ENV=development
HOST=${NODE_ENV}.example.com

The problem is that we cannot easily use dotenv-expand in the preloading phase, but luckily for us we still have options. dotenv-cli is a CLI wrapper around dotenv and dotenv-expand that allows us to do so:

dotenv -e ./.env.test node ./src/tests.js

This one is my personal favorite, as it enables us to use the full power of dotenv and dotenv-expand in our development scripts in a much less verbose way, while still keeping the responsibility of loading environment variables outside of our application.

Conclusion

This article was focused on dotenv, but if there are any takeaways I'd like to leave you with, they are these:

Be critical of the tools you use, and don't be afraid to challenge their most common usage patterns.
When possible, try to minimise the amount of dependencies in your production code.
Try to keep your application logic as simple as possible, don't pollute it to handle use cases that only apply to development or testing environments.

Thank you for reading me!

Credentials Leak with Knex

Andrés Correa Casablanca — Tue, 06 Jun 2023 06:00:00 +0000

Preamble

I wasn't planning on writing this post, but I think it's time to talk about this problem openly. First things first, let me be clear about something before I start: It's not my intention to bash on Knex or its maintainers.

This article will be focused on a security issue that I found in Knex and how to mitigate it, but I'll also talk briefly about the social aspects of this problem.

The Problem

Knex is (in essence) a query builder for Node.js, but its functionality does not end there. It also provides wrappers for database drivers, and a way to execute SQL queries directly (without having to pass them to a driver or client object). As we'll see, the ability to execute SQL queries directly is one of the reasons why it's possible to accidentally leak credentials without even knowing it.

Let's look at a minimal example of how to reproduce the problem:

import Knex from "knex";
import { pino } from "pino";

import { getConnectionConfig } from "./db.js";

const logger = pino();

const knex = Knex({
  client: "pg",
  connection: getConnectionConfig(),
});

const badQuery = knex.raw("CRASH AND BURN");
try {
  await badQuery;
} catch (err) {
  // We might want to log the failed query, so we can debug it later
  logger.error({ err, badQuery }, "It crashed and burned!");
}

So, taking aside the obvious bad code, where does the problem lie? Well, the problem is that the badQuery object holds a reference to the connection object, which in turn holds a reference to the credentials. This, by itself, is not enough to cause a credentials leak, but there are 2 other factors that make it happen:

The nested credentials object has enumerable properties (we'll see why this is important later)
Our logger (in this case, Pino) will serialize the badQuery object "too well".

What follows is an example of what could end up appearing in our logs:

[18:10:11.777] ERROR: It crashed and burned!
    badQuery: {
      "_events": {},
      "_eventsCount": 0,
      "client": {
        "_events": {},
        "_eventsCount": 4,
        "config": {
          "client": "pg",
          "connection": {
            "database": "svc-db-name",
            "host": "svc-db-host",
            "password": "svc-db-password", <-- here!
            "port": 5432,
            "user": "svc-db-user"
          }
        },
        "connectionConfigExpirationChecker": null,
        "connectionSettings": {
          "database": "svc-db-name",
          "host": "svc-db-host",
          "password": "svc-db-password", <-- and here too!
          "port": 5432,
          "user": "svc-db-user"
        },
                ...

We only wanted to log the poblematic query, but we ended up logging the credentials as well. Not good.

Mitigations

What can we do about it? Let's go over some mitigation strategies.

1. Use `toQuery` for logging

The first mitigation strategy is to convert the query object to string before logging it.

logger.error(
  { err, badQuery: badQuery.toQuery() },
  "It crashed and burned!"
);

I'm not a fan of this solution, because it requires Knex users to change their code, and to do it everywhere where a query is logged, but also because it's not entirely clear how good is the toQuery method at returning the real query that was executed... 😮‍💨

Don't get me wrong, doing this is better than nothing, and it can be useful in its own right if we want less verbose logs, but it's not a complete solution because it's too easy to reintroduce the problem again.

2. Enable redaction in your logger

Many loggers (including Pino) support redaction for sensitive data, and it is almost always advisable to enable it. The good thing about it is that we have a single place where we can configure redaction, and we don't have to worry about it anymore.

const logger = pino({
    redact: [
        "password",
        "connection.password"
        "connectionSettings.password",
        "*.password",
        "*.*.password",
        "*.*.*.password",
        "some.nested.list[*].password"
    ],
});

This approach is not free of problems either:

Although most "redacters" are quite flexible, they still require from us to know the path to the sensitive data, or at least the depth of the object where it is located.
- As you can see in the snippet above, we can use wildcards if we don't know the exact path, but the depth still matters (at least for Pino, other libraries might be more flexible, but I suspect they won't, mainly for performance reasons).
- Another problem that we can't solve with wildcards is that, although we can substitute array indices with [*] or property names with *, we still have to tell the redacter if it will encounter an array or a "regular" object.

Let's say for a moment that we have tested our redaction configuration thoroughly, and that we even overshoot by covering more cases than we need to. Sadly, this might not be enough.

A redaction system like this one is essentially a denylist, which means that new usage patterns might not be covered by it. For example, imagine that a developer decides to log an object that has nested query object, instead of directly logging the query. If there are arrays involved, the problem becomes even more complicated, wildcards only take us so far.

3. Do the right thing: fix Knex

I'm going to show the general idea behind the fix here. However, you can find a link to the actual PR in the References section at the end of this article.

The idea is to make the credentials property (in our case password) non-enumerable, so it won't be picked up by the logger serializer.

/**
 * Sets a hidden (non-enumerable) property on the `target` object, copying it
 * from `source`.
 *
 * This is useful when we want to protect certain data from being accidentally
 * leaked through logs, also when the property is non-enumerable on the `source`
 * object and we want to ensure that it is properly copied.
 *
 * @param {object} target
 * @param {object} source - default: target
 * @param {string} propertyName - default: 'password'
 */
function setHiddenProperty(target, source, propertyName = 'password') {
  if (!source) {
    source = target;
  }

    // THIS IS THE IMPORTANT PART
  Object.defineProperty(target, propertyName, {
    enumerable: false,
    value: source[propertyName],
  });
}

// Usage example:
// authentication.options.password becomes non-enumerable, therefore it won't be
// serialized by JSON.stringify, util.inspect or any other serializer.
setHiddenProperty(authentication.options);

We rely on Object.defineProperty to make the property non-enumerable, and we use a helper function to make it easier to use.

There is a caveat, though: as soon as we make a property non-enumerable, it won't be copied when we rely on Object.assign or the spread syntax (...) to perform object shallow copies (if we know the name of the property and access it explicitly, then we can copy it without problems).

4. Don't wait for Knex to fix it

I'll talk about this in the next section, but I don't have much trust in Knex applying the proposed fix any time soon. In the meantime, I recommend you to "patch" your knex dependency.

Yarn has a patch command (Yarn Berry only, Yarn classic doesn't support it).
PNPM has a patch command too.
NPM doesn't have a patch command, but you can use patch-package to achieve the same result.

You can base your patch on my PR, or at least you can use it as a reference.

5. Apply a combination of the previous strategies

None of the previous strategies is perfect, but combining them can result in a more robust solution that accounts for the shortcomings of each individual strategy.

Challenges in Open Source Maintenance

Some notes to set the context:

At the time of writing this article, Knex has 95 pending PRs and 737 open issues.
The only commits that are being merged are those from dependabot and from "regular" maintainers.
They have no clear security policy, so I had to report the security issue as a regular issue, in the open.
My report has been waiting for a response for 1 month now. The same goes for the solution that I proposed.
This was a known issue for pg developers, and they managed to fix it a long time ago (at the pg level), but the knowledge of this problem didn't reach Knex maintainers.

Ok, that doesn't sound good. But as I said at the beginning, I don't want to bash on Knex or its maintainers (I myself have maintained open source projects, and I neglected them due to lack of time and burnout).

Contributing code, triaging issues, and reviewing PRs all require a significant amount of time and effort. Sadly, most open source projects are understaffed, and most of their contributors are non-paid volunteers. You already know all this, but I think it's important to keep it in mind every time that we encounter these problems.

I have a few loose recommendations that (I believe) can help communities to cope with these problems (if they are not already doing it):

Establishing clear contribution guidelines, a code of conduct and a security policy can help a lot to reduce friction and to make the project more welcoming.
Defining stewardship rules can help to distribute the workload among maintainers and contributors, and to make it easier to onboard new maintainers.

Conclusion

I hope that this article will be helpful for those who are using Knex and want to ensure that they are not leaking credentials through their logs. I also hope that it served to raise awareness about some generic techniques that can be used to mitigate these kinds of problems, such as log redaction or making sensitive properties non-enumerable..

All the best 🤗

References

Reported Knex issue #5560.
Knex PR #5559, with a fix for the problem.

Exploring the LZ77 Compression Algorithm

Andrés Correa Casablanca — Mon, 05 Jun 2023 06:00:00 +0000

This article is the first in a series where we'll delve into the fascinating world of compression algorithms, starting with LZ77 (a lossless data compression algorithm). In future articles, we'll expand on its family: LZ78, LZW, LZSS, DEFLATE, and more.

You might be wondering, why explore a topic that's already been covered extensively in numerous articles and books? Well, I have some reasons:

During my previous work on (animated gosper curves), I found myself in need of reliable TypeScript/JavaScript libraries that could handle GIT, PNG and APNG files. However, the available options were either poorly maintained, tied to NodeJS, or not compatible with ECMAScript modules (ESM). This prompted me to create my own solution and document the journey.
I noticed a lack of engaging, visual and interactive explanations on the topic. This series aims to fill the gap by providing not just information, but also interactive simulations to help you understand these algorithms.

Stay tuned as we embark on this exciting journey of data compression!

A bit of history

LZ77 was first published in 1977 by Abraham Lempel and Jacob Ziv. It was by no means the first compression algorithm, but it was the first to introduce a sliding window that allowed processing arbitrary streams of data in a nearly optimal way without needing prior knowledge of the data.

Some terms & concepts

Before we dive into the algorithm, let's define some terms and concepts that will come in handy later on (don't worry if you don't understand them yet, they are here so you can refer to them later on):

The input stream is the sequence of symbols that we want to compress.
The output stream is the sequence of symbols that we get after compressing the input stream.
A character is a single symbol in the input stream, our basic data element.
The sliding window is a fixed-size¹ buffer that acts as a view into the input stream. It's composed of two parts:
- The search buffer is the part of the sliding window that contains the symbols that have already been processed.²
- The look-ahead buffer is the part of the sliding window that contains the symbols that are yet to be processed.²
The coding position is the position of the first symbol in the look-ahead buffer (but we use its absolute index respect to the input stream to refer to it).
A pointer is a pair of numbers that points to a sequence of symbols in the search buffer. In this article, we'll use the notation <length:offset>, but other notations are also used elsewhere. Also, the offset is often expressed respect to the coding position, but we'll compute it respect to the start of the search buffer.

The size of the sliding window is configurable, but it's usually a power of 2. Also, although its size is fixed during the process, it can be treated as if it was smaller at the beginning and end of the compression process.
As with the sliding window, the sizes of the search and look-ahead buffers are also configurable.

The algorithm

The algorithm is quite simple, it can be summarized in the following steps (don't fret, we'll see a simulation of it later on):

Initialize the sliding window with the first symbols of the input stream, and set the coding position to the beginning of the stream.
Find the longest sequence of symbols in the search buffer that matches the symbols in the look-ahead buffer (notice that we can find matches that span across both buffers!).
Output a <length:offset:char> tuple (a pointer followed by the first symbol in the look-ahead buffer that doesn't match the sequence) to the output stream.
If the look-ahead buffer is empty, we are done. Otherwise, move the sliding window as many symbols "to the right" as the length of the previous match, and repeat from step 2.

Notice that the algorithm does not prescribe the format of the output tuples, nor the size of the buffers, nor the pattern matching algorithm.

Compression examples

Let's see a simulation of the algorithm in action! For these examples we'll use a search buffer of size 4 and a look-ahead buffer of size 4.

The examples in this post are simple animations because I can't embed my own React components, but if you want to play with them, you can visit the original article.

The previous example is helpful, but it does not show how the match can span across both buffers. Let's see it in action:

By the way, did you notice that the output stream for the second example is shorter than the first one? That's because the second example has more redundancy (more repeated patterns), so it can be compressed more efficiently.

A decompression example

Now that we have seen how the compression process works, let's see how we can do it in reverse to recover the original input stream. For this example we'll use the output stream from the previous one.

When decompressing, we reconstruct the search buffer on the fly. In our case we do it character by character, because some of the pointers point to symbols that are yet to be decompressed.

Besides the minor complication of having pointers to symbols that are yet to be decompressed, the decompression process is simpler and faster than compression (we don't have to search for longest matches between strings).

Some observations

Up until this point, we have paid attention to the algorithm, however, apart from the algorithm itself, there are at least 3 other factors that affect how
well it performs:

The pattern matching algorithm (how we find the longest match) affects both the speed and memory usage.
The output format (how we encode the pointers) affects the compression ratio.
The parametrization (how big the search and look-ahead buffers are) directly affects the compression ratio and memory usage, but it can also affect the compression ratio indirectly by affecting the output format (a larger sliding window results in larger pointers, but it also increases the likelihood of finding longer matches).

Depending on how "clever" is the implementation, other relations between these factors and performance attributes can arise.

For example, let's imagine an encoding that uses smaller pointers for the first bytes of the stream (this is possible because the search buffer is smaller at the beginning of the compression process). This would result in a slightly better compression ratio, but the added complexity could affect how fast the algorithm is.

The pattern matching algorithm

The naive approach for pattern matching is easy to understand and implement, but has a quadratic time complexity of O(m·n) (where m is the length of the pattern and n is the length of the text).

A better (but not necessarily the best) approach is to use the Knuth-Morris-Pratt algorithm, which has a time complexity of O(m+n).

For the interactive examples in this article I implemented a modified version of KMP to account for partial matches and the fact that although we accept matches that overlap over the search and look-ahead buffers, we don't want matches with an offset greater than the size of the search buffer.

/**
 * Custom version of Knuth-Morris-Pratt algorithm that also returns the longest
 * partial match if no full match is found. It also introduces stopAfter
 * parameter to stop searching after a certain point.
 */
export const partialKMPSearch = (
    searchString: string,
    pattern: string,
    stopAfter: number = +Infinity
): [number, number] => {
    const n = searchString.length;
    const m = pattern.length;

    const table = kmpPreprocessPattern(pattern);

    let i = 0;
    let j = 0;

    let longestMatchLength = 0;
    let longestMatchIndex = -1;

    while (j < n) {
        if (j - i >= stopAfter) {
            break;
        }
        if (pattern[i] === searchString[j]) {
            j += 1;
            i += 1;
            if (i === m) {
                return [j - i, i];
            }
        } else {
            if (i === 0) {
                j += 1;
            } else {
                if (i > longestMatchLength) {
                    longestMatchLength = i;
                    longestMatchIndex = j - i;
                }

                i = table[i - 1];
            }
        }
    }

    return [longestMatchIndex, longestMatchLength];
};

/**
 * Preprocesses the pattern to generage the skip table used by the KMP algorithm
 */
const kmpPreprocessPattern = (pattern: string): number[] => {
    const m = pattern.length;
    const table = new Array(m).fill(0);

    let i = 0;
    let j = 1;

    while (j < m) {
        if (pattern[i] === pattern[j]) {
            i += 1;
            table[j] = i;
            j += 1;
        } else if (i === 0) {
            table[j] = 0;
            j += 1;
        } else {
            i = table[i - 1];
        }
    }

    return table;
};

Choosing the right pattern matching algorithm deserves an article of its own. In my case, I chose KMP because it's easy to implement and easy to adapt for partial matches. However, it's important to note that there are other notable algorithms that may be worth exploring, such as Boyer-Moore, Rabin-Karp, and many others (such as algorithms relying on suffix trees or
suffix arrays).

Parametrization & Encoding

There are many tradeoffs to consider when choosing an output format:

How many bits do we use for the length and offset?
Going further: Do we use a fixed or variable number of bits for the length and offset?
Should we use "fat pointers" (pointers that include the next symbol), or "thin pointers "(pointers that only include the length and offset)?
- In the case of "fat pointers" we could skip encoding the offset when the length is 0.
- In the case of "thin pointers", we could distinguish between pointers and symbols by looking at the first bit (0 for symbols, 1 for pointers).

Conclusion

As we have seen, LZ77 is a simple yet powerful lossless compression algorithm, or rather, a family of algorithms: It leaves many design decisions to the implementer, which can result in a wide range of implementations with different performance characteristics.

We have only scratched the surface, but I hope this article has piqued your interest in the topic. In future articles we'll explore other compression algorithms, as well as some of the topics we have left out in this one.

Animated Gosper Curves in JS

Andrés Correa Casablanca — Thu, 08 Dec 2022 23:10:00 +0000

This post is the continuation of a very old article I wrote more than 6 years ago. Back then I used Python instead of JavaScript/TypeScript, and I didn't bother about properly explaining how this works nor any other lesson we can learn from this example. Well, I'm here today to remediate that.

Note: I didn't finish GIF generation yet, so you can only see the animated version of this image in the original post, where I'm more free to include my own JS code.

L-Systems

The animated image you can see above is an example of a Gosper curve, which is an interesting particular case of an L-System that also happens to be a space filling curve.

We can understand an L-System as a tuple (A, i, P), where:

A denotes an alphabet (that is, a finite collection of symbols). Example: a, b, +, -
i denotes an initial state (in the form of a string of 1 or more symbols). Example: a
P denotes a production rule, or more plainly for our particular case: a function that takes a symbol from our alphabet A and returns us a string of symbols from that same alphabet. Example:
- - ➞ -
- + ➞ +
- a ➞ a-b--b+a++aa+b-
- b ➞ +a-bb--b-a++a+b

If we take our initial state i and recursively apply P until a certain predefined depth, we end up with a string that can be interpreted as instructions to draw our curve!

If all this formalism leaves you cold, don't fret, we'll jump straight to the code right now! 🎮.

How we structure code

There are three very important functions we need to generate our image:

The 1st one is precisely our "production rule" (the function in charge of generating the "drawing instructions").
The 2nd is the one responsible of interpreting the generated instructions and translate them into an ordered list of points on our canvas, that will act as the vertices of our drawing.
The 3rd function is responsible for taking the ordered list of points and draw coloured lines between them, giving us the final result.

Summary:

start ➞ generate "instructions string" ➞ generate points in space ➞ draw the gosper curve

The Production function

This code is in TypeScript, but it's pretty straightforward to simplify it and obtain plain JavaScript.

type ChainStep = 'a' | 'b' | '+' | '-'
type Chain = ChainStep[]

const generateChain = (
    level: number = 0,
    baseChain: Chain = ['a']
): Chain => {
    if (level === 0) {
        return baseChain.slice(0)
    }

    const newChain = baseChain.flatMap(rule => {
        if (rule === 'a') {
            return [
                'a','-','b','-','-','b','+','a','+','+','a','a','+','b','-',
            ] as const
        } else if (rule === 'b') {
            return [
                '+','a','-','b','b','-','-','b','-','a','+','+','a','+','b',
            ] as const
        }

        return rule
    })

    return generateChain(level - 1, newChain)
}

The baseChain parameter corresponds to our initial state i, while level is used to indicate how deep is going to go our recursive function. The deeper it is, the more complex our final drawing will be.

Generating points

To generate our list of points, we'll need a couple of helper functions (so we can rotate vectors and add them to points), plus some constants.

type Vector2D = readonly [number, number]
type VectorsIndex = 0 | 1 | 2 | 3 | 4 | 5

const DEG60 = Math.PI / 3 // 60º as radians
const SQUARE_SIDE = 512   // The size of our canvas

// This formula might seem tricky, as it was obtained empirically, although
// it has some solid grounding:
// - Each time we go one level further, each segment is divided into 7 more
//   segments.
// - Even at level 0, we only want it to be 3/4 of the square side
// - Powering the divisor to the square root of level is the most difficult
//   to justify bit, but it is because we're in a 2d space. If we were in
//   3d, then we would write LEVEL / 3.
const SEGMENT_SIZE = (0.75 * SQUARE_SIDE) / 7.0 ** (LEVEL * 0.5)

const toRight = [SEGMENT_SIZE, 0] as const
const toLeft = [-SEGMENT_SIZE, 0] as const

// See rotation matrix:
// https://en.wikipedia.org/wiki/Rotation_matrix#In_two_dimensions
const rotate2D = ([x, y]: Vector2D, angle: number): Vector2D => {
    const cosA = Math.cos(angle)
    const sinA = Math.sin(angle)
    return [x * cosA - y * sinA, x * sinA + y * cosA] as const
}

const addVector2D = ([x0, y0]: Vector2D, [x1, y1]: Vector2D): Vector2D => {
    return [x0 + x1, y0 + y1] as const
}

const inferPointsFromChain = (chain: Chain, firstPoint: Vector2D) => {
    const vectorsCircle = [
        toRight,
        rotate2D(toRight, +DEG60), // toNorthEast
        rotate2D(toLeft, -DEG60),  // toNorthWest
        toLeft,
        rotate2D(toLeft, +DEG60),  //toSouthWest
        rotate2D(toRight, -DEG60), // toSouthEast
    ] as const

    let vectorsIndex: VectorsIndex = 0
    let currentPoint = firstPoint

    const points = [firstPoint]
    for (const step of chain) {
        // `a` & `b` indicate us to "advance"
        // `+` & `-` indicate us to "change direction"
        if (['a', 'b'].includes(step)) {
            currentPoint = addVector2D(
                currentPoint,
                vectorsCircle[vectorsIndex]
            )
            points.push(currentPoint)
        } else if (step === '+') {
            vectorsIndex = ((vectorsIndex + 1) % 6) as VectorsIndex
        } else if (step === '-') {
            // Wee need to add 6 because fmod in JS can return negative
            vectorsIndex = ((vectorsIndex - 1 + 6) % 6) as VectorsIndex
        }
    }

    return points
}

Extra tweaks for our generated points

To make things easier, we also create a wrapper function that calls the two previously defined functions, and re-aligns our generated points to ensure they are centered in our canvas:

const computeGosperCurveByRules = () => {
    const gosperPoints = inferPointsFromChain(
        generateChain(LEVEL),
        // This 2nd parameter could actually be any point in space because
        // we'll re-align all the points afterwards. This was here before we
        // introduced the re-alignment code.
        [0.625 * SQUARE_SIDE, 0.90625 * SQUARE_SIDE] as const, // starting point
    );

    let maxX = 0;
    let minX = SQUARE_SIDE;
    let maxY = 0;
    let minY = SQUARE_SIDE;

    for (const [x, y] of gosperPoints) {
        maxX = Math.max(x, maxX);
        maxY = Math.max(y, maxY);
        minX = Math.min(x, minX);
        minY = Math.min(y, minY);
    }
    const width = maxX - minX;
    const height = maxY - minY;

    const shift = [
        (SQUARE_SIDE - width) / 2 - minX,
        (SQUARE_SIDE - height) / 2 - minY,
    ] as const;

    return gosperPoints.map((p) => addVector2D(p, shift));
}

Preparing the canvas and dealing with HiDPI

Ok! that was a lot of code, and we didn't draw anything yet. But we're getting close. Now that we know how to generate a list of points, it's time to draw them.

For that, we'll prepare our canvas, and we'll also take care of ensuring that our image doesn't become blurring in HiDPI screens. In our HTML, we'll place our canvas object (we could create it in JS as well, of course):

<canvas id='gosperCanvas' height='512px' width='512px'>
    Animated Gosper Curve
</canvas>

I wrote the following function after noticing that my generatd image was slightly blurry, and it is inspired by this article and this StackOverflow post.

function prepareHiDPIContext(canvas: HTMLCanvasElement) {
    const dpr = window.devicePixelRatio || 1;

    // Get canvas' size in CSS pixels
    const rect = canvas.getBoundingClientRect();

    // Give the canvas pixel dimensions of their CSS size * the device pixel ratio
    canvas.width = rect.width * dpr;
    canvas.height = rect.height * dpr;

    // Scale down the canvas by the same amount we increased its resolution
    canvas.style.width = `${rect.width}px`;
    canvas.style.height = `${rect.height}px`;

    const ctx = canvas.getContext("2d")!;

    // Scale all drawing operations by dpr, to avoid having to deal with it in
    // the drawing-focused code.
    ctx.scale(dpr, dpr);

    return ctx;
}

You can see the difference between implementing the rescaling trick or not:

Painting!

This piece of code is relatively big, so I'll try to explain it before you start with it, to make it easier to understand.

One relevant detail of this code is that we use an HSL color space. We fix saturation and lightness, and let the "hue" change over time, but in discrete steps (I decided to have a relatively small palette, of just 128 colors, plus black).

Another aspect I introduced is that colors change randomly, but they to change in the same direction to avoid too much flickery. To achieve that, I introduced a some "inertia" that makes the code a bit more complicated, but not by much.

// we limit the amount of color tones used in or drawing
const NUM_TONES = 128;

// just a cached coefficient
const HUES_MULTIPLIER = 360 / NUM_TONES;

// used to control lines thickness
const GAP = SEGMENT_SIZE * Math.cos(DEG30);

const getRandomInt = (max: number) => {
    return Math.floor(Math.random() * max)
}

const paint = (canvasId: string) => {
    let gosperPoints = computeGosperCurveByRules();

    // We assign a randum hue to each one of our points
    let hues = Array(gosperPoints.length)
        .fill(0)
        .map((_) => getRandomInt(NUM_TONES));

    // Because we animate color transitions, and we want them to be random, but
    // not too flickery, we introduce some "inertia"
    let huesInertia = Array(gosperPoints.length)
        .fill(0)
        .map((_) => getRandomInt(9) - 4);

    const gosperCanvas = document.getElementById(
        canvasId,
    ) as HTMLCanvasElement;

    const ctx = prepareHiDPIContext(gosperCanvas);

    ctx.lineWidth = (GAP * 2) / 3;
    ctx.lineCap = "round"; // our line caps are round
    ctx.fillStyle = "#000"; // We set black background

    // This inner function will take care of redrawing the gosper curve
    const animate = () => {
        // We update our hue inertia
        huesInertia = huesInertia.map((i) => {
            const r = Math.random();
            if (r < 0.25) {
                return Math.max(i - 1, -4);
            } else if (r > 0.75) {
                return Math.min(i + 1, 4);
            } else {
                return i;
            }
        });

        // We update our hues
        hues = hues.map((hue, index) => {
            const inertia = huesInertia[index] as number;
            if (inertia >= 3) {
                return (hue + 1) % NUM_TONES;
            } else if (inertia <= -3) {
                return (hue - 1 + NUM_TONES) % NUM_TONES;
            } else {
                return hue;
            }
        });

        ctx.fillRect(0, 0, 768, 768); // Clearing the canvas using black

        let [currentX, currentY] = gosperPoints[0]!;
        gosperPoints.slice(1).forEach(([x, y], i) => {
            ctx.beginPath();
            ctx.moveTo(currentX, currentY);
            ctx.strokeStyle = `hsl(${hues[i]! * HUES_MULTIPLIER}, 95%, 75%)`;
            ctx.lineTo(x, y);
            ctx.stroke();
            [currentX, currentY] = [x, y];
        });

        requestAnimationFrame(animate);
    };

    animate();
}

// Finally, we call the paint function
paint('gosperCanvas')

That would be it! Now, there are still a couple of details that I didn't work on yet, but I would like to improve:

Making this animation to be responsive, to adapt when presented on small screens, or when we resize our window. In the meantime, you could check this very interesting article about the topic, from Joel Gibson.
Capturing the generated animations into GIF images and/or videos. I intend to write about this, let's see if I really do it in the end 😅.

Final rant

You might have noticed that I didn't use any library for the code shown in this article, and that I directly relied on HTMLCanvasElement functions. There are reasons for that.

Before I started writting this post, I wrote the same code, but using p5js... and I decided it wasn't good:

The provided TS typings are basically a patch, they don't properly cover all the details of the library.
It's designed to always work in the global scope, making a lot of assumptions about how we organize our code.
It doesn't play well with ES Modules.
It's very heavy. Using ESBuild, the generated bundle was above 950Kb (versus less than 2Kb without it), and tree shaking basically doesn't work with it, probably because of its incompatibility with ES Modules.

Some extras

You can check how I integrated the animation into this article by checking the MDX source code.
You can check the Astro component that I created to be embedded into this article.
The Gosper Curve that I've shown here is actually not unique. There are many other curves with similar but different shapes, and there's even a method to find new ones. You can learn more by reading this article from Fukuda, Shimizu and Nakamura.

The comeback of the Fediverse and the Old Web

Andrés Correa Casablanca — Mon, 05 Dec 2022 07:00:00 +0000

Twitter is not dead (yet), but we act like it was, while the old web wants to come back from its grave.

These past weeks have been pretty interesting to me, leaving the Birdsite aside. I started using Mastodon again, and through it I discovered how some people are devoting their time & energy to resucitate some buried ideas from the old Web, or Yesterweb.

What about Mastodon?

Mastodon is far from perfect, but it brings many interesting things with it. To be clear, I'm not talking about its federated/decentralized nature, although we can't deny it plays a small role in why the user experience is somehow special.

The ones who lived the early years of the Web might know what I'm talking about. It might feel like plain nostalgia, but it's not only that; it could be the feeling of experiencing the birth of something new, the thrill of discovery, but that still amounts to just a small part of it.

What else there is beyond nostalgia and the thrill of discovery? And, before going forward, why did I mention "nostalgia" in the first place if Mastodon didn't exist on the early days of the Web? I don't know how to distill the essence of things in few words, so I'll tell you about my experience instead.

A first taste

I have many less followers on Mastodon than in the Birdsite (40 vs 341), yet my activity has generated many more interactions than there. Not only that, among the users who decided to interact with me I counted: a co-discoverer of the Laniakea supercluster, one of the lead developers behind F#, the author of many important books on Java & JVM, plus many others. I'm literally a nobody, but this time there was no algorithm relying on relevance and engament metrics to decide what to present to each one of us.

On the other side, when I look at both timelines (Twitter's & Mastodon's), the later seems much more interesting to me, and it took me much less effort to "curate" who I follow so I can find interesting content.

Both differences I mentioned share the same root cause: Controversial takes are not over-rewarded by any algorithm, network centrality measures are not relied upon to prioritise the content of famous people over the others', and knowledge about users' preferences is not abused by recommender systems to keep us in our own echo chamber bubble.

The felled jungle

All of this makes it increadibly easier to discover new people and new ideas, to pop out of our bubble. But here's the catch: "randomness", and worsened searchability. That's where some people could say that Mastodon is playing the nostalgia card, because that's how the Web was before Google. Search engines were terrible, people relied on directories to find what they were looking for, but it was also incredibly common to discover new & interesting stuff every day. Sometimes it was rubish, but many others it was small gems, written by other humble people like us.

Today, this is sadly uncommon. Most people don't wander off beyond the borders of some big walled gardens, such as Google, Facebook, or Medium. Independent projects are still out there, they're not a rarity, but it is increasingly difficult to find them.

The Indie Web & Yesterweb

Preamble

While Google was a bliss for users during its first years of existence (same can be said about other hypercentralized services), it soon fell into the temptation of abusing its position of power (again, same can be said about others).

Nowadays, they can choose what people see and what not. They can punish their oponents, and favor whoever pours money into its advertising business. That's the very same thing they criticized in the academic article where the PageRank algorithm was publicly described for the first time.

Other actors in the scene have also been harmful, yet in different ways. Facebook lead the way by creating a walled garden with high fences. Most content created there was unavailable withour first registering as a user. Almost everyone was there, and the more people joined, the more difficult it was to avoid them it you wanted to be connected.

Others attempted similar things, like MySpace or Friendster, but in the end they failed to stay relevant. MySpace still lives, but with its decline, it has also dragged down the visibility and relevance of the content created in it. Friendster was directly wiped out, and with it, all its content gone.

Free/cheap website publishing platforms such as Geocities, Tripod, or Angelfire started to fade. Angelfire & Tripod still live, but they're a shadow of what they were, and Geocities was closed (luckily, there were some interesting archival efforts).

Blogging services such as Blogger/Blogspot started to fade in popularity after 2010. Tumblr had a great interest peak on 2013, and some time later it also steadily declined because if its highly controversial porn ban (recently revoked) plus (sometimes) requiring users to register to read its contents. Wordpress never saw steep surges or falls in its usage, because it was used by less hype-driven users, but we can also see how the trend of blogging going down caught with it as well.

The return of the blog

Going back to Twitter for a second, one of its founders (who founded Blogger as well) is also behind the creation of Medium. One might think that these platforms are in the hands of too few people, but that's not the point I want to bring up.

Medium is a blogging platform as the previous ones, although a bit special, because it mixes content created by amateur & professional writers... and big part of it is behind a paywall (where only a few authors get a small chunk of that money).

So, walled gardens again. I see nothing bad about making consumers pay for what they read, but I don't find it very compelling from the perspective of amateur writers who only want to reach some readers for the sake of sharing their ideas.

It isn't only that some of these platforms are imposing registration and/or payment requirements to access the content published in them; they are also trying to pull users away from the Web. For some reason, they are trying to make us use their mobile apps, instead of accessing their websites (Hello Reddit, please stop already, it's annoying as fuck).

It was monetization, though, what drove Hackernoon to leave Medium
and start its own publication platform. Same as Medium and other sites such as Dev.to, they honor the canonical tag (so we can publish in our own personal blog, and then re-publish there for greater visibility)... well, they did, not anymore.

In theory, Hackernoon's business model was based on charging for branded publications. As of today, they seem to have engaged in some shady crypto bullshit, and on stealing credit for others' work.

I'm actually quite angry at them, because I wasn't aware of their tactics and I paid the price with my previous article. Google still points to their page as the authoritative source when I search for my content, even though I manually fixed the canonical tag after the they shadily stripped it without my consent nor informing me.

Closing the circle, the return of the old

Some people are not happy with the current state of the web, and they're fighting back with the few resources they have, working on projects such as:

IndieWeb: they provide resources to help independent content creators to thrive outside the umbrella of big corporations' services.
Neocities: they try to bring back the atmosphere or our old Geocities (believe me, they achieved that goal).
Yesterweb: It's a community dedicated to nurture an ecosystem of old-style websites, but without the centralization aspect of Neocities. To give you an idea of how commited they are to this, they manage a webring, a links directory, and even a periodical zine like the ones that got popularized in the old Bulletin Board Systems.
Ooh.Directory: Its name says it all, but it's worth mentioning that, as a directory, it seems to do a better job than Yesterweb. Curiously enough, I learnt about it through Mastodon, much before it reached HackerNews' front page.
Blogroll: Another directory, but only with a few curated links that are appealing enough for its maintainer.
Sadgrl.Online's directory of directories, and... webrings directory! One can always add or an extra level of indirection, right? right?!

All of these sites are almost marginal if we consider the size of the Web and the size of the corporations operating in it... But there must be something of value in there when both Ooh.Directory and Yesterweb had reached the front page of notable news aggregators and spurred so much enthusiasm.

And there must be something in the zeitgeist when these sites are rising in popularity precisely when Musk is destroying Twitter's legacy, and when most people have finally realized that Web3 is not the way to gain sovereignty over the Web, but a carefully crafted global marketing campaing to promote crypto scams.

Barrel Exports considered harmful

Andrés Correa Casablanca — Sun, 04 Dec 2022 21:13:15 +0000

The barrel exports pattern is described in Basarat's book (TypeScript Deep Dive), and looks like this:

/// index.ts
export * from "./a";
export * from "./b";
// [···]
export * from "./z";

It's pretty common, most people use it, and I don't blame them: it's actually very useful. It saves us from having to remember too many import paths, and from having to explicitly export every symbol from our submodules inside the package.

So... yes, I admit it, the title of this article was pure clickbait. I don't think this pattern is always harmful, only sometimes. When and why, then?

It's almost always about coupling

Side effects at module load time

Although not likely to happen (because most of us have learnt the hard way that it's a bad idea to perform side effects at module load time), that's the simplest problematic case we can explain.

When we "need" to perform some side effects at load time (whatever the reason: singletons, resource initialisation, monkey patching...) in one of the submodules that is star-exported on our "barrel", we'll be unable to avoid them when importing anything that is completely unrelated from that same barrel.

As an example, if we just wanted to import a utility function, we might be triggering some resource initialisation that had no relation to that task (affecting performance for no good reason), or performing unintentional monkey patching, leading to potential hard-to-debug problems.

/// lib/index.ts
export * from "./a";
export * from "./b";
export * from "./c";

/// lib/a.ts
export const greet = (name = "stranger") => console.log(`Hello ${name}!`);
console.log("Loading module a"); // Side effect

/// lib/b.ts
export const square = (x: number) => x * x;
console.log("Loading module b"); // Side effect

/// lib/c.ts
export const cube = (x: number) => x * x * x;
console.log("Loading module c"); // Side effect

/// main.ts
import { greet } from "./lib";
greet("World");

/// main.js's output:
// --------------------------------------------------------------------
// Loading module a
// Loading module b // <- We didn't want this
// Loading module c // <- We didn't want this
// Hello World!

"Type effects" at type checking time

You might be thinking that the previous case is not really that problematic, because no side effects are performed in most code you see every day (consider yourself lucky!), but that's not where the story ends.

First of all, I'm probably going to abuse the term "type effect" on the following lines (I suspect this term already has some different meaning attached, but given that I don't have a better word, I'll stick to it for now), I ask for your forgiveness in advance... and if you know of a better way to describe it, please feel free to reach out to me and tell me!

While most of the times types have to be explicitly imported in order to have any effect at typechecking time, there are some interesting cases that don't work like that, module augmentation and ambient modules (both rely on the same construct "declare module"):

declare module "some_external_module" {
    // typing stuff in here
}

What this does is to overwrite or augment the types exposed by the pointed module, and can be used (for example) when relying on autogenerated code. One interesting case of this is GraphQL to TypeScript code generation, and how this is integrated with Mercurius library ).

It's not a coincidence that I mention autogenerated code and Mercurius: I actually found myself having to deal with a barrel that was exporting one of these autogenerated files. That barrel was inside a supposedly generic types library
(in the context of a monorepo)... and because of it, it was "leaking" types from one federated "gateway" into another one that was supposed to expose a completely unrelated object graph. Untangling that wasn't fun.

Generated code can also suffer

When some of our modules only contain types (but no runtime symbols), we probably want that the generated code doesn't really perform a runtime import because it's totally unnecessary.

If we use import type instead of just import, or our import statement is explicitly importing something that it's just a type without a runtime counterpart (basically, not enum nor class), then we can expect tsc to optimize away that import from the generated JS code.

Sadly, tsc is not that smart when it comes to "start-exports", and we won't be able to optimize it away. It is true that the compiler can improve in the future, but I wouldn't hold my breath for now.

Indirect problems

Circular references

The barrel pattern makes it easier to introduce accidental circular references. It is true that the fault in this case doesn't really fall on the pattern, but I prefer to not have footguns at my disposal.

I've seen too many times how some people decide to import "symbols" from "sibling" modules through the index.ts file instead of directly addressing the source modules; the error here is not the barrel itself (because it's thought to be consumed from "outside"), but consuming "internal" code as if it was an external library.

Some dev tools become slower

When we use barrel exports in our code, static analysis becomes more complex. Some tools have explicit support for it, but others strugle to provide good results, or to give them in an acceptable amount of time.

It's worth noticing that not only static analysis tools can perform worse when in presence of barrel exports, test runners such as Jest can also see their performance badly hurt, affecting our productivity, and the associated costs of our CI pipelines.

Conclusion

Barrel exports are a great tool, if used correctly, but there's great risk of abusing them. As with any other technique, we should be aware of its associated trade-offs and under which circumstances they can become problematic, so we can consider these factors when deciding whether to use them or not.

DEV Community: Andrés Correa Casablanca

How to create a React components ESM+CJS library

Introduction

Project Setup

The Package Manager

Dependencies

TypeScript

Rollup

Creating a dummy React component

Building our dummy React component

Visualizing our dummy React component

Vitest

The package.json File

Validating our package

Publint

AreTheTypesWrong

Conclusion

A better way to use Dotenv

Preface

Introduction

Removing clutter

Final tweaks

Conclusion

Credentials Leak with Knex

Preamble

The Problem

Mitigations

1. Use toQuery for logging

2. Enable redaction in your logger

3. Do the right thing: fix Knex

4. Don't wait for Knex to fix it

5. Apply a combination of the previous strategies

Challenges in Open Source Maintenance

Conclusion

References

Exploring the LZ77 Compression Algorithm

A bit of history

Some terms & concepts

The algorithm

Compression examples

A decompression example

Some observations

The pattern matching algorithm

Parametrization & Encoding

Conclusion

Animated Gosper Curves in JS

L-Systems

How we structure code

The Production function

Generating points

Extra tweaks for our generated points

Preparing the canvas and dealing with HiDPI

Painting!

Final rant

Some extras

The comeback of the Fediverse and the Old Web

What about Mastodon?

A first taste

The felled jungle

The Indie Web & Yesterweb

Preamble

The return of the blog

Closing the circle, the return of the old

Barrel Exports considered harmful

It's almost always about coupling

Side effects at module load time

"Type effects" at type checking time

Generated code can also suffer

Indirect problems

Circular references

Some dev tools become slower

Conclusion

The `package.json` File

1. Use `toQuery` for logging