DEV Community: Camille Hodoul

Illustrate your comments

Camille Hodoul — Fri, 07 Jul 2023 12:56:48 +0000

I remember the day structural sharing in data structures "clicked" for me. It was illustrated by a drawing of boxes and arrows.

An image is a powerful way to communicate relationships between objects, distance, size, time, all of which are relevant in a program.

Comments in code are communication tools: they explain an abstraction, an implementation, or give context to complex technical problems. They are the documentation closest to the code, where we need it most.

We should use images in comments more often.

We should not, however, lose the convenience of plain text: it can be checked in version control, shows up in diffs and works out-of-the-box in any text editor.

Nor should we entirely remove sentences, because phrases can be searched for in a search engine and can be read by accessibility tools or people who don't like visual communication.

Text-based diagrams are, to me, a good compromise. Look at this example:


            No data duplication

┌───────────┐                 ┌───────────┐
│ Version A │                 │ Version B │
└───────────┘                 └───────────┘
      │                             │
      ├─────────────┬───────────────┘
      │             │
┌─────▼────┐  ┌─────▼────┐
│ Data A 1 │  │ Data A 2 │
└──────────┘  └──────────┘
              ──────────────
           With data duplication

┌───────────┐                 ┌───────────┐
│ Version A │                 │ Version B │
└───────────┘                 └───────────┘
      │                             │
      ├─────────────┐               ├─────────────┐
      │             │               │             │
┌─────▼────┐  ┌─────▼────┐    ┌─────▼────┐  ┌─────▼────┐
│ Data A 1 │  │ Data A 2 │    │ Data B 1 │  │ Data B 2 │
└──────────┘  └──────────┘    └──────────┘  └──────────┘

I generated this diagram with monodraw a paid (but not expensive) macos app, which can export text-based drawings with options to easily paste into common comment formats.

Communicating ideas with pictures seems to be obvious everywhere, from art to education and marketing. Why not use it in comments?

Meaning is thus conveyed not only by what's depicted, but through structure:
the size, shape, placement, and relationship of components - what they're next to, and what they're not, matters.
Unflattening, Nick Sousanis

Original article on my blog

Update to the latest version of many npm dependencies at once

Camille Hodoul — Mon, 17 May 2021 08:49:22 +0000

I couldn't find a npm command to update a subset of my dependencies to their latest version based on a name pattern, so here's a one-liner to do it with pipes and awk (to be modified for your needs).

In this example, I want to update all the dependencies containing the string "babel".

npm outdated |awk 'BEGIN{OFS="@"} $1 ~ /babel/ { print $1, "latest" }'| xargs npm install

Explanation of each command

npm outdated lists your outdated dependencies.

awk:

BEGIN{OFS="@"} sets @ as the output field separator (will be used by print)
$1 ~ /babel/ will match the lines containing "babel" in their first column
{ print $1, "latest" } will output each selected lines concatenated with "latest" (using "@" as the OFS)

xargs npm install will give the output of awk as input arguments to npm install, like so : npm install dependency1@latest dependency2@latest ...

Tweak it

The beauty of the command line: you could tweak this for different dependency managers, such as Composer for PHP.

Ship modern JavaScript with Rollup

Camille Hodoul — Sat, 23 Jan 2021 11:22:15 +0000

This post is a few months old, but it still gets a few hits on my blog everyday so maybe it can help people.

Using ES2017 (and newer) syntax in our codebase is one thing, sending it to our users is another.

Chances are, if we have to support IE11 and/or use @babel/preset-env and/or ship a single bundle (even with code-splitting), we're probably sending ES5 to everybody, even if their browser is perfectly capable of understanding classes or async/await.

This is suboptimal, because we're sending more code than needed and, long term, we'll miss out on some "free" performance gains.

You can read more about the peformance of ES6 features compared to their ES5 transpiled versions on the Six Speed project. The status quo isn't set in stone, as browser vendors regularly improve the performance of new syntax (see this and this).

Moreover, even if all our preset-env targets are up-to-date, Babel may still be transpiling some syntax because of edge-cases. Besides, our dependencies are probably transpiled to ES5 anyway.

A better thing to do would be to ship transpiled ES5 to legacy browsers and as much ES2017 as possible to modern browsers.

The steps we need to take to achieve this are:

create 2 distinct bundles
1. use preset-modules instead of preset-env for our modern build,
2. configure terser to output ES2017
load each bundle in the correct browsers.

You can see a working example in this repo.

Bundling

If you're using Rollup, you can already output ES modules or a fallback out-of-the-box.

To create each build with a different config however, we have to add a bit of configuration.

One way to do this could be to use an environment variable ROLLUP_BUILD_TYPE, in npm scripts.

Rollup and Terser configuration

// package.json
{
    /** ... **/
    "scripts": {
        "build": "npm-run-all --parallel \"build:*\"",
        "build:legacy": "env ROLLUP_BUILD_TYPE=\"legacy\" rollup -c",
        "build:modern": "env ROLLUP_BUILD_TYPE=\"modern\" rollup -c"
    },
    "devDependencies" : {
        "npm-run-all": "...",
        "rollup": "..."
    }
    /** ... **/
}

// rollup.config.js
export default () => {
    const buildType = typeof process.env.ROLLUP_BUILD_TYPE !== "undefined" ? process.env.ROLLUP_BUILD_TYPE : "modern";
    return {
        input: ["./src/index.jsx"],
        output: buildType === "modern" ?
        {
            dir: `/public/js/system/`,
            format: "system"
        } :
        {
            dir: `/public/js/esm/`,
            format: "esm"
        },
        plugins: [
            /** ... **/
            babel({
                /**
                * Uncomment to ignore node_modules. This will accelerate yur build,
                * but prevent you from using modern syntax in your dependencies
                */
                // exclude: "node_modules/**"
            }),
            terser({
                compress: {
                    unused: false,
                    collapse_vars: false
                },
                sourcemap: true,
                ecma: buildType === "legacy" ? 5 : 2017,
                safari10: true,
            })
            /** ... **/
        ]
    };
};

Notice how we didn't exclude node_modules/ from Babel. This will let us optimally transpile our dependencies if they export modern syntax. More on that later.

Terser is a minifier which needs to be told which version of the language we are using.

Babel configuration

Babel 7 can be configured with a function instead of a static object. We can take advantage of this feature in order to write conditional and annotated code.

Remember that our goal is to transpile as little as possible. In this example, I include a preset for react as well as plugins for ES2020 features, that will still be used for a while even in modern browsers.

// in babel.config.cjs
module.exports = function(api) {
    api.cache.invalidate(() => [
        process.env.NODE_ENV, 
        process.env.ROLLUP_BUILD_TYPE
    ].join("-"));
    const environment = api.env();
    const modern = process.env.ROLLUP_BUILD_TYPE === "modern";
    /**
     * Will be used for the legacy build
     */
    const presetEnv = [
        "@babel/preset-env",
        {
            modules: false,
            targets: {
                browsers: [">0.25%", "not op_mini all"],
            },
        },
    ];
    /**
     * Will be used for the modern build
     */
    const presetModule = [
        "@babel/preset-modules",
        {
            loose: true,
        },
    ];
    const alwaysUsedPresets = ["@babel/preset-react"];

    const alwaysUsedPlugins = [
        "@babel/plugin-syntax-dynamic-import",
        "@babel/plugin-proposal-optional-chaining",
        "@babel/plugin-proposal-nullish-coalescing-operator",
    ];
    /**
     * Only loaded in the legacy build
     */
    const legacyPlugins = [
        "@babel/plugin-proposal-object-rest-spread"
    ];

    const productionConfig =
        modern === "modern"
            ? {
                  /**
                   * Modern build
                   */
                  presets: [
                      presetModule, 
                      ...alwaysUsedPresets
                    ],
                  plugins: [...alwaysUsedPlugins],
              }
            : {
                  /**
                   * Legacy build
                   */
                  presets: [
                      presetEnv,
                      ...alwaysUsedPresets
                    ],
                  plugins: [
                      ...alwaysUsedPlugins,
                      ...legacyPlugins
                    ],
              };

    const developmentConfig =
        modern === "modern"
            ? {
                  /**
                   * Modern build
                   */
                  presets: [
                      presetModule,
                      ...alwaysUsedPresets
                    ],
                  plugins: [
                      ...alwaysUsedPlugins
                    ],
              }
            : {
                  /**
                   * Legacy build
                   */
                  presets: [
                      presetEnv, 
                      ...alwaysUsedPresets
                    ],
                  plugins: [
                      ...alwaysUsedPlugins, 
                      ...legacyPlugins
                      ],
              };
    return {
        env: {
            production: productionConfig,
            development: developmentConfig,
            test: {
                /**
                 * Chances are tests will run in node
                **/
                presets: [
                    "@babel/preset-env",
                    ...alwaysUsedPresets
                ],
                plugins: [
                    ...alwaysUsedPlugins, 
                    ...legacyPlugins
                ],
            },
        },
    };
};

We didn't use the .mjs alternative because it only works when babel is loaded asynchonously.

preset-modules will transpile less code than preset-env. Read the documentation to understand why.

Now that we have our bundles, we need a way to load them in their targets.

The module/nomodule pattern

This technique (from 2017) utilizes <script type="module"> support as a breakpoint between legacy browsers (mostly IE nowadays) and "evergreen" browsers (Chrome, Firefox, Safari, Edge, ...).

This gives us a way to send a modern bundle to modern browsers, and a legacy bundle to old browsers.

For a long time, things weren't as simple as some browsers did support modules, but didn't support import(). This has now been delt wih though, and <script type="module"> can be safely used.

The pattern:

<script type="module">
import("/js/esm/main.js").then((module) => {
    /** ... **/
});
</script>
<script nomodule src="/js/legacy/main.js"></script>

The future

While this post is about ES2017, the techniques explained here can be updated later to use newer versions of the language, as long as browser support keeps up.

One important limitation to note is that most third-party libraries still don't export untranspiled code. Unfortunately, this means the majority of the code you ship will still be ES5 for now.

However, if you stop excluding node_modules/ from your Babel config, and more people do the same, things might improve soon enough.

Pre-commit, pre-push or pre-merge ?

Camille Hodoul — Mon, 23 Nov 2020 07:03:19 +0000

On my codebases, I may want to run:

a linter,
static analysis,
tests.

While I can run them manually, I find it easier for everyone involved if it happens on its own.

It's about getting feedback as soon as possible without breaking the flow of work.

When and where should they run?

on save: runs every time I save a file on disk,
pre-commit: runs before I commit my changes,
pre-push: runs before I push mu changes to a remote,
pre-merge: runs on CI when I do a merge request (or "pull-request").

	When	How often	Where
on save	very early	very often	locally
pre-commit	early	often	locally
pre-push	late	sometimes	locally
pre-merge	later	rarely	CI

My advice is that the correct answer for a given task will depend on the "lost" time perceived by the developer. Perceived lost time ~= runtime + time to correct the code, if the task is blocking.

The slower the task, the later and less often it should run.

If something feels slow, it will get in the way and people (me included) will work around it.

If I have to wait a long time before knowing if my tests pass or not, I probably will switch to another task and waste time.

If the task itself can correct the code instead of rejecting the commit (e.g. linters), make it so.

Here is my rule of thumb, to be adapted to your needs:

Runtime	Earliest it can run
less than 100 milliseconds	on save
less than 3 seconds	pre-commit
less than 10 seconds	pre-push
above 10 seconds	pre-merge
above 5 minutes	split into sub-tasks

Fine-tuning

To state the obvious: every team, every codebase, every workflow is different. What works for one won't necessarily work for others.

The tools we use give us even more options to choose when to run what, according to our preferences:

IDE Integrations,
manual or conditional tasks on CI,
third party tools integrating with our version control system

For instance, I choose to run linters on pre-commit, not on save, even if it's fast, because I find it distracting to have code jumping around on the screen.

I run unit tests pre-merge but trigger end-to-end tests manually because they are very slow, and would delay feedback too much.

Experiment and gather feedback.

External links

Bitbucket Pipeline for PHP, MySQL & Elasticsearch

Camille Hodoul — Sun, 21 Jul 2019 10:58:19 +0000

If your application has dependencies that are not covered by Bitbucket Pipelines built-in services, or these dependencies are more complex that simply checking that a container is running, docker-compose can be a helpful tool.

In my case, I want to setup CI on a Symfony + MySQL + Elasticsearch (which is not available as a pipeline service) application. I just want to run phpunit tests, so I don't need an HTTP server. The dependencies are as follows:

MySQL has no dependencies,
Elasticsearch has no dependencies,
PHP depends on MySQL and Elasticsearch for bootstraping and running tests.

These look pretty straightforward and can be expressed in docker-compose. However as always, the devil is in the details: not only should the Elasticsearch container be up, the service should be ready for the tests to run correctly.

Pipelines use docker containers to run our application and let us use whatever image we need. We will use this to our advantage and use an image with docker-compose available.

Folder structure

We will create a ci/ folder which will contain the files we will use in our CI environment.

.env.ci contains environment variables if you use DotEnv
phpunit.xml contains configuration for phpunit
php-ini-overrides.ini is self-explanatory
docker-compose.yml describes the containers we need
setup-and-test.sh will bootstrap our application and run tests against it.

docker-compose.yml

version: "3.1"
services:

    mysql:
      image: mysql:5.7
      container_name: my-db-server
      working_dir: /application
      volumes:
        - ../:/application
      environment:
        - MYSQL_ROOT_PASSWORD=root
        - MYSQL_DATABASE=my-db
      ports:
        - "8002:3306"

    elasticsearch:
      image: docker.elastic.co/elasticsearch/elasticsearch:6.8.1
      container_name: elasticsearch
      working_dir: /application
      environment:
          - xpack.security.enabled=false
          - "transport.host=localhost"
          - "bootstrap.system_call_filter=false"
      volumes:
          - ../:/application
      ports:
          - "9200:9200"
      tty: true

    php-fpm:
      image: chodoul1egal2/php72-mysql
      working_dir: /application
      volumes:
        - ../:/application
        - ./php-ini-overrides.ini:/etc/php/7.2/fpm/conf.d/99-overrides.ini
      depends_on:
        - elasticsearch
        - mysql

A few things to note:

The docker image I use for php-fpm is a custom one which includes the modules I need for my application (notably php-mysql). You might need a different one.
No volumes are needed as nothing needs to be persisted in a CI context

setup-and-test.sh

#!/bin/bash

# Make sure CI stops as soon as something wrong happens
set -eu

# Inspiration for this script:
# https://github.com/elastic/elasticsearch-py/issues/778#issuecomment-384389668

host="http://elasticsearch:9200"


until $(curl --output /dev/null --silent --head --fail "$host"); do
    printf '.'
    sleep 1
done

# Wait for ES to be available
response=$(curl $host)

until [ "$response" = "200" ]; do
    response=$(curl --write-out %{http_code} --silent --output /dev/null "$host")
    echo "Elasticsearch is unavailable - sleeping"
    sleep 1
done


# Wait for ES to be ready
health="$(curl -fsSL "$host/_cat/health?h=status")"
health="$(echo "$health" | sed -r 's/^[[:space:]]+|[[:space:]]+$//g')"

until [ "$health" = 'green' ]; do
    health="$(curl -fsSL "$host/_cat/health?h=status")"
    health="$(echo "$health" | sed -r 's/^[[:space:]]+|[[:space:]]+$//g')"
    echo "Elasticsearch is unavailable - sleeping"
    sleep 1
done

echo "ES is up"

# Install dependencies
composer install --no-interaction

echo "bootstrapping"
#
# YOUR BOOTSTRAPPING LOGIC HERE
#

# php bin/console ...
# php bin/console fos:elastica:populate
echo "done"
echo "tests"
vendor/bin/simple-phpunit --stop-on-failure 
echo "over"

Pipelines configuration

Finally, the bitbucket-pipelines.yml file:

image: tiangolo/docker-with-compose

definitions:
  services: 
    docker:
      memory: 2048

pipelines:
  default:
    - step:
        size: 2x
        caches:
          - composer
          - docker
        script:
          - cp ci/.env.ci ./.env
          - cp ci/phpunit.ci ./phpunit.xml
          - docker-compose -f ci/docker-compose.yml up -d && sleep 5
          - docker-compose -f ci/docker-compose.yml exec -T php-fpm ci/setup-and-test.sh 
        services: 
          - docker

the sleep 5 is an ugly hack to avoid timing problems related to the - d flag on docker-compose. Removing -d implies having a lot of noise in your pipeline logs. If you have a better solution, please share!
Elasticsearch will use a lot memory, so we use size: 2x and give the docker service enough memory.

If you have ideas to improve this, I would love to hear about it!

This was originally posted on my blog.

UTF-8 csv fix for Excel

Camille Hodoul — Tue, 16 Jul 2019 20:36:48 +0000

Using utf-8 for your CSV files sounds obvious and works well in most software processing them.

Most software except Micrososft's Excel, which, considering its number of users, can sadden your day.

Excel seems to assume windows-1252 unless a Byte order mark is provided.

To fix this without asking your users to navigate a maze of hidden menus, you can add a BOM to a string before saving it to a file or triggering a download.

In JavaScript:

let csvString = ["a,b,c", "1,2,3"].join("\n");
csvString = "\ufeff" + csvString;

or in PHP:

$csvString = implode("\n", ["a,b,c", "1,2,3"]);
$csvString = chr(0xEF) . chr(0xBB) . chr(0xBF) . $csvString;

If you look at the hex dump, you can check for the presence of this byte sequence at the very beginning: EF BB BF.