DEV Community: Kev

Dev Containers - Part 5: Multiple Projects & Shared Container Configuration

Kev — Fri, 21 Jun 2024 06:46:15 +0000

Welcome to the fifth guide in the Dev Container series:

Dev Containers - Why You Need Them
Part 1: Quick Start - Basic Setup and Usage
Part 2: Image, Features, Workspace, Environment Variables
Part 3: Full Stack Dev - Docker Compose & Database
Part 4: Remote Dev - Develop on a Remote Docker Host
Part 5: Multiple Projects & Shared Container Configuration

To get started, you can clone my demo project using the following command:



git clone -b part-5-shared-configure-for-multiple-projects https://github.com/graezykev/dev-container.git

Introduction

In Part 3, we learned about using Docker Compose in Dev Containers to build containers for Node.js applications and databases.

Currently, you can only connect to one container per Visual Studio Code window. But what if you have multiple projects using different tech stacks like Node.js, Python, Go, etc., and need to create Dev Containers for each?

One option is to place a .devcontainer folder under each project:



.
└── path
 └── to
    ├── project-a-node-js
    │   └── .devcontainer
    │       ├── docker-compose.yml
    │       ├── ...
    │       └── devcontainer.json
    ├── project-b-node-js
    │   └── .devcontainer
    │       ├── ...
    │       └── devcontainer.json
    ├── project-c-python
    │   └── .devcontainer
    │       ├── ...
    │       └── devcontainer.json
    ├── project-d-go-lang
    │   └── .devcontainer
    │       ├── ...
    │       └── devcontainer.json
    └── project-...

If these applications need to share the same database, you must ensure they all use the same database container in their docker-compose.yml and the same volume:



services:

  app-name-...
    ...

  postgres:
    image: postgres:latest
    ...

...

volumes:
  postgres-data:

However, this can result in overlapping configurations across multiple projects, making maintenance tedious. A better approach is to share a common docker-compose.yml:



.
└── path
 └── to
  └── dev-container
      │
      ├── .devcontainer
      │   │
      │   ├── .env
      │   ├── docker-compose.yml
      │   ├── ...
      │   │
      │   ├── project-a-node-js
      │   │   └── devcontainer.json
      │   │
      │   ├── project-b-node-js
      │   │   └── devcontainer.json
      │   │
      │   ├── project-c-python
      │   │   └── devcontainer.json
      │   │
      │   ├── project-d-go-lang
      │   │   └── devcontainer.json
      │   │
      │   └── project-e-...
      │       └── devcontainer.json
      │
      ├── project-a-node-js
      │       └── index.js
      │
      ├── project-b-node-js
      │
      ├── project-c-python
      │       └── hello.py
      │
      ├── project-d-go-lang
      └── project-e-...

All projects and the .devcontainer folder share a common root-level folder, with each project having its own configuration folder under .devcontainer.

This setup allows you to define multiple Dev Containers (and a container for the database) in a common docker-compose.yml, and create a devcontainer.json for each project to reference the shared docker-compose.yml. This approach also helps manage each project's features and lifecycle scripts, avoiding configuration conflicts.

Common Docker Compose File

First, create a common docker-compose.yml inside the root-level .devcontainer:



services:
  project-a-node-js:
    image: graezykev/dev-container-base-image:latest
    volumes:
 - ..:/workspaces:cached
    ports:
 - 8001:8000
    depends_on:
 - postgres
    command: /bin/zsh -c "while sleep 1000; do :; done"

  project-b-node-js:
    ...
    volumes:
 - ..:/workspaces:cached
    ports:
 - 8002:8000

  project-c-python:
    ...

  project-d-go-lang:
    ...

  project-e-...

  postgres:
    ...

volumes:
  postgres-data:

See the complete file in my demo.

All projects share the same database container postgres.

We have learned most of the concepts in Part 3 but there are a few things we need to pay attention to.

Ports

Notice the port mappings: 8001:8000, 8002:8000, etc. Each project uses port 8000 within its own Dev Container, mapped to different ports on the host machine. This setup avoids port conflicts and allows access to each project's server via distinct ports (e.g., 8001, 8002, etc.).

See demo preveiw below.

Volumes & Workspace

Using ..:/workspaces for the volumes sections mounts the entire root-level folder (dev-container) on the host machine to /workspaces in the containers.

In each devcontainer.json, specify the project folder within the workspace:



"workspaceFolder": "/workspaces/project-b-node-js"

Service

In each devcontainer.json, reference the Docker Compose file docker-compose.yml and specify the service name:



{
  "name": "Dev Container",
  "dockerComposeFile": [
    "../docker-compose.yml"
  ],
  "service": "project-a-node-js",
  "shutdownAction": "none",
  "workspaceFolder": "/workspaces/project-a-node-js"
}



{
  "service": "project-b-node-js",
  "workspaceFolder": "/workspaces/project-b-node-js"
}



{
  "service": "project-c-python",
  "workspaceFolder": "/workspaces/project-c-python"
}

The "shutdownAction": "none" option will leave the containers running when VS Code closes -- which prevents you from accidentally shutting down both containers by closing one window (or switching containers).

Build and Switch Dev Containers

In VS Code, open the root-level folder (dev-container in my demo). Run Dev Containers: Reopen in Container from the Command Palette and select the project to build.

This triggers the Dev Container build for the selected project.

To switch between projects, use Dev Containers: Switch Container from the Command Palette and select the desired project.

After each Dev Container is built, we can use this Switch Container command to switch between projects, the current VS Code window will reload each time and connect to the selected Dev Container.

Demo Preview

In my demo, I have three projects: two Node.js projects and one Python project, each starts an HTTP server with a web page.

When visited, the projects write "visiting records" to the shared database and display all records on the page.

And remember the port mapping we mentioned before?

Dev Containers - Part 4: Remote Dev - Develop on a Remote Docker Host

Kev — Fri, 21 Jun 2024 06:46:04 +0000

Welcome to the fourth guide in the Dev Container series:

Dev Containers - Why You Need Them
Part 1: Quick Start - Basic Setup and Usage
Part 2: Image, Features, Workspace, Environment Variables
Part 3: Full Stack Dev - Docker Compose & Database
Part 4: Remote Dev - Develop on a Remote Docker Host
Part 5: Multiple Projects & Shared Container Configuration

To get started, you can clone my demo project using the following command:



git clone -b part-4-remote-dev https://github.com/graezykev/dev-container.git

Introduction

Here comes my favourite part: Remote Development, or more fashionably, cloud development.

In the previous parts, we focused on standalone PCs for building dev containers and development. In this part, I'll show you how to develop when you're not at the office or even without a working PC at your disposal.

Here's how the VS Code team explains the remote development architecture:

On the left, VS Code can be a native client installed on your development machine (desktop, laptop, tablet, etc.) or a VS Code web accessed via any web browser. This flexibility means you can work anywhere with an internet connection and a web browser.

There are two primary ways of remote development:

Connect to remote and virtual machines with Visual Studio Code via SSH.
Connect to a remote machine via a secure tunnel, without configuring SSH.

I'll guide you through the second method, which is simpler and avoids the lengthy, laborious tasks of installing and configuring SSH servers and clients, enabling us to work anywhere without a PC.

I won't delve into the concepts of Visual Studio Code Server, although we'll be leveraging its capabilities.

Developing with Remote Tunnels

0. Prepare

For remote development, you need at least two devices:

Remote Machine: This serves the code and dev environment. It can be your office desktop/laptop, a Virtual Machine, or a cloud machine (e.g., AWS EC2). In this guide, I'll use an Ubuntu Linux server.
Dev Machine: This can be another desktop, laptop, iPad, Surface Tablet, Android Tablet, or even a mobile phone (although using a phone is only practical for urgent scenarios).

Additionally, you'll need a GitHub/Microsoft account. This account acts as a bridge to connect the dev machine to the remote machine via Visual Studio Code Server, ensuring secure access.

1. Install Docker (Remote Machine)

Install Docker on the remote machine. You don't need it on the dev machine. Instructions for installing Docker are in Part 1. It's straightforward, especially on Mac/Windows PCs, and even on headless systems like an Ubuntu Linux server.

Interestingly, you can skip installing Docker for now, as you can install it at the end of step 4 (Connect to Remote Machine) using the terminal from the dev machine.

2. Install Code CLI (Remote Machine)

On the remote machine, download and uncompress the VS Code CLI from https://code.visualstudio.com/download based on your operating system.

For Ubuntu Linux, use these commands (replace cli-alpine-x64 with cli-alpine-arm64 or cli-linux-armhf as needed):



curl -Lk 'https://code.visualstudio.com/sha/download?build=stable&os=cli-alpine-x64' --output vscode_cli.tar.gz && \
tar -xf vscode_cli.tar.gz

You can also try my fully automatic shell script to install the CLI: https://gist.github.com/graezykev/b7c981c4966d49e580cf1fddc0c52559

After uncompressing, you'll find the executable binary file:

3. Create Secure Tunnel (Remote Machine)

Run the binary file to create a secure tunnel:



./code tunnel --accept-server-license-terms

Follow the steps to set up the tunnel:

Use the GitHub/Microsoft account mentioned earlier.

The CLI will output a vscode.dev URL tied to this remote machine, such as https://vscode.dev/tunnel/<machine_name>/<folder_name>.

4. Connect to Remote Machine

Now you can connect to the remote machine using the dev machine. You have two options:

Directly visit the vscode.dev URL from a web browser.

Use a VS Code client.
1. Install the Remote Development extension pack (search remote dev).
2. Use VS Code's Command Palette and choose Connect to Tunnel.
3. Login and verify with the GitHub/Microsoft account you used to create the tunnel.
4. Choose the remote machine name created in the previous step.
5. Wait for the connection and check the status.

Here are the steps in action:

Either option you choose, when you try to connect for the first time, you'll be prompted to log into your Github/Microsoft account at a https://github.com/login/oauth/authorize... URL.

Once connected, you’ll see the remote machine's name (e.g., my-remote-ubuntu) at the bottom left of VS Code. You now have terminal access to the remote machine.

As mentioned in step 1, you can install Docker from this terminal. In the example below, Docker is installed from the VS Code client's terminal following this guide. You can even do it from VS Code web.

Now, we’ve established a connection to the remote machine. In the next steps, we’ll clone the code and build a dev container on it to enjoy a unified development environment.

5. Clone Your Project

Clone your project (which has a .devcontainer folder and its configurations). You can use my project for testing:



git clone -b part-4-remote-dev https://github.com/graezykev/dev-container.git

Build the Dev Container

At the time of writing, building a dev container via VS Code web is not yet supported. You need to do this using a native VS Code client.

The dev container isn’t built yet, so we're still unable to use the container environment. Follow these steps:

Use VS Code's Open Folder to open the project.
Use VS Code's Command Palette and choose Reopen in Container.
Wait for the build to complete.

Here are the steps:

Once the container is built, enjoy the dev environment, ports mapping, VS Code extensions, settings, etc. Lifecycle commands in devcontainer.json will also run in the container after it’s built.

The Node.js program listens on port 8000 on the remote machine’s dev container, and the port is mapped directly to your dev machine, so you can visit it via localhost:8000.

Moreover, the VS Code extensions specified in devcontainer.json are installed on the remote machine’s dev container, not on your dev machine.

Furthermore, although the remote machine doesn’t have Node.js or Python installed, you can still enjoy the pre-installed software in the dev container built on your remote machine.

Work Anywhere

Now you’re able to work anywhere with internet access!

Use a spare laptop (with Windows, Mac, or Linux OS) to install VS Code, login with the GitHub/Microsoft account, and connect to the remote machine via "Remote Explorer" (Don't forget to install the Remote Development extension pack).

If you don’t want to install anything, use a tablet device like an iPad to visit the vscode.dev URL and start developing.

In urgent scenarios, you can even use a mobile phone to visit the vscode.dev URL, make code changes, or run commands on the remote machine.

Dev Containers - Part 3: Full Stack Dev - Docker Compose & Database

Kev — Fri, 21 Jun 2024 06:45:51 +0000

Welcome to the third guide in the Dev Container series:

Dev Containers - Why You Need Them
Part 1: Quick Start - Basic Setup and Usage
Part 2: Image, Features, Workspace, Environment Variables
Part 3: Full Stack Dev - Docker Compose & Database
Part 4: Remote Dev - Develop on a Remote Docker Host
Part 5: Multiple Projects & Shared Container Configuration

To get started, you can clone my demo project using the following command:



git clone -b part-3-use-docker-compose-and-db https://github.com/graezykev/dev-container.git

Alternatively, the faster approach is to use GitHub's Codespaces to run the demo (switch to branch part-3-use-docker-compose-and-db):

Introduction

Imagine you're developing server applications relying on both Node.js and PostgreSQL. You might initially use installation commands in a Dockerfile (as discussed in Part 1) or leverage Dev Container Features (covered in Part 2) to install PostgreSQL within your container.

However, managing PostgreSQL this way can lead to unexpected behaviours if not handled correctly.

For instance, if your database startup script exits unexpectedly, the container may stop unless it includes a command to keep running, such as starting a shell or a daemon process.

Environment Isolation

A more robust solution for development involves using Docker Compose to manage both your application container and your PostgreSQL service in separate containers. It's generally better to handle services like databases with separate containers or services, especially in production environments.

Using Docker Compose, you can manage two containers: one for development and the other for the database.

Install only the PostgreSQL client on your development container to connect to the PostgreSQL server running in the other container.

I. Compose Configuration

0. docker-compose.yml

Creating a Docker Compose configuration file docker-compose.yml is crucial for using multiple containers:



touch .devcontainer/docker-compose.yml



# docker-compose.yml
services:
  app:
    image: docker.io/your-user-name/your-image-name
    volumes:
      - ..:/workspaces/dev-container:cached
    ports:
      - 8001:8000
    environment:
      - POSTGRES_HOST=${POSTGRES_HOST}
      - POSTGRES_USER=${POSTGRES_USER}
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
      - POSTGRES_DB=${POSTGRES_DB}
    depends_on:
      - postgres

  postgres:
    image: postgres:latest
    restart: always
    environment:
      - POSTGRES_USER=${POSTGRES_USER}
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
      - POSTGRES_DB=${POSTGRES_DB}
    volumes:
      - postgres-data:/var/lib/postgresql/data  # Named volume for persisting PostgreSQL data
    ports:
      - 5432:5432

volumes:
  postgres-data:  # Define a named volume for PostgreSQL data persistence

Let's break down some basic concepts in this docker-compose.yml. You can also jump straight to the the next step of Entry Point and come back to the explanations below.

1. Define `services`

We define 2 services here, each as a container: app and postgres.

This setup allows extending the project to include multiple other services (containers) like app2, app3, redis, etc.

2. Specify `image`

Specify the image used in devcontainer.json in this configuration file to initiate the app container.

For the database container, use the official image postgres:latest.

3. Define `volumes`

Volumes are pretty similar to the workspace mounting mentioned in Part 2.



services:
  app:
    volumes:
      - ..:/workspaces/dev-container:cached
  postgres:
    volumes:
      - postgres-data:/var/lib/postgresql/data

volumes:
  postgres-data:

If using Docker Compose, eliminate the workspaceMount in devcontainer.json:



- "workspaceMount": "source=${localWorkspaceFolder},target=/workspaces/${localWorkspaceFolderBasename},type=bind,consistency=cached",

For the volumes under container app, the value ..:/workspaces/dev-container is separated by a colon :.

The .. before the colon represents the path on the host machine, while /workspaces/dev-container after the colon represents the path in the container.

The absolute path of the Docker Compose file is /path/to/dev-container/.devcontainer/docker-compose.yml, so .. equals /path/to/dev-container/.

This setup mounts /path/to/dev-container/ on the host machine to /workspaces/dev-container in the container.

Changes in one are reflected in the other. Whatever you change /path/to/dev-container/ on the host machine, you're making the same change in the /workspaces/dev-container of the container, and vice versa.

For the volumes under container postgres, the value is postgres-data:/var/lib/postgresql/data.

We don't have a path on the host machine to serve data for the database, so create a "virtual volume" via the configuration:



volumes:
  postgres-data:

postgres-data is the name used in postgres-data:/var/lib/postgresql/data, this volume is mounted to container postgres's path /var/lib/postgresql/data.

Creating a virtual volume enables the persistence of database data and the ability to share it between multiple applications.

For instance, if you delete the container postgres, the data saved in the volume postgres-data still exists and is reusable when creating another database with the same volume.

4. Environment variables

The docker compose command will automatically pick up a file called .env in the folder containing the docker-compose.yml. That's why I commented out this section:



...
    # env_file:
    #   - .env # This is a default setting
    #   - ...  # Put other xxx.env files here
...

Unless you use multiple .env files, you don't need to specify any other files here.

Also, we need to remove the runArgs in devcontainer.json:



- "runArgs": [
-   "--env-file",
-   "${localWorkspaceFolder}/.devcontainer/.env"
- ]

Variables in .env are not automatically injected into containers; pass them manually for each container:



services:
  app:
    environment:
      - POSTGRES_HOST=${POSTGRES_HOST}
      - POSTGRES_USER=${POSTGRES_USER}
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
      - POSTGRES_DB=${POSTGRES_DB}
  postgres:
    environment:
      - POSTGRES_USER=${POSTGRES_USER}
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
      - POSTGRES_DB=${POSTGRES_DB}

${POSTGRES_PASSWORD} represents the POSTGRES_PASSWORD defined in .env, and so forth.

Pass these variables to container postgres to create a database with specific credentials, and to container app to connect to the database in container postgres.

Note: In real scenarios, .env and database credentials should be handled by CI/CD systems.

5. Map `ports`

We learned "Forwarding Ports" in Part 1 but this is a bit different.

The ports section in docker-compose.yml exposes container ports to the host machine, specified as HOST:CONTAINER.

While the forwardPorts section in devcontainer.json is specific to VS Code Dev Containers and makes ports accessible within the container and to linked services.

In scenarios with multiple containers (app, app1, app2), each listening on port 8080, map the ports as follows:



services:
  app:
    ports:
      - 8001:8080
  app1:
    ports:
      - 8002:8080
  app2:
    ports:
      - 8003:8080

This setup allows using port 8001 on the host to access container app's port 8080, port 8002 for container app1, and so on.

II. Entry Point

Update devcontainer.json to use Docker Compose as the entry point:



  "name": "Dev Container",
- "image": "docker.io/your-user-name/your-image-name"
+ "dockerComposeFile": [
+   "docker-compose.yml"
+ ],
+ "service": "app",

The app points to the service defined in docker-compose.yml. The depends_on in docker-compose.yml ensures postgres starts with app.

III. Full-Stack Development

With everything prepared, build the Dev Containers (use VS Code's "Open in Container") and start full-stack development.

In my demo, I have a Node.js server (index.js). You can copy index.js and package.json to your project for the demo.

This program runs in container app, connects to the database in container postgres with credentials from .env, creates a table named clients (on first run), writes new data, and displays all data on the web page.

VI. Add Database Client (optional)

To use the psql command line in container app to connect to the PostgreSQL server in container postgres, install the PostgreSQL client via Dev Container Features in devcontainer.json:



  "features": {
    "ghcr.io/robbert229/devcontainer-features/postgresql-client:1": {}
  }

Then, in container app's terminal, connect to the database:



psql -h postgres -U postgres -d postgres
# or
# psql -h $POSTGRES_HOST -U $POSTGRES_USER -d $POSTGRES_DB

Our next guide will focus on developing remotely by putting Dev Containers on a remote machine (cloud machine). This allows you to use the VS Code client or web to connect to the remote machine and Dev Containers, enabling development from anywhere.

Dev Containers - Part 2: Image, Features, Workspace, Environment Variables

Kev — Fri, 21 Jun 2024 06:45:39 +0000

Welcome to the second guide in the Dev Container series:

Dev Containers - Why You Need Them
Part 1: Quick Start - Basic Setup and Usage
Part 2: Image, Features, Workspace, Environment Variables
Part 3: Full Stack Dev - Docker Compose & Database
Part 4: Remote Dev - Develop on a Remote Docker Host
Part 5: Multiple Projects & Shared Container Configuration

To get started, clone my demo project using the following command:



git clone -b part-2-use-image-and-features https://github.com/graezykev/dev-container.git

Alternatively, the faster approach is to use GitHub's Codespaces to run the demo (switch to branch part-2-use-image-and-features):

Introduction

In our last guide, we introduced the basics of using .devcontainer, devcontainer.json, and a Dockerfile to configure and build a Dev Container.

In this guide, we'll optimise the process by adding reusable images and extra software.

Problem

Using a Dockerfile in our .devcontainer folder to build a Dev Container can be time-consuming, especially when pre-installing multiple software.

Newcomers might face issues due to the time taken to install software while building via a Dockerfile. They might also end up with different versions of the software, leading to inconsistencies.

For example, the command:



RUN apt-get install -y nodejs

might install different versions of Node.js over time, leading to unexpected issues.

To address this, we can provide a fixed environment with specific software versions, like Go lang v1.22.4, and prevent unintended upgrades. We can pre-install multiple versions of Node.js in this environment, allowing new team members to switch between versions easily.

This fixed environment is known as a "Docker image" or simply "image."

Newcomers just need to "clone" (download) this image and run it, similar to starting an operating system, but much faster.

Use an Image

To streamline the process for newcomers, we build a Docker image from a basic Dockerfile and push it to a "cloud warehouse" - Docker Hub.

Docker Hub serves as a free file server for our built, ready-to-use Docker images, which anyone can download and reuse.

1. Build & Push Image

Building and pushing a Docker image is straightforward if you have a Dockerfile from our last guide.

You will also need to sign up for a free account on Docker Hub.

For detailed steps, check out this guide to register, build, and push Docker images.

In the end of this step you should get a URL of the image from Docker Hub like docker.io/your-user-name/your-image-name.

2. Configure Image

Specify this image in devcontainer.json and remove the Dockerfile reference:



- "build": {
-   "dockerfile": "Dockerfile"
- }
+ "image": "docker.io/your-user-name/your-image-name"

You can check out my demo and copy the image I've built here: https://github.com/graezykev/dev-container/blob/part-2-use-image-and-features/.devcontainer/devcontainer.json#L3

Using "image": "your-user-name/your-image-name" also works.

That's all! When new teammates use "Open in Container" in VS Code to open our project, the image will be automatically downloaded, cloned, and started as a Dev Container on their machines.

Features

Using an image to create Dev Containers is much faster, especially with a hefty Dockerfile, as we only need to build the image once (or rebuild it if modified).

However, it's yet to be ideal, consider scenarios with multiple tech stacks:



.
├── project-go-lang
├── project-node-js-1
├── project-node-js-2
├── project-postgresql
├── project-python
└── project-rust

Are you going to write a Dockerfile to pre-install all tech stacks (Node.js, Go, Python, etc.) and build an enormous image to use in each project?

No! You don't do this.

Instead, pre-install and pre-configure the most commonly used software in the image.

For projects with divergent tech stacks, install extra software as needed.

Adding extra software to a Dev Container is simple by specifying them in devcontainer.json using the features field. This allows different projects to use the same image with additional specific features.

For example, add the Python engine to devcontainer.json in Project A:



  "image": "docker.io/your-user-name/your-image-name",
+ "features": {
+   "ghcr.io/devcontainers/features/python:1": {
+     "version": "latest"
+   }
+ }

And add the Go language engine to devcontainer.json in Project B:



  "image": "docker.io/your-user-name/your-image-name",
+ "features": {
+   "ghcr.io/devcontainers/features/go:1": {}
+ }

Find available Dev Container features here: https://containers.dev/features. You can also create your own feature.

Adding features is like adding sprinkles to your ice cream!

Workspace

In the devcontainer.json of my demo, you'll see these lines:



  "workspaceMount": "source=${localWorkspaceFolder},target=/workspaces/${localWorkspaceFolderBasename},type=bind,consistency=cached",
  "workspaceFolder": "/workspaces/${localWorkspaceFolderBasename}",

These configurations aren't necessary for most cases but are worth explaining.

Well, workspaceMount and workspaceFolder always come together, to understand them, let me explain the variables of ${localWorkspaceFolder} and ${localWorkspaceFolderBasename} first.

Pre-defined Variables

${localWorkspaceFolder} and ${localWorkspaceFolderBasename} are some pre-defined variables in devcontainer.json.

If you have a project project-a on your host machine:



.
└── path
    └── to
        ├── project-a
        │   └── .devcontainer
        │       ├── Dockerfile
        │       └── devcontainer.json
        ├── project-b
        └── project-c

${localWorkspaceFolder} represents the absolute path /path/to/project-a, and ${localWorkspaceFolderBasename} represents the folder name project-a.

workspaceMount & workspaceFolder

When you set:



"workspaceMount": "source=/path/to/project-a,target=/workspaces/project-a, ..."

it means you're mounting the folder /path/to/project-a from the host machine to the folder /workspaces/project-a in the container.

Whatever you change /path/to/project-a on the host machine, you're making the same change in the /workspaces/project-a of the container, and vice versa.

The workspaceFolder:



"workspaceFolder": "/workspaces/project-a"

sets the default source code location in VS Code within the container.

i.e., when we connect to the container in VS Code, the default source code location will be set to workspaceFolder.

And, the postStartCommand (we mentioned in our last guide) is also run within workspaceFolder.

When to Use

You generally don't need to set workspaceMount and workspaceFolder. However, in scenarios where project-a needs to reference code in project-b or project-c without modifying them, you can set:



"workspaceMount": "source=/path/to,target=/workspaces, ..."
"workspaceFolder": "/workspaces/project-a"

This mounts all projects under /path/to into the container while only modifying project-a.

Environment Variables

In the devcontainer.json of my demo, I have:



"runArgs": [
  "--env-file",
  "${localWorkspaceFolder}/.devcontainer/.env"
]

Environment variables in Dev Containers are useful for managing configuration settings specific to your application or development environment, such as database connection strings, API keys, or feature flags.

By using environment variables, you can securely store sensitive information without hardcoding them in your code.

Moreover, creating separate environment variable files for development, testing, and production environments are very common use case.

How to Set Environment Variables

Create a .env file with key/value pairs (mostly handled by CI/CD systems), and reference it via runArgs in devcontainer.json:



"runArgs": [
  "--env-file",
  "${localWorkspaceFolder}/.devcontainer/.env"
]

These variables will be injected into the container as system variables:

Once the container is built and running, the variables can be used inside the container via Linux commands, shell scripts, or in your code.

In our next guide Full Stack Dev - Docker Compose & Database, we'll use environment variables to manage database passwords and usernames, avoiding exposure in version control.

Dev Containers - Part 1: Quick Start - Basic Setup and Usage

Kev — Fri, 21 Jun 2024 06:45:23 +0000

Welcome to the first guide in the Dev Container series:

Dev Containers - Why You Need Them
Part 1: Quick Start - Basic Setup and Usage
Part 2: Image, Features, Workspace, Environment Variables
Part 3: Full Stack Dev - Docker Compose & Database
Part 4: Remote Dev - Develop on a Remote Docker Host
Part 5: Multiple Projects & Shared Container Configuration

To get started, you can clone my demo project using the following command:



git clone -b part-1-use-docker-file https://github.com/graezykev/dev-container.git

Alternatively, the faster approach is to use GitHub's Codespaces to run the demo (switch to branch part-1-use-docker-file):

Introduction

In my previous post, I explained why Dev Containers are essential. Now, let's dive into setting up a simple Dev Container.

Imagine we have a Node.js project (or Python, Go etc.).

We want anyone joining our project to start developing without manually installing Node.js, or other required software (YARN, PNPM, etc.).

We also want to ensure they have the necessary VS Code extensions and adhere to our coding conventions, like using 2 spaces for indentation.

By defining a Dev Container, we can share all these setups seamlessly.

Future teammates will only need to build the Dev Container and start coding, without the hassle of setting up environments and installing software from scratch.

1. Install VS Code

First, install the VS Code client on your computer.

Note: JetBrains WebStorm also supports Dev Containers, though I haven't tried it yet.

2. Install Docker

Installing Docker is straightforward. Follow the official guide.

Windows PC: Install Docker Desktop with a few clicks. Installation guide.
Mac: Install Docker Desktop with a few clicks. Installation guide.
Linux PC: Install Docker Engine with a few commands. Installation guide.

For example, on Ubuntu Linux, use these commands:



# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update



# Install Docker Engine:
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin



# Add user to the Docker group:
sudo usermod -aG docker $USER # replace $USER with your Linux login username



# Restart the system:
sudo reboot



# Verify the installation:
docker -v

Here's a recording of the installation process:

3. Project Setup

Clone my demo project here: https://github.com/graezykev/dev-container/tree/part-1-use-docker-file



git clone -b part-1-use-docker-file https://github.com/graezykev/dev-container.git

My demo project above might be sort of hefty and take a relatively long time to build. If you prefer a simpler demo, follow these steps to set up a "Hello World" project:

Create a project folder and necessary files:



mkdir your-project-folder && \
cd your-project-folder && \
mkdir .devcontainer && \
touch .devcontainer/devcontainer.json && \
touch .devcontainer/Dockerfile && \
touch index.js

Modify index.js to create a "Hello World" web page:



// index.js
const http = require('http')
const hostname = '0.0.0.0'
const port1 = 8080
const server = http.createServer((req, res) => {
  res.statusCode = 200
  res.setHeader('Content-Type', 'text/plain')
  res.end('Hello, World Node.js!\n')
})
server.listen(port1, hostname, () => {
  console.log(`Server running at http://${hostname}:${port1}/`)
})

Create a simple Dockerfile:



FROM ubuntu:24.04
RUN apt-get update
RUN apt-get upgrade -y
RUN apt-get install -y nodejs
RUN apt-get install -y npm

Reference the Dockerfile in devcontainer.json:



{
  "name": "Dev Container",
  "build": {
    "dockerfile": "Dockerfile"
  }
}

This basic setup defines a Dev Container, creating a virtual Ubuntu system with Node.js installed.

To enhance the workflow, add an auto-run command in devcontainer.json:



{
  "name": "Dev Container",
  "build": {
    "dockerfile": "Dockerfile"
  },
+ "postStartCommand": "node index.js",
+ "forwardPorts": [8080]
}

This command will run the Node.js program every time the container starts.

Additionally, add configurations for VS Code extensions and editor settings:



  "customizations": {
    "vscode": {
      "settings": {
        "editor.tabSize": 2,
        "editor.insertSpaces": true,
        "editor.detectIndentation": false
      },
      "extensions": [
        "dbaeumer.vscode-eslint@3.0.5",
        "esbenp.prettier-vscode",
        "eamodio.gitlens"
      ]
    }
  }

I specify some useful VS Code extensions here, like ESLint, Prettier and the GitLens visualising tools, in your scenario you may add your extensions. You can even specify the version of an extension such as @3.0.5.

4. Build Dev Container

Configuration works are done now, the next step is to build the Dev Container based on what we have configured.

4.1 Open Project in VS Code

In VS Code, go to File -> Open Folder... and select your project folder (dev-container for my demo project or your-project-folder).

4.2 Install Dev Containers Extension

VS Code will prompt you to install the Dev Containers extension (ms-vscode-remote.remote-containers).

Click "Install" and wait for the installation.

4.3 Reopen in Container

After installing the extension, VS Code will prompt you to "Reopen" your project in a container.

Click Reopen in Container to start the building process.

Here is a recording of the process:

4.4 Dev Container Built

Once you reopen the project in a container, the building process will begin. This may take some time, depending on your Dockerfile and system performance. Subsequent starts will be faster unless you change the .devcontainer setup.

After building, enjoy a unified development environment with all applications, extensions, and settings pre-configured!

Moving forward, I'll refer to the computer you're using as the "host machine".

VS Code and Docker run on the host machine, while the Dev Container is built and executed within Docker.

When the Dev Container is built or started, the postStartCommand specified in devcontainer.json will run, starting the Node.js server:

This Node.js engine runs inside the Dev Container, not on your host machine. You can verify this from the terminal:

Similarly, the VS Code extensions specified in devcontainer.json are installed inside the Dev Container, not on your host machine:

Unified Working Environment

We've now created a shareable working environment with a Node.js engine, useful VS Code extensions, and consistent editor settings, all scoped to your project.

Next, we'll optimise this Dev Container using a reusable Docker image and add extra software without modifying the Dockerfile.

Dev Containers - Part 2: Image, Features, Workspace, Environment Variables

Appendix 1: Explanation of Basic Configurations

Building Entry Point

The build -> dockerfile field in devcontainer.json serves as the entry point for the Dev Container. VS Code uses this to build and run the Docker image.

Forwarding Ports

The "forwardPorts": [8080] configuration forwards ports from the container to the host machine, allowing you to access the Node.js server running inside the container from your host machine's browser.

"forwardPorts": [8080] is an abbreviation for "forwardPorts": ["8080:8080"], you can make your own adjustment like "forwardPorts": ["port-on-host:port-in-container"].

For more on forwardPorts, refer to this guide.

Appendix 2: Lifecycle Commands Explanation

Lifecycle scripts in devcontainer.json run at different points in the container’s lifecycle. These can be command lines or shell scripts. For detailed information, refer to this documentation.

postCreateCommand
- When you create a new Codespace, the postCreateCommand will run right after the container is set up.
- When you first open a project in a VS Code dev container, the postCreateCommand will run after the container is built or rebuilt.

If you want to install global npm packages or set environment variables, you should use this command.

postStartCommand
- You are working on a project in a Codespace. You stop the Codespace at the end of the day. The next day, you start the Codespace again to continue your work.
- You are developing an application in a VS Code dev container. You close VS Code or restart your computer, which stops the container. Later, you reopen VS Code and the container starts again.
postAttachCommand
- You are using a Codespace for your project, and you disconnect from it (e.g., by closing the browser tab or your laptop going to sleep). Later, you reconnect to the same Codespace.
- You are working on a project in a VS Code dev container. You close VS Code or restart your computer, then later reopen VS Code and attach to the same running container.

Dev Containers - Part 0: Why You Need Them

Kev — Fri, 21 Jun 2024 06:45:04 +0000

It Works on My Machine

Picture this: You're a newbie developer, excited to dive into your first big project. You clone the repository, open your terminal, and confidently type in the command to start the project. But instead of seeing the application running smoothly, you're greeted with a wall of error messages.

Frustrated, you turn to your mentor for help.

You: Hi mentor, I can't start this project! I'm encountering too many errors!
Mentor: Hmm, it works on my machine. Let me check. Oh, I see, I'm using Node.js v14. What about you?
You: I'm on v18.
Mentor: That explains it. Switch back to v14. It's easy, just run nvm use 14.
You: Sorry, what's NVM?
...

You realise that just one small version difference can cause big headaches, and you're left wondering how to manage these differences seamlessly.

Managing Software and Versions

Our team has several full-stack JavaScript projects, some relying on different versions of Node.js. Upgrading them all at once isn't feasible (it's annoying but common in real-world scenarios).

Whenever a new member joins, they must install NVM and multiple versions of Node.js in order to run those projects.

VS Code Extensions

VS Code extensions can significantly boost development efficiency.

For instance, many JavaScript developers use ESLint and Prettier extensions to auto-format code and fix conventions with a simple Command + S (Control + S):

As an experienced developer, you likely have other VS Code extensions to recommend to your team. However, getting them to install each one, and possibly specific versions of the extensions, can be challenging.

Editor Settings

Every team follows its own code conventions, such as using 2 or 4 spaces for indentation or preferring tabs.

To reduce the hassle for your team, consider how difficult it would be if your new colleague's editor auto-inserts 4 spaces instead of the team's "2 spaces" convention when hitting Tab.

Setting up a Working Environment is Painful

New team members often spend over two days installing and configuring necessary tools.

Network issues, firewalls, dependency, and dependencies of a dependency can be a nightmare, especially in some scenarios they need to run Unix-like commands on a Windows PC.

Easier and Faster Onboarding

All these problems can be resolved with Dev Containers.

Here's how a newcomer can quickly integrate into the development team:

Use any preferred machine or system (Mac, Windows, Ubuntu).
Install Docker (without needing to understand it), which is straightforward.
Install VS Code (just 2 to 3 clicks).
Download or clone the project.
Open the project in VS Code, follow the prompt to build a Dev Container, and start developing!

Emergency Situations

Imagine this: It's a sunny Saturday, and you're finally enjoying a much-needed day off. You're at the park with your family, soaking in the sun, when suddenly, your phone rings. It's your manager, and she sounds frantic.

Manager: We've got a huge problem! The code you pushed yesterday is causing memory leaks, and users are flooding us with complaints. We need a fix immediately!
You: I understand, but I'm not at home. I don't have my laptop with me.
Manager: This issue requires an urgent fix. Can you handle it?

Panic sets in. Fixing this issue requires just a tiny change, but how can you possibly manage it without your usual setup?

With Dev Containers, you could calmly say:

You: No problem. I can handle it right here.

You pull out your wife's iPad, connect to the internet, and open VS Code in the browser. Thanks to Dev Containers, you have an exact replica of your development environment, complete with all necessary tools and settings. Within minutes, you've made the fix and pushed the changes, all from a device you never thought you could code on.

What is a Dev Container?

What exactly is a Dev Container, and how does it work?

If you're familiar with backend development or containers like Docker, Dev Containers will seem familiar.

Even if you're new to containers, using Dev Containers in VS Code is simple and doesn't require in-depth knowledge of containers or Docker (though you need to install Docker).

A Dev Container runs a Docker container with all development software, configurations, VS Code extensions, and settings, sharing environments with the local OS via VS Code.

This creates reproducible development environments usable on any PC, anytime, anywhere.

Dev Containers offer:

Pre-installed and pre-configured systems and software:
- Latest stable LTS Ubuntu
- Tools like Git, wget, curl
- NVM and multiple versions of Node.js
- Package managers like PNPM, YARN
- Programming languages like Python, Ruby, Go, Java, Rust
- Databases like PostgreSQL and extensions
- Customised shells (e.g., Zsh with themes from Oh My Zsh)
- Other customised software or settings
Pre-installed VS Code extensions:
- Specific extensions and versions (including pre-releases)
- Self-made extensions
Pre-configured VS Code settings:
- Tab size: 2
- Auto-insert spaces on hitting Tab: true
- Font size and family
- And more...
Automatic workflow scripts:
- Automatically run npm install or npm start or other scripts as needed
Develop remotely without a PC!
And more...

Steps to Use Dev Containers

I have several guides on setting up and using Dev Containers, each brief and easy to follow:

A Guide to Understanding the Nuances of Git Merge and Rebase

Kev — Tue, 14 May 2024 09:24:38 +0000

Introduction

I've seen many developers avoid using rebase due to its complexity. I've also noticed that some teams even forbid rebase in their projects altogether.

In this guide, we will walk through the basics of managing code changes in Git using both merge and rebase strategies. This tutorial aims to clarify these often misunderstood commands with a hands-on demonstration.

I may be too verbose in walking you through these steps one by one, but I suggest you start from the Recap section, or jump straight to the Key Takeaways section.

Introduction
Scenario Setup
- Initialise a Git Repository and Create a Starting File
- Make Initial Commits A and B
- Bseline Commit Graph
- Create a Feature Branch
- Commit C
- Commit D (Developing on Feature Branch)
Initial Git Commit Graph
Decision Time: Merging vs. Rebasing
Option 1: Merge feature into main
- Merge
- Merge Conflict
- Graph After Merge
Option 2: Rebase feature onto main
- Rebase
- Rebase Conflict
- Graph After Rebase
Recap
- Commit History
- Commit Graph of Taking Each Option
Key Takeaways: Merge or Rebase?

Scenario Setup

Initialise a Git Repository and Create a Starting File

First, create a new directory and initialise a Git repository:

mkdir git-nuance-demo-merge && \
cd git-nuance-demo-merge && \
git init && \
git branch -M main

Next, create a text file named example.txt:

touch example.txt

Make Initial Commits `A` and `B`

Now, add a first line to example.txt and commit it as Commit A:

+Initial content

Proceed by adding a second line and commit it as Commit B:

Initial content
+Added by leader

Bseline Commit Graph

Now we can visualize the project's baseline with this commit graph:

A---B [main]

Create a Feature Branch

Assuming developer Kev needs to add a new feature, he branches off from main:

git checkout main && \
git checkout -b feature

Commit `C`

While Kev works on the feature branch, another developer, Dash, updates the main branch. Dash modifies the second line in Commit C:

git checkout main

Initial content
-Added by leader
+Modified by Dash

Commit `D` (Developing on Feature Branch)

Simultaneously, Kev modifies the same line on his branch in Commit D:

git checkout feature

Initial content
-Added by leader
+Modified by Kev

Initial Git Commit Graph

The commit graph is now:

A---B---C [main]
     \
      D [feature]

And check their Git history details by running git log main and git log feature:

The commit IDs (hashes) will be different if you try following my steps from the beginning on your device.

main

/workspaces/git-nuance-demo-merge (main) $ git log main

commit ac466bea2d13762608660e27798ba08b840529db (HEAD -> main)
Author: Dash <dash@test.com>

    C

commit fd6ecc11625c41c06e150015254b16a620c1cd44
Author: Leader <leader@test.com>

    B

commit 9d00689a138f8d1fd6b7d370bae29bcfb27fec19
Author: Leader <leader@test.com>

    A

feature

/workspaces/git-nuance-demo-merge (main) $ git log feature

commit 58beb1e4f1d47f9bf1364be906a2e22b5280be1d (HEAD -> feature)
Author: Kev <kev@test.com>

    D

commit fd6ecc11625c41c06e150015254b16a620c1cd44
Author: Leader <leader@test.com>

    B

commit 9d00689a138f8d1fd6b7d370bae29bcfb27fec19
Author: Leader <leader@test.com>

    A

Decision Time: Merging vs. Rebasing

Imagine Kev needs to release the features in branch feature, he faces a decision: to merge or rebase his feature branch onto the updated main branch.:

Merge: merge feature into main and release main to production environment.
Rebase: Rebase feature onto main, and release feature.

We all know that Kev and Dash both edited the second line, meaning a conflict is definitely going to happen in both options.

Next, we're going to apply both options and examine the different consequences of each.

Before continuing, copy the initial status into two separate folders so that we can apply each option in its own folder:
cd .. && \
cp -r git-nuance-demo-merge git-nuance-demo-rebase

Option 1: Merge `feature` into `main`

Merge

Merging results in a new commit on main that combines the changes from both branches:

cd git-nuance-demo-merge

git checkout main

git merge feature

Merge Conflict

This approach creates a merge conflict that Kev resolves by favouring his changes:

Initial content
<<<<<<< HEAD
Modified by Dash
=======
Modified by Kev
>>>>>>> feature

Pay attention to the terminal console:

/workspaces/git-nuance-demo-merge (main) $ git merge feature
Auto-merging example.txt
CONFLICT (content): Merge conflict in example.txt
Automatic merge failed; fix conflicts and then commit the result.

Check this out: "fix conflicts and then commit the result"

It instructs us to both fix and commit so that the conflicts can be resolved. This means a new commit is needed when resolving merge conflicts.

Resolve it by selecting "Accept Incoming Change", which in this case means accepting the changes from the feature branch.

If you check the log by running git log main, you'll notice the new merge commit in the logs saying "Merge branch...":

/workspaces/git-nuance-demo-merge (main) $ git log main

commit 4d6ac5eb811be1a6130c6bbcde215946c618dd95 (HEAD -> main)
Merge: ac466be 58beb1e
Author: Kev <kev@test.com>

    Merge branch 'feature'

commit 58beb1e4f1d47f9bf1364be906a2e22b5280be1d (feature)
Author: Kev <kev@test.com>

    D

commit ac466bea2d13762608660e27798ba08b840529db
Author: Dash <dash@test.com>

    C

commit fd6ecc11625c41c06e150015254b16a620c1cd44
Author: Leader <leader@test.com>

    B

...

Wondering what the change is in this new commit?

Run git diff 4d6ac5e~ 4d6ac5e (4d6ac5e is the commit ID (hash))

/workspaces/git-nuance-demo-merge (main) $ git diff 4d6ac5e~ 4d6ac5e
diff --git a/example.txt b/example.txt
index 22ad65d..d49a266 100644
--- a/example.txt
+++ b/example.txt
@@ -1,2 +1,2 @@
 Initial content
-Modified by Dash
+Modified by Kev

If you take a deeper look at the details of this new commit, pay special attention to the line:

Merge: ac466be 58beb1e

What does it mean?

This merge commit has two parents, pointing to commit C and commit D:

Graph After Merge

After resolving the conflict, the commit graph looks like:

A---B---C---M [main]
     \       /
      D [feature]

Option 2: Rebase `feature` onto `main`

Before continuing with the rebase process, take a look at the feature branch's logs at this point:

/workspaces/git-nuance-demo-merge (main) $ git log feature

commit 58beb1e4f1d47f9bf1364be906a2e22b5280be1d (Head -> feature)
Author: Kev <kev@test.com>

    D
...

The commit D's ID (hash) is 58beb1e....

The change in this commit is:

/workspaces/git-nuance-demo-merge (main) $ git diff 58beb1e~ 58beb1e
diff --git a/example.txt b/example.txt
index a53a0e2..d49a266 100644
--- a/example.txt
+++ b/example.txt
@@ -1,2 +1,2 @@
 Initial content
-Added by leader
+Modified by Kev

The commit ID and changes will be altered after the rebase; we'll see why.

Rebase

Let's now start the rebase process.

Alternatively to option 1, rebasing involves integrating the changes from main directly into the feature branch history.

cd git-nuance-demo-rebase

git checkout feature

git rebase main

Rebase Conflict

It creates a merge conflict that is almost identical to the one in option 1:

Initial content
<<<<<<< HEAD
Modified by Dash
=======
Modified by Kev
>>>>>>> 58beb1e (D)

But what's in the terminal console is different:

/workspaces/git-nuance-demo-rebase (feature) $ git rebase main
Auto-merging example.txt
CONFLICT (content): Merge conflict in example.txt
error: could not apply 58beb1e... D
hint: Resolve all conflicts manually, mark them as resolved with
hint: "git add/rm <conflicted_files>", then run "git rebase --continue".
hint: You can instead skip this commit: run "git rebase --skip".
hint: To abort and get back to the state before "git rebase", run "git rebase --abort".
Could not apply 58beb1e... D

It instructs us to "Resolve all conflicts manually", and continue with git rebase --continue.

Alright, let's follow the hint to resolve the conflict (also by selecting "Accept Incoming Change").

git add example.txt && \
git rebase --continue

Take a look at the content after running git rebase --continue:

D

# Please enter the commit message for your changes. Lines starting
# with '#' will be ignored, and an empty message aborts the commit.
#
# interactive rebase in progress; onto 05cfdd2
# Last command done (1 command done):
#    pick 58beb1e D
# No commands remaining.
# You are currently rebasing branch 'feature' on '05cfdd2'.
#
# Changes to be committed:
# modified:   example.txt
#

Git offers us an editor to edit the commit message of D.

You can rewrite commit D by editing the message (lines starting with "#" will be ignored).

I made no changes and just closed the commit editor, so the rebase process proceeded with the rewriting of commit D.

Now that the rebase process has finished, take a look at the commit history of the feature branch:

/workspaces/git-nuance-demo-rebase (feature) $ git log feature

commit 047c3480da20d9dd6c530d3698b4e047f1f1d9d9 (HEAD -> feature)
Author: Kev <kev@test.com>

    D

commit ac466bea2d13762608660e27798ba08b840529db (main)
Author: Dash <dash@test.com>

    C

...

Pay special attention to the commit ID of the new D.

It's now 047c348.... Remember what it was before the rebase? It was 58beb1e....

And the change in this commit is:

/workspaces/git-nuance-demo-rebase (feature) $ git diff 047c348~ 047c348
diff --git a/example.txt b/example.txt
index 22ad65d..d49a266 100644
--- a/example.txt
+++ b/example.txt
@@ -1,2 +1,2 @@
 Initial content
-Modified by Dash
+Modified by Kev

See? The change is also different from the previous commit D because the commit D was rewritten in the rebase process.

Graph After Rebase

Rebase modifies the original commit D on the feature branch:

A---B---C [main]
         \
          D' [feature]

In this case, we performed a history rewrite, causing the original commit D to disappear and the new commit D' to emerge.

Recap

Commit History

Now let's recap the primary steps and Git histories in both options we covered.

Option 1: Merge

Option 2: Rebase

Commit	Diff	Author
Merge	```diff -Modified by Dash +Modified by Kev ```	Kev
D	```diff -Add by leader +Modified by Kev ```	Kev
C	```diff -Add by leader +Modified by Dash ```	Dash
B	```diff +Add by leader ```	Leader
A	...	Leader

Commit	Diff	Author
D'	```diff -Modified by Dash +Modified by Kev ```	Kev
D (actually disappeared)	```diff -Add by leader +Modified by Kev ```	Kev
C	```diff -Add by leader +Modified by Dash ```	Dash
B	```diff +Add by leader ```	Leader
A	...	Leader

You can also view the histories in my demo repositories:

Option 1: Merge

Option 2: Rebase

Commit Graph of Taking Each Option

Merge

A---B---C---M [main]
     \       /
      D [feature]

Rebase

A---B---C [main]
         \
          D' [feature]

The obvious difference is that there are five commits (A-B-C-D-M) after a merge, and four (A-B-C-D') after a rebase (D disappears because it is rewritten).

Git Rebase can be Dangerous

Merge preserves the exact historical sequence of changes and is generally easier for beginners to understand and handle, especially in a collaborative environment.

In contrast, Rebase rewrites the commit history to make it appear as if your changes were created on top of the latest remote commits.

This can result in a cleaner, more linear commit history but can be confusing because it alters the commit history. This might be trickier in a collaborative project unless all contributors are comfortable with Git.

This is typical evidence of why Git rebase can be dangerous:

In more complex, real-world scenarios, there is a risk that you could rewrite, overwrite, or delete another developer's commit from the history, preventing your teammates from finding their previously made commits.

There are some basic principles in using rebase like "Don’t rebase a branch that’s been published remotely" and "Create a backup branch from the tip of the branch you’re about to rebase" etc.

There are some basic principles for using rebase, such as "Don’t rebase a branch that’s been published remotely" and "Create a backup branch from the tip of the branch you’re about to rebase".

Choosing between merge and rebase based on your project's needs is a broad topic that is not comprehensively covered within the scope of this post. For more detailed information, I recommend reading "Differences Between Git Merge and Rebase — and Why You Should Care".

Key Takeaways: Merge or Rebase?

Both strategies have their merits:

Merge: You create a new commit that shows how conflicts were resolved, allowing your teammates to clearly see the resolution process.
Rebase: You modify existing commits. When resolving conflicts during a rebase, you may alter code or commit messages that others have previously made. This can disrupt collaborative work if not used carefully.

A doggerel rhyme to ensure you won't get confused by HTTP 301 and 302

Kev — Fri, 10 May 2024 10:30:55 +0000

Three-O-One, the permanent run;
Three-O-Two, just passing through.

301 Moved Permanently: Imagine your favourite restaurant has moved to a new location. You know you won't find it at the old address anymore, so you update your address book and GPS to go directly to the new spot. It's like the restaurant has permanently changed its address.
302 Found (Moved Temporarily): picture a food truck that occasionally changes its parking spot. You might find it at one location today and another tomorrow. You keep checking the usual spot because you know it's just a temporary move.

Streamline Your Workflow: A Guide to Normalising Git Commit and Push Processes

Kev — Mon, 06 May 2024 03:07:45 +0000

Discover how to streamline your Git commit and push processes using a robust toolkit comprising Husky, ESLint, lint-staged, Commitlint, and simple Shell scripts. This guide will help you automate and standardise your version control workflow, ensuring consistency and efficiency in your development projects. Dive into practical setups and tips to make your Git operations seamless and error-free.

Introduction

Today's topic revolves around leveraging Git Hooks to streamline Git workflows, focusing on code linting before commits and pushes, and standardizing the format of commit messages.

Git Hooks are essentially custom scripts that trigger during key Git actions.

There are two categories of Git Hooks: client-side and server-side. Client-side hooks are activated by operations such as committing, merging, and pushing, whereas server-side hooks are triggered by network operations, like receiving pushed commits.

In this post, I will primarily discuss client-side hooks. I'll delve into three specific hooks: pre-commit, commit-msg, and pre-push. The central tool discussed will be Husky, which simplifies the configuration of Git Hooks, making it more straightforward.

Here’s a preview of what I will cover in this post:

GitHub Repo

If you'd prefer to run the demo I've created instead of following the steps individually, check out this GitHub repository for a quick overview and hands-on experience.

I would like to outline all the steps I'm gonna elaborate on in this post.

Tedious? Don't worry, each step is clear and straightforward. Most of the time, you can copy and paste my commands and codes to follow the guide.

1 - Init your Project
2 - Git Pre-Commit Hook
- 2.1 - Create Pre-Commit Hook
- 2.2 - Code Linting
- 2.3 - Integrate Linting into the Pre-Commit Hook
- 2.4 - Brief Sum-Up
3 - lint-staged
4 - Commit Message Hook
- 4.1 - Commit Message Format Linting
- 4.2 - Add Message Format Linting to Commit Message Hook
- 4.3 - Tailor your Commit Message Format
5 - Git Push Hook
- 5.1 - Incremental Code Linting
- 5.2 - Force test Before Push
- 5.3 - Force Push

1. Init your Project (if you haven't)

Init NPM

mkdir normalise-your-git-commit-and-push && \
cd normalise-your-git-commit-and-push && \
npm init -y && \
npm pkg set type="module"

Init Git Repo

git init && \
echo 'node_modules' >> .gitignore

2. Git Pre-Commit Hook

The Git Pre-Commit Hook is triggered just after you execute the git commit command, but before the commit message editor is opened (if you are not using the -m option), or before the commit is finalized (if you are using the -m option).

What is commit message editor?

The most simple way of specifying our commit message is to execute git commit -m 'my commit message'. However, there's a more interactive way to commit using git commit.

After you run git commit without the -m option and commit message, Git typically opens an editor for you to enter a commit message (This editor could be Vim, Emacs, Nano, or whatever your default command-line text editor is set to).

Returning to the Git Pre-Commit Hook, it is triggered before the editor is opened, or before Git receives you commit message if you're using -m.

We can use it to perform checks (such as running the ESLint command), and if these checks fail, the commit action is blocked.

By sharing this hook script, you can ensure that every teammate commits only code that has been checked.

Let's explore how we can accomplish this.

2.1. Create Pre-Commit Hook

Install Husky

The first important step is to install Husky.

npm install -D husky@9

Init Husky

npx husky init

It's necessary to break down what primarily happens after the execution of husky init.

Create the directory .husky/_ and add the configuration of hooksPath = .husky/_ to .git/config.

The .husky/_ contains the actual hook scripts that Husky will execute in response to specific Git events.

By setting hooksPath = .husky/_, it tells Git to look for hooks in the .husky/_ directory instead of the default .git/hooks.

Create executable scripts such as .husky/_/husky.sh and .husky/_/h etc.

.husky/_/husky.sh is the primary script that Husky injects into the hooks it manages, and .husky/_/h is its Shortcut.

Create a script .husky/pre-commit with the command npm test.

This executable file is the actual Git Pre-Commit Hook we mentioned before, whatever user-defined behaviours you want to add should be placed here.

It's invoked by .husky/_/husky.sh.

The npm test in it means: You want to commit the code? Oh, you have to pass the test first.

Create the prepare command in package.json with the execution of husky.

Anyone who clones your Git repo and runs npm install will automatically invoke the prepare script. Then, husky will initiate .husky/_ and configure .git/hooks.

Create a .gitignore in .husky/_ with a single line *.

This prevents everything inside .husky/_ from being pushed to the remote repository.

Why should .husky/_ be excluded from Git version control?

The scripts under .husky/_ should be generated dynamically according to different environments' configurations and behaviors (machines, operating systems, CI environments, etc.).

For example, the Windows system uses \ as a path separator, while Linux uses /. Each system may have Node.js installed in a specific path. Each system may have its own Node.js version.

...

You cannot create scripts that can be universally applied unless you're working inside the specific environment.

Try A Commit

Since our pre-commit hook has been initiated, let's try it.

git add .

git commit -m 'first commit'

Here is what you'll see from the terminal console:

It fails because the pre-commit hook runs the test job, which contains an exit 1, in the test script of package.json.

"scripts": {
  "test": "echo \"Error: no test specified\" && exit 1",

Changing it to exit 0 will make the commit work.

"scripts": {
-  "test": "echo \"Error: no test specified\" && exit 1",
+  "test": "exit 0",

Commit again:

git add . && \
git commit -m 'first commit'

In a real production project, you should specify your actual test command, such as Jest, Playwright, etc.

2.2. Code Linting

For now, our pre-commit hook only includes a test command. Next, we'll add a Lint command to check the Code Style before you commit JavaScript code.

Install & Configure Linting Tools

Before Linting code in pre-commit hook, let's install and configure ESLint.

npm install -D eslint@9 @eslint/js@9

Create an eslint.config.js file with the code below:

// eslint.config.js

import pluginJs from "@eslint/js";

export default [
  pluginJs.configs.recommended
];

Note: I use export default xxx here because my package.json includes the configuration "type": "module". If you don't have this configuration, use module.exports = xxx instead.

Add Linting Script to `package.json`

"scripts": {
+  "lint": "eslint .",

Create a Demo `index.js`

// index.js

export const field = {
    "b": process.evn.bit,
}

Lint the Demo

npm run lint

This will produce some errors because we haven't defined the variable process in the demo code, which is not allowed according to the ESLint rule.

2.3. Integrate Linting into the Pre-Commit Hook

In the last step, we configured a command to lint our code style. Now, we're going to integrate it with the pre-commit hook.

Put Linting Command to Pre-Commit hook

Add npm run lint to the first line of .husky/pre-commit.

+npm run lint
npm test

Try A Commit

Now, all commits will trigger the execution of this linting command.

git add . && \
git commit -m 'second commit'

You'll encounter a failure because you must fix all the linting errors (mentioned above) before committing the code.

Fix the Linting Errors

Fix it by editing index.js:

// index.js

+const process = {
+    env: {
+        bit: 2
+    }
+}

export const field = {
    "b": process.evn.bit,
}

Commit again and it should work.

git add . && \
git commit -m 'commit after fix index.js'

2.4. Brief Sum-Up

By now, both npm run lint and npm test must pass in the pre-commit hook before you can commit code.

Better Linting

I only made a very simple ESLint configuration here, which may be insufficient for a production project. To integrate a robust linting toolchain, you can check out another post of mine to learn more: Mastering Code Quality: Setting Up ESLint with Standard JS in TypeScript Projects.

Mastering Code Quality: Setting Up ESLint with Standard JS in TypeScript Projects

Kev ・ May 6

#codequality #standardjs #eslint #typescript

3. lint-staged

Have you noticed the problem with npm run lint in the pre-commit?

Yes, all your JavaScript files in the project are checked in this process. What is the problem, though?

Suppose you're working on a historical project with hundreds of JavaScript files that had never integrated linting tools or Git hooks before, meaning there may be numerous code style issues in the existing code.

Today, you integrated ESLint and Git commit hooks into the project, and tomorrow your teammate edits just one JavaScript file. However, they can't commit it because they are overwhelmed by all the historical linting issues in the whole project at once, which are identified by npm run lint.

Here’s another example:

You've spent a day developing a web page and have written several files like header.js, aside.js, main.js, footer.js, etc. But, only header.js is complete, the others are still under development.

Now it's 5 o'clock, time to call it a day! You decide to commit header.js first, but you encounter similar obstacles as in the previous example.

What we need is a way to commit only the "code we really want to commit right now", or more correctly, by Git terminology, the staged files.

In simple terms, staged files are those files you've added to the Git Staging Area by git add <filename>.

Only the code in the staging area should be linted with each commit.

Now, let's see how to get it done.

Install lint-staged

lint-staged is the second important tools we need here, install it via NPM.

npm install -D lint-staged

Configure lint-staged

Create a lint-staged.config.js file in the project root with the configuration below:

// lint-staged.config.js

export default {
  // You can lint other types of files with different tools.
  "*.{js,jsx,ts,tsx}": [
    // You can also add other tools to lint your JavaScript here.
    "eslint"
  ]
}

var b // I included this line to intentionally elicit an error output in ESLint later.

Note: I use export default xxx here because my package.json includes the configuration "type": "module". If you don't have this configuration, use module.exports = xxx instead.

Add lint-staged to NPM Script

  "lint": "eslint .",
+ "lint:staged": "lint-staged",

And of course, modify your hook command accordingly in .husky/pre-commit.

-npm run lint
+npm run lint:staged

Use lint-staged

Annotate the following code from index.js:

// index.js

+// const process = {
+//    env: {
+//        bit: 2
+//    }
+// }

export const field = {
    "b": process.evn.bit,
}

In this scenario, we have ESLint issues in both lint-staged.config.js and index.js, but let's say we don't want to commit index.js and ignore its ESLint issues for now.

To let lint-staged identify what you're going to commit, first add the file you want to commit to the Git Staging Area.

git add lint-staged.config.js # don't add index.js

Then, commit the file.

git commit -m 'test lint-staged'

This time, only the newly added(staged) file lint-staged.config.js is checked during your commit.

index.js was changed, and it obviously has ESLint errors, but it's not staged, so it's not checked in this commit.

You don't need to fix all the JavaScript files in the project, nor even all the JavaScript files you have modified, but just the staged file(s) you actually want to commit.

Let's comment out the line var b in lint-staged.config.js to fix its ESLint error, and then the commit will succeed.

-var b // I included this line to intentionally elicit an error output in ESLint later.
+// var b // I included this line to intentionally elicit an error output in ESLint later.

git add lint-staged.config.js && \
git commit -m 'test lint-staged'

lint-staged Other Files

There are more linting tools that I won't go into deeply, but you can integrate them with lint-staged. For example, you can lint your CSS content with Stylelint, or even lint your README files with markdownlint, etc.

Save other changes

Now we have some unsaved work.

git status

Before we move on to our next Git Hook, let's revert our last change in index.js and save our previous work.

// index.js

-// const process = {
-//    env: {
-//        bit: 2
-//    }
-// }
+const process = {
+    env: {
+        bit: 2
+    }
+}

export const field = {
    "b": process.evn.bit,
}

git add . && \
git commit -m "let's continue"

We're going to mention this "let's continue" later in our next Git Hook.

4. Commit Message Hook

In the previous steps, we introduced the Git Pre-Commit Hook, which activates immediately after you execute the git commit command but before the commit message editor opens or the commit is finalized.

And after that, after the Git Pre-Commit Hook is triggered, the next hook is the Commit Message Hook, or specifically speaking, the commit-msg script. In this hook, you're able to get the Commit Message from the commit message editor or from the message you provide via git commit -m 'message'.

Our next step is to validate the format of the commit message.

Why is format of the commit message so important?

Well, Git commit message is a semantic description of what you are going to do in a specific commit. It's a gateway to communicate with your teammates, code reviewers are easier to understand your intention without diving into the code.

Git commit message also improves traceability, by linking the commit to external resources like bug and task trackers through identifiers or tags, enhancing the traceability of work and the management of project documentation.

That's the significance of adhering to a standard format for commit messages promotes clarity, coherence, and collaboration within a team.

4.1. Commit Message Format Linting

Install Commit Message Linting Tools

commitlint is the most important tool we need for this step.

npm install --save-dev @commitlint/{cli,config-conventional}

Configure `commitlint`

Below is a simple (but conventional) configuration, adhering to the conventional commit format.

echo "export default { extends: ['@commitlint/config-conventional'] };" > commitlint.config.js

Note: I use export default xxx here because my package.json includes the configuration "type": "module". If you don't have this configuration, use module.exports = xxx instead.

Test `commitlint`

Run commitlint directly to verify the message we used in our last commit, checking if it meet the conventional commit format.

npx commitlint --from HEAD~1 --to HEAD --verbose

"from HEAD~1 to HEAD" is your latest commit

You will encounter the error:

Why Dose It Fail?

The command above is mimicking your last commit.

Do you recall what it was? Yes, it was git commit -m 'let's continue'.

In this case, your commit message is "let's continue", but we have a rule for the commit message format configured in commitlint.config.js, which stipulates that the commit message should be structured as follows:

<type>[optional scope]: <description>

[optional body]

[optional footer(s)]

i.e., your commit message must be at least formatted like "feat: your commit description ...".

However, Your message of "let's continue" doesn't satisfy the rule, which means your commit will fail.

We'll find out how to fix it later.

4.2. Add Message Format Linting to Commit Message Hook

Running commitlint directly is not appropriate; we should integrate it into an automatic Git Hook (Commit Message Hook).

Add Linting Script to the Hook

Create the Git Commit Message Hook file .husky/commit-msg and add execution of commitlint to it.

echo "npx --no -- commitlint --edit \$1" > .husky/commit-msg

You'll see a newly created file .husky/commit-msg with the content below:

+npx --no -- commitlint --edit \$1

Test the Hook

The Commit Message Hook is ready, now test it.

git add . && \
git commit -m "this will fail"

It fails because neither a type nor a subject is specified.

Make some slightly adjustments:

git commit -m "foo: this will also fail"

As foo is not a valid type, it fails again, But at least we have one less problem than the last commit.

Modify the message once more:

git commit -m "chore: this is a legal commit message"

Hooray!

4.3. Tailor your Commit Message Format

In real senarios, the conventional commit message format may not meet your team's requirements, sometimes you need to customise your rules.

For instance, your team is using Jira for project and product management, as well as issue tracking, etc. You and your teammates have agreed that every commit should include a Jira ticket ID, allowing you to trace back the real motivation (a product requirement, a technical optimization, a bug, etc.) of every code change.

To do this, edit your commitlint.config.js as follows:

// commitlint.config.js

export default {
  extends: ['@commitlint/config-conventional'],
  plugins: [
    {
      rules: {
        'subject-prefix-with-jira-ticket-id': parsed => {
          const { subject } = parsed
          const match = subject ? subject.match(/^\[[A-Z]{3,5}-\d+\]\s/) : null
          if (match) return [true, '']
          return [
            false,
            `The commit message's subject must be prefixed with an uppercase JIRA ticket ID.
    A correct commit message should be like: feat: [JIRA-1234] fulfill this feature
    Your subject: ${subject}
    Please revise your commit message.
    `
          ]
        }
      }
    }
  ],
  rules: {
    'subject-prefix-with-jira-ticket-id': [2, 'always']
  }
}

Note: I use export default xxx here because my package.json includes the configuration "type": "module". If you don't have this configuration, use module.exports = xxx instead.

Test it.

git add . && \
git commit -m 'chore: try to commit'

Oops! The commit fails as we just add a new rule to force a "JIRA ticket ID" in the commit message's subject.

Try another one:

git commit -m 'chore: [PRJ-1234] a commit with sample id' # the [PRJ-1234] is a fake ID

You've got it!

5. Git Push Hook

We have taken some steps to ensure our code style and commit message format in the previous steps. One further step is to ensure the code quality before it's pushed to the remote repository.

A Git Push Hook is triggered before a push (git push origin <branch>) occurs. You can use the Git Push Hook as another "firewall" to validate the code before it is pushed to the remote repository.

5.1. Incremental Code Linting

We have run npm run lint:staged in pre-commit. Does it ensure that there will be no unchecked code contaminating our codebase?

The answer is No. You can still commit code that hasn't been fixed by this:

git add . && \
git commit -m 'whatever I like' --no-verify

See the --no-verify flag? It can help you to skip pre-commit and commit-msg hooks we made above.

This allows for a forced commit. It's like a hidden time bomb! You're not even able to stop your teammates from doing it sneakily!

So, here we need a second defence line before the forced committed code is pushed to our remote repository and pollute the codebase.

I need to clarify what I mean by Incremental Code here.

Say we have the following commits: commit-a and commit-b have been pushed to the remote repository, and commit-c to commit-N (not sure how many commits in ...) have not been pushed yet.

Commit	Commit time	push status
commit-N	today	not pushed
...	today	not pushed
commit-d	today	not pushed
commit-c	today	not pushed
commit-b	yesterday	pushed
commit-a	the-day-before-yesterday	pushed

Right now, we're going to push from commit-c to commit-N, and commit-c to commit-N represent the incremental code in this context.

First Push

Before making any changes and testing the incremental changes since the last push, it's important to push the code first.

Because later, we're going to make another push, which will be compared to this push as Incremental Code.

git push origin main

Incremental Code Linting Shell Script

Create a Shell script file named scripts/lint-incremental-push-files.sh.

mkdir scripts && \
touch scripts/lint-incremental-push-files.sh

In this Shell script, we'll identify those incremental JavaScript files we want to push, and then run the ESLint command only on them. Input the following code into lint-incremental-push-files.sh.

#!/bin/bash

# Ensure you have the latest info from your remote
git fetch

# Automatically identify the current branch and corresponding remote branch
BRANCH=$(git rev-parse --abbrev-ref HEAD)
REMOTE_BRANCH="origin/${BRANCH}"

# Find the last commit from the remote branch that has been pushed
LAST_PUSHED_COMMIT=$(git rev-parse ${REMOTE_BRANCH})

# Find the current commit
CURRENT_COMMIT=$(git rev-parse HEAD)

# List changed files since the last pushed commit that match the desired extensions
CHANGED_FILES=$(git diff --name-only $LAST_PUSHED_COMMIT $CURRENT_COMMIT | grep -E '\.(js|jsx|ts|tsx)$')

echo "Files to lint:"
echo $CHANGED_FILES

# Run ESLint on these files if any are found
if [ -z "$CHANGED_FILES" ]
then
    echo "No JavaScript/TypeScript files to lint."
else
    echo "Linting files..."
    ./node_modules/.bin/eslint $CHANGED_FILES
    if [ $? -ne 0 ]; then
        echo "Linting issues found, please fix them."
        exit 1
    fi
fi

Make the script executable:

chmod +x scripts/lint-incremental-push-files.sh

Add this script to a NPM script in package.json.

  "scripts": {
    "prepare": "husky",
    "lint": "eslint .",
    "lint:staged": "lint-staged",
+   "lint:incremental-push": "./scripts/lint-incremental-push-files.sh",
    "test": "exit 0"
  },

Create the Git Push Hook configuration file named .husky/pre-push, and add the NPM script to it.

echo "npm run lint:incremental-push" > .husky/pre-push

Now, this Shell script will run every time before your push. Your git push command will fail if the incremental code doesn't pass the ESLint check, meaning no force-committed code can pass through!

Test Incremental Code Linting

Now, let's make some changes.

Open index.js to add a simple line.

// index.js

const process = {
    env: {
        bit: 2
    }
}

export const field = {
    "b": process.evn.bit,
}

+var a

You can not commit it because we have a pre-commit hook to lint the file and we just made a ESLint error in it.

git add . && \
git commit -am 'bypass eslint to commit'

But you can bypass the check with --no-verify.

git commit -am 'bypass eslint to commit' --no-verify

Do a similar thing to eslint.config.js with a new line:

import pluginJs from "@eslint/js";

export default [
  pluginJs.configs.recommended
];

+var b;

Bypass the check process again.

git add . && \
git commit -am 'bypass eslint again to commit' --no-verify

Check and Push Incremental Code

Just now, we have two commits including index.js and eslint.config.js, in which there are actually ESLint issues, but they were committed using tricks (--no-verify).

The commits in the red rectangle are the so-called incremental changes to be pushed, but they should not be pushed since they have ESLint issues!

But don't panic, we've got your back! They won't be able to be pushed because they will face the punishment of the Git Push Hook we made above!

If you try to push the code:

git push origin main

All Incremental Errors will be caught!

You can not push them until fix the errors and commit again.

index.js:

-var a
+// var a

eslint.config.js:

-var b;
+// var b;

git add . && \
git commit -m 'fix: [TEST-01] fix ESLint errors' && \
git push origin main

5.2. Force `test` Before Push

Next, I'm going to modify our team's workflow.

I will move the npm test command from the Git Pre-Commit Hook hook to the Git Push Hook named pre-push.

This is because I decide to allow linted code commits, without the test verification. I mean, we allow "un-test" code to be commit, but don't allow them to be pushed.

To do that, edit ./husky/pre-commit:

npm run lint:staged
- npm test

And move the test command to the Git Push Hook.

echo "npm test" >> .husky/pre-push

Let's edit the package.json's test command to intentionally make the test fail:

"scripts": {
-  "test": "exit 0",
+  "test": "exit 1",

Now, if you try to push any code, you'll fail because we have an exit 1 in the command. This means the test process is not passed, preventing you from pushing the "un-tested" code to the remote repository.

git add . && \
git commit -am 'chore: [TEST-1234] test commit'

This would have failed if we hadn't removed npm test from pre-commit.

git push origin main

The push will fail because we just added a test in the pre-push hook, which means we need to ensure that the test succeeds before we can push our code.

Revert exit 1 to exit 0, or in a real production project, use your actual test scripts that can pass, your code push to the remote repository will succeed!

5.3 Force Push

Unfortunately, you can still bypass the pre-push check and force push the code by using the --no-verify flag, just as you can force a commit.

git push origin main --no-verify

Or, if you're using GitHub, you can even commit and push your code directly on the website.

To address this, we need server-side Git hooks or CI systems. However, these are more complex topics, and I won't delve deeply into them now. Perhaps I'll write another post to introduce them in the future.

Conclusion: DIY your Workflows

In this post, I've delved into detailed steps to normalise our teams' Git workflow. My approach is just a basic framework; there are many Git hooks I haven't mentioned here. You should customize your Git hooks to tailor them to your team's workflow.

For example, you may not agree with me that I move the npm test command from the Git Pre-Commit Hook to the Git Push Hook. Based on this post, I think you already know how to make your own choice and tailor your own workflows.

Here are some additional steps you might consider:

Make use of the other Git Hooks I didn't introduce in this post to enhance your teamwork, such as post-checkout, post-commit, pre-rebase, etc.
Implement specific rules for commit message formatting after discussing with your teammates.
Perform both linting and testing in commits and pushes.
Add other commands or scripts to your Git commit/push hooks, such as sending an IM message or an email to notify your teammates of your changes.
Use your imagination to customize the workflow as you see fit.

Last but not least, any suggestions, corrections, questions, or disagreements will be greatly appreciated.

Mastering Code Quality: Setting Up ESLint with Standard JS in TypeScript Projects

Kev — Mon, 06 May 2024 02:55:00 +0000

Learn how to enhance your TypeScript project by integrating JavaScript Standard Style rules using eslint-config-love in ESLint. This guide will also provide you with essential tips for setting up automatic formatting in Visual Studio Code, ensuring your code remains clean and consistent. Furthermore, we'll explore the inclusion of ESLint plugins specifically designed for React and React Hooks to optimise your development workflow.

Introduction

Configuring ESLint in a TypeScript project is nothing much, the more challenging aspect is choosing a Code Style for your team.

You may be torn between those famous code styles, struggling to choose one between Airbnb JavaScript Style, Google JavaScript Style Guide, JavaScript Standard Style, or XO, among others.

What's worse, you and your teammates may be arguing about whether you should use ; at the end of each line, whether if() { should have a space after if, whether function foo () {return true} is a bad way and should change it to function foo () { return true }, and so on.

In my personal perspective, it doesn't matter which style you choose, but you need a unified and strict style across a team. Once the rules are set, there's no room for argument, as long as the rules are detailed and cover most aspects.

So, self-assertively, I have chosen JavaScript Standard Style, also known as Standard JS, even though it is not at all standard—the name is somewhat misleading.

Standard JS is simple, clear, straightforward, and detailed, and I think sticking to the rules makes JS/TS code very clean.

It doesn't allow configurations, rules are rules. No configurations, which is by design to avoid too much bikeshedding over style choices, So I don't need to argue with my teammates.

Examples.



window.alert('hi')   // ✓ ok
window.alert('hi');  // ✗ avoid



console.log('hello there')    // ✓ ok
console.log("hello there")    // ✗ avoid
console.log(`hello there`)    // ✗ avoid



/* comment */ // ✓ ok
/*comment*/   // ✗ avoid

...

I have to admit that Standard JS is opinionated, and choosing it is a subjective thing. Some people even hate it—we're all right.

In this post, I also use ESLint + Standard JS as my code formatting tools. Formatting JS/TS code by using ESLint is also subjective and opinionated, arguably most people would rather use Prettier instead, which provides more configurable options.

But like I said before, Standard JS's philosophy is "rules are rules", its rules are detailed, some arguable rules are strictly normalised and no compromises are allowed, customising Prettier's options will lead me back to endless arguments with my teammates.

Sorry, I've gone too far. I'm not here to persuade you to use Standard JS. My intention is to provide information and guidance on configuring JavaScript Standard Style for your team, should you agree with me or have other reasons to choose it.

GitHub Repo

If you'd prefer to run the demo I've created instead of following the steps individually, check out this GitHub repository for a quick overview and hands-on experience.

I primarily divide this post into 5 parts.

Initial Setup
ESLint Configuration
Editor (VS Code) Integration
Automate Linting and Formatting
Linting React & React Hooks

1. Initial Setup

If you're not setting up a new TypeScript project, you can skip this part 1.

Init Git

Initialize a Git repository if you're starting a new project.



mkdir ts-eslint-standard-js && \
cd ts-eslint-standard-js && \
git init && \
echo 'node_modules' >> .gitignore

Init Project

Initialize your NPM package.



npm init -y

I recommend enabling ES6 modules in your project, indicating that your source code should use import syntax.



npm pkg set type="module"

I also recommend upgrading your Node.js version and specifying the minimum version requirements in your project.



npm pkg set engines.node=">=18.18.0 <19 || >=20.9.0 <21 || >=21.1.0"

Install TypeScript

Run this in your terminal to install TypeScript.



npm install -D typescript

Init TypeScript

Use the following command to initialise a new TypeScript project.



npx tsc --init

A tsconfig.json will be created under your project.

2. ESLint Configuration

Install ESLint

Run the command below to install ESLint packages.



npm install -D \
  eslint@^8.57.0 \
  typescript-eslint@^7.6.0 \
  eslint-plugin-promise@^6.0.0 \
  eslint-plugin-import@^2.25.2 \
  eslint-plugin-n@^15.0.0 \
  @typescript-eslint/eslint-plugin@^7.0.1 \
  eslint-config-love@latest \
  globals@^15.0.0

Note: These versions can be changed in the future.

The packages installed include:

ESLint Core
ESLint shared configs and plugins for Standard JS, with the most important one being eslint-config-love.

ESLint configuration

Create a file named eslint.config.js with the code below:



import globals from 'globals'
import tseslint from 'typescript-eslint'

import path from 'path'
import { fileURLToPath } from 'url'
import { FlatCompat } from '@eslint/eslintrc'
import pluginJs from '@eslint/js'

// mimic CommonJS variables -- not needed if using CommonJS
const __filename = fileURLToPath(import.meta.url)
const __dirname = path.dirname(__filename)
const compat = new FlatCompat({
  baseDirectory: __dirname,
  recommendedConfig: pluginJs.configs.recommended
})

let a

export default [
  { files: ['**/*.js'], languageOptions: { sourceType: 'script' } },
  { languageOptions: { globals: globals.browser } },
  ...tseslint.configs.recommended,
  ...compat.extends('love')
]

I use import and export here because in Part 1 I set the package type to module to enable ES6 modules. If your project does not use ES6 modules, you may name this file as eslint.config.mjs or modify the import and export syntax to use CommonJS.

There are only 2 lines you need to notice now.



  ...tseslint.configs.recommended,
  ...compat.extends('love')

The first line instructs ESLint to parse and check TypeScript syntax, while the second line enforces adherence to Standard JS rules for both JavaScript and TypeScript code.

Try Out

Run this command to verify the configuration file itself:



npx eslint eslint.config.js

Yes, there are indeed some errors in the configuration file.

Try `.ts`

Create an index.ts with the code:



const x = {
    a: "b",
    b: 123
};

And check its code style:



npx eslint index.ts

Showing the errors indicates that the configurations also work for TypeScript!

Besides, you can use npx eslint . to check the style of all your files with one command.

Fix Code Style Issues

You can use the --fix flag in ESLint to automatically correct some of the illegal syntax and style issues in your code.



npx eslint --fix index.ts # npx eslint --fix .

Some issues will be automatically fixed, such as changing " to ', removing unnecessary semicolons at the end of lines, and so on.

Certainly, there's still 1 problem left, as not all issues can be automatically resolved. For instance, ESLint typically won't fix unused variables; you'll need to address those manually.

3. Editor (VS Code) Integration

Using npx eslint . and npx eslint --fix . to check and format every JS/TS file in your codebase can be a nightmare, which is laborious and sometimes uncontrollable.

Imagine you need to edit the code in the editor and run the commands in the terminal, you may be facing a huge amount of errors at the time you run npx eslint.

A way to ease the anxiety is by integrating with the ESLint VS Code Extension. It checks and formats the code problems along with your coding. For example, if you make a mistake, you'll be alerted by the editor immediately and can fix it right away.

You may not be using VS Code, but other editors like WebStorm offer similar solutions.

VS Code ESLint Extension

Search for dbaeumer.vscode-eslint in the Extensions panel and install it.

At the time I write this, I'm using version 3.0.5 (pre-release) of this extension. Other versions may have some unknown issues.

First, create the configuration file.



mkdir .vscode && \
touch .vscode/settings.json

Edit .vscode/settings.json to enable the ESLint extension in your editor.



{
  "eslint.enable": true
}

Reloading your VS Code window is required. You'll see errors detected by the ESLint extension showing up in the editor while you are editing the code. These errors will be highlighted with wavy lines.

Hovering over each wavy line will toggle a pop-up displaying the error details.

Auto Format on Save

This VS Code ESLint extension can also automatically fix your code's linting issues. It operates similarly to running npx eslint on the file while you're editing it.

Modify .vscode/settings.json with some additional configurations:



  "eslint.format.enable": true,
  "editor.formatOnSave": true,
  "[javascript]": {
    "editor.defaultFormatter": "dbaeumer.vscode-eslint"
  },
  "[typescript]": {
    "editor.defaultFormatter": "dbaeumer.vscode-eslint"
  },
  "[javascriptreact]": {
    "editor.defaultFormatter": "dbaeumer.vscode-eslint"
  },
  "[typescriptreact]": {
    "editor.defaultFormatter": "dbaeumer.vscode-eslint"
  }

Now, every time you type some code and press Command + S to save them, the problems in the file that can be automatically fixed will be corrected automatically, just looks like your are running eslint --fix eslint.config.js.

The semicolons will disappear after you press Command + S to save your code.

By sharing this settings.json in Git or any other version control tools, if your teammates have the extension dbaeumer.vscode-eslint installed, the same behaviour will occur in their VS Code.

Recommend Installing the Extension

You can recommend your teammates to install dbaeumer.vscode-eslint via the following configure.



touch .vscode/extensions.json && \
echo '{   
  "recommendations": ["dbaeumer.vscode-eslint"]
}' > .vscode/extensions.json

VS Code prompts your teammates to install dbaeumer.vscode-eslint when the project is opened for the first time.

devcontainer

If your team is using a Cloud Development Environment such as GitHub Codespaces, or Dev Containers such as Docker, you can even share the installation of dbaeumer.vscode-eslint with your teammates, via devcontainer.json.

When someone opens the project in VS Code, the specified extension (as well as the specific version) dbaeumer.vscode-eslint@3.0.5 will be automatically installed within the devcontainer.

The automatic installation only applies to the devcontainer environment.

I love this, but it's a larger topic, and I won't elaborate too much on it here, you can just give it a try.



mkdir .devcontainer && \
touch .devcontainer/devcontainer.json && \
echo '{
  "name": "ts-eslint-standard-js",
  "customizations": {
    "vscode": {
      "extensions": [
        "dbaeumer.vscode-eslint@3.0.5"
      ]
    }
  }
}' > .devcontainer/devcontainer.json

The extension dbaeumer.vscode-eslint will be automatically installed if you try it from here.

Extra Tricks

Different developers have their own default editor behaviours.

For instance, in your VS Code, pressing the Tab key may produce a tab with a width of 4 spaces, while for your teammates, it may produce 2 spaces or 4 spaces, depending on their default settings.

Under the rules of Standard JS, the use of real tabs for code indentation is not allowed; instead, you have to use 2 spaces.

To collaborate better with your teammates or enforce the rules of our code style more strictly, I recommend configuring the VS Code editor and sharing the configurations to unify this behaviour. For example, you can set it to automatically insert 2 spaces after pressing the Tab key instead of inserting a real tab, following the rule defined by Standard JS.

To achieve this, add the following 3 configurations to .vscode/settings.json:



  "editor.tabSize": 2,
  "editor.insertSpaces": true,
  "editor.detectIndentation": false

From now on, whenever someone opens your project in VS Code, pressing the Tab key will insert 2 spaces, and pressing the Backspace key will delete 2 spaces.

4. Automate Linting and Formatting

In this section, we'll integrate checking and formatting commands into your project's NPM scripts. This enables you to execute batched tasks or run them in your CI workflows and Git Hooks.

Linting Command

Edit your package.json.



{
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
+   "lint": "eslint ."

Don't forget to exclude some files that should not be linted. Create a .eslintignore file and add the necessary exclusions to it.



node_modules
test
coverage
public
dist

Now try the script:



npm run lint

All ESLint problems in your JavaScript/TypeScript files will be displayed.

Formatting Command

As mentioned above, some problems can be fixed by the command eslint --fix. So We can leverage this command as a way to batch format your code.

Edit package.json:



{
  "scripts": {
+   "format": "eslint --fix .",

And try formatting it:



npm run format

Since some problems have been fixed, only those that can't be fixed will be displayed.

Why do I need NPM scripts

You must find that the commands of npm run lint and npm run format have no difference from npx eslint . and npx eslint --fix . respectively.

Well, when you run npx eslint . directly, it attempts to execute the ESLint tool from your project’s local dependencies (ESLint under the /node_modules/.bin/ directory of your project). However, if ESLint is not installed within your project, npx will attempt to run the globally installed ESLint instead.

However, the NPM script "lint": "eslint ." ensures that ESLint is only executed within your project. If ESLint is not installed, running this script will result in an error.

By strictly defining the specific version (or version ranges) of ESLint in your package.json, you can avoid conflicts between ESLint versions installed globally and those required for your project.

What's more, your lint script might include additional flags or options specific to your project. NPM scripts, with more semantic indicators, allow you to abstract away those complex commands or tool configurations.

I have another post on Streamline Your Workflow: A Guide to Normalising Git Commit and Push Processes, leveraging the NPM scripts. Check it out for more ideas that can enhance your workflow.

Streamline Your Workflow: A Guide to Normalising Git Commit and Push Processes

Kev ・ May 6

#githook #husky #commitlint #lintstaged

5. Linting React & React Hooks

JavaScript Standard Style is less opinionated about JSX formatting and largely leaves JSX as-is. In a React project, you should integrate with React-specific linting rules for ESLint. The generally accepted configurations are eslint-plugin-react and eslint-plugin-react-hooks, enforcing some best practices of writing React code.

Dependencies

First, install React and its type definitions.



npm install react react-dom



npm install -D @types/react @types/react-dom

Disable VS Code Extension (temporary)

In the next steps, I'm going to create some code, but I don't want them to be auto-fixed by ESLint.

So, before we proceed, let's disable the ESLint VS Code Extension to better demonstrate how the following linting configurations work. (But this is only an intermediate step; don't forget to undo the change in the end.)

Let TypeScript to Recognize JSX

Next, let's create a sample file react-code-1.tsx to write some React (JSX) code.



import React from "react";

const SignupButton = () => {
  const handleSignup = () => {
    alert("Sign up successful!");
  };

  return (
    <button onClick={handleSignup} className="signup-button">
      Sign Up
    </button>
  );
};

const MyComponent = (props) => {
  return (<div id={props.id} />)
};

There are some errors visible in the editor.

That's because JSX syntax is not allowed yet in TypeScript configuration.

Edit tsconfig.json to enable it:



-    // "jsx": "preserve",                                /* Specify what JSX code is generated. */
+    "jsx": "react",                                /* Specify what JSX code is generated. */

Then the error above will disappear.

Next, lint the .tsx file:



npx eslint react-code-1.tsx

You'll find 14 problems, indicating that ESLint is enabled for .tsx files.

Ask ESLint to Recognize `.jsx`

Create a react-code-2.jsx file with the same code as react-code-1.tsx, and lint it.



npx eslint react-code-2.jsx

You may see some problems, but they are not actual code style issues.

That's because the .jsx file extension is not specified in the ESLint configuration. Let's configure it in eslint.config.js:



export default [
-  { files: ['**/*.js'], languageOptions: { sourceType: 'script' } },
+  { files: ['**/*.{js,ts,jsx,tsx}'], languageOptions: { sourceType: 'script' } },

Lint it again, and you'll get 14 errors, the same as react-code-1.tsx.

These errors in react-code-1.tsx and react-code-2.jsx are code style problems, however, they are still not React-specific problems. We need to do more.

Install React & React Hooks ESLint plugins



npm install -D eslint-plugin-react eslint-plugin-react-hooks

Configure React ESLint Plugin

First, configure eslint-plugin-react in eslint.config.js with just a new line.



export default [
  { files: ['**/*.{js,ts,jsx,tsx}'], languageOptions: { sourceType: 'script' } },
  { languageOptions: { globals: globals.browser } },
+ ...compat.extends('plugin:react/recommended'),
  ...tseslint.configs.recommended,
  ...compat.extends('love')

Run command to lint it.



npx eslint react-code-1.tsx

You'll find 15 problems. This is one more issue compared to the previous 14 problems, because an additional problem related to React, specified in the plugin eslint-plugin-react, has been detected.

Self-Customize your Own Rule(s)

Ok, let's take one further step and edit eslint.config.js to extend your custom rule:



export default [
  { files: ['**/*.{js,ts,jsx,tsx}'], languageOptions: { sourceType: 'script' } },
  { languageOptions: { globals: globals.browser } },
  ...compat.extends('plugin:react/recommended'),
  ...tseslint.configs.recommended,
  ...compat.extends('love'),
+  {
+    rules: {
+      'react/destructuring-assignment': ['warn', 'always']
+    }
+  }
]

Run npx eslint react-code-1.tsx, and you'll notice one additional problem defined by you.

You can define or modify any rules as much as you like to tailor your team's rules here.

Configure React Hooks ESLint Plugin

Let's now integrate the best practice rules for React Hooks.

Edit react-code-1.tsx to include some React Hooks code.



-import React from "react";
+import React, { useContext } from "react";

...

const MyComponent = (props) => {

+  if (true) {
+    const theme = useContext(ThemeContext);
+  }

  return (<div id={props.id} />)
};

Now lint it.



npx eslint react-code-1.tsx

And you'll find some errors, but none of them are related to React Hooks.

You just need to edit the ESLint configuration with one additional line.



  ...compat.extends('plugin:react/recommended'),
+ ...compat.extends('plugin:react-hooks/recommended'),
  ...tseslint.configs.recommended,

Lint react-code-1.tsx again, and you'll get errors related to React Hooks.

There's one more thing you may have noticed: the warning.

Add the following configurations to automatically detect the React version and eliminate the warning.



  ...compat.extends('love'),
  {
+    settings: {
+      react: {
+        version: 'detect'
+      }
+    },
    rules: {

Conclusion

Now we have a somewhat comprehensive toolchain for linting and formatting your code.

The TypeScript and JavaScript code, including features developed with React and React Hooks, will be validated and formatted using CLI scripts and Editor Tools.

We also have a method to customise the rules for React and React Hooks or to extend them with your own rules.

Next, you may want to integrate your linting scripts with your Git Hooks or CI/CD workflows. For Git Hooks, I have another post that elaborates on the steps. Check it out if you're interested.

Streamline Your Workflow: A Guide to Normalising Git Commit and Push Processes

Kev ・ May 6

#githook #husky #commitlint #lintstaged

Last but not least, any suggestions, corrections, questions, or disagreements will be greatly appreciated.

DEV Community: Kev

Dev Containers - Part 5: Multiple Projects & Shared Container Configuration

Introduction

Common Docker Compose File

Ports

Volumes & Workspace

Service

Build and Switch Dev Containers

Demo Preview

Dev Containers - Part 4: Remote Dev - Develop on a Remote Docker Host

Introduction

Developing with Remote Tunnels

0. Prepare

1. Install Docker (Remote Machine)

2. Install Code CLI (Remote Machine)

3. Create Secure Tunnel (Remote Machine)

4. Connect to Remote Machine

5. Clone Your Project

Build the Dev Container

Work Anywhere

Dev Containers - Part 3: Full Stack Dev - Docker Compose & Database

Introduction

Environment Isolation

I. Compose Configuration

0. docker-compose.yml

1. Define services

2. Specify image

3. Define volumes

4. Environment variables

5. Map ports

II. Entry Point

III. Full-Stack Development

VI. Add Database Client (optional)

Next

Dev Containers - Part 2: Image, Features, Workspace, Environment Variables

Introduction

Problem

Use an Image

1. Build & Push Image

2. Configure Image

Features

Workspace

Pre-defined Variables

workspaceMount & workspaceFolder

When to Use

Environment Variables

How to Set Environment Variables

Next

Dev Containers - Part 1: Quick Start - Basic Setup and Usage

Introduction

1. Install VS Code

2. Install Docker

3. Project Setup

4. Build Dev Container

4.1 Open Project in VS Code

4.2 Install Dev Containers Extension

4.3 Reopen in Container

4.4 Dev Container Built

Unified Working Environment

Appendix 1: Explanation of Basic Configurations

Building Entry Point

Forwarding Ports

Appendix 2: Lifecycle Commands Explanation

Dev Containers - Part 0: Why You Need Them

It Works on My Machine

Managing Software and Versions

VS Code Extensions

Editor Settings

Setting up a Working Environment is Painful

Easier and Faster Onboarding

Emergency Situations

What is a Dev Container?

Steps to Use Dev Containers

A Guide to Understanding the Nuances of Git Merge and Rebase

Introduction

Table of Contents

Scenario Setup

Initialise a Git Repository and Create a Starting File

Make Initial Commits A and B

Bseline Commit Graph

1. Define `services`

2. Specify `image`

3. Define `volumes`

5. Map `ports`

Make Initial Commits `A` and `B`

Commit `C`

Commit `D` (Developing on Feature Branch)

Option 1: Merge `feature` into `main`

Option 2: Rebase `feature` onto `main`

Add Linting Script to `package.json`

Create a Demo `index.js`

Configure `commitlint`

Test `commitlint`

5.2. Force `test` Before Push