DEV Community: Kengo TODA

Use NodeJS v16 as runtime for JavaScript actions

Kengo TODA — Wed, 29 Dec 2021 08:49:24 +0000

This month GitHub supported NodeJS v16 as runtime for JavaScript actions. It is also supported by GitHub Enterprise Servers (GHES) 3.0 and later.

Motivation to use NodeJS v16

The end-of-life of NodeJS v12 is the end of next April. So, if you maintain your own JavaScript actions, it is suggested to migrate their runtime to NodeJS v16.

How to update the project

We can complete the minimal migration by updating run.using in the actions.yml file. We may need several additional updates like below:

Update `engines.node` in `package.json`

If you have specified the version of NodeJS to use in package.json, you will need to update it.
16.13.1 is the current latest version, so the preferred configuration would be as follows:

  "engines": {
    "node": "^16.13.1"
  }

Update `tsconfig.json`

NodeJS v16 lets us use ES2021. Based on the official TSConfig settings recommendation for NodeJS v16, update the compilerOptions as follows:

  "compilerOptions": {
    "lib": ["ES2021"],
    "module": "commonjs",
    "target": "ES2021"
  }

Update TypeScript

If your TypeScript is too old to use ES2021, it is suggested to update.

Note that TypeScript v4.4 changed the type of caught error from any to unknown by default.
So if your action catches error to invoke core.setFailed() method, you need to check the type of caught object in runtime:

-     core.setFailed(error)
+     if (error instanceof Error) {
+       core.setFailed(error)
+     } else {
+       core.setFailed(JSON.stringify(error))
+     }

Update ESLint

If you've updated your TypeScript, it is necessary to update ESLint too. If you've created projects based on the official template project, you can run the following command to upgrade ESLint and its plugins in batch:

$ npm add -D eslint@^8.5.0 @typescript-eslint/parser@^5.8.0 eslint-plugin-github@^4.3.5 eslint-plugin-jest@^25.3.0

If you found lint failure caused by changes on the ESLint side, copy the default .eslintrc.json from the template project, or follow ESLint's migration guide documented in its GitHub Releases.

Update GitHub Actions Workflow to use NodeJS v16

If you use the default NodeJS on the PATH, you need no action because GitHub hosted runners use NodeJS v16 by default since this Dec/10th.

If you use actions/setup-node to specify the version of NodeJS, you need to update the node-version config in the workflow definitions.

Or if you have a configuration file like .node-version and .nvmrc, you can use the node-version-file config supported from v2.5.0.

Wrap up

NodeJS v12 will reach its end-of-life soon. All system running on NodeJS needs to be migrated, and GitHub JavaScript Actions are also in the target.

The migration process is quite simple, just update metadata and run the build with NodeJS v16. You may refer to my PR for gradle/wrapper-validation-action as example. Enjoy!

SpotBugs supports SARIF that helps integration with other SAST tools

Kengo TODA — Sun, 17 Oct 2021 02:17:53 +0000

Are you using SAST tools? How many tools you're using? Just one? Then it's fine. A few? It's still working, probably. More? Welcome to one of difficulties of DevSecOps!

Currently, when we maintain a system, it's quite common to depend on multiple languages and ecosystems. If system is enough small, a one-stop SAST solution should be enough... but later you may find it's not enough.

Then how to integrate reports generated by multiple SAST tools, to grab the overview of your system? The Static Analysis Results Interchange Format (SARIF) could be a solution.

SARIF is an OASIS standard and an industry standard format for the output of static analysis tools. Configure each SAST tool to generate a SARIF report, then we can merge reports to get a simple overview of the service. The GitHub Code scanning is one example, it works as a dashboard of all SAST tools like below:

How to configure SpotBugs to generate a SARIF report

First, it's better to use SpotBugs 4.4.1 and above, that includes a fix to make SARIF report compatible with Github code scanning API requirements.

If you use command line interface to run SpotBugs, append -sarif option.

If you are using Gradle, configure tasks with SpotBugsTask to set reports.sarif.enabled:

spotbugsMain {
    reports {
        sarif {
            enabled = true
        }
    }
}

If you are using Maven, configure the plugin with <sarifOutput>true</sarifOutput>:

      <plugin>
        <groupId>com.github.spotbugs</groupId>
        <artifactId>spotbugs-maven-plugin</artifactId>
        <configuration>
          <sarifOutput>true</sarifOutput>
        </configuration>
      </plugin>

Refer to spotbugs/spotbugs-gradle-plugin as a living example with GitHub Code scanning integration.

Hope that this guide helps you to find new way to handle SAST tool reports, and make your hacking awesome!

How to replace @types/jest with @jest/globals and jest-mock

Kengo TODA — Sun, 22 Aug 2021 09:38:31 +0000

I believe that it's always better to prefer official type definitions rather than unofficial DefinitelyTyped. And Jest v27 introduced new defaults so I thought that it's a nice timing to replace @types/jest with @jest/globals and jest-mock. I'll share several problems I met:

Dependency Update

jest-circus is now default test runner, so we don't need to depend on it directly. Also remove @types/jest to replace with @jest/globals and jest-mock:

npm remove jest-circus @types/jest
npm add -D jest@^27.0.6 ts-jest@^27.0.5

Configuration

I'm using ts-jest to transpile TypeScript code, and its configuration is like below. Now we can remove both testEnvironment and testRunner in most cases:

const { defaults: tsjPreset } = require('ts-jest/presets')

module.exports = {
  clearMocks: true,
  moduleFileExtensions: ['js', 'ts'],
  // no need `testEnvironment: "node"` any more
  testMatch: ['**/*.test.ts'],
  // no need `testRunner: "jest-circus"` any more
  transform: {
    ...tsjPreset.transform,
  },
  verbose: true
}

I do not use preset to configure ts-jest, to let other integrations like puppeteer use it.

Import from `@jest/globals` and `jest-mock`

As described in the official API Document, we can import Jest API from @jest/globals module.

About SpyInstance, @types/jest provides it as jest.SpyInstance. But official type definition provides it as import { SpyInstance } from 'jest-mock'; ... with generics!

import { beforeEach, jest } from '@jest/globals';
import { SpyInstance } from 'jest-mock';

let spyOSHomedir: SpyInstance<string, []>;

beforeEach(() => {
  spyOSHomedir = jest.spyOn(os, 'homedir');
  spyOSHomedir.mockReturnValue(__dirname);
});

This generics will bring better type safety, and tons of code change. For instance, mocked function sometimes wants to return the value that satisfies required type partially. In the following case, mocked fs.statSync returns an object that has only isFile() method even though the required type fs.Stats has more properties and methods. In such case we can use the Partial utility type:

import { beforeEach, jest } from '@jest/globals';
import { SpyInstance } from 'jest-mock';
import fs from 'fs';

let spyFsStat: SpyInstance<Partial<fs.Stats | fs.BigIntStats>, [fs.PathLike, fs.StatOptions?]>

beforeEach(() => {
  spyFsStat = jest.spyOn(fs, 'statSync');
  spyFsStat.mockImplementation((file: fs.PathLike) => {
    return { isFile: () => file === expectedJdkFile };
  });
});

One more example: when you mock overloaded methods, you'll face some difficulty like:

https://javascript.plainenglish.io/mocking-ts-method-overloads-with-jest-e9c3d3f1ce0c

By applying the solution introduced in that article, we may lose integrity in test codes. So it would be an acceptable solution to use unknown in some cases:

let spyFsReadDir: SpyInstance<unknown, Parameters<typeof fs.readdirSync>>;

beforeEach(() => {
  // returned value is typed as `Dirent[]` but we want to return `string[]`, so use `unknown` to relax mismatch
  spyFsReadDir = jest.spyOn(fs, 'readdirSync');
  spyFsReadDir.mockImplementation(() => ['JavaTest']);
});

That's all, you may refer to my commit replacing DefinitelyTyped with official type definitions. Thank you!

A complete guide to use dependabot with semantic-release and @vercel/ncc for GitHub Actions

Kengo TODA — Fri, 16 Jul 2021 12:05:34 +0000

Depandabot is a really productive solution to keep our products secure and updated.

It is also easy to set up, however, it provides no chance to run a command in the PR, so we have additional considerations if we have some pre-release processes. For instance, for a GitHub Actions repository, we want to keep the dist directory updated by @vercel/ncc right after we update its dependencies.

This article introduces semantic-release as a solution. It automates the release process, and its exec plugin helps us to run @vercel/ncc before the release automatically.

Overview of the development process is like below:

Dependabot creates a PR to update dependencies
GitHub Actions test the change, and reflect its result to GitHub Checks
Merge the PR manually or automatically
GitHub Actions run on the default branch, and trigger the semantic-release
semantic-release creates a Git tag, a GitHub Release, and a npm release

Check the following chapters for detail, or refer to my GitHub repo as a working example.

Install necessary node packages

Make sure you have a npm project in your working directory, and run the following command to install semantic-release and its plugins:



npm add -D semantic-release @semantic-release/exec @semantic-release/git

Configure npm to run `@vercel/ncc`

In package.json, confirm that we have a script to run ncc to optimize JS files like below:



  "scripts": {
    "package": "ncc build --source-map"
  }

Configure semantic-release

Follow the official documents to configure semantic-release.

In my case, it became like below to use the main branch as default branch, and run exec plugin and git plugin:



  "release": {
    "branches": "main",
    "plugins": [
      "@semantic-release/commit-analyzer",
      "@semantic-release/release-notes-generator",
      "@semantic-release/npm",
      "@semantic-release/github",
      ["@semantic-release/exec", {
        "prepare": "npm run package"
      }],
      [
        "@semantic-release/git",
        {
          "assets": [
            "dist",
            "package.json",
            "package-lock.json"
          ],
          "message": "chore(release): ${nextRelease.version} [skip ci]\n\n${nextRelease.notes}"
        }
      ]
    ]
  }

Configure the dependabot to follow the Conventional Commit

Configure .github/dependabot.yml to enable dependabot. Make sure commit-message part is configured with prefix: fix to following the Conventional Commits.



version: 2
updates:
  - package-ecosystem: npm
    directory: "/"
    schedule:
      interval: daily
    commit-message:
      prefix: fix
      prefix-development: chore
      include: scope

(optional) Configure the auto-merge probot

To merge PRs made by dependabot automatic, you may introduce probot-auto-merge. .github/auto-merge.yml will be like below:



minApprovals:
  NONE: 0
requiredLabels:
  - dependencies
updateBranch: true
mergeMethod: rebase

If we apply NONE: 0 to minApprovals configuration to merge without manual review, make sure the branch protection rule is created for the default branch, to make sure PRs will be merged only when required status checks have passed.

There is no way to configure the branch protection rule by file, so you need to use GUI at github.com like below:

Create a PAT (Personal Access Token)

semantic-release needs a GitHub token to push a commit. But when we have branch protection rules for the target branch, secrets.GITHUB_TOKEN provides not enough permission so you'll face an error like below:



error: GH006: Protected branch update failed for refs/heads/main.
error: 2 of 2 required status checks are expected.

To avoid this error, create a PAT and store it as an encrypted secret.

To keep our development environment secure, it's better to keep PAT's scope minimal as much as possible. In my case, just a public_repo scope is enough.

Configure GitHub Actions

We have a PAT so now we can configure GitHub Actions workflow!
It'll be like below:



      - uses: actions/checkout@v2
        with:
          # Make sure the release step uses its own credentials.
          persist-credentials: false
      - run: |
          npm ci
          npm run all
      - name: Run semantic-release
        # This process will run `ncc`, commit files, push a Git commit, and release to GitHub and npmjs.
        run: |
          npx semantic-release
        env:
          GITHUB_TOKEN: ${{ secrets.PAT_TO_PUSH }}
          NPM_TOKEN: ${{ secrets.NPM_TOKEN }}

Or you may refer to my workflow config as a working example.

Important points are as follows:

Run actions/checkout with persist-credentials: false, or git plugin will use the GitHub token generated by actions/checkout.
Set the PAT created at the previous chapter to GITHUB_TOKEN when we run npx semantic-release.

Wrap up

Dependabot is really productive, even to GitHub Actions project that needs to keep the dist directory updated. semantic-release helps us not only by tagging but also by running commands to make the project ready to release.

There are several pitfalls around permission control, hope that this article helps you to avoid them. Enjoy hacking! :)

New GitHub Action to set up docker-compose

Kengo TODA — Fri, 28 Aug 2020 12:23:36 +0000

My Action

This setup-docker-compose action downloads the docker-compose command and add it to the PATH for following executions. At this moment, It supports the Linux environment only.

Submission Category:

Wacky Wildcards. Usually, FOSS projects don't need to use this action, because action runners hosted by GitHub already have docker-compose in its PATH. So other categories aren't suitable to submit.

Yaml File or Link to Code

KengoTODA / actions-setup-docker-compose

the GitHub Action setting up docker-compose command

GitHub action to setup `docker-compose` command

This action downloads the docker-compose command and add it to the PATH for following execution. It supports Linux environment only.

How to use

Add a step to your workflow like below:

  steps:
  - uses: KengoTODA/actions-setup-docker-compose@main
    with:
      version: '1.26.2'

The version parameter is required, specify the full version of docker-compose command.

View on GitHub

Additional Resources / Info

As noted above, FOSS projects usually don't need to use this action. So there is no FOSS project depending on it for now.

This action is for who's trying to integrate FOSS power into in-house development with the self-hosted runner.

Deploying to OSSRH with Gradle in 2020

Kengo TODA — Wed, 29 Apr 2020 01:22:29 +0000

It seems that the method described in the official page is a little bit old. Here I'll introduce a modern way.

Metadata and Signing

Use the maven-publish plugin instead of the maven plugin. No need to configure the artifacts extension.

Jar Files

The classifier property of Jar task is now deprecated. Instead, we can use withJavadocJar() and withSourcesJar() in the java extension.

java {
    withJavadocJar()
    withSourcesJar()
}

Signing artifacts

We're using maven-publish plugin, so the signing extension in our build script will be like the following:

ext.isReleaseVersion = !version.endsWith("SNAPSHOT")

// -------------------------------------
// here 'publishing' extension will come
// -------------------------------------

signing {
    sign publishing.publications.mavenJava
}
tasks.withType(Sign) {
    onlyIf { isReleaseVersion }
}

Metadata Definition and Upload

Instead of uploadArchives extension for the maven plugin, configure publishing extension for the maven-publish plugin.

publishing {
    repositories {
        maven {
            def releaseRepo = "https://oss.sonatype.org/service/local/staging/deploy/maven2/"
            def snapshotRepo = "https://oss.sonatype.org/content/repositories/snapshots/"
            url = isReleaseVersion ? releaseRepo : snapshotRepo
            credentials {
                username = project.hasProperty('ossrhUsername') ? ossrhUsername : "Unknown user"
                password = project.hasProperty('ossrhPassword') ? ossrhPassword : "Unknown password"
            }
        }
    }

    publications {
        mavenJava(MavenPublication) {
            pom {
                groupId = 'com.example'
                name = 'example project'
                description = 'Example Project to learn how to deploy to OSSRH'
                url = 'https://example.com/'
                from components.java
                licenses {
                    license {
                        name = 'The Apache License, Version 2.0'
                        url = 'http://www.apache.org/licenses/LICENSE-2.0.txt'
                    }
                }
                scm {
                    connection = 'scm:git:git@github.com:example.com/example.git'
                    developerConnection = 'scm:git:git@github.com:example.com/example.git'
                    url = 'https://github.com/example.com/example/'
                }
            }
        }
    }
}

Deployment

Use the publishToMavenLocal task to publish artifacts to maven local repository.

Use the publish task to publish artifacts to maven remote repositories. Don't forget to visit oss.sonatype.org to close and release your staging repository.

SpotBugs v4.0.0 is out!

Kengo TODA — Wed, 19 Feb 2020 11:04:58 +0000

This week we've released SpotBugs v4.0.0.

We've spent about a year to deliver this stable version, however, this version has less breaking changes. We removed only unused features based on this investigation. I wish you can enjoy the new version with fewer migration costs.

Note that default detectors in this version still has some problems around Java 9+, check related issues if you run SpotBugs for projects using Java 11, 14 or later. To avoid this problem, you can limit activated detectors by command line option.

We also provide the official migration guide. If you're SpotBugs plugin author, please check it to ease your migration.

BTW if you're using IntelliJ IDEA, watch this issue and contribute to help the SpotBugs-IDEA project. I'm not member of it, but it seems pretty cool.
And if you're using Gradle, watch this issue and help us to verify the experimental implementation. Current implementation has many problems, and it's quite hard to solve incrementally, so I started this project from scratch. It seems working and keep most of API, but we still need your contribution especially from Android project developers.

Tips to implement Gradle Plugin

Kengo TODA — Thu, 09 Jan 2020 20:59:13 +0000

Now I'm implementing a new Gradle Plugin. The official document is really nice, and I wish my experience will help you to code yours.

Use groovy for extension and task

The extension is a major way to configure your plugin. It's a class that uses Property and Provider as its fields, then users can set values like below:

extension {
  intProp = 10
  fileProp = file('config.yml')
  listProp = [ 'bar', 'baz' ] 
}

Groovy helps you to generate necessary accessors, so what you need is simply private final fields:

class MyExtension {
    @NonNull
    final Property<Boolean> boolProp;

    @Inject
    MyExtension(ObjectFactory objects) {
        boolProp = objects.property(Boolean).convention(false)
    }
}

class MyTask extends DefaultTask {
    @Input
    @NonNull
    final Property<Boolean> boolProp;

    @Inject
    MyTask(ObjectFactory objects) {
        extension = project.exntensions.getByType(MyExtension)
        boolProp = objects.property(Boolean).convention(extension.boolProp)
    }
}

Always prefer lazy-approach

For better performance, we need to learn about build lifecycle, or face performance problem like this.

Prefer the lazy-approach. Pick APIs that lets you specify a configurationAction as its parameter like TaskContainer#register(...).

Refer the official document for detail.

Understand the difference between getXxx() and findXxx()

In Groovy modules, methods named getXxx() throw an exception if you found nothing. On the other hand, methods named findXxx() return null (not Optional!).

So when you expect that you always find the value (e.g. get the extension created by your plugin), use getXxx(). Otherwise, prefer findXxx() and check the nullness of returned value.

Worker API

To run heavy computations like static analysis, it is better to use the Gradle Worker API.

In my case, I need to launch a JVM process to kick SpotBugs, and it makes build especially in multi-module projects. Here is the result of my benchmark:

It requires Gradle 5.6+ but is actually valuable. You may have a try :)

Sign Eclipse plugin by Gradle

Kengo TODA — Wed, 21 Nov 2018 10:28:15 +0000

When we distribute Eclipse plugin, it is better to sign your .jar files by jarsigner bundled with JDK. By this process user can install your plugin without warning like this.
In Eclipse wiki we have Jar Signing page, but it explains no detailed method to sign. Here let's see how Gradle can sign your .jar files.

To sign by jarsigner, you need Java Keystore file that stores keychain. We can use keytool to generate self-signed signature, but here we'll use Let's Encrypt to generate non self-signed signature.

First, follow interaction of Let's Encrypt, then you can generate privkey.pem and fullchain.pem. Second, use openssl command to generate .p12 file that is necessary to generate Java Keystore. Here is example:

$ openssl pkcs12 -export \
  -in fullchain.pem -inkey privkey.pem \
  -out your.p12 \
  -name eclipse-plugin

Third, generate .jks file by keytool command:

$ keytool -importkeystore \
  -srckeystore your.p12 -srcstorepass $SRC_PASS -srcstoretype PKCS12 \
  -destkeystore your.jks -deststorepass $DEST_PASS

We use this .jks file to sign, so remember your $DEST_PASS. You also need to keep this .jks file secret. If you're using Travis CI, encrypt-file command is your friend.

OK now we have all necessary things, let's config our build.gradle file.
What you need is a function to sign both plugins jar file and feature jar files. Groovy itself doesn't provide feature to sign, so use Ant's SignJar task instead like below:

// sign all .jar file under specified dir
ext.signJar = { File dir ->
  def keystorepass = project.hasProperty('DEST_PASS') ? keystorepass : '';
  if (keystorepass.isEmpty()) {
    print 'to sign eclipse plugins, set "DEST_PASS" project property'
    return
  }

  dir.traverse(type: FILES, nameFilter: ~/.*\.jar$/) {
    def relativePath = rootDir.toPath().relativize( it.toPath() )
    println "signing ${relativePath}"
    ant.signjar(
        destDir: it.parentFile,
        jar: it,
        alias: 'eclipse-plugin',
        keystore: "path/to/your.jks",
        storepass: keystorepass,
        tsaurl: 'http://timestamp.digicert.com',
        verbose: true
    )
  }
}

Well done! You can simply invoke this function between jar task and artifacts.xml file generation (because this artifacts.xml contains md5 hash of .jar files).

Here is a PR that introduces jarsigner to SpotBugs build, you may check it as working solution. It uses Gradle 5.0 RC3, but it should work with Gradle 4 too.

Build stage on Travis CI

Kengo TODA — Fri, 27 Jul 2018 10:55:48 +0000

Today I found a Travis CI blog post regarding build stage on Travis CI. Now it goes GA so we all can enjoy this feature.

Here is the my PR to introduce build stage. In my feeling, it has one good point:

No more `if` in analysis and deploy part

When I run analysis like SonarCloud, it was necessary to limit target JVM & environment variable to make PR page easy to read. For example, here is a .travis.yml snippet that runs analysis only with specific condition:

jdk:
  - oraclejdk8
  - oraclejdk9
script:
  - ./mvnw org.jacoco:jacoco-maven-plugin:prepare-agent verify -B
  - if [[ $TRAVIS_JDK_VERSION == "oraclejdk8" ]]; then ./mvnw sonar:sonar -B; fi

In case of build stage, script runs with the first value in env, jdk and others by default. So we can remove if from .travis.yml:

jdk:
  - oraclejdk8
  - oraclejdk9
script:
  - ./mvnw verify -B
jobs:
  include:
    - stage: analysis
      script:
        - ./mvnw org.jacoco:jacoco-maven-plugin:prepare-agent verify sonar:sonar -B

One point, is that, each build stage and job doesn't share generated files. So here we need to run verify phase even in analysis stage, to generate analysis targets (.class file in this case).

This merit works even to deploy phase. This feature should reduce complexity in our .travis.yml.

That's all. Enjoy hacking with Travis CI!

DEV Community: Kengo TODA

Use NodeJS v16 as runtime for JavaScript actions

Motivation to use NodeJS v16

How to update the project

Update engines.node in package.json

Update tsconfig.json

Update TypeScript

Update ESLint

Update GitHub Actions Workflow to use NodeJS v16

Wrap up

SpotBugs supports SARIF that helps integration with other SAST tools

How to configure SpotBugs to generate a SARIF report

How to replace @types/jest with @jest/globals and jest-mock

Dependency Update

Configuration

Import from @jest/globals and jest-mock

A complete guide to use dependabot with semantic-release and @vercel/ncc for GitHub Actions

Install necessary node packages

Configure npm to run @vercel/ncc

Configure semantic-release

Configure the dependabot to follow the Conventional Commit

(optional) Configure the auto-merge probot

Create a PAT (Personal Access Token)

Configure GitHub Actions

Wrap up

New GitHub Action to set up docker-compose

My Action

Submission Category:

Yaml File or Link to Code

KengoTODA / actions-setup-docker-compose

the GitHub Action setting up docker-compose command

GitHub action to setup docker-compose command

How to use

Additional Resources / Info

Deploying to OSSRH with Gradle in 2020

Metadata and Signing

Jar Files

Signing artifacts

Metadata Definition and Upload

Deployment

SpotBugs v4.0.0 is out!

Tips to implement Gradle Plugin

Use groovy for extension and task

Always prefer lazy-approach

Understand the difference between getXxx() and findXxx()

Worker API

Sign Eclipse plugin by Gradle

Build stage on Travis CI

No more if in analysis and deploy part

Update `engines.node` in `package.json`

Update `tsconfig.json`

Import from `@jest/globals` and `jest-mock`

Configure npm to run `@vercel/ncc`

GitHub action to setup `docker-compose` command

No more `if` in analysis and deploy part