DEV Community: Yuya Takeyama

Mask multiple lines text in GitHub Actions Workflow

Yuya Takeyama — Sun, 26 May 2024 13:36:10 +0000

This article is a translation of an article I originally wrote in Japanese, translated into English using ChatGPT with some modifications:

GitHub Actions の Workflow 内で複数行の文字列をマスクする

tl; dr

Although it might seem like a bit of bad practice, the content of $multiple_lines_text can be masked with the following one-liner:

- run: |
    echo "::add-mask::$(echo "$multiple_lines_text" | sed ':a;N;$!ba;s/%/%25/g' | sed ':a;N;$!ba;s/\r/%0D/g' | sed ':a;N;$!ba;s/\n/%0A/g')"

`add-mask` Command in GitHub Actions

GitHub Actions has a feature called workflow commands.

Workflow commands for GitHub Actions

The add-mask command is used to mask specific strings in subsequent outputs. For example, by executing the following command, the string "Hello, World!" will be masked as "***" in subsequent outputs.

echo "::add-mask::Hello, World!"

Values retrieved from secrets can be masked this way, but using the add-mask command allows dynamically specifying strings to be masked.

Additionally, this command is implemented in the NPM package @actions/core. By using core.setSecret('Hello, World!'); in JavaScript code, the same masking can be achieved.

The setSecret function essentially builds and outputs the ::add-mask::~~~ string to the standard output, so the mechanism is identical.

Issues When Masking Multiple Lines

If you try to mask a string like Foo\nBar\nBaz in a shell script without any considerations, it would look like this:

::add-mask::Foo
Bar
Baz

The command breaks due to the line breaks, and only "Foo" gets masked. This can lead to unwanted side effects like the string "Foo Fighters" being masked as "*** Fighters".

The workflow commands documentation briefly mentions handling multiple strings:

Multiline strings

However, these techniques are meant for inputting multiline environment variables or output values into $GITHUB_ENV or $GITHUB_OUTPUT, and based on my tests, they don’t work with workflow commands.

Examining the `core.setSecret` Code

By examining the core.setSecret function in @actions/core, it appears that the values to be masked are escaped using a function called escapeData.

https://github.com/actions/toolkit/blob/d1df13e178816d69d96bdc5c753b36a66ad03728/packages/core/src/command.ts#L80-L85

function escapeData(s: any): string {
  return toCommandValue(s)
    .replace(/%/g, '%25')
    .replace(/\r/g, '%0D')
    .replace(/\n/g, '%0A')
}

The toCommandValue function simply returns the string as-is if it’s already a string, so it can be ignored.

https://github.com/actions/toolkit/blob/d1df13e178816d69d96bdc5c753b36a66ad03728/packages/core/src/utils.ts#L11-L18

Then, the replace method replaces % with %25, \r with %0D, and \n with %0A.

By executing a command like this, Foo\nBar\nBaz gets masked correctly:

echo "::add-mask::Foo%0ABar%0ABaz"

After asking ChatGPT about how to perform these replacements in a shell script, I arrived at the following conclusion:

echo "::add-mask::$(echo "$multiple_lines_text" | sed ':a;N;$!ba;s/%/%25/g' | sed ':a;N;$!ba;s/\r/%0D/g' | sed ':a;N;$!ba;s/\n/%0A/g')"

While the s/%/%25/g part is clear, the preceding :a;N;$!ba; part is obscure. I understand from ChatGPT that it’s necessary for handling line breaks, but I can’t explain it in detail here.

When Is This Necessary?

This article is written for those wondering how to mask multiline strings, regardless of the purpose. For me, there was a specific use case: safely storing and using a GitHub App's Private Key.

To achieve this, I believe the following steps are necessary:

Encode the Private Key in Base64 and store it in AWS Secrets Manager.
Use OIDC for authentication with AWS.
Retrieve the secret in the workflow, decode it from Base64 to get the Private Key.

Assuming AWS usage, the steps can be adapted similarly for Google Cloud using Secret Manager, etc. (not verified).

This can be implemented in a workflow as follows:

steps:
  - name: Configure AWS credentials
    id: aws-credentials
    uses: aws-actions/configure-aws-credentials@v4
    with:
      role-to-assume: arn:aws:iam::012345678901:role/role-name
      aws-region: ap-northeast-1

  - name: Retrieve secret from AWS Secrets Manager
    id: aws-secrets
    run: |
      secrets=$(aws secretsmanager get-secret-value --secret-id secret-name --query SecretString --output text)
      gh_app_private_key="$(echo "$secrets" | jq .GH_APP_PRIVATE_KEY_BASE64 -r | base64 -d)"
      echo "::add-mask::$(echo "$gh_app_private_key" | sed ':a;N;$!ba;s/%/%25/g' | sed ':a;N;$!ba;s/\r/%0D/g' | sed ':a;N;$!ba;s/\n/%0A/g')"
      echo "gh-app-private-key<<__EOF__"$'
'"$gh_app_private_key"$'
'__EOF__ >> "$GITHUB_OUTPUT"

  - uses: actions/create-github-app-token@v1
    id: app-token
    with:
      app-id: ${{ vars.GH_APP_ID }}
      private-key: ${{ steps.aws-secrets.outputs.gh-app-private-key }}

Further details are explained in the Q&A format below.

Why Not Store It in Organization or Repository Secrets?

Because it is considered unsafe.

With some specific conditions, the secret's value can be exposed if someone can create a Pull Request and place a GitHub Actions workflow file in the repository.

For more details, refer to this presentation from a recent event. (company blog)

AWS知見共有会でTerraformのCI/CDパイプラインのセキュリティ等について発表してきました + GitHub新機能Push rulesについて (in Japanese)

Why Do You Need to Mask the Private Key?

Without masking, the Private Key passed as input to actions/create-github-app-token@v1 can be viewed in the GitHub Actions UI.

Why Encode the Private Key in Base64 for AWS Secrets Manager?

It’s not mandatory but simplifies storing it without line breaks.

While AWS Secrets Manager can store secrets with line breaks, the key/value mode in the Management Console does not handle line breaks well. Plaintext mode allows entering line breaks, but it’s cumbersome. Therefore, encoding it in Base64 for storage without line breaks is more convenient.

Representation of a Data Structure Like an Array of Hash Tables in Bash Using jq

Yuya Takeyama — Mon, 17 Sep 2018 20:41:55 +0000

Have you ever needed a Hash Table in Bash?
I needed it when I was writing a deploy script.

I found that Bash 4 has a data structure called Dictionary.
https://stackoverflow.com/questions/1494178/how-to-define-hash-tables-in-bash

But the version of Bash in my laptop is 3.2.57 and I couldn't use it.
Of course, it's easy to update the Bash.
But we may need to work on environments with an older version of it.
So I thought up about the other way.

How to

#!/bin/bash -eu

# A data structure like an array of hash tables
# Strictly, it's a stream of JSONs
PARAMETERS=$(cat <<_EOT_
{
  "foo":"FOO 1",
  "bar":"BAR 1"
}
{"foo":"FOO 2","bar":"BAR 2"}
{
  "foo": "フー",
  "bar": "\u30d0\u30fc"
}
_EOT_
)

# Transform into a line-delimited JSONs
PARAMETERS=$(echo "${PARAMETERS}" | jq . --monochrome-output --compact-output)

# Enumerate JSONs and output foo and bar of each item
while IFS='' read -r param || [[ -n "${param}" ]]; do
  foo=$(echo "${param}" | jq .foo -r)
  bar=$(echo "${param}" | jq .bar -r)

  echo "foo=${foo}"
  echo "bar=${bar}"

  echo "---"
done < <(echo "${PARAMETERS}")

$ bash json_stream.sh
foo=FOO 1
bar=BAR 1
---
foo=FOO 2
bar=BAR 2
---
foo=フー
bar=バー
---

Build a JSON as a string using heredoc
Transform the JSON into a line-delimited JSONs
Read a JSON from each line
- https://stackoverflow.com/a/10929511
Retrieve each property like jq .foo in the loop

Hope you enjoyed it!

How I measure Response Times of Web APIs using curl

Yuya Takeyama — Wed, 06 Dec 2017 09:39:27 +0000

Why `curl`?

There is a bunch of specific tools for benchmarking HTTP requests.
ab, JMeter, wrk...
Then why still use curl for the purpose?

It's because curl is widely-used and it's a kind of common language for Web Developers.

Also, some tools have a feature to retrieve an HTTP request as a curl command.

It's quite useful because it copies not only the URL and parameters but also request headers including Authorization or Cookie.

Tools

In this article, I use these tools:

Measure response time using `curl`

At first, let's prepare a curl command. In this time, I got the command of the request to my personal blog using Google Chrome. (Cookie is removed)

$ curl 'https://blog.yuyat.jp/' -H 'Accept-Encoding: gzip, deflate, sdch' -H 'Accept-Language: en-US,en;q=0.8,ja;q=0.6' -H 'Upgrade-Insecure-Requests: 1' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.86 Safari/537.36' -H 'Connection: keep-alive' --compressed

It just outputs the response body from the server.

Let's append these options.

-s -o /dev/null -w  "%{time_starttransfer}\n"

-s is to silence the progress, -o is to dispose the response body to /dev/null.

And what is important is -w.
We can specify a variety of format and in this time I used time_starttransfer to retrieve the response time (time to first byte).

It shows like below:

$ curl 'https://blog.yuyat.jp/' -H 'Accept-Encoding: gzip, deflate, sdch' -H 'Accept-Language: en-US,en;q=0.8,ja;q=0.6' -H 'Upgrade-Insecure-Requests: 1' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.86 Safari/537.36' -H 'Connection: keep-alive' --compressed -s -o /dev/null -w  "%{time_starttransfer}\n"
0.188947

The response time is 0.188947 second (188 msec).

To simplify, I also created a wrapper command curlb:

#!/bin/sh
curl -s -o /dev/null -w '%{time_starttransfer}\n' "$@"

Measure the percentile of the response times

It's not proper to benchmark from just a single request.

Then let's measure the percentile of 100 requests.

ntimes is useful for such purposes.

https://github.com/yuya-takeyama/ntimes

You can install with go get github.com/yuya-takeyama/ntimes or the repository has pre-built binaries.

Let's append ntimes 100 -- at the beginning of the curl command.

$ ntimes 100 -- curlb 'https://blog.yuyat.jp/' -H 'Accept-Encoding: gzip, deflate, sdch' -H 'Accept-Language: en-US,en;q=0.8,ja;q=0.6' -H 'Upgrade-Insecure-Requests: 1' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.86 Safari/537.36' -H 'Connection: keep-alive' --compressed
0.331915
0.064085
0.059883
0.074047
0.059774
...

And to measure the percentile of the numbers, the command called percentile may be the easiest option.

https://github.com/yuya-takeyama/percentile

Install it by go get github.com/yuya-takeyama/percentile or download the pre-built binary from the repo.

And append | percentile to the end of the command.

$ ntimes 100 -- curlb 'https://blog.yuyat.jp/' -H 'Accept-Encoding: gzip, deflate, sdch' -H 'Accept-Language: en-US,en;q=0.8,ja;q=0.6' -H 'Upgrade-Insecure-Requests: 1' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.86 Safari/537.36' -H 'Connection: keep-alive' --compressed | percentile
50%:    0.061777
66%:    0.06412
75%:    0.06872300000000001
80%:    0.07029000000000001
90%:    0.07496700000000001
95%:    0.076153
98%:    0.077226
99%:    0.07957
100%:   0.109931

That's it!