Our returns API is built and deployed by one GitHub Actions workflow with two jobs. The build job pushes the image to ECR and publishes the full image reference as a job output. The deploy job reads that output, takes the tag from the end of it and runs helm upgrade with --set image.tag. In October it deployed a build from March to production, and for most of a working day customers could not create returns for the two carriers we had added since the spring.
The workflow had not been edited in months. What changed was a tidy-up of repository settings. Our AWS account ID had been a plain repository variable, used to build the role ARN for the credentials step, and someone moved it into secrets on the reasoning that anything with an account number in it should be hidden.
The image reference begins with the registry host, and the registry host contains the account ID. The runner will not pass a job output that contains a secret value to another job. It writes a warning in the build job's log, Skip output 'image' since it may contain secret, and the next job receives an empty string. Nothing fails. The build job was green, and nobody reads warnings on a green job.
The deploy step cut the tag off an empty string and got an empty string. Our chart still had the template that helm create writes, which uses the image tag or, if that is empty, the chart's appVersion. Somebody had set appVersion to 2.3.1 during a manual release in March and nobody had touched it since. Helm deployed 2.3.1, the pods were healthy, and the rollout succeeded.
The account ID is a repository variable again, because AWS does not treat it as a secret. The deploy step now refuses an empty tag with ${TAG:?} before it calls Helm. The chart no longer falls back to appVersion: the template uses required, so a release without a tag fails to render. And after every deploy the job compares the digest the pods are running with the digest the build job pushed.
Masking is meant to keep secrets out of places where people can read them. Between two jobs it also kept out a value we needed, and the only record of it was a warning.
– Sergey Shinder
Top comments (0)