Docker Hub's access token vulnerability: how signing let me verify my images

A critical bug in Docker Hub's access controls meant some personal access tokens, including the ones I use to publish compose-lint, a security linter for Docker Compose files, could have been used by someone else. Not the email you want on a Friday afternoon.

I actually got two of them on Friday, October 2, about an hour apart. The one I happened to open first was addressed to me as the organization owner. It was pretty vague and said there was no action for me to take, so I skimmed it and moved on. The other one, sent to my user account, was titled "Action Requested." When I had a chance to read both more closely, I realized my user was the one flagged, and Docker was planning to force-deactivate my two CI tokens that Monday, October 5.

On September 30, an outside researcher had reported a flaw in Docker Hub's identity and access management. Under certain conditions around converting a personal account to an organization, one user could take on another user's capabilities through a personal access token, which means pushing or pulling images as them. Docker said they fixed it within about four hours and hadn't seen any sign of it being exploited.

At that point I knew I needed to rotate my tokens and check that my published images hadn't been touched.

Why I could check

I started compose-lint because I couldn't find a Compose linter that met my needs or that I was happy with, and I'd been looking for a chance to build something open source. I also wanted a project where I could build with complete freedom and put everything I know about security engineering into it, including the release pipeline.

So every compose-lint image is signed with cosign, keyless, using GitHub Actions' OIDC identity. The signing certificate records the workflow and tag that produced the image, something like publish.yml@refs/tags/v0.34.0. Someone with Docker Hub push access could push an image, but they couldn't produce that signature without running my workflow in my repo on a release tag.

That meant I didn't have to take Docker's word for it. I could go check.

What I checked

I verified every tag against my workflow identity. Checking that an image has a valid signature isn't enough on its own, since anyone can sign an image. You have to pin who signed it.

This is where I hit something I hadn't thought about. Most of my images were signed by publish.yml, but one older release, 0.3.4, was signed by a different workflow I'd used to republish it. Different workflow, different identity in the certificate. I could still verify everything, but it meant using a regexp that allowed both workflows instead of one exact identity. It works, it's just not as clean as I'd like. If you're setting this up, it's worth keeping signing in one workflow, or at least knowing every identity you've ever signed with.

for tag in $(curl -s "https://hub.docker.com/v2/repositories/composelint/compose-lint/tags?page_size=100" \
  | jq -r '.results[].name'); do
  cosign verify "docker.io/composelint/compose-lint:${tag}" \
    --certificate-identity-regexp '^https://github\.com/tmatens/compose-lint/\.github/workflows/(publish|publish-channel)\.yml@refs/tags/v[0-9]+\.[0-9]+\.[0-9]+$' \
    --certificate-oidc-issuer https://token.actions.githubusercontent.com \
    > /dev/null && echo "ok   ${tag}" || echo "FAIL ${tag}"
done

All 42 images verified, and every one traced back to the CI run that built it.

A signature check only tells you what a tag points to today, though. Someone could point a tag at a bad image, wait for people to pull it, then point it back, and everything would still verify afterward. So I also pulled each tag's last_pushed time from the Docker Hub API and matched it against my release runs. Every push lined up with a real run. I also compared the full list of tags against my releases to make sure nothing extra had shown up and nothing was missing.

Everything came back clean. I rotated the tokens and split them up so each job only gets the one it needs: read-only for the nightly scans, push for releases, and a separate one for updating the Docker Hub description.

What I changed afterward

Going back through the pipeline, I found a couple of things I wanted to fix.

The release workflow signed whatever the tag pointed to right after pushing it. In the few seconds between push and sign, someone with write access to the registry could have moved the tag and gotten their image signed by my workflow. The window is small, but it's there. Now the workflow checks that the tag points to exactly what the build pushed, and stops before signing if it doesn't.

The verify step in CI was also using a loose regexp. It now checks the exact workflow with --certificate-identity "https://github.com/${GITHUB_WORKFLOW_REF}".

And signing doesn't help much if nobody verifies, so I added the command to the docs:

VERSION=0.34.0
cosign verify "docker.io/composelint/compose-lint:${VERSION}" \
  --certificate-identity "https://github.com/tmatens/compose-lint/.github/workflows/publish.yml@refs/tags/v${VERSION}" \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

The docs cover how to verify compose-lint images in more detail, including the SBOM and VEX attestations.

If you maintain images and aren't signing yet

On GitHub Actions it's about twenty lines. The job needs id-token: write so cosign can get a short-lived certificate tied to your workflow, and you sign by digest, not by tag:

permissions:
  contents: read
  id-token: write

steps:
  - uses: sigstore/cosign-installer@<pinned-sha>
  # build and push, capturing the digest
  - run: cosign sign --yes "docker.io/your-org/your-image@${DIGEST}"

You can see the sign and verify steps in compose-lint's release workflow, and the rest of the project is on GitHub.

Twenty lines is the open source on GitHub version. At a bigger org, signing is the easy part. Managing keys, keeping private repo names out of public transparency logs, deciding which builds to trust, and actually enforcing verification at deploy time is where it gets hard.

For a small project, though, it's cheap. And the next time a vendor tells you there's no evidence of exploitation, you'll be able to check for yourself.

Personal project, personal views.