By establishing a signature chain with Cosign and Sigstore, you can automatically reject unsigned container images in your CI/CD pipeline. This guide explains the signing, verification, and policy-based rejection steps with real command examples and potential rollback scenarios.
Core Concepts of Cosign and Sigstore
As part of the sigstore project, Cosign secures container images with cryptographic signatures (GitHub - sigstore/cosign). The cosign sign command adds a signature layer to the image's image manifest (GitHub - sigstore/cosign). The cosign verify command verifies this signature; if there is no signature, the verification fails and the image is rejected. Sigstore offers keyless signing, which allows signatures to be generated via OIDC-based authentication without requiring private key management (GitHub - sigstore/cosign, GitLab Docs). The flow specified in the official documentation is as follows:
# Keyless signing (OIDC authentication)
cosign sign --yes ghcr.io/example/app:1.0.0
This command redirects the user to the OIDC provider via the browser; once authentication is complete, Cosign obtains a temporary certificate from the Fulcio CA and records the signature (GitHub - sigstore/cosign). To verify the same image:
cosign verify ghcr.io/example/app:1.0.0
If there is no signature, the cosign verify command returns an error and the CI stage stops. This behavior ensures that an unsigned image is automatically rejected.
Keyless Signing Workflow
The Mermaid diagram below shows the steps of OIDC-based keyless signing. The workflow consists of a sign stage, a verify stage, and a policy enforcement stage within the CI/CD pipeline.
In this flow, the policy enforcement stage can be implemented, for example, by checking the exit code of the cosign verify command inside a GitLab CI job. If the exit code is 0, the image is considered valid; if it is !=0, the pipeline fails and the image deployment is halted.
Policy Definition to Reject Unsigned Images
Here is a simple policy example in GitLab CI that checks the result of the cosign verify command inside a script block:
verify_image:
stage: verify
image: ghcr.io/sigstore/cosign:latest
script:
- cosign verify $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG || exit 1
The || exit 1 statement forces the job to fail when verification fails. This way, an unsigned image is automatically rejected within the CI. For the policy to be effective, the images in the registry must be signed; otherwise, the pipeline will stop every time (GitLab Docs).
Post-Verification Rollback Strategy
When signature verification fails, you need to roll back the faulty image. The most common method is to restore an older tag and re-verify it. Example steps:
# 1. Remove the failed image
docker rmi ghcr.io/example/app:1.0.0
# 2. Re-tag the previous version
docker pull ghcr.io/example/app:0.9.5
docker tag ghcr.io/example/app:0.9.5 ghcr.io/example/app:1.0.0
# 3. Re-sign the old version (keyless)
cosign sign --yes ghcr.io/example/app:1.0.0
# 4. Verify again
cosign verify ghcr.io/example/app:1.0.0
These steps allow you to resolve the issue by replacing a version that could not be deployed due to a missing signature with the previous stable version while maintaining the same tag. Since the cosign sign command re-signs the same tag, the signature chain is not broken; only the image content changes and a new signature layer is added.
Performance and Security Trade-off Analysis
Keyless signing reduces operational complexity by eliminating the burden of key management. However, the OIDC flow requires an HTTP request to be made to an external identity provider (e.g., GitHub, Google) every time a signature is generated. This can add network latency; a typical OIDC token retrieval takes a few seconds, but in high-latency environments, it can extend CI times. On the other hand, key-based signing (cosign sign --key <key.pem>) happens entirely locally, thus providing lower latency, but it requires secure storage and rotation of private keys.
Edge case: If the CI environment is air-gapped (offline), OIDC-based keyless signing will not work; in this case, you must use a pre-prepared PEM key with the --key option (GitHub - sigstore/cosign).
Example Scenario: Full Integration in CI/CD Pipeline
The following GitLab CI example brings together the build → sign → verify → deploy stages. This scenario only allows the deployment of images whose signatures are verified.
stages:
- build
- sign
- verify
- deploy
build_image:
stage: build
image: docker:stable
services:
- docker:dind
script:
- docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG .
- docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG
sign_image:
stage: sign
image: ghcr.io/sigstore/cosign:latest
script:
- cosign sign --yes $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG
verify_image:
stage: verify
image: ghcr.io/sigstore/cosign:latest
script:
- cosign verify $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG || exit 1
deploy_image:
stage: deploy
image: alpine/k8s:1.24.0
script:
- kubectl set image deployment/myapp myapp=$CI_REGISTRY_IMAGE:$CI_COMMIT_TAG
In this pipeline, if the verify_image step fails (for example, if the signature is missing), the exit 1 command prevents the subsequent deploy_image stage from running. Consequently, an unsigned image never makes it to production.
Multiple Signatures Support with Cosign
While a single signature might seem sufficient for container images, security teams often prefer to add multiple signature layers to increase the integrity of the signature chain. Cosign supports generating multiple signatures on the same image; each signature can be created with a different identity or key set (GitHub - sigstore/cosign). For example, while an OIDC-based keyless signature is automatically created in the CI environment using cosign sign --yes, a key-based signature can be added to the same image by another team member using the cosign sign --key ./team.key command. Both signatures are added to the image as separate signature layers and pushed to the registry.
During verification, Cosign checks by default whether at least one valid signature exists (GitHub - sigstore/cosign). However, in scenarios where all signatures must be valid, the cosign verify --certs option can be used. This flag verifies all signatures against the specified certificate files; if any signature is incorrect or missing, the verification fails. Multiple signature support makes the layers of the signature chain more transparent and facilitates coordination between teams at different security levels. Thus, even if one signature is invalid, other valid signatures guarantee the usability of the image, while in the case of a completely invalid signature, the CI/CD pipeline can be aborted.
Image Tagging Policy and Version Management
Image tags play a critical role in version tracking and deployment management within the CI/CD process. A well-defined tagging policy prevents version conflicts, accelerates rollback operations, and secures automatic updates. The most common practice is to follow semantic versioning (semver) principles: tags in the MAJOR.MINOR.PATCH format clearly indicate API changes, backward-incompatible updates, and bug fixes. To implement this policy inside GitLab CI, it is sufficient to add a step that checks whether the CI_COMMIT_TAG variable is in a valid semver format.
validate_tag:
stage: validate
image: alpine:3.18
script:
- echo "Checking: $CI_COMMIT_TAG"
- if ! echo "$CI_COMMIT_TAG" | grep -E '^[0-9]+\.[0-9]+\.[0-9]+$'; then
echo "Invalid semver tag: $CI_COMMIT_TAG";
exit 1;
fi
The example above verifies whether the tag is in the major.minor.patch format. If the tag is not valid, the pipeline is terminated, and the faulty deployment is prevented. Additionally, by using the git tag --list --sort=creatordate | tail -1 command, the latest tag can be retrieved to ensure that the new version is automatically one step ahead of the previous version. This way, every new build creates a consistent version history independent of the previous stable version, making a quick rollback possible in rollback scenarios.
Conclusion
Cosign and Sigstore provide a secure way to sign and verify container images. While keyless signing eliminates key management overhead, adding the cosign verify command as a policy step in your CI/CD pipeline ensures that unsigned images are automatically rejected. Considering latency risks and the limitations of air-gapped environments, combining key-based and keyless methods as needed is the most balanced approach. Rollback procedures allow you to quickly revert to an older, signed version after a failed signature verification. Integrating these steps into your CI/CD pipeline establishes a robust defense layer against supply-chain attacks.
Top comments (0)