A practical guide to building, scanning, signing, and deploying trusted container images.
Key Takeaways
In container-focused CI/CD, the container image is the primary release artifact. Build once, promote the same digest across multiple environments, and never rebuild separately for production.
Secure CI/CD for containers means protecting both the build process (workflows, credentials, base images) and the resulting images through scanning, SBOM generation, signing, and deployment policy enforcement.
Immutable image digests, SBOM and provenance attestations, and signature verification together create a traceable chain from Git commit to running container.
A problem arises when development teams treat container registries as generic storage instead of security boundaries. Registry controls, admission policies, and runtime monitoring close the gap.
At AppRecode, container CI/CD is treated as an artifact supply chain. This article provides a step-by-step blueprint for designing and auditing such pipelines.
Introduction: CI/CD Containers as an Artifact Supply Chain
Picture this: a team deploys a "patched" container built manually on a laptop, tagged api:latest, which bypasses all automated testing and contains a known OpenSSL vulnerability. Production breaks, the rollback takes hours, and nobody can identify which commit produced the image.
Containers eliminate the "it works on my machine" issue by packaging dependencies into immutable images, and Docker ensures consistent environments across development and production. But consistent packaging alone does not guarantee that what runs in the production environment is trusted, tested, or traceable.
Insecure CI/CD pipelines can still publish images containing known security vulnerabilities, outdated base layers, embedded secrets in image layers, mutable tags without digests, or artifacts that cannot be traced to a specific Git commit. CI/CD pipelines automate software delivery processes, but automation without controls just ships problems faster.
Implementing CI/CD pipelines with container technologies streamlines software delivery by ensuring consistency from development to production, but only when both the build process and the resulting container images are secured end to end.
What Is CI/CD for Containers?
CI/CD for containers is an automated delivery process where source code and dependencies are packaged into a container image, validated, stored in a registry, and promoted through environments.
Continuous integration involves frequently merging code changes into a central repository and triggering automated tests. In container pipelines, this means every change to the Git repository produces a tested candidate image. CI/CD promotes integrating code changes early to avoid complex merge conflicts.
Continuous delivery ensures that code changes are always in a deployable state through automated pipelines. A verified container image, identified by an immutable digest, stays ready for controlled deployment to staging and production.
Continuous deployment extends continuous delivery by automatically releasing every validated change to production once all gates pass.
The container image becomes the deployable artifact. Unlike traditional pipelines that may rebuild application binaries per environment, a CI/CD pipeline for containers builds once and promotes that exact digest across dev, QA, staging environment, and production. CI/CD pipelines enable faster, more reliable application delivery when this principle is followed. Docker is one ecosystem for building docker containers, while "container image" refers to the broader OCI-compatible concept.
How Container CI/CD Differs from Traditional Pipelines
Instead of outputting a binary or archive, the build stage creates an OCI-compatible image containing application code, runtime, OS packages, and metadata. Unlike traditional virtual machines or bare-metal deploys, lightweight containers allow for faster deployment and efficient resource utilization in CI/CD pipelines.
Base images like ubuntu:22.04 or distroless images become part of the software supply chain. They must be versioned, scanned, and updated like any other dependency in the development process.
A container registry (Docker Hub, Amazon ECR, Harbor) is now a critical delivery component. Tags like :latest are mutable pointers that can change without warning. Digests (@sha256:...) are immutable. Deploying by digest guarantees that staging and production run the same bytes.
Container images are version-controlled artifacts, simplifying the rollback process in case of deployment issues. Containers also support microservices architecture, enabling independent CI/CD pipelines for different services, so multiple teams can ship through a single pipeline pattern without blocking each other.
Image scanning must cover both OS-level packages and application dependencies, unlike legacy pipelines focused only on app libraries. Containers facilitate immutable infrastructure, simplifying software updates and rollbacks. Environment-specific configuration and secrets should remain outside the image, supplied via manifests or secret managers to maintain consistent environments while keeping runtime configuration separate.
The Complete CI/CD Pipeline for Containers
A CI/CD pipeline typically includes build, test, and deployment stages. For containers, the flow expands: source commit → source validation → dependency checks → container build → image tests → vulnerability and secret scanning → SBOM and provenance generation → image signing → push to registry → deploy to staging → integration testing and smoke tests → policy verification → production promotion → runtime monitoring.
Each stage operates on the same container image digest. Using the same container image throughout the CI/CD pipeline reduces deployment errors and increases release confidence. Automated CI/CD pipelines reduce manual intervention in software delivery, and containerized CI/CD workflows often involve committing code, building and containerizing, testing, and deploying.
Using container images improves CI/CD pipeline efficiency and speeds up the build-test-deploy cycle. The build environment should be deterministic, with pinned tooling images rather than mutable machine images.
Source validation before building
Running source-level checks before any Docker build saves compute time and shortens feedback loops. Detecting code vulnerabilities during the build stage improves security. Testing should occur at every stage of the CI/CD process to catch bugs immediately.
Typical checks include:
Linting and code formatting
Unit tests against the source code
Static analysis (SAST) and software composition analysis (SCA)
Secret detection on the code repository to prevent committing keys into images
Kubernetes manifest validation with tools like kube-linter or Helm linting
Dockerfile checks via hadolint to enforce best practices
Run these inexpensive checks before starting a resource-intensive image build process.
Building a secure, reproducible container image
The build process should be deterministic. Pin base images by digest where possible, use language lockfiles (package-lock.json, poetry.lock), and specify explicit dependency versions so rebuilding from the same commit produces the same Docker image.
Multi stage builds separate build-time dependencies (compilers, test frameworks) from the final runtime image. Use .dockerignore to exclude tests, docs, and local configs from the build context. This is how you build container images that are lean and auditable.
Choose trusted base images from known sources. Establish non-root users in Dockerfiles where applications allow it and avoid unnecessary package managers or debugging tools in the final layer. Smaller images improve pull times and may reduce exposure, but still require vulnerability scanning. An automated build triggered by new code changes ensures consistency.
Testing the built image, not just the source
Testing only source code is insufficient. The exact container image intended for deployment must be started and tested. Automated testing in CI/CD pipelines improves code quality, and automated testing can include security checks for container images.
Start the built Docker image (via docker commands like docker run or docker compose) and run:
Health endpoint checks
Startup behavior and environment-variable handling
Exposed port validation
Read-only filesystem compatibility where applicable
Verification that processes do not require root privileges
Integration testing with ephemeral databases or queues
Validate Kubernetes manifests or Helm charts alongside the image. When tests pass, the image is a candidate for promotion. Running multiple test suites against the actual container catches issues that source-only testing misses.
How to Secure Containers in CI/CD Pipelines
Container security protects applications from potential risks and security threats throughout the development lifecycle. A strong posture combines workflow integrity, least-privilege credentials, trusted base images, multi-layer scanning, and policy enforcement.
In AppRecode's container and CI/CD work, an important first step is tracing the full path from Git commit and base image through CI jobs and the artifact repository to what runs in production clusters. Organizations can improve CI/CD pipelines by implementing automated tests and vulnerability scanning before deployment.
Secure the source and workflow definitions
Protect branches like main with required pull-request reviews and status checks. Workflow configuration (GitHub Actions, GitLab CI, Jenkinsfiles) should be treated as code in your version control system, requiring review before changes that affect deployments or secrets.
Isolate untrusted pull requests from forks by preventing them from accessing production secrets. Pin third-party CI actions to specific commit hashes instead of floating tags. Split automated workflows so that build pipelines and deployment pipelines are separate, with stricter permissions around anything touching production.
A malicious workflow change can compromise the entire container supply chain even when source control shows clean application code. Refer to GitHub's deployment security guidance for hardening workflows with OIDC and environment-level protections.
Protect pipeline credentials and secrets
Use the CI platform's encrypted secret store (GitHub Actions secrets, GitLab masked variables, Jenkins credentials). Short-lived credentials and workload identity via OpenID Connect reduce the blast radius compared to long-lived static keys.
Separate credentials by environment: distinct identities for build-time registry push, staging deploy, and production deploy, each with least-privilege access. Never embed cloud keys, registry passwords, SSH keys, TLS private keys, .env files, or Kubernetes service-account tokens inside Docker images, build arguments, or base images. Deleting a secret in a later Docker layer does not fully remove it from earlier layers.
Rotate credentials regularly and revoke promptly on incidents or personnel changes.
Use trusted and controlled base images
Base images define much of the OS and runtime behavior of containerized applications. Rely on official or internally approved images mirrored into a private registry.
Pin base images by digest (e.g., FROM ubuntu@sha256:...) to ensure CI/CD builds are reproducible. Maintain an internal catalog of allowed base images with owners responsible for tracking upstream advisories and triggering rebuilds when critical CVEs appear. Avoid pulling unknown images directly from public registries for production containerized workloads. Official images have better maintenance but still require scanning and update policies.
Scan dependencies, container images, and manifests
Differentiate between scanning source dependencies (reading requirements.txt, pom.xml) and scanning the built Docker image for OS and language packages actually present in layers. Regularly scanning containers for vulnerabilities is a best practice.
Run security scans, secret detection, and malware checks using tools like Trivy, Grype, Docker Scout, Snyk Container, Prisma Cloud, Aqua, or Anchore. Include Kubernetes manifest scanning for privileged containers or missing resource limits.
Scanners differ in databases and update cadence. Define policies that fail builds on exploitable critical issues rather than on every CVE. Create time-boxed vulnerability exceptions with owners and justification. Integrating these controls is a common topic in AppRecode's DevSecOps services.
SBOM, Provenance, and Container Image Signing
SBOM, build provenance, and signing are three distinct mechanisms improving traceability of ci/cd docker containers. Together they answer: what is inside this image, how it was built, and who approved this specific digest.
Software Bill of Materials (SBOM)
An SBOM is a structured inventory (SPDX, CycloneDX) of components, libraries, and packages contained in a given container image. SBOMs support vulnerability response and license analysis: when a new CVE is disclosed, development teams can quickly identify affected images.
An SBOM does not guarantee security or completeness. It is as trustworthy as the build process and tooling that produced it. Docker's BuildKit-based workflows can generate SBOM attestations in CI. Refer to Docker's SBOM and provenance documentation for implementation details. Store SBOMs alongside images in registries, queryable by image digest.
Build provenance
Provenance captures how an artifact was built: the shared repository and commit, build workflow, CI system identity, timestamps, base images, and dependencies. It helps prove a given image digest came from a specific CI/CD pipeline rather than a manual or untrusted build.
The SLSA framework models increasing supply-chain integrity levels. Generating provenance does not automatically make a pipeline SLSA compliant. Provenance needs protection itself (signing, access controls) and should be linked to the image digest in the artifact repository.
Container image signing
Image signing creates a cryptographic binding between an image digest and a signing identity. Tools like Cosign can sign Docker images in CI using key-based or keyless OIDC signing and store signatures alongside images in the registry. See Sigstore's container signing documentation for key-based and keyless workflows.
Signing does not indicate vulnerability status. It proves identity and integrity. Use dedicated signing keys with rotation, restricted access, and optionally KMS services to avoid long-lived local keys.
Registry Security and Image Promotion
The container registry is part of the trusted delivery path. If attackers can push or modify images there, they bypass CI controls entirely.
Use private repositories with least-privilege access. CI build jobs push to specific repos; production environments pull only allowed images. Enable immutable tags where available, but deploy by digest. Versioned container images allow for quick rollbacks during deployment.
Separate registries or repositories by environment (e.g., app-staging vs app-prod). Promote by copying the same digest that passed staging, not by rebuilding. Enable audit logs, built-in scanning, encryption, and retention rules that respect rollback needs.
Secure Deployment to Kubernetes
Kubernetes manages containerized workloads and enhances security practices. It facilitates automation and declarative configuration, making it the most common target for CI/CD pipelines that deploy containers.
Store manifests (YAML, Kustomize, Helm charts) in a Git repository alongside or linked to application code. A GitOps-style approach using Argo CD or Flux lets a controller in the Kubernetes cluster reconcile desired state, improving auditability. Kubernetes supports zero downtime deployments and instant rollbacks, and Kubernetes deployments can be automated using CI/CD tools. Terraform automates Kubernetes cluster provisioning in CI/CD pipelines.
Apply runtime security controls: namespaces, service accounts, RBAC, resource limits, security contexts (runAsNonRoot, dropped capabilities, seccomp profiles), and network policies. Use admission policies (Kyverno, OPA Gatekeeper, Sigstore policy controllers) to deny deployments using unapproved registries, unsigned images, missing provenance, or privileged containers. Kubernetes enables efficient scaling of containerized applications, and managing container orchestration introduces complexity that requires specialized skills and configuration. For aligning Kubernetes deployment models with CI/CD and security policies, teams sometimes engage container orchestration consulting support.
Example Container CI/CD Architecture
Pull-request pipeline: Checkout code, run linting and unit tests, secret detection, Dockerfile checks, temporary image build, smoke tests, non-blocking image scans. No production credentials, no push to production registries. Operations teams and development teams both benefit from fast feedback here.
Main-branch pipeline: Repeat critical checks, build the candidate image with BuildKit, tag with commit SHA, generate SBOM and provenance, scan the final image, sign it, push to a controlled registry, and deploy that exact digest to a staging environment for integration tests. This is where the release process becomes traceable.
Production deployment pipeline: Manual approval or automated policy-based promotion, verification of image digest, signature, and provenance. Deploy via Kubernetes with progressive rollout (canary or blue-green). Health verification, observability checks, and rollback to the last known good digest on failure. Kubernetes manages containerized workloads and services across Kubernetes nodes, enabling faster deployments. The pipeline should not rebuild the application after final approval.
Docker CI/CD Pipeline Example (Illustrative)
Below is a simplified GitHub Actions-style pseudocode showing how a CI/CD with containers workflow ties commits to digests:
steps:
- checkout code from repository
- authenticate to registry via OIDC (short-lived token)
- build image: Docker buildx build --sbom=true --provenance=mode=max -t ghcr.io/org/my-api-service:$COMMIT_SHA .
- run container tests: docker run ghcr.io/org/my-api-service:$COMMIT_SHA /healthcheck
- scan image: trivy image ghcr.io/org/my-api-service:$COMMIT_SHA
- push image (immutable tag by commit SHA)
- sign image digest: cosign sign ghcr.io/org/my-api-service@sha256:...
- deploy by digest to staging
- verify deployment health
This example is illustrative only. Environment-specific controls, secret management, and the computing environment configuration must be adapted per organization. Jenkins, GitHub Actions, and CircleCI all automate CI/CD processes with similar stage patterns. Never store credentials directly in workflow files.
Common CI/CD Container Mistakes and Better Practices
| Mistake | Risk | Better Practice |
|---|---|---|
| Deploying only by :latest tag | Image content can change silently; ubuntu latest steps may differ between pulls | Deploy by immutable digest |
| Rebuilding separately for staging and production | Untested code reaches prod | Promote the same digest |
| Embedding secrets in image layers | Secrets are retrievable from layer history | Use external secret managers |
| Running all containers as root | Expanded attack surface | Use non-root users in Dockerfiles |
| Pulling unverified public base images | Supply-chain risk from unknown sources | Mirror and vet images internally |
| Ignoring base-image updates | Known CVEs persist in production | Schedule periodic rebuilds |
| Scanning only source code, not the final image | Misses OS-level vulnerabilities | Scan the built Docker image |
| Scanning without enforcing policy | Findings are informational noise | Fail builds on critical issues |
| Giving CI permanent cloud admin credentials | Blast radius of compromise is total | Use short-lived, scoped credentials |
| Signing images but not verifying at deploy | Signing provides no value without enforcement | Enforce verification via admission control |
| Generating SBOMs but never storing them | Compliance gaps during audits | Store alongside images, query by digest |
| Publishing images before tests pass | Broken images reach registries | Gate pushes on test and scan completion |
Containerization improves deployment speed and flexibility in software delivery, but only when these mistakes are addressed. Treat your registry as a security boundary, not a storage bucket.
Tools for Container CI/CD (Comparison Table)
| Tool or Category | Role in the Pipeline | Best Suited For | Main Consideration |
|---|---|---|---|
| Docker BuildKit / Buildx | Image build, multi-stage builds, SBOM/provenance attestation | Reproducible, optimized builds | Requires flag configuration for attestations |
| GitHub Actions / GitLab CI / Jenkins | Workflow orchestration across build, test, scan, deploy | Teams with existing CI platforms | Securing workflow definitions and secrets |
| Argo CD / Flux | GitOps-based deployment and reconciliation | Kubernetes-native delivery | Requires Git-driven manifest management |
| Trivy / Grype | Image and dependency scanning | Shift-left vulnerability detection | Different vulnerability databases; tune thresholds |
| Cosign (Sigstore) | Image signing, verification, attestations | Supply-chain integrity enforcement | Key management; registry compatibility |
| Kyverno / OPA Gatekeeper | Kubernetes admission policy enforcement | Pre-deployment policy gates | Policy language learning curve |
| Harbor / Managed Cloud Registry | Image storage, access control, scanning, replication | Central governance and compliance | Cost; feature differences across providers |
These orchestration tools and scanning platforms solve different problems and are not direct replacements for each other. Deploy applications using the combination that fits your orchestration platform and development teams.
Container CI/CD Metrics That Indicate Readiness
Track these to understand whether your pipeline is production-ready:
Delivery performance: Median and p95 pipeline duration, queue time, build success rate, image build time, cache hit rate, deployment frequency, failed deployment rate, mean time to restore
Security and compliance: Vulnerability exception count, age of unresolved critical findings, percentage of images with SBOMs, percentage with verified provenance, percentage of production deployments using verified signatures and immutable digests
Operational health: Image size trend, time to rebuild after base-image CVEs, rollback rate, application performance impact of new version releases
Avoid universal benchmark numbers. Establish a baseline and measure improvement over time as your pipeline and container platform mature. These metrics help improve efficiency across the development lifecycle and give operations teams clear signals.
When Your Container Pipeline Needs an Audit
Warning signs that your container CI/CD pipeline needs review:
Nobody can prove which Git commit produced a running container
Production uses images never tested in the staging environment
CI has long-lived cloud administrator credentials
Widespread use of mutable tags in production
Images contain unknown packages or untracked base images
SBOMs are generated but never stored or inspected
Vulnerability exceptions have no owner or expiration
A single engineer controls signing keys
Build failures are routinely rerun without investigation
Dockerfiles contain environment-specific secrets
An audit should examine the complete artifact path: the code repository, CI workflows, build environment, registry, Kubernetes manifests, and runtime policies. Organizations sometimes bring in external expertise via CI/CD consulting services to perform end-to-end assessments. Containers bundle application code and dependencies, but that bundle must be traceable.
Final Checklist for Secure CI/CD Pipelines for Containers
[ ] Protected branches with reviewed pipeline changes
[ ] Trusted, pinned base images (by digest)
[ ] Multi stage builds with minimal runtime images
[ ] No embedded secrets in Dockerfiles or images
[ ] Non-root execution where feasible
[ ] Source and image scanning integrated into CI
[ ] Tests run against the final built image
[ ] SBOM and provenance generated and stored with artifacts
[ ] Image signing with verification policies at deployment stage
[ ] Private registries with immutable digests for deployments
[ ] Controlled promotion using the same digest across multiple environments
[ ] Short-lived deployment credentials via OIDC or workload identity
[ ] Kubernetes admission policies checked into version control
[ ] Runtime monitoring tools and log aggregation active
[ ] Documented rollback strategy using known good digests
[ ] Clear ownership for pipelines and containerized services
Conclusion: Making CI/CD and Containers Verifiable
CI/CD with containers only improves reliability when the entire path from commit, through build, SBOM, signing, registry, and Kubernetes deployment is treated as a verifiable chain. Containers ensure consistent environments across development and production, but consistency without integrity is incomplete. Kubernetes manages containerized workloads in CI/CD pipelines and the Kubernetes control plane enforces policies, but Kubernetes play a role only when the images it runs are already trustworthy.
A mature pipeline should answer:
which commit and base image produced this container?
What does it contain?
Was it tested and scanned?
Who signed it?
Which exact digest runs in production?
Can the organization roll back safely?
Secure CI/CD pipeline practices do not require exotic tooling. They require consistent implementation: immutable artifacts, strong identity, least-privilege access, and policy-driven promotion across isolated environments.
AppRecode helps engineering teams assess and improve container platforms, CI/CD pipelines, DevSecOps controls, and Kubernetes delivery workflows.
FAQ about CI/CD for Containers
How often should we rebuild container images if nothing in our code changed?
Schedule periodic rebuilds (weekly or when base images receive security updates) even without code changes, so CI/CD pipelines pick up patched layers and refresh SBOMs. Use dependency and base-image scanning to trigger rebuilds when high-severity vulnerabilities appear rather than relying purely on a fixed calendar. This approach uses the open-source platform tooling you already have.
Can we safely use Docker-in-Docker (DinD) in CI for container builds?
Docker-in-Docker is widely used but can increase complexity and, in privileged modes, expand the attack surface. Evaluate whether rootless build options or daemonless builders (e.g., BuildKit with Docker buildx) are feasible. If DinD is used, scope it to isolated runners, restrict network and registry access, and apply the same scanning and signing practices to resulting images.
How do we handle multi-tenant teams sharing one Kubernetes cluster in CI/CD?
Separate workloads by namespace with dedicated service accounts and RBAC. Limit which CI/CD pipelines can deploy to which namespaces and registries. Use admission policies to enforce per-team constraints (allowed image registries, labels, capabilities) and separate observability dashboards per tenant. Multiple containers from different teams can coexist when one or more containers per namespace are governed by clear policies.
Is it necessary to sign images for internal, non-production environments?
Signing pre-production images helps test the policy and tooling that will later protect production, and supports traceability for internal incident response. Start by enforcing signature checks only in higher environments. Plan for a future state where at least all staging and production images are signed and verified, with lower environments gradually adopting the same controls as pipelines in the software development lifecycle mature.
Top comments (0)