Postmortem: How a Docker 26.0 Image Vulnerability Led to a Production Breach

Published: October 12, 2024 | Incident ID: INC-2024-0927 | Severity: Critical

Executive Summary

On September 27, 2024, our security team detected unauthorized access to our production Kubernetes cluster. The root cause was traced to a hardcoded CI/CD token exposed in a container image built with Docker Engine 26.0.0, which contained a known vulnerability (CVE-2024-3092) in its multi-stage build process. The token was extracted by an external attacker, who used it to deploy cryptomining workloads and exfiltrate 1.2TB of customer metadata. This postmortem details the timeline, root causes, remediation, and lessons learned from the incident.

Timeline of Events

• September 18, 2024, 09:00 UTC: Engineering team upgrades all CI runner hosts to Docker Engine 26.0.0 to adopt new build cache features.
• September 20, 2024, 14:30 UTC: First production image is built with Docker 26.0.0 using multi-stage build process; CVE-2024-3092 causes CI token to be persisted in final image layer.
• September 22, 2024, 11:15 UTC: Misconfigured registry policy pushes the vulnerable image to a public Amazon ECR repository instead of the private internal registry.
• September 25, 2024, 03:45 UTC: External attacker scans public ECR repositories, identifies the vulnerable image, and extracts the hardcoded CI token using docker save and layer inspection.
• September 27, 2024, 08:20 UTC: Security team detects anomalous pod creation in production EKS cluster; attacker has used the CI token to create privileged pods for cryptomining.
• September 27, 2024, 08:45 UTC: Incident response team rotates all CI/CD tokens, isolates the compromised EKS node group, and revokes the attacker's access.
• September 28, 2024, 17:00 UTC: All vulnerable images are purged from registries, Docker Engine is downgraded to 25.0.5 (patched version), and multi-stage build policies are updated.

Root Cause Analysis

Three contributing factors led to the breach:

1. Unpatched Docker 26.0.0 Vulnerability: CVE-2024-3092 is a flaw in Docker 26.0.0's handling of multi-stage builds where COPY --from=build-stage /tmp/ci-token /root/.ci-token instructions (used to pass CI credentials to build stages) were not properly cleaned up in final image layers. Docker fixed this in 26.0.1, but our team skipped the patch due to "stability concerns".
2. Registry Misconfiguration: A Terraform typo in our registry policy set the ECR repository's public-access flag to true instead of false, exposing all images in the repository to the public internet.
3. Overprivileged CI Token: The CI token used in builds had eks:DescribeCluster and eks:CreatePod permissions for all environments, including production, violating the principle of least privilege.

Impact Assessment

• 1.2TB of customer metadata (email addresses, subscription tiers, last login timestamps) was exfiltrated to an external S3 bucket.
• Cryptomining workloads consumed $42,000 in unplanned AWS EC2 costs over 48 hours.
• Production API latency increased by 300% during the attack, causing a 2-hour partial outage for 12% of customers.
• Regulatory notification requirements triggered under GDPR and CCPA, with potential fines up to €4M.

Remediation Steps Taken

1. Downgraded all Docker Engine instances to 25.0.5 (LTS patched version) and implemented automated patch management for all container tooling.
2. Audited all ECR repositories and fixed public access misconfigurations; implemented mandatory registry access reviews via CI checks.
3. Rotated all CI/CD tokens, scoped tokens to only required environments (e.g., staging tokens cannot access production), and migrated to short-lived OIDC tokens for CI builds.
4. Deployed image scanning in CI pipelines using Trivy to block images with known vulnerabilities or hardcoded secrets.
5. Added runtime security monitoring with Falco to detect anomalous pod creation or credential use in Kubernetes clusters.

Lessons Learned

• Never skip patching critical infrastructure components like container runtimes, even for minor version upgrades. CVE-2024-3092 had a CVSS score of 8.8 (High) and was publicly disclosed 14 days before our incident.
• Registry configurations must be validated via automated tests (e.g., Checkov, Terratest) to prevent public exposure of private images.
• CI/CD tokens must follow least privilege: no token should have cross-environment access, and tokens should be short-lived with automatic rotation.
• Image scanning must include secret detection, not just vulnerability scanning, to catch hardcoded credentials before images are pushed to registries.

Conclusion

This incident highlights the cascading risks of unpatched container tooling, misconfigured infrastructure, and overprivileged credentials. By adopting automated patching, strict least privilege, and end-to-end image security checks, we have reduced our container attack surface by 72% in the 30 days since the breach. We will continue to publish postmortems for all critical incidents to maintain transparency with our customers and the broader DevOps community.