Postmortem: We Leaked 10k User Records After Snyk 2.0 Missed a Vulnerability in Log4j 2.23 and Spring Boot 3.3

Date: October 26, 2024

Status: Resolved

Executive Summary

On October 12, 2024, our team discovered an unauthorized data exposure affecting 10,427 user records, including email addresses, hashed passwords, and partial billing information. The root cause was a critical remote code execution (RCE) vulnerability in a transitive dependency chain combining Log4j 2.23 and Spring Boot 3.3, which was not detected by Snyk 2.0’s vulnerability scanner during our CI/CD pipeline checks.

Incident Timeline

• October 8, 2024, 09:00 UTC: Snyk 2.0 scans our Spring Boot 3.3 application, returns 0 critical vulnerabilities.
• October 9, 2024, 14:30 UTC: Attackers exploit the unpatched Log4j-Spring Boot vulnerability to gain initial access via a malicious JNDI lookup payload.
• October 10, 2024, 02:15 UTC: Unauthorized database queries extract 10,427 user records over a 4-hour window.
• October 12, 2024, 11:00 UTC: Internal monitoring alerts on abnormal database read volume; incident response team is activated.
• October 12, 2024, 13:45 UTC: Vulnerability is identified, application is patched, and access is revoked.
• October 12, 2024, 18:00 UTC: All affected users are notified via email and in-app alerts.

Root Cause Analysis

Our investigation confirmed three contributing factors:

1. Snyk 2.0’s Vulnerability Database Gap: Snyk 2.0’s database did not include CVE-2024-XXXX (the Log4j 2.23/Spring Boot 3.3 RCE vulnerability) at the time of our scan, as the CVE was published 48 hours after our last pipeline run. Snyk 2.0 does not support retroactive scan updates for closed pipeline runs.
2. Transitive Dependency Blind Spot: The vulnerable Log4j 2.23 instance was a transitive dependency of a Spring Boot 3.3 starter package, which our manual dependency review process failed to flag.
3. Missing Runtime Protection: We did not have RASP (Runtime Application Self-Protection) enabled for this service, allowing the exploit to execute without triggering blocking rules.

Impact Assessment

• 10,427 user records exposed (emails, bcrypt-hashed passwords, last 4 digits of credit cards, billing zip codes).
• No evidence of data exfiltration beyond the 10k records; no financial fraud reported to date.
• 2-hour service downtime during patching.
• Temporary drop in user trust: 12% of active users deactivated accounts in the 72 hours post-notification.

Remediation Steps

1. Upgraded Log4j to 2.24.1 (patched version) and Spring Boot to 3.3.2, which removed the vulnerable transitive dependency.
2. Migrated to Snyk 3.0, which supports real-time vulnerability database updates and retroactive pipeline scan reruns.
3. Enabled RASP for all production services to block untrusted JNDI lookups and RCE payloads.
4. Implemented daily dependency audit reports, including transitive dependencies, sent to the security team.
5. Forced password resets for all affected users and added mandatory 2FA enrollment for high-risk accounts.

Lessons Learned

• Never rely on a single vulnerability scanner: We now run Snyk 3.0, OWASP Dependency Check, and GitHub Advanced Security scans in parallel.
• Transitive dependencies require explicit review: We added a step to our CI/CD pipeline to generate a full dependency tree report for every build.
• Runtime protection is non-negotiable: RASP and WAF rules must be updated in lockstep with new CVE disclosures.
• Incident response plans must account for zero-day vulnerabilities: We now conduct quarterly tabletop exercises simulating unpatched CVE exploitation.

Conclusion

This incident highlighted gaps in our DevSecOps pipeline that we have since addressed. We apologize to our users for the inconvenience and remain committed to transparency and continuous security improvement. For questions, contact our security team at [e-mail].