DEV Community: Whitney

Lessons from Log4Shell: Building a CRA-Ready Log4j

Whitney — Wed, 06 May 2026 22:01:56 +0000

By: Piotr P. Karwasz, VP Logging, Apache Software Foundation

The disclosure of Log4Shell (CVE-2021-44228) on December 9, 2021 did not just expose a vulnerability: it exposed a way of building software that was no longer fit for purpose, and it helped bring the European Cyber Resilience Act into being.

I recently hosted a session for the Open Regulatory Compliance community’s CRA Monday series to tell the story from the inside: what the Apache Logging team actually did in the years after Log4Shell to rebuild the project as something CRA-ready.

This blog recaps and expands upon that session; you can also watch the recording or view the slides.

A Wake-Up Call for the Software Ecosystem

Log4Shell’s impact was unprecedented in scale. Apache Log4j is embedded so deeply across the software ecosystem that the vulnerability propagated almost everywhere at once and most organizations had no idea where they were exposed. The rush to assess risk revealed a fundamental problem: few teams maintained a reliable Software Bill of Materials (SBOM), and the question “are we affected?” had no quick answer.

The scramble had at least one useful side effect: it pushed many teams to finally migrate from Log4j 1, already end-of-life since 2015, to Log4j 2.

Lessons from Log4j perspective

Since Log4j is mostly consumed as a dependency rather than built upon, the lessons the Apache Software Foundation’s Logging Services team drew from Log4Shell were different from those of the broader ecosystem. The problems were not about visibility into our own dependencies, but about the state of the project itself:

Documentation was hard to navigate, with many features either undocumented or described only in terms a new contributor could not act on.
The release process was antiquated, understood by only a handful of people, and run on personal hardware: a single point of failure that nobody had reason to address until a crisis made it unavoidable.
Builds were slow and tests were flaky, meaning a failure late in a multi-hour process sent you back to the beginning.

None of these were unique to Log4j. Log4Shell made them impossible to ignore, and addressing them put us on a path that anticipates much of what the CRA now asks of software maintainers.

Documentation: from maintainer knowledge to public record

Logging is not always safe. There are real security concerns: CRLF injection from unstructured logging; sensitive information leaking into debug output; and injection of Log4j formatting patterns through user-supplied strings. Before Log4Shell, much of this knowledge lived in the heads of a few maintainers: not written down, not discoverable, and not actionable for the thousands of teams depending on the library.

We rewrote the documentation website from scratch. The goal was to turn that private knowledge base into a public record by:

Covering security best practices and an explicit security model
Providing reference documentation generated directly from code, so it stays in sync as the library evolves
Making Log4j’s versioning policy and support status explicit and visible, both required for CRA attestationsMoving the issue tracker from JIRA closer to the code in GitHub Issues
Mirroring some discussions on both GitHub
Discussions and mailing lists

The results were measurable: more documentation pull requests, more site visits, a useful proxy for coverage and clarity, and noticeably better answers from LLMs trained on our new content.

Release process: from manual to reproducible

In December 2021, Log4j’s tests ran on a Jenkins instance, binaries were built on maintainer machines, signing was manual, and builds were not reproducible. A full binary and site build literally took hours. This was not unusual for open source projects, but it created real risks around build integrity, and it was clearly not sustainable.

By September 2024 we had migrated to GitHub Actions, achieved reproducible builds signed by a CI GPG key only known to ASF admins, parallelized tests, and reduced the build-and-deploy cycle to around 30 minutes.

Currently:

The CI pipeline now automatically stages releases up to the voting phase: the first project in The ASF to do this.
We are working on integration with Apache Trusted Releases, which will bring automation to the voting and publishing steps as well.
We are working on full-SLSA build and source attestations, which will make us one of the first ASF projects to achieve this. This includes SLSA source level 4, requiring a non-author review for every commit: a critical guarantee for a project at the center of the most significant supply-chain incident in recent memory.

Machine-readable metadata: SBOMs, VEX, and beyond

One of the most concrete CRA requirements is the expectation that software comes with machine-readable security information. We now publish CycloneDX SBOMs to Maven Central, which references a Vulnerability Disclosure Report, a machine-readable version of our CVE list, on our website. This gives downstream users a complete, well-curated source of vulnerability information, unaffected by the data loss that public vulnerability databases sometimes introduce when converting between formats. It is also open to improvements by contributors.

The next step is Vulnerability Exploitability eXchange (VEX) statements, generated automatically through an open source toolset we are developing with OpenRefactory. The system combines:

An AI-backed Root Cause Service that identifies vulnerable methods
A Call Graph Service that maps per-component call graphs
A VEX Generation Service that determines the maximum reachable path and generates enriched VEX statements, which we call VEXplanations

We are currently testing this within Apache Solr and plan to extend it to Log4j and Commons. The goal is to give downstream users a machine-readable guarantee of no known exploitable vulnerabilities, assessed automatically rather than by hand.

We are also planning to support Common Lifecycle Enumeration (ECMA-428), the machine-readable equivalent of our supported versions list, generated ASF-wide through Apache Trusted Releases.

Vulnerability handling: from ad hoc to structured

Since Log4Shell, the Logging Services team has put in place several structural improvements to vulnerability handling:

A dedicated reporting address (security@logging.apache.org) separate from the general ASF security team
An explicit threat model that clearly separates the security responsibilities of Log4j from those of its consumers
A bug bounty program hosted on YesWeHack, funded by the Sovereign Tech Resilience program

Since July 2024, the program has received 140 reports across Log4cxx, Log4j, and Log4net, yielding 10 CVEs.

At the architectural level, Log4j 3 addresses the attack surface problem directly. Log4j 2 was built for a pre-Maven world and shipped as a monolithic core with many optional dependencies bundled in. Log4j 3 modularises most of that core, so each module only pulls in what it actually needs. Smaller surface area means fewer things that can go wrong.

Community sustainability: the hardest and unsolved problem

All of the above is meaningless if the project burns out. And that risk is real. Log4j currently has two active maintainers doing most of the work. Log4cxx and Log4net are in a similar position. This is not a criticism of the community. It is the structural reality of most open source projects, and it is the problem that CRA compliance pressure will make worse if it is not addressed alongside the technical work.

We are exploring two directions:

The Open Source Economy initiative: offering consulting and compliance attestations as a funded model, where fees support maintainers, infrastructure, and upstream dependencies.
ECMA TC-54’s CONTRIBUTING.yaml specification: a machine-readable format for describing a project’s maintenance status, contribution needs, and support expectations, so that users and organisations can understand what they are depending on.

There is also a cultural dimension. Many open source communities grew up in an era when maintainers could commit directly to the repository and build software on their own machines. Shifting to a model where maintainers review all contributions and CI builds everything is the right move for security, but it is genuinely less fun, and communities resist it. Making that transition while keeping contributors engaged is one of the real challenges of building a CRA-ready project.

What CRA readiness actually looks like

The CRA introduces a set of “due diligence” requirements for manufacturers and voluntary attestations for OSS projects. What I hope this post makes clear is that meeting them is not a compliance exercise you bolt on at the end. It is the output of years of work on documentation, build integrity, machine-readable metadata, vulnerability processes, and community health.

The good news is that the direction was right before the regulation arrived. Log4Shell forced hard questions that led, eventually, to a project that is genuinely more secure, more transparent, and more sustainable than it was in 2021. The CRA gives that work a formal framework and, ideally, the organisational backing to fund it. For other open source projects facing the same pressures, that is perhaps the most useful takeaway: the path to CRA readiness and the path to a healthier, more sustainable project are the same path.

Get involved

The path to a CRA-ready ecosystem is not walked by one project. If any of this matters to you, here is where to start:

Contribute to Log4j directly. Code, documentation, and review all count. Start at logging.apache.org, or report security issues to security@logging.apache.org.
Help across the ASF. The Contributing to Apache Security guide is the way in.
Shape OSS regulatory response. The Open Regulatory Compliance Working Group is where CRA implementation for open source is being worked out in public.
Define what sustainable maintenance means. ECMA TC54 TG4 is building the machine-readable standards for project health and support.
Secure the supply chain. The SLSA Community is advancing the build-integrity framework Log4j will soon use.

Apache Geode 2.0: Revival, Reinvention, and the Road Ahead

Whitney — Tue, 03 Mar 2026 18:22:06 +0000

Originally published at https://news.apache.org/foundation/entry/apache-geode-2-0-revival-reinvention-and-the-road-ahead

By: Jinwoo Hwang
Lead Developer, Project Lead, and Release Manager, Apache Geode 2.0
https://JinwooHwang.com

This post is divided into three parts. Part I explains why Apache Geode 2.0 matters. Part II walks through how it was modernized. Part III looks ahead—what we learned, what changed, and how you can help shape what comes next.

Apache Geode 2.0, Part I: The Revival of Apache Geode

Legacy, purpose, and the moment a terminated project came back to life

Apache Geode 2.0 is not just a new release—it is a statement of intent. Before diving into code, frameworks, or version numbers, it is worth understanding why this release exists at all.

This story begins with a platform that once powered mission-critical systems, drifted toward obsolescence, and then found new life through conviction, persistence, and community. This first section sets the stage: the purpose behind the work, the legacy of Apache Geode, and the moment when a seemingly-finished project began its comeback.

I have the privilege of serving as a Committer and Release Manager for Apache Geode 2.0. This release represents one of the most ambitious modernization efforts in the project’s history. For me, it has been more than engineering work—it has been a journey shaped by purpose, responsibility, and a deep belief in the value of our shared open source legacy.

When I stepped into these roles, it became clear that Apache Geode could not survive on incremental change. The Java ecosystem had moved forward—Jakarta EE, Spring, Jetty, Tomcat, and security practices had all evolved—while Geode had effectively stood still. At the same time, unpatched vulnerabilities threatened user trust. To remain relevant, Geode needed a fundamental reset: technically, architecturally, and culturally.

Why I Took on This Project

I do not earn additional compensation for the nights and weekends spent on Apache Geode. I am grateful to my employer for supporting my open source contributions, but this work did not replace my day job. I carried the same responsibilities while taking on this effort.

The reason I stayed with it is simple: purpose. I believe in this project and the community behind it. Friedrich Nietzsche famously wrote, “He who has a why to live can bear almost any how,” an idea later echoed by Viktor Frankl in his work on meaning and resilience. That sense of why—of keeping something valuable alive—carried me through the hardest moments of this journey.

With that context, it is worth stepping back and answering a foundational question.

What Exactly Is Apache Geode?

Apache Geode is a distributed, in‑memory data management platform designed for low‑latency, scalable, and consistent data access. It is built for systems that must react in real time, handle massive data volumes, and remain operational under failure. Data is dynamically partitioned or replicated across a cluster, with built‑in fault tolerance and optional persistence to disk.

As modern applications have shifted toward real-time analytics, event-driven architectures, and microservices, latency has become a central architectural constraint. Disk-backed storage systems, while durable and cost-efficient, often introduce millisecond-scale access times that are incompatible with sub-millisecond response requirements. In-memory data platforms address this gap by keeping active or frequently-accessed data in RAM, significantly reducing access latency and increasing throughput. This approach is particularly important in domains such as financial services, telecommunications, e-commerce, and IoT, where responsiveness, scale, and availability directly influence user experience and operational outcomes.

At its core, Geode aggregates memory, CPU, and network resources across multiple nodes into a single, coherent data fabric. Applications continue running even when individual nodes fail—no blinking, no downtime. Geode supports multiple deployment models, including peer‑to‑peer, client/server, and multi‑site configurations, enabling it to scale from tightly coupled application clusters to geographically distributed systems.

From GemFire to Geode—and Almost to Obsolescence

Apache Geode’s lineage traces back to 2002, when GemStone Systems introduced GemFire, a commercial platform widely used in financial services for real‑time workloads. Through acquisitions—GemStone to SpringSource, then to VMware, and later to Pivotal—the technology evolved before being open sourced in 2015 and donated to The Apache Software Foundation as Apache Geode.

For several years, the project thrived. But after 2019, corporate shifts and changing priorities reduced contributor engagement. By 2022, most committers were inactive. By mid‑2023, development had stopped entirely. In 2024, the PMC voted to terminate the project. Apache Geode appeared to be finished.

Apache Geode Contributors Over Time

Then came 2025. I began upstreaming my internal fork. The community delivered Apache Geode 1.15.2 in September, followed by Apache Geode 2.0 in December. What looked like an ending became a comeback—a transition from long winter to spring.

By the time Apache Geode’s revival began, it was clear that survival alone was not enough. To remain viable, trusted, and relevant, the platform would need far more than incremental fixes—it would need a complete modernization from the ground up.

Learn what Apache Geode is https://geode.apache.org/

How DiDi Scaled to Hundreds of Petabytes with Apache Ozone

Whitney — Thu, 29 Jan 2026 23:42:28 +0000

Building a cost-effective, high-performance data foundation for global mobility

When you’re operating one of the world’s largest ride-hailing and mobility platforms, every millisecond and megabyte counts. For DiDi Global, which generates over one petabyte of new data every day, scaling storage isn’t just a technical challenge—it’s a business imperative.

As the company’s data footprint grew to more than 500PB annually, DiDi’s engineers found themselves battling the limits of their legacy Apache HadoopⓇ Distributed File System (HDFS) storage layer. The infrastructure was struggling to keep pace with the company’s explosive data growth, slowing downstream analytics and machine learning (ML) workloads that power everything from route optimization to dynamic pricing.

The Challenge: Scaling Without Compromise

DiDi’s HDFS-based infrastructure had served the company well, but it was beginning to show its age under the weight of petabyte-scale workloads. The team faced several interconnected problems:

Metadata bottlenecks: File count limits in HDFS created stress on metadata services, driving up latency and throttling performance.
Read-heavy workloads: RPC congestion and HDD I/O bottlenecks introduced lag for analytics and AI pipelines.
Escalating costs: Triple replication inflated storage use and operational expenses.
Operational risk: Even routine maintenance, such as decommissioning, carried stability concerns.

These issues had tangible business impacts. Slow metadata operations increased latency for end users, inflated costs, and created risks during peak demand periods.

“Metadata latency wasn’t just a technical problem—it slowed down business units that rely on real-time analytics and AI insights,” said JiangHua Zhu, Software Engineer, DiDi’s Storage Team.

The Solution: Apache Ozone

After a rigorous evaluation, DiDi selected Apache Ozone™, a next-generation distributed storage system designed for scalability and performance in large, unstructured data environments.

Ozone’s modern architecture—featuring RocksDB-based metadata management, separation of Object Manager (OM) and Storage Container Manager (SCM) services, and containerized data storage—provide the foundation DiDi needed to scale with confidence.

Key Benefits

Massive scalability: Ozone comfortably supports tens of billions of files, removing HDFS metadata constraints.
Performance optimizations: Features like OM Follower Read, multi-cluster routing, and NVMe caching help minimize latency and balance system load.
Cost efficiency through Erasure Coding:
Transitioning from 3x replication to EC 6-3 reduce storage overhead from 3.0x to roughly 1.5x—saving hundreds of petabytes.
Enhanced resilience: Container-based data granularity improves fault tolerance and streamlined operations.

“Ozone gave us the flexibility to scale elastically across hundreds of petabytes without sacrificing performance,” said Wei Ming, DiDi engineer.

The Results: Faster, Leaner, and More Reliable

The move to Apache Ozone delivered measurable, cross-functional benefits across DiDi’s data ecosystem:

Latency: P90 GetMetaLatency improved from 90ms to 17ms.
Throughput: Production read throughput increased by more than 20% with OM follower reads.
Cost savings: Erasure Coding cut the storage footprint nearly in half, saving both capital and operational expenses.
Stability under load: The platform now operates smoothly even during cluster maintenance and peak traffic periods.
Developer productivity: Application teams no longer need to manage small-file compaction, reducing complexity and accelerating data delivery.

Smooth Adoption Through Planning and Community Collaboration

DiDi’s migration to Ozone was meticulous and deliberate. Engineers ensured data consistency with DistCp COMPOSITE_CRC checksums, implemented dual-write for rollback safety, and validated end-to-end compatibility with Hadoop, Apache Spark™, and S3 APIs.

The company also leaned heavily on the Apache Ozone open source community—which contribute bug fixes, performance enhancements, and feedback that benefit all users.

“The open source community was instrumental in our success—we gained support, shared knowledge, and received bug fixes that help everyone,” said Shilun Fan, DiDi’s storage leadership.

DiDi engineers even became active contributors, helping resolve issues such as metadata inconsistencies and Erasure Coding container handling. The collaboration ultimately strengthened both DiDi’s deployment and the broader Ozone ecosystem.

Technical Highlights

Storage savings: Hundreds of petabytes saved through Erasure Coding (6-3).
Read efficiency: 20%+ improvement from OM follower reads and NVMe caching.
Unified access: Hadoop API and S3 compatibility for batch, interactive, and ML workloads.
Scalability: A single Ozone cluster can handle ~5 billion files, with the potential to scale to tens of billions.

Looking Ahead

DiDi’s storage team continues to push the boundaries of performance and efficiency. Upcoming initiatives include:

Integrating IO_URING and SPDK to enhance I/O performance.
Developing AI-driven operational insights for anomaly detection and auto-remediation.
Piloting tiered storage strategies for hot, warm, and cold data layers to optimize cost and performance.

“Ozone is more than a storage layer—it’s the backbone of DiDi’s data ecosystem and future AI innovation,” said Hongbing Wang, DiDi technical lead.

The Takeaway

By embracing Apache Ozone, DiDi transformed its data storage infrastructure from a limitation into a competitive advantage. The move delivered lower costs, higher reliability, and faster access to the insights that power intelligent mobility.

At petabyte scale, even incremental improvements deliver outsized impact—and with Apache Ozone, DiDi has built a storage foundation ready for the next decade of data-driven innovation.

To learn more about Apache Ozone:

Apache Ozone GitHub: https://github.com/apache/ozone
Apache Ozone Getting Started: https://ozone.apache.org/docs/edge/start/startfromdockerhub.html
Apache Ozone LinkedIn page: https://www.linkedin.com/company/apache-ozone/
Apache Ozone X.com handle: https://x.com/ApacheOzone
Apache Ozone Best Practices at Didi: https://ozone.apache.org/assets/ApacheOzoneBestPracticesAtDidi.pdf