DEV Community: DASWU

How D-Robotics Manages Massive Small Files in a Multi-Cloud Environment with JuiceFS

DASWU — Fri, 06 Mar 2026 06:44:16 +0000

D-Robotics, founded in 2024 and spun off from Horizon Robotics' robotics division, specializes in the research and development of foundational computing platforms for consumer-grade robots. In 2025, we released an embodied AI foundation model.

In robot data management, training, and inference, the sheer volume of data is immense. Using object storage presents challenges such as handling small files and managing multi-cloud data. After trying some solutions and replacing private MinIO with SSD storage, we still faced difficulties in addressing these challenges. Ultimately, we selected JuiceFS as our core storage solution.

JuiceFS' inherent adaptability for cross-cloud operations efficiently supports data sharing needs in multi-cloud environments. In training scenarios, JuiceFS' cache mechanism, specifically designed for small file data, effectively replaces traditional caching solutions while achieving a cost-effective balance between cost and efficiency, fully meeting storage performance requirements. Currently, we manage tens of millions of files.

In this article, we’ll share our application characteristics, storage pain points, solution selection, implementation practices, and production tuning experiences. We hope our experience offers useful insights for those facing similar challenges in the industry.

Storage pain points in the robotics industry

The cloud platform serves as our core technical hub, undertaking key application functions such as simulation environment setup, data generation and model training, model lightweighting and deployment, and visual verification. The data types involved in the platform are diverse, mainly including sensor image data, LiDAR point cloud data, model weights and configuration data, motor operational data, and map construction data.

While object storage meets basic storage needs for massive data, its performance limitations become particularly obvious when handling the massive small files frequently encountered in robotics applications. Our storage system faced four challenges:

Metadata performance bottleneck with massive small files: Robot model training involves tens of millions to billions of sensor images, LiDAR data, and model files. Traditional object storage (like standard S3) exhibits significant metadata operation bottlenecks at this scale. The fixed API latency for routine operations like listing files or retrieving attributes is typically 10–30 ms. This directly constrains queries per second (QPS) performance during training and inference and impacts overall R&D efficiency.
Inefficient multi-cloud collaboration and data flow: As robotics companies increasingly adopt multi-cloud architectures for their R&D and production applications, ensuring efficient data synchronization and sharing across different cloud platforms and geographical regions has become a common challenge for the industry. Traditional storage solutions typically suffer from low cross-cloud data transfer efficiency and are often deeply integrated with a single cloud provider. This leads to technical lock-in and makes it difficult to achieve flexible cross-cloud deployment and data collaboration.
The impossible trinity of performance, cost, and operations: High-performance parallel file systems offer high throughput and low latency but typically rely on all-flash arrays or dedicated hardware. This leads to high hardware investment and ongoing operational costs, plus complex deployment. Low-cost object storage offers good elasticity but is difficult to support the high-throughput I/O demands of GPU clusters in AI training scenarios. A common industry workaround is using a high-speed file system as a cache synchronized with S3. However, the extra data synchronization steps significantly reduce usability and fail to achieve efficient storage-compute synergy.
Difficulty in dataset version management: The rapid iteration cycle of robot models requires efficient and granular management of multiple dataset versions. Using physical copies for version control directly leads to exponentially higher underlying storage consumption, significantly increasing costs. Moreover, the difficulty of retrieving, reusing, and maintaining multi-version data also increases substantially.

Storage selection: JuiceFS vs. MinIO/S3 vs. PFS

To address these storage challenges, we established a clear evaluation framework for storage selection. A comprehensive comparative test was conducted on mainstream storage solutions across seven core dimensions: storage architecture, protocol compatibility, metadata performance, scalability, multi-cloud adaptability, cost efficiency, and operational complexity.

Comparison basis	JuiceFS	MinIO / Public Cloud S3	CephFS / Public Cloud FS (CPFS)
Storage architecture	Separation of metadata and data	Unified object storage	Metadata and data typically coupled, often with kernel-space parallel design
Protocol support	Full compatibility: POSIX, HDFS, S3 API, Kubernetes CSI	Primarily S3 API, with weak POSIX compatibility	POSIX-oriented; HDFS or S3 compatibility often requires plugins
Metadata performance	Very high: sub-millisecond latency, supports hundreds of billions of files per volume	Lower: high metadata overhead for massive small files; API call overhead about 10–30 ms	Medium to high: performance bottlenecks and complexity challenges at ultra-large scale (100M+ files)
Scalability	High: horizontal scaling, supports tens to hundreds of billions of files per volume	High: near-infinite storage capacity, but small-file management efficiency degrades with scale	Moderate: scaling limited by metadata nodes; operational complexity grows exponentially with scale
Multi-cloud adaptability	Native support	Relies on sync tools; cross-cloud data flow inefficient; global unified view difficult	Limited: often tightly bound to specific hardware or cloud provider; cross-cloud deployment is complex
Cost efficiency	High performance-to-cost ratio	Low (storage only): cheap storage, but low GPU utilization in high-throughput scenarios like AI training	High: often requires all-flash architecture or dedicated hardware; high operational labor cost

Based on the comparison results above, JuiceFS demonstrates significant advantages in core performance, scalability, multi-cloud adaptability, and cost efficiency. This makes it the preferred choice for our unified storage solution.

Furthermore, JuiceFS has been widely adopted in the autonomous driving industry. Leading companies such as Horizon Robotics have leveraged JuiceFS to manage data at the exabyte scale. This demonstrates its maturity and effectiveness in large-scale production environments.

For our specific application scenarios, JuiceFS' core technical advantages:

Decoupled architecture: JuiceFS adopts a metadata-data separation architecture, persisting data in cost-effective object storage (like S3 or OSS) while storing metadata in databases like Redis or TiKV. This decoupled design enables elastic storage scaling and reduces dependence on any single cloud provider.
Chunking and caching mechanisms: JuiceFS uses chunks, slices, and blocks to significantly improve small file read efficiency and enhance concurrent read/write performance. In addition, multi-level caching (memory, local SSD, distributed cache) reduces access latency for hot data. This meets the demands of high-throughput training workloads.
Cloud-native adaptability: By providing a CSI Driver, JuiceFS delivers persistent storage decoupled from compute nodes in Kubernetes environments, supporting stateless container deployment and cross-cloud migration. It enables data sharing, enhances application high availability and flexibility, and adapts to various Kubernetes deployment methods.
Full-stack support for AI training: JuiceFS fully supports POSIX, HDFS, and S3 API, and is compatible with mainstream AI frameworks such as PyTorch and TensorFlow. It can be integrated without code modifications, lowering the technical barrier for adoption.
Multi-cloud support: Its cross-cloud capabilities and high-performance metadata engine ensure efficient data flow, perfectly aligning with our strategy of "computing power on demand."

From a cost perspective, JuiceFS does not offer a significant cost advantage in the early stages of small-scale deployment. However, when data volume reaches the petabyte level—especially at the 10 PB or 100 PB scale—and is compared against all-flash storage solutions, its cost-efficient architecture built on object storage becomes fully evident. In addition, JuiceFS requires minimal operational overhead. Currently, we need only one engineer to manage the entire cloud platform and storage system, a fraction of the personnel required by traditional solutions.

From Community Edition to Enterprise Edition: addressing larger-scale scenarios

As our application continued to expand, we encountered limitations when using Redis as the metadata engine—specifically, physical memory capacity constrained data scalability. When the number of files approached the hundred-million level, metadata query latency increased significantly. This impacted the concurrency efficiency of training tasks. After using the clone feature, the metadata volume grew substantially. In addition, in cross-cloud scenarios, we faced higher demands for metadata synchronization and mirror file system capabilities. We also required more granular capacity controls and permission management at the directory level.

Considering these requirements—along with our desire to leverage local SSDs on GPU nodes to build a distributed cache layer for improved performance—we decided to deploy JuiceFS Enterprise Edition in parallel, migrating core scenarios such as ultra-large-scale directory management and multi-node collaborative training to this version. Through this scenario-based approach, we’ve effectively enhanced the adaptability of our overall storage system and established a solid foundation for future application growth. Below are the key features of the Enterprise Edition that we’ve applied in real-world scenarios.

High-performance metadata engine: solving the bottleneck of large-scale directory retrieval

For high-frequency operations such as traversing directories with hundreds of millions of files and deep pagination queries, we previously encountered the "slower as you query" problem with traditional storage solutions. When the number of files in a single directory exceeded 10 million, and the pagination offset surpassed 100,000 entries, response latency would spike from hundreds of milliseconds to several seconds. This severely impacted data filtering efficiency.

After switching to JuiceFS Enterprise Edition, its native tree-structured metadata storage architecture played a key role. Unlike the flat key-value storage used—which stores file metadata in a disordered manner—this tree structure allows direct navigation to directory levels, reducing the scope of metadata scans. In our actual tests, deep pagination queries (with an offset of 500,000 entries) in a directory containing 120 million files saw latency drop from 3.8 seconds to just 210 milliseconds. This fully met the retrieval needs of large-scale datasets. In addition, this engine supports storing hundreds of billions of files per volume, and we’ve already used it to manage three petabyte-scale training datasets stably, aligning with our application growth expectations.

Enterprise-grade distributed cache: improving data sharing efficiency in multi-node, multi-GPU training

In multi-node, multi-GPU training scenarios, we previously faced challenges such as low cache hit rates and cross-node bandwidth congestion. The open-source version only supports local caching on each node. This means that when multiple nodes pull the same dataset simultaneously, each node must access object storage independently. This resulted in single-node bandwidth utilization exceeding 90%, with average training job startup delays of up to 20 minutes.

With JuiceFS Enterprise Edition's distributed caching feature, we set up a distributed cache across a 12-node training cluster using just three commands. The dataset only needs to be pulled from object storage once and is cached in a pool built from local SSDs across the nodes. As a result, the cache hit rate for multi-node collaborative training increased from 45% to 92%, cross-node bandwidth utilization dropped to below 15%, and training job startup time was reduced to under three minutes. This significantly improved compute utilization.

Enhanced cross-cloud collaboration: building a low-operational-cost cross-cloud data foundation

Since our R&D environments are distributed across two cloud environments, we previously encountered challenges with slow cross-cloud data synchronization and high operational costs. Using traditional synchronization tools to maintain data consistency between the two clouds required configuring eight scheduled tasks, with an average synchronization delay of four hours, and dedicated personnel needed to investigate sync failures weekly.

By using the JuiceFS sync tool combined with our internal AI operations tools, we achieved automated configuration of synchronization policies. The system automatically adjusts sync priorities based on data heat levels, keeping cross-cloud data latency within 10 minutes. In addition, tasks such as failure retries and log alerts for synchronization are fully automated, eliminating the need for dedicated monitoring. This has reduced operational overhead by 70%, and we now stably support multiple training projects across two cloud platforms sharing the same dataset. Going forward, we plan to use the Enterprise Edition's mirror file system feature to further enhance cross-cloud data collaboration.

JuiceFS optimization

Client cache and write performance tuning

We need to pay attention to compatibility issues between caching strategies and Kubernetes resource limits. For example, using memory as a local cache path with improper configuration may lead to abnormal memory growth in the Mount Pod, or insufficient resource quota reservations may cause checkpoint loss or file handle write exceptions during long-running training tasks.

Regarding write performance tuning, enabling writeback mode can improve small file write throughput to some extent. However, considering production environment requirements for data consistency, we still adopt write-through synchronous mode to reduce data risks in extreme crash scenarios. It’s recommended to cautiously enable writeback mode only in scenarios with lower data reliability requirements, such as temporary computing or offline data cleaning, based on actual needs.

Deployment and network topology optimization

For more stable performance, it’s strongly recommended to deploy the metadata engine and compute nodes within the same region during deployment. In actual operations, we observed that cross-region deployment could increase metadata operation latency by several to ten times. This significantly impacted I/O-intensive operations such as data decompression. Deploying metadata services and GPU computing resources within the same region helps maintain performance while controlling network transmission costs, improving overall resource utilization efficiency.

Data warm-up and cache optimization

In a 10-gigabit network environment, fully utilizing JuiceFS' data warm-up and reasonably adjusting data block sizes based on application scenarios can better leverage network bandwidth capabilities and improve read throughput. Combined with the distributed cache architecture, this can effectively enhance data sharing efficiency in multi-node concurrent scenarios and improve cache hit rates during high-concurrency reads. This thereby optimizes the overall performance of large-scale AI training tasks.

Resource quotas and high availability guarantee

In enterprise-level multi-role operations and storage responsibility separation scenarios, to avoid operational risks caused by inconsistent configurations, it’s recommended to finely control resource quotas for JuiceFS CSI Driver in Kubernetes environments. By appropriately setting CPU and memory request/limit for Mount Pods, Pod restarts or node anomalies caused by resource preemption can be reduced. In practice, resource reservation ratios can be dynamically adjusted based on cluster load.

In addition, for scenarios with high application continuity requirements, the automatic mount point recovery feature for Mount Pods can be enabled to achieve automated fault recovery for storage services, further ensuring underlying storage stability.

Multi-tenancy

We provide independent file systems and storage buckets for large enterprise customers, while achieving isolation for small and medium-sized enterprises and end users through subdirectory-level directory isolation and permission control.

Large enterprises can flexibly scale throughput and capacity, avoiding performance bottlenecks associated with shared storage buckets. For small and medium-sized enterprises and end users, we ensure data security and independence through subdirectory isolation and permission control, while enabling accurate metering and billing.

This architecture ensures tenant isolation while flexibly allocating resources, improving system management efficiency.

Version management

Using the juicefs clone command, copies of original datasets can be quickly created and modified independently without affecting the source data. The clone operation only copies file metadata, while data only stores additional changes, saving underlying storage space. This feature supports managing multiple versions, facilitating rollback and recovery and ensuring data security and version control.

Summary

JuiceFS' characteristics in metadata performance, scalability, cross-cloud adaptability, and comprehensive cost efficiency have made it our choice for building a unified storage layer. Currently, we adopt both JuiceFS Community Edition and Enterprise Edition to accommodate different storage requirements across various application scenarios.

In the future, we plan to further implement JuiceFS in the embodied intelligence field, addressing specific storage needs in this scenario. These include high-throughput processing of time-series data, precise multi-modal data alignment, edge-cloud collaborative storage, and integrated management of simulation and real-world data.

If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and community on Slack.

JuiceFS Enterprise 5.3: 500B+ Files per File System & RDMA Support

DASWU — Wed, 04 Feb 2026 09:18:32 +0000

JuiceFS Enterprise Edition 5.3 has recently been released, achieving a milestone breakthrough by supporting over 500 billion files in a single file system. This upgrade includes several key optimizations to the metadata multi-zone architecture and introduces remote direct memory access (RDMA) technology for the first time to enhance distributed caching efficiency. In addition, version 5.3 enhances write support for mirrors and provides data caching for objects imported across buckets. It aims to support high-performance requirements and multi-cloud application scenarios.

JuiceFS Enterprise Edition is designed for high-performance scenarios. Since 2019, it has been applied in machine learning and has become one of the core infrastructures in the AI industry. Its customers include large language model (LLM) companies such as MiniMax and StepFun; AI infrastructure and applications like fal and HeyGen; autonomous driving companies like Momenta and Horizon Robotics; and numerous leading technology enterprises across various industries leveraging AI.

Single file system supports 500 billion+ files

The multi-zone architecture is one of JuiceFS' key technologies for handling hundreds of billions of files, ensuring high scalability and concurrent processing capabilities. To meet the growing demands of scenarios like autonomous driving, version 5.3 introduces in-depth optimizations to the multi-zone architecture, increasing the zone limit to 1,024 and enabling a single file system to store and access at least 500 billion files (each zone can store 500 million files, with a maximum of 2 billion).

The figure below shows JuiceFS Enterprise Edition architecture, with a single zone in the lower left corner:

This breakthrough presents exponentially increasing challenges in system performance, data consistency, and stability, backed by a series of complex underlying optimizations and R&D efforts.

Cross-zone hotspot balancing: automated monitoring and hotspot migration, with manual ops tools

In distributed systems, hotspots are a common challenge. Especially when data is distributed across multiple zones, some zones may experience higher loads than others. This leads to imbalance that impacts system performance.

When the number of zones reaches hundreds, hotspot issues become more prevalent. Particularly with smaller datasets and larger numbers of files, read/write hotspots exacerbate latency fluctuations.

We introduced an automated hotspot migration mechanism to move frequently accessed files to other zones, distributing the load and reducing pressure on specific zones. However, in practice, relying solely on automated migration cannot fully resolve all issues. In certain special or extreme scenarios, automated tools may not respond promptly. Therefore, alongside automated monitoring and migration, we added manual operational tools, allowing administrators to intervene in complex scenarios, perform manual analysis, and implement optimization solutions.

Large-scale migration: improved migration speed, small-batch concurrent migration

Facing zones with excessive hotspots, early migration operations were simple. However, as the system scale expanded, migration efficiency gradually decreased. To address this, we introduced a small-batch concurrent migration strategy, breaking down high-access directories into smaller chunks and migrating them in parallel to multiple lower-load zones. This quickly scatters hotspots and restores normal application access.

Enhanced reliability self-checks: automatic repair and cleanup of intermediate migration states

In large-scale clusters, the probability of distributed transaction failures increases significantly, especially during extensive migration processes. To address this, we enhanced reliability detection mechanisms, adding periodic background checks to scan cross-zone file states, particularly focusing on intermediate state issues, and automatically performing repairs and cleanup.

Previously, the system encountered issues with residual intermediate state data. While these did not affect operations in the short term, over time they could lead to errors. Through enhanced self-check mechanisms, we ensure the background periodically scans and promptly handles intermediate state issues, improving system stability and reliability.

Beyond the three key optimizations above, we also made multiple improvements to the console to better adapt to managing more zones. We optimized concurrent processing, operational tasks, and query displays, enhancing overall performance and user experience. Specifically, we refined UI design to better showcase system states in large-scale zone environments.

Performance stress test for hundreds of billions of files

We conducted large-scale tests using a custom mdtest tool on Google Cloud, deploying 60 nodes, each with over 1 TB of memory. In terms of software configuration, we increased the number of zones to 1,024. The deployment method was similar to previous setups, but to reduce memory consumption, we deployed only one service process, with two others as cold backups.

JuiceFS Enterprise Edition 5.3 test:

Test duration: Approximately 20 hours
Total files written: About 400 billion files
Write speed: 5 million files per second
Memory usage: About 35% to 40%
Disk usage: 40% to 50%, primarily for metadata persistence, with good utilization

Based on our experience, if using a configuration with one service process, one hot backup, and one cold backup, memory usage increases by 20% to 30%.

Due to limited cloud resources, this test only wrote up to 400 billion files. During stress testing, the system performed stably, with hardware resources still remaining. We’ll continue to attempt larger-scale tests in the future.

Support for RDMA: increased bandwidth cap, reduced CPU usage

This new version introduces support for RDMA technology for the first time. Its basic architecture is shown in the diagram below. RDMA allows direct access to remote node memory, bypassing the operating system's network protocol stack. This significantly improves data transfer efficiency.

The main advantages of RDMA include:

Low latency: By enabling direct memory-to-memory transfers and bypassing the OS network protocol layers, it reduces CPU interrupts and context switches. This lowers latency.
High throughput: RDMA uses hardware for direct data transfer, better utilizing the bandwidth of network interface cards (NICs).
Reduced CPU usage: In RDMA, data copying is almost entirely handled by the NIC, with the CPU only processing control messages. This allows the NIC to handle hardware transfers, freeing up CPU resources.

In JuiceFS, network request messages between clients and metadata services are small, and existing TCP configurations already meet the needs. However, in distributed caching, file data is transferred between clients and cache nodes. Using RDMA can effectively improve transfer efficiency and reduce CPU consumption.

We conducted 1 MB random read tests using 160 Gbps NICs, comparing versions 5.1, 5.2 (using TCP networking) with version 5.3 (RDMA), and observed CPU usage.

Tests showed that RDMA effectively reduces CPU usage:

In version 5.2, CPU usage was nearly 50%.
In version 5.3, with RDMA optimization, CPU usage dropped to about one-third. Client and cache node CPU usage decreased to 8 cores and 5 cores respectively, with bandwidth reaching 20 GiB/s.

In previous tests, we found that while TCP ran stably on 200G NICs, fully saturating bandwidth was challenging, typically achieving only 85%-90% utilization. For customers requiring higher bandwidth (such as 400G NICs), TCP could not meet demands. However, RDMA can more easily reach hardware bandwidth limits, providing better transfer efficiency.

If customers have RDMA-capable hardware and high bandwidth requirements (for example, NICs greater than 100G) and wish to reduce CPU usage, RDMA is a technology worth trying. Currently, our RDMA feature is in public testing and has not yet been widely deployed in production environments.

Enhanced write support for mirrors

Initially, mirror clusters were primarily used for read-only mirroring in enterprise products. As users requested capabilities like writing temporary files (such as training data) in mirrors, we provided write support for mirrors.

The mirror client implements a read-write separation mechanism. When reading data, the client prioritizes fetching from the mirror cluster to reduce latency. When writing data, it still writes to the source cluster to ensure data consistency. By recording and comparing metadata version numbers, we ensure strong consistency between the mirror client and source cluster client views of the data.

To improve availability, version 5.3 introduces a fallback mechanism. When the mirror becomes unavailable, client read requests automatically fall back to the source cluster. This ensures application continuity and avoids interruptions caused by mirror cluster failures. We also optimized deployments in multi-mirror environments. Previously, the mirror end required two hot backup nodes to ensure high availability. Now, with the improved fallback feature, deploying a single mirror node can achieve similar effects. This ensures application continuity and reduces costs, especially beneficial for users requiring multiple mirrors.

Through this improvement, we not only reduced hardware costs but also found a balance between high availability and low cost. For users deploying mirrors in multiple locations, reducing metadata replicas further lowers overall costs.

Simplified operations & increased flexibility: providing cross-bucket data cache for imported objects

In JuiceFS, users can use the import command to bring existing files from object storage under unified management. This is convenient for users already storing large amounts of data (for example, tens of petabytes). However, in previous versions, this feature only supported caching for objects within the same data bucket. This meant imported objects had to reside in the same bucket as the existing file system data. This limitation had certain practical constraints.

In version 5.3, we improved this feature. Users can now provide caching capability for any imported objects, regardless of whether they come from the same data bucket. This allows users more flexibility in managing objects across different data buckets, avoiding strict bucket restrictions and enhancing data management freedom.

In addition, previously, if users had data distributed across multiple buckets and wanted to provide caching for that data, they needed to create a new file system for each bucket. In version 5.3, users only need to create one file system (volume) to uniformly manage data from multiple buckets and provide caching for all buckets.

Other important optimizations

Trace feature

We added the trace feature, a feature provided by the Go language itself. Through this, advanced users can perform tracing and performance analysis, gaining more information to help quickly locate issues.

Trash recovery

In previous versions, especially with multiple zones, sometimes the paths recorded in the trash were incomplete. This led to anomalies during recovery, where files were not restored to the expected locations. To address this, in version 5.3, when deleting files, we record the original file path, ensuring more reliable recovery capabilities.

Python SDK improvements

In earlier versions, we released the Python SDK, providing basic read/write functionalities for Python users to interface with our system. In version 5.3, we not only strengthened basic read/write functions but also added support for operational subcommands. For example, users can directly call commands like juicefs info or warmup via the SDK without relying on external system commands. This simplifies coding efforts and avoids potential performance bottlenecks from frequently calling external commands.

The Windows client

We previously launched a beta version of the Windows client and have received some user feedback. After improvements, the current version shows significant enhancements in mount reliability, performance, and compatibility with Linux systems. In the future, we plan to further refine the Windows client, providing an experience closer to Linux for users reliant on Windows.

Summary

Compared to expensive dedicated hardware, JuiceFS helps users balance performance and cost when addressing data growth by flexibly utilizing cloud or existing customer storage resources. In version 5.3, by optimizing the metadata zone architecture, a single file system can support over 500 billion files. The first-time introduction of RDMA technology significantly improves distributed caching bandwidth and data access efficiency, reduces CPU usage, and further optimizes system performance. In addition, we enhanced features like write support for mirrors and caching, improving the performance and operational efficiency of large-scale clusters and optimizing user experience.

Cloud service users can now directly experience JuiceFS Enterprise Edition 5.3 online, while on-premises deployment users can obtain upgrade support through official channels. We’ll continue to focus on high-performance storage solutions, partnering with enterprises to tackle challenges brought by continuous data growth.

If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and community on Slack.

From GlusterFS to JuiceFS: Lightillusions Achieved 2.5x Faster 3D AIGC Data Processing

DASWU — Fri, 09 Jan 2026 07:43:46 +0000

Lightillusions is a company specializing in spatial intelligence technology, integrating 3D vision, computer graphics, and generative models to build innovative 3D foundation models. Our company is led by Ping Tan, a professor at the Hong Kong University of Science and Technology (HKUST) and Director of the HKUST-BYD Joint Laboratory.

Unlike 2D models, a single 3D model can be several gigabytes in size, especially complex models like point clouds. When our data volume reached petabyte scales, management and storage became significant challenges. After trying solutions like NFS and GlusterFS, we chose JuiceFS, an open-source high-performance distributed file system, to build a unified storage platform. This platform now serves multiple scenarios, supports cross-platform access including Windows and Linux, manages hundreds of millions of files, improves data processing speed by 200%–250%, enables efficient storage scaling, and greatly simplifies operations and maintenance. This allows us to focus more on core research.

In this article, we’ll break down the unique storage demands of 3D AIGC, share why we selected JuiceFS over CephFS, and walk through the architecture of our JuiceFS-based storage platform.

Storage requirements for 3D AIGC

Our research focuses on perception and generation. In the 3D domain, task complexity is different from image and text processing. This placed higher demands on our AI models, algorithms, and infrastructure.

We illustrate the complexity of 3D data processing through a typical pipeline. On the left side of the diagram below is a 3D model containing texture (top-left) and geometry (bottom-right) information. First, we generate rendered images. Each model has text labels describing its content, geometric features, and texture features, which are tightly coupled with the model. In addition, we process geometry data, such as sampled points and necessary numerical values obtained from data preprocessing, like signed distance fields (SDFs). It's important to note that 3D model file formats are highly diverse, and image formats are also different.

Our work spans language models, image/video models, and 3D models. As data volume grows, so does the storage burden. The main characteristics of data usage in these scenarios are as follows:

Language models: Data typically consists of a vast number of small files. Although individual text files are small, the total file count can reach millions or even tens of millions as data volume increases. This makes the management of such a large number of files a primary storage challenge.
Image and video data: High-resolution images and long videos are usually large. A single image can range from hundreds of kilobytes to several megabytes, while video files can reach gigabytes. During preprocessing—such as data augmentation, resolution adjustment, and frame extraction—data volume increases significantly. Especially in video processing, where each video is typically decomposed into a large number of image frames, managing these massive file collections adds considerable complexity.
3D models: Individual models, especially complex ones like point clouds, can be several gigabytes in size. 3D data preprocessing is more complex than other data types, involving steps like texture mapping and geometry reconstruction, which consume great computational resources and can increase data volume. Furthermore, 3D models often consist of multiple files, leading to a large total file count. As data grows, managing these files becomes increasingly difficult.

Based on the storage characteristics discussed above, when we chose a storage platform solution, we expected it to meet the following requirements:

Diverse data formats and cross-node sharing: Different models use different data formats, especially the complexity and cross-platform compatibility issues of 3D models. The storage system must support multiple formats and effectively manage data sharing across nodes and platforms.
Handling data models of different sizes: Whether it's small files for language models, large-scale image/video data, or large files for 3D models, the storage system must be highly scalable to meet rapidly growing storage demands and handle the storage and access of large-size data efficiently.
Challenges of cross-cloud and cluster storage: As data volume increases, especially with petabyte-level storage needs for 3D models, cross-cloud and cluster storage issues become more prominent. The storage system must support seamless cross-region, cross-cloud data access and efficient cluster management.
Easy scaling: The need for scaling is constant, whether for language, image/video, or 3D models, and is particularly high for 3D model storage and processing.
Simple operations and maintenance: The storage system should provide easy-to-use management interfaces and tools. Especially for 3D model management, operational requirements are higher, making automated management and fault tolerance essential.

Storage solutions: from NFS, GlusterFS, CephFS to JuiceFS

Initial solution: NFS mount

Initially, we tried the simplest solution—using NFS for mounting. However, in practice, we found that the training cluster and rendering cluster required independent clusters for mount operations. Maintaining this setup was very cumbersome. Especially when adding new data, as we needed to write mount points separately for each new dataset. When the data volume reached about 1 million objects, we could no longer sustain this approach and abandoned it.

Mid-term solution: GlusterFS

GlusterFS was an easy-to-start-with choice, offering simple installation and configuration, acceptable performance, and no need for multiple mount points—just add new nodes.

While GlusterFS greatly reduced our workload in the early stages, we also discovered issues with its ecosystem:

Many GlusterFS execution scripts and features required writing custom scheduled tasks. Particularly when adding new storage, it had additional requirements, such as needing to increase nodes by specific multiples.
Support for operations like cloning and data synchronization was weak. This led us to frequently consult documentation.
Many operations were unstable. For example, when using tools like fio for speed testing, results were not always reliable.
A more serious problem was that GlusterFS performance would drastically decline when the number of small files reached a certain scale. For example, one model might generate 100 images. With 10 million models, that would produce 1 billion images. GlusterFS struggled severely with addressing in later stages, especially with an excessive number of small files. This led to significant performance drops and even system crashes.

Final selection: CephFS vs. JuiceFS

As storage demands grew, we decided to use a more sustainable solution. After evaluating various options, we compared CephFS and JuiceFS.

Although Ceph is widely used, through our own practice and reviewing documentation, we found Ceph's operational and management costs to be very high. Especially for a small team like ours, handling such complex operational tasks proved particularly difficult.

JuiceFS had two native features that strongly aligned with our needs:

The client data cache. For our model training clusters, which are typically equipped with high-performance NVMe storage, fully utilizing client caching could significantly accelerate model training and reduce pressure on the JuiceFS storage backend.
JuiceFS' S3 compatibility was crucial for us. As we had developed some visualization platforms based on storage for data annotation, organization, and statistics, S3 compatibility allowed us to rapidly develop web interfaces supporting visualization, data statistics, and other features.

The table below compares basic features of CephFS and JuiceFS:

Storage platform practice based on JuiceFS

Metadata engine selection and topology

JuiceFS employs a metadata-data separation architecture with several metadata engine options. We first quickly validated the Redis storage solution, which is well-documented by the JuiceFS team. Redis' advantage lies in its lightweight nature; configuration typically takes only a day or half a day, and data migration is smooth. However, when the number of small files exceeded 100 million, Redis' speed and performance significantly declined.

As mentioned earlier, each model might render 100 images. With other miscellaneous files, the number of small files increased dramatically. While we could mitigate the issue by packing small files, performing modifications or visualization on packed data greatly increased complexity. Therefore, we preferred to retain the original small image files for subsequent processing

As the file count grew and soon exceeded Redis' capacity, we decided to migrate the storage system to a combination of TiKV and Kubernetes (K8s). The TiKV-K8s setup provided us with a more highly available metadata storage solution. Furthermore, through benchmarking, we found that although TiKV's performance was slightly lower, the gap was not significant, and its support for small files was better than Redis'. We also consulted JuiceFS engineers and learned that Redis has poor scalability in cluster mode. Therefore, we switched to TiKV.

The table below shows read/write performance test results for different metadata engines:

Latest architecture: JuiceFS+TiKV+SeaweedFS

We use JuiceFS to manage the object storage layer. For the metadata storage system, we built it with TiKV and K8s. For object storage, we used SeaweedFS. This allows us to quickly scale storage capacity and provides fast access for both small and large files. In addition, our object storage is distributed across multiple platforms, including local storage and platforms like R2 and Amazon S3. Through JuiceFS, we were able to integrate these different storage systems and provide a unified interface.

To better manage system resources, we built a resource monitoring platform on K8s. The current system consists of about 60 Linux nodes and several Windows nodes handling rendering and data processing tasks. We monitored read stability, and the results show that even with multiple heterogeneous servers performing simultaneous read operations, the overall system I/O performance remains stable, able to fully utilize the bandwidth resources.

Problems we encountered

During the optimization of the storage solution, we initially tried an erasure code (EC) storage scheme aimed at reducing storage requirements and improving efficiency. However, in large-scale data migration, EC storage computation was slow, and its performance was unsatisfactory in high-throughput and frequent data change scenarios. Especially when combined with SeaweedFS, bottlenecks existed. Based on these issues, we decided to abandon EC storage and switch to a replication-based storage scheme.

We set up independent servers and configured scheduled tasks for large-volume metadata backups. In TiKV, we implemented a redundant replica mechanism, adopting a multi-replica scheme to ensure data integrity. For object storage, we used dual-replica encoding to further enhance data reliability. Although replica storage effectively ensures data redundancy and high availability, storage costs remain high due to processing petabyte-scale data and massive incremental data. In the future, we may consider further optimizing the storage scheme to reduce costs.

In addition, we found that using all-flash servers with JuiceFS did not bring significant performance improvements. The bottleneck mainly appeared in network bandwidth and latency. Therefore, we plan to consider using InfiniBand to connect storage servers and training servers to maximize resource utilization efficiency.

Summary

When using GlusterFS, we could process at most 200,000 models per day. After switching to JuiceFS, the processing capacity increased significantly. Our daily data processing capacity has grown by 2.5 times. Small file throughput also improved notably. The system remained stable even when storage utilization reached 70%. Furthermore, scaling became very convenient, whereas the previous architecture involved troublesome scaling processes.

Finally, let's summarize the advantages JuiceFS has demonstrated in 3D generation tasks:

Small file performance: Small file handling is a critical point, and JuiceFS provides an excellent solution.
Cross-platform features: Cross-platform support is very important. We found that some data can only be opened in Windows software, so we need to process the same data on both Windows and Linux systems and perform read/write operations on the same mount point. This requirement makes cross-platform features particularly crucial, and JuiceFS' design addresses this well.
Low operational cost: JuiceFS' operational cost is extremely low. After configuration, only simple testing and node management (for example, discarding certain nodes and monitoring robustness) are needed. We spent about half a year migrating data and have not encountered major issues so far.
Local cache mechanism: Previously, to use local cache, we needed to manually implement local caching logic in our code. JuiceFS provides a very convenient local caching mechanism, optimizing performance for training scenarios by setting mount parameters.
Low migration cost: Especially when migrating small files, we found using JuiceFS for metadata and object storage migration to be convenient, saving us a lot of time and effort. In contrast, migrating with other storage systems was very painful.

In summary, JuiceFS performs excellently in large-scale data processing, providing an efficient and stable storage solution. It not only simplifies storage management and scaling but also significantly improves system performance. This allows us to focus more on advancing core tasks. In addition, the JuiceFS tools are very convenient. For example, we used the sync tool for small file migration with extremely high efficiency. Without additional performance optimization, we successfully migrated 500 TB of data, including a massive number of small data and image files. It was done in less than 5 days, exceeding our expectations.

If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and community on Slack.

AI Data Storage: Challenges, Capabilities, and Comparative Analysis

DASWU — Fri, 19 Dec 2025 07:42:58 +0000

Note: This article was first published on DZone and featured on its homepage.

The explosion in the popularity of ChatGPT has once again ignited a surge of excitement in the AI world. Over the past five years, AI has advanced rapidly and has found applications in a wide range of industries. As a storage company, we’ve had a front-row seat to this expansion, watching more and more AI startups and established players emerge across fields like autonomous driving, protein structure prediction, and quantitative investment.

AI scenarios have introduced new challenges to the field of data storage. Existing storage solutions are often inadequate to fully meet these demands.

In this article, we’ll deep dive into the storage challenges in AI scenarios, critical storage capabilities, and comparative analysis of Amazon S3, Alluxio, Amazon EFS, Azure, GCP Filestore, Lustre, Amazon FSx for Lustre, GPFS, BeeGFS, and JuiceFS Cloud Service. I hope this post will help you make informed choices in AI and data storage.

Storage challenges for AI

AI scenarios have brought new data patterns:

High throughput data access challenges: In AI scenarios, the growing use of GPUs by enterprises has outpaced the I/O capabilities of underlying storage systems. Enterprises require storage solutions that can provide high-throughput data access to fully leverage the computing power of GPUs. For instance, in smart manufacturing, where high-precision cameras capture images for defect detection models, the training dataset may consist of only 10,000 to 20,000 high-resolution images. Each image has several gigabytes in size, resulting in a total dataset size of 10 TB. If the storage system lacks the required throughput, it becomes a bottleneck during GPU training.
Managing storage for billions of files: AI scenarios need storage solutions that can handle and provide quick access to datasets with billions of files. For example, in autonomous driving, the training dataset consists of small images, each about several hundred kilobytes in size. A single training set comprises tens of millions of such images, each sized several hundred kilobytes. Each image is treated as an individual file. The total training data amounts to billions or even 10 billion files. This creates a major challenge in effectively managing large numbers of small files.
Scalable throughput for hot data: In areas like quantitative investing, financial market data is smaller compared to computer vision datasets. However, this data must be shared among many research teams, leading to hotspots where disk throughput is fully used but still cannot satisfy the application's needs. This shows that we need storage solutions that can handle a lot of hot data quickly.

The basic computing environment has also changed a lot.

These days, with cloud computing and Kubernetes getting so popular, more and more AI companies are setting up their data pipelines on Kubernetes-based platforms. Algorithm engineers request resources on the platform, write code in Notebook to debug algorithms, use workflow engines like Argo and Airflow to plan data processing workflows, use Fluid to manage datasets, and use BentoML to deploy models into apps. Cloud-native technologies have become a standard consideration when building storage platforms. As cloud computing matures, AI businesses are increasingly relying on large-scale distributed clusters. With a significant increase in the number of nodes in these clusters, storage systems face new challenges related to handling concurrent access from tens of thousands of Pods within Kubernetes clusters.

IT professionals managing the underlying infrastructure face significant changes brought about by the evolving business scenarios and computing environments. Existing hardware-software coupled storage solutions often suffer from several pain points, such as no elasticity, no distributed high availability, and constraints on cluster scalability. Distributed file systems like GlusterFS, CephFS, and those designed for HPC such as Lustre, BeeGFS, and GPFS, are typically designed for physical machines and bare-metal disks. While they can deploy large capacity clusters, they cannot provide elastic capacity and flexible throughput, especially when dealing with storage demands in the order of tens of billions of files.

Key capabilities for AI data storage

Considering these challenges, we’ll outline essential storage capabilities critical for AI scenarios, helping enterprises make informed decisions when selecting storage products.

POSIX compatibility and data consistency

In the AI/ML domain, POSIX is the most common API for data access. Previous-generation distributed file systems, except HDFS, are also POSIX-compatible, but products on the cloud in recent years have not been consistent in their POSIX support:

Compatibility: Users should not solely rely on the description "POSIX-compatible product" to assess compatibility. You can use pjdfstest and the Linux Test Project (LTP) framework for testing. We’ve done a POSIX compatibility test of the cloud file system for your reference.
Strong data consistency guarantee: This is fundamental to ensuring computational correctness. Storage systems have various consistency implementations, with object storage systems often adopting eventual consistency, while file systems typically adhere to strong consistency. Careful consideration is needed when selecting a storage system.
User mode or kernel mode: Early developers favored kernel mode due to its potential for optimized I/O operations. However, in recent years, we’ve witnessed a growing number of developers "escaping" from kernel mode for several reasons:
- Kernel mode usage ties the file system client to specific kernel versions. GPU and high-performance network card drivers often require compatibility with specific kernel versions. This combination of factors places a significant burden on kernel version selection and maintenance.
- Exceptions of kernel mode clients can potentially freeze the host operating system. This is highly unfavorable for Kubernetes platforms.
- The user-mode FUSE library has undergone continuous iterations, resulting in significant performance improvements. It has been well-supported among JuiceFS customers for various business needs, such as autonomous driving perception model training and quantitative investment strategy training. This demonstrates that in AI scenarios, the user-mode FUSE library is no longer a performance bottleneck.

Linear scalability of throughput

Different file systems employ different principles for scaling throughput. Previous-generation distributed storage systems like GlusterFS, CephFS, the HPC-oriented Lustre, BeeGFS, and GPFS primarily use all-flash solutions to build their clusters. In these systems, peak throughput equals the total performance of the disks in the cluster. To increase cluster throughput, users must scale the cluster by adding more disks.

However, when users have imbalanced needs for capacity and throughput, traditional file systems require scaling the entire cluster, leading to capacity wastage.

For example, for a 500 TB capacity cluster using 8 TB hard drives with 2 replicas, 126 drives with a throughput of 150 MB/s each are needed. The theoretical maximum throughput of the cluster is 18 GB/s (126 ×150 = 18 GB/s). If the application demands 60 GB/s throughput, there are two options:

Switching to 2 TB HDDs (with 150 MB/s throughput) and requiring 504 drives
Switching to 8 TB SATA SSDs (with 500 MB/s throughput) while maintaining 126 drives

The first solution increases the number of drives by four times, necessitating a corresponding increase in the number of cluster nodes. The second solution, upgrading to SSDs from HDDs, also results in a significant cost increase.

As you can see, it’s difficult to balance capacity, performance, and cost. Capacity planning based on these three perspectives becomes a challenge, because we cannot predict the development, changes, and details of the real business.

Therefore, decoupling storage capacity from performance scaling would be a more effective approach for businesses to address these challenges. When we designed JuiceFS, we considered this requirement.

In addition, handling hot data is a common problem in AI scenarios. JuiceFS employs a cache grouping mechanism to automatically distribute hot data to different cache groups. This means that JuiceFS automatically creates multiple copies of hot data during computation to achieve higher disk throughput, and these cache spaces are automatically reclaimed after computation.

Managing massive amounts of files

Efficiently managing a large number of files, such as 10 billion files has three demands on the storage system:

Elastic scalability: The real scenario of JuiceFS users is to expand from tens of millions of files to hundreds of millions of files and then to billions of files. This process is not possible by adding a few machines. Storage clusters need to add nodes to achieve horizontal scaling, enabling them to support business growth effectively.
Data distribution during horizontal scaling: During system scaling, data distribution rules based on directory name prefixes may lead to uneven data distribution.
Scaling complexity: As the number of files increases, the ease of system scaling, stability, and the availability of tools for managing storage clusters become vital considerations. Some systems become more fragile as file numbers reach billions. Ease of management and high stability are crucial for business growth.

Concurrent load capacity and feature support in Kubernetes environments

When we look at the specifications of the storage system, some storage system specifications specify the maximum limit for concurrent access. Users need to conduct stress testing based on their business. When there are more clients, quality of service (QoS) management is required, including traffic control for each client and temporary read/write blocking policies.

We must also note the design and supported features of CSI in Kubernetes. For example, the deployment method of the mounting process, whether it supports ReadWriteMany, subPath mounting, quotas, and hot updates.

Cost analysis

Cost analysis is a multifaceted concept, encompassing hardware and software procurement, often overshadowed by operational and maintenance expenses. As AI businesses scale, data volume grows significantly. Storage systems must exhibit both capacity and throughput scalability, offering ease of adjustment.

In the past, the procurement and scaling of systems like Ceph, Lustre, and BeeGFS in data centers involved lengthy planning cycles. It took months for hardware to arrive, be configured, and become operational. Time costs, notably ignored, were often the most significant expenditures. Storage systems that enable elastic capacity and performance adjustments equate to faster time-to-market.

Another frequently underestimated cost is efficiency. In AI workflows, the data pipeline is extensive, involving multiple interactions with the storage system. Each step, from data collection, clear conversion, labeling, feature extraction, training, backtesting, to production deployment, is affected by the storage system's efficiency.

However, businesses typically utilize only a fraction (often less than 20%) of the entire dataset actively. This subset of hot data demands high performance, while warm or cold data may be infrequently accessed or not accessed at all. It’s difficult to satisfy both requirements in systems like Ceph, Lustre, and BeeGFS.

Consequently, many teams adopt multiple storage systems to cater to diverse needs. A common strategy is to employ an object storage system for archival purposes to achieve large capacity and low costs. However, object storage is not typically known for high performance, and it may handle data ingestion, preprocessing, and cleansing in the data pipeline. While this may not be the most efficient method for data preprocessing, it's often the pragmatic choice due to the sheer volume of data. Engineers then have to wait for a substantial period to transfer the data to the file storage system used for model training.

Therefore, in addition to hardware and software costs of storage systems, total cost considerations should account for time costs invested in cluster operations (including procurement and supply chain management) and time spent managing data across multiple storage systems.

Storage system comparison

Here's a comparative analysis of the storage products mentioned earlier for your reference:

Category	Product	POSIX compatibility	Elastic capacity	Maximum supported file count	Performance	Cost (USD)
	Amazon S3	Partially compatible through S3FS	Yes	Hundreds of billions	Medium to Low	About $0.02/GB/ month
	Alluxio	Partial compatibility	N/A	1 billion	Depends on cache capacity	N/A
Cloud file storage service	Amazon EFS	NFSv4.1 compatible	Yes	N/A	Depends on the data size. Throughput up to 3 GB/s, maximum 500 MB/s per client	$0.043~0.30/GB/month
	Azure	SMB & NFS for Premium	Yes	100 million	Performance scales with data capacity. See details	$0.16/GiB/month
	GCP Filestore	NFSv3 compatible	Maxmium 63.9 TB	Up to 67,108,864 files per 1 TiB capacity	Performance scales with data capacity. See details	$0.36/GiB/month
Lustre	Lustre	Compatible	No	N/A	Depends on cluster disk count and performance	N/A
	Amazon FSx for Lustre	Compatible	Manual scaling, 1,200 GiB increments	N/A	Multiple performance types of 50 MB~200 MB/s per 1 TB capacity	$0.073~0.6/GB/month
GPFS	GPFS	Compatible	No	10 billion	Depends on cluster disk count and performance	N/A
	BeeGFS	Compatible	No	Billions	Depends on cluster disk count and performance	N/A
	JuiceFS Cloud Service	Compatible	Elastic capacity, no maximum limit	10 billion	Depends on cache capacity	JuiceFS $0.02/GiB/month + AWS S3 $0.023/GiB/month

Conclusion

Over the last decade, cloud computing has rapidly evolved. Previous-generation storage systems designed for data centers couldn't harness the advantages brought by the cloud, notably elasticity. Object storage, a newcomer, offers unparalleled scalability, availability, and cost-efficiency. Still, it exhibits limitations in AI scenarios.

File storage, on the other hand, presents invaluable benefits for AI and other computational use cases. Leveraging the cloud and its infrastructure efficiently to design the next-generation file storage system is a new challenge, and this is precisely what JuiceFS has been doing over the past five years.

If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and community on Slack.

JuiceFS+MinIO: Ariste AI Achieved 3x Faster I/O and Cut Storage Costs by 40%+

DASWU — Mon, 15 Dec 2025 03:18:17 +0000

Ariste AI is a company specializing in AI-driven trading, with businesses covering proprietary trading, asset management, high-frequency market making, and other fields. In quantitative trading research, data read speed and storage efficiency often determine the speed of research iteration.

In the process of building quantitative research infrastructure, facing market and factor data with a total scale exceeding 500 TB, we went through four stages—from local disks to eventually choosing the JuiceFS file system on top of MinIO object storage. Through caching mechanisms and a layered architecture, we achieved fast access to high-frequency data and centralized management. This practice validates the feasibility of the integrated solution of cache acceleration + elastic object storage + POSIX compatibility in quantitative scenarios. We hope our experience can provide some reference for you.

Storage challenges in quantitative investment: balancing scale, speed, and collaboration

The quantitative investment process sequentially includes the data layer, factor and signal layer, strategy and position layer, and execution and trading layer. They form a closed loop from data acquisition to trade execution.

Throughout this process, the storage system faced multiple challenges:

Data scale and growth rate: Quantitative research requires processing a large total volume of data, covering historical market data, news data, and self-calculated factor data. Currently, the total volume of this data is close to 500 TB. Furthermore, our company adds hundreds of gigabytes of new market data daily. Using traditional disks for storage would clearly be unable to meet such massive data storage demands.
High-frequency access and low-latency requirements: High-frequency data access relies on low-latency data reads. The data read rate directly determines research efficiency. Faster data reads allow the research process to advance rapidly; conversely, slower reads lead to inefficient research.
Multi-team parallelism and data management: During quantitative research, multiple teams often conduct different experiments simultaneously. To ensure the independence and data security of each team's research work, secure isolation is necessary to avoid data confusion and leakage.

To address the data storage needs of the entire quantitative process and build a future-proof storage system, we wanted to achieve high performance, easy scalability, and management capability:

High performance: Single-node read/write bandwidth should exceed 500 MB/s, and access latency should be below the local disk perception threshold.
Easy scalability: The solution should support on-demand horizontal scaling of storage and computing resources, enabling smooth elastic scaling without application modification.
Management capability: The solution should provide one-stop management capabilities for fine-grained permission control, operation auditing, and data lifecycle policies.

Evolution of the storage architecture

Stage 1: Local disk

In the initial phase of the project, we adopted the QuantraByte research framework. It had a built-in exchange-traded fund (ETF) module allowing data to be stored directly on local disks. This resulted in fast data read speeds. Researchers could directly run the data they needed, and the iteration process was quick.

However, this stage had some issues:

Resource waste from repeated downloads: Multiple researchers downloading the same data led to redundant efforts.
Insufficient storage capacity: Research servers had limited storage capacity, only about 15 TB. This could not meet growing data storage needs.
Collaboration difficulties: The process was not convenient when needing to reuse others' research results.

Stage 2: MinIO centralized management

To solve the problems of the first stage, we introduced MinIO for centralized management. All stored data was centralized on MinIO, with a split-out module handling all data ingestion. Specific factor data was also stored in MinIO. This enabled unified downloads of public data. Permission isolation facilitated multi-team data sharing and improved storage space utilization.

However, new bottlenecks emerged in this stage:

High latency for high-frequency random reads: High-latency I/O operations during high-frequency data access impacted data read speeds.
Slow reads/writes due to lack of cache: Since the MinIO community edition lacked caching, reading and writing high-frequency public data was slow.

Stage 3: Introducing JuiceFS for cache acceleration

To address the above bottlenecks, after thorough research, we finally introduced JuiceFS’ cache acceleration solution. This solution involved mounting via client-side local RAID5 storage. With an efficient caching mechanism, it improved read/write performance by about three times. This significantly enhanced the access experience for high-frequency shared data.

As application data volume surpassed 300 TB, the scaling limitations of local storage became apparent. Since data was stored locally, scaling required reconfiguring storage devices. Scaling under a RAID5 architecture was slow and risky, making it difficult to meet the needs of continuous application growth.

Stage 4: JuiceFS + MinIO cluster

To solve the scaling challenge, we ultimately adopted the JuiceFS + MinIO cluster architecture. This solution offers the following advantages:

Sustained high performance: JuiceFS provides good caching capability, fully meeting the performance demands of high-frequency data access scenarios.
Easy cluster scaling: Based on the clustered solution, we quickly achieved horizontal scaling. Simply by adding disks of the same type, we can flexibly increase storage capacity. This greatly enhances system scalability.

Through this four-stage evolution, we validated the feasibility of the integrated solution combining cache acceleration, elastic object storage, and POSIX compatibility in quantitative scenarios. This solution can provide the industry with a replicable, implementable best practice template, achieving an excellent balance between performance, cost, and management.

Performance and cost benefits

By adopting the combined storage architecture of JuiceFS and MinIO, our system bandwidth and resource utilization efficiency greatly improved. Now they fully meet the storage performance requirements of the research application. After introducing the JuiceFS cache layer, backtesting task execution efficiency increased dramatically. The time required for backtesting 100 million entries of tick data was reduced from hours to tens of minutes.

We implemented a tiered storage strategy for managing the data lifecycle:

Hot data (0-90 days): This tier handles data that is accessed frequently. To ensure maximum performance, it’s automatically cached on local SSDs.
Warm data (90-365 days): Data with medium access frequency resides here. It’s stored on MinIO's standard object storage drives, striking an optimal balance between cost and performance.
Cold data (>365 days): This tier is for rarely accessed, archival data. It’s automatically migrated to a low-frequency access storage layer, which is compatible with the S3 Glacier strategy.

Based on this tiered storage strategy, we achieved a smooth transition from higher to lower storage unit costs. Our overall storage costs were reduced by over 40%.

Operational practices

Multi-tenant management

Regarding data isolation and permission management, we’ve established a comprehensive management system:

Logical isolation is achieved through namespaces, using path planning like /factor/A and /factor/B to ensure clear data boundaries for each application. For permission control, we implemented fine-grained management across three dimensions: user, team, and project. They seamlessly integrate with the POSIX ACL permission system.

We’ve also established a complete audit log system. It enables real-time tracking of access behaviors and historical change backtracking. This fully meets compliance requirements.

Observability and automated operations

We built a complete monitoring system around four critical metrics: cache hit rate, I/O throughput, I/O latency, and write retry rate. The system automatically triggers alerts when metrics are abnormal.

We implemented closed-loop operations management based on Grafana to continuously monitor node health and storage capacity. Before each scaling operation, we did simulated stress tests to verify system capacity and ensure no application impact. The overall operations system achieves high-standard goals of automation, predictability, and rollback capability.

Data update design in the backtesting system

In our backtesting system design, we adopted an architecture based on directed acyclic graphs (DAGs) to improve computational efficiency and maintainability. This framework centers on computational nodes and dependency relationships, abstracting data processing, feature calculation, signal generation, and other steps into nodes, all managed uniformly through a dependency graph. The system has a built-in version control mechanism. When data versions are updated, the dependency graph automatically identifies affected nodes, precisely locating parts that need recalculation, thereby enabling efficient incremental updates and result traceability.

Plans for the future

In future planning, we’ll continue to optimize the storage architecture in the following three aspects:

Metadata high availability upgrade: We plan to migrate metadata storage from Redis to TiKV or PostgreSQL to build a cross-data-center high-availability architecture. This can significantly improve our system disaster recovery and rapid recovery capabilities.
Hybrid cloud tiered storage: By integrating with public cloud S3 and Glacier storage services, we aim to build an intelligent hot/cold tiering system to achieve unlimited storage elasticity while optimizing costs.
Unified management for the research data lake: We’ll build a unified research data lake platform, integrating core services such as schema registration, automatic data cleansing, and unified catalog management. We hope to comprehensively improve the discoverability and management efficiency of data assets.

If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and community on Slack.

Design and Performance Optimization of juice sync for Enterprise Data Synchronization

DASWU — Wed, 10 Dec 2025 03:01:54 +0000

juicefs sync is a powerful data synchronization tool that supports data transfer between object storage, JuiceFS, and local file systems. It can maximize the efficiency of large-scale full migrations, achieving transfer speeds close to the upper limit of the network bandwidth. This makes petabyte-level data migration feasible within days or weeks (the actual duration depends on bandwidth and cloud throttling policies). Beyond this, it also supports accessing remote directories via SSH, HDFS, and WebDAV, while offering advanced features such as incremental synchronization, pattern matching (similar to rsync), and distributed synchronization.

In this article, we’ll focus on the core execution principles of sync, introducing its architecture and task scheduling mechanisms in both standalone and cluster modes. We’ll also provide a detailed explanation of how performance is improved through concurrency optimizations. In addition, the article will cover sync’s multi-dimensional filtering capabilities, flexible path-based pattern matching, and data consistency verification mechanisms. We aim to help users better understand and utilize this tool for efficient and reliable data synchronization.

How it works

The core execution principle of sync is consistent between standalone mode and cluster mode. The main differences lie in the deployment approach and the task distribution mechanism. Therefore, this section uses standalone mode as an example to explain the principles. We’ll explain the differences in cluster mode later.

As shown in the figure below, we can abstract the architecture of sync as a typical producer–consumer model:

The producer traverses all keys in the source and target object storage, merges the ordered results from both ends, determines whether objects need to be synchronized, deleted, or validated based on this, and writes the generated tasks into a task queue.
The consumer continuously retrieves tasks from the queue and performs the actual operations.

In standalone mode, the producer and consumer run within the same process and collaborate concurrently through goroutines (lightweight threads).

At the producer stage, the reason for simultaneously traversing the target end is that sync synchronizes data from the source and also supports the deletion of redundant objects on the target according to configuration. By combining the existence and different status of objects on the target, it determines what type of task to generate for each key. Currently, it supports five types of tasks: sync, delete from source, delete from target, copy permissions, and verify.

Next, we’ll further elaborate on the specific mechanism of task generation at the producer stage.

To improve task generation efficiency, sync requires that the traversal (list) interface of the object storage must return results in an ordered manner, that is, sorted by key in ascending order (such as lexicographically). Relying on this ordering, the producer can perform linear scans on both the source and target object storage simultaneously and compare the current key values one by one to determine whether a synchronization task needs to be generated.

Taking the merge process shown in the figure as an example, sync traverses the source and target synchronously. It reads the key at the current position from each side, for example, file1 from the source and file2 from the target:

juicefs sync file filtering workflow:

If src key < dst key (for example, file1 < file2), a synchronization task is generated. Then, the source pointer moves forward to continue comparing with the current target key.
If the keys at both ends are the same (for example, both are file2), it means the object exists on both sides. Whether to skip or perform further verification can be judged based on policy. Both pointers then move forward simultaneously.
If src key > dst key (for example, file4 > file3), it indicates an extra object exists on the target. This possibly requires a deletion task. Then, the target pointer moves forward.

The comparison logic continues until all keys from both source and target have been traversed. Thereby this constructs a complete task queue. The primary advantage of this strategy is that it requires only a linear scan to assess the state of all objects and quickly generate tasks.

Performance optimization

After understanding what the producer and consumer do, let’s examine where performance bottlenecks mainly occur and how to optimize them.

Produce optimization

During producer execution, list operations on object storage can become a performance bottleneck. The list interfaces of most object storage systems do not support specifying a range to return results for a particular interval, unlike segmented downloads where list operations for the same directory can be split into multiple intervals for parallel execution. Therefore, sync's optimization strategy is: execute list operations sequentially within a single directory, but execute list operations concurrently across multiple directories, and then aggregate the results from each directory into a complete key set.

In terms of the traversal method:

The first step is a breadth-first traversal with limited depth on the directory tree to find all subdirectories. This achieves directory-level splitting.
The second step involves a full traversal of each subdirectory, listing all objects under that prefix. This step requires that the list of objects be ordered.

Consumer optimization

The entire transfer process is an end-to-end pipeline: the consumer reads data from the source and writes it directly to the target, without passing through the local disk. Therefore, any bottleneck in either reading from the source or writing to the target can become a bottleneck in the synchronization flow.

During the upload process, sync supports concurrent execution of multiple tasks, handled by multiple goroutines. From a task perspective, the system possesses a degree of parallel transmission capability. The optimization focus is on enhancing read/write concurrency for large files.

Therefore, sync utilizes the object storage's multipart upload capability. It splits a large file into multiple parts, uploads them in parallel via multiple goroutines, and finally completes the merge via a combine interface.

For extremely large files, we can further split the parts:

Upload several intermediate objects.
Use the UploadPartCopy interface to perform splitting and merging of these intermediate objects directly within the object storage service.

This two-level splitting + two-level merging for the same file significantly improves the concurrency and efficiency of large file transfers.

Compared to the write path, the read-side concurrency logic is simpler: split the object into chunks and read each chunk concurrently. These concurrently read chunks form a readahead window. This means data blocks ahead of the current read position (offset) are pre-downloaded and cached as much as possible. The readahead mechanism helps reduce wait times when source read performance is insufficient or network jitter occurs, preventing the read side from becoming the performance bottleneck in the transfer pipeline.

Cluster mode

For larger data scales where single-node throughput is insufficient, sync also supports cluster mode. By horizontally scaling workers across multiple nodes to execute tasks in parallel, overall synchronization efficiency can be further enhanced.

Cluster mode does not introduce a new synchronization algorithm. The changes relative to standalone mode are mainly reflected in the physical deployment method and task distribution channels.

Physical deployment differences

In standalone mode, the producer and consumer run within the same process, cooperating via different goroutines. In cluster mode, consumers can be independently deployed to multiple nodes as worker nodes, while the producer serves as the manager node. This enables horizontal scaling.

Before enabling cluster mode, SSH password-free login needs to be configured between the manager and each worker node. After starting cluster mode, the manager distributes the juicefs binary to the /tmp directory of each worker node and starts the worker process remotely via SSH to join the synchronization task.

Task distribution mechanism differences

In standalone mode, the producer and consumer pass tasks via Go's chan within the same process. In cluster mode, task distribution switches to an HTTP-based approach—the manager provides a task allocation API, and each worker periodically polls the manager to pull tasks for execution. Except for the communication method, the rest of the flow is consistent with standalone mode.

Task filtering

In real-world synchronization scenarios, users typically only want to perform synchronization or cleanup operations on objects meeting specific criteria. sync supports filtering objects from multiple dimensions during the task generation stage and decides whether to generate a task and its type based on the filtering results. Overall, filtering and adjustment mainly include the following categories:

Based on time: Filter keys according to the file's mtime.
Based on size: Filter keys according to the file's size.
Based on key: Filter keys through pattern matching rules for paths/names.
Based on result: Filter or adjust task types by combining the key's existence status or differences on the target.

Among these, filtering based on key is the most commonly used and flexible. sync uses the --exclude and --include parameters for path pattern matching to precisely control the synchronization scope.

Currently, two matching modes are supported: the layer-by-layer filtering mode and the full-path filtering mode. The layer-by-layer mode is compatible with rsync's rules but complex to use. For details, see this document. The following will focus on the more intuitive and easier-to-use full-path filtering mode.

Since v1.2.0, the sync command added the --match-full-path option to enable the full-path filtering mode. In this mode, sync will match the full path of the object to be processed sequentially against multiple configured rules. Once a rule is matched, the processing result for that object (synchronize or exclude) is immediately determined, and subsequent rules are no longer evaluated.

For example, consider a file with the a1/b1/c1.txt path and three matching patterns: --include 'a*.txt', --include 'c1.txt', and --exclude 'c*.txt'. In full-path filtering mode, the string a1/b1/c1.txt is directly evaluated against the patterns sequentially:

Evaluate a1/b1/c1.txt against --include 'a*.txt'. The result is no match because * cannot match the / character. See Matching rules.
Evaluate a1/b1/c1.txt against --include 'c1.txt'. According to the matching rules, this will match successfully. Although the subsequent --exclude 'c*.txt' would also match according to the rules, based on the logic of the full-path filtering mode, once a pattern is matched, subsequent patterns are no longer attempted. Therefore, the final match result is "synchronize".

More examples:

--exclude '/foo**' excludes all files or directories whose root directory name is foo.
--exclude '**foo/**' excludes all directories ending with foo.
--include '*/', --include '*.c', and --exclude '*' includes only all directories and files with the .c suffix; all other files and directories are excluded.
--include 'foo/bar.c' and --exclude '*' includes only the foo directory and the foo/bar.c file.

Data integrity verification

Beyond high-concurrency transfers, correctness of the synchronization result is crucial. To mitigate risks from network jitter, concurrent writes during synchronization, and differences in consistency semantics across storage backends, juicefs sync provides a multi-level data integrity verification mechanism. This allows users to balance reliability and resource overhead based on application requirements.

Specifically, sync supports different verification levels when executing sync tasks:

Stricter verification provides stronger guarantees for data consistency between source and target but introduces higher read amplification and CPU consumption.
More relaxed verification has less impact on performance but correspondingly increases tolerance for potential inconsistencies.

Users can choose suitable verification parameters based on data importance and performance costs.

Taking the highest-level verification check-all as an example, its process has three steps:

Pre-transfer comparison: When determining whether an object needs synchronization, sync reads and compares the data content from the source and target. If the content differs, a synchronization task is generated and executed. In extreme scenarios, it might be necessary to read to the last byte to confirm inconsistency. At this point, both source and target undergo one full read before the transfer. This results in approximately 1x additional read traffic.
Transfer process verification: During synchronization, source data is read entirely again and written to the target. This results in approximately 1x read traffic on the source and 1x write traffic on the target. Simultaneously, sync calculates a checksum while reading the source data, obtaining the object's verification value after the read is complete.
Post-transfer verification: To verify the correctness of network transmission and target-side writing, sync reads the target object once more, calculates its checksum, and compares it with the checksum obtained during the transfer. If they are inconsistent, a data error has occurred, and an error is reported.

The table below shows comparison of sync verification modes and traffic consumption.

check-new can be seen as a subset of check-all: it only performs the above content verification on new or objects requiring transmission to achieve a balance between higher reliability and lower overhead.

check-change adopts a weaker verification approach: it does not directly compare content but infers whether an object has changed based on metadata like mtime and size. Since this strategy does not verify the actual content, there is a possibility that metadata remains unchanged while content is different. Therefore, this offers the lowest verification level with the smallest performance overhead

Conclusion

This article systematically introduced the architectural design, performance optimization, cluster mode, and task filtering mechanisms of juicefs sync. We aim to help users gain a deeper understanding of its implementation principles and usage methods. We’ll continue to optimize and evolve sync.

If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and community on Slack.

NAS vs. Object Storage vs. JuiceFS: Storage Selection of Billion-Dollar Quantitative Firms

DASWU — Fri, 28 Nov 2025 06:09:28 +0000

In the quantitative investment field, the performance and scalability of the storage system support efficient research and computational tasks. JuiceFS, an open-source high-performance distributed file system, has become the storage backbone for multiple top-tier, billion-dollar quantitative investment firms. It delivers high-performance, cost-effective, and elastically scalable storage infrastructure for their core operations—including backtesting and model training.

This article shares the critical storage challenges faced by quantitative firms and how JuiceFS addresses them. We’ll explore typical case studies focusing on cost optimization, metadata performance improvement, and seamless cloud migration.

Data types and storage challenges in the quantitative industry

First, let's understand the data usage patterns in quantitative research. The programming languages and data processing tools used are diverse, including Python, R, Java, MATLAB, and Conda. The data types involved are very rich, such as CSV, Parquet, TXT, Excel, and HDF. This requires the storage platform to be flexible enough to support various data formats efficiently.

The table below shows common data types in quantitative investing:

Data type	Data characteristics	Data format	Use case
Order book	GB~TB per day	Parquet, NPY	High-frequency backtesting, market making
Tick data	GB~TB per day	Parquet, NPY	High-frequency backtesting
K-line / bar data	MB~GB per year	Parquet	Low-frequency backtesting
Factor library	Tens of GB~TB per year	Parquet, Feather	Trading reference

In quantitative investing, data usage exhibits a distinct read-intensive, write-light pattern. Researchers frequently read data from unified data sources, while data writing and updating occur infrequently. Various types of market data and factor data are typically stored centrally and shared for read access by all researchers. Both factor research and trading strategy development involve repeated access and utilization of foundational data. Consequently, storage systems must deliver high-performance read capabilities to support rapid data access and analytical requirements.

This data usage pattern is also fully reflected in the critical task of quantitative research: backtesting. In a typical backtesting workflow, researchers repeatedly read historical market data to construct factors, generate trading signals, and validate strategy performance. Stable factors with strong predictive power are incorporated into factor libraries, serving as critical references for investment decisions.

Consequently, storage systems face a series of challenges:

The ability to handle diverse programming languages and data processing tools.
The requirement for high performance despite low capacity needs. Although storage capacity requirements typically range from tens to hundreds of terabytes, the performance demands on the storage system are extremely high, particularly regarding throughput and IOPS. Since quantitative tasks involve extensive machine learning and deep learning computations, the storage system must provide efficient sequential read throughput and fast random read IOPS to facilitate rapid data processing and analysis.
Adaptability to cloud environments. As more quantitative teams adopt public or hybrid clouds to build their technology platforms, storage systems need to possess high availability, elastic scalability, and flexible resource management capabilities. This meets dynamic storage demands and allows for quick adjustment of storage resources based on computational needs.
Integration with Kubernetes, supporting CSI drivers. This is because Kubernetes has become the mainstream solution for resource orchestration in quantitative research. This integration enhances platform flexibility and scalability and enables more efficient resource management for storage systems in cloud environments.
Data security and access control requirements. Protecting sensitive information like factor data and trading strategies is a top priority. The storage system must provide strong access control mechanisms to ensure data privacy and security. As more data moves to the cloud, features like encryption, auditing, and compliance management become essential. These measures guarantee that sensitive data remains secure in cloud environments.

Analysis of main storage solutions: NAS vs. object storage vs. JuiceFS

NAS

Many quantitative teams initially choose network-attached storage (NAS) for its out-of-the-box simplicity, typically using NFS or Samba protocols. This works well for data scales from tens to hundreds of terabytes in on-premises data centers. It’s easy to deploy and manage with controllable costs. However, several issues often emerge:

Cost: High-performance storage in on-premise data centers often requires all-flash NAS systems, which are expensive. Hybrid flash systems (mixing SSD and HDD) often lack the performance for high-concurrency access. This becomes a bottleneck when multiple researchers read data simultaneously.
Cloud NAS performance tied to capacity: In public cloud environments, the throughput of many NAS or parallel file system products is directly tied to purchased capacity. If a team needs 100 TB of storage but requires high throughput for backtesting (for example, 50+ GB/s reads or millions of IOPS), they might be forced to purchase PB-level capacity. This leads to waste.
Protocol-level bottlenecks: NAS protocols rely on kernel clients. Under high load, data and metadata competing for the same channel (prior to NFS v4.2) can cause congestion, performance drops, and latency spikes, especially during peak research or backtesting periods.

While NAS offers simplicity and good compatibility, its cost, scalability, and high-concurrency performance are significant limitations.

Object storage

Object storage is a low-cost alternative. While it has higher single-request latency, it handles large sequential reads and writes effectively. Many teams adapt their code for sequential access, but this approach has disadvantages:

Object storage can only be accessed via the S3 protocol. For applications already using POSIX mount points, migrating to object storage requires code modifications, which introduces intrusion into the existing system. Furthermore, object storage natively lacks efficient metadata management. Due to its flat structure, the response speed for I/O operations is slow, especially for operations like file renaming. Renaming a file essentially requires deleting the original file and creating a copy with the new name, which performs poorly.
Files on object storage are immutable. This means that if data needs to be modified (for example, adding a column to a Parquet file), it can only be achieved by creating a new file, rather than modifying the original file in place. This immutability can cause significant inconvenience in certain scenarios, such as when dealing with database files.
Frequent read operations on object storage incur extra data transfer fees. For the high-frequency read/write demands typical in quantitative finance, this can lead to substantial costs. Moreover, although object storage performs well for large sequential throughput, its performance is entirely dependent on the storage system itself. Under high-concurrency access patterns, it may still fail to meet the low-latency and real-time requirements crucial for quantitative research.

JuiceFS

JuiceFS is a POSIX-compatible distributed file system built on object storage. It manages both structured and unstructured data, providing a unified, high-performance storage platform for quantitative research. With CSI Driver support, it integrates deeply with Kubernetes and offers cloud-native flexibility for deployment across public, private, or hybrid cloud environments.

Architecture design

JuiceFS employs a separated architecture for data and metadata management:

Data is split into chunks and stored in object storage.
Metadata is handled by a dedicated engine and cached in memory.

This design delivers significant performance improvements. File metadata operations—including rename, lookup, and getattr—avoid frequent access to underlying storage. The system achieves high throughput, low latency, and high IOPS performance.

Security and access control

JuiceFS provides comprehensive security features:

Token-based access control
POSIX ACL support
Data encryption during transmission and at rest
Data chunking in object storage and cache for enhanced security and compliance

Product editions

JuiceFS offers two editions:

Community Edition: Users select their preferred metadata engine for scenario flexibility.
Enterprise Edition: Includes high-performance metadata service and distributed caching for demanding workloads.

Deployment process

Users deploy JuiceFS through a simple process:

Download and install the JuiceFS binary.
Mount using the juicefs mount command.
Access via POSIX mount points.

This approach maintains compatibility with existing applications, working like a local file system.

Advantages for quantitative research

This architecture delivers significant advantages for the quantitative industry:

High throughput and IOPS support: Data usage patterns in quantitative research are typically read-heavy write-light. Especially for tasks like backtesting and factor research, frequent reading of large datasets is required. JuiceFS' chunk-based storage and efficient multi-level data caching provide high throughput and IOPS support. This meets the demanding data access requirements of quantitative research.
Elastic scaling and cost reduction: Quantitative research often faces dynamic storage demands. JuiceFS inherently provides elastic scaling, allowing it to dynamically adjust storage and cache resources based on requirements. By using object storage as the underlying architecture combined with distributed caching, JuiceFS significantly reduces storage costs and avoids the high expense of all-flash storage.
Data security: In quantitative investing and research, factor data and trading strategies are core assets, and they are very important. JuiceFS effectively ensures data security through data chunking and encryption mechanisms. Particularly for institutions hesitant about moving to the cloud, this chunking and encryption strategy significantly reduces the risk of data leakage. Even if partial data is intercepted, attackers cannot reconstruct the original content without the metadata. This ensures the security and compliance of sensitive information.
Cloud environment compatibility: With the increasing adoption of cloud computing and public clouds, quantitative research is gradually migrating to cloud environments. JuiceFS supports cross-cloud synchronization and can operate efficiently in multi-cloud or hybrid cloud environments. This provides greater flexibility and scalability for quantitative research.

For on-premises quantitative firms, JuiceFS is also a viable and efficient choice. Traditional on-premises storage often uses hybrid flash or hard disk drives. Their performance often cannot meet the high-concurrency demands of quantitative research. By fully utilizing the local cache disks of application nodes and an efficient caching mechanism, JuiceFS can achieve performance close to all-flash at a low cost, significantly reducing hardware investment. Its operational simplicity and proven stability ensure reliable performance with light maintenance, as confirmed by customer feedback.

Case studies

Enhancing object storage performance and reducing costs

To further reduce storage costs, one of our customers migrated data to object storage. In their backtesting operations, they initially used a self-developed interface to directly access Amazon S3 object storage via the S3 API. While this solution could handle their primary workload of writing large data blocks and performing sequential reads—even without caching—it had several notable drawbacks.

Using the S3 API was intrusive to the application code. This means that if API issues arose, debugging and maintaining the application code would become very difficult. In addition, this API didn't implement caching and only handled protocol conversion. This imposed certain performance limitations. For these reasons, the application team didn't want to rely on this model for a long time.

To improve this situation, the company decided to migrate to JuiceFS Enterprise Edition. After migration, the application code no longer depended on the S3 API. JuiceFS, as a rich client, was directly mounted on the application nodes. This approach made the application code almost non-intrusive, allowing developers to continue compiling and running code in the original way. Meanwhile, the NVMe drives in the application nodes were used as distributed cache. This effectively optimized data access efficiency.

Using JuiceFS brought significant performance improvements. This enabled the client to achieve high-performance read/write operations on low-cost object storage:

Local cache avoided repeated access to remote storage. This significantly reduced read latency.
JuiceFS’ chunking and readahead mechanisms fully utilized object storage bandwidth. They enabled even single-file reads to reach system performance limits.

Overall, the migrated system showed noticeable improvements in both performance and maintenance convenience.

This company’s neural network training tasks also benefited from JuiceFS optimizations. During training, the system read approximately 100,000 files (each about 24 MB) and repeatedly read them after random shuffling. Without caching, each training epoch required reloading all files, generating massive network traffic. After implementing JuiceFS, data from the first read was automatically cached locally. This allowed subsequent training to directly fetch data from the cache pool. Bandwidth usage dramatically reduced and the overall training efficiency significantly improved. This fully met the company’s storage requirements for training.

Extremely high metadata demands

Another customer primarily uses JuiceFS for backtesting operations. They purchase market data from external sources, typically in CSV format, and store it in directories. Unlike other customers who aggregate data, this company stores raw data directly in folders. This creates a heavier data access burden. Historical market data for each stock is stored separately, with each file representing one year of data for a single stock. These files are distributed across different directories by month and year. Each time market data for a specific date needs to be read, the system must randomly read and process about 6,000 files and then arrange these files in sequence. To support backtesting, this data is used to launch hundreds of virtual machines, running 4,000 to 6,000 backtesting tasks.

This process of randomly reading large numbers of files placed enormous pressure on the storage system, particularly in high-concurrency read/write scenarios where metadata access efficiency severely impacted overall performance. The company initially used JuiceFS Community Edition combined with cloud-based Redis service to cache some data. While Redis provided good performance, its QPS limit of 500,000 couldn't meet the demands of the company’s high-concurrency backtesting tasks. When backtesting tasks reached 12,000, Redis performance hit a bottleneck. This caused the entire system to slow down significantly and prevented completion of backtesting tasks.

To solve this problem, the company tried JuiceFS Enterprise Edition. The metadata service in the Enterprise Edition showed significant performance improvements, particularly in handling large volumes of metadata requests with higher concurrency support. The Enterprise Edition's metadata service adopts a directory-tree based management approach and provides more efficient request processing. Customers no longer need to request multiple operations through key-value storage; instead, they simply inform the metadata system to read files through simple requests. The per-request processing efficiency of the Enterprise Edition metadata service is approximately 6 times that of the Community Edition. This means that for the same application operations, the required QPS is only one-sixth of the Community Edition's requirement. This results in higher overall execution efficiency.

In addition, JuiceFS Enterprise Edition provides a more powerful metadata client caching mechanism: numerous read-only requests (such as getattr and lookup) can directly hit the cache, significantly reducing the access pressure on the metadata service.

As shown in the figure below, the system handled about 500,000 QPS of read-only metadata requests, with only about 40,000 actually reaching the metadata server. Combined with the efficient processing capability of the Enterprise Edition metadata service, the overall system maintained stable operation in high-concurrency scenarios, with CPU utilization at only about 40%, enabling the company to stably support up to 12,000 backtesting tasks.

From this case, we can see that JuiceFS Enterprise Edition effectively addresses the challenges of high-concurrency read/write operations, high IOPS, and a lot of metadata requests in the quantitative industry. Even without optimizing for specific application scenarios, JuiceFS efficiently handles these intensive operational demands. It meets the quantitative industry's stringent requirements for data access speed and responsiveness.

Overcoming cloud NAS IOPS bottlenecks

In this case, our customer faced cloud storage performance issues when migrating their application to the cloud. They initially used cloud NAS to support their storage needs but quickly discovered that as the application expanded, the cloud NAS performance couldn't meet IOPS requirements. While using cloud NAS, the company’s 120 nodes already saturated the IOPS limits of standard cloud NAS. Even using high-speed cloud NAS (providing about 200,000 IOPS) still couldn't fully meet performance demands.

To address this bottleneck, the company decided to switch to JuiceFS. Through JuiceFS' distributed caching, the company could aggregate more IOPS performance. Theoretically, 5 cache nodes could provide 200,000 IOPS each. With local hard drive support, they could ultimately deliver over 1 million IOPS. This approach allowed the company to flexibly combine cache resources and scale storage performance on demand.

In extreme scenarios, the company achieved 7.5 GB/s throughput and nearly 3 million IOPS. This was sufficient to handle large-scale random read/write scenarios. Specifically, the company could increase throughput by adding network bandwidth and improve IOPS performance by increasing the number of cache nodes. Each cache node can provide about 100,000 to 200,000 IOPS. The company can also expand capacity by mounting additional hard drives or utilizing memory as needed. This allows the company to freely combine and adjust cache pool capabilities according to actual application requirements, ensuring system performance.

Summary

Based on the multiple cases discussed above, we can summarize several distinctive characteristics of quantitative research regarding storage:

Quantitative applications typically have small data volumes but extremely high performance requirements. JuiceFS significantly improves access efficiency through data and metadata separation and caching mechanisms. Even without caching, it can fully utilize bandwidth through readahead and data chunking upload mechanisms. This ensures high-performance data processing capabilities.
JuiceFS' data chunking and encryption mechanisms enhance cloud storage security. Data is split into chunks and encrypted before storage. Even if partial data is stolen, attackers cannot reconstruct the files without metadata. This design is particularly suitable for protecting sensitive data like quantitative factors and trading strategies.
As quantitative applications gradually migrate to cloud environments, JuiceFS provides a solution that balances performance and cost. It achieves high-performance access and unified management on top of object storage. This meets diverse computing needs in public and hybrid cloud environments and builds scalable, reliable data infrastructure for quantitative institutions.

If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and community on Slack.

Deep Dive into the JuiceFS Garbage Collection Mechanism

DASWU — Fri, 07 Nov 2025 07:12:45 +0000

When using JuiceFS, you might have these questions:

Why isn't the object storage space freed up immediately after deleting files?
How can I efficiently clean up a large number of files accumulating in the trash?
Why is the deletion operation so slow or performance degraded when deleting files in batches within a short period of time?

The workflow behind JuiceFS' garbage collection (GC) is quite intricate. It’s hard to intuitively grasp when file states change and resources are truly released.

To pull back the curtain, this article will systematically break down the key processes and underlying logic of GC, explain the roots of common issues, and offer practical advice for efficient cleanup and system maintenance.

Background knowledge: JuiceFS file storage structure

JuiceFS handles file deletion through an asynchronous mechanism. It responds to user delete requests instantly but postpones the actual cleanup for later, when the system is not busy. This approach smooths out workload spikes, reduces system load, and enhances system stability. Furthermore, this phased cleanup strategy offers greater flexibility for potential data recovery.

This efficient design is rooted in JuiceFS' underlying architecture. It employs a separation of metadata and data storage, and manages file data in chunks. When you delete a file, the system doesn't immediately remove the actual data. Instead, it only adds a deletion marker in the metadata layer. The real data reclamation happens later in an asynchronous process.

To fully grasp this asynchronous deletion logic, we need to understand how JuiceFS organizes data internally. The system manages file data through chunks, slices, and blocks. This structure not only dictates file performance during storage and access but also directly influences how data is located, referenced, and cleaned up during deletion.

Chunks: Each file consists of one or more chunks, each with a maximum size of 64 MB. Regardless of file size, all read and write operations locate the corresponding chunk based on offset. This boosts retrieve and access efficiency.
Slices: This is the unit of writing within a chunk. Each continuous write operation generates one slice. A slice must reside entirely within a single chunk, so its size cannot exceed 64 MB. The process of writing a file typically produces multiple slices.
Blocks: To improve write efficiency to object storage, each slice is further split into multiple blocks (default maximum size is 4 MB) before being written. Multi-threaded concurrent writing is used to improve throughput.

JuiceFS GC process

Now that we've covered the background, let's walk through the main file deletion workflow. JuiceFS' deletion mechanism can be divided into three key processes:

Trash management
File deletion processing
Underlying slice cleanup

Trash management

The file deletion journey begins with a user's delete request. When an upper-layer interface initiates a file deletion, the system doesn't remove the file immediately. Instead, it moves the file to the trash (enabled by default). In this step, only the pointer from the file's parent directory changes; the file's other metadata and its actual content remain unchanged.

Once a file stays in the trash beyond its configured retention period, a background cleanup task or a manual operation converts it into a "pending deletion" state, pushing it into the subsequent cleanup pipeline.

To better understand and manage files in the trash, let's look at its directory structure and operation.

The trash resides in the .trash directory under the mount point. Deleted files are saved into subdirectories named based on the deletion time (accurate to the hour), for example, .trash/2024-01-15-14/. To avoid file name conflicts, files moved to the trash are automatically renamed following this pattern: <ParentDirID>_<FileID>_<OriginalFileName>.

The system also provides a flexible recovery mechanism for files in the trash. See Trash for details. Files in the trash aren't kept forever. JuiceFS cleans them up in two ways:

Automatic cleanup: A background task (cleanupTrash) runs hourly. It gradually cleans files based on the set retention period, marking them for deletion.
Manual cleanup: You can use the OS rm command or the juicefs rmr tool to directly delete files or directories from the trash. Manual cleanup typically reclaims space faster than waiting for the background task, but the process is asynchronous. The actual deletion of objects from storage is handled by subsequent tasks, which will be covered later.

Furthermore, trash cleanup performance is constrained by the processing capacity of a single client. When dealing with massive file deletions, consider mounting JuiceFS on multiple clients and performing manual cleanup simultaneously to significantly boost the overall efficiency of large-scale deletion operations.

File deletion

In this stage, the system first checks if the file is still in use:

If the file hasn't been closed (for example, it's still held open by a process), it's placed in a sustained state, waiting for all references to be released before re-entering the cleanup workflow.
If the file is closed, the system marks it as a deletable file (internal state delfile), indicating it's ready for the deletion queue.

Next, the system attempts to add these deletable files (delfile) to the file deletion queue. This queue acts as a buffer for files awaiting cleanup. To prevent impacting system performance, the queue has a limited capacity. If the queue is full, deletion requests for regular files are temporarily skipped, letting the background cleanup task handle them later.

For files in the trash, if a user performs a manual delete operation while the deletion queue is full, that operation will block synchronously until space becomes available in the queue. This ensures manual deletions are queued within the current process and won't be skipped.

To guarantee all deletable files are eventually processed, JuiceFS provides a background task named cleanupDeletedFiles that runs hourly. This task scans for files still marked as delfile and adds them into the deletion queue in batches, ensuring the cleanup pipeline keeps moving.

Once a file enters the deletion queue, the system scans all its associated chunks. For each chunk, it decrements the reference count for every contained slice and then cleans up the chunk metadata. Slices whose reference count drops to zero are marked as pending cleanup, entering the slice cleanup stage.

Slice cleanup and space reclamation

In the previous stage, the reference counts for all slices associated with the deleted file were updated. Now, the system checks the current reference count for these slices:

If it's still greater than 0, the slice is still being used by another file and won’t be cleaned up.
If the reference count has reached 0, it means the slice is completely obsolete, and the system adds it to the slice deletion queue for the final cleanup step.

A scheduled background task, cleanupSlices, periodically scans this queue and performs batch deletion. Users can also trigger a manual GC to actively remove these invalid slices. Slices entering this workflow are precisely located down to their physical blocks in the object storage. The actual delete operations are performed, finally freeing up the storage space.

It's important to note that the slice deletion queue itself is also subject to concurrency and capacity limits. You can use the --max-deletes parameter to adjust the number of concurrent threads. This controls how many delete operations happen simultaneously and thus balances resource usage and cleanup speed.

Compaction

The previous sections covered the complete file deletion and reclamation process. However, JuiceFS deals with another type of "hidden garbage" – redundant or useless file fragments. Because files are stored in fixed-size chunks, and each chunk contains multiple slices, frequently overwriting or performing random writes to the same file can leave the slices within certain chunks scattered and non-contiguous. This is known as fragmentation.

While fragmentation doesn't prevent files from being read correctly, it has bad effects on several performance and resources:

Read amplification: Reading a single chunk might require putting many dispersed slices together. This increases I/O operation complexity.
Increased resource usage: More fragments mean more metadata and potentially more object storage space is consumed. This strains the system and reduces overall performance.

To combat fragmentation, JuiceFS provides a compaction feature. It identifies and merges redundant or scattered slices within chunks, reconstructing them into larger, more contiguous data blocks. This significantly reduces the number of fragments and improves storage efficiency. JuiceFS supports two triggering mechanisms:

Automatic triggering: During file read/write operations, the system monitors the number of slices in each chunk. When the slice count for a chunk reaches a predefined threshold, the system automatically starts an asynchronous compaction task. This process involves:

1. Applying for a new slice ID at the metadata layer to build the new, contiguous data.  
2. Reading the data from the old, scattered slices from object storage and writing it into the new, contiguous block. 
3. Decrementing the reference count for the old slices. If the trash feature is enabled, these old slices are marked for delayed processing, handed over to background tasks for reference count updates and eventual cleanup.

Manual triggering: Users can use the juicefs compact command to proactively scan specified paths and perform concurrent defragmentation. The --threads parameter controls the number of concurrent threads, allowing you to adjust based on system load to balance performance and resource usage. Check the command document for detailed usage.

# Compact one or multiple paths; concurrency can be specified. 
juicefs compact /mnt/jfs/pathA /mnt/jfs/pathB --threads 16 
# Shorthand 
juicefs compact /mnt/jfs/path --p 16

To help users monitor the status of compaction handling, JuiceFS provides metrics viewable via the juicefs status command. For details, see the next section.

GC tools principles and usage

The GC tool

The GC tool in JuiceFS is an auxiliary utility for manual cleanup, supporting modes like check-only (dry-run) and scan-and-clean. Its core principle involves scanning and comparing metadata with the actual contents of the object storage. It identifies and processes objects that are no longer referenced or managed by the metadata, ensuring data consistency and reclaiming underlying space occupied by invalid data.

In certain corner cases or due to operational errors, data segments might exist in the object storage that are detached from JuiceFS' metadata management and create what's known as "object storage leaks." While rare, the GC tool can scan for and remove these. Beyond that, GC can also clean up invalid records within the metadata, such as files or slice information that wasn't fully removed during the standard cleanup process, and can accelerate various stages of the GC process, like manual compaction or manual cleanup of delfile files.

When running, the GC command displays a progress bar. It allows you to monitor the task's status in real time. The status items shown in the progress bar correspond directly to the concepts illustrated in the earlier figures, helping you better understand the GC workflow.

Some users have expressed confusion about the meaning of these status items, especially in contexts like resource billing. Correlating the figures with these status messages can greatly enhance comprehension of the entire GC mechanism's operation.

GC metrics:

Pending deleted files/data: Number of files / amount of data awaiting deletion.
Cleaned pending files/data: Number of files / amount of data deleted by this GC run.
Listed slices: Total number of slices in the file system.
Trash slices: Slices within the trash.
Cleaned trash slices/data: Number of trash slices / amount of data cleaned by this GC run.
Scanned objects: Total objects in object storage.
Valid objects/data: Valid objects / amount of data in object storage.
Compacted objects/data: Objects / amount of data after being processed by compaction.
Leaked objects/data: Leaked objects / amount of data.
Skipped objects/data: Objects / amount of data skipped during this GC run.

Status metrics

juicefs status displays various status information. Metrics relevant to GC are listed below. These metrics correspond to different stages of the cleanup process and serve as crucial evidence for judging the system's cleanup progress and task status. If you notice certain fragments continuously increasing, or the background processing can't keep up, consider using manual cleanup methods to accelerate the process to prevent system performance degradation.

Trash files: Files in the trash.
Pending deleted files: Files marked for deletion.
To be sliced: Fragmentation tasks pending compaction.
Trash slices: Hidden slices in the trash (resulting from compaction), potentially usable for recovery.
Pending deleted slices: Slices with zero reference count, waiting for cleanup.

Operational suggestions

Finally, here are some fundamental yet practical operational suggestions to help manage JuiceFS system resources more efficiently and reduce operational risks in daily use:

Set an appropriate trash retention period based on your specific application needs, balancing data recoverability against storage cost. For systems with frequent file turnover or high storage cost sensitivity, consider shortening the retention period to minimize space waste.
It's recommended that operators periodically use tools like juicefs status to monitor key cleanup metrics, such as pending deleted files, delayed slice count, and trash space usage. This helps identify potential cleanup lag or resource buildup early.
If automatic background tasks are disabled for your system, schedule regular execution of GC or related cleanup commands to remove invalid residues from metadata and object storage. This prevents long-term performance issues and resource waste.
Schedule all cleanup tasks to run during off-peak application hours whenever possible to minimize interference with normal read/write operations and ensure overall system stability. By adopting this approach, users can establish a more predictable rhythm for managing the deletion and reclamation lifecycle, achieving ongoing optimization of resource utilization efficiency.

If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and community on Slack.

Building AI Inference with JuiceFS: Supporting Multi-Modal Complex I/O, Cross-Cloud, and Multi-Tenancy

DASWU — Fri, 24 Oct 2025 07:40:50 +0000

The demand for inference services is rapidly growing in the market and has become a key application of widespread interest across various industries. With the continuous enhancement of model inference capabilities through techniques like chain-of-thought and slow thinking, their application scope keeps expanding, holding significant importance for enterprises deploying AI infrastructure.

Inference services impose higher requirements on storage systems. There is a need not only for high-performance, low-latency data access capabilities but also for features like elastic scaling, resource isolation, and multi-cloud deployment. Choosing the right storage solution has become a crucial issue enterprises must address when building AI infrastructure. JuiceFS has been widely adopted in various inference service scenarios, serving different types of customers including foundational model platforms, multi-modal applications, and enterprise-grade AI services. Based on in-depth analysis of these practical cases, we’ve summarized a series of representative storage requirements and challenges.

This article is divided into two sections:

The first section focuses on the storage access patterns in inference services, combining JuiceFS application practices to outline key requirements and common problems in inference scenarios.
The second section compares the advantages and limitations of several mainstream storage solutions, helping users make more informed selection decisions when building inference systems.

Storage requirements of inference services

JuiceFS is already widely used in inference scenarios. Through collaboration with customers, we’ve identified several typical storage requirements for inference workloads:

More complex multi-modal I/O patterns: Inference is rapidly evolving towards multi-modal directions, involving extensive small file management, mmap-style random reads, and other access patterns. These scenarios are extremely sensitive to - I/O throughput and latency, directly impacting model response speed and user experience.
Cloud-native deployment needs: Inference traffic typically has distinct peaks and troughs, requiring the underlying storage to quickly respond to sudden requests and coordinate with cloud-native scheduling systems to achieve scaling within minutes.
Clear trend towards multi-cloud/multi-region architectures: To enhance service availability and disaster recovery capabilities, more customers deploy inference services across multiple clouds or regions. The storage system needs to possess strong cross-region consistency and cost-control capabilities.
Emphasis on both resource sharing and isolation: Inference tasks prioritize resource reuse, especially in clusters shared by multiple tenants or application lines. Balancing effective isolation and data security with high resource utilization becomes fundamental for stable system operation.

Model loading performance challenges

We can analyze file access requirements through the various stages of an inference service. The main stages of an inference service include:

As mentioned, file access requirements are different at each stage. I/O bottlenecks are primarily concentrated during the cold start phase (model loading) and runtime model switching, especially concerning how to efficiently read model files.

Solution 1: Packaging model files into container images

In small-scale inference scenarios, many users package model files, code dependencies, and extension packs into container images and start them via Kubernetes. While this method was once simple, it exposes several issues as inference scale increases:

Oversized image files: As model files grow larger, container images also become massive, leading to timeouts during image pulls.
Slow extraction: Container images use layered storage; unpacking each layer is a serial operation. With large models, this unpacking process becomes slow and inefficient.
Resource waste: If multiple inference services need the same model, sharing the model efficiently is difficult. This leads to frequent download and unpack operations, wasting significant resources.
Frequent model switching: This approach is even less efficient in scenarios requiring frequent model changes.

Solution 2: Storing model files in object storage

Each inference instance fetches model files from object storage upon startup. While this avoids image pull timeouts, it still faces challenges:

Bandwidth competition: In large-scale clusters, when multiple inference instances pull models simultaneously, bandwidth contention can occur. This prolongs startup times and affects user experience.
Inefficiency: Many model file formats (like Safetensors) use mmap for lazy loading. However, object storage doesn't support mmap. This forces the system to download the file locally before reading. This wastes local storage space and reduces read efficiency.

Solution 3: Using a high-performance file system like JuiceFS

Compared to object storage and container images, storing model files in a high-performance file system is a more effective solution. JuiceFS, as an implementation of this solution, offers the following advantages:

On-demand configuration: Unlike traditional high-performance file systems, JuiceFS allows flexible configuration of performance and capacity independently, avoiding the tight coupling of performance and capacity.
Multi-cloud data distribution capability: Traditional cloud file systems often lack support for efficient data distribution across cloud environments. JuiceFS possesses cross-cloud distribution capabilities, enabling efficient data distribution across multiple cloud environments, enhancing flexibility. We’ll explain later how JuiceFS is used in multi-cloud architectures.

The figure below shows the improvement in the model file loading process after introducing JuiceFS. When building the container image, we separate the model files from the image, storing them in JuiceFS, while the container image only contains the application code and the basic runtime environment.

This approach brings significant benefits:

Model files are decoupled from the image cold start process. This avoids the need to unpack large model layers locally.
Compared to the object storage solution, during cold start with JuiceFS, there's no need to download the complete model file to the local node, nor load the entire model file (such as Safetensors files). Inference instances can complete cold starts quickly. This on-demand loading mode is particularly critical in elastic scaling scenarios, ensuring new instances start rapidly and begin serving requests promptly. Although the first inference might experience slight latency due to on-demand model loading, the speed for subsequent inference requests is significantly improved.

Beyond these optimizations, JuiceFS further enhances inference service performance through its powerful caching capabilities. In addition to the local cache functionality provided in the Community Edition, JuiceFS Enterprise Edition further offers distributed cache functionality. This allows the construction of large-scale cache pools, centrally storing frequently used model files in the cache, thereby significantly boosting read performance.

Especially in scenarios involving large-scale instance startups or scaling, such as applications like Stable Diffusion that require frequent model switching, the distributed cache cluster can substantially reduce model loading times. This markedly improves the front-end user experience. The figure below shows the JuiceFS distributed cache architecture. After deploying the inference service, we mount JuiceFS to the inference nodes. When a node reads a model file, it first attempts to retrieve the data from the cache cluster.

Since the cache cluster consists of high-speed media and communication occurs over high-speed internal networks, data retrieval performance is excellent. On a cache miss, the respective cache node fetches the data from the object storage, returns it to the node, and fills the cache. We can also use the warmup command to pre-fill the cache, loading the required model files into the cache layer before nodes start. When inference instances start, all instances hit the model files directly from the cache cluster. This eliminates the need for each instance to pull files individually from object storage and thus avoids bandwidth competition issues on the object storage during high-concurrency file pulls.

Multi-region model file synchronization and consistency management

An increasing number of enterprises are adopting multi-cloud architectures to deploy inference services, flexibly scheduling resources across different cloud providers to optimize cost structures. As shown in the figure below, in a typical inference service deployment case, a customer built infrastructure spanning multiple cloud service providers and data centers to support high-concurrency, low-latency inference requests. A multi-cloud architecture not only effectively mitigates the risk of application interruption from single points of failure but also supports dynamically selecting the most suitable compute nodes based on factors like regional network conditions, resource pricing, and computing power, thus achieving optimal cost control while ensuring performance.

To ensure consistency of inference models and configuration files across different regions, customers typically use object storage in a primary site as a unified model repository, centrally managing model versions and configurations. During the inference task scheduling process, the system can select suitable computing resources based on availability, while model files need to be deployed as close as possible to the computing resources or within high-throughput network environments to improve cold start efficiency and service response speed.

In traditional practice, model files often required manual migration to the scheduled computing cluster. As models iterate frequently, this leads to cumbersome operations and potential data inconsistency issues. Furthermore, inference clusters are typically large-scale. During cold start, concurrent model pulling by nodes can cause bandwidth contention. This creates performance bottlenecks, especially in multi-cloud architectures where model distribution and loading become a significant challenge.

JuiceFS addresses this by using a mirror file system to achieve real-time metadata synchronization from the primary site to various mirror sites. Clients local to each site see a unified file system view. For data files, it offers two schemes regarding whether to perform full synchronization or to cache and load on demand:

Asynchronous replication: it copies the object storage data completely to multiple sites. Each site prioritizes reading from its local object storage.
Deploying only a distributed cache layer at the mirror sites without persistent data storage. Required data is warmed up into the cache on demand, thereby improving local read performance.

In large-scale inference services, considering cost-effectiveness, users often choose the second scheme, deploying distributed cache locally within the site. A pipeline task is added before node startup and scaling to warm up the required model files in one go into the local site's cache cluster where the computing resources are allocated. This stage essentially performs a single high-concurrency pull of the model files from the object storage.

The benefit is that when a large number of inference services perform cold start simultaneously, they only need to concurrently read the model files from the local domain's cache cluster, without repeatedly pulling from the object storage over the dedicated line. This solves the problem of dedicated line bandwidth bottlenecks caused by high-concurrency repeated pulls from object storage.

Model version iterations can also be handled through warm-up scripts, pre-loading new versions into the cache while removing expired versions. The entire data distribution and cache warming process is fully handled by the JuiceFS mirror file system feature. This is transparent to the user, avoiding a significant amount of manual data copy work.
The benefits brought by the JuiceFS Mirror Filesystem include:

Cost reduction: Each mirror site only needs to deploy an appropriately sized cache cluster, avoiding the need for excessive capacity configuration.
Efficiency improvement: The distributed cache uses local high-speed media, providing excellent throughput performance. The network is the local region's internal network, offering good performance.
Ease of use: No manual intervention in data copying. The unified data view automatically handles data distribution for the user. Even on a cache miss, data is automatically fetched from the object storage.
Scalability and stability: Resolves bandwidth competition issues during concurrent model file pulls by multiple nodes, enhancing system performance and user experience.

Cloud-native support and cluster management

Inference services typically have bursty and intermittent load characteristics, so they demand high resource elasticity. As the scale of inference services expands, the number and node scale of compute clusters also increase. It often requires scheduling across clusters and regions. This demands that the storage system can easily support deployment in multi-cluster environments.

Furthermore, in elastic environments, the file system needs to ensure transparency to the application during resource expansion, upgrades, or failure recovery, without affecting the normal operation of applications. This means the file system must possess capabilities like automatic failure recovery, resource isolation, and seamless upgrades to ensure application continuity and stability.

JuiceFS provides a range of capabilities to support inference services in cloud-native environments.

JuiceFS offers a standard Kubernetes CSI Driver. Users can conveniently use PVCs to access the JuiceFS file system in Kubernetes environments, supporting both static and dynamic provisioning. One of its advantages is multi-environment compatibility: to match the resource elasticity needs of inference services, JuiceFS offers multiple operational modes suitable for both standard and serverless Kubernetes environments. Regardless of the elastic load type used by the inference service, JuiceFS deployment is simple and easy.
In terms of resource optimization, JuiceFS supports multiple applications using the same PVC to share the mounted volume and allows fine-grained definition of resources. This enables optimized resource utilization and isolation between tenants to ensure different tenants do not interfere with each other when using different PVs.
For stability optimization, JuiceFS supports automatic recovery of mount points and smooth upgrades of Mount Pods. When a Mount Pod fails and needs restarting, or when an upgrade of the CSI Driver or Mount Pod image is required, the application Pod does not need to restart and can continue running normally.
Regarding automated management, JuiceFS provides various Operators, including features for cache group management, warm-up, and data synchronization. Users can conveniently manage these resources via CRDs, enabling automated operations.
For visualized monitoring, JuiceFS offers a complete visual management platform, supporting monitoring for both Mount Pods and application Pods. Users can also perform operations like smooth upgrades on the platform, all manageable through a unified interface.
Many users integrate the Karmada multi-cluster management framework with JuiceFS CSI Driver to achieve unified management of resource scheduling and storage configuration in multi-cluster environments. Thanks to the simplicity of the JuiceFS architecture, the only background service that needs independent deployment is the metadata service. All other features are implemented within the client. Only the client needs deployment on the computing nodes. In complex multi-K8s cluster scenarios, users can use a multi-cluster management control plane like Karmada to distribute JuiceFS-related CRDs to different clusters according to policies. This enables simple storage volume deployment and efficient operations.

Multi-tenancy and data isolation

In AI inference service scenarios, the demand for multi-tenant data isolation is particularly pressing. On one hand, for AI foundational platforms and AI applications serving external users, requirements for data security and isolation are high. On the other hand, large enterprises also need data isolation and access control between different internal departments and project teams. Overall, requirements in multi-tenant environments mainly include:

Data isolation & access control: Data needs to be isolated between different tenants, with user-specific access permission management.
Storage quota management: Storage capacity needs to be allocated and managed for different tenants.
Resource isolation and performance management: Different tenants may require independent resource isolation, with managed and optimized performance (QoS).

To address these needs, JuiceFS provides three data isolation schemes:

Scheme 1: Isolation via independent file systems

This method allocates different JuiceFS file systems to different tenants for strong data isolation. It means different tenants use their own separate JuiceFS file systems. If further resource and performance isolation is required, the metadata services for the file systems and the associated object storage buckets can be completely isolated. This scheme is generally applicable to tenant environments in AI foundational platforms serving business customers (toB). These tenants typically have large-scale application requirements and high demands for service quality. This scheme fully ensures the tenant's data access performance is unaffected by load from other tenants sharing resources. Furthermore, data is physically completely isolated, providing the highest level of security assurance.

Scheme 2: Subdirectory isolation within the same file system

This scheme allocates different subdirectories on the same file system to different tenants. Subdirectories are invisible to each other. Access control for mounting subdirectories can be enforced using JuiceFS' tokens, strictly ensuring data isolation between subdirectories. This scheme is more commonly used in tenant environments like AI foundational platforms serving consumers (toC), or for data isolation needs between different project teams within an enterprise.

In K8s environments, JuiceFS CSI Driver, by default, allocates different subdirectories on the file system for dynamically provisioned PVCs, and different PVCs do not reuse the JuiceFS client by default. This achieves data isolation for different applications while allowing separate resource control and parameter optimization for each application client. Of course, in non-K8s environments, data security can also be ensured by using tokens to control client mounting permissions for subdirectories. In this scheme, the chunk data in object storage is not isolated, but these JuiceFS chunks cannot be read directly as original file content from the object storage, so there is no risk of data content leakage.

Scheme 3: Fine-grained access control based on POSIX ACL

Based on Linux ACL, JuiceFS Enterprise Edition extends group permission control capabilities. Each file can have permissions set separately for different groups. Combined with the UID/GID mapping feature, consistent ACL permission management can be achieved even across different nodes, as long as the usernames and group names are consistent. This scheme is suitable for small-scale users requiring fine-grained access control over individual directories or files. It has high configuration complexity and limited use cases.

Furthermore, JuiceFS Enterprise Edition provides two key features supporting multi-tenant management:

Client access tokens: Used to authorize client access to the file system. It can control access IP ranges, read/write permissions, subdirectory mount permissions, and permissions for clients to perform background tasks. It can also limit object storage access traffic, implementing quota or rate limiting management.
Quota management: Allows allocation of storage capacity and file count quotas for each subdirectory. Combined with the subdirectory data isolation scheme, this enables precise control over storage resource usage for each tenant.

Comparison of different storage solutions for inference services

In practical applications for inference services, different types of storage solutions have significant differences in performance, stability, scalability, and cost. The following section analyzes and compares several typical categories to help enterprises evaluate their suitability more comprehensively:

Object storage is a low-cost solution suitable for scenarios involving sequential reads of large files. However, its performance is poor when handling small files, model files (like Safetensors format), and multi-modal data frequently used in inference tasks. When multiple inference nodes read data in parallel, the bandwidth of object storage can easily be exhausted. This leads to performance bottlenecks. While it supports eventual consistency and nearly unlimited storage capacity, it lacks optimization for high concurrency and small file access and does not support automated data distribution across heterogeneous multi-cloud environments.
High-performance file systems offer higher performance, especially suitable for scenarios with high demands on computing resources and performance. They fully support POSIX-compliant frameworks and provide strong consistency. However, they often tightly couple performance and capacity, requiring large storage capacities to achieve optimal performance. While generally stable, because the client deploys in kernel space, a single client failure can potentially crash the entire system. Operational costs are also high.
Pure cache systems + object storage is a compromise, configuring distributed cache to provide high throughput and low-latency read/write performance. However, their performance in scenarios involving massive small files and high-frequency metadata requests needs further validation. This type of solution is hard to guarantee strong data consistency in multi-cloud/multi-region architectures.
JuiceFS + object storage provides a multi-tier caching architecture, allowing flexible configuration of local and distributed cache. It can deliver high throughput and low-latency data read/write performance, particularly suitable for the high-concurrency, low-latency demands of inference services. It not only supports efficient metadata operations but also fully utilizes Ethernet bandwidth without relying on expensive proprietary hardware, ensuring excellent performance among similar systems. JuiceFS also possesses strong consistency and supports automated data distribution in multi-cloud environments, guaranteeing consistency across regions and clusters.
From a scalability perspective, JuiceFS supports up to 100 billion files and TB/s level throughput, configures cache on-demand, and scales massively. Object storage is suitable for scenarios with virtually unlimited file count and capacity, but its bandwidth is limited by the cloud provider's offerings. The scalability of high-performance file systems is often constrained by the tight coupling of performance and capacity.

The table below is a comparison summary of different storage solutions:

Summary

Compared to training tasks, inference services impose different and more stringent requirements on infrastructure. They especially emphasize low-latency response, high-concurrency access, resource isolation, and flexible scalability. Therefore, selecting an appropriate storage solution has become a key decision point for enterprises deploying AI inference systems.

This article has outlined the core storage requirements and typical practical challenges in inference scenarios, covering extensive random access during model cold starts, high-concurrency reads of small files, multi-modal data processing, resource sharing among multiple tenants, and complex scenarios like cross-region multi-cloud deployment. Based on JuiceFS' practical implementation experience with numerous inference users, we’ve proposed a series of optimization strategies to address these challenges.

The article also compared current mainstream storage solutions—including object storage, high-performance file systems, pure cache systems, and the architecture combining JuiceFS with object storage—helping users make more targeted selection judgments from multiple dimensions such as performance, stability, scalability, and cost.

With its elastically scalable architecture, excellent I/O performance, and strong cloud-native compatibility, JuiceFS has become a stable choice for inference services in many enterprises. We hope this article can provide practical selection reference and architectural insights for more enterprises building or expanding their inference capabilities, assisting them in moving forward robustly in the increasingly complex landscape of AI applications.

If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and community on Slack.

Zelos Tech Manages Hundreds of Millions of Files for Autonomous Driving with JuiceFS

DASWU — Fri, 10 Oct 2025 07:00:48 +0000

Zelos Tech is a high-tech company focused on autonomous driving and unmanned delivery technologies, possessing independent R&D and large-scale deployment capabilities for autonomous driving systems and AI chips. Our core products have been widely deployed in 200+ cities across China, with a vehicle sales market share exceeding 90%. We maintain a leading position in the Chinese L4 urban autonomous delivery sector.

As our application expanded rapidly, our data volume grew from terabytes to petabytes. The original Ceph-based storage solution faced high costs and maintenance pressures, while performance bottlenecks gradually emerged in handling small files, metadata, high concurrency, and latency. This impacted simulation and training efficiency.

Furthermore, accelerated algorithm iteration and cross-regional application deployment increased the frequency of data transfer and resource scheduling needs across multi-cloud environments. Challenges included dispersed data, high migration costs, and complex scheduling. Limited community support and slow response times for some storage tools increased the operational complexity.

Facing these challenges, we needed to build a cloud-native storage architecture with high cost-effectiveness, strong scalability, and easy maintainability. After evaluating solutions like Alluxio and CephFS, we chose JuiceFS as our unified storage infrastructure. Currently, we’ve fully migrated critical data—including production, simulation, training, and inference—to JuiceFS, establishing an efficient, flexible, and unified storage foundation that comprehensively supports massive data processing requirements for various autonomous driving scenarios.

In this article, I’ll share the storage challenges we faced at Zelos Tech in autonomous driving R&D, why we chose JuiceFS over Alluxio and CephFS, and how we use it effectively in multi-cloud environments.

Autonomous driving training process and storage challenges

We’re currently dedicated to the R&D of L4 autonomous driving technology, primarily focusing on urban intelligent delivery logistics. The training process for autonomous driving models generates vast amounts of data and involves complex processing steps. The basic steps are as follows:

Data collection and upload: Data is collected from vehicles after calibration work and subsequently uploaded.
Algorithm processing: The algorithm team extracts relevant data for model training or algorithm improvement and then passes the results to the simulation stage for scoring.
Simulation verification and modification: If simulation fails, the process returns to the algorithm team for modification. If successful, it proceeds to the simulated environment verification stage.
Test vehicle validation: After passing simulated environment verification, tests are conducted on test vehicles. If the test fails, the process returns to the modification stage. If successful, the model is released via over-the-air (OTA) technology.

Throughout this process, data volume increases dramatically. Each vehicle uploads 10+ gigabytes of data daily. As the vehicles grew, the total data volume reached petabytes. Efficiently extracting and utilizing this massive data, especially during model training, places extremely high demands on the storage system's performance, scalability, and stability.

To meet the data requirements of the entire autonomous driving R&D pipeline, we needed to establish a storage platform with the following characteristics:

High-performance I/O: Capable of supporting high-concurrency reads and low-latency access for massive data during training and simulation.
Elastic scalability: Able to flexibly scale to petabyte-level or higher storage needs as the vehicle scale grows.
Multi-cloud compatibility: Supporting unified access across multiple clouds and self-built environments, ensuring data transfer and consistency.
Ease of operation: Providing simplified management and monitoring capabilities to reduce operational complexity and ensure long-term stability.
Cost-effectiveness: Controlling overall storage costs while guaranteeing performance and stability, maximizing resource utilization.

Storage selection: JuiceFS vs. Alluxio vs. CephFS

We tried various storage solutions, including Alluxio, JuiceFS, and CephFS.

Alluxio uses master nodes for metadata management, which has a steep learning curve. It requires separate deployment of master and worker clusters, increasing operational complexity. We encountered issues like hangs and I/O exceptions when using the community edition.

Regarding CephFS, its metadata is stored in its own Ceph Metadata Server (MDS), while data is stored in RADOS. In contrast, JuiceFS supports various backend storage options (like S3 and OSS) and can use external databases (like Redis and TiKV) for metadata management. This results in a more flexible architecture.

Furthermore, deploying and tuning CephFS is extremely complex, requiring deep involvement from specialized teams. When building our own Ceph cluster, we found that expanding OSDs and data rebalancing were time-consuming, small file write performance was poor, and tuning was difficult due to the complex architecture. Therefore, we abandoned this solution.

The table below compares key features of JuiceFS and CephFS. For a detailed comparison, see this document.

JuiceFS enables seamless integration of various object storage systems into local environments and supports multi-host concurrent reads/writes across platforms and regions. By separating data and metadata storage, it stores file data in object storage after splitting, while metadata is stored in databases like Redis, MySQL, PostgreSQL, TiKV, or SQLite. Users can flexibly choose based on actual scenarios and performance needs, greatly simplifying operational work.

In comparison, JuiceFS excels in multiple aspects—particularly in high-concurrency reads of small files, where its performance meets our requirements. Thus, we fully migrated to JuiceFS about a year ago and have widely adopted it in our multi-cloud architecture.

JuiceFS in multi-cloud environments

JuiceFS adopts a storage architecture that decouples metadata and data:

The metadata layer supports multiple database engines, such as Redis, MySQL, PostgreSQL, TiKV, and SQLite, allowing users to choose based on application scale and reliability needs.
The data layer uses mainstream object storage for seamless integration with diverse storage systems.

Currently, our system is deployed across multiple cloud platforms like AWS, with JuiceFS serving as the core storage component. In different environments, we flexibly pair backend storage with metadata engines:

In self-built data centers, we use MinIO as object storage with Redis for metadata management.
In public clouds, we combine OSS with Redis.

This architecture enhances flexibility and has remained stable over a year of operation, fully meeting application needs with excellent availability and user experience.

In Kubernetes clusters, we deploy JuiceFS using its CSI driver, ensuring strong compatibility with Kubernetes. We use the official Helm charts to create and manage JuiceFS storage volumes, configuring charts tailored to different applications to connect with backend Redis and OSS storage.

On the local node, we allocate NVMe SSDs for JuiceFS caching, mounted under the /data directory. As a cache layer, this significantly improves read performance. Once data is cached, subsequent reads for the same file are served directly from the local NVMe disk, achieving high read/write efficiency.

JuiceFS in the training platform

Our training platform architecture consists of multiple layers:

The infrastructure layer includes storage, network, compute resources, and several data service instances and hardware devices.
The upper layer is the containerization layer, built on a Kubernetes cluster, supporting various services. The platform provides deep GPU computing support, multiple development language environments, and mainstream deep learning frameworks.

On the deep learning platform, users can submit training jobs via Notebook or a VR interface. After submission, the system schedules and allocates resources through the Training Operator. For storage, we pre-provisioned storage resources based on Persistent Volume Claims (PVCs) and used JuiceFS to automatically associate and supply the underlying storage.

We integrated JuiceFS into the Kubeflow machine learning platform for model training tasks. When creating a training job in the Notebook environment, the system automatically associates with the StorageClass provided by the backend JuiceFS, enabling dynamic storage allocation and management. Simultaneously, a monitoring system deployed in the cluster observes storage performance in real time. Current monitoring shows read throughput about 200 MB/s, with low write requests—consistent with the read-heavy I/O pattern typical of our training and inference scenarios.

During performance tuning, we referenced JuiceFS community users’ sharing and compared Redis and TiKV as metadata engines. Test results showed that TiKV significantly outperformed Redis in read-intensive scenarios. Therefore, we plan to gradually migrate the metadata engine for some training clusters to TiKV to further improve read efficiency.

Currently, one of our storage buckets holds approximately 700 TB of data, comprising 600 million files. This includes a vast number of small files, typical for data organization in AI training tasks. In practice, JuiceFS has demonstrated excellent and stable performance under high-concurrency small file read/write operations, with no anomalies, fully meeting production demands.

In simulation scenarios, the data scale has reached petabyte levels, with storage buckets primarily containing large files. The underlying storage resources rely on a cloud object storage solution, mainly used for centralized storage of simulation data. This storage solution has also proven stable in practice, supporting the continuous operation of large-scale simulation tasks.

Data synchronization across multi-cloud environments with JuiceFS

To achieve data synchronization across multi-cloud environments, we deployed multiple dedicated lines between different cloud service providers and established cross-cloud network connectivity beforehand. For training data requiring consistency across clouds, we developed a synchronization tool that integrates the underlying juicefs sync command. It efficiently syncs the same dataset to multiple cloud environments. Although cross-cloud mounting is supported, we do not recommend this method due to its high dependence on network stability. The core challenge lies in network reliability; synchronization processes are susceptible to network fluctuations and must be used cautiously.

Summary

In key segments such as production, simulation, training, and inference, JuiceFS uses flexible metadata engine choices, diverse object storage backend options, and good compatibility with Kubernetes and Kubeflow to effectively support requirements for high-concurrency small file access, cross-cloud data flow, and performance scaling. Under large-scale data scenarios, JuiceFS operates stably. This significantly reduces operational complexity and overall costs, while meeting current application scale needs in terms of system scalability.

Looking forward, with the gradual adoption of metadata engines like TiKV and continuous optimization of cross-cloud synchronization mechanisms, JuiceFS' overall performance and adaptability still have room for improvement. It will continue to provide sustained support for Zelos' massive data processing needs in autonomous driving research and development.

If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and community on Slack.

MLPerf Storage v2.0: JuiceFS Leads in Bandwidth Utilization and Scalability for AI Training

DASWU — Fri, 26 Sep 2025 06:48:18 +0000

On August 5, the global authoritative AI engineering consortium MLCommons released the latest MLPerf® Storage v2.0 benchmark results. This evaluation attracted numerous vendors, including those from categories such as cloud, shared file, fabric-attached block, and direct-attached block storage.

Due to differences in hardware configurations, node scales, and application scenarios among vendors, direct cross-vendor comparisons have limitations. Therefore, this article will focus on the shared file system category, analyzing their performance under the same testing standards.

JuiceFS is a high-performance distributed file system supporting both cloud and on-premises deployments. Across multiple AI training workloads, JuiceFS achieved excellent results, particularly leading in bandwidth utilization and scalability. In this article, we’ll analyze the specific test results and further introduce the key features underpinning this performance.

MLPerf Storage v2.0 and its workloads

MLPerf is a universal AI benchmark suite launched by MLCommons. MLPerf Storage simulates real AI workload access to storage systems through multiple clients. It replicates storage loads in large-scale distributed training clusters to comprehensively evaluate the practical performance of storage systems in AI training tasks.

The latest 2.0 version includes three types of training workloads, covering the most representative I/O patterns in deep learning training.

3D U-Net (medical image segmentation): This workload involves sequential and concurrent reads of large-volume 3D medical images. Each sample averages about 146 MB and is stored as an independent file. This task primarily tests the storage system's throughput in large-file sequential read scenarios and its ability to maintain stable responsiveness under concurrent multi-node access.
ResNet-50 (image classification): This workload is characterized by highly concurrent random reads of small samples. Each sample averages only 150 KB, with data packaged and stored in large files using TFRecord format. This data organization leads to significant random I/O and frequent metadata access during training. It places extremely high demands on the storage system's IOPS. It’s a crucial test for measuring concurrent performance in small-file scenarios.
CosmoFlow (cosmology prediction): This workload emphasizes concurrent small-file access and bandwidth scalability across nodes. Each sample averages 2 MB, typically stored as individual files in TFRecord format. As it involves distributed reading of massive numbers of small files, the system requires sufficient aggregate throughput and must demonstrate stability in metadata handling and tail latency control. Otherwise, as nodes increase, latency fluctuations will significantly amplify and slow down overall training speed.

In addition, the 2.0 version introduces a new checkpointing workload to simulate checkpoint saving and recovery in large model training. This workload primarily involves concurrent sequential writes of large files. In the JuiceFS architecture, checkpoint data is written through JuiceFS to object storage. Its performance bottleneck depends on the bandwidth limit of the object storage serving as the persistent data layer.

Performance comparison: product categories, elastic scalability, and resource utilization

Numerous vendors participated in this MLPerf Storage v2.0 test, involving various types like block storage and shared file systems. However, due to significant architectural and application scenario differences, coupled with different hardware configurations and node scales used by vendors, a direct horizontal comparison has limited value.

In this article, we’ll focus on results within the shared file system category. This category can be further subdivided:

Ethernet-based systems: This group includes Alluxio, JuiceFS, and Oracle. These cloud systems rely on Ethernet environments to provide distributed storage capabilities. Some vendors, like Nutanix, used RoCE-based Ethernet solutions, typically with higher-bandwidth NICs per instance.
InfiniBand-based storage solutions: Vendors like DDN, Hewlett Packard, and Ubix offer complete storage appliances based on InfiniBand networks. These systems feature very high hardware specifications and overall cost, delivering extremely high bandwidth and performance.

Before we explain the results, let's clarify the evaluation criteria.

MLPerf Storage documentation requires submitted results to meet a GPU utilization threshold while maximizing the number of GPUs (scale). The thresholds are 90% for 3D U-Net and ResNet-50, and 70% for CosmoFlow. Given these thresholds are met, the key metric that truly differentiates performance is the maximum number of GPUs a storage system can support. This scale is essentially determined by the system's maximum aggregate bandwidth. A system supporting more GPUs indicates stronger scalability and stability in large-scale training scenarios, especially crucial in workloads like CosmoFlow which involve vast numbers of small files and are highly latency-sensitive.

Comparison should consider resource utilization. For software vendors, the key is whether the storage software can fully leverage the potential of the underlying hardware. As the network bandwidth is often the bottleneck for storage systems, we use NIC bandwidth utilization as a reference metric: higher utilization indicates greater software efficiency and implies higher performance and cost-effectiveness for a given hardware configuration.

JuiceFS test results

3D U-Net workload: JuiceFS achieved a data read bandwidth of up to 108 GiB/s, supporting a training scale of 40 H100 GPUs across 10 nodes. The network bandwidth utilization reached 86.6%, and the GPU utilization was 92.7%.
CosmoFlow workload: JuiceFS supported a training scale of 100 H100 GPUs across 10 nodes, with a GPU utilization of 75%. This workload places extreme demands on storage latency stability and has lower network bandwidth requirements. The performance bottleneck is not bandwidth. Handling massive concurrent small-file access means that I/O latency and stability determine overall scalability and limit GPU utilization.
ResNet-50 workload: JuiceFS achieved a data read bandwidth of 90 GiB/s, with a network bandwidth utilization of 72% and an overall GPU utilization of 95%.

Note: The maximum number of stable elastic instances available in a single zone on the chosen GCP instance type for this test was limited. Therefore, our submitted maximum scale was 10 nodes. This does not represent the maximum capacity of the JuiceFS system, which can provide higher aggregate bandwidth and support larger training scales with more nodes.

Background: GPU scale limited by storage network bandwidth capacity

JuiceFS has been participating in MLPerf tests for three years. In the first year, based on the 0.5 version, we conducted full tests using V100 GPUs for simulation.

Because the per-GPU bandwidth requirement of V100 is low, the GPU utilization remained near 99% for a long interval. It slowly decreased to about 98% as the training GPU scale increased. Only when the total bandwidth approached the NIC bandwidth limit, the inflection point occurred. After that, utilization dropped rapidly with additional GPUs. For detailed analysis, see 98% GPU Utilization Achieved in 1k GPU-Scale AI Training Using Distributed Cache. According to the data in the figure below, at 90% GPU utilization, about 110 V100 GPUs could be supported. This scale limit is primarily constrained by the storage network's bandwidth capacity.

3D U-Net: JuiceFS leads peer systems with highest bandwidth and utilization

The 3D U-Net training workload involves large-file sequential reads, demanding high read bandwidth from the storage system.

The figure below clearly shows that among all storage systems based on conventional Ethernet, JuiceFS performed best: achieving 108 GiB/s data read bandwidth at a scale of 10 nodes (40 H100 GPUs), with a network bandwidth utilization of 86.6%, both the highest among comparable systems. This indicates JuiceFS not only provides higher total bandwidth but also utilizes network and hardware resources more efficiently, offering significant cost-performance advantages.

The following section shows results from participants using InfiniBand networks and RDMA over Converged Ethernet (RoCE). These vendors' hardware specifications are generally very high, with minimum total network bandwidth about 400 GiB/s, and the highest exceeding 1,500 GiB/s. Consequently, they can provide extremely high aggregate bandwidth for training workloads. It’s worth noting that JuiceFS can achieve similar bandwidth levels after elastically scaling the number of distributed cache nodes. In recent tests, an aggregate read bandwidth of 1.2 TB/s was achieved based on a distributed cache cluster of 100 GCP 100 Gbps nodes.

From a utilization perspective, as per-NIC bandwidth increases, achieving high bandwidth utilization becomes increasingly challenging. Achieving utilization close to 80% is very difficult with 400 Gb/s or higher NIC configurations.

CosmoFlow: JuiceFS supports hundred-GPU scale, leading in scalability

The CosmoFlow training workload requires reading vast numbers of small files. This places extreme demands on the storage system's metadata performance and read latency. The test requires GPU utilization to be above 70%. Due to the high latency sensitivity, as the number of H100s increases, the read latency variance in multi-node distributed training increases significantly, causing GPU utilization to drop rapidly and making horizontal scaling difficult. Compared to 3D U-Net, fewer total results were submitted for CosmoFlow. This reflects the greater optimization challenge this workload presents.

The chart's x-axis represents the total number of GPUs supported, that is, the number of H100 GPUs. JuiceFS continues to lead among comparable systems. Running simultaneously through 10 clients, it successfully supported the CosmoFlow training task with 100 H100 GPUs, maintaining GPU utilization above the required threshold.

Systems based on InfiniBand networks performed particularly well in this workload. This benefits from the end-to-end extremely low and highly stable latency systematically provided by InfiniBand networks. Despite their higher cost, the performance advantage of InfiniBand networks is significant for latency-sensitive tasks.

ResNet-50: JuiceFS supports the most H100s, achieving highest bandwidth utilization

The ResNet-50 training workload involves highly concurrent random reads within large files, demanding extremely high IOPS from the storage system and requiring GPU utilization to stay above 90%.

In this test, JuiceFS supported the largest number of 500 H100 GPUs among comparable systems and achieved the highest network bandwidth utilization (72%) among all Ethernet-based solutions, far exceeding the approximately 40% level of other vendors. Simultaneously, GPU utilization remained at 95%. This result demonstrates that under high-concurrency random I/O scenarios, JuiceFS can fully leverage its software optimizations to efficiently utilize hardware resources, exhibiting leading performance and efficiency.

JuiceFS system configuration in the MLPerf test

The JuiceFS system topology in this test consisted of three main layers:

The client layer: The top layer consisted of 10 client nodes, all running on the same GCP instance type, with the same bandwidth configurations per node. This ensures client-side balance.
The cache cluster layer: The middle layer was the cache cluster, comprising 10 cache nodes. These nodes used cloud disks combined with local memory as cache to accelerate data access.
The cold data storage layer: Data was ultimately stored in Google Cloud Storage (GCS).

Before training started, cold data was warmed up from GCS to the cache cluster. During training, clients read data from the cache cluster. This approach avoids accessing high-latency object storage during training, providing stable high bandwidth and low-latency access to meet the sustained data demands of large-scale AI training.

The MLPerf Storage v2.0 rules allowed for an additional "warm-up test" round. Thus, even systems without an active warm-up feature could load data into cache before the official test, ensuring fairness. In addition, all participants could benefit from performance gains offered by page cache. This is critical for latency-sensitive workloads like CosmoFlow.

In the current configuration, 10 clients and the cache cluster could support a training scale of about 40 H100 GPUs. If the training scale increases further, for example, doubling the number of GPUs, simply scaling the cache cluster nodes proportionally can match the new bandwidth requirements. This architecture demonstrates high flexibility and elasticity, adaptable to larger-scale training tasks.

The strong performance of JuiceFS in this test relies firstly on its high-performance metadata engine. This engine delivers very high IOPS and extremely low latency, with integrated metadata caching on the client, significantly reducing access latency. This capability is particularly critical for the CosmoFlow workload, which involves massive concurrent small-file access and is highly latency-sensitive. Stable low latency ensures high GPU utilization and overall scalability.

Another key advantage is the distributed cache. A test shows that the JuiceFS cache cluster can provide up to 1.2 TB/s of aggregate bandwidth (based on 100 GCP 100 Gbps nodes, scale limit constrained by instance availability in the selected GCP zone) and reduce access latency to sub-millisecond levels. The cache cluster supports elastic scaling, providing extremely high performance in bandwidth and allowing expansion of IOPS and bandwidth according to task needs. In the ResNet-50 workload, JuiceFS' high IOPS performance was decisive. JuiceFS stood out in the Ethernet-based comparison relying on its IOPS advantage, achieving 72% bandwidth utilization and maintaining 95% GPU utilization.

Summary

In this article, we analyzed the results of the MLPerf Storage v2.0 test. Before evaluating a storage product's GPU utilization, users must understand the product type, including its architecture, hardware resources, and cost. Assuming GPU utilization thresholds are met, the key difference among storage systems is the maximum number of GPUs they can support. A system supporting more GPUs indicates better scalability and stability in large-scale training scenarios. Furthermore, resource utilization must be considered—whether the storage software can fully leverage the potential of the underlying hardware.

JuiceFS demonstrated stable low latency and high resource utilization across different AI training workloads in these tests. As a fully user-space, cloud-native distributed file system, it utilizes a distributed cache to achieve high throughput and low latency, while leveraging object storage for cost advantages. It presents a viable option for large-scale AI training that does not rely on expensive proprietary hardware.

Achieving TB-Level Aggregate Bandwidth: How JuiceFS Optimized Distributed Cache Network

DASWU — Mon, 22 Sep 2025 09:54:50 +0000

With the explosive growth of data volume and model scale, scenarios where multiple clients frequently access the same data have become increasingly common. Distributed caching aggregates the local caches of multiple nodes to form a large-capacity cache pool. This improves cache hit rates, enhances read bandwidth and IOPS, reduces read latency, and meets high-performance demands.

However, data exchange between nodes heavily relies on network performance. Insufficient bandwidth can limit data transfer speeds and increase latency; high network latency can affect cache responsiveness and reduce system efficiency; meanwhile, the CPU resources consumed by network data processing can also become a bottleneck, constraining overall performance.

To address these issues, the recently released JuiceFS Enterprise Edition 5.2 introduces multiple optimizations for network transmission between cache nodes. This effectively resolves some performance-intensive aspects and improves network interface card (NIC) bandwidth utilization. After optimization, client CPU overhead was reduced by 50%+, and cache node CPU overhead dropped to one-third of pre-optimization levels. The aggregate read bandwidth reached 1.2 TB/s, nearly saturating the TCP/IP network bandwidth (based on a distributed cache cluster of 100 GCP 100 Gbps nodes).

As the most widely supported network standard, the TCP/IP protocol can be used almost everywhere out-of-the-box. With built-in congestion control and retransmission mechanisms, it achieves sufficiently high performance and stability in complex networks. This makes it the preferred choice for distributed caching. Currently, JuiceFS can fully saturate 100 Gb NIC bandwidth under TCP/IP. In the future, we’ll also introduce RDMA to further unleash the potential of 200 Gb and 400 Gb NICs.

In this article, we’ll share the specific implementations of these TCP/IP network optimizations.

Golang network performance optimization

In large-scale environments, such as when thousands of clients access data from over 100 distributed cache nodes, the number of network connections to cache nodes increases significantly. This leads to substantial Golang scheduling overhead and reduced network bandwidth utilization.

We addressed two key points—connection reuse and epoll event triggering—by introducing multiplexing mechanisms and event trigger threshold optimizations. This effectively alleviated system pressure and improved data processing performance.

Multiplexing

If a connection can only handle one request at a time, the system’s overall concurrency capability will be limited by the number of connections itself. To improve concurrency, a large number of connections must be established. Therefore, we introduced connection multiplexing. It allows multiple request packets to be sent simultaneously over the same connection to the peer end. Through multiplexing, the throughput capacity of a single connection can be effectively improved. This reduces the need for numerous connections, thereby enhancing concurrent performance while lowering resource consumption and network burden.

Since single TCP connections have performance bottlenecks, we further introduced the ability to dynamically adjust the number of connections based on real-time traffic within the multiplexing architecture to achieve the best balance between performance and resources. This improved overall throughput and network efficiency. The system automatically increases or decreases the number of connections based on current network traffic:

When user request volume is high, total traffic continues to rise, and existing connections can no longer meet bandwidth requirements, the system automatically increases the number of connections to improve bandwidth utilization efficiency.
When requests become inactive and total bandwidth decreases, the system automatically reduces the number of connections to avoid resource waste and packet fragmentation issues.

Another advantage of multiplexing is support for small packet merging. Through efficient connection management, multiple small packets’ data streams can be transmitted together over the same physical connection. This reduces network transmissions along with the overhead from system calls and kernel-user space switching.

Specifically, on the sender side, the sending thread continuously retrieves multiple requests from the sending channel until the channel is empty or the cumulative data to be sent reaches 4 KiB before sending them together. This merges multiple small packets into a larger data block, reducing system calls and network transmissions. On the receiver side, data from the NIC is read in bulk into a ring buffer, from which upper-layer services extract segments one by one and parse out complete application-layer data packets.

Receive watermark setting (`SO_RCVLOWAT`)

In Golang’s network framework, epoll’s edge-triggered mode is used by default to handle sockets. An epoll event is generated when the socket state changes (for example, from no data to having data).

To reduce the additional overhead caused by frequent events triggered by small amounts of data, we set SO_RCVLOWAT (socket receive low watermark) to control the kernel to trigger epoll events only when the data volume in the socket receive buffer reaches a specified number of bytes. This reduces the number of system calls triggered by frequent events and lowers network I/O overhead.

For example:

conn, _ := net.Dial("tcp", "example.com:80")
syscall.SetsockoptInt(conn.(net.TCPConn).File().Fd(), syscall.SOL_SOCKET, syscall.SO_RCVLOWAT, 512*1024)

After testing, in high-concurrency connection scenarios, epoll events dropped by 90% to about 40,000 per second. Cache performance became more stable, with CPU overhead maintained at about 1 core per GB bandwidth.

Zero-copy optimization: reducing CPU and memory consumption

In Linux network communication, zero-copy is a technique that reduces or eliminates unnecessary data copying between kernel space and user space to lower CPU and memory consumption, thereby improving data transfer efficiency. It’s particularly suitable for large-scale data transfer scenarios. Classic products such as nginx, Kafka, and haproxy use zero-copy technology to optimize performance.

Common zero-copy technologies include the mmap system call, as well as mechanisms such as sendfile, splice, tee, vmsplice, and MSG_ZEROCOPY.

sendfile is a system call provided by Linux to read file data directly from disk into the kernel buffer and transmit it to the socket buffer via DMA without passing through user space.
splice allows direct data movement in kernel space, supporting data transfer between any two file descriptors (with at least one end being a pipe). Using a pipe as an intermediary, data is transferred from the input descriptor (e.g., a file) to the output descriptor (e.g., a socket), while the kernel only manipulates page pointers, avoiding actual data copying.

Compared to sendfile, splice is more flexible, supporting non-file transmission scenarios (such as forwarding between sockets), and is more suitable for high-concurrency network environments, such as proxy servers. sendfile can reduce the number of system calls but blocks the current thread, affecting concurrent performance. We used splice and sendfile in different scenarios to optimize data transmission flows for the best results.

Taking splice as an example, when a client requests file data, it sends a request to the cache node via distributed caching. We used splice zero-copy technology to optimize the data flow on the sender side. The specific data transmission path is as follows:

When cache data needs to be sent to the client, the process uses the splice interface to read data directly from the file into the kernel buffer (page cache). The data is then transmitted via splice to a pre-created pipe, and the data in the pipe is further transmitted via splice to the socket buffer. Since the data remains in kernel space throughout the process, only two DMA copies occur without any CPU copying, achieving zero-copy cache service.

Although there is also optimization potential on the receiver side, due to the high hardware and runtime environment requirements for zero-copy implementation and its lack of universality, we haven’t introduced related solutions in the current architecture.

Optimizing the CRC check process

In previous versions, reading a data block from distributed caching required two cyclic redundancy checks (CRCs):

Disk loading phase: When loading data from disk into memory, to check if data on the disk has experienced bit rot.
Network transmission phase: The sender recalculates the CRC and writes it into the packet header. The receiver recalculates the CRC after receiving the data and compares it with the value in the packet header to ensure the data has not been tampered with or corrupted during transmission.

To reduce unnecessary CPU overhead, the new version optimizes the CRC check process during network transmission. The sender no longer recalculates the CRC but instead uses the CRC value saved on the disk, merging it into a total CRC written into the packet header. The receiver performs only one CRC after receiving the data.

To support random reads, cached data blocks need to have CRC values calculated segmentally—every 32 KB. However, the network transmission process requires the CRC value for all transmitted data. Therefore, we efficiently merge segmented CRC values into an overall CRC value using a lookup table method, the overhead of which is almost negligible.
This approach reduces one CRC calculation while ensuring data consistency and integrity, effectively lowering the sender’s CPU consumption and improving overall performance.

Summary

In scenarios such as large-scale model training and inference, data scale is growing at an unprecedented rate. As a critical component connecting computing and storage, distributed caching distributes hot data across multiple nodes, significantly improving system access performance and scalability. This effectively reduces the pressure on backend storage, and enhances the stability and efficiency of the overall system.

However, when facing continuously growing data and client access pressure, building high-performance large-scale distributed caching still faces many challenges. Therefore, JuiceFS’ distributed caching system, implemented based on Golang, has undergone in-depth optimizations around Golang’s I/O mechanisms in practice and introduced key technologies such as zero-copy. This significantly reduces CPU overhead, improves network bandwidth utilization, and makes the system more stable and efficient under high-load scenarios.

We hope that some of the practical experiences in this article can provide references for developers facing similar issues.
If you have any questions or feedback for this article, feel free to join JuiceFS discussions on GitHub and community on Slack.