HeyGen is an AI video generation and digital avatar platform with more than 30 million users across 196 countries. It’s also used by 85% of Fortune 100 companies. Our development, training, and inference workloads run across more than 10 GPU clusters worldwide. As our platform has grown, sharing data efficiently across these clusters has become a key challenge for our storage infrastructure.
In this article, we’ll explore how our team uses JuiceFS Enterprise Edition to address this challenge. A unified file namespace and multi-level cache warmup help distribute and load models and dependencies for inference workloads. For training, our team combines RDMA's high-bandwidth data transfer with batch-based prefetching to reduce GPU idle time.
The architecture has been running in production for more than a year. It supports approximately 6 PB of data, around 1 billion inodes, and 1,000–2,000 client nodes. Most notably, cold reads of 100 GB models have dropped from 20–30 minutes to just 60–120 seconds.
Challenges of data access across 10+ global GPU clusters
Our compute resources span multiple data centers and GPU providers. Workloads are dynamically scheduled across clusters based on available compute capacity, so the required datasets, models, and dependencies may not reside in the same region as the compute resources.
If each cluster maintains its own data copy, the team must replicate or synchronize data across clusters. This process takes time and may incur additional network egress costs. Different clusters may also end up with different data versions, increasing the effort required to maintain synchronization and consistency. When compute resources change, data must move accordingly, adding further complexity to job startup and operations.
Therefore, we needed a file system that could provide a unified namespace across clouds and regions. This would allow different clusters to access data through consistent paths without requiring users to manage data replication and transfers manually.
Storage architecture: unified namespace, cross-cluster sharing, and POSIX compatibility
During our evaluation, we considered cloud and on-premises NAS solutions, as well as object storage combined with data replication.
Cloud or on-premises NAS can provide a file system interface, but these solutions are typically designed for deployment at a single site, and their throughput is constrained by the underlying storage hardware. Object storage combined with replication can deliver high bandwidth, but cross-region access latency, data replication, and synchronization still require additional work.
Another consideration was our development team's reliance on POSIX interfaces for data access. Switching to object storage APIs would require changes to existing code and increase migration costs.
After evaluating these options, we chose JuiceFS Enterprise Edition. By using object storage as the data foundation, we avoid having to maintain storage nodes itself, reducing operational overhead. JuiceFS also provides a unified namespace, POSIX interfaces, and cluster-level caching to meet the requirements for cross-cluster data sharing and performance.
The architecture consists of four layers:
- Access layer: Workloads such as R&D, training, and inference access files through POSIX mounts.
- Cache layer: Client-side local caches and distributed cluster caches handle data reads.
- Metadata layer: JuiceFS Enterprise Edition manages file system metadata.
- Data layer: Object storage across regions and clouds stores file data.
Cross-cluster sharing: from separate copies to unified access
Under the previous approach, different clusters could use different versions of the same dataset. For example, one cluster might use data v3 while another still uses data v2. We had to handle data replication, version updates, and synchronization separately.
With JuiceFS, we connect multiple clusters to the same global file system. All clusters access data through a unified mount point and use their own cache groups to accelerate reads. This reduces redundant data replication and version synchronization.
Preserving POSIX access
Beyond cross-cluster sharing, we wanted to minimize the impact of storage changes on application code. Our development and data processing workloads already relied primarily on POSIX file paths rather than direct object storage APIs. Switching to object storage access would require additional code migration and adaptation.
JuiceFS preserves the file system interface, so applications can continue accessing files through regular paths, for example, open('/jfs/data/...').
Developers can keep using their existing file access patterns, making it easier to maintain consistency between local development and production environments. POSIX compatibility reduces application migration costs and shields application code from changes to the underlying storage for us.
Multi-level caching architecture
A unified file system allows different clusters to access the same data. However, if workloads still read models and training datasets directly from object storage across regions and clouds, load times remain subject to network distance, available bandwidth, and concurrent read pressure.
This is especially problematic when many jobs start simultaneously and repeatedly access the same files. Data loading can delay job startup and even leave GPUs idle while they wait for data.
To address this, we use a multi-level caching architecture that keeps frequently accessed data closer to compute resources. This reduces the time spent repeatedly fetching data from remote storage.
The data access path has four levels:
- Memory: Per-node read/write buffers and metadata caches.
- Local cache: Clients cache read data on local NVMe disks.
- Distributed cache: Each cluster runs its own cache group, allowing workloads within the cluster to share cached data.
- Global storage: Object storage across regions and clouds holds the complete dataset and serves as the shared data foundation.
Client memory includes read/write buffers and a metadata cache. The read/write buffers are allocated to the client process, and their size is controlled by --buffer-size. Both read and write operations pass through these buffers, so this setting affects client prefetching and read/write concurrency.
The metadata cache reduces repeated requests to the metadata service, such as directory listings (ls) and metadata lookups. This improves metadata access performance. It also supports active invalidation to maintain strong consistency.
At the disk caching layer, local caches serve individual clients. A cache group, in contrast, combines the available NVMe capacity across multiple nodes in a cluster into a distributed cache that GPU workloads can share. We deploy a separate cache group in each cluster, while all clusters continue to access the complete dataset through the same global JuiceFS file system.
Making efficient use of limited cache capacity requires prioritizing frequently accessed data. JuiceFS cache groups support cache priorities at the directory or file level. When cache space runs low, data with higher priorities is evicted later.
In addition to controlling eviction order through cache priorities, we use warmup to preload data that upcoming jobs are expected to need into cache groups. This reduces the initial read latency after jobs start.
The following sections show how we apply this multi-level caching architecture to two workloads: model and Python dependency distribution, and training data loading.
Use case 1: Distributing models and Python dependencies
Before training or inference jobs can read their datasets, they typically need to load models and install Python dependencies. When many job instances start simultaneously, fetching these files independently from upstream storage or Python package repositories can make startup times dependent on network conditions and upstream service performance.
We wanted to preserve its existing development workflow even as jobs scaled from a single node to thousands of nodes. Data processing requirements change rapidly, sometimes requiring dozens of versions to be synchronized in a single day. Rebuilding and distributing OCI images for every change would complicate the development workflow and increase the operational burden on users.
For this reason, we chose to keep its existing Python dependency installation workflow rather than package every dependency change into a new OCI image.
This shifted the infrastructure challenge: how could jobs across different clusters quickly access the same models and Python packages without changing existing development workflows?
Model caching and warmup
Model files are often large. Jobs need to load them at startup, and multiple jobs may reuse the same files. To reduce the delay caused by initial reads, we assign model files a high cache priority: /demo/models: cache priority 3.
We also periodically warm up high-priority models in cache groups so that jobs can read the data from prepared caches as soon as they start. On the client, page cache and prefetch further optimize the model read path.
The model warmup configuration is as follows:
juicefs mount models /mnt/jfs-warmup
--cache-group=$CACHE_GROUP
--no-sharing
--group-network=$NIC
--cache-size=0
--buffer-size=8192 -d
juicefs warmup
/mnt/jfs-warmup/models
-c 64
--cache-priority=3
The warmup job runs hourly. The --cache-size=0 setting allows the warmup process to populate the cache group without using its own local disk cache.
Python dependency distribution
Python dependencies are also reused across jobs. However, cache capacity is limited, so we must allocate it between models and dependencies.
Therefore, Python dependencies receive a lower cache priority than models: /demo/deps: cache priority 2.
Other data is assigned priority 0. When cache space runs low, the system retains models first, followed by commonly used Python dependencies. Other data is evicted earlier.
To avoid downloading dependencies separately every time a job starts, we store Python packages under a JuiceFS path and warm up the required packages when jobs are submitted. During installation, jobs point their package index to this path, for example: --index-url /jfs/pypi/simple.
Jobs in different clusters can access the same dependencies through a unified JuiceFS path. On a cache miss, a node fetches the file from upstream storage. Then, other workloads can reuse the data already cached within the cluster. This reduces repeated upstream downloads and eliminates the need for each cluster to maintain its own external Python package mirror.
We tested model and Python dependency loading under workloads with more than 1,000 concurrent jobs. The results were:
- 100 GB model cold reads: Reduced from approximately 20–30 minutes to 60–120 seconds.
- uv dependency installation: Reduced from approximately 5 minutes to around 30 seconds.
Actual performance varies with the workload and runtime environment. For us, the key benefit is that model and dependency distribution no longer depends entirely on the external network conditions of each cluster. Frequently used data can be prepared in caches close to compute resources, reducing delays during job startup.
Use case 2: Training data loading and reducing GPU idle time
Unlike model and dependency distribution, which focuses on job startup, training workloads require a continuous supply of data throughout execution.
Training data may reside in one region while training jobs run in another cluster with large GPU capacity. If data loading cannot keep up with the training workload, GPUs may sit idle while waiting for data.
To address this, we use NVMe storage on training nodes to build larger cache groups. It combines RDMA with batch-based warmup to reduce data read overhead and GPU idle time.
Building large-scale caches with NVMe on training nodes
Training nodes typically have substantial local NVMe capacity. We combine these resources into cluster-level cache groups, allowing training jobs to read data from the cluster cache whenever possible.
To improve data transfer between clients and cache group nodes, we use RDMA. This capability is available in JuiceFS Enterprise Edition 5.3 and later. Both clients and cache nodes must be configured with the corresponding rdma-network setting.
In our production environment, combining cache groups with RDMA allows cache reads to saturate the available network bandwidth.
Faster cache reads are only part of the solution. We also wanted to prevent data loading from blocking training, so it further optimized the workflow by overlapping warmup of the next batch with training on the current batch.
Preloading the next batch of training data
We process training data in batches. While a training job works on batch N, a warmup job uses otherwise idle network bandwidth to preload batch N+1 into the cache group:
- The training job reads and processes batch N. At the same time, the warmup job preloads batch N+1.
- By the time training on batch N finishes, batch N+1 is already available in the cache.
- Training continues with batch N+1 while the warmup job prepares subsequent batches.
This allows data loading to run in parallel with training on the current batch, reducing wait time and GPU idle time caused by data I/O. Data from completed training jobs is automatically evicted as new data is loaded, making more efficient use of limited cache capacity.
Conclusion
By combining a unified namespace, POSIX compatibility, and multi-level caching, we’ve addressed data sharing, application migration, and workload startup challenges across multi-cloud, multi-cluster environments.
The architecture delivers three main benefits:
- Less waiting during startup and training: Models and Python dependencies can be warmed up in advance, while the next batch of training data can be preloaded during current-batch processing. This reduces cold-read latency and GPU idle time.
- Simpler data management and operations: Multiple clusters access data through a unified namespace, reducing data replication, version synchronization, and multi-cluster operational overhead.
- More flexible storage costs: Object storage provides the underlying data capacity, while multi-level caching handles frequently accessed data. This balances storage costs and access performance.
For AI infrastructure spanning multiple clouds and GPU clusters, combining a unified namespace, POSIX compatibility, and multi-level caching can help balance cross-cluster data sharing, application compatibility, and data access performance.
We welcome others working in storage and AI infrastructure to share their experiences with cross-cluster data sharing, model distribution, and training data loading.
If you have any feedback on this article or ideas to share, we invite you to participate in the discussions on GitHub and join the JuiceFS community on Discord.





Top comments (0)